When AI Servers Demand Higher Cooling Technology! Why the Trend Is Shifting from “Air Cooling” to “Liquid Cooling”

 

NVIDIA CEO Jensen Huang made a special appearance at COMPUTEX 2023 to support Supermicro Chairman Charles Liang's speech. Fanner Chairman Lin Yushen pointed out that most of the server products displayed on stage featured Fanner's water-cooling modules. Fanner, having invested in water-cooling modules for many years, is well-positioned to capitalize on this AI wave. However, as AI servers demand increasingly higher cooling standards, why is the trend shifting from "air cooling" to "liquid cooling"?

 

 

I From High-Speed Operation to Liquid Cooling

 

In terms of cooling technology, Lin Yushen noted that current cooling modules mainly use hybrid cooling technology that incorporates heat pipes. These heat pipe cooling modules combine components like fans, heat sinks, and heat pipes to create a balanced thermal environment for internal electronic components, thereby enhancing the stability of electronic devices. However, as downstream terminal electronic products become more multifunctional and compact, cooling module manufacturers have shifted to designing cooling solutions centered on vapor chambers and heat pipes.

 

Currently, cooling modules are divided into two types: "air cooling" and "liquid cooling." Air cooling uses air as a medium, with materials like thermal interface materials, vapor chambers (VCs), or heat pipes conducting heat, which is then dissipated through heat sinks or fans via air convection. In contrast, liquid cooling dissipates heat through liquid convection, including immersion cooling, which cools chips more efficiently. However, as chips generate more heat and become smaller, and as thermal design power (TDP) increases, air cooling is gradually becoming insufficient.

 

▲ A comparison between air cooling and liquid cooling technologies

 

With the rise of ChatGPT, generative AI has driven an increase in server shipments, which in turn has led to upgraded specifications for cooling modules, pushing them towards liquid cooling solutions to meet the stringent requirements for server cooling and stability. Lin Yushen highlighted that Fanner started with air cooling technology and, as early as ten years ago, began acquiring liquid cooling technology through IBM technology transfer. They provided water-cooling backdoors that allowed clients to add water-cooling to cabinets without modifying existing data center infrastructure.

 

 

II By 2025, a New Era of Concurrent Air and Liquid Cooling

 

Driven by the development of semiconductor technology related to AI applications, the introduction of GPT-3 in ChatGPT has grown AI algorithm parameters to 175 billion, necessitating a hundredfold increase in GPU computing power. The industry is primarily using single-phase immersion cooling technology within liquid cooling to solve the heat dissipation challenges of high-density servers or components. However, this method has a 600W limit, while the cooling needs for ChatGPT or more advanced servers exceed 700W.

 

▲ Illustration of single-phase immersion cooling technology

 

With the development of IoT, edge computing, and 5G applications, data AI is propelling global computing power into a high-growth phase. The next generation of cooling module designs will follow two main directions: upgrading existing cooling modules with 3D vapor chambers (3DVC) or introducing liquid cooling systems that use liquid as a thermal convection medium to improve cooling efficiency. As a result, the number of liquid cooling tests increased significantly in 2023. However, 3DVC is ultimately a transitional solution, and it is expected that by 2024-2025, the era of concurrent air and liquid cooling will begin.

 

According to TrendForce, in 2022, AI servers equipped with GPGPUs (General Purpose GPUs) accounted for about 1% of total shipments. However, driven by the application of ChatGPT, AI server shipments are expected to grow by 38.4% in 2023, with the compound annual growth rate of AI server shipments between 2022 and 2026 reaching 29%.

 

 

III "Liquid Cooling" Will Become the Mainstream for AI Chips

 

As the TDP of the new generation of servers approaches the limits of air cooling, leading tech companies are starting to test liquid cooling or increase cooling space. For example, Intel's Eagle Stream and AMD's Genoa TDP of 350-400W have reached the limits of air cooling, making liquid cooling the mainstream solution for AI chips. NVIDIA's H100 has a TDP of 700W, and air cooling using 3DVC generally requires more than 4U of space, which is not suitable for high-density deployment architectures.

 

▲ NVIDIA's H100

 

Given that cooling systems account for approximately 33% of the total energy consumption in data centers, reducing total power consumption and improving Power Usage Effectiveness (PUE) involves optimizing cooling systems, IT equipment, and using renewable energy. Since water's thermal capacity is four times that of air, implementing liquid cooling systems requires only 1U of space for the liquid cooling plate. According to NVIDIA's tests, to achieve the same computing power, liquid cooling can reduce the number of cabinets required by 66%, energy consumption by 28%, and PUE from 1.6 to 1.15, while also improving computing performance.

 

 

IV Supermicro's Use of Fanner Cooling Modules Is Crucial

 

Liquid cooling is further divided into "water cooling" and "oil cooling," with water cooling currently being the most widely used. Lin Yushen noted that almost all AI servers now use water cooling solutions. For example, NVIDIA's GH100, with a TDP exceeding 700W, must use water cooling. Although water cooling currently accounts for a small proportion of Fanner's revenue, the average selling price (ASP) of AI servers is ten times higher than that of traditional servers, which will help transform Fanner's product structure in the second half of the year. It is estimated that AI servers could account for 5-10% of their business in 2023.

 

Lin Yushen emphasized that the key reason Supermicro uses water cooling modules from Fanner for servers equipped with NVIDIA GH100 GPUs is that Fanner has been working on liquid cooling solutions for over a decade. While the challenges of implementing water cooling mainly involve costs and leak prevention, Fanner's over ten years of research have gradually overcome these leakage issues. Fanner's long-term investment in water cooling modules has positioned them to capitalize on this AI wave.

 

▲ Supermicro Custom Liquid Cooling

 

Lin Yushen stressed that the continuous increase in TDP driven by high-speed computing and the growing cooling demands of AI servers have pushed traditional heat pipe cooling to its limits, necessitating the adoption of water cooling modules. Fanner already has several customers, including Supermicro and Meta, that have adopted their solutions. Despite the faster-than-expected adoption of water cooling, it is unlikely to be fully implemented in 2023. However, it is expected to see a significant breakthrough by 2024, with explosive growth anticipated in 2025.