Liquid Cooling Technology: Tackling the Cooling Challenges of Data Centers in the AIGC Era

 

With the rapid development of Artificial Intelligence Generated Content (AIGC), the demand for computing power has exploded, leading to a sharp increase in the power consumption and thermal management needs of data centers. The high computing resource requirements during AI model training and inference significantly increase server heat generation, raising the bar for cooling technologies. According to a report by Colocation America, the average power per cabinet in global data centers had increased to 16.5 kW by 2020, a 175% rise compared to 2008. As a result, liquid cooling technology has become a new focal point for data center cooling solutions.

 

At this year's GTC conference, NVIDIA not only showcased the B200 and GB200 chips but also highlighted the accompanying liquid cooling technology. Additionally, at the 2024 SIEPR Economic Summit, NVIDIA CEO Jensen Huang revealed that the next-generation DGX GPU servers would fully adopt liquid cooling. NVIDIA's decision has set a trend in the industry, injecting new momentum into the development of liquid cooling technology. As AI technology continues to advance, the importance of liquid cooling is becoming increasingly evident. Liquid cooling technology not only significantly reduces the energy consumption of data centers but also enhances server operational efficiency and extends equipment lifespan. Therefore, liquid cooling is gradually becoming a priority consideration for data center cooling solutions.

 

 

I Comparison of Data Center Cooling Methods

 

Currently, data center cooling systems are primarily divided into two types: air cooling and liquid cooling. Liquid cooling technology replaces air with a liquid medium to exchange heat with the server's heat-generating components, thus carrying away the heat and ensuring the server operates stably within an optimal temperature range. In contrast, air cooling relies on fans and air conditioning systems to dissipate heat through the movement of air. Liquid cooling directly cools the heat-generating components, achieving a thermal conduction efficiency 25 times greater than air, with a specific heat capacity 1,000 to 3,500 times higher, and a convective heat transfer efficiency 10 to 40 times greater than air. Thus, under the same conditions, liquid cooling technology far surpasses air cooling in cooling efficiency.

 

▲ Data center cooling systems

 

▲  Liquid cooling technology & Air cooling 

 

Compared to air cooling, liquid cooling provides higher cooling efficiency and lower energy consumption. In high-density computing environments, air cooling systems often struggle to meet cooling demands, whereas liquid cooling can effectively address this challenge. Additionally, liquid cooling offers advantages such as low noise and a smaller footprint, making it well-suited for the high-density configurations and green energy-saving requirements of modern data centers.

 

 

II What Drives Liquid Cooling Development in the AI Era?

 

1. Rising Heat Power of Computing Chips: Air Cooling Reaches Its Limit

With the rapid development of AI technology, the demand for computing power continues to rise, leading to increasing heat generation and heat flux density in chips. When chips operate at high temperatures for extended periods, their performance and lifespan are negatively impacted, and failure rates increase. Research indicates that when a chip's operating temperature approaches 70-80°C, every 10°C increase can reduce its performance by about 50%.

 

Currently, Intel's CPUs have a thermal design power (TDP) of up to 350W, NVIDIA's H100 reaches 700W, and the future B100 could reach 1,000W, nearing the 800W single-point cooling limit of air cooling. As computing chip power consumption continues to grow, and with CPU and GPU power consumption accounting for about 80% of total AI server power, continuing to use air cooling will lead to a significant increase in in-row air conditioning needs. In high-density cooling scenarios, liquid cooling offers significant cost and performance advantages.

 

Apart from the chip side, the power density per cabinet in data centers is also on the rise. Traditional air cooling typically meets cabinet cooling needs in the 12KW to 15KW range. According to the 2022 Global Data Center Survey Report by Uptime Institute, the maximum power for a single NVIDIA DGX A100 server is 6.5KW, and a standard 42U high cabinet can house around five 5U high AI servers, with a total power exceeding 20KW per cabinet. Traditional air cooling cannot meet the cooling needs of AI server cabinets.

 

2. Driven by Data Center Energy-Saving Needs: Higher PUE Requirements

PUE (Power Usage Effectiveness) is a key indicator for evaluating data center energy efficiency, calculated as: PUE = Total Data Center Energy Consumption / IT Equipment Energy Consumption. The closer the PUE value is to 1, the higher the data center's energy efficiency; conversely, the higher the PUE value, the lower the overall efficiency.

 

Statistics divide data center power consumption into several parts: IT equipment accounts for 45%, cooling systems for 43%, power supply and distribution systems for 10%, and lighting and other uses for 2%. Among these, air conditioning system energy consumption is second only to IT equipment, so reducing air conditioning system energy consumption becomes particularly important when IT systems cannot be upgraded.

 

In the context of the national goals of achieving "carbon peak" and "carbon neutrality" and the "Eastern Data, Western Computation" strategy, the newly released Green Data Center Government Procurement Demand Standards (Trial) imposes stricter PUE requirements. This standard stipulates that from June 2023, data center PUE must not exceed 1.4, and by 2025, the requirement will be a PUE of no more than 1.3. According to data from CDCC and Inspur Information, data centers using air cooling typically have a PUE between 1.4 and 1.5, while liquid cooling technology can reduce PUE to below 1.2. Thus, adopting more energy-efficient and effective liquid cooling technology has become a trend.

 

Energy consumption in data centers has long been a focus of industry attention, especially against the backdrop of global energy resource constraints and heightened environmental awareness. Improving data center energy efficiency is particularly crucial. Liquid cooling technology, by providing more efficient cooling solutions, reduces air conditioning system energy consumption, thereby significantly lowering data center PUE values. This technology not only helps reduce operational costs but also lowers carbon emissions, aligning with sustainable development goals.

 

 

▲ Data Center Energy Consumption

 

 

III Classification of Liquid Cooling Technology

 

Liquid cooling systems can be classified into direct liquid cooling and indirect liquid cooling based on how the liquid interacts with the hardware. Direct liquid cooling involves the liquid coming into direct contact with the hardware components to transfer heat. This method can be further divided into immersion cooling and spray cooling. Immersion cooling submerges the hardware components entirely in the liquid, while spray cooling involves spraying the liquid directly onto the hardware.

 

Indirect liquid cooling, on the other hand, uses an intermediary component (such as a heat exchanger or cooling plate) to conduct heat away, preventing the liquid from directly contacting the hardware. A common indirect liquid cooling system is the cold plate liquid cooling system, which can be further divided into single-phase and two-phase cold plate cooling based on whether the cooling medium undergoes a phase change.

 

▲ Introduction to Liquid Cooling Methods

 

1. From Cold Plates to Immersion Cold Plates

Liquid cooling technology transfers heat from heat-generating components to a cooling liquid through cold plates, and the cooling liquid then dissipates the heat through its refrigerating properties. In this system, the working liquid does not directly contact the electronic components, resulting in minimal modifications to the computer system. The original air-cooling heat sink can simply be replaced with a liquid cooling kit, and the coolant pipes can be routed outside the chassis. This technology is particularly suitable for cooling requirements with medium to high heat flux densities.

 

A cold plate liquid cooling system primarily consists of a cooling tower, a Coolant Distribution Unit (CDU), primary & secondary liquid cooling circuits, cooling medium, and a liquid-cooled cabinet. The primary circuit refers to the loop that discharges heat from the secondary circuit to the outdoor environment or other heat recovery units, while the secondary circuit refers to the loop that removes heat from the servers and dissipates it through the primary circuit. The two circuits exchange heat through the CDU, or Coolant Distribution Unit.

 

The working principle of the cold plate liquid cooling system is relatively simple, but in practical applications, considerations need to be made for the design of the cold plates, the selection of cooling liquids, and system maintenance. Additionally, cold plate liquid cooling systems perform exceptionally well in high heat flux density environments, making them highly suitable for the high-density layout requirements of modern data centers.

 

▲ Schematic Diagram of the Cold Plate Liquid Cooling System

 

Immersion liquid cooling systems achieve efficient heat dissipation by directly submerging heat-generating components in non-conductive cooling liquids. Depending on whether the cooling liquid undergoes a phase change during circulation, immersion liquid cooling can be divided into single-phase immersion cooling and two-phase immersion cooling.

 

In single-phase immersion cooling, the cooling liquid only undergoes a temperature change during the heat exchange process without a phase change. Heat transfer relies entirely on the sensible heat change of the liquid, utilizing the characteristic that the liquid expands and decreases in density when heated. The warmer cooling liquid naturally rises and is cooled by an external cooling loop's heat exchanger. The cooled liquid then sinks under gravity, completing the cooling cycle. In this method, the cooling liquid remains in a liquid state throughout the entire process. In contrast, two-phase immersion cooling involves the cooling liquid undergoing a phase change from liquid to gas during heat dissipation and then returning from gas to liquid.

 

An immersion liquid cooling system includes both indoor and outdoor components. The outdoor side includes a cooling tower, primary pipeline network, and primary cooling liquid; the indoor side includes a Coolant Distribution Unit (CDU), immersion tank (cabinet), IT equipment, secondary pipeline network, and secondary cooling liquid. During use, the IT equipment is fully immersed in the cooling liquid, so the selection of the cooling liquid needs to consider non-conductive fluids, such as silicone oil or fluorinated liquids.

 

▲ Schematic Diagram of Single-Phase lmmersion Liquid Cooling

 

Although spray cooling exists, its application is relatively limited and not suitable for high-density servers and large-scale data centers. In the short term, cold plate liquid cooling is highly suitable for the cooling needs of the AI era and the transition of data centers from air cooling to liquid cooling due to its maturity, compatibility with existing systems, ease of maintenance, and low retrofit costs. In the long run, immersion liquid cooling, with its excellent thermal conductivity, efficient waste heat recovery capability, and support for higher cabinet power, will be more suitable for the evolving cooling needs of future data centers. Especially as cabinet unit power continues to increase, immersion liquid cooling can provide more efficient cooling solutions, helping to reduce the overall Power Usage Effectiveness (PUE) of data centers.

 

2. Preferred Choice for Intelligent Computing Centers – Liquid Cooling

As power density increases, liquid cooling solutions are becoming the choice for more newly constructed GPU computing centers. According to IDC's "China Semiannual Liquid-Cooled Server Market (H1 2023) Tracker" report, the Chinese liquid-cooled server market reached $1.51 billion in 2023. IDC predicts that from 2022 to 2027, the compound annual growth rate of the Chinese liquid-cooled server market will reach 54.7%, with the market size expected to reach $8.9 billion by 2027.

 

The application of liquid cooling technology in intelligent computing centers not only enhances computing performance but also significantly reduces energy consumption and operating costs. The promotion of liquid cooling technology will drive data centers towards more efficient, green, and intelligent development, providing a solid foundation for meeting data processing needs in the AI era.

 

▲ Liquid Cooling Server Market Size

 

IV Liquid Cooling Industry Chain

 

The liquid cooling industry chain encompasses three main segments: upstream product component suppliers, midstream liquid-cooled server manufacturers, and downstream computing power users. Among the current downstream users, domestic companies like Alibaba are focusing on the development of single-phase immersion liquid cooling, while others, such as Baidu, Tencent, and JD.com, mainly use cold plate liquid cooling. Overseas, immersion cooling is more advanced than cold plate cooling, with leading U.S. companies such as Intel, Google, and Meta driving the rapid development of immersion liquid cooling, especially with AI support.

 

▲ Liquid Cooling lndustry Chain

 

 

V Potential Issues with Immersion Liquid Cooling Technology

 

1. Coolant Selection

Coolant is one of the key raw materials in liquid cooling technology and presents a high technical barrier. In immersion liquid cooling technology, the coolant needs to directly contact electronic products, thus imposing high requirements on the coolant's performance, such as excellent thermal conductivity, good insulation, and material compatibility. Additionally, environmental characteristics like odor, toxicity, and ease of degradation are also important, and the coolant should be as user-friendly and environmentally friendly as possible.

 

The most commonly used immersion coolants currently include hydrocarbons and organosilicons (commonly referred to as "oils," such as mineral oil) and fluorinated compounds (such as fluorinated liquids). Fluorinated liquids have good overall performance and are considered ideal liquid cooling materials. However, the major challenge with fluorinated liquids is their high cost. With increasingly stringent environmental protection requirements, silicone oil, which has a higher thermal conductivity and lower density, is also more environmentally friendly. The choice of cooling medium mainly depends on the cooling process.

 

2. Optical Path Sealing Issues

Coolants such as fluorinated liquids or silicone oils possess excellent insulating properties, effectively preventing circuit short circuits. Under low-frequency signal conditions, these coolants have minimal interference with signal transmission. However, under high-frequency signals, the impact of the coolant on signal transmission needs careful assessment and control. Overall, the impact on circuits is manageable.

 

Regarding optical paths, most optical modules in data centers are designed with non-hermetic packaging, meaning that without appropriate modifications, the coolant may enter the optical cavity, affecting optical performance. Even with hermetic packaging, some passive optical paths, such as lenses, remain outside the hermetic chamber.

 

The design of optical paths is typically based on the refractive index of air (approximately 1.0). When optical components are immersed in a coolant, the refractive index of the coolant, which differs from air, can cause changes in focal points and coupling efficiency. For example, the refractive index of fluorinated oil is usually around 1.3, and this change in refractive index may require adjustments to the optical path design parameters.

 

To address the potential impact of immersion liquid cooling on optical and electrical paths, the industry is taking various measures, such as developing new optical module packaging technologies adapted to the coolant environment, optimizing circuit design for high-frequency signals, and researching optical materials and structures more suitable for immersion cooling.

 

3. Integrated Delivery vs. Decoupled Delivery

Currently, there are three delivery models for cold plate liquid-cooled servers:

① IT equipment side delivers only the liquid-cooled server;

② IT side delivers the "liquid-cooled server + liquid-cooled cabinet";

③ IT side delivers the "liquid-cooled server + liquid-cooled cabinet + CDU + secondary circuit".

 

The third delivery model, integrated delivery, where the entire cabinet is delivered by the same manufacturer with a self-defined standard for integrated design and development, is the most widely used. Decoupled delivery involves following user-defined interface design specifications between the liquid-cooled cabinet and the liquid-cooled server, with the cabinet and server delivered by different manufacturers. Infrastructure and server manufacturers need to coordinate and cooperate. Decoupled delivery is easier to scale and deploy flexibly.

 

▲ Cold Plate Liquid-Cooled Server Delivery Mode Differentiation

 

Currently, the level of standardization in liquid cooling technology within China is relatively low. Different server equipment, coolants, refrigeration pipelines, and power supply products vary in form, and there is no unified interface standard, which presents challenges for standardization and large-scale application. The white papers published by the three major domestic telecom operators outline a three-year vision for liquid cooling technology, gradually verifying and testing the technology, with plans to begin large-scale applications of liquid cooling by 2025. It is expected that over 50% of data projects will adopt this technology, promoting the standardization and large-scale implementation of liquid cooling and supporting decoupled delivery.