Application of Liquid-Cooled Servers in Data Centers
Sep 19, 2024
Leave a message
In recent years, industries are actively promoting digital transformation, and China's digital economy is rapidly growing, with computing power injecting new momentum into economic and social development. To meet the increasing demand for computing power, data center scales have been growing rapidly across the country. By the end of 2023, the total number of racks in use across China's data centers exceeded 8.1 million standard racks, and the total computing power reached 230 EFLOPS, ranking second globally. With the expansion of computing power, the electricity consumption of data centers is also rapidly increasing.
According to the "China Data Center Industry Development White Paper (2023)," it is expected that by 2025, the national electricity consumption of data centers will reach 1.2×10¹¹ kW•h, and total carbon emissions will be around 100 million tons, accounting for approximately 1.23% of the national total. To implement "carbon peaking and carbon neutrality," the government has issued a series of policies to promote energy saving and consumption reduction in data centers, with PUE (Power Usage Effectiveness) requirements quickly evolving to below 1.3 and even 1.25. Traditional cooling technologies in data centers are gradually failing to meet these new demands.
Simultaneously, as the computing power density of single chips continues to increase, air cooling has gradually become a bottleneck, and the emergence of high-power equipment such as AI servers has exacerbated the issue of cabinet space shortages. Under the dual pressure of national carbon policies and increasing chip computing power density, the introduction of liquid cooling technology is becoming increasingly urgent.
I Advantages of Liquid Cooling Technology
Compared to traditional air cooling, liquid cooling can significantly improve cooling efficiency and operational reliability, with good environmental adaptability and reduced land occupation.
1. Reducing Equipment Energy Consumption and Supporting Energy Conservation and Carbon Reduction
In the digital economy era, the demand for computing power is rapidly growing, coupled with the widespread application of 5G, driving the development of information infrastructure. This has significantly increased the electricity consumption and carbon emissions of data centers. Liquid cooling can reduce data center PUE to around 1.1–1.2, effectively reducing energy consumption and carbon emissions. For example, for 100 20kW liquid-cooled racks, reducing PUE from 1.45 to 1.15 can save more than 10⁷ kW•h of electricity annually, saving over 7 million yuan in electricity costs and reducing carbon emissions by 6,000 tons. Additionally, liquid cooling environments can reduce or eliminate fans, further reducing server energy consumption.
2. Extending Equipment Life and Reducing Maintenance Pressure
According to the 10°C rule for chips, the lifespan of components decreases by about 50% for every 10°C increase in temperature. Liquid cooling can effectively lower the CPU junction temperature by over 10°C (the junction temperature is the actual operating temperature of the semiconductor in electronic devices), extending the server's lifespan. Compared to traditional air cooling, immersion liquid cooling can remove fans, reduce noise and dust, and has a lower thermal failure rate. Data shows that the failure rate of hard drives in immersion cooling environments is about 50% lower than in air-cooled environments. The reduction in server failure rates and the extension of server lifespans can significantly improve business operational stability and reduce maintenance pressure.
3. Good Environmental Adaptability and High Rack Deployment Rate
Traditional air-cooled racks have high environmental requirements and are significantly affected by external environmental factors. Liquid-cooled racks have low environmental requirements and can maintain stable PUE in regions with large temperature differences. Liquid-cooled racks have a high deployment density, approximately 3–4 times that of traditional air-cooled racks, saving about 75% of data center space for the same computing power. Liquid-cooled data centers can eliminate air conditioning and reduce the need for chiller units, further saving space.
II Liquid Cooling Technology Routes
Air-cooled servers use air as the heat transfer medium, while liquid-cooled servers use liquids such as water or fluorinated fluids. Due to differences in density, specific heat capacity, and thermal conductivity, the cooling capacity of liquid at the same flow rate can be up to 3,500 times higher than that of air. The highly efficient cooling of liquid cooling can significantly increase server deployment density, reducing the occupation of rack and room space.
Liquid cooling technologies can be divided into non-contact liquid cooling and contact liquid cooling based on how the coolant interacts with heat sources.
1. Non-Contact Liquid Cooling
Non-contact liquid cooling primarily refers to cold plate liquid cooling, where the liquid flows through the cold plates to dissipate heat from components like the CPU/GPU/VRD/DIMM, cooling parts of the server. The liquid in cold plate liquid cooling does not directly contact the heat source.
With its higher specific heat capacity and thermal conductivity, cold plate liquid cooling can reduce energy consumption by 60%–90% compared to traditional air cooling and reduce the PUE of data centers to around 1.2. Cold plate liquid-cooled servers use traditional rack deployment, making the retrofitting of existing data centers relatively easy and cost-effective. However, as liquid cooling is only applied to certain high-heat components, some fans are still needed, and air conditioning is required for cooling other non-liquid-cooled components in the server.
Heat pipe liquid cooling also falls under non-contact liquid cooling, transferring the heat from components like CPUs out of the server through heat pipes, which is then dissipated by circulating water. It primarily uses water, refrigerants, and alcohol-based liquids, though it has limited current application.
2. Contact Liquid Cooling
2.1 Immersion Liquid Cooling
In immersion liquid cooling, the entire server is submerged in a coolant, with the coolant directly contacting the server. The circulating flow or evaporative condensation of the coolant removes the heat. Non-conductive, non-corrosive liquids like mineral oil, silicone oil, or fluorinated liquids are typically used.
In immersion liquid cooling, all heat-generating components dissipate heat through liquid cooling, allowing for the complete removal of fans and eliminating the need for air conditioning in the data center. Compared to cold plate liquid cooling, immersion liquid cooling provides better heat transfer, energy saving, and noise reduction, potentially reducing PUE to 1.1 or lower. However, immersion cooling often requires box-type deployment, with lower deployment density than cold plate liquid cooling, and the retrofitting of supporting infrastructure is more complex and costly. Immersion liquid-cooled servers also only support SSDs and helium-filled drives, and optical connectors and cables need to be sealed or customized for corrosion resistance.
Based on whether the coolant undergoes phase change, immersion liquid cooling can be divided into single-phase and phase-change immersion cooling. In single-phase immersion cooling, the liquid does not change phase, removing heat through temperature increases, and remains in liquid form throughout the cooling cycle. In phase-change immersion cooling, the liquid evaporates upon heating, carrying away heat in its gaseous state, and is cooled back into liquid form by a condenser, where it flows back to the chamber by gravity, completing the cooling cycle.
2.2 Spray Liquid Cooling
Spray liquid cooling involves spraying coolant from the top of the server chassis through a spray module, with heat being transferred through convection between the coolant and the heat sink, cooling all the server's heat-generating components. The heat is then dissipated through a heat exchanger. Non-conductive, non-corrosive liquids such as mineral oil and fluorinated liquids are commonly used as coolants.
Like immersion cooling, spray liquid cooling can completely remove server fans, eliminating the need for air conditioning in data centers, with PUE reduced to around 1.1. Unlike immersion cooling, spray liquid-cooled servers typically use blade-style deployment, requiring modifications to the rack and server chassis. Spray liquid cooling has lower coolant consumption and maintenance costs.
3. Comparison of Liquid Cooling Technologies
Refer to Table 1 for a comparison of different liquid cooling technologies.
▲ Liquid Cooling Technology
Considering factors such as initial investment costs, maintainability, PUE effectiveness, and industry maturity, cold plate liquid cooling and single-phase immersion liquid cooling are more advantageous than other liquid cooling technologies, becoming the mainstream solutions in the industry. Cold plate liquid cooling enables a smooth transition from traditional air-cooled models and is more widely applied in data centers. The following analysis will primarily focus on cold plate liquid cooling.
III Main Components of Cold Plate Liquid Cooling
The main components of cold plate liquid cooling systems that interact closely with IT equipment are primarily on the secondary side, including coolant, quick connectors, and the Cooling Distribution Unit (CDU).
1. Coolant
Current industry choices for coolants include ethylene glycol solutions, propylene glycol solutions, and deionized water. Ethylene glycol and propylene glycol solutions are industrial standard products and are readily available, with ethylene glycol being more cost-effective. Deionized water has good heat transfer properties, is non-toxic and safe, but requires consideration of anti-freezing at 0°C.
Coolants need to be supplemented with corrosion inhibitors and biocides to prevent bacterial growth that can cause blockages or leaks. A concentration of 20%–30% is recommended, as concentrations that are too high can affect heat dissipation, while concentrations that are too low can reduce the coolant's ability to prevent freezing and inhibit microbial growth.
2. Quick Connectors
Quick connectors link the cold plate module in liquid-cooled servers to the manifold in the liquid-cooled rack. There are two types of quick connectors: manual quick connectors and blind-mate quick connectors. Manual connectors require higher skill from maintenance personnel, with plugging and unplugging dependent on manual operation by data center staff, making it easier to decouple the server from the rack. Blind-mate connectors have no specific operational requirements and automatically connect during server insertion into the liquid-cooled rack. Quick connectors must have strong corrosion resistance and minimal liquid loss when disconnected to prevent coolant spills.
3. Cooling Distribution Unit (CDU)
The CDU is a system used for distributing coolant between liquid-cooled servers, providing secondary side flow distribution, pressure control, physical isolation, and anti-condensation features. It can also isolate the primary side and secondary side to accommodate different water quality requirements. Based on its form and deployment location, the CDU can be classified into centralized and distributed CDUs.
A centralized CDU is typically connected to multiple server cabinets, providing cooling capacity to several server cabinets simultaneously. Multiple CDU clusters can achieve N+M redundancy, offering high reliability and making it suitable for large-scale deployment of liquid-cooled server cabinets. A distributed CDU, which does not require secondary piping, provides cooling for only the servers in its own cabinet and cannot offer cross-cabinet redundancy, thus having lower reliability compared to a centralized CDU. A comparison of centralized and distributed CDUs can be found in Table 2.
▲ Comparison between centralized CDU and distributed CDU
IV Technical Requirements for Cold Plate Liquid-Cooled Servers and Cabinets
1. Server Technical Requirements
The design of cold plate liquid-cooled servers must meet the technical requirements listed in Table 3.
▲ Design requirements of cold plate liquid cooling server
In addition to meeting the various requirements listed in the table, the design of cold plate liquid-cooled servers must also comply with the design requirements for traditional air-cooled servers, including components such as the CPU, memory, hard drives, and motherboard.
2. Cabinet Technical Requirements
In liquid cooling scenarios, the coupling between servers and cabinets is high. The design of liquid-cooled cabinets must meet the technical requirements listed in Table 4.
▲ Liquid-Cooled Cabinets
In addition to meeting the requirements listed in the table, the design of liquid-cooled cabinets must also adhere to the design requirements for traditional air-cooled cabinets, such as transportation and installation, cabling within the cabinet, and functional partitioning.
V Changes Brought by the Introduction of Liquid Cooling in Data Centers
When liquid-cooled servers are introduced in data centers, there will be changes in supporting infrastructure, operations and maintenance, as well as delivery models.
1. Supporting Infrastructure
The introduction of liquid-cooled servers increases deployment density, which places higher demands on the load-bearing capacity of the data center. In addition, changes occur in areas such as the height of raised floors, power supply, and maintenance space. The changes in supporting infrastructure are outlined in Table 5.
▲ Supporting Infrastructure Changes
2. Operations and Maintenance
Currently, liquid-cooled servers are mostly customized deployments, differing from traditional air-cooled servers in terms of component compatibility, deployment methods, and maintenance tools. Specific differences are shown in Table 6.
▲ Operations and Maintenance Changes
3. Delivery Models
Liquid cooling in data centers involves interfacing between the primary cooling system, secondary cooling system, CDU, liquid-cooled cabinets, and liquid-cooled servers. The standardization level is low, and compatibility between different manufacturers is limited. Based on the delivery content, there are currently three delivery models for liquid-cooled servers, as shown in Table 7.
▲ Delivery Models
When introducing liquid cooling systems into data centers, factors such as industry maturity, ease of maintenance, and construction timelines must be considered. In the initial stages, Model 2 or Model 3 can be considered to gain experience in construction and maintenance. Over time, the interface standardization between the primary system, secondary system, CDU, liquid-cooled cabinets, and liquid-cooled servers can be gradually promoted, advancing towards Model 1.
VI Conclusion
The introduction of liquid-cooled servers in data centers can significantly reduce PUE values, improve the stability of equipment operation and business availability, increase server installation density, reduce land usage in data centers, and achieve energy savings and carbon reduction. Additionally, the increasing power density of chips and the gradual introduction of high-power equipment such as AI servers provide even greater opportunities for the adoption of liquid cooling, which is gradually becoming an important cooling solution for data centers.
However, the standardization of liquid cooling is relatively low, with different manufacturers working independently. The coupling between liquid-cooled servers and liquid-cooled cabinets is particularly high, so the subsequent standardization of relevant interfaces is necessary. Over time, it is important to achieve blind mating of water, power, and network interfaces, making liquid cooling not only functional but also user-friendly, fully realizing its role in the dual carbon goals.