Design of Efficient Liquid Cooling Systems for Data Centers

 

I Components of the Data Center Cooling System

 

The majority of the electrical energy consumed by IT equipment is converted into waste heat. To ensure the IT equipment operates within an appropriate temperature range, data centers are equipped with cooling and heat dissipation systems, including chillers, cooling towers, and precision air conditioning units, which remove waste heat from the data center. The heat transfer process is illustrated in Figure 1. Key energy consumption points include chillers, cooling towers, pumps, and precision air conditioning units.

 

▲ Figure 1: Heat Transfer in Data Centers

 

Currently, the primary heat transfer media in data centers are air or water. Water, with a constant pressure specific heat capacity of 1.004 kJ/(KgK) and a specific heat capacity of 4200 kJ/(KgK), has a heat transfer capacity approximately 1,000 times greater than air. Therefore, using water as a cooling medium is an effective energy-saving method in the design of cooling systems. To improve the energy efficiency of cooling systems, measures such as high-efficiency radiators and precise air delivery are used to capture and transfer heat.

 

In precision air conditioning, cooling has evolved from room-level to modular data rooms and rack-level cooling, moving closer to the heat source and reducing energy consumption in coolant transport. The generation of cooling sources has progressed from air cooling to water cooling and natural cooling, enhancing external heat transfer efficiency.

 

 

Traditional cooling systems feature independent control systems and operating strategies for precision air conditioning, chillers, and cooling towers, optimizing efficiency locally. However, overall cooling efficiency still requires improvement.

 

Systematic improvements can be achieved by end-to-end management and precise control of heat collection, cooling source preparation, and external heat transfer, thereby reducing the power consumption of the cooling system.

 

 

II End-to-End Liquid Cooling System Design

 

1. Board-Level Liquid Cooling Design

With the exponential growth in computing power demand, the integration and power consumption of CPUs and GPUs have increased significantly, with single-chip power consumption reaching 300W. Traditional chip heatsinks and air cooling solutions have encountered cooling bottlenecks. Since the chip is the heat source, the primary challenge for the data center cooling system is to efficiently remove heat from within the chip.

 

From a heat dissipation perspective, the heat generated by the chip must first be transferred to the board-level heatsink. More efficient heatsink solutions will facilitate better heat collection.

 

For single chips with power consumption below 200W and IT equipment with power consumption below 20kW per rack, air can continue to be used as the heat transfer medium. Heat pipe heatsinks and vapor chamber (VC) heatsinks, combined with high thermal conductivity TIM materials (such as graphite sheets/graphene), effectively reduce the thermal resistance between the chip and the heatsink base, improving the heatsink's efficiency.

 

For single chips with power consumption exceeding 200W and IT equipment with power consumption exceeding 20kW per rack, air is no longer sufficient as the heat transfer medium, and liquid coolant must be used for cooling. Liquid-cooled cold plate technology is currently a mature solution for board-level chip cooling. A liquid-cooled cold plate consists of inlet and outlet connectors, an upper cover, and a base plate, which are connected by vacuum brazing to form a sealed liquid heat exchange chamber. The chamber includes distribution chambers and various width flow channels, which control fluid flow and increase turbulence, enhancing local cooling capabilities and eliminating hot spots caused by high-power chips. The internal structure is shown in Figure 2.

 

▲ Figure 2: Cross-Section of a Liquid-Cooled Cold Plate

 

Different types of boards within the same rack have varying power levels and hot spots, but the supply pressure at the inlet connector of the liquid supply line is generally the same, requiring the cold plate's distribution chamber to control throttling. For boards with lower chip power consumption, throttling reduces the flow of coolant. In practice, liquid-cooled cold plates cover the CPU, memory, and other high-power components, but components like resistors and capacitors, which are not covered, generate residual heat that requires fan cooling. This results in a combination of liquid and air cooling within the system, leaving room for improvement in cooling efficiency.

 

By using TIM materials to cover all components during cold plate design, 100% liquid cooling can be achieved technically, but this increases the cost and complexity of the cold plate. While pursuing efficient cooling, the initial cost investment must also be considered. If the node board types are uniform, a fully covered board can be considered, with initial costs offset by scaling production, achieving a balance between energy savings and investment.

 

 

Deionized water is typically used as the coolant in liquid cooling due to its high specific heat capacity, allowing for rapid heat absorption while being non-corrosive, thus not affecting pipeline reliability. Cold plate liquid cooling is indirect, with the chip not directly contacting the liquid coolant, resulting in high reliability and mature technology.

 

However, thermal resistance exists between the chip and the liquid coolant, leading some manufacturers to promote immersion cooling solutions. In immersion cooling, IT equipment is submerged in circulating liquid, with the chip directly contacting the coolant, reducing thermal resistance while leveraging phase change to remove more heat, making it a new hotspot in liquid cooling. Fluorinated liquids are commonly used as coolants in immersion cooling, but their high cost is a barrier to large-scale commercial use.

 

2. Rack-Level Liquid Cooling

In data centers, IT equipment is arranged by racks, which house data center information equipment such as servers, storage devices, and network switches. While board-level cooling removes heat from individual IT devices, rack-level cooling collects and transfers the heat outdoors. Key components in rack-level liquid cooling include inlet and outlet manifolds, monitoring units, temperature sensors, solenoid valves, and check valves, as shown in Figure 3.

 

▲ Figure 3: Rack-Level Liquid Cooling Configuration

 

The Manifold connects externally to the room-level liquid cooling distribution unit and internally via quick connectors to the liquid-cooled cold plate's inlet and outlet connectors, facilitating the transfer of system heat to the outside of the rack.

The main functions of the solenoid valve and check valve are to control the flow of liquid and to limit the extent of failure to a single rack in the event of a leak.

 

The primary role of the temperature sensor is to continuously monitor the inlet and outlet water temperatures. By utilizing the temperature difference between the inlet and outlet water, it controls the opening of the solenoid valve, thereby controlling the flow of water and ensuring that the heat and flow are matched.

The liquid cooling system uses deionized water as the working fluid, which theoretically will not cause short circuits.

 

However, circuit boards or electronic components often have dust particles, and when deionized water comes into contact with the circuit board, it can cause a short circuit. This is one of the major obstacles and concerns in the implementation of liquid cooling. To address the issue of cold plate leakage, measures such as quality control, micro-leakage monitoring, and prevention of sudden large leaks are employed.

 

Quality control is divided into production and installation application stages. In the production stage, the process reliability is ensured, 100% of cold plates undergo pressure testing, and ultrasound is used for random sampling and defect detection. Quick-connect fittings must be validated for effective insertion and long-term reliability. In the installation application stage, the secondary piping must be flushed clean before installation to prevent impurities from causing blockages in quick-connect fittings, spring jams, or rubber ring failures, thereby preventing leaks during operation. The above measures aim to prevent leaks as much as possible.

 

 

If a cold plate develops a micro-leak, it must be detectable and trigger an alarm to prompt maintenance personnel to repair it promptly. There are two detection methods: one is using a water immersion sensor, which is installed on the drip tray. The main function of the drip tray is to facilitate leak detection and prevent liquid from leaking outside the rack, reducing the spread of faults.

 

While water immersion sensor detection is mature and reliable, it requires the leaking fluid to accumulate in the drip tray after flowing along the hardware board and rack fittings, by which time the total amount of leaked fluid may be considerable and may have already damaged the board and components during the flow.

 

The other method is real-time monitoring. A tracer substance with a low boiling point is mixed into the working fluid, and in the event of a leak, a gas sensor built into the board detects it. Large-scale sudden leaks are rare but highly impactful. To prevent such incidents, check valves are installed at the inlet and outlet of the manifold on the rack. These check valves automatically close when a significant pressure difference is detected.

 

3. Room-Level Liquid Cooling Design

Room-level cooling is designed to transfer the heat extracted from the racks to the outdoors. The room-level liquid cooling solution includes a liquid-cooled modular data room, chillers, water pumps, cooling towers, piping, and more, as shown in Figure 4.

 

▲ Figure 4: Room-Level Liquid Cooling Configuration

 

Typically, a liquid-cooled modular data room contains two backup liquid cooling distribution units (CDUs), 10-20 IT racks, 1-2 row-level air conditioners, and power supply equipment, as shown in Figure 4.

 

The liquid cooling distribution unit (CDU) is responsible for distributing the working fluid among IT liquid-cooled racks, providing secondary-side flow distribution, pressure control, physical isolation, and anti-condensation functions. During actual operation, the CDU supplies a certain flow and temperature of cooling water to the IT liquid-cooled racks, which enters the liquid-cooled cold plates via the manifold, removes the heat generated by processors and key components, and returns the heated cooling water to the intermediate heat exchange unit of the CDU. The heat is then released into the outdoor return water pipeline, and this portion of the heat is discharged to the outdoor environment via chillers or dry coolers, completing the heat management process for the liquid-cooled servers.

 

The CDU regulates the temperature and flow of the coolant entering the liquid-cooled cold plates, thereby providing cooling to the IT racks and distributing cooling power. The internal heat exchange unit also plays a role in isolating the liquid supply circuit between the modular data room and the outdoor environment. Due to its critical role, the CDU usually employs a 1+1 redundancy configuration. The CDU controls the flow of the liquid coolant by detecting the inlet and outlet water temperatures and supply pressure and adjusting the speed of the supply water pump.

 

 

Currently, most CDU control systems do not link with the temperature sensors within the racks, resulting in relatively coarse control. To address this issue, some applications have replaced centralized CDUs with distributed CDUs, which are integrated into the racks. This way, CDU flow adjustments are entirely based on the operational status and power consumption fluctuations of the rack. Centralized CDUs are suitable for scenarios with a large number of liquid-cooled racks that can be consolidated into a modular data room, while distributed CDUs are better suited for situations with only 2-3 liquid-cooled racks, making deployment easier.

 

 

III Conclusion

 

Under the guidance of dual-carbon goals, data centers bear a dual mission: on one hand, through intensive and scaled operations, they provide sufficient computing power for the digital economy. The widespread application of high-density racks and high-power chips, driven by the efficiency of data center computing power, has caused traditional air cooling to encounter bottlenecks. On the other hand, by utilizing high-efficiency heat exchangers, liquid cooling, and natural cooling from dry coolers, data centers can reduce their own energy consumption.

 

After adopting liquid cooling, cooling efficiency is significantly improved, with the energy consumption of cooling systems reduced from 37% to about 10%, resulting in significant energy savings and carbon reduction. If 50% of newly built data centers nationwide adopt liquid cooling, 45 billion kWh of electricity can be saved annually, and 3 million tons of CO₂ emissions can be reduced.