Revolution of Full Liquid-Cooled Servers Efficient Cooling Solutions for CPU, Memory, and PCIe

 

Under the backdrop of China's 14th Five-Year Plan, which emphasizes the development of the digital economy, data centers serve as the core infrastructure supporting digital transformation, but they also face significant carbon emission pressures. With the increase in chip and server power consumption, the power density per rack is rising, and traditional air cooling is gradually becoming limited in terms of heat dissipation and energy optimization.

 

▲ Data Centers

 

Liquid cooling, as an emerging cooling technology, uses liquid coolant to carry away the heat generated by components. Compared to air cooling, liquid cooling offers several advantages, including support for high-power chips, extended chip lifespan, reduced PUE (Power Usage Effectiveness) of data centers, improved heat transfer efficiency, minimized heat spots, support for higher rack densities, reduced noise, and improved environmental adaptability. Therefore, liquid cooling will become an important part of future data center construction, crucial for achieving green computing and carbon neutrality goals.

 

The nodes of fully liquid-cooled servers are composed of a node chassis, motherboard, CPU chips, memory modules, memory cold plates, CPU cold plates, IO cold plates, power supplies, and power supply heat exchangers.

 

 

I CPU Cold Plate Design

 

The CPU cold plate module is designed based on the requirements for the Intel 5th generation Xeon platform scalable processor cold plate. It takes into account factors such as heat dissipation, structural performance, yield, cost, and compatibility with different materials in cold plate design, resulting in an optimized reference design. The CPU cold plate primarily consists of an aluminum bracket, cold plate, and cold plate connectors.

 

▲ CPU Cold Plate

 

 

II Memory Liquid Cooling Design

 

The memory liquid cooling design employs an innovative "rail tie" liquid cooling heatsink, named for its resemblance to the sleepers on railway tracks when memory slots are fully occupied. This design combines traditional air cooling with cold plate cooling. The heatsink, incorporating heat pipes (or made of pure aluminum/copper, VaporChamber, etc.), transfers the heat from the memory to both ends, which then contacts the cold plate through selected thermal pads, allowing the liquid coolant in the cold plate to carry away the heat.

 

The memory and heatsink can be assembled into a minimal maintenance unit (referred to as the memory module) outside the system using fixtures. The memory cold plate is designed with a structure to ensure good contact between the heatsink and the memory cold plate. This structure can be secured with screws or maintained without tools as needed. The top of the memory cold plate cools the memory, while the bottom can cool other heat-generating components on the motherboard, such as VR, maximizing the use of the memory cold plate. To simplify the cold plate design, an adapter bracket between the memory and motherboard can be introduced to meet the height clearance of different motherboards.

 

▲ Memory Cold Plate

 

Compared to existing tubing-based liquid cooling solutions in the market, the "rail tie" liquid cooling design has the following advantages:

 

Ease of Maintenance: During memory maintenance, the memory module is serviced just like an air-cooled memory module, without the need to remove the heatsink and fasteners. This greatly improves assembly efficiency and reliability while reducing potential damage to memory chips and thermal pads during installation and removal.

 

Good Compatibility: The cooling performance is unaffected by different memory chip thicknesses or spacing. The solution supports a minimum memory spacing of 7.5 mm and is upwardly compatible. The decoupled design of the heatsink and cold plate allows for reuse and standardization of liquid-cooled memory.

 

Higher Cost-Effectiveness: The heatsink can be selected based on memory power consumption, and the number of heatsinks can be configured according to memory requirements. For a memory spacing of 7.5 mm, this solution can meet the cooling needs of memory modules with power consumption exceeding 30W.

 

Easy to Manufacture and Assemble: There are no liquid cooling tubes between memory slots, eliminating the need for complex tubing welding and process control. The heatsink can be manufactured using traditional air cooling and standard CPU cold plate manufacturing techniques. The thermal performance is not sensitive to the tolerances between the heatsink and motherboard in the direction perpendicular to the memory chip plane, making assembly easier.

 

High Reliability: The "rail tie" liquid cooling design avoids potential damage to memory chips and thermal pads during assembly and meets the requirements for multiple insertions/removals. Additionally, it eliminates the risk of signal contact issues between memory and sockets due to misalignment, greatly enhancing system reliability.

 

 

III SSD Liquid Cooling Design

 

The innovative SSD liquid cooling solution transfers heat from the SSD area through a heatsink with built-in heat pipes. The heat is then conducted to the cold plate outside the SSD area through direct contact with thermal pads.

 

This SSD liquid cooling solution mainly consists of an SSD module with a heatsink, SSD cold plate, SSD module locking mechanism, and SSD bracket. The locking mechanism on the SSD bracket ensures proper preloading to maintain reliable long-term contact between the SSD module and the cold plate. To facilitate installation in confined spaces, the SSD bracket adopts a drawer-type design in the depth direction of the server.

 

▲ SSD Liquid Cooling Design

 

Compared to existing SSD liquid cooling attempts, the advancements in this solution include:

• Supports over 30 hot-swappable insertions/removals without powering off.
• No risk of shearing damage to thermal interface materials during SSD installation; the locking mechanism ensures long-term contact reliability.
• Low manufacturing complexity, requiring only traditional air cooling and CPU cold plate manufacturing processes.
• No water pathways between SSDs, allowing multiple SSDs to share a single cold plate, reducing the number of connectors and lowering the risk of leakage.
• Flexible adaptation to different SSD thicknesses and system configurations.

 

 

IV NPCIe/OCP Card Liquid Cooling Design

 

1. PCIe Liquid Cooling Solution

The PCIe card liquid cooling solution is based on the existing air-cooled PCIe card. It achieves cooling for the optical module and main chips on the PCIe card by developing a cooling module that contacts the system's cold plate. The heat from the optical module is transferred via heat pipes to the main heatsink module on the PCIe card, which then dissipates heat through contact with the IO cold plate using appropriate thermal interface materials.

 

The liquid-cooled PCIe card primarily consists of a QSFP heatsink clamp, PCIe chip heatsink module, and PCIe card. The QSFP clamp must have sufficient elasticity to ensure proper floating contact during installation, preventing damage to the optical module while ensuring good contact for optimal cooling performance.

 

▲ PCIe Liquid Cooling

 

2. OCP 3.0 Liquid Cooling Solution

The OCP 3.0 card liquid cooling solution is similar to the PCIe card. It customizes a liquid cooling heatsink for the OCP 3.0 card, transferring heat from the card's main chips to the liquid cooling heatsink. Heat is then removed through contact between the heatsink and the system's IO cold plate.

 

The OCP 3.0 liquid cooling module mainly consists of a heatsink module, OCP 3.0 card, and its bracket. Due to space limitations, the locking mechanism uses spring screws to ensure long-term contact reliability between the heatsink module and the IO cold plate.

 

▲ OCP 3.0 Liquid Cooling

 

Given the need for easy maintenance and multiple hot-swappable insertions/removals of the OCP 3.0 card, the locking mechanism and thermal interface materials have been optimized to improve overall reliability and maintenance convenience.

 

 3. IO Cold Plate Solution 

The IO cold plate is a multifunctional cold plate, cooling not only the heat-generating components in the motherboard's IO area but also the liquid-cooled PCIe and OCP 3.0 cards.

 

▲ IO Cold Plate

 

The IO cold plate primarily consists of an aluminum alloy body and copper pipes for coolant flow and enhanced heat dissipation. The design must be optimized according to the motherboard layout and heat dissipation requirements. The liquid-cooled PCIe and OCP 3.0 card modules contact the IO cold plate along designated pathways. The coolant materials must be compatible with the system's pipeline coolant and wetting agents.

 

▲ IO Cold Plate

 

This liquid cooling solution for the IO cold plate meets the multi-dimensional assembly needs of several components, using a combination of copper and aluminum materials to resolve compatibility issues. It ensures effective heat dissipation, reduces the cold plate weight by 60%, and lowers costs.

 

 

V Power Supply Cold Plate Design

 

The power supply liquid cooling solution integrates an external air-to-liquid heat exchanger with the existing air-cooled power supply (PSU), cooling the air expelled by the PSU fan and reducing the preheating effect on the external data center environment.

 

The PSU rear heat exchanger features a multi-layer structure with overlapping flow channels and fins. The heat exchanger dimensions are optimized for space and functional needs without affecting the PSU cable connections. The heat exchanger is independently mounted on the node chassis.

 

▲ Power Supply Liquid Cooling

 

This innovative power supply liquid cooling solution eliminates the need for developing new liquid-cooled PSUs, shortening development time and reducing costs. Its high adaptability allows it to be flexibly applied to various PSU designs, saving more than 60% compared to custom liquid-cooled PSUs.

 

For full-rack applications, a centralized air-to-liquid heat exchanger can be used instead of distributed rear heat exchangers for each PSU. This centralized structure replaces individual PSU heat exchangers, providing cooling through a system that integrates with the rack's airflow pathways, ensuring no impact on the server room environment.

 

A single centralized heat exchanger can handle 8kW of cooling capacity, supporting at least 150 PSUs. The main components of the centralized air-to-liquid heat exchanger include a heat exchanger core, water inlet and outlet ports, copper cooling tubes, an aluminum casing, and flow guide fins. This setup allows for efficient and scalable PSU cooling in high-density data centers.

 

 

Conclusion

 

▲ Fully Liquid-cooled Server

 

Liquid cooling technology, as exemplified by these optimized designs, is key to managing the increasing heat output of modern data centers while driving efficiency and sustainability goals. With innovations in cold plate solutions for CPUs, memory, SSDs, PCIe/OCP cards, and power supplies, these liquid-cooled servers are paving the way toward a future of greener, higher-performing data centers.