Last Updated: Nov 5, 2025
Introduction
NVIDIA's GB200 systems leverage a tightly coupled, hardware-coherent design where the default driver setting, NUMA mode, unifies the entire CPU (host) and GPU (device) memory into a single pool for the operating system.
This design offers powerful flexibility, allowing standard Linux APIs like malloc and mmap to allocate memory on either processor and even facilitating automatic memory migration by the kernel.
However, this introduces a critical performance challenge. By treating high-speed GPU memory as a generic pool, the ability to isolate it from general OS functions is lost. This can lead to undesirable "spillover," where general system memory encroaches on the GPU, creating contention and resource bottlenecks.
Affected Versions
- NVIDIA Driver Version 570.158.08
- CUDA Version 12.8
- Firmware Version 1.2.2
Solution
CDMM is an alternative operating mode for NVIDIA drivers designed to prevent the OS from managing GPU memory as part of its general NUMA pool. CDMM creates a clear boundary: the Linux kernel manages system memory, and the NVIDIA driver manages device memory.
By giving the NVIDIA driver sole responsibility for the GPU memory, this mode ensures that high-speed device memory is isolated from general system functions. This granular control is crucial for performance, as it eliminates the risk of OS-level memory "spillover" and allows the driver to optimize resource use directly.
Short Term Workaround
- Launch Jobs using numactl with bind to nodes 0,1 to prevent spillover of system process memory into GPUs during model loads
numactl --membind=0-1 --cpunodebind=0-1 <command>- However, this workaround is not comprehensive. Residual memory allocations may still accumulate, necessitating a device reset to clear them.
Long Term Resolution
- CDMM is enabled by default starting from NVIDIA Driver version 580.65.06 and greater.
- Crusoe is currently in the process of certifying NVIDIA Driver version > 580.XX.XX along with compatible CUDA version 13.0 and Firmware version 1.3.
Additional Information
NVIDIA - Understanding Memory Management on Hardware-Coherent Platforms