Last Updated: Jan 24, 2026
Introduction
NVIDIA Collective Communications Library (NCCL) is the standard library used for fast communication between GPUs for certain primitives (i.e. all-reduce, broadcast) that make up the core operations of AI model training jobs.
NCCL implements a graph search algorithm to automatically analyze the system's hardware topology such as NVLink, NVSwitch and NICs, to compute the most efficient communication paths for multi-GPU training. Graph search algorithm is often highly recursive, especially on GB200 NVL72, where many GPUs (or ranks in a training job) are interconnected via RDMA, in a system called NVIDIA Multi-Node NVLink (MNNVL). Because of this, your GPU nodes may require more than 2MB of thread stack during NCCL initialization.
Potential Issue
Due to a known bug, setting the stack size to unlimited may result in a crash because GNU libc changes the stack size down to 2MB, which will not be sufficient. To avoid this potential issue, Crusoe Cloud has published its GB200 NVL72 Curated Images for the VMs with the following: ulimit -s 8192 (8MB). However, you may still see training jobs fail after NCCL initialization, but prior to starting a training epoch.
Solution
When you submit a job, you will need to explicitly set the stack limit as part of the job submission, to ensure the current environment value does not get passed into the job. For example, using a container image to submit a Slurm job:
srun --container-image=/data/images/training_job.sqsh \
--container-mounts=/data:/data,/home:/home \
bash -c "
ulimit -s 8192 && \
cd /home/$USER/app && \
...