Introduction
Every Crusoe GPU VM has one Ethernet network interface for general-purpose traffic: storage mounts, checkpoint writes, SSH, telemetry. It sits alongside the InfiniBand interfaces that NCCL uses for GPU-to-GPU communication. Inside the VM, the Ethernet interface is driven by mlx5_core and presents 11 combined TX/RX queues. You can lower the queue count from inside the VM with ethtool -L, but you cannot raise it above 11.
Each of those 11 queues has its own MSI-X completion interrupt (mlx5_comp0 through mlx5_comp10). When the driver initialises, it pins those interrupts sequentially to logical CPUs 0 through 10. With hyperthreading enabled, that means all Ethernet packet processing for the VM, hard IRQs, NAPI polling and the resulting softirq work, lands on just six physical cores.
On its own, that layout is fine. It becomes a problem when a CPU-hungry workload also runs on those same CPUs. Training data loaders, tokenisers and checkpoint serialisers happily spread across every core the scheduler gives them, including cores 0–10. When they do, network softirq work and training compute fight for the same six cores. The kernel's NAPI poll budget gets exhausted before the queues are drained, packets wait, and both storage throughput and training step time suffer, even though the NIC itself is barely loaded.
The intuitive fix is "give me more queues". That does not help. More queues would spread interrupts across more CPUs, but if training still runs on those CPUs, you have the same contention on more cores. The actual fix is to separate the two: pin the NIC interrupts to CPUs your training job does not use, and keep the training job off the interrupt CPUs. Both are guest-side changes you can make yourself, and this article shows you how to confirm the problem, apply the fix, and make it persist.
Prerequisites
- Root or sudo Access on the VM
- sysstat Package Installed for mpstat (Diagnosis Only)
- Choosing A Known Set of CPUs Your Training Workload Should Not Use
- irqbalance Inactive or Masked (Verified in Step 1)
Instructions
Step 1: Identify the Ethernet VF and its interrupt layout
Find the Ethernet interface (not the InfiniBand ones), its PCI address, and confirm the queue count. Replace ens7 with your interface name throughout.
ethtool -i ens7 | grep bus-info
ethtool -l ens7You should see Combined: 11 under both "Pre-set maximums" and "Current hardware settings". That is the expected value for this interface, not a misconfiguration.
Now list the 11 completion interrupts and where they are pinned:
PCI=$(ethtool -i ens7 | awk '/bus-info/{print $2}')
for irq in $(grep -E "mlx5_comp[0-9]+@pci:${PCI}" /proc/interrupts | awk -F: '{gsub(/ /,"",$1); print $1}'); do
echo "IRQ $irq affinity=$(cat /proc/irq/$irq/smp_affinity_list) effective=$(cat /proc/irq/$irq/effective_affinity_list)"
done
systemctl is-active irqbalanceOn a freshly provisioned VM you will typically see IRQs pinned one-to-one to CPUs 0-10 and irqbalance reporting inactive.
Note: If irqbalance is active, it will periodically overwrite any affinity you set by hand. Stop and mask it (sudo systemctl disable irqbalance; sudo systemctl stop irqbalance) before continuing, or your pins will not stick.
Step 2: Confirm you actually have contention
Do not tune blind. Three signals together tell you whether interrupt/compute contention is real on your VM. Take these measurements while your training job is running and generating storage traffic. On an idle VM every check below will correctly report nothing wrong.
Signal 1: time_squeeze is climbing on the IRQ CPUs. Column 3 of /proc/net/softnet_stat counts, in hex, how often a CPU exhausted its NAPI poll budget with packets still waiting. Sample it twice, 10 seconds apart, and show the 15 busiest CPUs:
snap() { n=0; while read -r a b c rest; do printf 'cpu%-4s %d\n' "$n" "0x$c"; n=$((n+1)); done < /proc/net/softnet_stat; }
snap > /tmp/sq1; sleep 10; snap > /tmp/sq2
paste /tmp/sq1 /tmp/sq2 | awk '{d=$4-$2; printf "%-7s +%-6d (cumulative %d)\n", $1, d, $4}' | sort -k2 -t+ -nr | head -15A healthy machine shows +0 everywhere. If CPUs 0-10 show hundreds of squeezes per 10 seconds while the rest of the machine shows none, packets are waiting because those CPUs are too busy to drain the queues.
Signal 2: softirq time is concentrated on CPUs 0-10 and those CPUs are also busy with user work.
mpstat -P ALL 10 1 | awk 'NR>3 && $2 ~ /^[0-9]+$/ {printf "cpu%-4s usr=%5.1f sys=%5.1f soft=%5.1f idle=%5.1f\n", $2, $3, $5, $8, $12}' | head -14The pattern you are looking for: soft in the 5-20% range on CPUs 0-10 versus well under 1% elsewhere, combined with usr in the 40-70% range on those same CPUs. High softirq on idle CPUs is not a problem. High softirq on CPUs that are also running your training job is.
Signal 3: the link is nowhere near saturated.
read -r rx1 tx1 < <(awk '$1=="ens7:"{print $2, $10}' /proc/net/dev); sleep 10
read -r rx2 tx2 < <(awk '$1=="ens7:"{print $2, $10}' /proc/net/dev)
echo "rx $(( (rx2-rx1)*8/10/1000000 )) Mbit/s tx $(( (tx2-tx1)*8/10/1000000 )) Mbit/s"If you see squeezes and high softirq while using only a few percent of the interface's bandwidth, the bottleneck is CPU scheduling, not queue count. Eleven queues comfortably handle that throughput on dedicated cores. Proceed to Step 3.
Tip: If squeezes are near zero and softirq is low even under full training load, you do not have this problem. Stop here. Repinning will not make anything faster.
Step 3: Pin the NIC interrupts to CPUs your training job does not use
Pick 11 logical CPUs that your workload explicitly excludes/does not use.. Good candidates are CPUs on the second NUMA node if your training job is confined to the first, or a block of hyperthread siblings you are willing to reserve. Check the interface's NUMA affinity first so you stay local where possible:
cat /sys/class/net/ens7/device/numa_node
lscpu -e | head -20Note: If numa_node returns -1, no NUMA hint is exposed for the interface. In that case, prefer CPUs on the same NUMA node your storage-consuming processes run on, and avoid splitting hyperthread siblings between IRQ work and compute.
Assign one IRQ per CPU. This example moves them to CPUs 160-170. Substitute your own list:
PCI=$(ethtool -i ens7 | awk '/bus-info/{print $2}')
TARGET=(160 161 162 163 164 165 166 167 168 169 170)
i=0
for irq in $(grep -E "mlx5_comp[0-9]+@pci:${PCI}" /proc/interrupts | awk -F: '{gsub(/ /,"",$1); print $1}'); do
echo "${TARGET[$i]}" | sudo tee /proc/irq/$irq/smp_affinity_list > /dev/null
i=$((i+1))
doneRe-run the loop from Step 1 to confirm. smp_affinity_list should immediately show your target CPUs for all 11 IRQs.
Note: effective_affinity_list may keep showing the old CPU for some IRQs. That is normal. The kernel migrates an interrupt to its new CPU the next time that interrupt fires, so queues with no traffic since your change have not moved yet. Push some traffic through the interface (reading a few GB from your storage mount to /dev/null is enough) and re-check. All 11 will then report the new CPUs. Do not "fix" this by re-writing the affinity.
Warning: Affinity set via /proc/irq does not survive a reboot, and IRQ numbers can change between boots. Do not hard-code IRQ numbers in scripts. Always rediscover them from /proc/interrupts as shown above. See Step 5 for persistence.
Step 4: Keep the training job off the interrupt CPUs
Step 3 only holds if nothing else lands on the CPUs you just dedicated to interrupts. If your training processes float freely across all CPUs, they will drift back onto the IRQ CPUs and you are back where you started. Constrain them using whichever method matches how you launch jobs:
- Direct launch: wrap the process: taskset -c 0-159 python train.py (excludes CPUs 160-170 from the example above; adjust to your layout)
-
Slurm: reserve the IRQ CPUs for system use with CpuSpecList=160-170 on the node definition in slurm.conf so jobs are never scheduled onto them, and use --cpu-bind in srun for tighter placement.
⚠️ Verify: confirm against your Slurm version's cgroup configuration - Containers: set --cpuset-cpus=0-159 on the container runtime or a cpuset in the pod spec
-
Kernel-level: add isolcpus=160-170 to the kernel command line and reboot. This is the heaviest option and also removes those CPUs from general scheduling and load balancing, so only use it if the lighter options are not available to you
Tip: Once the NIC IRQs are on dedicated CPUs and the job is fenced off them, time_squeeze on the IRQ CPUs should drop to near zero and idle on CPUs 0-10 should rise. Re-run Step 2 to confirm before declaring victory.
Step 5: Make the IRQ pinning persistent
Create a oneshot systemd service that rediscovers the IRQs and pins them at boot. Save as /etc/systemd/system/nic-irq-affinity.service:
[Unit]
Description=Pin Ethernet completion IRQs to dedicated CPUs
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pin-nic-irqs.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.targetAnd the script at /usr/local/sbin/pin-nic-irqs.sh (make it executable with chmod +x):
#!/bin/bash
IFACE=ens7
TARGET=(160 161 162 163 164 165 166 167 168 169 170)
PCI=$(ethtool -i "$IFACE" | awk '/bus-info/{print $2}')
i=0
for irq in $(grep -E "mlx5_comp[0-9]+@pci:${PCI}" /proc/interrupts | awk -F: '{gsub(/ /,"",$1); print $1}'); do
echo "${TARGET[$i]}" > "/proc/irq/$irq/smp_affinity_list"
i=$((i+1))
doneEnable and start it:
sudo systemctl daemon-reload
sudo systemctl enable nic-irq-affinity.service
sudo systemctl start nic-irq-affinity.serviceNote: If you would rather leave the hard IRQs where they are and only spread the softirq half of the work, Receive Packet Steering (/sys/class/net/ens7/queues/rx-*/rps_cpus) can fan NAPI processing out to additional CPUs. It is a reasonable complement but not a substitute. RPS adds an inter-CPU hop per packet, and the cleaner fix is dedicated IRQ CPUs as described above.
Step 6: Take checkpoint writes off the critical path
IRQ pinning fixes the contention, but checkpoint writes over NFS still hold your training processes while multi-gigabyte files stream out over the Ethernet interface. If your VM has local NVMe, stage checkpoints there first and copy to NFS in the background, so the training loop resumes as soon as the local write completes:
# write the checkpoint to local NVMe (fast, no network involved)
torch.save(state, "/mnt/nvme/ckpt/step_001000.pt")
# then hand the copy off to a background process
nohup rsync -a --remove-source-files /mnt/nvme/ckpt/ /mnt/nfs/ckpt/ > /dev/null 2>&1 &Adjust paths to your mount points. Most training frameworks also have an async or background checkpoint option that does the same thing natively. Prefer that if yours offers it.
Warning: Local NVMe on a VM is ephemeral. Anything not yet copied to shared storage is lost if the VM is stopped or recreated. Only treat a checkpoint as durable once it has landed on NFS. Keep the staging directory small and make the background copy retry on failure.
Example
An 8x H100 training node with 176 vCPUs mounts a shared NFS filesystem for datasets and writes multi-gigabyte checkpoints every few hundred steps. All of that traffic goes over the single Ethernet interface; the InfiniBand interfaces carry only NCCL collectives.
Under load, mpstat shows CPUs 0-10 at roughly 55-65% user time (data loader workers) plus 8-17% softirq, while the remaining 165 CPUs sit at under 0.3% softirq with 30% idle headroom. softnet_stat shows 100-200 squeeze events per 10 seconds on CPUs 0-10 and none anywhere else. Link throughput during checkpointing peaks at about 7 Gbit/s on a 200 Gbit/s interface.
That is the pattern we are looking for. The interface has capacity to spare, but the six cores servicing it are shared with compute. Moving the 11 mlx5_comp IRQs to CPUs 160-170 and reserving those CPUs from the job scheduler drops squeezes to zero and returns roughly 10% of CPU time on cores 0-10 to the training workers. Staging checkpoints to local NVMe then takes the remaining NFS write time off the training loop entirely, with no change to queue count or instance type.