How-To Diagnose and Resume a Drained Slurm Node on Crusoe Managed Slurm

Introduction

Crusoe Managed Slurm runs on top of Crusoe Managed Kubernetes (CMK). Each Slurm worker node is a pod running on a GPU VM in a CMK node pool, meaning a "drained Slurm node" is actually a Kubernetes pod backed by a physical GPU host. When Slurm drains a node, it marks it as unavailable for new job scheduling and records a reason string. The drain can be triggered automatically by the Slinky operator (e.g. when the underlying K8s node is cordoned), by a job failure, or manually by an administrator.

Before resuming a drained node, you must verify the underlying hardware and software stack are actually healthy. A node that looks idle at the Slurm layer can still have GPU memory errors, NVLink faults, hung NFS mounts, or D-state processes that will cause the next job to fail immediately. This article walks through every required health check and the exact resume procedure.

The Comment= field on a drained Slurm node contains the Kubernetes pod name and namespace needed to exec into the worker for inspection. This is the bridge between the Slurm layer and the Kubernetes layer.

⚠️ Warning: Do not skip or short-circuit any health check, especially dcgmi diag -r 2. A node that passes idle checks can still fail under GPU load. All six checks must be clean before resuming.

Prerequisites

• SSH Access to the Slurm Login Node
• Kubeconfig for the CMK Cluster
• kubectl Installed and Configured on Your Local Machine
• KUBECONFIG Environment Variable Set to the Cluster Kubeconfig File
• Slurm User Account with scontrol Privileges on the Login Node

Instructions

Step 1: Save Pre-Action Cluster State

Before making any changes, capture the current state of the cluster from your local machine. This gives you a baseline for the support ticket and a rollback reference.

export KUBECONFIG=<path-to-your-kubeconfig>.yml
kubectl get pods -n slurm -o wide > ~/slurm-pre-action-pods.txt
kubectl get nodes -o wide > ~/slurm-pre-action-k8s-nodes.txt

Verify the kubeconfig loaded correctly:

kubectl config current-context

This should return your cluster name. If it returns an error or falls back to localhost:8080, the kubeconfig path is wrong. Use the absolute path rather than ~:

export KUBECONFIG=/Users/<your-username>/<your-kubeconfig>.yml

Step 2: Identify the Drained Node

From the Slurm login node, inspect the node state:

scontrol show node <node-name>

Or list all nodes to find drained ones:

scontrol show nodes | grep -A5 -iE 'DRAIN|DOWN'

Note the following fields:

• State= look for DRAIN, DOWN+DRAIN, or IDLE+DYNAMIC_NORM with Reason=Not responding
• Reason= the drain reason message and timestamp
• Comment= contains the Kubernetes pod name and namespace in JSON format, e.g.:

Comment={"namespace":"slurm","podName":"<cluster-name>-h100-0","node":"np-xxxxxxxx-1.us-southcentral1-a.compute.internal"}

Extract the podName and namespace values. You will need them for Step 3.

Save the full node output:

scontrol show nodes > ~/slurm-node-state-before.txt

Step 3: Exec into the Worker Pod

From your local machine, exec into the worker pod identified in Step 2:

kubectl exec -it <podName> -n <namespace> -- /bin/bash

Example:

kubectl exec -it <cluster-name>-h100-0 -n slurm -- /bin/bash

You should land at a root shell inside the worker pod. All checks in Steps 4–9 run from inside this pod.

Step 4: Check for Stuck D-State Processes

ps -eo pid,stat,wchan,cmd | awk '$2 ~ /D/'
pgrep -af slurmstepd

• No output from either command — ✅ clean, continue.
• D-state processes found — the node likely needs a pod restart. Do not resume. Exit the pod and delete it:

kubectl delete pod <worker-pod-name> -n slurm

Wait for the pod to restart (kubectl get pods -n slurm -w), then restart this article from Step 2.

• Orphaned slurmstepd processes found: kill them, then continue:

kill -9 <pid>

Step 5: Check NFS / Shared Filesystem Connectivity

df -h /home
ls /home
stat /home
mount | grep nfs

All four commands should return immediately without hanging.

• Commands return normally: ✅ clean, continue.
• Commands hang or time out: NFS is unhealthy. Do not resume. Contact Crusoe Support with the mount and df output. Also check any other shared mounts the cluster uses (e.g. /scratch, /shared):

mount | grep nfs
showmount -e <nfs-server> 2>/dev/null

A hung NFS mount will cause D-state processes and job kill failures on the next workload.

Step 6: Check for Post-Boot GPU Errors

dmesg -T | grep -iE 'NVRM|xid' | awk '{print $1, $2, $3, $4, $5}' | sort -u

Save the output:

dmesg -T | grep -iE 'NVRM|xid' | awk '{print $1, $2, $3, $4, $5}' | sort -u > /tmp/dmesg-gpu.txt

• Timestamps match boot time only: ✅ benign driver initialization noise, continue.
• Errors appear after boot time: do not resume. Contact Crusoe Support with the full dmesg output and note the Xid error codes. Pay particular attention to:
  • Xid 79: GPU fallen off the bus
  • Xid 13, 31, 63: ECC-related errors

ℹ️ Note: Cross-reference the GPU error timestamps against the node's BootTime= field from scontrol show node to distinguish boot-time noise from runtime faults.

Step 7: Check ECC Errors

nvidia-smi -q -d ECC | grep -A1 -iE 'aggregate|remapped|retired|sram|dram uncorrect'

• All values are 0 and SRAM Threshold Exceeded: No — ✅ clean, continue.
• Any non-zero value — do not resume. Contact Crusoe Support with the nvidia-smi -q -d ECC output. Remapped or retired pages indicate physical VRAM degradation.

Step 8: Check NVLink Errors

nvidia-smi nvlink -e

• All Replay, Recovery, and CRC counters are 0 across all GPUs and links: ✅ clean, continue.
• Any non-zero counter: do not resume. Contact Crusoe Support with the nvidia-smi nvlink -e output. NVLink errors under load will cause NCCL job failures that are difficult to diagnose after the fact.

Step 9: Run GPU Stress Test (Mandatory Gate)

This is the required final check before any resume. It stress-tests SMs, memory, and PCIe across all GPUs and takes approximately 5 minutes. A node that looks clean at idle can fail under load.

dcgmi diag -r 2 | tee /tmp/dcgmi-diag.txt

• All tests show Pass — ✅ proceed to resume.
• Any Fail result — do not resume. Contact Crusoe Support with the full output from /tmp/dcgmi-diag.txt.

⚠️ Warning: Do not skip dcgmi diag -r 2 even if all prior checks are clean. This is the only check that exercises the GPU under realistic compute load.

Decision Reference

Before proceeding to resume, verify every check passed using this summary table:

Step 10: Resume the Node

Once all six checks are clean, exit the worker pod and exec into the Slurm controller pod from your local machine:

kubectl get pods -n slurm | grep controller
kubectl exec -it <controller-pod-name> -n slurm -- /bin/bash

Resume the node with a timestamped reason documenting what was checked:

scontrol update nodename=<node-name> state=resume reason="<YYYY-MM-DD> health checks clean: dcgmi -r2 pass, ECC/NVLink zero, no Xid errors, NFS ok"

ℹ️ Note: state=resume is only valid when the node is in DRAIN or DOWN state. If you see slurm_update error: Invalid node state transition, the node is already IDLE and does not need to be resumed.

Verify the state transition:

scontrol show node <node-name> | grep State

Expected output: State=IDLE or State=ALLOCATED (if a job was immediately scheduled).

Step 11: Post-Resume Verification

From the login node, confirm the cluster view is consistent:

sinfo
scontrol show node <node-name>

Save the final state:

scontrol show nodes > ~/slurm-node-state-after.txt

Note in the support ticket: what drain reason was found, which checks were run, what (if anything) was found, and that the node was resumed. If the node had any boot-time GPU errors (e.g. FECS PRI errors), add: "If this node drains again, escalate directly to hardware — do not re-resume."

Related Articles

• Slurm Command Basics
• Getting Started with SLURM on Crusoe Cloud
• How-To Recover Slurm Nodes From a Drain State
• Nvidia Driver Upgrade Causing Invalid Slurm Compute Nodes