Introduction
Training jobs on Crusoe Managed Slurm can suddenly start failing on specific nodes with an import error deep in the Python stack, most commonly through OpenCV:
import albumentations as A ... import cv2 ... ImportError: libGL.so.1: cannot open shared object file: No such file or directory srun: error: <node>: task 0: Exited with exit code 1
The confusing part is that the same job worked yesterday, and it still works on other nodes in the same nodeset. Nothing in your code changed.
The cause is how Managed Slurm compute nodes are built. Each "node" is a Kubernetes pod running Crusoe's Slurm node image. That image ships the GPU and networking stack (CUDA, drivers, fabric tooling) but not every userspace library, and it does not include the Mesa GL runtime (libgl1) that the non-headless build of OpenCV links against.
When someone installs the missing package with apt, it lands in the container's ephemeral writable layer, not in the image. The moment that pod is recreated, or the slurmd container inside it restarts, it comes back with a fresh filesystem from the base image, and the library is silently gone.
Node recreation is a routine part of the platform: node maintenance, GPU health remediation, and image updates all recreate compute pods. So an apt install done directly on a node is never durable, and the error returns on exactly the nodes that were recreated most recently. This applies to any package installed this way, not only the GL libraries, and equally to pip install into the image's system Python. libGL is the most common case because of OpenCV.
This article shows how to confirm the cause, restore the affected nodes immediately, and then make the fix durable so node recreation stops mattering.
Prerequisites
- Crusoe Managed Slurm Cluster
- kubectl Access to the Backing CMK Cluster (
crusoe kubernetes clusters get-credentials <slurm-cluster-name>) - Shared
/homeVolume (Present by Default on Managed Slurm)
Instructions
Step 1: Confirm the Cause
Check whether the failing nodes were recently recreated or restarted. Recreated pods show a much lower AGE than their healthy peers. A slurmd container that restarted in place keeps the pod's age but shows a non-zero RESTARTS count, and it has lost its writable layer the same way:
kubectl get pods -n slurm -o wide
If you only have the Slurm node name from the srun: error line, the node's Comment= field holds the pod name. Run this from the login node:
scontrol show node <NODE_NAME> | grep Comment
Then confirm the library is missing on an affected node and present on a healthy one:
kubectl exec -n slurm <WORKER_POD_NAME> -c slurmd -- sh -c "ldconfig -p | grep libGL.so.1"
No output on the failing node plus a recent pod age or restart is the full signature: the node was recreated or restarted and lost everything that had been installed into its ephemeral layer.
Step 2: Restore the Affected Nodes Now
To get jobs running again immediately, install the GL runtime packages on each affected node:
kubectl exec -n slurm <WORKER_POD_NAME> -c slurmd -- sh -c "apt-get update || true; DEBIAN_FRONTEND=noninteractive apt-get install -y libgl1 libglvnd0 libegl1 libgles2 libglib2.0-0"
ℹ️ Note: An error from
apt-get updateabout an unverified repository GPG key can appear and is harmless here; the packages install from the main Ubuntu repositories.
This fixes the node until the next time it is recreated. Do not stop here, or you will be running this command again after the next maintenance event. Continue with one of the three durable fixes below.
Step 3: Durable Fix, Option A: Switch to the Headless OpenCV Build
Compute nodes have no display, so most training jobs never use OpenCV's GUI functions (cv2.imshow and the rest of highgui). The opencv-python-headless wheel drops the GUI backends and does not link against libGL.so.1 at all, which removes the dependency instead of working around it.
albumentations already depends on the headless build. If you see this error, the full opencv-python build is also installed, directly or through another dependency, and because both provide the same cv2 module, the full build wins. Check which builds are installed:
pip list 2>/dev/null | grep -i opencv
Remove every OpenCV build first, then install only the headless one. Uninstalling one build after the other is installed deletes shared cv2 files, so do not skip the uninstall:
pip uninstall -y opencv-python opencv-contrib-python opencv-python-headless pip install opencv-python-headless
If you need the contrib modules, install opencv-contrib-python-headless instead. Pin the headless package in your requirements.txt so the next dependency install does not bring the full build back.
⚠️ Warning: Run these commands in a Python environment that survives node recreation, such as a virtual environment on shared
/homeor your container image. Apip installinto the node image's system Python is ephemeral and disappears on the next recreation, exactly like theaptinstall in Step 2.
Step 4: Durable Fix, Option B: Run Jobs in Your Own Container Image
The most robust fix is to stop depending on the node's filesystem for your job's userspace at all. Managed Slurm supports containerized jobs through Pyxis and Enroot, so you can build an image that contains your full Python environment including the GL libraries, and host it in a registry such as Crusoe Container Registry. Bake the libraries into the image at build time:
FROM <YOUR_BASE_IMAGE> RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends libgl1 libglib2.0-0 \ && rm -rf /var/lib/apt/lists/*
Then run your job in that image:
srun --container-image=<YOUR_REGISTRY>#<YOUR_IMAGE>:<TAG> python train.py
ℹ️ Note: Pyxis uses Enroot's image URI format, where
#separates the registry host from the image path (for examplenvcr.io#nvidia/pytorch:23.10-py3). A private registry also needs credentials in Enroot's credentials file (~/.config/enroot/.credentials) on the cluster.
Because the container is unpacked per job, node recreation has no effect on your dependencies.
Step 5: Durable Fix, Option C: Self-Healing Nodes via a Custom Prolog Check
If changing your Python environment or job workflow is not practical right now, you can make nodes repair themselves. Managed Slurm lets you add your own pre-job (prolog) scripts through the SlurmClusterHealthCheck custom resource, documented in Node Health Checks. Your script runs as root on each allocated node before every job, so the first job that lands on a freshly recreated node reinstalls the libraries automatically.
⚠️ Warning: Do not add a
Prolog=line to the Controller'sspec.extraConf. The documentation explicitly warns against this: it conflicts with the managed prolog dispatcher. Custom prolog scripts belong in the<cluster>-customSlurmClusterHealthCheckobject only.
Step 5a: Stage the Packages Once on Shared /home. The shared volume survives pod recreation, so downloaded packages staged there are always available to the prolog. Run this from a node that is currently missing the libraries (apt only downloads packages the node does not already have):
kubectl exec -n slurm <UNPATCHED_WORKER_POD> -c slurmd -- sh -c "mkdir -p /home/scripts/gl-debs/partial && (apt-get update || true) && apt-get install --download-only -y -o Dir::Cache::archives=/home/scripts/gl-debs libgl1 libglvnd0 libegl1 libgles2 libglib2.0-0 && chown -R root:root /home/scripts/gl-debs && chmod -R go-w /home/scripts/gl-debs && ls /home/scripts/gl-debs/*.deb | wc -l"
Expect the GL packages plus their dependency chain (a few dozen .deb files). Installing from these pre-staged packages instead of over the network keeps the pre-job step to a few seconds and removes any dependency on external package mirrors.
⚠️ Warning: The prolog runs
dpkg -ias root on every.debin this directory, on every node. Keep/home/scripts/gl-debsroot-owned and not writable by other users, or anyone who can write to it can run code as root across the cluster.
ℹ️ Note: The staged set matches the node image at the time you staged it. After a node image update, delete the directory and re-run this step from an unpatched node on the new image, so the prolog does not install mismatched package versions.
Step 5b: Add the Prolog Script to Your Cluster's Custom Health-Check Object. The <CLUSTER_NAME>-custom object exists, empty, on every cluster. Open it and add the entry below under spec.scripts, keeping any scripts that are already there:
kubectl edit schc <CLUSTER_NAME>-custom -n slurm
With only this script added, the object looks like this:
apiVersion: slurm.crusoe.ai/v1alpha1
kind: SlurmClusterHealthCheck
metadata:
name: <CLUSTER_NAME>-custom
namespace: slurm
spec:
scripts:
- name: "60-restore-gl-libs"
type: prolog
enabled: true
source: |
#!/usr/bin/env bash
# Reinstalls GL libraries on a freshly recreated node before the job starts.
# Fails open on purpose: this script must never block a job.
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
if ! ldconfig -p | grep -q 'libGL\.so\.1'; then
(
flock -w 120 9 || exit 0
if ! ldconfig -p | grep -q 'libGL\.so\.1'; then
dpkg -i /home/scripts/gl-debs/*.deb || true
ldconfig || true
fi
) 9>/tmp/gl-restore.lock || true
fi
exit 0Three details in the script matter. The export PATH line is required because prolog scripts can run with a minimal environment, and dpkg fails outright without a PATH. The flock serializes concurrent jobs landing on the same fresh node so they do not race the install, and the second ldconfig check inside the lock makes the jobs that waited skip the install. And the script always exits 0: it repairs what it can and never fails the job, because this is a convenience repair, not a health gate. A non-zero prolog exit would requeue the job.
On a node that already has the library, the script costs one ldconfig -p lookup per job start.
Changes to the -custom object propagate to all nodes within seconds, with no reconfigure step. Your script runs after Crusoe's built-in checks on every job start.
Step 5c: Verify End to End. Delete nothing in production to test this; watch the next recreated node instead. The first job scheduled onto it pauses a few seconds while the prolog installs the packages, then runs cleanly. You can confirm the script is deployed on the nodes (custom prolog scripts appear under prolog.d.custom/, with .sh appended to the name):
kubectl exec -n slurm <WORKER_POD_NAME> -c slurmd -- ls /opt/crusoe/healthcheck/prolog.d.custom/
Resolution
The import failures were caused by node recreation, not by anything in the job. Managed Slurm compute nodes are pods, and packages installed with apt on a running node live only in the container's ephemeral layer, so every recreated or restarted node reverts to the base image, which does not include the GL runtime that non-headless OpenCV requires.
Installing the packages restores a node temporarily. A durable fix removes the dependency on the node's filesystem: switch to opencv-python-headless in a persistent environment, bake libgl1 and libglib2.0-0 into your own container image, or add a custom prolog script via the SlurmClusterHealthCheck resource that reinstalls the packages from a pre-staged copy on shared /home before each job.
Example
A training team's jobs import albumentations, which pulls in OpenCV. After a routine maintenance window recreates three of their H200 nodes overnight, every job scheduled onto those three nodes fails with ImportError: libGL.so.1, while identical jobs on the untouched nodes run fine. The team had installed libgl1 manually weeks earlier, so the failure looks random until pod ages reveal that exactly the failing nodes were recreated. They stage the GL packages on /home and add the prolog script above to their -custom health-check object. The next time a node is recreated, the first job on it installs the packages in a few seconds and completes normally, with no ticket and no manual action.
Related Articles
- How-To Run NCCL Tests Using Crusoe Managed Slurm
- How-To Diagnose and Resume a Drained Slurm Node on Crusoe Managed Slurm
- FAQ: Using Enroot and Pyxis on Slurm Clusters