How-To Fix NVIDIA Container Toolkit CrashLoopBackOff Due to Containerd Configuration Issue on Crusoe RKE2 Cluster

Last Updated: Dec 24th, 2025

Introduction

This guide addresses a common issue where the nvidia-container-toolkit daemonset enters a CrashLoopBackOff state on RKE2-based Kubernetes clusters. The root cause is a misconfigured containerd configuration file that prevents containerd from starting, which in turn prevents the NVIDIA container toolkit from connecting and completing its setup.

Users experiencing this issue will observe the following:

• NVIDIA container toolkit pods restarting continuously
• containerd service failing to start with exit code 1
• GPU nodes unable to run GPU workloads
• Error messages indicating inability to connect to containerd socket
• Pod logs showing: unable to dial: dial unix /runtime/sock-dir/containerd.sock: connect: connection refused
• systemctl status containerd showing: Active: activating (auto-restart) (Result: exit-code)
• Nodes may be cordoned due to GPU unavailability

By following this guide, you will restore containerd functionality and allow the NVIDIA container toolkit to configure GPU support properly.

Prerequisites

• SSH access to affected Kubernetes worker nodes
• Root or sudo privileges on the nodes
• Basic familiarity with Linux text editors (nano or vi)
• kubectl access to check pod status

Root Cause

The issue stems from an incorrectly formatted disabled_plugins directive in the containerd configuration file (/etc/containerd/config.toml). The configuration contains:

disabled_plugins = ["cri"]

This is invalid syntax. containerd expects the full plugin URI format:

disabled_plugins = ["io.containerd.grpc.v1.cri"]

However, the CRI (Container Runtime Interface) plugin is essential for Kubernetes operation and should not be disabled.

Step-by-Step Instructions

1. Identify Affected Nodes

From your local machine with kubectl access:

# List nodes with failing nvidia-container-toolkit pods
kubectl get pods -n gpu-operator -o wide | grep nvidia-container-toolkit | grep -i crash

# List cordoned nodes
kubectl get nodes | grep SchedulingDisabled

Note the node names - you'll need to SSH into each one.

2. SSH into an Affected Node

ssh ubuntu@<node-hostname>

Replace <node-hostname> with the actual hostname from Step 1 (e.g., crusoe-rke-worker-12).

3. Verify containerd Status

Check the containerd service status:

sudo systemctl status containerd

Expected output (failing state):

root@crusoe-rke-worker-12:~# sudo systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2025-10-15 13:53:52 UTC; 384ms ago
       Docs: https://containerd.io
    Process: 1696416 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 1696418 ExecStart=/usr/bin/containerd (code=exited, status=1/FAILURE)
   Main PID: 1696418 (code=exited, status=1/FAILURE)
        CPU: 30ms

Key indicators of the issue:

• Active: activating (auto-restart) - Service is continuously restarting
• Result: exit-code - containerd is exiting with an error
• code=exited, status=1/FAILURE - The process failed to start

If you see Active: active (running), containerd is already working - skip to the next node.

4. Check containerd Logs (Optional)

sudo journalctl -u containerd -n 50 --no-pager | grep -i "invalid\|error\|failed"

Look for error messages like:

containerd: failed to load TOML from /etc/containerd/config.toml: invalid disabled plugin URI "cri" expect io.containerd.x.vx

5. Locate and Backup the containerd Config

The config file is located at /etc/containerd/config.toml

Create a backup before making changes:

sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.backup

# Verify backup was created
ls -la /etc/containerd/config.toml*

Expected output:

-rw-r--r-- 1 root root 1234 Oct 14 08:00 /etc/containerd/config.toml
-rw-r--r-- 1 root root 1234 Oct 15 14:00 /etc/containerd/config.toml.backup

6. Edit the containerd Configuration File

Open the config file for editing:

sudo nano /etc/containerd/config.toml

Find the problematic line:

1. Press Ctrl+W to search
2. Type: disabled_plugins
3. Press Enter

You should see:

disabled_plugins = ["cri"]

Replace with the correct format:

Change ["cri"] to ["io.containerd.grpc.v1.cri"] so the line looks like this:

disabled_plugins = ["io.containerd.grpc.v1.cri"]

Save and exit:

1. Press Ctrl+X
2. Press Y to confirm
3. Press Enter to save

7. Restart containerd Service

IMPORTANT: This restarts only the containerd service, NOT the entire node.

sudo systemctl restart containerd

Wait 2-3 seconds, then verify:

sudo systemctl status containerd

Expected output (successful):

● containerd.service - containerd container runtime
     Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2025-10-14 08:29:46 UTC; 8s ago
     ...
     Main PID: 28890 (containerd)

Look for Active: active (running) - this confirms containerd is now working.

8. Verify NVIDIA Container Toolkit Recovery

From your local machine:

# Watch the nvidia-container-toolkit pod on this node
kubectl get pods -n gpu-operator -o wide | grep <node-hostname>

# The pod should transition from CrashLoopBackOff to Running within 1-2 minutes
# You can watch it live:
kubectl get pods -n gpu-operator -w

Expected output:

nvidia-container-toolkit-daemonset-xxxxx   1/1   Running   0   30s   <node-ip>   <node-hostname>

The restart count will remain high, but the status should be Running now.

9. Uncordon the Node

Once the nvidia-container-toolkit pod is running:

kubectl uncordon <node-hostname>

Verify:

kubectl get nodes | grep <node-hostname>

The node should show Ready status without SchedulingDisabled.

10. Repeat for Remaining Nodes

Repeat Steps 2-9 for each affected node identified in Step 1.

Example

Scenario: Multiple GB200 nodes with failing nvidia-container-toolkit

This example demonstrates multiple GB200 nodes experienced nvidia-container-toolkit CrashLoopBackOff issues.

Initial state:

$ kubectl get pods -n gpu-operator | grep nvidia-container-toolkit
nvidia-container-toolkit-daemonset-mql4f   0/1   CrashLoopBackOff   3247   14d
nvidia-container-toolkit-daemonset-b6k6w   0/1   CrashLoopBackOff   3156   14d
[... multiple failing pods ...]

Resolution applied to crusoe-rke-worker-12:

1. Checked containerd status:

root@crusoe-rke-worker-12:~# sudo systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2025-10-15 13:53:52 UTC; 384ms ago
       Docs: https://containerd.io
    Process: 1696416 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 1696418 ExecStart=/usr/bin/containerd (code=exited, status=1/FAILURE)
   Main PID: 1696418 (code=exited, status=1/FAILURE)
        CPU: 30ms

2. Edited /etc/containerd/config.toml:
   • Found the line: disabled_plugins = ["cri"]
   • Changed it to: disabled_plugins = ["io.containerd.grpc.v1.cri"]
3. Restarted containerd service:

root@crusoe-rke-worker-12:~# sudo systemctl restart containerd
root@crusoe-rke-worker-12:~# sudo systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2025-10-15 14:00:15 UTC; 5s ago
       Docs: https://containerd.io
    Process: 1698543 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 1698544 (containerd)
      Tasks: 16
     Memory: 17.8M
        CPU: 69ms
     CGroup: /system.slice/containerd.service
             └─1698544 /usr/bin/containerd

Result:

$ kubectl get pods -n gpu-operator | grep nvidia-container-toolkit | grep crusoe-rke-worker-12
nvidia-container-toolkit-daemonset-xxxxx   1/1   Running   3247   14d

The nvidia-container-toolkit pod successfully connected to containerd and completed its configuration. The node was then uncordoned and returned to service.

Troubleshooting

Issue 1: Config file not found at /etc/containerd/config.toml

Symptom: cat: /etc/containerd/config.toml: No such file or directory

Resolution: Check for RKE2-specific location:

cat /var/lib/rancher/rke2/agent/etc/containerd/config.toml

If found, edit this file instead. If neither file exists, containerd may not be installed properly - contact support.

Issue 2: containerd still fails after editing config

Symptom: After editing and restarting, containerd still shows exit-code status

Resolution:

Check for syntax errors in the config:

sudo containerd config dump

Review detailed error logs:

sudo journalctl -u containerd -n 100 --no-pager

If you see other errors, restore the backup and contact support:

sudo cp /etc/containerd/config.toml.backup /etc/containerd/config.toml
sudo systemctl restart containerd

Issue 3: NVIDIA toolkit pod still crashing after containerd is running

Symptom: containerd is active (running) but the toolkit pod continues crashing

Resolution:

Check the toolkit pod logs for different errors:

kubectl logs -n gpu-operator <pod-name>

Delete the pod to force a fresh restart:

kubectl delete pod -n gpu-operator <pod-name>

If issues persist, this may be a different problem - contact support with the new pod logs.

Issue 4: Cannot find the disabled_plugins line

Symptom: The config file doesn't contain disabled_plugins = ["cri"]

Resolution: The issue may already be fixed, or the error is different. Check:

sudo journalctl -u containerd -n 200 --no-pager | grep -i error

Look for the actual containerd error message and contact support with those logs.

Additional Resources

• NVIDIA GPU Operator Documentation
• containerd Configuration Guide
• RKE2 containerd Configuration

Related Articles

• How-to Install GPU and Network Operators on Kubernetes Clusters
• How-To Capture NVIDIA Bug Report in CMK
• Run NCCL Tests On Crusoe Managed Kubernetes (CMK) Cluster

Still Need Help? If you continue to experience issues after following this guide, please contact support and include:

• Output of systemctl status containerd
• Output of journalctl -u containerd -n 100
• Output of kubectl logs -n gpu-operator <pod-name>
• The modified /etc/containerd/config.toml file