Containerd Failing to Start with Error "invalid disabled plugin URI cri"

Last Updated: Oct 27, 2025

Overview

After installing the NVIDIA GPU Operator on an RKE2 (Rancher Kubernetes Engine) cluster, some pods may fail to start and enter a CrashLoopBackOff state. System logs /var/log/syslog  indicate that containerd cannot load its configuration due to an invalid path reference.

Jun 20 18:27:35 rke-worker-0 containerd[26464]: containerd: failed to load TOML from /etc/containerd/config.toml: invalid disabled plugin URI "cri" expect io.containerd.x.vx

Prerequisites

1. RKE2 (Rancher Kubernetes Engine) cluster
2. NVIDIA GPU Operator
3. Containerd 2.0

Step-by-Step Instructions

During the NVIDIA GPU Operator installation, the operator configures the NVIDIA Container Runtime and assumes that containerd’s default configuration file is located at: /etc/containerd/config.toml. However, RKE-managed clusters store their containerd configuration in a different path: /var/lib/rancher/rke2/agent/etc/containerd/config.toml. As a result, containerd attempts to load a non-existent or invalid configuration file from the default location. This leads to initialization errors that prevent the GPU-related containers from starting, causing repeated pod restarts (CrashLoopBackOff). You can resolve or work around this issue using the following steps:

1. Verify if you are observing the same error
   Check your system logs to confirm if containerd is failing to load the configuration file from the default path.

   sudo cat /var/log/syslog | grep -i 'invalid disabled plugin'

   If you see an output similar to the following, you are encountering the same issue:

   Jun 20 18:27:35 rke-worker-0 containerd[26464]: containerd: failed to load TOML from /etc/containerd/config.toml: invalid disabled plugin URI "cri" expect io.containerd.x.vx

   This confirms that the NVIDIA GPU Operator is attempting to read containerd’s configuration from /etc/containerd/config.toml instead of the RKE-specific path.

2. Update the GPU Operator Configuration
   Modify the GPU Operator’s values.yaml file to reference the correct containerd configuration file used by RKE. Locate and update the toolkit section as shown below:

   toolkit:
     enabled: true
     repository: nvcr.io/nvidia/k8s
     image: container-toolkit
     version: v1.17.8-ubuntu20.04
     imagePullPolicy: IfNotPresent
     imagePullSecrets: []
     env:
     # Add the following configuration override to use RKE2 containerd config for adding the NVIDIA runtime
     # This also ensures containerd is restarted with the same RKE2 configuration
     - name: CONTAINERD_CONFIG  
       value: /var/lib/rancher/rke2/agent/etc/containerd/config.toml

3. Apply the Updated Configuration
   Upgrade (or redeploy) the NVIDIA GPU Operator using the updated values.yaml file.

   helm upgrade -n gpu-operator nvidia-gpu-operator -f values.yaml

   This will apply the updated configuration and restart the affected components with the correct containerd path.

4. Verify Pod Status
   After the upgrade completes, verify that all GPU Operator pods are running successfully:

   kubectl get pods -n gpu-operator

Additional Resources

• Configuration options for containerd
• Github Open Issue #11747
• Crusoe RKE2 Github repository