How-To Install NVIDIA GPU and Network Operators with Custom Tolerations via Helm

Last Updated: Jan 15, 2025

Introduction

By default, Crusoe Managed Kubernetes (CMK) Addons do not currently support custom tolerations. To ensure GPU and Network Operator pods can schedule on nodes with custom taints (such as maintenance or "bad-node" taints), you must install these components manually using Helm instead of enabling them as CMK Addons.

Prerequisites

• A Crusoe Managed Kubernetes (CMK) cluster.

• Helm CLI installed and configured.

• Important: Ensure that the GPU Operator and Network Operator Addons are disabled in the Crusoe Cloud Console.

Step-by-Step Instructions

1. Add the NVIDIA Helm Repository
    

   helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
   helm repo update

2. Create Custom Values Files Create a file named gpu-operator-values.yaml. You must define tolerations for both the operator and the various components it deploys (drivers, container toolkit, etc.).

   Example gpu-operator-values.yaml:
    

   operator:
     tolerations:
     - key: "example.ai/bad-node"
       operator: "Exists"
       effect: "NoSchedule"
   
   daemonsets:
     tolerations:
     - key: "nvidia.com/gpu"
       operator: "Exists"
       effect: "NoSchedule"
     - key: "example.ai/bad-node"
       operator: "Exists"
       effect: "NoSchedule"
   

3. Install the GPU Operator
    

   helm install gpu-operator nvidia/gpu-operator \
     -f gpu-operator-values.yaml \
     --namespace gpu-operator \
     --create-namespace
   

4. Install the NVIDIA Network Operator Create a file named network-operator-values.yaml and apply similar toleration logic to ensure the network stack initializes on tainted nodes.
    

   helm install nvidia-network-operator nvidia/nvidia-network-operator \
     -f network-operator-values.yaml \
     --namespace nvidia-network-operator \
     --create-namespace

Validation

Verify that the pods are successfully scheduled on your tainted nodes:

kubectl get pods -A -o wide | grep -E 'gpu-operator|network-operator'

Impact and Considerations

• Downtime: Updating these Helm charts triggers a rolling restart of the operator pods. This process is not expected to cause downtime or interruptions to running training jobs or GPU workloads.

• Component Coverage: Most components within these charts support customizable tolerations in values.yaml. If specific pods do not adopt the toleration, verify that the tolerations field is correctly placed under that specific component's hierarchy in the Helm chart documentation.

Additional Resources

• Nvidia Network Operator Helm Chart
• NVIDIA GPU Operator