Nvidia GPU Operator-enabled Workloads Show in the Logs: "NVIDIA peer memory driver not detected" and "CUDA Forward Compatibility mode ENABLED"

Overview

Workloads running in environments where Nvidia GPU Operator is enabled may show the following or similar notes in their logs:

│ NOTE: CUDA Forward Compatibility mode ENABLED.                                                                                                                                                                                                                                                                                                     │
│   Using CUDA 13.0 driver version 580.65.06 with kernel driver version 570.158.08.                                                                                                                                                                                                                                                                  │
│   See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.                                                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                                                                                                    │
│ NOTE: Mellanox network driver detected, but NVIDIA peer memory driver not                                                                                                                                                                                                                                                                          │
│       detected.  Multi-node communication performance may be reduced.

This article explains the reason behind these log entries and whether or not you should be concerned about them.

Resolution

1. CUDA Forward Compatibility mode ENABLED.
   
   This is just an informational note indicating your workload is using CUDA Forward Compatibility mode.
   This mode allows higher CUDA versions to run on older Nvidia driver versions.
   Essentially, this generally means that the Nvidia driver inside the workload's containers is newer than the Nvidia driver on the host machine.
   More information can be found in Forward Compatibility 
2. Mellanox network driver detected, but NVIDIA peer memory driver not detected.

nvidia-peermem is a legacy kernel module to allow RDMA, however, currently Nvidia recommends using DMA-BUF Linux kernel module instead of nvidia-peermem.
   
   See reference: GPUDirect RDMA and GPUDirect Storage
   
   In most cases, the above note can be ignored, as DMA-BUF is already in place substituting nvidia-peermem.
   However, if any part of your custom code requires nvidia-peermem module, you can follow the above reference to get GPU Operator installed with it.

References

• Forward Compatibility 
• GPUDirect RDMA and GPUDirect Storage