How-To Validate InfiniBand Hardware and Test RDMA Connectivity for GPU VMs in Crusoe Cloud

Last Updated: Oct 21, 2025

Introduction

This guide explains how to validate InfiniBand (IB) connectivity and link health on virtual machines with GPU acceleration (H100, A100, etc.) and high-performance computing workloads. Many GPU workloads rely on InfiniBand for RDMA (Remote Direct Memory Access), NCCL, NVSHMEM, and CUDA-aware MPI communication, making it critical to ensure the underlying IB interfaces and hardware are functioning correctly.

InfiniBand operates at the hardware level for ultra-low latency, high-bandwidth communication, bypassing the kernel networking stack entirely. This is why standard networking tools may not reflect the true operational state of IB hardware.

By following this guide, you can:

• Confirm IB hardware is present and active
• Understand why IB interfaces appear DOWN in standard network tools
• Validate end-to-end RDMA connectivity
• Troubleshoot common InfiniBand configuration issues
• Test GPU-specific InfiniBand features like GPUDirect RDMA

Prerequisites

• Hardware: Infiniband supported VMs with GPU acceleration and Mellanox ConnectX HCAs
• Network: Ensure to whitelist perftest ports mentioned below to perform the tests.
• Software Stack:
  • Mellanox OFED drivers (pre-installed on Crusoe cloud curated images for GPU instances)
  • infiniband-diags package
  • GPU drivers (NVIDIA) if testing GPUDirect features

Understanding InfiniBand vs. Ethernet Networking

Why IB interfaces show as "DOWN" on running the command ip a:

InfiniBand operates fundamentally differently from Ethernet:

• Hardware-Level Communication: IB uses its own addressing scheme (LIDs/GIDs) and doesn't require IP addresses for RDMA operations
• IPoIB (IP over InfiniBand): This is an optional layer that allows IP traffic over IB, but most GPU workloads bypass it entirely
• Kernel Bypass: RDMA applications communicate directly with the hardware, not through the Linux network stack

Therefore, an IB interface showing "DOWN" in ip a is completely normal and does not indicate hardware problems unless you specifically need IPoIB functionality.

Step-by-Step Validation Process

Step 1: Verify InfiniBand Hardware Detection

Check if the system recognizes InfiniBand devices:

# List all InfiniBand devices with detailed information
ibv_devinfo

Expected output:

hca_id: mlx5_1
    transport:                  InfiniBand (0)
    fw_ver:                     28.37.1700
    node_guid:                  714f:f106:21b9:2e48
    sys_image_guid:             946d:ae03:0063:0ee2
    vendor_id:                  0x02c9
    vendor_part_id:             4126
    hw_ver:                     0x0
    board_id:                   MT_0000000224
    phys_port_cnt:              1
        port:   1
            state:              PORT_ACTIVE (4)
            max_mtu:            4096 (5)
            active_mtu:         4096 (5)
            sm_lid:             1
            port_lid:           2425
            port_lmc:           0x00
            link_layer:         InfiniBand

Key indicators:

• state: PORT_ACTIVE (4) - Port is operational
• link_layer: InfiniBand - Confirms IB mode (vs. Ethernet)
• port_lid - Valid LID assigned by Subnet Manager

Step 2: Check Physical Link Status

Verify the physical state of InfiniBand ports:

# Check link-level status for all HCAs
ibstat

Expected output for InfiniBand devices:

CA 'mlx5_1'
    CA type: MT4126
    Number of ports: 1
    Firmware version: 28.37.1700
    Hardware version: 0
    Node GUID: 0x714ff10621b92e48
    System image GUID: 0x946dae0300630ee2
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 400
        Base lid: 2425
        LMC: 0
        SM lid: 1
        Capability mask: 0xa651ec48
        Port GUID: 0x714ff10621b92e48
        Link layer: InfiniBand

Note about device naming:

• mlx5_0 is typically configured for Ethernet (Rate: 200, Link layer: Ethernet)
• mlx5_1, mlx5_2, etc. are typically InfiniBand devices (Rate: 400, Link layer: InfiniBand)

Critical fields:

• State: Active - Port is ready for communication
• Physical state: LinkUp - Cable/optical connection is good
• Rate: 400 - Link speed
• Link layer: InfiniBand - Confirms this is an IB device

Step 3: Map IB Devices to Network Interfaces

Understanding the relationship between hardware and OS interfaces:

# Map InfiniBand devices to network interfaces
ibdev2netdev

Sample output:

mlx5_0 port 1 ==> ens7 (Up)
mlx5_1 port 1 ==> ibP2p0s9 (Down)
mlx5_2 port 1 ==> ibP2p0s10 (Down)

Interpretation:

• ens7 (Up) - Ethernet interface, likely for management traffic
• ibP2p0s9 (Down) - IPoIB interface, DOWN is normal for RDMA-only usage
• ibP2p0s10 (Down) - Another IPoIB interface, also DOWN normally

Step 4: Network Interface Status (Understanding the "DOWN" State) 

# Check all network interfaces
ip a

Expected IB interface output:

3: ibP2p0s9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc noop state DOWN group default qlen 256
    link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:71:4f:f1:06:21:b9:2e:49 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

Why it shows DOWN:

• No IP Configuration: The interface has no IP address assigned
• IPoIB Not Configured: Most GPU workloads don't need IP over InfiniBand
• RDMA Bypass: Applications communicate directly with the HCA hardware
• Normal Behavior: This does NOT indicate a hardware or driver problem

Step 5: RDMA Connectivity Testing

RDMA (Remote Direct Memory Access) testing validates that two nodes can communicate directly through the InfiniBand fabric without involving the operating system kernel. This is the core functionality that GPU workloads like NCCL depend on. The test measures both bandwidth and latency while confirming that the InfiniBand hardware, drivers, and fabric are working correctly.

Why this matters:

• Validates end-to-end InfiniBand connectivity between nodes
• Tests the actual data path that GPU applications will use
• Measures real-world performance you can expect from ML/AI workloads
• Bypasses IP networking entirely, testing pure RDMA functionality

Practical Example: Testing Between Two VMs

For this example, we'll test between:

test1: 204.52.27.127 (172.27.20.115) - Server node

test2: 204.52.24.106 (172.27.28.27) - Client node

Step 1: On test1 (Server node), start the bandwidth test server:

# SSH to test1 (204.52.27.127)
ssh user@204.52.27.127

# Find your InfiniBand device (should be mlx5_1 or similar)
ibstat | grep "CA '"

# Start the RDMA write bandwidth test server
ib_write_bw -d mlx5_1 -p 18515

The server will start and display output like:

************************************
* Waiting for client to connect... *
************************************

Step 2: On test2 (Client node), connect to the server:

# SSH to test2 (204.52.24.106)
ssh user@204.52.24.106

# Get the server's InfiniBand address (LID or GID)
# Option 1: Use the server's LID (if known from ibstat on server)
ib_write_bw -d mlx5_1 -p 18515 <server_lid>

# Option 2: Use the private IP if IPoIB is configured
ib_write_bw -d mlx5_1 -p 18515 172.27.20.115

# Option 3: Use the public IP
ib_write_bw -d mlx5_1 -p 18515 204.52.27.127

Expected successful output:

ib_write_bw -d mlx5_1 -p 18515 172.27.20.115
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_1
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x9ff QPN 0x012b PSN 0x10fa7b RKey 0x040500 VAddr 0x00146bb6a3e000
 remote address: LID 0x979 QPN 0x012b PSN 0xe7a7d6 RKey 0x040500 VAddr 0x007fe25b9cd000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      5000             44039.93            44014.54		   0.704233
---------------------------------------------------------------------------------------

Additional RDMA Tests

These additional RDMA tests validate different communication patterns and operations that GPU workloads might use. Each test focuses on a specific RDMA operation type, helping you understand the performance characteristics of different communication methods your applications might employ.

Latency Testing:

# Test latency instead of bandwidth - measures round-trip time for RDMA Send operations
# On server node:
ib_send_lat -d mlx5_1

# On client node (connect to server):
ib_send_lat -d mlx5_1 <server_address>

Expected output:

#bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]    t_avg[usec]    t_stdev[usec]   99% percentile[usec]   99.9% percentile[usec] 
 2       1000          1.71           9.68         1.80     	       1.82        	0.19   		2.02    		9.68   

RDMA Read Bandwidth Testing:

# Test RDMA Read operations - server provides memory that client reads from
# On server node:
ib_read_bw -d mlx5_1 -p 18516

# On client node:
ib_read_bw -d mlx5_1 -p 18516 <server_address>

RDMA Send Bandwidth Testing:

# Test RDMA Send operations - traditional message passing pattern
# On server node:
ib_send_bw -d mlx5_1 -p 18517

# On client node:
ib_send_bw -d mlx5_1 -p 18517 <server_address>

Why test different RDMA operations:

• RDMA Write: Direct memory writes, lowest CPU overhead
• RDMA Read: Remote memory reads, useful for data gathering patterns
• RDMA Send: Message-based communication, used by MPI and similar frameworks

Common Issues and Troubleshooting

perftest Commands Fail with "Couldn't get context for device"

Cause: Wrong device name or device not available 

Solution:

• List available devices: ibv_devinfo | grep hca_id
• Use the correct device name (typically mlx5_1 for InfiniBand)
• Ensure the device shows PORT_ACTIVE state

Low Bandwidth Results in perftest

Cause: Various performance factors 

Solution:

• Verify link speed matches expected rate: ibstat should show Rate: 400 for NDR
• Test with larger message sizes: ib_write_bw -d mlx5_1 -s 65536
• Check both nodes are using InfiniBand devices, not Ethernet

Connection Refused or Timeout in perftest

Cause: Network connectivity or firewall issues 

Solution:

• Verify both VMs can reach each other on management network first
• Check if custom ports (like 18515) are blocked by firewall
• Try using the private IP addresses instead of public IPs

GPUDirect Tests Show No Improvement

Cause: GPUDirect RDMA not properly configured 

Solution:

• Verify NVIDIA drivers are loaded: lsmod | grep nvidia
• Check GPU memory access: nvidia-smi
• GPUDirect may require specific driver versions or kernel modules

Summary

A properly functioning InfiniBand setup should show:

• ibv_devinfo lists devices with PORT_ACTIVE state
• ibstat shows State: Active, Physical state: LinkUp for IB devices
• Valid LID assignment (Base lid > 0, SM lid > 0)
• perftest achieves expected bandwidth/latency between nodes
• IB interfaces may show DOWN in ip a this is expected

Additional Resources

1. NVIDIA Mellanox OFED Documentation
2. NVIDIA NCCL Documentation
3. GPU-Direct RDMA
4. Performance Testing Tools (perftest)