How-To Reconfigure Docker Storage to Prevent “No Space Left on Device” Errors on GPU Instances

Introduction

This guide explains how to resolve and prevent no space left on device errors when using Docker on GPU cloud instances, especially when pulling large containers like NVIDIA’s NeMo, PyTorch, or TensorFlow. These errors typically occur because Docker stores images and layers on the OS disk (/var/lib/docker), which fills up quickly. By moving Docker's storage to a larger attached disk (e.g., /raid0), you can avoid failures related to disk exhaustion and improve container management.

Prerequisites

• Root or sudo access to the VM
• Docker already installed
• An attached secondary disk (e.g., 500GB NVMe) mounted or mountable at /raid0
• Your workloads involve large GPU containers (e.g., NeMo, PyTorch with CUDA/DALI)

Step-by-Step Instructions

Step 1: Identify and Format the Secondary Disk

If the secondary disk (like NVMe) isn't already formatted and mounted:

# Identify NVMe disks
nvme list

# Format the first NVMe device (WARNING: this will erase all data)
sudo mkfs.ext4 /dev/nvme0n1

# Create a mount point and mount the disk
sudo mkdir -p /raid0
sudo mount /dev/nvme0n1 /raid0

💡 If you use RAID0 across multiple disks, ensure the array is created first using mdadm or automated startup scripts provided by your cloud provider.

Step 2: Stop Docker

Before reconfiguring storage, stop Docker:

sudo systemctl stop docker

Step 3: Create a New Docker Storage Directory

sudo mkdir -p /raid0/docker
sudo chown root:root /raid0/docker

Step 4: Point Docker to the New Data Directory

Edit the Docker daemon configuration:

sudo vi /etc/docker/daemon.json

Paste the following JSON configuration:

{
    "data-root": "/raid0/docker",
    "default-ulimits": {
        "memlock": {
            "Name": "memlock",
            "Soft": -1,
            "Hard": -1
        }
    },
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

 data-root: tells Docker to use the new disk
 memlock ulimits: ensures CUDA workloads don’t run into locked memory errors
 nvidia runtime: required for GPU-accelerated containers

Step 5: Restart Docker and Validate

sudo systemctl start docker
docker info | grep "Docker Root Dir"

You should see:

Docker Root Dir: /raid0/docker

Step 6: Test by Pulling a Large Container

docker pull nvcr.io/nvidia/nemo:25.04

This time, the image should download and extract successfully using the space on /raid0.

Example

Scenario: You are pulling NVIDIA’s NeMo container

Without reconfiguring Docker, the following error may occur:

failed to register layer: write /usr/...: no space left on device

After performing the steps in this guide, pulling the same container works without issue:

docker pull nvcr.io/nvidia/nemo:25.04

Result:

Status: Downloaded newer image for nvcr.io/nvidia/nemo:25.04

Troubleshooting & Common Issues

Docker still uses /var/lib/docker

• Cause: Docker was restarted before daemon.json was edited correctly.
• Fix: Double-check /etc/docker/daemon.json and ensure the data-root path is correctly set. Then restart Docker.

mount: /raid0 is busy

• Cause: /raid0 is already mounted or used by another process.
• Fix: Use lsof | grep /raid0 or fuser -vm /raid0 to identify processes, then unmount and retry.

Ephemeral data lost after reboot

• Cause: You’re using ephemeral disks (e.g., Crusoe /raid0), which are not persistent across reboots.
• Fix: Add disk mount logic to your startup script or use persistent block storage if needed.

NeMo still fails to run even after pulling

• Cause: Missing NVIDIA runtime or CUDA driver version mismatch.
• Fix: Make sure nvidia-smi shows valid output and that the nvidia-container-runtime is installed.

Additional Resources

• Docker Storage Configuration
• NVIDIA Container Toolkit
• How to Create RAID0 on NVMe Disks
• Docker System Cleanup