Resolving Slow SGLang Model Loading on Crusoe Shared Disks

Overview

This article provides a solution for slow model loading performance when using older versions of the SGLang framework on Crusoe Cloud, particularly on MI300x nodes with Shared Disks. Customers may experience read throughput far below the documented performance of the storage, with large models taking an unexpectedly long time to load.

Prerequisites

• Access to a Crusoe Cloud project with a provisioned Crusoe Managed Kubernetes (CMK) cluster.
• A running VM or pod with access to the docker or container runtime.
• Model weights (e.g., Deepseek-V3) stored on a Crusoe Shared Disk, accessible from the CMK cluster.

Steps

1. Step 1: Update Your Container Image
   • The multi-threaded loading feature is available in a newer version of the sglang container. Update your Kubernetes deployment YAML or container run command to use the following image or a later version:
   • lmsysorg/sglang:v0.4.10.post2-rocm630-mi30x
2. Step 2: Modify the SGLang Launch Command
   • To activate the new feature, you must add the --model-loader-extra-config flag to your sglang.launch_server command. This flag takes a JSON string that enables multi-threading and specifies the number of threads to use. A good starting point is to match the number of threads to the tensor parallelism (--tp) degree.

Example Command:

python3 -m sglang.launch_server \
    --model deepseek-ai/DeepSeek-V3 \
    --port 30000 \
    --tp 8 \
    --trust-remote-code \
    --enable-metrics \
    --model-loader-extra-config "{\"enable_multithread_load\": true, \"num_threads\": 8}"

Resolution

Applying this solution has been shown to resolve critical performance issues for this workload. In a benchmark scenario, the following improvements were observed:

• Before: Using older sglang versions resulted in slow, single-threaded I/O, causing a significant performance bottleneck when loading large models from a Shared Disk.
• After: Upgrading the container and enabling the multi-threaded loader allows the application to perform parallel reads, resolving the performance bottleneck and leading to significantly improved model loading times.

Enabling multi-threaded loading allows sglang to fully utilize the available storage bandwidth, leading to drastically faster model load times.

Additional Resources

• Crusoe Docs: Managing Shared Disks
• SGLang Project: SGLang GitHub Repository