Introduction
Crusoe provides two storage options for model data, each suited to different workloads:
-
Crusoe Shared Storage — NFS-backed persistent volumes (
crusoe-csi-driver-fs-sc) withReadWriteManyaccess. Ideal when multiple pods or nodes need to read the same data simultaneously, or when you need a single persistent copy shared across jobs. -
Local NVMe — node-local storage available on
s1ainstances, pre-mounted at/mnt/nvme. Ideal for single-node workloads where you want the fastest possible model load time and the data only needs to be accessible on one node.
This guide covers the local NVMe path using rclone with parallel streams, which can significantly reduce cold-start time for large model downloads on a single node. If your workload requires shared access across multiple nodes, see How To Download Data From AWS S3 Using Rclone Parallel Streams.
Prerequisites
-
kubectlconfigured with access to your Crusoe Kubernetes cluster - An
s1ainstance node in your cluster (e.g.s1a.60x) — these have local NVMe pre-mounted at/mnt/nvme - AWS credentials with read access to your S3 bucket
- The hostname of your
s1anode (runkubectl get nodesto find it)
Step-by-Step Instructions
1. Create the AWS credentials secret
Store your AWS credentials as a Kubernetes secret so they can be injected into the download pod:
kubectl create secret generic aws-credentials \ --from-literal=AWS_ACCESS_KEY_ID=<your-access-key-id> \ --from-literal=AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
2. Find your s1a node hostname
kubectl get nodes
Look for your s1a node. Copy the full hostname — you'll need it in the next step.
NAME STATUS ROLES AGE np-04b37f88-1.eu-iceland1-a.compute.internal Ready <none> 5h np-b60557ea-1.eu-iceland1-a.compute.internal Ready <none> 10d
3. Create the download Job
Create a file named download-to-nvme.yaml. Replace the following before applying:
-
<your-node-hostname>— thes1anode hostname from step 2 -
s3:your-bucket/your-prefix— your S3 source path -
/nvme/model— destination path inside the container (maps to/mnt/nvme/modelon the host)
apiVersion: batch/v1
kind: Job
metadata:
name: s3-to-nvme-download
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
hostNetwork: true
nodeSelector:
kubernetes.io/hostname: <your-node-hostname>
containers:
- name: downloader
image: rclone/rclone
command: ["/bin/sh", "-c"]
args:
- |
mkdir -p /root/.config/rclone
printf '[s3]\ntype = s3\nprovider = AWS\nenv_auth = true\nregion = us-east-1\n' \
> /root/.config/rclone/rclone.conf
rclone copy s3:your-bucket/your-prefix /nvme/model \
--transfers 16 \
--multi-thread-streams 16 \
--s3-chunk-size 64M \
--fast-list \
--buffer-size 256M \
--no-check-dest \
--stats 5s \
--log-level INFO
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-credentials
key: AWS_SECRET_ACCESS_KEY
volumeMounts:
- name: nvme-storage
mountPath: /nvme
resources:
requests:
memory: "32Gi"
cpu: "16"
limits:
memory: "64Gi"
cpu: "32"
volumes:
- name: nvme-storage
hostPath:
path: /mnt/nvme
type: Directory
Key parameters explained:
| Parameter | Purpose |
hostNetwork: true |
Bypasses the Kubernetes network overlay, improving download throughput |
--transfers 16 |
Downloads 16 files simultaneously |
--multi-thread-streams 16 |
Splits each file into 16 parallel HTTP range requests (256 total connections) |
--s3-chunk-size 64M |
Sets the S3 multipart chunk size |
--fast-list |
Uses S3 List v2 API — faster and cheaper for large buckets |
--buffer-size 256M |
In-memory read-ahead buffer per stream |
--no-check-dest |
Skips destination file existence checks for faster startup |
4. Apply the Job and monitor progress
kubectl apply -f download-to-nvme.yaml
Get the pod name:
kubectl get pods -l job-name=s3-to-nvme-download
Follow the logs:
kubectl logs -f <pod-name>
5. Verify the download
Once the job completes (STATUS: Completed), verify the files landed correctly:
kubectl run verify --rm -it --restart=Never \
--image=busybox \
--overrides='{
"spec": {
"nodeSelector": {"kubernetes.io/hostname": "<your-node-hostname>"},
"containers": [{"name": "verify", "image": "busybox",
"command": ["sh"], "stdin": true, "tty": true,
"volumeMounts": [{"name": "nvme", "mountPath": "/nvme"}]}],
"volumes": [{"name": "nvme",
"hostPath": {"path": "/mnt/nvme", "type": "Directory"}}]
}
}'
Inside the pod:
df -h /nvme # check disk usage and free space du -sh /nvme/model # verify total model size ls /nvme/model # list downloaded files
Example
The following example downloads DeepSeek-V3 (~688 GB, 163 shards) from an S3 bucket in us-east-1 to an s1a.60x node in the Iceland datacenter.
Example log output:
2026/02/24 18:42:01 INFO Starting download: s3:crusoe-model-download/DeepSeek-V3 → /nvme/model 2026/02/24 18:42:06 INFO Transferred: 14.2 GiB / 688 GiB, 2%, 2.91 GiB/s, ETA 3m58s 2026/02/24 18:42:11 INFO Transferred: 29.1 GiB / 688 GiB, 4%, 2.95 GiB/s, ETA 3m51s ... 2026/02/24 18:46:02 INFO Transferred: 688 GiB in 3m58s, 2.89 GiB/s
Note: Actual throughput varies by node type, source S3 region, and available network bandwidth. Local NVMe throughput is higher than Crusoe Shared Storage because it eliminates shared filesystem write overhead. For workloads that need the model available across multiple nodes simultaneously, Crusoe Shared Storage is the appropriate choice and can be optimized using the multi-pod approach described in How To Download Data From AWS S3 Using Rclone Parallel Streams.
Additional Resources
- How To Download Data From AWS S3 Using Rclone Parallel Streams — multi-node approach using Crusoe Shared Storage
- rclone documentation
- Crusoe s1a instance types