How-To perform distributed training on Crusoe Managed Kubernetes using Ray Train and KubeRay

Last Updated: November 20th, 2025

Introduction

Ray Train simplifies the use of PyTorch and other common frameworks by abstracting away the complexities of creating, using and scaling the distributed training environment. It's fault-tolerant, provides simple and efficient parallelized checkpointing, and allows you to leverage other Ray features, such as hyper-parameter tuning with Ray Tune, and parallelized data transfer operations with Ray Data. This article provides a multi-node, multi-gpu model-training example using the common Fashion MNIST dataset which saves its checkpoints in object storage.

Prerequisites

• A CMK cluster with sufficient GPU capacity.
• Kubectl and Helm installed locally and configured to access the cluster.

• An S3-compatible object store with a pre-existing bucket and valid credentials. You can create a suitable object store on your CMK cluster itself using the Minio helm chart (ensure that the Crusoe SSD CSI driver storage class is installed on your cluster to provide the persistent volumes). For example:

  helm repo add minio https://charts.min.io/ && helm repo update
  kubectl create ns minio
  
  helm install minio-release minio/minio --n minio \
    --set mode=standalone \
    --set rootUser="minioadmin" --set rootPassword="minioadmin"

• (only for optional step 5). A container registry with credentials that allow pushing and pulling of images. We will use Crusoe Container Registry (CCR), but you could use a third-party registry such as Dockerhub or run a Harbor instance in your CMK cluster. Create a CCR registry and obtain its URL and an access token. Refer to the CCR documentation for full details about all available options. Instruction steps 1 and 2 of this article show how to create an ImagePull Secret in the namespace where your RayJobs will run.

Step-by-Step Instructions

1.  Install the KubeRay operator:

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
helm -n kuberay install kuberay-operator kuberay/kuberay-operator --create-namespace
watch kubectl -n kuberay get po #cancel the watch command when the kuberay-operator pod is Running

2. Clone the example repo:

git clone https://github.com/crusoecloud/raytrain-kuberay-example.git
cd raytrain-kuberay-example

3. Run the quick-start example (code in configmap):

Review the 'rayjob-train-code-in-configmap.yaml' file to see the RayJob object and the ConfigMap containing the PyTorch training code. The RayJob head and worker pods mount the contents of train.py from the ConfigMap and run it. Therefore, to quickly experiment with the training code, all you need to do is re-apply the yaml file (after deleting any previous RayJob objects that have the same name).

kubectl apply -f rayjob-train-code-in-configmap.yaml
watch kubectl get po 

You will see 3 new pods created immediately (the head pod and 2 workers specified by the RayCluster config in the job) and then one pod created for the job itself. Now, check the services created for the job:

The service and port number highlighted in the screenshot are for the Ray Dashboard representing the job that you have created. Set up port forwarding to it:

kubectl port-forward svc/raytrain-fashion-mnist-head-svc 8265:8265 &
# if you used the Minio example to provide the object store, do the same for its console
kubectl -n minio port-forward svc/minio-console 9001 &

Access the job's Ray Dashboard in your browser to monitor the progress of the running job:

4. Confirm that the checkpoints from the training job are successfully pushed into your object store:

From here, you can use the checkpoints as required (e.g for an inferencing task) - this is not covered by this article, but it could be done with the Ray Serve framework.

5. (Optional) Create and use a dedicated container image with your training code and Python dependencies

If you will use the same training code regularly, you can save a lot of job startup time - of the order of minutes per job - by building a custom image that contains your training code and all of its dependencies pre-installed. To do this: 

• Update your copy of the Dockerfile included in the cloned repo to suit your use case (by adding your required Python packages to the 'pip install' command or requirements.txt.
• Build the image with a suitable tag for the registry that you will push it to. In this case, the tag is for a registry in Crusoe Container Registry.

docker build -t registry.us-southcentral1-a.ccr.crusoecloudcompute.com/myregistry.abc1234/raytrain-mnist-example:v1 .
docker push registry.us-southcentral1-a.ccr.crusoecloudcompute.com/myregistry.abc1234/raytrain-mnist-example:v1

• Update rayjob-train-code-in-image.yaml with your new image tag where indicated in the head group and worker group container specs.
• Create the job with kubectl apply -f rayjob-train-code-in-image.yaml and monitor it as shown in step 3 above. 

When accessing the dashboard, you should see that the job transitions from 'Pending' to 'Running' state much more quickly than in the ConfigMap example. In both cases, the first time a particular Kubernetes node is used in a job, the creation of the pods themselves will take a while as the images are pulled from the container registry, especially if you use a registry other than Crusoe Container Registry.