OpenAI-compatible multi-node model serving on CMK with Ray Serve and vLLM

Last Updated: March 3rd, 2026

Introduction

Ray Serve on Crusoe Managed Kubernetes allows you to serve simple and complex model compositions across multiple worker nodes in a quick and scalable fashion. Combined with a Crusoe Load Balancer and TLS-enabled ingress, you can securely expose OpenAI-compatible model endpoints by using or adapting the example explained here.

Prerequisites

• Admin access to a Crusoe Managed Kubernetes cluster with configured Shared Volumes storage class
• Up-to-date KubeRay operator installed (instructions provided in the github repo). Version 1.5.1 was used for the creation of this article.
• A NodePool with available GPU resources
• Access to your desired model on Hugging Face via a valid hf_token

Step-by-Step Instructions

1. Confirm your chosen model's vLLM support status
   • Review the 'architectures' property of your chosen model's config.json file in the Hugging Face repository and check that the architecture is contained in the vLLM 'Architecture' table here.
   • Ensure that you have a valid HuggingFace 'read' token - even for non-gated models
   • If your model is gated in HuggingFace, request access to it before proceeding.
2. Clone the repo or copy example YAML
   • Edit the deepseek-ray-service.yaml to replace model_id (your own choice), model_source (HF model reference) and hf_token (token from step 1 above) as needed.
   • In the same yaml file, update replicas, minReplicas, maxReplicas, min_replicas and max_replicas to suit the number of workers that you want to be included in the Ray Cluster that will be created.

   • Apply the yaml: 

     kubectl apply -f deepseek-ray-service.yaml
     # expected output:
     rayservice.ray.io/ray-serve-llm created
     secret/hf-token configured

3. Monitor, troubleshoot and access the rayservice
   • Run watch kubectl get po to see the Ray head and worker pods get created (don't wait for the worker pods to get to 'Ready' state, this takes several minutes as they download the model weights), then kubectl get svc to see corresponding services

   • An initial service with a name of the form 'ray-serve-llm-XXXXX-head-svc' is available as soon as the head pod is ready. Access the Ray Cluster UI of the initial service using kubectl port-forwarding to port 8265 (replace the service name in the example below to match your own):

     kubectl port-forward svc/ray-serve-llm-2q47c-head-svc 8265:8265 &

   • Navigate to http://localhost:8265 and select the 'Serve' tab. Monitor the logs as the vLLM starts up in the cluster:
      

     

      

   • It takes several minutes for the Ray Serve deployment to come to 'Healthy' state. If deployment fails, the UI will indicate this clearly and will show the Python traceback so you can determine the reason why. A common reason is that gated model weights cannot be accessed because Hugging Face permission hasn't been granted yet. 
   • When the 'ray-serve-llm-serve-svc' service is ready, port-forward to port 8000 of that service ('kubectl port-forward svc/ray-serve-llm-serve-svc 8000:8000 &') and test it using the curl examples given in the repo's README file, or whatever OpenAI-compatible AI requests are required for your use case.
   • If you have Load Balancer functionality enabled on your Crusoe Cloud organization, you can expose the endpoint to the Internet without requiring the use of port-forward. Edit the service to change its type from ClusterIP to LoadBalancer and wait for the service to display its static IP address (as shown with a kubectl get svc command). For a more secure production implementation, create a TLS-enabled ingress controller and point it at port 8000 of the (unedited) ray-serve-llm-serve-svc. You can now send curl requests to the model endpoint via the public-facing port and IP address of the Ingress service.

Additional Resources

• Ray Serve documentation