How-To Troubleshoot Crusoe Container Registry (CCR): Push and Pull Failures

Last Updated: Dec 30, 2025

Introduction

The Crusoe Container Registry (CCR) is a regional, Docker-compliant registry built to provide high-speed, local image access for your workloads. However, because CCR is strictly scoped to specific Crusoe locations, common misconfigurations in regional tagging or credential handling can lead to 500 Internal Server Errors or 401 Unauthorized responses. This guide identifies the root causes behind these failures and provides verified solutions to restore your CI/CD pipeline.

Prerequisites

• Access to the Crusoe Cloud Console.

• The Crusoe CLI and Docker CLI are installed on your local machine or VM.

• If using Crusoe Managed Kubernetes (CMK): kubectl access to your cluster.

Step-by-Step Instructions

1. Resolve 500 Internal Server Errors (Regional Mismatch)

If you receive a 500 Internal Server Error during a docker push, the most likely cause is a mismatch between the image tag and the repository’s physical location.

• The Symptom: Your logs show received unexpected HTTP status: 500 Internal Server Error. 

• The Cause: You are likely trying to push to a US-East endpoint (registry.us-east1-a...) for a repository that was created in US-Southcentral (us-southcentral1-a). The regional gateway cannot resolve metadata for resources outside its scope.

• The Fix: Ensure your image tag matches the registry URL for your repository’s region.

  Verified Command Flow:

  1. Identify the correct regional URL in the Console (e.g., us-southcentral1-a)

  2. Re-tag the image specifically for that region's endpoint

     docker tag my-image:latest registry.us-southcentral1-a.ccr.crusoecloudcompute.com/my-repo-name.project-id/my-image:latest

  3. Push to the correct regional endpoint

     docker push registry.us-southcentral1-a.ccr.crusoecloudcompute.com/my-repo-name.project-id/my-image:latestResolve 401 Unauthorized / ImagePullBackOff (Token Corruption)

2. Resolve 401 Unauthorized / ImagePullBackOff (Token Corruption)

• If your CMK pods fail with ImagePullBackOff and a 401 Unauthorized error, the Crusoe Container registry token in your Kubernetes Secret has likely been corrupted by your local shell during creation.

• The Symptom: kubectl describe pod shows failed to resolve reference... unexpected status from HEAD request: 401 Unauthorized.

• The Cause: Crusoe Registry tokens often contain special characters (e.g., $, ., /). When running kubectl create secret, your shell (bash/zsh) may attempt to interpret these characters as variables, altering the token before it reaches Kubernetes.

• The Fix: Always enclose your token in single quotes (') during secret creation.

  Verified Secret Creation:

  kubectl create secret docker-registry ccr-credentials \
    --docker-server=<your-ccr-repository-url> \
    --docker-username=<your-crusoe-email> \
    --docker-password='<paste-your-token-here-inside-single-quotes>' \
    --namespace=<your-namespace>

  Note: Using double quotes (") will not prevent the shell from interpreting $ characters.

Resolution

After implementing these verified fixes, you can expect the following:

• Successful Authentication: Registry tokens will remain uncorrupted, allowing for stable ImagePullBackOff resolution.

• Optimised Throughput: By pushing to the correct regional endpoint, your images will reside in the same physical location as your compute nodes, leveraging the high-bandwidth backbone of the Crusoe network.

Additional Resources

1. Using CCR with Crusoe Managed Kubernetes

2. Managing Registry Tokens