How-To Resolve '403 Forbidden' Errors When Creating Load Balancers in CMK

Last Updated: Jan 15, 2026

Introduction

Crusoe's Load Balancer service allows you to easily expose services running on your instances, including those in a Crusoe Managed Kubernetes (CMK) cluster. For performance and security, each project has a default quota for the maximum number of load balancers that can be created.

When using an automated tool like the Crusoe Load Balancer controller for Kubernetes, you might see a '403 Forbidden' error in the controller's logs. This typically indicates that your attempt to create a new load balancer has been blocked because the project has reached its quota limit.

This guide will help you confirm if you're hitting this quota limit and explain the next steps to get it resolved.

Prerequisites

• You will need kubectl access to your Crusoe Managed Kubernetes cluster to view pod logs.

Step-by-Step Instructions

1. Check the Load Balancer Controller Logs

   • If a Kubernetes service of type LoadBalancer is failing to provision, the first step is to inspect the logs from the controller pod responsible for creating the resource in Crusoe Cloud. Use kubectl to find the controller pod and view its logs.

   # Find the controller pod in the relevant namespace
   $ kubectl get pods -n <namespace> | grep lb-controller
   
   # View the logs for that pod
   $ kubectl logs <lb-controller-pod-name> -n <namespace>

2. Identify the '403 Forbidden' Error

   • Scan the logs for error messages related to the API call. If you've hit the quota limit, you will see a log entry that looks very similar to this:

   ERROR Failed to create load balancer via API {"controller": "service", "controllerGroup": "" , "controllerKind": "Service", "Service": {"name": "<Service Name>", "namespace": "default"}, [...], "error": "403 Forbidden"}

   • The key parts of this message are Failed to create load balancer via API and the final error": "403 Forbidden". This is a strong indicator that the action was denied due to a policy, which in this case is the quota limit.

Resolution

1. Request a Quota Increase
   • It looks like you've run into the default load balancer limit for your project. To resolve this, the Crusoe Support team will need to increase the quota for you.
   • Please open a support ticket with the necessary details. To make this process as efficient as possible, you can use our pre-filled link: Request Load Balancer Quota Increase.
2. Confirmation and Creation
   • Once you've submitted the ticket, our team will review the request and apply the new quota to your project.
   • After the quota has been increased, the Kubernetes controller will automatically retry the API call, and the Load Balancer should be created successfully without any further action on your part.