How-To Manage Load Balancer Firewall Rules Automatically on CMK

Introduction

Crusoe Cloud load balancers are Layer 4 pass-through: they forward connections to your backends, but creating one does not open any firewall path to those backends. On CMK, the backends are your worker nodes' NodePorts, so a new type: LoadBalancer Service needs an ingress rule for its NodePort before any client traffic can arrive.

This gap has a sharp edge. Load balancer health checks originate on the infrastructure side and do not traverse your firewall rules, so every backend can show Online while 100% of real client traffic is dropped. A missing firewall rule is the most common reason a brand-new load balancer "doesn't work".

The Crusoe Load Balancer Controller can close this gap for you. With a single annotation on the Service, the controller creates a firewall rule scoped to that Service's NodePorts, keeps it in sync when your allowed source ranges change, and removes it when the Service is deleted. The rule's allowed sources come from the standard Kubernetes loadBalancerSourceRanges field.

The annotation is opt-in and per-Service. It does not turn on firewall management for the cluster or the VPC, and it never touches rules it did not create.

Prerequisites

CMK Cluster with the Crusoe Load Balancer Controller Installed (v0.0.23 or Later)
kubectl Access to the Cluster
Crusoe CLI Configured for Your Project (Verification Steps Only)
Available Load Balancer Quota in the Project

Instructions

Step 1: Verify the Controller Is Installed

Check the controller release and version:

helm list -n crusoe-system

You should see a crusoe-lb-controller release at chart version 0.0.23 or later. The deployment name follows the Helm release name, so adjust the commands below if yours differs:

kubectl get deploy -n crusoe-system | grep lb-controller

ℹ️ Note: Run exactly one controller instance. A duplicate Helm release of the controller in the same cluster causes both instances to reconcile the same Services, which surfaces as 409 Conflict errors in the controller logs.

Step 2: Enable Firewall Management on a New Service

Add the crusoe.ai/manage-firewall-rule: "true" annotation and set loadBalancerSourceRanges to the CIDRs that should be allowed to reach the load balancer:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    crusoe.ai/manage-firewall-rule: "true"
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080
  type: LoadBalancer
  loadBalancerSourceRanges:
    - 198.51.100.4/32

The controller creates the load balancer and the firewall rule in the same reconcile pass. No separate step is needed.

⚠️ Warning: If you omit loadBalancerSourceRanges, the rule defaults to 0.0.0.0/0 and your NodePort is reachable from the entire internet. Always set source ranges for anything you would not expose publicly.

Step 3: Verify the Rule

Get the Service's NodePort:

kubectl get svc my-service
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
my-service   LoadBalancer   10.233.62.51   203.0.113.10   80:30368/TCP   19s

List your VPC firewall rules and look for the NodePort:

crusoe networking vpc-firewall-rules list --project-id <PROJECT_ID>
name                          direction   protocols   sources           destination ports
default-my-service-e7fbf039   ingress     tcp         198.51.100.4/32   30368

Controller-managed rules follow the naming pattern <namespace>-<service-name>-<hash>, and the destination is the Service's NodePort rather than a port range. A Service with multiple ports (for example an ingress controller with HTTP and HTTPS) gets all of its NodePorts in one rule.

💡 Tip: The CLI's table output truncates long source lists. To verify a rule with many CIDRs, use JSON output instead: crusoe networking vpc-firewall-rules list --project-id <PROJECT_ID> -f json and read the rule's sources array.

Confirm the data path from an allowed source:

nc -zv <EXTERNAL_IP> 80

You can also watch the controller act in its logs. Firewall operations happen inside the same Service reconcile that manages the load balancer, so filter for the relevant lines:

kubectl logs -n crusoe-system deploy/<LB_CONTROLLER_DEPLOYMENT> --since=1h | grep -iE "firewall|rule"

Rule creation is asynchronous: the controller submits an operation and polls it across reconciles. A healthy create looks like this (trimmed for readability):

INFO  Creating firewall rule            {"Service": {"name":"my-service"}, "name": "default-my-service-e7fbf039", "destinationPorts": ["30368"], "protocols": ["tcp"]}
INFO  Created firewall rule operation   {"name": "default-my-service-e7fbf039", "operationId": "<OPERATION_ID>"}
INFO  Firewall rule operation in progress
INFO  Firewall rule operation complete  {"ruleInfo": {"name":"default-my-service-e7fbf039","sources":[{"cidr":"198.51.100.4/32"}],"destination_ports":["30368"],"state":"active"}}
INFO  Firewall rule matches service, no action needed

The rule is often functional before the operation complete line appears, and matches service, no action needed is the steady state you will see on every reconcile afterwards.

When you change loadBalancerSourceRanges, the controller logs the diff and the fix:

INFO  Rule sources do not match                    {"ruleSources": [{"cidr":"198.51.100.4/32"}], "argsSources": [{"cidr":"198.51.100.4/32"},{"cidr":"203.0.113.0/24"}]}
INFO  Firewall rule does not match service, patching

And on Service deletion with the annotation still "true":

INFO  Deleted firewall rule    {"ruleID": "<RULE_ID>"}

A silent grep with no matching lines means the controller never attempted a firewall operation for your Service. Check that the annotation is spelled exactly crusoe.ai/manage-firewall-rule and its value is the string "true", then check kubectl describe svc <SERVICE> for reconcile events. An ERROR line means it tried and failed, and the message will say why. Those are two different problems, so read the logs before assuming the feature is broken.

Step 4: Enable It on an Existing Service

You do not need to recreate anything. Annotate a running Service and the rule appears on the next reconcile, typically within a minute:

kubectl annotate svc my-existing-service crusoe.ai/manage-firewall-rule=true

This is the migration path if you already run load balancers behind manually created rules: annotate the Services, verify the controller-managed rules exist, then remove your broad manual rules.

Step 5: Update the Allowed Sources

Edit loadBalancerSourceRanges on the Service and the controller updates the rule to match:

kubectl patch svc my-service --type=merge -p '{"spec":{"loadBalancerSourceRanges":["198.51.100.4/32","203.0.113.0/24"]}}'

Re-list the firewall rules to confirm both CIDRs appear on the rule.

Step 6: Stop Managing the Rule Without Deleting Anything (Optional)

If you want to keep the Service and the firewall rule but take the rule back under manual control, set the annotation to "false":

kubectl annotate svc my-service crusoe.ai/manage-firewall-rule=false --overwrite

The controller stops touching the rule from that point on. The rule stays exactly as it was, and you now own it like any manually created rule: changes to loadBalancerSourceRanges no longer sync to it, and deleting the Service later will not remove it.

This is useful when you want to hand the rule over to your own firewall management, for example to edit it in ways the annotation does not support, or to freeze it during a migration.

ℹ️ Note: Unmanaging is reversible. Set the annotation back to "true" (with --overwrite, since the annotation already exists) and the controller resumes managing the rule on the next reconcile.

💡 Tip: Before unmanaging, make sure loadBalancerSourceRanges reflects what you actually want frozen. If you removed the ranges earlier, the rule is sitting at 0.0.0.0/0 and will stay that way until you edit it yourself.

Step 7: Clean Up

Delete the Service while the annotation is still "true" and the controller removes the firewall rule with it:

kubectl delete svc my-service

⚠️ Warning: Setting the annotation to "false" tells the controller to stop managing the rule. It does not remove the rule, and deleting the Service afterwards leaves the rule behind. If you turned management off (Step 6), delete the rule yourself with crusoe networking vpc-firewall-rules delete <RULE_ID> or from the console. The controller records the rule ID in the Service's crusoe.ai/firewall-rule-id annotation, so note it before you delete the Service. Watch for the worst combination: removing loadBalancerSourceRanges (rule widens to 0.0.0.0/0), then setting the annotation to "false", then deleting the Service leaves a permanent world-open rule that nothing manages.

Behavior Notes

The annotation is scoped to its own Service. Other Services, manually created rules, and CMK system rules are never touched.
The controller creates exactly one rule per Service. All ports and all source CIDRs live on that single rule.
Source-range changes keep the same rule ID and are applied as an atomic transaction on the platform side: the old rule stays in force until the new one takes effect, so there is no moment during an update where no rule applies and no gap in packet processing.
The flip side of that atomicity: changes take effect late, not instantly. New lists typically propagate within seconds, and until they land the previous list is still enforced. In particular, removing a CIDR does not cut that source off immediately; it keeps access until the updated rule takes effect. Do not treat a source-range removal as instant revocation.
Identical source lists are deduplicated before the rule is touched. A patch that does not materially change the deduplicated set results in no rule change at all.
Automated refreshes of large allow-lists (for example a CDN or cloud provider prefix list on a cron) are safe from an availability standpoint, since updates do not gap traffic. Be considerate with frequency all the same: each material refresh rewrites the whole rule, so batch changes rather than patching per-CIDR.
The reconcile loop patches the rule whenever it disagrees with the Service (the Firewall rule does not match service, patching log line). Expect manual edits to a managed rule to be reverted on a later reconcile. If you need manual control of the rule, unmanage it first (Step 6).
Firewall policy is the union of all rules. A broad manual rule (for example an all-ports allow) masks the effect of the controller-managed rule, so tighten or remove broad rules to get the benefit.
If the controller logs show 403 Forbidden when creating the load balancer itself, the project's load balancer quota is exhausted (use the same kubectl logs command from Step 3). Delete unused load balancers or request a quota increase, and let the controller retry. See How-To Resolve '403 Forbidden' Errors When Creating Load Balancers in CMK.

Example

A team exposes an inference endpoint through a type: LoadBalancer Service on CMK. The load balancer provisions, every backend shows Online, and client requests still time out. The cause is the classic one: no ingress rule for the NodePort, invisible to health checks.

They add crusoe.ai/manage-firewall-rule: "true" to the Service and set loadBalancerSourceRanges to their office and VPN CIDRs. Within a minute a rule named default-inference-api-3fa2c1b7 appears, scoped to the Service's NodePort and their two CIDRs. Requests from the allowed ranges succeed, requests from anywhere else still fail, and when the team later deletes the Service during a redesign, the rule disappears with it. No firewall tickets, no orphaned rules, no all-ports allow left behind.

Additional Resources

Related to

firewall how-to cmk load balancer network acl

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Article is closed for comments.