Introduction
Crusoe Managed Kubernetes (CMK) clusters run Cilium as the CNI, and by default Cilium allocates pod IPs using its cluster-pool IPAM mode. In this mode, the Cilium operator owns one or more parent CIDR blocks (configured as ipam.operator.clusterPoolIPv4PodCIDRList) and carves off a fixed-size block — a /24 by default (clusterPoolIPv4MaskSize) — for each node that joins the cluster. That per-node block becomes the node's podCIDR, and the Cilium agent on that node can't come up healthy until it receives one.
This matters because the parent CIDR is finite. A single /16 sliced into /24 blocks yields exactly 256 available node allocations. Once every block in the pool is handed out, any additional node that joins the cluster — via node pool scale-up, autoscaler, or new pool creation — will register with the Kubernetes API and show Ready, but its Cilium agent has nothing to allocate. The node looks healthy at the Kubernetes level while being completely non-functional for pod networking.
This article covers how to recognize pod CIDR pool exhaustion, confirm it's actually your root cause, and safely expand the pool so new nodes can schedule and network correctly.
ℹ️ Note: This applies specifically to clusters using Cilium's cluster-pool IPAM mode, which is the CMK default. If your cluster uses a different IPAM mode (e.g., CRD- or ENI-based), the symptoms and fix described here don't apply.
Prerequisites
- Cluster Running Crusoe Managed Kubernetes With Cilium CNI in Cluster-Pool IPAM Mode
- kubectl Access With Permissions to Read and Modify Resources in the kube-system Namespace
- Helm 3 Installed and Configured Against the Target Cluster
- Ability to Identify the Currently Deployed Cilium Chart Version Before Making Changes
Instructions
Step 1: Confirm the Symptoms
Check whether new nodes are Ready but not actually schedulable:
kubectl get nodes kubectl get pods -A -o wide | grep <new-node-name>
If pods assigned to the new node(s) are stuck in Pending or ContainerCreating, move to the next step rather than assuming it's a generic scheduling issue.
Step 2: Inspect the Cilium Agent Logs
Find the Cilium agent pod running on the affected node and check its logs:
kubectl get pods -n kube-system -l k8s-app=cilium -o wide kubectl -n kube-system logs <cilium-agent-pod-on-affected-node>
The signature error for this condition is:
level=warning msg="Waiting for k8s node information" error="required IPv4 PodCIDR not available" subsys=daemon
The Cilium agent pod itself typically shows 0/1 ready in kubectl get pods -n kube-system.
Step 3: Check CiliumNode Allocation
Inspect the CiliumNode custom resources to confirm the affected nodes never received a pod CIDR:
kubectl get ciliumnode -o wide
Healthy nodes will show a CILIUMINTERNALIP value. Affected nodes will have this field blank, even though the VM-level networking (VPC IP) looks completely normal — this is the key differentiator between a pod CIDR exhaustion issue and a general node health problem.
Step 4: Confirm Pool Exhaustion
Count how many nodes currently hold an allocation versus your parent CIDR's capacity:
kubectl get ciliumnode -o wide | grep -c <parent-cidr-prefix>
Also pull the current IPAM configuration to see the exact pool size and mask being used — don't assume /24, since clusterPoolIPv4MaskSize can be customized:
helm get values cilium -n kube-system | grep -A5 clusterPool
If the node count is at or near the theoretical ceiling for your configured pool (e.g., 254–256 nodes for a single /16 at /24 masks), pool exhaustion is confirmed.
💡 Tip: If you're planning to scale a node pool past 256 nodes, check your pod CIDR capacity ahead of time rather than after nodes start failing to schedule.
Step 5: Identify the Currently Deployed Cilium Chart Version
Before making any changes, confirm the exact chart version already running. Upgrading without pinning --version can pull in an unrelated version bump alongside your CIDR change:
helm list -n kube-system helm history cilium -n kube-system
⚠️ Warning: Always pass
--versionexplicitly matching the currently deployed release. Omitting it lets Helm pick up whatever is latest in the repo, which can trigger an unintended Cilium upgrade.
Step 6: Export the Current Cilium Values
If the cilium Helm repo isn't already configured on the host you're running this from, add it first:
helm repo add cilium https://helm.cilium.io/ helm repo update
Then export the release's current values to a local file rather than editing blind:
helm get values -n kube-system cilium > cilium-values.yaml
ℹ️ Note:
helm get valuesprefixes its output with aUSER-SUPPLIED VALUES:header line. Delete that first line fromcilium-values.yamlbefore proceeding — it isn't part of the actual values and will break the upgrade in Step 7 if left in.
Step 7: Append the New CIDR and Apply
Edit cilium-values.yaml and append the new block to the existing clusterPoolIPv4PodCIDRList. Do not remove or replace the existing entry, or nodes already using the old range can lose their allocation on the next operator restart:
ipam:
mode: cluster-pool
operator:
clusterPoolIPv4MaskSize: 24
clusterPoolIPv4PodCIDRList:
- <EXISTING_POD_CIDR>
- <NEW_POD_CIDR>⚠️ Warning: Before choosing
<NEW_POD_CIDR>, confirm it doesn't overlap with your existing pod CIDR(s), service CIDR, VPC ranges, or anynative-routing-cidrconfiguration in use. If your cluster relies on native routing (rather than encapsulation), the underlay routes for the new block may also need to be updated — verify this against your routing setup, not just the IPAM pool. Also don't assume the mask is/24— checkclusterPoolIPv4MaskSizein the exported values, since it can be customized.
📝 Example: if your cluster's current pool is
10.234.0.0/16, adding a second/16to double capacity looks like this — note the first entry is untouched and the second is the new addition:
ipam:
mode: cluster-pool
operator:
clusterPoolIPv4MaskSize: 24
clusterPoolIPv4PodCIDRList:
- 10.234.0.0/16 # existing pool — retain as is
- 10.235.0.0/16 # new pool — this entry is the additionApply the updated values, pinning the version you identified in Step 5:
helm upgrade -n kube-system cilium cilium/cilium --values ./cilium-values.yaml --version <CURRENT_CILIUM_VERSION>
Step 8: Restart the Cilium Operator and Verify
The operator needs to pick up the new pool before it can allocate from it:
kubectl -n kube-system rollout restart deploy/cilium-operator kubectl -n kube-system rollout status deploy/cilium-operator
Once the rollout completes, confirm the previously affected nodes now have an allocation:
kubectl get ciliumnode -o wide
All nodes should now show a populated CILIUMINTERNALIP. Pods that were stuck Pending on the affected nodes should transition to Running shortly after.
Resolution
Cilium's cluster-pool IPAM carves your configured pod CIDR into fixed-size, per-node blocks — by default a /24 per node. A single /16 parent CIDR therefore supports a hard ceiling of 256 nodes across all pools drawing from it. When that ceiling is reached, additional nodes join the Kubernetes API and report Ready, but the Cilium operator has no block left to hand out, so the agent on those nodes can never leave PodCIDR not available and pods can't be scheduled or networked on them. Adding a second (non-overlapping) CIDR block to clusterPoolIPv4PodCIDRList and restarting the operator gives it new blocks to allocate, resolving the issue without requiring any change to already-healthy nodes.
Example
A team running a CMK cluster with a single /16 pod CIDR scales a node pool from around 200 nodes to 260 to support a larger training job. The first ~250 nodes join normally, but the last handful come up Ready in kubectl get nodes with no pods scheduling. kubectl get ciliumnode -o wide shows the new nodes missing CILIUMINTERNALIP, and the Cilium agent logs on those nodes show required IPv4 PodCIDR not available. Following the steps above, the team confirms they're at the 256-node ceiling for their /16 pool, adds a second /16 block via helm upgrade, restarts cilium-operator, and the new nodes pick up pod CIDR allocations within a minute or two.