CMK Cluster Autoscaler: Best practices and Known Limitations

Introduction

The Cluster Autoscaler add-on for Crusoe Managed Kubernetes (CMK) watches pending pods, picks a node pool whose template node can host them, and resizes that pool by calling the Crusoe Cloud API. It also evicts under-utilized nodes when their workloads can be rescheduled elsewhere. The add-on is a Crusoe-maintained build of the upstream kubernetes/autoscaler cluster-autoscaler with a Crusoe Cloud provider that speaks to the CMK node-pool API.

A few CMK-specific behaviors are important to understand before you put the autoscaler in front of production workloads:

• The autoscaler only resizes existing node pools — it cannot create, delete, or autoprovision them. Every pool you want to scale must already exist in Crusoe Cloud.
• The autoscaler does not see reservations when picking which pool to scale. Reservation discounts are applied automatically by the billing system whenever your usage is within reserved capacity for that instance type, but the autoscaler will still scale a pool past its reserved size if pending pods can fit — which can push you into on-demand spend or, if on-demand inventory is also constrained, into an unhealthy-pool deadlock.
• Any pool not explicitly listed in the --nodes flag inherits a built-in default of min=1, max=254, which can produce unexpected on-demand spend if you intended that pool to stay at a fixed size.

This article documents the best practices that prevent these failure modes, the limitations baked into the current autoscaler release, and the recovery procedure for the unhealthy-pool deadlock. Behavior described below is validated against the crusoe-cluster-autoscaler-release-1.30.3 source in the crusoecloud/k8s-autoscaler repository.

Prerequisites

• Running CMK Cluster With the Cluster Autoscaler Add-on Installed
• Existing Node Pools (Autoscaler Cannot Create Them)
• kubectl Access With Permission to Edit Deployments in crusoe-system
• Crusoe CLI Installed and Authenticated (crusoe kubernetes nodepools Commands)
• Awareness of Which Pools Are Reserved Versus On-Demand

Best Practices

1. List Every Node Pool in --nodes

The autoscaler iterates over every node pool attached to your cluster on each refresh. For every pool, it looks up a --nodes=<min>:<max>:<pool-name> argument matching the pool's name. If no match exists, it falls back to a built-in default of min=1, max=254. That pool then becomes eligible for scale-up to 254 nodes the moment any pending pod can fit on it.

⚠️ Warning: A pool not listed in --nodes is not "ignored" by the autoscaler — it is treated as 1:254. Always list every pool, including pools you intend to leave at a fixed size.

Edit the autoscaler Deployment and add one --nodes argument per pool:

kubectl -n crusoe-system edit deployment cluster-autoscaler

args:
  - --cloud-provider=crusoecloud
  - --skip-nodes-with-local-storage=false
  - --nodes=1:10:cpu-pool
  - --nodes=0:8:gpu-pool
  - --nodes=3:3:fixed-pool        # freeze a pool by setting min == max

To prevent the autoscaler from ever changing a pool's size, set min and max equal to that pool's current desired count.

2. Set --skip-nodes-with-local-storage=false

The upstream default is true, which prevents scale-down of any node running pods with emptyDir or hostPath volumes. CMK clusters include system components (such as CSI drivers and log collectors) that use emptyDir, so with the default in place the autoscaler refuses to remove any node:

Node <node-name> unremovable: pod with local storage present

Set --skip-nodes-with-local-storage=false in the Deployment args. If a specific workload of your own uses emptyDir but is safe to restart, annotate the pod instead of relying on the global flag:

metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"

3. Always Set Resource Requests on Your Workloads

The autoscaler decides whether a pending pod will fit on a hypothetical new node by comparing the pod's resources.requests against the pool's template node, which is built from CPU cores, memory, and GPU count in the Crusoe Cloud VM-type catalog. Pods without requests can be incorrectly matched — or not matched at all — which causes either over-scaling or scale-up to silently fail.

resources:
  requests:
    cpu: "4"
    memory: "16Gi"
    nvidia.com/gpu: "1"    # GPU workloads only

GPU requests must use the nvidia.com/gpu resource name. AMD GPUs are not currently in the supported list (see Limitations).

4. Cap On-Demand Spend With a Fixed-Size Reserved Pool

Reservation discounts apply automatically: whenever your usage at a given moment is within your reserved capacity for that instance type, you pay the reserved rate, regardless of which node pool the VM came from. Usage that exceeds your reserved quantity at that moment is billed at on-demand rates.

The autoscaler itself is not reservation-aware. If a single pool is configured as --nodes=1:10:my-pool against a 5-VM reservation, the autoscaler will happily request 10 VMs — the first 5 effectively bill at the reserved rate, and the additional 5 bill on-demand. If on-demand inventory is also constrained, the pool can transition to STATE_UNHEALTHY and block further scale-up entirely (see Limitation 1).

To put a hard ceiling on on-demand spend (and to keep the reserved pool out of the unhealthy-pool failure mode), use two separate pools of the same VM type:

  # Reserved pool — sized to your reservation, never scales beyond it
  - --nodes=5:5:reserved-l40s-pool

  # On-demand pool — optional burst capacity, scale to zero when idle
  - --nodes=0:10:ondemand-l40s-pool

If you only have the reserved pool, the autoscaler cannot burst into on-demand even when pending pods would otherwise force it. If you want burst behavior, keep the on-demand pool. With the two-pool pattern both behaviors are explicit and independently controllable. To express a preference for one pool over the other (for example, fill the reserved pool first), use --expander=priority with a priority ConfigMap.

ℹ️ Note: Reservation auto-application is documented under Reservations Overview and Managing Your Reservations. No manual VM-to-reservation association is required.

5. Tune Scale-Down for Cost Optimization

The upstream defaults are conservative. For workloads where you want capacity released quickly, the following tuned values are a reasonable starting point:

• --scale-down-utilization-threshold — Upstream default 0.5; suggested 0.3. A node is eligible for scale-down only when its requested-resource usage drops below this fraction of allocatable.
• --scale-down-unneeded-time — Upstream default 10m; suggested 5m. How long a node must remain below the threshold before removal.
• --scale-down-delay-after-add — Upstream default 10m; suggested 5m. Cooldown after a scale-up event before scale-down is allowed again.
• --expander — Upstream default random; suggested least-waste.

If you see Node <name> unremovable: cpu requested (X% of allocatable) is above the scale-down utilization threshold, the node's pod requests exceed the threshold. Lower the threshold or right-size the pod requests.

6. Use Scale-to-Zero for Expensive GPU Pools

Setting min=0 on a GPU pool drops it to zero nodes when no GPU workloads are pending, eliminating idle GPU cost.

  - --nodes=0:8:gpu-pool

The trade-off is that the first GPU workload waits 2–5 minutes for VM provisioning plus 1–2 minutes for the node to join the cluster. If your GPU workloads are latency-sensitive, keep min ≥ 1.

7. Use PodDisruptionBudgets for Graceful Scale-Down

PodDisruptionBudgets let the autoscaler safely evict pods during scale-down. Without a PDB, scale-down can be blocked, or replicas can be removed in a way that disrupts your workload mid-request.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app

Known Limitations

Limitation 1: Unhealthy Node Pool Deadlock

When a node pool's desired count exceeds what Crusoe Cloud can provision, the pool transitions to STATE_UNHEALTHY. The autoscaler then forces its internal target size to the number of currently active nodes, but the cloud-side desired count is still the higher (unfulfillable) number. Because cloud-count target-size, every IncreaseSize call is silently aborted:

IncreaseSize,PoolID=<pool-id>, aborting IncreaseSize. Current node pool count on Crusoe Cloud already exceeds node group's target size

The autoscaler logs this every ~10 seconds with no scale-up progress until the cloud-side desired count is reset. See the Recovery section below.

Scope of the deadlock

The deadlock is scoped to the unhealthy pool, not the whole cluster:

• Other node pools continue to scale up and down normally.
• The autoscaler pod itself does not crash or stop reconciling.
• Pods already running on the unhealthy pool's nodes are unaffected — only new scale-up requests for that pool are blocked.
• Scale-down on the unhealthy pool still works, and as a side effect it lowers the cloud-side count and can clear the deadlock (see Limitation 2).

One subtle interaction to watch for: if the upstream cluster-autoscaler expander selects the unhealthy pool as the best target for a pending pod, the silent IncreaseSize abort means the pod stays pending across multiple loops until either the pool recovers or the expander picks a different pool. If you are seeing pending pods that are not landing on a healthy alternate pool, check whether an unhealthy pool is "shadowing" it.

Limitation 2: Self-Healing Only Works on Scale-Down

When the autoscaler scales an unhealthy pool down, it lowers the cloud-side desired count as part of the delete operation, which clears the deadlock automatically. But if your workloads are growing, the autoscaler cannot self-heal — you must manually reset the count using the recovery procedure below.

Limitation 3: Cannot Create or Delete Node Pools

The autoscaler can only resize existing pools. Pool creation, deletion, and autoprovisioning return ErrNotImplemented. Pre-create every pool you need (including burst pools), and do not rely on the autoscaler to spin up new instance types dynamically.

Limitation 4: No Pricing Awareness

The autoscaler does not know what any instance type costs. It cannot prefer cheaper pools or make cost-optimized scaling decisions. Use --expander=least-waste to minimize wasted capacity, or --expander=priority with a priority ConfigMap to express your own preference between pools.

Limitation 5: Soft Affinity Is Ignored During Scale-Up

Only requiredDuringSchedulingIgnoredDuringExecution influences scale-up decisions. preferredDuringSchedulingIgnoredDuringExecution is not — the autoscaler will not provision a new node solely to satisfy a soft preference. Use hard affinity for placement rules that must drive scaling.

Limitation 6: 30-Minute Operation Timeout

Scale-up and scale-down API operations have a 30-minute timeout with 1-second polling. If a provisioning operation does not complete within that window, the autoscaler logs a timeout error and treats the operation as failed. Repeated timeouts typically point to a capacity or quota issue in the region.

Limitation 7: GPU Support Is NVIDIA-Only Today

The autoscaler's supported GPU type list is hard-coded and currently includes only NVIDIA:

nvidia-a100-80gb, nvidia-a100-80gb-sxm-ib, nvidia-h100-80gb-sxm-ib, nvidia-l40s-48gb.

AMD MI300X is not in the supported list as of release 1.30.3. AMD pools can still be resized, but the autoscaler will not match amd.com/gpu resource requests when picking a pool for a pending pod.

Recovery: Unhealthy Node Pool Deadlock

Symptom

The autoscaler logs the following message every ~10 seconds with no scale-up progress, and crusoe kubernetes nodepools list shows the pool in STATE_UNHEALTHY:

IncreaseSize,PoolID=<pool-id>, aborting IncreaseSize. Current node pool count on Crusoe Cloud already exceeds node group's target size

Cause

The pool's cloud-side desired count exceeds the number of VMs that successfully provisioned. The autoscaler has set its internal target to the active node count, but the cloud is still holding the higher unfulfilled count. Until the cloud-side count is reset, scale-up will not proceed.

Recovery Procedure

1. Inspect the pool to find its current desired count and instance list:

   crusoe kubernetes nodepools get <pool-name-or-id>

2. Count the actual running nodes for that pool, either from the output above or by cross-referencing kubectl get nodes.

3. Reset the cloud-side count to match the actual running count:

   crusoe kubernetes nodepools update <pool-name-or-id> --count <actual-running-count>

4. Wait for the pool state to transition back to STATE_RUNNING (up to a minute). The autoscaler resumes normal behavior on its next refresh.

💡 Tip: If the unhealthy pool also has idle nodes that scale-down would remove, the autoscaler will self-heal the pool on its next scale-down by lowering the cloud-side count automatically. This only helps when utilization is dropping; for growing workloads, run the CLI command.

Prevention

• Use the two-pool reserved + on-demand pattern (Best Practice #4).
• Avoid manually scaling a pool beyond known cloud capacity.
• Freeze fixed-size pools with min == max in --nodes so the autoscaler never tries to grow them.

Autoscaler Resilience

The autoscaler pod can continue functioning even when its host node is NotReady. The container runtime operates independently of the kubelet status, so as long as the pod has network connectivity to the Kubernetes API and the Crusoe Cloud API, it will continue making scaling decisions and provisioning replacement nodes.

Example: End-to-End Args Block

A complete args block for a cluster with one CPU pool (on-demand, 1–10 nodes), one reserved L40S pool (fixed at 5), and one on-demand burst L40S pool (0–10 nodes):

spec:
  template:
    spec:
      containers:
        - name: cluster-autoscaler
          command:
            - ./cluster-autoscaler
            - --namespace=crusoe-system
            - --logtostderr=true
            - --v=4
            - --cloud-provider=crusoecloud
            - --skip-nodes-with-local-storage=false
            - --nodes=1:10:cpu-pool
            - --nodes=5:5:reserved-l40s-pool
            - --nodes=0:10:ondemand-l40s-pool
            - --scale-down-utilization-threshold=0.3
            - --scale-down-unneeded-time=5m
            - --scale-down-delay-after-add=5m
            - --expander=least-waste

Additional Resources

• Crusoe Managed Kubernetes Add-ons
• Managing CMK Node Pools
• Crusoe k8s-autoscaler source (cloudprovider/crusoecloud)
• Upstream Cluster Autoscaler FAQ
• Pod Disruption Budget reference