How to Set Minimum and Maximum Node Count per Node Pool with Cluster Autoscaler in CMK

Introduction

In CMK, desired number of nodes can be set during Nodepool creation. If Cluster Autoscaler is enabled, this article highlights steps to set desired minimum and maximum count for number of nodes for a nodepool.

Prerequisites

Before starting, ensure you have the following:

• Access to a Crusoe Cloud project with appropriate permissions
• A CMK cluster with the latest image version (>= 1.30.8-cmk.26)
• Kubeconfig to access your CMK cluster
• Access to Crusoe CLI or Console 

Step-by-Step Instructions

• Fetch details about the Nodepools:

$ crusoe kubernetes nodepools get crusoe-test-nodepool
node pool name: crusoe-test-nodepool
id: 81067549-9bf7-4971-b993-bcb36f15ec3e
state: STATE_RUNNING
type: c1a.4x
desired count: 2
running count: 2
image: 1.31.7-cmk.5
node labels: { crusoe.ai/instance.class: "c1a", crusoe.ai/project.id: "4d012e45-f891-47f6-bace-d8a61a7c32c7"  }
instance ids: [89e08573-edf3-442d-92gd-ca1fdb400ae6 3987fc5d-c2fb-45f9-a76d-9724b8ab7e7f]
subnet id: 73a12f6d-388d-40e6-a4e7-456598af18j8
cluster name: crusoe-test
cluster id: 6596fa96-dcc1-4922-943d-2bbf95675592

$ kubectl get nodes
NAME                                        STATUS   ROLES    AGE    VERSION
np-41067549-1.us-east1-a.compute.internal   Ready    <none>   23m    v1.31.7
np-41067549-2.us-east1-a.compute.internal   Ready    <none>   23m    v1.31.7

• Verify that the cluster autoscaler is part of the cluster.

$ kubectl get deploy cluster-autoscaler -n crusoe-system
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
cluster-autoscaler   1/1     1            1           78m

• Edit the cluster-autoscaler deployment and add parameter --nodes=2:4:crusoe-test-nodepool in the command args section.

$ kubectl edit deploy cluster-autoscaler -n crusoe-system
...
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
    kubectl.kubernetes.io/last-applied-configuration: |...
  generation: 3
  labels:
    app: cluster-autoscaler
  name: cluster-autoscaler
  namespace: crusoe-system
spec:
  ...
    spec:
      containers:
      - command:
        - ./cluster-autoscaler
        - --namespace=crusoe-system
        - --logtostderr=true
        - --v=4
        - --cloud-provider=crusoecloud
        - --nodes=2:4:crusoe-test-nodepool     ----> Line to be added
        envFrom:
        - secretRef:
            name: crusoe-secrets
        image: ghcr.io/crusoecloud/cluster-autoscaler-crusoecloud:1.30.3-crusoe.9
        imagePullPolicy: Always
        name: cluster-autoscaler

Note: Add as command argument - --nodes=1:5:crusoe-second-nodepool for second additional nodepool and so on.

• To test, create a sample deployment that requests high CPU and Memory.

$ kubectl create deployment scale-up-test --image=busybox --replicas=50 -- sleep 3600
$ kubectl set resources deployment scale-up-test --requests=cpu=1000m,memory=1Gi

• Autoscaler pod logs show scaling of 2 more nodes for the nodepool:

static_autoscaler.go:306] Starting main loop

crusoe_node_group.go:511] current target size for node pool with id 81067549-9bf7-4971-b993-bcb36f15ec3e 
is 4, where node pool's current desired count is 4 and it contains 2 active nodes

1 klogx.go:87] failed to find place for default/scale-up-test-5c6d7fcf79-mmcgf based on similar pods scheduling
1 klogx.go:87] There were also 4 other logs from HintingSimulator.TrySchedulePods func that were capped.
1 filter_out_schedulable.go:120] 6 pods marked as unschedulable can be scheduled.
1 filter_out_schedulable.go:75] Schedulable pods present
1 filter_out_daemon_sets.go:40] Filtering out daemon set pods
1 filter_out_daemon_sets.go:49] Filtered out 0 daemon set pods, 18 unschedulable pods left

1 orchestrator.go:112] Upcoming 2 nodes
1 orchestrator.go:444] Skipping node group 81067549-9bf7-4971-b993-bcb36f15ec3e - max size reached

1 static_autoscaler.go:306] Starting main loop
1 crusoe_cloud_provider.go:156] Refresh,ProjectID=4d012e45-f891-47f6-bace-d8a61a7c32c7,ClusterID=6596fa96-dcc1-4922-943d-2bbf95675592
1 crusoe_manager.go:204] Refresh,ProjectID=4d012e45-f891-47f6-bace-d8a61a7c32c7,ClusterID=6596fa96-dcc1-4922-943d-2bbf95675592 ListNodePools returns 1 IDs
1 crusoe_node_group.go:511] current target size for node pool with id 81067549-9bf7-4971-b993-bcb36f15ec3e
is 4, where node pool's current desired count is 4 and it contains 4 active nodes

• Two additional nodes have now scaled up:

$ kubectl get nodes
NAME                                        STATUS   ROLES    AGE    VERSION
np-41067549-1.us-east1-a.compute.internal   Ready    <none>   23m    v1.31.7
np-41067549-2.us-east1-a.compute.internal   Ready    <none>   23m    v1.31.7
np-41067549-3.us-east1-a.compute.internal   Ready    <none>   112s   v1.31.7
np-41067549-4.us-east1-a.compute.internal   Ready    <none>   112s   v1.31.7

• Modify the parameter to set new minimum value of 1 in the cluster-autoscaler deployment.

    spec:
      containers:
      - command:
        - ./cluster-autoscaler
        - --namespace=crusoe-system
        - --logtostderr=true
        - --v=4
        - --cloud-provider=crusoecloud
        - --nodes=1:4:crusoe-test-nodepool     ----> Line to be changed

• After resource requirement is reduced, autoscaler eventually scaled down to the minimum count i.e. 1.

$ kubectl get nodes
NAME                                        STATUS   ROLES    AGE   VERSION
np-41067549-4.us-east1-a.compute.internal   Ready    <none>   54m   v1.31.7

Note: 

• To safely enable scale-to-zero for a node pool, you need ensure there are multiple nodepools in the cluster where 1 dedicated node pool will run system critical pods.

• System critical pods (DaemonSets like Cilium, fluent-bit, metrics-server, etc. and Deployments like CoreDns) must always run somewhere to keep your cluster functional. So, if your node pool hosts these system pods, it cannot scale down to zero because the last node must stay up to run them.

Additional Resources

• Cluster Autoscaler
• Startup parameters Supported for Cluster Autoscaler