How-To Disable Root Login and SSH on CMK Nodes

Introduction

Crusoe Managed Kubernetes (CMK) provisions and manages the lifecycle of your node pool VMs for you. Because CMK owns the node image and boot process, it does not currently support startup scripts or cloud-init customization on node pool VMs. There is no built-in hook to apply OS-level configuration at boot time.

That matters for security hardening. Common audit controls — CIS benchmarks, SOC 2, internal security baselines — frequently require that root login over SSH be disabled, or that SSH access to nodes be removed entirely. Since you can't bake this into the node at provision time, the configuration needs to be applied after the node joins the cluster, and reapplied automatically whenever the autoscaler or a node pool operation brings up a new node.

The Kubernetes-native answer is a privileged DaemonSet. A DaemonSet is guaranteed to run one pod on every node, including any node that joins later, which makes it the right mechanism for enforcing node-level configuration in a managed environment. The pod uses nsenter to enter the host's mount and process namespaces and modifies the node's SSH configuration directly, then sleeps so the DaemonSet doesn't churn through restarts.

This article walks through deploying a hardening DaemonSet that disables root login via SSH, and optionally disables the SSH service entirely, across all current and future nodes in your CMK cluster.

Disabling SSH entirely also removes the most common break-glass path onto a node. The article therefore also covers how to get back in when you need to.

Prerequisites

CMK Cluster
kubectl Access to the Cluster With Permission to Create Resources in kube-system
Ability to Deploy Privileged Workloads to the Cluster
Crusoe CLI Installed and Configured (Serial Console and VM Replacement Recovery Only)

Instructions

⚠️ Warning: Decide on your break-glass path before you disable SSH entirely. Once SSH is off, kubectl debug and kubectl exec only work while the node's kubelet is healthy. If a node goes NotReady, your remaining options are the serial console, which needs a local password set while you still have access, or replacing the VM. See How-To Setup Serial-Console to Access Your VM. If your control only requires disabling root login, follow the Tip in Step 1 and keep SSH for a non-root user.

Step 1: Create the Hardening DaemonSet Manifest

Save the following manifest as ssh-hardening.yaml. As written, it disables both root login and the SSH service on every node:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ssh-hardening
  namespace: kube-system
  labels:
    app: ssh-hardening
spec:
  selector:
    matchLabels:
      app: ssh-hardening
  template:
    metadata:
      labels:
        app: ssh-hardening
    spec:
      hostPID: true
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        - operator: Exists
      containers:
        - name: ssh-hardening
          image: busybox:1.37
          command:
            - /bin/sh
            - -c
            - |
              set -e

              echo "=== SSH Hardening: starting ==="

              # Disable root login via SSH
              nsenter -t 1 -m -- sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config

              # Disable SSH service entirely
              nsenter -t 1 -m -u -i -n -p -- systemctl stop sshd || true
              nsenter -t 1 -m -u -i -n -p -- systemctl disable sshd || true

              echo "=== SSH Hardening: done ==="

              # Keep the pod running so the DaemonSet doesn't restart in a loop
              exec sleep infinity
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 50m
              memory: 32Mi
          securityContext:
            privileged: true
      terminationGracePeriodSeconds: 5

A few notes on how this works:

hostPID: true combined with privileged: true lets the container use nsenter -t 1 to enter the namespaces of the node's init process (PID 1), so the sed and systemctl commands run against the host, not the container.
tolerations: - operator: Exists ensures the pod schedules onto every node, including tainted GPU nodes.
The script is idempotent. Every step converges to the same end state, so it is safe for the script to re-run whenever the pod restarts, including after a node reboot.
The trailing exec sleep infinity keeps the pod alive after hardening completes, so the DaemonSet controller doesn't treat it as crashed and restart it in a loop.

💡 Tip: If your audit control only requires disabling root login while keeping SSH available for non-root users, remove the two systemctl lines from the script. The PermitRootLogin no change alone satisfies that control. If you keep SSH enabled, also restart the daemon so the config change takes effect: add nsenter -t 1 -m -u -i -n -p -- systemctl restart sshd || true after the sed line.

ℹ️ Note: A new node is reachable over SSH from first boot until its hardening pod starts, which is usually seconds to a few minutes while the image pulls. The DaemonSet is not a network control. Restrict port 22 with your VPC security group rules if exposure during that window matters for your audit.

ℹ️ Note: Each hardening pod stays running as a privileged container in the host's PID namespace. Anyone who can kubectl exec into pods in kube-system can use it to get a root shell on the node. Scope pods/exec in kube-system with RBAC accordingly.

Step 2: Deploy the DaemonSet

Apply the manifest to your cluster:

kubectl apply -f ssh-hardening.yaml

Step 3: Verify the Hardening Applied

Confirm one pod is running per node:

kubectl get pods -n kube-system -l app=ssh-hardening -o wide

Check the logs on any pod to confirm the script completed:

kubectl logs -n kube-system -l app=ssh-hardening --tail=20

You should see the === SSH Hardening: done === line. If you disabled SSH entirely, an SSH connection attempt to a node's IP should now be refused.

Because this is a DaemonSet, any new node that joins the cluster, whether added manually to a node pool or brought up by the cluster autoscaler, will automatically receive a hardening pod and have the same configuration applied. No further action is needed when scaling.

⚠️ Warning: Once SSH is disabled, direct node access via SSH is no longer possible. For node-level troubleshooting, use kubectl debug node/<YOUR_NODE_NAME> -it --image=busybox or kubectl exec into a privileged pod instead. If you later need SSH restored on a node, you can deploy a similar DaemonSet that re-enables the service (systemctl enable --now sshd). See Common Issues below.

Example

A team preparing for a SOC 2 audit needs to demonstrate that root login is disabled on all compute infrastructure, including the nodes backing their CMK training cluster. Their cluster uses the autoscaler, so nodes come and go throughout the day; a one-time manual fix on each node would drift out of compliance as soon as a new node joined.

They deploy the ssh-hardening DaemonSet with only the PermitRootLogin change, keeping SSH available for their non-root break-glass user. Every existing node is hardened within seconds of the apply, and every autoscaled node is hardened automatically on join. For the audit evidence, they export the DaemonSet manifest and a kubectl logs sample showing the hardening ran cluster-wide.

Common Issues

Restoring SSH Access to Nodes

Deleting the DaemonSet does not revert the changes, because the edits live on each node's disk. Delete the ssh-hardening DaemonSet before you deploy one that re-enables SSH (systemctl enable --now sshd), so a hardening pod restart can't disable SSH again.

If a node is NotReady, neither kubectl debug nor kubectl exec can reach it. Use the serial console if you set a local password beforehand. Otherwise, replace the VM: see How-To Recover a CMK Node When a Full Boot Disk Blocks Access for the reset and delete-and-replace procedure.

⚠️ Warning: A replacement node is hardened again on join if the ssh-hardening DaemonSet is still deployed. Deleting a VM also destroys all node-local state, including the boot disk and local NVMe.

Additional Resources

Related to

ssh kubernetes cmk security hardening daemonset

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Article is closed for comments.