How-To: Add SSH Keys to CMK Nodes After Node Pool Creation

Introduction

If you created a Crusoe Managed Kubernetes (CMK) node pool without specifying an SSH key, its nodes come up with no authorized key, and you won't be able to SSH into them. There is currently no console option to add an SSH key to nodes after a node pool has been created.

This guide explains how to add one or more public SSH keys to your existing nodes by deploying a small DaemonSet.

The key(s) are supplied to the DaemonSet through a Kubernetes Secret, and the DaemonSet appends them to /home/ubuntu/.ssh/authorized_keys on every node in the cluster. Because it runs as a DaemonSet, it also applies the key(s) automatically to any new node that later joins a pool (for example, when a pool scales up).

Prerequisites

• A Running CMK Node Pool
• Crusoe CLI Installed and Configured for the Project That Owns the Cluster
• kubectl Installed on Your Local Machine
• The Public SSH Key(s) You Want to Add

💡 Tip: If you don't have a key pair, generate one with ssh-keygen -t ed25519 -C "you@example.com", which creates ~/.ssh/id_ed25519 (private) and ~/.ssh/id_ed25519.pub (public). Only the public key is used in this guide.

Instructions

Step 1: Connect kubectl to Your Cluster

Download the cluster's credentials so kubectl can reach it:

crusoe kubernetes clusters get-credentials <cluster-name-or-id>

By default this writes to ~/.kube/config and sets it as your current context.

Confirm connectivity and note the node's external IP:

kubectl get nodes -o wide

Record the value in the EXTERNAL-IP column — you'll use it in the final step.

ℹ️ Note: If the node pool was created without public IPs, the EXTERNAL-IP column shows <none>. In that case, connect from a jump host inside the same VPC using the INTERNAL-IP instead.

Step 2: Gather the Public Key(s) You Want to Add

• Create a file named keys.pub and add one public key per line — one line for each person who needs access.
• A user can print their own public key with cat ~/.ssh/id_ed25519.pub.

⚠️ Warning: Only ever use the public key. Never request or handle anyone's private key.

Step 3: Store the Key(s) in a Kubernetes Secret

Create a Secret in the kube-system namespace from your keys.pub file:

kubectl -n kube-system create secret generic ssh-public-keys \
  --from-file=authorized_keys=./keys.pub

This keeps the key(s) out of the DaemonSet manifest — useful if you store manifests in version control — and makes rotation easier.

Step 4: Create the DaemonSet Manifest

Save the following as append-ssh-key.yaml. It mounts the Secret read-only at /keys and appends any keys that aren't already present on the node:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: append-ssh-key
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: append-ssh-key
  template:
    metadata:
      labels:
        name: append-ssh-key
    spec:
      tolerations:
      - operator: Exists
      containers:
      - name: writer
        image: alpine:latest
        command: ["/bin/sh", "-c"]
        args:
        - |
          SRC="/keys/authorized_keys"
          DEST="/ssh_dir/authorized_keys"

          chown 1000:1000 /ssh_dir
          chmod 700 /ssh_dir

          if [ ! -f "$DEST" ]; then
            touch "$DEST"
            chown 1000:1000 "$DEST"
            chmod 600 "$DEST"
          fi

          ADDED=0
          while IFS= read -r line || [ -n "$line" ]; do
            [ -z "$line" ] && continue
            if ! grep -qF -- "$line" "$DEST"; then
              echo "$line" >> "$DEST"
              ADDED=$((ADDED+1))
            fi
          done < "$SRC"

          echo "Done. Added $ADDED new key(s)."

          while true; do sleep 3600; done
        volumeMounts:
        - name: ssh-directory
          mountPath: /ssh_dir
        - name: ssh-key
          mountPath: /keys
          readOnly: true
      volumes:
      - name: ssh-directory
        hostPath:
          path: /home/ubuntu/.ssh
          type: DirectoryOrCreate
      - name: ssh-key
        secret:
          secretName: ssh-public-keys

• Why tolerations: - operator: Exists? GPU node pools (and any pool created with custom taints) reject pods that don't tolerate their taints. The blanket toleration ensures the DaemonSet schedules on every node in the cluster — which is the whole point of using it to distribute keys. Without it, tainted nodes silently never receive the key.
• Why DirectoryOrCreate? Because the node pool was created without an SSH key, the /home/ubuntu/.ssh directory may not exist yet. DirectoryOrCreate creates it if it's missing; the container then sets the correct ownership (1000:1000, the ubuntu user) and permissions so sshd accepts the key. If you use type: Directory instead, the pod will stay stuck in Pending on any node where the directory is absent.

Step 5: Apply the DaemonSet

Apply the manifest to your cluster:

kubectl apply -f append-ssh-key.yaml

Step 6: Verify the Rollout and Confirm the Key

Wait for the DaemonSet to roll out and check that a pod is running on each node:

kubectl -n kube-system rollout status ds/append-ssh-key
kubectl -n kube-system get pods -l name=append-ssh-key -o wide

Check the logs to confirm the key was written:

kubectl -n kube-system logs -l name=append-ssh-key --tail=5

You should see a line like Done. Added 1 new key(s). (a value of 0 means the key was already present).

Optionally, confirm the key is present on the node directly:

kubectl -n kube-system exec ds/append-ssh-key -- cat /ssh_dir/authorized_keys

Step 7: Connect via SSH

Using the EXTERNAL-IP recorded in Step 1, log in as the ubuntu user with the matching private key:

ssh -i ~/.ssh/id_ed25519 ubuntu@<EXTERNAL-IP>

ℹ️ Note: The script appends keys once, when its pod starts. To add or rotate keys later, update the Secret and restart the DaemonSet so it re-runs on every node: kubectl -n kube-system create secret generic ssh-public-keys --from-file=authorized_keys=./keys.pub --dry-run=client -o yaml | kubectl apply -f - followed by kubectl -n kube-system rollout restart ds/append-ssh-key. The script is append-only — removing a key still requires editing authorized_keys on the nodes (or via a modified script).

Example

Suppose you created a node pool without an SSH key and now need to grant a teammate access to a node whose EXTERNAL-IP is 203.0.113.24.

1. You put the key in keys.pub:

   # keys.pub
   ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleKeyValue teammate@example.com

   Then create the Secret:

   $ kubectl -n kube-system create secret generic ssh-public-keys \
       --from-file=authorized_keys=./keys.pub
   secret/ssh-public-keys created

2. You apply it and check the rollout:

   $ kubectl apply -f append-ssh-key.yaml
   daemonset.apps/append-ssh-key created
   
   $ kubectl -n kube-system rollout status ds/append-ssh-key
   daemon set "append-ssh-key" successfully rolled out
   
   $ kubectl -n kube-system logs -l name=append-ssh-key --tail=5
   Done. Added 1 new key(s).

3. Your teammate then connects successfully:

   $ ssh ubuntu@203.0.113.24
   Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 5.15.0-generic x86_64)
   ubuntu@np-xxxxxxxx-1:~$

The key is now present on every node in the cluster, and any node added later will receive it automatically.

Additional Resources

• Managing your CMK Clusters — retrieve cluster credentials and list clusters
• Managing your Node Pools
• How-To Add Additional SSH Keys to Instances
• Crusoe CLI Reference