Slurm login node restarts with OOMKilled (exit code 137)

Introduction

On Crusoe Managed Slurm, the login node is the shared entry point for your whole team. It runs as a pod on the underlying Crusoe Managed Kubernetes (CMK) cluster, and every user's SSH session, editor, and interactive process runs inside that one login container, drawing from the same memory allocation. When that memory is exhausted, the kernel's OOM killer terminates the login container and Kubernetes restarts it, disconnecting everyone at once.

The symptom is a login pod whose container restarts repeatedly. Active SSH sessions drop, and the pod may briefly fail its readiness probe with a message like Readiness probe failed: command timed out: "test -S /run/slurm/sack.socket". The container's last state shows OOMKilled with exit code 137.

The most common cause is not Slurm itself, and not a Crusoe-installed component. It is remote IDE backends. When a user connects Cursor or VS Code Remote-SSH to the login node, the IDE starts server processes on the node: a file watcher, an extension host, and language servers.

The file watcher's memory grows with the number of files it monitors, not with their size. If a user opens a workspace that contains dataset, checkpoint, or wandb directories with a very large number of files, a single file watcher process can grow to tens of GB. Language servers add to this, and some run with heaps allowed to reach 32 GB (--max-old-space-size=32768). With several users connected, these processes stack until the login container runs out of memory.

Python jobs run directly on the login node (preprocessing scripts, data pipeline tests) are a related risk. They cause sudden memory bursts, and on a node already running close to its limit from IDE load, a burst like that can trigger the OOM kill. Such jobs belong on compute nodes.

Prerequisites

• Crusoe Managed Slurm Cluster
• SSH Access to the Login Node
• Kubeconfig Access to the Underlying CMK Cluster (Diagnosis Steps Only)

Instructions

Step 1: Confirm the Restarts Are OOM Kills

Find the login pod. The RESTARTS column gives a first indication:

kubectl -n slurm get pods | grep -i login

Check the login container's restart count and last exit reason:

kubectl -n slurm get pod <LOGIN_POD_NAME> -o jsonpath='{range .status.containerStatuses[*]}{.name}: restarts={.restartCount}, lastExit={.lastState.terminated.reason} at {.lastState.terminated.finishedAt}, exitCode={.lastState.terminated.exitCode}{"\n"}{end}'

An OOM kill looks like this:

login: restarts=4, lastExit=OOMKilled at 2026-07-28T13:12:44Z, exitCode=137

You can also check recent events on the pod:

kubectl -n slurm get events --field-selector involvedObject.name=<LOGIN_POD_NAME> --sort-by=.lastTimestamp | tail -10

Events typically show the readiness probe failures and BackOff restarts around each kill. Kubernetes only retains events for a short window (one hour by default), so the containerStatuses output above is the more reliable record of the OOM kill itself.

ℹ️ Note: Exit code 137 is 128 + 9: the process was ended with SIGKILL. Together with the reason OOMKilled, it means the kernel OOM killer ended the container. On cgroup v2 nodes, Kubernetes kills every process in the container together when any one of them is OOM-killed, which is why a single user's runaway process disconnects everyone. If the reason is not OOMKilled, or the exit code is different, the restarts have another cause and this article does not apply.

Step 2: Identify What Is Consuming Memory

First, find the ceiling that actually triggers the kill: the login container's memory limit. From a machine with kubeconfig access:

kubectl -n slurm get pod <LOGIN_POD_NAME> -o jsonpath='{range .spec.containers[*]}{.name}: request={.resources.requests.memory}, limit={.resources.limits.memory}{"\n"}{end}'

Run the remaining commands directly on the login node. Current usage against the container's limit, as the kernel sees it:

awk '{ if ($1 == "max") print FILENAME ": no limit"; else printf "%s: %.1f GiB\n", FILENAME, $1/1073741824 }' /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current

ℹ️ Note: free -h inside a container reports the memory of the whole Kubernetes node, not the container's limit. Use memory.max and memory.current above to judge how close the login container is to being OOM-killed.

Top consumers:

ps aux --sort=-rss | head -11

Top processes with age, in GB, with the full command line:

ps -eo user:12,pid,etime,rss,args --sort=-rss --no-headers | head -8 | awk '{cmd=""; for (i=5; i<=NF; i++) cmd=cmd " " $i; printf "%-12s %8s %11s %7.1fGB%s\n", $1, $2, $3, $4/1048576, cmd}'

Total memory per user:

ps aux --sort=-rss | awk 'NR>1 {sum[$1]+=$6} END {for (u in sum) printf "%-12s %6.1f GB\n", u, sum[u]/1048576}' | sort -k2 -rn

RSS counts shared pages once per process, so per-user totals overstate real usage. Treat them as a ranking of who to look at, not an exact accounting.

Look for processes under ~/.cursor-server/ or ~/.vscode-server/. A bootstrap-fork --type=fileWatcher process with a large RSS is the classic signature. Also look for node processes started with a large --max-old-space-size value; these are language servers.

💡 Tip: For immediate relief, the affected user can kill their own IDE server processes with pkill -u "$(whoami)" -f cursor-server (or vscode-server). The IDE restarts them cleanly on the next connection. Run it from a plain SSH session rather than the IDE's integrated terminal, which is closed along with the server.

Step 3: Exclude Large Directories from IDE File Watching

What matters to the file watcher is the number of files, not their size in GB. Dataset, checkpoint, and experiment-tracking directories often contain a very large number of files, and the watcher's memory grows with each one it tracks.

Each user connecting an IDE to the login node should exclude those paths in their remote or workspace settings (settings.json in Cursor or VS Code):

{
  "files.watcherExclude": {
    "**/datasets/**": true,
    "**/checkpoints/**": true,
    "**/wandb/**": true,
    "**/.venv/**": true
  },
  "search.exclude": {
    "**/datasets/**": true,
    "**/checkpoints/**": true
  }
}

Alternatively, open a narrower workspace root that does not contain those directories at all, for example the code repository rather than the parent directory that also holds data and checkpoints.

Step 4: Keep Compute Jobs Off the Login Node

This step is a best practice for any shared login node rather than a fix for the OOM kills themselves. Python processes and data jobs run directly on the login node cause sudden memory bursts, and on a node already carrying heavy IDE load, a burst can be what tips it over. These jobs belong on compute nodes, where they get dedicated memory and cannot affect the shared login node:

srun --mem=64G python preprocess.py

For interactive work, request a shell on a compute node instead:

srun --mem=64G --pty bash

Or submit them as batch jobs with sbatch.

Step 5: Monitor

After making the changes, re-run the commands from Step 2 periodically. memory.current should sit well below the container's memory.max, and the per-user totals will show quickly if one workspace starts growing again.

Resolution

Login node OOM kills on Crusoe Managed Slurm are typically caused by memory exhaustion inside the login container from remote IDE backends, not by a fault in Slurm or any Crusoe-managed component.

File watchers grow with the number of files in the open workspace, so a workspace that includes dataset, checkpoint, or experiment-tracking directories can push a single watcher to tens of GB, and language server heaps add to the standing load across every connected user. Excluding high-file-count paths from IDE file watching and opening narrower workspace roots removes that load at the source. Keeping compute jobs off the login node removes the bursts that tip an already loaded node over.

ℹ️ Note: These fixes live in each user's IDE settings and working habits, not in the cluster configuration, so Managed Slurm upgrades do not undo them. If the login container's memory limit is still too small for your team's normal load after these changes, contact Crusoe Support to discuss login node sizing.

Example

A team of several engineers all connect Cursor to their Slurm login node. One engineer opens a workspace whose root contains a checkpoint directory. Over two days, that user's file watcher process grows past 45 GB, and combined with other users' file watchers and language servers, the login container ends up running within a few GB of its memory limit. The kernel OOM killer terminates the login container, dropping every user's session, and the container keeps restarting as the watchers grow back. After the team adds files.watcherExclude entries for their checkpoint and dataset paths and opens narrower workspace roots, memory settles at less than half the container's limit and the restarts stop.

Related Articles

• How-To Customize slurm.conf on Crusoe Managed Slurm
• How-To Diagnose and Resume a Drained Slurm Node on Crusoe Managed Slurm
• FAQ: Slurm Commands Basics
• Getting Started with SLURM on Crusoe Cloud
• How-To Recover a Slurm Node Drained Due to Full Root Disk

Additional Resources

• Crusoe Managed Slurm Overview