Prolog Script Updates Are Not Taking Effect — Nodes Drain After Job Submission

Introduction

When you update a Slurm prolog script on a Crusoe Managed Slurm cluster, the change is expected to be visible on compute nodes immediately — /home is a shared PVC mounted across all login and worker pods. But a missing execute permission on the script file will cause slurmd to reject it with a Permission denied error, drain the node, and cancel any job that lands on it.

This article shows how to spot the permission problem, fix it, resume drained nodes, and prevent it from happening again. The root cause is simple — a missing chmod +x — but the blast radius is large: every node that tries to run the broken script will drain.

You may see errors like the following in Slurm logs:

error: run_command: prolog can not be executed (/home/scripts/10-gpu-sweep-prolog.sh) Permission denied
error: prolog didn't run: status:127 reason:Run command failed - configuration error
error: JobId=<id> prolog failed status=0:0
Could not launch JobId=<id> and not able to requeue it, cancelling job

Prerequisites

• Crusoe Managed Slurm Cluster Already Provisioned
• SSH Access to the Slurm Login Node
• Slurm User Account with scontrol Privileges on the Login Node

Instructions

Step 1: Ensure the Prolog Script Has Executable Permissions

After creating or updating the script on the login node, set executable permissions:

chmod +x /home/scripts/<YOUR_SCRIPT>.sh

Verify the permission is set:

ls -al /home/scripts/

The script should show -rwxr-xr-x (or similar with the execute bit set for the owner):

-rwxr-xr-x 1 root root 5377 May 21 07:28 prolog.sh

A file missing the execute bit will appear as:

-rw-r--r-- 1 root root 5377 May 21 07:28 prolog.sh   ← will fail

ℹ️ Note: Writing to a file with a text editor or copying it does not preserve or set executable permissions automatically. Always run chmod +x after creating or updating a prolog or epilog script.

Step 2: Return Drained Nodes to the Idle Pool

After fixing the script permissions, resume any nodes that were drained due to the failed prolog. From the login node:

# Check which nodes are drained and why
sinfo -R

# Resume a specific node
scontrol update NodeName=<NODE_NAME> State=resume

Step 3: Verify the Prolog Runs Correctly

Submit a minimal test job to confirm the prolog executes successfully before returning your full workload:

sbatch --nodes=1 --wrap="echo hello"
squeue

If the job runs and the node stays idle (not drained), the prolog is working correctly.

Step 4: Prevent the Issue From Recurring

When creating or updating prolog or epilog scripts, always follow this pattern to ensure the execute bit is set immediately:

# Create or update the script
cat > /home/scripts/prolog.sh << 'EOF'
#!/bin/bash
# your prolog logic here
exit 0
EOF

# Always set executable permissions immediately after
chmod +x /home/scripts/prolog.sh

ℹ️ Note: If you back up and restore scripts (e.g. from /home/<user>/scripts/prolog.backup), the backup may not have executable permissions. Always re-apply chmod +x after restoring from a backup.

Resolution

There are two related causes for this issue:

1. Missing executable permissions. When you create or overwrite a prolog script file, the file may not have the executable (+x) permission set. Slurm requires the prolog script to be executable. If it is not, slurmd cannot run it, logs a Permission denied error, and drains the node.

2. Script path lives under /home, which is shared — but permissions are per-file. The shared /home filesystem is mounted on all login and worker nodes. If your prolog script path is under /home (e.g. /home/scripts/prolog.sh), edits made on the login node will be visible on worker nodes immediately — no manual sync required. However, if the script was written without executable permissions, the update will still fail on every node.

The fix is simple but unforgiving: always run chmod +x immediately after creating or modifying any prolog or epilog script. A single missing execute bit drains every node that touches the script.

Example

A cluster operator adds a GPU health check to their prolog and deploys it:

cat > /home/scripts/gpu-check-prolog.sh << 'EOF'
#!/bin/bash
nvidia-smi > /dev/null 2>&1
if [ $? -ne 0 ]; then
    echo "[prolog] $(date) GPU not available on $(hostname)" >&2
    exit 1
fi
echo "[prolog] $(date) GPU check passed on $(hostname)" >> /home/prolog.log
exit 0
EOF

They forget chmod +x. Within minutes, jobs start draining nodes:

$ sinfo -R
REASON               USER      TIMESTAMP           NODELIST
Prolog failed        root      2026-05-27T14:02:00 gpu-[01-04]

All four GPU nodes drained because the script was not executable — even though the file content was correct and visible from every node. Running chmod +x /home/scripts/gpu-check-prolog.sh and then scontrol update NodeName=gpu-[01-04] State=resume resolves it in under a minute.

Related Articles

• Slurm Command Basics
• Slurm Prolog and Epilog Scripts