How-To Shut Down Slurm Cluster Nodes While Preserving NFS Storage

Last Updated: Dec 15th, 2025

Introduction

This guide explains how to shut down Slurm cluster compute nodes, head nodes, and login nodes while preserving the NFS storage node when using the Crusoe Cloud Slurm deployment. This is useful for cost optimization during periods of inactivity, maintenance windows, or when you need to preserve data while temporarily scaling down your cluster infrastructure.

Unlike simply stopping VMs through your Crusoe Cloud's console, using Terraform to manage Slurm cluster shutdowns ensures proper cluster state management and prevents job scheduling conflicts. By following this guide, you can significantly reduce your costs while maintaining all your shared data, user directories, and applications on the NFS server for future use.

Prerequisites

• Access to your Slurm cluster's Terraform configuration files
• Terraform installed and configured on your local machine
• Appropriate permissions to modify and apply Terraform configurations
• Understanding of your current cluster configuration in terraform.tfvars

Step-by-Step Instructions

Approach 1: Shut Down Compute Nodes Only (Partial Shutdown)

1. Locate Your Terraform Configuration
   • Navigate to your Slurm cluster's Terraform directory
   • Open the terraform.tfvars file in your preferred text editor
   • Note: This approach keeps head and login nodes running while stopping only compute nodes
2. Modify Compute Node Count
   • Find the line containing slurm_compute_node_count

   • Change the value to 0:

     slurm_compute_node_count = 0

   • Save the file
3. Apply the Configuration

   • Run the following command in your terminal:

     terraform apply

   • Review the planned changes when prompted
   • Type yes to confirm and apply the changes
   • Note: The Ansible playbook will automatically run as part of the apply process

Approach 2: Complete Cluster Shutdown (Full Shutdown)

1. Locate Your Variables Configuration
   • Navigate to your Slurm cluster's Terraform directory
   • Open the variables.tf file in your preferred text editor
   • Note: This approach shuts down all nodes except the NFS server
2. Modify All Node Count Defaults
   • Find the variable definitions for each node type

   • Set the default value to 0 for each:

     variable "slurm_head_node_count" {
            description = "Number of head nodes"
            type        = number
            default     = 0
          }
          
          variable "slurm_login_node_count" {
            description = "Number of login nodes"
            type        = number
            default     = 0
          }
          
          variable "slurm_compute_node_count" {
            description = "Number of compute nodes"
            type        = number
            default     = 0
          }

   • Save the file
3. Apply the Configuration

   • Run the following command in your terminal:

     terraform apply

   • Review the planned changes when prompted
   • Type yes to confirm and apply the changes
   • Wait for Terraform to complete the shutdown process
4. Verify NFS Node Remains Running
   • Check your Crusoe Cloud's console to confirm only the NFS node is still running
   • The NFS node should remain active and accessible

Troubleshooting

Issue: Nodes don't shut down after terraform apply

• Resolution: Check the Terraform output for errors. Verify the configuration syntax in your files. Try running terraform plan first to preview changes

Issue: Unable to access NFS data after shutdown

• Resolution: Ensure the NFS node is still running. Check network security groups and firewall rules. Verify NFS service is running on the remaining node

Additional Resources

• Crusoe Cloud Slurm GitHub Repository