How-To Configure Topology-Aware Scheduling for Slurm Nodes

Last Updated: December 16th, 2025

Introduction

This guide explains how to configure topology-aware scheduling in Slurm for virtual machine (VM) environments. Topology-aware scheduling enhances job performance by ensuring that multi-node jobs are placed on nodes connected to the same network switch. This reduces network latency and improves communication speed.

In VM environments, traditional tools like UFM commands or ibswitches are not effective for discovering network topology. This guide provides a solution for automatically discovering VM placement and generating the necessary Slurm topology configuration.

Prerequisites

• Root or sudo access to the Slurm head node
• A Slurm cluster that is already configured and running
• Access to compute nodes for service restarts
• A Python environment and cloud platform access for VM topology discovery

Step-by-Step Instructions

1. Check Current Topology Status

First, verify that topology-aware scheduling is not already configured:

# Check current topology plugin
scontrol show config | grep -i topology

# Try to view topology (should be empty)
scontrol show topology

Expected output for unconfigured topology:

TopologyPlugin = topology/default
# Empty output from scontrol show topology

2. Determine Your Network Topology

The provided Python script (topology.py) will:

• Query your Crusoe API for VM placement information
• Group nodes by pod_id (nodes with the same pod_id are on the same network fabric)
• Generate a topology.conf file based on actual VM locations

To use the script, you need to set your project_id within the topology.py file. You can also customize the exclude_list within the script to specify VMs that should not be included in the topology.

topology.py Script

To utilize this script, copy the content below into a file named topology.py on your system. Remember to replace <project-id> with your actual project ID and adjust the exclude_list as needed.

import json
import hmac
import hashlib
import base64
import datetime
import requests
import configparser
from pathlib import Path

class TopologyGenerator:
    def __init__(self):
        self.config = self.load_config()
        self.api_access_key = self.config['default'].get('access_key_id').replace('"', '')
        self.api_secret_key = self.config['default'].get('secret_key').replace('"', '')
        # IMPORTANT: Replace <project-id> with your actual project ID
        self.project_id = "<project-id>"
        self.api_version = "/v1alpha5"
        self.base_url = "https://api.crusoecloud.com"
        # Customize this list to exclude specific VMs from the topology
        self.exclude_list = [
            "slurm-head-node-0", "slurm-login-node-0", "slurm-nfs-node-0"
        ]

    def load_config(self):
        """Loads API configuration from ~/.crusoe/config."""
        config = configparser.ConfigParser()
        config_path = Path.home() / '.crusoe' / 'config'
        config.read(config_path)
        if 'default' not in config:
            raise ValueError("Default section not found in ~/.crusoe/config")
        return config

    def generate_signature(self, request_path, request_verb, query_params):
        """Generates the HMAC-SHA256 signature for API requests."""
        dt = datetime.datetime.now(datetime.UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z")
        payload = (
            self.api_version + request_path + "\n" +
            query_params + "\n" +
            request_verb + "\n" +
            f"{dt}\n"
        )
        decoded_secret_key = self.decode_base64_urlsafe(self.api_secret_key)
        signature_hash = hmac.new(
            decoded_secret_key,
            msg=payload.encode('utf-8'),
            digestmod=hashlib.sha256
        ).digest()
        signature = base64.urlsafe_b64encode(signature_hash).decode('ascii').rstrip("=")
        return dt, signature

    @staticmethod
    def decode_base64_urlsafe(data):
        """Decodes a base64url-encoded string."""
        data = data.replace('-', '+').replace('_', '/')
        padding = '=' * (4 - len(data) % 4)
        return base64.b64decode(data + padding)

    def make_api_request(self, request_path, request_verb, query_params=""):
        """Makes an authenticated API request to Crusoe Cloud."""
        dt, signature = self.generate_signature(request_path, request_verb, query_params)
        headers = {
            'X-Crusoe-Timestamp': dt,
            'Authorization': f'Bearer 1.0:{self.api_access_key}:{signature}'
        }
        url = self.base_url + self.api_version + request_path
        response = requests.get(url, headers=headers)
        if response.status_code != 200:
            raise Exception(f"API request failed with status code {response.status_code}: {response.text}")
        return response.json()

    def get_instances(self):
        """Fetches VM instance information from the Crusoe Cloud API."""
        request_path = f"/projects/{self.project_id}/compute/vms/instances"
        data = self.make_api_request(request_path, "GET")
        # Optionally save data to a JSON file for debugging
        with open('instances.json', 'w') as f:
            json.dump(data, f, indent=4)
        items = data.get('items', [])
        if not items:
            raise Exception("No instances found in the API response.")
        return items

    def group_nodes_by_pod(self, instances):
        """Groups VM instances by their pod_id."""
        pod_dict = {}
        for instance in instances:
            name = instance.get('name')
            pod_id = instance.get('pod_id')
            if not name:
                print(f"Warning: Instance missing name. Skipping instance: {instance}")
                continue
            node_name = name.strip()
            if node_name in self.exclude_list:
                continue
            if pod_id:
                pod_name = f"pod_{pod_id}"
            else:
                # Assign to a default 'no_pod' if pod_id is missing
                pod_name = "no_pod"
            if pod_name not in pod_dict:
                pod_dict[pod_name] = []
            pod_dict[pod_name].append(node_name)
        return pod_dict

    def generate_topology_content(self, pod_dict):
        """Generates the Slurm topology configuration lines."""
        topology_lines = []
        # Sort pods by number of nodes, from most to least for better readability
        sorted_pods = sorted(pod_dict.items(), key=lambda x: len(x[1]), reverse=True)
        for pod_name, nodes in sorted_pods:
            node_list = ','.join(sorted(nodes)) # Sort nodes alphabetically within each pod
            topology_lines.append(f"SwitchName={pod_name} Nodes={node_list}")
        
        # Create the root switch that connects all pod switches
        pod_switches = ','.join([pod[0] for pod in sorted_pods])
        topology_lines.append(f"SwitchName=root Switches={pod_switches}")
        return topology_lines

    def write_topology_file(self, topology_lines):
        """Writes the generated topology configuration to topology.conf."""
        with open('topology.conf', 'w') as topo_file:
            topo_file.write("# Generated by script\n")
            topo_file.write("# Slurm topology configuration file\n")
            topo_file.write("\n".join(topology_lines))

    def generate_topology(self):
        """Main function to fetch instances, group them, and generate the topology file."""
        print("Fetching VM instances from Crusoe Cloud API...")
        instances = self.get_instances()
        print("Grouping nodes by pod ID...")
        pod_dict = self.group_nodes_by_pod(instances)
        
        # Calculate pod counts and sort from most to least for comments
        pod_counts = {pod: len(nodes) for pod, nodes in pod_dict.items()}
        sorted_pod_counts = sorted(pod_counts.items(), key=lambda item: item[1], reverse=True)
        
        # Prepare comment lines for the topology file
        comment_lines = ["# Pod counts:"]
        for pod, count in sorted_pod_counts:
            comment_lines.append(f"# {pod}: {count}")
        
        print("Generating topology content...")
        topology_lines = self.generate_topology_content(pod_dict)
        
        # Combine comments, an empty line, and topology content
        final_topology = comment_lines + [""] + topology_lines
        
        print("Writing topology.conf file...")
        self.write_topology_file(final_topology)
        print("topology.conf has been generated successfully in the current directory.")
        print("Remember to move it to /etc/slurm/ and update slurm.conf.")

if __name__ == "__main__":
    topology_generator = TopologyGenerator()
    try:
        topology_generator.generate_topology()
    except Exception as e:
        print(f"An error occurred: {e}")

Run the script:

python3 topology.py

This script will generate a topology.conf file in the current directory.

Important: VM placement can change due to migrations. If VMs move, you should re-run the topology.py script to regenerate an updated topology file.

3. Create or Update topology.conf File

The topology.py script will generate a topology.conf file. You need to move this file to the /etc/slurm/ directory:

sudo mv topology.conf /etc/slurm/topology.conf

The generated topology.conf will include comments indicating the pod counts, sorted from most to least nodes, followed by the topology configuration.

Example of a generated topology.conf for a VM environment:

# Generated by script
# Slurm topology configuration file
# Pod counts:
# pod_12345: 8
# pod_67890: 4

SwitchName=pod_12345 Nodes=slurm-compute-node-0,slurm-compute-node-1,slurm-compute-node-2,slurm-compute-node-3,slurm-compute-node-4,slurm-compute-node-5,slurm-compute-node-6,slurm-compute-node-7
SwitchName=pod_67890 Nodes=slurm-compute-node-8,slurm-compute-node-9,slurm-compute-node-10,slurm-compute-node-11
SwitchName=root Switches=pod_12345,pod_67890

4. Enable Topology Plugin

Edit the main Slurm configuration file:

sudo nano /etc/slurm/slurm.conf

Find the line with TopologyPlugin and change it to:

TopologyPlugin=topology/tree
TopologyParam=dragonfly

Note: If the line doesn't exist, add it to the configuration file.

5. Restart Slurm Services

On the head node:

sudo systemctl restart slurmctld

On all compute nodes (not on the head node or login nodes):

sudo systemctl restart slurmd

Important: You must restart slurmd on all compute nodes to sync the configuration. Without this, you may encounter plugin compatibility errors or segmentation faults. Do not run slurmd on the head node unless it is also configured as a compute node.

6. Verify Configuration

Test that topology-aware scheduling is working:

# Check topology plugin is active
scontrol show config | grep -i topology

Expected output:

TopologyParam           = dragonfly
TopologyPlugin          = topology/tree

# Verify the topology.conf file syntax
cat /etc/slurm/topology.conf

Expected output:

# Generated by script
# Slurm topology configuration file
# Pod counts:
# pod_467e0e97-70d8-55e9-6268-92feeeae30df: 2

SwitchName=pod_467e0e97-70d8-55e9-6268-92feeeae30df Nodes=slurm-compute-node-0,slurm-compute-node-1
SwitchName=root Switches=pod_467e0e97-70d8-55e9-6268-92feeeae30df

# Check if slurm.conf has correct topology settings
grep -i topology /etc/slurm/slurm.conf

Expected output:

TopologyPlugin=topology/tree
TopologyParam=dragonfly

# View topology hierarchy
scontrol show topology

Expected output:

SwitchName=pod_467e0e97-70d8-55e9-6268-92feeeae30df Level=0 LinkSpeed=1 Nodes=slurm-compute-node-0,slurm-compute-node-1
SwitchName=root Level=1 LinkSpeed=1 Switches=pod_467e0e97-70d8-55e9-6268-92feeeae30df

# Check node status
sinfo -N

Expected output:

NODELIST              NODES PARTITION STATE
slurm-compute-node-0      1    normal idle
slurm-compute-node-1      1    normal idle

7. Test Topology-Aware Scheduling

Submit a test job to verify placement:

# Submit multi-node job
sbatch --nodes=2 --wrap="sleep 60"

# Check job placement
squeue --format="%i %N %T"

# View job details
scontrol show job <jobid>

# Example of running a multi-node job with verbose output
srun -vvv -N2 nvidia-smi -L

Expected output should include lines showing the topology plugin is active:

srun: topology/tree: init: topology tree plugin loaded
srun: debug2: Tree head got back 0 looking for 2
srun: debug2: Tree head got back 1
srun: debug2: Tree head got back 2

Example

Scenario

A Slurm cluster with nodes distributed across different pod_ids, as discovered by the topology.py script.

Configuration Files

/etc/slurm/topology.conf:

# Generated by script
# Slurm topology configuration file
# Pod counts:
# pod_a1b2c3d4: 4
# pod_e5f6g7h8: 2

SwitchName=pod_a1b2c3d4 Nodes=node-0,node-1,node-2,node-3
SwitchName=pod_e5f6g7h8 Nodes=node-4,node-5
SwitchName=root Switches=pod_a1b2c3d4,pod_e5f6g7h8

/etc/slurm/slurm.conf (add/modify):

TopologyPlugin=topology/tree
TopologyParam=dragonfly

Result

$ scontrol show topology
SwitchName=pod_a1b2c3d4 Level=0 LinkSpeed=1 Nodes=node-0,node-1,node-2,node-3
SwitchName=pod_e5f6g7h8 Level=0 LinkSpeed=1 Nodes=node-4,node-5
SwitchName=root Level=1 LinkSpeed=1 Switches=pod_a1b2c3d4,pod_e5f6g7h8

When submitting a multi-node job, Slurm will preferentially place all nodes on the same pod_id (representing the same network fabric) for optimal network performance.

Troubleshooting

Problem: Jobs not being placed optimally

Solution:

1. Verify node names in /etc/slurm/topology.conf exactly match with sinfo -N
2. Check that SelectType is configured for resource-aware scheduling:

# Check current SelectType setting
scontrol show config | grep SelectType

Expected output:

SelectType=select/cons_tres

If not set correctly, edit /etc/slurm/slurm.conf:

sudo nano /etc/slurm/slurm.conf

Add or modify this line:

SelectType=select/cons_tres

Then restart services:

sudo systemctl restart slurmctld
sudo systemctl restart slurmd  # On all compute nodes

3. Review job submission parameters and use topology constraints if needed:

# Request nodes from the same switch
sbatch --switches=1 --nodes=2 myjob.sh

Problem: VM topology changes after migration

Solution:

• Re-run the topology.py script
• Update /etc/slurm/topology.conf with the new VM placement
• Restart Slurm services

Checking Topology of Currently-Allocated Nodes

Once topology-aware scheduling is configured, you can check the topology of allocated nodes using:

# View complete topology hierarchy
scontrol show topology

# Check currently running jobs and their nodes
squeue --format="%N %j %T" --states=RUNNING

# Get detailed node information including topology
scontrol show node <nodename>

# See which nodes are allocated
sinfo -t alloc

# Check specific job placement
scontrol show job <jobid>

Additional Resources

• Slurm Topology Guide