Summary
In large-scale training clusters that span multiple Infiniband leaf switches, efficient job scheduling can significantly impact performance. Users often need to optimize pod locality to ensure that NCCL traffic traverses InfiniBand links in the most efficient manner possible. Slurm addresses this need by allowing system administrators to define a topology.conf file, which maps out the physical and logical topology of nodes across the InfiniBand fabric. This configuration enables Slurm to make more informed scheduling decisions, enhancing overall cluster performance.
Reference Slurm Docs: https://slurm.schedmd.com/topology.conf.html
An example of a topology.conf file is as follows:
##################################################################
# Slurm's network topology configuration file for use with the
# topology/tree plugin
##################################################################
SwitchName=s0 Nodes=dev[0-5]
SwitchName=s1 Nodes=dev[6-11]
SwitchName=s2 Nodes=dev[12-17]
SwitchName=s3 Switches=s[0-2]
Rail Optimized Topology at Crusoe
At Crusoe, each VM scheduled within an InfiniBand fabric is assigned a metadata flag called pod_id. This pod_id identifies VMs that belong to the same InfiniBand Rail Optimized Pod. Within a Rail Optimized Pod of 32 hosts, the topology is structured such that GPU0 on each host connects to IB Leaf Switch 0, GPU1 connects to IB Leaf Switch 1, and so on. This design ensures optimal traffic routing and performance. For a deeper exploration of this topology, you can refer to NVIDIA’s blog here .
Generating Topology File
As a Crusoe Cloud customer, you can dynamically generate a topology.conf file by querying the Crusoe API or CLI, reflecting the real-time scheduling of your instances. However, keep in mind that VM placement within Crusoe is not guaranteed to be fully within the same pod. Additionally, if a VM migrates to a new host due to a hardware issue, its associated pod_id may change. In such cases, you will need to regenerate the topology file and reconfigure Slurm to reflect the updated placement.
To generate a topology file, start by querying the Crusoe CLI to retrieve details of your running instances in JSON format. For example, you can use the following command to list all h100-80gb-sxm-ib.8x instances in the RUNNING state within your account, saving the JSON output to a slurm.json file for further processing.
# crusoe compute vms list --types b200-180gb-sxm-ib.8x --states STATE_RUNNING -f json > vms.json
Then you can run the following topology.py python script to generate the topology file for you dynamically
import json
import re
import sys
from collections import defaultdict
# Default filenames - can be overridden via command line args
DEFAULT_INPUT = 'vms.json'
DEFAULT_OUTPUT = 'topology.conf'
def node_list_to_bracket_notation(nodes):
"""
Convert a list of nodes like 'b200-compute-node-1', 'b200-compute-node-2',
'b200-compute-node-3', 'b200-compute-node-5' to a compact form like
'b200-compute-node-[1-3,5]'.
Preserves zero-padding: if any input number has a leading zero (e.g.
'fignore-compute-node-001'), all numbers in the output are padded to the
width of the longest input number string. SLURM treats 'node[001-009]'
and 'node[1-9]' as different names, so the padding must round-trip.
"""
if not nodes:
return ""
# Extract the prefix and node numbers
prefix_pattern = re.compile(r'(.*?)(\d+)$')
node_prefix = None
node_numbers = [] # (int value, original string) pairs
for node in nodes:
match = prefix_pattern.match(node)
if match:
prefix, number_str = match.groups()
if node_prefix is None:
node_prefix = prefix
elif prefix != node_prefix:
# If we have different prefixes, fall back to comma-separated list
return ','.join(nodes)
node_numbers.append((int(number_str), number_str))
else:
# If any node doesn't match the pattern, fall back to comma-separated list
return ','.join(nodes)
# Detect zero-padding from the input: if any number string has a leading
# zero (and is longer than one char), treat the whole set as padded and
# use the longest string's width.
has_padding = any(s.startswith('0') and len(s) > 1 for _, s in node_numbers)
pad_width = max(len(s) for _, s in node_numbers) if has_padding else 0
def fmt(n):
return f"{n:0{pad_width}d}" if pad_width else str(n)
# Sort unique node numbers
sorted_numbers = sorted({n for n, _ in node_numbers})
# Group consecutive numbers into ranges
ranges = []
start = sorted_numbers[0]
prev = start
for num in sorted_numbers[1:] + [None]: # None sentinel to flush the last range
if num is None or num > prev + 1:
# End of a range
if prev == start:
ranges.append(fmt(start))
else:
ranges.append(f"{fmt(start)}-{fmt(prev)}")
if num is not None:
start = num
prev = num if num is not None else prev
# Construct the final string
if len(ranges) == 1:
if '-' in ranges[0]:
return f"{node_prefix}[{ranges[0]}]"
else:
return f"{node_prefix}{ranges[0]}"
else:
return f"{node_prefix}[{','.join(ranges)}]"
def generate_topology_conf(json_data):
"""
Generate SLURM topology.conf content from JSON metadata.
Groups nodes by pod_id and creates switch entries accordingly.
Uses bracket notation for consecutive node numbers.
NOTE: This always uses the VM's `name` field as it appears in the JSON
(e.g. "b200-compute-node-363"), regardless of the hostname actually
configured on the VM. SLURM must be configured to use these same names
as NodeName entries for the topology to match.
"""
# Group nodes by pod_id
pod_groups = defaultdict(list)
# Parse JSON if it's a string, otherwise use as is
if isinstance(json_data, str):
nodes = json.loads(json_data)
else:
nodes = json_data
# If the input is a single node, convert to list
if isinstance(nodes, dict):
nodes = [nodes]
# Group nodes by pod_id, using the JSON `name` field verbatim
for node in nodes:
pod_id = node['pod_id']
node_name = node['name'] # always use JSON name, ignore on-VM hostname
pod_groups[pod_id].append(node_name)
# Sort node names within each pod
for pod_id in pod_groups:
pod_groups[pod_id].sort()
# Generate topology.conf content
topology_content = []
switch_names = []
for i, (pod_id, nodes) in enumerate(sorted(pod_groups.items())):
switch_name = f"leaf{i}"
switch_names.append(switch_name)
# Use bracket notation for nodes
node_list = node_list_to_bracket_notation(nodes)
topology_content.append(f"SwitchName={switch_name} Nodes={node_list}")
# Add a final line connecting all switches together
if switch_names:
all_switches = f"leaf[0-{len(switch_names) - 1}]"
topology_content.append(f"SwitchName=spine0 Switches={all_switches}")
return '\n'.join(topology_content)
def main():
input_file = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_INPUT
output_file = sys.argv[2] if len(sys.argv) > 2 else DEFAULT_OUTPUT
try:
with open(input_file, 'r') as f:
data = json.load(f)
# Generate topology configuration
topology_conf = generate_topology_conf(data)
# Write to topology.conf file
with open(output_file, 'w') as f:
f.write(topology_conf)
print(f"Successfully generated {output_file} from {input_file}")
print("\nGenerated content:")
print(topology_conf)
except FileNotFoundError:
print(f"Error: {input_file} file not found")
except json.JSONDecodeError:
print(f"Error: Invalid JSON format in {input_file}")
except Exception as e:
print(f"Error: {str(e)}")
if __name__ == "__main__":
main()
This will generate a topology file like the following:
SwitchName=leaf0 Nodes=b200-compute-node-[003,006,010,013,019-020,041,046,048,052-053,057,061,063-064]
SwitchName=leaf1 Nodes=b200-compute-node-[001,005,008,011-012,016,024,026,028,030,032-033,035-036,044-045,065]
SwitchName=leaf2 Nodes=b200-compute-node-[000,017,022,031,042,047,050-051,060]
SwitchName=leaf3 Nodes=b200-compute-node-[004,034]
SwitchName=leaf4 Nodes=b200-compute-node-055
SwitchName=leaf5 Nodes=b200-compute-node-009
SwitchName=leaf6 Nodes=b200-compute-node-043
SwitchName=leaf7 Nodes=b200-compute-node-018
SwitchName=leaf8 Nodes=b200-compute-node-[007,014-015,021,038,040,049,054,056,062,066]
SwitchName=leaf9 Nodes=b200-compute-node-002
SwitchName=leaf10 Nodes=b200-compute-node-[023,025,027,029,037,039,058-059]
SwitchName=spine0 Switches=leaf[0-10]
From here, you can copy the output to a topology.conf file to the /etc/slurm directory within your Slurm headnode.
Additionally, Crusoe Infiniband fabrics resemble a dragonfly network topology, so for the topology.conf to take effect it is required to include the following Topology Parameters in /etc/slurm/slurm.conf .
# slurm.conf
...
TopologyPlugin=topology/tree
TopologyParam=dragonfly
...
At this point, please run scontrol reconfigure on the head node and restart slurmd on all compute nodes and slurmctld. This will make the system aware of the new configuration files.
For testing purposes, start with a simple job requesting a command type function like nvidia-smi -L using srun, an example might be: srun -vvv -N(x) -n(x) nvidia-smi -L.
The output should contain something like this (it will be fairly verbose output), the important parts are highlighted:
|
Additional Resources