Last Updated: December 4th, 2025
Introduction
A distributed PyTorch job requires multiple individual PyTorch processes on different nodes to coordinate their activities. This coordination uses a low-bandwidth control plane called Rendezvous which specifies a key-value store at a designated network endpoint, and is in addition to the high-bandwidth InfiniBand network that GPUs use for collective communications. All the processes in the group write to and read from the store to discover each other, coordinate the ProcessGroup membership, and synchronize training activities. PyTorch supports various Rendezvous backends, including etcd, but the one most commonly used in Crusoe SLURM clusters is c10d. This is because PyTorch is able to start the c10 process itself on a designated master node - so you don't need to worry about running and maintaining a separate service. The code snippet below shows the relevant parts of a SLURM batch script (this is not a runnable example):
...
# Get the master node hostname
MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)
MASTER_PORT=29500
...
srun torchrun --rdzv_id=$SLURM_JOB_ID --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT my_training_program.py
...
# rdzv_id: A unique identifier for all the processes related to this job
# rdzv_backend: The type of KV store to be used for Rendezvous - c10d, etcd or Identifying Rendezvous problems
Because Rendezvous plays an essential role in creation of the process group, any problems manifest before any training cycles have even occurred. Each node connects to the Rendezvous service with a default timeout of one minute, so it's a strong possibility that a Torchrun job that fails after 1 minute is suffering from Rendezvous service connection problems.
In the error log of such a job, you will see a timeout error similar to:
[E1205 22:19:56.415927090 socket.cpp:1031] [c10d] The client socket has timed out after 60000ms while trying to connect to (172.27.31.26, 29500).
[W1205 22:19:56.416166836 TCPStore.cpp:340] [c10d] TCP client failed to connect/validate to host 172.27.31.26:29500 - retrying (try=0, timeout=60000ms, delay=23511ms): The client socket has timed out after 60000ms while trying to connect to (172.27.31.26, 29500).
Main causes and solutions for Rendezvous socket timeout errors
Cause 1: No Rendezvous service was started at the specified location
When a torchrun process starts up on any node and rdzv_endpoint is specified and the rdzv_backend is c10d, the process inspects the hostname/address component of rdzv_endpoint and compares it with the outputs of various Linux 'hostname' commands that it executes on the host OS to try and ascertain if it is the master node. If the 'hostname' output matches the rdzv_endpoint hostname, the torchrun process knows that it has to act as the master, and it starts up the c10d backend service listening on the specified master port.
Common reasons why this might cause no Rendezvous service to start:
1 - the hostname or address information provided in rdzv_endpoint do not match any of the 'hostname' outputs from any of the nodes in the torchrun process group. As a result, none of the nodes attempts to start a c10d backend. They will still try to connect to whatever the specified rdzv_endpoint is and these connection attempts will all time out.
Solutions: Run the 'hostname' and 'hostname -i' commands on the compute nodes in your cluster. If you specify the rdzv_endpoint by hostname (which is the method preferred by the PyTorch master node determination logic, and is what Crusoe recommends), then 'hostname' should output the node's hostname in a format that matches how the Slurm compute nodes are named (example: slurm-compute-node-0). If you specify rdzv_endpoint by IP address, then hostname -i should output the host's primary IP address and not 127.0.0.1. If hostname -i does return 127.0.0.1, you can fix it by updating /etc/hosts to include the host's primary IP address instead, for example:
172.27.31.26 slurm-compute-node-0.us-east1-a.compute.internal slurm-compute-node-0
127.0.0.2 localhost2 - The specified port is already in use by another process, or the user starting the torchrun process does not have permissions to use the specified port (which could happen if you specify a port number < 1024).
Solution: Use the default c10d port 29500, which is an ephemeral port and unlikely to be blocked for user capability reasons, or to be in use by another process. To verify that the port is available, run sudo ss -tulnp
grep 29500 on the master node and check that no process is bound to it. Conversely, you can use the same method to determine that the c10d process did start up correctly on the master node. The command output below shows the working case. In a typical Slurm cluster, you won't know what the master node is going to be prior to submitting the Slurm job, but you can include a command like 'echo $MASTER_ADDR' in your Slurm batch script and then check the job logs immediately after submitting that job to find it out; then connect to the master node via ssh and run the ss command to see if the c10d process was successfully started on that node.
ubuntu@slurm-compute-node-1:~$ ss -tulnp|grep 29500
tcp LISTEN 0 4096 *:29500 *:* users:(("pt_elastic",pid=312118,fd=78))Cause 2: DNS or firewall configuration
When rdzv_endpoint specifies a hostname, all compute hosts should be able to resolve that hostname to an IP address either by using DNS or by using static configuration in their local /etc/hosts file. Check that the compute nodes are pingable by name from other compute nodes (for example - ping slurm-compute-node-0). Also, check that no firewall rules have been added that would prevent the compute nodes connecting to other compute nodes on the specified master port, within the private VPC network.
Cause 3: The socket timeout errors are a knock-on effect of another failure.
Unrelated bugs in the PyTorch training script that torchrun is executing ('my_training_program.py' in the example above) could cause it to crash on some or all of the nodes in the process group. If it crashes on the master node, the c10d process will also be halted. This causes un-crashed torchrun processes on other nodes in the same process group to record 'socket timeout' errors in the job logs. The solution in this case is, of course, to find and fix the cause of the original crash. To identify this situation, and as matter of general good practice, be sure to always read the error log from the top down to see the cause of the first crash or traceback, rather than simply tailing a few lines from the error log, seeing 'socket timeout' and immediately jumping into a network troubleshooting process.