How To Fix the Ansible Playbook Error When Mounting Shared Volumes in Crusoe Slurm Terraform Setup

Introduction

When deploying the Crusoe Slurm cluster solution using the published Terraform and Ansible playbooks, you may encounter an error related to mounting volumes on compute or login nodes. This error occurs because the Ansible loop directive expects a list but receives a string from JSON parsing inconsistencies. This article explains the root cause, how to fix the issue, and how to properly specify shared volumes.

Prerequisites

• Access to the Crusoe Slurm GitHub repository
• Basic familiarity with Terraform, Ansible, and JSON handling
• Valid Crusoe Cloud shared disk created via Web UI or CLI
• Understanding of the Slurm cluster architecture using Crusoe

Step-by-Step Instructions

Step 1: Recognise the Error

When running the Slurm Ansible playbook, you may see:

TASK [slurm_compute_node : Mount all volumes] **********************************
[ERROR]: The `loop` value must resolve to a 'list', not 'str'.

This indicates the variable volumes is treated as a string rather than a list of objects, causing failure in the volume mounting task.

Step 2: Apply the Required Code Fixes

Modify two files in the Slurm repository to explicitly parse the JSON lists:

• In slurm/ansible/roles/slurm_compute_node/tasks/main.yaml, change:

  - name: Mount all volumes
    ansible.posix.mount:
      src: "{{ item.name }}"
      path: "{{ item.mount_point }}"
      opts: rw,nofail
      state: mounted
      fstype: virtiofs
    loop: "{{ volumes }}" <----- this line need to be changed

  Change to

  loop: "{{ volumes | from_json }}"

• In slurm/ansible/slurm.yml, change:

  - hosts: all
    remote_user: ubuntu
    become: true
    tasks:
      - name: "Add users"
        ansible.builtin.include_role:
          name: user
        vars:
          user_name: "{{ item.name }}"
          user_uid: "{{ item.uid }}"
          user_ssh_pubkey: "{{ item.ssh_pubkey }}"
        loop: "{{ slurm_users }}" <----- this line need to be changed
        when: slurm_users is defined

  Change to

  loop: "{{ slurm_users | from_json }}"

These changes ensure the variables are treated as lists, resolving the Ansible loop error.

Step 3: Specify Shared Volumes in Your Terraform Variables

• Clone the Slurm repo:

  git clone https://github.com/crusoecloud/slurm.git

• Edit or create your .tfvars file (copy from slurm/examples/) and add shared volume info: 

  slurm_shared_volumes = [{
    id = "<your-shared-disk-id>"
    name = "data-01"
    mode = "read-write"
    mount_point = "/mnt/data-01"
  }]

You can obtain the shared disk ID from Crusoe Cloud Console under Storage > Shared Disk or via CLI.

Step 4: Run Terraform Commands

Execute Terraform commands in your Slurm directory:

terraform init
terraform plan -var-file="yourfile.tfvars"
terraform apply -var-file="yourfile.tfvars"

Step 5: Use Appropriate VM Instance Types for Shared Volumes

• Shared volumes currently can only be attached to the largest instance types, e.g., l40s.10x, not smaller sizes such as l40s.1x.
• Alternatively, use instance types like c1a series, which support shared volumes on all sizes.

Additional Notes

• The Slurm playbook fix is a workaround; a Slurm architecture rework is in progress, so this fix may not be committed to the repo.
• For shared disk limitations, see the Crusoe Shared Disk FAQ.

Additional Resources

1. Crusoe Github Slurm Solution