How-To Resolve Duplicate Entries When Paginating Crusoe Cloud VM Instances API

Problem:

When paginating VM instances using the Crusoe Cloud API's listInstances endpoint, users may encounter the following issues:

• Duplicate VM entries appear across pages.
• Not all instances are returned across paginated requests.
• A 401 bad_credentials error occurs when using sort with next_token due to query string misordering.  

These issues stem from either missing or misordered query parameters when making signed requests to the API.

Environment:

1. Product: Crusoe Cloud
2. Component: REST API
3. Endpoint: GET /v1alpha5/projects/{project_id}/compute/vms/instances
4. Docs: Crusoe API Reference – listInstances
5. Affected Parameters: limit , sort , next_token , prev_token

Cause:

• The API does not enforce a deterministic default sort order. Without the sort parameter, results may be returned in arbitrary order, causing inconsistent pagination and duplicate results.
• When using signed requests, Crusoe Cloud's API requires query parameters to be lexicographically sorted in both the URL and the signature payload. A mismatch in this order leads to a 401 bad_credentials error.

Resolution:

• Include the sort=created_at query parameter in your request to ensure stable, consistent pagination. This resolves duplicate and missing entry issues.

• Sort all query parameters lexicographically by key name before both:

  • Appending to the request URL

  • Including in the HMAC signature payload

Failure to do so will result in a 401 Unauthorized  error, even if the request appears syntactically valid.

Procedure:

Example Signed Python Request With Correct Pagination:

import hmac
import hashlib
import base64
import datetime
import requests
import json

api_access_key = "<your-access-key>"
api_secret_key = "<your-secret-key>"

project_id = "<your_project_id>"

limit = 10
next_token = ""  # Leave blank

def generate_signature(path, query, verb, timestamp):
    payload = f"{path}\n{query}\n{verb}\n{timestamp}\n"
    decoded = base64.urlsafe_b64decode(api_secret_key + '=' * (-len(api_secret_key) % 4))
    signature = hmac.new(decoded, msg=bytes(payload, 'ascii'), digestmod=hashlib.sha256).digest()
    return base64.urlsafe_b64encode(signature).decode('ascii').rstrip('=')

def list_instances(next_token=None):
    base_url = "https://api.crusoecloud.com"
    api_version = "/v1alpha5"
    http_method = "GET"
    endpoint = f"/projects/{project_id}/compute/vms/instances"

    # Canonical order: limit, next_token, sort
    query_parts = [f"limit={limit}", "sort=created_at"]
    if next_token:
        query_parts.append(f"next_token={next_token}")

    # Sort lexicographically
    query_string = '&'.join(sorted(query_parts))

    dt = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0).isoformat()

    signature = generate_signature(api_version + endpoint, query_string, http_method, dt)

  
    response = requests.get(
        f"{base_url}{api_version}{endpoint}?{query_string}",
        headers={
            "X-Crusoe-Timestamp": dt,
            "Authorization": f"Bearer 1.0:{api_access_key}:{signature}"
        }
    )

    print(f"\n Request URL: {base_url}{api_version}{endpoint}?{query_string}")
    print(f"Timestamp: {dt}")
    print("Signature generated.")

    if response.status_code != 200:
        print("Request failed:", response.status_code, response.text)
        return None

    result = response.json()
    print("Response:")
    print(json.dumps(result, indent=4))
    return result.get("next_page_token")

# Run pagination loop
print("Fetching VM instances with pagination...\n")
while True:
    next_token = list_instances(next_token)
    if not next_token:
        break

Note:

• Ensure query parameters are lexicographically sorted in both the signature payload and the final URL.
• Do not assume default sort behaviour, always use sort=created_at or another supported field for consistent pagination.

References:

• https://docs.crusoecloud.com/reference/api/ 
• https://docs.crusoecloud.com/api/#section/Authentication 
• https://docs.crusoecloud.com/api/#tag/VMs/operation/listInstances