How-To Get Started With Text Generation on Crusoe Managed Inference (Python)

Last Updated: Jan 15, 2026

Introduction

Crusoe Managed Inference, delivered via the Intelligence Foundry, is the first vertically integrated inference service powered by MemoryAlloy™. This proprietary, cluster-native memory fabric enables persistent sessions and eliminates duplicate prefills, delivering up to 9.9x faster TTFT and 5x higher throughput than standard vLLM deployments. This guide demonstrates how to use the OpenAI Python SDK to handle single requests, inspect full JSON responses, and manage multi-turn conversation state.

Prerequisites

• Crusoe Account: Access to the Intelligence Foundry enabled.

• Inference API Key: Generated from the Security tab in the Crusoe Cloud Console.

• Environment: Python 3.8+ and the openai library (pip install openai).

• Model Selection: This guide uses google/gemma-3-12b-it (128k context), optimised for reasoning and multilingual tasks.

Step-by-Step Instructions

1. Configure Your Environment

For "Safety First" security, always use environment variables. 

Note: Crusoe API keys often contain $ characters. If you wrap your key in double quotes ("), shells like Bash or Zsh will attempt to interpret segments of the key as variables (e.g., $2a), which will mangle the key and result in a 401 Unauthorized error. Always use single quotes (').

export CRUSOE_API_KEY='your_actual_api_key_here'

2. Implement the Script

Create crusoe_inference.py. This script is a basic Python script demonstrating how to parse individual usage fields and maintain a conversation "buffer."

import os
from openai import OpenAI

# Initialize the client with the Crusoe Intelligence Foundry endpoint
client = OpenAI(
    base_url='https://api.inference.crusoecloud.com/v1/',
    api_key=os.getenv('CRUSOE_API_KEY', '<YOUR_API_KEY>'),
)

# Model selection: google/gemma-3-12b-it (128k context window)
MODEL_ID = 'google/gemma-3-12b-it'

# ============================================================================
# EXAMPLE 1: Single Request & JSON Inspection
# ============================================================================
print("--- EXAMPLE 1: Inspecting Full JSON Response ---")
response = client.chat.completions.create(
    model=MODEL_ID,
    messages=[{'role': 'user', 'content': 'Explain Digital Flare Mitigation in one sentence.'}],
    temperature=0.7,
)

# Observe all metadata fields provided by the Crusoe Gateway
print(response.to_json())

# ============================================================================
# EXAMPLE 2: Multi-Turn Conversation (Context Persistence)
# ============================================================================
print("\n--- EXAMPLE 2: Multi-Turn Conversation ---")
conversation_history = []

def chat_turn(user_input):
    global conversation_history
    
    # Build the message list including previous history
    messages = conversation_history + [{'role': 'user', 'content': user_input}]
    
    # MemoryAlloy™ recognizes 'messages' prefixes to speed up the prefill stage
    resp = client.chat.completions.create(
        model=MODEL_ID,
        messages=messages,
        temperature=1.0
    )
    
    reply = resp.choices[0].message.content
    
    # Update history to maintain state
    conversation_history.append({'role': 'user', 'content': user_input})
    conversation_history.append({'role': 'assistant', 'content': reply})
    
    print(f"\nUser: {user_input}")
    print(f"Assistant: {reply}")
    print(f"[Resource Efficiency] Tokens Used: {resp.usage.total_tokens}")

# Execute turns
chat_turn("What is Python?")
chat_turn("Can you give me a code example?")

python crusoe_inference.py

After executing the script, 

• The script outputs the generated text and token usage information. For JSON output, you'll see all available response fields. You can verify usage metrics in the Crusoe Cloud Portal Usage Dashboard.

  {
    "id": "chatcmpl-8c6544158bdd4d8a84b6c1d8c51fefbc",
    "choices": [
      {
        "finish_reason": "stop",
        "index": 0,
        "logprobs": null,
        "message": {
          "content": "Digital flare mitigation involves using data analytics and automation to proactively identify and address potential flare events in industrial processes, minimizing emissions and maximizing efficiency.\n",
          "role": "assistant"
        }
      }
    ],
    "created": 1767097790,
    "model": "google/gemma-3-12b-it",
    "object": "chat.completion",
    "service_tier": "paygo",
    "usage": {
      "completion_tokens": 29,
      "prompt_tokens": 17,
      "total_tokens": 46
    }
  }

• You can now extract prompt_tokens and completion_tokens individually to monitor your Resource Efficiency.

• By maintaining conversation_history, you enable the MemoryAlloy™ cache to "remember" the prefix, significantly reducing latency on subsequent turns.

• For a complete list and model details please refer, Managed Inference Overview

Common Error Messages & Troubleshooting

Additional Resources

• Managed Inference Overview

• MemoryAlloy™ Technical Deep Dive

• Crusoe Intelligence Foundry Dashboard