Run Production Batch Inference with openai/gpt-oss-120b model on Crusoe Intelligence Foundry

1. Create project directory

   mkdir crusoe-batch-inference
   cd crusoe-batch-inference

2. Create virtual environment

   python3 -m venv .venv
   source .venv/bin/activate

3. Install dependencies

   pip install --upgrade pip
   pip install openai

4. Store Managed Inference API key in a file .env:

   CRUSOE_API_KEY=your_api_key_here

5. Create input file for batch inputs.json with multiple entries. Example:

   [
     "This service processes user jobs synchronously and writes results to a single PostgreSQL database. It is expected to scale to 5,000 requests per second.",
     "This service manages image uploads, resizes images, and stores them on S3. Users can upload files up to 5GB."
   ]

6. Create batch inference script run_batch_inference.py:

   import os
   import json
   import time
   from openai import OpenAI, RateLimitError
   
   # Load API key
   api_key = os.environ.get("CRUSOE_API_KEY")
   if not api_key:
       raise RuntimeError("CRUSOE_API_KEY is not set")
   
   # Create Crusoe client
   client = OpenAI(
       base_url="https://api.inference.crusoecloud.com/v1/",
       api_key=api_key,
   )
   
   SYSTEM_PROMPT = (
       "You are a production analysis engine. "
       "You produce deterministic, structured output. "
       "Do not include explanations."
   )
   
   MAX_RETRIES = 5
   BACKOFF_SECONDS = 10
   DELAY_BETWEEN_REQUESTS = 2  # seconds
   
   with open("inputs.json", "r") as f:
       inputs = json.load(f)
   
   results = []
   
   for text in inputs:
       USER_PROMPT = f"""
   Analyze the following text.
   
   Rules:
   - Do not hallucinate information
   - If information is missing, return null
   - Output must be valid JSON
   
   Return this schema:
   {{
     "summary": string,
     "key_points": [string],
     "risks": [string]
   }}
   
   Input:
   ```{text}
   """
   
       for attempt in range(MAX_RETRIES):
           try:
               response = client.chat.completions.create(
                   model="openai/gpt-oss-120b",
                   messages=[
                       {"role": "system", "content": SYSTEM_PROMPT},
                       {"role": "user", "content": USER_PROMPT},
                   ],
                   temperature=0.0,
                   max_tokens=600,
               )
               break
   
           except RateLimitError:
               if attempt == MAX_RETRIES - 1:
                   raise
               print(
                   f"Rate limited. Retrying in {BACKOFF_SECONDS} seconds "
                   f"(attempt {attempt + 1}/{MAX_RETRIES})"
               )
               time.sleep(BACKOFF_SECONDS)
   
       content = response.choices[0].message.content
   
       try:
           output = json.loads(content)
       except json.JSONDecodeError:
           raise RuntimeError("Model output was not valid JSON")
   
       results.append(output)
   
       # Prevent sustained rate limiting
       time.sleep(DELAY_BETWEEN_REQUESTS)
   
   with open("outputs.json", "w") as f:
       json.dump(results, f, indent=2)
   
   print("Batch inference completed. Results written to outputs.json")

7. Export API key

   export CRUSOE_API_KEY=$(cat .env | cut -d= -f2)

8. Run batch inference

   python3 run_batch_inference.py

9. Example output (outputs.json)

   [
     {
       "summary": "A service processes user jobs synchronously, storing results in a single PostgreSQL database, and aims to handle 5,000 requests per second.",
       "key_points": [
         "Jobs are processed synchronously",
         "Results are written to a single PostgreSQL database",
         "Target throughput is 5,000 requests per second"
       ],
       "risks": [
         "Synchronous processing can become a performance bottleneck under high load",
         "A single PostgreSQL instance may not scale to 5,000 rps, leading to latency or failures",
         "Single point of failure: database outage would halt the entire service",
         "Potential connection pool exhaustion and contention in the database"
       ]
     },
     {
       "summary": "The service handles image uploads, resizes the images, and stores them on S3, supporting files up to 5GB.",
       "key_points": [
         "Manages image uploads",
         "Resizes uploaded images",
         "Stores images on S3",
         "Supports file uploads up to 5GB"
       ],
       "risks": [
         "Large (up to 5GB) uploads may lead to performance bottlenecks or timeouts",
         "High storage and bandwidth costs due to large file sizes",
         "Potential security and access control issues if S3 permissions are misconfigured"
       ]
     }
   ]

   
   Rate Limiting and Backoff 
   
   Large models such as openai/gpt-oss-120b enforce strict request limits. Production inference workflows must implement:

   • Fixed delays between requests
   • Retry logic with backoff on HTTP 429 errors
   • Controlled concurrency

Additional Resources

• Crusoe Managed AI models for text generation

• Crusoe Foundry openai-oss-gpt-120b model (UI)