Introduction
The Crusoe Watch Agent (CWA) is the vector.dev-based collector that publishes host, GPU, and fabric telemetry to the Crusoe metrics backend every 60 seconds. Command Center metrics and instance health, the Logs API, and the Telemetry Conduit scrape endpoint are all fed by it.
New collectors ship in point releases. DCGM, InfiniBand, NVLink, and NFS mountstats all arrived this way. An old agent breaks nothing, but it does mean the signal Support needs during an investigation may not be collected.
The upgrade path differs by deployment: a Helm-managed DaemonSet on CMK, a script-managed service on standalone VMs.
Prerequisites
- Helm 3 and kubectl Configured Against the Cluster (CMK Only)
- Root or Sudo Access on the Instance (VM Only)
- Crusoe Watch Agent Already Installed
Instructions
Follow Step 1 on a CMK cluster or Step 2 on a standalone VM, then verify with Step 3.
Step 1: Upgrade on a CMK Cluster
Refresh the chart index, check your current version, and upgrade:
helm repo update helm list --namespace crusoe-system --filter '^crusoe-watch-agent$' helm upgrade crusoe-watch-agent crusoe-watch-agent/crusoe-watch-agent --namespace crusoe-system
If the repository isn't configured yet, add it first:
helm repo add crusoe-watch-agent https://crusoecloud.github.io/crusoe-watch-agent/k8s/helm-charts
ℹ️ Note: Opening that URL in a browser returns a 404. This is expected and doesn't mean the repository is down. It's a Helm chart repository, not a web page. Helm reads
index.yamlbeneath that path, so append/index.yamlto the URL if you want to confirm it's reachable and see the published chart versions.
⚠️ Warning: If the upgrade reports success but the pods keep running the old image, previously set Helm values are overriding the chart's image tags. Re-run with
--reset-values. This discards your custom values, so capture them first withhelm get values crusoe-watch-agent --namespace crusoe-system --all.
Step 2: Upgrade on a Standalone VM
Download the latest script and run its upgrade command:
wget https://crusoecloud.github.io/crusoe-watch-agent/vm/latest/crusoe_watch_agent.sh chmod +x crusoe_watch_agent.sh sudo ./crusoe_watch_agent.sh upgrade
If you installed using the symlink method, run sudo crusoe-watch-agent upgrade instead.
ℹ️ Note: The agent reuses the mode it was installed with, Docker or native (
--no-docker). Don't switch modes during an upgrade. Your monitoring token persists. If authentication fails afterward, runsudo crusoe-watch-agent refresh-token.
Step 3: Verify the Upgrade
On CMK, confirm the rollout completes and the images actually changed:
kubectl -n crusoe-system rollout status daemonset/crusoe-watch-agent
kubectl -n crusoe-system get ds crusoe-watch-agent crusoe-metrics-exporter \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].image}{"\n"}{end}'On a VM, check the service logs. Use docker container logs crusoe-vector in Docker mode, or sudo systemctl status crusoe-watch-agent.service in native mode.
Then confirm data is landing in Command Center > Metrics in the Crusoe Console. Allow two 60-second intervals before concluding something is wrong.
ℹ️ Note: The Crusoe Metrics Exporter (CME) is a separate DaemonSet with its own image. A chart upgrade normally moves both, but verify it explicitly when Support asks you to upgrade for a specific fix.
Common Issues
| Symptom | Cause and fix |
|---|---|
| Upgrade succeeds, images unchanged | Stale Helm values. Re-run with --reset-values (see Step 1). |
| Pods evicted right after upgrade | The node is under DiskPressure. Reclaim disk space first, or telemetry won't stay up. |
Example
A team running distributed training opens a ticket about intermittent stalls on shared NFS storage. Support asks for mountstats telemetry to determine whether the stall is queue backlog or genuinely slow I/O.
Checking with helm list --namespace crusoe-system, the team finds they're several minor versions behind, far enough back that the NFS collector was never present. They run helm repo update followed by helm upgrade, watch the DaemonSet roll, and confirm in Command Center that the new metrics are flowing.
The next stall is captured automatically, with nobody needing to SSH into a node mid-incident to hand-collect /proc/self/mountstats while the problem is still happening.