How-To Install Observability in a Crusoe Managed Kubernetes Cluster

This guide walks you through deploying a full observability stack on a Crusoe Managed Kubernetes (CMK) cluster using the kube-prometheus-stack Helm chart.

At its core, the stack runs Prometheus — a pull-based metrics engine that scrapes endpoints across your cluster on a configurable interval and stores them as time-series data you can query with PromQL.

Alongside it, Prometheus Operator manages the lifecycle of your Prometheus and Alertmanager instances as Kubernetes-native custom resources (ServiceMonitor, PrometheusRule, etc.), meaning your scrape configs and alerting rules live in version-controlled YAML rather than a config file someone SSHed in to edit once and forgot about.

Grafana sits on top as your visualization layer, and ships pre-loaded with dashboards covering node CPU/memory, pod resource consumption, kube-state-metrics, and more — useful from the moment it's running.

For GPU workloads, we'll also wire up NVIDIA's DCGM Exporter, which surfaces per-GPU metrics like utilization, memory bandwidth, temperature, and NVLink throughput directly into Prometheus — giving you the visibility you actually need when debugging training jobs or diagnosing thermal throttling on H100 or H200 nodes.

Prerequisites

• Kubernetes API Access to Your CMK Cluster
• A Valid kubeconfig for the Cluster
• helm Installed Locally
• NVIDIA GPU Operator Installed on the Cluster (GPU Metrics Only)

Step 1: Add the Helm Repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Step 2: Install kube-prometheus-stack

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n kube-prometheus-stack --create-namespace

Note: The default installation uses ephemeral storage.

Prometheus and Grafana data will not persist across pod restarts. For production use, configure persistent volumes before installing — see Additional Resources below.

Step 3: Expose the Grafana UI

Expose the Grafana deployment as a Kubernetes service.

Choose the service type appropriate for your setup:

• NodePort — accessible on a specific port of each node's IP
• LoadBalancer — provisions a Crusoe load balancer with an external IP (recommended for persistent access)

kubectl -n kube-prometheus-stack expose deploy/kube-prometheus-stack-grafana \
  --name grafana-np --type <NodePort|LoadBalancer>

Step 4: Import NVIDIA DCGM Metrics (GPU Clusters Only)

If your cluster has GPU node pools with the NVIDIA GPU Operator installed, label the DCGM ServiceMonitor so Prometheus scrapes it:

kubectl -n nvidia-gpu-operator label servicemonitor nvidia-dcgm-exporter \
  release=kube-prometheus-stack

Step 5: Import the DCGM Grafana Dashboard

Import the NVIDIA DCGM Exporter dashboard into your Grafana instance:

https://grafana.com/grafana/dashboards/12239-nvidia-dcgm-exporter-dashboard/

Example

You've just spun up a new CMK cluster running H100 nodes for a distributed training workload. Jobs are queued, GPUs are allocated — but you have no visibility into whether they're actually saturated, throttling, or sitting idle between iterations.

After following this guide, you open Grafana and pull up the DCGM dashboard. You immediately see that three of your eight H100s are sitting at 40% utilization while the others are pegged at 100% — a data pipeline bottleneck that would have taken hours to diagnose from logs alone. You also notice memory bandwidth on two nodes spiking in a pattern that correlates with your checkpoint saves, confirming that your storage I/O is the real constraint.

That's the loop this stack closes: GPU time is expensive, and you shouldn't be flying blind on it.

Additional Resources

• Custom Grafana Credentials: Set a custom username and password by creating a values override file — see the upstream values reference.
• Additional Dashboards: Browse community dashboards at https://grafana.com/grafana/dashboards/ — for example, the K8s Dashboard.
• Persistent Storage for Prometheus: Configure a persistent volume using the Prometheus values reference.
• Persistent Storage for Grafana: Configure a persistent volume using the Grafana values reference.