Enable and Visualize Hubble Metrics in Cilium on Crusoe Managed Kubernetes

Last Updated: March 30th, 2025

Introduction

This guide provides steps to enable Hubble for visualizing eBPF flow events in Crusoe Managed Kubernetes using Prometheus and Grafana.

Hubble builds on Cilium and eBPF to provide programmable visibility into service communication and network infrastructure with minimal overhead.

Prerequisites

1. Access to a Crusoe Cloud project with appropriate permissions

2. Existing Crusoe Managed Kubernetes Cluster 

3. Access to helm and kubectl CLI

Step-by-Step Instructions

Step 1: Monitoring Stack Setup

If you do not have a monitoring stack configured, install Prometheus, Grafana, and the Prometheus Operator CRDs.

• Add and update the Helm repository

  helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  helm repo update

• Install the kube-prometheus-stack

  helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --create-namespace \
    --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
    --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false

Note: If you have an existing Prometheus setup, verify it is configured to pick up ServiceMonitors cluster-wide. 

You can check the selector with: 

kubectl get prometheus -A -o jsonpath='{.items[*].spec.serviceMonitorSelector}' 

If the output is not empty, ensure your Cilium ServiceMonitors match those labels or patch the Prometheus object to allow empty selectors: 

kubectl patch prometheus -n monitoring kube-prometheus-stack-prometheus --type merge --patch '{"spec":{"serviceMonitorSelector":{}}}'

Step 2: Upgrade Cilium to Enable Hubble Metrics

Current State:

$ kubectl -n kube-system exec -it ds/cilium -c cilium-agent -- cilium status | grep Hubble
Hubble: Ok Current/Max Flows: 4095/4095 (100.00%), Flows/s: 76.77  Metrics: Disabled

Run the Helm upgrade command upgrades Cilium in-place to enable Hubble relay and flow visibility, expose Hubble/Cilium metrics via Prometheus ServiceMonitors, and deploy Envoy as a DaemonSet for L7 HTTP inspection.

helm upgrade cilium cilium/cilium \
  --version 1.16.1 \
  --namespace kube-system \
  --reuse-values \
  --set hubble.relay.enabled=true \
  --set hubble.metrics.enabled="{drop,tcp,flow,httpV2}" \
  --set hubble.metrics.enableOpenMetrics=true \
  --set envoy.enabled=true \
  --set prometheus.enabled=true \
  --set prometheus.serviceMonitor.enabled=true \
  --set operator.prometheus.enabled=true \
  --set operator.prometheus.serviceMonitor.enabled=true \
  --set hubble.metrics.serviceMonitor.enabled=true

Post upgrade, Cilium pods will restart via a rolling update with maxUnavailable: 2  based on updateStrategy. Modify the value based on cluster size and application throughput requirement to ensure traffic interruption is minimal.

$ kubectl get ds cilium -n kube-system -o jsonpath='{.spec.updateStrategy}'
{"rollingUpdate":{"maxSurge":0,"maxUnavailable":2},"type":"RollingUpdate"}

Step 3: Verify Hubble Activation

• Check the Cilium status

  $ kubectl -n kube-system exec -it ds/cilium -c cilium-agent -- cilium status | grep Hubble
  Hubble: Ok Current/Max Flows: 4095/4095 (100.00%), Flows/s: 96.01  Metrics: Ok

• Confirm ServiceMonitors are created

  $ kubectl -n kube-system get servicemonitors
  NAME              AGE
  cilium-agent      4h13m
  cilium-operator   4h13m
  hubble            4h13m

• Envoy DaemonSet

  NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
  cilium-envoy   2         2         2       2            2           kubernetes.io/os=linux   4h14m

Step 4: Verify Scrapping in Prometheus

Port-forward the Prometheus service to your local machine

kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090

Open http://localhost:9090/targets in your browser and confirm that the Cilium and Hubble entries show a status of UP.

Step 5: Import Grafana Dashboards

• Retrieve the Grafana admin password

  kubectl --namespace monitoring get secrets kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

• Port-forward to the Grafana U/I

  export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=kube-prometheus-stack" -oname)
  kubectl --namespace monitoring port-forward $POD_NAME 3000

• Import Dashboards

  Access Grafana at http://localhost:3000. Navigate to New > Import and upload the JSON files or provide the following URLs:

  • Cilium Agent Dashboard

  • Cilium Operator Dashboard

  • Hubble Dashboard

Ensure CiliumNetworkPolicy with L7 HTTP rules exists for the target workload to enable Envoy interception and populate L7 visibility metrics.

Additional Resources

Cilium Docs - Hubble Metrics