Introduction

You create a new Kubernetes cluster or have an existing Kubernetes cluster and would like to set up observability on the cluster. The popular open source solution commonly used is Prometheus and Grafana.  

Prerequisites

• Kubernetes API access
• Kubeconfig
• Helm

Step-by-Step Instructions

1. Step 1: Install the kube-prometheus-stack helm chart

   helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack -n kube-prometheus-stack --create-namespace

2. Step 2: Expose the Grafana UI through a kubernetes service

   kubectl -n kube-prometheus-stack expose deploy/kube-prometheus-stack-grafana --name grafana-np --type <service-type>

Additional Resources

1. Configure the username and password for the Grafana chart by creating a custom values file:
   https://github.com/prometheus-community/helm-charts/blob/ae248069a3c9aac30262eb5d6f93a12db52fb065/charts/kube-prometheus-stack/values.yaml#L1246
2. Grafana installs the default Kubernetes dashboards, but you can import custom dashboards provided here https://grafana.com/grafana/dashboards/
   Example: https://grafana.com/grafana/dashboards/15661-k8s-dashboard-en-20250125/
3. To keep the metrics and dashboards persistent, create a persistent volume for both Prometheus and Grafana:
   https://github.com/prometheus-community/helm-charts/blob/ae248069a3c9aac30262eb5d6f93a12db52fb065/charts/kube-prometheus-stack/values.yaml#L4251
   https://github.com/prometheus-community/helm-charts/blob/ae248069a3c9aac30262eb5d6f93a12db52fb065/charts/kube-prometheus-stack/values.yaml#L1292