# Setting up monitoring with Grafana and Prometheus in Kubernetes: A practical guide Author: Malay Hazarika Published: 2024-09-13 Tags: metrics, prometheus, self-hosting Canonical: https://osuite.io/articles/collect-metrics-with-prometheus-and-grafana/ ## Introduction: The importance of monitoring for startups Zuck said, "Move fast and break things". But you gotta know when and how things broke for you to take action on it. You can not do that without reliable monitoring. Imagine this: you’ve just rolled out a much-anticipated feature, and initial user feedback is overwhelmingly positive. But then, suddenly, your application crashes under the load, leaving customers frustrated and searching for alternatives. Such downtime not only costs you money but also the brand value you have painstakingly created. In this blog post, we’ll explore how to set up monitoring with Grafana and Prometheus in Kubernetes, allowing your startup to maintain speed, break things in the process and know as soon as it happens. ## The setup Prometheus would be at the heart of the system. Grafana will sit on top of Prometheus for you to build dashboards. The whole thing will run on Kubernetes. This article also assumes you are running your apps on Kubernetes. In later installments, we will cover how to monitor applications that are running outside Kubernetes environments. ### Why Prometheus and Grafana Prometheus + Grafana is the gold standard for monitoring. Prometheus is open-source with a large community, providing all the resources and support you'd need. Grafana is so popular that you'll be able to find prebuild dashboards for standard components, saving you ton of engineering effort. ### How to set up Prometheus using the Prometheus operator in Kubernetes Ready to get started? Let’s set up Prometheus using the Prometheus Operator and Helm charts for efficient deployment. If you haven’t installed Helm yet, you can follow the [official Helm installation guide](https://helm.sh/docs/intro/install/). #### Install the Prometheus Operator We will be using the Prometheus operator to manage our Prometheus installation. Prometheus operator also comes with a few useful CRDs that will be helpful as we extend our setup in the future. ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install prom-operator prometheus-community/kube-prometheus-stack --namespace=observability --create-namespace --set grafana.enabled=false ``` Note: We are not installing the Grafana that comes bundled with the operator, we will install it separately. #### Create Prometheus instance Using the following manifest create a Prometheus instance ```yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: my-prometheus namespace: observability spec: retention: 7d shards: 1 replicas: 1 scrapeConfigSelector: matchLabels: prometheus: my-prometheus storage: volumeClaimTemplate: spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: ``` #### Create `ServiceMonitor` to monitor your services `ServiceMonitor` is a custom resource definition(CRD) that allows Prometheus to discover a service to scrape metrics from. You can have a service monitor for every service or you can have a default monitor that matches all the services exposed. Following a common way to match all the services running in your k8s cluster ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: default-service-monitor namespace: observability spec: endpoints: - interval: 10s path: /metrics port: metrics scheme: http jobLabel: app.kubernetes.io/name namespaceSelector: any: true sampleLimit: 1000 selector: selector: matchLabels: monitoring: prometheus ``` This `ServiceMonitor` will allow Prometheus to discover all services with the label `monitoring: prometheus` . Create this `ServiceMonitor` and update the services to include the label and these services will be scraped for metrics on the defined intervals. #### Monitor your infrastructure Monitoring your services is crucial, but don’t overlook the infrastructure itself. Essential metrics to track include CPU usage, memory consumption, and Disk I/O. With the `kube-prometheus-stack` we have installed a component called `node-exporter` which will make all the essential metrics available to Prometheus. ### Setup Grafana Install Grafana as follows ```bash helm repo add grafana https://grafana.github.io/helm-charts helm install grafana grafana/grafana -f values.yaml ``` The `values.yaml` file will be as follows ```yaml persistence: enabled: true storageClassName: size: admin: existingSecret: "observability-grafana-admin-credentials" datasources: datasources.yaml: apiVersion: 1 datasources: - name: Prometheus type: prometheus url: http://my-prometheus.observability.svc.cluster.local:9090 ``` Note: - You have to create a secret named `observability-grafana-admin-credentials` that will contain the keys `admin-user` and `admin-password` - The URL points to the Prometheus instance you just created above. #### Tips on building dashboards How to build dashboards in Grafana is beyond the scope of this article. In upcoming blog posts, we will dive deeper into that topic. For now, here is how to get started with Grafana 1. **Log into Grafana** (usually found at `http://:3000`). 2. **Add a Data Source:** Choose Prometheus as the data source. 3. **Create a New Dashboard:** Click on “+” and select “Dashboard.” When building your dashboard, align your metrics with business objectives. For instance, tracking user engagement metrics alongside response times can provide actionable insights. To measure the success of your dashboards, gather feedback from your team to see how well the dashboard aids in decision-making. Be sure to avoid clutter—clarity is key. Use consistent color coding and clear labels to enhance usability. ## Conclusion Implementing effective monitoring solutions like Grafana and Prometheus is essential for the success of your startup. By addressing performance issues proactively and optimizing your infrastructure, you’ll create a reliable experience for your customers. So, what’s next? Stay tuned for more such articles from us. Explore the documentation for Prometheus and Grafana for more advanced configurations, and consider joining community forums for additional support. Your journey towards enhanced operational efficiency starts now—let’s get monitoring!