Prerequisites: prometheus, promql
Grafana is the presentation and exploration layer. It connects to data sources such as Prometheus, runs queries, and maps results into visual panels. It does not scrape node metrics, store Prometheus time series, or make an unhealthy system healthy.
Why Devata needs an overview dashboard
kube-prometheus-stack ships many detailed dashboards. They are excellent once the failing subsystem is known, but a poor first page during an incident because the operator must first decide where to look.
Devata Overview is a triage surface. Its order encodes an operating method:
- read aggregate health first;
- find whether the problem is workload, host, network, DNS, or alert related;
- follow a linked specialist dashboard or open Explore for diagnosis.
Its 14 panels cover four layers:
| Layer | Panels | First question |
|---|---|---|
| Cluster state | Node readiness, pods needing attention, unavailable replicas, restarts | Is Kubernetes reporting a control-loop problem? |
| Monitoring state | Scrape targets, firing alerts, alert details | Can the observer see its targets, and are rules firing? |
| Resource state | Node CPU, node memory, pods by phase, restarts by namespace | Is pressure or churn localized to a node or namespace? |
| Network state | Hubble flow rate, Hubble drops, DNS non-success responses | Is traffic moving, being dropped, or failing at name resolution? |
The dashboard also links to the Kubernetes cluster, node-exporter, Cilium, and Prometheus dashboards. An overview should route investigation, not duplicate every detailed view.
Dashboard versus alert
A dashboard answers a question when a human looks. An alert evaluates a defined condition continuously and seeks a human when action may be needed. A dashboard is therefore not a monitoring strategy by itself. Devata Overview is valuable for routine checks, change verification, and incident triage, while Prometheus rules remain responsible for continuous detection.
UI work and code work
Experienced SREs use both interfaces:
- Grafana UI and Explore for learning metric shape, testing PromQL, adjusting panel layout, and investigating a live event;
- files, APIs, Terraform, or other automation for repeatable dashboards that need review, history, and recovery.
The split is not “professionals avoid the UI.” The UI is a workbench. Git is the durable contract. Devata’s dashboard has editable: false and its provider has allowUiUpdates: false, so experiments should happen in Explore or a scratch dashboard. Accepted changes go into the JSON manifest and through pull-request review.
What to look after the dashboard exists
On a normal check, scan in this order:
- nodes are 100 percent Ready;
- scrape targets are near 100 percent, with any miss explained;
- firing alerts exclude only the expected Watchdog behavior;
- pods needing attention and unavailable replicas are zero;
- restarts are not increasing unexpectedly;
- CPU and memory have headroom and no unexplained step change;
- Hubble drops and DNS failures are zero or understood.
After a deployment, reboot, storage change, or network change, keep the selected time range wide enough to include the event and watch for delayed restarts or resource changes. During an incident, the first abnormal panel is a lead, not a verdict. Drill into its source dashboard and confirm with logs, Kubernetes objects, and recent changes.
Official reference: About Grafana.