PromQL: Turning Metrics into Operational Questions

3 min read

Prerequisites: prometheus

PromQL is Prometheus’s query language. A dashboard panel is useful only when its query expresses a precise operational question. The visual layer comes after the semantics.

Start with the simplest Devata Overview query:

100 * sum(up) / count(up)

Read it inside out:

  1. up selects the latest sample for every scrape target;
  2. sum(up) counts successful targets because success is 1 and failure is 0;
  3. count(up) counts all selected targets;
  4. division produces the successful fraction;
  5. multiplication converts it to a percentage.

That query answers “what percentage of configured scrape targets answered the last scrape?” It does not answer “is the whole cluster healthy?”

Four query shapes used by the dashboard

Filter and aggregate current state

sum(kube_pod_status_phase{phase=~"Pending|Unknown|Failed"} == 1) or vector(0)

The label matcher keeps only concerning phases. == 1 keeps series whose current phase is active. sum collapses them to a count. or vector(0) makes the healthy empty result display as zero rather than “No data.”

Turn a counter into a rate

sum by (reason) (rate(hubble_drop_total[5m]))

A counter only goes up. Its raw value is usually not actionable. rate(...[5m]) estimates per-second change over the previous five minutes, and sum by (reason) groups the resulting flow by drop reason.

Count events in a window

round(sum(increase(kube_pod_container_status_restarts_total[1h]))) or vector(0)

increase(...[1h]) asks how much the restart counter rose during the last hour. This is an event count over a range, not the current lifetime counter.

Join useful identity onto a hardware series

100 * (1 - avg by (instance) (
  rate(node_cpu_seconds_total{mode="idle"}[5m])
)) * on(instance) group_left(nodename) node_uname_info

The first half converts idle CPU time into utilization. The vector match joins nodename from node_uname_info using the shared instance label so the legend can show a human-readable node identity.

Instant versus range queries

An instant query evaluates an expression at one time. The top-row stat panels use that shape for current health. A range query evaluates the same expression repeatedly over a time interval. CPU, memory, Hubble, and DNS panels use the repeated values to draw trends.

The dashboard JSON records the expression and whether the target is instant. Grafana supplies the selected time range and step when it asks Prometheus for a graph.

A query review checklist

Before putting a query on the landing dashboard, ask:

  • What exact question does this answer?
  • Which component produces the metric?
  • Do labels multiply the result unexpectedly?
  • Is a counter being converted with rate or increase?
  • Does an empty healthy result need or vector(0)?
  • Is the time window fast enough to notice trouble but stable enough to avoid noise?
  • Does the panel title state what the query really proves?

Use Grafana Explore to develop and inspect queries interactively. Once the semantics are settled, put the expression in the version-controlled dashboard JSON. That is the practical SRE split between exploratory UI work and reproducible configuration.

Official reference: Prometheus querying basics.