Prometheus: Time-Series Memory for the Cluster

Prerequisites: kubernetes

Prometheus collects numeric observations over time and stores them as time series. Grafana does not collect the measurements shown on Devata Overview. It asks Prometheus questions, then turns the returned samples into stats, lines, bars, and tables.

A Prometheus time series has three parts:

metric name + label set + timestamped samples

For example:

kube_node_status_condition{node="talos-opt-7040",condition="Ready",status="true"}

The metric name describes the measurement family. Labels identify dimensions within that family. Each scrape appends a value and timestamp to the matching series. Changing a label value creates a different series.

Where Devata Overview gets its signals

Signal family	Producer	What it describes
`up`	Prometheus itself	Whether the last scrape of each target succeeded
`kube_*`	kube-state-metrics	Kubernetes object state reported through the API
`node_*`	node-exporter	Host CPU, memory, filesystem, and operating-system measurements
`ALERTS`	Prometheus rule evaluation	Current alert states
`hubble_*`	Cilium and Hubble	Network flow processing and drops
`coredns_*`	CoreDNS metrics endpoint	DNS request and response behavior

This provenance prevents a common category error. A green kube_node_status_condition result means Kubernetes reports the node Ready. It does not mean every application is healthy. A green up means Prometheus scraped a target. It does not mean the target’s business function is correct. The dashboard combines several signal families because no single metric proves platform health.

The scrape loop

Prometheus discovers configured targets, makes HTTP requests to their metrics endpoints, parses the returned samples, and stores them locally. ServiceMonitors and PodMonitors in kube-prometheus-stack describe much of that discovery. Prometheus then exposes an HTTP query API that Grafana uses as a datasource.

flowchart LR
  E[Exporters and instrumented apps] -->|metrics HTTP endpoints| P[Prometheus]
  K[Kubernetes API state] --> KS[kube-state-metrics]
  KS -->|metrics endpoint| P
  P -->|PromQL result| G[Grafana panel]

The separation gives each system one job:

exporters translate system state into metrics;
Prometheus scrapes, stores, and queries time series;
Grafana visualizes selected query results;
Alertmanager would route notifications, but it is currently disabled in devata’s kube-prometheus-stack values.

Prometheus in devata retains seven days or up to 4 GiB and stores its TSDB on Longhorn. That is durable historical state, unlike the disposable dashboard file under /tmp.

Official references: Prometheus data model and Prometheus overview.