ServiceMonitor: Declaring a Prometheus Scrape Target

2 min read

Prerequisites: kubernetes-manifest, service, prometheus

A ServiceMonitor is a Prometheus Operator custom resource that describes which Kubernetes Services expose metrics and how Prometheus should scrape them.

It is not the metrics endpoint, a network proxy, or a Prometheus server. It is declarative discovery configuration.

flowchart LR
  P[application Pod /metrics] --> S[Kubernetes Service named port]
  SM[ServiceMonitor selectors and endpoint] --> S
  PO[Prometheus Operator] --> SM
  PO --> C[generated Prometheus config]
  C --> PR[Prometheus scrape]
  PR --> S

The Prometheus Operator design explains the two selector layers. A Prometheus resource selects ServiceMonitor objects, then each ServiceMonitor selects Services. The selected Service and its EndpointSlice lead Prometheus to the application Pods.

The labels must meet twice

A valid ServiceMonitor can still be ignored.

First, the Prometheus resource must select the ServiceMonitor by namespace and labels. Second, the ServiceMonitor must select the application Service by its labels.

Inspect all three objects on devata:

kubectl -n monitoring get prometheus \
  kps-kube-prometheus-stack-prometheus -o yaml
 
kubectl -n logging get servicemonitor loki -o yaml
kubectl -n logging get service loki -o yaml

The endpoint refers to the Service port name, not an arbitrary container port name:

spec:
  endpoints:
    - port: http-metrics
      path: /metrics
  selector:
    matchLabels:
      app.kubernetes.io/instance: loki
      app.kubernetes.io/name: loki

The exact rendered labels come from the Loki Helm chart. The important verification is that the selector resolves to the intended Service and that the named Service port resolves to the ready Loki endpoint.

Object existence is not scrape success

Use this proof ladder:

  1. the ServiceMonitor exists;
  2. Prometheus selects it;
  3. its selector matches the intended Service;
  4. the Service has a ready EndpointSlice;
  5. Prometheus reports the target up;
  6. expected application metrics are present and changing.
kubectl -n logging get servicemonitor loki
kubectl -n logging get service loki
kubectl -n logging get endpointslice \
  -l kubernetes.io/service-name=loki

Then inspect Prometheus targets or query:

up{namespace="logging",service="loki"}

The Prometheus Operator ServiceMonitor troubleshooting guide covers selector mismatches, missing named ports, rejected resources, and checking the generated configuration.

Application metrics change the incident boundary

Before Loki had a ServiceMonitor, Prometheus could show its container memory, restart count, and last termination reason through Kubernetes and node exporters. Those signals proved the OOM loop but not the state of Loki’s own WAL and flush machinery.

Direct scraping adds signals such as:

loki_ingester_wal_disk_full_failures_total
loki_ingester_wal_corruptions_total
loki_ingester_chunks_flush_failures_total
loki_ingester_flush_queue_length

Infrastructure metrics show what happened to the container. Application metrics show what the application was trying to do.