Loki Log Pipeline: From Node Files to Searchable Streams

4 min read

Prerequisites: kubernetes, daemonset, service, write-ahead-log, prometheus, longhorn-volume

Devata’s logging path has three distinct responsibilities:

flowchart LR
  C[container stdout and stderr] --> F[node log files]
  F --> PT[Promtail DaemonSet]
  PT -->|push API| L[Loki single binary]
  L --> W[WAL on Longhorn]
  L --> CH[chunks and index on Longhorn]
  G[Grafana or curl] -->|LogQL| L
  PM[Prometheus] -->|scrape /metrics| L
  • container runtimes write pod logs to node-local files;
  • Promtail tails those files, adds labels, and pushes batches;
  • Loki accepts, indexes, stores, and queries log streams;
  • Prometheus stores numeric health metrics about the system;
  • Grafana presents LogQL queries and results.

Prometheus cannot replace Loki. It can record that a container restarted or crossed a memory boundary, but it does not retain the container’s log lines as searchable streams.

Promtail runs once per node

Promtail is a daemonset, so devata runs one Pod on each of its three nodes. Each instance reads node log files and persists a positions file that records how far it has read.

kubectl -n logging get daemonset promtail
kubectl -n logging get pods -l app.kubernetes.io/name=promtail -o wide

The positions file prevents every restart from rereading every log from the beginning. It is delivery state, not the log store itself.

If Loki remains unavailable, Promtail retries with backoff. Once Loki is recovered, an old Promtail process may still be waiting at a long backoff interval. Restarting the DaemonSet can clear that wait, but only after confirming positions are persisted and then proving delivery end to end.

Promtail is also now operational debt. Grafana marks Promtail end of life as of March 2, 2026 and directs users toward Grafana Alloy or another supported client. This case study documents the stack that handled the incident. It does not make Promtail a future-facing choice.

Loki groups entries into streams

A Loki stream is identified by its label set. Entries with the same labels belong to the same stream and carry timestamps plus log text.

{namespace="logging", pod="loki-0"}

Labels should describe bounded dimensions such as namespace, pod, container, or application. Putting unbounded values such as request IDs into labels creates excessive stream cardinality.

The ingester buffers incoming entries into compressed chunks. Loki stores chunk data and an index that maps label sets and time ranges to chunks. Grafana’s Loki storage documentation explains this split.

The WAL protects accepted entries

Before an in-memory chunk is finalized, the ingester writes accepted entries to its write-ahead-log. On restart, Loki replays the checkpoint and newer segments before it becomes ready.

Devata runs Loki as a one-replica StatefulSet with one retained Longhorn-backed claim. That fixed identity and volume allow the new process to see the same WAL as the old process.

The reliability dependency is therefore intentional:

same StatefulSet identity + same persistent volume -> replay acknowledged entries

It also means a retained pathological backlog follows every restart until Loki successfully replays, flushes, and checkpoints it.

Readiness and memberlist are secondary signals

The single-binary chart still uses memberlist for its internal ring. During startup, the only Loki Pod can temporarily fail to resolve or join the memberlist Service because that Service has no ready endpoint yet. Loki then reports an empty ring until startup advances.

This can delay readiness, but it is not automatically the root cause of a crash. In the incident, memberlist eventually joined; the process then died while replaying and flushing under its memory limit. Timing established the difference.

Check all three layers:

kubectl -n logging get pod loki-0
kubectl -n logging get endpointslice \
  -l kubernetes.io/service-name=loki-memberlist
kubectl -n logging logs loki-0 -c loki --previous

Prove the pipeline, not just each Pod

Green Pods do not prove that a new line can traverse the complete path. A final test should create a unique log line, wait for the node shipper, and query Loki for that exact token.

The incident recovery used five unique lines and required all five to return through LogQL. That single test covered:

  • container log creation;
  • node file discovery;
  • Promtail tailing and positions;
  • push connectivity;
  • Loki ingestion and indexing;
  • query readiness.

Application metrics then provided the second axis: the pipeline worked while WAL corruption, disk-full, and flush-failure counters remained zero.