Recovering Loki from a WAL Replay OOM Loop

Prerequisites: ethernet-link-negotiation, write-ahead-log, kubernetes-container-memory-limit, longhorn-volume, loki-log-pipeline, service-monitor, gitops, reconciliation, application, promql

Loki stayed broken after the Ethernet problem was repaired because the physical fault and the active application failure were different stages of one incident.

Both devata workers lost or downshifted their Ethernet links. Kubernetes node state and Longhorn replicas churned. Longhorn emitted enough recovery logs to leave Loki with a 1.2 GB write-ahead-log containing 1,117 segment files. Loki then replayed that durable backlog inside a container limited to 384 MiB, began flushing unusually large streams, crossed the memory limit, and was killed. The StatefulSet restarted it with the same persistent volume, so the same recovery repeated.

This case study follows the evidence from cable to cgroup, recovers the backlog without deleting acknowledged logs, turns the live repair into Git-managed state, and proves the final pipeline with a new write and read.

How to work through the case

Keep one question visible for the entire session:

Why did Loki remain down after both Ethernet links were healthy again?

Do not read all of this in one sitting. Use four passes and stop after each checkpoint. If attention breaks, return to the working question, state the last completed output in one sentence, and resume at the next pass. There is no need to reread the completed passes.

The commands that inspect state are safe to repeat. Commands that pause reconciliation, restart workloads, or change resources describe the completed incident response. Do not rerun them on a healthy system merely to follow the chapter.

Pass 1: Build the incident model

Case file

The causal chain

flowchart TD
  A[Both worker Ethernet links degrade] --> B[Kubernetes node readiness changes]
  B --> C[Longhorn replica and controller churn]
  C --> D[Large Longhorn log storm]
  D --> E[Loki accepts logs into persistent WAL]
  E --> F[Loki restarts and replays 1.2 GB backlog]
  F --> G[Large streams flush concurrently]
  G --> H[384 MiB cgroup limit crossed]
  H --> I[OOMKilled, exit 137]
  I --> F

The cable problem was the original trigger. The retained WAL plus the resource mismatch sustained the outage. Fixing only the cable could not remove data already stored on disk. Raising memory without explaining the backlog would hide the mechanism. Deleting the WAL would remove the mechanism by discarding the data it existed to protect.

Checkpoint 1: hold four stages, not one root cause

Stop here and write these four lines without copying the diagram:

trigger:
amplifier:
retained state:
enforcement boundary:

For this incident, the answer is degraded Ethernet, Longhorn recovery log churn, the persistent Loki WAL backlog, and the 384 MiB container memory limit. Continue only when the four stages can be explained as one chain.

Pass 2: Diagnose from outside in

First question: what is actually unhealthy?

The incident began with a component-role check:

The Loki Pod had two containers. Its rules sidecar was healthy while the main loki container failed. A Pod-level 1/2 summary was therefore not enough. The container state provided the boundary:

kubectl -n logging get pod loki-0 -o wide
kubectl -n logging describe pod loki-0
kubectl -n logging logs loki-0 -c loki --previous

The decisive fields were:

reason: OOMKilled
exit code: 137
memory limit: 384Mi

This proved the immediate termination mechanism. It did not yet explain why memory rose.

Map node names to physical machines

Loki ran on talos-opt-7040, but that name had to be tied to a machine before touching cables:

Talos logs showed both workers changed link state at the same time. The OptiPlex returned at 10 Mbps. The Acer returned at 100 Mbps and reported a downshift from 1 Gbps. This ruled out the first simple story that only the node hosting Loki had a fault.

The physical repair was accepted only after both interfaces reported 1,000 Mbps full duplex and their counters stayed stable through an observation window. See ethernet-link-negotiation for the commands and counter interpretation.

Separate the stopped trigger from retained damage

After the links were healthy:

• all Kubernetes nodes were Ready;
• attached Longhorn volumes were healthy;
• current Longhorn manager logs were quiet;
• Loki still crashed.

That combination was not contradictory. It localized time.

Longhorn healthy now       -> the original storage churn stopped
Loki still OOM-looping     -> a retained consequence remains
same persistent volume    -> the consequence survives every restart

The retained consequence was visible on Loki’s Longhorn volume: about 1.2 GB across 1,117 WAL segment files.

Read the seconds before death

The previous Loki logs showed successful WAL recovery followed by flushes of Longhorn log streams with about 68, 75, 122, and 144 MB of uncompressed data. The process then crossed its 384 MiB limit.

The size of one uncompressed stream is not the process memory total. Several streams, chunk encoding, indexes, Go heap overhead, and concurrent storage work contribute. The logs nevertheless established the sequence:

WAL recovery finished
large recovered streams begin flushing
container exits 137

Loki’s default replay ceiling was 4 GB. Grafana documents that the ceiling controls when WAL replay applies backpressure by flushing recovered data, and suggests sizing it relative to available memory in the Loki WAL guide. A 4 GB application threshold inside a 384 MiB container could never activate in time. The kernel enforced the smaller boundary first.

Memberlist was real but secondary

During startup, the single Loki Pod initially could not resolve its own memberlist Service because the only endpoint was not yet ready. It logged an empty ring and retried joining.

That was a recovery delay, not the OOM root cause:

1. memberlist eventually became available;
2. WAL replay completed;
3. large flushes began;
4. the process was OOM-killed.

An error near a crash is not automatically causal. Ordering and recurrence matter.

Checkpoint 2: rank the evidence

Close the note and answer three questions:

1. What proved the immediate kill mechanism?
2. What proved the operation consuming memory?
3. Why was memberlist not the primary cause?

The answers are OOMKilled with exit 137, replay followed by large recovered-stream flushes immediately before death, and memberlist eventually recovering before the OOM sequence continued. This distinction is the difference between collecting errors and diagnosing a system.

Pass 3: Recover without losing the evidence

The recovery contract

The repair had five constraints:

The first live configuration used:

loki:
  ingester:
    wal:
      replay_memory_ceiling: 256MB
 
singleBinary:
  resources:
    limits:
      memory: 1Gi
    requests:
      cpu: 50m
      memory: 512Mi

The 256 MB ceiling was deliberately conservative. It made Loki flush recovered state in bounded batches. The 1 GiB container limit supplied room for those flushes and runtime overhead. The first successful backlog recovery peaked around 400 MiB, proving that the old 384 MiB limit had no meaningful safety margin.

Hold reconciliation during emergency recovery

Argo CD correctly considered the old Git values authoritative. Applying emergency live changes while self-heal remained active would let reconciliation restore the crashing configuration.

The Loki Application was therefore paused with a visible annotation before the temporary ConfigMap and StatefulSet changes were applied:

kubectl -n argocd annotate application loki \
  argocd.argoproj.io/skip-reconcile=true

This is an incident bridge, not a normal deployment method. While paused, Git no longer protects the application from drift. The pause had an explicit exit condition: merge the validated values, remove the annotation, and require Argo to roll and report the final desired state healthy.

Let Loki recover its own WAL

With bounded replay and sufficient headroom, Loki:

1. loaded the existing checkpoint;
2. replayed the retained segments;
3. triggered backpressure flushes;
4. became Ready;
5. created a native checkpoint;
6. removed the old segments covered by that checkpoint.

The initial successful replay completed in about 8.2 seconds. The temporary checkpoint.001582.tmp became a completed checkpoint, and the directory fell from 1,117 historical segment files to the current rolling set. No WAL file was manually removed.

Restarting during checkpoint creation would have interrupted useful native compaction. The healthy Pod was left running until the completed checkpoint appeared and the old segments disappeared.

Clear the shipper’s outage backoff

Loki was healthy before new logs began arriving normally. The three Promtail Pods were still in exponential retry backoff from the outage.

Their positions files were persisted on their nodes, so the DaemonSet was restarted to clear the stale wait without intentionally rereading the entire log history:

kubectl -n logging rollout restart daemonset/promtail
kubectl -n logging rollout status daemonset/promtail

All three replacements became Ready with zero restarts and no new client errors. This step repaired delivery timing, not Loki storage.

Add the missing application telemetry

Before the incident, Prometheus showed the container symptom through Kubernetes metrics but did not scrape Loki’s native metrics. The durable Helm values enabled a service-monitor.

The new target exposed the application side of recovery:

loki_ingester_wal_disk_full_failures_total = 0
loki_ingester_chunks_flush_failures_total = 0
loki_ingester_flush_queue_length = 0

The corruption counter was absent or zero and no corruption appeared in Loki logs. A Prometheus target marked up proved scraping. These values proved the recovery finished without the specific disk-full, corruption, or flush-failure modes being investigated.

Prove the entire data path

The final smoke test emitted five unique lines from a disposable Pod. Promtail read the node log, pushed it to Loki, and LogQL returned every line:

loki-case-study-proof-20260726-0
loki-case-study-proof-20260726-1
loki-case-study-proof-20260726-2
loki-case-study-proof-20260726-3
loki-case-study-proof-20260726-4

Query shape:

{namespace="logging", pod="loki-case-study-smoke"}
  |= "loki-case-study-proof-20260726"

This test was repeated after the merged Git state caused Argo to roll Loki. The final rollout recovered its compacted checkpoint and current segments in about 0.52 seconds, returned Ready with zero restarts, used about 181 MiB resident memory at the check, and returned all five new proof lines.

Turn the repair into desired state

Lab PR #51 changed only Loki’s Helm values:

• replay ceiling: 256MB;
• memory request: 512Mi;
• memory limit: 1Gi;
• ServiceMonitor: enabled.

The exact Loki chart version passed helm lint. Rendered manifests passed Kubernetes server-side dry-run. Repository CI passed Helm rendering and kubeconform. After merge:

kubectl -n argocd annotate application loki \
  argocd.argoproj.io/skip-reconcile-
 
kubectl -n argocd annotate application loki \
  argocd.argoproj.io/refresh=hard --overwrite

Argo briefly moved through OutOfSync and Progressing, applied the merged source, rolled the StatefulSet, and settled at Synced/Healthy. The emergency live state and Git now agree.

Checkpoint 3: explain the safety contract

State the recovery in this order without looking at the YAML:

1. stop the upstream churn;
2. preserve the WAL;
3. make application backpressure activate below the cgroup limit;
4. provide measured recovery headroom;
5. let native checkpointing compact the backlog;
6. clear the shipper backoff;
7. commit the same state and restore reconciliation.

If “increase memory” is the whole explanation, return to write-ahead-log and kubernetes-container-memory-limit. The portable lesson is the relationship between the replay ceiling, runtime overhead, container limit, and available node capacity.

Pass 4: Prove the service and transfer the method

Final proof matrix

Checkpoint 4: require two proof axes

Write one proof for resource health and one proof for service behavior:

resource health:
service behavior:

For devata, the first is the healthy node, Pod, volume, target, queue, and failure-metric matrix. The second is a unique log line completing the full Promtail-to-Loki-to-LogQL path. A production incident is not closed with only one axis.

Production transfer: reuse the method, not the numbers

Devata’s 256MB, 512Mi, and 1Gi values are measured answers for one small cluster. They are not production defaults.

Grafana’s current deployment-mode guidance describes monolithic mode as suitable for small volumes, roughly up to 20 GB per day. A production design that needs horizontal availability requires shared object storage and an appropriate replicated topology. Simple Scalable Deployment is being deprecated for Loki 4.0, so a new long-lived platform should evaluate highly available monolithic or microservices mode rather than treating SSD as the automatic middle step.

The topology decision and the incident method are separate. A larger Loki cluster changes which components fail independently, but the same diagnostic questions remain:

What triggered the event?
What amplified it?
What durable state survived it?
Which boundary terminated or throttled recovery?
What proves the user-facing path works again?

Production controls before the incident

The ServiceMonitor added during recovery is the start of monitoring, not the finished production package. Grafana’s key Loki metrics and meta-monitoring guidance recommend watching the service from a separate working observability path and using the Loki mixin as a baseline for dashboards, alerts, and recording rules.

Longhorn exposes volume state and robustness through metrics such as longhorn_volume_state and longhorn_volume_robustness; the exact label model is documented in the Longhorn metrics reference. Alert thresholds should come from the platform’s normal baseline and service objective. A queue of one for a few seconds during a rollout is not the same event as a queue that grows for twenty minutes while write latency rises.

Do not make Loki its only witness. If Loki stores its own component logs and the only Prometheus instance shares the same cluster and storage failure domain, a broad incident can remove both the service and the evidence needed to diagnose it. Production meta-monitoring should preserve at least critical Loki metrics, alerts, and component logs outside that shared failure domain.

A reusable incident worksheet

Fill this before proposing a fix. Unknown is an acceptable value. An untested assumption written as fact is not.

This worksheet prevents a common production failure mode: jumping from the loudest error directly to a configuration change without identifying the retained state or the data-loss boundary.

Change control during recovery

Before a production mutation, write six lines in the incident record:

evidence snapshot:
change and expected effect:
rollback command or manifest:
data-loss boundary:
observation window:
exit condition:

For the Loki recovery, the exit condition was not “Pod becomes Ready.” It was:

1. WAL replay completes without corruption or disk-full failures;
2. native checkpointing removes the historical segments;
3. Promtail resumes delivery;
4. a unique log transaction succeeds;
5. the same configuration merges;
6. reconciliation is restored;
7. the reconciler rolls Loki successfully from declared state.

An emergency pause must be visible, owned, and time-bounded. If the incident ends while self-heal remains disabled, the recovery has created a new production risk.

Read-only rehearsal on devata

This pass exercises the diagnostic order without recreating the outage or changing the cluster. Predict what each command should prove before reading its output.

# 1. Physical placement and Kubernetes state
kubectl get nodes -o wide
kubectl -n logging get pod loki-0 -o wide
talosctl -n 192.168.1.9 read /sys/class/net/enp3s0/speed
talosctl -n 192.168.1.9 read /sys/class/net/enp3s0/duplex
talosctl -n 192.168.1.10 read /sys/class/net/enp0s31f6/speed
talosctl -n 192.168.1.10 read /sys/class/net/enp0s31f6/duplex
 
# 2. Runtime boundary and restart state
kubectl -n logging get statefulset loki \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="loki")].resources}{"\n"}'
kubectl -n logging get pod loki-0 \
  -o jsonpath='{.status.containerStatuses[?(@.name=="loki")].restartCount}{"\n"}'
 
# 3. Retained storage and current robustness
kubectl -n logging get pvc storage-loki-0
LOKI_VOLUME=$(kubectl -n logging get pvc storage-loki-0 \
  -o jsonpath='{.spec.volumeName}')
kubectl -n longhorn-system get volumes.longhorn.io "$LOKI_VOLUME"
 
# 4. Application-level observability
kubectl -n logging get servicemonitor loki
kubectl -n logging get endpointslice \
  -l kubernetes.io/service-name=loki
 
# 5. Durable ownership
kubectl -n argocd get application loki \
  -o jsonpath='{.status.sync.status}{"/"}{.status.health.status}{"\n"}'

The expected story is one sentence: Loki runs on the OptiPlex with the declared recovery budget, its Longhorn volume is attached and healthy, its native metrics endpoint is discoverable, and Argo owns a Synced/Healthy application.

Production exit criteria

Close a similar incident only when all of these are true:

• the upstream trigger is stopped and observed through a meaningful window;
• retained backlog is draining or compacted through the application’s supported mechanism;
• no destructive recovery shortcut was taken without an explicit data-loss decision;
• restart, memory, queue, error, and storage signals are stable;
• a synthetic write and read succeeds;
• emergency drift has been converted into reviewed desired state;
• the reconciler is active and reports healthy;
• remaining risks have separate owners and are not hidden inside the recovery verdict.

What the incident teaches

Green hardware does not erase durable backlog

Recovery is often temporal. A healthy link describes now. A WAL describes accepted work from before. Both statements can be true at once.

Persistence can preserve a failure loop

The Longhorn volume did its job. It kept the WAL across crashes. The defect was the mismatch between the retained workload, Loki’s replay behavior, and its container budget.

Root cause has stages

The useful explanation names all three:

trigger: degraded Ethernet
amplifier: node and Longhorn recovery log storm
sustaining mechanism: retained WAL replay and flush under a 384 MiB limit

Calling only the last stage “the root cause” loses the physical incident. Calling only the first stage “the root cause” cannot explain why Loki stayed down after the cable was fixed.

Recovery needs two proof axes

Resource health proves that components are running. A synthetic transaction proves that the service still performs its job. The recovery was not complete until both axes passed.

A live fix is unfinished until reconciliation owns it

Pausing Argo made the emergency change possible. Merging the same state and restoring reconciliation made it durable. The final rollout was a deliberate test that the next restart would not recreate the outage.

Follow-up debt kept outside this repair

Promtail reached end of life on March 2, 2026. Replacing it with Grafana Alloy is a separate migration, not part of this incident repair.

The control-plane Talos machine configuration also retains an old Kubernetes endpoint at 192.168.1.2. The active kubeconfig and both worker configurations use 192.168.1.8, and the complete Talos health check passes when given that current endpoint. Correcting the stale control-plane value is separate configuration hygiene. It was not allowed to blur the Loki recovery verdict.