Write-Ahead Log: Durable State Before Final Storage

Prerequisites: none. root note.

A write-ahead log, usually shortened to WAL, records a change before the application finishes placing that change in its final storage format. The order is the contract:

receive data -> append durable record -> acknowledge -> organize or flush later

If the process crashes after the acknowledgement but before the later work completes, it can replay the WAL and reconstruct the accepted state.

Why applications use one

Writing every small update directly into its final optimized representation is expensive. Applications therefore buffer, batch, compress, or reorganize data. That improves normal throughput but creates a vulnerable interval between receiving data and completing the final write.

The WAL closes that interval with an append-oriented record. Sequential appends are usually cheaper than rebuilding a final index or storage object for every input.

The tradeoff appears at restart. Recovery must read retained records and rebuild the in-memory state they represent. A large WAL can therefore turn a storage guarantee into a startup resource problem.

Segments, checkpoints, and replay

A WAL is normally split into numbered segment files rather than one file that grows forever.

00001581
00001582
00001583

A checkpoint summarizes an older prefix of the log. Once that checkpoint is complete and durable, the segments it covers can be removed. Restart then loads the checkpoint and replays only newer segments.

flowchart LR
  A[old WAL segments] --> B[checkpoint]
  B --> C[remove covered segments]
  C --> D[keep current segments]
  B --> E[restart recovery]
  D --> E

A directory named like checkpoint.001583.tmp is work in progress. It must not be treated as a completed checkpoint or used as permission to delete old segments manually.

Replay is not a free operation

Replay commonly rebuilds data structures in memory. The on-disk WAL size is not a direct prediction of peak resident memory because encoding, object overhead, indexes, concurrent streams, and flush work all change the ratio.

The dangerous pattern is:

large retained WAL
  -> restart
  -> rebuild large in-memory state
  -> cross memory limit
  -> process killed
  -> restart from the same retained WAL

Persistence makes the loop repeatable. The WAL correctly survives the crash, so every new process encounters the same recovery burden.

Backpressure bounds recovery

Loki provides a replay memory ceiling. Once replayed data passes that threshold, the ingester flushes recovered data to its durable chunk store and continues. Grafana documents this as a backpressure mechanism for recovering a WAL that is larger than available memory in the Loki WAL guide.

This ceiling is not the container memory limit. It is an application-level point at which recovery starts shedding reconstructed state. The container still needs room for the Go runtime, indexes, compression buffers, concurrent flushes, networking, and ordinary service work.

Preserve first, delete only with an explicit loss decision

Deleting a WAL can make a process start because there is nothing left to replay. It can also discard acknowledged data that has not reached final storage. That is a data-loss operation, not a repair shortcut.

Safe recovery begins with:

1. stop the upstream event that made the backlog grow;
2. verify the persistent volume and free disk space;
3. inspect replay, corruption, and disk-full evidence;
4. bound replay memory at the application layer;
5. provide enough temporary container headroom for the bounded recovery path;
6. allow native checkpointing to compact the retained segments;
7. prove new writes and reads end to end.

The WAL is healthy when it performs its intended recovery and returns to a small rolling set of current segments. A large directory alone is not corruption.