Longhorn Volume: Replicated Block Storage for a Kubernetes Workload

Prerequisites: kubernetes, pod-volume

A Longhorn volume presents block storage to one Kubernetes workload while storing replicas of that volume on eligible cluster nodes. Kubernetes sees a PersistentVolume and a claim. Longhorn manages the engines, replicas, attachment, rebuilds, and data path behind those objects.

The Longhorn concepts documentation describes the core arrangement: an engine runs on the node where the volume is attached, and replica processes hold copies on the nodes selected for replica placement.

flowchart LR
  P[workload Pod] --> PVC[PersistentVolumeClaim]
  PVC --> E[Longhorn engine on attached node]
  E --> R1[replica on worker A]
  E --> R2[replica on worker B]

Devata uses two replicas. The design can survive one replica or worker becoming unavailable, but only if the surviving path stays healthy and the system has enough network and disk capacity to recover.

The network is part of the storage path

Replicated storage turns writes into network traffic. The workload writes to the attached volume, and the engine coordinates those writes with replicas. A worker Ethernet fault is therefore also a storage event.

When a link drops or downshifts:

• Kubernetes may mark the node NotReady;
• a replica may become unavailable;
• Longhorn managers and controllers emit state changes;
• the surviving system may rebuild or resynchronize data;
• recovery traffic competes with application and logging traffic.

Healthy volume status after the link returns proves that Longhorn recovered. It does not prove that secondary effects, such as a log backlog already accepted by Loki, disappeared with it.

Read both Kubernetes and Longhorn state

Start with the claim and volume attachment:

kubectl -n logging get pvc storage-loki-0
kubectl -n longhorn-system get volumes.longhorn.io
kubectl -n longhorn-system get replicas.longhorn.io
kubectl -n longhorn-system get engines.longhorn.io

The fields answer different questions:

None of these alone proves application-level correctness. A healthy volume can faithfully preserve a WAL backlog that repeatedly crashes its owner.

Storage recovery can amplify logs

Controllers are supposed to report topology and health changes. During repeated link loss, those useful messages can become a storm:

link event
  -> node readiness change
  -> volume and replica state change
  -> controller reconciliation
  -> more logs

If the logging system itself uses Longhorn, it is both observing the disturbance and depending on the disturbed subsystem. This is not a reason to remove persistent storage. It is a reason to bound the logging system’s recovery path and monitor it directly.

Verify quiet recovery, not only green objects

After the physical link is fixed:

kubectl -n longhorn-system get volumes.longhorn.io
kubectl -n longhorn-system logs -l app=longhorn-manager --since=10m \
  --prefix --max-log-requests=2

Look for attached healthy volumes, stable replicas, no repeating rebuild loop, and no continuing node readiness churn. Then inspect downstream retained state separately. The storage system can be healthy now while an application still carries the consequences of the earlier incident.