Troubleshooting devata without stopping at green objects

3 min read

Document contract

  • Role: operations concept
  • Truth boundary: portable method generalized from documented devata incidents
  • Last verified: 2026-07-26

Prerequisites: kubernetes, devata-system-map

The most expensive troubleshooting mistake is stopping when one layer turns green. A node can be Ready while its physical link is degraded. A PVC can be Bound while the expected filesystem is not mounted. Loki can be Ready while new logs never arrive. Argo can be Synced while a user transaction still fails.

Use a causal chain that distinguishes six things:

flowchart LR
  symptom[Visible symptom] --> loop[Current failure loop]
  trigger[Original trigger] --> retained[Retained state]
  retained --> loop
  loop --> mitigation[Mitigation]
  mitigation --> recovery[Resource recovery]
  recovery --> proof[End-to-end proof]

1. Bound the symptom

Write what failed from the observer’s perspective, when it began, and which adjacent behavior still works. “Loki is broken” is weak. “The Loki Pod restarts during WAL replay, while its Service and volume remain present, and fresh Promtail lines cannot be queried” gives testable boundaries.

2. Identify the current failure loop

The condition keeping the system broken may not be the event that started it. Repeated OOM kills, a controller reconciling bad intent, a queue replaying retained work, or a mount missing after reboot can persist after the original trigger disappears.

Ask what state is reintroduced on every retry.

3. Search for retained state

Look for WALs, queues, caches, volumes, controller specs, finalizers, stale endpoints, or machine configuration that outlives the initiating event. Retained state explains why “the network is fixed” or “the node is Ready again” may not end the failure.

4. Rank evidence by layer

Collect the smallest set that can disprove competing hypotheses:

  1. physical and OS facts;
  2. Talos services and mounts;
  3. Kubernetes nodes, events, ownership, and resource limits;
  4. controller status and desired-state diff;
  5. application metrics and logs;
  6. a boundary-to-boundary transaction.

Prefer exact timestamps, previous-container logs, exit reasons, status fields, and metrics over screenshots or summaries.

5. Define recovery before changing state

A recovery contract names:

  • the smallest intended mutation;
  • rollback input and stop condition;
  • resource and controller health expected afterward;
  • the synthetic transaction that proves the path;
  • the declarative change required to make the result survive reconciliation.

A live patch may diagnose or temporarily unblock an incident. It becomes durable only when the authoritative source is updated and reconciliation confirms the same state.

6. Prove the actual user path

Choose a proof that enters where real input enters and exits where real consumers observe it:

  • networking: resolve the intended name and make the request;
  • logging: emit a unique line through the configured collector and retrieve it with LogQL;
  • metrics: generate a fresh observation and query Prometheus;
  • storage: write a sentinel, exercise the claimed failure, and read the same bytes;
  • GitOps: introduce bounded drift and watch the controller correct it;
  • backup: delete the disposable source and restore into a new namespace.

Incident worksheet

QuestionEvidence
What does the user observe?
What is failing repeatedly now?
What event likely started it?
Which state survived that event?
What evidence would disprove the leading hypothesis?
What is the smallest safe mitigation?
What is the rollback and stop condition?
Which declared source must change?
What end-to-end transaction closes the incident?

Check yourself

  1. Why can an original trigger be gone while its incident continues?
  2. Give one example each of a green resource, controller, and infrastructure layer that still fails a transaction.
  3. What turns a mitigation into a durable recovery?

Worked examples