Reconstructing devata

Document contract

• Role: operations
• Scope: dependency order, recovery inputs, gates, and proof; exact machine values remain in private rendered configurations
• Truth boundary: current declarative design, not a claim that total-loss recovery has been completed
• Last verified: against the lab default branch, 2026-07-26

Prerequisites: devata-system-map, talos, argocd, sealedsecret, longhorn-volume

This page answers a harder question than “is the YAML in Git?” It asks whether someone with replacement hardware and the authorized recovery set could restore useful behavior without relying on memory.

The honest current answer is partially proven. Node replacement, Argo self-heal, Longhorn worker failure, and a bounded Velero restore have each been exercised. A complete empty-site rebuild has not. The sequence below is therefore a reconstruction contract and gap detector, not a completed disaster-recovery claim.

Recovery inputs

Before touching a failed system, inventory the inputs without printing secret material.

Stop condition

Do not install Talos until the target disk and rollback boundary are identified from the booted replacement machine. Do not claim secret or backup recovery until decryption and restore have been exercised.

Dependency order

flowchart TD
  A[Recover Git and private trust material] --> B[Inspect hardware and render Talos configs]
  B --> C[Build control plane and etcd]
  C --> D[Join workers and verify Cilium]
  D --> E[Bootstrap Argo CD and root Application]
  E --> F[Restore Sealed Secrets key]
  F --> G[Reconcile platform controllers]
  G --> H[Attach Longhorn and recover selected data]
  H --> I[Attach Velero repository and restore off-cluster data if needed]
  I --> J[Verify workloads and end-to-end transactions]

Each gate depends on the one above. Applying application manifests before networking, secrets, and storage exist produces noise, not useful progress.

Gate 1: establish operator identity and target

Work from a clean clone of the lab repository at a recorded commit. Read its README.md, docs/conventions.md, and talos/machineconfigs/README.md before rendering anything.

Record:

• physical machine to intended node-name mapping;
• MAC address, active NIC, and install disk from the booted machine;
• control-plane endpoint and LAN address plan;
• exact Git revision used for reconstruction;
• location and checksum of every private recovery artifact.

Gate proof: every physical target is unambiguous, machine configs validate, and no destructive command depends on a guessed device name.

Gate 2: restore the Talos and Kubernetes control plane

Apply the control-plane machine configuration using the version-matched Talos tooling and installation media declared by the repository inputs. A single control-plane design has an unavoidable API and etcd outage during its loss or disruptive maintenance.

Verify the OS before Kubernetes:

talosctl config info
talosctl -n <control-plane-ip> version
talosctl -n <control-plane-ip> health
talosctl -n <control-plane-ip> services

Then recover or bootstrap etcd according to the failure scenario. Do not mix “new empty cluster” and “restore existing etcd” procedures. They have different identity and data contracts.

Gate proof: Talos reports healthy services, the Kubernetes API responds, etcd has the expected membership, and the kubeconfig names the intended context.

Gate 3: join workers and recover the dataplane

Apply each worker configuration only after verifying its install disk and storage layout. The committed machine patches declare Longhorn mounts and required system extensions. Cilium must initialize before higher-level service tests are meaningful.

kubectl config current-context
kubectl get nodes -o wide
kubectl -n kube-system get pods -l k8s-app=cilium -o wide
kubectl -n kube-system get ciliumnodes.cilium.io

Gate proof: every intended node is Ready, every node has a Cilium agent, pod-to-pod traffic works across workers, and DNS resolves from a disposable pod.

Gate 4: transfer Kubernetes ownership to Git

The current bootstrap is pinned and Kustomize-based. From the repository root:

kubectl config current-context
kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -k kubernetes/bootstrap/argocd
kubectl wait --for=condition=Established crd/applications.argoproj.io --timeout=120s
kubectl apply -f kubernetes/bootstrap/root.yaml

This is the manual anchor. The root Application then creates the child Applications under kubernetes/clusters/devata/, and the Argo child adopts its own installation.

Gate proof: devata-root and every expected child appear. Investigate any OutOfSync, Degraded, or missing Application before applying its objects by hand.

Gate 5: restore the secret-decryption capability

Let Argo install Sealed Secrets, then restore the backed-up controller key using the version-matched upstream recovery procedure. The encrypted resources in Git are not useful until this key exists in the controller namespace.

Verify one known derived Secret and its consumer without printing secret data:

kubectl -n sealed-secrets get pods
kubectl get sealedsecrets -A
kubectl get secrets -A --field-selector type=Opaque

Gate proof: the controller is healthy, committed SealedSecrets report successful synchronization, and at least one consumer starts using its derived Secret.

Gate 6: recover storage before stateful workloads

Argo installs Longhorn, but installation health and data recovery are different facts. Confirm the Talos mount, node eligibility, Longhorn node conditions, StorageClasses, and replica placement.

If the original Longhorn disks survive, follow Longhorn recovery guidance before provisioning replacement claims. If they do not, create clean storage and restore selected data from Velero. Retained local-path claims are migration rollback artifacts, not replicated recovery.

Gate proof: a disposable Longhorn claim gets replicas on two workers, survives a controlled worker interruption, and returns the same sentinel checksum.

Gate 7: attach off-cluster backup and restore deliberately

Velero uses a private R2 bucket and Kopia for selected file data. Restore the SealedSecret-derived credentials first, confirm the backup location is available, then enumerate backups before choosing one.

Respect the bucket capacity guard and the fact that file-system backup is opt-in. Never infer that a PVC was backed up merely because Velero is installed.

Gate proof: restore a bounded backup into a remapped namespace, start its workload, and compare the content checksum. Only then use the same path for required application data.

Gate 8: prove the platform as a user would

Reconstruction is complete only when these transactions pass from the system boundaries where users and operators enter.

Remaining recovery gaps

• No complete empty-site rebuild has validated this entire sequence.
• The single control plane remains a control-plane failure domain.
• Backup coverage is intentionally selective and capacity-bounded.
• One Longhorn replica path is inside Talos EPHEMERAL storage and will not survive a node wipe.
• Recovery depends on private artifacts whose periodic restore checks must remain an operating habit.
• Version upgrades require their own compatibility and rollback rehearsal.

References

• Talos disaster recovery
• Argo CD disaster recovery
• Sealed Secrets backup and recovery
• Longhorn disaster recovery
• Velero restore reference