Document contract
- Role: architecture
- Scope: durable control, trust, data, and recovery boundaries
- Truth boundary: intended state comes from the default branch of the
labrepository; runtime observations below retain their verification dates- Last verified:
labrevisionbbc2473, recovery evidence from 2026-07-28, and public endpoint checks on 2026-07-30
Prerequisites: kubernetes, talos, gitops
devata is one system built from several independent reconcilers. Talos keeps each machine aligned with a machine configuration. Kubernetes controllers keep API objects aligned with their specs. Argo CD keeps the Kubernetes API aligned with Git. Longhorn keeps selected volume replicas aligned across two workers. Cloudflare adds a public edge outside those reconcilers, reached through outbound-only connectors inside the cluster. None of these mechanisms replaces the others, and each stops at a different boundary.
That boundary map is the shortest answer to what was built.
The platform in one diagram
flowchart TB operator[Operator workstation] lab[lab repository] recovery[Off-cluster recovery material] public[Public client] lanclient[LAN client] subgraph edge[Cloudflare account outside devata] cfconfig[DNS, tunnel routes,<br/>and Access policies] cfedge[Cloudflare edge] end subgraph lan[Home LAN behind CGNAT] cp[Control-plane node<br/>Talos + etcd + Kubernetes API] w1[Worker<br/>Talos + Longhorn disk] w2[Worker<br/>Talos + Longhorn disk] argo[Argo CD] platform[Cilium, MetalLB, Gateway API,<br/>cert-manager, Sealed Secrets] tunnel[Two cloudflared connectors] gateway[Cilium Gateway] observability[Prometheus, Grafana, Loki, Hubble] storage[Longhorn replicas] velero[Velero + Kopia] end r2[Private R2 backup bucket] snapshot[Public allowlisted snapshot] operator -->|talosctl applies reviewed inputs| cp operator -->|talosctl applies reviewed inputs| w1 operator -->|talosctl applies reviewed inputs| w2 operator -->|manages until imported into OpenTofu| cfconfig lab -->|outbound pull| argo argo --> platform argo --> tunnel argo --> observability argo --> storage argo --> velero platform --> gateway public --> cfedge cfconfig --> cfedge cfedge <-->|outbound tunnel sessions| tunnel tunnel -->|HTTPS with LAN host and SNI| gateway lanclient -->|LAN DNS and MetalLB address| gateway gateway --> observability recovery -. trust and decryption material .-> operator observability --> storage velero -->|selected resource and file backups| r2 observability -->|allowlisted health only| snapshot
At the 2026-07-26 verification point, the live cluster had one control-plane node, two workers, and 17 Argo CD Applications. The public path was proven after the 2026-07-28 policy recovery. On 2026-07-30, public Grafana reached its login redirect and public Hubble reached Cloudflare Access. These observations are dated evidence, not architectural promises. Query the cluster, its generated snapshot, and the endpoints for current state.
Five ownership planes
| Plane | Authoritative input | Applies it | What it owns | What it cannot recover alone |
|---|---|---|---|---|
| Machine | reviewed files under talos/ plus locally rendered machine configs | talosctl | OS image, node identity, networking, mounts, system extensions | cluster trust material and data stored outside the node |
| Kubernetes platform | kubernetes/ | Argo CD after one bootstrap apply | controllers, networking, storage, observability, workloads | Talos itself and private decryption material |
| External exposure | Git-managed cloudflared connectors plus remotely managed Cloudflare DNS, tunnel routes, and Access policies | Argo CD and the Cloudflare control plane | outbound tunnel sessions, public routing, and edge access | remote Cloudflare state is not yet reconstructable from lab |
| Data protection | Longhorn replicas and selected Velero/Kopia backups | Longhorn and Velero controllers | node-failure continuity and bounded off-cluster restore | data never selected for backup, lost credentials, or an untested restore assumption |
| Public evidence | an allowlisted snapshot schema and publisher | read-only CronJob plus a write-only deploy key | a bounded public view of health | private cluster state or recovery |
The lab repository conventions draw the most important line: kubernetes/ is reconciled; talos/ is versioned but applied out of band; lab-experiments/ is deliberately unreconciled.
The request path
flowchart TB public[Public client] --> edge[Cloudflare edge] edge <-->|outbound tunnel session| connector[cloudflared connector] connector -->|HTTPS using .lab SNI and host| gateway[Cilium Gateway] lan[LAN client] --> dns[LAN DNS] dns --> address[MetalLB address] address --> gateway gateway --> route[HTTPRoute] route --> service[Kubernetes Service] service --> endpoints[EndpointSlice] endpoints --> pod[Workload Pod] cilium[Cilium eBPF dataplane and policy] -. implements and filters .-> gateway cilium -. selects backend .-> pod
Cilium is both the CNI and the kube-proxy replacement. It allocates pod networking, implements Service translation without kube-proxy, and enforces the policies around this path. MetalLB gives the Gateway its stable LAN address, while cert-manager owns the certificate used by both LAN clients and the tunnel origin connection.
Cloudflare terminates the public connection and sends it over one of two outbound-only cloudflared sessions. The connector then opens HTTPS to the Gateway’s cluster-local Service using the existing .lab hostname for certificate verification and HTTP routing. No inbound home-network port is opened. Grafana relies on its own login; Hubble is stopped by Cloudflare Access before reaching the tunnel unless the client is authorized.
The public and LAN paths converge at the same Gateway but fail independently before that point. Disabling the Cloudflare routes or connectors removes public access without changing the .lab DNS records or MetalLB address. A healthy tunnel does not prove that Cilium policy, the Gateway, a backend Service, or application authentication succeeded. tracing-cloudflared-through-cilium-gateway-policy follows the recovered public path, and going-cilium-only traces the inner Service path.
The persistent-data path
flowchart LR pod[Stateful Pod] --> pvc[PVC] pvc --> sc[Longhorn StorageClass] sc --> volume[Longhorn volume] volume --> r1[Replica on worker A] volume --> r2[Replica on worker B] pod -. selected volume opt-in .-> agent[Velero node agent] agent --> kopia[Kopia repository] kopia --> cloud[Private R2 bucket]
Two Longhorn replicas protect selected workloads from one worker becoming unavailable. They do not protect against loss of the cluster, deletion propagated to every replica, or loss of both local disks. Velero provides a separate failure domain for explicitly selected resources and file data. Its R2 bucket is intentionally bounded, so backup input is checked and backups are created on demand rather than scheduled without a capacity guard.
The OptiPlex stores Longhorn data on a dedicated XFS Talos user volume. The second worker stores it inside Talos EPHEMERAL space, which survives ordinary reboot and upgrade but not a wipe. That asymmetry matters during disaster planning.
The secret and trust boundary
Git contains encrypted SealedSecret objects, never their plaintext. The controller private key is the bridge that makes those ciphertexts useful after a rebuild. Losing the cluster and that key together makes the encrypted files unrecoverable.
Other recovery material also stays outside public Git:
- rendered Talos machine configurations and the
talosconfigclient identity; - Kubernetes client credentials;
- the Sealed Secrets controller key backup;
- Velero object-store credentials and the Kopia repository password;
- any hardware-specific recovery archives retained by a case study.
The presence, readability, and restore procedure for each item must be verified. A note saying that a backup exists is weaker evidence than restoring from it.
How to inspect the current system safely
These commands are read-only. They prove different layers, so no single green result is enough.
kubectl config current-context
kubectl get nodes -o wide
kubectl -n argocd get applications.argoproj.io
kubectl get storageclass
kubectl get pvc -A
kubectl get gateway,httproute -A
kubectl get pods -A
curl -I https://grafana.pragalva.me
curl -I https://hubble.pragalva.me- Nodes answer whether Kubernetes can see healthy machines.
- Argo Applications compare runtime objects with Git.
- StorageClasses and claims show which persistence contract a workload requested.
- Gateway and route objects show the declared exposure path.
- Pods show scheduled process health, not end-to-end service behavior.
- The public requests test two edge behaviors: Grafana should redirect to its login, while Hubble should redirect to Cloudflare Access for an unauthorized client.
Finish with an application transaction: resolve a name, make the request, write and read a log, or write and reread persistent data. reconstructing-devata turns that principle into an ordered proof matrix.