journey

53 min read

🧭 Journey

The living state of the work — read this first, update it when something changes. The homelab mentor skill reads this file; the skill holds the method, this file holds the facts.

This file lives at homelab-vault/content/devata/planning/journey.md (private repo; renders at vault.pragalva.me/devata/planning/journey behind Cloudflare Access). The former /planning/journey URL redirects here. It moved here from ~/Desktop/homelab-dashboard/JOURNEY.md on 2026-07-02 (vault issue #12); the old path holds a pointer stub, do not edit it. Snapshots: planning/snapshots/. Operational runbooks: planning/runbooks/. Pre-vault lesson/lab drafts were moved outside the published content tree to archive/legacy-docs/ on 2026-07-26.

Last updated: 2026-08-17.

Where each thread stands

Homelab — devata

  • Snapshot publisher quantity parsing recovered (2026-08-17, lab PR #72). After a control-plane reboot, Linux reported 12 KiB more usable memory. Kubernetes therefore changed the canonical capacity string from a Ki value to the exactly equivalent 3336Mi. The publisher’s jq helper stripped only Ki, failed while parsing 3336Mi, and stopped before schema validation or push. PR #72 made the conversion accept bytes plus Ki, Mi, Gi, and Ti while preserving the public GiB values. Argo reconciled merge 1d4d01b and returned Synced/Healthy. A temporary Job rendered the live mixed-unit node set, passed the schema gate, pushed snapshot commit 17373ae, and restored both public files at 2026-08-17T10:52:50Z; the verification Job was then removed.
  • Talos-native NetBird recovery access complete on all three nodes (2026-08-13; Cilium boundary corrected 2026-08-16, lab PRs #67, #68, and #71). NetBird 0.66.2 runs below Kubernetes as a Talos system extension, with one-off node enrollment keys and the workstation as the administrator peer. The OptiPlex canary exposed a multihoming failure: kubelet selected the new wt0 address as the Kubernetes InternalIP, which broke control-plane, Cilium, and CSI paths even though NetBird itself worked. The first durable fix pins machine.kubelet.nodeIP.validSubnets to 192.168.1.0/24. A later two-replica Cloudflare Tunnel outage proved a second automatic-consumer failure: Cilium selected NetBird’s MTU-1280 wt0 as a datapath device and lowered cilium_host from MTU 1500 to 1280. Existing QUIC sessions survived until reconnect; new handshakes then timed out, and cloudflared’s liveness probe restarted each connector before HTTP/2 fallback completed. Lab PR #71 set Cilium devices: "enp+", preserving NetBird while excluding host overlays from Cilium device and MTU selection. After merge 37baab3, Argo returned Synced/Healthy, all three Cilium agents selected only their physical NIC at MTU 1500, both cloudflared replicas became Ready, and each registered four QUIC connections. The OptiPlex, single control plane, and Nitro worker run Talos v1.12.11 while Kubernetes remains v1.34.1; each retains its physical LAN InternalIP, and the Talos API passes through each NetBird address. Before the control-plane upgrade, a verified 51.8 MB etcd snapshot and complete machine configuration were stored off-node. Before the Nitro upgrade, every attached Longhorn volume was healthy with two replicas and its complete machine configuration was stored off-node. Nitro returned Ready and schedulable, Longhorn rebuilt to zero unhealthy attached volumes and zero failed replicas, every pod became Ready, and every Argo application returned Synced/Healthy. Its NVIDIA extensions advanced from 535.247.01 to 580.126.20, all four modules loaded, the device plugin stayed Ready, and the boot log contained no RmInitAdapter, VBIOS, or Xid failure. A real nvidia-smi call from the device-plugin container identified the GTX 1650 Ti, driver 580.126.20, CUDA 13.0, and 4096 MiB VRAM. That overturns the earlier hardware-fault conclusion for adapter initialization and NVML; a real CUDA workload remains untested. Final off-LAN proof used an alternate network where every direct LAN Talos and Kubernetes port was unreachable: all three Talos APIs passed through P2P wt0 routes, the Kubernetes API returned readyz=ok through the control-plane peer, all nodes remained Ready, and there were zero unhealthy pods or Argo applications. The remote recovery-access rollout has no remaining gate. The complete architecture, rollout, failure analysis, kubeconfig explanation, operating drill, and reconstruction path are in building-talos-native-netbird-recovery-access.
  • Aggregate Hubble traffic stream complete (2026-08-01; lab PRs #61 and #63, platform-hub PR #64). The TypeScript Cloudflare Worker remains beside the Go producer and Kubernetes resources in lab while their payload, authentication, deployment, and rollout order form one feature contract; hubble-traffic-relay-repository-boundary records that exception and its split triggers. The first main-branch Worker deployment uploaded the script and Durable Object but could not attach telemetry.pragalva.me until the account token gained zone-scoped Workers Routes write access. The first producer sync then reached CoreDNS but its L7 DNS allowlist omitted the Prometheus Service; PR #63 changed both the URL and policy to the exact cluster FQDN. Final platform proof showed the main workflow green, Argo Synced/Healthy at 5284064, one Ready producer with zero restarts, fresh five-second samples in the public six-hour history, rejected unauthenticated writes, and a WebSocket 101 followed by the latest and next live samples. PR #64 added the public consumer. A production browser rendered live within two seconds, updated the chart within six, moved through reconnecting to stale snapshot fallback when its relay tunnel was cut, and returned to live after restoration. streaming-aggregate-hubble-traffic records the architecture, code, failure causes, recovery, and next focus areas.
  • Grafana 13 gzip OOM recovery verified (2026-07-31, lab PR #62 merged). The provisioned home dashboard began leaving panels blank while Grafana exhausted its explicit 256 MiB limit, became unready, timed out internal datasource and SQLite operations, and was OOM-killed. Prometheus answered the same panel queries in 8-16 ms, while the previous Grafana pod had already recorded six OOM restarts in 24 hours, isolating the failure to Grafana rather than the metrics backend or browser. PR #62 changed only server.enable_gzip to false, preserving the 128 MiB request and 256 MiB limit so the Grafana 13 compression regression could be tested independently. Argo reconciled merge 9f52838 and returned Synced/Healthy; the new pod was Ready with zero restarts, idle memory fell from 216 MiB to 153 MiB, one browser session completed 14 datasource query batches with HTTP 200, and two 30-second-refresh sessions completed 60 query batches in five minutes while memory stayed at 172-184 MiB, one-minute CPU stayed below 32m, and no HTTP errors or readiness failures appeared. The immediate load-triggered OOM is recovered; the unchanged limit remains a 24-hour canary for slower Grafana 13 growth before any resource increase is considered.
  • Cloudflare Tunnel to Cilium Gateway policy incident recovered (2026-07-28, lab PRs #59 and #60 merged). Public Grafana returned Envoy 403 Access denied while LAN Grafana, the Gateway, tunnel connectors, and Grafana remained healthy. Removing Grafana from Cloudflare Access did not repair the path. The first policy repair in PR #59 removed a listener-port constraint from the selectorless generated Gateway Service, passed validation, reconciled, and still failed. A disposable same-label probe reproduced the 403 without Cloudflare; Cilium’s L7 verdict exposed a hairpin transition through the special ingress identity before the routed backend identity. PR #60 replaced the Service-based rule with explicit ingress identity access plus Grafana TCP 3000 and Hubble UI TCP 8081 endpoint permissions. Final proof showed Argo Synced/Healthy at bbc2473, public Grafana 302 then the Grafana login page, LAN Grafana 302, Hubble redirected to Cloudflare Access, two healthy connector replicas, and no diagnostic resources. The complete case study is tracing-cloudflared-through-cilium-gateway-policy.
  • Roadmap v3 adopted (2026-07-26). A read-only cluster, repository, issue, vault, and external-practice audit replaced the stale 2026-07-02 roadmap with roadmap and its evidence layer, next-goals-research. The new portfolio does not relabel the existing LLM, tunnel, showcase, security-study, chaos, operator, or reconstruction backlog. It starts with live integrity gaps: unexplained non-Argo resources, disabled Alertmanager, a stale control-plane Talos endpoint, client-version skew, scheduler and public GPU disagreement, hardcoded storage posture, and insufficient capacity evidence. Six goals now follow: restore the operating contract, build actionable alerting, declare Cloudflare state, operate PostgreSQL with Miniflux, prove a disposable delivery path, and add off-cluster UPS control when hardware exists. A bounded trace experiment follows only after a real multi-hop workload and memory reserve exist.
  • Cilium Gateway API and LAN TLS complete (2026-07-26, lab issue #10; PRs #52, #53, #54, and #55 merged). Cilium Gateway API is enabled on the existing pinned v1.18.6 release with Gateway API CRDs v1.2.0; no Cilium upgrade was combined with the change. cert-manager v1.21.0 uses Let’s Encrypt production and Cloudflare DNS-01 without an ACME email. The Cloudflare token is stored as a strict-scope SealedSecret, and DNS-only A records send grafana.lab.pragalva.me and hubble.lab.pragalva.me to the dedicated MetalLB address 192.168.1.244. The shared ECDSA certificate covers both names and is valid from 2026-07-26 through 2026-10-24. HTTP returns 301 to HTTPS; verified HTTPS returns Grafana 302 and Hubble 200 with a valid chain. Existing direct LoadBalancer access remains available at Grafana .242 and Hubble .243, which is also the documented rollback path. Prometheus scrapes certificate readiness, renewal, and expiration metrics; alerts cover not-ready certificates, overdue renewal, and expiration within 14 days. Final proof showed all 17 Argo applications Synced/Healthy, all three nodes Ready, no unhealthy pods, the Kubernetes API ready, Cilium and Envoy healthy, and all Gateway listeners and routes Accepted, Programmed, and ResolvedRefs. Live rollout findings became durable configuration: Cilium pod checksums force the required restart after Gateway API config changes, the generated Service uses the annotation prefix required by MetalLB v0.14.5, Gateway API defaults are explicit to prevent false Argo drift, and the cert-manager ServiceMonitor preserves certificate namespace labels.
  • Loki WAL recovery complete (2026-07-26, lab PR #51 merged). Both worker Ethernet links are back at 1 Gbps full duplex with no carrier changes or interface errors during the observed hour, so the hardware trigger is no longer active. The earlier network and Longhorn log storm left Loki with a 1.2 GB WAL and 1,117 segment files. Loki then used its 4 GB default replay ceiling inside a 384 MiB container limit, recovered the WAL, began flushing 68 to 144 MB Longhorn streams, and was repeatedly OOM-killed. Recovery preserved the WAL: a 256 MB replay ceiling forced bounded flushes, 1 GiB headroom completed replay, and Promtail was restarted with its persisted positions to clear the outage backoff. Native checkpointing compacted the old segments without manual deletion. The durable values keep the 256 MB replay ceiling, set a 1 GiB limit with a 512 MiB request, and enable the Loki ServiceMonitor. After merge, Argo rolled Loki from Git and returned Synced/Healthy; the compacted WAL replayed in about 0.52 seconds, Loki was Ready with zero restarts at about 181 MiB resident memory, failure metrics stayed at zero, and a fresh five-line Promtail-to-LogQL proof passed. The end-to-end case study is recovering-loki-wal-replay-oom.
  • Devata Grafana home dashboard live (2026-07-19, lab PR #49). Devata Overview is now the provisioned Grafana home dashboard: a Git-managed ConfigMap carries its JSON, the chart’s dashboard sidecar watches the grafana_dashboard=1 label and writes it to the shared /tmp/dashboards/Devata/devata-overview.json path, and Grafana’s file provider loads it into the Devata folder with UI writes disabled. The stable UID is devata-overview; Grafana reports it provisioned with 14 panels, and the home API returns the same dashboard. All 14 PromQL expressions returned successfully against the live datasource. A real drift test deleted the dashboard ConfigMap; Argo recreated it in four seconds, briefly reported OutOfSync, then returned kps to Synced/Healthy while the Grafana home remained correct. The end-to-end learning chapter is building-devata-overview.
  • Storage durability complete (2026-07-17, lab PR #48). Longhorn 1.12.0 is live through Argo CD with the V1 data engine, a non-default two-replica StorageClass, and /var/mnt/longhorn on both Talos workers. .9 gained iscsi-tools, util-linux-tools, and the shared kubelet mount without repartitioning; .10 kept its existing XFS UserVolume. A 1 GiB disposable PVC preserved its checksum across pod recreation, a 71-second normal reboot, and a 126-second forced cross-node failover, then returned to two healthy replicas; Longhorn used 109-130m CPU and 586-588 MiB. Grafana, Loki, and Prometheus were cold-backed up and migrated one at a time, each passed a real rebind to its retained local PV and a final return to Longhorn. Velero 1.17.1 now sends opt-in Kopia file-system backups to private R2 with sealed cloud and repository credentials, no schedule, and a preflight ceiling below 1,000,000,000 bytes. The real game day backed up 8,388,629 bytes from a 16 MiB PVC, deleted the source namespace and Longhorn volume, restored into a mapped namespace, and returned the exact checksum. R2 finished at 8,437,201 bytes; Velero used 5-6m CPU and 121 MiB. The successful backup retains a seven-day TTL. All 14 Argo applications were Synced/Healthy after returning every source to main.
  • OptiPlex bare metal migration complete (2026-07-16). Retired Proxmox on the Dell OptiPlex 7040 after copying and checksum-verifying the host configuration plus both Talos worker VM archives off-host. The replacement worker is talos-opt-7040 at 192.168.1.10, running Talos v1.11.5 and Kubernetes v1.34.1 directly on the 256 GB NVMe. Its disk layout is a 43 GB Talos EPHEMERAL partition plus a 193 GB XFS user volume mounted at /var/mnt/longhorn; intel-ucode, iscsi-tools, and util-linux-tools are loaded. The two VM node objects were removed. Grafana, Prometheus, and Loki received fresh claims under /var/mnt/longhorn/local-path, an interim single-node persistence bridge until Longhorn is installed; their former histories remain only in the VM backups. A full firmware power-cycle boot succeeded, the XFS volume remounted, a proof file in Grafana’s PVC survived, and all 12 Argo CD Applications returned Synced/Healthy. Article evidence collection then caught a post-migration regression: a label-only main MachineConfig update had dropped the auxiliary UserVolumeConfig from active state and unmounted u-longhorn while Kubernetes still looked healthy. Reapplying the complete three-document configuration restored the volume; a second firmware reboot remounted /dev/nvme0n1p7 and preserved a new PVC proof file. Future node changes must preserve the complete document set and verify both volumestatus and mountstatus. Merged lab PR #45 holds the bare metal machine and placement configuration; merged PR #46 holds the reconciled local-path mapping. Tailscale and remote access are deliberately deferred, so the node is LAN-only for now. The full case study is deleting-the-hypervisor.
  • M0 done (2026-06-18): Cilium-only (Flannel + kube-proxy removed), all control-plane dashboards live, Grafana secured, observability up.
  • GPU hardware-fault verdict from 2026-06-27 superseded on 2026-08-13. The earlier 535 and 570 matrix repeatedly failed RmInitAdapter, produced Xid 140 with GSP enabled, and could not read the GTX 1650 Ti VBIOS, which justified treating the GPU as unusable at the time. After moving Nitro to Talos v1.12.11 and the NVIDIA 580.126.20 LTS extensions, the same bare-metal node loaded all modules without those signatures and nvidia-smi successfully identified the GPU, driver, CUDA 13.0, temperature, power, and 4096 MiB VRAM. The old conclusion no longer holds for adapter initialization or NVML. A scheduled CUDA workload is still required before declaring compute end to end. The historical failure matrix remains useful evidence of a driver/Talos/headless compatibility boundary rather than dead silicon.
  • GPU enablement (2026-06-26): full K8s stack lit, blocked at the lowest rung — the GPU’s own firmware bring-up. Built a Talos Image Factory image with proprietary 570 extensions (nonfree-kmod-nvidia-production + nvidia-container-toolkit-production, 570.172.08), upgraded talos-lqv-w4u, loaded the kernel modules (node-scoped patch), added RuntimeClass nvidia, labeled the node + ns privileged, and ran the device plugin. Every rung verified: /dev/nvidia* present, libs/toolkit inject correctly. BUT nvidia.com/gpu never appears because RmInitAdapter failed! (0x62:0x40:2520) + Xid 140 — the 570 GSP firmware handshake fails on this Turing card, so NVML enumerates 0 GPUs. Proven not-a-k8s-problem: even a privileged pod with /dev/nvidia0 gets “No devices found”. All config staged in lab/lab-experiments/kubernetes/gpu-enable/ (schematic, patches, RUN.md, before-state).
    • Fix attempt 1 (done): NVreg_EnableGpuFirmware=0 (GSP off) — cleared the Xid 140 firmware error but RmInitAdapter still fails, new code 0x24:0x72:1512. Ruled out (via reads, no reboot): not a BIOS/BAR problem (VRAM aperture BAR1 mapped), bare-metal Acer Nitro AN515-44, failure moves with driver config → not dead silicon, it’s the driver RM.
    • Fix attempt 2 (LTS 535) — BLOCKED on infra, not GPU: 535 schematic built (af8eb824…, 535.247.01), but the node can’t pull the Factory installer image — machined times out after 7 min (failed to pull … context/pull). Node has flaky DNS (resolver errors in dmesg). Node currently sits on 570 + GSP-off, healthy/Ready, GPU still dark. To resume: fix node DNS / retry the pull, then talosctl upgrade -e .9 -n .9 --image factory.talos.dev/installer/af8eb82417d3deaa94d2ef19c3b590b0dac1b2549d0b9b35b3da2bc325de75f7:v1.11.5.
    • 535 pull blocked by node connectivity, NOT the GPU: talos-lqv-w4u’s NIC firmware fails to load (r8169: Unable to load firmware rtl_nic/rtl8168h-2.fw (-2)) → flaky external transfers → the ~hundreds-of-MB Factory installer pull stalls/times out repeatedly. Half-fixed: pointed node DNS at the router (machine.network.nameservers: [192.168.1.1, 1.1.1.1], live patch) — DNS errors stopped, but the bulk-transfer half still stalls. Node pulled smaller images (CUDA, original 570) fine, so it’s intermittent. Strategy: let talosctl upgrade grind with a long timeout (don’t kill it — containerd resumes partial layers, machined retries ~every 2 min). 2026-06-26 outcome: gave up — pull never completed even with router DNS + a patient 15-min grind. 535 untested. Node clean on 570. Resume needs the node’s connectivity fixed first (wired-cable/port check, or mirror the Factory image to a LAN registry the node can reach).
    • 2026-06-26 deeper finding — it’s an external CDN path issue, NOT the node or its hardware. Measured NIC link: clean, 100Mbps full-duplex, zero errors (same as healthy nodes) — retracted the “bad NIC” theory. Real blocker: the 365MB installer image’s big layer (341MB) lives on cdn.factory.talos.dev (S3/ENAM). Downloads flow fast (~12MB/s) for the first ~82MB then hard-stall to ~12KB/s; byte-ranges past ~82MB return HTTP 206 + 0 bytes. Reproduces on the workstation too (not just the node) → external path/throttle to Factory’s CDN, while general internet (github/ghcr/cloudflare) is fine. LAN-mirror plan blocked because the workstation can’t pull the image either. Resume options: VPN/different egress (e.g. Cloudflare WARP) to route around it → pull → local registry (docker run -p5000 registry:2) → talosctl upgrade --image <ws-ip>:5000/...; or pull from a different network/hotspot; or just retry later (may be transient Factory CDN trouble). Node clean on 570 throughout.
    • Honest odds on 535 itself: a real gamble even if it pulls — 570-GSP-off already fails past the firmware handshake, deep in adapter init. If 535 also fails RmInitAdapter, conclusion is a card/VBIOS-level fault on this laptop dGPU. Possible NIC fix worth bundling into a future image: add siderolabs/realtek-firmware extension (but that also needs a pull to land).
  • Repo structure baseline (2026-06-28): PR open, awaiting Pragalva’s review/merge. lab is being turned from a notebook into the declarative source of truth for devata. Issue #1 set the skeleton + conventions; PR #2 (repo-structure-baseline) implements it: three-plane layout (talos/ applied by talosctl, kubernetes/ as the only GitOps-watched path, lab-experiments/ the never-reconciled sandbox), kubernetes/{bootstrap,clusters/devata,infra/{networking,controllers,observability,ingress,storage},apps/showcase}, a purpose README in every dir, hygiene rules in docs/conventions.md, and the reasoning as ADR-0001. Renamed ansible-1/ → ansible/. Installs/migrates nothing — each component lands later under its own follow-up issue (GitOps, Sealed Secrets, ingress+TLS, Cloudflare Tunnel, showcase publisher, storage). The in-progress lab-experiments/kubernetes/gpu-enable/ WIP was deliberately left untracked, out of this PR.
  • GitOps engine LIVE on devata (confirmed 2026-06-29). Argo CD is installed, self-managing, and reconciling git. argocd app list shows three apps all Synced/Healthy: argocd (self-manage, prune:false), devata-root (the app-of-apps watching kubernetes/clusters/devata/), and the throwaway hello demo. Control-plane node 192.168.1.8 is back, all 4 nodes Ready. The bootstrap walkthrough (content/gitops/bootstrapping-argocd/) was run end to end, and a second vault chapter content/gitops/migrating-the-imperative-stack/ is now written: helm-application + adoption concept notes plus the per-component plan to bring the still-imperative stack (Cilium, MetalLB, kube-prometheus-stack, Loki, Promtail, nvidia-device-plugin) under git, least-blast-radius first. dictionary.json carries both chapters.
    • Adoption runbook written + navigation fixed (2026-06-29), vault PR #6 MERGED to master. Pragalva will do the remaining Helm adoptions himself to learn, so wrote a dedicated command-by-command note content/gitops/migrating-the-imperative-stack/adopting-a-helm-component.md: discover the next component, recover values, write the YAML, diff to empty, flip automation, retire the Helm record. The migrating chapter links it as the followable companion to its plan; dictionary.json registers it. Same PR fixed the front door: content/index.md was not linking the GitOps chapters at all (orphaned), now has a GitOps section linking both in order. npx quartz build clean (29 files, no broken links). Live via Cloudflare Pages off master.
    • First real adoption MERGED (lab PR #7, 2026-06-29). nvidia-device-plugin child Application + recovered values committed to lab on gitops-p1 (kubernetes/infra/controllers/nvidia-device-plugin/{values.yaml,README.md} + clusters/devata/nvidia-device-plugin.yaml), multi-source $values Helm pattern, chart pinned 0.19.3, automation deliberately OFF. The safe rehearsal: GPU is a confirmed hardware fault, so the DaemonSet advertises nothing and adoption cannot hurt anything. Merged; the hands-on follow-through (diff-to-empty, flip automation, retire the sh.helm.release.v1.nvidia-device-plugin.v1 Secret) is the first checkbox of lab issue #9.
    • nvidia follow-through CONFIRMED DONE (2026-07-02, later session): helm list -A no longer shows nvidia-device-plugin (release record retired) and the child Application in git has automated: {selfHeal, prune} on — Pragalva completed the first checkbox of lab #9 hands-on. Next component: kube-prometheus-stack.
    • Migration nearly complete — Pragalva ran four adoptions solo (verified live 2026-07-04). kps, loki, promtail, metallb all adopted hands-on (lab PRs #22–#24 among them): Synced/Healthy, automation on, Helm release records retired — helm list -A shows only cilium. Cilium itself is adopted in git too (PR #25, automation off per the runbook) but diff-to-empty is NOT met: benign first-sync noise (tracking-id annotations, three empty-string cilium-config keys) plus the real blocker — cilium-ca + both hubble TLS Secrets differ because the chart default hubble.tls.auto.method=helm regenerates certs on every render (values.yaml sets nothing under hubble.tls), so the diff can never empty and selfHeal would churn Hubble TLS forever. Decision before automation (Pragalva’s): hubble.tls.auto.method: cronJob (Cilium’s documented GitOps pattern) vs ignoreDifferences on the three Secrets. Then: manual sync → automation on → retire sh.helm.release.v1.cilium.v3. Loose thread: kps app shows ExcludedResourceWarning(3), unexamined.
    • MIGRATION COMPLETE — lab #9 CLOSED (2026-07-04, verified live). Pragalva finished cilium’s last mile same-day: chose ignoreDifferences (jsonPointers /data on cilium-ca + both hubble cert Secrets) over the cronJob method, correctly paired with RespectIgnoreDifferences=true in syncOptions so syncs skip the ignored fields too (without it, every sync would re-push freshly rendered certs). Lab PRs #26 (TLS fix) + #27 (automation on), all through PRs per policy. End state: cilium Synced/Healthy/Automated-Prune, argocd app diff cilium empty, helm list -A returns NOTHING — zero imperative Helm state on devata; every component is git-described and Argo-reconciled. Known tradeoff, recorded: git does not describe how hubble’s certs come to exist (helm render minted them once at first sync); hubble.tls.auto.method: cronJob is the upgrade path if cert provenance ever matters.
    • kps walkthrough note written (2026-07-02): content/gitops/migrating-the-imperative-stack-notes/adopting-kube-prometheus-stack.md — the runbook instantiated for kps, standalone. Key finding from the live cluster: helm get values kps carries grafana.adminPassword in PLAINTEXT, so the note adds a phase 0 before the standard drill: create a manual grafana-admin Secret (same password, from the local .grafana-admin-password file), switch values to grafana.admin.existingSecret, one last imperative helm upgrade -f (never --reuse-values, which would keep the password in the release record) — then diff-to-empty proceeds on clean values. Also covers: what a Secret is (base64 ≠ encryption), ServerSideApply, the admission-webhook caBundle diff residue + ignoreDifferences fix, retiring the Helm records as destroying the plaintext copies, and a sealed-secrets (lab #8) preview incl. the key-backup warning. Dictionary +5 keys (secret, sealed secrets, …). The grafana-admin Secret is the one debt the adoption leaves; lab #8 pays it and unblocks deleting the old dashboard folder (vault #12). Reviewed adopting-a-helm-component.md for accuracy before Pragalva runs it. Live facts: root created the nvidia child app, automation still off, and argocd app diff nvidia-device-plugin is ALREADY EMPTY (app shows Synced/Healthy without any sync) — the diff-to-empty step is done for free; his remaining hands-on is flip automation on, drift test, retire the Helm Secret (still present). Three doc fixes: (1) the big one — the Application must be named after the release (or set helm.releaseName) because charts embed the release name in resource names; devata’s kube-prometheus-stack release is kps (kps-grafana etc.), so naming the app by chart would render a parallel stack and the diff could never empty; (2) “OutOfSync is expected” rewritten — Synced-with-empty-diff immediately is the success case, and argocd app diff normalizes Argo’s tracking label (no “label leftover” appears); (3) drift test now has a DaemonSet variant (kubectl set env ds/...) since the first two components (nvidia, promtail) ship DaemonSets, not Deployments. Chapter note now flags the kps release name in its kube-prometheus-stack section.
    • Sealed Secrets chapter written (2026-07-03), install-ahead pattern again; hands-on is Pragalva’s (lab #8). Written while off the homelab network (cluster unreachable, untouched) — Sealed Secrets is the one workstream that needed no live cluster (fresh install, no values to recover), and it gates the kps adoption’s secret handling. New chapter content/gitops/installing-sealed-secrets-notes/: sealedsecret + kubeseal concept notes plus step-by-step landing note installing-sealed-secrets.md covering: discover/pin the chart, child Application (release sealed-secrets into kube-system with fullnameOverride: sealed-secrets-controller so bare kubeseal works, values file is that one line), automation off → manual first sync → verify key born → flip on, same-day key backup to ~/devata-secrets/ (proposed as the new local home for ALL the old dashboard folder’s secrets/backups — clears vault #12’s blocker) + second copy off-laptop + refresh-after-renewal rule + restore procedure, commit pub-cert.pem for offline sealing, throwaway round-trip, then seal grafana-admin (annotate sealedsecrets.bitnami.com/managed=true so the controller may take over the hand-made Secret; delivered via a third directory source on the kps Application pointing at infra/observability/kps/secrets/, since the ref: values source applies nothing). Closure shrank from the proposed 4 notes to 3: the plain-Secret concept was already covered in adopting-kube-prometheus-stack (dictionary hit), linked not re-taught. Dictionary +5 keys, sealed secrets/sealed-secrets repointed from the kps sketch to the dedicated note; index GitOps section now lists all three chapters in order.
  • Sealed Secrets LIVE, lab #8 DONE (2026-07-05, verified on cluster). Pragalva ran the walkthrough hands-on: controller Running in kube-system (Argo app sealed-secrets Synced/Healthy, 10 apps total), the grafana-admin SealedSecret in monitoring shows Synced=True, and the sealing key is backed up same-day to ~/devata-secrets/sealed-secrets-keys-2026-07-05.yaml (dir 700 / file 600). ~/devata-secrets/ now exists as the local home for cluster secrets, which clears the blocker vault #12 named for the password file. Remaining tail: (1) the off-laptop second copy of the sealing key, (2) lab branch kubeseal-cert holds the uncommitted pub-cert.pem (verified byte-identical to the live controller cert via kubeseal --fetch-cert) awaiting its PR, (3) .grafana-admin-password in the old dashboard folder can now die, but machineconfig-backups/ and the meshery backup still need moving to ~/devata-secrets/ before the folder deletes. Key connection noticed: sealed secrets was the gate on the Cloudflare Tunnel (lab #13), whose credentials Secret can now live in git, so the Showcase v2 spine is buildable declaratively.
  • STORAGE BUG FOUND (2026-07-05): Prometheus loses its entire TSDB on every node reboot; the lab #14 parking trigger has fired. Discovered while trying to measure cluster uptime (an initial “up 20% of the week” claim was RETRACTED as an artifact of this bug). Evidence chain, all verified live: Prometheus’s PVC is Bound (5Gi local-path on talos-igk-9ep) but the PV directory on the host has been untouched since June 9 (a single stale zero-byte queries.active, no prometheus-db subPath dir); Prometheus’s startup log at the 2026-07-03 23:26 UTC boot shows a fresh empty TSDB (WAL segment=0 maxSegment=0, replay 1.3ms); its container has 34 restarts, meaning ~every reboot started from zero. Meanwhile Loki’s PV on talos-3eq-vo2 (same /opt/local-path-provisioner path) persists fine across reboots (June 9 through July 5 files on disk). Ruled out: per-node machine config differences (no extraMounts on any node, configs identical), provisioner nodePathMap (single DEFAULT /opt entry), Talos version skew (all four nodes v1.11.5). Mechanism unresolved; needs on-node forensics (where do Prometheus’s writes actually land on igk-9ep). Known Talos guidance says local-path must live under /var with kubelet extraMounts; /opt is unsupported, so Loki persisting is the accident, not Prometheus losing data. Also surfaced: local-path-provisioner (ns local-path-storage, 150 days old) is kubectl-applied, NOT under Argo — the one component helm list could never see; bring it into GitOps as part of the fix. Uptime facts that survive the retraction: three desktop nodes power-cycled together 2026-07-03 23:26 UTC (05:11 NPT), ~19 reboots since June 9, cluster on July 2 evening through July 5 per Loki chunk timestamps (Loki retention ~72h caps how far back that record reaches). An honest public uptime number needs off-cluster memory; the snapshot publisher’s commit history provides it for free.
  • Roadmap v2 + the issue backlog exist (2026-07-02, vault PR #11 merged). The 06-28 “100-day roadmap” was replaced: it had gone stale (still said “install Argo CD”) and, the big one, it parked the LLM bridge for 100 days, which the 06-29 career decision overrode — the keystone (train the book’s GPT + serve a quantized GGUF via llama.cpp on devata’s CPU) is now the lead, showcase demoted to supporting. Bundle refreshed in homelab-vault/content/planning/ (roadmap v2, baseline delta, serving research addendum). Also fixed v1’s false privacy claim: planning/ was actually being built and deployed (Access-gated, so not publicly reachable); it was added to ignorePatterns — superseded the next morning: Pragalva reverted the ignore (feccdb9, 2026-07-02), deliberately publishing planning/ behind the Access gate; the repo is private and the site is gated, so the exposure is a signed-in Pragalva only. Confirmed by observation: pragalva.me = Vercel origin behind Cloudflare proxy (v1’s blocking decision 1, answered); vault.pragalva.me live behind Cloudflare Access. Cluster was OFF at check time (no route to 192.168.1.8:6443). The per-workstream issues promised by lab issue #1 now exist: lab #8 Sealed Secrets, #9 finish the imperative-stack migration (checklist), #10 Gateway API + cert-manager DNS-01, #11 llama.cpp keystone serve, #12 snapshot publisher, #13 tunnel (stretch), #14 storage (parked, with trigger); vault #12 journal migration; hub #32 snapshot consumer (v1.5, between #21 and #23).
  • Ahead: roadmap is the current next-goal map. Existing repository issues remain authoritative for their own already-recorded workstreams.

LLM from scratch

  • Chapter 3 started (2026-07-10): attention. Chapters 1 and 2 covered tokenization (regex → BPE/tiktoken), sliding-window dataloaders, and token + positional embeddings.
  • Project foundation reframed (2026-07-12): attention-is-all-you-need exposed the need for a learning system before more implementation. llm-from-scratch-project-overhaul now defines a notebook-first collaboration: recover Chapter 2 into concept-driven labs, make Pragalva predict/run/explain, then extract tested reusable code only after understanding converges. Recommended project identity: GPT from First Principles (gpt-from-first-principles).
  • CPU is the design now, not a stopgap: training stays on the laptop, serving on devata is CPU llama.cpp (lab #11); the dead GPU is no longer a blocker anywhere.

Writing

  • Active Hugo blog; LLM series in progress (“Understanding LLMs”, “Working with Text Data”).
  • GitOps migration post MERGED (hub PR #38, merged 2026-07-05; drafted 2026-07-04). “How I Made My Homelab Declarative”, built from his skeleton draft (~/Desktop/archives/meshery/meshery-vault/How I made my homelab declarative.md, which explicitly asked for the detail to be filled in). All facts sourced real: lab manifests, adoption PRs 20–#26, live cluster verification (helm list -A empty, 9 apps Synced/Healthy). His empty-helm screenshot is the closing figure (blog/static/images/posts/gitops-helm-list-empty.png); new theme-matched SVG cover of the reconcile loop. The ownership contrast with M0 (below): this time the engineering is genuinely his (he ran the adoptions solo), only the prose was delegated — the review pass on PR #38 is where he makes the words his own.
  • M0 post is NOT just unwritten — it’s unowned. M0 engineering was done by Claude; Pragalva has little hands-on sense of what happened, so the lessons read as notes about someone else’s work. Can’t write a credible post from that. → new track: understand-by-experiment first, blog second.
  • Docs revamp to Obsidian vault + Quartz — COMPLETE except deleting the old folder (2026-07-02). Vault live at vault.pragalva.me (Pages + Access), concept-note system running (dictionary, prereqs, link-check CI), journal now lives here. The original HANDOFF-vault-revamp.md is superseded and deliberately left in the old folder to die with it. Voice rules unchanged: mentor/documentation voice, long-form prose, no templates, no em-dashes, reader does the work.
    • 2026-06-27 — vault scaffolded and serving. Quartz is now v5 (was v4 in the handoff). Repo cloned to ~/Desktop/projects/homelab-vault/ with the Obsidian template (npx quartz create --template obsidian --strategy new), which auto-wires wikilinks/callouts/mermaid and shortest-path links. Config is quartz.config.yaml (YAML, not the v4 .ts). Pragalva ran the scaffold himself; npx quartz build --serve confirmed local.
    • Migrated in this session: going-cilium-only.md → content/networking/ (H1 converted to frontmatter title so it renders once); the ledger → content/private/vocabulary-covered.md (the template’s ignorePatterns already excludes private/, so it never publishes — verified: npx quartz build emits no vocabulary file to public/); homepage content/index.md rewritten with a wikilink to the article. Config fixes: pageTitle: devata homelab, analytics: null (killed the template’s Plausible script).
    • Journal migration DONE (2026-07-02, vault issue #12). Live-state home decided: this file, content/planning/journey.md, one source of truth; the old path was stubbed to a pointer the same day, so there was no drift window. Reader-facing cut: none needed, planning/ is unpublished and durable lessons keep flowing into concept notes as before. ACTION-PLAN.md dropped (stale by two roadmaps, not migrated). Also recreated in the vault: the 06-18 cluster snapshot (planning/snapshots/), the Cloudflare runbooks plus DNS baseline (planning/runbooks/), and the old lessons/labs drafts. The drafts moved to archive/legacy-docs/ on 2026-07-26 so they no longer render as current learning material.
    • Old folder awaiting deletion (NOT deleted yet, deliberate): ~/Desktop/homelab-dashboard/ still holds .grafana-admin-password (dies when lab #8 seals the password into git properly), machineconfig-backups/ and meshery-removal-backup-2026-06-27/ (secrets/backups, never committed raw — they need a local home first), plus the superseded dashboard machinery (index.html, build.py, template, README, ACTION-PLAN, HANDOFF). Deleting the folder is the last step of vault issue #12.
    • Hosting decided (2026-06-27): going Cloudflare Access at vault.pragalva.me, which means moving pragalva.me DNS to Cloudflare (free plan = authoritative for the whole apex zone). Confirmed pragalva.me email is unused, so it’s a clean move with no email migration. DNSSEC is OFF. Baseline captured in dns-baseline-pragalva-me-2026-06-27.md; step-by-step in cloudflare-migration-runbook.md. Pragalva runs the console steps; blog stays up with zero downtime if the apex A is staged before the nameserver flip. The earlier “lab-doc template” approach is dropped (he rejected templates).
    • Hosting — Phase A done, Phase B speced (2026-06-27). DNS is now actually on Cloudflare (dig NS pragalva.me → aragorn/stevie.ns.cloudflare.com; blog apex returns HTTP 200). Phase B delivery method chosen: Cloudflare Pages via Git-connect (CF builds on push to master), not GitHub Actions or self-host on devata (cluster has no public ingress/TLS yet — that’s the future migration path, behind a Cloudflare Tunnel, Access app + DNS unchanged). Verified the CF build command locally: npx quartz plugin install && npx quartz build → output public/ (the plugin-install step is mandatory because .quartz/ is gitignored and config pulls a remote github: plugin). Full step-by-step (Pages project + custom domain + Access gate, all console-side) in cloudflare-pages-vault-runbook.md. Watch-item: apex A is currently proxied (orange) not DNS-only as the DNS runbook specified — blog still 200, flip to grey if Vercel TLS misbehaves.
    • Concept index + prerequisite system added (2026-06-27). New layer on the vault: dictionary.json at the vault root is the single {concept: path} index of every covered concept (seeded with the 14 inline terms → going-cilium-only, plus kubernetes and a talos stub). Every article now opens with a prerequisites table of [[wikilinks]] built from the dictionary; an undefined prerequisite becomes a short concept note or a draft: true stub, then is registered so the index never goes stale. content/private/vocabulary-covered.md is frozen (superseded by the dictionary, not dual-maintained). Worked the system end-to-end on the one existing article: wrote content/foundations/kubernetes.md (real, published) as its prerequisite, added a talos.md draft stub, and verified the build — kubernetes publishes; talos (draft), private/, and dictionary.json do not. Method now lives in the homelab skill’s Writing the docs section. Deliberately did not mass-produce concept notes (those come per-session, with Pragalva tracing).

Current goal portfolio

  • Goal 0, restore the operating contract: classify non-Argo state, repair Talos endpoint and client integrity, derive the public storage and accelerator truth, and establish a measured worker-loss reserve.
  • Goal 1, actionable operations: route one useful alert outside the cluster, dispose current alert noise, and monitor Argo CD, Longhorn, and Velero as recovery-critical controllers.
  • Goal 2, declarative external boundary: import existing Cloudflare resources into encrypted OpenTofu state with zero replacement, then give any future ExternalDNS records a non-overlapping ownership boundary.
  • Goal 3, PostgreSQL and a useful canary: prove CloudNativePG failover and point-in-time recovery with checksums, then operate Miniflux for a four-week utility trial. Paperless follows only if a real document habit exists.
  • Goal 4, bounded delivery path: create and destroy a quota-limited pull-request preview from an immutable first-party image digest. Add progressive delivery only after a stable service has a meaningful error metric.
  • Goal 5, power-loss recovery: when compatible UPS hardware exists, control shutdown from outside devata and prove worker-first, control-plane-last recovery through storage and application transactions.
  • Next observability experiment: add OpenTelemetry and monolithic Tempo only after a real multi-hop request and measured memory reserve exist.

Existing lanes still tracked elsewhere

  • LLM Chapter 3 and CPU inference remain in llm-from-scratch-project-overhaul and lab issue #11.
  • Cloudflare Tunnel and the showcase program remain in lab issue #13 and the hub issue set.
  • Security, reconstruction, NetworkPolicy, chaos, operator, and upgrade learning remain in kubernetes-coverage, reconstructing-devata, and their existing records.
  • Vault and repository housekeeping remain separate from the new platform goals.

Where the threads meet (notice, don’t force)

  • The bridge is CPU serving now, not the GPU (dead, hardware fault): llama.cpp on the 16-vCPU worker (lab #11) carries the “AI workload on my platform” story, and the book’s own GPT rides the same pattern later as the bounded stretch.
  • Every milestone and chapter is blog fuel — the writing thread documents the other two.
  • Size ambitions to CPU + ~1.5–3 GB free RAM: 1B-class quantized models, toy-scale own GPT; not large models.

Decisions / notes log

  • 2026-07-16: The public repository is the engineering artifact; this journey is the journal. Lab PR #47 replaces the hand-maintained hardware inventory, status report, and roadmap prose with stable architecture, trust boundaries, repository navigation, and links to generated evidence. The live badges stay because the cluster owns their values. docs/conventions.md now assigns current work to issues, decisions to ADRs, runtime values to generated surfaces, and chronology to this gated file. The bootstrap README owns the exact handoff commands, and the snapshot-publisher README now describes a reusable first-deployment gate and present limitation instead of its PR history. A deterministic docs workflow checks active relative links and rejects journal-style root sections such as Status, Current Progress, Next, and Roadmap. The homelab skill carries the same boundary and both agent clients resolve one canonical copy. No cluster state changed.
  • 2026-07-09 — Schema 1.3.0 found stranded by a stacked-PR merge-order slip; repair PR opened (lab #44, his to merge). Pragalva asked why the live snapshot still said 1.2.0 when 1.3.0 had been created. Trace: lab PR #43 (feat/snapshot-derived — Argo-tracking-gated services, per-app tier labels, publisher emitting 1.3.0) was stacked on chore/snapshot-copy and merged at 18:35:49 UTC on 2026-07-08, 69 seconds AFTER its base had merged to main as PR #42 (18:34:40). The base branch was never deleted, so GitHub never retargeted #43; its merge commit (3235fa9) landed on the already-consumed branch and main never received the work. Argo, correctly Synced/Healthy at main’s head, kept publishing 1.2.0 hourly — the cluster was honest, the git graph wasn’t. Repair: lab PR #44 opened, feat/snapshot-derived into main, the same two commits with no rewrites; once merged, Argo syncs and the next hourly tick publishes 1.3.0 with no cluster-side action. Lesson: a stacked PR’s Merged badge only means merged into its base branch as of click time; GitHub auto-retargets a stacked PR to main only if the base branch is deleted when the base PR merges. Habit adopted: merge stacks bottom-up with delete-branch checked, and read the “into ” line in the merge box before clicking. After #44 merges, chore/snapshot-copy and feat/snapshot-derived should both be deleted.
  • 2026-07-07 (evening) — Lab README front page shipped from the snapshot, no tunnel needed: lab PR #41 open, his to merge; hub #35’s stepping stone. Pragalva picked hub #35 as the session’s work; honest scoping said the kromgo-through-tunnel half is hard-blocked on lab #13, but the publisher changed the economics: shields.io dynamic-JSON badges read devata-snapshot/snapshot.json directly (CORS open, hourly), so the README gets live cluster-published numbers today. Shipped in the PR: nine badges (talos/k8s versions, nodes ready, pods, argo apps via $.apps.length, vCPU, memory, cluster age, last-snapshot timestamp), every query tested against the live snapshot before landing; provenance line under the badges; hardware table (ProBook cp, two Proxmox VMs, Nitro 5 with the dead 1650 Ti stated plainly, linked to lab-experiments/kubernetes/gpu-enable/); mermaid CGNAT topology (both outbound flows, ending at pragalva.me/homelab); Status section rewritten (it still said “contents land next” after 12 all landed). Badge honesty without kromgo: colors deliberately neutral (never green), the timestamp badge is the freshness signal — stale-coloring, utilization %, alerts, tokens/sec remain kromgo’s, so #35 stays open; commented there (incl. that its done-when’s “delete cluster.ts” line was overtaken by hub #42). Repo topics left as his one-command call. Worked from a fresh worktree ~/Desktop/projects/lab-readme because the lab checkout sits on unmerged branch feat/snapshot-logos (one commit, “Artifact Hub marks” — someone’s open lane, untouched). Also confirmed this session: the heartbeat pair is fully merged (lab PR #39 AND hub PR #50), and the ledger already earned its keep — kubectl got “no route to host” (workstation off the LAN) while the snapshot’s generatedAt proved devata up and publishing. Standing recommendation unchanged: the storage bug (lab #14) is the next depth project; the README was the light win in the meantime.; the sleep narrative is retired (lab PR #39 and hub PR #50 open, both his to merge; ledger seeded).** Pragalva flagged the live page’s strip going all dark with a rate-limit message, challenged the “powered down when idle” copy, and questioned the architecture (why a separate snapshot repo; why not a CI relay lab→hub with a scheduled checker). Rulings from the session: the separate repo STANDS — the in-cluster deploy key must never be able to write to the GitOps source (a leaked key would mean cluster compromise via Argo, not JSON vandalism), and a CI relay would add a second write credential plus GitHub scheduled Actions as the clock, which run late or skip and would record false darks (and auto-disable after 60 idle days). Root cause of the dark strip was the consumer, not the repo layout: it walked the commit log through api.github.com (60 req/hr anonymous, StrictMode doubling fetches in dev), and the rate-limited state rendered the same dark track as a week of downtime. Fix: every publisher run now maintains heartbeat.json beside the snapshot (rolling 336 beats, the job’s own clock only, outside the allowlist and schema gate by construction; commit history stays the audit trail) — lab PR #39; the hub hook fetches that one raw CDN file instead (hub PR #50, one commit on the #42 branch). **NARRATIVE RULING (his): devata is always on and is never described as sleeping; dark hours are unplanned outages, cause implied, never spelled out.** Applied across the lab README/cronjob/schema annotations and the page (beacon “asleep”→“offline”; caption now “devata runs around the clock; any dark hour is a real outage, recorded rather than hidden”). Ledger seeded same session: heartbeat.json backfilled in devata-snapshot from the 29 existing publisher commits (workstation commit, labeled, same pattern as the bootstrap), so the file starts with the full history and merge order cannot show a fake dark week; old script ignores the file, new script appends to it. Verified: ConfigMap YAML/schema JSON/sh -n clean, jq ledger logic exercised (first run, append, corrupt rebuild, 336 trim), hub branch npm ci + tsc + vite build clean, seeded file confirmed serving at the raw URL.
  • 2026-07-06 — PUBLISHER LIVE, the fake-data era is over: devata publishes its own snapshot (lab PRs #37 merged, #38 merged; lab #12 done-when met). Pragalva merged hub #42 and said “make it work end to end”; the lap ran the vault chapter as the spec with the settled rulings folded in (hourly 0 * * * *, commit-every-run ledger, schema 1.1.0 fields, the osVersion capture fix). Shipped: devata-snapshot repo created; publisher manifests under kubernetes/apps/showcase/snapshot-publisher/ (read-only SA/ClusterRole extended to pods+apps-kinds+argoproj applications for 1.1.0, script+pinned-known_hosts ConfigMap, schema ConfigMap now the canonical copy, CronJob born suspend: true); toolbox image built by a new publisher-image.yaml workflow with GITHUB_TOKEN (no registry PAT exists); child Application automation-off first, per the drill. Two open grill questions got answered by construction: (b) suspend-first won; (c) push credential is a DEPLOY KEY, not a fine-grained token (same one-repo blast radius, cannot expire into a permanently stale page, mintable via API so the lap could finish end to end; his to veto, swap documented in the chapter). Verification, all live: rehearsal render byte-identical to the merged stub except ageDays+1; gate PROVEN to reject a stray internalIp on additionalProperties; leak grep clean; bootstrap commit pushed from the workstation (labeled as such), then the first IN-CLUSTER heartbeat 14634df (11 apps incl. snapshot-publisher itself, ns 17→18, pods 57→58, exactly the sync’s own footprint); raw URL serves with CORS *; drift test passed (deleted CronJob, Argo selfHealed it back in seconds). Surprise worth keeping: the ghcr package came up anonymously pullable immediately (source label + workflow token; the chapter’s flip-to-public trap never fired), so zero console steps remained. Tails: (1) confirm the first scheduled tick DONE, the 12:00 UTC tick ran unattended (job Complete in 13s, commit “snapshot 2026-07-06T12:00:02Z”), lab #12 CLOSED with the evidence; (2) runAsNonRoot hardening for the image (pod runs as root, restricted-PSS warning recorded in the lab README); (3) hub #32 done-when met, commented, close is Pragalva’s; (4) the snap-manual pod’s PodSecurity warning is advisory only, devata enforces baseline. Concept-note proposal pending his accept/defer: deploy key, container registry publishing via the workflow token (neither in the dictionary). The page’s uptime ledger starts counting from today; the storage-bug session’s wish for off-cluster uptime memory is now being written one commit per hour. Two more rulings landed: (1) #49 must show the future BY DEFAULT — opening the preview shows the beast, no ?snapshot=future toggle; fixture upgraded to the roadmap at 100% (six bare-metal nodes 48 vCPU/64 GiB, four public surfaces — ask-devata, Grafana ops room, Gatus, kromgo — each with sloth SLOs, Longhorn 500 GiB replicated + Velero/MinIO, Gateway API + cert-manager, Dex OIDC, devata-operator, chaos-mesh, 23 green apps, ledger at 95.2%; the dead GPU stays dead). PR #49 body rewritten as the vision doc: each module mapped to the lab/hub issue that makes it real. (2) The packets note from #48 meant a real redesign, not a caption removal: topology signals rebuilt on #42 (e7f3a4c) — solid per-link gradient hairlines (faint at worker, bright at cp, direction reads in a still), heartbeats as bright streaks sweeping the link (stroke-dash sweep), a departure glint at the worker, and the control plane answering every arrival with an expanding ring; cycle scales with vCPU, reduced-motion leaves a calm constellation. Both branches verified (tsc/eslint/build, headless renders).
  • 2026-07-06 — Hub PR #49 opened: the stepping-stone half of issue #48, the page rendered as if the roadmap were finished. Grill settled two things first: the hero stays without “baremetal” for now (his ruling: “it will be baremetal later” — VM workers to be replaced with bare metal, a roadmap fact), and #48’s “another cluster” was a typo for “another node,” so multi-cluster and schema v2 are off the table entirely. The PR stacks on #42: /homelab is byte-identical; ?snapshot=future loads a code-split fixture (schema 1.2.0: services[].url + optional slo block, the additive delta #46 sketched — now the executable spec for lab #12) showing a fifth node, bare-metal workers, breathing hourly ledger, ask-devata + status badges public through the tunnel, sloth SLOs with error budgets, Longhorn replicated storage, 19 apps. Honesty extends to the future: amber banner + beacon + footer all say simulated, the ledger pattern is a labeled constant (a real ledger cannot testify about a future that hasn’t happened), and the dead GPU stays dead. New page capability is all conditional-on-data (SLO panel renders only when the block exists; stack tiles become links only when a service publishes a url), so nothing changes for real visitors until the cluster earns it. Ruling same day: #49 NEVER MERGES — he wanted only to look at the beast-at-+3-months with fake data; the dynamic capability belongs in the original PR. Restructured accordingly: the dormant modules (1.2 types, SLO panel, tile links — all data-gated, render nothing today) moved to #42 as f241ac6; #49 rebased down to demo-only (fixture, ?snapshot=future loader, watermarks, simulated ledger) and converted to draft. #42 awaits his judgment; never self-merge on hub.
  • 2026-07-06 — Round 4 on PR #42: Pragalva’s issue #48 teardown applied (commit fea8bdb), the anti-slop pass. His rule, now a comment at the top of the page: a line that is not derived from the snapshot and does not change what the reader knows, goes. Hero cut to one typed line (his copy), topology caption removed and ring placement computed for any node count (the old three fixed angles would overlap a fifth node), strip stats each carry their snapshot field as hover provenance (namespaces/volumes out, apps synced in), cat snapshot.json promoted to the FIRST panel fused with the pipeline row (his read: the drawer is the story of the page), gitops/manifest/footer fillers cut, trajectory panel deleted (hand-written intent; the map stays in hub #46). Stack rebuilt derived-not-curated: snapshot platform statements produce tiles, a registry only translates names to official simple-icons marks (new dep, tree-shaken; hand-inlined logo-paths.ts deleted), unknown tools render as monograms, so the tiles-vs-extras arbitrariness is gone and the wall grows with the cluster. Fact-check en route (talosctl systeminformation, first use of the newly working Talos access): worker-1/-2 are QEMU/Proxmox VMs, cp is the HP ProBook, worker-3 the Nitro 5 — so his suggested hero adjective “baremetal” is half-true; shipped without the word, grill question posed. Issue #48 also orders a stepping-stone PR: the page as it looks when the cluster is not the bottleneck (roadmap-complete future snapshot, builds on #46’s schema 1.2 sketch) — deliberately not started until the grill lands. Verified before claiming: tsc/eslint/vite clean, rendered headlessly at 1440px and 390px.
  • 2026-07-06 — Design converged: Pragalva closed 44 (“slop”), PR #42 is THE page; round 3 applied his review. Removed in-memoriam/storage panels (honesty folded into the footer), raw snapshot moved to the bottom as a distinct dashed strip, hero status collapsed to one typed line, topology’s cryptic “4c/16c” replaced by explicit vCPU/GiB sublabels, vCPU deprioritized. Cadence decision (his): publisher runs HOURLY (0 * * * *), consumer uptime window is 7 days at one cell per hour — supersedes both the lab #12 issue text (_/30, commit-on-change) and the chapter’s _/30; the commit-every-run-vs-on-change half of the parked grill question is still open, but at hourly cadence every-run is ~24 commits/day and stays the uptime ledger. The future question answered as hub issue #46: module-by-module map of what the page becomes as lab 11 and hub #36 land (page gains a hand-labeled // trajectory section mirroring it; schema 1.2 sketch: optional slo block + per-service public exposure). The second Claude session shipped competing PR #45 (“datasheet”, same contract) — still open, Pragalva’s to judge; lanes held. Meta-lesson recorded to memory: iterate one design deeply with his review; parallel variants read as slop to him.
  • 2026-07-05 — Design round 2: three /homelab candidates open (hub PRs #42, #43, #44); Pragalva picks one, closes the rest. A second Claude session joined mid-lap. Collaboration protocol: lanes claimed publicly in a PR #42 comment (this session owns the three design branches; main hub checkout parked on main; this session works from a git worktree at ~/Desktop/projects/hub-fable). Cluster op done en route: nvidia-device-plugin’s OutOfSync root-caused to the June 26 kubectl rollout restart annotation (restartedAt lives outside last-applied, so kubectl-apply syncs never remove it); dropped with a JSON patch, app Synced/Healthy, all 10 apps green, stub recaptured. PR #42 (mission control) v3 applied his feedback: English-led DEVATA wordmark with देवता demoted to an accent (remote-hiring audience), intro prose cut and first-person softness removed, heartbeat cells status-green with a solid empty state, pipeline row made unwrappable (his screenshot showed it broken mid-wrap), stack tiles in official brand colors (simple-icons hexes) on an auto-fit grid, provenance panel folded into the footer, raw snapshot.json added as a drawer per his ask. PR #43: amber-phosphor terminal session (pure transcript, block-char meters). PR #44: as-built engineering drawing (schematic + balloons + dimension line, node schedule, BOM with the GPU struck through, general notes, revision table, title block, red HOMELAB stamp). All three share the same data layer and live-captured stub, so lab #12 feeds whichever wins. Ponytail plugin disabled at his request.
  • 2026-07-05 — Hub PR #42 v2: hallmark redesign on Pragalva’s review (“this should be the coolest shit in my portfolio”), same branch, pushed same day. The page left the site’s Layout entirely: full-bleed mission-control surface, देवता Devanagari wordmark, first-person cluster voice, typed terminal rendering the fetched snapshot, constellation topology (node size = vCPU), HUD panels with corner ticks, real simple-icons brand marks, animated outbound pipeline with a hatched CGNAT wall, GPU memorial panel, and a 48-hour uptime ledger whose cells light from devata-snapshot commit timestamps (GitHub commits API, honest empty state until the publisher exists). recharts dropped from the page. Schema bumped to 1.1.0, additive (vault snapshot.schema.v1.json): optional apps[] (Argo name/synced/healthy — names already public in lab) and workloads{} (deployment/daemonset/statefulset/pod counts only); stub regenerated live (10 apps, nvidia-device-plugin honestly OutOfSync at capture; 22/7/3/57 workloads). Publisher-lap deltas this creates for lab #12: the CronJob’s RBAC needs get/list on applications.argoproj.io plus deployments/daemonsets/statefulsets/pods (counts only), and the chapter’s jq needs the 1.1.0 fields on top of the earlier osVersion capture() fix. Ponytail plugin disabled at Pragalva’s request (settings.json ponytail@ponytail: false); design-boundary rule for the hub: /homelab is deliberately allowed to break the site’s visual system.
  • 2026-07-05 — Showcase consumer lap done: hub PR #42 opened (stub-first /homelab), awaiting Pragalva’s review — never self-merge on hub. Grill (cut short by Pragalva: “consumer first, don’t ask the obvious questions”) locked two decisions: (1) full consumer now — fetch raw.githubusercontent.com/PragalvaXFREZ/devata-snapshot/main/snapshot.json, fall back to a bundled copy, render the freshness contract (fresh / stale / bundled-fallback states); the repo name devata-snapshot is now load-bearing, baked into the shipped URL before the repo exists. (2) Stub produced by the real pipeline, not by hand: the vault chapter’s jq render run locally against live devata (2026-07-05), schema-validated, leak-grepped clean. Stub-first paid immediately: the chapter’s jq selects the whole osImage (“Talos (v1.11.5)”) into cluster.osVersion where the schema documents “v1.11.5” — corrected with a capture() in the stub’s render; fold the same one-line fix into publishing-the-cluster-snapshot.md step 5 before building the CronJob. Salvage executed: #21’s rendering retyped to schema v1 (its cluster.ts published real hostnames/IPs/MetalLB addresses/namespace list — everything the schema bans); its chart was module-scope-importing data (would freeze on the stub forever) — made props-driven, and vCPU/GiB split into two single-unit panels instead of sharing one axis. Publisher-lap grill questions, deliberately still open: (a) commit cadence — lab #12 issue says commit-only-on-change, the chapter deviates to commit-every-run as the off-cluster uptime ledger (the storage-bug session made that valuable); (b) the chapter has no suspend: true staging, so after the first manual sync the schedule can push before the eyeball step — decide suspend-first vs accept; (c) push-token expiry policy (real date + calendar); (d) lab #12’s done-when includes “the portfolio page renders”, which now belongs to hub #32. His to close: hub PR #21 (unmerged, superseded) and #32’s stale “Depends on PR #21 (merge first)” line.
  • 2026-07-05 — Lab repo hygiene batch, PRs #32–#36, all merged same day; cluster verified Synced/Healthy after each. (1) Infra tree aligned to the documented layout (network/→networking/, logging/→observability/, four Application path refs updated in the same commit so main stays atomically renderable). (2) CI + Renovate: kubeconform strict over kubernetes/ (datree CRD catalog; CRD kinds skipped) plus a helm-template job that discovers chart/values/namespace from the Application manifests themselves, so new apps are covered with zero CI edits; renovate.json uses the argocd manager, weekly window (Monday pre-4am), Chore: prefix. Renovate’s first scan found nothing — teachable: bots read the default branch only, and the config sat in the then-unmerged PR. (3) De-vendored both installers (35.5k lines deleted): argocd install.yaml → kustomization on manifests/cluster-install?ref=v3.4.4, metallb-native.yaml → config/native?ref=v0.14.5 + the local pool config. ?ref= form chosen because Renovate’s kustomize extractor requires a query string (raw URLs are invisible to it). Both renders were diffed against the vendored files before deletion: identical resource sets, argocd namespace now explicit. The blog’s published “deliberately vendored” reasoning was revisited, not steamrolled — Renovate changed the economics, and the post now owns the change of mind (hub PR #41, merged). Bootstrap is now kubectl apply -k; fresh-cluster bootstrap needs GitHub reachable, tradeoff accepted. (4) Learning material archived under archive/ (trailing-space dirname fixed, generated log report dropped). (5) GPU work graduated: schematics → talos/schematics/nvidia-{production,lts535}.yaml with derived image IDs as comments; the nvidia RuntimeClass → device-plugin infra dir as a third Application source (the kps secrets pattern). This closed a real gap: the DaemonSet sets runtimeClassName: nvidia but the RuntimeClass was hand-applied only, so a from-git rebuild would have failed admission. Open flags: promtail chart is deprecated upstream (Grafana → Alloy, future migration); talos/machineconfigs/ carries no factory.talos.dev installer pin (wire it on the next node upgrade); nvidia vertical keep-vs-teardown on the dead GPU — mentor recommends keep (near-zero cost, the LLM thread wants the path warm), Pragalva undecided.
  • 2026-07-05 — Showcase scaffold analysed, deliberately not built. The lab task spec (Deployment/Service/Ingress/cert-manager Certificate placeholders) contradicts the revived v1.5 publisher design (push-only CronJob, keeps nothing open, CGNAT), the Gateway API direction (lab #10: HTTPRoutes, never Ingress), and today’s cluster (no cert-manager CRDs — the first sync would fail). Recommended shape: publisher skeleton per the vault’s publishing-the-cluster-snapshot chapter — read-only RBAC + suspended CronJob + snapshot-push-token SealedSecret sealed offline against the committed pub-cert (kubeseal strict scope: lock name/namespace before sealing) + child Application with automation off (the nvidia adoption drill). Decision pending Pragalva: publisher skeleton, literal spec, or SealedSecret+Application wiring only.
  • 2026-07-02 — Showcase v1/v1.5 superseded by Showcase v2 (“the cluster serves the showcase”); hub issues #33–#37 created. Pragalva judged hub PR #21 and the snapshot plan neither scalable nor impressive; research agreed (home-ops community gold standard is live kromgo badges through a tunnel on the repo itself; hiring managers discount “has a cluster” in favor of systems strangers can touch, measured reliability, incident maturity). New architecture: one Cloudflare Tunnel (lab #13, stretch → spine) plus an edge honesty layer (cached “last live Xh ago” when devata sleeps), after which every showcase surface is a manifest in the lab repo, zero portfolio code. Pillars: hub #33 Ask-devata public LLM chat (the keystone’s public stage, edge rate-limited — raw llama.cpp must never face the internet, so the edge work is itself the demo), #34 public ops room (curated anonymous read-only Grafana + Gatus; public read-only Argo CD rejected on ecosystem consensus, its status-badge endpoint is the safe GitOps evidence), #35 kromgo living README (supersedes hub #23’s custom status API; hub /homelab reads the same endpoints), #36 honest SLOs via sloth with the SLO defined on the edge-served surfaces rather than raw cluster uptime, plus public postmortems (GPU saga is postmortem #1, CDN stall #2), #37 umbrella with the full v1/v1.5 disposition (close #21 unmerged, supersede #32 + lab #12, absorb 27; 28 demoted to ordinary workloads). Nothing auto-closed — the old PR/issues are Pragalva’s to close. Cluster untouched throughout.
  • 2026-07-02 — Journal migration executed (vault issue #12, partially: deletion deferred on Pragalva’s instruction to recreate first). Live state moved to content/planning/journey.md in the private vault (versioned for the first time; the old folder was never a git repo), old path stubbed to a pointer, skill updated to read the new path. Snapshots, Cloudflare runbooks, and the legacy lesson/lab drafts recreated under planning/. Deliberately not migrated: ACTION-PLAN.md and HANDOFF-vault-revamp.md (superseded). Deliberately kept local-only: the Grafana password file and the machineconfig/Meshery backups (secrets never go in git raw; they need a local home before the folder can be deleted). The dashboard (index.html + build.py) is retired as an interface: not rebuilt anymore, dies with the folder.
  • 2026-07-02 — Replaced the 06-28 roadmap with v2 (vault PR #11, self-merged per vault policy) and created the issue backlog across all three repos (lab #8–#14, vault #12, hub #32). Reasoning: the old roadmap parked the on-cluster LLM work for its whole window, but the 06-29 career decision made train+serve the keystone artifact for the hired-by-end-2026 bet, so the plan now leads with it (book chapters 3–5 on the laptop in parallel with GitOps completion, then llama.cpp CPU serving through Argo, then Gateway/TLS; tunnel demoted to stretch, storage still parked with the same trigger). Also fixed the planning bundle’s false “never publishes” claim by adding planning to the vault’s ignorePatterns (it had been deploying behind Access), and answered v1’s hosting question by observation (Vercel behind Cloudflare proxy). Cluster was off throughout; nothing on it was touched.
  • 2026-06-28 — Wrote the GitOps bootstrap chapter in the vault (content/gitops/bootstrapping-argocd/) ahead of any install, because Pragalva wants to run the Argo CD setup himself and understand each step. Deliberate scope call: the first chapter stands up the engine and proves the loop on one throwaway app; it does NOT migrate devata’s existing imperative infra (Cilium/MetalLB/monitoring/logging) into git. Reason: adopting GitOps on a running cluster is safest done incrementally and reversibly, so brownfield migration is named as later, per-component chapters rather than a big-bang takeover. Install method chosen in the doc: raw upstream install.yaml (transparent) over Helm/Autopilot (which hide the moving parts), self-managed via app-of-apps. Bottom-up closure: 5 concept notes + index, only new dependency is the existing kubernetes note; dictionary.json +10 keys; build passes, links resolve. Did NOT touch the cluster (no argocd namespace created) so the hands-on stays Pragalva’s.
  • 2026-06-26 — Reframed M0 from “write the post” to “own it first.” The original M0 was done by Claude end-to-end; writing a blog from notes about work he didn’t do produced the focus-loss / hollow-notes problem. New method: experiment-driven notes (template + map note in the lab repo), forensics-on-devata then before→after in throwaway kind. Blog is the output of understanding, not a substitute for it.
  • 2026-06-26 — Left Meshery; no longer a homelab workload or focus (kept only as past background).
  • 2026-06-27 — Vault docs revamp: scaffolded Quartz v5 (Obsidian template), migrated the first article + ledger + homepage, verified the ledger does not publish. Chose Cloudflare Access for the gate (DNS move to Cloudflare); confirmed pragalva.me email unused so the move is clean. Journal migration deliberately deferred (live-state drift + stale ACTION-PLAN). Runbook + DNS baseline written beside the dashboard.
  • 2026-06-27 — Vault serving (Phase B) speced. DNS confirmed live on Cloudflare. Chose Cloudflare Pages via Git-connect as the deploy mechanism (over GitHub Actions and over self-hosting on devata, which isn’t ready — no ingress/TLS/GitOps). Reasoning: lowest maintenance, auto-deploys on push; the in-repo-config tradeoff is acceptable for a static site, and the homelab-native path (self-host behind a Cloudflare Tunnel) is a clean later migration that keeps the same Access app + DNS. Proved the CF build command (npx quartz plugin install && npx quartz build) locally — plugin-install is required because .quartz/ is gitignored. Wrote cloudflare-pages-vault-runbook.md. Console steps (Pages project, custom domain vault.pragalva.me, Access policy for dhirajs6342@gmail.com) are Pragalva’s to run.
  • 2026-06-27 — Added the concept-index + prerequisite system. dictionary.json (vault root, flat {concept: path}) is now the single index of covered concepts and drives a prerequisites table at the top of every article; vocabulary-covered.md is frozen and superseded by it (reopened handoff lock #6 deliberately, kept reversible — the file is preserved, not deleted). Reasoning: the prereq rule “link to where defined” needs inline-only terms (DNAT, CoreDNS) in the index too, which the dictionary’s concept→note mapping subsumes; two live ledgers would be the duplication the handoff says to kill. Undefined prereqs get a short note or a draft:true stub, never a recursive deep-dive (bounds the pragalva-learns-by-doing focus-loss risk). Proven on going-cilium-only + new kubernetes note; build confirms drafts/private/dictionary stay unpublished. Method recorded in the homelab skill.