research-findings

21 min read

Research findings for roadmap v2

Historical decision record

This page supported the 2026-07-02 roadmap. Its GitOps, secrets, ingress, storage, and power-state baseline is no longer current. Use next-goals-research for current recommendations and current-state for the source order used to verify live claims. The page remains intact as evidence for the choices made at the time.

This was the evidence layer under roadmap v2. Each recommendation in that roadmap pointed to a section here, and every URL below was fetched and confirmed during the research, not quoted from memory. Where a source was silent or a claim is reasoned rather than documented, it says so.

The two discriminators (read this first)

Your stack has two hard constraints that decide almost every choice below:

  1. CGNAT forbids inbound. Anything that “must be reached” needs an outbound tunnel, or it must be inverted so the cluster pushes out instead of being polled.
  2. The cluster is intermittently powered off. A pushed static snapshot keeps serving its last-known JSON while the cluster is dark; any live exposure (a server behind a tunnel, a metrics endpoint) goes blank the moment power drops.

These two axes are what separate “fits your homelab” from “fights it” throughout.

1. Showcase: publishing cluster state safely (the lead workstream)

The model: an in-cluster job renders a public-safe JSON document, pushes it outbound, and a static site consumes it. The schema is the contract. This has solid, verifiable prior art.

Upptime, the keystone for the output model

Upptime is a fully serverless status page. There is no Upptime server: a GitHub Actions workflow runs on GitHub’s cloud, commits response-time history to git, opens and closes GitHub Issues for incidents, and publishes a Svelte page to GitHub Pages. All public data is read-only and lives in a repo you own.

  • Transfers: the output half is exactly your model. Git is the datastore, commits are the update mechanism, a static host renders it, nothing accepts inbound. This is mainstream, production-proven precedent that a commit-driven static public surface works.
  • Does not: the monitoring half is backwards for you. Upptime’s runner sits in GitHub’s cloud and pulls your endpoints; under CGNAT it cannot reach your cluster. Your design correctly inverts this: the cluster pushes outbound. Upptime also watches external URLs only and has no notion of internal state.
  • Anti-pattern: Upptime is safe because its data is whatever a public URL already returns; there is nothing internal to leak. In your inverted version the leak risk moves to what your job writes into the JSON. “Static like Upptime” does not automatically mean “safe like Upptime.”

Kromgo, the keystone for the safety technique

Kromgo sits in front of Prometheus and exposes only predefined endpoints (/badges/{id}, /graphs/{id}), each a fixed PromQL query authored in YAML. Query parameters are never interpolated into PromQL, so only the values you curated are reachable; Prometheus itself stays unexposed.

  • Transfers: this is the gold-standard public-safe pattern to copy into your publisher: an allowlist of curated outputs, no user-controlled input, never expose the metrics backend. It is exactly how you derive counts, uptime, and node summaries without leaking everything Prometheus knows.
  • Does not: Kromgo is a live pull server. A public request triggers a query at that instant, so it needs inbound (a Cloudflare Tunnel under CGNAT) and the cluster up. When devata is off, it serves nothing.
  • Anti-pattern: an over-broad PromQL query (or exposing Prometheus directly) leaks internal series, labels, hostnames, and IPs. The allowlist is only as safe as the queries you write.

The judgment call: Kromgo is the strongest live alternative to the snapshot, so it is worth being explicit about why you push instead. On both discriminators the push-snapshot wins: it needs no inbound, and it survives power-off by serving the last JSON. So borrow Kromgo’s curation discipline and reject its liveness. This is the evidence behind blocking decision 2 in the roadmap (push now, defer the live API).

Push targets, all confirmed CGNAT-safe (outbound HTTPS only)

  1. Git commit via a scoped token, which triggers a Pages or Hugo rebuild. This is literally Upptime’s mechanism, with the commit coming from the cluster instead of a cloud runner.
  2. Cloudflare R2 PutObject over the S3 API. Nuance: R2 leaves S3 ACLs unimplemented, so objects are made public through the bucket’s custom domain or the rate-limited r2.dev subdomain, not via x-amz-acl. Buckets are private by default.
  3. Cloudflare Pages deploy hook, a unique POST URL that triggers a rebuild, explicitly intended for scheduled or cron jobs. The hook URL is itself a secret.

The roadmap’s default (a dedicated devata-snapshot repo written by a fine-grained contents:write token) is option 1, chosen for least blast radius.

What does not fit, and why (the foils that make the static choice deliberate)

  • Gatus, Uptime Kuma, Statping-ng, Cachet are all always-on servers with a database that must accept inbound and that go dark when the cluster powers off. Cachet’s curated component-and-incident data model is genuinely public-safe and worth imitating in spirit, but its operational weight is not worth it here.
  • Homepage (gethomepage) is documented as private and internal; it queries the cluster live and proxies real service APIs (holding their keys server-side). Public exposure is the canonical mistake; it is the thing your read-model exists to avoid being.
  • kube-state-metrics, kube-ops-view, Headlamp, Backstage are internal tools. kube-state-metrics is an excellent internal source for your job to read counts from, but its raw output carries namespaces, workload and node names, and labels and must never be the public surface. Headlamp is a credentialed read-write console; never public. Backstage is an enterprise portal framework, massive over-engineering for one home cluster.

Snapshot-specific anti-patterns (designed-against in the artifacts)

  • Stale-as-live: the static site keeps serving the last JSON while the cluster is off, so it can show a dead cluster as healthy. Mitigation, already in the schema: generatedAt plus freshness.maxAgeHours, and the consumer marks it stale past the threshold.
  • Identifiers or secrets in the JSON: anything pushed is permanently public and edge-cached. Mitigation, already in the publisher: an allowlist transform that selects safe fields only, never copy-then-redact, and the schema’s additionalProperties: false gate.
  • Leaking the push credential: scope it to one repo or one bucket, write-only, and rotate it.
  • No-op churn: committing an identical timestamped file every minute spams history and rebuilds. Mitigation, already in the publisher: commit only on change.

2. GitOps backbone (Argo CD vs Flux)

Both run as ordinary Kubernetes workloads; nothing about Talos or Cilium changes their install, and GitOps pairs especially well with Talos because the platform is already declarative.

  • Argo CD bootstraps via app-of-apps (a parent Application pointing at a Git path of child Applications); the docs now also present ApplicationSets as the contemporary pattern. Its web UI gives visual sync status, diffs, and manual sync or rollback.
  • Flux bootstraps with flux bootstrap, installing the GitOps Toolkit controllers; idempotent and lighter on resources.
  • Recommendation for you: Argo CD. The UI directly serves a learn-by-doing engineer who traces things himself, and a community comparison recommends Argo CD for small teams new to GitOps for exactly that reason. Honest counter-point from the same source: Argo CD has a higher resource footprint than Flux’s modular controllers, which matters on a 16 GB cluster, though Argo CD core runs fine in that budget. Flux plus SOPS is the lighter alternative if footprint becomes binding.
  • Sharp edge (the one boundary to internalize): GitOps here manages the Kubernetes layer only. The Talos machine config and OS are managed by talosctl, not Argo CD or Flux. Keep that split clear so you do not expect your GitOps tool to reconcile node-level OS state.
  • Why first: every later milestone (cert-manager, the Cilium Gateway, cloudflared, and eventually Longhorn and Velero) should be deployed through GitOps. Standing it up first means each subsequent step is a Git change, not another manual Helm drift source. The first action is migrating your existing hand-applied Helm releases (kube-prometheus-stack, Loki, Promtail, Cilium values) into Git.

3. Secrets in GitOps (Sealed Secrets vs SOPS+age vs External Secrets)

  • Sealed Secrets (Bitnami): a cluster-side controller plus kubeseal, which encrypts a Secret with the controller’s public key so only that controller can decrypt; the encrypted SealedSecret is safe in Git. Encryption is namespace-and-name scoped by default.
  • SOPS + age (via Flux): encrypt only the secret data with an age public key; Flux’s kustomize-controller decrypts at reconcile.
  • External Secrets Operator: syncs values from an external manager (Vault, cloud KMS). For a single CGNAT homelab with no such backend, it adds infrastructure for little gain; skip it.
  • Recommendation, coupled to the GitOps choice: Argo plus Sealed Secrets (the standard Argo-friendly pattern), or Flux plus SOPS+age. This is the evidence behind recommended default 5.
  • Sharp edge (operational, not Talos-platform): the single point of failure is the decryption key (the Sealed Secrets controller key, or the age private key). Back it up off-cluster, because Talos nodes are immutable with no shell and a wiped or rebuilt cluster will not regenerate the same key. This is the one line to carry into WS-1.
  • Why second: the next two milestones each need a credential committed to Git (a Cloudflare API token for cert-manager, a tunnel token for cloudflared). Establishing encrypt-before-commit now is what makes ingress and the tunnel safe to GitOps.

4. Ingress and TLS (Cilium Gateway API + cert-manager Cloudflare DNS-01)

  • Cilium Gateway API: Cilium implements Gateway API natively (eBPF forwards to an embedded Envoy), so you do NOT deploy a separate ingress controller. Prerequisites per the docs: kubeProxyReplacement=true (you have it), l7Proxy=true (default), the Gateway API CRDs preinstalled (you have them), and a way to expose the Gateway, which your MetalLB pool provides. Enablement is gatewayAPI.enabled=true in the Cilium Helm values.
  • cert-manager DNS-01 with Cloudflare: an Issuer with dns01.cloudflare.apiTokenSecretRef pointing at a Secret holding a scoped token (Zone:DNS Edit plus Zone:Read). DNS-01 is the correct and essentially only choice here because CGNAT means no inbound port 80, which rules out HTTP-01.
  • Transfers cleanly, and this is where your existing stack pays off: the CRDs are installed and kube-proxy replacement is done, so this is mostly enabling gatewayAPI and defining your first Gateway (currently zero) bound to a MetalLB IP.
  • Sharp edge: enabling Gateway API changes Cilium config and typically needs a Cilium restart or rollout, so do that change through GitOps. The Cloudflare token must come from the Sealed Secret in milestone 3, never committed plaintext. No Talos-specific gotcha.
  • Why third: depends on the token from milestone 3, and should itself be GitOps-managed. It comes before the tunnel so you have a real Gateway and valid internal TLS to point external exposure at.

5. Cloudflare Tunnel on Kubernetes (the CGNAT-safe outbound path)

  • Official approach: run cloudflared as a Deployment (multiple replicas for HA) with the tunnel token in a Secret injected as TUNNEL_TOKEN; public-hostname-to-internal-service routing is configured on the Cloudflare side. The connection is outbound-only, which is exactly what makes it work behind CGNAT with no inbound ports. Pair with Cloudflare Access for auth.
  • Sharp edge: the phrase “Tunnel operator” is worth flagging. The official Kubernetes path is cloudflared as a Deployment with a TUNNEL_TOKEN Secret; third-party “operators” exist in the community but are not Cloudflare-official, so treat them as optional convenience, not canonical. The token flows through milestone 3.
  • Why fourth: no hard dependency on the in-cluster Gateway (a tunnel can point straight at Services), so the strict prerequisite is only the token plus GitOps. Sequenced after ingress so the tunnel fronts your Gateway through one consistent entrypoint, but it can legitimately precede the Gateway work if you want public access sooner. This is what later unlocks the heavier live-data services (#23, #27).

6. Talos upgrade and Image Factory (the lifecycle skill under storage)

  • Approach: talosctl upgrade --nodes <ip> --image factory.talos.dev/metal-installer/<schematic-id>:<version>. Talos uses an A-B image scheme, so a failed upgrade auto-rolls-back to the prior OS. Control-plane nodes get extra etcd checks: Talos refuses an upgrade that would break etcd quorum and serializes control-plane upgrades to one at a time. Image Factory turns a schematic (system extensions, kernel args) into a stable schematic ID and the matching installer.
  • Recommended practice: upgrade control-plane nodes first, one at a time, confirm each rejoins and etcd is healthy, then upgrade workers.
  • Sharp edge: the immutable OS means an upgrade is a whole-image swap, not in-place patching, and you cannot add a system extension at runtime; you build a schematic and upgrade to the installer that bakes it in. With only 4 nodes, mind quorum. The deprecated Talos docs paths (www.talos.dev) now redirect to docs.siderolabs.com.
  • You already have a head start: the GPU experiment exercised exactly this (a Factory schematic plus a talosctl upgrade on the worker), and left you a LAN registry mirror that bypasses the flaky Factory CDN. Keep that mirror; it is your upgrade lifeline. This is why WS-0 folds the upgrade competency in rather than making it a separate workstream.

7. Storage durability (Longhorn + Velero), the most Talos-specific milestone

This is parked in the roadmap, but here is the evidence for why it is invasive and where it sits.

  • Longhorn gives replicated block storage to replace single-node local-path. General prerequisites: open-iscsi/iscsid with the iscsi_tcp module on every node, mount propagation, an NFSv4 client for RWX or backups, and root-capable workloads.
  • The Talos cost is real: you cannot apt install open-iscsi on an immutable, shell-less node. Longhorn on Talos requires two system extensions baked in via Image Factory, siderolabs/iscsi-tools and siderolabs/util-linux-tools, which is an Image Factory plus talosctl upgrade operation (hence milestone 6 is its prerequisite). The machine config needs a kubelet extraMounts bind for the Longhorn data path with rshared propagation (/var/mnt/longhorn on v1.10+), and the longhorn-system namespace must be set to privileged PodSecurity (Talos defaults to baseline, which blocks what Longhorn needs).
  • Resource reality: 16 GB across 4 nodes is tight for Longhorn’s per-volume replica processes plus default 3-way replication; lower the replica count and watch memory.
  • Velero backs up cluster resources and volumes; File System Backup uses a node-agent DaemonSet with kopia. It requires an object-storage target, which on a CGNAT homelab with no cloud means external storage (Cloudflare R2 or Backblaze B2) or an in-cluster MinIO. The object-storage requirement is from the Velero docs; the specific providers are general practice.
  • Why parked and placed last: it is the most invasive milestone, consumes the Image Factory and controlled-upgrade competency, and does not block the showcase. Pull it forward before you run anything whose PVC loss would actually hurt. Its natural companion experiment is the Velero restore game-day in section 9.

Ordering verdict (dependency-grounded): GitOps, then Secrets, then Ingress and TLS, then Tunnel, with the Talos upgrade competency folded under foundations and Longhorn plus Velero parked last. The day-number split (GitOps and secrets in the first ~30 days, ingress and tunnel through ~day 70, storage after) is a synthesis, not from any doc.

8. The LLM touchpoints (the honest intersection)

Context that shapes all of it: the 7.4 GB node already hosts kube-prometheus-stack, Loki, and Cilium/Hubble, so genuinely free RAM for an LLM pod is well under 7 GB. That makes a 1B-class model the realistic default and a 3B model tight. Touchpoints, ranked by whether they connect the book’s output to the homelab or just add an LLM-flavored workload:

  • B1, serve the book’s own from-scratch GPT, is the keystone genuine bridge. You train the tiny model on your laptop per the book, containerize the weights with a minimal FastAPI or Flask /generate endpoint, and deploy it as Deployment plus Service plus HTTPRoute. CPU is trivially enough for a toy model. This is the one touchpoint where the book literally produces a homelab artifact, and it is the ideal first workload for a HomelabApp operator if you ever build one. Output quality is a toy, and that is fine; as a bridge it is the best one available.
  • B2, inference metrics into your existing Prometheus and Grafana, is a strong genuine bridge because it reuses infrastructure you already operate. llama.cpp’s server exposes a native Prometheus endpoint behind --metrics (throughput and token-count series), a clean ServiceMonitor scrape. vLLM has the richest native /metrics but its docs are GPU-centric throughout, so cite it as the reference design, not the thing you run on CPU. Ollama has no native metrics endpoint and needs a community proxy. For your own GPT wrapper, instrument the FastAPI app directly with prometheus_client; that is the cleanest version because you control the code.
  • B3, on-cluster CPU inference of off-the-shelf small models (Ollama or llama.cpp), is real homelab work but a weak book-bridge. Ollama officially supports CPU as a fallback. By quantization math (a Q4 model is roughly 0.5 to 0.6 GB per billion parameters), a 1B model is about 1 GB of weights (comfortable on the free RAM) and a 3B model is 2 to 3 GB (tight once the monitoring stack is resident), plus KV-cache that grows with context. A real container gotcha: llama.cpp sizes its thread pool from host CPU count, not the pod’s cgroup limit, so set threads explicitly. Verdict: do it for the operating skills (limits, probes, model storage), not because it ties to the book.
  • B4, training on-cluster, is forced. A tiny nanoGPT on CPU took about 11 hours for 143k iterations in a real run; the cluster’s per-core CPU is weaker than a modern laptop, RAM is scarce, and training gains nothing from Kubernetes scheduling. Bring the trained artifact to the cluster (B1), not the training loop.

Net: the tracks are mostly parallel; the one clean bridge is B1 (serve your own model) once the book produces one, with B2 (metrics into the stack you already run) riding alongside. This is the evidence behind the roadmap parking the LLM bridge and the budget cut agreeing with the assessment.

9. Experiments worth exploring (the depth backlog)

Each tagged for what it costs you. All run on the eBPF and monitoring foundation you already have, so the cost is effort, not hardware. Talos caveats are flagged because the immutable, shell-less node model can constrain privileged tooling.

  • Cilium and Hubble plus Tetragon deep-dive [high-learning]. Tetragon adds eBPF runtime security (process, file, network, capability monitoring, plus enforcement) on the substrate Cilium already proves works. Talos caveat: deeper tracing can depend on kernel BTF and tracepoints; likely fine given Cilium runs, but verify. Anti-pattern: turning on in-kernel enforcement before running observe-only; you can break workloads cluster-wide.
  • Cilium policy testing [high-learning]. cilium connectivity test plus reading Hubble verdicts (hubble observe --verdict DROPPED) teaches default-deny and identity-based policy, and how to prove a policy works rather than guess. Anti-pattern: applying cluster-wide default-deny before observing flows; you lock out DNS and yourself.
  • Velero backup and restore game-days [high-learning]. Teaches that a backup is worthless until a restore is proven, plus namespace remapping for safe rehearsal. Needs an object-storage target (MinIO or external); PV data protection is CSI-driver-dependent on Talos, so verify the storage layer. Anti-pattern: storing backups in the same cluster they protect, or never actually restoring. This is the natural pair for the parked storage workstream.
  • Custom operator with kubebuilder [high-learning] [high-effort]. A HomelabApp CRD reconciling Deployment plus Service plus HTTPRoute plus Certificate is the single highest-leverage way to understand Kubernetes itself (controller-runtime, owner references, status conditions, idempotent reconcile). Cost is Go plus the controller-runtime curve. Anti-pattern: edge-triggered thinking; Reconcile is level-triggered and idempotent.
  • Chaos Mesh [high-learning] [high-effort]. CRD-driven fault injection (pod-kill, network latency and partition, CPU and memory stress) that validates your alerts actually fire. Talos caveat: the Chaos Daemon is a privileged DaemonSet with host-namespace access, which hardened Talos may constrain (uncertain, and some fault types may not work). Hardware caveat: stress faults on a 7.4 GB node can OOM real workloads. Anti-patterns: targeting the monitoring stack itself, or skipping a steady-state hypothesis before injecting.

10. Serving addendum (2026-07-02, for roadmap v2’s WS-2)

Section 8 already established llama.cpp’s native Prometheus /metrics, the CPU thread/cgroup gotcha, and the quantization RAM math. What v2 adds, checked 2026-07-02:

  • llama.cpp on Kubernetes CPU is a current, mainstream pattern, not an improvisation. There is an open KServe issue (kserve#5334, active April 2026) proposing llama-server as an official CPU-optimized ServingRuntime for GGUF models, and community deployment guides for the plain Deployment + Service shape. For one cluster, the plain Deployment is the right rung; KServe would be over-engineering.
  • Model choice for ~1.5 to 3 GB free RAM at Q4_K_M: Llama 3.2 1B Instruct runs in ~1.5 GB and is the default; Qwen2.5 1.5B (~2 GB) is the quality step-up; TinyLlama 1.1B (~1 GB) is the floor. Rule of thumb from the guides: GGUF file size + ~0.5 GB overhead = minimum RAM, plus KV cache growing with context. Q4_K_M remains the standard quality/size tradeoff.
  • Serving the book’s own GPT stays exactly as section 8 B1 designed it (FastAPI + prometheus_client), and stays a bounded stretch after both keystone artifacts, per the 2026-06-29 decision that GGUF-converting a hand-written nano-GPT is a rabbit hole to avoid.

Addendum sources:

References

Sources for sections 1, 8, and 9 (each fetched and confirmed):

Showcase and safe state publishing:

Stack progression (GitOps, secrets, ingress, tunnel, Talos, storage):

LLM touchpoints:

Experiments:

Uncertainty flags carried from research: Talos compatibility for Tetragon’s deeper tracing and for Chaos Mesh’s privileged DaemonSet was not confirmed by a Talos-specific source (likely but verify). Absolute Ollama RAM-per-model numbers are reasoned from quantization math, since the docs publish only the scaling formula. Re-check exact model tags at pull time.