Publishing the cluster snapshot

15 min read

Prerequisites: gitops, application, serviceaccount, rbac, cronjob, sealedsecret, kubeseal

This walkthrough has devata describe itself to the public. A cronjob inside the cluster reads cluster state through a read-only serviceaccount, renders a public-safe JSON document, validates it against a schema, and pushes it to a dedicated GitHub repo. The portfolio’s /homelab page then fetches that one raw file. The original design sketch is publisher-sketch; this note is the buildable version, updated for what devata actually is now (the sketch predates the GitOps migration and still described a manual-Helm cluster).

Push, not pull, for two reasons that will not change: CGNAT means nothing outside can reach in, and the cluster is allowed to be powered off, in which case a pushed static file simply keeps serving its last state. The document carries its own generatedAt timestamp so the page can say “as of 20 minutes ago” honestly instead of pretending a sleeping cluster is live.

A second property falls out for free and is worth naming: the snapshot repo’s commit history is a durable, off-cluster record of when the cluster was up. Each commit is a heartbeat. As of 2026-07-05 this is the only uptime memory devata has that survives a reboot.

The safety model

Read this before any YAML, because every implementation choice below follows from it.

  1. Read-only on the cluster side. The ServiceAccount can get and list exactly what the document publishes: nodes, namespaces, services, persistentvolumeclaims, pods, the apps workload kinds (deployments, daemonsets, statefulsets), and Argo Applications. The last four exist for the schema 1.1.0 fields the consumer renders (apps and workloads, counts and Argo app names only). No secrets, no writes, no watch. The full reasoning is in rbac.
  2. Allowlist, never denylist. The public document is built by selecting safe fields into a new object. It is never built by copying a Kubernetes object and deleting the unsafe parts, because a field Kubernetes adds next year would then leak by default. Internal IPs, real node names, and namespaces beyond a curated list are never selected in the first place.
  3. The schema is the gate. The job validates its own output against snapshot.schema.v1.json and refuses to push on failure. The schema sets additionalProperties: false everywhere, so an accidental extra field fails the run instead of shipping.
  4. One secret, tightly scoped. Reading the cluster needs no credential (the in-cluster token does that), but pushing needs one: a GitHub deploy key that can write to exactly one repo, committed as a sealedsecret.
  5. Graceful staleness. A powered-off cluster misses runs and the last good snapshot stays in place. The consumer compares generatedAt against freshness.maxAgeHours and labels old data as old. Never overwrite the published file with an empty or partial document; failing the run is always better than pushing garbage.

Step 0: the snapshot repo

Create a new public repo PragalvaXFREZ/devata-snapshot with a README saying what it is (machine-published, one JSON file, do not edit by hand). Public, because the whole point is that the portfolio page and anyone else can fetch raw.githubusercontent.com/PragalvaXFREZ/devata-snapshot/main/snapshot.json without auth. A dedicated repo keeps the machine token’s blast radius to this one file and keeps thousands of heartbeat commits out of any repo humans work in.

Step 1: the push credential

As built, the credential is a deploy key, not the fine-grained token the first draft of this note specified. The blast radius is identical (write access to the contents of one public data repo and nothing else), but a deploy key does not expire, so the publisher cannot silently die into a permanently stale page when a calendar entry is missed, and it can be minted and registered from the terminal in one sitting:

ssh-keygen -t ed25519 -N "" -C "devata-snapshot-publisher" -f deploykey
gh api repos/PragalvaXFREZ/devata-snapshot/keys \
  -f title="devata snapshot publisher (in-cluster CronJob)" \
  -f key="$(cat deploykey.pub)" -F read_only=false

The private half goes into the cluster in step 2 and nowhere else; once sealed, delete the local files. If the key ever leaks, revoke it on the repo’s deploy key page and mint a new one. The fine-grained token remains a valid alternative if per-credential expiry is ever wanted; only the push block of the script changes.

SSH also means the job must trust GitHub’s host identity. Pin it instead of trusting on first use: gh api meta --jq '.ssh_keys[]' returns GitHub’s published host keys, and prefixing each line with github.com produces a known_hosts file that ships in the scripts ConfigMap.

Step 2: seal the key

Strict scope (see sealedsecret) bakes the name and namespace into the ciphertext, so decide them now: Secret snapshot-push-credential, namespace showcase. From the lab repo root, using the committed controller cert so this works offline:

mkdir -p kubernetes/apps/showcase/snapshot-publisher
kubectl create secret generic snapshot-push-credential \
  --namespace showcase \
  --from-file=ssh-privatekey=deploykey \
  --dry-run=client -o yaml \
| kubeseal --cert kubernetes/infra/controllers/sealed-secrets/pub-cert.pem --format yaml \
  > kubernetes/apps/showcase/snapshot-publisher/sealedsecret-snapshot-push-credential.yaml

--dry-run=client means the plaintext Secret is never applied anywhere; it exists for one pipe. What lands in the file is ciphertext only. The sealing works with the namespace not yet existing, because sealing is offline math against the controller’s public key.

Step 3: the manifests

Everything lives in kubernetes/apps/showcase/snapshot-publisher/ in the lab repo (plus a README per repo convention). One trap to know before filing anything: an Argo directory source applies every .yaml and .json file in the path, so the schema must never sit in this directory as a loose .json file or Argo will try to apply it as a manifest. It exists only embedded inside a ConfigMap. Subdirectories are ignored (directory sources do not recurse by default), which is why the image build files can live in an image/ subfolder untouched.

serviceaccount.yaml, rbac.yaml: exactly the objects worked through in serviceaccount and rbac (ServiceAccount snapshot-publisher in showcase, ClusterRole snapshot-reader, ClusterRoleBinding snapshot-publisher-can-read).

configmap-schema.yaml: a ConfigMap snapshot-schema-v1 whose single data key snapshot.schema.v1.json holds the schema verbatim. The schema was drafted in the planning bundle (publisher-sketch sits next to it); the copy in the lab repo is the canonical one from now on, because it is the one the running job enforces.

configmap-scripts.yaml: a ConfigMap snapshot-publisher-scripts with two data keys: publish.sh, the script from step 5 verbatim, and known_hosts, GitHub’s pinned SSH host keys from step 1. The ConfigMap in git is the script’s single source of truth; do not keep a second loose copy anywhere.

cronjob.yaml:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: snapshot-publisher
  namespace: showcase
spec:
  schedule: "0 * * * *"
  suspend: true
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 2
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 600
      template:
        spec:
          serviceAccountName: snapshot-publisher
          restartPolicy: Never
          containers:
            - name: publish
              image: ghcr.io/pragalvaxfrez/snapshot-publisher:0.1.0
              command: ["/bin/sh", "/scripts/publish.sh"]
              resources:
                requests: { cpu: 50m, memory: 64Mi }
                limits: { memory: 256Mi }
              securityContext:
                allowPrivilegeEscalation: false
                capabilities: { drop: ["ALL"] }
                seccompProfile: { type: RuntimeDefault }
              volumeMounts:
                - name: scripts
                  mountPath: /scripts
                - name: schema
                  mountPath: /schema
                - name: credentials
                  mountPath: /credentials
          volumes:
            - name: scripts
              configMap: { name: snapshot-publisher-scripts }
            - name: schema
              configMap: { name: snapshot-schema-v1 }
            - name: credentials
              secret:
                secretName: snapshot-push-credential
                defaultMode: 0400

Every non-default line is a decision from the concept notes: hourly because each commit is one cell of the uptime ledger the portfolio renders (the cadence ruling of 2026-07-03, superseding this note’s original half-hour schedule), suspend: true at birth so the schedule cannot race the eyeball step after the first manual sync (it flips off in the follow-up PR that also turns on automation), Forbid because two publishers must not race a git push, startingDeadlineSeconds: 600 because without it a cluster off for two days wedges the CronJob permanently (the sharp edge in cronjob), backoffLimit: 1 because the next hourly tick is the retry, activeDeadlineSeconds: 600 so a hung push cannot block every later run behind Forbid, restartPolicy: Never so failed pods keep their logs. The deploy key mounts as a volume with mode 0400 rather than an environment variable, because ssh refuses group-readable keys and files never show up in kubectl describe pod. The sketch’s optional R2 push target is dropped entirely; one push path, git.

One honest wrinkle the first run surfaced: the pod runs as root (Alpine’s default user) and Kubernetes warns it would violate the restricted PodSecurity profile. The warning is advisory (devata enforces baseline), and a read-only job whose worst failure is a missing commit does not justify blocking the lap on it, but runAsNonRoot is the recorded hardening item for the next image version.

Step 4: the image

The job needs kubectl, jq, git, ssh, and check-jsonschema in one small image, defined at kubernetes/apps/showcase/snapshot-publisher/image/Dockerfile:

FROM alpine:3.21
 
LABEL org.opencontainers.image.source="https://github.com/PragalvaXFREZ/lab"
 
RUN apk add --no-cache git jq curl openssh-client python3 py3-pip \
 && pip install --break-system-packages --no-cache-dir check-jsonschema \
 && curl -fsSLo /usr/local/bin/kubectl \
      "https://dl.k8s.io/release/v1.34.1/bin/linux/amd64/kubectl" \
 && chmod +x /usr/local/bin/kubectl

Pin the kubectl version to the cluster’s minor (devata runs v1.34.1). As built, this note’s original plan of a local docker build plus a classic PAT with write:packages was skipped entirely: the workflow .github/workflows/publisher-image.yaml in the lab repo builds and pushes ghcr.io/pragalvaxfrez/snapshot-publisher:0.1.0 whenever the image directory changes on main, authenticating with the workflow’s own GITHUB_TOKEN and permissions: packages: write. No long-lived registry credential exists anywhere.

The expected trap did not fire: a first-push ghcr package is documented as private by default, but with the org.opencontainers.image.source label pointing at the public lab repo and the push coming from that repo’s own workflow token, the package came up anonymously pullable at once. Verify rather than assume, in either direction: request an anonymous token from ghcr.io/token?scope=repository:pragalvaxfrez/snapshot-publisher:pull and fetch the manifest with it before concluding the visibility needs a manual flip.

Step 5: the publish script

Shell and jq, deliberately, so every transform can be read line by line. Notice the shape of the jq program: it constructs a new object from nothing (jq -n) and selects fields into it. Nothing is copied and redacted; that is the allowlist rule made mechanical. Node identities become positional aliases (cp-1, worker-2), IPs and real hostnames are simply never read out of the source objects.

The canonical, running copy lives in the lab repo at kubernetes/apps/showcase/snapshot-publisher/configmap-scripts.yaml; this listing documents its shape and the decisions in it.

#!/bin/sh
set -eu
 
# 1. Read state, read-only. The in-cluster ServiceAccount token authenticates automatically.
kubectl get nodes -o json > /tmp/nodes.json
kubectl get namespaces -o json > /tmp/ns.json
kubectl get services -A -o json > /tmp/svc.json
kubectl get persistentvolumeclaims -A -o json > /tmp/pvc.json
kubectl get applications.argoproj.io -n argocd -o json > /tmp/apps.json
kubectl get deployments -A -o json > /tmp/deployments.json
kubectl get daemonsets -A -o json > /tmp/daemonsets.json
kubectl get statefulsets -A -o json > /tmp/statefulsets.json
kubectl get pods -A -o json > /tmp/pods.json
 
NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
 
# 2. Build the public document by selecting safe fields into a new object (allowlist).
jq -n \
  --arg now "$NOW" \
  --slurpfile nodes /tmp/nodes.json \
  --slurpfile ns /tmp/ns.json \
  --slurpfile svc /tmp/svc.json \
  --slurpfile pvc /tmp/pvc.json \
  --slurpfile apps /tmp/apps.json \
  --slurpfile deployments /tmp/deployments.json \
  --slurpfile daemonsets /tmp/daemonsets.json \
  --slurpfile statefulsets /tmp/statefulsets.json \
  --slurpfile pods /tmp/pods.json '
  def iscp($n): $n.metadata.labels | has("node-role.kubernetes.io/control-plane");
  def ready($n): ([$n.status.conditions[] | select(.type == "Ready") | .status] | first) == "True";
  def gib($q): ($q | rtrimstr("Ki") | tonumber / 1048576);
  def nodebody($n): {
    role: (if iscp($n) then "control-plane" else "worker" end),
    status: (if ready($n) then "Ready" else "NotReady" end),
    vcpu: ($n.status.capacity.cpu | tonumber),
    memoryGiB: ((gib($n.status.capacity.memory) * 10) | round / 10),
    ageDays: (((now - ($n.metadata.creationTimestamp | fromdateiso8601)) / 86400) | floor)
  };
  ($nodes[0].items) as $N |
  ($svc[0].items) as $S |
  ($pvc[0].items) as $P |
  {
    schemaVersion: "1.1.0",
    generatedAt: $now,
    generator: {
      name: "devata-snapshot-publisher",
      version: "0.1.0",
      method: "read-only Kubernetes API via a scoped ServiceAccount"
    },
    freshness: { clusterPowered: true, maxAgeHours: 24 },
    cluster: {
      name: "devata",
      distribution: "Talos Linux",
      osVersion: ($N[0].status.nodeInfo.osImage | capture("\\((?<v>v[^)]+)\\)").v),
      kubernetesVersion: ($N[0].status.nodeInfo.kubeletVersion),
      cni: { name: "Cilium", notes: "eBPF dataplane, kube-proxy replacement, Hubble observability" },
      loadBalancer: "MetalLB (L2)",
      gitops: "Argo CD app-of-apps; zero imperative Helm state",
      ingress: "none yet; Gateway API planned",
      secretsManagement: "Sealed Secrets",
      publicExposure: "none; behind CGNAT, outbound-only"
    },
    totals: {
      nodes: ($N | length),
      nodesReady: ([$N[] | select(ready(.))] | length),
      vcpu: ([$N[].status.capacity.cpu | tonumber] | add),
      memoryGiB: ((([$N[] | gib(.status.capacity.memory)] | add) * 10) | round / 10),
      namespaces: ($ns[0].items | length),
      exposedServices: ([$S[] | select(.spec.type == "LoadBalancer")] | length),
      persistentVolumeClaims: ($P | length)
    },
    nodes: (
      ([ $N[] | select(iscp(.)) ]       | to_entries | map({id: "cp-\(.key + 1)"}     + nodebody(.value))) +
      ([ $N[] | select(iscp(.) | not) ] | to_entries | map({id: "worker-\(.key + 1)"} + nodebody(.value)))
    ),
    services: ([
      { name: "Grafana",   match: "grafana",       purpose: "dashboards over cluster metrics" },
      { name: "Hubble UI", match: "hubble-ui",     purpose: "eBPF network flow observability" },
      { name: "Argo CD",   match: "argocd-server", purpose: "GitOps reconciler UI" }
    ] | map(. as $item | {
      name: $item.name,
      exposure: "lan-only",
      status: (if ([$S[] | select(.metadata.name | test($item.match))] | length) > 0 then "up" else "unknown" end),
      purpose: $item.purpose
    })),
    observability: [
      { name: "kube-prometheus-stack", purpose: "metrics, alerting, Grafana dashboards" },
      { name: "Loki",     purpose: "log aggregation" },
      { name: "Promtail", purpose: "log shipping from every node" },
      { name: "Hubble",   purpose: "eBPF network flow visibility" }
    ],
    storage: {
      defaultClass: "local-path",
      replicated: false,
      claims: ($P | length),
      totalGiB: ([$P[].spec.resources.requests.storage | rtrimstr("Gi") | tonumber] | add // 0 | floor)
    },
    gpu: {
      present: true,
      usable: false,
      model: "GTX 1650 Ti (4 GB)",
      reason: "card firmware fault, driver cannot read the VBIOS; nvidia.com/gpu never advertised"
    },
    apps: ($apps[0].items | map({
      name: .metadata.name,
      synced: (.status.sync.status == "Synced"),
      healthy: (.status.health.status == "Healthy")
    }) | sort_by(.name)),
    workloads: {
      deployments: ($deployments[0].items | length),
      daemonsets: ($daemonsets[0].items | length),
      statefulsets: ($statefulsets[0].items | length),
      pods: ($pods[0].items | length)
    }
  }' > /tmp/snapshot.json
 
# 3. The gate. check-jsonschema exits non-zero on any violation and set -eu stops the run,
#    so a document that does not match the contract is never pushed.
check-jsonschema --schemafile /schema/snapshot.schema.v1.json /tmp/snapshot.json
 
# 4. Push. Every run commits, even when nothing changed but the timestamp:
#    the commit history doubles as the cluster's off-cluster uptime record.
export GIT_SSH_COMMAND="ssh -i /credentials/ssh-privatekey -o UserKnownHostsFile=/scripts/known_hosts -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes"
git clone --depth 1 git@github.com:PragalvaXFREZ/devata-snapshot.git /tmp/repo
cp /tmp/snapshot.json /tmp/repo/snapshot.json
cd /tmp/repo
git add snapshot.json
git -c user.name="devata snapshot publisher" -c user.email="publisher@devata.local" commit -m "snapshot ${NOW}"
git push origin HEAD:main

The deviations from publisher-sketch, each on purpose. The commit-on-change logic is gone: generatedAt changes every run anyway, and committing every run is what makes the history an uptime ledger. The hardcoded prose values are updated to the truth (gitops is now Argo CD, secretsManagement is Sealed Secrets); the schema’s own description of cluster.gitops says the value evolves with the roadmap, and this is that evolution. osVersion extracts the bare version with capture(), because the schema documents v1.11.5 while the node object’s osImage says Talos (v1.11.5); the consumer stub work surfaced this and the fix landed here before the CronJob was built. The document is schemaVersion 1.1.0: apps and workloads are rendered by the same select-only pattern (Argo app names are already public in the lab repo; workload numbers are counts, never names). The push is SSH with the deploy key and pinned host keys, so no token ever appears in a URL or an environment variable. And the storage.totalGiB sum assumes every PVC requests in Gi, which is true on devata today (5, 2, 5); if a Mi-sized claim ever appears, this line is the one to fix.

Step 6: the Application, and the drill you already know

kubernetes/clusters/devata/snapshot-publisher.yaml, watched by the root app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: snapshot-publisher
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/PragalvaXFREZ/lab.git
    targetRevision: main
    path: kubernetes/apps/showcase/snapshot-publisher
  destination:
    server: https://kubernetes.default.svc
    namespace: showcase
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

Automation deliberately off at first, exactly like every adoption in adopting-a-helm-component: PR the manifests, let the root app create the child, sync manually, verify, and only then flip automated: {prune: true, selfHeal: true} together with the CronJob’s suspend flag in a follow-up PR. This app is the easiest one yet to trust: it is new (nothing to adopt), read-only against the cluster, and its worst failure mode is a missing commit in a data repo.

Verification, including breaking it on purpose

  1. After the manual sync: kubectl get cronjob,sa,clusterrole -n showcase and argocd app get snapshot-publisher showing Synced/Healthy.
  2. Fire one without waiting for the clock, and watch it end to end:
kubectl create job snap-manual --from=cronjob/snapshot-publisher -n showcase
kubectl logs -n showcase job/snap-manual -f
  1. Fetch what the world will see and eyeball it against the safety model: no IPs, no real node names, nothing credential-shaped.
curl -s https://raw.githubusercontent.com/PragalvaXFREZ/devata-snapshot/main/snapshot.json | jq .
curl -s https://raw.githubusercontent.com/PragalvaXFREZ/devata-snapshot/main/snapshot.json | grep -E '192\.168|talos-' || echo "clean"
  1. Prove the gate actually rejects. Take the real document, inject one stray field, and watch the validator refuse (this is lab issue #12’s done-when, and it is the difference between having a gate and believing you have one):
kubectl get cm snapshot-schema-v1 -n showcase -o jsonpath='{.data.snapshot\.schema\.v1\.json}' > /tmp/schema.json
curl -s https://raw.githubusercontent.com/PragalvaXFREZ/devata-snapshot/main/snapshot.json \
  | jq '. + {internalIp: "192.168.1.8"}' > /tmp/broken.json
check-jsonschema --schemafile /tmp/schema.json /tmp/broken.json   # must fail on additionalProperties
  1. Flip automation on, drift-test if you want the rehearsal (delete the CronJob, watch Argo put it back), and let the schedule run. After a day, git log --oneline in devata-snapshot reads as a heartbeat log of the cluster.

What consumes this

The portfolio’s /homelab page (hub issue #32) fetches the raw file at load, renders the data, and honors the freshness contract: past maxAgeHours it shows the snapshot visibly marked stale instead of presenting a powered-off cluster as live. With the hourly cadence, a visitor of a powered-on cluster sees data at most an hour old, and the page’s uptime ledger lights one cell per heartbeat commit. The page never talks to the cluster, the cluster never accepts a connection, and everything a stranger can reach is one static JSON file whose every field passed an allowlist and a schema.