Streaming Aggregate Hubble Traffic Without Exposing devata

Document contract

• Role: dated architecture and troubleshooting case study
• Scope: the complete path from aggregate Hubble metrics inside devata to the public traffic chart, including delivery, trust boundaries, failures, recovery, and rollback; raw flow export and general-purpose telemetry platforms are deliberately excluded
• Truth boundary: merged source and runtime evidence collected during the issue #57 rollout
• Last verified: 2026-08-01 at lab revision 5284064 and platform-hub revision fd436bd

Prerequisites: prometheus, cilium, sealedsecret, publishing-the-cluster-snapshot, gitops

The public homelab page already had a safe traffic chart. An hourly publisher queried aggregate Hubble metrics, wrote six hours of fifteen-minute samples to snapshot.json, and pushed that allowlisted document out of the private cluster. The snapshot was durable and honest during an outage, but it could never make the chart feel live.

The unsafe shortcut would have been to expose Hubble Relay or Prometheus through an inbound tunnel. Both interfaces reveal far more than the chart needs. Individual flows can carry workload identities, addresses, ports, and service relationships. The actual requirement was much smaller: publish only the total flow rate and total drop rate every five seconds.

lab issue #57 therefore became an exercise in narrowing interfaces. The finished design sends one closed document outward, retains it outside the home network, lets browsers read but never write, and keeps the existing snapshot as a separate fallback.

Case file

Mental model

There are two outward data paths. The live path optimizes for freshness. The snapshot path optimizes for durable last-known state. They meet only in the browser.

flowchart LR
  subgraph Home[Private home network]
    P[Prometheus aggregate series]
    G[Go producer]
    S[Snapshot publisher]
    P -->|fixed PromQL| G
    P -->|15 minute aggregates| S
  end

  subgraph Edge[Cloudflare edge]
    W[Worker authentication and validation]
    D[(Durable Object SQLite)]
    W --> D
    D -->|six-hour history| W
    D -->|new sample| W
  end

  subgraph Public[Public read side]
    R[(devata-snapshot repository)]
    B[pragalva.me traffic chart]
  end

  G -->|outbound HTTPS every 5 seconds| W
  S -->|outbound Git push hourly| R
  W -->|GET history and WebSocket| B
  R -. hourly fallback .-> B

No arrow points from the public network into devata. Prometheus has no public route. The producer has no Kubernetes Service, listening port, or mounted service-account token. Its only job is to transform one fixed query result into one smaller document and send it outward.

The contract is smaller than the source

The producer does not forward a Prometheus response. It constructs a new value:

{
  "timestamp": "2026-07-31T10:12:05Z",
  "flowsPerSecond": 146.4,
  "dropsPerSecond": 0.048
}

Those three fields are the entire public contract. Pod names, namespaces, services, labels, IP addresses, ports, nodes, and individual Hubble flows are prohibited.

The fixed query in image/main.go sums the five-minute rate of hubble_flows_processed_total and hubble_drop_total. label_replace marks the two results as flows and drops. The decoder rejects unknown Prometheus response fields, unexpected result kinds, duplicate values, missing values, negative numbers, NaN, and infinity. Rates are rounded to three decimal places before publication.

The producer also fails visibly. Six consecutive collection or delivery failures terminate the process. Kubernetes then exposes a restart instead of leaving a nominally healthy process that only writes errors to its log.

What code was written

The Go module declares go 1.23. That line is the minimum language and module-semantics baseline, not a claim that the deployed container carries a Go 1.23 runtime. The binary is statically compiled in the build stage, and the final distroless image contains no Go toolchain. The delivery workflow and builder used Go 1.26.5 during this rollout.

The TypeScript code is also not running inside the homelab. It targets the Cloudflare Workers runtime. It remains beside the producer in lab because payload validation, authentication, deployment order, and tests form one feature contract. hubble-traffic-relay-repository-boundary records the conditions that should eventually split it into a separate repository.

Relay enforcement

The Worker narrows the public write path before the Durable Object sees it:

1. compare the bearer credential through SHA-256 digests;
2. require application/json and a body no larger than 512 bytes;
3. require exactly the three public keys;
4. require second-precision UTC timestamps no more than ten seconds in the future and no more than thirty seconds old;
5. reject rates outside the finite non-negative range;
6. reject a timestamp that is not newer than the stored latest row;
7. reject writes less than three seconds after the previous accepted write;
8. delete rows older than six hours;
9. broadcast only the reconstructed public sample.

The history endpoint is read-only and returns { windowSeconds: 21600, samples: [...] }. The WebSocket accepts only the portfolio origin in browsers, sends the latest sample in a ready message, and then sends each accepted sample. Client messages other than ping close the socket as read-only misuse.

The browser repeats the schema check. It does not trust data merely because it came from the relay. Samples are deduplicated by timestamp, sorted, capped to the six-hour maximum, and discarded when too old or implausibly far in the future.

The state machine does not invent continuity

The portfolio derives status from two facts: whether the WebSocket is open and how old the latest relay sample is.

stateDiagram-v2
  [*] --> reconnecting
  reconnecting --> live: socket open and sample age at most 12 seconds
  live --> reconnecting: socket closes or sample age exceeds 12 seconds
  reconnecting --> stale: latest sample age exceeds 30 seconds
  stale --> live: reconnect succeeds and a fresh sample arrives

While reconnecting, the last relay data remains visible. Once stale, the hook switches to the latest valid hourly snapshot. Snapshot samples are normalized relative to their own publication window, not the browser wall clock, so an old but valid snapshot does not disappear simply because the relay has been down for hours.

This is not uptime monitoring. live means that this browser has a fresh aggregate sample. stale means the stream cannot currently prove freshness. Neither state claims that every workload in devata is healthy.

The release sequence

The rollout was deliberately staged because each boundary required a different kind of proof.

This order prevented a broken relay or producer from becoming a broken public feature. The hourly snapshot remained the only traffic source until the live contract had passed its own runtime checks.

Failure one: the Worker uploaded, but the route did not attach

The first main deployment did not fail while compiling or uploading the Worker. Wrangler reported that devata-hubble-traffic-relay had uploaded, then its request to the zone Workers Routes API failed with Cloudflare error 10000, an authentication error.

The API token had account-level Workers Scripts edit permission. That was sufficient to upload the Worker. Attaching telemetry.pragalva.me was a different operation against the pragalva.me zone, and it required zone-scoped Workers Routes edit permission.

The correction was to add a separate policy for the pragalva.me zone with Workers Routes edit, then rerun workflow attempt 2. The Worker deployment, secret installation, and health probe then passed.

Why it appeared only after merge

The pull-request workflow ran verify-producer and verify-relay. The image publish and Worker deploy jobs were intentionally conditional on a push to main or a manual dispatch. A pull request could therefore prove TypeScript behavior and Wrangler’s build configuration without mutating a real Cloudflare zone.

The credential policy also lived outside GitHub source. Static validation could not infer whether that external token authorized the exact zone route operation. The first post-merge deployment was the first test of that permission boundary.

There was a second credential lesson during setup. The Cloudflare API token was briefly configured as a plaintext GitHub Actions variable rather than an encrypted secret. It was treated as exposed, rotated, and stored correctly. The account ID is not sensitive, but the API token and producer token must always use the encrypted secrets interface. No token value belongs in Git or this case study.

Failure two: DNS packets were allowed, but the Prometheus name was not

After the relay passed, the manually synced producer reached CoreDNS but could not resolve the Prometheus Service. The lookup returned a DNS server failure. Each collection failed, and the producer exited after six consecutive failures. Kubernetes restarted it, making the fault visible.

The Cilium policy looked reasonable at a packet-level glance:

• TCP and UDP port 53 were allowed to the CoreDNS endpoints;
• Prometheus TCP 9090 was allowed to the monitoring endpoint;
• telemetry.pragalva.me was allowed through DNS and HTTPS rules.

The missing detail was the L7 DNS rule. Once rules.dns is present, Cilium’s DNS proxy applies the listed names as an allowlist. The policy listed only telemetry.pragalva.me. DNS traffic could reach CoreDNS, but the query for Prometheus was not authorized.

PR #63 made the dependency exact in both places:

- name: PROMETHEUS_URL
  value: http://kps-kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/api/v1/query
 
rules:
  dns:
    - matchName: kps-kube-prometheus-stack-prometheus.monitoring.svc.cluster.local
    - matchName: telemetry.pragalva.me

Why it appeared only after merge

The child Argo CD Application intentionally had no automated first sync. No producer pod existed during the pull request. Server-side dry-run and kubeconform proved that the Deployment and CiliumNetworkPolicy were accepted object shapes. They did not perform a DNS lookup from a selected endpoint through Cilium’s DNS proxy.

This was a semantic policy error, not a YAML error. Every object could be valid while the actual query name remained forbidden. The failure surfaced only when the merged Application was synced and a real pod used the real resolver under the real policy.

The producer’s restart was not the root cause. It was the current failure loop created by the deliberate six-failure threshold. The root cause was the omitted DNS name.

What each validation layer actually proved

The lesson is not that CI failed. Each layer answered the question it was built to answer. The mistake would have been to treat those answers as broader proof.

Production proof

The final verification crossed every boundary rather than stopping at a healthy component.

The disconnect test did not stop the producer or relay. A local CONNECT proxy cut only one browser’s tunnel, so production traffic remained untouched. That isolated the consumer state machine from backend availability changes.

In devata

The authoritative Kubernetes source is kubernetes/apps/showcase/hubble-traffic-stream/. The hubble-traffic-stream Argo CD Application points at that directory and deploys into showcase.

The pod runs as UID and GID 65532 with a read-only root filesystem, RuntimeDefault seccomp, no Linux capabilities, no privilege escalation, no Kubernetes API token, and a 64 MiB memory limit. strategy: Recreate prevents two producers from writing simultaneously during a rollout. The credential comes from the hubble-traffic-producer Secret generated by a SealedSecret.

The Cilium policy declares ingress: []. Egress is limited to Prometheus TCP 9090, CoreDNS TCP and UDP 53 for the two exact names, and relay TCP 443. The producer directory contains no Service, Ingress, HTTPRoute, or Cloudflare Tunnel route.

Try it safely

• Objective: prove freshness and the public data boundary without changing the cluster.
• Environment: lab repository, the devata kubectl context, jq, and public network access.
• Safety: read-only. Do not send a valid producer credential and do not disable the live workload. Rollback is unnecessary because these commands do not mutate state.

Predict which layer each command proves before running it.

kubectl -n argocd get application hubble-traffic-stream \
  -o jsonpath='{.status.sync.status} {.status.health.status} {.status.sync.revision}{"\n"}'
 
kubectl -n showcase get pod \
  -l app.kubernetes.io/name=hubble-traffic-streamer
 
kubectl -n showcase logs deployment/hubble-traffic-streamer --tail=8
 
curl -fsS -H 'Origin: https://pragalva.me' \
  https://telemetry.pragalva.me/v1/history \
  | jq '{windowSeconds, count: (.samples | length), latest: .samples[-1], keys: (.samples[-1] | keys)}'
 
curl -sS -o /dev/null -w '%{http_code}\n' \
  -X POST https://telemetry.pragalva.me/v1/samples \
  -H 'content-type: application/json' \
  --data '{}'

Expected observation: Argo is Synced/Healthy, one producer is Ready without a restart loop, logs advance in five-second steps, history contains only the three contract keys, and the unauthenticated write returns 401. None of these commands alone proves the browser fallback state.

Reconstruct or recover

The required order is part of the design:

1. recover the Sealed Secrets controller key and reconcile the producer SealedSecret;
2. configure encrypted GitHub secrets for the Cloudflare account ID, API token, and producer token;
3. give the Cloudflare token account Workers Scripts edit and zone Workers Routes edit for pragalva.me;
4. run the delivery workflow to deploy the Worker, Durable Object, custom domain, and Worker secret;
5. prove /health, unauthorized write rejection, and the history contract;
6. reconcile the Argo CD Application;
7. prove Prometheus DNS, TCP reachability, five-second producer logs, and zero restarts;
8. deploy the platform consumer;
9. prove browser live, disconnect, stale fallback, and recovery behavior.

For rollback, remove the consumer’s live hook first so the public page uses only snapshot.json. Then remove the child Argo CD Application and let Argo prune the producer. Delete the Worker only after the producer is gone. The hourly snapshot path remains independent throughout rollback.

What to focus on next

1. Test deployed permissions as interfaces. A Cloudflare token is part of the release contract. Add a safe preflight that confirms the account and zone scopes needed by Wrangler before the first mutable deployment.
2. Test policy semantics, not only policy schemas. A disposable pod selected by the same Cilium policy should resolve the exact Prometheus and relay names, reach the allowed ports, and fail a deliberately forbidden destination.
3. Make image updates immutable. The current producer uses the fixed 0.1.0 tag. Future code changes should publish a unique tag or digest and update Git so reconciliation cannot reuse a stale node-cached image.
4. Keep secret handling boring. API tokens belong in encrypted secrets, never variables, screenshots, logs, or shell history. Rotation should be documented and practiced as a two-sided relay-then-producer change.
5. Preserve the two read models. WebSocket freshness and hourly snapshot durability solve different failures. Do not merge them into one ambiguous health claim.
6. Keep the public contract closed. Any new field needs an explicit privacy review and matching producer, relay, consumer, and test changes.
7. Decide the steady-state Argo policy explicitly. The Application remained manual for the first deployment gate. Either enable automated prune and self-heal in a dedicated reviewed change or document manual reconciliation as the continuing operating policy.

Check yourself

1. Why did allowing UDP and TCP 53 to CoreDNS still produce a DNS failure for Prometheus?
2. Which evidence would distinguish a broken producer from a healthy producer whose browser consumer is disconnected?
3. Why does a successful pull-request Worker test not prove that a production API token can attach a custom domain?
4. If the producer stops but the WebSocket remains open, why must sample age participate in the browser state instead of socket state alone?
5. Which fields would you inspect to disprove the claim that no workload identity leaves devata?

References

• lab issue #57
• lab PR #61
• lab PR #63
• platform-hub PR #64
• Main deployment workflow, failed attempt
• Main deployment workflow, successful rerun
• Cloudflare Durable Objects documentation
• Cilium DNS policy documentation
• Kubernetes NetworkPolicy documentation