Trace LAN TLS through Cilium Gateway API

Document contract

• Role: lab
• Scope: Gateway API request routing, DNS-01 certificate issuance, GitOps ownership, monitoring, and rollback; NetworkPolicy and certificate rotation drills are left for separate sessions
• Truth boundary: declared state at lab revision 1d97807 plus read-only runtime evidence observed on 2026-07-26
• Last verified: devata, 2026-07-26

Prerequisites: kubernetes, service, gitops, cilium, metallb, sealedsecret, service-monitor

This session answers one working question:

What exact chain turns https://grafana.lab.pragalva.me into a valid response from the Grafana workload, and what evidence proves each controller did its part?

Continue into the public and policy path

After this LAN foundation, tracing-cloudflared-through-cilium-gateway-policy follows the Cloudflare Tunnel hairpin path and reconstructs the Cilium identity-policy incident that returned Envoy 403 before Grafana.

The goal is not to memorize YAML. The goal is to separate declared intent, controller work, runtime objects, and end-to-end proof. The session is read-only and takes about one hour. Each pass has a stop point, so an interruption does not force a restart.

Mental model

Two control paths meet at the HTTPS listener.

flowchart LR
  client[LAN client] -->|DNS A lookup| vip[192.168.1.244]
  vip -->|MetalLB advertises Service IP| gateway[Cilium Gateway Service]
  gateway --> envoy[Cilium Envoy]
  envoy -->|HTTPRoute hostname and path| service[Grafana Service]
  service --> endpoints[EndpointSlice]
  endpoints --> pod[Grafana Pod]

flowchart LR
  sealed[SealedSecret in Git] --> sealer[Sealed Secrets controller]
  sealer --> token[Cloudflare token Secret]
  certificate[Certificate] --> certmanager[cert-manager]
  token --> certmanager
  certmanager -->|temporary TXT record| cloudflare[Cloudflare DNS]
  cloudflare --> letsencrypt[Let's Encrypt validation]
  letsencrypt --> certmanager
  certmanager --> tlssecret[TLS Secret]
  tlssecret --> listener[Gateway HTTPS listener]

The request path moves traffic. The certificate path proves control of the DNS name and supplies the key pair. DNS-01 does not route application traffic, and MetalLB does not terminate TLS.

Actors and contracts

Pass 1: reconstruct the declared state

Time box: 12 minutes
Objective: predict creation order and ownership before looking at runtime.
Environment: /home/pragalva/Desktop/projects/lab, branch main.
Safety: read-only.

Start by proving the repository and cluster context.

cd /home/pragalva/Desktop/projects/lab
git status --short --branch
git log -1 --oneline
kubectl config current-context

Expected repository revision for this dated session: 1d97807. Expected context: pragalva@devata. If either differs, treat the rest of the page as a model and inspect the current files before trusting exact values.

Before opening the files, predict which must exist first: Gateway API CRDs, cert-manager, the Certificate, or the Gateway.

sed -n '1,160p' kubernetes/clusters/devata/gateway-api-crds.yaml
sed -n '1,180p' kubernetes/clusters/devata/cert-manager.yaml
sed -n '1,160p' kubernetes/clusters/devata/lan-gateway.yaml
sed -n '1,180p' kubernetes/infra/networking/gateway/gateway.yaml
sed -n '1,220p' kubernetes/infra/networking/gateway/routes.yaml

Read the sync waves as dependency hints:

1. Gateway API CRDs at wave -2 make the resource kinds available.
2. cert-manager at wave -1 installs its CRDs and controllers.
3. the LAN Gateway application at wave 1 declares the certificate, listeners, routes, and grants.

An important limit: these waves order the parent Application objects. They do not guarantee every controller webhook inside a child application is ready before another child starts. The initial certificate sync exposed that distinction when the cert-manager webhook was not yet reachable.

Stop and recall

Without looking back, name the declared input, applying controller, and output object for each of the three waves. If you cannot, repeat only this pass.

Resume marker: start Pass 2 with the Git files closed.

Pass 2: trace one request from name to Pod

Time box: 15 minutes
Objective: prove every routing boundary for Grafana.
Safety: read-only.

Predict the IP before resolving the name.

dig +short grafana.lab.pragalva.me A
kubectl -n gateway-system get gateway lan-gateway
kubectl -n gateway-system get service cilium-gateway-lan-gateway

All three observations should converge on 192.168.1.244. They prove DNS, Gateway status, and the generated LoadBalancer Service agree. They do not yet prove the route or backend works.

Inspect the route conditions.

kubectl -n gateway-system get httproute grafana -o json \
  | jq '.status.parents[].conditions[] | {type, status, reason}'
 
kubectl -n gateway-system get gateway lan-gateway -o json \
  | jq '.status.listeners[] | {name, attachedRoutes, conditions}'

Accepted=True means the parent accepted the route. ResolvedRefs=True means referenced objects passed resolution and permission checks. Programmed=True means Cilium prepared the listener. None of these is an HTTP transaction.

Follow the backend reference.

kubectl -n monitoring get service kps-grafana
kubectl -n monitoring get endpointslice \
  -l kubernetes.io/service-name=kps-grafana -o wide
kubectl -n monitoring get referencegrant allow-grafana-route -o yaml

The HTTPRoute lives in gateway-system, while the Service lives in monitoring. The ReferenceGrant is deliberately stored in monitoring because the target namespace owns permission to be referenced.

Finish with the transaction.

curl -sS -o /dev/null \
  -w 'code=%{http_code} ip=%{remote_ip} tls=%{ssl_verify_result} redirect=%{redirect_url}\n' \
  http://grafana.lab.pragalva.me
 
curl -sS -o /dev/null \
  -w 'code=%{http_code} ip=%{remote_ip} tls=%{ssl_verify_result} redirect=%{redirect_url}\n' \
  https://grafana.lab.pragalva.me

Expected observations are HTTP 301 toward HTTPS, then HTTPS 302 toward Grafana login, remote IP .244, and TLS verification result 0.

Stop and recall

Draw the request path from memory. Put DNS, MetalLB, the generated Service, Envoy, HTTPRoute, backend Service, EndpointSlice, and Pod in order. Circle the first layer that an incorrect hostname would fail.

Resume marker: if the transaction works, start Pass 3. If not, stop at the first layer whose evidence disagrees with the preceding layer.

Pass 3: trace certificate issuance without reading secrets

Time box: 15 minutes
Objective: explain how a publicly trusted certificate can protect a private LAN address.
Safety: read-only. Do not print Secret data or private keys.

Predict whether Let’s Encrypt ever connects to 192.168.1.244 during DNS-01 validation.

kubectl get clusterissuer letsencrypt-production
kubectl -n gateway-system get certificate lan-services-tls
kubectl -n gateway-system get certificaterequest,order,challenge
kubectl -n cert-manager get sealedsecret cloudflare-api-token
kubectl -n gateway-system get secret lan-services-tls \
  -o custom-columns='NAME:.metadata.name,TYPE:.type,CREATED:.metadata.creationTimestamp'

Let’s Encrypt does not need LAN reachability. cert-manager places temporary TXT proof under the public DNS zone using the scoped Cloudflare token. Successful validation lets cert-manager write the certificate and private key into gateway-system/lan-services-tls. The Gateway listener references that Secret.

Inspect only the public certificate presented over the network.

echo | openssl s_client \
  -connect 192.168.1.244:443 \
  -servername grafana.lab.pragalva.me \
  -verify_return_error 2>/dev/null \
  | openssl x509 -noout -issuer -dates -ext subjectAltName

The SAN list should contain both LAN names, and the validity window should match the Certificate status. -servername supplies TLS SNI, allowing the listener to select a certificate for the requested hostname.

Now inspect the monitoring contract.

kubectl -n monitoring get prometheusrule cert-manager-certificate-health -o yaml
kubectl -n cert-manager get servicemonitor cert-manager -o json \
  | jq '.spec.endpoints[] | {port, honorLabels}'

The three alerts cover prolonged not-ready state, an overdue renewal timestamp, and expiration within 14 days. honorLabels: true preserves the certificate namespace exported by cert-manager instead of replacing it with the ServiceMonitor namespace.

Stop and recall

Explain why an A record pointing to a private address can still receive a public certificate. Then name the credential that must remain private, the object committed to Git, and the controller key needed to recover that ciphertext after cluster loss.

Resume marker: start Pass 4 only after you can explain DNS-01 without saying that Let’s Encrypt reaches the application.

Pass 4: diagnose the real rollout failures

Time box: 12 minutes
Objective: identify the ownership boundary behind each symptom.
Safety: reasoning only. Do not recreate the failures on the live cluster.

For each row, state which evidence remained green and which evidence exposed the problem. The reusable lesson is that configuration existence, controller process state, object status, and end-to-end behavior are different layers.

Stop and recall

Pick one failure and explain why restarting everything would have hidden the real cause. Name the narrowest observation that distinguished stale controller state from bad declarative intent.

Reconstruct and roll back

Reconstruction uses dependency order:

1. install Gateway API CRDs;
2. reconcile Sealed Secrets and cert-manager;
3. confirm the Cloudflare token Secret and ClusterIssuer become usable;
4. enable Cilium Gateway API and confirm its agents and operators restart;
5. reconcile the Certificate, Gateway, routes, and grants;
6. wait for the dedicated LoadBalancer address and certificate readiness;
7. create DNS-only A records;
8. prove redirects, TLS, routes, legacy access, monitoring, and cluster health.

Rollback reverses the consumer path without removing the safety path:

1. keep Grafana .242 and Hubble .243 available;
2. revert Gateway resources and Cilium Gateway API enablement through Git;
3. wait for Argo to reconcile the revert;
4. remove the two .244 A records;
5. remove cert-manager only after no remaining consumer references its Secrets.

The old LoadBalancer Services are intentional migration infrastructure, not duplicate cleanup debt.

Final explain-back

Close the manifests and answer aloud:

1. What is the complete Grafana request path from DNS to Pod?
2. Which controller creates the Gateway’s LoadBalancer Service, and which controller assigns its LAN IP?
3. Why must a ReferenceGrant live in the backend namespace?
4. How does DNS-01 issue a trusted certificate for a private address?
5. Why did changing the Cilium ConfigMap not activate Gateway API immediately?
6. What evidence proves TLS is valid, rather than merely proving the Certificate object is Ready?
7. Which old endpoints make rollback possible?

You have completed the session when you can draw both diagrams from memory, answer all seven questions causally, and point to one command that would disprove each answer.

References

• devata Gateway resources at the verified revision
• devata cert-manager resources at the verified revision
• Cilium Gateway API documentation
• Gateway API concepts
• cert-manager ACME DNS-01 documentation
• Sealed Secrets scope documentation