Tracing a Cloudflare Tunnel 403 Through Cilium Gateway API

Document contract

• Role: case study with a read-only lab
• Scope: Gateway API foundations, Cilium’s implementation, the LAN and Cloudflare Tunnel request paths, identity-aware policy, the failed and successful repairs, and reusable debugging practice; creating the tunnel, issuing the certificate, and editing Cloudflare dashboard state are outside this page
• Truth boundary: portable concepts plus the completed 2026-07-28 incident at lab revision bbc2473
• Last verified: devata, 2026-07-28, after lab PR #60 reconciled

Prerequisites: kubernetes, service, endpointslice, cilium, ebpf, metallb, gitops, reconciliation

grafana.pragalva.me returned 403 Access denied while grafana.lab.pragalva.me continued to work. The Cloudflare Access application was removed, both tunnel connectors were healthy, the Cilium Gateway was programmed, and Grafana itself was healthy. A first policy correction passed static validation, merged, reconciled, and still returned the same 403.

The incident only became understandable after the request was treated as a chain of ownership and identity transitions instead of one connection to one Service.

Keep one question visible while working through this chapter:

Which component made the 403 decision, and which identity transition was absent from the policy that selected cloudflared?

The companion lesson, tracing-lan-tls-through-cilium-gateway, explains certificate issuance, MetalLB address allocation, and the original LAN Gateway rollout. This chapter starts from the Gateway API object model, then follows the new public path and its policy failure.

How to work through the case

Use five passes. Stop at each checkpoint and reconstruct the causal chain without the page.

The live inspection commands are read-only. The policy experiments and temporary debug Pod describe completed incident work. Do not recreate them on the healthy cluster merely to follow the chapter.

Pass 1: Build the object model

First separate four overloaded words

Gateway, Ingress, Service, and route are often used casually, but each has several meanings.

Cloudflare also has a product named Gateway. It is unrelated to the Kubernetes Gateway in this case. Cloudflare Tunnel is the public transport used before traffic reaches Kubernetes.

Gateway API is an API, not a proxy

Gateway API provides Kubernetes resource types. Those objects describe intent. An implementation must watch them and construct the runtime data plane.

In devata, Cilium is the implementation:

flowchart LR
  class[GatewayClass cilium] --> gateway[Gateway lan-gateway]
  gateway --> controller[Cilium operator and agents]
  controller --> service[Generated LoadBalancer Service]
  controller --> envoy[Cilium Envoy configuration]
  routes[HTTPRoutes] --> controller
  grants[ReferenceGrants] --> controller

The object model separates responsibilities:

The official Gateway API model calls these separate infrastructure, Gateway, and routing concerns. That separation supports different owners without requiring an application developer to control the shared listener.

What each devata object declares

The GatewayClass named cilium selects Cilium’s controller. The Gateway named lan-gateway declares two listeners:

listeners:
  - name: http
    protocol: HTTP
    port: 80
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
        - name: lan-services-tls

TLS termination means Envoy presents the certificate and decrypts the request. After termination, an HTTPRoute can inspect HTTP fields such as hostname and path.

The redirect route attaches to the http listener and returns a 301 toward HTTPS. The Grafana route attaches to the https listener, matches grafana.lab.pragalva.me, and references monitoring/kps-grafana:80.

parentRefs:
  - name: lan-gateway
    sectionName: https
hostnames:
  - grafana.lab.pragalva.me
rules:
  - backendRefs:
      - name: kps-grafana
        namespace: monitoring
        port: 80

The route references a Service, not a Pod. The Service translates its port 80 to the Grafana container’s target port 3000 and uses an EndpointSlice to find the current Pod.

Why the ReferenceGrant lives with the backend

The route is in gateway-system; Grafana is in monitoring; Hubble UI is in kube-system. A route author must not be able to claim another namespace’s Service without consent.

The ReferenceGrant therefore lives in the target namespace:

metadata:
  namespace: monitoring
spec:
  from:
    - kind: HTTPRoute
      namespace: gateway-system
  to:
    - kind: Service
      name: kps-grafana

This is target-owned permission. It prevents a cross-namespace confused-deputy problem. It is not a NetworkPolicy and does not permit packets by itself.

Declared intent, reconciled state, and transaction proof

These are three different claims:

manifest exists
    ↓
controller accepts and programs it
    ↓
a real request succeeds through every layer

Accepted=True, ResolvedRefs=True, and Programmed=True are valuable. They prove controller decisions. They do not prove DNS, TLS, network policy, backend readiness, and the application response as one transaction.

Checkpoint 1

Close the manifests and explain the distinct jobs of Gateway, HTTPRoute, ReferenceGrant, Service, and EndpointSlice. Then name one command or observation that could disprove each explanation.

Pass 2: Trace both devata request paths

The LAN path

The LAN hostname resolves directly to the dedicated Gateway address:

flowchart LR
  client[LAN client] -->|DNS A record| vip[192.168.1.244]
  vip -->|MetalLB advertises IP| lb[Generated Gateway LoadBalancer Service]
  lb -->|Cilium eBPF interception| envoy[Cilium Envoy]
  envoy -->|TLS termination| route[HTTPRoute hostname and path match]
  route --> service[kps-grafana Service port 80]
  service --> slice[EndpointSlice target port 3000]
  slice --> pod[Grafana Pod]

MetalLB makes the address reachable on the LAN. It does not terminate TLS and does not choose Grafana. Cilium creates the Gateway Service and Envoy configuration. Envoy terminates TLS and evaluates the HTTP route. The backend Service and EndpointSlice finish the path to the Pod.

The public path

Cloudflare Tunnel does not open an inbound home-router port. Two cloudflared Pods maintain outbound connections to Cloudflare. A public request travels back over one of those established connections.

flowchart LR
  browser[Public client] --> edge[Cloudflare edge]
  edge -->|existing outbound tunnel| connector[cloudflared Pod]
  connector -->|Gateway Service HTTPS| envoy[Cilium Envoy]
  envoy --> route[HTTPRoute]
  route --> service[Grafana Service]
  service --> pod[Grafana Pod]

The remotely managed tunnel route uses this origin Service:

https://cilium-gateway-lan-gateway.gateway-system.svc.cluster.local:443

The connection still needs the LAN application identity:

TLS SNI:   grafana.lab.pragalva.me
HTTP Host: grafana.lab.pragalva.me

These fields serve different layers:

If the TCP connection fails, neither SNI nor Host matters yet. If SNI is wrong, certificate selection or verification fails. If TLS succeeds but Host is wrong, the HTTPRoute will not match.

Where the paths join

The LAN client approaches the Gateway from the network. The public path approaches the same Gateway from a Pod already inside Kubernetes.

LAN client ───────────────┐
                         ├─> Cilium Gateway -> Route -> backend
Cloudflare -> cloudflared ┘

The public path is therefore a hairpin path: an in-cluster workload enters the cluster’s ingress proxy and is routed back to another in-cluster workload.

That one detail explains why LAN Grafana could work while public Grafana failed. The two requests shared the listener, route, Service, EndpointSlice, and Pod, but they did not share the same source identity or policy path.

Authentication is a separate boundary

Cloudflare Tunnel and Cloudflare Access are independent:

• Tunnel transports a request from Cloudflare to the origin.
• Access decides whether the client must authenticate before Cloudflare releases the request into that tunnel route.
• Grafana’s own login is an application authentication boundary after the request reaches Grafana.

Removing a Cloudflare Access application can change edge authentication. It cannot repair a Cilium policy denial occurring after cloudflared reaches the origin.

Checkpoint 2

Draw the LAN and public paths from memory. Circle the first component both paths share. Then explain why a healthy LAN path proves the listener, route, Service, and Grafana backend but does not prove the cloudflared egress policy.

Pass 3: Add Cilium identities and policy

NetworkPolicy changes the default

Devata uses Cilium’s default policy-enforcement mode. An endpoint begins unrestricted in a direction. Once a policy with an egress section selects that endpoint, unspecified egress becomes denied. The same rule applies independently to ingress.

The cloudflared policy selects:

endpointSelector:
  matchLabels:
    app.kubernetes.io/name: cloudflared

That is a security boundary, not documentation. After selection, every required egress leg must be represented by an allowed identity and, where appropriate, port.

Cilium prefers identities over individual Pod IPs

Cilium derives security identities from labels. Recreated Pods may receive different IP addresses but retain the same application identity if their security-relevant labels stay the same.

The portable names matter more than the numeric IDs:

The numeric values observed during the incident, including ingress identity 8, are diagnostic evidence, not configuration. Do not hardcode dynamic workload identity numbers into policy.

Cilium has two logical enforcement points around Envoy

Cilium’s Gateway API data plane is closely integrated with the CNI. eBPF intercepts traffic for the generated Gateway Service and transparently forwards it to the per-node Envoy proxy. Cilium assigns ingress proxy traffic the special ingress identity.

The logical model is:

flowchart LR
  source[Original source identity] --> first[Policy before Envoy]
  first --> ingress[Cilium ingress identity]
  ingress --> envoy[Envoy listener and HTTPRoute]
  envoy --> second[Policy toward backend]
  second --> backend[Backend endpoint identity]

For the public Grafana hairpin, the relevant identities were:

cloudflared endpoint
    ↓
Cilium ingress identity
    ↓
Grafana endpoint on TCP 3000

The existing Cloudflare connections also required world egress on QUIC or HTTPS ports. DNS required CoreDNS identity and port 53.

Why the generated Gateway Service was misleading

The generated Service was selectorless. Its listener exposed port 443, while its Cilium-managed EndpointSlice used a synthetic address and internal target port observed as 192.192.192.192:9999.

That address and port were implementation artifacts used by Cilium to steer traffic into Envoy. They were not the Grafana backend and were not a stable application contract.

The initial policy tried to express the origin permission through the Service:

- toServices:
    - k8sService:
        serviceName: cilium-gateway-lan-gateway
        namespace: gateway-system
  toPorts:
    - ports:
        - port: "443"

This looked reasonable from the Service frontend. It did not represent the identity transitions through Envoy and onward to the routed backend.

The final policy expresses the real path

The corrected egress policy contains five contracts:

egress:
  # Resolve internal names.
  - toEndpoints:
      - matchLabels:
          k8s:io.kubernetes.pod.namespace: kube-system
          k8s:k8s-app: kube-dns
    toPorts:
      - ports:
          - port: "53"
            protocol: UDP
          - port: "53"
            protocol: TCP
        rules:
          dns:
            - matchPattern: "*"
 
  # Enter Cilium's Gateway proxy identity.
  - toEntities:
      - ingress
 
  # Reach only the routed Grafana backend target.
  - toEndpoints:
      - matchLabels:
          k8s:io.kubernetes.pod.namespace: monitoring
          k8s:app.kubernetes.io/name: grafana
    toPorts:
      - ports:
          - port: "3000"
            protocol: TCP
 
  # Reach only the routed Hubble UI backend target.
  - toEndpoints:
      - matchLabels:
          k8s:io.kubernetes.pod.namespace: kube-system
          k8s:app.kubernetes.io/name: hubble-ui
    toPorts:
      - ports:
          - port: "8081"
            protocol: TCP
 
  # Maintain the outbound Cloudflare tunnel and fallback transport.
  - toEntities:
      - world
    toPorts:
      - ports:
          - port: "7844"
            protocol: TCP
          - port: "7844"
            protocol: UDP
          - port: "443"
            protocol: TCP

The policy does not grant arbitrary cluster egress. It permits the ingress identity and then limits backend access to the two current route targets on their translated container ports.

Checkpoint 3

Explain why allowing only Gateway Service port 443 was not equivalent to allowing the full request. Name the identities and target ports required after the request entered Envoy.

Pass 4: Replay the incident

The symptom matrix

The first useful evidence was disagreement between paths:

A Cloudflare response header did not prove Cloudflare generated the body. Cloudflare can relay an origin response. The plain Access denied body and Envoy behavior made Cilium policy a stronger hypothesis, but the decision still needed flow evidence.

The decisive flow

Cilium’s monitor recorded the request at L7:

Request http ... identity 8->2, verdict Denied
GET https://grafana.lab.pragalva.me/

Identity 8 was reserved:ingress. The event showed that the request reached Envoy and was rejected by Cilium’s HTTP policy integration. Grafana was never asked to serve it.

This localized the failure:

Cloudflare edge       passed
Tunnel transport      passed
cloudflared DNS/TCP    passed
Cilium Envoy listener reached
policy identity lookup denied
Grafana                not reached

First repair: plausible, validated, still wrong

The first theory focused on Service port translation. The Gateway Service exposed 443, but its synthetic EndpointSlice target used 9999. Lab PR #59 removed the toPorts: 443 constraint from the toServices rule.

Why the theory was plausible:

• the observed Service and EndpointSlice ports differed;
• Cilium enforces policy after Service translation in relevant paths;
• Envoy returned the denial;
• the manifest rendered and passed server-side validation.

Why it was not proven:

• static validation only proved the resource was structurally accepted;
• the selectorless Service rule still did not express the special ingress identity;
• no successful synthetic transaction had been run under the exact cloudflared policy before merge.

PR #59 merged. Argo reconciled its merge commit. Public Grafana still returned the identical 403.

This is an important incident lesson:

A reasonable mechanism, a valid manifest, a green CI run, and a successful GitOps sync can all coexist with a wrong runtime theory.

Reproduce without Cloudflare

A disposable curl Pod was created in cloudflare-tunnel with the same app.kubernetes.io/name=cloudflared label. That caused the real Cilium policy to select it. The probe connected directly to the Gateway Service while preserving the LAN SNI and HTTP host.

Both requests returned Envoy 403:

grafana.lab.pragalva.me -> 403 Access denied
hubble.lab.pragalva.me  -> 403 Access denied

This removed Cloudflare from the reproduction. The failure could be created entirely inside Kubernetes by combining:

1. the cloudflared endpoint identity;
2. the cloudflared egress policy;
3. the Cilium Gateway hairpin path.

Narrow the policy by experiment

Temporary additive policies selected only the disposable probe Pod. Each experiment changed one identity assumption, waited for the endpoint policy to regenerate, and then repeated both requests.

The broad success was not accepted as the final configuration. It was a diagnostic midpoint. The final rule was narrowed from all cluster endpoints to the two actual route targets.

Second repair: match identities, then prove the transaction

Lab PR #60 replaced the selectorless Gateway Service rule with:

• the Cilium ingress entity;
• the Grafana endpoint identity on TCP 3000;
• the Hubble UI endpoint identity on TCP 8081.

After merge, Argo still reported PR #59’s revision because its last reconciliation preceded the merge. A normal refresh moved the Application to merge commit bbc2473. Only then was the new live policy inspected and the public transaction repeated.

Final evidence:

A final verification trap: curl -L can lie by omission

An earlier Hubble probe followed redirects and reported final status 200. That did not mean Hubble was publicly available without authentication. The final URL was a Cloudflare Access login page.

For a redirecting application, record at least:

initial status
Location header
final status
final URL
content type or page identity

An HTTP code is not an application identity. A 200 login page and a 200 Hubble page are different outcomes.

Checkpoint 4

Explain why PR #59 was a good hypothesis but an incomplete repair. Then name the single experiment that removed Cloudflare from the problem and the experiment that revealed the missing backend identity class.

Pass 5: Transfer the method

Try the healthy path safely

• Objective: prove the current declared policy, controller revision, and public transaction agree.
• Environment: /home/pragalva/Desktop/projects/lab, Kubernetes context pragalva@devata, read-only commands.
• Safety: read-only. Do not apply, patch, restart, or delete resources.

Start by verifying context and source:

cd /home/pragalva/Desktop/projects/lab
git status --short --branch
git fetch origin main
git show origin/main:kubernetes/infra/ingress/cloudflared/network-policy.yaml
kubectl config current-context

Inspect reconciliation and the live rule:

kubectl -n argocd get application cloudflare-tunnel \
  -o jsonpath='{.status.sync.status}{"\t"}{.status.health.status}{"\t"}{.status.sync.revision}{"\n"}'
 
kubectl -n cloudflare-tunnel get ciliumnetworkpolicy cloudflared -o yaml
kubectl -n cloudflare-tunnel get pods -l app.kubernetes.io/name=cloudflared -o wide

Inspect Gateway API status separately:

kubectl -n gateway-system get gateway lan-gateway
kubectl -n gateway-system get httproute
kubectl -n gateway-system get service cilium-gateway-lan-gateway
kubectl -n gateway-system get endpointslice \
  -l kubernetes.io/service-name=cilium-gateway-lan-gateway -o wide

Predict the first public response before running it:

curl -sS -o /dev/null -D - https://grafana.pragalva.me/
 
curl -sS -L -o /dev/null \
  -w 'status=%{http_code} final=%{url_effective} type=%{content_type}\n' \
  https://grafana.pragalva.me/
 
curl -sS -o /dev/null \
  -w 'status=%{http_code} redirect=%{redirect_url}\n' \
  https://hubble.pragalva.me/

Expected observation: public Grafana first redirects to /login, following the redirect ends on the Grafana login page, and public Hubble redirects to Cloudflare Access. These are dated expectations. If Cloudflare or application authentication changes, inspect the response identity rather than forcing the old code to match.

Diagnose from the first disagreement

Use this order:

1. Public DNS and edge: does the hostname resolve and does Cloudflare answer?
2. Access boundary: is the response an Access challenge, bypass, or application response?
3. Tunnel health: are both connector replicas Ready and maintaining connections?
4. Origin DNS and TCP: can the selected identity resolve and connect to the Gateway Service?
5. TLS: does SNI select a valid certificate for the LAN hostname?
6. Gateway listener: is the listener Programmed and is Envoy accepting the connection?
7. HTTPRoute: do hostname, path, attachment, and reference conditions agree?
8. Policy identities: which source, ingress, and backend identities appear in the verdict?
9. Service translation: what frontend port, target port, and EndpointSlice are actually used?
10. Application: did the backend log or serve the request?

Stop at the first layer whose evidence disagrees with the previous layer. Do not restart all controllers to erase the transition evidence.

Use a proof ladder for fixes

theory
  ↓
rendered manifest
  ↓
API validation
  ↓
isolated live reproduction
  ↓
successful synthetic transaction
  ↓
signed Git change and CI
  ↓
Argo at the exact merge revision
  ↓
fresh public transaction

PR #59 reached Git, CI, and Argo but skipped a successful transaction under the exact policy theory. PR #60 included the isolated reproduction and synthetic transaction before publication.

Beginner mentor notes

1. A Kubernetes object is a contract, not the implementation. A Gateway needs a controller and data plane. A Service needs EndpointSlices. A Certificate needs cert-manager.
2. A Service and a Gateway solve different problems. Service provides a stable L4 backend destination. Gateway listeners and Routes make L7 decisions.
3. The listener and route are deliberately separate. The shared platform can own ports and certificates while application owners attach routing rules.
4. Cross-namespace references require consent. ReferenceGrant belongs with the target. It is not packet authorization.
5. TLS SNI and HTTP Host are not interchangeable. One exists before decryption; the other is inspected after TLS termination.
6. Network policy is identity-aware state. Once an egress policy selects a Pod, every required destination identity must be explicit.
7. Proxy paths create more than one policy boundary. Hairpin traffic can enter a special ingress identity and then leave toward a backend identity.
8. Implementation artifacts are not durable contracts. Synthetic addresses, proxy ports, and numeric security IDs are valuable diagnostics but poor policy intent.
9. A 403 is not a root cause. Cloudflare Access, Envoy policy, and Grafana can all return 403 for different reasons.
10. A healthy connector does not prove an application. Tunnel continuity and service uptime are separate contracts.
11. A successful build does not prove runtime semantics. Schema checks cannot predict every CNI identity transition.
12. A merged PR is not live state. Verify the GitOps controller’s exact revision before retesting.
13. Following redirects can hide the responder. Always capture the final URL and page identity.
14. Use broad permissions only as a diagnostic midpoint. Once all proves the missing class, narrow to exact identities and ports.
15. Reproduce with the same labels. A debug Pod outside the policy selector can produce a false success.
16. Preserve the healthy rollback path. The direct LAN LoadBalancer addresses stayed available while the public path was repaired.

Reconstruct or recover

The authoritative inputs are:

1. Gateway API CRDs and the Cilium Gateway API feature configuration;
2. the lan-gateway Gateway, HTTPRoutes, ReferenceGrants, and certificate manifests;
3. the cloudflared Deployment, SealedSecret, and CiliumNetworkPolicy;
4. remotely managed Cloudflare tunnel routes, origin SNI and Host settings, and Access applications;
5. the off-cluster Sealed Secrets recovery key and Cloudflare account access.

Reconstruct in dependency order:

1. install Gateway API CRDs;
2. install and reconcile Cilium with Gateway API enabled;
3. reconcile cert-manager, the certificate, the Gateway, routes, and grants;
4. verify the Gateway listener and LAN transaction;
5. reconcile the cloudflared token and Deployment;
6. apply the identity-aware cloudflared policy;
7. verify both connector replicas and tunnel metrics;
8. verify Cloudflare route and Access behavior;
9. run fresh public Grafana and Hubble transactions;
10. confirm Argo is Synced/Healthy at the expected Git revision.

Rollback public exposure without breaking LAN access:

1. disable the published Cloudflare routes;
2. revert the cloudflared application through Git;
3. wait for Argo to reconcile the revert;
4. keep grafana.lab.pragalva.me and hubble.lab.pragalva.me on the LAN Gateway;
5. retain direct Grafana .242 and Hubble .243 LoadBalancer paths until a separate change removes them.

Check yourself

1. Why is Gateway API not itself a load balancer or proxy?
2. Which object owns listeners, which owns HTTP matches, and which owns current backend addresses?
3. Why does the Grafana ReferenceGrant live in monitoring?
4. At what point can Envoy inspect the HTTP Host header?
5. Where do the LAN and Cloudflare paths first converge?
6. Why is the public path a hairpin path?
7. What happens to unspecified egress after a Cilium policy selects cloudflared?
8. Why did a toServices rule for the generated Gateway Service fail to describe the full policy path?
9. Which flow evidence proved the denial occurred in Cilium rather than Grafana?
10. Why did removing toPorts: 443 pass validation yet fail to restore service?
11. What did the disposable same-label Pod prove that an external curl could not?
12. Why was toEntities: all useful but unacceptable as the final rule?
13. Which exact identities and ports replaced it?
14. Why must Argo’s sync revision be checked after merge?
15. How can curl -L produce a misleading 200?
16. What evidence now proves public Grafana reached Grafana, rather than merely proving Cloudflare returned a response?

The chapter is complete when both request diagrams and the identity transition can be drawn from memory, PR #59 can be defended as plausible but disproven, and one falsifying observation can be named for every layer in the final proof.

References

• Gateway API introduction
• Gateway API resource model
• Gateway API HTTPRoute
• Gateway API ReferenceGrant
• Cilium 1.18 Gateway API and policy model
• Cilium policy enforcement modes
• Cloudflare Tunnel
• Kubernetes Service
• devata Gateway resources at the recovered revision
• devata cloudflared resources at the recovered revision
• lab PR #59, incomplete Service-port repair
• lab PR #60, recovered identity-aware policy