Prerequisites: service

A ClusterIP is the address a service gets by default, and the strange thing about it is that no machine in your cluster owns it. See one:

kubectl get svc -n monitoring kps-grafana

The CLUSTER-IP column shows an address in the 10.96.0.0/12 range. No network card on any of devata’s four nodes has that address. Nothing answers ARP for it. It exists only as an agreement: a packet sent there should come out at one of the pods behind the Service.

That agreement is the whole job. A virtual address does nothing on its own. Some software on every node has to notice a packet headed for the ClusterIP and rewrite its destination to a real pod that is listening. That rewrite of the destination address is called DNAT, covered in iptables. Which component does the rewriting, and how, is what separates one cluster’s networking from another’s. On most clusters kube-proxy keeps the agreement. On devata cilium does, and going-cilium-only follows one request to the exact spot where that handoff happens.

The range is fixed when the cluster is created (the service CIDR) and is independent of the pod network and your home LAN. A ClusterIP is reachable only from inside the cluster. Reaching a Service from your laptop needs a different address class, a NodePort or an external IP from metallb.

Reference: Services, virtual IPs and Service proxies.