Prerequisites: service, clusterip
Inside the cluster you reach a service by name, like kps-grafana.monitoring, not by its address. Something has to turn that name into the clusterip. That something is CoreDNS, a DNS server running as pods in the cluster, with its own Service at a well-known ClusterIP that every pod is told to use as its nameserver.
Watch it on devata. Run a throwaway pod and look at how it resolves names:
kubectl run trace --image=busybox:1.36 --rm -it --restart=Never -- \
sh -c 'cat /etc/resolv.conf; echo ---; nslookup kps-grafana.monitoring'The resolv.conf nameserver is not your router or 8.8.8.8. It is 10.96.0.10, the CoreDNS ClusterIP, and the nslookup hands back the same ClusterIP you see for Grafana. There is a sharp consequence hiding here: that nameserver address is itself a ClusterIP, so the very first hop of every request already depends on ClusterIP routing working. If that routing breaks, DNS breaks, and nothing in the cluster resolves, even though every node still reports healthy. That is why node health tells you nothing about whether traffic actually flows.
Reference: DNS for Services and Pods.