Reconciliation: the loop that closes the gap between desired and actual

Prerequisites: kubernetes

Reconciliation is a control loop. It reads a desired state, observes the actual state, computes the difference, takes whatever action narrows that difference, and then starts over. It never finishes. The loop is the unit of work, and the steady state is not “done” but “no difference found this time round”. This single pattern is the engine under both Kubernetes itself and the GitOps tooling layered on top of it, so it is worth watching it run before meeting argocd, which is just one more reconciler.

Kubernetes is already a pile of reconcilers

You do not need to install anything to see a reconciliation loop, because kubernetes is built from them. Every controller in the cluster runs the same loop against a different kind of object. Watch the simplest one, the Deployment controller, do its job. Make a throwaway Deployment on devata.

kubectl create deployment recon-demo --image=nginx --replicas=3
kubectl get pods -l app=recon-demo -w

Three pods appear and go Running. The desired state is “3 replicas”, written in the Deployment’s spec. The actual state, the number of pods that exist, started at zero. The controller saw the gap and created three pods to close it. Now create drift by hand, in another terminal.

kubectl delete pod -l app=recon-demo --field-selector status.phase=Running | head -1

In the watch window a pod goes Terminating, and before you can think about it a replacement is already ContainerCreating. You did not tell anything to make a new pod. The controller observed that actual (now 2) no longer matched desired (still 3) and acted. That is the entire loop: observe, diff, act, repeat. Clean up when you have seen it.

kubectl delete deployment recon-demo

The key thing you just watched is that the desired state is durable and the actual state is disposable. The spec said 3, so the controller defended 3 against your deletion. Reconciliation does not care how the actual state got wrong. It only ever asks “does actual match desired, and if not, what do I do about it”.

What GitOps adds: an outer loop

The Deployment controller’s desired state lives in the cluster’s own database, etcd. GitOps adds a loop one level out, where the desired state lives in a git repository instead. The GitOps reconciler reads the manifests in git, observes the live objects in the cluster, diffs them, and applies whatever is needed to make the cluster match git. It is the same observe-diff-act loop, with git playing the role that spec played above.

This is why drift correction in GitOps feels like the pod coming back. When you kubectl delete a Deployment that git still declares, the GitOps reconciler sees actual (gone) diverge from desired (declared in git) and recreates it, exactly as the Deployment controller recreated the pod. The mechanism is identical; only the source of the desired state moved from etcd to a repository. The two terms for the GitOps reconciler noticing and then fixing that divergence are out-of-sync (it noticed) and self-heal (it fixed it), and both are properties of the application it is reconciling.

A reconciler is therefore not a deploy tool that runs once. It is a process that holds the cluster against a written description indefinitely. The next note is the specific reconciler this homelab uses to hold devata against git.