Kubernetes as APIs and control loops

5 min read

Document contract

  • Role: root concept
  • Scope: the minimum control model used throughout this vault
  • Truth boundary: portable Kubernetes concepts with read-only devata observations
  • Last verified: Kubernetes documentation, 2026-07-26

Prerequisites: none.

Kubernetes is easier to reason about when it is not treated as a bag of commands. It is an API containing records of intent, plus controllers that repeatedly move observed reality toward that intent.

devata is a pool of Talos machines, but operators usually address it as one API. They submit objects that describe workloads, networking, storage, permissions, and policy. The control plane stores and processes those objects. Node-level components turn them into running containers and network paths.

The controller model

flowchart LR
  human[Human or GitOps controller] -->|writes spec| api[Kubernetes API]
  api --> etcd[(etcd)]
  api --> controller[Controller watches objects]
  controller -->|creates or updates dependants| runtime[Pods, endpoints, volumes]
  runtime -->|status and events| api
  api -. observed state .-> controller

Most Kubernetes objects separate spec from status:

  • spec is desired state supplied by a user or controller;
  • status is observed state reported by the system.

A Deployment spec can request three replicas. Its controller sees that zero Pods exist, creates them, and keeps replacing failed ones. A Service spec selects labels. EndpointSlice controllers keep the current backend addresses aligned with that selection. A PersistentVolumeClaim requests a storage contract. A storage provisioner tries to satisfy it.

This explains why deletion is not always durable. Deleting a Pod owned by a Deployment removes observed state while leaving desired state unchanged, so the controller creates another Pod. To change the outcome, change the owning object or its source of truth.

The control plane and nodes

The control plane contains the API server, etcd, scheduler, and controller managers. It accepts intent, stores it, decides where unscheduled Pods should run, and maintains object relationships.

Nodes run the kubelet and container runtime. The kubelet watches the Pods assigned to its node and asks the runtime to create their containers. A CNI supplies Pod networking. In devata, cilium also implements Service routing without kube-proxy.

Inspect the current machines:

kubectl config current-context
kubectl get nodes -o wide

Ready means the node is reporting health to Kubernetes. It does not prove that every workload, mount, route, or physical link is healthy. recovering-loki-wal-replay-oom is an example where healthy higher layers did not erase a retained application failure loop.

Pods are scheduled execution units

Kubernetes schedules Pods, not individual containers. Containers in one Pod share a network identity and can share volumes. A controller usually owns the Pod because a standalone Pod has no higher-level object to recreate it after deletion or failure.

kubectl get pods -A -o wide

Read a row as connected evidence:

  • namespace scopes the name and policy context;
  • readiness reports whether declared readiness checks currently pass;
  • status summarizes lifecycle state;
  • restarts show container replacement within the Pod;
  • node and Pod IP connect scheduling to networking.

The controller hierarchy matters more than the row. Use owner references to trace upward:

kubectl -n <namespace> get pod <pod> -o jsonpath='{.metadata.ownerReferences}'

Namespaces scope names, not complete isolation

A namespace groups namespaced objects and gives RBAC, quota, policy, and defaulting rules a boundary. It is not a security wall by itself. Nodes, PersistentVolumes, StorageClasses, and many custom resources are cluster-scoped.

kubectl api-resources --namespaced=true
kubectl api-resources --namespaced=false

The output prevents a common reasoning error: assuming every object lives inside the namespace shown by a workload.

Services give changing Pods a stable identity

Pods are replaceable and their addresses change. A service gives callers a stable name and virtual address, while endpointslice objects track the current eligible backends. CoreDNS resolves the Service name. Cilium or kube-proxy implements the hop from virtual address to backend.

flowchart LR
  client --> dns[CoreDNS]
  dns --> service[Service ClusterIP]
  service --> dataplane[Cilium service translation]
  endpoints[EndpointSlice] -. backend set .-> dataplane
  dataplane --> pod[Ready Pod]

No one layer proves the whole request. A Service can exist without endpoints. Endpoints can exist while policy blocks traffic. A Pod can be Ready while the client-facing Gateway is wrong. going-cilium-only walks the complete path.

kubectl is an API client

kubectl reads a kubeconfig, selects a context, and sends authenticated requests to the API server. It is one client among many. Argo CD, operators, scripts, and applications use the same API under different identities.

Before any lab command:

kubectl config current-context
kubectl auth can-i get pods -A

Prefer declarative, reviewed state for durable changes. Imperative commands are useful for observation and disposable experiments, but they do not explain how to recreate the final system.

A practical evidence ladder

When checking a feature, move through the layers instead of stopping at the first green signal:

  1. Declaration: the intended object exists in the authoritative Git path.
  2. Reconciliation: the owning controller reports that desired and observed state agree.
  3. Resources: dependent Pods, endpoints, volumes, or Secrets are healthy.
  4. Infrastructure: the node, network, mount, and storage layers support them.
  5. Transaction: a user-visible request, write/read, metric query, or log query succeeds.

The same ladder is used in devata-system-map and reconstructing-devata.

Try it safely

All commands below are read-only. Predict which controller owns each object before revealing the answer.

kubectl config current-context
kubectl -n monitoring get deploy,rs,pods
kubectl -n monitoring get service,endpointslice
kubectl -n monitoring describe pod <pod-name>
kubectl -n monitoring get events --sort-by=.lastTimestamp

Trace Deployment to ReplicaSet to Pod, then Service to EndpointSlice to the same Pod IP. describe and Events explain why the controllers made their recent decisions.

Check yourself

  1. A Pod is deleted and returns. Which object should you inspect next, and why?
  2. A Service resolves but a request fails. Which independent layers must be tested?
  3. What does a Bound PVC prove, and what durability claim does it not prove?
  4. Why can Talos, Kubernetes, and Argo CD all be healthy reconcilers while an application transaction still fails?

References