Talos, the API-managed Kubernetes OS

5 min read

Document contract

  • Role: concept
  • Scope: Talos operating and recovery model, not an installation walkthrough
  • Truth boundary: portable Talos v1.11 concepts grounded in the current devata repository design
  • Last verified: Talos v1.11 documentation and lab/talos, 2026-07-26

Prerequisites: kubernetes

Talos is the operating system on every devata node. It has no SSH server, interactive shell, or package manager. An operator does not log in and accumulate changes. They submit a declarative machine configuration through the Talos API, and Talos keeps the machine aligned with it.

That choice is not only about a smaller attack surface. It changes where knowledge must live. A package installed by hand, an edited file, or an undocumented mount cannot become a hidden requirement because those familiar mutation paths do not exist.

Two APIs, two ownership boundaries

flowchart LR
  operator[Operator workstation]
  talosconfig[talosconfig<br/>Talos client identity]
  kubeconfig[kubeconfig<br/>Kubernetes client identity]
  apid[Talos API<br/>machine and services]
  kubeapi[Kubernetes API<br/>cluster objects]
  machine[OS image, mounts,<br/>network, kubelet]
  objects[Deployments, Services,<br/>Secrets, controllers]

  operator --> talosconfig --> apid --> machine
  operator --> kubeconfig --> kubeapi --> objects

talosctl speaks to the Talos API and changes or inspects the machine layer. kubectl speaks to the Kubernetes API and changes or inspects cluster objects. Kubernetes RBAC cannot grant Talos access, and a Talos client certificate does not automatically grant Kubernetes permissions.

This distinction explains common failure investigations. A Running Pod and Bound PVC say nothing about whether the Talos mount below the volume is active. Conversely, a healthy Talos node does not prove that a Kubernetes Service has endpoints.

The machine configuration is intent, not a setup script

A rendered machine configuration contains node role, cluster trust, networking, installation, kubelet, and other machine settings. Talos validates and applies that desired state. Some changes can reconcile immediately; others require a reboot or staged application.

The public lab/talos directory deliberately holds only reviewable non-secret inputs:

  • per-node patches for hostname, address, install image, labels, and mounts;
  • volume documents for EPHEMERAL and user storage;
  • Image Factory schematics that select system extensions and kernel configuration.

Rendered configurations include cluster trust material and stay outside public Git. That split gives reviewable infrastructure without publishing the credentials that control it.

Immutable does not mean stateless

Talos has explicit writable and persistent locations. The important question is not “can this path be written?” but “which Talos volume backs it, and what event is it designed to survive?”

In devata, the OptiPlex worker has a dedicated XFS user volume mounted at /var/mnt/longhorn. The kubelet receives that path through a shared bind mount so Longhorn can use it. Another worker keeps its Longhorn directory inside Talos EPHEMERAL storage. Both survive ordinary reboots, but their wipe and disk-failure boundaries differ.

deleting-the-hypervisor shows why Kubernetes object health alone was insufficient: a PVC could remain Bound while the expected Talos user volume was no longer declared or mounted.

System extensions replace package installation

There is no apt install on a Talos node. Required host capabilities are selected in an Image Factory schematic and become part of the installed image. Longhorn needs iSCSI and utility support, so devata declares iscsi-tools and util-linux-tools extensions. A GPU worker uses a different schematic for its hardware.

This makes a node image reproducible, but it also moves a package-like change into the OS upgrade path. The correct verification is:

  1. the schematic declares the extension;
  2. the installed image matches the intended schematic and Talos version;
  3. the service or binary capability is visible to Talos;
  4. the Kubernetes controller that depends on it is healthy.

Recovery follows from the design

A Talos node should be replaceable from a known image, rendered machine configuration, and durable data source. The machine does not need a hand-maintained history of shell actions.

That does not make recovery automatic. The operator still needs:

  • the correct install disk and network identity;
  • private machine configuration and talosconfig material;
  • an etcd recovery decision for a control-plane loss;
  • persistent data or backups that live beyond the failed node;
  • version-matched tooling and a tested proof sequence.

reconstructing-devata orders these dependencies for the whole platform.

Inspect without changing state

Confirm the target before every command. These examples are read-only:

talosctl config info
talosctl -n <node-ip> version
talosctl -n <node-ip> health
talosctl -n <node-ip> services
talosctl -n <node-ip> get machineconfig -o yaml
talosctl -n <node-ip> get mounts

Expected observations:

  • config info names the active Talos context, endpoints, and role.
  • version separates local client version from remote server version.
  • services shows Talos-managed services such as kubelet and, on the control plane, etcd.
  • machineconfig shows active intent but may contain sensitive fields, so do not paste or commit it.
  • mounts lets you verify the OS storage layer beneath Kubernetes.

Applying configuration is a separate operation

talosctl apply-config, reset, upgrade, reboot, and power-cycle actions can disrupt or erase a node. Their target, rollback, quorum effect, and storage impact must be established before use.

Check yourself

  1. Why can a Kubernetes object look healthy while its Talos storage layer is wrong?
  2. Which changes belong in the public talos/ inputs, and which rendered artifacts must remain private?
  3. What must survive off the node before “cattle, not pets” becomes a defensible recovery claim?
  4. When would kubectl be the wrong tool even though the symptom appears inside Kubernetes?

References