Deleting the Hypervisor: Moving devata from Proxmox VMs to Bare-Metal Talos

Dated case study

This page records the migration state on 2026-07-16. Its statements that storage was single-node and Longhorn was future work are historically correct for the end of that change, not current architecture. Longhorn, observability-volume migration, and bounded Velero backup were completed afterward. See longhorn-and-velero for the successor design and devata-system-map for the current platform map.

The dangerous command was not the interesting part.

By the time the Dell OptiPlex finally wiped its own Proxmox installation, the decision had already been made safe by everything around that command: two off-host VM archives, a host-configuration backup, a healthy surviving control plane, a Talos kernel booted only into memory, a hardware inspection from maintenance mode, a machine configuration reviewed in Git, and a person standing beside the power button.

The actual wipe took seconds. Making those seconds boring took the whole migration.

This case study records how devata moved from two Talos worker VMs inside Proxmox to one Talos worker running directly on the OptiPlex. The change began as a hardware simplification, but storage became the real subject. Prometheus had a Bound PersistentVolumeClaim and still lost its entire TSDB after node reboots. The migration therefore had to answer a harder question than “does the node rejoin Kubernetes?”

It had to prove that bytes written by a workload would still exist after a real firmware reboot.

Case file

The PVC was Bound. The data was not durable.

The trigger was Prometheus, not Proxmox.

Kubernetes reported the Prometheus claim as Bound. That only proved that a PVC object had been matched to a PV object. It did not prove that the PV’s host path lived on storage that survived a Talos reboot. The Kubernetes storage model keeps the control-plane object and the actual storage medium deliberately separate. A green object can describe a bad path perfectly.

The evidence on devata was blunt:

• Prometheus started with an empty WAL after node reboot.
• Its supposed PV directory under /opt/local-path-provisioner contained no real TSDB.
• Loki appeared to survive on a similar path, but that did not make the path supported or safe. It made Loki the lucky workload.
• Talos treats /opt as an overlay backed by the EPHEMERAL system volume. It was not a storage contract I could trust for monitoring history.

The misleading signal

PVC=Bound is a scheduling and binding fact. It is not a durability test. Durability begins only when the backing path is identified, the underlying device is known, and data survives the failure or reboot being claimed.

That changed the shape of the work. Installing Longhorn immediately would have added a storage controller while the physical host was still hidden behind a hypervisor and its disk was divided into small virtual disks. The cleaner order was:

1. expose the real NVMe to Talos;
2. create a durable Talos user volume;
3. restore observability on that volume with the existing provisioner;
4. install Longhorn later as a separate, observable change.

The final system is not yet replicated storage. It is a sound foundation for it.

What actually changed

The control plane at .8 and the separate worker at .9 did not move. That was the most important architectural fact in the whole operation: the OptiPlex could disappear without taking the Kubernetes API or etcd with it.

Was deleting Proxmox directionally sound?

Yes, for this machine and this cluster. Not as a universal rule.

Proxmox is valuable when a host needs to run heterogeneous operating systems, isolate unrelated labs, provide VM snapshots, or multiplex spare capacity among workloads that should not know about one another. None of those jobs remained on this OptiPlex. It existed mainly to divide four physical CPU cores and 8 GB of memory into two nearly identical Talos workers.

The strongest argument was not performance. No before-and-after benchmark was run, so this case study makes no speed claim. The strongest argument was legibility: one physical machine became one Kubernetes node, and one real disk became one explicitly managed Talos disk layout.

The migration contract

The plan had six gates. A failed gate stopped the migration before the next irreversible step.

The migration followed this sequence:

Preparing the replacement before touching the host

Talos installation was treated as an artifact build, not a console session.

The Image Factory schematic selected three extensions:

• intel-ucode for CPU microcode;
• iscsi-tools for the iSCSI userspace and service Longhorn will require;
• util-linux-tools for utilities such as fstrim that Longhorn expects on Talos.

The official Talos boot-assets documentation describes the schematic as the reproducible input to kernels, initramfs images, ISOs, and installer images. The installer for schematic 36cd6536...c010 and Talos v1.11.5 was mirrored to the LAN registry so the new node did not depend on the previously unreliable Factory CDN path during installation.

The configuration had two different homes for two different reasons:

• non-secret machine patches and volume documents live in the public lab repository;
• the rendered worker configuration containing cluster trust material stays outside Git with mode 0600.

The public inputs are small enough to audit directly:

machine:
  nodeLabels:
    node-role.kubernetes.io/worker: ""
  install:
    disk: /dev/nvme0n1
    image: 192.168.1.13:5000/installer/<schematic>:v1.11.5
    wipe: true
  kubelet:
    extraMounts:
      - destination: /var/mnt/longhorn
        type: bind
        source: /var/mnt/longhorn
        options: [bind, rshared, rw]
  network:
    hostname: talos-opt-7040

The important word here is wipe. A correct YAML file can still destroy the wrong disk. That is why disk selection was checked from the booted machine rather than trusted from the old Proxmox inventory.

Building a rollback that did not share the failure domain

A backup stored on the OptiPlex would have been erased by the operation it was supposed to protect against. The recovery set was therefore copied to the workstation before the VM shutdown:

• Proxmox host configuration archive;
• full compressed VMA archive for VM 100;
• full compressed VMA archive for VM 101;
• a SHA256SUMS file covering all three.

The proof command was intentionally dull:

cd ~/devata-secrets/proxmox-retirement-2026-07-16
sha256sum -c SHA256SUMS

All three returned OK. File existence was not accepted as proof because an interrupted network copy can leave a large, convincing, unusable archive.

An etcd snapshot and a dated cluster inventory were captured separately. They protected Kubernetes state, while the VMA files protected the retired machines. Those are different recovery problems and needed different backups.

Draining Kubernetes before shutting down machines

Each VM worker was drained on its own. Kubernetes documents kubectl drain as the safe boundary before hardware maintenance because it cordons the node and evicts ordinary pods while respecting graceful termination and disruption policy. The official drain guide also explains why DaemonSet pods are ignored: their controller owns one pod per eligible node and would immediately recreate them.

kubectl drain talos-3eq-vo2 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force \
  --timeout=10m
 
# Verify the surviving cluster, then repeat for talos-igk-9ep.

The order mattered. After .2 drained, Cilium and the remaining workloads were checked before .3 was touched. Only when both nodes were SchedulingDisabled and the surviving Cilium operators were healthy were VM 100 and VM 101 shut down from Proxmox.

This was not an availability-perfect migration. Grafana, Prometheus, and Loki were deliberately pinned to the future hostname by the merged Git configuration, so they entered a visible Pending or Degraded state until talos-opt-7040 existed and received new storage. That downtime was bounded and explained rather than hidden.

Booting Talos without writing the disk

The OptiPlex had no IPMI or Intel AMT endpoint. A failed remote reboot therefore required a human at the machine. That made a reversible first boot essential.

The Talos v1.11.5 kernel and initramfs were copied into Proxmox and loaded with Linux kexec. kexec transfers control directly from the running Linux kernel to another kernel. In this case, Talos started entirely from RAM and DHCP without modifying the NVMe.

This was safe only because the disk write had not happened yet

If maintenance mode had failed to appear, a physical power cycle would have booted the unchanged Proxmox installation. After talosctl apply-config, that rollback ceased to exist.

The maintenance address appeared at .12. From there, Talos reported the facts that mattered:

talosctl -n 192.168.1.12 get disks --insecure
talosctl -n 192.168.1.12 get links --insecure
talosctl -n 192.168.1.12 get addresses --insecure

The results matched the rendered configuration:

• active interface: enp0s31f6;
• interface MAC: 18:66:da:28:d4:25;
• writable device: nvme0n1;
• device model: UMIS RPETJ256MGE2MDQ;
• device size: 256 GB.

Only then was the irreversible command allowed:

talosctl apply-config \
  --insecure \
  --nodes 192.168.1.12 \
  --file "$WORKER_CONFIG"

Talos installed from the LAN registry, rebooted, claimed the static address .10, and registered as talos-opt-7040. The node was not considered added when its API answered. It was considered added after Cilium initialized, Kubernetes marked it Ready, and the node-level DaemonSets were healthy.

What the NVMe became

Talos v1.11 accepts extra block-configuration documents alongside the main machine configuration. A UserVolumeConfig allocates a partition and mounts it under /var/mnt/<name>. Naming this one longhorn produces the partition label u-longhorn and mount point /var/mnt/longhorn.

The reported 43 GB and 193 GB values are not evidence that Talos ignored the 40 GiB and 180 GiB limits. They are the same byte counts displayed with decimal GB units.

The committed documents are:

apiVersion: v1alpha1
kind: VolumeConfig
name: EPHEMERAL
provisioning:
  diskSelector:
    match: system_disk
  minSize: 20GiB
  maxSize: 40GiB
  grow: false
---
apiVersion: v1alpha1
kind: UserVolumeConfig
name: longhorn
provisioning:
  diskSelector:
    match: system_disk
  minSize: 170GiB
  maxSize: 180GiB
  grow: false
filesystem:
  type: xfs

The explicit upper bounds stop EPHEMERAL from consuming the disk before the user volume is allocated. This disk layout had to be decided at first installation, not after Longhorn arrived.

The persistence bridge

Longhorn was intentionally not installed during the hardware migration. Its Talos prerequisites were installed, and its data path was reserved, but introducing a CSI system while replacing two nodes would have combined two failure domains into one change.

The interim bridge reuses the existing local-path provisioner with a node-specific path:

{
  "nodePathMap": [
    {
      "node": "talos-opt-7040",
      "paths": ["/var/mnt/longhorn/local-path"]
    },
    {
      "node": "DEFAULT_PATH_FOR_NON_LISTED_NODES",
      "paths": ["/opt/local-path-provisioner"]
    }
  ]
}

Lab PR #46 put that ConfigMap under its own Argo CD Application. The storage path is therefore not a one-off live patch. Git records it, Argo reconciles it, and drift becomes visible.

This distinction must stay explicit:

The volume is named longhorn; Longhorn is not installed

The name reserves the intended data path and carries the extensions Longhorn needs. Today the live PVs remain local-path volumes with node affinity and no replicas. The official Longhorn on Talos guide is the contract for the later installation: iscsi-tools, util-linux-tools, a kubelet-visible /var/mnt/longhorn, privileged Pod Security for its namespace, and the UserVolumeConfig-backed path.

Why the old claims did not move

The old PVC objects were bound to PVs whose node affinity named the deleted VM workers and whose host paths existed inside those VMs. Changing the workload nodeSelector could not make that storage appear on a different physical machine. Kubernetes correctly left the pods Pending.

The migration therefore treated the old live data and the workload definitions separately:

1. the VM archives preserved the old disks as rollback evidence;
2. Grafana, Prometheus, and Loki were quiesced;
3. their dead VM-bound claims were deleted;
4. the local-path provisioner created fresh PVs on talos-opt-7040;
5. the workloads restarted against empty, persistent directories.

This was data loss from the live cluster by choice, not by surprise. The old observability history was already unreliable, and importing it during a node-and-storage migration would have made failures harder to attribute. The archives remain available if forensic recovery is ever worth the effort.

The first proof looked conclusive

After the new claims bound, a proof file was written inside Grafana’s actual PVC. The node was then rebooted with Talos powercycle mode, which skips kexec and exercises the firmware boot path and installed disk.

The test checked four separate layers:

1. firmware could boot the installed Talos disk;
2. Talos returned to running and ready;
3. Kubernetes returned the node and three stateful workloads to Ready;
4. the file inside Grafana’s PVC still contained the expected value.

All four passed. Every Argo CD Application returned Synced/Healthy.

That still was not the end.

The verification that caught the verification bug

While collecting live evidence for this article, talosctl get volumestatus no longer listed u-longhorn. The PV objects were still Bound. Grafana, Prometheus, and Loki were still Running. The filesystem path still existed. At the Kubernetes layer, the cluster looked healthy.

At the Talos layer, the 193 GB partition was no longer mounted.

The evidence reconstructed the failure precisely:

The worker label had been added with a main-document talosctl patch machineconfig operation after the first reboot proof. That operation was reported as successful and required no reboot, but the active auxiliary volume document disappeared and Talos unmounted the user volume.

The partition was not destroyed. That is why recovery stayed small:

1. add the worker label to the secure, complete rendered configuration;
2. validate the complete three-document file;
3. reapply the main MachineConfig, EPHEMERAL VolumeConfig, and longhorn UserVolumeConfig together;
4. verify u-longhorn returned to volumeconfigs and volumestatus;
5. reboot so the restored user-volume document remounted /dev/nvme0n1p7;
6. read a new proof file from Grafana after the reboot.

The second proof passed:

MountStatus   u-longhorn   /dev/nvme0n1p7   /var/mnt/longhorn   xfs
persisted-proof=full-config-volume-proof-2026-07-16

The corrected operational rule

The Talos worker configuration is a document set, not just machine: YAML. Any future node change must preserve and reapply the main MachineConfig plus every VolumeConfig and UserVolumeConfig. Post-change verification must inspect both volumestatus and mountstatus; Kubernetes health alone is insufficient.

This was the most valuable finding in the migration. The first reboot proved the disk layout. The second round proved that the configuration lifecycle could preserve it.

The end state, verified live

The declarative record is split across two merged lab pull requests:

• PR #45: Image Factory schematic, OptiPlex machine and volume documents, and observability placement;
• PR #46: worker role plus the Argo-managed persistent local-path mapping.

What this design did not provide at completion

Calling the migration successful does not erase its limits.

• No storage replication. If the OptiPlex is down, its three local PVs are unavailable.
• No automatic volume failover. Node affinity correctly keeps those pods attached to the only node that has the bytes.
• No live restoration of old observability history. The prior VM disks exist only in backup archives.
• No remote-access replacement yet. Tailscale was deliberately deferred, so the node is LAN-only.
• No performance conclusion. Removing the hypervisor simplified resource ownership, but no controlled benchmark was run.
• A brittle registry address. The installer mirror is pinned to the workstation’s current LAN address. A DHCP reservation or stable local DNS name should precede the next Talos upgrade.
• Client version skew. The workstation used talosctl v1.12.4 against Talos v1.11.5. It worked, but a matching client would remove warnings and reduce one variable during future lifecycle operations.

At the conclusion of this migration, Longhorn was the next storage phase rather than part of the same change. Keeping it separate made its prerequisites, failure experiment, workload migration, and rollback independently observable. longhorn-and-velero records the completed successor and its remaining limits.

What I would keep and what I would change

Keep

• Back up outside the machine being destroyed.
• Verify hashes rather than trusting file sizes.
• Merge desired state before removing old capacity.
• Drain one node, verify the surviving system, then drain the next.
• Boot the replacement OS in memory before writing disk.
• Confirm device identity from the replacement OS.
• Test a full firmware reboot, not only a kexec reboot.
• Put a file inside a real application volume and read it after reboot.
• Inspect the OS storage layer as well as Kubernetes objects.

Change next time

• Treat Talos multi-document configuration as one indivisible artifact from the start.
• Add a regression check that fails if any expected VolumeConfig or MountStatus disappears after a machine update.
• Land the Argo-managed local-path mapping before changing the observability node selectors, reducing the intentional degraded window.
• Stabilize the LAN registry address before embedding it in machine configuration.
• Decide the remote-access replacement before retiring the jump host if physical access will not be available.
• Use the same talosctl minor version as the nodes during a destructive migration.

A reusable runbook

This is the compact form of the migration, suitable for another single-host hypervisor retirement.

1. Prove the architecture can tolerate removal

kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get applications -n argocd

Do not proceed if the hypervisor contains the only control plane, the only healthy CNI path, or a workload that has no recovery plan.

2. Capture rollback

• export host configuration;
• create full VM archives;
• copy everything off-host;
• generate and verify checksums;
• capture etcd and cluster inventories;
• record the physical power-cycle fallback.

3. Prepare declarative state

• pin the Talos version and schematic;
• validate the installer image and registry path;
• commit non-secret machine patches and volume documents;
• render the secret-bearing full configuration outside Git;
• validate the complete document set;
• merge workload placement only when the maintenance window begins.

4. Remove virtual workers cleanly

kubectl drain <worker-one> --ignore-daemonsets --delete-emptydir-data --force
# Verify the surviving cluster.
kubectl drain <worker-two> --ignore-daemonsets --delete-emptydir-data --force

Shut down the VMs only after both drains return successfully.

5. Boot maintenance mode reversibly

• boot Talos kernel and initramfs from RAM, PXE, ISO, or another reversible path;
• find the maintenance address;
• inspect links, addresses, disks, model, size, and writability;
• compare all of them with the rendered configuration.

6. Cross the irreversible boundary

talosctl apply-config --insecure --nodes <maintenance-ip> --file <full-worker-config>

The file must include every auxiliary block document, not only the main MachineConfig.

7. Admit the node

kubectl wait --for=condition=Ready node/<new-node> --timeout=5m
talosctl --nodes <new-ip> get extensions
talosctl --nodes <new-ip> get volumeconfigs
talosctl --nodes <new-ip> get volumestatus
talosctl --nodes <new-ip> get mountstatus

Remove old node objects only after the replacement is Ready.

8. Recreate local storage deliberately

• point the provisioner at a persistent host path;
• reconcile that mapping from Git;
• quiesce stateful workloads;
• delete only claims whose old nodes and paths are gone;
• wait for new PVs to bind to the intended node and path;
• restore workloads and wait for readiness.

9. Prove persistence

• write a unique file inside a real PVC;
• perform a full power-cycle reboot;
• verify the OS mount source and target;
• verify the node, CNI, workloads, PV paths, and GitOps health;
• read the file after reboot;
• remove the proof file.

10. Close the loop

• delete orphaned node and PV objects;
• retain backups for an explicit period;
• update the living journey;
• record remaining risks without calling them solved;
• schedule replicated storage and remote access as separate changes.

Run the evidence on devata

These commands are read-only and should agree with the end-state table:

kubectl get nodes -o wide
 
kubectl get pv \
  -o custom-columns='PV:.metadata.name,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0],PATH:.spec.hostPath.path'
 
kubectl get applications -n argocd \
  -o custom-columns='NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status'
 
talosctl --nodes 192.168.1.10 get extensions
talosctl --nodes 192.168.1.10 get volumeconfigs
talosctl --nodes 192.168.1.10 get volumestatus
talosctl --nodes 192.168.1.10 get mountstatus

The most important pair is the last two. volumestatus proves Talos found the partition. mountstatus proves Talos mounted that partition at the path Kubernetes is using. Neither should be inferred from the other.

The lesson

Deleting Proxmox simplified the diagram, but a simpler diagram was not the success condition.

The success condition was a chain of claims that could each be demonstrated: the cluster survived two worker drains; the off-host backup checksums matched; maintenance mode identified the right hardware; Talos installed to the intended NVMe; the node rejoined through Cilium; the user volume mounted from the expected partition; Kubernetes provisioned PVs under that mount; the workloads recovered; a real file survived a firmware reboot; and a later verification caught the volume document being dropped before that regression became another month of missing metrics.

The hypervisor was the visible thing removed. Ambiguity was the more important thing removed.

Connected vault notes

• kubernetes for the node, pod, and control-plane vocabulary used here;
• gitops for why the machine-adjacent Kubernetes state was reconciled from Git;
• adopting-kube-prometheus-stack for the monitoring stack whose storage exposed the original bug;
• publishing-the-cluster-snapshot for the off-cluster heartbeat that remains trustworthy even when Prometheus history is not.