Building Talos-Native NetBird Recovery Access

Document contract

• Role: architecture and operations case study with read-only drills
• Scope: the problem, product decision, NetBird mechanics, Talos integration, secret handling, kubelet and Cilium multihoming incidents, staged rollout, kubeconfig contexts, off-LAN proof, diagnostics, and reconstruction; self-hosting NetBird and exposing arbitrary LAN services are outside this page
• Truth boundary: portable concepts plus the completed Devata rollout on 2026-08-13 at lab revisions 4b34290 and 9914372, followed by the Cilium device-selection correction at merge 37baab3
• Last verified: Devata, 2026-08-16; all three nodes on Talos v1.12.11, NetBird 0.66.2, and Kubernetes v1.34.1; Cilium selects only physical interfaces at MTU 1500, and both cloudflared replicas hold four QUIC connections

Prerequisites: talos, kubernetes, cilium, longhorn and velero, reconstructing devata

Devata needed a way to reach its Talos and Kubernetes APIs when Pragalva was away from home. The obvious answer, “run a VPN,” hid the real constraint: there was nowhere independent to run it.

The only always-on computers at home were the bare-metal Talos nodes themselves. There was no Proxmox host, utility VM, router appliance, Raspberry Pi, or separate server. A VPN Pod inside Kubernetes would disappear behind exactly the control-plane, kubelet, CNI, or scheduling failures that remote recovery must help diagnose.

The solution was to put a NetBird peer inside each Talos image as a system extension. That gives every physical node its own encrypted overlay address and exposes only the required management APIs to the administrator workstation.

Keep this question visible while working through the chapter:

What must still be healthy for each management command to reach its target?

The answer changes between talosctl and kubectl, between LAN and NetBird, and between node recovery access and true out-of-band management.

How to work through this chapter

Use six passes. Stop at each checkpoint and explain the system without looking back.

The commands in the drills are read-only unless a warning explicitly says otherwise. Do not recreate the completed failure experiments on the healthy cluster merely to follow the chapter.

Pass 1: Start with the failure boundary

What “remote access” had to mean

The desired capability was narrower than “join the whole home LAN from anywhere.” The administrator workstation needed to reach:

The physical map matters because every rollout and recovery command targets a real machine, not an abstract worker number:

These mappings were rechecked from each node’s Talos SystemInformation resource on 2026-08-13. The NetBird overlay adds another address to each machine but does not change this physical identity.

No workload Service, Pod CIDR, storage network, SSH port, or arbitrary LAN device had to be exported. Each Talos node would be a NetBird peer and receive traffic addressed to its own overlay IP. Devata did not need a subnet router.

That distinction reduces both complexity and authority:

flowchart LR
  workstation[Administrator workstation] -->|TCP 50000| cp[Talos control plane peer]
  workstation -->|TCP 50000| worker1[OptiPlex Talos peer]
  workstation -->|TCP 50000| worker2[Nitro Talos peer]
  workstation -->|TCP 6443| cp

  workstation -. no policy .-> lan[Arbitrary home LAN]
  workstation -. no policy .-> pods[Pod and Service networks]

Why a Pod was the wrong layer

A DaemonSet can place a networking agent on every Kubernetes node, but the process still depends on a chain above the operating system:

Kubernetes API
  -> scheduler and controllers
  -> kubelet
  -> container runtime
  -> CNI networking
  -> VPN Pod

If the CNI is broken, the Pod may run but have no useful network. If the kubelet is down, Kubernetes cannot start or restart it. If the API is unreachable, GitOps cannot repair it. Putting the recovery path there makes the recovery tool share too many failure domains with the thing being recovered.

The Talos extension path is shorter:

Talos kernel and services
  -> NetBird extension service
  -> encrypted overlay
  -> Talos API

Kubernetes can be completely unhealthy while the Talos API and NetBird extension remain available.

Why adding Proxmox would have solved a different problem

If Devata still ran inside Proxmox, a tunnel on the Proxmox host could survive a guest-cluster outage. That would be a sensible host-level recovery boundary.

Devata had deliberately removed that layer. Reintroducing a hypervisor or utility VM only to host remote access would add another OS, lifecycle, failure surface, and recovery procedure. It would reverse part of the bare-metal migration instead of extending Talos at its intended customization boundary.

Why plain WireGuard was not enough here

WireGuard provides the encrypted tunnel. It does not by itself provide all of the coordination needed for peers behind changing addresses, NAT, and CGNAT.

Devata’s home connection cannot reliably accept an unsolicited inbound connection from the public internet. A manually managed WireGuard design would still need a reachable rendezvous or relay, endpoint discovery, key distribution, peer membership, and access rules. Building those missing pieces was not the objective.

NetBird uses WireGuard for peer encryption and adds management, signaling, NAT traversal, relaying when direct connectivity is impossible, peer groups, and policy distribution.

Why NetBird was selected

NetBird matched the actual boundary:

• a supported Talos system extension existed in the v1.12 extension catalog;
• a headless node could enroll with a one-off setup key;
• the workstation and nodes could discover each other through CGNAT;
• direct P2P WireGuard was preferred, with relay available as a fallback;
• groups and unidirectional policies could expose only TCP 50000 and 6443;
• no new always-on LAN machine was required.

Tailscale belongs to the same broad solution class. The decision was not that Tailscale is generally incapable. The implemented path used the NetBird extension and its policy/enrollment model because those matched the Talos v1.12 image and the chosen account configuration.

Checkpoint 1

Why would a healthy VPN DaemonSet fail the recovery requirement even if it worked perfectly during normal operation? Name at least three dependencies it shares with Kubernetes.

Pass 2: Understand what NetBird actually does

Peer, overlay, and underlay

A peer is a machine enrolled in the NetBird network. In this design, the workstation and each physical Talos node are peers.

The underlay is the ordinary network that can carry packets toward the internet: home Ethernet or Wi-Fi, a hotspot, an ISP, NAT devices, and the public internet.

The overlay is the private logical network built over that underlay. NetBird assigns each peer a private overlay address and creates the wt0 WireGuard interface on Linux.

flowchart TB
  subgraph underlay[Underlay networks]
    hotspot[Alternate Wi-Fi or hotspot]
    internet[Public internet and NAT]
    home[Home LAN]
  end

  subgraph overlay[NetBird overlay]
    laptop[Workstation peer wt0]
    control[Talos control-plane peer wt0]
    opti[OptiPlex peer wt0]
    nitro[Nitro peer wt0]
  end

  hotspot --> internet --> home
  laptop <-->|encrypted WireGuard| control
  laptop <-->|encrypted WireGuard| opti
  laptop <-->|encrypted WireGuard| nitro

The overlay does not replace the LAN. Each node remains reachable at its physical 192.168.1.x address at home and gains a separate NetBird address on wt0.

Control plane versus data plane

NetBird has several roles. They should not be collapsed into “the VPN server.”

The preferred data path is peer to peer:

workstation NetBird client
  -> encrypted WireGuard packet
  -> Talos node NetBird client
  -> local Talos or Kubernetes API

The management and signal services coordinate that connection. They are not reverse proxies terminating Talos or Kubernetes traffic. If a relay is needed, the relay carries ciphertext and the WireGuard endpoints remain the peers.

Identity and authorization

Four NetBird groups were relevant to the design:

The policies are intentionally unidirectional from administrators to nodes. A worker does not gain permission to initiate arbitrary traffic to the workstation merely because both are enrolled.

The default full-mesh policy was disabled only after the explicit policies existed. That order prevented an accidental lockout while moving from permissive defaults to least privilege.

Setup keys are enrollment credentials, not tunnel keys

A setup key authorizes a new machine to register without interactive browser login. It is not the long-lived WireGuard private key used for every packet.

For Devata:

• each node received a distinct one-off key;
• the key auto-assigned the peer to either the control-plane or worker group;
• the key was never placed in Git, a PR, an issue, chat output, or shell history;
• the key was streamed into the complete Talos configuration and consumed by first enrollment;
• the node generated and retained its own peer identity after enrollment.

The practical threat is clear: before consumption or expiry, possession of a setup key may allow enrollment with its assigned groups. Treat it as a secret even though it is temporary.

Checkpoint 2

If NetBird reports a peer as registered but talosctl cannot connect, which separate layers still need proof? Include route selection, peer connection, policy, target service, and client authentication.

Pass 3: Place NetBird inside Talos correctly

System extension, not package installation

Talos has no apt, SSH session, or writable general-purpose root filesystem. Host software is added through a system extension embedded in a Talos image.

The Image Factory schematic is the source for that customized image:

customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/netbird

Devata has three hardware-specific schematics because the workers need different additional capabilities:

The schematic content produces a content-addressed ID. That ID and the Talos version identify the installer image used for the upgrade. talosctl get extensions later proves which schematic and extension versions actually booted.

Why the Talos upgrade was necessary

NetBird was absent from the Talos v1.11.5 extension catalog and present in the v1.12.11 catalog. An extension compiled and packaged for a different Talos release was not mixed into the old OS.

The change therefore bundled two related machine-layer operations:

1. move Talos from v1.11.5 to v1.12.11;
2. boot a v1.12.11 installer image containing the NetBird extension.

Kubernetes deliberately remained on v1.34.1. Talos OS and Kubernetes versions are separate lifecycle controls; changing both during this recovery-access rollout would have multiplied the possible causes of failure.

The extension binary and its runtime configuration are separate

The custom Talos image provides the NetBird software. An ExtensionServiceConfig document provides service-specific runtime input:

apiVersion: v1alpha1
kind: ExtensionServiceConfig
name: netbird
environment:
  - NB_SETUP_KEY=REDACTED

Before the upgrade, the configuration document could be applied while the old image was still running. No NetBird service started because the old image did not contain the binary. After the new image booted, Talos discovered both the extension and its named service configuration, started ext-netbird, and performed enrollment.

This separates two questions:

Does the OS image contain NetBird?
                  +
Does Talos have configuration for the netbird extension service?
                  =
Can ext-netbird start and enroll?

Why the complete machine configuration mattered

A Talos machine configuration can contain multiple YAML documents. The main machine document is not necessarily the whole active configuration.

The OptiPlex also depended on auxiliary storage documents for its EPHEMERAL layout and dedicated Longhorn user volume. Applying only a new ExtensionServiceConfig as though it were the complete desired configuration could remove omitted auxiliary documents.

That had already happened in an earlier storage incident: Kubernetes remained superficially healthy while a Talos volume document disappeared and the intended mount was lost.

The safe operation was:

1. retrieve the complete live main configuration;
2. append every required auxiliary document;
3. append the NetBird ExtensionServiceConfig;
4. dry-run the complete multi-document result;
5. verify additions and removals;
6. apply only after zero unintended removals were proven.

flowchart LR
  live[Live main machine config] --> complete[Private complete config]
  storage[Required auxiliary storage docs] --> complete
  netbird[NetBird service config] --> complete
  complete --> dry[Dry-run diff]
  dry -->|only intended addition| apply[Apply without reboot]
  dry -->|removal or mutation| stop[Stop and reconstruct input]

The retrieved configuration contains cluster trust material. Both the configuration and dry-run output remained private, mode 0600, and were shredded after use.

Three sources of truth

Do not confuse the following artifacts:

The public source is necessary for reproducibility. The private complete configuration is necessary for recovery. Runtime inspection is necessary to prove that either one took effect.

Checkpoint 3

Explain why committing siderolabs/netbird to a schematic does not enroll a node, and why applying an ExtensionServiceConfig to a v1.11.5 node does not make the NetBird service appear immediately.

Pass 4: The multihoming incident

What changed when wt0 appeared

Before NetBird, each node had one relevant host address on the physical LAN. After NetBird, the node had at least two:

physical interface -> 192.168.1.x
NetBird wt0        -> overlay address

This is multihoming: one machine participates in multiple networks through multiple addresses or interfaces.

Multihoming is not itself a bug. The first failure came from leaving kubelet free to choose an address from a larger candidate set. A later incident showed that kubelet was not the only automatic consumer that needed an explicit boundary.

The visible symptom

The OptiPlex canary enrolled successfully. NetBird worked in normal kernel mode and its Talos API was reachable over the overlay.

At the same time, Kubernetes began using the node’s NetBird address as its InternalIP. Control-plane-to-kubelet requests, Cilium node addressing, and CSI traffic then followed an address that was not intended to be the cluster’s physical node network. Pods and storage became unhealthy even though the new tunnel itself was functioning.

The causal chain was:

flowchart TD
  extension[NetBird extension starts] --> wt0[wt0 and overlay address appear]
  wt0 --> candidates[Kubelet sees another node-address candidate]
  candidates --> wrong[Kubelet publishes overlay address as InternalIP]
  wrong --> cp[Control-plane to kubelet path changes]
  wrong --> cilium[Cilium node addressing changes]
  wrong --> csi[CSI and storage paths fail]
  cp --> unhealthy[Kubernetes workloads become unhealthy]
  cilium --> unhealthy
  csi --> unhealthy

This is why “the VPN connected” was not a completion gate. A new host interface can change unrelated consumers that automatically select addresses.

Durable fix 1: constrain kubelet address selection

The kubelet was constrained to select its node IP only from the LAN subnet:

machine:
  kubelet:
    nodeIP:
      validSubnets:
        - 192.168.1.0/24

This does not disable wt0, remove the overlay route, or prevent talosctl from reaching the node’s NetBird address. It narrows only kubelet’s node-address selection.

After the patch:

The fix was applied live without reboot, verified in the Node object, committed into both worker machine patches, and documented as a mandatory pre-enrollment gate.

Why kubectl get nodes still shows LAN addresses remotely

When the workstation uses NetBird to reach the Kubernetes API, the API server returns Node objects stored by Kubernetes. Their InternalIP fields describe how the cluster identifies its nodes. They do not describe the path the current client used to reach the API server.

Therefore this output is correct during an off-LAN session:

client route to API: workstation wt0 -> control-plane NetBird address
Node InternalIP:     control plane and workers -> 192.168.1.x

The remote transport and the cluster’s internal node identity are intentionally different.

A second automatic consumer: Cilium device and MTU detection

NetBird’s wt0 interface uses MTU 1280. Cilium had no explicit devices value, so its automatic device detection added both the physical NIC and wt0 to the datapath. Prometheus then showed cilium_host changing from MTU 1500 to 1280 on every node in the same windows where wt0 appeared.

The maximum transmission unit, or MTU, is the largest packet an interface can carry without fragmentation or another packet-size adjustment. The smaller overlay MTU was correct for NetBird, but it should not have become the cluster pod network’s selected device MTU.

Long-lived Cloudflare Tunnel QUIC connections initially survived that change. When those connections later recycled, both cloudflared replicas could resolve DNS, reach the Cloudflare API, and complete HTTP/2 checks over TCP 7844, but new QUIC handshakes over UDP 7844 timed out. Cilium policy allowed the traffic and Hubble observed UDP packets in both directions. The failure was no longer at policy or routing; it was at the QUIC transport after Cilium had adopted the overlay interface and its MTU.

The liveness probe turned that transport failure into a restart loop. /ready returned 503 while the connector had no edge connection, so kubelet terminated cloudflared before its automatic HTTP/2 fallback completed.

flowchart TD
  netbird[NetBird starts] --> wt0[wt0 appears at MTU 1280]
  wt0 --> detect[Cilium auto-detects wt0 as a datapath device]
  detect --> mtu[Cilium host and endpoint MTU fall from 1500 to 1280]
  mtu --> recycle[Existing QUIC connections eventually recycle]
  recycle --> handshake[New QUIC handshakes time out]
  handshake --> unready[cloudflared readiness returns 503]
  unready --> restart[Liveness probe restarts the connector before fallback]

Durable fix 2: keep Cilium on physical interfaces

The repair did not disable NetBird or force Cloudflare Tunnel onto HTTP/2. Cilium’s Helm values now select only the physical NIC naming class:

devices: "enp+"

This keeps wt0 available for Talos recovery while excluding it from Cilium’s NodePort, masquerading, device, and MTU selection. After Argo reconciled lab PR #71:

• each Cilium agent listed only its node’s physical enp* interface;
• Cilium reported MTU 1500 on all three nodes;
• wt0 and the NetBird recovery path remained active;
• both cloudflared replicas became Ready;
• each connector registered four Cloudflare edge connections using QUIC.

The broader rule is not “VPN interfaces are dangerous.” It is: after adding a host interface, enumerate every component that automatically discovers addresses, devices, routes, or MTUs, then constrain each consumer to the network it owns.

Checkpoint 4

NetBird was healthy during both incidents. Why does that rule out “NetBird itself failed” while still leaving its new interface as the trigger for kubelet and Cilium behavior?

Pass 5: Roll out through evidence gates

Why the order mattered

The rollout changed one physical node at a time:

1. OptiPlex worker canary;
2. single control plane;
3. Nitro worker.

The OptiPlex was the safest place to discover image, extension, networking, and storage interactions without risking etcd quorum. The control plane came next only after the kubelet address bug was fixed. Nitro came last because its image also changed NVIDIA extensions and therefore added a hardware-specific verification surface.

Gate 0: prepare access policy and rollback

Before any node changed:

• the workstation was enrolled interactively;
• explicit administrator-to-node policies existed;
• a fresh one-off setup key was created for only the target node;
• the current machine configuration was captured privately;
• the running Talos client version matched the pre-upgrade node;
• the exact Image Factory installer and extension catalog were verified;
• all cluster nodes, workloads, Argo applications, and attached Longhorn volumes were healthy.

An upgrade was never used to repair an already unexplained unhealthy baseline.

Gate 1: worker canary

The OptiPlex gate required proof across multiple layers:

The canary failed the Kubernetes identity gate, which stopped the rollout and produced the validSubnets fix. Only after every downstream surface recovered did the control-plane stage begin.

Gate 2: single control plane

A single control plane is a special risk because there is no second API server or etcd member to carry the cluster while it reboots.

Before upgrade, the off-node rollback package included:

• a complete live machine configuration;
• a fresh, non-empty etcd snapshot;
• the existing Talos client identity and physical/LAN access plan.

After boot, verification included:

• Talos v1.12.11;
• healthy etcd;
• Kubernetes API readiness;
• stable 192.168.1.8 Kubernetes InternalIP;
• all Pods and Argo applications healthy;
• Talos TCP 50000 over the control-plane NetBird peer;
• Kubernetes TCP 6443 over the same peer with correct TLS identity.

Gate 3: Nitro worker

Before Nitro rebooted, every attached Longhorn volume was healthy with two replicas, no replica had a failure timestamp, and Nitro’s complete machine configuration was stored off-node.

After boot:

• Talos reached v1.12.11 and the node returned Ready and schedulable;
• InternalIP remained 192.168.1.9;
• NetBird 0.66.2 started and the Talos API worked through wt0;
• Longhorn rebuilt to zero unhealthy attached volumes and zero failed replicas;
• all Pods became Ready and every Argo application returned Synced/Healthy;
• NVIDIA LTS moved from 535.247.01 to 580.126.20;
• all four NVIDIA modules loaded;
• the device plugin registered one GPU;
• nvidia-smi identified the GTX 1650 Ti, 4096 MiB VRAM, driver 580.126.20, and CUDA 13.0.

That last transaction superseded the earlier conclusion that the GPU could not initialize at the hardware level. It proved adapter initialization and NVML. It did not yet prove a scheduled CUDA workload.

How the Cloudflare observation became a root cause

The Nitro rollout initially produced only a useful correlation. Stopping ext-netbird let the same cloudflared container connect, but that one-variable recovery did not identify which interface, route, firewall rule, or packet property had changed.

The later two-replica outage supplied the missing evidence:

1. Prometheus placed each wt0 appearance beside a Cilium MTU change from 1500 to 1280.
2. Live Cilium status showed wt0 selected as a datapath device alongside the physical NIC.
3. Cloudflared prechecks isolated the failure to QUIC while DNS, TCP 7844, and the Cloudflare API passed.
4. Hubble showed UDP 7844 allowed and returning, ruling out the declared Cilium policy as the drop point.
5. The declarative devices: "enp+" correction removed only wt0 from Cilium selection and restored MTU 1500.
6. Without disabling NetBird or forcing HTTP/2, both cloudflared replicas returned Ready with four QUIC connections each.

The earlier experiment remains valid evidence, but it is no longer the final verdict. The mechanism was Cilium’s automatic adoption of the NetBird overlay device and MTU, followed by a cloudflared liveness loop when QUIC connections recycled.

The final proof had to remove the LAN path

Testing a NetBird IP while still at home proves the overlay route and API listener, but it does not prove that recovery survives loss of direct LAN access.

The final test moved the workstation to an alternate internet connection. At that point:

• the workstation had a different 192.168.1.x address on an unrelated network;
• direct connections to Devata’s LAN addresses on TCP 50000 and 6443 returned unreachable;
• routes to all three NetBird peer addresses used wt0;
• all three peers reported Connected and P2P;
• talosctl version passed against all three overlay addresses;
• Kubernetes /readyz returned ok through the control-plane overlay address;
• kubectl get nodes returned all three Ready nodes;
• unhealthy Pod, Argo application, attached Longhorn volume, and failed replica counts were all zero.

That is transaction proof. It tests the actual recovery path under the condition it was built for.

Checkpoint 5

Why would ping to a peer, a Connected dashboard badge, or a successful test while still on the home LAN each be insufficient as the only acceptance test?

Pass 6: Understand the kubeconfig change

Why k get nodes initially failed off-LAN

k is a shell alias for kubectl. It does not automatically ask NetBird which endpoint to use.

kubectl reads a kubeconfig. The active kubeconfig context still pointed to:

https://192.168.1.8:6443

On the alternate network, that address referred to a device on the current local Wi-Fi or had no reachable host. It was not routed through NetBird. The resulting error was therefore correct:

Unable to connect to the server: dial tcp 192.168.1.8:6443: connect: no route to host

NetBird was healthy. kubectl had simply been told to dial the wrong address for the current location.

Kubeconfig has three reusable object types

flowchart LR
  context[Context] --> cluster[Cluster entry]
  context --> user[User entry]
  context --> namespace[Default namespace]
  cluster --> server[API server URL]
  cluster --> ca[Certificate authority]
  cluster --> tls[TLS server name]
  user --> credentials[Client credentials]

The two Devata contexts reuse the same user identity but select different cluster entries:

Creating a kubeconfig context does not create a Kubernetes cluster, change the API server, change RBAC, or deploy anything. It changes only how this workstation finds and authenticates to an existing API.

Why the NetBird cluster entry needs tls-server-name

The TCP destination and the TLS identity are related but not identical.

The NetBird context dials:

https://<control-plane-netbird-ip>:6443

The Kubernetes API certificate is validated as:

talos-k3t-9cz

Without an explicit TLS server name, a client dialing an IP normally expects that IP to appear in the certificate’s subject alternative names. The overlay IP was not the stable certificate identity used here. Setting tls-server-name: talos-k3t-9cz tells the client to keep strict CA and hostname verification while using the NetBird IP as transport.

Do not replace this with insecure-skip-tls-verify. Encryption without server authentication would make the client unable to prove it reached the intended API server.

What was added to the workstation

The existing kubeconfig was backed up. A second cluster and context were added, while the original LAN context remained intact. In conceptual YAML, the result is:

clusters:
  - name: devata
    cluster:
      server: https://192.168.1.8:6443
      certificate-authority-data: REDACTED
  - name: devata-netbird
    cluster:
      server: https://NETBIRD_CONTROL_PLANE_IP:6443
      tls-server-name: talos-k3t-9cz
      certificate-authority-data: REDACTED
 
contexts:
  - name: pragalva@devata
    context:
      cluster: devata
      user: pragalva
      namespace: argocd
  - name: pragalva@devata-netbird
    context:
      cluster: devata-netbird
      user: pragalva
      namespace: argocd

The certificate and user fields are deliberately redacted. A kubeconfig can contain credentials and must be protected like a secret.

Daily commands

See the current selection:

kubectl config current-context
kubectl config get-contexts pragalva@devata pragalva@devata-netbird

Use NetBird from outside home:

kubectl config use-context pragalva@devata-netbird
kubectl get --raw=/readyz
kubectl get nodes

Use the direct LAN at home:

kubectl config use-context pragalva@devata
kubectl get --raw=/readyz
kubectl get nodes

The LAN context is useful even when NetBird also works at home. Keeping both paths independently selectable makes diagnosis explicit and retains a local path if the external coordination service is unavailable.

How the context was safely constructed

This reconstruction is shown so the mechanism is understandable. It is a mutating workstation operation and should not be rerun while both contexts already exist.

# MUTATING: modifies the current user's kubeconfig.
# Back up ~/.kube/config first and keep the backup mode 0600.
 
CONTROL_PLANE_NETBIRD_IP="<read from the connected NetBird peer>"
DEVATA_CA="$(mktemp)"
trap 'shred -u "$DEVATA_CA"' EXIT
chmod 600 "$DEVATA_CA"
 
# Extract the existing trusted CA without printing credentials.
kubectl config view --raw -o json \
  | jq -r '.clusters[] | select(.name == "devata") | .cluster["certificate-authority-data"]' \
  | base64 -d > "$DEVATA_CA"
 
kubectl config set-cluster devata-netbird \
  --server="https://${CONTROL_PLANE_NETBIRD_IP}:6443" \
  --tls-server-name=talos-k3t-9cz \
  --certificate-authority="$DEVATA_CA" \
  --embed-certs=true
 
kubectl config set-context pragalva@devata-netbird \
  --cluster=devata-netbird \
  --user=pragalva \
  --namespace=argocd
 
kubectl config use-context pragalva@devata-netbird
kubectl get --raw=/readyz

Notice what is reused and what changes:

same Kubernetes CA
same user and client credentials
same API server process
different network destination
explicit TLS identity for that destination

Checkpoint 6

Why did adding NetBird to every node not automatically make the old kubectl context remote-aware? Explain the separate jobs of a route, a kubeconfig server URL, and a TLS server name.

Two APIs, two remote-access tests

The NetBird peer does not collapse Talos and Kubernetes into one API.

Talos API

talosctl talks to TCP 50000 on a specific node. Its client identity comes from talosconfig.

From outside home, a direct one-command test can override both the connection endpoint and target node:

TALOS_NETBIRD_IP="<target-node-netbird-ip>"
 
talosctl \
  --endpoints "$TALOS_NETBIRD_IP" \
  --nodes "$TALOS_NETBIRD_IP" \
  version --short

--endpoints selects where the Talos client establishes its API connection. --nodes selects the node the request targets. They can differ when Talos request forwarding is intentionally used, but direct per-node recovery makes them the same here.

Kubernetes API

kubectl talks to TCP 6443 on the control plane. Its client identity comes from kubeconfig. Once the API request arrives, Kubernetes may report on all nodes and workloads.

This means:

• a working worker Talos peer does not imply the Kubernetes API is healthy;
• a working control-plane Talos peer can help inspect a broken Kubernetes API;
• a working Kubernetes API does not imply every Talos node service or mount is healthy;
• kubectl authentication and Talos authentication remain independent.

Read-only operating drill

Objective

Prove which path is active without changing the cluster.

Environment

• workstation with NetBird connected;
• kubectl, talosctl, netbird, jq, and ip available;
• pragalva@devata-netbird context already configured.

Safety

Every command below is read-only. Do not display kubectl config view --raw, full Talos machine configurations, or setup keys.

Predict

Before running anything, write down:

1. which interface should route the control-plane NetBird address;
2. which server URL the active kubeconfig context should contain;
3. why Node InternalIP values should still be on the LAN;
4. which API can remain useful if Cilium is down.

Observe

kubectl config current-context
 
kubectl config view --minify -o json \
  | jq '{context: .["current-context"], server: .clusters[0].cluster.server, tlsServerName: .clusters[0].cluster["tls-server-name"]}'
 
netbird status --json \
  | jq '[.peers.details[] | select(.fqdn | startswith("talos-")) | {fqdn, status, connectionType}]'
 
CONTROL_PLANE_NETBIRD_IP="<obtain without publishing it>"
ip route get "$CONTROL_PLANE_NETBIRD_IP"
 
kubectl get --raw=/readyz
kubectl get nodes -o wide

Expected observation: the active context is the NetBird context; the server uses the control-plane overlay address; TLS validation names talos-k3t-9cz; the route uses wt0; the API returns ok; all Node InternalIP fields remain on 192.168.1.0/24.

Explain

Say this aloud without using “because VPN”:

The workstation’s kubeconfig selects the control-plane overlay address as its TCP destination. The Linux route sends that destination into wt0. NetBird encrypts the packet to the enrolled control-plane peer. The API server presents a certificate validated as talos-k3t-9cz. Kubernetes then returns Node objects whose internal addresses remain pinned to the physical LAN.

Reconstruct

Close this page and draw five boxes:

kubeconfig context -> cluster server -> Linux route -> NetBird peer -> Kubernetes API

Under each arrow, write the command that proves it.

Failure diagnosis by layer

Start with the exact destination in the error. Do not begin by restarting NetBird or Kubernetes.

A compact diagnostic sequence

1. What exact IP and port did the client dial?
2. Which kubeconfig or talosconfig selected it?
3. Which local interface owns the route?
4. Is the target NetBird peer connected and authorized?
5. Does TCP reach the target API?
6. Does TLS authenticate the intended server?
7. Does the client authenticate and authorize?
8. Is the target API healthy?
9. Are downstream cluster systems healthy?

Stopping at step 4 because the dashboard says Connected would skip most of the system.

Recovery boundaries and honest limitations

This is below Kubernetes, not outside the node

NetBird survives many Kubernetes failures because Talos starts it as an extension service. It does not survive every machine failure.

This is in-band node recovery access. A BMC with independent power and network would be true out-of-band access. Devata currently has no BMC-class path.

The system retains two recovery planes

At home:
  workstation -> LAN IP -> Talos or Kubernetes API
 
Away:
  workstation -> wt0 -> node NetBird IP -> Talos or Kubernetes API
 
At the machine:
  physical console and boot/recovery media

Do not delete the LAN contexts or physical-console knowledge merely because the overlay works.

Reconstruct the design

Authoritative public inputs

The public lab repository holds the secret-free design:

• docs/decisions/0003-talos-native-remote-access.md: why this layer and product were selected;
• talos/netbird/README.md: rollout, gates, and rollback;
• talos/patches/netbird.node-ip.patch.yaml: kubelet LAN address constraint;
• talos/patches/netbird.extensionserviceconfig.example.yaml: secret-free enrollment document shape;
• talos/schematics/: hardware-specific extension sets and Image Factory IDs;
• talos/machineconfigs/: non-secret per-node network, storage, kubelet, and installer intent;
• kubernetes/infra/networking/cilium/values.yaml: physical-interface selection that excludes host overlays from the cluster datapath;
• kubernetes/infra/networking/cilium/README.md: the Cilium device-selection and verification contract.

Private material that must exist off-cluster

Public Git is insufficient for full recovery. Preserve these privately:

• talosconfig and its client identity;
• kubeconfig and its user credentials;
• current complete machine configurations for each node;
• a fresh etcd snapshot before control-plane risk;
• knowledge of NetBird account ownership, peer groups, and policies;
• a way to create a fresh one-off setup key for replacement enrollment;
• physical-console and boot-media procedure.

A consumed setup key is not a durable backup. Recovery requires the ability to issue a new one through the NetBird account.

Rebuild order after a total loss

The broad dependency order is:

flowchart TD
  account[NetBird account, admin peer, groups, policies] --> image[Version-matched Talos images with NetBird]
  private[Private Talos configs and credentials] --> control[Rebuild control plane]
  image --> control
  snapshot[Etcd snapshot when state restoration is chosen] --> control
  control --> workers[Rebuild and enroll workers]
  workers --> storage[Recover Longhorn and application data]
  storage --> gitops[Reconcile GitOps workloads]
  gitops --> proof[Repeat LAN and off-LAN transaction proofs]

The exact choice between restoring etcd and rebuilding cluster state belongs to reconstructing devata. NetBird provides a path to the nodes; it does not decide the disaster-recovery strategy.

Upgrade invariant

Every future Talos image must retain the NetBird extension. Before each node upgrade:

1. verify the target Talos catalog still contains the extension;
2. reproduce the schematic ID;
3. preserve machine.kubelet.nodeIP.validSubnets;
4. preserve Cilium’s physical devices: "enp+" selector;
5. preserve every auxiliary configuration document;
6. capture rollback material;
7. upgrade one node at a time;
8. verify the NetBird service, Cilium device and MTU state, and API transaction before continuing.

An image upgrade that silently omits NetBird can remove remote recovery on reboot even if Kubernetes itself returns healthy.

What the rollout disproved

Check yourself

1. Why is the Talos NetBird extension a better recovery boundary than a Kubernetes DaemonSet in this cluster?
2. Which NetBird components coordinate peer connections, and which endpoints own the WireGuard encryption keys?
3. What is the difference between an underlay address, a NetBird overlay address, and a Kubernetes Node InternalIP?
4. Why did validSubnets repair Kubernetes without disabling remote Talos access?
5. Why must the OptiPlex’s auxiliary volume documents be included when adding an unrelated extension service configuration?
6. What does a one-off setup key authorize, and what persists after it is consumed?
7. Why does the NetBird kubeconfig context reuse the same user and CA but change the cluster server and TLS server name?
8. What exactly does tls-server-name protect against?
9. Why can talosctl work while kubectl fails?
10. Which evidence proves external recovery rather than local overlay reachability?
11. What failure would make both the LAN and NetBird APIs unavailable but still be reachable through a true BMC?
12. Reconstruct the path of kubectl get nodes from an alternate network, naming every identity and routing decision.

References

• NetBird: How NetBird Works
• NetBird: Register Machines Using Setup Keys
• Talos v1.12: Image Factory
• Talos v1.12: Editing Machine Configuration
• Talos v1.12: Configuration Patches
• Talos v1.12: Upgrading Talos Linux
• Kubernetes: Organizing Cluster Access Using kubeconfig Files
• Lab PR #67: Add Talos-native NetBird recovery access
• Lab PR #68: Pin kubelet node IP for NetBird
• Lab PR #71: Keep Cilium on physical interfaces
• Vault PR #82: Final alternate-network proof