Few months back, I had a crucial decision to make, turn my homelab into baremetal Talos cluster or allocate memory to proxmox in a constrained enviroment. The only thing rational holding me back was a vm that was running tailscale in it.
From that day onwards, since I turned bare-metal. I had no access to my homelab when I was outside. I have a fairly modest uptime (apart from some electricity outbreaks) but being locked out didn’t sit right with me. The homelab being pre-production didn’t mean I could just let it be broken, therefore here we are. This blog is all about what I did to access my homelab when i touch grass.
Figure: This strip is rendered from commit-driven snapshots published by an in-cluster service to pragalva.me/homelab. It is an inferred activity signal, not an availability monitor or production SLO. A missing commit can indicate several failures besides cluster downtime.
Figuring out the right tool
Simply said, I could’t indepently run a VPN, there was no Proxmox host, utility VM, raspberry pi or seprate server. All the devices I had handy were part of my bedroom lab - configured with Talos.
A VPN pod inside kubernetes would not work either, as it depends on a chain above the operating system with higher failure domains:
Kubernetes API -> scheduler and controllers -> kubelet -> container runtime -> CNI networking -> VPN Pod
Therefore I needed a tool, that would still work when Kubernetes is unhealthy, given that talosctl works completely fine.
I could have used cloudfare tunnels or wireguard, but neither did I want the full internet to be able to access my private network, nor did I want too much hassle. That narrowed down the hit list by a margin.
Let’s welcome NetBird to the picture.
What is NetBird and why was it chosen ?
I got lazy here so here is what the official site says: “NetBird is an open source platform consisting of a collection of components, responsible for handling peer-to-peer connections, tunneling, authentication, and network management (IPs, keys, ACLs, etc).
It uses open-source technologies like WireGuard®, Pion ICE (WebRTC), Coturn, and softwaredeveloped by NetBird to make secure private networks deployment and management simple.”
Why Netbird? Because it matched the actual boundary:
- a supported Talos system extension existed in the v1.12 (this was an undersight, more regarding it in bonus section) extension catalog;
- a headless node could enroll with a one-off setup key;
- the workstation and nodes could discover each other through CGNAT;
- direct P2P WireGuard was preferred, with relay available as a fallback;
- groups and unidirectional policies could expose only TCP 50000 and 6443;
- no new always-on LAN machine was required.
What does the bird do ?
Firstly, let’s understand some key concepts:
A peer is a machine enrolled in the NetBird network. In this design, the workstation and each physical Talos node are peers.
The underlay is the ordinary network that can carry packets toward the internet: home Ethernet or Wi-Fi, a hotspot, an ISP, NAT devices, and the public internet.
The overlay is the private logical network built over that underlay. NetBird assigns each peer a private overlay address and creates the wt0 WireGuard interface on Linux.
Note that: The overlay does not replace the LAN. Each node remains reachable at its physical 192.168.1.x address at home and gains a separate NetBird address on wt0.
flowchart TB subgraph underlay[Underlay networks] hotspot[Alternate Wi-Fi or hotspot] internet[Public internet and NAT] home[Home LAN] end subgraph overlay[NetBird overlay] laptop[Workstation peer wt0] control[Talos control-plane peer wt0] opti[OptiPlex peer wt0] nitro[Nitro peer wt0] end hotspot --> internet --> home laptop <-->|encrypted WireGuard| control laptop <-->|encrypted WireGuard| opti laptop <-->|encrypted WireGuard| nitro
NetBird has several roles. They should not be collapsed into “the VPN server.”
| Component | Responsibility | Does application traffic normally pass through it? |
|---|---|---|
| management | authenticates peers, assigns overlay addresses, stores public peer state, distributes policies and network maps | no |
| signal | helps peers exchange connection candidates | no |
| relay | carries encrypted peer traffic when a direct path cannot be established | only when needed |
| client | owns the peer’s WireGuard key, creates wt0, applies routes and policies, and establishes peer connections | yes, at each endpoint |
The prefered data path is peer to peer. The managment and signal services coordinate that conncetion. They are not reverse proxies terminating Talos or kubernetes traffic. If a relay is needed, the relay carries ciphertext and the WireGuard endpoints remain the peers.
3 peer groupds were created in netbird, not just for namespacing. Devata-admins -> my workstation -> approved outbound traffinc to the homelab control plane -> Talos TCP 50000 and Kubernetes TCP 6443 destinations workers -> Talos TCP 50000 destinations
Policies were created unidirectional deliberately. A worker does not gain permission to initiate arbitary traffic to the workstation.
Placing NetBird inside Talos correctly
Netbird was added as a system extension, not a simple package intallation as done in ubuntu.
I started my homelab about a year ago, and a talos version was not granted yet, and the talos version we were currently at didn’t support netbird, so we had to do the due-diligence as a pre-requiste to get it right.
Machine configuration Previously, when installed an applications, things were pretty simple, create an argocd manifest, pin it watch it reconsile and the work was pretty much done most of the time.
As mentioned above, this was not aan application, it’s a system extension.
- talos image configuration
- ExtensionSpecificConfig
During the installation process, these documentes were edited, both had different functionality. The extension file unlocked the runtime pathway which was merged first.
Why? This had no affect as the talos image (the prev version) did not support the netbird application, therefore the previous was a no-op untill this was cleared.
We need to understand that the talos configration could contain multiple YAML documents, which infact did exist in my context.
Evidence to add
Link the exact Talos image configuration and explain the multi-document patch boundary.
Post Installation and testing off-lan access
When I thought that the installation was finished, things started getting tricky, I hadn’t covered each corners that should’ve been. This section will be covered in Part 2 of this blog, I will work on it this weekend. But, I will share a small obeservation that I hadn’t noticed previously.
The concepts of kubeconfig and contexts were not particularly new to me, I had worked with them during my time at meshery. But, being used to tailscale previously, this error message caught be offguard:
Unable to connect to the server: dial tcp 192.168.1.8:6443: connect: no route to host
while talosctl was working perfectly fine.
Explanation to add
Explain why
talosctlcould use the overlay while the kubeconfig still targeted the LAN API address.
Soo, this was it for today, stay tuned for the next blog. I promise it’s going to more interesting. We’ll figure out some :bugs together.