Bringing devata's imperative stack under GitOps

Prerequisites: bootstrapping-argocd, helm-application, adoption, app-of-apps

Historical migration record

The inventory and commands in this chapter describe the start of the GitOps adoption, when three Argo Applications coexisted with imperative infrastructure. The migration is complete, and a current helm list -A is empty because Argo owns the chart-backed platform. Read this page for the adoption method and decision order. Use devata-system-map and reconstructing-devata for current state and rebuild procedure.

bootstrapping-argocd left devata with a working engine: Argo CD reconciling itself and one disposable hello app from Git, with the app-of-apps root watching kubernetes/clusters/devata/ for more. At this chapter’s starting point, everything that actually ran the cluster was still imperative. The walkthrough closed that gap one component at a time until Git described the whole cluster. It carried the roadmap’s GitOps backbone past one proven app to the whole stack.

What is still imperative

Ask the cluster what it runs outside Argo:

argocd app list                           # today: argocd, devata-root, hello
helm list -A                              # the Helm releases git does not yet own
kubectl -n metallb-system get deploy,ds   # installed with no Helm release at all

At the time of writing, three Argo Applications exist (the engine, its root, and the hello demo) against this imperative stack:

The homes come from the repo conventions: one concern per directory under kubernetes/infra/, and each adopted component a child application under kubernetes/clusters/devata/ that the app-of-apps root already watches.

The per-component move

For every Helm component the move is the same five steps, and it is the two concept notes joined. The exact command-by-command version, the one to keep open while you actually do it, is adopting-a-helm-component; the summary here is the map of why each step exists.

1. Recover the running values. helm get values <release> -n <ns> into kubernetes/infra/<concern>/<name>/values.yaml.
2. Write the child application with a Helm source (see helm-application): the upstream chart, the version pinned to exactly what helm list -A shows, and that values file.
3. Commit on a branch, open a PR, merge to main. Argo CD reads main, not your working tree, so nothing reconciles until the files are on the branch the Applications track.
4. Adopt it (see adoption): automation off, argocd app diff until empty, then turn on selfHeal and prune.
5. Prove it with a drift test, then retire the leftover Helm release Secret.

The root app-of-apps notices each new child file under clusters/devata/ and creates the Application; you never kubectl apply a component by hand again. MetalLB is the one exception to step 2, because it has no chart; its variant is below.

The order, least blast radius first

Adopt in an order where a mistake costs the least, and leave the things Argo itself depends on for last.

1. nvidia-device-plugin, the safe rehearsal

The GPU is a confirmed hardware fault (see the current-state baseline), so this DaemonSet advertises nothing and does nothing, which makes it the ideal first adoption: a real Helm chart whose failure cannot hurt the cluster. Run the five-step move on it end to end to learn the diff-to-empty loop on something disposable before touching anything that matters.

2. kube-prometheus-stack, the stopping rule and the first secret

This is the component the roadmap names as the backbone’s stopping rule: when monitoring is Argo-managed and survives a drift test, the pattern is proven. It has its own full walkthrough, adopting-kube-prometheus-stack, because it is the first adoption that is not plain. Note the release is named kps, not kube-prometheus-stack, so the Application, its folder, and every helm get values kps -n monitoring use kps; the naming rule and why it matters are in adopting-a-helm-component. Two more specifics make it more than a plain adoption:

• It ships CRDs over the client-side size limit, so set ServerSideApply=true on the Application from the start (see adoption).
• Its Grafana admin password is currently a value kept in a file off-cluster, not in git, and it must never enter git as plaintext. This is the point where secrets management stops being optional. The clean answer is a sealed-secrets controller (a child in infra/controllers/, its own chapter), after which the password lives in git encrypted and the chart reads it through grafana.admin.existingSecret. Until that controller exists, migrate the rest of kube-prometheus-stack and keep the Grafana secret as a manually-created Secret the values reference by name, so nothing secret is committed. When you do install the controller, back its key up off-cluster the same day: a rebuilt Talos cluster will not regenerate it, and without it every sealed secret becomes undecryptable. Controller and key caveat: sealed-secrets.

3. Loki and Promtail, the logging pair

Two ordinary Helm adoptions in the logging namespace, the same five-step move. No CRD size issue, no secret. Do them after monitoring so you migrate the higher-volume observability data while the pattern is fresh.

4. MetalLB, the one that is not Helm

MetalLB was installed from raw manifests, not a chart, so helm-application does not apply to it. Vendor the pinned upstream manifest into infra/networking/metallb/ (the install YAML for the version you run, from the MetalLB install docs), and beside it commit the two custom resources that hold devata’s actual config, recovered from the live cluster:

kubectl get ipaddresspool lan-pool -n metallb-system -o yaml      # the 192.168.1.240-250 pool
kubectl get l2advertisement lan-adv -n metallb-system -o yaml

Strip the live-only fields (status, resourceVersion, uid, creationTimestamp, managedFields) before committing. The child Application points at the folder as a plain directory source. One ordering note: the pool and advertisement are custom resources whose CRDs ship in the same manifest, and on a first sync Argo may try to apply the CRs before their CRDs are established. If it does, give the manifest sync-wave 0 and the two CRs sync-wave 1 with the argocd.argoproj.io/sync-wave annotation, so the CRDs land first. Reference: sync waves.

5. Cilium, last, because Argo rides on it

Cilium is the dataplane every pod uses, including Argo CD itself. That makes it the most dangerous to adopt: a bad render or a destructive sync can drop cluster networking, and then the controller that would fix it has no network either. This is the same chicken-and-egg that makes Argo manage itself with prune: false in bootstrapping-argocd, so adopt Cilium with matching caution:

• Recover the exact values with helm get values cilium -n kube-system, and pin targetRevision to 1.18.6.
• Keep prune: false, and leave selfHeal off until you have watched one clean manual sync.
• Drive argocd app diff to genuinely empty before any sync. For Cilium, a non-trivial diff is a values mismatch to fix in git, never something to apply blindly.
• Keep the imperative escape hatch ready: helm rollback cilium can undo a bad sync from outside Argo, as long as networking survives enough to run it.

Enabling Gateway API later (gatewayAPI.enabled=true) is a values change in this same file that triggers a Cilium rollout, which is exactly why it belongs in this Application rather than a manual helm upgrade. That change, and devata’s first Gateway (currently zero), are the ingress chapter, not this one.

From hand-written children to an ApplicationSet

Once all five are migrated, clusters/devata/ holds one near-identical child Application per folder under infra/. That repetition is what an ApplicationSet removes: a directory generator over kubernetes/infra/* templates one Application per folder automatically, which is the pattern the lab README and app-of-apps both anticipate. It is a refactor to make once the components are all adopted and the shape has stopped changing, not something to reach for mid-migration. Reference: ApplicationSet.

When devata is fully GitOps

The stopping rule for the whole effort, the thing that says the roadmap’s backbone is done:

• helm list -A no longer shows any release Argo does not own (the records are retired), and nothing in kubernetes/’s scope is running without a manifest behind it.
• argocd app list shows every component Synced and Healthy.
• A drift test passes on each: hand-edit a value, watch Argo revert it.
• The honest end-to-end proof: standing the stack up on a throwaway kind cluster from kubernetes/bootstrap/root.yaml alone reconstructs the whole thing from git, with no remembered helm install.

At that point the gap bootstrapping-argocd left is closed: git is the single source of truth, drift is corrected automatically, and a rollback is a git revert. The OS layer stays out of it, applied by talosctl, for the reason in gitops.