Bootstrapping GitOps on devata with Argo CD

8 min read

Prerequisites: kubernetes, gitops, reconciliation, argocd, application, app-of-apps

Historical build record

This walkthrough preserves the cluster state and first-install reasoning from the Argo CD bootstrap. devata is no longer imperative. The current platform is reconciled through the lab repository, and the rebuild anchor is the pinned Kustomize input under kubernetes/bootstrap/argocd. Use reconstructing-devata for current recovery order. Treat the commands below as dated evidence unless a safety boundary explicitly says they are repeatable.

At this walkthrough’s starting point, devata was an imperative cluster. Cilium, MetalLB, the monitoring stack, the logging stack, and the Gateway API had been installed by hand, and nothing in Git described them. The first change did not try to fix all of that at once. It installed the engine, argocd, proved the reconciliation loop on one disposable app, and left the cluster reconciling Git going forward. Migrating the existing infrastructure became a later job, one component at a time, because adopting gitops on a running cluster is safest in small reversible steps.

The original session used the commands below. Have kubectl pointed at the intended disposable or recovery target and the argocd CLI installed before adapting them.

kubectl config current-context     # expect: pragalva@devata
argocd version --client

Step 0: confirm the starting point

Prove to yourself that Argo CD is not already there, so you know exactly what each later step changed.

kubectl get ns argocd               # expect: NotFound
kubectl get crd | grep argoproj.io  # expect: nothing

Both empty. This is a clean install, and the argocd namespace not existing yet is the marker for it.

Step 1: the one manual install

This is the single imperative action in the whole process. You apply Argo CD’s install manifest by hand, because there is no reconciler yet to apply it for you. Everything after this step is driven by git.

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

The stable ref is fine to get going, but pin a real release for anything you intend to keep. Pick the latest from the releases page and use its tag in the URL, for example .../argo-cd/v3.0.6/manifests/install.yaml. Pinning means the version is a fact recorded in your commands and later in git, not whatever stable happened to point at the day you ran it.

What this did: it created the workloads from argocd (the application-controller, repo-server, server, redis) and registered the argoproj.io CRDs, including the application kind. Watch them come up.

kubectl get pods -n argocd -w

Wait until every pod is Running and ready. The argocd-application-controller-0 StatefulSet pod is the reconciler; the rest support it.

Step 2: log in

The api-server is not exposed outside the cluster yet (ingress and TLS are a later chapter), so reach it with a port-forward.

kubectl -n argocd port-forward svc/argocd-server 8080:443

In another terminal, read the initial admin password. Argo CD writes it to a secret on first install.

argocd admin initial-password -n argocd
# or:
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

Log in with the CLI, and open the same thing in a browser at https://localhost:8080 if you want the UI.

argocd login localhost:8080 --username admin --password <the-password> --insecure
argocd app list      # expect: no apps yet

--insecure is only because the port-forward presents Argo CD’s self-signed certificate; it is local traffic to your own machine. Change the admin password (argocd account update-password) once you are in.

Step 3: prepare the repo paths

The lab repo already has the directories this needs, currently holding only READMEs. You are going to put the root app-of-apps under kubernetes/bootstrap/ and the child Applications under kubernetes/clusters/devata/, exactly as the repo’s conventions intend. Work on a branch and open a PR, per the repo’s policy.

Create a throwaway workload for the children to manage, so the first thing Argo CD reconciles is something you can safely break. Under kubernetes/apps/hello/, add deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  namespace: hello
spec:
  replicas: 1
  selector:
    matchLabels: { app: hello }
  template:
    metadata:
      labels: { app: hello }
    spec:
      containers:
        - name: hello
          image: nginx:1.27
          ports:
            - containerPort: 80

Step 4: write the root and its first child

Under kubernetes/clusters/devata/, add hello.yaml, a child application that points at the workload you just wrote.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/PragalvaXFREZ/lab.git
    targetRevision: main
    path: kubernetes/apps/hello
  destination:
    server: https://kubernetes.default.svc
    namespace: hello
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true

Under kubernetes/bootstrap/, add root.yaml, the root app-of-apps that points at the folder of children.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: devata-root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/PragalvaXFREZ/lab.git
    targetRevision: main
    path: kubernetes/clusters/devata
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

Commit these, open the PR, and merge it, so main actually contains them. Argo CD reads from git, not from your working tree, so nothing reconciles until the files are on the branch the Applications track.

Step 5: apply the root, once

This is the second and last manual kubectl apply. After it, git is the source of truth.

kubectl apply -n argocd -f kubernetes/bootstrap/root.yaml

Watch what unfolds without you doing anything else.

argocd app list
argocd app get devata-root
argocd app get hello

The root syncs, sees hello.yaml in clusters/devata/, and creates the hello child Application. The child syncs, sees kubernetes/apps/hello, creates the hello namespace (because of CreateNamespace=true), and applies the Deployment. You ran one kubectl apply and a Deployment you never applied directly is now running.

kubectl get all -n hello

Step 6: prove reconciliation against git

You have read about the loop; now watch it defend the cluster. Drift the live state by hand.

kubectl -n hello scale deployment hello --replicas=5
kubectl get pods -n hello -w

For a moment there are five pods, then Argo CD pulls it back to one. Git says replicas: 1, selfHeal is on, so the live cluster is corrected to match git. This is the same defence the Deployment controller gave the pod in reconciliation, now run by Argo CD against git instead of against etcd. Try to delete the workload entirely.

kubectl delete deployment hello -n hello
kubectl get deploy -n hello -w

It comes back, because git still declares it. The only way to actually remove hello is to remove its declaration: delete kubernetes/apps/hello/ and clusters/devata/hello.yaml from git, and because prune: true, Argo CD deletes the live objects to match. Drift is corrected; intent expressed through git is honoured.

Step 7: let Argo CD manage itself

Argo CD is still the one thing not under git, since you installed it by hand in step 1. Close that gap by vendoring its install into the repo and adding a child Application for it. Put the pinned install manifest at kubernetes/bootstrap/argocd/install.yaml (download the exact version you applied), then add kubernetes/clusters/devata/argocd.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/PragalvaXFREZ/lab.git
    targetRevision: main
    path: kubernetes/bootstrap/argocd
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: false      # never let Argo CD prune itself out of existence

Commit it. The running root sees the new child and Argo CD adopts its own install. From now on, upgrading Argo CD is editing the version in bootstrap/argocd/ and letting the running copy apply it. prune: false is deliberate, for the reason in app-of-apps: a stray deletion in git must not be able to tear the controller down with nothing left to rebuild it.

Step 8: the boundary, restated on the live cluster

Argo CD now reconciles kubernetes/. It does not, and must not, touch the operating system. The Talos machine layer (the OS image, kernel extensions, node network config) stays applied by talosctl, outside the reconciler, for the reason given in gitops: the controller runs on the cluster and cannot rebuild the cluster’s own nodes. And lab-experiments/ is outside the watched path entirely, so you can keep breaking things there without Argo CD fighting you. The line is: kubernetes/ is reconciled and authoritative, talos/ is versioned but applied by hand, lab-experiments/ is ignored.

Where this leaves devata, and what comes next

devata now has a working GitOps engine reconciling one app from git, and Argo CD managing itself. The existing imperative infrastructure (Cilium, MetalLB, monitoring, logging) is untouched and still imperative. Bringing it under git is the next stretch of work, done one component at a time so each move is reversible: render or locate each component’s manifests, commit them under kubernetes/infra/, add a child Application in clusters/devata/, and let Argo CD adopt the already-running workload. Two things to handle before that is comfortable: secrets, which must be encrypted before they enter git (a sealed-secrets or SOPS controller in infra/controllers/, its own chapter), and the repetition of writing one child Application per component, which is where an ApplicationSet replaces the hand-written files. Clean up the demo when you are done with it by deleting kubernetes/apps/hello/ and clusters/devata/hello.yaml from git and letting the prune remove it.

Current successor

The adoption work described as next is complete. The current bootstrap creates the Argo namespace, applies the pinned Kustomize package with kubectl apply -k kubernetes/bootstrap/argocd, waits for the Application CRD, and applies kubernetes/bootstrap/root.yaml. The root then reconciles the platform children, including Cilium, observability, sealed secrets, Longhorn, and Velero. devata-system-map describes the resulting ownership planes.