Adoption: taking over a running workload without recreating it

Prerequisites: application, reconciliation, helm-application

devata already runs Cilium, MetalLB, and the monitoring and logging stacks, all installed by hand. Bringing them under git must not mean deleting and recreating them: that would drop Prometheus history, restart every pod, and risk an outage on the live cluster. Adoption is the property that makes the migration safe. When you create an application whose rendered manifests match objects that already exist, Argo CD does not recreate anything. It runs the same compare step as any other reconciliation, sees that git and the live objects are identical, and reports the app Synced and Healthy having changed nothing on the cluster.

So migrating a running component is not “deploy it”. It is “make Argo’s render match what is already running, then hand it the keys”. The whole job is driving the diff to empty.

The procedure

Create the Application with automation off, so nothing is applied until you have looked at what would change. Leave out the automated: block entirely:

  syncPolicy:
    syncOptions: [ CreateNamespace=true ]   # no automated block yet

Then read the difference between git and the live cluster:

argocd app get <app>          # Synced already means the render matched exactly
argocd app diff <app>         # exactly what Argo would change; empty means done

If the render matches the live objects exactly, the app reports Synced with an empty diff from the moment it is created, before any sync has run; that is the compare step finding nothing to do, which is adoption working. Otherwise every line in the diff is something your committed manifests do not yet match. For a Helm component it is almost always a value you set at install time and have not copied into git, recovered with helm get values (see helm-application), or a renamed resource because the Application name does not match the original release name. Adjust, commit, diff again, and repeat until empty. Then enable automation:

  syncPolicy:
    automated: { selfHeal: true, prune: true }

and sync once. Because the diff was empty, that first sync is a no-op against the workload; from then on the loop defends it.

Helm ownership metadata

A Helm-installed object carries the label app.kubernetes.io/managed-by: Helm and the annotations meta.helm.sh/release-name and meta.helm.sh/release-namespace. Argo CD adopts the object regardless; it marks ownership with its own app.kubernetes.io/instance label, the two markers coexist, and argocd app diff normalizes Argo’s own tracking label out of the comparison, so a fully recovered component reads as an empty diff.

What does not go away on its own is the Helm release record, the sh.helm.release.v1.<name>.v<n> Secret sitting in the namespace. Once Argo is authoritative it is vestigial, but it is a loaded gun: a stray helm upgrade would fight Argo, and a helm uninstall would delete the very workload Argo just adopted. The lab repo conventions say hand-applied state is drift to be removed, so once the diff is clean and Argo is in control, retire the release record without touching the workload:

# deletes only Helm's bookkeeping, not the running objects
kubectl delete secret -n <ns> -l owner=helm,name=<release>

Never reach for helm uninstall to do this. It deletes the resources, not just the record.

ServerSideApply, the one that bites on big charts

Client-side apply stores a copy of the whole manifest in a last-applied-configuration annotation, and Kubernetes caps any annotation at 262144 bytes. kube-prometheus-stack ships CRDs larger than that, so adopting it client-side fails with Too long: must have at most 262144 bytes. The fix is server-side apply, where the API server tracks field ownership instead of stuffing the manifest into an annotation:

  syncPolicy:
    syncOptions:
      - ServerSideApply=true

See the sync-options docs. Server-side apply also takes over fields previously owned by Helm more cleanly, so it is the safer default whenever you adopt a resource another controller created, not only when you hit the size limit.

Proving adoption worked

The real test is the drift test from reconciliation: change a live value by hand and watch Argo pull it back.

kubectl -n <ns> scale deploy <something> --replicas=9   # for a Deployment
kubectl -n <ns> set env ds/<name> DRIFT_TEST=1          # for a DaemonSet
# Argo reverts either to what git renders, because selfHeal is on

When that revert happens against a workload you never recreated, the component is genuinely under git, not merely re-deployed.