Adopting kube-prometheus-stack: the kps walkthrough

11 min read

Prerequisites: adopting-a-helm-component, adoption, helm-application

This is adopting-a-helm-component instantiated for devata’s monitoring stack, the component the roadmap names as the GitOps backbone’s stopping rule. It is written as its own note because kps is the first adoption that is not plain. Three things separate it from the nvidia rehearsal:

  1. The release is named kps, not kube-prometheus-stack, and the Application must be named after the release.
  2. The chart ships CRDs too large for client-side apply, so the Application needs ServerSideApply=true from the first sync.
  3. The recovered values contain a plaintext password. This is the point where secrets management stops being optional, and most of this note is about handling it correctly.

Everything else is the standard drill and is not repeated here; keep the runbook open beside this note.

Step 0: read the live state, and find the trap

helm list -n monitoring                  # kps, chart kube-prometheus-stack-86.2.0
helm get values kps -n monitoring        # the overrides that become values.yaml
kubectl get deploy,ds,sts -n monitoring  # what "unchanged" looks like later

The live objects are kps-grafana, kps-prometheus-node-exporter, prometheus-kps-kube-prometheus-stack-prometheus, and so on. The chart embedded the release name kps into every one of them, which is why the Application, its folder, and its child file must all be named kps; the full reasoning is in adopting-a-helm-component step 2.

Now look closely at the helm get values output. Near the top sits:

grafana:
  adminPassword: <the actual Grafana admin password, in the clear>

For every previous component the recovery command produced a file you could commit as-is. Here it produces a file you must not commit. Committing it would put the password into git history, and git history is forever: a later commit that deletes the line removes it from the tree, not from history, and every clone of the repository carries every past commit. Scrubbing a secret out of git history means rewriting history on every copy, which in practice means treating the password as burned and rotating it. The only winning move is to never commit it.

What a Secret actually is

Kubernetes stores this kind of value in a Secret object. Read the one the chart rendered for Grafana:

kubectl -n monitoring get secret kps-grafana -o yaml

The data: fields look scrambled, but they are base64, an encoding, not encryption. Anyone who can read the object can decode it:

kubectl -n monitoring get secret kps-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo

That prints the admin password. A Secret’s protection is access control (RBAC decides who can read it, and it lives only in etcd and in the containers that mount it), not cryptography. This is fine inside the cluster and catastrophic in a public git repository, which has no access control at all. Reference: Secrets.

Count where this password currently lives: the kps-grafana Secret, the local ~/Desktop/homelab-dashboard/.grafana-admin-password file, and inside every sh.helm.release.v1.kps.v* record in the namespace, because each Helm revision record embeds the full values of its revision. All of those are on the cluster or on your disk. The adoption must not add “the public lab repo” to that list.

Step 1: get the password out of the values, before any GitOps

The strategy: make the live release stop carrying the password in its values first, imperatively, with one last helm upgrade. Then the standard adoption drill runs unmodified on values that are clean, and diff-to-empty works exactly as the runbook describes. Decoupling the two changes also means each one is separately verifiable and separately reversible.

The chart supports this directly: instead of adminPassword, the Grafana subchart can read credentials from a Secret you own via admin.existingSecret (grafana chart docs). First create that Secret by hand, with the same password, so nothing about the running system changes:

kubectl -n monitoring create secret generic grafana-admin \
  --from-literal=admin-user=admin \
  --from-literal=admin-password="$(awk -F': ' '/^password:/{print $2}' ~/Desktop/homelab-dashboard/.grafana-admin-password)"

The local file is not a bare password, it is a four-line notes file (URL, user, password, rotated date), so the awk extracts only the value of its password: line. A --from-file here, or a plain $(cat ...), would stuff all four lines into the Secret as the password, and the mistake is silent: the Secret creates fine and nothing complains until a login fails. After creating it, verify the content is exactly the password before moving on:

kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.admin-password}' | base64 -d | wc -l   # 0: one line, no trailing newline

The keys admin-user and admin-password match the chart’s defaults so no key mapping is needed.

Now build the values file where it will live in the lab repo, so the file you upgrade with and the file git tracks are literally the same file:

cd ~/Desktop/projects/lab
mkdir -p kubernetes/infra/observability/kps
helm get values kps -n monitoring > kubernetes/infra/observability/kps/values.yaml

Edit that file three ways, keeping everything else exactly as recovered:

  1. Delete the USER-SUPPLIED VALUES: first line. It is helm get values framing that rode along with the redirect, not a value; it parses as a stray top-level key and does not belong in git.
  2. Delete the adminPassword: line.
  3. In its place add:
grafana:
  admin:
    existingSecret: grafana-admin

The nesting is load-bearing and its failure mode is silent. Helm does not validate values keys against the chart, so existingSecret: placed directly under grafana: is not an error, it is simply ignored: the upgrade succeeds, the chart keeps rendering its own kps-grafana Secret, and the Deployment keeps reading the old credentials, which the verification below is what catches. Then run the upgrade, pinned to the running chart version, with -f and the full file:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade kps prometheus-community/kube-prometheus-stack --version 86.2.0 \
  -n monitoring -f kubernetes/infra/observability/kps/values.yaml

Use -f with the complete file, never --reuse-values --set ...: reuse merges on top of the stored values, so adminPassword would survive inside the release record and helm get values would keep showing it.

Verify all four consequences:

helm get values kps -n monitoring | grep -i password        # nothing
kubectl -n monitoring get secret kps-grafana                 # NotFound: chart stopped rendering it, Helm removed it
kubectl -n monitoring get deploy kps-grafana -o yaml | grep -B1 -A3 secretKeyRef   # now points at grafana-admin
kubectl -n monitoring get pods -l app.kubernetes.io/name=grafana   # one fresh pod, Running

Log in to Grafana with the old password. It works for two reasons: the manual Secret carries the same password, and Grafana only reads GF_SECURITY_ADMIN_PASSWORD when it first initialises its database; with persistence enabled the real credential lives in grafana.db on the PersistentVolume and survives every rollout regardless.

The live state is now secret-free in its values, and one object in the namespace, grafana-admin, is managed by hand rather than by any chart. That is the one piece of debt this adoption leaves; the last section is about paying it.

Step 2: the two files

The standard shape from the runbook, filled in. The values file already exists from step 1. The child Application goes in kubernetes/clusters/devata/kps.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kps
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://prometheus-community.github.io/helm-charts
      chart: kube-prometheus-stack
      targetRevision: 86.2.0
      helm:
        valueFiles:
          - $values/kubernetes/infra/observability/kps/values.yaml
    - repoURL: https://github.com/PragalvaXFREZ/lab.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: monitoring
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    # automated:                    # uncomment only after the diff is empty
    #   selfHeal: true
    #   prune: true

The one line that is new against the nvidia child is ServerSideApply=true. The chart’s CRDs (PrometheusRule, ServiceMonitor, and friends) are bigger than the 256KiB annotation that client-side apply uses to track state, so without it the first sync fails on the CRDs; adoption has the full mechanism. Reference: server-side apply.

Pre-merge: what to check before it leaves your machine

The review gate matters more here than for any previous component, because a mistake is public and permanent. On the branch, before pushing:

git diff --cached                                    # read every line you are about to publish
grep -ri adminpassword kubernetes/                    # must print nothing
grep -c existingSecret kubernetes/infra/observability/kps/values.yaml   # 1

And eyeball the four facts against the cluster one last time: Application name kps equals the release name, targetRevision: 86.2.0 equals what helm list shows, the valueFiles path matches where the file actually sits, and the automation block is still commented out. Then commit and open the PR as usual:

git checkout -b adopt-kps
git add kubernetes/infra/observability/kps kubernetes/clusters/devata/kps.yaml
git commit -s -m "Adopt kube-prometheus-stack (kps) into GitOps"

Post-merge: diff to empty

After the merge, the root app-of-apps creates the kps Application, automation off, nothing applied. Because step 1 already made the live values match the committed file, the expected result is the same immediate success the nvidia rehearsal showed:

argocd app get kps       # Synced, Healthy, before any sync has run
argocd app diff kps      # empty

If the diff is not empty, read it line by line. A line traceable to a value is a recovery mistake: fix values.yaml, commit, diff again. One residue is expected on this chart specifically: the operator’s admission webhook patch job injects a caBundle certificate into the live ValidatingWebhookConfiguration and MutatingWebhookConfiguration after install, and git renders that field empty, so those lines can never converge by editing values. If every remaining diff line is a caBundle, that is not a mistake, it is a field the cluster legitimately owns; tell Argo to ignore it on the Application:

  ignoreDifferences:
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle

Reference: diffing customization. Only once the diff is genuinely empty, uncomment the automated: block, commit, merge. The first automated sync is a no-op against the running stack; Prometheus keeps scraping and Grafana keeps serving throughout, because an empty diff means Argo has nothing to change.

Note what prune: true now protects rather than threatens: grafana-admin is safe from pruning because Argo never rendered or tracked it; pruning only removes objects Argo itself once applied.

Post-merge: prove it, then destroy the plaintext copies

The drift test, on the Deployment this chart does ship:

kubectl -n monitoring set env deploy/kps-grafana DRIFT_TEST=1
argocd app get kps        # OutOfSync, then self-healed back to Synced

Watch Argo revert it, then retire the Helm release records:

kubectl -n monitoring get secret -l owner=helm,name=kps    # one record per revision
kubectl -n monitoring delete secret -l owner=helm,name=kps

This is more than bookkeeping here. Every pre-upgrade revision record still embeds the old values, plaintext password included, so deleting the records is also the step that destroys those on-cluster copies. After it, the password exists in exactly two places: the grafana-admin Secret and the local file, both off git.

This is the roadmap’s stopping rule met: monitoring is Argo-managed and survives a drift test, the pattern is proven on the component that matters.

Encrypting for real: where the manual Secret goes next

grafana-admin is correct but unmanaged: it is not in git, so a rebuilt cluster would not recreate it, and kubectl get secret is the only record it exists. The fix is sealed-secrets (lab issue #8), and the idea is worth having in your head now even though installing it is its own adoption:

  • The controller runs in-cluster and holds a private key. The kubeseal CLI encrypts a Secret with the matching public certificate, producing a SealedSecret custom resource that only that controller can ever decrypt.
  • A SealedSecret is safe to commit publicly. Argo applies it from git, the controller decrypts it in-cluster into the real Secret, and encrypt-then-commit becomes the normal way any credential enters the repo.
  • The day the controller is installed, back its private key up off-cluster. A rebuilt Talos cluster generates a new key and cannot decrypt anything sealed under the old one; without the backup, every SealedSecret in git becomes ciphertext with no key.

When that lands, grafana-admin gets re-created as a SealedSecret in infra/observability/kps/, the local .grafana-admin-password file becomes deletable, and the last blocker on retiring the old dashboard folder goes with it.

The rest of the stack, same drill or not

  • Loki and Promtail: the same drill with none of this note’s complications. No oversized CRDs, no secret in the values, ordinary release names. Recover, two files, merge, diff to empty, automate, drift test, retire the record; the runbook alone covers them.
  • MetalLB: not Helm at all, so the drill does not apply; its directory-source variant is in migrating-the-imperative-stack.
  • Cilium: the drill applies but with the blast-radius caveats in migrating-the-imperative-stack; it stays last because Argo itself rides on it.

When those are done, helm list -A shows nothing Argo does not own, and the migration chapter’s stopping rule takes over.