Installing Sealed Secrets and sealing the first real secret

8 min read

Prerequisites: sealedsecret, kubeseal, application, helm-application, app-of-apps, adopting-kube-prometheus-stack

This walkthrough installs the sealed-secrets controller on devata as an Argo CD child Application, proves the seal/unseal round-trip on a throwaway secret, then seals the first real credential: the grafana-admin password the kps adoption left as manual debt. It is lab issue #8. Unlike the migration chapter this is a fresh install, not an adoption: nothing is running yet, there are no live values to recover, and the git side can be prepared before ever touching the cluster.

Every credential the roadmap needs later rides on this: the Cloudflare API token for cert-manager, the GitHub token for the snapshot publisher. Encrypt-then-commit has to exist before any of them can enter the repo.

Step 1: discover and pin the chart

The controller ships as a Helm chart from its own repo, bitnami-labs/sealed-secrets:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm repo update
helm search repo sealed-secrets/sealed-secrets   # note CHART VERSION and APP VERSION

Pin the chart version the search shows, exactly as targetRevision. APP VERSION is the controller version; match your kubeseal download to it in step 5.

Step 2: the two files

The standard shape from adopting-a-helm-component step 2, minus the recovery work. First kubernetes/infra/controllers/sealed-secrets/values.yaml, which is one line:

fullnameOverride: sealed-secrets-controller

That line is the whole configuration, and it is there for kubeseal. The CLI’s compiled-in defaults look for a controller named sealed-secrets-controller in kube-system; a Helm release named sealed-secrets would render objects named sealed-secrets, and every kubeseal invocation forever after would need --controller-name flags. Overriding the rendered name and installing into kube-system makes the bare command work.

Then the child Application, kubernetes/clusters/devata/sealed-secrets.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://bitnami-labs.github.io/sealed-secrets
      chart: sealed-secrets
      targetRevision: <version from step 1>
      helm:
        valueFiles:
          - $values/kubernetes/infra/controllers/sealed-secrets/values.yaml
    - repoURL: https://github.com/PragalvaXFREZ/lab.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system
  syncPolicy:
    # automated:                    # uncomment after step 3 verifies the controller
    #   selfHeal: true
    #   prune: true

No CreateNamespace (kube-system exists), no ServerSideApply (the CRD is small). Automation starts off, not because adoption caution applies here, but to keep one deliberate moment between “git describes a controller” and “a controller that will hold every future credential’s key is running”: you review what the sync will create, then pull the trigger yourself.

Commit both on a branch, PR, merge to main, per lab repo policy.

Step 3: sync by hand, watch the key be born

After the merge the root app-of-apps creates the app, automation off, nothing applied:

argocd app get sealed-secrets     # OutOfSync: Argo knows what it would create
argocd app sync sealed-secrets
kubectl -n kube-system get pods -l app.kubernetes.io/name=sealed-secrets   # Running

On first start the controller generates its key pair and logs it. Look at both the log line and the object:

kubectl -n kube-system logs deploy/sealed-secrets-controller | grep -i "new key"
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key

That Secret is the private key: the single object that can ever decrypt anything this cluster seals. Which makes the next step the most important one in this walkthrough.

Step 4: back up the key, today

sealedsecret has the reasoning: a rebuilt Talos cluster generates a fresh key and every SealedSecret in git becomes undecryptable ciphertext. The backup happens the same day as the install, before the first real secret is sealed, so there is never a sealed credential whose key exists in only one place.

mkdir -p ~/devata-secrets && chmod 700 ~/devata-secrets
kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml \
  > ~/devata-secrets/sealed-secrets-keys-$(date +%F).yaml
chmod 600 ~/devata-secrets/sealed-secrets-keys-*.yaml

The label selector, without =active, captures every key the controller has, which is what you want once renewal has produced more than one. Three rules for the backup:

  1. Never into a git repo. It is the private key; committing it would undo the entire system.
  2. Not only on this laptop. A second copy somewhere that survives the laptop: a USB stick, or an attachment in your password manager.
  3. Refresh after renewal. The controller mints a new key every 30 days (sealedsecret); a backup from before a renewal cannot decrypt secrets sealed after it. Re-run the two commands when the key Secret count grows.

~/devata-secrets/ is also the local home the old dashboard folder’s leftovers have been waiting for (vault issue #12): move machineconfig-backups/ and the Meshery backup in here and the folder’s last blocker but one is gone.

Read the restore before you need it (docs): on a rebuilt cluster, kubectl apply -f the backup file into kube-system before or after installing the controller, then restart it (kubectl -n kube-system rollout restart deploy/sealed-secrets-controller) so it picks the keys up. That is the whole procedure; the point of reading it now is knowing the backup file alone is sufficient.

Now flip automation on: uncomment the automated: block, commit, merge. From here the controller is GitOps-managed like everything else.

Step 5: install kubeseal, save the certificate

Match the controller’s APP VERSION from step 1:

KUBESEAL_VERSION=<app version, e.g. 0.27.1>
curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
tar -xzf "kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz" kubeseal
sudo install -m 755 kubeseal /usr/local/bin/kubeseal && rm kubeseal*

Then fetch the public certificate and commit it next to the controller’s values:

kubeseal --fetch-cert > ~/Desktop/projects/lab/kubernetes/infra/controllers/sealed-secrets/pub-cert.pem

The cert is public material, safe in the public repo, and committing it means future sealing works with --cert pub-cert.pem from anywhere, including away from the homelab network (kubeseal).

Step 6: round-trip a throwaway secret

Prove the loop end to end on something worthless before trusting it with something real:

kubectl create secret generic seal-test -n default --from-literal=hello=world \
  --dry-run=client -o yaml | kubeseal -o yaml > /tmp/seal-test.yaml
 
cat /tmp/seal-test.yaml                  # kind: SealedSecret, hello: <ciphertext>
kubectl apply -f /tmp/seal-test.yaml
kubectl -n default get secret seal-test -o jsonpath='{.data.hello}' | base64 -d; echo   # world

Ciphertext went in, the controller unsealed it, a real Secret came out. Clean up by deleting the SealedSecret and watch the ownerReference cascade take the Secret with it:

kubectl -n default delete sealedsecret seal-test
kubectl -n default get secret seal-test    # NotFound

Step 7: seal grafana-admin for real

This step assumes the grafana-admin Secret from adopting-kube-prometheus-stack step 1 exists in monitoring; if the kps adoption has not run yet, create that Secret per that note first.

Two preparations. First, the controller refuses to overwrite a Secret it does not own, so the existing hand-made object must consent to takeover:

kubectl -n monitoring annotate secret grafana-admin sealedsecrets.bitnami.com/managed=true

Second, seal the same password under the same name and namespace (strict scope bakes both in). The awk extracts the password line from the four-line local notes file; the reason it must, and the silent failure if it does not, are documented in adopting-kube-prometheus-stack:

cd ~/Desktop/projects/lab
mkdir -p kubernetes/infra/observability/kps/secrets
kubectl create secret generic grafana-admin -n monitoring \
  --from-literal=admin-user=admin \
  --from-literal=admin-password="$(awk -F': ' '/^password:/{print $2}' ~/Desktop/homelab-dashboard/.grafana-admin-password)" \
  --dry-run=client -o yaml | kubeseal -o yaml \
  > kubernetes/infra/observability/kps/secrets/grafana-admin.yaml

Now wire it into GitOps. Nothing applies files sitting in infra/observability/kps/ on its own (the kps Application’s ref: values source only serves the values file), so the SealedSecret needs a source that renders it: add a third, directory source to kubernetes/clusters/devata/kps.yaml:

    - repoURL: https://github.com/PragalvaXFREZ/lab.git
      targetRevision: main
      path: kubernetes/infra/observability/kps/secrets

Before pushing, the same review gate as the kps adoption: read the diff, and confirm the only new data leaving your machine is ciphertext:

git diff                                          # the sealed file plus three lines in kps.yaml
grep -c AgB kubernetes/infra/observability/kps/secrets/grafana-admin.yaml   # ciphertext present
grep -ri "$(awk -F': ' '/^password:/{print $2}' ~/Desktop/homelab-dashboard/.grafana-admin-password)" kubernetes/ | wc -l   # 0

Commit, PR, merge. Argo syncs kps, applies the SealedSecret, and the controller overwrites the annotated Secret with its own managed copy. Verify nothing about Grafana changed and ownership moved:

argocd app get kps                                # Synced, one new SealedSecret resource
kubectl -n monitoring get secret grafana-admin -o jsonpath='{.metadata.ownerReferences[0].kind}'; echo   # SealedSecret
kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.admin-password}' | base64 -d | wc -c  # same length as before

Log in to Grafana once. Then the local .grafana-admin-password file has no remaining job: git now carries the credential in sealed form, the cluster carries it live, and the key backup from step 4 guarantees a rebuild can recover it. Delete the file, and with it the last content blocker on retiring ~/Desktop/homelab-dashboard/ entirely.

What this closes and opens

Issue #8’s done-when is met: kubeseal output committed to git round-trips into a live Secret, and the controller key backup exists off-cluster with its restore procedure read. Operating principle 6, secrets never go in raw, is now machinery instead of a rule to remember. Every later credential follows step 7’s shape: annotate if taking over an existing Secret, dry-run pipe through kubeseal, commit ciphertext, let a directory source apply it. The kps adoption’s last debt is paid, and cert-manager’s Cloudflare token (lab #10) and the snapshot publisher’s GitHub token (lab #12) have their path into the repo.