Prerequisites: gitops, adopting-kube-prometheus-stack
The kps adoption ended with a deliberate piece of debt: the grafana-admin Secret is correct on the cluster but exists nowhere in git, because a Secret cannot be committed (its data: fields are base64, an encoding anyone can reverse; the kps note’s “What a Secret actually is” section demonstrates it). That leaves a contradiction with gitops: git is supposed to describe the whole cluster, and rebuilding from git is supposed to work, but the one object that holds a credential is invisible to it. SealedSecrets is the standard resolution: bitnami-labs/sealed-secrets, a controller plus a custom resource that make encrypted secrets a normal, committable Kubernetes object.
The design
The system is asymmetric encryption split across two halves:
- In the cluster runs the sealed-secrets controller. On first start it generates an RSA key pair and stores it as a Secret in its own namespace. The private key never leaves the cluster.
- On your workstation runs kubeseal, the client CLI. It takes an ordinary Secret manifest, encrypts each data field with the controller’s public certificate, and emits a
SealedSecret: a custom resource whose spec is nothing but ciphertext.
Only the controller’s private key can decrypt what the public certificate encrypted, so a SealedSecret is safe to publish anywhere: the public lab repo, a pastebin, a billboard. Git only ever holds ciphertext.
The unseal loop closes in-cluster. The controller watches SealedSecret resources; when one appears (applied by argocd from git, or by hand), it decrypts the fields and creates the real Secret, setting an ownerReference on it. The relationship is the same one a Deployment has to its Pods: the SealedSecret is the declared source object, the Secret is derived runtime state. Delete the SealedSecret and the owned Secret is garbage-collected with it; hand-edit the Secret and the controller’s next reconcile puts it back. From git’s point of view the Secret does not exist, only its sealed form does.
Scopes
Encryption binds more than the value. In the default strict scope, the Secret’s name and namespace are mixed into the encryption as authenticated data, so the ciphertext is only valid for exactly that name in exactly that namespace. Move the same ciphertext into a manifest with a different name or namespace and the controller refuses to unseal it. This blocks a replay attack: without it, anyone who can read the public repo and can create objects in some namespace of the cluster could copy your ciphertext into their own namespace and have the controller decrypt your credential for them.
Two looser scopes exist, selected at sealing time: namespace-wide (rename freely within the namespace) and cluster-wide (unseal anywhere). They trade that protection for convenience; nothing on devata needs them, so strict stays the default. Details: scopes.
A practical consequence of strict scope: the final name and namespace must be decided before sealing, because they are baked into the ciphertext.
Key renewal
The controller does not keep one key forever. Every 30 days by default it generates a fresh key pair and starts sealing new secrets against it, while keeping every old private key for decryption. Existing SealedSecrets in git therefore keep working without resealing; renewal only means the newest public certificate is the one kubeseal should fetch. It also means the set of private keys grows over time, which matters for the next point.
The key is the cluster’s memory
Every private key lives only as a Secret in etcd on this one cluster. Talos nodes are immutable and a cluster rebuild starts the controller from scratch: it generates a brand-new key and cannot decrypt anything sealed under the old ones. At that moment every SealedSecret in git, the entire point of the system, becomes ciphertext with no key in existence. Backing up the controller’s keys off-cluster is therefore not an optional hardening step, it is part of the install, done the same day, and refreshed after renewals. The procedure is in installing-sealed-secrets.
What it does not protect
SealedSecrets secures exactly one path: the trip through git. On the cluster the unsealed object is an ordinary Secret, readable by anyone RBAC lets read Secrets, base64 and all. It is not a vault, not encryption at rest for etcd, and not protection against a compromised cluster admin. The threat it removes is the one the kps adoption ran into: a credential in a public repository, where there is no access control at all.