We often talk about Infrastructure as Code (IaC) which means every aspect of the infrastructure should be in a code and reproducible.
In the Kubernetes ecosystem, we use various projects to keep manifests in code, whether in plain YAML, Kustomize, or Helm. However, one component that is often manually created and not stored as code is the Secret.
Let’s take a quick example.
$ cat secret.yaml apiVersion: v1 data: foo: YmFy # This is a base64 encoded bar kind: Secret metadata: name: mysecret namespace: default
Now anyone with access to this secret can just run base64 decode and obtain the secret value:
$ echo "YmFy" | base64 -d bar
This leads to manual management of secrets, and we don’t achieve 100% Infrastructure as Code.
This tutorial will guide you on how to enhance secure Kubernetes management with Sealed Secrets. Additionally, we will dive into installation, usage, and best practices for a robust Infrastructure-as-Code approach.
Introducing Sealed Secrets
Sealed Secrets is an open-source project started at Bitnami and is used to encrypt Kubernetes secrets. Once encrypted, you can store this encrypted secret in your Git repository safely. It allows DevOps practices without exposing sensitive data.
Only the Sealed Secrets controller can decrypt these encrypted secrets.
How do Sealed Secrets work?
By following these steps, users can ensure the secure management and deployment of their Kubernetes secrets.
- Encryption: The user encrypts Kubernetes secrets using a Sealed Secrets controller.
- Storage: The Sealed Secrets are safely stored in Git.
- Deployment: The user deploys the Sealed Secrets to the Kubernetes cluster.
- Decryption and Creation: The Sealed Secrets controller decrypts the sealed secrets and creates a Kubernetes secrets resource on the cluster.
The Sealed Secrets controller in the Sealed Secrets ecosystem is responsible for watching for Sealed Secret custom resources. When it detects one, it decrypts the enclosed secret using its private key and then creates a standard Kubernetes Secret. Its significance lies in its ability to manage and decrypt Sealed Secrets securely, ensuring that only the cluster with the corresponding private key can access the original secret content.
Before we begin, there are some prerequisites you’ll need to have in place:
Installing Sealed Secrets through Civo Marketplace
There are two ways to install the Sealed Secrets through Civo Marketplace: the UI or the CLI. This section will explore how to install Sealed Secrets using both methods.
Civo Marketplace UI
After successfully creating the Kubernetes cluster, log in to the Civo dashboard and go to "Kubernetes" > "Kubernetes Cluster" > "Marketplace," as shown in the screenshot below:
In the Marketplace section, select the “Architecture” tab, and click “Sealed Secrets”. Click “Install Apps” from here to install the Sealed Secrets application inside the cluster.
For more information, check out the Git Repository here.
If you want to install the controller separately, you can run the following:
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.23.1/controller.yaml
The following resources will be created in the
- serviceaccount sealed-secrets-controller
- Deployment sealed-secrets-controller
- CustomResourceDefinition sealedsecrets.bitnami.com
- Service sealed-secrets-controller
- RBAC resources
Once you’ve installed Sealed Secrets, we will also need something on the client side to encrypt the secrets and for other operations. For this purpose, the project has a
kubeseal command-line utility which we will install. It's used to encrypt Kubernetes Secrets, turning them into "Sealed Secrets" which can be safely stored in version control systems such as Git.
Installing Kubeseal on Mac
To install Kubeseal on Mac, you can run the following command:
brew install kubeseal
Installing Kubeseal on Linux
To install Kubeseal on Linux, you can run the following command:
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-v0.24.0-linux-amd64.tar.gz tar -xvzf kubeseal-v0.24.0-linux-amd64.tar.gz kubeseal sudo install -m 755 kubeseal /usr/local/bin/kubeseal
Till now, we have installed a sealed secrets controller and
Before we start encrypting secrets, we need to understand the concept of scopes.
The scope is nothing but the context or visibility of a sealed secret within a Kubernetes cluster. The scope of a Sealed Secret relates to where and how the Sealed Secret can be decrypted and used within your cluster.
There are 3 types of scopes in Sealed Secrets:
|strict||The name and namespace of the secret are included in the encrypted data. Therefore, you must seal the secret using the same name and namespace.|
|namespace-wide||You can freely rename the sealed secret within a given namespace.|
|cluster-wide||You have the flexibility to unseal the secret using any name and in any namespace.|
strict scope is selected unless you pass the
--scope flag to kubeseal CLI with a different value.
Sealed Secrets Use Case
In this section, we will walk through a practical example of converting a secret from a file to a Sealed Secret. This step-by-step guide will provide a clearer understanding of the use of Sealed Secrets in Kubernetes.
Before diving into the commands, let's understand the workflow. We will take a
secret.yaml file, convert it into a Sealed Secret, and then apply this Sealed Secret to a Kubernetes cluster. The Sealed Secret will then be unsealed by the controller in the cluster, creating a usable secret.
Ensure you have a
secret.yaml file as used in earlier sections. This file contains the secret data in base64 encoded format.
Converting Secret to Sealed Secret
Step 1: Run the
kubeseal command to convert the
secret.yaml file to a Sealed Secret. The
-f flag specifies the input file, and the
-w flag writes the output to
kubeseal -f secret.yaml -w sealedsecret.yaml
This command will generate a new file,
sealedsecret.yaml, containing the encrypted secret.
Step 2: Review the Sealed Secret file
sealedsecret.yaml file to review the contents. It should look similar to the example below:
apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: mysecret namespace: default spec: encryptedData: foo: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....
This file is safe to store in version control systems as it is encrypted.
Applying the Sealed Secret
Step 1: Apply the Sealed Secret
kubectl apply command to apply the
sealedsecret.yaml file to your Kubernetes cluster.
kubectl apply -f sealedsecret.yaml
This command will create a Sealed Secret in the cluster.
Step 2: Verify the Sealed Secret
Confirm the creation of the Sealed Secret and the corresponding secret by running the following commands:
kubectl get sealedsecrets kubectl get secrets
These commands will list the Sealed Secrets and secrets in the cluster, respectively.
Unsealing the Secret
Step 1: Describe the Sealed Secret
kubectl describe command to view the details of the Sealed Secret.
kubectl describe sealedsecrets mysecret … In the events section, you should see a message indicating that the Sealed Secret was unsealed successfully: … Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Unsealed 67s sealed-secrets SealedSecret unsealed successfully
This message confirms that the Sealed Secret has been successfully converted back into a secret in the cluster.
Creating and Converting a Secret Simultaneously
In some scenarios, you may want to create a secret and convert it to a Sealed Secret in one go. Follow the steps below to achieve this:
Step 1: Create a Secret
kubectl create secret command to create a secret and output it to a
secret.yaml file. The
--dry-run option ensures the command doesn’t execute but returns the output as YAML.
kubectl create secret generic mysecret –from-literal=foo=bar –dry-run -o yaml > secret.yaml
Step 2: Convert the Secret to a Sealed Secret
kubeseal command to convert the
secret.yaml file to a
kubeseal -f secret.yaml -o sealedsecret.yaml
Step 3: Apply the Sealed Secret
sealedsecret.yaml to your Kubernetes cluster.
kubectl apply -f sealedsecret.yaml
This sequence of commands creates a secret and converts it to a Sealed Secret efficiently.
Managing Existing Secrets
If you are installing Sealed Secrets on an existing cluster with pre-existing secrets, follow the steps below:
Step 1: Annotate the existing Secret
This annotation marks the secret for management as a Sealed Secret.
Updating Keys in Existing Sealed Secrets
In instances where you need to update or add new values to an existing Sealed Secret, follow the steps below:
Step 1: Append New Secret Value
Use the following command to append a new value to the existing Sealed Secret without updating the entire secret.
echo -n baz | kubectl create secret generic mysecret --dry-run=client --from-file=bar=/dev/stdin -o yaml | kubeseal --merge-into sealedsecret.yaml -o yaml
This command adds a new value to the
Validating Sealed Secrets Locally
Before applying Sealed Secrets to the cluster, it’s crucial to validate the Sealed Secrets file to ensure its correctness.
Step 1: Validate the Sealed Secrets File
kubeseal --validate command to validate the
cat sealedsecret.yaml | kubeseal –validate
If the Sealed Secret is not valid, the command will return an error message:
cat sealedsecret.yaml | kubeseal --validate error: unable to decrypt sealed secret
This validation step helps in avoiding the application of invalid Sealed Secrets to the cluster.
If you run a sealed secrets controller in a namespace other than kube-system, you will have to pass the
--controller-namespace flag every time you run the
kubeseal commands. Deploying Sealed Secrets outside the
kube-system namespace can enhance security by isolating it from critical cluster components and offer more granular control through namespace-specific configurations and RBAC policies.
Alternatively, you can set the environment variable
Through this tutorial, we’ve learned how to leverage Sealed Secrets to securely store secrets in Git.
Sealed Secrets has become essential software to complete the story of Infrastructure as a code and GitOps.
If you have any issues or doubts, feel free to reach out to me on Twitter.
If you want to know more about using Sealed Secrets, take a look at these resources given here: