Sealed secrets: Securely storing Kubernetes secrets in Git

We often talk about Infrastructure as Code (IaC), which means every aspect of the infrastructure should be in code and reproducible.

In the Kubernetes ecosystem, we use various projects to keep manifests in code, whether in plain YAML, Kustomize, or Helm. However, one component that is often manually created and not stored as code is the Secret.

Let’s take a quick example.

1$ cat secret.yaml
2apiVersion: v1
3data:
4  foo: YmFy # This is a base64 encoded bar
5kind: Secret
6metadata:
7  name: mysecret
8  namespace: default

Now, anyone with access to this secret can just run base64 decode and obtain the secret value:

1$ echo "YmFy" | base64 -d
2bar

This leads to manual management of secrets, and we don’t achieve 100% Infrastructure as Code.

This tutorial will guide you on how to enhance secure Kubernetes management with Sealed Secrets. Additionally, we will dive into installation, usage, and best practices for a robust Infrastructure-as-Code approach.

Introducing sealed secrets

Sealed Secrets is an open-source project started at Bitnami and is used to encrypt Kubernetes secrets. Once encrypted, you can store this encrypted secret in your Git repository safely. It allows DevOps practices without exposing sensitive data.

Only the Sealed Secrets controller can decrypt these encrypted secrets.

How do sealed secrets work?

By following these steps, users can ensure the secure management and deployment of their Kubernetes secrets.

• Encryption: The user encrypts Kubernetes secrets using a Sealed Secrets controller.
• Storage: The Sealed Secrets are safely stored in Git.
• Deployment: The user deploys the Sealed Secrets to the Kubernetes cluster.
• Decryption and creation: The Sealed Secrets controller decrypts the sealed secrets and creates a Kubernetes secrets resource on the cluster.

The Sealed Secrets controller in the Sealed Secrets ecosystem is responsible for watching for Sealed Secret custom resources. When it detects one, it decrypts the enclosed secret using its private key and then creates a standard Kubernetes Secret. Its significance lies in its ability to manage and decrypt Sealed Secrets securely, ensuring that only the cluster with the corresponding private key can access the original secret content.

Prerequisites

Before we begin, there are some prerequisites you’ll need to have in place. This tutorial assumes a basic understanding of Kubernetes concepts, plus:

• A Civo account
• Kubernetes cluster launched

Installing sealed secrets

Install the Sealed Secrets through Civo CLI.

If you want to install the controller separately, you can run the following:

1kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.23.1/controller.yaml

The following resources will be created in the kube-system namespace:

• serviceaccount sealed-secrets-controller
• Deployment sealed-secrets-controller
• CustomResourceDefinition sealedsecrets.bitnami.com
• Service sealed-secrets-controller
• RBAC resources

Installing Kubeseal

Once you’ve installed Sealed Secrets, we will also need something on the client side to encrypt the secrets and for other operations. For this purpose, the project has a kubeseal command-line utility which we will install. It's used to encrypt Kubernetes Secrets, turning them into "Sealed Secrets" which can be safely stored in version control systems such as Git.

Installing Kubeseal on Mac

To install Kubeseal on Mac, you can run the following command:

1brew install kubeseal

Installing Kubeseal on Linux

To install Kubeseal on Linux, you can run the following command:

1wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-v0.24.0-linux-amd64.tar.gz
2tar -xvzf kubeseal-v0.24.0-linux-amd64.tar.gz kubeseal
3sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Till now, we have installed a sealed secrets controller and kubeseal CLI. Before we start encrypting secrets, we need to understand the concept of scopes.

Understanding scopes

The scope is nothing but the context or visibility of a sealed secret within a Kubernetes cluster. The scope of a Sealed Secret relates to where and how the Sealed Secret can be decrypted and used within your cluster.

There are 3 types of scopes in Sealed Secrets:

By default, strict scope is selected unless you pass the --scope flag to kubeseal CLI with a different value.

Sealed secrets use case

In this section, we will walk through a practical example of converting a secret from a file to a Sealed Secret. This step-by-step guide will provide a clearer understanding of the use of Sealed Secrets in Kubernetes.

Overview

Before diving into the commands, let's understand the workflow. We will take a secret.yaml file, convert it into a Sealed Secret, and then apply this Sealed Secret to a Kubernetes cluster. The Sealed Secret will then be unsealed by the controller in the cluster, creating a usable secret.

Initial Setup

Ensure you have a secret.yaml file as used in earlier sections. This file contains the secret data in base64 encoded format.

Converting secret to sealed secret

Step 1: Run the kubeseal command

Use the kubeseal command to convert the secret.yaml file to a Sealed Secret. The -f flag specifies the input file, and the -w flag writes the output to sealedsecret.yaml.

1kubeseal -f secret.yaml -w sealedsecret.yaml

This command will generate a new file, sealedsecret.yaml, containing the encrypted secret.

Step 2: Review the sealed secret file

Open the sealedsecret.yaml file to review the contents. It should look similar to the example below:

1apiVersion: bitnami.com/v1alpha1
2kind: SealedSecret
3metadata:
4  name: mysecret
5  namespace: default
6spec:
7  encryptedData:
8    foo: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....

This file is safe to store in version control systems as it is encrypted.

Applying the sealed secret

Step 1: Apply the sealed secret

Use the kubectl apply command to apply the sealedsecret.yaml file to your Kubernetes cluster.

1kubectl apply -f sealedsecret.yaml

This command will create a Sealed Secret in the cluster.

Step 2: Verify the sealed secret

Confirm the creation of the Sealed Secret and the corresponding secret by running the following commands:

1kubectl get sealedsecrets
2kubectl get secrets

These commands will list the Sealed Secrets and secrets in the cluster, respectively.

Unsealing the secret

Step 1: Describe the sealed secret

Use the kubectl describe command to view the details of the Sealed Secret.

1kubectl describe sealedsecrets mysecret
2…
3
4In the events section, you should see a message indicating that the Sealed Secret was unsealed successfully:
5
6…
7Events:
8  Type    Reason    Age   From            Message
9  ----    ------    ----  ----            -------
10  Normal  Unsealed  67s   sealed-secrets  SealedSecret unsealed successfully

This message confirms that the Sealed Secret has been successfully converted back into a secret in the cluster.

Creating and converting a secret simultaneously

In some scenarios, you may want to create a secret and convert it to a Sealed Secret in one go. Follow the steps below to achieve this:

Step 1: Create a secret

Use the kubectl create secret command to create a secret and output it to a secret.yaml file. The --dry-run option ensures the command doesn’t execute but returns the output as YAML.

1kubectl create secret generic mysecret –from-literal=foo=bar –dry-run -o yaml > secret.yaml

Step 2: Convert the secret to a sealed secret

Use the kubeseal command to convert the secret.yaml file to a sealedsecret.yaml file.

1kubeseal -f secret.yaml -o sealedsecret.yaml

Step 3: Apply the sealed secret

Apply the sealedsecret.yaml to your Kubernetes cluster.

1kubectl apply -f sealedsecret.yaml

This sequence of commands creates a secret and converts it to a Sealed Secret efficiently.

Managing existing secrets

If you are installing Sealed Secrets on an existing cluster with pre-existing secrets, follow the steps below:

Step 1: Annotate the existing secret

1sealedsecrets.bitnami.com/managed: "true"

This annotation marks the secret for management as a Sealed Secret.

Updating keys in existing sealed secrets

In instances where you need to update or add new values to an existing Sealed Secret, follow the steps below:

Step 1: Append new secret value

Use the following command to append a new value to the existing Sealed Secret without updating the entire secret.

1echo -n baz | kubectl create secret generic mysecret --dry-run=client --from-file=bar=/dev/stdin -o yaml | kubeseal --merge-into sealedsecret.yaml -o yaml

This command adds a new value to the sealedsecret.yaml file.

Validating sealed secrets locally

Before applying Sealed Secrets to the cluster, it’s crucial to validate the Sealed Secrets file to ensure its correctness.

Step 1: Validate the sealed secrets file

Use the kubeseal --validate command to validate the sealedsecret.yaml file.

1cat sealedsecret.yaml | kubeseal –validate

If the Sealed Secret is not valid, the command will return an error message:

1cat sealedsecret.yaml | kubeseal --validate
2error: unable to decrypt sealed secret

Key considerations

If you run a sealed secrets controller in a namespace other than kube-system, you will have to pass the --controller-namespace flag every time you run the kubeseal commands. Deploying Sealed Secrets outside the kube-system namespace can enhance security by isolating it from critical cluster components and offer more granular control through namespace-specific configurations and RBAC policies.

Alternatively, you can set the environment variable SEALEDSECRETSCONTROLLER_NAMESPACE

Summary

Through this tutorial, we’ve learned how to leverage Sealed Secrets to securely store secrets in Git.

Sealed Secrets has become essential software to complete the story of Infrastructure as a code and GitOps.

If you have any issues or doubts, feel free to reach out to me on Twitter.

Related resources

If you want to know more about using Sealed Secrets, take a look at these resources given here:

• Civo Academy: Kubernetes configuration and security
• Video: Store Secrets in Git - Sealed Secrets
• Sealed Secrets Documentation