Efficient Secrets Management in Kubernetes using Bitwarden Secrets Manager

This piece with co-written by Keith Hubner, Integration Engineer (EMEA), at Bitwarden.

Handling secrets directly in Kubernetes presents a myriad of challenges that can complicate the management of sensitive information. Hardcoding secrets can lead to significant difficulties, as they become harder to manage and may create gaps in access control and auditing features. Additionally, the processes of rotating and revoking secrets can be cumbersome, increasing the risk of exposure.

To add to these challenges, integrating Kubernetes with external secrets management tools often proves complicated, making secure and efficient secrets management within Kubernetes a daunting task. Learn more about some of these challenges.

Effective secrets management is crucial in DevOps, as it securely handles sensitive information such as passwords, API keys, and tokens. Proper secrets management ensures secure storage, access control, rotation, and auditing, preventing the risk of exposure within code or pipelines. Tools like Bitwarden Secrets Manager play a vital role in automating secret management, seamlessly integrating it into DevOps workflows to enhance both security and efficiency.

By addressing these challenges and implementing robust secrets management strategies, organizations can protect their sensitive data while benefiting from the collaborative and agile nature of DevOps practices.

Prerequisites

Before getting started with this tutorial, you will need to ensure you have the following:

Access to Bitwarden Secrets Manager account. You can read the help documentation here.
You will also need a Civo account. Instructions on how to do this can be found here.

Implementing secrets management for Kubernetes with Bitwarden

Bitwarden Secrets Manager is a unique solution to efficiently and securely store and retrieve secrets. In order to create a more secure digital world, we need to make security convenient and effective.

Adding secrets to Kubernetes workflows is one way Bitwarden is helping make security fast and effective. Using the Secrets Manager Kubernetes Operator, which is deployed using Helm package manager, secrets can be stored and retrieved from Secrets Manager. In this tutorial, we will show you how this solution can be enabled and used with your Civo workflow.

Setting up Secrets Manager

To start, we will need to open the Bitwarden Secrets Manager on the web app. Use the product switcher in the bottom left corner of the web app to navigate to Secrets Manager if you have not yet used Secrets Manager.

From the Secrets Manager web app, create a new project, which will be the primary way of grouping secrets and assigning access later.

Step 1: Select Projects from the navigation bar and create a New project using the dropdown menu.

Step 2: Next, create a secret by selecting Secrets on the navigation. Create a new secret to store your needed access information, such as a cluster access token.

Step 3: Create a Machine Account by navigating to Machine Accounts on the navigation menu. Create a machine account and generate a new access token. This token will be used by your machine to allow Bitwarden Secrets Manager to retrieve secrets from the project.

Step 4: Return to the project that was created and assign the following items to the project:

The created secrets
Assign users to the project
Machine account created

Setting up Civo

In this example we will create a new cluster with one medium node and then save and merge our kubeconfig. For more information, you can refer to the Civo documentation.

Create a cluster:

1civo k3s create -n 1 -s g4s.kube.medium --merge --save --wait

Save the Civo API key in Secrets Manager Once the Civo CLI has been installed, you will be required to retrieve a Civo API key. This can be securely stored in secrets manager. Retrieve the API key from your Civo Dashboard. Once the API key has been saved, copy the value and Run:

For additional information on Chart setup and adding the repository to Helm, see https://bitwarden.com/help/secrets-manager-kubernetes-operator/#add-the-repository-to-helm.

Create values:

1helm show values bitwarden/sm-operator --devel &gt; my-values.yaml

You may wish to edit fields:

1# How often the secrets synchronize in seconds. Minimum value is 180.
2bwSecretManagerResreshInterval: 300
3
4# Cloud region for sync
5cloudRegion: US

After changes have been made, upgrade the release to a new chart by running:

1helm upgrade sm-operator bitwarden/sm-operator -i --debug -n sm-operator-system --create-namespace --values my-values.yaml --devel

Return to the Secrets Manager web vault and copy the secret IDs and Machine account access token.

We will deploy a test app and therefore create a test namespace for this and the secrets:

1kubectl create namespace test

1kubectl create secret generic bw-auth-token -n test --from-literal=token="&lt;your_machine_token_here&gt;"

Create a secret

If using the following example, be sure to replace the values for your_org_id_here and your_secret_id_here. These values can be retrieved from the Secrets Manager web vault.

1kubectl apply -f - &lt;&lt;EOF
2apiVersion: k8s.bitwarden.com/v1
3kind: BitwardenSecret
4metadata:
5  namespace: test
6  labels:
7    app.kubernetes.io/name: bitwardensecret
8    app.kubernetes.io/instance: bitwardensecret-sample
9    app.kubernetes.io/part-of: sm-operator
10    app.kubernetes.io/managed-by: kustomize
11    app.kubernetes.io/created-by: sm-operator
12  name: bitwardensecret-sample
13spec:
14  organizationId: "&lt;your_org_id_here&gt;"
15  secretName: bw-sample-secret
16  map:
17    - bwSecretId: &lt;your_secret_id_here&gt;
18      secretKeyName: test__secret__1
19    - bwSecretId: &lt;your_secret_id_here&gt;
20      secretKeyName: test__secret__2
21  authToken:
22    secretName: bw-auth-token
23    secretKey: token
24
25EOF

Check that the secrets and operator are working in K8s:

1kubectl get bitwardensecrets -n test                                                                                    
2NAME                     AGE
3bitwardensecret-sample   27m

Check the health of the operator:

1kubectl describe bitwardensecrets -n test

Check the sync status:

1Message:                  Completed sync for test/bitwardensecret-sample
2    Reason:                   ReconciliationComplete
3    Status:                   True
4    Type:                     SuccessfulSync
5  Last Successful Sync Time:  2024-09-25T10:36:12Z

Check secrets:

1kubectl get secrets -n test                                                                                                
2NAME               TYPE     DATA   AGE
3bw-auth-token      Opaque   1      7m47s
4bw-sample-secret   Opaque   2      6m56s

More secret information:

1kubectl describe secrets bw-sample-secret -n test                                                                            
2Name:         bw-sample-secret
3Namespace:    test
4Labels:       k8s.bitwarden.com/bw-secret=a9468555-5e5e-4621-93a1-eeaf028862bd
5Annotations:  k8s.bitwarden.com/custom-map:
6                [
7                  {
8                    "bwSecretId": "&lt;your_secret_id_here&gt;",
9                    "secretKeyName": "test__secret__1"
10                  },
11                  {
12                    "bwSecretId": "&lt;your_secret_id_here&gt;",
13                    "secretKeyName": "test__secret__2"
14                  }
15                ]
16              k8s.bitwarden.com/sync-time: 2024-09-25T10:36:12.922941793Z
17
18Type:  Opaque
19
20Data
21====
22test__secret__1:  10 bytes
23test__secret__2:  12 bytes

Check the secret contents:

1kubectl get secret bw-sample-secret -n test -o jsonpath="{.data}" | jq 'to_entries[] | "\(.key): \(.value | @base64d)"'
2
3"test__secret__1: &lt;your_secret_shown_here&gt;"
4"test__secret__2: &lt;your_secret_shown_here&gt;"

Test app to deploy a Bitwarden Secret

We can create and use Bitwarden secrets with the operator. The Kubernetes secret belongs to a namespace and will be injected with the data that the Secrets Manager machine account has access to.

Create test app:

1kubectl apply -f - &lt;&lt;EOF
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: my-deployment
6  namespace: test
7  labels:
8    app: my-deployment
9spec:
10  selector:
11    matchLabels:
12      app: my-deployment
13  template:
14    metadata:
15      labels:
16        app: my-deployment
17    spec:
18      containers:
19      - name: my-deployment
20        image: nginx
21        envFrom:
22        - secretRef:
23            name: bw-sample-secret
24      imagePullSecrets:
25      - name: bw-sample-secret
26
27EOF

Check that these secrets have been injected into the app:

1kubectl get pods -n test

Find the test pod and copy the name:

1my-deployment-7ff457d764-qsp5m   1/1     Running   0          15m

View the env vars of the pod:

1kubectl exec -it my-deployment-7ff457d764-qsp5m -n test -- env

1PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2HOSTNAME=my-deployment-7ff457d764-g82dx
3NGINX_VERSION=1.27.1
4NJS_VERSION=0.8.5
5NJS_RELEASE=1~bookworm
6PKG_RELEASE=1~bookworm
7DYNPKG_RELEASE=2~bookworm
8test__secret__2=&lt;your_secret_shown_here&gt;"
9test__secret__1=&lt;your_secret_shown_here&gt;"
10KUBERNETES_SERVICE_HOST=10.43.0.1
11KUBERNETES_SERVICE_PORT=443
12KUBERNETES_SERVICE_PORT_HTTPS=443
13KUBERNETES_PORT=tcp://10.43.0.1:443
14KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
15KUBERNETES_PORT_443_TCP_PROTO=tcp
16KUBERNETES_PORT_443_TCP_PORT=443
17KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
18TERM=xterm
19HOME=/root

If we change the secret value in Secrets Manager, we can restart the deployment and then check the value has changed:

1kubectl get secret bw-sample-secret -n test -o jsonpath="{.data}" | jq 'to_entries[] | "\(.key): \(.value | @base64d)"'     
2"test__secret__1: &lt;your_new_secret_vaule_shown_here&gt;"
3"test__secret__2: &lt;your_new_secret_vaule_shown_here&gt;"

You will see the value has changed in the secret, but not yet in the pod, until it is restarted and it grabs the new values:

1kubectl exec -it my-deployment-7ff457d764-p2fcl -n test -- env                                                               
2PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
3HOSTNAME=my-deployment-7ff457d764-p2fcl
4NGINX_VERSION=1.27.1
5NJS_VERSION=0.8.5
6NJS_RELEASE=1~bookworm
7PKG_RELEASE=1~bookworm
8DYNPKG_RELEASE=2~bookworm
9test__secret__2=&lt;your_new_secret_vaule_shown_here&gt;"
10test__secret__1=&lt;your_new_secret_vaule_shown_here&gt;"
11KUBERNETES_SERVICE_PORT=443
12KUBERNETES_SERVICE_PORT_HTTPS=443
13KUBERNETES_PORT=tcp://10.43.0.1:443
14KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
15KUBERNETES_PORT_443_TCP_PROTO=tcp
16KUBERNETES_PORT_443_TCP_PORT=443
17KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
18KUBERNETES_SERVICE_HOST=10.43.0.1
19TERM=xterm
20HOME=/root

Summary

In this tutorial we demonstrated using the Secrets Manager Helm Operator to deploy a secret stored in Secrets Manager. This setup has allowed us to store and retrieve the secrets for use in any Kubernetes project.

Overview of steps completed:

Setup Secrets Manager from the UI to include: Project, Secret, and Machine Account.
Setup Kubernetes cluster using Civo’s CLI.
Setup the Bitwarden Helm repository and created a custom configuration file.
Created a BitwardenSecret object to be deployed by the operator.

Secure secrets management in Kubernetes with Bitwarden and Civo

Prerequisites

Implementing secrets management for Kubernetes with Bitwarden

Setting up Secrets Manager

Setting up Civo

Create a secret

Test app to deploy a Bitwarden Secret

Summary

Further Reading

Sealed secrets: Securely storing Kubernetes secrets in Git

Kubectl: Essential tips, hidden shortcuts, and expert workflows for seamless cluster management

Sealed secrets: Securely storing Kubernetes secrets in Git

Kubectl: Essential tips, hidden shortcuts, and expert workflows for seamless cluster management

Company

Company

Public Cloud

Public Cloud

Private Cloud

Private Cloud

Civo AI

Civo AI

Solutions

Solutions

Resources

Resources

Contact

Contact

Legal

Social