Unveiling our data centre rollout built for NVIDIA Vera Rubin
DocsSystem statusSupport

Running OpenBao on Civo Kubernetes

HashiCorp changed the rules. OpenBao kept them open. Learn how to run this community-led Vault fork on Civo Kubernetes — from empty cluster to live secret reads.

4 minutes reading time

Written by

Jubril Oyetunji
Jubril Oyetunji

Technical Writer at Civo

HashiCorp relicensed Vault under the Business Source License in August 2023, so we’ve switched to OpenBao, the Linux Foundation’s community-maintained fork of the last open-source release. The API, CLI, and Kubernetes integrations are all preserved, so the tutorial remains just as applicable, just genuinely open source.

If you have been around the HashiCorp ecosystem for a while, you probably already know that in August 2023 they moved their products, Vault included, from the Mozilla Public License to the Business Source License. Following this, a number of projects stood up to keep the tools free and truly open source. OpenTofu picked up Terraform under the Linux Foundation within weeks, and in December 2023 the Linux Foundation announced OpenBao, a fork of the last MPL-licensed release of Vault, now developed as a sandbox project of the OpenSSF.

OpenBao is an identity-based secrets and encryption management system. It keeps the Vault API, the CLI (renamed bao), the storage model, and the Kubernetes integrations, so most of what you know carries over. Essentially, what OpenTofu is to Terraform, OpenBao is to Vault.

In this guide we will walk through running it on Civo Kubernetes, from an empty cluster to an application reading a secret that only OpenBao knows.

Prerequisites

Creating a Civo Kubernetes cluster

civo k3s create --create-firewall --nodes 2 --size g4s.kube.medium \
  --save --switch --wait openbao-demo
kubectl get nodes
NAME                                            STATUS   ROLES    AGE     VERSION
k3s-bao-demo-f027-4aab43-node-pool-01d9-9foki   Ready    <none>   4h13m   v1.35.0+k3s1
k3s-bao-demo-f027-4aab43-node-pool-01d9-gmvgt   Ready    <none>   4h13m   v1.35.0+k3s1

Installing OpenBao

Create a namespace, then the two things the server needs before it starts: a 32-byte unseal key in a Secret, and the ServiceAccount that will act as admin.

kubectl create namespace openbao




kubectl -n openbao create secret generic openbao-unseal \
  --from-literal=key="$(openssl rand -base64 32)"




kubectl -n openbao create serviceaccount bao-admin

For the Helm values. The seal "static" block points to the key through an environment variable, and the initialize block is used to bootstrap a KV version 2 mount at apps/. 

The Kubernetes auth method points to the in-cluster API, an admin policy that can manage secrets, roles, and policies, a read-only guardian policy for the application, and one Kubernetes auth role for each.

cat > openbao-values.yaml <<'EOF'
injector:
  enabled: true
server:
  extraSecretEnvironmentVars:
    - envName: BAO_UNSEAL_KEY
      secretName: openbao-unseal
      secretKey: key
  standalone:
    enabled: true
    config: |
      ui = true
      disable_mlock = true
      listener "tcp" {
        address         = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_disable     = true
      }
      storage "file" {
        path = "/openbao/data"
      }
      seal "static" {
        current_key_id = "key-1"
        current_key    = "env://BAO_UNSEAL_KEY"
      }
      initialize "bootstrap" {
        request "enable-kv" {
          operation = "update"
          path      = "sys/mounts/apps"
          data      = { type = "kv", options = { version = "2" } }
        }
        request "enable-kubernetes-auth" {
          operation = "update"
          path      = "sys/auth/kubernetes"
          data      = { type = "kubernetes" }
        }
        request "kubernetes-auth-config" {
          operation = "update"
          path      = "auth/kubernetes/config"
          data      = { kubernetes_host = "https://kubernetes.default.svc:443" }
        }
        request "policy-admin" {
          operation = "update"
          path      = "sys/policies/acl/admin"
          data      = { policy = <<-POL
            path "apps/*"                 { capabilities = ["create", "read", "update", "delete", "list"] }
            path "auth/kubernetes/role/*" { capabilities = ["create", "read", "update", "delete", "list"] }
            path "sys/policies/acl/*"     { capabilities = ["create", "read", "update", "delete", "list"] }
            path "sys/mounts/*"           { capabilities = ["create", "read", "update", "delete", "list"] }
          POL
          }
        }
        request "policy-guardian" {
          operation = "update"
          path      = "sys/policies/acl/guardian"
          data      = { policy = "path \"apps/data/guardian\" { capabilities = [\"read\"] }" }
        }
        request "role-admin" {
          operation = "update"
          path      = "auth/kubernetes/role/admin"
          data      = { bound_service_account_names = "bao-admin", bound_service_account_namespaces = "openbao", token_policies = "admin", token_ttl = "1h" }
        }
        request "role-guardian" {
          operation = "update"
          path      = "auth/kubernetes/role/guardian"
          data      = { bound_service_account_names = "guardian-sa", bound_service_account_namespaces = "default", token_policies = "guardian", token_ttl = "1h" }
        }
      }
EOF

Before you install, there are two side effects of revoking the root token. The admin policy has to cover roles and policies as well as secrets, because once the root token is revoked, there is no other way to add them. And every role the platform will ever need should be in this block from the start, for the same reason.

Install the chart:

helm repo add openbao https://openbao.github.io/openbao-helm
helm repo update




helm install openbao openbao/openbao --namespace openbao \
  --version 0.29.4 -f openbao-values.yaml --wait
kubectl -n openbao get pods

The server pod and the injector webhook are both ready without an init or unseal step.

The pod is 1/1 without you doing anything, which is the part that used to need operator init and operator unseal. Check the status to see why:

kubectl -n openbao exec openbao-0 -- bao status

Logging in without a root token

There is no root token to log in with. The admin role is bound to the bao-admin ServiceAccount, so a token for that ServiceAccount is the login. 

Port-forward to the server, then exchange a ServiceAccount token for an OpenBao token:

kubectl -n openbao port-forward svc/openbao 8200:8200 &
export BAO_ADDR=http://127.0.0.1:8200




export BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
  role=admin jwt="$(kubectl -n openbao create token bao-admin)")
bao token lookup

The token carries the admin policy and expires in an hour. Run the same command for a new one.

The same address serves the UI at http://127.0.0.1:8200 while the port-forward is up, and this token logs you in there too.

Writing a secret

Store the two values the demo application expects:

bao kv put apps/guardian \
  guardian_message="Hello from OpenBao on Civo" \
  guardian_encryption_key="k3y-0001"
bao kv get apps/guardian
====== Data ======
Key                        Value
---                        -----
guardian_encryption_key    k3y-0001
guardian_message           Hello from OpenBao on Civo

Deploying an application that reads it

The application is guardian, a small Go service that reads /vault/secrets/guardian.txt and reports what it found on /secrets. It runs under the guardian-sa ServiceAccount, which the guardian role was bound to during bootstrap.

The annotations do the rest. agent-inject asks the injector for a sidecar, role names the OpenBao role it logs in with, and the agent-inject-secret and agent-inject-template pair render apps/guardian into a file as key:value lines:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: guardian-sa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: guardian
spec:
  replicas: 1
  selector:
    matchLabels:
      app: guardian
  template:
    metadata:
      labels:
        app: guardian
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'guardian'
        vault.hashicorp.com/agent-inject-secret-guardian.txt: 'apps/data/guardian'
        vault.hashicorp.com/agent-inject-template-guardian.txt: |
          {{ with secret "apps/data/guardian" }}
          guardian_encryption_key:{{ .Data.data.guardian_encryption_key }}
          guardian_message:{{ .Data.data.guardian_message }}
          {{- end }}
    spec:
      serviceAccountName: guardian-sa
      containers:
        - name: guardian
          image: ghcr.io/s1ntaxe770r/guardian:v1.2
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: guardian
spec:
  selector:
    app: guardian
  ports:
    - port: 8080
      targetPort: 8080
EOF

Presumably, for backwards compatibility, the annotations still carry the vault.hashicorp.com prefix. 

kubectl get pods -l app=guardian
NAME                        READY   STATUS    RESTARTS   AGE
guardian-6846d44b7c-fv48b   2/2     Running   0          6s

2/2 is the application plus the agent sidecar. Read the file the agent rendered, then ask the application for it over HTTP:

kubectl exec deploy/guardian -c guardian -- cat /vault/secrets/guardian.txt




kubectl port-forward svc/guardian 8080:8080 &
curl -s http://localhost:8080/secrets

The agent logged in with the pod's ServiceAccount, rendered the template, and the application read it.

Rotating the secret

Write a new version and restart the Deployment. The agent renders the current version when the pod starts, and the application reads the file on each request:

bao kv put apps/guardian \
  guardian_message="Rotated on Civo" \
  guardian_encryption_key="k3y-0002"




kubectl rollout restart deploy/guardian
kubectl rollout status deploy/guardian
curl -s http://localhost:8080/secrets
{"msg":"Egads!! I have some secrets, I must inform you at once","secrets":{"encryption_key":"k3y-0002","lets hear":"Rotated on Civo"}}

Cleaning up

civo k3s remove openbao-demo

Wrapping up

You now have OpenBao running on Civo Kubernetes with an unseal key in a Secret, its auth method, policies, and roles declared in configuration, an admin path that is a ServiceAccount, and an application reading its secrets through the Agent Injector.

From here, the OpenBao Helm configuration covers the HA Raft mode for more than one replica, and the self-initialization docs cover everything else the bootstrap block can declare.

Tags:Security & IdentityKubernetes & Cluster Management
Jubril Oyetunji
Jubril Oyetunji

Technical Writer at Civo

Jubril Oyetunji is a DevOps engineer and technical writer with a strong focus on cloud-native technologies and open-source tools. His work centers on creating practical tutorials that help developers better understand platforms such as Kubernetes, NGINX, Rust, and Go.

As a contract technical writer, Jubril authored an extensive library of technical guides covering cloud-native infrastructure and modern development workflows. Many of his tutorials achieved strong search rankings, helping developers around the world learn and adopt emerging technologies.

View author profile
PrerequisitesCreating a Civo Kubernetes clusterInstalling OpenBaoLogging in without a root tokenWriting a secretDeploying an application that reads itRotating the secretCleaning upWrapping up

Share this article

Further Reading

22 August 2022

Managing SSH access with HashiCorp Vault

Read more

9 October 2023

Sealed secrets: Securely storing Kubernetes secrets in Git

Read more

10 October 2024

Secure secrets management in Kubernetes with Bitwarden and Civo

Read more
Slide 1 of 3