Introduction

In this tutorial, you will learn how to create and inject secrets into your application running on Kubernetes using Hashicorp Vault. According to the official website, HashiCorp Vault can be defined as:

HashiCorp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, and certificates.

Vault allows you to manage all your secrets in one place, as it takes care of encryption, leasing, and renewal, as well as secrets revocation when required. Implementing any one of these by itself is quite the challenge which is why Vault is a good choice if you are looking for a cloud-agnostic way to manage secrets.

Prerequisites

This tutorial assumes some familiarity with Kubernetes and helm. In addition, you will also need:

Creating a cluster using the Civo CLI

We’ll begin by creating a Kubernetes cluster using the Civo command line:

$ civo kubernetes create vault-experiments --size "g4s.kube.medium" --nodes 2 --wait --save --merge --region NYC1 

The above command will create a cluster with two nodes and save your KUBECONFIG so you can access your cluster.

Next, you will need to run the following command to switch your Kubernetes context to your new cluster:

kubectl config use-context vault-experiments 

Installing Vault

Hashicorp recomends installing vault in a separate namespace for logical separation and isolation.

Run the following command to create a namespace:

kubectl create namespace vault

Next, add the hashicorp helm repository:

helm repo add hashicorp https://helm.releases.hashicorp.com

Install Vault using helm:

helm install vault hashicorp/vault --namespace vault

This will install vault in standalone mode. This is ideal for simple use cases such as this demo, however, if you plan to run Vault in production, take a look at this page on running Vault in high availability mode.

Confirm the installation to the cluster has been successful. You should see something like the following when you run kubectl get pods -n vault:

$ kubectl get pods -n vault 
vault         vault-agent-injector-5c5b87595-4fr7v   1/1     Running     0          2m23s
vault         vault-0                                0/1     Running     0          2m23s

At first glance it will appear that vault-0 is not running:

This is because the status check defined in a readinessProbe returns a non-zero exit code.

Unsealing Vault

By default, every Vault installation is sealed. This means Vault knows where and how to access the physical storage, but doesn't know how to decrypt any of it.

Vault uses Shamir’s secret sharing algorithm to distribute the keys required to unseal a vault. As we are running in standalone mode, we only have to unseal one Vault instance. In a production environment where Vault is running in HA mode you would be required to unseal all the Vault instances. For an easier approach to this operation, have a look into auto unsealing.

Initialise Vault:

kubectl exec -n vault vault-0 -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json > keys.json

Unseal Vault:

kubectl exec -n vault --stdin=true --tty=true vault-0 -- vault operator unseal #unseal_key_b64
# output 

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.11.3
Build Date      2022-08-26T10:27:10Z
Storage Type    file
Cluster Name    vault-cluster-2695a1ff
Cluster ID      023d5e2a-ca48-c1ed-94c6-1268d6f20b23
HA Enabled      false

NOTE: This would output your vault unseal keys and root token into a file named keys.json, both of which are vital to Vault's operation. Make sure to store them somewhere secure.

Enabling Kubernetes Authentication

Vault supports several methods for authentication. This is useful for assigning identity and a set of policies to a user. In this tutorial we will be using Kubernetes as the authentication provider.

For a more comprehensive list of supported providers take a look at this section of the Vault documentation.

First let's provide the Vault CLI access to the Vault instance running in your cluster:

# export vault server address 
# portfoward vault-0 
export VAULT_ADDR=http://localhost:8200
kubectl -n vault port-forward vault-0 8200:8200

In another terminal window, log in using the vault CLI:

vault login # your_root token 
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                #token
token_accessor       #token accessor 
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

You can also explore the Vault UI which should be available here during the port-forward.

Secrets engines view in the Vault UI

Enable Kubernetes auth:

vault auth enable kubernetes

Kubernetes authentication is now enabled but we need to provide Vault some information about our cluster:

# exec into vault-0 
kubectl exec -n vault -it vault-0 -- /bin/sh
# login to vault 
vault login root_token
#pass auth info 
vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

In the command above we provided vault with a jwt token, Kubernetes CA certificate and the Kubernetes API server address. This will allow Vault to make authenticated API requests.

We’re not quite done yet, though. Next we need to create a service account with a cluster-role binding of system:auth-delegator .

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: vault
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: vault

This will need to be applied to the cluster:

kubectl apply -f cluster-rolebinding.yaml

To ensure our application has access to secrets that would be passed to it , we will create a vault policy that has read and list permissions. Create a file called policy.hcl and add the following:

# policy.hcl 
path "apps/data/guardian" {
    capabilities = ["read", "list"]
}

Write the policy to vault:

vault policy write guardian-ro policy.hcl

The command above shows the policy definition for an application called guardian and subsequent secrets would be placed under the path apps/data/guardian.

You can view the applied policy in the Vault UI by clicking on the policies tab

ACL Policies on Vault

https://file.coffee/u/69BNOegCiPpz8ns4ce0PD.png

Guardian app secrets management

Finally bind the policy and service account to the role for the guardian app by running:

vault write auth/kubernetes/role/guardian \
        bound_service_account_names=guardian-sa \
        bound_service_account_namespaces=default \
        policies=guardian-ro \
        ttl=24h
# Success! Data written to: auth/kubernetes/role/guardian

In the command above we bind a service account guardian-sa to a the default Kubernetes namespace and give specify a time to live of 24hrs.

Deploying an application

Now that we have vault all set up, let's deploy an application that we can inject secrets into. As mentioned earlier we would be deploying an app called guardian which spits out the secrets vault would pass to it. Save this app definition in a file called deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: guardian
spec:
  selector:
    matchLabels:
      app: guardian
  template:
    metadata:
      labels:
        app: guardian
    spec:
      serviceAccountName: guardian-sa
      containers:
      - name:  guardian
        image: ghcr.io/s1ntaxe770r/guardian:v1.1
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: guardian
spec:
  selector:
    app: guardian
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: guardian-sa
  namespace: default

Apply the deployment:

kubectl apply -f deployment.yaml 

Notice we are using guardian-sa as the service account in this deployment. This is required for Vault to grab secrets we would pass shortly.

Next, once applied to your cluster, expose the deployment through a local port-forward:

kubectl port-forward svc/guardian 8080:8080

Explore the secrets endpoint:

curl http://localhost:8080/secrets
# result 
{
  "msg": "I hold no secrets",
  "secrets": {}
}

Guardian currently holds no secrets so lets give it some using Vault.

Injecting Vault secrets

Begin by enabling the key/value store:

vault secrets enable -path=apps kv-v2

Add the following secrets:

vault kv put apps/guardian guardian_message="a word from our sponsor civo cloud" guardian_encryption_key="zsdasdfaskfjhj4534"

We specifically enable version 2 of Vault’s key value store. There are some differences in how Vault handles data stored in v1 and v2 of the store. In this example, secrets stored in apps/guardian would be available at apps/data/guardian. The defined Guardian app looks for the secrets placed under /vault/secrets/guardian.txt in the pod

Update the deployment to tell it to grab the secrets:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: guardian
spec:
  selector:
    matchLabels:
      app: guardian
  template:
    metadata:
      labels:
        app: guardian
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'guardian'
        vault.hashicorp.com/agent-inject-secret-guardian.txt: 'apps/data/guardian'
        vault.hashicorp.com/agent-inject-template-guardian.txt: |
          {{ with secret "apps/data/guardian" }}
          guardian_encryption_key:{{ .Data.data.guardian_encryption_key }}
          guardian_message:{{ .Data.data.guardian_message }}
          {{- end }}  
    spec:
      serviceAccountName: guardian-sa
      containers:
      - name:  guardian
        image: ghcr.io/s1ntaxe770r/guardian:v1.2
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: guardian
spec:
  selector:
    app: guardian
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: guardian-sa
  namespace: default

Apply the changed deployment:

kubectl apply -f deployment.yaml 

The updated deployment has a few new annotations, let walk through each of them:

vault.hashicorp.com/agent-inject: 'true' enable tells Vault to inject a sidecar container into the application pod. If you don't want a sidecar running alongside your application consider adding vault.hashicorp.com/agent-pre-populate-only: 'true'. This would still inject secrets without a sidecar.

vault.hashicorp.com/agent-inject-secret-guardian.txt: 'apps/data/guardian' this tells Vault to inject the secret we added earlier into a file called guardian.txt.

vault.hashicorp.com/role: 'guardian' tells Vault what role to use, in this example the role named “guardian” which we created earlier.

vault.hashicorp.com/agent-inject-template-guardian.txt tells Vault how secrets should be rendered in the file. This would be available in /vault/secrets/guardian.txt inside the container.

Once applied, you can curl query the secrets endpoint:

curl http://localhost:8080/secrets
{
  "msg": "Egads!! I have some secrets, I must inform you at once",
  "secrets": {
    "encryption_key": "zsdasdfaskfjhj4534",
    "lets hear": "a word from your sponsor civo cloud  "
  }
}

You can explore the container to see how the secrets are placed:

# exec into guardian 
kubectl exec --stdin=true --tty=true svc/guardian -- sh
# cat guardian.txt
cat /vault/secrets/guardian.txt

Summary

In this tutorial, we deployed Hashicorp Vault and injected secrets into a secure app that tells us exactly what those secrets are. No doubt Vault has a learning curve, however, when you take into account that rotating credentials and secrets can now be managed from a single place, you start to see where Vault shines. The application example used in this tutorial is available over here.

Next Steps

From here you might be wondering what to do next. Here are a few ideas:

  • Check out the helm chart reference for more values that can be used to tune Vault here .

  • Vault has a few more annotations, asides the ones covered in this guide. Take a look at the annotations reference for more information here

  • Finally, take a look at the Go SDK for vault if you would like to programmatically interact with Vault here.