Cloud-Native stateful storage for Kubernetes with Rancher Labs' Longhorn

In this learn guide you will get up and running with Longhorn from Rancher Labs on Civo using K3s, a lightweight Kubernetes-compatible daemon.

For this, you'll need a Kubernetes cluster, but we'll link to Rancher's guide on installing K3s. We'll then install Longhorn and give an example of how to use it.

One of the principles of Cloud Native applications is that they aim to be stateless, so you can purely horizontally scale your application. However, unless your site is just a brochure, you need to store things somewhere!

The big gorillas in our industry (such as Google and Amazon) have custom systems with their own scalable storage solutions to plug in, but what about options for the dev clusters, the small shops, the tinkerers...

Rancher Labs announced their Project Longhorn in March 2018 to try to fill this gap. What Longhorn does is use the existing disks of your Kubernetes nodes to provide replicated and stable storage for your Kubernetes pods.

Longhorn installation prerequisites

Before we can play with Longhorn, you need to have a Kubernetes cluster up and running. You can do this with a simple manually installed K3s cluster or if you're a tester of our Kubernetes service, you can use that. Although it's not publicly available, we'll create the cluster with our Kubernetes service for ease (although the steps on the K3s link above are really easy to follow manually).

We'd recommend using a minimum of Medium instances because we'll be testing stateful storage for MySQL and it can be RAM hungry.

1$ civo k8s create longhorn-test --wait
2Building new Kubernetes cluster longhorn-test: \
3Created Kubernetes cluster longhorn-test

Your cluster needs to have open-iscsi installed on each of the nodes, so if you aren't using our Kubernetes service you also will need to run the following on each node, in addition to the instructions linked above:

1sudo apt-get install open-iscsi

You need to have a Kubernetes configuration file either downloaded and saved to ~/.kube/config or set an environment variable called KUBECONFIG to its filename:

1cd ~/longhorn-play
2civo k8s config longhorn-test > civo-longhorn-test-config
3export KUBECONFIG=civo-longhorn-test-config

Installing Rancher Longhorn

There are two steps to installing Longhorn on an existing Kubernetes cluster - install the controller and extensions for Longhorn, and then create a StorageClass so it's usable by pods. The first step is as simple as:

1$ kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/master/deploy/longhorn.yaml
2namespace/longhorn-system created
3serviceaccount/longhorn-service-account created
4...

To create the StorageClass is another one command away, however as an additional step you can make the new class the default, so you don't need to specify it every time:

1$ kubectl apply -f https://raw.githubusercontent.com/rancher/longhorn/master/examples/storageclass.yaml
2storageclass.storage.k8s.io/longhorn created
3
4$ kubectl get storageclass
5NAME       PROVISIONER           AGE
6longhorn   rancher.io/longhorn   3s
7
8$ kubectl patch storageclass longhorn -p \
9  '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
10    storageclass.storage.k8s.io/longhorn patched
11
12$ kubectl get storageclass
13NAME                 PROVISIONER           AGE
14longhorn (default)   rancher.io/longhorn   72s

Accessing Longhorn's dashboard

Longhorn has a lovely simple dashboard that lets you see used space, available space, volume listing, etc. First off, we need to create the authentication details (so the whole world can't see it):

1$ htpasswd -c ./ing-auth admin
2$ kubectl create secret generic longhorn-auth \
3  --from-file ing-auth --namespace=longhorn-system

Now we'll create an Ingress object that uses K3s' built-in Traefik to expose the dashboard to the outside world. Make a file called longhorn-ingress.yaml and put this in it:

1apiVersion: extensions/v1beta1
2kind: Ingress
3metadata:
4  name: longhorn-ingress
5  annotations:
6    ingress.kubernetes.io/auth-type: "basic"
7    ingress.kubernetes.io/auth-secret: "longhorn-auth"
8spec:
9  rules:
10  - host: longhorn-frontend.example.com
11    http:
12      paths:
13      - backend:
14          serviceName: longhorn-frontend
15          servicePort: 80

Then apply it with:

1$ kubectl apply -f longhorn-ingress.yaml -n longhorn-system
2ingress.extensions/longhorn-ingress created

You now need to add an entry to your /etc/hosts file to point any one of your Kubernetes IP addresses to longhorn-frontend.example.com:

1echo "1.2.3.4 longhorn-frontend.example.com" >> /etc/hosts

Now you can visit https://longhorn-frontend.example.com in your browser, and after authenticating with admin and the password you typed when using htpasswd and see something like:

Installing MySQL with persistent storage

There's not much point in running MySQL in a single container if when the underlying node (or the container) dies you lose all of your customers, orders, etc. So let's configure it with a new Longhorn persistent volume.

First, we need to create a few resources in Kubernetes. Each of these are a yaml file in a single empty directory, or you can put them all in a single file separated with a line containing just ---.

A persistent volume in mysql/pv.yaml:

1apiVersion: v1
2kind: PersistentVolume
3metadata:
4  name: mysql-pv
5  namespace: apps
6  labels:
7    name: mysql-data
8    type: longhorn
9spec:
10  capacity:
11    storage: 5G
12  volumeMode: Filesystem
13  storageClassName: longhorn
14  accessModes:
15    - ReadWriteOnce
16  csi:
17    driver: io.rancher.longhorn
18    fsType: ext4
19    volumeAttributes:
20      numberOfReplicates: '2'
21      staleReplicaTimeout: '20'
22    volumeHandle: mysql-data

A claim to that volume (like an abstract request so that something can use the volume) in mysql/pv-claim.yaml:

1apiVersion: v1
2kind: PersistentVolumeClaim
3metadata:
4  name: mysql-pv-claim
5  labels:
6    type: longhorn
7    app: example
8spec:
9  storageClassName: longhorn
10  accessModes:
11    - ReadWriteOnce
12  resources:
13    requests:
14      storage: 5Gi

And a pod that will run MySQL and use the claim to the volume above (note: we use password here as the MySQL root password, but really you should use a secure password AND you should use Kubernetes secrets to store it, not in the YAML file - we're just keeping it simple here) in mysql/pod.yaml:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: my-mysql
5  labels:
6    app: example
7spec:
8  selector:
9    matchLabels:
10      app: example
11      tier: mysql
12  strategy:
13    type: Recreate
14  template:
15    metadata:
16      labels:
17        app: example
18        tier: mysql
19    spec:
20      containers:
21      - image: mysql:5.6
22        name: mysql
23        env:
24        - name: MYSQL_ROOT_PASSWORD
25          value: password
26        ports:
27        - containerPort: 3306
28          name: mysql
29        volumeMounts:
30        - name: mysql-persistent-storage
31          mountPath: /var/lib/mysql
32      volumes:
33      - name: mysql-persistent-storage
34        persistentVolumeClaim:
35          claimName: mysql-pv-claim

Now apply either the folder or the single file (depending on which you chose):

1$ kubectl apply -f mysql.yaml
2persistentvolumeclaim/mysql-pv-claim created
3persistentvolume/mysql-pv created
4deployment.apps/my-mysql created
5
6# or
7
8kubectl apply -f ./mysql/
9persistentvolumeclaim/mysql-pv-claim created
10persistentvolume/mysql-pv created
11deployment.apps/my-mysql created

Testing MySQL with persistent storage works

Our test here will be simple, we'll create a new database, delete the container (which Kubernetes will recreate for us) and then connect back in and hopefully still see our new database.

Let's create a new database called should_still_be_here (great name for our production DB!)

1$ kubectl get pods | grep mysql
2my-mysql-d59b9487b-7g644   1/1     Running   0          2m28s
3$ kubectl exec -it my-mysql-d59b9487b-7g644 /bin/bash
4root@my-mysql-d59b9487b-7g644:/# mysql -u root -p mysql
5Enter password: 
6mysql> create database should_still_be_here;
7Query OK, 1 row affected (0.00 sec)
8
9mysql> show databases;
10+----------------------+
11| Database             |
12+----------------------+
13| information_schema   |
14| #mysql50#lost+found  |
15| mysql                |
16| performance_schema   |
17| should_still_be_here |
18+----------------------+
195 rows in set (0.00 sec)
20
21mysql> exit
22Bye
23root@my-mysql-d59b9487b-7g644:/# exit
24exit

Now we'll delete the container:

1kubectl delete pod my-mysql-d59b9487b-7g644

After a minute or so, we'll look again for the new container name, connect to it and see if our database still exists:

1$ kubectl get pods | grep mysql
2my-mysql-d59b9487b-8zsn2   1/1     Running   0          84s
3$ kubectl exec -it my-mysql-d59b9487b-8zsn2 /bin/bash
4root@my-mysql-d59b9487b-8zsn2:/# mysql -u root -p mysql
5Enter password: 
6mysql> show databases;
7+----------------------+
8| Database             |
9+----------------------+
10| information_schema   |
11| #mysql50#lost+found  |
12| mysql                |
13| performance_schema   |
14| should_still_be_here |
15+----------------------+
165 rows in set (0.00 sec)
17
18mysql> exit
19Bye
20root@my-mysql-d59b9487b-7g644:/# exit
21exit

So that was a complete success, our storage persisted across containers being killed.

Wrapping up

We've now installed scalable cloud storage on a Kubernetes cluster and ensured it works by deleting our container and watching our existing state still exist.

If you would like to continue learning you can:

• Try deleting and recreating one of your Kubernetes nodes and see if the storage persists
• Join our Community to chat with the other cloud native developers

Let us know how you get on by tweeting us @CivoCloud!