Kubernetes StatefulSets: Concept and working example

What are StatefulSets and why do they exist?

Deployments create pods with random names in any order. For stateless applications this is fine — any replica can serve any request. For stateful applications like databases this is a problem.

Consider a database Deployment with two replicas. The pods get names like demo-a3f7b and demo-x9k2p. They start in any order, they have no stable identity, and any service can route traffic to either of them. This works for serving read traffic but is dangerous for writes. Writing to any replica without coordination creates data inconsistency.

StatefulSets solve this by giving pods predictable, ordered names and creating them in sequence. If you name your StatefulSet web, the pods are named web-0, web-1, web-2. The next pod does not start until the previous one is ready. You can define web-0 as the primary and web-1, web-2 as replicas with confidence because the order is guaranteed.

StatefulSet vs Deployment

Why headless services?

A regular Service load-balances across all pods. For stateless applications this is exactly what you want. For a database it is dangerous — writes routed to a replica instead of the primary create inconsistency.

A headless Service sets clusterIP: None. Instead of a single virtual IP that load-balances, Kubernetes creates individual DNS entries for each pod:

1web-0.nginx.default.svc.cluster.local
2web-1.nginx.default.svc.cluster.local
3web-2.nginx.default.svc.cluster.local

This lets you address specific pods directly by DNS name. Your application can always send writes to web-0.nginx.default.svc.cluster.local regardless of which node it is running on. The identity is stable.

Working example

This example creates a headless Service and a StatefulSet with three nginx replicas, each with its own PersistentVolumeClaim.

Prerequisites: install local-path-provisioner to provide a StorageClass for dynamic PVC provisioning. Check the latest release on GitHub and replace the version number:

1kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.26/deploy/local-path-storage.yaml

Verify the StorageClass exists:

1kubectl get storageclass

Expected output:

1NAME         PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE
2local-path   rancher.io/local-path   Delete          WaitForFirstConsumer

Create the headless Service and StatefulSet

1apiVersion: v1
2kind: Service
3metadata:
4  name: nginx
5  labels:
6    app: nginx
7spec:
8  clusterIP: None
9  selector:
10    app: nginx
11  ports:
12  - port: 80
13    name: web
14---
15apiVersion: apps/v1
16kind: StatefulSet
17metadata:
18  name: web
19spec:
20  serviceName: nginx
21  replicas: 2
22  selector:
23    matchLabels:
24      app: nginx
25  template:
26    metadata:
27      labels:
28        app: nginx
29    spec:
30      containers:
31      - name: nginx
32        image: nginx
33        ports:
34        - containerPort: 80
35          name: web
36        volumeMounts:
37        - name: www
38          mountPath: /usr/share/nginx/html
39  volumeClaimTemplates:
40  - metadata:
41      name: www
42    spec:
43      accessModes: ["ReadWriteOnce"]
44      storageClassName: local-path
45      resources:
46        requests:
47          storage: 1Gi

1kubectl create -f statefulset.yaml

Watch the pods start in order:

1kubectl get pods

Expected output — web-0 starts first and web-1 only starts once web-0 is Running:

1NAME    READY   STATUS    RESTARTS   AGE
2web-0   1/1     Running   0          30s
3web-1   1/1     Running   0          15s

Check the StatefulSet:

1kubectl get statefulset

Expected output:

1NAME   READY   AGE
2web    2/2     45s

Check the PVCs created automatically — one per pod:

1kubectl get pvc

Expected output:

1NAME        STATUS   VOLUME                     CAPACITY   STORAGECLASS
2www-web-0   Bound    pvc-abc123...              1Gi        local-path
3www-web-1   Bound    pvc-def456...              1Gi        local-path

Check the PVs created automatically by the provisioner:

1kubectl get pv

Scale up to three replicas

1kubectl scale --replicas=3 statefulset/web

Watch the new pod appear:

1kubectl get pods

Expected output:

1NAME    READY   STATUS    RESTARTS   AGE
2web-0   1/1     Running   0          2m
3web-1   1/1     Running   0          2m
4web-2   1/1     Running   0          10s

A new PVC is created automatically for web-2:

1kubectl get pvc

Access a pod by DNS name

Exec into web-0 and curl another pod directly using its stable DNS name:

1kubectl exec -it web-0 -- bash

1curl web-0.nginx.default.svc.cluster.local

Expected output (if an index.html exists in the volume):

1Hello world!

You can also use the short form within the same namespace:

1curl web-0.nginx

Exit the pod:

1exit

Scale down to one replica

Scaling down happens in reverse order. web-2 is deleted first, then web-1:

1kubectl scale --replicas=1 statefulset/web

1kubectl get pods

Expected output once scaling completes:

1NAME    READY   STATUS    RESTARTS   AGE
2web-0   1/1     Running   0          5m

Check the PVCs:

1kubectl get pvc

The PVCs for web-1 and web-2 are not deleted. This is intentional. If you scale back up, web-1 and web-2 will reconnect to the same PVCs and find their data intact. The sticky identity is preserved across scale-down and scale-up events.

To permanently remove the storage, delete the PVCs manually:

1kubectl delete pvc www-web-1 www-web-2

What Kubernetes does and what you are responsible for

Kubernetes handles pod naming and ordering, sequential creation and deletion, PVC creation per pod via volumeClaimTemplates, and reattaching each pod to its original PVC when rescheduled.

You are responsible for:

• Choosing and installing a storage provisioner that suits your workload
• Implementing data replication between pods if needed. For databases like MySQL or MongoDB, you must configure the primary/replica replication yourself. StatefulSets give you predictable pod names and stable storage to build that configuration on top of, but they do not do replication for you.
• Defining a backup and recovery strategy for your data