A/B testing using the Traefik Kubernetes ingress controller

A/B testing, sometimes called split testing, is the practice of routing a percentage of your users to a newer version of an application so you can watch how it behaves in production before rolling it out to everyone. Instead of flipping every user over to a new release at once, you send a slice of live traffic to it, measure, and decide.

You can reach for a full service mesh like Istio or Linkerd to do this, but that is a lot of moving parts to introduce for traffic splitting alone. There is a good chance you already run an ingress controller in your cluster, so extending that to shift traffic between two versions is a much simpler approach.

This tutorial uses Traefik, a fast and lightweight ingress controller you can run on Civo Kubernetes. Traefik can split traffic across two backends using weighted round-robin, and it can route based on a request header, both without any extra components.

Why use Traefik for A/B testing?

• No service mesh required. Weighted traffic splitting is built into Traefik through its TraefikService resource, so there is nothing extra to install on top of the ingress controller.
• Two ways to split. You can split by percentage weight for a classic canary, or by request header when you want to target a specific set of users or testers.
• Native Kubernetes objects. The split is expressed as ordinary Custom Resources you can keep in Git and roll forward or back with kubectl.
• A gradual dial, not a switch. You start a new version at a small weight, watch it, and turn the dial up as your confidence grows.
• One lightweight deployment. Traefik runs as a single ingress controller pod and asks for a Civo Load Balancer, so there is very little to operate.

Prerequisites

• A Civo account
• The Civo CLI installed and authenticated (civo apikey save <name> <key>)
• kubectl installed and on your PATH
• Familiarity with Kubernetes Deployments, Services, and ingress

Launching a cluster using the Civo CLI

Create a small two-node cluster to work in:

1civo k3s create --create-firewall --nodes 2 --size g4s.kube.medium \
2  --save --switch --wait ab-demo

The --save --switch flags merge the new cluster into your kubeconfig and set it as the current context, so kubectl points at it as soon as the command returns. Confirm the nodes are ready:

1kubectl get nodes
2NAME                                              STATUS   ROLES    AGE   VERSION
3k3s-traefiklab-2a58-53387a-node-pool-9c18-rqgrh   Ready    <none>   37m   v1.35.0+k3s1
4k3s-traefiklab-2a58-53387a-node-pool-9c18-yyv1p   Ready    <none>   37m   v1.35.0+k3s1

Installing Traefik

Install Traefik with Helm and ask for a Civo Load Balancer so it gets a public IP:

1helm repo add traefik https://traefik.github.io/charts
2helm repo update
3
4
5helm install traefik traefik/traefik \
6  --namespace traefik --create-namespace \
7  --set service.type=LoadBalancer \
8  --wait

Civo provisions a Load Balancer and attaches a public IP to the Traefik Service. Wait for that IP to appear:

1kubectl -n traefik get svc traefik
2NAME      TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)                      AGE
3traefik   LoadBalancer   10.43.14.41   212.2.240.191   80:30853/TCP,443:30211/TCP   2m

Traefik has a public IP from a Civo Load Balancer; this is the address your split test will answer on.

Deploying the application

We need two versions of an application to split traffic between. To keep the focus on the traffic split rather than on building an app, we'll use hashicorp/http-echo, a tiny web server that returns whatever text you give it. Version 1 returns one string and version 2 returns another, so you can tell at a glance which one served your request.

Deploy version 1:

1kubectl apply -f - <<'EOF'
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: webapp-v1
6  labels:
7    app: webapp
8    version: v1
9spec:
10  replicas: 2
11  selector:
12    matchLabels:
13      app: webapp
14      version: v1
15  template:
16    metadata:
17      labels:
18        app: webapp
19        version: v1
20    spec:
21      containers:
22        - name: webapp
23          image: hashicorp/http-echo:1.0.0
24          args:
25            - "-text=Hello from version 1"
26            - "-listen=:5678"
27          ports:
28            - containerPort: 5678
29---
30apiVersion: v1
31kind: Service
32metadata:
33  name: webapp-v1
34spec:
35  selector:
36    app: webapp
37    version: v1
38  ports:
39    - port: 80
40      targetPort: 5678
41EOF

Now deploy version 2. It is identical except for the labels and the text it returns:

1kubectl apply -f - <<'EOF'
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: webapp-v2
6  labels:
7    app: webapp
8    version: v2
9spec:
10  replicas: 2
11  selector:
12    matchLabels:
13      app: webapp
14      version: v2
15  template:
16    metadata:
17      labels:
18        app: webapp
19        version: v2
20    spec:
21      containers:
22        - name: webapp
23          image: hashicorp/http-echo:1.0.0
24          args:
25            - "-text=Hello from version 2"
26            - "-listen=:5678"
27          ports:
28            - containerPort: 5678
29---
30apiVersion: v1
31kind: Service
32metadata:
33  name: webapp-v2
34spec:
35  selector:
36    app: webapp
37    version: v2
38  ports:
39    - port: 80
40      targetPort: 5678
41EOF

Check that both versions are running:

1kubectl get pods -l app=webapp

Two replicas each of version 1 and version 2, ready to receive traffic.

Splitting traffic with a weighted service

Traefik expresses a weighted split through a TraefikService, a Custom Resource that sits in front of two or more ordinary Services and hands each request to one of them according to the weights you set.

We'll start by sending 60% of traffic to version 2 and keep 40% on version 1. Because Civo assigns your ingress a public IP through a Civo Load Balancer, we'll use a nip.io hostname built from that IP so you don't need to own a domain. First grab the IP that Traefik is serving on:

1LB_IP=$(kubectl -n traefik get svc traefik \
2  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
3HOST=ab.${LB_IP}.nip.io
4echo "Using HOST=$HOST"

Now create the weighted TraefikService and an IngressRoute that points your hostname at it:

1kubectl apply -f - <<EOF
2apiVersion: traefik.io/v1alpha1
3kind: TraefikService
4metadata:
5  name: webapp-canary
6spec:
7  weighted:
8    services:
9      - name: webapp-v1
10        port: 80
11        weight: 40
12      - name: webapp-v2
13        port: 80
14        weight: 60
15---
16apiVersion: traefik.io/v1alpha1
17kind: IngressRoute
18metadata:
19  name: webapp
20spec:
21  entryPoints:
22    - web
23  routes:
24    - match: Host(\`${HOST}\`)
25      kind: Rule
26      services:
27        - name: webapp-canary
28          kind: TraefikService
29EOF

Note the kind: TraefikService on the route's service reference. That is what tells Traefik to route through the weighted split rather than straight to a plain Kubernetes Service.

Performing the split test

Send a handful of requests and watch the responses alternate between the two versions:

1for i in $(seq 1 10); do curl -s http://${HOST}/; done
2Hello from version 1
3Hello from version 2
4Hello from version 1
5Hello from version 2
6Hello from version 2
7Hello from version 1
8Hello from version 2
9Hello from version 2
10Hello from version 1
11Hello from version 2

Roughly six out of every ten requests land on version 2 and the rest on version 1, which matches the 60/40 weight we set. Run the loop a few more times and the ratio holds. To count it directly:

1for i in $(seq 1 100); do curl -s http://${HOST}/; done | sort | uniq -c
2  42 Hello from version 1
3  58 Hello from version 2

Out of 100 requests, close to 60 are served by version 2 and 40 by version 1.

Routing by header instead of by weight

Percentage weights are the right tool when you want a random slice of all users on the new version. Sometimes you instead want a specific group, your QA team or a set of beta users, to always get version 2 while everyone else stays on version 1. Traefik can match on a request header for that.

Replace the IngressRoute with one that has two routes: a specific one that sends any request carrying X-Canary: always straight to version 2, and the original catch-all that keeps everyone else on the weighted split:

1kubectl apply -f - <<EOF
2apiVersion: traefik.io/v1alpha1
3kind: IngressRoute
4metadata:
5  name: webapp
6spec:
7  entryPoints:
8    - web
9  routes:
10    - match: Host(\`${HOST}\`) && Header(\`X-Canary\`, \`always\`)
11      kind: Rule
12      services:
13        - name: webapp-v2
14          port: 80
15    - match: Host(\`${HOST}\`)
16      kind: Rule
17      services:
18        - name: webapp-canary
19          kind: TraefikService
20EOF

When two routes could both match a request, Traefik picks the one with the higher priority, and unless you set a priority yourself it derives one from the length of the match rule. The header route's rule is longer than the plain host rule, so it automatically wins for requests that carry the header. Resist the urge to pin an explicit low priority on the header route: a small number like priority: 10 is very likely lower than the length-derived priority of the plain host rule, and your header requests would end up on the weighted split instead.

A request with the header always gets version 2:

1curl -s -H "X-Canary: always" http://${HOST}/
2Hello from version 2

While a request without it still follows the weighted split:

1curl -s http://${HOST}/
2Hello from version 1

The X-Canary: always header pins a request to version 2; without it, the same host follows the 40/60 split.

Rolling out

Once version 2 has proven itself, you finish the rollout by dialing its weight up. Edit the TraefikService to send all traffic to version 2:

1kubectl patch traefikservice webapp-canary --type=merge -p '
2spec:
3  weighted:
4    services:
5      - name: webapp-v1
6        port: 80
7        weight: 0
8      - name: webapp-v2
9        port: 80
10        weight: 100
11'

Every request now returns version 2:

1for i in $(seq 1 10); do curl -s http://${HOST}/; done | sort | uniq -c
2  10 Hello from version 2

At this point version 2 is your production release. You can point the IngressRoute directly at the webapp-v2 Service and delete the canary resources, then retire the version 1 Deployment when you are ready.

Cleaning up

Delete the whole cluster when you are done so it stops billing:

1civo k3s remove ab-demo

Conclusion

You have split live traffic between two versions of an application using nothing but Traefik, first by percentage weight for a classic canary and then by request header to target a specific audience. Because the split is expressed as ordinary Kubernetes resources, you can version it in Git and turn the dial up or down with a single kubectl command.

From here, you might look at exposing a gRPC service through Traefik, or at wiring metrics into the rollout so the decision to promote a version is driven by data rather than by eye.