Practical guide to Kubernetes probes

A container that is still running is not the same as a container that is still working. A process can be alive in every sense the kernel cares about while the application inside it has wedged on a deadlock, lost its database connection, or filled its heap. Kubernetes, left to its own devices, sees a running process and assumes all is well, so traffic keeps flowing to a pod that can no longer serve it. 

In this guide, we'll walk through the three Kubernetes probes, the methods they use to check health, and how to tune them so your deployments heal themselves instead of quietly failing.

Prerequisites

To follow along, you'll need:

• A Civo account
• A Kubernetes cluster created on Civo
• The latest kubectl utility to interact with the cluster
• The KUBECONFIG file pointing kubectl at your cluster, downloadable from the cluster page on your Civo dashboard

If you do not have a cluster yet, you can create one from the Civo dashboard and download the kubeconfig in a couple of minutes.

Why probes matter

Kubernetes has become the de-facto standard for deploying cloud-native applications, driven in large part by the shift to microservices. That style of system, with lightweight, loosely coupled, autonomous services, suits distributed applications well. The trade-off is that managing those applications at scale is hard, especially when many components depend on one another and you need the system to stay correct even during partial failures.

By default, the only health signal Kubernetes has is whether the container's main process is running. That is a weak signal. An application can return errors on every request, refuse new connections, or hang indefinitely while its process stays up. Probes give Kubernetes an application-specific health signal so it can act: restart a wedged container, stop routing traffic to one that is not ready, or wait patiently for a slow boot to finish.

Before we add any probes, let's set up a namespace and a plain deployment to work against.

1kubectl create namespace probe-demo
2
3
4kubectl apply -f - <<'EOF'
5apiVersion: apps/v1
6kind: Deployment
7metadata:
8  name: nginx
9  namespace: probe-demo
10spec:
11  selector:
12    matchLabels:
13      app: nginx
14  replicas: 2
15  template:
16    metadata:
17      labels:
18        app: nginx
19    spec:
20      containers:
21      - name: nginx
22        image: nginx
23        ports:
24        - containerPort: 80
25EOF

This is a healthy baseline with no probes. Everything below builds on it.

The three probe types

Kubernetes gives us three probes, each answering a different question:

• Liveness probe: Constantly checks whether the container is healthy and functional. If it detects an issue, by default it restarts the container.
• Readiness probe: Checks whether the container is ready to accept incoming requests. If it is, requests are sent to the container; if not, the pod is removed from the Service's endpoints.
• Startup probe: Determines whether a container has started. While it runs, the liveness and readiness probes are held off so a slow boot does not trigger a premature restart.

Which probe when

If you remember nothing else, remember this:

• Use a liveness probe to restart a container that has wedged but is still running. If it fails, the kubelet kills and restarts the container.
• Use a readiness probe to decide whether a container should receive traffic. If it fails, the pod is pulled from Service endpoints but is not restarted.
• Use a startup probe for slow-starting apps. While it runs, liveness and readiness are held off, so a long boot does not trigger premature restarts.

Probe methods

Every probe type checks health using one of four methods. The first three have been around since the early days of Kubernetes; the fourth, gRPC, is the modern addition we'll cover at the end of this section.

Exec

An exec probe runs a command inside the container. If the command exits with status 0, the probe succeeds; any non-zero exit is a failure. Here we check that nginx's default index page is present on disk.

1kubectl apply -f - <<'EOF'
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: nginx
6  namespace: probe-demo
7spec:
8  selector:
9    matchLabels:
10      app: nginx
11  replicas: 2
12  template:
13    metadata:
14      labels:
15        app: nginx
16    spec:
17      containers:
18      - name: nginx
19        image: nginx
20        ports:
21        - containerPort: 80
22        livenessProbe:
23          exec:
24            command:
25            - ls
26            - /usr/share/nginx/html/index.html
27EOF

The file exists in the stock image, so ls exits 0 and the probe passes.

TCP

A TCP probe succeeds if the kubelet can open a TCP connection to the given port. Note that the manifest below points the probe at port 8080, while nginx listens on 80. That mismatch is deliberate: the connection will fail, and you can watch the kubelet restart the container as a result.

1kubectl apply -f - <<'EOF'
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: nginx
6  namespace: probe-demo
7spec:
8  selector:
9    matchLabels:
10      app: nginx
11  replicas: 2
12  template:
13    metadata:
14      labels:
15        app: nginx
16    spec:
17      containers:
18      - name: nginx
19        image: nginx
20        ports:
21        - containerPort: 80
22        livenessProbe:
23          tcpSocket:
24            port: 8080
25EOF

Point the probe at port 80 instead and it will pass.

HTTP

An HTTP probe sends a GET request to a path and port. Any response code from 200 to 399 counts as success; anything else is a failure. Let's start by asking for a path that does not exist so we can see a failure.

1kubectl apply -f - <<'EOF'
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: nginx
6  namespace: probe-demo
7spec:
8  selector:
9    matchLabels:
10      app: nginx
11  replicas: 2
12  template:
13    metadata:
14      labels:
15        app: nginx
16    spec:
17      containers:
18      - name: nginx
19        image: nginx
20        ports:
21        - containerPort: 80
22        livenessProbe:
23          httpGet:
24            path: /non-existing-endpoint
25            port: 80
26EOF

That path returns a 404, so the probe fails and the container restarts. Point it at / and it succeeds.

1kubectl apply -f - <<'EOF'
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: nginx
6  namespace: probe-demo
7spec:
8  selector:
9    matchLabels:
10      app: nginx
11  replicas: 2
12  template:
13    metadata:
14      labels:
15        app: nginx
16    spec:
17      containers:
18      - name: nginx
19        image: nginx
20        ports:
21        - containerPort: 80
22        livenessProbe:
23          httpGet:
24            path: /
25            port: 80
26EOF

Tip: name your ports

Instead of repeating a port number across the manifest, name the container port and reference it by name in the probe. This keeps the probe in sync if the port ever changes.

1kubectl apply -f - <<'EOF'
2apiVersion: v1
3kind: Pod
4metadata:
5  name: named-port-probe
6  namespace: probe-demo
7spec:
8  containers:
9    - name: app
10      image: registry.k8s.io/e2e-test-images/agnhost:2.40
11      args: ["liveness"]
12      ports:
13        - name: health
14          containerPort: 8080
15      livenessProbe:
16        httpGet:
17          path: /healthz
18          port: health
19        initialDelaySeconds: 5
20        periodSeconds: 5
21EOF

The probe references the named port and resolves with no error:

1Liveness:  http-get http://:health/healthz delay=5s timeout=1s period=5s

gRPC

If your service speaks gRPC, you don't need to bolt on an HTTP endpoint or ship a separate probe binary. Since Kubernetes 1.27 you can point a probe straight at the gRPC health checking endpoint with the grpc field. This method is generally available on every supported cluster and needs no feature gate.

The example below uses the maintained agnhost grpc-health-checking test image, which implements the gRPC health checking protocol, so it runs with no extra setup.

1kubectl apply -f - <<'EOF'
2apiVersion: v1
3kind: Pod
4metadata:
5  name: grpc-liveness
6  namespace: probe-demo
7spec:
8  containers:
9    - name: agnhost
10      image: registry.k8s.io/e2e-test-images/agnhost:2.40
11      args: ["grpc-health-checking"]
12      ports:
13        - containerPort: 5000
14        - containerPort: 8080
15      livenessProbe:
16        grpc:
17          port: 5000
18        initialDelaySeconds: 5
19        periodSeconds: 10
20EOF

The grpc block takes an optional service field. Leave it unset to check the server's overall health, or set it to a specific registered service name. With the probe passing, the pod stays healthy with no restarts:

1NAME            READY   STATUS    RESTARTS   AGE
2grpc-liveness   1/1     Running   0          2m10s

The gRPC liveness probe: pod Running with 0 restarts, no feature gate needed on current Kubernetes.

Seeing liveness restart a container

A liveness probe earns its keep when a container that was healthy goes bad. Let's force that. The deployment below starts nginx, then immediately deletes its index page in a postStart hook. The HTTP liveness probe against / will start failing and the kubelet will restart the container.

1kubectl apply -f - <<'EOF'
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: nginx
6  namespace: probe-demo
7spec:
8  selector:
9    matchLabels:
10      app: nginx
11  replicas: 2
12  template:
13    metadata:
14      labels:
15        app: nginx
16    spec:
17      containers:
18      - name: nginx
19        image: nginx
20        ports:
21        - containerPort: 80
22        livenessProbe:
23          httpGet:
24            path: /
25            port: 80
26        lifecycle:
27          postStart:
28            exec:
29              command: ["/bin/bash", "-c", "rm -f /usr/share/nginx/html/index.html"]
30EOF

Watch the restart count climb:

1kubectl get pods -n probe-demo -w

The liveness probe is not a cure for every problem. It helps when a restart can actually clear the fault, such as an intermittent hang or a corrupted in-memory state. If the underlying issue is permanent, restarting just produces a crash loop, so reach for liveness when a fresh start has a real chance of fixing things.

Adding a readiness probe

Restarting a broken container is good, but while it is broken we should also stop sending it traffic. That is the readiness probe's job. The deployment below keeps the same self-sabotaging postStart hook and adds a readiness probe alongside the liveness probe.

1kubectl apply -f - <<'EOF'
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: nginx
6  namespace: probe-demo
7spec:
8  selector:
9    matchLabels:
10      app: nginx
11  replicas: 2
12  template:
13    metadata:
14      labels:
15        app: nginx
16    spec:
17      containers:
18      - name: nginx
19        image: nginx
20        ports:
21        - containerPort: 80
22        livenessProbe:
23          httpGet:
24            path: /
25            port: 80
26        lifecycle:
27          postStart:
28            exec:
29              command: ["/bin/bash", "-c", "rm -f /usr/share/nginx/html/index.html"]
30        readinessProbe:
31          httpGet:
32            path: /
33            port: 80
34EOF

When the readiness probe fails, the pod is removed from the Service's endpoints, so clients are not routed to it. The container is not restarted by readiness alone; that is the liveness probe's responsibility. Combining the two gives you the robust pattern most production deployments want: unhealthy pods stop receiving traffic and unhealthy containers get restarted.

Adding a startup probe

Some applications are slow to boot. If a liveness probe starts checking before the app has finished starting, it can kill the container during a perfectly normal, if lengthy, startup. The startup probe solves this. While it runs, both the liveness and readiness probes are disabled, and only once it succeeds do the other two take over.

1kubectl apply -f - <<'EOF'
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: nginx
6  namespace: probe-demo
7spec:
8  selector:
9    matchLabels:
10      app: nginx
11  replicas: 2
12  template:
13    metadata:
14      labels:
15        app: nginx
16    spec:
17      containers:
18      - name: nginx
19        image: nginx
20        ports:
21        - containerPort: 80
22        livenessProbe:
23          httpGet:
24            path: /
25            port: 80
26        lifecycle:
27          postStart:
28            exec:
29              command: ["/bin/bash", "-c", "rm -f /usr/share/nginx/html/index.html"]
30        readinessProbe:
31          httpGet:
32            path: /
33            port: 80
34        startupProbe:
35          httpGet:
36            path: /
37            port: 80
38EOF

This is the key point that trips people up: when a startup probe is configured, it disables the other two probes until it succeeds. Give it a generous failureThreshold and periodSeconds so even a slow application has time to come up.

Advanced probe configuration

Every probe shares the same five tunables. Sensible defaults are built in, but tuning them is what makes a probe fit your application instead of fighting it.

A common combination is a longer initialDelaySeconds to cover startup, a periodSeconds short enough to catch failures quickly, and a failureThreshold high enough to ride out a brief blip without overreacting.

Failing fast with a probe-level grace period

By default, when a liveness probe fails, the container gets the pod's full terminationGracePeriodSeconds to shut down. For a process that is already hung, that wait is wasted. You can set a probe-level terminationGracePeriodSeconds so a liveness failure overrides the pod value and restarts faster. This field is valid on liveness and startup probes only; the API server rejects it on readiness probes.

1kubectl apply -f - <<'EOF'
2apiVersion: v1
3kind: Pod
4metadata:
5  name: liveness-fast-restart
6  namespace: probe-demo
7spec:
8  terminationGracePeriodSeconds: 600
9  containers:
10    - name: app
11      image: registry.k8s.io/e2e-test-images/agnhost:2.40
12      args: ["liveness"]
13      livenessProbe:
14        httpGet:
15          path: /healthz
16          port: 8080
17        initialDelaySeconds: 5
18        periodSeconds: 5
19        failureThreshold: 1
20        # a liveness failure overrides the pod's 600s grace period
21        terminationGracePeriodSeconds: 5
22EOF

The agnhost liveness image starts returning 500s on /healthz shortly after boot. With failureThreshold set to 1 and a probe-level grace period of 5 seconds, the wedged container is killed and restarted promptly instead of waiting out the 600-second pod grace period:

1Liveness:  http-get http://:8080/healthz ... #failure=1
2Warning  Unhealthy  Liveness probe failed: HTTP probe failed with statuscode: 500
3Normal   Killing    Container app failed liveness probe, will be restarted
4RESTARTS 3

A failing liveness probe with a probe-level terminationGracePeriodSeconds restarting the container promptly.

Cleaning up

When you are finished experimenting, remove everything you created:

1kubectl delete pod grpc-liveness liveness-fast-restart named-port-probe -n probe-demo
2kubectl delete namespace probe-demo

Summary

We covered how to configure probes in Kubernetes, the three probe types and the questions each one answers, the four methods (exec, TCP, HTTP, and gRPC) they use to check health, and the advanced settings that let you tune probe behavior to match your application. Used together, a liveness probe restarts containers that have wedged, a readiness probe keeps traffic away from pods that are not ready, and a startup probe protects slow-booting apps from premature restarts. Add the five tunables and a probe-level grace period on top, and you have the building blocks for highly available, self-healing deployments on Civo.