How to mitigate Kubernetes runtime security threats

It’s no secret that containers and Kubernetes are by far the most used cloud native technologies. According to stackoverflow’s latest survey, 53.9% of their 65,000 respondents use Docker, and 19% use Kubernetes as well.

With the mass adoption of containers, a unique opportunity for attackers presents itself for attackers to take advantage of vulnerabilities at runtime. In a report by Sysdig, 91% of runtime scans are failing within organizations, indicating that many teams are still in reactive mode, identifying security issues after deployment rather than preventing them proactively.

In this article, we will discuss runtime security, what it is, and how you can mitigate it.

What is runtime security?

Runtime security is a proactive approach to securing hosts and applications. It involves continuous monitoring workloads for violations of user-defined security policies. As the name suggests, “runtime" refers to protecting your workloads while they are running rather than after the fact.

Why is it important?

When thinking about the impact of runtime security, it is essential to remember that attackers are constantly looking for vulnerabilities in your applications. As such, even if your code passes security scans, this does not guarantee it is safe once deployed.

Runtime security can help prevent various threats, such as credential theft, where attackers attempt to extract sensitive information like API keys or sensitive credentials from running containers.

It also protects against malware injection attempts, where compromised containers might be used as entry points for introducing malicious code. Additionally, runtime security helps detect privilege escalation attempts where threat actors try to gain unauthorized access to sensitive resources.

Embracing runtime security enables you to take a proactive approach to security incidents.

How does it work?

At a high level, an agent sits on each node in your Kubernetes cluster and monitors events such as the creation of pods, deployments or replicasets. For example, if a user attempts to create a privileged pod, the agent will block the event and log the action while sending a notification. Beyond pod creation, runtime security can monitor other events, such as network calls, file creation, or file access.

Beyond the examples above, runtime security can monitor other critical activities, such as network calls, file creation and access to sensitive files, as we will demonstrate in the hands-on example with Tetragon.

How eBPF is powering the runtime security space

Central to runtime security is a technology called eBPF (Extended Berkeley Packet Filter). For readers unfamiliar with it, eBPF is a technology that extends the Linux kernel to allow sandboxed programs to run within the operating system. It provides powerful capabilities like syscall monitoring and network tracing, which are critical for runtime security. Explaining ePBF can be tricky, checkout this article for a more in depth explanation.

Why is eBPF relevant?

eBPF is relevant to the runtime security space because of its ability to monitor syscalls and do network tracing. In more practical terms this means if a suspicious process tries to escalate privileges, runtime security agents are able to block and fire alerts about suspicious activity by monitoring the execve syscall. Later in this article we will touch on a few open source projects using eBPF.

eBPF also excels network monitoring. Through eXpress Data Path (XDP), eBPF is able to do high speed packet processing. This allows developers to add packet filtering and redirection with very minimal latency.

Top runtime security tools

Key takeaways from the comparison:

• Tetragon is ideal for users who prioritize real-time security and observability with minimal resource overhead.
• Falco stands out for its integrations and compliance focus, making it suitable for organizations with extensive regulatory requirements.
• Tracee excels in post-event forensics and is valuable for teams focused on understanding and mitigating root causes of incidents.
• KubeArmor is a versatile choice for environments with IoT or edge devices, offering extended runtime protection beyond Kubernetes workloads.

Hands-on with Tetragon

Taking things up a notch, let’s explore how you can protect your workloads using Tetragon. To follow along with this section of this post, you will need the following installed locally:

• A Civo Account
• Kubectl installed
• Helm installed
• Civo CLI (optional)

Create a cluster

If you haven’t already, create a new cluster using the Civo CLI

1civo k3s create --create-firewall --nodes 1 -m --save --switch --wait tetragon-lab

This will provision a one node cluster named tetragon-lab in your Civo account.

Install Tetragon

To install and deploy Tetragon, run the following commands:

Add the Helm repo:

1helm repo add cilium https://helm.cilium.io
2helm repo update

Install the Tetragon chart:

1helm install tetragon ${EXTRA_HELM_FLAGS[@]} cilium/tetragon -n kube-system

Verify the installation:

1kubectl rollout status -n kube-system ds/tetragon -w

Output is similar to:

1daemon set "tetragon" successfully rolled out

Deploy a sample application

To test Tetragon, we will need an application to test it on; thankfully Cilium has a deployment we can use:

1kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.3/examples/minikube/http-sw-app.yaml

Verify the pods were created:

1kubectl get pods

Output is similar to:

1NAME                                     	READY   STATUS    RESTARTS   AGE
2deathstar-b4b8ccfb5-dsrnt        			1/1  	Running   0          8s
3deathstar-b4b8ccfb5-lt6fm        			1/1  	Running   0          8s
4xwing                                       1/1     Running   0          7s
5tiefighter                                  1/1     Running   0          7s

Protect sensitive files

The xwing pod contains a file called default.json, which contains some sensitive information. To view the contents of the file, run the following command:

1kubectl exec -ti xwing -- bash -c 'cat default.json'

Output is similar to:

1{
2  "private": [
3{ "id": 1, "body": "secret information" }
4  ],
5  "public": [
6{ "id": 1, "body": "public information" }
7  ],
8  "auth-header-required": [
9{ "id": 1, "body": "super secret information" }
10  ]
11}

To protect access to this file, we can leverage a custom resource definition Tetragon provides called TracingPolicy, a tracing policy allows you to define custom, kernel-level tracing policies which can monitor specific system calls or kernel functions and set conditions for when these calls should be traced or acted upon, in our case, we want the system call to open the default.json to be acted upon, i.e., terminated.

Apply the following tracing policy

1kubectl apply -f -
2apiVersion: cilium.io/v1alpha1
3kind: TracingPolicy
4metadata:
5 name: "prevent-default-json-read"
6spec:
7 kprobes:
8 - call: "security_file_permission"
9   syscall: false
10   return: true
11   args:
12   - index: 0
13     type: "file"
14   - index: 1
15     type: "int"
16   returnArg:
17     index: 0
18     type: "int"
19   returnArgAction: "Post"
20   selectors:
21   - matchArgs:     
22     - index: 0
23       operator: "Equal"
24       values:
25       - "/default.json"
26     - index: 1
27       operator: "Equal"
28       values:
29       - "4" # MAY_READ
30     matchActions:
31     - action: Sigkill
32EOF

With the policy applied, let’s try to access the file once more:

1kubectl exec -ti xwing -- bash -c 'cat default.json'

Output is similar to:

1command terminated with exit code 137

We can verify the command was indeed terminated by Tetragon, by taking a look at the corresponding event generated.

1kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | jq

Output is similar to:

1"process_exec": {
2	"process": {
3  	"exec_id": "azNzLXRldHJhZ29uLWxhYi1mZGVjLWY2NWM0Zi1ub2RlLXBvb2wtNTE5My1zamtsZToxMDM0MjU4MDYzMTE3MjE6MzIyMg==",
4  	"pid": 3222,
5  	"uid": 0,
6  	"cwd": "/",
7  	"binary": "/usr/bin/bash",
8  	"arguments": "-c \"cat default.json\"",
9  	"flags": "execve rootcwd clone",
10  	"start_time": "2024-09-15T11:39:11.959890661Z",
11  	"auid": 4294967295,
12  	"pod": {
13    	"namespace": "default",
14    	"name": "xwing",
15    	"container": {
16      	"id": "containerd://430b112a9a05082c7fb8d4c0c012e4e2cf40d147c68fca54e57fbbb78b5da1a1",
17      	"name": "spaceship",
18      	"image": {
19        	"id": "quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603",
20        	"name": "sha256:adcc2d0552708b61775c71416f20abddad5fd39b52eb4ac10d692bd19a577edb"
21      	},
22      	"start_time": "2024-09-14T17:39:24Z",
23      	"pid": 36
24    	},
25    	"pod_labels": {
26      	"app.kubernetes.io/name": "xwing",
27      	"class": "xwing",
28      	"org": "alliance"
29    	},
30    	"workload": "xwing",
31    	"workload_kind": "Pod"
32  	},
33  	"docker": "430b112a9a05082c7fb8d4c0c012e4e",
34  	"parent_exec_id": "azNzLXRldHJhZ29uLWxhYi1mZGVjLWY2NWM0Zi1ub2RlLXBvb2wtNTE5My1zamtsZTozODYzNDg1NTQxNjkzMDoyODU1Nw==",
35  	"tid": 3222
36	}
37  },
38  "node_name": "k3s-tetragon-lab-fdec-f65c4f-node-pool-5193-sjkle",
39  "time": "2024-09-15T11:39:11.959888807Z"
40}

We get a rich response, which includes information about the time, node, pod, and arguments which were used to execute the command.

Clean up

If you followed this section of the tutorial, you might want to delete some of the resources we provisioned

Uninstall Tetragon:

1helm uninstall tetragon -n kube-system

Delete the Kubernetes cluster:

1civo k3s rm tetragon-lab

Conclusion

Runtime security is reshaping how teams protect their workloads, emphasizing proactive measures to mitigate threats in real-time. Tools like Tetragon, Falco, Tracee, and KubeArmor each bring unique strengths to runtime security, allowing developers to tailor their strategies based on their environment and priorities. Whether you’re securing Kubernetes workloads, addressing compliance requirements, or analyzing post-incident forensics, these tools provide robust solutions to modern runtime security challenges.

For those exploring container security further, technologies like eBPF are at the forefront of innovation, powering advanced solutions not only in runtime security but also across service meshes and supply chain security.

Interested in learning more about container security? Here are some ideas:

• Check out this article on how eBPF is helping out in the service mesh space
• Supply security is a big threat to containers, learn how you can defend your images using sigstore tooling