Why are network policies in Kubernetes so hard to understand?

In Kubernetes, the concept of network policies allows you to control the traffic flow within a cluster. Essentially, by creating policies, you determine which pods can access others, streamlining the process of restricting traffic between different applications within the cluster.

You will also run into many microservices in different namespaces within a Kubernetes environment. These applications are run as pods, which in turn run containers. These containers are your applications and are capable of communicating with every other pod, either directly or through services. However, this open communication model isn't always secure. Fortunately, Kubernetes offers the concept of network policy, implemented by various network providers, to provide out-of-the-box functionality for controlling this aspect securely.

The community often voices that network policies are complex, but by exploring concrete examples, we can gain a clearer understanding of how they work in action.

Prerequisites

To follow along with this tutorial, you need to ensure you have the following in place:

Creating a Kubernetes cluster with Cilium

To begin with, let’s create a Civo Kubernetes cluster with Cilium as the CNI. You can create the cluster from the UI or the CLI.

For the purpose of this tutorial, we will be using Civo Kubernetes, but you can go with any Kubernetes cluster and CNI where network policies will work.

Interacting with the cluster

Once you have the cluster created, you can export the KUBECONFIG variable in your terminal and point it to the downloaded kubeconfig file for the cluster. From here, you should be able to interact with the cluster:

1kubectl get nodes
2NAME                                                   STATUS   ROLES    AGE   VERSION
3k3s-networkpolicies-7aed-fb151a-node-pool-71b0-krmn8   Ready    <none>   71s   v1.28.2+k3s1
4k3s-networkpolicies-7aed-fb151a-node-pool-71b0-jl4q3   Ready    <none>   69s   v1.28.2+k3s1
5k3s-networkpolicies-7aed-fb151a-node-pool-71b0-6ko85   Ready    <none>   67s   v1.28.2+k3s1

Create 2 namespaces dev1 and dev2:

1kubectl create ns dev1
2namespace/dev1 created
3kubectl create ns dev2
4namespace/dev2 created

Create a pod demo1 and pod demo2 in respective namespaces with NGINX image:

1kubectl run demo1 --image=nginx -n dev1
2pod/demo1 created
3kubectl run demo2 --image=nginx -n dev2
4pod/demo2 created
5
6kubectl get pods -owide -n dev1
7NAME    READY   STATUS    RESTARTS   AGE   IP           NODE                                                   NOMINATED NODE   READINESS GATES
8demo1   1/1     Running   0          97s   10.0.1.147   k3s-networkpolicies-7aed-fb151a-node-pool-71b0-6ko85   <none>           <none>
9
10kubectl get pods -owide -n dev2
11NAME    READY   STATUS    RESTARTS   AGE   IP           NODE                                                   NOMINATED NODE   READINESS GATES
12demo2   1/1     Running   0          94s   10.0.1.185   k3s-networkpolicies-7aed-fb151a-node-pool-71b0-6ko85   <none>           <none>

Testing the connectivity

Let’s now test the connectivity of one pod from another:

1kubectl exec demo1 -n dev1 -- curl 10.0.1.185
2
3  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
4                                 Dload  Upload   Total   Spent    Left  Speed
5  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE html>
6<html>
7<head>
8<title>Welcome to nginx!</title>
9<style>
10html { color-scheme: light dark; }
11body { width: 35em; margin: 0 auto;
12font-family: Tahoma, Verdana, Arial, sans-serif; }
13</style>
14</head>
15<body>
16<h1>Welcome to nginx!</h1>
17<p>If you see this page, the nginx web server is successfully installed and
18working. Further configuration is required.</p>
19
20<p>For online documentation and support please refer to
21<a href="http://nginx.org/">nginx.org</a>.<br/>
22Commercial support is available at
23<a href="http://nginx.com/">nginx.com</a>.</p>
24
25<p><em>Thank you for using nginx.</em></p>
26</body>
27</html>
28100   615  100   615    0     0   405k      0 --:--:-- --:--:-- --:--:--  600k

In this code, we exec into pod demo1 in the dev1 namespace and try to curl the IP of pod demo2 in dev2 namespace, this shows that any pod can communicate with any other pod in any namespace.

Now, how can we fix this? You are right! Using NetworkPolicy.

Creating a network policy

Let’s create a network policy in the dev2 namespace so that no traffic can reach the pods in the dev2 namespace.

1cat << EOF | kubectl apply -f -
2apiVersion: networking.k8s.io/v1
3kind: NetworkPolicy
4metadata:
5  name: deny-all
6  namespace: dev2
7spec:
8  podSelector: {}
9  policyTypes:
10  - Ingress
11EOF
12networkpolicy.networking.k8s.io/deny-all created

Above is the manifest for all the pods in dev2 namespace. The purpose of this policy is to restrict all incoming traffic to the pods within the dev2 namespace. Here's a breakdown of how it works:

apiVersion: networking.k8s.io/v1: Specifies the API version for the network policy resource.
kind: NetworkPolicy: This specifies the kind of Kubernetes resource you're defining, which in this case is a Network Policy.
metadata: Contains metadata about the network policy, including its name (deny-all) and the namespace (dev2) it is applied to.
spec: Defines the specifications of the Network Policy.
- podSelector: This is set to an empty object ({}), which means the policy applies to all pods within the specified namespace (dev2 in this case). You could specify label selectors here if you wanted to target specific pods.
- policyTypes: Specifies the types of policies. In this case, it includes - Ingress, which means the policy will apply to incoming traffic to the pods. By not specifying Egress in the policy types, this policy does not restrict egress (outgoing) traffic from the pods.
Ingress: Since no rules are defined under the ingress section (which is implicitly understood from the lack of an ingress field under spec), it means no inbound connections are allowed to any pods in the dev2 namespace. You would define rules here if you wanted to allow specific types of ingress traffic.

The following outcome should appear:

1kubectl exec demo1 -n dev1 -- curl --connect-timeout 5 10.0.1.185
2  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
3                                 Dload  Upload   Total   Spent    Left  Speed
4  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
5curl: (28) Failed to connect to 10.0.1.185 port 80 after 5001 ms: Timeout was reached
6command terminated with exit code 28

You can see that after applying this policy no traffic can reach the pods in the dev2 namespace.

Receiving incoming TCP traffic

Next, let’s try to cover a couple more scenarios to understand the concept more clearly. Next, we will create a network policy that would allow the traffic from pods in dev1 namespace to dev2 namespace over port 80.

1cat << EOF | kubectl apply -f -
2apiVersion: networking.k8s.io/v1
3kind: NetworkPolicy
4metadata:
5  name: kube-demo
6  namespace: dev2
7spec:
8  podSelector: {}
9  ingress:
10  - from:
11    - namespaceSelector:
12        matchLabels:
13          kubernetes.io/metadata.name: dev1
14    ports:
15    - protocol: TCP
16      port: 80
17EOF
18networkpolicy.networking.k8s.io/kube-demo created

This network policy allows pods in the dev2 namespace to receive incoming TCP traffic on port 80 from any pod in the dev1 namespace. No other ingress traffic is permitted by this policy, effectively isolating the pods in dev2 from unwanted or unsolicited incoming traffic from pods in other namespaces, except for the allowed traffic from dev1.

Network policy example

Interestingly, if you have a previous policy of deny all and this policy as well, it will be a combination, and the resultant will allow traffic on port 80 from pods in dev1 namespace to dev2 namespace.

You can check the output below:

1kubectl exec demo1 -n dev1 -- curl --connect-timeout 5 10.0.1.185
2  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
3                                 Dload  Upload   Total   Spent    Left  Speed
4  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--   100   615  100   615    0     0   267k      0 --:--:-- --:--:-- --:--:--  300k
5<!DOCTYPE html>
6<html>
7<head>
8<title>Welcome to nginx!</title>
9<style>
10html { color-scheme: light dark; }
11body { width: 35em; margin: 0 auto;
12font-family: Tahoma, Verdana, Arial, sans-serif; }
13</style>
14</head>
15<body>
16<h1>Welcome to nginx!</h1>
17<p>If you see this page, the nginx web server is successfully installed and
18working. Further configuration is required.</p>
19
20<p>For online documentation and support please refer to
21<a href="http://nginx.org/">nginx.org</a>.<br/>
22Commercial support is available at
23<a href="http://nginx.com/">nginx.com</a>.</p>
24
25<p><em>Thank you for using nginx.</em></p>
26</body>
27</html>

To check that traffic is not allowed from any other namespace or on any other port, let's create a pod in the dev2 namespace listening on a different port than 80 and a pod in default namespace as well:

1kubectl run default --image=nginx
2pod/default created
3
4kubectl get pods -owide
5NAME                                 READY   STATUS      RESTARTS   AGE    IP           NODE                                                   NOMINATED NODE   READINESS GATES
6default                              1/1     Running     0          26s    10.0.2.237   k3s-networkpolicies-7aed-fb151a-node-pool-71b0-jl4q3   <none>           <none>
7
8kubectl exec default -- curl --connect-timeout 5 10.0.1.185
9  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
10                                 Dload  Upload   Total   Spent    Left  Speed
11  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
12curl: (28) Failed to connect to 10.0.1.185 port 80 after 5000 ms: Timeout was reached
13command terminated with exit code 28

Now, lets try to create a pod and service to listen on port 8080:

1kubectl run http-echo --image=hashicorp/http-echo -n dev2 -- -listen=:8080 -text="Hello from http-echo"
2
3pod/http-echo created
4
5kubectl get pods -n dev2 -owide
6NAME         READY   STATUS    RESTARTS   AGE   IP           NODE                                                   NOMINATED NODE   READINESS GATES
7demo2        1/1     Running   0          80m   10.0.1.185   k3s-networkpolicies-7aed-fb151a-node-pool-71b0-6ko85   <none>           <none>
8http-echo   1/1     Running   0          26s   10.0.1.115   k3s-networkpolicies-7aed-fb151a-node-pool-71b0-6ko85   <none>           <none>

Creating a service

Create the service using the following:

1kubectl expose pod http-echo -n dev2 --port=8080
2service/http-echo exposed
3kubectl get svc -n dev2
4NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
5http-echo   ClusterIP   10.98.167.221    <none>        8080/TCP   28s

1kubectl exec demo1 -n dev1 -- curl --connect-timeout 5 10.98.167.221
2  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
3                                 Dload  Upload   Total   Spent    Left  Speed
4  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
5curl: (28) Failed to connect to 10.98.167.221 port 80 after 5000 ms: Timeout was reached
6command terminated with exit code 28

The kube-demo NetworkPolicy that was applied to all pods in the dev2 namespace (since podSelector is empty), allows them to receive incoming TCP traffic on port 80 only from pods within the dev1 namespace (identified by the label kubernetes.io/metadata.name: dev1). All other incoming traffic from different namespaces, or different ports, will be denied by default, as this is the standard behavior of Kubernetes NetworkPolicies when they are applied to a set of pods. We proved that with a pod in the default namespace and also by running a pod on port 8080 and trying to connect to it also failed.

Summary

Throughout this tutorial, you should now have a better understanding of how you can apply network policies within your Kubernetes cluster to limit the ingress/egress traffic for the pods. Another interesting way to learn more about this topic is by using this tool which allows you to create network policies for Kubernetes and gain a better understanding of the concept.

If you want other resources to keep learning more about this topic, I recommend checking out the following:

Why are network policies in Kubernetes so hard to understand?

Prerequisites

Creating a Kubernetes cluster with Cilium

Interacting with the cluster

Testing the connectivity

Creating a network policy

Receiving incoming TCP traffic

Network policy example

Creating a service

Summary

Further Reading

Exploring network policies with Cilium

Networking 101 - IP addressing

Networking 101 - IP packets

Exploring network policies with Cilium

Networking 101 - IP addressing

Networking 101 - IP packets

Company

Company

Public Cloud

Public Cloud

Private Cloud

Private Cloud

Civo AI

Civo AI

Solutions

Solutions

Resources

Resources

Contact

Contact

Legal

Social