Kubernetes CNI plugins explained: Choosing the right network plugin

Kubernetes defines a networking model but does not implement it. It leaves the actual implementation to Container Network Interface (CNI) plugins. The CNI project, hosted under the CNCF, combines various network technologies and produces a set of plugins that can be switched in and out of Kubernetes clusters through the kubelet to offer different features and capabilities.

CNI plugins are a requirement for inter-node networking. Without one, pods on different nodes cannot communicate at all. They handle two core responsibilities: IP address management (IPAM), assigning IP addresses to pods, and packet routing, ensuring packets can travel between pods on different nodes.

For a deeper comparison of Cilium, Calico, and Flannel on Civo, see the Comparing networking solutions for Kubernetes blog.

CNI comparison

Cilium uses eBPF to implement networking directly at the Linux kernel level, bypassing iptables entirely. This gives better performance, lower latency, and richer observability. Cilium goes beyond standard Kubernetes NetworkPolicy by introducing CiliumNetworkPolicies, which support application-level filtering for protocols like HTTP, gRPC, and Kafka. It also integrates with Hubble, a network observability platform that provides visibility into network flows and security events. Cilium is a CNCF graduated project and the best choice for new clusters where performance, security, and observability are priorities.

Calico provides a purely IP-based networking solution without NAT within the cluster. It supports Kubernetes NetworkPolicy and adds its own Calico network policies for more granular control. Calico uses iptables by default with an optional eBPF mode for better performance. It is the most widely deployed CNI in production clusters and a well-tested, reliable choice for most workloads.

Flannel is the simplest CNI to set up and is Civo's default for k3s clusters. It creates a flat overlay network and requires minimal configuration. The trade-off is that Flannel does not support NetworkPolicy natively. If you need to control pod-to-pod traffic with NetworkPolicy rules you need a different CNI.

How Flannel works under the hood

For readers who want to understand the implementation detail, here is how Flannel routes packets between nodes.

Flannel creates a virtual overlay network with a large address range (typically a /16) on top of your existing VPC or physical network. Within that overlay, each node gets a smaller subnet (typically a /24). This gives Flannel three layers of IP addressing: the VPC level, the cluster overlay level, and the per-node level. The per-node subnet assignment is how Flannel maps which IP ranges live on which nodes.

The packet journey

When a pod sends a packet to another pod on a different node, the following sequence happens:

1. The packet leaves the pod through its veth pair and crosses the network namespace
2. It reaches the L3 Linux bridge on the node
3. The bridge forwards it to flannel0, which is a TUN device that simulates a network layer device
4. flannel0 operates by sending bidirectional packets between user space and the kernel
5. The flanneld daemon intercepts the packet and queries etcd to find the subnet range of the target host, resolving it to the underlying node IP on the VPC
6. flanneld sends the packet via UDP to the destination node
7. The flanneld daemon on the destination node receives it and passes it back down through the flannel0 tunnel into the kernel
8. The kernel routes it to the local bridge, up through the veth pair, across the network namespace, and into the target pod

This architecture works reliably but carries a performance cost. Every packet crossing nodes must pass between user space and kernel space twice, once on the source node and once on the destination. This is the main reason Cilium and Calico outperform Flannel on high-throughput workloads.

CNI on Civo

On Civo managed Kubernetes clusters, Flannel is installed by default. For most workloads this is sufficient. If you need NetworkPolicy support or higher networking performance, you can select Calico or Cilium as your CNI when creating your cluster via the dashboard or the Civo CLI.