Kubernetes does not prescribe a single way to build a container network. It does, however, define a model that any network implementation has to satisfy. That model is best understood as a small set of admission rules plus a set of communication goals that shape the topology.
Three admission rules
For a container network to be considered suitable, three conditions matter:
- Any two pods can communicate directly, without requiring explicit NAT for data and address translation.
- A node and a pod can communicate directly, without visible address translation.
- A pod sees the same IP that others see when they address it. There is no translation in between.
The third rule is the one that gives Kubernetes much of its simplicity. A per-pod IP remains stable all the way through the path, which helps with service management, request tracing, and monitoring. When an IP is consistent end to end, many operational cases become easier to reason about.
Four communication paths
When designing a Kubernetes system that serves the outside world, the network has to answer four questions:
- How does the external world communicate with a Service?
- How does a Service communicate with the pods behind it?
- How do pods communicate with one another?
- How do containers inside the same pod communicate?
The end goal is straightforward: traffic from the outside world should be able to reach the container that provides the application.
Underlay, overlay, and the per-pod IP
Container networking is complicated because it sits on top of the host network. Broadly, implementations fall into two camps.
Underlay networks stay close to the host network. They typically use the same subnet, the same underlying devices, and container IPs that are coordinated with the host network, often through a central allocation or a shared address plan.
Overlay networks take a different route. They do not need to request IPs from the host network's IP management components. In general, the only requirement is that the IP range does not conflict with the host network, leaving the overlay free to allocate addresses on its own.
Why did the community settle on the simple per-pod-per-IP model? In my view, it brings real advantages for service tracking and performance monitoring later on. A single IP that stays consistent from start to finish is valuable for many cases, large and small.
What a network namespace actually provides
At a narrow technical level, runC-style containers do not depend on any particular hardware. Their execution foundation is the kernel, where a process is represented by a task. If no isolation is needed, the process uses the host namespace and does not require a separate namespace data structure. With an independent network proxy or mount proxy, however, the namespace has to hold its own private data.
From the outside, an isolated network namespace looks like a network of its own. It has network devices, which may be virtual or physical. It has its own IP addresses, IP tables, routing tables, and protocol stack state, specifically for TCP/IP. It has its own iptables and IPVS state. The result feels like a completely separate network, isolated from the host network. The protocol stack code is still shared; what differs is the data structures.
Pods and network namespaces
Each pod has its own network namespace, and the pod network container shares that namespace. Kubernetes commonly recommends the loopback interface for communication between containers in the same pod, while all containers expose services through the pod IP. The root network namespace on the host can be treated as a special network namespace whose PID is 1.
Common container network designs
Container networking is one of the most diverse areas in Kubernetes. Implementations vary widely because they have to coordinate with the underlying IaaS network and make trade-offs between performance and flexible IP allocation.
The main options usually include Flannel, Calico, Canal, and WeaveNet. Most of the options in the middle of that list use policy-routing approaches similar to Calico.
Flannel is a broad, unified solution with multiple backends. Different backends implement different topologies and cover different scenarios.
Calico relies mainly on policy routing, with nodes synchronizing routes through BGP. It is feature-rich, especially in its support for Network Policy. Calico also has requirements for the underlying network: in general, MAC addresses need to be able to pass through directly, and the network cannot cross an L2 boundary.
Some community projects combine ideas from Flannel and Calico. Cilium, for example, has been presented as a graft-style innovation that brings together strengths from those approaches.
WeaveNet is worth considering when data encryption is needed. Its dynamic design can provide relatively strong encryption.
Flannel in practice
Flannel is one of the most widely used options. A typical container network first has to solve how a container packet reaches the host, and Flannel does this by adding a bridge. The backend is independent: once the packet needs to leave the host, the implementation can choose whether to encapsulate it and which encapsulation method to use.
There are three main backends:
- User-space UDP, the earliest implementation.
- Kernel VXLAN, an overlay option with better performance, though it requires a kernel version that supports VXLAN features.
- Host-gw, suitable when the cluster is not large and all nodes are in the same L2 domain. It works through a set of broadcast routing rules and offers high performance.
What Network Policy is for
The basic Kubernetes network model assumes pods can reach one another. That creates a problem: not every call chain in a cluster should be allowed to talk. Two departments, for example, may not want one department probing the other department's services. This is where policy comes in.
Network Policy uses selectors, based on labels or namespaces, to find a group of pods or the two ends of a communication path. It then describes the flow characteristics that determine whether the two sides may connect. The mechanism works like a whitelist.
Before using Network Policy, several API server flags must be enabled. More importantly, the network plugin you choose must support enforcement. Network Policy is only an object provided by Kubernetes; there is no built-in component that implements it. Whether it actually works depends on your container network solution and how completely it supports the standard. If you choose something like Flannel that does not enforce the policy, trying Network Policy will not accomplish much.
When designing a Network Policy, I think three decisions matter:
- The control target. In the spec, a podSelector or namespace selector chooses the set of pods that will receive the control.
- The direction. Should ingress be controlled, egress, or both?
- The flow description. For the chosen direction, which streams are allowed in or out? Think of a five-tuple. Selectors decide which remote endpoints are acceptable; IPBlock can allow or deny specific IP ranges; protocols and ports define the rest. Together these flow characteristics select the traffic that may pass.
Choosing a path and using policy well
In a pod network, IP is the central concept. It is the address foundation for pod communication, and it must remain consistent inside and outside the pod to match the Kubernetes model.
When comparing network options, topology is the most important factor for performance. You need to understand how a packet travels end to end: how it gets from the container to the host, whether the host encapsulates or decapsulates it, whether policy routing is involved, and how the far end extracts it again.
If you do not know much about the external network, or if you need the most broadly applicable option, Flannel with VXLAN as its backend is a reasonable choice. It does not assume that MAC addresses can connect directly or that you control the external router's routing table. If you are confident that your network supports direct L2 connectivity, Calico or Flannel with host-gw can be a better fit.
Network Policy is a powerful operational tool for precise control over ingress and egress flows. The key is to be clear about two things: what you want to control, and how the flow should be defined.
Open questions
A few questions are worth leaving open:
- Why is the interface standardized through CNI, while container networking itself still has no standard implementation built into Kubernetes?
- Why does Network Policy not have a standard controller or standard implementation, but instead leaves enforcement to the container network owner?
- Could container networking be implemented without network devices altogether, given alternatives such as RDMA that differ from TCP/IP?
- Network problems are common in operations and often hard to diagnose. Would it be worth building an open-source tool that clearly shows the network state from container to host, from host to host, and through encapsulation and decapsulation, so failures can be located quickly? As far as I know, no such tool exists today.