In Linux, CPU, memory, and I/O give you a set of metrics to chase. Network problems are often less polite. The complaint is usually some version of it doesn’t work: a ping gets no reply, the physical link is down, DNS resolves to the wrong place, or a container cannot reach another container because of network isolation. Then there is the slower, more annoying category: the bandwidth was never large to begin with, a burst of requests fills it, and everything starts timing out—at the interface, in nginx, or inside the request itself.
What to measure
sar -n DEV 1 gives a per-interface view of network throughput:
[root@Linkin ~]# sar -n DEV 1
Linux 3.10.0-1062.9.1.el7.x86_64 (Linkin) 07/04/2020 _x86_64_ (2 CPU)
03:03:40 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
03:03:41 PM br-22dd849a79f5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:03:41 PM br-dac186ab0c70 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:03:41 PM veth860c89a 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:03:41 PM eth0 5.00 4.00 0.30 2.43 0.00 0.00 0.00
03:03:41 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:03:41 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rxpck/s and txpck/s are received and sent packets per second, or PPS. rxkB/s and txkB/s are receive and send throughput in KB/s.
iperf3 tests speed and bandwidth. Start the server:
# -s starts the server, -i sets reporting interval, -p sets listening port
$ iperf3 -s -i 1 -p 10000
# -c starts the client; 192.168.10.30 is the target server IP
# -b sets target bandwidth (bits/s)
# -t sets test duration
# -P sets concurrency, -p sets target server listening port
$ iperf3 -c 192.168.10.30 -b 1G -t 15 -P 2 -p 10000
Do not underestimate DNS. Resolution problems are common, especially when caching is involved.
# Install
yum install bind-utils
# Use
nslookup www.baidu.com
# +trace enables trace query
# +nodnssec disables DNS security extensions
dig +trace +nodnssec www.baidu.com
dig linkinstar.com @localhost
tcpdump is a powerful packet capture tool with many options. Common filters look like this:
# -i eth0 specifies the network interface
# -nn host 10.0.2.1 filters by host
# -nn tcp filters by protocol
# -nn dst port 80 filters by port
Many network tuning knobs live in /etc/sysctl.conf (vi /etc/sysctl.conf). Treat them with respect: buffer maximums, queue lengths, and related values can have wide effects. If there is no active problem, leaving them alone is often better. When tuning is needed, hand it to someone with operational experience. Dropping in a config file copied from the internet is risky—when something breaks, you may not know which value caused it.
A practical order
Start with reachability. Ping is crude, but if packets cannot get through, other analysis is wasted. Next, check whether request volume or traffic is overwhelming the link. Then look at connection counts and the TCP/UDP state distribution; those often show what kind of problem you have.
Network trouble splits into two broad outcomes: unreachable and slow. The reasons behind them are messy. Some common cases:
Firewall
A firewall is a rite of passage. A service is unreachable because the firewall is still on or the port was never opened. On cloud instances there is another layer: the OS firewall may be off, but the provider’s security group rules still need to be opened in the web console.
DNS resolution
DNS has two sides. External resolution may fail because the DNS server is not configured or the resolver itself is broken. Internal DNS also matters; internal services often use DNS names, and trouble usually appears as a failed ping, then a dig query. Load balancers such as nginx can cache DNS answers; if a backend IP changes, the cached answer can send traffic to the wrong place or break the load-balancing layer.
A large number of TIME_WAIT sockets
This usually points to a program bug: many clients connect and then close abnormally. TIME_WAIT is a TCP state, and the count can be revealing.
Many SYN_RECV entries
This may mean a SYN flood. Kernel parameter tuning can reduce the impact, but for an internet-facing service with frequent attacks, blocking the source IP is often the first move, along with cloud provider protection or a CDN.
Container networking
Container networking still produces unreachable addresses. Whether using host networking or a bridge, services sometimes cannot be reached. Packet capture is often the most reliable way through. Ping the target, capture the ICMP packets, and follow them across eth0, docker0, flannel0, and other interfaces.
Too many requests
Sometimes the network is not broken; response time is just too long. Users call it lag, and a poor network makes it worse. If bandwidth is saturated, even logging into the server can become difficult. Find out why. Often the underlying cause is slow data queries or an I/O bottleneck; database query optimization is frequently the key fix. Caching can improve performance immediately. Security policies and rate limiting also matter: every server has a limit, and a flood of requests can simply fill the pipe. Without sensible controls, a service can be knocked over quickly.
Network protocols wrap data in layers so it can cross links with some reliability. Learning the protocols and their principles is useful because it shortens the path from symptom to cause and helps prevent repeat failures. A network engineer may describe the job as ping was failing, now it works, but the simple result can hide ports, protocols, DNS, and any number of steps in between. When a network problem appears, avoid jumping around; start at the beginning and follow the path until the weak point shows itself.