Networking Under the Hood: Diagnosing Latency and Packet Drops in the Linux Kernel

You've been there. Grafana shows p99 latency spiking on one service. The load balancer logs a smattering of 502s. You ping the box — 0.2ms, 0% loss. You traceroute it — clean. The network team says "the network is fine," and by their definition, they're right. The packets are arriving at the NIC.

The problem is that arriving at the NIC is maybe 30% of a packet's journey. Between the wire and your application's read() call sits a deep pipeline: DMA into a ring buffer, softirq processing, protocol stack traversal, netfilter hooks, socket queue accounting, and finally a wakeup of your process. Every stage has a queue, every queue has a limit, and almost every limit fails silently — the kernel increments an obscure counter and moves on. No log line. No error. Just a retransmission 200ms later, which your user experiences as latency and your LB experiences as a timeout.

ping tests exactly none of this. ICMP echo is answered in the kernel's fast path — it never touches your listen backlog, your socket buffers, your conntrack table, or your application. traceroute maps router hops, not kernel queues. In a microservices world where "the network" includes veth pairs (the virtual cable linking each pod's network namespace to the host), bridges, VXLAN encapsulation, iptables/nftables chains, and IPVS (the in-kernel L4 load balancer behind kube-proxy's ipvs mode) — all inside the Linux kernel — the tools you need are the ones that read the kernel's own accounting.

This post is the diagnostic playbook I actually use: where packets get dropped, which counters betray each drop point, and how to walk an incident from symptom to sysctl fix. Everything here is copy-pasteable.

1. The Anatomy of a Packet Drop

The receive path, one paragraph per stage

NIC → ring buffer. The NIC DMAs incoming frames — writes them straight into host RAM, no CPU involved — into a fixed-size RX ring buffer, a circular array of descriptors: slots, each pointing at a buffer the NIC may fill. If the ring is full because the kernel isn't draining it fast enough, the NIC drops the frame in hardware. The kernel never sees it. Only ethtool -S driver statistics record it (rx_missed_errors, rx_no_buffer_count, or similar — naming is driver-specific).

Ring buffer → softirq (NAPI). The NIC raises an interrupt; the real work happens later in NET_RX softirq context — a softirq is deferred kernel work that runs just after the hard interrupt returns, so the interrupt handler itself stays microscopic — scheduled via NAPI, which polls frames off the ring in batches instead of taking an interrupt per frame. Each frame becomes an sk_buff ("skb") — the kernel's universal packet container. An skb is a metadata structure (pointers to the packet data, protocol headers, timestamps, the owning socket, netfilter state) that travels through the entire stack; layers adjust header pointers rather than copying data. When you see kernel memory attributed to networking, it's largely skbs and the buffers they reference.

Softirq budget. NAPI polling is budgeted (net.core.netdev_budget, netdev_budget_usecs) so networking can't starve the CPU. If a core exhausts its budget with packets still pending, that's a time squeeze — not a drop, but deferred processing, i.e. latency. If packets are funneled to a per-CPU backlog queue (as with RPS — software steering of packets to other CPUs — or older non-NAPI drivers) and that queue exceeds net.core.netdev_max_backlog, packets are dropped, counted only in /proc/net/softnet_stat.

Protocol stack → socket queue. The skb climbs IP → netfilter hooks (the attach points where your iptables/nftables rules actually run) → TCP/UDP, and lands in a socket receive queue, bounded by the socket's receive buffer (sk_rcvbuf, governed by net.core.rmem_max and net.ipv4.tcp_rmem). If the application is slow to read() and the buffer fills: for UDP, the datagram is dropped (UdpRcvbufErrors); for TCP, the window closes — zero-window, the sender stalls, which is latency — and under memory pressure the kernel starts "pruning" already-received segments (TcpExtPruneCalled, TcpExtRcvPruned), forcing retransmissions.

TCP connection setup has two queues of its own. This is the stage that causes the most production incidents, so it gets its own section.

The SYN queue and the accept queue

Every TCP listener maintains two queues:

• The SYN queue (incomplete connections): SYN received, SYN-ACK sent, waiting for the final ACK. Sized by net.ipv4.tcp_max_syn_backlog.
• The accept queue (completed connections): handshake done, waiting for the application to call accept(). Sized by min(application_backlog, net.core.somaxconn) — where application_backlog is the second argument your app passes to listen(), and the sysctl silently clamps it.

Here's the killer detail: when the accept queue is full and a final ACK arrives, the default kernel behavior (net.ipv4.tcp_abort_on_overflow = 0) is to silently ignore the ACK. The client believes the connection is established — its connect() returned success — and starts sending data. The server retransmits SYN-ACKs; the client's data goes unacknowledged. Eventually one side gives up, seconds later. From the client's perspective: a connection that opened fine and then hung. From your LB's perspective: an upstream timeout → 502/504. From the server's logs: nothing, because the application never saw the connection.

This is the canonical "silent drop." We'll hunt it down in section 3.

Other silent culprits worth naming

• Socket memory limits. Global TCP memory pressure (net.ipv4.tcp_mem) causes collapse/prune behavior — the kernel compacting, then outright discarding, already-queued receive data to claw memory back — across all sockets. The nstat counters TCPRcvCollapsed, PruneCalled, and TCPMemoryPressures are the tell.
• conntrack exhaustion. On any box doing NAT or stateful firewalling — which is every Kubernetes node — a full connection-tracking table drops new flows outright. dmesg shows nf_conntrack: table full, dropping packet, and it's the only drop in this list that actually logs. Check net.netfilter.nf_conntrack_count against nf_conntrack_max.
• qdisc drops on transmit. The egress queueing discipline — the qdisc, the kernel's transmit buffer/scheduler sitting between the IP stack and the NIC (fq_codel, pfifo_fast, …) — drops when the NIC can't drain fast enough. Visible only via tc -s qdisc.
• MTU / PMTUD black holes. Overlay networks shrink the effective MTU (VXLAN eats 50 bytes, WireGuard 60–80, IPsec varies). TCP sets the Don't Fragment bit and relies on ICMP "Fragmentation Needed" messages to discover the path MTU (that's PMTUD). If a firewall drops ICMP — depressingly common — small packets flow and large packets vanish. The symptom is beautifully weird: TLS handshakes succeed, small API responses work, large responses hang forever. The handshake fits in small packets; the first full-MSS data segment — a maximum-size packet, exactly the kind that no longer squeezes through the shrunken path — doesn't.

2. The Diagnostic Toolbox

ss — the socket X-ray

Forget netstat for socket inspection; ss reads netlink directly and exposes kernel TCP state that netstat never could.

Listener queue depth — the first command in any 502 investigation:

ss -lnt

State   Recv-Q  Send-Q  Local Address:Port
LISTEN  129     128     0.0.0.0:8080

For sockets in LISTEN state these columns are overloaded: Recv-Q is the current accept-queue occupancy, Send-Q is the accept-queue limit. Recv-Q at or above Send-Q means the queue is overflowing right now and connections are being silently discarded. This single line has closed more "intermittent timeout" tickets for me than any other command.

Per-connection TCP internals:

ss -nti dst 10.0.4.12

ESTAB 0 148744 10.0.4.7:44832 10.0.4.12:5432
     cubic wscale:7,7 rto:204 rtt:1.2/0.4 cwnd:10 ssthresh:7
     bytes_retrans:14480 retrans:0/12 unacked:8 lost:2 sacked:4
     notsent:120000 pacing_rate 92.3Mbps delivery_rate 41.1Mbps

How to read the fields that matter:

• rtt:1.2/0.4 — smoothed RTT / variance in ms. If this reads 1.2ms but your app sees 200ms, the network path is innocent; look at queues and the application.
• rto:204 — the retransmission timeout. Every lost packet costs at least this much. On low-RTT LANs the floor is ~200ms, which is why a 0.1% loss rate produces those characteristic ~200ms spikes in your p99.
• retrans:0/12 — current/total retransmissions on this connection. A nonzero total on an intra-VPC connection means something is dropping.
• cwnd:10 with a shrunken ssthresh — the congestion window (how much data TCP will keep in flight unacknowledged) collapsed, and a shrunken ssthresh is the scar tissue: this connection has experienced loss recently.
• notsent:120000 — 120KB sitting in the send buffer that TCP hasn't even attempted to transmit. The bottleneck is the path or the receiver, not your application.
• A large plain Send-Q (second column — on ESTAB sockets it reverts to its normal meaning, bytes sent but unacknowledged) with small notsent: the receiver or path is slow to ACK.

Socket memory accounting:

ss -ntm dst 10.0.4.12

skmem:(r0,rb2985780,t0,tb87040,f0,w0,o0,bl0,d37)

Decode: r = bytes currently in the receive queue, rb = receive buffer limit, t/tb = same for transmit, w = queued write bytes, and — the one everyone misses — d37 = 37 packets dropped on this socket because the receive buffer was full. A nonzero d on a socket whose owning process looks "healthy" means that process isn't reading fast enough. That's not a network problem; that's a blocked event loop, a GC pause, or a saturated thread pool.

nstat — the kernel's confession log

netstat -s works, but nstat is the modern interface and, critically, shows deltas since its last invocation — perfect for "is this counter moving right now?":

# Deltas of the counters that matter, every 2 seconds:
watch -n2 'nstat | grep -Ei "listen|retrans|prune|collapse|overflow|backlog|abort"'
 
# Or the full absolute dump:
nstat -az | grep -Ei "listendrops|listenoverflows|retranssegs|pruned|backlogdrop"

The rogues' gallery:

A note on baselines: on any busy fleet these counters are rarely zero. What you care about is the rate during the symptom window versus quiet periods. nstat's delta behavior makes that trivial.

Interface and driver level: /proc/net/dev, ethtool, softnet

Quick per-interface totals:

cat /proc/net/dev
# or nicer:
ip -s -s link show eth0

The doubled -s adds detail rows: rx: dropped overrun mcast / tx: dropped carrier collsns. Kernel-level dropped here usually means out-of-buffer or an unwanted protocol; overrun means frames arrived faster than the CPU drained the ring.

Driver/hardware statistics — where NIC ring drops live:

ethtool -S eth0 | grep -iE 'drop|miss|err|fifo|full' | grep -v ' 0$'

Names vary by driver (rx_missed_errors on igb, rx_out_of_buffer on mlx5, per-queue rx_queue_N_drops on virtio). Any of these climbing means the ring buffer is overflowing. Check and raise the ring:

ethtool -g eth0          # shows preset max vs current
ethtool -G eth0 rx 4096  # raise RX ring to hardware max

Bigger rings absorb bursts at the cost of a little latency and memory — almost always the right trade on a server. Caveat: ethtool -G briefly resets the interface on many drivers; do it in a maintenance window or behind working LB health checks.

Softirq-level pressure:

awk '{ printf "cpu%-3d processed=%d dropped=%d time_squeeze=%d\n", NR-1, strtonum("0x"$1), strtonum("0x"$2), strtonum("0x"$3) }' /proc/net/softnet_stat

Column 2 (dropped) counts packets discarded because the per-CPU backlog exceeded net.core.netdev_max_backlog. Column 3 (time_squeeze) counts NAPI budget exhaustions. Climbing dropped on specific CPUs usually also means your IRQ affinity — which CPU each NIC queue's interrupt is pinned to — is funneling all traffic to one core. Check /proc/interrupts and fix the spreading (RSS is the NIC hashing flows across multiple hardware queues; RPS is its software fallback) before you just crank the backlog sysctl.

Egress qdisc drops — the transmit side everyone forgets:

tc -s qdisc show dev eth0
# qdisc fq_codel 0: root ... dropped 1182, overlimits 0

tcpdump — beyond host and port

The filters that earn their keep in kernel-level debugging:

# All SYNs and RSTs — connection churn, refusals, and who's aborting whom:
tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'
 
# Only RSTs — an unexpected RST storm from an upstream is a classic
# backlog-overflow or conntrack-drop signature:
tcpdump -ni eth0 'tcp[tcpflags] & tcp-rst != 0'
 
# SYN retries specifically (SYN set, ACK clear) toward a service —
# every one you see is a handshake the server ignored at least once:
tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and dst port 8080'
 
# Zero-window advertisements — the receiver saying "stop, I'm full"
# (window is bytes 14:2 of the TCP header; pair with a host filter in practice):
tcpdump -ni eth0 'tcp[14:2] = 0 and tcp[tcpflags] & tcp-rst == 0'
 
# ICMP fragmentation-needed — if you suspect a PMTUD black hole, run this
# where the ICMP *should* arrive; silence while large transfers hang = black hole:
tcpdump -ni eth0 'icmp[icmptype] = 3 and icmp[icmpcode] = 4'

Two pro moves:

Capture to a file, analyze offline. tcpdump -ni eth0 -s 128 -w /tmp/cap.pcap -W 5 -C 100 — the 128-byte snaplen keeps headers and drops payload, and the ring of 5×100MB files won't fill your disk. Then let tshark find retransmissions for you; retransmissions aren't a wire-format flag, they're an analysis of repeated sequence numbers:

tshark -r /tmp/cap.pcap \
  -Y 'tcp.analysis.retransmission or tcp.analysis.rto or tcp.analysis.zero_window' \
  -T fields -e frame.time_relative -e ip.src -e ip.dst -e tcp.analysis.flags

Capture on both sides. A packet visible leaving the client but never arriving in the server's tcpdump was dropped between the two TCP stacks. And since tcpdump on the receive side taps in early (before netfilter, before socket queues), a packet visible in the server-side capture that the application never received was dropped inside the server's own kernel. That bisection instantly tells you whether to blame the fabric or the host.

eBPF — asking the kernel directly

Counters tell you drops are happening; eBPF tells you where and why, per-packet, on a live production kernel, with negligible overhead. Since kernel 5.17, kfree_skb — the function that frees a dropped skb — carries an explicit drop reason enum. One bpftrace one-liner against its tracepoint — a stable kernel hook you can attach to on a live box, nothing to deploy or restart — is the single highest-signal tool on this page:

# Histogram of every packet-drop reason on the box, live:
bpftrace -e 'tracepoint:skb:kfree_skb { @[args->reason] = count(); }'

Run it during the symptom window. Reasons like TCP_OVERFLOW (listen overflow), SOCKET_RCVBUFF, NO_SOCKET, and NETFILTER_DROP map directly to root causes. On older kernels, dropwatch -l kas gives you the kernel symbol where drops occur instead.

BCC tools (the bcc-tools package on most distros — a stock collection of ready-made eBPF utilities) — precompiled answers to common questions:

tcpretrans -c    # live retransmissions per flow — WHICH connections are suffering
tcpdrop          # stack trace of every TCP drop: the exact kernel path that discarded the skb
tcpconnlat 10    # connection-establishment latency over 10ms — handshake delay is queue delay
tcplife          # per-connection lifespans and volumes — spot short-lived connection churn

tcpdrop deserves emphasis: it prints the kernel call stack for each drop, so instead of inferring from counters you read the code path — tcp_v4_rcv → tcp_v4_syn_recv_sock → tcp_conn_request is a full accept queue; no guessing.

pwru (packet, where are you?) — Cilium's tool for the "where in the kernel did my packet go" question. It attaches to every kernel function that takes an skb and traces a filtered packet's entire journey — through bridges, veth pairs, netfilter verdicts, encapsulation — until it's delivered or dropped:

pwru --filter-dst-ip 10.0.4.12 --filter-port 8080 --output-tuple
# each matching packet prints the chain of kernel functions it traversed;
# a journey ending in kfree_skb, plus the function before it, is your drop point

On a Kubernetes node with three layers of virtual networking, pwru turns "the packet disappears somewhere between the pod and the node interface" from an afternoon of guessing into five minutes of reading. It's especially decisive for netfilter drops: you see the exact hook that issued the verdict.

3. A Real Incident, Walked End to End

The symptom. An internal API service (Go, behind nginx) starts throwing sporadic 502s under peak load — roughly 0.3% of requests. p50 latency is flat; p99 has spikes at suspiciously round values: ~1s, ~3s. The service's own logs show no errors and no slow requests. CPU sits at 60%. Ping between the nginx and upstream nodes: perfect.

That combination — errors at the LB, silence in the app, quantized latency spikes — should already whisper handshake retries. 1s and 3s are TCP SYN retransmission timeouts (1s initial, then exponential backoff).

Step 1 — Is the listener overflowing? On the upstream host:

$ ss -lnt sport = :8080
State   Recv-Q  Send-Q  Local Address:Port
LISTEN  129     128     0.0.0.0:8080

Recv-Q (129) has passed Send-Q (128): the accept queue is full at this instant. And 128 is a suspicious limit — the app was almost certainly clamped, because…

$ sysctl net.core.somaxconn
net.core.somaxconn = 128

An older kernel default (modern kernels default to 4096). Whatever backlog the Go runtime requested, the kernel clamped it to 128.

Step 2 — Confirm drops are occurring, and at what rate:

$ nstat | grep -i listen     # nstat shows deltas since its last run
TcpExtListenOverflows   183   0.0
TcpExtListenDrops       183   0.0

183 overflows in the sampling interval. Each one is a completed handshake the kernel threw away while the client believed it was connected. Cross-check with eBPF for certainty:

$ bpftrace -e 'tracepoint:skb:kfree_skb /args->reason == SKB_DROP_REASON_TCP_OVERFLOW/ { @ = count(); }'
@: 178

Step 3 — Prove the client-side experience. On the nginx node:

$ tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and dst port 8080' -c 20
# ...same source ports SYN-ing twice, 1s apart → SYN retries confirmed

And nginx's error log, now that we know what to grep for:

upstream timed out (110: Connection timed out) while connecting to upstream

"While connecting" — not while reading the response. The 502s were never about the application at all.

Step 4 — Why is the queue filling? A full accept queue means accept() isn't being called fast enough. Two possible worlds: the app is stalling, or the burst rate simply exceeds a 128-slot buffer. Check the connect-latency pattern:

$ tcpconnlat 10   # connections taking over 10ms to establish
# bursts of 200-400ms connect latency clustered at :00 and :30 of each minute

Clustered bursts on the half-minute — a thundering herd from cron-aligned clients, amplified by connection churn: the callers weren't using keep-alive, so every request paid a fresh handshake. During each burst, connections arrived faster than the Go accept loop drained them, 128 slots filled in milliseconds, and the overflow was silent.

Step 5 — The fix, in layers.

Kernel — raise the clamp and the SYN queue, persist it properly:

cat <<'EOF' >/etc/sysctl.d/90-tcp-listen.conf
# Accept-queue clamp: listen() backlogs are min(app, somaxconn).
net.core.somaxconn = 4096
# Half-open (SYN_RECV) queue; relevant under connection bursts/floods.
net.ipv4.tcp_max_syn_backlog = 4096
# Keep syncookies on: under genuine SYN-queue exhaustion the kernel
# still completes handshakes statelessly instead of dropping.
net.ipv4.tcp_syncookies = 1
EOF
sysctl --system

Application — sysctls don't resize existing listeners. The backlog is fixed at listen() time, so the service must be restarted (or redeployed) to pick up the new clamp. If the app hardcodes its own backlog lower than somaxconn, fix that too. (Go's net.Listen uses somaxconn automatically; other stacks — Redis, nginx's own listen ... backlog= — have explicit knobs.)

Clients — enable upstream keep-alive in nginx so you stop paying a handshake per request:

upstream api { server 10.0.4.12:8080; keepalive 64; }
# and in the location block:
proxy_http_version 1.1;
proxy_set_header Connection "";

Optional diagnostic honesty — on internal services, consider net.ipv4.tcp_abort_on_overflow = 1, which sends an RST instead of silently ignoring the ACK. Clients fail fast and loudly instead of hanging into a timeout. Don't do this on internet-facing listeners (transient bursts become hard client errors), but for east-west traffic — service-to-service, inside your own perimeter — behind a retrying LB, a loud failure beats a silent 3-second stall.

Step 6 — Verify. After rollout: ss -lnt shows Send-Q 4096 with Recv-Q peaking around 300 during bursts; nstat shows ListenOverflows flat at zero across three peak windows; the LB 502 rate drops to baseline; the 1s/3s p99 spikes vanish. Close the incident with the counters in the postmortem — next time, whoever's on call greps for the counter instead of the symptom.

4. Key Takeaways & Production Checklist

The mental model: latency is a queue that's filling; a drop is a queue that's full. Find the queue.

When latency spikes or timeouts appear, walk the path in order:

• ss -lnt — any listener with Recv-Q near Send-Q? Accept-queue saturation is the #1 silent killer.
• nstat deltas during the symptom window — ListenOverflows, ListenDrops, RetransSegs rate, TCPBacklogDrop, PruneCalled, UdpRcvbufErrors.
• ss -nti on the affected flows — is rtt actually high, or is the path fine and the delay in queues? Is retrans climbing? Is notsent piling up? rto explains your spike magnitude.
• ss -ntm — a nonzero d (drops) in skmem means the application isn't reading; stop blaming the network.
• ethtool -S | grep -iE 'drop|miss|fifo' and softnet_stat column 2 — ring-buffer and softirq drops; check IRQ spread before raising limits.
• tc -s qdisc — egress drops on the transmit side.
• conntrack headroom — nf_conntrack_count vs nf_conntrack_max, and dmesg | grep conntrack. A mandatory reflex on Kubernetes nodes.
• MTU sanity on overlay networks — ping -M do -s 1472 <peer> (adjust for your overlay's overhead); hangs on large-but-not-small payloads mean a PMTUD black hole; net.ipv4.tcp_mtu_probing=1 is the pragmatic mitigation when you can't fix the ICMP filtering.
• eBPF for ground truth — bpftrace on skb:kfree_skb for drop reasons; tcpdrop/tcpretrans for per-flow stacks; pwru when a packet vanishes inside a multi-layer virtual network path.
• Capture on both ends when in doubt — tcpdump's tap position (early on RX, late on TX) makes two-sided captures a precise bisection tool.

Standing hygiene, before the incident:

• Ship nstat-level TCP counters (ListenOverflows, RetransSegs, prune counters) and conntrack occupancy to your metrics stack — node_exporter's netstat and conntrack collectors cover most of this. An alert on ListenOverflows rising is nearly free and catches this entire class of incident before users do.
• Know your listeners' effective backlogs (the ss -lnt Send-Q column) and your somaxconn — at deploy time, not at 3 a.m.
• Remember the restart rule: backlog sysctls apply at listen() time; buffer sysctls (tcp_rmem/tcp_wmem) apply to new sockets. Tuning without restarting changes nothing you're measuring.
• Raise NIC rings toward the hardware max on servers; re-verify after every image or driver bump — defaults regress.

The kernel keeps meticulous books on every packet it discards. It just never volunteers them. Ask.