Introduction
On Crusoe Managed Kubernetes (CMK) clusters, a pod's traffic to an external storage service can degrade over time and across consecutive sessions. Throughput falls steadily, yet nothing looks broken: there are no errors, no dropped packets, and no obvious retransmission storm.
Packet captures show telltale signs. The measured RTT is unrealistically low, such as 0.02 ms, and many streams show SACK and DSACK packets.
The cause is in the kernel's forwarding path. BBR relies on packet pacing, which the fq qdisc enforces using the earliest-departure-time timestamp stored in skb->tstamp. On Linux kernels before 5.18, including worker kernel 5.15.0-185-generic, forwarding packets from the pod network namespace into the host clears that timestamp. Pacing breaks and the result is unpaced bursts, spurious retransmissions, and corrupted rate samples that pin BBR's congestion window.
Pods using hostNetwork: true don't cross a namespace boundary, so they don't hit this problem. That makes them a useful control test.
ℹ️ Note: Worker images before
ubuntu-k8s-worker-1.35.5-cmk.20default to BBR withfq.
Prerequisites
-
kubectlAccess to the Affected CMK Cluster - Shell Access to the Affected Worker Node With
tcpdump -
tcptraceand Wireshark Installed on an Analysis Workstation - Permission to Create Nodepools (Solution 1)
- Cluster-Admin Permissions in
kube-system(Solution 2)
Instructions
Step 1: Capture Traffic to the Storage Endpoint
Resolve the storage hostname to an IP address. This example uses storage.googleapis.com:
nslookup storage.googleapis.com
On the affected node, capture traffic in both directions while you reproduce the problem. Both directions are required for RTT and SACK analysis.
sudo tcpdump -i any -nn -w capture.pcap host <STORAGE_IP>
Step 2: Confirm Throughput Degrades Over Time
Open capture.pcap in Wireshark and go to Statistics > I/O Graphs, with all packets selected. Throughput should trend downward over the session.
Step 3: Check for Unrealistic RTT Values
tcptrace -n -l -r capture.pcap | grep -E '(RTT\ m|\=)' | grep -v last
Each stream prints one column per direction. In affected streams, one direction reports an RTT min of 0.0 ms, which is not physically realistic:
RTT min: 22.3 ms RTT min: 0.0 ms
RTT max: 29.3 ms RTT max: 6.7 ms
================================
RTT min: 41.7 ms RTT min: 0.0 ms
RTT max: 67.3 ms RTT max: 18.3 ms
================================Step 4: Look for SACK and DSACK Packets
tcptrace -r -l capture.pcap | grep -E "sack|dsack"
Affected streams show non-zero sack pkts sent and, in some streams, non-zero dsack pkts sent:
sack pkts sent: 0 sack pkts sent: 20
dsack pkts sent: 0 dsack pkts sent: 4
max sack blks/ack: 0 max sack blks/ack: 1
req sack: Y req sack: Y
sacks sent: 0 sacks sent: 20Step 5: Run a hostNetwork Control Test
Run a test pod with hostNetwork: true on the same worker node and repeat the transfer. If throughput stays stable and RTTs look realistic, the issue is confirmed.
Step 6 - Solution 1: Migrate to a Fixed Nodepool (Recommended)
Create a nodepool running worker image ubuntu-k8s-worker-1.35.5-cmk.20 or newer, which defaults to cubic and fq_codel. Then move workloads to the new pool and remove the old one. This avoids running privileged workloads or changing sysctls on existing nodes.
- Create the new nodepool with the fixed image.
- Cordon the old nodes:
kubectl cordon <OLD_NODE_NAME> - Drain the old nodes:
kubectl drain <OLD_NODE_NAME> --ignore-daemonsets --delete-emptydir-data - Delete the old nodepool once workloads have rescheduled.
Step 6 - Solution 2: Alternatively, Apply the Node Sysctl DaemonSet
This one-shot privileged DaemonSet switches the host sysctls to cubic and fq_codel. It was tested on CMK 1.35.5 with kernel 5.15.0-181-generic. Once it runs, standard unprivileged pods inherit cubic without changes to application code, Helm charts, or security contexts.
⚠️ Warning: The change applies to new pods only. Network namespace sysctls and active TCP sockets are set at initialization, so running workloads keep using BBR until restarted. Restart them with
kubectl rollout restartor by re-creating the Job or Pod.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: set-tcp-cubic
namespace: kube-system
spec:
selector:
matchLabels:
app: set-tcp-cubic
template:
metadata:
labels:
app: set-tcp-cubic
spec:
priorityClassName: system-node-critical
hostNetwork: true # Required: applies sysctl to the root host network namespace instead of an isolated pod namespace
tolerations:
- operator: Exists
containers:
- name: sysctl-setter
image: alpine:3.20
securityContext:
privileged: true
command:
- /bin/sh
- -c
- |
sysctl -w net.ipv4.tcp_congestion_control=cubic net.core.default_qdisc=fq_codel
sleep infinityℹ️ Note:
sysctl -wdoes not persist across node reboots. The DaemonSet reapplies the settings, but a pod scheduled before the DaemonSet pod runs can still start with BBR.net.core.default_qdiscalso applies only when a qdisc is created, so existing interfaces may keepfq.
Step 7: Verify the Fix
Restart a test pod and confirm the congestion control in use:
kubectl exec <YOUR_POD_NAME> -n <YOUR_NAMESPACE> -- sysctl net.ipv4.tcp_congestion_control
The output should show cubic. Then repeat the transfer from Step 1 and confirm throughput no longer decays and RTT values look realistic.
Resolution
The root cause is a kernel defect, not the application or the storage service. On kernels before 5.18, forwarding packets from the pod network namespace into the host strips the pacing timestamp, which breaks fq pacing and corrupts BBR's rate sampling. Moving to cubic with fq_codel, either by migrating to a nodepool with the fixed image or by applying the sysctl DaemonSet and restarting pods, removes the dependency on that timestamp.
Example
A training job pulls datasets and writes checkpoints to external object storage from pods on a CMK cluster. The first transfer runs at the expected rate, but each consecutive session is slower while the storage service and the network show no errors. The same transfer from a hostNetwork: true pod on the same node stays fast. After migrating the job to a nodepool running the fixed worker image, throughput stays stable across sessions.