Introduction
If you're pulling large container images on a Crusoe Managed Kubernetes (CMK) cluster, you may see pulls fail partway through with an error like this from kubectl describe pod:
Failed to pull image "...": rpc error: code = Canceled desc = failed to pull and unpack image "...": failed to extract layer (application/vnd.oci.image.layer.v1.tar+gzip sha256:...) to overlayfs as "extract-...": context canceled
and this in the node's containerd logs around the same time:
level=error msg="cancel pulling image ... because of no progress in 5m0s"
ℹ️ Note: This is not specific to CCR (Crusoe Container Registry) — it can happen when pulling from any registry, upstream or otherwise. CCR is only responsible for delivering the layer's bytes over the network; extraction/unpacking to overlayfs happens entirely on the node afterward, and that's the phase this timeout actually trips on.
This is caused by containerd's CRI image_pull_progress_timeout setting, which defaults to 5 minutes. It's an idle-progress watchdog: if no new data is read from the registry for that long, containerd cancels the pull. The subtlety is that this same watchdog also covers the extraction phase, after the layer has finished downloading. Unpacking a large layer to overlayfs can itself take several minutes with zero new network bytes moving — which looks identical to a stalled pull from the watchdog's point of view, even though extraction is actively running.
The workaround is a DaemonSet that raises image_pull_progress_timeout to 15 minutes on every node so extraction of large layers has enough headroom to complete.
Prerequisites
- Running CMK Cluster
- CMK Nodes Running containerd 1.7.x or 2.x (CMK's Default Runtime)
-
kubectlInstalled and Configured for the Cluster - Cluster-Admin Access to Deploy a Privileged DaemonSet to
kube-system
Instructions
Step 1: Deploy the containerd Pull-Timeout DaemonSet
This DaemonSet runs on every node (including GPU and tainted nodepools, via its tolerations), detects your containerd version's CRI config section, patches image_pull_progress_timeout to 15m, and restarts containerd. It's idempotent — nodes already at 15m are skipped — and a restart of containerd does not affect already-running containers, since containerd supervises them through separate shim processes; only a pull that happens to be mid-flight at that exact moment would need to retry.
ℹ️ Note: This DaemonSet edits
/etc/containerd/config.tomldirectly on the host and restarts the containerd service. It backs up the original file to/etc/containerd/config.toml.bak-pull-timeoutbefore making any change.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: containerd-pull-timeout
namespace: kube-system
labels:
app: containerd-pull-timeout
spec:
selector:
matchLabels:
app: containerd-pull-timeout
template:
metadata:
labels:
app: containerd-pull-timeout
spec:
# Run everywhere, including GPU/tainted nodes
tolerations:
- operator: Exists
hostPID: true
priorityClassName: system-node-critical
containers:
- name: patch
image: ubuntu:24.04
securityContext:
privileged: true
resources:
requests:
cpu: 5m
memory: 16Mi
command:
- /bin/bash
- -c
- |
set -euo pipefail
CFG=/etc/containerd/config.toml
WANT=15m
NS="nsenter -t 1 -m -u -i -n --"
if $NS grep -Eq "image_pull_progress_timeout *= *['\"]${WANT}['\"]" "$CFG"; then
echo "image_pull_progress_timeout already ${WANT}, nothing to do"
else
$NS cp "$CFG" "${CFG}.bak-pull-timeout"
if $NS grep -q image_pull_progress_timeout "$CFG"; then
# Default config dumps (containerd 1.7 and 2.x) already carry the key; replace its value
$NS sed -i -E "s/(image_pull_progress_timeout *= *)['\"][^'\"]+['\"]/\1'${WANT}'/" "$CFG"
elif $NS grep -q "io.containerd.cri.v1.images" "$CFG"; then
# containerd 2.x section (cmk = 1.32 images)
$NS sed -i "/\[plugins\.'io\.containerd\.cri\.v1\.images'\]/a image_pull_progress_timeout = '${WANT}'" "$CFG"
elif $NS grep -q "io.containerd.grpc.v1.cri" "$CFG"; then
# containerd 1.7.x section (older cmk images)
$NS sed -i "/\[plugins\.\"io\.containerd\.grpc\.v1\.cri\"\]/a image_pull_progress_timeout = \"${WANT}\"" "$CFG"
else
echo "ERROR: no CRI section found in $CFG; leaving untouched" >&2
exit 1
fi
echo "patched $CFG - image_pull_progress_timeout = ${WANT}; restarting containerd"
$NS systemctl restart containerd
fi
# Keep the pod alive so the DaemonSet stays Ready and covers nodes that join later
sleep infinityStep 2: Verify the Change Took Effect
Check the DaemonSet's logs to confirm each node was patched (or already compliant):
kubectl logs -n kube-system daemonset/containerd-pull-timeout
Optionally confirm directly on a node:
sudo containerd config dump | grep image_pull_progress_timeout
Resolution
Raising image_pull_progress_timeout to 15 minutes gives large-layer extraction enough headroom to finish before the watchdog cancels the pull, resolving the context canceled / no progress in 5m0s failure for most large-image cases.
This comes with a tradeoff worth knowing about: a genuinely stuck pull (a real network or registry problem, not just a large layer) now takes up to 15 minutes to fail and retry instead of 5, which can make actual problems slower to notice.
💡 Tip: If the same image is pulled repeatedly across many nodes or pods — a common pattern for training jobs that scale out — consider enabling Spegel, CMK's peer-to-peer image distribution. With Spegel enabled, only the first pull on the cluster needs to fetch the image from the registry (CCR or upstream); every subsequent pull on any other node is served from a peer that already has it. Since this timeout only affects pulls that actually go through the registry-and-extract path, Spegel effectively removes the problem for all pulls after the first.
Example
A team ships a 40 GB inference image whose largest layer is a 22 GB model bundle. Pulls from CCR download at full line rate, but the pod repeatedly lands in ImagePullBackOff with context canceled, and the node's containerd log shows no progress in 5m0s — the download finished, but unpacking the 22 GB layer to overlayfs took longer than the 5-minute watchdog allows.
They deploy the DaemonSet above, confirm each node logs either "patched" or "already 15m", and re-create the pod. Extraction now completes with headroom to spare, and because the DaemonSet stays Running, nodes added by the cluster autoscaler later get the same setting the moment they join.