Last Updated: Mar 31, 2026
Introduction
This article highlights troubleshooting checklist for network connectivity failures on Crusoe Managed Kubernetes (CMK) with Cilium, covering Loadbalancer, pod-to-pod, and node-level checks.
Prerequisites
Access to a Crusoe Cloud project with appropriate permissions
Existing Crusoe Managed Kubernetes Cluster
Access to
crusoeandkubectlCLI
For the purpose of this article, we have a CMK cluster with couple of nodes and a web-test application deployed on it.
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP
np-e03515ab-1 Ready <none> 12d v1.33.4 172.27.61.24 160.211.67.2
np-e03515ab-2 Ready <none> 12d v1.33.4 172.27.63.68 160.211.67.26
$ kubectl get deployment web-test
NAME READY UP-TO-DATE AVAILABLE AGE
web-test 2/2 2 2 3d13h
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
web-test-66ccc8489d-jj58p 1/1 Running 0 3d13h 10.234.1.116 np-e03515ab-2
web-test-66ccc8489d-mnrzq 1/1 Running 0 3d13h 10.234.0.43 np-e03515ab-1
$ kubectl get svc web-test-svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web-test-svc LoadBalancer 10.233.39.231 160.211.64.124 80:30318/TCP 3d13hApplication pods may have exec disabled via RBAC or lack network tooling — deploying a dedicated debug pod on a particular node gives us a controlled environment to run connectivity tests.
apiVersion: v1
kind: Pod
metadata:
name: net-test
namespace: default
spec:
nodeName: <name>
restartPolicy: Never
containers:
- name: net-test
image: nicolaka/netshoot
command: ["sleep", "infinity"]Step-by-Step Instructions
1. Service/LoadBalancer connectivity.
$ curl -v 160.211.64.124:80
* Trying 160.211.64.124:80...
* Connected to 160.211.64.124 (160.211.64.124) port 80
...
<p><em>Thank you for using nginx.</em></p>If there are any issues:
crusoe networking load-balancers list— verify the LB is active, the external IP is assigned, and all backends showonline.If any backend is
offline, check the node statuskubectl get nodesand VM health.Verify any blocking firewall rules via
crusoe networking vpc-firewall-rules list.
2. Test ICMP connectivity between pods. With net-test running on node np-e03515ab-1
net-test:~# ping -c 2 10.234.1.116
PING 10.234.1.116 (10.234.1.116) 56(84) bytes of data.
64 bytes from 10.234.1.116: icmp_seq=1 ttl=63 time=2.40 ms
64 bytes from 10.234.1.116: icmp_seq=2 ttl=63 time=0.510 ms
--- 10.234.1.116 ping statistics ---
net-test:~# ping -c 2 10.234.0.43
PING 10.234.0.43 (10.234.0.43) 56(84) bytes of data.
64 bytes from 10.234.0.43: icmp_seq=1 ttl=63 time=0.041 ms
64 bytes from 10.234.0.43: icmp_seq=2 ttl=63 time=0.040 ms
--- 10.234.0.43 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001msVerify the same with net-test pod running on node np-e03515ab-2
3. Verify NodePort 30318 is programmed in Cilium's BPF datapath. If missing, NodePort traffic won't reach backends.
CILIUM_POD=$(kubectl -n kube-system get pods -l k8s-app=cilium --field-selector spec.nodeName=np-e03515ab-1.eu-iceland1-a.compute.internal -o jsonpath='{.items[0].metadata.name}')
$ kubectl -n kube-system exec -it $CILIUM_POD -c cilium-agent -- cilium bpf lb list | grep 30318
172.27.61.24:30318 (2) 10.234.1.116:80 (24) (2)
172.27.61.24:30318 (0) 0.0.0.0:0 (24) (0) [NodePort]
0.0.0.0:30318 (1) 10.234.0.43:80 (5) (1)
172.27.61.24:30318 (1) 10.234.0.43:80 (24) (1)
0.0.0.0:30318 (0) 0.0.0.0:0 (5) (0) [NodePort, non-routable]
0.0.0.0:30318 (2) 10.234.1.116:80 (5) (2)
---
$ CILIUM_POD=$(kubectl -n kube-system get pods -l k8s-app=cilium --field-selector spec.nodeName=np-e03515ab-2.eu-iceland1-a.compute.internal -o jsonpath='{.items[0].metadata.name}')
$ kubectl -n kube-system exec -it $CILIUM_POD -c cilium-agent -- cilium bpf lb list | grep 30318
172.27.63.68:30318 (0) 0.0.0.0:0 (34) (0) [NodePort]
0.0.0.0:30318 (1) 10.234.0.43:80 (35) (1)
0.0.0.0:30318 (0) 0.0.0.0:0 (35) (0) [NodePort, non-routable]
172.27.63.68:30318 (1) 10.234.0.43:80 (34) (1)
172.27.63.68:30318 (2) 10.234.1.116:80 (34) (2)
0.0.0.0:30318 (2) 10.234.1.116:80 (35) (2) 4. If pod-to-pod is fine — check ICMP for to pod-to-node and node-to-node.
$ kubectl exec -it net-test -- bash
net-test:~# ping 172.27.63.68
PING 172.27.63.68 (172.27.63.68) 56(84) bytes of data.
64 bytes from 172.27.63.68: icmp_seq=1 ttl=63 time=7.11 ms
64 bytes from 172.27.63.68: icmp_seq=2 ttl=63 time=1.06 ms
^C
--- 172.27.63.68 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
net-test:~# ping 172.27.61.24
PING 172.27.61.24 (172.27.61.24) 56(84) bytes of data.
64 bytes from 172.27.61.24: icmp_seq=1 ttl=63 time=0.072 ms
64 bytes from 172.27.61.24: icmp_seq=2 ttl=63 time=0.081 ms
^C
--- 172.27.61.24 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006msnp-e03515ab-1:~$ ping 172.27.63.68
PING 172.27.63.68 (172.27.63.68) 56(84) bytes of data.
64 bytes from 172.27.63.68: icmp_seq=1 ttl=64 time=5.60 ms
64 bytes from 172.27.63.68: icmp_seq=2 ttl=64 time=0.852 ms
^C
--- 172.27.63.68 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.852/3.227/5.603/2.375 ms
np-e03515ab-2:~$ ping 172.27.61.24
PING 172.27.61.24 (172.27.61.24) 56(84) bytes of data.
64 bytes from 172.27.61.24: icmp_seq=1 ttl=64 time=7.40 ms
64 bytes from 172.27.61.24: icmp_seq=2 ttl=64 time=3.30 ms
^C
--- 172.27.61.24 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 3.300/5.351/7.402/2.051 ms5. Test reachability over HTTP between pods.
net-test:~# curl -s 10.234.1.116:80
...
<p><em>Thank you for using nginx.</em></p>
net-test:~# curl -s 10.234.0.43:80
...
<p><em>Thank you for using nginx.</em></p>6. If there are any failures in above, check ciliumnetworkpolicy configuration for ingress/egress rules set for entities and port/protocol.
$ kubectl get ciliumnetworkpolicy -A
NAMESPACE NAME AGE
default web-test-l7 3d17h7. Verify health of core-dns pods.
$ kubectl get pods -A | grep coredns
kube-system coredns-5464ff8584-6rdc4 1/1 Running 2 12d
kube-system coredns-5464ff8584-gszcg 1/1 Running 0 11d
kube-system coredns-5464ff8584-xtcks 1/1 Running 2 12d