Introduction
Note: In a Crusoe Managed Kubernetes (CMK) environment, native cloud provider LoadBalancer support is now available.
Please, refer to How-To Create a Crusoe Cloud Load Balancer for a CMK Cluster
If native LoadBalancer doesn't suit your needs, to expose your Kubernetes services externally and load balance traffic efficiently, setting up an HAProxy instance as a reverse proxy and load balancer is a reliable solution.
This guide walks you through deploying a simple test application on your CMK cluster, configuring it to identify which node serves the request, and setting up HAProxy on a dedicated VM to distribute incoming requests evenly across your worker nodes via NodePort services. By following this guide, you'll have a working load balancer that can handle external traffic and distribute it across multiple Kubernetes worker nodes.
Prerequisites
- Access to a Crusoe Managed Kubernetes cluster with at least two worker nodes
- A VM with Ubuntu (or similar Linux) where you can install and configure HAProxy
-
kubectlaccess configured to manage your CMK cluster - Basic familiarity with Linux command line and editing files with
viornano - External IP addresses for your Kubernetes worker nodes
- Appropriate firewall rules allowing traffic on required ports
Step-by-Step Instructions
Step 1: Deploy the Sample Application with Node Info
Deploy a test application that will help you verify load balancing is working correctly across your nodes.
- Create a file named
testapp.yamlwith the following content:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
spec:
replicas: 3
selector:
matchLabels:
app: hello-world
template:
metadata:
labels:
app: hello-world
spec:
containers:
- name: hello-world
image: nginx:alpine
command: ["/bin/sh", "-c"]
args:
- |
echo "<html><body><h1>Hello from $(hostname) on node $NODE_NAME</h1></body></html>" > /usr/share/nginx/html/index.html && \
nginx -g "daemon off;";
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
---
apiVersion: v1
kind: Service
metadata:
name: hello-world
spec:
type: NodePort
selector:
app: hello-world
ports:
- port: 80
targetPort: 80
protocol: TCP
nodePort: 31858
- Apply the deployment and service:
kubectl apply -f testapp.yaml
- Verify the deployment and get node information:
kubectl get svc hello-world
kubectl get nodes -o wide
- Note: The application creates a simple webpage that displays the pod hostname and the Kubernetes node name, making it easy to verify which node is serving each request.
Step 2: Install and Configure HAProxy on a Separate VM
Set up HAProxy on a dedicated VM to act as your load balancer.
- Install HAProxy on your Ubuntu VM:
sudo apt update
sudo apt install haproxy -y
- Edit the HAProxy configuration file:
sudo vi /etc/haproxy/haproxy.cfg
- Replace the entire contents with the following configuration, updating the worker node IP addresses:
global
log /dev/log local0
log /dev/log local1 notice
maxconn 2048
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
timeout connect 5s
timeout client 50s
timeout server 50s
frontend http_front
bind *:80
default_backend k8s_nodeport_backend
backend k8s_nodeport_backend
balance roundrobin
option httpchk GET /
server worker1 <worker1_external_ip>:31858 check
server worker2 <worker2_external_ip>:31858 check
-
Important: Replace
<worker1_external_ip>and<worker2_external_ip>with your actual Kubernetes worker node external IP addresses. - Validate the HAProxy configuration syntax:
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
- Restart and enable HAProxy service:
sudo systemctl restart haproxy
sudo systemctl enable haproxy
-
Tip: You can check the HAProxy service status with
sudo systemctl status haproxyto ensure it's running properly.
Step 3: Test Load Balancing
Verify that your HAProxy setup is working correctly and distributing traffic across your worker nodes.
- Test the load balancer by making HTTP requests to your HAProxy VM:
curl http://<haproxy_vm_public_ip>
- You should receive responses similar to:
<html><body><h1>Hello from hello-world-xxxx on node np-48ce1093-1.us-southcentral1-a.compute.internal</h1></body></html>
- Repeat the curl command multiple times to observe load balancing in action. The responses should show different pod hostnames and potentially different node names, confirming that traffic is being distributed.
-
Additional Testing: You can also test from a web browser by navigating to
http://<haproxy_vm_public_ip>and refreshing the page multiple times to see the load balancing effect.
Example
Here's a practical example of what you should see when testing your setup:
First request:
$ curl http://192.168.***.***
<html><body><h1>Hello from hello-world-7d4f8c9b5-abc12 on node worker-node-1</h1></body></html>
Second request:
$ curl http://192.168.***.***
<html><body><h1>Hello from hello-world-7d4f8c9b5-def34 on node worker-node-2</h1></body></html>
Third request:
$ curl http://192.168.***.***
<html><body><h1>Hello from hello-world-7d4f8c9b5-ghi56 on node worker-node-1</h1></body></html>
This demonstrates that HAProxy is successfully distributing requests across different pods running on different worker nodes in a round-robin fashion.
Troubleshooting
Issue: HAProxy service fails to restart
- Check configuration syntax with
sudo haproxy -c -f /etc/haproxy/haproxy.cfg - Review error messages and fix any syntax errors
- Ensure all IP addresses and ports are correctly specified
- Retry starting the service after fixing issues
Issue: Cannot access NodePort service externally
- Confirm Crusoe Cloud firewall rules allow ingress traffic on the NodePort (31858 in this example)
- Verify that worker node IP addresses and ports are correct
- Test direct access to NodePort on worker nodes before testing through HAProxy
- Check that the Kubernetes service is running with
kubectl get svc
Issue: HAProxy backend shows server down
- Verify the health check path matches your application's endpoint
- Check network connectivity between HAProxy VM and worker nodes
- Ensure worker nodes are accessible on the specified NodePort
- Review HAProxy logs with
sudo journalctl -u haproxy -f
Issue: Load balancing not working as expected
- Verify that multiple replicas of your application are running across different nodes
- Check HAProxy statistics page (if configured) to see backend server status
- Ensure the balance algorithm is set correctly in the configuration
- Test with sufficient requests to observe the distribution pattern