How-To Setup HA Proxy Load Balancer using CMK (Crusoe Managed Kubernetes) – Crusoe Support

Introduction

In a Crusoe Managed Kubernetes (CMK) environment, native cloud provider LoadBalancer support is not available. To expose your Kubernetes services externally and load balance traffic efficiently, setting up an HAProxy instance as a reverse proxy and load balancer is a reliable solution.

This guide walks you through deploying a simple test application on your CMK cluster, configuring it to identify which node serves the request, and setting up HAProxy on a dedicated VM to distribute incoming requests evenly across your worker nodes via NodePort services. By following this guide, you'll have a working load balancer that can handle external traffic and distribute it across multiple Kubernetes worker nodes.

Prerequisites

Access to a Crusoe Managed Kubernetes cluster with at least two worker nodes
A VM with Ubuntu (or similar Linux) where you can install and configure HAProxy
kubectl access configured to manage your CMK cluster
Basic familiarity with Linux command line and editing files with vi or nano
External IP addresses for your Kubernetes worker nodes
Appropriate firewall rules allowing traffic on required ports

Step-by-Step Instructions

Step 1: Deploy the Sample Application with Node Info

Deploy a test application that will help you verify load balancing is working correctly across your nodes.

Create a file named testapp.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: nginx:alpine
        command: ["/bin/sh", "-c"]
        args:
          - |
            echo "<html><body><h1>Hello from $(hostname) on node $NODE_NAME</h1></body></html>" > /usr/share/nginx/html/index.html && \
            nginx -g "daemon off;";
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

---

apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  type: NodePort
  selector:
    app: hello-world
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 31858

Apply the deployment and service:

kubectl apply -f testapp.yaml

Verify the deployment and get node information:

kubectl get svc hello-world
kubectl get nodes -o wide

Note: The application creates a simple webpage that displays the pod hostname and the Kubernetes node name, making it easy to verify which node is serving each request.

Step 2: Install and Configure HAProxy on a Separate VM

Set up HAProxy on a dedicated VM to act as your load balancer.

Install HAProxy on your Ubuntu VM:

sudo apt update
sudo apt install haproxy -y

Edit the HAProxy configuration file:

sudo vi /etc/haproxy/haproxy.cfg

Replace the entire contents with the following configuration, updating the worker node IP addresses:

global
    log /dev/log local0
    log /dev/log local1 notice
    maxconn 2048
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    timeout connect 5s
    timeout client  50s
    timeout server  50s

frontend http_front
    bind *:80
    default_backend k8s_nodeport_backend

backend k8s_nodeport_backend
    balance roundrobin
    option httpchk GET /
    server worker1 <worker1_external_ip>:31858 check
    server worker2 <worker2_external_ip>:31858 check

Important: Replace <worker1_external_ip> and <worker2_external_ip> with your actual Kubernetes worker node external IP addresses.
Validate the HAProxy configuration syntax:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

Restart and enable HAProxy service:

sudo systemctl restart haproxy
sudo systemctl enable haproxy

Tip: You can check the HAProxy service status with sudo systemctl status haproxy to ensure it's running properly.

Step 3: Test Load Balancing

Verify that your HAProxy setup is working correctly and distributing traffic across your worker nodes.

Test the load balancer by making HTTP requests to your HAProxy VM:

curl http://<haproxy_vm_ip>

You should receive responses similar to:

<html><body><h1>Hello from hello-world-xxxx on node np-48ce1093-1.us-southcentral1-a.compute.internal</h1></body></html>

Repeat the curl command multiple times to observe load balancing in action. The responses should show different pod hostnames and potentially different node names, confirming that traffic is being distributed.
Additional Testing: You can also test from a web browser by navigating to http://<haproxy_vm_ip> and refreshing the page multiple times to see the load balancing effect.

Example

Here's a practical example of what you should see when testing your setup:

First request:

$ curl http://192.168.1.100
<html><body><h1>Hello from hello-world-7d4f8c9b5-abc12 on node worker-node-1</h1></body></html>

Second request:

$ curl http://192.168.1.100
<html><body><h1>Hello from hello-world-7d4f8c9b5-def34 on node worker-node-2</h1></body></html>

Third request:

$ curl http://192.168.1.100
<html><body><h1>Hello from hello-world-7d4f8c9b5-ghi56 on node worker-node-1</h1></body></html>

This demonstrates that HAProxy is successfully distributing requests across different pods running on different worker nodes in a round-robin fashion.

Troubleshooting

Issue: HAProxy service fails to restart

Check configuration syntax with sudo haproxy -c -f /etc/haproxy/haproxy.cfg
Review error messages and fix any syntax errors
Ensure all IP addresses and ports are correctly specified
Retry starting the service after fixing issues

Issue: Cannot access NodePort service externally

Confirm firewall allows traffic on the NodePort (31858 in this example)
Verify that worker node IP addresses and ports are correct
Test direct access to NodePort on worker nodes before testing through HAProxy
Check that the Kubernetes service is running with kubectl get svc

Issue: HAProxy backend shows server down

Verify the health check path matches your application's endpoint
Check network connectivity between HAProxy VM and worker nodes
Ensure worker nodes are accessible on the specified NodePort
Review HAProxy logs with sudo journalctl -u haproxy -f

Issue: Load balancing not working as expected

Verify that multiple replicas of your application are running across different nodes
Check HAProxy statistics page (if configured) to see backend server status
Ensure the balance algorithm is set correctly in the configuration
Test with sufficient requests to observe the distribution pattern

Additional Resources