CMK: Fluentbit - Kube API Upstream Connection Error

Overview

• A fluent-bit manifest has been deployed to the CMK cluster whereas the pod is exhibiting a TLS failure, and an inability to connect to the upstream Kubernetes API, e.g.
  

  [2025/03/17 19:54:33] [error] [tls] error: unexpected EOF
  [2025/03/17 19:54:33] [error] [filter:kubernetes:kubernetes.0] kube api upstream connection error
  [2025/03/17 19:54:33] [ warn] [output:stackdriver:stackdriver.0] failed to find a corresponding entry for resource label entry [container_name=$kubernetes['container_name']]
  [2025/03/17 19:54:33] [ warn] [output:stackdriver:stackdriver.0] failed to find a corresponding entry for resource label entry [namespace_name=$kubernetes['namespace_name']]
  [2025/03/17 19:54:33] [ warn] [output:stackdriver:stackdriver.0] failed to find a corresponding entry for resource label entry [pod_name=$kubernetes['pod_name']]

• This error indicates that the Fluent Bit Kubernetes filter is unable to establish a secure connection to the Kubernetes API server due to TLS verification failure.

Prerequisites

• Crusoe Managed Kubernetes (CMK)
• A Fluent Bit DaemonSet or Deployment configured to use the Kubernetes filter.
• Access to the cluster via kubectl.

Steps

Step 1: Modify the Manifest

• Locate the ConfigMap used to configure Fluent Bit. This is typically named something like fluent-bit-config and deployed in the kube-system namespace.
• Edit the ConfigMap using kubectl edit configmap <configmap-name> -n kube-system.
• Within the filter-kubernetes.conf section, add the line Tls.verify Off.
• The relevant portion of the ConfigMap should now resemble:

filter-kubernetes.conf: |
  [FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On
    Keep_Log Off
    K8S-Logging.Parser On
    K8S-Logging.Exclude On
    Tls.verify Off # Add this line
  [FILTER]
    # ... rest of your filter config ...

• Save the changes to the ConfigMap.

Step 2: Restart the Fluent Bit Pods

• To apply the configuration changes, restart the Fluent Bit pods. If you are using a DaemonSet, delete one of the pods, and the DaemonSet will recreate it with the new configuration. If you are using a deployment, delete all of the pods.
• Use the following kubectl command to delete the pods:

kubectl delete pods -n kube-system -l app.kubernetes.io/name=<fluent-bit-pod-label>

• Replace <fluent-bit-pod-label> with the label used to identify your Fluent Bit pods (e.g., app.kubernetes.io/name=fluent-bit).

Resolution

The addition of the Tls.verify Off directive to the Kubernetes filter configuration in the Fluent Bit ConfigMap resolves the TLS verification failure. This allows Fluent Bit to connect to the Kubernetes API server and collect the necessary metadata for log enrichment and forwarding.

Important Note: Disabling TLS verification introduces a security risk. This is a temporary workaround until full certificate chain validation is implemented in the CMK environment.

After implementing these steps, the Fluent Bit logs should no longer display the unexpected EOF and Kube API connection errors. The Stackdriver output errors related to missing resource labels should also be resolved, as Fluent Bit will now be able to retrieve the required Kubernetes metadata.