How-To Increase the Open Files (nofiles) ulimit on Kubernetes Worker Nodes

Last Updated: [Jan 15, 2025]

Introduction

Processes running in pods are constrained by a low ulimit for the number of open files (often defaulting to 1024). This can cause applications that require a high number of file descriptors to fail.

You can verify the current limit by running a command inside a pod:

$ kubectl run debug -n default -i -t --image=ubuntu --restart=Never --command -- /bin/sh -c "ulimit -a"
...
nofiles                      1024
...

Or by checking directly on the worker node:

ubuntu@<Hostname>:~$ ulimit -n
1024

Prerequisites

1. SSH access to the worker nodes and sudo privileges to modify system configuration files and restart services.

2. kubectl installed and configured with administrative access to the CMK cluster.

Note: Since these steps require draining nodes and restarting the container runtime, they will cause temporary pod displacement or downtime. We recommend performing these actions during a scheduled maintenance window.

Step-by-Step Instructions

Explained Solution

To increase the nofiles limit (e.g., to 1048576), you must apply changes to both the system-wide security limits and the containerd service configuration on each affected worker node. 

The value 1048576 represents a high, commonly used limit that equals 2^20, or roughly one million open file descriptors per process. This value is large enough to effectively avoid the "Too many open files" error in demanding workloads that require a significant number of simultaneous open files or network sockets, such as databases, message brokers, or high-concurrency web servers. 

While 1048576 is a safe upper bound, setting this limit blindly to such a high value may cause unnecessary memory overhead and performance degradation, By evaluating actual application requirements and usage patterns, you can avoid over-provisioning the file descriptor limits that might lead to unnecessary overhead on your worker nodes.

• Each open file descriptor uses kernel memory. Extremely high limits can increase memory usage and slow down system operations like process scanning.

• For workload-specific tuning:

  • Monitor your application's current open files usage with commands like lsof or checking ls /proc/<pid>/fd | wc -l.
  • Understand peak concurrency and maximum expected open files.
  • Add a safety buffer (e.g., 20-50% above peak usage).
  • Set the limit to this calculated value rather than an arbitrary high number.

Steps to Perform:

1. SSH into the worker node.

2. Edit the system-wide limits configuration file:

   • $ sudo vi /etc/security/limits.conf

   • Add the following lines to the end of the file. This sets the soft and hard limits for all users (*) and for the root user specifically.

     * soft   nofile   1048576
     * hard   nofile   1048576
     root  soft   nofile   1048576
     root  hard   nofile   1048576

3. Edit or create the systemd override file for containerd. This step is critical for ensuring the new limit is applied to all new containers.

   • $ sudo vi /etc/systemd/system/containerd.service.d/override.conf

   • Add the following content to this file:

     [Service]
     LimitMEMLOCK=infinity
     LimitNOFILE=1048576

4. Reload the systemd daemon and restart the containerd service for the changes to take effect for new pods:

   • Before restarting containerd, drain the node to safely evict pods by running:

     kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

   • Once the node is drained, restart containerd:

     $ sudo systemctl daemon-reload
     $ sudo systemctl restart containerd

     Note: Restarting the containerd service on a Kubernetes worker node will disrupt all running pods on that node because containerd manages container lifecycles directly. This may cause temporary downtime for your workloads.

   • After the restart, uncordon the node to allow pod scheduling:

     kubectl uncordon <node-name>

5. After completing the steps above, the new nofiles limit will be applied to newly created pods. You can verify this by exec-ing into a pod:

   • $ kubectl exec -it <Pod Name> -n <Namespace> -- /bin/sh
     sh-5.1$ ulimit -n
     1048576

Rollback Plan for ulimit and containerd configuration changes

In case the changes cause issues, you can revert them with these steps:

1. Cordon and Drain the Node

   Prevent new pods from being scheduled on the node and safely evict running pods:

   kubectl cordon <node-name>
   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

2. Restore Original ulimit Settings

   Edit the system-wide limits configuration file to remove or revert your previous changes in /etc/security/limits.conf:

   sudo vi /etc/security/limits.conf

   Remove or comment out the lines setting nofiles to the higher value, restoring them to their original defaults (or as appropriate).

3. Revert containerd Override Configuration

   Remove or revert changes to the containerd systemd override file:

   sudo vi /etc/systemd/system/containerd.service.d/override.conf

   If you have a backup original file, restore it instead.

4. Reload systemd and restart containerd

   Apply the configuration changes by reloading systemd and restarting containerd:

   sudo systemctl daemon-reload
   sudo systemctl restart containerd

5. Reboot the Node

   To apply node-level ulimit changes completely (especially for user sessions), reboot the node:

   sudo reboot

   Note: Rebooting will cause downtime on this node; ensure pods are drained first as shown in step 1.

6. Uncordon the Node

   After the node is back online and stable, allow pod scheduling again:

   kubectl uncordon <node-name>

7. Validate Rollback

   Verify the ulimit settings on the node by logging in and running:

   ulimit -n

   1. Validate containerd is running properly.
   2. Check newly scheduled pods on the node are operating without issues.

Important: Node-Level vs. Pod-Level Limits

You may observe that after restarting containerd (and even after logging out and logging back in), the ulimit -n command run directly on the node still shows the old, lower limit:

This is expected behaviour. The containerd overrides (override.conf) directly instructs the container runtime to use the new limit (1048576) for the containers it launches. This solves the immediate problem for your containerised workloads.

The changes in /etc/security/limits.conf apply to new user login sessions, but for these changes to be applied system-wide to all processes, including the user sessions on the node itself, a full reboot of the worker node is required.

sudo reboot

After the node reboots, logging in as ubuntu or root will show the new limit at the node level as well:

Conclusion: A reboot is required to see the limit reflected at the node's user session level, but a containerd restart is sufficient to apply the new limit to pods, which typically resolves the user's workload issue.

Additional Resources

• Crusoe Managed Kubernetes (CMK)