Ingress controller reported: epoll_create() failed (24: Too many open files)
Last Updated: 2023-08-22
Ingress controller does not work with a POWER machine, which has 160 cores. Ingress controller might fail when it is running on a node with lots of cores.
Causes
Ingress controller might be running on a node that has too many cores. The maximum number of open file descriptors is calculated with the following formula: *RLIMIT_NOFILE/worker-processes) - 1024. To resolve, you can either decrease the value of the worker processes, or increase the value of the RLIMIT_NOFILE of the container.
Solution one: Edit the configMap of nginx-ingress-controller with a decreased value of worker-processes.
-
To edit the
configmapofnginx-ingress-controller, run the following command:kubectl -n kube-system edit cm nginx-ingress-controller -
Add
worker-processes: "2"to theconfigMap, as it is in the following example. Note: The value might not be2, depending on yoursysctlconfiguration.# Edit the following object. Lines beginning with a '#' are ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 data: body-size: "0" disable-access-log: "true" worker-processes: "2"
Solution two: Set docker LimitNOFILE to a larger value to increase the value of RLIMIT_NOFILE of your ingress controller container.
Note: The maximum number of open file descriptors is calculated with the formula RLIMIT_NOFILE / worker-processes) - 1024. You can increase RLIMIT_NOFILE by changing the Docker LimitNOFILE to a larger value.
-
Run the following command to list your pods that run ingress:
$ kubectl -n kube-system get pods -owide | grep nginx-ingress nginx-ingress-lb-amd64-sknw6 1/1 Running 0 35m 9.111.255.21 9.111.255.21 -
Log on to the node where the ingress controller is running and edit the file
/lib/systemd/system/docker.serviceand updateLimitNOFILEto a large value. IfLimitNOFILEis set to infinity, it means that the value is65536or1048576 (2^20), based on your operating system.[Service] LimitNOFILE=<your value>LimitNOFILEcannot be larger thanfs.file-max. If you need to updatefs.file-maxto a larger value, run the following command:sysctl -w fs.file-max=<your value> -
Restart Docker.
systemctl daemon-reload systemctl restart docker