Ingress controller reported: epoll_create() failed (24: Too many open files)
Last Updated: 2023-08-22
Ingress controller does not work with a POWER machine, which has 160 cores. Ingress controller might fail when it is running on a node with lots of cores.
Causes
Ingress controller might be running on a node that has too many cores. The maximum number of open file descriptors is calculated with the following formula: *RLIMIT_NOFILE/worker-processes) - 1024
. To resolve, you can either decrease the value of the worker processes
, or increase the value of the RLIMIT_NOFILE
of the container.
Solution one: Edit the configMap
of nginx-ingress-controller
with a decreased value of worker-processes
.
-
To edit the
configmap
ofnginx-ingress-controller
, run the following command:kubectl -n kube-system edit cm nginx-ingress-controller
-
Add
worker-processes: "2"
to theconfigMap
, as it is in the following example. Note: The value might not be2
, depending on yoursysctl
configuration.# Edit the following object. Lines beginning with a '#' are ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 data: body-size: "0" disable-access-log: "true" worker-processes: "2"
Solution two: Set docker LimitNOFILE to a larger value to increase the value of RLIMIT_NOFILE of your ingress controller container.
Note: The maximum number of open file descriptors is calculated with the formula RLIMIT_NOFILE / worker-processes) - 1024
. You can increase RLIMIT_NOFILE
by changing the Docker LimitNOFILE
to a larger value.
-
Run the following command to list your pods that run ingress:
$ kubectl -n kube-system get pods -owide | grep nginx-ingress nginx-ingress-lb-amd64-sknw6 1/1 Running 0 35m 9.111.255.21 9.111.255.21
-
Log on to the node where the ingress controller is running and edit the file
/lib/systemd/system/docker.service
and updateLimitNOFILE
to a large value. IfLimitNOFILE
is set to infinity, it means that the value is65536
or1048576 (2^20)
, based on your operating system.[Service] LimitNOFILE=<your value>
LimitNOFILE
cannot be larger thanfs.file-max
. If you need to updatefs.file-max
to a larger value, run the following command:sysctl -w fs.file-max=<your value>
-
Restart Docker.
systemctl daemon-reload systemctl restart docker