supervisor使用注意问题

遇到的问题

core file

程序崩溃无法显示core file

max open file

程序报错 [Errno 24] Too many open files:

备注: 是在/etc/security/limits.conf 已经调整情况下

---

预备知识

resource,python系统资源管理库

如下脚本能显示进程系统资源限制(soft/hard)

import resource

LIMITS = [
    ('RLIMIT_CORE', 'core file size'),
    ('RLIMIT_CPU', 'CPU time'),
    ('RLIMIT_FSIZE', 'file size'),
    ('RLIMIT_DATA', 'heap size'),
    ('RLIMIT_STACK', 'stack size'),
    ('RLIMIT_RSS', 'resident set size'),
    ('RLIMIT_NPROC', 'number of processes'),
    ('RLIMIT_NOFILE', 'number of open files'),
    ('RLIMIT_MEMLOCK', 'lockable memory address'),
]

print('Resource limits (soft/hard):')
for name, desc in LIMITS:
    limit_num = getattr(resource, name)
    soft, hard = resource.getrlimit(limit_num)
    print('{:<23} {}/{}'.format(desc, soft, hard))

读supervisor源码:

github.com/Supervisor/…

 def set_rlimits_or_exit(self):
        """Set the rlimits of the supervisord process.  Called during
        supervisord startup only.  No return value.  Exits the process via
        usage() if any rlimits could not be set."""
        limits = []
        if hasattr(resource, 'RLIMIT_NOFILE'):
            limits.append(
                {
                'msg':('The minimum number of file descriptors required '
                       'to run this process is %(min_limit)s as per the "minfds" '
                       'command-line argument or config file setting. '
                       'The current environment will only allow you '
                       'to open %(hard)s file descriptors.  Either raise '
                       'the number of usable file descriptors in your '
                       'environment (see README.rst) or lower the '
                       'minfds setting in the config file to allow '
                       'the process to start.'),
                'min':self.minfds,
                'resource':resource.RLIMIT_NOFILE,
                'name':'RLIMIT_NOFILE',
                })
        if hasattr(resource, 'RLIMIT_NPROC'):
            limits.append(
                {
                'msg':('The minimum number of available processes required '
                       'to run this program is %(min_limit)s as per the "minprocs" '
                       'command-line argument or config file setting. '
                       'The current environment will only allow you '
                       'to open %(hard)s processes.  Either raise '
                       'the number of usable processes in your '
                       'environment (see README.rst) or lower the '
                       'minprocs setting in the config file to allow '
                       'the program to start.'),
                'min':self.minprocs,
                'resource':resource.RLIMIT_NPROC,
                'name':'RLIMIT_NPROC',
                })

        for limit in limits:
            min_limit = limit['min']
            res = limit['resource']
            msg = limit['msg']
            name = limit['name']
            name = name # name is used below by locals()

            soft, hard = resource.getrlimit(res)

            if (soft < min_limit) and (soft != -1): # -1 means unlimited
                if (hard < min_limit) and (hard != -1):
                    # setrlimit should increase the hard limit if we are
                    # root, if not then setrlimit raises and we print usage
                    hard = min_limit

                try:
                    resource.setrlimit(res, (min_limit, hard))
                    self.parse_infos.append('Increased %(name)s limit to '
                                '%(min_limit)s' % locals())
                except (resource.error, ValueError):
                    self.usage(msg % locals())

由此可知supervisor将 RLIMIT_CORE 限制为0 ,RLIMIT_NOFILE限制为1024

解决方案

方案一 修改supervisord配置 ,解决open file 问题，未解决core

/etc/supervisor/supervisord.conf

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor
minfds=100000

方案二 修改 /proc/{pid}/limits 缺陷：只是临时生效,进程重启后需要再次修改

例如

Limit                     Soft Limit           Hard Limit           Units
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max open files            100000               100000               files

方案三 修改supervisor 源码

find /usr/lib/python* -name options.py|grep supervisor

查找源码路径

/usr/lib/python2.7/dist-packages/supervisor/options.py

在函数 set_rlimits_or_exit 添加

soft, hard = resource.getrlimit(resource.RLIMIT_CORE)
resource.setrlimit(resource.RLIMIT_CORE, (-1, hard))

soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_CORE, (100000, 100000))