构建带插件的nginx镜像

118 阅读4分钟

用户的安全基线要求系统中只能使用nginx作为网关,openresty都不行,kong、apisix就更不用谈了。因此只能自己编译构建带插件的nginx镜像。

用到的模块是lua-nginx-module和headers-more-nginx-module。构建过程中主要遇到的问题是版本兼容问题。

验证时用到的 nginx.conf

worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /usr/local/nginx/conf/mime.types;
    default_type  application/octet-stream;

    # Lua 模块配置
    lua_package_path "/usr/local/share/lua/5.1/?.lua;;";
    lua_package_cpath "/usr/local/lib/lua/5.1/?.so;;";

    server {
        listen       80;
        server_name  localhost;

        # 测试 more-header 模块:设置自定义响应头
        location /test-more-headers {
            # more_set_headers 是 more-header 模块的指令
            more_set_headers 'X-Test-More-Header: Hello from more-header module';
            more_set_headers 'X-Server-Version: Nginx-1.24.0';
            return 200 'Check response headers for more-header test\n';
        }

        # 测试 Lua 模块:执行 Lua 脚本
        location /test-lua {
            # content_by_lua_block 是 Lua 模块的指令
            content_by_lua_block {
                ngx.say("Hello from Lua module!")
                ngx.say("Current time: ", os.date("%Y-%m-%d %H:%M:%S"))
                ngx.say("Server address: ", ngx.var.server_addr)
            }
        }

        # 测试组合功能:Lua 动态设置头 + more-header 追加头
        location /test-combined {
            # Lua 动态生成内容和头
            content_by_lua_block {
                ngx.header["X-Lua-Generated"] = "true"
                ngx.say("This response uses both modules")
            }
            # more-header 追加额外头
            more_set_headers 'X-Combined-Mode: Lua+more-header';
        }
    }
}

Dockerfile

# 基于Alpine Linux构建
FROM alpine:3.18 as builder

# 更换为国内镜像源
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories

# 安装编译依赖
RUN apk add --no-cache \
    gcc \
    g++ \
    make \
    libc-dev \
    pcre-dev \
    zlib-dev \
    openssl-dev \
    lua-dev \
    luajit-dev \
    git \
    wget

# 定义版本
ENV NGINX_VERSION=1.24.0
ENV LUA_NGINX_MODULE_VERSION=0.10.25
ENV NGX_DEVEL_KIT_VERSION=0.3.3
ENV NGX_HTTP_MORE_MODULE_VERSION=0.34
ENV LUAJIT_LIB=/usr/lib
ENV LUAJIT_INC=/usr/include/luajit-2.1
ENV RESTY_CORE_VERSION=0.1.27 
ENV RESTY_LRUCACHE_VERSION=0.13 

# 下载并解压源码;完成之后,目录下应该有6个文件:(1)nginx-1.24.0 (2)ngx_devel_kit-0.3.3 (4)headers-more-nginx-module-0.34 (3)lua-nginx-module-0.10.25 (5)lua-resty-core (6)lua-resty-lrucache-0.13  
# 1 wget http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz
# 2 wget https://github.com/simpl/ngx_devel_kit/archive/v$NGX_DEVEL_KIT_VERSION.tar.gz
# 3 wget https://github.com/openresty/lua-nginx-module/archive/v$LUA_NGINX_MODULE_VERSION.tar.gz
# 4 wget --timeout=30 --tries=3 https://github.com/openresty/headers-more-nginx-module/archive/v$NGX_HTTP_MORE_MODULE_VERSION.tar.gz
# 5 wget https://github.com/openresty/lua-resty-core/archive/v$RESTY_CORE_VERSION.tar.gz
# 6 wget --timeout=30 --tries=3 https://github.com/openresty/lua-resty-lrucache/archive/v$RESTY_LRUCACHE_VERSION.tar.gz

COPY . .

# 编译Nginx
RUN cd /nginx-$NGINX_VERSION && \
    ./configure \
        --prefix=/usr/local/nginx \
        --with-http_ssl_module \
        --with-http_v2_module \
        --with-http_realip_module \
        --with-http_addition_module \
        --with-http_sub_module \
        --with-http_dav_module \
        --with-http_flv_module \
        --with-http_mp4_module \
        --with-http_gunzip_module \
        --with-http_gzip_static_module \
        --with-http_random_index_module \
        --with-http_secure_link_module \
        --with-http_stub_status_module \
        --with-http_auth_request_module \
        --with-mail \
        --with-mail_ssl_module \
        --with-stream \
        --with-stream_ssl_module \
        --with-stream_realip_module \
        --add-module=../ngx_devel_kit-$NGX_DEVEL_KIT_VERSION \
        --add-module=../lua-nginx-module-$LUA_NGINX_MODULE_VERSION \
        --add-module=../headers-more-nginx-module-$NGX_HTTP_MORE_MODULE_VERSION && \
    make && \
    make install

# 构建最终镜像
FROM alpine:3.18
RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories

# 安装运行时依赖
RUN apk add --no-cache \
    pcre \
    zlib \
    openssl \
    lua \
    luajit \
    ca-certificates

# 从构建阶段复制Nginx
COPY --from=builder /usr/local/nginx /usr/local/nginx

# 复制resty.coreLua标准库路径
RUN mkdir -p /usr/local/share/lua/5.1/resty
COPY /lua-resty-core/lib/resty/ /usr/local/share/lua/5.1/resty/
COPY /lua-resty-lrucache-0.13/lib/resty/ /usr/local/share/lua/5.1/resty/


# 配置Lua模块路径(覆盖所有可能的搜索路径)
ENV LUA_PACKAGE_PATH="/usr/local/share/lua/5.1/?.lua;/usr/share/lua/5.1/?.lua;/usr/local/lib/lua/5.1/?.lua;;"
ENV PATH="/usr/local/nginx/sbin:${PATH}"

# 创建用户和目录
RUN addgroup -S nginx && adduser -D -S -h /var/cache/nginx -s /sbin/nologin -G nginx nginx && \
    mkdir -p /var/log/nginx && \
    chown -R nginx:nginx /var/log/nginx && \
    mkdir -p /var/cache/nginx && \
    chown -R nginx:nginx /var/cache/nginx

# 复制配置文件
COPY nginx.conf /usr/local/nginx/conf/nginx.conf

EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

TODO: 使用dumb-init解决nginx在容器中的优雅退出

验证过程

部署

docker run -d -p 80:80 --name ngx ngx:1.24.0

验证more-header模块

预期有自定义的响应头

curl -I http://localhost:8080/test-more-headers

HTTP/1.1 200 OK
Server: nginx/1.24.0
Date: Sat, 20 Sep 2025 06:17:28 GMT
Content-Type: application/octet-stream
Content-Length: 44
Connection: keep-alive
X-Test-More-Header: Hello from more-header module
X-Server-Version: Nginx-1.24.0

验证lua模块

预期返回lua脚本生成的动态内容

curl http://localhost:8080/test-lua

Hello from Lua module!
Current time: 2025-09-20 06:23:03
Server address: 172.17.0.2

验证组合功能

curl -i http://localhost:80/test-combined

HTTP/1.1 200 OK
Server: nginx/1.24.0
Date: Sat, 20 Sep 2025 06:25:06 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive
X-Lua-Generated: true
X-Combined-Mode: Lua+more-header

This response uses both modules

在Kubernetes集群中直接使用nginx的问题

“直接使用nginx”是指将nginx二进制程序丢到镜像中运行(不使用controller)。

虽然nginx性能很好,但是在kubernetes中直接使用是有些问题的:

  1. 上游使用Service IP
    • 不能对Pod做主动健康检测
    • 无法使用IP亲和,因为无法区分后端副本
  2. 配置无法动态下发,至少需要手动更新nginx.conf然后reload。在集群规模增大或者需要频繁操作的场景这是不可接受的
  3. worker_processes设置为auto时,在容器环境中通常不能正确识别cgroup 的 CPU 限制
  4. 当 Nginx 收到 SIGTERM 信号时,默认不会优雅退出,而是会立即终止所有进程(包括正在处理的连接),可能导致客户端请求被中断。SIGQUIT才是 Nginx 定义的优雅退出信号,会等待所有活跃连接处理完毕后再退出。可以用dumb-init
  5. 必须在nginx.conf中使用Service IP;如果使用服务名,比如一个Service,删掉之后又创建了新的,nginx中解析到的IP还是之前那个。