nginx自定义格式日志

nginx 的日志默认是写在文件里的，在多台服务器情况下需要同步日志，可以通过MQ来集中收集 我们先看一个普通的web服务以及nginx配置，先用go起一个http服务 代码如下

func main() {
	s := g.Server()
	s.BindHandler("/index", func(r *ghttp.Request) {
		r.Response.WriteJsonExit(g.Map{
			"tip": "欢迎来到hello world:" + gconv.String(gtime.Datetime()),
		})
	})
	s.SetPort(9999)
	s.Run()
}

跑起来

配上nginx，我是用虚拟机桥接的,http服务在192.168.0.220的9999端口，先改一下本机的host文件

192.168.0.220 www.nginxlog.com

nginx配置文件

server {
    listen 80;
    server_name www.nginxlog.com;
    client_max_body_size 1G;
    location  / { 
        proxy_pass http://127.0.0.1:9999;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header REMOTE-HOST $remote_addr;

        proxy_hide_header content-security-policy;
        proxy_connect_timeout 30s;
        proxy_read_timeout 86400s;
        proxy_send_timeout 30s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_redirect off;
        #add_header X-Cache $upstream_cache_status;#¨º1¨®?¡Á??¡§¦Ì?cache2???
        add_header Cache-Control no-store;
        
        #这里就是记录日志
        access_log  /home/wwwlogs/nglog_access.log;
        error_log   /home/wwwlogs/nglog_api_error.log;

    }
}

重启nginx，访问物理机访问www.nginxlog.com/index

也是可以的

这个时候查看日志文件

[root@localhost ~]# tail -f /home/wwwlogs/nglog_access.log 
192.168.0.173 - - [09/Oct/2023:15:23:09 +0800] "GET / HTTP/1.1" 404 40 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
192.168.0.173 - - [09/Oct/2023:15:23:09 +0800] "GET /favicon.ico HTTP/1.1" 404 40 "http://www.nginxlog.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
192.168.0.173 - - [09/Oct/2023:15:23:18 +0800] "GET /index HTTP/1.1" 200 64 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"

我们希望采集到的格式是json格式，原格式不好做入库处理 ，在nginx的配置文件http处配置一下我们期望的格式

 log_format logstash_json '{ "@timestamp": "$time_local", '
         '"@fields": { '
         '"remote_addr": "$remote_addr", '
         '"remote_user": "$remote_user", '
         '"body_bytes_sent": "$body_bytes_sent", '
         '"request_time": "$request_time", '
         '"status": "$status", '
         '"request": "$request", '
         '"request_method": "$request_method", '
         '"http_referrer": "$http_referer", '
         '"body_bytes_sent":"$body_bytes_sent", '
         '"http_x_forwarded_for": "$http_x_forwarded_for", '
         '"http_user_agent": "$http_user_agent" } }';

在自己的server上启用此日志格式

server {
    listen 80;
    server_name www.nginxlog.com;
    client_max_body_size 1G;
    location  / { 
        proxy_pass http://127.0.0.1:9999;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header REMOTE-HOST $remote_addr;

        proxy_hide_header content-security-policy;
        proxy_connect_timeout 30s;
        proxy_read_timeout 86400s;
        proxy_send_timeout 30s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_redirect off;
        #add_header X-Cache $upstream_cache_status;#¨º1¨®?¡Á??¡§¦Ì?cache2???
        add_header Cache-Control no-store;
        
        #这里就是记录日志，加上自己配置的日志格式logstash_json
        access_log  /home/wwwlogs/nglog_access.log logstash_json;
        error_log   /home/wwwlogs/nglog_api_error.log;

    }
}

重启nginx ，再访问，查看nginx日志

{ "@timestamp": "09/Oct/2023:15:42:46 +0800", "@fields": { "remote_addr": "192.168.0.173", "remote_user": "-", "body_bytes_sent": "64", "request_time": "0.002", "status": "200", "request": "GET /index HTTP/1.1", "request_method": "GET", "http_referrer": "-", "body_bytes_sent":"64", "http_x_forwarded_for": "-", "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" } }