Grafana对一些中间的监控实践

1.容器监控

1.1 容器监控工具cAdvisor

对于docker容器可以使用监控工具cAdvisor进行监控，同时可以整合Prometheus

docker安装： (ps:本服务器没开8080安全组，所以将本机 9004 端口映射到容器的 8080 端口)访问：IP+9004

docker run -d --name cadvisor -p 9004:8080 -v /:/rootfs:ro -v /var/run/:/var/run/:rw -v /sys/:/sys/:ro -v /var/lib/docker/:/var/lib/docker/:ro -v /dev/disk/:/dev/disk/:ro google/cadvisor:v0.32.0

可以看的已经可以通过cadvisor监控的容器的信息，并有图表展示。 下面采用 grafana，进行集中式展示，方便查看各个主机的信息，在使用 grafana 之前，我们还需要将这些数据用 Prometheus 进行收集。

1.2 grafana集中式展示

Prometheus.yml配置

  - job_name: "docker"
    file_sd_configs:
      - files: ['/usr/local/prometheus/sd_config/docker.yml']
        refresh_interval: 5s

docker.yml

- targets:
  - 192.168.0.26:9004
  - 192.168.0.27:9004
  - 192.168.0.28:9004

grafana展示

可以在grafana.com/dashboards/ 下载别的模板目前展示两个模板

导入模板：193

导入模板：11600

2.数据库监控

2.1 mysql数据库监控

创建监控用户、赋权限

CREATE USER 'exporter' IDENTIFIED BY 'ctsi@123';
grant PROCESS, REPLICATION CLIENT, SELECT on *.* to 'exporter'@'%';

mysql_exporter：用于收集MySQL性能信息

下载 github.com/prometheus/… 解压

tar -xvf mysqld_exporter-0.14.0.linux-amd64.tar.gz

mysqld_exporter-0.14.0.linux-amd64目录下创建my.cnf

vi my.cnf

增加数据库连接信息
[client]
user=root
password=lzz@189.
host=127.0.0.1
port=9005

写进系统服务

vi /usr/lib/systemd/system/mysqld_exporter.service

[Unit]  
Description=[https://prometheus.io](https://prometheus.io/)

[Service]  
Restart=on-failure  
ExecStart=/usr/local/mysqld_exporter/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter/my.cnf

[Install]  
WantedBy=multi-user.target

启动

systemctl daemon-reload
systemctl start mysqld_exporter.service 

Prometheus增加配置

  - job_name: mysql
    static_configs:
    - targets: ['192.168.0.27:9104']

查看

2.2 grafana集中式展示

导入模板：7362

3.nginx监控

插曲：本想用Prometheus日志收集grafana展示来着，后来看到基于 Loki+grafana的nginx监控页面更好看些。于是决定搞一下

3.1 Loki介绍

Loki 是受Prometheus启发的水平可扩展、高可用、多租户日志聚合系统。它的设计非常经济高效且易于操作。它不索引日志的内容，而是索引每个日志流的一组标签。

基于 Loki 的日志堆栈由 3 个组件组成：

• promtail是代理，负责收集日志并发送给 Loki。
• loki是主服务器，负责存储日志和处理查询。
• Grafana用于查询和显示日志。

3.2 Loki安装

Loki架构

Loki 分两部分，Loki 是日志引擎部分，Promtail 是收集日志端。

• Loki 是主服务器，负责存储日志和处理查询 。
• promtail 是代理，负责收集日志并将其发送给 loki 。

安装参考：Loki安装部署_loki部署-CSDN博客 (blog.csdn.net/qq_30614345…)

nginx安装Nginx + GeoIP2 获取客户端地理位置及限制指定地区访问 blog.csdn.net/weimeilayer…

nginx.conf配置

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;
error_log /usr/local/nginx/logs/error.log;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}

stream {
    log_format proxy '$remote_addr [$time_local] ' '$protocol $status $bytes_sent $bytes_received ' '$session_time "$upstream_addr" ' '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';
    access_log /usr/local/nginx/logs/tcp-access.log proxy ;
    open_log_file_cache off;
    include /usr/local/nginx/conf/conf.d/*.stream;
}
http {
    include       mime.types;
    default_type  application/octet-stream;

	log_format json_analytics escape=json '{'
                            '"msec": "$msec", ' # request unixtime in seconds with a milliseconds resolution
                            '"connection": "$connection", ' # connection serial number
                            '"connection_requests": "$connection_requests", ' # number of requests made in connection
                    '"pid": "$pid", ' # process pid
                    '"request_id": "$request_id", ' # the unique request id
                    '"request_length": "$request_length", ' # request length (including headers and body)
                    '"remote_addr": "$remote_addr", ' # client IP
                    '"remote_user": "$remote_user", ' # client HTTP username
                    '"remote_port": "$remote_port", ' # client port
                    '"time_local": "$time_local", '
                    '"time_iso8601": "$time_iso8601", ' # local time in the ISO 8601 standard format
                    '"request": "$request", ' # full path no arguments if the request
                    '"request_uri": "$request_uri", ' # full path and arguments if the request
                    '"args": "$args", ' # args
                    '"status": "$status", ' # response status code
                    '"body_bytes_sent": "$body_bytes_sent", ' # the number of body bytes exclude headers sent to a client
                    '"bytes_sent": "$bytes_sent", ' # the number of bytes sent to a client
                    '"http_referer": "$http_referer", ' # HTTP referer
                    '"http_user_agent": "$http_user_agent", ' # user agent
                    '"http_x_forwarded_for": "$http_x_forwarded_for", ' # http_x_forwarded_for
                    '"http_host": "$http_host", ' # the request Host: header
                    '"server_name": "$server_name", ' # the name of the vhost serving the request
                    '"request_time": "$request_time", ' # request processing time in seconds with msec resolution
                    '"upstream": "$upstream_addr", ' # upstream backend server for proxied requests
                    '"upstream_connect_time": "$upstream_connect_time", ' # upstream handshake time incl. TLS
                    '"upstream_header_time": "$upstream_header_time", ' # time spent receiving upstream headers
                    '"upstream_response_time": "$upstream_response_time", ' # time spend receiving upstream body
                    '"upstream_response_length": "$upstream_response_length", ' # upstream response length
                    '"upstream_cache_status": "$upstream_cache_status", ' # cache HIT/MISS where applicable
                    '"ssl_protocol": "$ssl_protocol", ' # TLS protocol
                    '"ssl_cipher": "$ssl_cipher", ' # TLS cipher
                    '"scheme": "$scheme", ' # http or https
                    '"request_method": "$request_method", ' # request method
                    '"server_protocol": "$server_protocol", ' # request protocol, like HTTP/1.1 or HTTP/2.0
                    '"pipe": "$pipe", ' # "p" if request was pipelined, "." otherwise
                    '"gzip_ratio": "$gzip_ratio", '
                    '"http_cf_ray": "$http_cf_ray",'
                    '"geoip_country_code": "$geoip2_country_code"'
                    '}';	
	access_log /usr/local/nginx/logs/access.log json_analytics; # 这边修改为上边promtail配置文件中指定的日志路径

    sendfile        on;
    keepalive_timeout  65;
    # 加载GeoIP2模块
    # load_module modules/ngx_http_geoip2_module.so;
    # 配置GeoIP2数据库路径

	#IP地址库解析映射 
	geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
			auto_reload 5m;
			$geoip2_metadata_country_build metadata build_epoch; #变量名可以自定义
			$geoip2_country_code default=US country iso_code;
			$geoip2_country_name country names zh-CN;
		}
    server {
        listen       9000;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

    }
}

4.在Grafana增加loki数据源

loki地址：

下载模板导入 Loki NGINX Service Mesh - JSON version | Grafana Labs

参考

grafana+loki+promtail 搭建Nginx日志高级监控_loki grafana采集nginx日志：blog.csdn.net/manongwangz…

Prometheus完整搭建及实战各种监控_prometheus监控部署:blog.csdn.net/qq_42684940…