fluent-bit是一款轻量级, 高性能的日志采集系统, 具有丰富的input, output 插件支持. 官网: docs.fluentbit.io/manual
下面记录实战中的配置过程及问题记录(踩坑记)
安装及配置
- 配置yum.repo, 创建 /etc/yum.repos.d/fluent-bit.repo
[fluent-bit]
name = Fluent Bit
baseurl = https://packages.fluentbit.io/centos/$releasever/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
repo_gpgcheck=1
enabled=1
- 安装, yum install fluent-bit
- 安装完后, 配置文件默认在 /etc/fluent-bit/ 目录下, 下面以采集nginx 日志到 es 为例进行配置:
-- 在 fluent-bit.conf 中增加:
# 添加hostname字段
[FILTER]
Name modify
Match *
Set hostname misc-01
@INCLUDE inputs.conf
@INCLUDE outputs.conf
-- 新建 inputs.conf:
[INPUT]
name cpu
tag cpu.local
# Read interval (sec) Default: 1
interval_sec 1
[INPUT]
Name tail
Path /data/log/nginx/beta-api-coshare.ccss.tv/access.log
Tag nginx.*
Parser nginx_access
[INPUT]
Name tail
Path /data/log/mitee-srv/mitee-srv0.log
Tag slog-mitee
Skip_Empty_Lines true
Mem_Buf_Limit 100MB
-- 新建 outputs.conf:
#[OUTPUT]
#Name stdout
#Match *
#[OUTPUT]
# Name es
# Match cpu.local
# Host 172.20.151.160
# Port 9200
# Logstash_Format On
[OUTPUT]
Name es
Match nginx.*
Index nginx.beta-api-coshare
Host 172.20.151.160
Port 9200
Generate_ID on # 低版本es需要生成id 字段
Logstash_Format On
Logstash_Prefix nginx-beta-api-coshare
# 采集业务服务日志到 kafka
[OUTPUT]
Name kafka
Match slog-mitee
Brokers 172.16.32.225:9092
Topics fluent-slog-mitee-3
Timestamp_Key _time_
-- 配置自定义的 nginx 日志parser, 在 parser.conf 中添加:
[PARSER]
Name nginx_access
Format regex
#Regex /^(?<remote>[^ ]*) "(?<host>[^ ]*)" (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>\S+)(?: +(?<path>[^"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<request_time>[^ ]*)" "(?<cUDID>[^ ]*)" "(?<MAC>[^ ]*)" "(?<cModel>[^ ]*)" "(?<cChip>[^ ]*)" "(?<cSize>[^ ]*)" "(?<Resolution>[^ ]*)" "(?<aSdk>[^ ]*)" "(?<cTcVersion>[^ ]*)" "(?<cAppVersion>[^ ]*)" "(?<cWebViewVersion>[^ ]*)" "(?<cPkg>[^"]*)" "(?<cFMode>[^ ]*)" "(?<cPattern>[^ ]*)" "(?<cOpenId>[^ ]*)" "(?<cBrand>[^ ]*)" "(?<cSwId>[^ ]*)" "(?<cSkyDevice>[^ ]*)" "(?<country>[^ ]*)" "(?<languag>[^ ]*)" "(?<cCustomId>[^ ]*)" "(?<headerVersion>[^ ]*)" "(?<cEmmcCID>[^ ]*)" "(?<request_body>[^ ]*)" "(?<http_x_forwarded_for>[^ ]*)" "(?<http_soapaction>[^ ]*)" "(?<upstream_response_time>[^ ]*)"(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")/
#Regex ^(?<remote>[^ ]*) "(?<host>[^ ]*)" (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>\S+)(?: +(?<path>[^? ]*+)(?<query>[^ ]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<request_time>[^ ]*)" "(?<cUDID>[^ ]*)" "(?<MAC>[^ ]*)" "(?<cModel>[^"]*)" "(?<cChip>[^ ]*)" "(?<cSize>[^ ]*)" "(?<Resolution>[^"]*)" "(?<aSdk>[^ ]*)" "(?<cTcVersion>[^"]*)" "(?<cAppVersion>[^"]*)" "(?<cWebViewVersion>[^ ]*)" "(?<cPkg>[^"]*)" "(?<cFMode>[^ ]*)" "(?<cPattern>[^ ]*)" "(?<cOpenId>[^ ]*)" "(?<cBrand>[^"]*)" "(?<cSwId>[^ ]*)" "(?<cSkyDevice>[^ ]*)" "(?<country>[^ ]*)" "(?<languag>[^ ]*)" "(?<cCustomId>[^ ]*)" "(?<headerVersion>[^"]*)" "(?<cEmmcCID>[^ ]*)" "(?<request_body>[^"]*)" "(?<http_x_forwarded_for>[^"]*)" "(?<http_soapaction>[^ ]*)" "(?<upstream_response_time>[^ ]*)"(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")
Regex ^(?<remote>[^ ]*) "(?<host>[^ ]*)" (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>\S+)(?: +(?<path>[^? ]*+)(?<query>[^ ]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<request_time>[^ ]*)" "(?<cUDID>[^ ]*)" "(?<MAC>[^ ]*)" "(?<cModel>[^"]*)" "(?<cChip>[^ ]*)" "(?<cSize>[^ ]*)" "(?<Resolution>[^"]*)" "(?<aSdk>[^ ]*)" "(?<cTcVersion>[^"]*)" "(?<cAppVersion>[^"]*)" "(?<cWebViewVersion>[^ ]*)" "(?<cPkg>[^"]*)" "(?<cFMode>[^ ]*)" "(?<cPattern>[^ ]*)" "(?<cOpenId>[^ ]*)" "(?<cBrand>[^"]*)" "(?<cSwId>[^ ]*)" "(?<cSkyDevice>[^ ]*)" "(?<country>[^ ]*)" "(?<languag>[^ ]*)" "(?<cCustomId>[^ ]*)" "(?<headerVersion>[^"]*)" "(?<cEmmcCID>[^ ]*)" "(?<request_body>.{0,1024})[^"]*" "(?<http_x_forwarded_for>[^"]*)" "(?<http_soapaction>[^ ]*)" "(?<upstream_response_time>[^ ]*)"(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
配置parser 时, 可以在 https://cloud.calyptia.com/regex 网站上对正则表达式进行校验, 并实时查看解析结果.
- 启动服务: service fluent-bit start, 并通过 service fluent-bit status 及 journalctl -u fluent-bit.service --since "20 min ago" --no-pager 查看服务运行状态及日志
问题记录:
- 服务启动后, 发现日志信息没有采集 es 中, 通过journalctl查看日志看到报如下错误:
[circuit_breaking_exception] [parent] Data too large, data for [<http_request>]
google 上查到答案是由于 es 的 内存配置太小导致, 调整配置即可 ES_JAVA_OPTS:"-Xmx256m -Xms256m"
- 报 fluent-bit requires a larger buffer size 错误:
[error] [input:tail:tail.1] file=/data/log/mitee-srv/mitee-srv0.log requires a larger buffer size
需调整input 配置, 增加 buffer_chunk_size 和 Buffer_max_size, 如下:
[INPUT]
Name tail
Path /data/log/mitee-srv/mitee-srv0.log
Tag slog-mitee
Skip_Empty_Lines true
Mem_Buf_Limit 50MB
Buffer_Chunk_Size 256KB
Buffer_Max_Size 512KB
- 配置文件块之间的空行不要有空字符, 不然会报错
[error] [config] error in /etc/fluent-bit//inputs.conf:60: invalid indentation level
- es 写入报错 cannot increase buffer, 如下
[http_client] cannot increase buffer: current=512000 requested=544768 max=512000
可以调整output的 Buffer_Size
Buffer_Size 64KB
- 如何进行日志截断
# 通过正则表达式 {0,100}, 如下示例, 可以截取body前1024个字符
# parsers.conf
Regex ^"(?<request_body>.{0,1024})[^"]*" "(?<http_x_forwarded_for>[^"]*)" "(?<http_soapaction>[^ ]*)" "(?<upstream_response_time>[^ ]*)"(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")
