介绍
Logstash 是一款开源的数据收集引擎。Logstash 可以从不同来源归集数据并动态地转化数据。在筛选和转化数据后，可用于不同的下游分析用例（如 Elasticsearch）和可视化用例（如 Kibana）。
Logstash 除了在日志收集方面不断改进，也有很强的扩展能力。任何的事件类型都可以进行收集和转换，通过配置多种 input、filter、output 插件，以及 codec 简化数据处理。
Logstash版本
基于Logstash 7.17，支持的操作系统及JVM
Docker安装Logstash
拉取镜像
docker pull logstash:7.17.0
Logstash配置
Pipeline配置
Logstash通过pipeline配置收集数据
pipeline配置在/usr/share/logstash/pipeline/
默认的配置文件是/usr/share/logstash/pipeline/logstash.conf
1.logstash.conf配置示例
input {
beats {
port => 5044
}
}
output {
stdout {
codec => rubydebug
}
}
2.将宿主机的pipeline配置目录挂载到容器
docker run --rm -it -v ~/pipeline/:/usr/share/logstash/pipeline logstash:7.17.0
Logstash配置
logstash配置在/usr/share/logstash/config/
通常使用一个自定义的logstash.yml文件
1.默认的logstash.yml配置
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
2.将宿主机的logstash配置目录挂载到容器内
docker run --rm -it -v ~/settings/:/usr/share/logstash/config logstash:7.17.0
3.将宿主机的单个logstash配置文件挂载到容器内
docker run --rm -it -v ~/settings/logstash.yml:/usr/share/logstash/config/logstash.yml logstash:7.17.0
Logging配置
在 Docker 中，Logstash 日志默认地会去到标准输出。
配置在/usr/share/logstash/config/log4j2.properties
插件配置
插件配置类似于模块，包括input，output，filter 和 codec。
配置示例：
input {
file {
path => "/var/log/messages"
type => "syslog"
}
file {
path => "/var/log/apache/access.log"
type => "apache"
}
}
配置示例
配置filter
input {}中定义使用stdin {}即标准输入作为数据来源；
filter {}中定义解析grok {}任意格式文本，其中message是对%{COMBINEDAPACHELOG}匹配模式自定义的字段名；
filter {}中定义解析date{}日期文本；其中timestamp是对dd/MMM/yyyy:HH:mm:ss Z匹配格式自定义的字段名；
output {}中定义将数据输出到elasticsearch和stdout。
input { stdin { } }
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
输入示例：
127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
输出示例：
{
"message" => "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/status.php HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
"@timestamp" => "2013-12-11T08:01:45.000Z",
"@version" => "1",
"host" => "cadenza",
"clientip" => "127.0.0.1",
"ident" => "-",
"auth" => "-",
"timestamp" => "11/Dec/2013:00:01:45 -0800",
"verb" => "GET",
"request" => "/xampp/status.php",
"httpversion" => "1.1",
"response" => "200",
"bytes" => "3891",
"referrer" => "\"http://cadenza/xampp/navi.php\"",
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\""
}
处理系统日志
配置示例：
input {
tcp {
port => 5000
type => syslog
}
udp {
port => 5000
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
发起 telnet：
telnet localhost 5000
输出示例：
Dec 23 12:11:43 louis postfix/smtpd[31499]: connect from unknown[95.75.93.154]
Dec 23 14:42:56 louis named[16000]: client 199.48.164.7#64817: query (cache) 'amsterdamboothuren.com/MX/IN' denied
Dec 23 14:30:01 louis CRON[619]: (www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)
Dec 22 18:28:06 louis rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="2253" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.
{
"message" => "Dec 23 14:30:01 louis CRON[619]: (www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)",
"@timestamp" => "2013-12-23T22:30:01.000Z",
"@version" => "1",
"type" => "syslog",
"host" => "0:0:0:0:0:0:0:1:52617",
"syslog_timestamp" => "Dec 23 14:30:01",
"syslog_hostname" => "louis",
"syslog_program" => "CRON",
"syslog_pid" => "619",
"syslog_message" => "(www-data) CMD (php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log)",
"received_at" => "2013-12-23 22:49:22 UTC",
"received_from" => "0:0:0:0:0:0:0:1:52617",
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice"
}
参考
[1] Logstash Introduction
[2] Running Logstash on Docker
[3] Configuring Logstash for Docker
[4] Elastic 支持一览表
