这是我参与11月更文挑战的第4天，活动详情查看：2021最后一次更文挑战
今天主要想介绍一下logstash 这个工具，真的很不错，最近在工作中使用到了，对于日志传输收集等都很友好，可以很方便的同步数据到elasticsearch 或者 kafka等工具均可以，今天主要介绍同步文件数据到elasticsearch中
docker pull logstash:6.4.0
km_log_pattern 文件：
STIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}:?%{SECOND},?%{MSECONDS}
logstash.conf 配置参数：
读取 文件数据 并写入 elasticsearch
input {
file {
path => ["/home/work/testVolume/test_map_log/*.log","/home/work/testVolume/test_map_log/*.log"]
type => "test_map_new"
start_position => "beginning"
}
}
\
filter {
grok {
patterns_dir => ["/config-dir/cmap_log_pattern"]
match => {
"message" => [
"\[%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second},%{MSECONDS:mill_seconds}\]\[user_id:%{GREEDYDATA:user_id},mobile:%{GREEDYDATA:user_mobile},status:%{GREEDYDATA:user_status},real_name:%{GREEDYDATA:real_name},email:%{GREEDYDATA:user_email},city:%{GREEDYDATA:user_city},permission_info:%{GREEDYDATA:permission_info},b_stree_permission:%{GREEDYDATA:b_stree_permission},together_permission:%{GREEDYDATA:together_permission},is_admin:%{GREEDYDATA:is_admin}\]\[URL:%{GREEDYDATA:uri}\]\[params:%{GREEDYDATA:params_json_content}\]",
"\[%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second},%{MSECONDS:mill_seconds}\]\[user_id:%{GREEDYDATA:user_id},mobile:%{GREEDYDATA:mobile},platformCompany:%{GREEDYDATA:platformCompany},real_name:%{GREEDYDATA:real_name},email:%{GREEDYDATA:email},city:%{GREEDYDATA:city},role:%{GREEDYDATA:role},platformCompany:%{GREEDYDATA:platformCompany}\]\[URL:%{GREEDYDATA:uri}\]\[params:%{GREEDYDATA:params_json_content}\]",
"\[%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second},%{MSECONDS:mill_seconds}\]\[user_id:%{GREEDYDATA:user_id}\]\[URL:%{GREEDYDATA:uri}\]\[params:%{GREEDYDATA:params_json_content}\]"
]
}
}
json {
source => "params_json_content"
target => "params_json"
remove_field => ["paramsjson"]
}
}
\
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "test_log"
user => "test"
password => "xxxxx"
}
stdout { codec => line }
}
读取kafka数据写入elasticsearch
input {
kafka {
bootstrap_servers => ["xxx.xxx.xxx.xxx:9092"]
auto_offset_reset => "latest"
consumer_threads => 5
decorate_events => true
group_id => "xxx"
topics => ["xxxxxxxxxx"]
type => "xxxxx"
}
}
output {
stdout {}
elasticsearch {
hosts => ["xxx.xxx.xxx.xxx:9200"]
index => "kafka-xxx-%{+YYYY.MM.dd}"
}
}
启动docker命令：
docker run -d --name logstash_test --log-opt max-size=10m --log-opt max-file=3 -v /config-dir:/config-dir -v /home/work/logstash_test/logstash:/home/work/logstash_test/logstash -v logstash -f /config-dir/logstash.conf
以上是通过读取文件然后写入elasticsearch 的方式去进行部署还有一种方式是通过部署logstash服务，其他服务进行服务调用去写入
相关logstash.conf 配置：
input {
tcp {
host => "0.0.0.0"
port => "5044"
codec => json
}
}
filter{
if [type] == "logstash" {
ruby {
code => "event.set('timestamp', event.timestamp.time.localtime.strftime('%Y-%m-%d %H:%M:%S'))"
}
}
}
output {
elasticsearch {
hosts => ["xx.xx.xx.xx:9200","xx.xx.xx.xx:9200"] #可以配置多个机器 一般为集群
user => "xxxxxx"
password => "xxxxxx"
index => "xxxxxx"
codec => "json"
}
stdout { codec => json }
}
启动命令：
docker run -it -d -p 5044:5044--name logstash --net somenetwork -v /docker/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml -v /docker/logstash/conf.d/:/usr/share/logstash/conf.d/ logstash:6.4.0
通过上面就可以搭建一个logstash的服务了，然后其他应用就可以直接调用 xx.xx.xx.xx:5044 传输日志文件进入elasticsearch 了
