二、部署logstash：

logstash环境准备及安装：

Logstash是一个开源的数据收集引擎，可以水平伸缩，而且logstash整个ELK当中拥有最多插件的一个组件，其可以接收来自不同来源的数据并统一输出到指定的且可以是多个不同目的地。

logstash1(103)

安装openjdk

apt install openjdk-8-jdk -y

安装包：logstash

cd /usr/local/src/

#安装
dpkg -i logstash-6.8.3.deb

logstash帮助

/usr/share/logstash/bin/logstash --help

测试logstash：

1.测试标准输入和输出

/usr/share/logstash/bin/logstash -e 'input { stdin{}} output { stdout{ codec => rubydebug }}'
...中间部分省略
[INFO ] 2020-05-08 16:13:07.216 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}   <--Successfully表示成功
hello world!   <--输入后、返回值
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
          "host" => "logstash1",    <--拿到了主机名
       "message" => "hello world!",    <--消息的具体内容
    "@timestamp" => 2020-05-08T08:16:06.760Z,    <--当前事件的发生时间
      "@version" => "1"    <--事件版本号，一个事件就是一个ruby对象
}
中文    <--也可以解析中文
{
          "host" => "logstash1",
       "message" => "中文",
    "@timestamp" => 2020-05-08T08:19:38.869Z,
      "@version" => "1"
}

/usr/share/logstash/bin/logstash -e 'input { stdin { type => "syslog" }} output { stdout { codec => rubydebug }}'
...中间部分省略
linux01
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
      "@version" => "1",
    "@timestamp" => 2020-05-08T08:28:01.318Z,
       "message" => "linux01",
          "type" => "syslog",
          "host" => "logstash1"
}

把配置放到文件目录中

#切换目录
cd /etc/logstash/conf.d/

#编辑
vim test.conf
#输入
input {
  stdin {
    type => "syslog" 
  }
}

#输出
output {
  stdout {
    codec => rubydebug
  }
}

检查文件

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf -t

[INFO ] 2020-05-08 16:47:00.294 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash    <--表示正常

修改文件

vim test.conf
#输入
input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
    stat_interval => "3"
    type => "syslog"
  }
}

#输出
output {
  elasticsearch {
    hosts => ["http://192.168.37.101:9200"]
    index => "syslog-37-103-%{+YYYY.MM.dd}"
  }
}

检查文件

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf -t

启动

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

查看文件是否有权限

#可以看到权限是640、logstash属于其他、所以没有权限
ll /var/log/syslog 
-rw-r----- 1 syslog adm 2562408 May  8 17:17 /var/log/syslog

启动服务

systemctl enable logstash
systemctl start logstash

需要修改权限

#查看日志
tail -f /var/log/logstash/logstash-plain.log
...中间部分省略
[2020-05-08T17:30:02,500][WARN ][filewatch.tailmode.handlers.createinitial] failed to open /var/log/syslog: #<Errno::EACCES: Permission denied - /var/log/syslog>, ["org/jruby/RubyIO.java:1236:in `sysopen'", "org/jruby/RubyFile.java:367:in `initialize'", "org/jruby/RubyIO.java:1155:in `open'"]

编辑启动脚本

#修改权限
vim /etc/systemd/system/logstash.service

[Service]
User=root    <--
Group=root    <--

重新加载配置文件、重启服务

systemctl daemon-reload
systemctl restart logstash

host1(101)和host2(102)

可以看到一些数据

ll /esdata/data/nodes/0/
indices/   node.lock  _state/

ll /esdata/data/nodes/0/indices/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 May  8 21:21 ./
drwxr-xr-x 4 elasticsearch elasticsearch 4096 May  8 21:21 ../
drwxr-xr-x 8 elasticsearch elasticsearch 4096 May  8 21:21 NrrrXyllSWmd-Vo3cgEJLA/

host1(101)

安装kibana、安装包：kibana

cd /usr/local/src/

#安装
dpkg -i kibana-6.8.3-amd64.deb

修改kibana配置信息

vim /etc/kibana/kibana.yml
server.port: 5601    <--端口
server.host: "0.0.0.0"    <--服务器主机
elasticsearch.hosts: ["http://192.168.37.102:9200"]    <--elasticsearch主机(101或102)

启动服务并设为开机启动

systemctl enable kibana
systemctl start kibana

可以看到一些数据

logstash1(103)

添加一个值、追加到文件最后

echo "111" >> /var/log/syslog

刷新页面后、可以看到新的日志信息、此时就要通过Kibana收集日志

关于sincedb：

cat /var/lib/logstash/plugins/inputs/file/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc
264896 0 2050 2386149 1683639321.210242 /var/log/syslog    <--记录了收集文件的inode信息


ll /var/log/syslog -i
264896 -rw-r----- 1 syslog adm 2386149 May  9 21:35 /var/log/syslog    <--

2.测试输出到文件

logstash1(103)

#当前所在目录
pwd
/etc/logstash/conf.d

#修改配置文件
vim test.conf
#输入
input {
#  stdin {
#    type => "syslog" 
#  }
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
    stat_interval => "3"
    type => "syslog"
  }
}

#输出
output {
#  stdout {
#    codec => rubydebug
#  }

  elasticsearch {
    hosts => ["http://192.168.37.101:9200"]
    index => "syslog-37-103-%{+YYYY.MM.dd}"
  }

  file {
    path => "/tmp/test.txt"
  }
}

重启服务

systemctl restart logstash

添加一个值、追加到文件最后

echo "222" >> /var/log/syslog

刷新页面后、可以看到新的日志信息

ll /tmp/test.txt
-rw-r--r-- 1 root root 7086 May  9 22:53 /tmp/test.txt

host1(101)

修改kibana语言为中文

vim /etc/kibana/kibana.yml
i18n.locale: "zh-CN"

重启服务

systemctl restart kibana

刷新页面后可看到、页面为中文