优雅的查看系统日志附上ELK+Kafka+filebeat的日志管理系统搭建优雅的查看系统日志附上ELK+Kafka+f

持续创作，加速成长！这是我参与「掘金日新计划 · 10 月更文挑战」的第2天，点击查看活动详情

docker安装kafka和zookeeper

拉去镜像

docker pull zookeeper:latest
docker pull wurstmeister/kafka:latest

启动zookeeper

docker run -d --name zookeeper --publish 2181:2181 --volume /etc/localtime:/etc/localtime zookeeper:latest

启动kafka

docker run -d --name kafka --publish 9092:9092 \
--link zookeeper \
--env KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
--env KAFKA_ADVERTISED_HOST_NAME=192.168.66.128 \
--env KAFKA_ADVERTISED_PORT=9092 \
--volume /etc/localtime:/etc/localtime \
wurstmeister/kafka:latest

测试zookeeper和kafka是否安装成功

# 进入kafka的安装文件内部
docker exec -it kafka /bin/bash
# 进入bin目录
cd /opt/kafka_2.13-2.8.1/bin/


# 创建生产者之后会输入一些内容
./kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic elk_kafka 

# 新开窗口运行消费者，指定同样的topic， 注意替换kafkaIp
./kafka-console-consumer.sh --bootstrap-server 192.168.58.129:9092 --topic elk_kafka --from-beginning 


# 在生产者页面输入消息后，消费者可以看到，表示kafka搭建成功

docker安装ELK

修改mmap计数大于等于262144的限制

#在/etc/sysctl.conf文件最后添加一行
vm.max_map_count=655360
#并执行命令
sysctl -p

下载elk镜像，整体下载，docker里面有一个完整的ELK包

docker pull sebp/elk:740    #780指的是下载7.8.0的版本，更多版本可以查看dockerhub官网

运行镜像

docker run -p 5601:5601 -p 9200:9200 -p 9300:9300 -p 5044:5044 --name elk -d sebp/elk:740

运行完毕后可以访问elasticSearch的端口和kibana的端口，查看是否运行成功
进入elk容器内部
```
docker exec -it elk bash
```

修改文件

vim /etc/logstash/conf.d/02-beats-input.conf

# 修改成这个文件的内容如下所示  删除ssl开头的三个内容 表示是否使用证书 不删除的话再启动filebeat的时候可能出现错误，需要把logstash.crt文件拷贝到客户端，然后再filebeat.yml文件中添加路径。

input {
  kafka {
    codec => "json"
    topics => ["sparksys-log"]
    bootstrap_servers => "kafka的IP:9092"
    auto_offset_reset => "latest"
    group_id => "logstash-g1"
  }
}

vim /etc/logstash/conf.d/30-output.conf

# 修改30-output.conf配置，索引自定义
# 192.168.58.128表示elasticsearch的IP

output {
  elasticsearch {
    hosts => ["192.168.58.128:9200"]     
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
}

过滤器文件
vim /etc/logstash/conf.d/10-syslog.conf

退出重启容器
```
docker restart elk（或者容器ID）
```

可以把配置文件映射到主机永久保留

复制elasticsearch的配置出来
mkdir /opt/elk/elasticsearch/conf
docker cp elk:/etc/elasticsearch/elasticsearch.yml /opt/elk/elasticsearch/conf
#复制logstash的配置出来
mkdir /opt/elk/logstash/conf
docker cp elk:/etc/logstash/conf.d/. /opt/elk/logstash/conf/   docker -cp 容器id/容器名称:容器位置 宿主机位置   
                                            详细docker-cp的命令https://blog.csdn.net/wx940627/article/details/106843650
                                            

启动时候挂载卷的麻烦可以在/opt/elk目录下建立一个start.sh文件
docker stop elk
docker rm elk

docker run -p 5601:5601 -p 9200:9200 -p 9300:9300 -p 5044:5044 \
    -v $PWD/logstash/conf:/etc/logstash/conf.d \
    -v $PWD/elasticsearch/log:/var/log/elasticsearch \
    -v $PWD/logstash/log:/var/log/logstash \
    --name elk \
    -d sebp/elk:740                                                 $PWD表示start.sh文件所在的文件目录/opt/elk


./start.sh  运行镜像

安装filebeat

直接访问artifacts.elastic.co/downloads/b…网站下载后上传到linux的centos7上，这样速度较快
rz -E 或者使用客户端工具XFTP上传文件
安装filebeat sudo rpm -vi filebeat-6.2.4-x86_64.rpm
安装完之后配置文件默认在下图所示目录 /etc/filebeat

修改配置文件filebeat.yml

vim /etc/filebeat/ filebeat.yml

修改部分

enabled:fasle修改成true（filebeat 6.0后,enabled默认为关闭,必须要修改成true）
paths:表示你想要抓取分析的日志所在路径,我这里就不改了,一般指向项目产生日志的所在目录  - /var/log/logs/*.log  自定义捕获的文件路径
注释掉elasticsearch和logstash的输出添加kafka的输出
#----------------------------- kafka output --------------------------------
output.kafka:
  enabled: true
  hosts: ["192.168.58.128:9092"]
  topic: sparksys-log

附上完整filebeat完整文件

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- type: log

  # Change to true to enable this prospector configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/logs/*.log
    #- c:\programdata\elasticsearch\logs*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after


#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

#============================= Elastic Cloud ==================================

# These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
 # hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["192.168.58.128:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#----------------------------- kafka output --------------------------------
output.kafka:
  enabled: true
  hosts: ["192.168.58.128:9092"]
  topic: sparksys-log

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:

启动（重启）filebeat 他会自动在elasticsearch中建立索引
```
/etc/init.d/filebeat restart
```

访问kibana并创建默认索引并测试环境搭建是否成功

调用elasticsearch的方法可以查看所有索引 ----- http://192.168.58.128:9200/_cat/indices

2. 访问kibana 3. 选中filebeat进入下一步

选择时间模式查看

测试：现在我们手动写入一个日志文件到/var/log/下,如下图所示

在/var/log/logs目录下创建.log结尾的目录，然后再kibana中马上就可以看见到消息

Tips内存溢出问题

最后附上自己在运行过程出现内存不够的问题见下面这个博客 -- OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Out of memory' (errno=12) - 朝明 - 博客园 (cnblogs.com)

查看内存使用情况

free -m

创建swapfile

# dd  if=/dev/zero  of=swapfile  bs=1024  count=500000  
# count=空间大小 of空间名字

将swapfile设置为swap空间

mkswap swapfile

启用交换空间，这个操作有点类似于mount操作

# swapon  swapfile （删除交换空间 swapoff swapfile）

至此增加交换空间的操作结束了，可以使用free命令查看swap空间大小是否发生变化

free -m