小白学习docker-compose部署elk

前言

ELK 是指 Elasticsearch、Logstash 和 Kibana 这三个开源软件的组合。

• Elasticsearch 是一个分布式的搜索和分析引擎，用于日志的存储,搜索,分析,查询。
• Logstash 是一个数据收集、转换和传输工具，用于收集过滤和转换数据，然后将其发送到 Elasticsearch 或其他目标存储中。
• Kibana 是一个数据可视化平台，通过与 Elasticsearch 的集成，提供了强大的数据分析和仪表盘功能。
• Filebeat 是 Elastic Stack（ELK）中的一个组件，用于轻量级的日志文件收集和转发。它能够实时监控指定的日志文件，并将其发送到 Elasticsearch 或 Logstash 进行处理和分析。

本篇 ELK 的版本为 v7.8.1,本篇使用的容器网络为 devopsnetwork ，需创建 docker network create devopsnetwork

部署Elasticsearch

1. 创建对应目录，并更改权限为777

mkdir -p /usr/local/docker/elasticsearch/{conf,logs,data,plugins}

chmod 777 /usr/local/docker/elasticsearch/{conf,logs,data,plugins}

2. 新建docker-compose.yml

• 指定了jvm参数：4g
• 暴露端口 9200：该端口是Elasticsearch REST API的默认端口。
• 暴露端口 9300：该端口是Elasticsearch节点之间的内部通信端口，默认用于节点之间的集群通信
• 挂载数据目录 ./data及配置文件./conf/elasticsearch.yml
• 需要对两个目录进行授权，这里直接用了777，也可以根据官网使用对应es的用户id 1000

version: '3.1'
services:
  elk_elasticsearch:
    image: elasticsearch:7.8.1
    container_name: elk_elasticsearch
    restart: always
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms4096m -Xmx4096m
    ports:
      - 9200:9200 
      - 9300:9300 
    volumes:
      # 授权 chmod 777 ./config/ && chmod 777 ./data/
      - ./data:/usr/share/elasticsearch/data
      - ./conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ./plugins/analysis-ik:/usr/share/elasticsearch/plugins/ik
    networks:
      - devopsnetwork

networks:
  devopsnetwork:
    external: true

3. 进入conf目录，新建elasticsearch.yml

network.host: 0.0.0.0
xpack:
  ml.enabled: false
  monitoring.enabled: false
  security.enabled: false
  watcher.enabled: false

4. 回到elasticsearch目录

docker compose up -d

5. 验证访问

部署Logstash

1. 创建对应目录，并更改权限为777

mkdir -p /usr/local/docker/logstash/{conf,data,logs,pipeline}

chmod 777 /usr/local/docker/logstash/{conf,data,logs,pipeline}

2. 新建docker-compose.yml

• 暴露端口5044：用于接收来自其他主机的日志数据、
• 挂载的 ./pipeline和./config目录可以运行容器复制出来
• 需要将./conf/logstash.yml 和 ./pipeline/logstash.conf 改成es地址，参考

version: '3.1'
services:
  elk_logstash:
    image: logstash:7.8.1
    container_name: elk_logstash
    restart: always
    ports:
      - 5044:5044 
    volumes:
     # 授权 chmod 777 ./logs/ && chmod 777 ./data/ && chmod 777 ./pipeline/ && chmod 777 ./conf/ 
      - /etc/timezone:/etc/timezone
      - /etc/localtime:/etc/localtime:ro
      - ./logs:/usr/share/logstash/logs
      - ./data:/usr/share/logstash/data
      - ./pipeline:/usr/share/logstash/pipeline
      - ./conf:/usr/share/logstash/config
    networks:
      - devopsnetwork

networks:
  devopsnetwork:
    external: true

3. 进入pipeline目录，新建logstash.conf

• 根据需要修改 output ，这里将推送到es地址中
• 输出插件文档

hosts需改成你的elasticsearch访问地址

input {
    beats {
        port => 5044
        codec => json {
            charset => "UTF-8"
        }
    }

}

filter {  
 
}

output {
    elasticsearch { 
      hosts => ["http://192.168.3.5:9200"]
      index => "%{[app]}-%{+YYYY.MM.dd}" 
  }    
  stdout { 
    codec => rubydebug 
  }
}

4. 进入conf目录，新建如下文件

• jvm.options

## JVM configuration

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1g
-Xmx1g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

## Locale
# Set the locale language
#-Duser.language=en

# Set the locale country
#-Duser.country=US

# Set the locale variant, if any
#-Duser.variant=

## basic

# set the I/O temp directory
#-Djava.io.tmpdir=$HOME

# set to headless, just in case
-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one
#-Djna.nosys=true

# Turn on JRuby invokedynamic
-Djruby.compile.invokedynamic=true
# Force Compilation
-Djruby.jit.threshold=0
# Make sure joni regexp interruptability is enabled
-Djruby.regexp.interruptible=true

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${LOGSTASH_HOME}/heapdump.hprof

## GC logging
#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime

# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${LS_GC_LOG_FILE}

# Entropy source for randomness
-Djava.security.egd=file:/dev/urandom

# Copy the logging context from parent threads to children
-Dlog4j2.isThreadContextMapInheritable=true

• log4j2.properties

status = error
name = LogstashPropertiesConfig

appender.console.type = Console
appender.console.name = plain_console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c]%notEmpty{[%X{pipeline.id}]}%notEmpty{[%X{plugin.id}]} %m%n

appender.json_console.type = Console
appender.json_console.name = json_console
appender.json_console.layout.type = JSONLayout
appender.json_console.layout.compact = true
appender.json_console.layout.eventEol = true

rootLogger.level = ${sys:ls.log.level}
rootLogger.appenderRef.console.ref = ${sys:ls.log.format}_console

• logstash.yml

hosts改成你的elasticsearch地址

http.host: "0.0.0.0"
xpack.monitoring.enabled: false
xpack.monitoring.elasticsearch.hosts: [ "http://192.168.3.5:9200" ]

• pipelines.yml

# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: main
  path.config: "/usr/share/logstash/pipeline/logstash.conf"

• startup.options

################################################################################
# These settings are ONLY used by $LS_HOME/bin/system-install to create a custom
# startup script for Logstash and is not used by Logstash itself. It should
# automagically use the init system (systemd, upstart, sysv, etc.) that your
# Linux distribution uses.
#
# After changing anything here, you need to re-run $LS_HOME/bin/system-install
# as root to push the changes to the init script.
################################################################################

# Override Java location
#JAVACMD=/usr/bin/java

# Set a home directory
LS_HOME=/usr/share/logstash

# logstash settings directory, the path which contains logstash.yml
LS_SETTINGS_DIR=/etc/logstash

# Arguments to pass to logstash
LS_OPTS="--path.settings ${LS_SETTINGS_DIR}"

# Arguments to pass to java
LS_JAVA_OPTS=""

# pidfiles aren't used the same way for upstart and systemd; this is for sysv users.
LS_PIDFILE=/var/run/logstash.pid

# user and group id to be invoked as
LS_USER=logstash
LS_GROUP=logstash

# Enable GC logging by uncommenting the appropriate lines in the GC logging
# section in jvm.options
LS_GC_LOG_FILE=/var/log/logstash/gc.log

# Open file limit
LS_OPEN_FILES=16384

# Nice level
LS_NICE=19

# Change these to have the init script named and described differently
# This is useful when running multiple instances of Logstash on the same
# physical box or vm
SERVICE_NAME="logstash"
SERVICE_DESCRIPTION="logstash"

# If you need to run a command or script before launching Logstash, put it
# between the lines beginning with `read` and `EOM`, and uncomment those lines.
###
## read -r -d '' PRESTART << EOM
## EOM

5. 回到logstash目录

docker compose up -d

部署Kibana

1. 创建对应目录，并更改权限为777

mkdir -p /usr/local/docker/kibana/{conf}

chmod 777 /usr/local/docker/kibana/{conf}

2. 新建docker-compose.yml

• 指定es节点是单节点，多节点使用zen
• 挂载配置文件 ./conf/kibana.yml
• 暴露端口 5601：面板访问端口

version: '3.1'
services:
  elk_kibana:
    image: kibana:7.8.1
    container_name: elk_kibana
    restart: always
    environment:
      - discovery.type=single-node
    ports:
      - 5601:5601 
    volumes:
      - ./conf/kibana.yml:/usr/share/kibana/config/kibana.yml
    networks:
      - devopsnetwork

networks:
  devopsnetwork:
    external: true

3. 新建conf/kibana.yml

elasticsearch.hosts修改成你的elasticsearch访问地址

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.3.5:9200/"]
i18n.locale: "zh-CN"
xpack:
  apm.ui.enabled: false
  graph.enabled: false
  ml.enabled: false
  monitoring.enabled: false
  reporting.enabled: false
  security.enabled: false
  grokdebugger.enabled: false
  searchprofiler.enabled: false

4. 打包并运行

docker compose up -d

部署Filebeat

1. 创建对应目录，并更改权限为777

mkdir -p /usr/local/docker/filebeat/{conf}

chmod 777 /usr/local/docker/filebeat/{conf,data,logs,app/logs}

2. 新建docker-compose.yml

• 挂载filebeat的配置文件，数据目录及日志目录，需要设置权限
• 挂载容器外的日志到容器内的日志采集目录

version: '3.1'
services:
  elk_filebeat:
    image: elastic/filebeat:7.8.1
    container_name: elk_filebeat
    restart: always
    volumes:
      # 授权 chmod 777 ./config/ && chmod 777 ./data/ && chmod 777 ./logs/ && chmod 777 /app/logs
      - ./conf/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./data:/usr/share/filebeat/data 
      - ./logs:/usr/share/filebeat/logs
      - ./app/logs:/app/logs
    networks:
      - devopsnetwork

networks:
  devopsnetwork:
    external: true

3. 新建conf/filebeat.yml

output.logstash:
  #logstash hosts
  hosts: ["192.168.3.5:5044"]
fields_under_root: true    
filebeat.inputs: 
 - type: log
   enabled: true
   paths:
       - /app/logs/*/*.log    
   close_older: 24h
   ignore_older: 24h   
   json.keys_under_root: true
   json.overwrite_keys: true
   encoding: utf-8 
filebeat.config.modules: 
  path: ${path.config}/modules.d/*.yml 
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3  
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~  
  - drop_fields:    
      fields: ["log","@version","ecs","agent","beat","host","beat.hostname","beat.version","beat.name","prospector.type","input.type","host.id","host.name","host.os.build","host.os.family","host.os.name","host.os.platform","host.os.platform","log.file.path","tags","offset","host.architecture","host.os.version"]

4. 打包运行

docker compose up -d

参考

• 使用 Docker Compose V2 快速搭建日志分析平台 ELK (Elasticsearch、Logstash 和 Kibana)