Filebeat日志采集

• 使用版本

  filebeat: 6.8.10
  logstash: 6.8.10
  elasticsearch: 7.2

• 下载地址

  www.elastic.co/cn/download…

• 版本匹配地址

  www.elastic.co/support/mat…

• 配置

  • filebeat配置

  filebeat.yml
  
  - type: log
  
    # Change to true to enable this input configuration.
    enabled: true
  
    # Paths that should be crawled and fetched. Glob based paths.
    paths:
      #- /var/log/*.log
      - E:\logs\*
  tags: ["192.168.1.3"]
  # fields字段可自定义用于过滤
  fields:
    # es 要小写
    mac: d4-5d-64-b7-0d-4d
  # 表示字段加在根节点下
  fields_under_root: true
  output.logstash:
    # The Logstash hosts
    hosts: ["192.168.1.3:5044"]
    bulk_max_size: 8192
    pipelining: 10
    worker: 6
  # 同时只能有一个输出  
  #output.console:
      #pretty: true
      #enable: true  
  

  • logstash配置

  logstash.conf
  
  input {
    beats {
      host => "192.168.1.3"
      port => 5044
    }
  }
  
  output {
    elasticsearch {
      hosts => ["http://192.168.1.3:9200"]
      #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      index => "%{[mac]}_%{[tags][0]}_%{+YYYY.MM.dd}"
    }
  }
  

• 启动

  filebeat 启动
  ./filebeat.exe -e
  logstash 启动
  logstash-6.8.10\bin>logstash.bat -f ../config/logstash.conf

• 内容结构

  {
  "@timestamp": "2023-02-28T08:39:38.767Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.8.10"
  },
  "input": {
    "type": "log"
  },
  "offset": 22223,
  "log": {
    "file": {
      "path": "E:\\log.log"
    }
  },
  "message": "日志内容",
  "prospector": {
    "type": "log"
  },
  "host": {
    "architecture": "x86_64",
    "os": {
      "name": "Windows 10 Enterprise",
      "build": "19045.2604",
      "platform": "windows",
      "version": "10.0",
      "family": "windows"
    },
    "name": "DESKTOP-1CT7LFO",
    "id": "7a4bc374-3c38-4557-9244-38752a541481"
  },
  "source": "E:\\log.log",
  "tags": [
    "192.168.1.1"
  ],
  "mac": "d4-5d-64-b7-0d-5d",
  "beat": {
    "name": "DESKTOP-1CT7LFO",
    "hostname": "DESKTOP-1CT7LFO",
    "version": "6.8.10"
  }
}

• 注意事项

  filebeat-> logstash -> es 配置相对好处理, 单独filebeat ->es/kafka 都报无法链接

• 问题及解决

  1. logstash 报Unable to get Charset 'sun.stdout.encoding', using default UTF-8
  解决:

  jvm.options 添加
  -Dsun.stdout.encoding=UTF-8 
  -Dsun.stderr.encoding=UTF-8