Flume 案例:聚合

93 阅读1分钟

aggregation-case.png

Flume 1 (服务器 adp-01)配置

a1.sources = r1
a1.channels = c1
a1.sinks = k1

a1.sources.r1.type = exec
a1.sources.r1.command = tail -F ./group.log

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.sinks.k1.type = avro
a1.sinks.k1.hostname = adp-03
a1.sinks.k1.port = 30000

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

Flume 2(服务器 adp-02)配置

a2.sources = r1
a2.channels = c1
a2.sinks = k1

a2.sources.r1.type = netcat
a2.sources.r1.bind = localhost
a2.sources.r1.port = 44444

a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100

a2.sinks.k1.type = avro
a2.sinks.k1.hostname = adp-03
a2.sinks.k1.port = 30000

a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

Flume 3(服务器 adp-03)配置

a3.sources = r1
a3.channels = c1
a3.sinks = k1

a3.sources.r1.type = avro
a3.sources.r1.bind = adp-03
a3.sources.r1.port = 30000

a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100

a3.sinks.k1.type = logger

a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1

启动 flume agent:

# 先启动监听 avro 端口的 agent
# 在服务器 adp-03 上启动
$ flume-ng agent -n a3 -c conf -f a3.conf
# 在服务器 adp-01 上启动
$ flume-ng agent -n a1 -c conf -f a1.conf
# 在服务器 adp-02 上启动
$ flume-ng agent -n a2 -c conf -f a2.conf

在 adp-02 上发送数据:

[admin@adp-02 ~]$ nc localhost 44444
hello flume

在 adp-01 上写数据到文件中

[admin@adp-01 ~]$ echo hello flume > group.log

在 Flume 3 (adp-03) 上接收到全部数据:

[admin@adp-03 ~]
2023-04-27 01:58:31,212 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 66 6C 75 6D 65                hello flume }
2023-04-27 01:58:40,538 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 66 6C 75 6D 65                hello flume }