Flume 案例：负载均衡

Flume 1 （服务器 adp-01）配置

a1.sources = r1
a1.channels = c1
a1.sinkgroups = g1
a1.sinks = k1 k2

a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type = load_balance
# round_robin 轮询
# random 随机
a1.sinkgroups.g1.processor.selector = round_robin
# 当给其中一个 sink 发送数据失败后，是否在一段时间内不再给该 sink 发送数据
# 设置为 true，代表启动此机制，给某个 sink 发送数据失败后
# 则在一段时间内，不会在尝试给此 sink 发送数据
# 这个时间是变化的，每次尝试失败都会增加此时间，但是最大不超过 processor.selector.maxTimeOut
# 设置为 false，即使上次给 sink 发送数据失败，本次轮训或随机到此 sink 后
# 仍然会尝试给此 sink 发送数据
# 默认 false
a1.sinkgroups.g1.processor.backoff = false
# 单位毫秒，默认 30000
a1.sinkgroups.g1.processor.selector.maxTimeOut = 30000

a1.sinks.k1.type = avro
a1.sinks.k1.hostname = adp-02
a1.sinks.k1.port = 30000

a1.sinks.k2.type = avro
a1.sinks.k2.hostname = adp-03
a1.sinks.k2.port = 30000

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1

Flume 2（服务器 adp-02）配置

a2.sources = r1
a2.channels = c1
a2.sinks = k1

a2.sources.r1.type = avro
a2.sources.r1.bind = adp-02
a2.sources.r1.port = 30000

a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100

a2.sinks.k1.type = logger

a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

Flume 3（服务器 adp-03）配置

a3.sources = r1
a3.channels = c1
a3.sinks = k1

a3.sources.r1.type = avro
a3.sources.r1.bind = adp-03
a3.sources.r1.port = 30000

a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100

a3.sinks.k1.type = logger

a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1

启动 flume agent：

# 先启动监听 avro 端口的 agent
# 在服务器 adp-02 上启动
$ flume-ng agent -n a2 -c conf -f a2.conf
# 在服务器 adp-03 上启动
$ flume-ng agent -n a3 -c conf -f a3.conf
# 在服务器 adp-01 上启动
$ flume-ng agent -n a1 -c conf -f a1.conf

在 adp-01 上发送数据：

~ nc localhost 44444
hello a2
OK
hello a3
OK

在 adp-02 和 adp-03 上会分别收到一条数据：

[admin@adp-02 ~]
2023-04-27 00:36:44,665 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 61 32                         hello a2 }

[admin@adp-03 ~]
2023-04-27 00:36:48,243 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 61 33                         hello a3 }