一、安装scala 2.11.4

1、将课程提供的scala-2.11.4.tgz使用WinSCP拷贝到spark1的/usr/local目录下。
2、对scala-2.11.4.tgz进行解压缩：tar -zxvf scala-2.11.4.tgz。
3、对scala目录进行重命名：mv scala-2.11.4 scala
4、配置scala相关的环境变量
vi .bashrc
export SCALA_HOME=/usr/local/scala
export PATH=$SCALA_HOME/bin
source .bashrc
5、查看scala是否安装成功：scala -version
6、按照上述步骤在spark2和spark3机器上都安装好scala。使用scp将scala和.bashrc拷贝到spark2和spark3上即可。

二、 安装Kafka包

1、将kafka_2.9.2-0.8.1.tgz压缩包使用Xshell拷贝到spark1的/usr/local目录下。
2、对kafka_2.9.2-0.8.1.tgz进行解压缩：tar -zxvf kafka_2.9.2-0.8.1.tgz。
3、对kafka目录进行改名：mv kafka_2.9.2-0.8.1 kafka
4、配置kafka
vim /usr/local/kafka/config/server.properties
broker.id：依次增长的整数，0、1、2、3、4，集群中Broker的唯一id
zookeeper.connect=192.168.1.107:2181,192.168.1.108:2181,192.168.1.109:2181
5、安装slf4j
将slf4j-1.7.6.zip压缩包上传到/usr/local目录下
unzip slf4j-1.7.6.zip
把slf4j中的slf4j-nop-1.7.6.jar复制到kafka的libs目录下面
6、按照上述步骤在spark2和spark3分别安装kafka。用scp把kafka拷贝到spark2和spark3行即可。
7、唯一区别的，就是server.properties中的broker.id，要设置为1和2

三、 启动kafka集群

1、在三台机器上分别执行以下命令：
前台启动：
[root@weekend03 bin]./kafka-server-start.sh ../config/server.properties

后台启动:
[root@weekend03 bin]./kafka-server-start.sh ../config/server.properties 1>/dev/null 2>&1 &
./kafka-server-start.sh -daemon ../config/server.properties

指定监听端口
JMX_PORT=2898 ./
kafka-server-start.sh ../config/server.properties 1>/dev/null 2>&1 &

2、解决kafka Unrecognized VM option 'UseCompressedOops'问题
vi bin/kafka-run-class.sh 
if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
  KAFKA_JVM_PERFORMANCE_OPTS="-server  -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"
fi
去掉-XX:+UseCompressedOops即可

3、使用jps检查启动是否成功

四、 测试kafka集群

#创建topic：
bin/kafka-topics.sh --zookeeper 192.168.88.200:2181,192.168.88.101:2181,192.168.88.102:2181 --topic wangcc --replication-factor 1 --partitions 1 --creat
#往Kafka的topic中写入数据(命令行的生成者)
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wangcc
#启动消费者
./kafka-console-consumer.sh --zookeeper weekend01:2181,weekend02:2181,weekend03:2181 --topic wangcc --from-beginning

安装搭建

官方安装搭建

Kafka配置文件详解

1. kafka创建topic出现：

Error while executing topic command org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids

原因： 没有在kafka目录下创建zookeeper ，指定myid

解决：

cd /uardata10/chbtmp/package/kafka_2.10-0.8.2.1
mkdir zookeeper
cd zookeeper 
touch myid 
echo 0 > myid 
重新启动kafka就ok

1.1 、创建topic的paration大于可用的boker

创建topic 指定partitions 为5  而 borker只有一个
 ./bin/kafka-topics.sh -zookeeper idc007128:2181,idc007124:2181,idc007123:2181  -topic test -replication-factor 2 -partitions 5 -create 
 ##问题
Error while executing topic command replication factor: 2 larger than available brokers: 1

解决：

 -replication-factor 1  副本数为1
[root@idc007130 kafka_2.10-0.8.2.1]# ./bin/kafka-topics.sh -zookeeper idc007128:2181,idc007124:2181,idc007123:2181  -topic test -replication-factor 1 -partitions 1 -create  
Created topic "test".

2、kafka的broker.id 重复导致出错

原因： 复制config/server.properties没有修改主机名和borker.id

[2017-06-14 18:07:55,583] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured brokerId 0 doesn't match stored brokerId 1 in meta.properties
        at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
        at kafka.Kafka$.main(Kafka.scala:67)
        at kafka.Kafka.main(Kafka.scala)

解决：

第一步： 修改config/server.properties中的boker.id 及主机名

第二步：删除log.dirs=/tmp/kafka-logs目录， 重新启动

3、kafka配置文件错误

[2018-10-12 08:35:04,553] ERROR Error when sending message to topic TestTopic with key: null, value: 12 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

[2018-10-12 08:35:05,335] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-10-12 08:35:06,326] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-10-12 08:35:07,309] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-10-12 08:35:08,444] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

解决：修改server.properties文件
listeners=PLAINTEXT://:9092

常见命令：

1.创建topic
kafka-topics.sh --zookeeper 192.168.88.200:2181,192.168.88.101:2181,192.168.88.102:2181 --topic wangcc (有空格)--replication-factor 1 --partitions 1 --creat
2.删除指定topic
kafka-topics.sh --zookeeper localhost:2181  --delete --topic topic-test
3.查看已创建的topic列表
kafka-topics.sh --zookeeper localhost:2181 --list
4.查看topic的信息
kafka-topics.sh --zookeeper localhost:2181 --describe --topic myTopic
5.往Kafka的topic中写入数据(命令行的生成者)
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic wangcc
6.启动消费者
./kafka-console-consumer.sh --zookeeper weekend01:2181,weekend02:2181,weekend03:2181 --topic wangcc --from-beginning