Spark踩坑-kafka StringDeserializer/StringSerializer could not be found

启动时kafka报错。StringDeserializer/StringSerializer 同理。

配置如下

props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");

Class org.apache.kafka.common.serialization.StringDeserializer could not be found.

日志如下

org.apache.kafka.common.config.ConfigException: Invalid value org.apache.kafka.common.serialization.StringDeserializer for configuration key.deserializer: Class org.apache.kafka.common.serialization.StringDeserializer could not be found.
	at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:728)
	at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:474)
	at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:467)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
	at org.apache.kafka.clients.consumer.ConsumerConfig.<init>(ConsumerConfig.java:539)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)

原因及解决

issue is caused by way Kafka load classes - it seems to be using wrong classloaded. Workaround with passing deserializer classes:\ classLoad 拿不到类。改成通过类获取。

props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());