Kafka源码系列之使用要点总结及重要错误解决

1，创建一个topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

牢记一点，副本数要要小于Broker的总数。

2，topic级别的配置

要多使用topic级别的配置

bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --partitions 1

--replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic

--config max.message.bytes=128000

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic

--deleteConfig max.message.bytes

3，kafka的client

非java客户端的位置

https://cwiki.apache.org/confluence/display/KAFKA/Clients

Java客户端的位置

贴这两张原图虽然，显得愚笨，确实有新手搞了一周没找到JAVA API的demo所在位置，所以浪尖在这里为了节省大家时间提升效率截了图两张。

4，常见错误处理

虚拟机内装了kafka，自己在windows主机写的代码，无法生产或者消费消息，而虚拟机里的命令行客户端却可以。需要配置广播参数，原理请看浪尖kafka系列文章前几篇。

0.8.2.2版本参数

# Hostname the broker will advertise to producers and consumers. If not set, it uses the
# value for "host.name" if configured.  Otherwise, it will use the value returned from
# java.net.InetAddress.getCanonicalHostName().
#advertised.host.name=<hostname routable by clients>

# The port to publish to ZooKeeper for clients to use. If this is not set,
# it will publish the same port that the broker binds to.
#advertised.port=<port accessible by clients>

0.9+版本参数

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = security_protocol://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092

# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092

要学会用的两个指令是telnet(网路连通性测试和端口存活探测)和ping(网络连通性测试)。

该篇是kafka源码系列的最后一篇，kafka与spark Streaming结合及与flume的整合多种形式，会在flume和spark专题里再讲解。