Kafka - Stream Word Count demo

> Data Integration Tool (ETL/ELT) > Kafka (Event Hub)

1 - About

Stream Word count demo adapted from the stream quickstart.

3 - Prerequisites

192.168.99.100   broker
192.168.99.100   zookeeper
Advertising

4 - Steps

4.1 - Start the services

cd /kafka/docker/stream
export COMPOSE_PROJECT_NAME="stream-demo"
docker-compose up -d
Creating network "streamdemo_default" with the default driver
Creating streamdemo_zookeeper_1 ...
Creating streamdemo_zookeeper_1 ... done
Creating streamdemo_broker_1 ...
Creating streamdemo_broker_1 ... done
  • Service status
docker-compose ps
         Name                     Command            State                     Ports
-------------------------------------------------------------------------------------------------------
streamdemo_broker_1      /etc/confluent/docker/run   Up      0.0.0.0:9092->9092/tcp
streamdemo_zookeeper_1   /etc/confluent/docker/run   Up      0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

4.2 - Create a topic

# Enter in the broker container and start bash
docker-compose exec broker bash
# Create the topic
kafka-topics --create \
    --zookeeper zookeeper:2181 \
    --replication-factor 1 \
    --partitions 1 \
    --topic streams-plaintext-input
Created topic "streams-plaintext-input".
  • Create the output topic
kafka-topics --create \
    --zookeeper zookeeper:2181 \
    --replication-factor 1 \
    --partitions 1 \
    --topic streams-wordcount-output \
    --config cleanup.policy=compact
Created topic "streams-wordcount-output".
  • Describe them
kafka-topics \
    --zookeeper zookeeper:2181 \
    --describe
Topic:streams-plaintext-input   PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: streams-plaintext-input  Partition: 0    Leader: 1       Replicas: 1     Isr: 1

Topic:streams-wordcount-output  PartitionCount:1        ReplicationFactor:1     Configs:cleanup.policy=compact
        Topic: streams-wordcount-output Partition: 0    Leader: 1       Replicas: 1     Isr: 1
Advertising

4.3 - Code

4.4 - Packaging

java -cp ./target/kafka-streams-0.1-standalone.jar com.gerardnico.kafka.demo.WordCountDemo