Kafka - Docker Single Node (Multiple Service Broker + Zookeeper)

> Data Integration Tool (ETL/ELT) > Kafka (Event Hub)

1 - About

Docker Single Node step by step tutorial adapted from the Quickstart documentation.

Made:

  1. on Windows 7
  2. with Git Bash for Windows as console
  3. with Confluent docker image version 3.3.0
docker-compose version
docker-compose version 1.16.1, build 6d1ac219
docker-py version: 2.5.1
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.2j  26 Sep 2016
Advertising

3 - Step by Step

3.1 - Get the docker-compose.yml file

  • With git
git clone https://github.com/confluentinc/cp-docker-images.git
cd cp-docker-images/examples/kafka-single-node
cat docker-compose.yml

3.2 - Modification of the docker-compose.yml file

Docker - docker-compose.yml need to be modified to:

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:3.3.0
    network_mode: host
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"
      - "default:127.0.0.1"
  kafka:
    image: confluentinc/cp-kafka:3.3.0
    network_mode: host
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: localhost:32181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    extra_hosts:
      - "moby:127.0.0.1"
      - "default:127.0.0.1"

3.3 - Start the services

docker-compose up -d
Starting kafkasinglenode_zookeeper_1 ...
Starting kafkasinglenode_zookeeper_1 ... done
Starting kafkasinglenode_kafka_1 ...
Starting kafkasinglenode_kafka_1 ... done
  • Up ? If the state is not Up, rerun the docker-compose up -d command.
docker-compose ps
           Name                        Command            State   Ports
-----------------------------------------------------------------------
kafkasinglenode_kafka_1       /etc/confluent/docker/run   Up
kafkasinglenode_zookeeper_1   /etc/confluent/docker/run   Up
  • Log Check Healthy Zookeeper
docker-compose logs zookeeper | grep -i binding
zookeeper_1  | [2017-10-12 13:17:35,956] INFO binding to port 0.0.0.0/0.0.0.0:32181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
zookeeper_1  | [2017-10-12 13:19:03,213] INFO binding to port 0.0.0.0/0.0.0.0:32181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
  • Log Check Healthy Kafka
docker-compose logs kafka | grep -i started
kafka_1      | [2017-10-12 13:20:31,103] INFO [Socket Server on Broker 1], Started 1 acceptor threads (kafka.network.SocketServer)
kafka_1      | [2017-10-12 13:20:31,353] INFO [Replica state machine on controller 1]: Started replica state machine with initial state -> Map() (kafka.controller.ReplicaStateMachine)
kafka_1      | [2017-10-12 13:20:31,355] INFO [Partition state machine on Controller 1]: Started partition state machine with initial state -> Map() (kafka.controller.PartitionStateMachine)
kafka_1      | [2017-10-12 13:20:31,490] INFO [Kafka Server 1], started (kafka.server.KafkaServer)
Advertising

3.4 - Produce and Consume

docker-compose exec kafka  bash -c "seq 42 | kafka-console-producer --request-required-acks 1 --broker-list localhost:29092 --topic foo && echo 'Produced 42 messages.'"
Produced 42 messages.
docker-compose exec kafka  \
  kafka-console-consumer --bootstrap-server localhost:29092 --topic foo --new-consumer --from-beginning --max-messages 42
...
36                                 
37                                 
38                                 
39                                 
40                                 
41                                 
42                                 
Processed a total of 42 messages