Kafka - Installation and services

> Data Integration Tool (ETL/ELT) > Kafka (Event Hub)

1 - About

3 - Steps

3.1 - Installation

  • With zip file
<path-to-confluent>/bin/        # Driver scripts for starting/stopping services
<path-to-confluent>/etc/        # Configuration files
<path-to-confluent>/share/java/ # Jars
  • With package, If you installed from deb or rpm packages, the contents are installed globally and you’ll need to adjust the paths used below:
/usr/bin/                  # Driver scripts for starting/stopping services, prefixed with <package> names
/etc/<package>/            # Configuration files
/usr/share/java/<package>/ # Jars
Advertising

3.1.1 - Windows

!!! Important on WINDOWS Version 3.3.0 !!!

  • With MINGW, in ./bin/kafka-run-class if running on Windows with Mingwin, replace the CYGWIN test at the begining:
# if [[ $(uname -a) =~ "CYGWIN" ]]; then
# becomes
if [[ $(uname -a) =~ "CYGWIN" || $(uname -a) =~ "MINGW" ]]; then
  • Schema registry file (./bin/schema-registry-run-class) must be modified if you are using bash on Windows. At the end of the CLASSPATH and SCHEMA_REGISTRY_LOG4J_OPTS init, add the below line
# Before the setting of SCHEMA_REGISTRY_LOG4J_OPTS
# The block above the assignment is adpated in order to have the variable SCHEMA_REGISTRY_LOG4J_OPTS that contains only a directory
# (ie -Dlog4j.configuration=file: is suppressed)
# Then we modifed the path to turn them in Windows form wikth cygpath
if [[ $(uname -a) =~ "CYGWIN" || $(uname -a) =~ "MINGW" ]]; then
  LOG_DIR=$(cygpath --path --mixed "${LOG_DIR}")
  SCHEMA_REGISTRY_LOG4J_OPTS=$(cygpath --path --mixed "${SCHEMA_REGISTRY_LOG4J_OPTS}")
fi
SCHEMA_REGISTRY_LOG4J_OPTS="-Dschema-registry.log.dir=$LOG_DIR -Dlog4j.configuration=file:$SCHEMA_REGISTRY_LOG4J_OPTS"
# before the call of the java command
if [[ $(uname -a) =~ "CYGWIN" || $(uname -a) =~ "MINGW" ]]; then
  CLASSPATH=$(cygpath --path --mixed "${CLASSPATH}")
fi
  • Rest file (./bin/kafka-rest-run-class) must be modified if you are using bash on Windows. At the end of the CLASSPATH and KAFKAREST_LOG4J_OPTS init, add the below line
# After the setting of KAFKAREST_LOG4J_OPTS
# The block above the assignment is adapted in order to have the variable KAFKAREST_LOG4J_OPTS that contains only a directory
# (ie -Dlog4j.configuration=file: is suppressed)
# Then we modifed the path to turn them in Windows form wikth cygpath
# CYGINW == 1 if Cygwin is detected, else 0.
if [[ $(uname -a) =~ "CYGWIN" || $(uname -a) =~ "MINGW" ]]; then
  KAFKAREST_LOG4J_OPTS=$(cygpath --path --mixed "${KAFKAREST_LOG4J_OPTS}")
  KAFKAREST_LOG4J_OPTS="-Dlog4j.configuration=file:${KAFKAREST_LOG4J_OPTS}"
fi
# before the call of the java command
if [[ $(uname -a) =~ "CYGWIN" || $(uname -a) =~ "MINGW" ]]; then
  CLASSPATH=$(cygpath --path --mixed "${CLASSPATH}")
fi

3.2 - Start the services

3.2.1 - With the CLI

  • Default Start with the cli
$ export PATH=<path-to-confluent>/bin:$PATH
$ confluent start
# confluent help
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]
  • To start just Zookeeper, Kafka and Schema Registry run:
$ export PATH=<path-to-confluent>/bin:$PATH
$ confluent start schema-registry
Advertising

3.2.2 - One by one

  • Start Zookeeper in its own terminal. Zookeeper is used to track the health of the nodes.
./bin/zookeeper-server-start ./etc/kafka/zookeeper.properties
  • Start Kafka in its own terminal.
./bin/kafka-server-start ./etc/kafka/server.properties
./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties
./bin/kafka-rest-start ./etc/kafka-rest/kafka-rest.properties
  • Start Connect in distributed mode. Run this command in its own terminal.
$ ./bin/connect-distributed ./etc/schema-registry/connect-avro-distributed.properties

4 - Support

4.1 - Log

confluent current
/tmp/confluent.tbVCwQps
cd /tmp/confluent.tbVCwQps
ls
connect  kafka  kafka-rest  schema-registry  zookeeper
  • Querying the connect log for instance
less connect/connect.stderr
Advertising

5 - Documentation / Reference