Kafka - Utility

> Data Integration Tool (ETL/ELT) > Kafka (Event Hub)

1 - Utility Belt

1.1 - Dub (Docker Utility Belt)

dub [-h] {template,ensure,ensure-atleast-one,wait,http-ready,path} Actions...

1.1.1 - Ensure (Verification of an env parameter)

dub ensure [-h] name
dub ensure KAFKA_ZOOKEEPER_CONNECT
dub ensure KAFKA_ADVERTISED_LISTENERS
dub ensure KAFKA_SSL_KEYSTORE_FILENAME
dub ensure KAFKA_SSL_KEY_CREDENTIALS
Advertising

1.1.2 - Path

dub path [-h] path {writable,readable,executable,exists}
dub path /etc/kafka/ writable
dub path "$KAFKA_SSL_KEYSTORE_LOCATION" exists

1.1.3 - Template

conversion from the env variable to a properties files through templating

dub template "/etc/confluent/docker/${COMPONENT}.properties.template" "/etc/${COMPONENT}/${COMPONENT}.properties"
dub template "/etc/confluent/docker/log4j.properties.template" "/etc/${COMPONENT}/log4j.properties"
dub template "/etc/confluent/docker/tools-log4j.properties.template" "/etc/${COMPONENT}/tools-log4j.properties"

where:

Example: ${COMPONENT}.properties.template. kafka.properties.template

kafka.properties.template
{% set excluded_props = ['KAFKA_VERSION',
                         'KAFKA_HEAP_OPTS'
                         'KAFKA_LOG4J_OPTS',
                         'KAFKA_OPTS',
                         'KAFKA_JMX_OPTS',
                         'KAFKA_JVM_PERFORMANCE_OPTS',
                         'KAFKA_GC_LOG_OPTS',
                         'KAFKA_LOG4J_ROOT_LOGLEVEL',
                         'KAFKA_LOG4J_LOGGERS',
                         'KAFKA_TOOLS_LOG4J_LOGLEVEL']
-%}
{% set kafka_props = env_to_props('KAFKA_', '', exclude=excluded_props) -%}
{% for name, value in kafka_props.iteritems() -%}
{{name}}={{value}}
{% endfor -%}
 
{% set confluent_support_props = env_to_props('CONFLUENT_SUPPORT_', 'confluent.support.') -%}
{% for name, value in confluent_support_props.iteritems() -%}
{{name}}={{value}}
{% endfor -%}

becomes for kafka /etc/${COMPONENT}/${COMPONENT}.properties

broker.id=1
advertised.listeners=PLAINTEXT://localhost:29092
offsets.topic.replication.factor=1
zookeeper.connect=localhost:32181
log.dirs=/var/lib/kafka/data
listeners=PLAINTEXT://0.0.0.0:29092
Advertising

1.2 - Cub (Confluent Platform Utility Belt)

cub [-h] {zk-ready,kafka-ready,sr-ready,kr-ready,listeners,ensure-topic} Actions...

Example:

KAFKA_LISTENERS=$(cub listeners "$KAFKA_ADVERTISED_LISTENERS")
  • Check if Zookeeper is healthy
cub zk-ready "$KAFKA_ZOOKEEPER_CONNECT" "${KAFKA_CUB_ZK_TIMEOUT:-40}"
  • Check if Kafka is ready
cub kafka-ready \
    "${CONNECT_CUB_KAFKA_MIN_BROKERS:-1}" \
    "${CONNECT_CUB_KAFKA_TIMEOUT:-40}" \
    -b "$CONNECT_BOOTSTRAP_SERVERS" \
    --config /etc/"${COMPONENT}"/kafka-connect.properties