Kafka Connect - Standalone Mode

> Data Integration Tool (ETL/ELT) > Kafka (Event Hub) > Kafka - Connect

1 - About

The runtime standalone mode of connect when running/starting a worker

Standalone mode is best suited for:

  • testing,
  • one-off jobs
  • or single agent (such as sending logs from webservers to Kafka)

You can still test a distributed mode locally by setting a different rest port. See locally

Advertising

3 - Management

3.1 - Start

To run a worker in standalone:

./bin/connect-standalone [-daemon] worker.properties connector1.properties [connector2.properties connector3.properties ...]
 
# Example
./bin/connect-standalone ./etc/schema-registry/connect-avro-standalone.properties ./etc/kafka-connect-jdbc/source-quickstart-sqlite.properties

where:

Users may optionally provide connector configurations at the command line as only a single worker instance exists and no coordination is required in standalone mode.

3.2 - Log

When started as a deamon, the log is located to /logs/connectStandalone.out

  • Check the log
cat /logs/connectStandalone.out | grep -i "finished"
[2017-10-18 12:43:26,306] INFO Finished creating connector test-source-sqlite-jdbc-autoincrement (org.apache.kafka.connect.runtime.Worker:225)
[2017-10-18 12:43:26,425] INFO Source task WorkerSourceTask{id=test-source-sqlite-jdbc-autoincrement-0} finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:143)

3.3 - Configuration

3.3.1 - Worker

configuration file in standalone mode.

See Work Config Reference If you run multiple standalone instances on the same host, there are a couple of settings that must be unique between each instance:

  • offset.storage.file.filename - storage for connector offsets, which are stored on the local filesystem in standalone mode; using the same file will lead to offset data being deleted or overwritten with different values
  • rest.port - the port the REST interface listens on for HTTP requests
Advertising

3.3.2 - Connector

name=local-file-sink
connector.class=FileStreamSinkConnector
tasks.max=1
file=test.sink.txt
topics=connect-test