Kafka Connect - Connector Plugin

> Data Integration Tool (ETL/ELT) > Kafka (Event Hub) > Kafka - Connect

1 - About

Connector is a component of the connect framework that coordinates data streaming by managing tasks

A connector instance is a logical job. Each connector instance coordinates a set of tasks that actually copy the data.

A connector is run by a worker.

Advertising

3 - Management

See:

The easiest way to create, configure, and manage connectors is with Confluent Control Center.

3.1 - State

The following states are possible for a connector or one of its tasks:

  • UNASSIGNED: The connector/task has not yet been assigned to a worker.
  • RUNNING: The connector/task is running.
  • PAUSED: The connector/task has been administratively paused.
  • FAILED: The connector/task has failed (usually by raising an exception, which is reported in the status output).

4 - Concept

4.1 - Type

Advertising

4.2 - Built-in consumer, producer

Kafka Connect uses the standard Java producer and consumers to communicate with Kafka.

All producer configs and consumer configs can be overridden by prefixing them with producer. or consumer., respectively.

4.3 - List

4.3.1 - Actual Connector

See list connector available for a worker (in distributed mode).

4.3.2 - Externally Available

4.4 - Dev Guide / Build

mvn archetype:generate -DarchetypeGroupId=com.github.jcustenborder.kafka.connect -DarchetypeArtifactId=kafka-connect-quickstart -DarchetypeVersion=0.11.0.0-cp1
Advertising

4.5 - Configuration

Connector configuration file are given:

  • when a worker is started in standalone mode
  • via Rest Api for the distributed mode. See connector. The Connector configuration is persisted into Kafka. Every time you start a worker, it will start all the connectors that were running when it was stopped. The configuration data will be persisted in a the internal topic connect-configs.

Common properties:

  • name - Unique name for the connector. Attempting to register again with the same name will fail.
  • connector.class - The Java class for the connector
  • tasks.max - The maximum number of tasks that should be created for this connector. The connector may create fewer tasks if it cannot achieve this level of parallelism.
  • key.converter and value.converter - (optional) to override the default converter configuration

Sink connectors:

  • topics - A list of topics to use as input for this connector

4.6 - Offset

Even if you delete and recreate the connector, if the connector has the same name it will retain the same offsets previously stored.

5 - Documentation / Reference