Kafka - Connect

> Data Integration Tool (ETL/ELT) > Kafka (Event Hub) > Kafka - Connect

1 - About

Kafka Connect is a framework service based around connector to move data into and out of Kafka.

The service is a worker. By starting a worker, you are starting connect.

The management is done through the rest api and the connect name is often used to describe its rest api.

Advertising

3 - Concept

Connectors and tasks are logical units of work:

  • Connectors – the high level abstraction that coordinates data streaming by managing tasks
  • Tasks – the implementation of how data is copied to or from Kafka

Runtime:

  • Workers – the running processes that execute connectors and tasks

Transform:

  • Converters – Read/write implementation of a data format
  • Transforms – simple logic to alter each message produced by or sent to a connector

4 - Management

4.1 - Start

Kafka Connect has only one hard prerequisite in order to get started: a set of Kafka brokers because it use it as a back-end database to store connector and task configurations, offsets, and status in internal Kafka topics

Connect (and therefore a worker) can be started

  • as a standalone process for testing, one-off jobs and single agent (such as sending logs from webservers to Kafka)
  • or as a distributed, scalable, fault tolerant service
Advertising

5 - Documentation / Reference