Kafka Connect - Transform (Single Message Transform - SMT)

> Data Integration Tool (ETL/ELT) > Kafka (Event Hub) > Kafka - Connect

1 - About

Transform is a concept of connect to apply simple transformation.

A transform is a simple function that accepts one record as input and outputs a modified record. Transform are chained.

More complex transformations and operations that apply to multiple messages are best implemented with Kafka Streams.

Advertising

3 - Implementation

You can implement the Transformation interface with your own custom logic, package them as a Kafka Connect plugin, and use them with any connectors.

4 - Example

  • TimestampConverter: Cast a string field to Timestamp
"transforms": "convert_op_my_field", // Name of the transform
"transforms.convert_op_my_field.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value", // Timestamp converter
"transforms.convert_op_my_field.target.type": "Timestamp",  // Desired type (i.e. string, long, Date, Time, Timestamp)
"transforms.convert_op_my_field.field": "field_name", //  the field name 
"transforms.convert_op_my_field.format": "yyyy-MM-dd HH:mm:ss.SSSSSS" // in case converting to or from a string, a SimpleDateFormat-compatible format string

5 - Documentation / Reference