Kafka - Schema Registry

> Data Integration Tool (ETL/ELT) > Kafka (Event Hub)

1 - About

schema registry

Schema registry is recommended if you plan to use Avro for a data format because it can help you with:

  • serialization
  • and schema evolution.
Advertising

3 - Connector

with schema knowledge

4 - Management

4.1 - Start

  • Localhost: Start the Schema Registry in its own terminal.
./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties
  • Docker
docker run -d \
  --net=host \
  --name=schema-registry \
  -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=localhost:32181 \
  -e SCHEMA_REGISTRY_HOST_NAME=localhost \
  -e SCHEMA_REGISTRY_LISTENERS=http://localhost:8081 \
  confluentinc/cp-schema-registry:4.0.0

4.2 - Produce with Schema registry

create data with a schema. See Kafka - kafka-avro-console-producer utility

4.3 - Compatibility level

You can change the compatibility level of Schema Registry to allow incompatible schemas or other compatibility levels. There are two ways to do this:

  • Set the compatibility level for subjects which are used by the connector using PUT /config/(string: subject).
  • Configure the Schema Registry to use other schema compatibility level by setting avro.compatibility.level in Schema Registry. Note that this is a global setting that applies to all schemas in the Schema Registry.
Advertising

4.4 - Subject

The subjects have format of topic-key and topic-value where the topic is determined by topic.prefix config and table name.

5 - Documentation / Reference