Kafka - Partition
Table of Contents
1 - About
Data Partitions (Clustering of data) in Kafka
Each partition is an:
- immutable sequence of records
that is continually appended to—a structured commit log.
Each partition in the Kafka cluster has:
- a leader
2 - Articles Related
3 - Purpose
- act as the unit of parallelism
* 1 Partition = 1 process * 2 partition = 2 process * ...
4 - Structure
Topic * -> partition 1 * -> segment 11 * -> segment 12 * -> partition 2 * -> segment 21 * -> segment 22 .......
5 - Physic Server Distribution
6 - Record Partition Assignment
7 - Replication
Each partition is replicated across a configurable number of servers for fault tolerance.
Replicated partitions are called replicas
8 - Leader / Follower
The leader handles all read and write requests for the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader.
Each server acts as a leader for some of its partitions and a follower for others so load is well balanced within the cluster.