Working with Partitions

1. Understanding Partition Assignment

Property	Description	Default Value	Notes
partition.assignment.strategy	Strategy used to assign partitions to consumers in a consumer group	`org.apache.kafka.clients.consumer.RangeAssignor`	Can be `RangeAssignor`, `RoundRobinAssignor`, or `StickyAssignor`
num.partitions	Default number of partitions for new topics	`1`	Used when creating topics without specifying partition count
replication.factor	Default replication factor for new topics	`1`	Number of replicas for each partition

Example: Consumer Group with Partition Assignment

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "my-consumer-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.StickyAssignor");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("my-topic"));

Note: Partition assignment strategies determine how partitions are distributed among consumers in a group. StickyAssignor is recommended for better load balancing and stability.

Key Concepts

Partition Assignment ensures that each partition is consumed by exactly one consumer in a group.
Assignments are managed by the coordinator and can change during rebalances.
Strategies like StickyAssignor aim to minimize reassignment overhead.

2. Specifying Partition Key

Property	Description	Values	Notes
partition.key	Key used to determine partition	`string`	Used for partitioning messages
partitioner	Partitioning strategy	`default`, `random`, `roundrobin`	Default is `default` which uses hash of key

Example: Setting Partition Key

const kafka = new Kafka({ brokers: ['localhost:9092'] });
const producer = kafka.producer();

await producer.connect();
await producer.send({
  topic: 'my-topic',
  messages: [
    {
      key: 'user-123',
      value: JSON.stringify({ message: 'Hello World' })
    }
  ]
});

Note: If no key is provided, Kafka uses a random partitioner to distribute messages.

3. Viewing Partition Distribution

Property	Description	Example
partition.id	Unique identifier for each partition within a topic	`0`, `1`, `2`
leader	Broker ID of the leader replica for a partition	`1001`
replicas	List of broker IDs that host replicas of the partition	`[1001, 1002, 1003]`
isr	List of in-sync replicas (replicas that are up-to-date)	`[1001, 1002]`

Example: Using Kafka CLI to describe partitions

kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092

Note: The isr list may be smaller than the replicas list if some replicas are out of sync.

4. Adding Partitions

Note: Adding partitions to an existing topic is a one-time operation and cannot be undone. It's important to plan partition counts carefully to avoid performance issues or data loss.

Property	Value	Description
topic.name	`my-topic`	Name of the Kafka topic to modify
new.partition.count	`10`	New total number of partitions for the topic

Example: Adding Partitions to a Topic

kafka-topics.sh --alter --topic my-topic --partitions 10

Note: The number of partitions can only be increased, not decreased. Reducing the number of partitions is not supported and will result in an error.

Warning: Increasing the partition count of a topic can lead to rebalancing of consumer groups and may cause temporary performance degradation.

Key Considerations When Adding Partitions

Partition count can only be increased, not decreased.
Adding partitions triggers a rebalance of consumer groups.
Each new partition must be assigned to a broker in the cluster.
Partitioning strategy affects data distribution and consumer parallelism.

5. Understanding Leader Election

Property	Description	Default Value	Notes
leader.imbalance.per.broker.percentage	Maximum imbalance percentage allowed per broker	10	Used to control leader imbalance in the cluster
leader.imbalance.check.interval.ms	Interval between leader imbalance checks	300000	Default 5 minutes
unclean.leader.election.enable	Allow follower brokers to become leaders	false	When set to true, can lead to data loss

Example: Configuring Unclean Leader Election

{
  "unclean.leader.election.enable": true,
  "leader.imbalance.per.broker.percentage": 5
}

Note: Enabling unclean leader election can lead to data inconsistency and should be used with caution.

Leader Election Process

Kafka uses a distributed consensus protocol to elect leaders
Leaders are responsible for handling read and write requests for a partition
When a leader broker becomes unavailable, a new leader is elected from the ISR (In-Sync Replicas)
Leaders are chosen based on availability and replication status

6. Understanding Log Segments

Property	Description	Default Value	Notes
log.segment.bytes	Maximum size of a single log segment file	`1073741824`	1 GB
log.segment.ms	Time interval after which a new log segment is created	`604800000`	7 days
log.roll.hours	Deprecated in favor of `log.segment.ms`	`168`	Deprecated since 0.10.0
log.retention.hours	Retention period for log segments in hours	`168`	7 days
log.retention.bytes	Maximum size of log segments before deletion	`-1`	Unlimited

Example: Configuring Log Segment Settings

{
  "log.segment.bytes": "1073741824",
  "log.segment.ms": "604800000",
  "log.retention.hours": "168"
}

Note: Log segments are immutable once created and can only be deleted when retention policies are met.

Warning: Setting log.segment.bytes too low can lead to excessive file system overhead.

7. Configuring Preferred Leader Election

Example: Setting Preferred Leader Election Policy

kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --alter --add-config preferred.leader.election.policy=offline

Property	Values	Description
preferred.leader.election.policy	`offline`, `prefer-local`	Determines the policy for preferred leader election. `offline` allows any broker to become leader, while `prefer-local` prefers leaders from the local broker.

Note: The preferred.leader.election.policy setting is used to configure how Kafka selects the preferred leader for a partition when a new leader is required.

8. Configuring Min In-Sync Replicas

Property	Default Value	Description
min.insync.replicas	2	Minimum number of replicas that must acknowledge a write for the write to be considered successful

Example: Setting min.insync.replicas

{"min.insync.replicas": 3}

Note: Setting min.insync.replicas to a value higher than the total number of replicas in the topic will cause writes to fail.

9. Setting Partition Assignment Strategy

Strategy	Description	Default	Notes
range	Partitions are assigned to consumers in a range-based manner	Yes	Simple and efficient for small numbers of consumers
round-robin	Partitions are assigned to consumers in a round-robin fashion	No	Ensures even distribution across consumers
sticky	Partitions are assigned to consumers with a focus on minimizing rebalances	No	Optimizes for stability and reduces consumer group rebalances

Example: Configuring Partition Assignment Strategy

Properties props = new Properties();
props.put("group.id", "my-group");
props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.StickyAssignor");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

Note: The sticky strategy is recommended for production environments to reduce rebalances and improve stability.

10. Manual Partition Reassignment

Example: Reassigning partitions manually

kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file reassignment.json --execute

Parameter	Description	Values
--bootstrap-server	Bootstrap server address	`localhost:9092`
--reassignment-json-file	Path to JSON file containing reassignment plan	`reassignment.json`
--execute	Execute the reassignment plan	`--execute`

Note: The reassignment plan must be created using kafka-reassign-partitions.sh with the --generate option before executing.

Example: Reassignment JSON structure

{
  "version": 1,
  "partitions": [
    {
      "topic": "my-topic",
      "partition": 0,
      "replicas": [1, 2, 3]
    }
  ]
}

Workflow for manual partition reassignment

Generate a reassignment plan using kafka-reassign-partitions.sh --generate
Review and modify the generated plan if necessary
Execute the reassignment using kafka-reassign-partitions.sh --execute
Verify the reassignment status using kafka-reassign-partitions.sh --verify