Understanding Kafka Architecture

1. Understanding Broker Role

Property	Description	Notes
broker.id	Unique identifier for each Kafka broker in the cluster	Must be unique per broker, typically a positive integer
listeners	Network interfaces the broker listens on for client connections	Can include multiple listeners (e.g., PLAINTEXT, SSL)
advertised.listeners	Address advertised to clients for connecting to the broker	Used when Kafka is behind a NAT or proxy
log.dirs	Directory where Kafka stores log segments	Can be a comma-separated list of directories for better I/O performance
num.partitions	Default number of partitions for new topics	Can be overridden per topic during creation

Example: Broker Configuration

broker.id=0
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kafka-logs
num.partitions=1

Note: The broker ID must be unique across the entire Kafka cluster and is used for leader election and replication.

2. Understanding Topics and Partitions

Concept	Description	Key Details
Topic	A category or feed name to which records are published	Logical identifier for a stream of messages; can have multiple partitions
Partition	A sub-division of a topic that allows for parallelism and scalability	Each partition is an ordered, immutable sequence of records; stored on a broker
Partition Key	Determines which partition a record is sent to within a topic	Hash of the key is used to route records; ensures same key always goes to same partition

Example: Publishing to a Topic with Partitions

const kafka = new Kafka({ brokers: ['localhost:9092'] });
const producer = kafka.producer();

await producer.connect();
await producer.send({
  topic: 'user-events',
  messages: [
    { key: 'user-123', value: JSON.stringify({ action: 'login', timestamp: Date.now() }) },
    { key: 'user-456', value: JSON.stringify({ action: 'logout', timestamp: Date.now() }) }
  ]
});

Note: Kafka uses the partition key to determine which partition a message belongs to. If no key is provided, messages are distributed in round-robin fashion across partitions.

Partitioning Strategy

By default, Kafka distributes messages to partitions using a hash of the message key
If no key is provided, it uses a round-robin algorithm to distribute messages
Partitioning allows Kafka to scale horizontally and support high throughput

3. Understanding Producers and Consumers

Component	Description	Key Features
Producer	Application that sends messages to Kafka topics	Message batching, compression, retries, partitioning
Consumer	Application that reads messages from Kafka topics	Consumer groups, offset management, commit strategies

Example: Producer Configuration

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});

const producer = kafka.producer();

await producer.connect();
await producer.send({
  topic: 'user-events',
  messages: [
    { value: JSON.stringify({ userId: '123', action: 'login' }) }
  ]
});
await producer.disconnect();

Example: Consumer Configuration

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});

const consumer = kafka.consumer({ groupId: 'user-group' });

await consumer.connect();
await consumer.subscribe({ topic: 'user-events' });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log("Received message:", message.value.toString());
  }
});

Note: Producers and consumers are designed to be highly scalable and fault-tolerant, with built-in retry mechanisms and automatic failover capabilities.

4. Understanding Consumer Groups

Property	Description	Notes
group.id	Unique identifier for the consumer group	Required field; determines which consumers belong to the same group
session.timeout.ms	Timeout for consumer session	If a consumer doesn't send heartbeats within this time, it's considered dead
heartbeat.interval.ms	Interval between heartbeat requests	Defaults to 3 seconds; must be less than session.timeout.ms
max.poll.interval.ms	Maximum time between poll() calls	Prevents consumer from being marked as dead if processing takes too long
auto.offset.reset	What to do when no offset is found	Values: earliest, latest, none. Default is latest

Example: Consumer Group Configuration

{
  "bootstrap.servers": "localhost:9092",
  "group.id": "user-activity-group",
  "session.timeout.ms": 45000,
  "heartbeat.interval.ms": 3000,
  "auto.offset.reset": "earliest"
}

Note: Kafka consumer groups ensure that each message in a topic partition is consumed by only one consumer within the group.

Key Points About Consumer Groups

Consumers in the same group share the load of processing messages
If a consumer leaves the group, its partitions are redistributed among remaining consumers
Each partition can only be consumed by one consumer in a group at any time
Multiple consumer groups can read from the same topic independently

Consumer Groups vs. Independent Consumers

Consumer Groups: Message distribution among group members
Independent Consumers: Each consumer processes all messages in topic

5. Understanding Replication

Replica Type	Description	Role in Kafka
Leader Replica	Primary replica that handles all read and write requests for a partition.	Acts as the main source of data for a partition.
Follower Replica	Replica that replicates data from the leader and can become leader if needed.	Provides fault tolerance and ensures availability.
Offline Replica	Replica that is not currently part of the ISR (In-Sync Replicas) set.	Replica is not in sync with the leader and may be out of date.

Example: Replication Configuration

{"replication.factor": 3,"min.insync.replicas": 2}

Note: The min.insync.replicas setting ensures that a write is acknowledged by at least this many replicas before being considered successful.

Warning: If min.insync.replicas is set to a value higher than the replication factor, writes will block until enough replicas are available.

Configuration Parameter	Default Value	Description
replication.factor	1	Number of replicas for each partition.
min.insync.replicas	1	Minimum number of replicas that must acknowledge a write.
unclean.leader.election.enable	false	Allows follower replicas to become leaders even if they are not in sync.

6. Understanding Log Segments

Property	Description	Notes
segment.size	Maximum size of a log segment file	Default: 1GB. When reached, a new segment is created.
segment.ms	Maximum age of a log segment before rotation	Default: 7 days. Ensures segments are rotated based on time.
segment.jitter.ms	Random jitter to avoid segment rotation storms	Default: 0ms. Helps distribute load during rotation.
log.retention.hours	Retention period for log segments (in hours)	Default: 168 hours (7 days). Older segments are deleted.

Example: Log Segment Configuration

{
  "segment.size": "1073741824",
  "segment.ms": "604800000",
  "log.retention.hours": "168"
}

Note: Log segments are stored in kafka-logs directory by default. Each partition has its own set of log segments.

7. Understanding Offsets

Offset Type	Description	Usage	Notes
Beginning Offset	First offset available in a partition	Used to read all messages from the start	Can be fetched using `seekToBeginning()`
End Offset	Last offset available in a partition	Used to read all messages from the end	Can be fetched using `seekToEnd()`
Committed Offset	Offset committed by a consumer group	Used to resume consumption from last committed point	Stored in Kafka's internal topic `__consumer_offsets`
Log End Offset	Offset of the last message in a partition	Used to determine how much data is available	Represents the highest offset in a partition

Example: Fetching Offsets

import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
TopicPartition partition = new TopicPartition("my-topic", 0);

// Get beginning offset
long beginningOffset = consumer.beginningOffsets(Collections.singletonList(partition)).get(partition);

// Get end offset
long endOffset = consumer.endOffsets(Collections.singletonList(partition)).get(partition);

// Get committed offset
long committedOffset = consumer.committed(partition).offset();

Note: Offsets are unique per partition and are monotonically increasing. They are crucial for maintaining consumer state and ensuring message delivery guarantees.

8. Understanding Message Flow

Component	Description	Role in Message Flow
Producer	Application that sends messages to Kafka topics	Initiates message flow by publishing records to a topic
Topic	Category or feed name to which messages are published	Serves as the channel for message distribution
Partition	Divided segment of a topic that allows parallelism and scalability	Messages are distributed across partitions based on a partitioning strategy
Broker	Individual Kafka server instance in a cluster	Stores and manages messages, handles client requests, coordinates replication
Consumer	Application that reads messages from Kafka topics	Consumes messages from partitions in a topic, processes them
Consumer Group	Collection of consumers that work together to consume messages from a topic	Ensures each message is consumed by only one member of the group

Example: Producer to Consumer Flow

// Producer sends a message to a topic
const producer = new KafkaProducer({
  brokers: ['localhost:9092'],
  clientId: 'my-app'
});

await producer.connect();
await producer.send({
  topic: 'user-events',
  messages: [
    { key: 'user-123', value: JSON.stringify({ event: 'login', timestamp: Date.now() }) }
  ]
});

// Consumer reads from the topic
const consumer = new KafkaConsumer({
  brokers: ['localhost:9092'],
  clientId: 'my-app-consumer',
  groupId: 'user-events-group'
});

await consumer.connect();
await consumer.subscribe({ topic: 'user-events' });

consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log('Received:', message.value.toString());
  }
});

Message Flow Steps

Producer creates and sends a message to a Kafka topic
Kafka broker receives the message and appends it to the appropriate partition
Consumer group subscribes to the topic and reads messages from partitions
Each consumer in a group consumes messages from different partitions to ensure parallelism

Note: Messages are stored in Kafka with a timestamp and can be consumed multiple times by different consumer groups.

9. Understanding ZooKeeper Role

Aspect	Description	Notes
ZooKeeper as Metadata Store	`Broker metadata, topic configuration, consumer group state`	Centralized storage for Kafka cluster metadata
ZooKeeper as Coordination Service	`Leader election, broker registration, partition assignment`	Manages cluster coordination and state changes
ZooKeeper as Configuration Store	`Cluster-wide configuration settings`	Stores and distributes Kafka configurations across the cluster
ZooKeeper Session Management	`Broker session timeouts, client reconnection logic`	Ensures fault tolerance and session consistency

Example: ZooKeeper Ensemble Configuration

server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

Note: ZooKeeper is not part of Kafka's core architecture in newer versions (e.g., Kafka 3.0+), but still required for older deployments.

10. Understanding KRaft Architecture

KRaft (Kafka Raft Metadata) is the modern approach to managing Kafka's metadata, replacing the older ZooKeeper-based architecture. It uses the Raft consensus algorithm to handle metadata operations in a more scalable and simpler way.

Component	Description	Status
Controller	Manages cluster state, topic/partition assignments, and leader elections.	NEW
Metadata Quorum	Stores metadata in a Raft log, managed by the Controller.	NEW
Kafka Brokers	Handle data storage and client requests, coordinating with the metadata quorum.	NEW

Example: KRaft Configuration

# Enable KRaft mode
process.roles=broker
node.id=1

# Metadata storage (in KRaft, this is managed by the broker)
metadata.log.dirs=/tmp/kafka-logs

# ZooKeeper is not required in KRaft mode
# zookeeper.connect=localhost:2181

Note: KRaft is the default in Kafka 3.0+, but requires a minimum of 3 brokers to form a quorum.

KRaft vs ZooKeeper Architecture

KRaft: Uses Raft consensus for metadata, no external dependencies.
ZooKeeper: Requires separate ZooKeeper ensemble for metadata management.