Managing Cluster Metadata
1. Understanding Metadata Topics
| Topic | Description | Retention Policy | Partition Count |
|---|---|---|---|
| __consumer_offsets | Stores consumer group offsets and metadata | 14 days | 50 |
| __transaction_state | Manages transaction state for idempotent producers | 14 days | 50 |
| __cluster_metadata | Internal topic storing cluster metadata (Kafka 2.8+) | Unlimited | 1 |
Note: These internal topics are managed by Kafka and should not be modified directly.
Example: Consumer Group Offset Commit
const consumer = new KafkaConsumer({
groupId: 'my-group',
topics: ['my-topic'],
autoCommit: true,
sessionTimeout: 30000
});
consumer.on('commit', (payload) => {
console.log("Committed offsets:", payload);
});2. Viewing Cluster Metadata
| Metadata Type | Description | Command | Notes |
|---|---|---|---|
| Broker Information | Lists all active brokers in the cluster | kafka-broker-api-versions.sh --bootstrap-server localhost:9092 |
Shows version compatibility per broker |
| Topic Metadata | Retrieves metadata for all topics | kafka-topics.sh --list --bootstrap-server localhost:9092 |
Includes topic names, partitions, and replication factors |
| Consumer Group Information | Displays consumer groups and their status | kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list |
Useful for monitoring consumer lag and offsets |
| Partition Assignment | Shows partition distribution across brokers | kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092 |
Displays leader, replicas, and in-sync replicas (ISR) |
Example: Describe a specific topic
kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
Note: The
--describe command provides detailed information about topic configuration, including partition count, replication factor, and leader assignments.3. Describing Cluster Configuration
| Property | Default Value | Description |
|---|---|---|
| broker.id | -1 |
Unique identifier for the broker in the cluster. Must be unique across all brokers. |
| listeners | PLAINTEXT://localhost:9092 |
Comma-separated list of listeners for the broker to listen on. Format: PLAINTEXT://host:port. |
| advertised.listeners | PLAINTEXT://localhost:9092 |
Listeners advertised to clients for connecting to the broker. Useful in containerized environments or when behind NAT. |
| log.dirs | /tmp/kafka-logs |
Comma-separated list of directories where log segments are stored. |
| num.partitions | 1 |
Default number of partitions for topics created without specifying a partition count. |
| default.replication.factor | 1 |
Default replication factor for topics created without specifying a replication factor. |
| min.insync.replicas | 1 |
Minimum number of replicas that must acknowledge a write for it to be considered successful. |
Example: Configuring Broker ID and Listeners
broker.id=0
listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
advertised.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
log.dirs=/var/lib/kafka/data
Note: The
broker.id must be unique across all brokers in the cluster. If set to -1, Kafka will auto-generate an ID.
4. Managing Broker Configuration
| Property | Default Value | Description | Notes |
|---|---|---|---|
| broker.id | -1 | Unique identifier for the broker in the cluster | Must be unique across all brokers in the cluster |
| listeners | PLAINTEXT://localhost:9092 | List of listeners for the broker to accept client connections | Can include multiple protocols (PLAINTEXT, SSL, SASL_SSL) |
| advertised.listeners | Same as listeners | List of listeners for clients to connect to the broker | Useful when Kafka is behind a proxy or NAT |
| log.dirs | /tmp/kafka-logs | Directories where Kafka logs are stored | Can be a comma-separated list of directories |
| num.partitions | 1 | Default number of partitions for new topics | Affects topic scalability and performance |
| default.replication.factor | 1 | Default replication factor for new topics | Must be less than or equal to the number of brokers |
| min.insync.replicas | 1 | Minimum number of replicas that must acknowledge a write | Used to enforce durability guarantees |
Example: Setting Broker Configuration
broker.id=0
listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
advertised.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
log.dirs=/var/lib/kafka/data
num.partitions=12
default.replication.factor=3
min.insync.replicas=2
Note: Broker configuration can be updated dynamically without restarting the broker, but some settings require a restart to take effect.
5. Understanding Controller Role
| Aspect | Description | Details |
|---|---|---|
| Controller Election | Process of selecting a broker to act as the cluster controller | The controller is elected by the Kafka brokers using a ZK-based mechanism or Raft consensus in KRaft mode |
| Controller Responsibilities | Core duties of the controller node | Manages topic and partition creation, handles broker failures, maintains ISR lists, and synchronizes metadata |
| Metadata Synchronization | How metadata is propagated across the cluster | Controller sends metadata updates to all brokers, ensuring consistency in topic/partition states and configurations |
| Failover Handling | Behavior during controller failure | If the current controller fails, a new one is elected from the available brokers to maintain cluster operations |
Example: Controller Election Process
# Simulated Kafka controller election log
[2026-04-05 10:00:00] INFO Controller started on broker 3
[2026-04-05 10:00:01] INFO Controller elected for topic "user_events"
[2026-04-05 10:00:02] INFO ISR list updated for partition 0
Note: In KRaft mode, the controller is managed by the Kafka metadata log rather than ZooKeeper.
6. Monitoring Controller Metrics
| Metric | Description | Unit | Notes |
|---|---|---|---|
| kafka.controller.controller_activation_count | Number of times the controller has been activated | count | Tracks leadership changes in the cluster |
| kafka.controller.offline_partitions_count | Number of partitions currently offline | count | Indicates potential availability issues |
| kafka.controller.active_controller_count | Number of active controllers in the cluster | count | Should be 1 in a healthy cluster |
| kafka.controller.leader_election_rate_and_time_ms | Rate and average time of leader elections | ms | High values may indicate instability |
| kafka.controller.partition_count | Total number of partitions in the cluster | count | Useful for capacity planning |
| kafka.controller.replica_count | Total number of replicas in the cluster | count | Includes both online and offline replicas |
Example: Collecting Controller Metrics with JMX
echo "kafka.controller.controller_activation_count" | jconsole
Note: Controller metrics are essential for understanding the health and stability of your Kafka cluster's metadata management.
7. Handling Metadata Updates
| Property | Description | Default Value | Notes |
|---|---|---|---|
| metadata.max.age.ms | Maximum age of metadata before it is refreshed | 300000 |
Controls how often the client refreshes metadata from the broker |
| metadata.fetch.timeout.ms | Timeout for metadata fetch requests | 60000 |
Used when fetching metadata from brokers |
| metadata.max.downgrades.ms | How long to wait before downgrading metadata | 30000 |
Applies to metadata version downgrade logic |
Example: Configuring Metadata Refresh
{
"metadata.max.age.ms": 60000,
"metadata.fetch.timeout.ms": 30000
}
Note: Kafka clients automatically handle metadata updates in the background. Applications should not manually refresh metadata unless dealing with specific edge cases.
Warning: Setting
metadata.max.age.ms too low can lead to excessive metadata fetches, increasing network overhead and potentially causing broker throttling.
8. Configuring Metadata Max Age
| Property | Default Value | Description |
|---|---|---|
| metadata.max.age.ms | 300000 |
Maximum age of metadata before refreshing from the cluster. |
Note: Lower values increase metadata freshness but may increase network overhead.
9. Understanding Partition Metadata
| Property | Description | Values |
|---|---|---|
| partition.id | Unique identifier for the partition within the topic | 0, 1, etc. |
| leader | Broker ID of the leader replica for this partition | 0, 1, etc. |
| replicas | List of broker IDs that are replicas of this partition | [0, 1, 2] |
| isr | List of broker IDs in the In-Sync Replicas set | [0, 1] |
| offline.replicas | List of replicas that are currently offline | [2] |
Example: Partition Metadata in Topic Description
kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092
Note: The leader is responsible for all read and write operations for the partition.
Warning: If the leader replica is unavailable, Kafka elects a new leader from the ISR set.
10. Debugging Metadata Issues
| Issue | Description | Solution |
|---|---|---|
| Unreachable Brokers | Metadata cannot be fetched due to broker unavailability. | Check network connectivity and broker status. |
| Stale Metadata Cache | Client holds outdated metadata, causing misrouting. | Restart client or force metadata refresh via metadata.max.age.ms. |
| ZooKeeper Misconfiguration | Metadata stored in ZooKeeper is inconsistent or corrupted. | Verify ZK connectivity and sync metadata manually if needed. |
Note: Use
kafka-topics.sh with --describe to inspect topic metadata and identify inconsistencies.