Optimizing Broker Performance
1. Tuning Number of Partitions
| Parameter | Default Value | Description | Notes |
|---|---|---|---|
| num.partitions | 1 | Default number of partitions for new topics | Can be overridden per topic |
| max.partition.fetch.bytes | 1048576 (1MB) | Maximum bytes a fetch request can return | Higher values may increase latency |
| replication.factor | 1 | Default replication factor for new topics | Higher values increase durability but reduce performance |
Example: Setting Partition Count for a Topic
kafka-topics.sh --create --topic my-topic --partitions 12 --replication-factor 3 --bootstrap-server localhost:9092
Note: Increasing partitions improves parallelism but also increases memory usage and metadata overhead.
Warning: Reducing the number of partitions after topic creation is not allowed.
Best Practices for Partition Tuning
- Start with a reasonable number (e.g., 12) based on expected throughput
- Monitor consumer lag and adjust if needed
- Avoid too many small partitions as they increase metadata overhead
- Ensure partition count aligns with consumer group size for optimal parallelism
2. Optimizing Replica Fetcher Threads
| Property | Default Value | Description |
|---|---|---|
| replica.lag.time.max.ms | 30000 |
Maximum time a replica may lag behind the leader before being considered out of sync. |
| replica.fetch.max.bytes | 1048576 |
Maximum bytes a follower can fetch in one request. |
| replica.fetch.wait.max.ms | 500 |
Maximum time a follower will wait for a fetch request before sending an empty response. |
| replica.fetch.min.bytes | 1 |
Minimum bytes a follower must have before responding to a fetch request. |
| replica.fetch.backoff.ms | 1000 |
Backoff time between fetch requests to a follower when it is not ready. |
Example: Configuring Replica Fetcher Threads
{
"replica.lag.time.max.ms": 60000,
"replica.fetch.max.bytes": 2097152,
"replica.fetch.wait.max.ms": 1000
}
Note: Increasing
replica.fetch.max.bytes can improve throughput but may increase memory usage on the follower.
Warning: Setting
replica.lag.time.max.ms too low can cause unnecessary leader reassignments during normal operations.
3. Configuring Network Threads
| Property | Default Value | Description |
|---|---|---|
| num.network.threads | 3 |
Number of threads handling network requests. Increasing this can improve throughput for high-concurrency workloads. |
| num.io.threads | 8 |
Number of threads used for I/O operations. Should typically be set to the number of CPU cores or slightly higher. |
| socket.send.buffer.bytes | 102400 |
Size of the TCP send buffer (SO_SNDBUF) in bytes. Larger buffers can improve throughput for large messages. |
| socket.receive.buffer.bytes | 102400 |
Size of the TCP receive buffer (SO_RCVBUF) in bytes. Larger buffers can help with high-throughput consumers. |
| max.connections.per.ip | 2147483647 |
Maximum number of connections allowed from a single IP address. Set to 100 or lower to prevent resource exhaustion. |
Example: Adjusting Network Thread Configuration
# Increase network threads for high-concurrency environments
num.network.threads=6
# Set I/O threads to match CPU cores (e.g., 8 cores)
num.io.threads=8
# Increase socket buffer sizes for large messages
socket.send.buffer.bytes=262144
socket.receive.buffer.bytes=262144
# Limit connections per IP to prevent abuse
max.connections.per.ip=50
Note: Increasing
num.network.threads beyond the number of CPU cores can lead to context switching overhead and reduced performance.
Warning: Setting
socket.send.buffer.bytes or socket.receive.buffer.bytes too high can cause memory pressure on the broker.
4. Tuning I/O Threads
| Property | Default Value | Description |
|---|---|---|
| num.io.threads | 8 |
Number of I/O threads to use for handling network requests. Increasing this can improve throughput under high load. |
| io.thread.idle.timeout.ms | 500 |
Timeout in milliseconds for idle I/O threads before they are shut down. Lower values reduce memory usage but may increase thread creation overhead. |
| io.thread.max.idle.ms | 60000 |
Maximum idle time in milliseconds before an I/O thread is shut down. This setting helps manage resource usage under low load. |
Example: Configuring I/O Threads
num.io.threads=12
io.thread.idle.timeout.ms=1000
io.thread.max.idle.ms=120000
Note: Increasing
num.io.threads beyond the default can help handle more concurrent requests, but it also increases memory consumption and context switching overhead. Monitor performance after changes to ensure optimal throughput.
5. Optimizing Log Segment Size
| Property | Default Value | Description |
|---|---|---|
| log.segment.bytes | 1073741824 |
Maximum size of a single log segment file in bytes (1 GB) |
| log.roll.hours | 168 |
Time in hours after which a new log segment is created (1 week) |
| log.roll.ms | 604800000 |
Time in milliseconds after which a new log segment is created (1 week) |
Note: Larger segment sizes can improve throughput by reducing the number of files and metadata operations, but may increase recovery time if a segment becomes corrupted.
6. Configuring Zero-Copy Transfer
| Property | Default Value | Description |
|---|---|---|
| replication.zerocopy | false |
Enables zero-copy transfer for replication. Requires replication.enable to be true. |
| replication.max.bytes.per.chunk | 1048576 |
Maximum bytes per chunk for zero-copy transfer. Increasing this can improve throughput. |
Example: Enabling Zero-Copy Transfer
{
"replication.zerocopy": true,
"replication.max.bytes.per.chunk": 2097152
}
Note: Zero-copy transfer is supported only on Linux and requires the
sendfile system call to be available.
Warning: Enabling zero-copy transfer may increase memory usage due to buffering.
7. Tuning Flush Settings
| Property | Default Value | Description |
|---|---|---|
| log.flush.interval.messages | 10000 |
Number of messages to accumulate before flushing logs to disk |
| log.flush.interval.ms | 1000 |
Time in milliseconds to wait before flushing logs to disk |
| log.flush.scheduler.interval.ms | 9223372036854775807 |
Scheduler interval for flush operations (effectively disables scheduler if set to max value) |
Example: Configuring Flush Settings
{
"log.flush.interval.messages": 5000,
"log.flush.interval.ms": 2000
}
Note: Lower flush intervals reduce data loss risk but increase I/O load.
8. Optimizing Page Cache
| Property | Default Value | Description |
|---|---|---|
| log.flush.interval.messages | 10000 |
Number of messages to accumulate before flushing log segments to disk. Lower values increase durability but reduce throughput. |
| log.flush.interval.ms | 900000 |
Time in milliseconds to wait before flushing log segments to disk. Ensures data is persisted even if log.flush.interval.messages is not reached. |
| log.flush.scheduler.interval.ms | 9223372036854775807 |
Interval in milliseconds to schedule log flushes. Set to -1 to disable. |
Example: Configuring Log Flush Settings
{log.flush.interval.messages: 5000,log.flush.interval.ms: 30000}
Note: Adjusting these settings can significantly impact performance and data durability. Lower flush intervals improve durability but may reduce throughput.
9. Setting Socket Buffer Sizes
| Property | Default Value | Description |
|---|---|---|
| socket.send.buffer.bytes | 1024 * 1024 |
Size of the TCP send buffer for sockets in bytes. Increasing this can improve throughput for high-volume producers. |
| socket.receive.buffer.bytes | 1024 * 1024 |
Size of the TCP receive buffer for sockets in bytes. Increasing this can improve throughput for high-volume consumers. |
Example: Configuring Socket Buffer Sizes
{socket.send.buffer.bytes: 2097152,socket.receive.buffer.bytes: 2097152}10. Monitoring Broker Metrics
| Metric | Description | Units | Notes |
|---|---|---|---|
| broker.topic.count | Number of topics on the broker | count | Useful for capacity planning and topic management |
| broker.partition.count | Number of partitions across all topics | count | Indicates the total partition load on the broker |
| broker.network.io.bytes.in | Total bytes received from clients | bytes/sec | High values may indicate high consumer load or network issues |
| broker.network.io.bytes.out | Total bytes sent to clients | bytes/sec | High values may indicate high producer load or network issues |
| broker.request.count | Number of requests handled by the broker | count/sec | Useful for identifying request volume and performance bottlenecks |
| broker.request.time.avg | Average time to process a request | ms | High average times may indicate performance issues or resource constraints |
| broker.producer.message.count | Number of messages produced to the broker | count/sec | Useful for tracking producer throughput and identifying bottlenecks |
| broker.consumer.message.count | Number of messages consumed from the broker | count/sec | Useful for tracking consumer throughput and identifying bottlenecks |
| broker.partition.leader.count | Number of partitions for which this broker is the leader | count | High leader count may indicate load imbalance or broker issues |
| broker.partition.follower.count | Number of partitions for which this broker is a follower | count | Indicates the total follower load on the broker |
Example: Monitoring Kafka Broker Metrics with JMX
bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool \
--jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi \
--object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec \
--attributes Count,OneMinuteRate
Note: Kafka exposes JMX metrics by default, which can be monitored using tools like Prometheus or Grafana for real-time performance tracking.