Skip to main content
Mapagam

Capacity Planning and Sizing

1. Estimating Message Throughput

Factor Description Typical Range Notes
Message Size Average size of a single message in bytes 1KB - 10MB Larger messages reduce throughput efficiency
Producer Rate Number of messages produced per second 100 - 1M+ Measured in messages/sec
Partition Count Number of partitions per topic 1 - 1000+ Higher count increases parallelism
Replication Factor Number of replicas per partition 1 - 3 Higher replication increases latency
Network Bandwidth Available bandwidth per broker 100 Mbps - 10 Gbps+ Must be sufficient for message traffic

Example: Throughput Calculation

// Example calculation for message throughput
const avgMessageSize = 1024; // 1KB
const messagesPerSecond = 5000;
const partitions = 10;
const replicationFactor = 3;

// Estimated throughput in bytes/sec
const throughputBytesPerSecond = messagesPerSecond * avgMessageSize;

// Estimated network load per broker (in bytes/sec)
const networkLoadPerBroker = throughputBytesPerSecond * replicationFactor / partitions;

console.log(`Throughput: ${throughputBytesPerSecond} bytes/sec`);
console.log(`Network Load per Broker: ${networkLoadPerBroker} bytes/sec`);
Note: Actual throughput depends on hardware, network configuration, and Kafka settings like compression and batch size.

2. Calculating Partition Count

Factor Description Guidance
Throughput Requirements Expected message rate (msgs/sec) Estimate total messages per second across all topics
Message Size Average size of each message (bytes) Typically 1KB–100KB; affects disk I/O and network bandwidth
Retention Policy How long messages are kept (days) Higher retention increases disk usage and requires more partitions
Consumer Parallelism Number of consumers per topic Each consumer can process one partition; more partitions = more parallelism
Broker Count Number of Kafka brokers in cluster Partitions should be distributed evenly across brokers

Example: Estimating Partition Count

# Example: 100,000 messages/sec, 50KB avg message size
# Retention: 7 days, 3 brokers

# Estimate disk usage per day:
messages_per_day = 100000 * 60 * 60 * 24
disk_usage_per_day = messages_per_day * 50 / 1024 / 1024

# Estimate partitions needed for throughput:
partitions_needed = messages_per_day / 10000 # assuming 10k msg/sec per partition
Note: Kafka performance degrades when a single partition is overwhelmed. Aim for partitions that can handle ~10k messages/sec each under typical load.

Partition Count Best Practices

  • Start with 10–100 partitions per topic for moderate throughput
  • Use fewer partitions if message rate is low (e.g., < 1k msg/sec)
  • Scale up to 1000+ partitions for high-throughput systems
  • Avoid too many partitions (can increase metadata overhead and memory usage)

3. Sizing Broker Hardware

Component Recommendation Notes
CPU 8-16 cores Multi-core processors recommended for handling multiple partitions and threads
Memory 32-64 GB RAM Should accommodate at least 1GB per partition for optimal performance
Storage 1-2 TB SSDs SSDs preferred over HDDs for better I/O performance; local storage recommended
Network 1 Gbps minimum Higher bandwidth (10 Gbps) preferred for high-throughput environments

Example: Broker Memory Configuration

# Kafka server configuration example
heap.memory=32g
direct.memory=16g
Note: Memory allocation should account for JVM overhead and OS cache usage.

Key Considerations

  • Use dedicated hardware or VMs with consistent performance characteristics
  • Ensure sufficient disk I/O to avoid bottlenecks during high throughput
  • Monitor CPU usage and adjust accordingly based on consumer lag

4. Estimating Storage Requirements

Component Storage Type Estimation Method Notes
Log segments Local disk Segment size × number of segments Default segment size is 1GB
Replication factor Multiple copies Storage × replication factor Typical values: 2 or 3
Retention policy Time-based or size-based Max retention time × throughput Configurable via log.retention.hours

Example: Storage Calculation

# Example: 10GB/s throughput, 7-day retention, replication factor 3
# Total storage = 10GB/s * 24h * 7d * 3 replicas
# = 5040 GB or ~5.04 TB
Note: Include buffer for growth and temporary spikes in throughput.

5. Planning Network Bandwidth

Component Bandwidth Requirement Notes
Producer 100 MB/s Typical for high-throughput applications
Consumer 200 MB/s Higher due to potential for multiple consumers
Broker-to-Broker 500 MB/s Replication traffic between brokers

Example: Producer Bandwidth Calculation

const avgMessageSize = 1024; // bytes
const messagesPerSecond = 1000;
const bandwidth = (avgMessageSize * messagesPerSecond) / 1024 / 1024; // MB/s
console.log(`Estimated bandwidth: ${bandwidth} MB/s`);
Note: Network bandwidth should be planned with a 20-30% buffer to accommodate bursts and unexpected traffic spikes.

6. Determining Replication Factor

Factor Description Recommendation
replication.factor Number of replicas for each partition Minimum 3 for production; higher for critical systems
min.insync.replicas Minimum number of replicas that must be in sync Set to replication.factor or less for availability vs consistency trade-off

Example: Setting Replication Factor

kafka-topics.sh --create --topic my-topic --partitions 12 --replication-factor 3 --bootstrap-server localhost:9092
Note: Increasing replication factor beyond 3 provides diminishing returns in most cases.
Warning: Ensure sufficient broker count to support desired replication factor.

7. Sizing Consumer Groups

Parameter Description Recommended Value Notes
max.poll.records Maximum number of records returned in a single poll 500-1000 Higher values can increase throughput but may impact latency
session.timeout.ms Timeout for consumer session 45000 Should be higher than the time to process a batch of records
heartbeat.interval.ms Interval between heartbeat requests 3000 Should be less than session.timeout.ms
fetch.min.bytes Minimum bytes to fetch in a single request 1024 Higher values can reduce network overhead
fetch.max.wait.ms Maximum time to wait for new records 500 Controls how long consumers wait for new messages

Example: Consumer Group Configuration

{
  "bootstrap.servers": "localhost:9092",
  "group.id": "my-consumer-group",
  "enable.auto.commit": true,
  "auto.commit.interval.ms": 1000,
  "max.poll.records": 500,
  "session.timeout.ms": 45000,
  "heartbeat.interval.ms": 3000,
  "fetch.min.bytes": 1024,
  "fetch.max.wait.ms": 500
}

Consumer Group Sizing Guidelines

  • Each consumer in a group should process records at a similar rate
  • Adjust max.poll.records based on processing capacity
  • Ensure session.timeout.ms is sufficient to avoid rebalances
  • Use fetch.min.bytes to reduce network overhead for low-volume topics

8. Planning for Peak Load

Example: Peak Load Calculation

// Calculate peak throughput for Kafka topics
const topicConfig = {
  partitions: 12,
  replicationFactor: 3,
  retentionMs: 604800000, // 7 days
  segmentBytes: 1073741824 // 1GB
};

const peakThroughput = {
  producerRate: 1000000, // messages/sec
  messageSize: 1024, // bytes
  totalBytesPerSecond: 1000000 * 1024,
  totalMBPerSecond: (1000000 * 1024) / (1024 * 1024)
};

console.log(
  `Peak throughput: ${peakThroughput.totalMBPerSecond} MB/sec`
);
Factor Description Impact on Capacity
Producer Rate Number of messages produced per second Higher rates require more brokers and partitions
Message Size Average size of each message in bytes Larger messages increase network and disk I/O requirements
Retention Period How long messages are retained before deletion Longer retention increases storage needs
Replication Factor Number of copies of each partition Higher replication increases storage and network load
Note: For peak load planning, use historical traffic data or stress testing to estimate maximum throughput and latency under load.

Peak Load Planning Steps

  1. Collect historical message rates and sizes from production systems
  2. Estimate peak message rate during high-traffic periods
  3. Calculate required throughput in MB/sec
  4. Determine number of partitions needed to handle peak load
  5. Estimate broker resource requirements (CPU, memory, disk)
  6. Plan for replication overhead and retention storage

Key Metrics for Peak Load

  • Throughput: Messages per second (MB/sec)
  • Partitions: Number of partitions per topic
  • Replication: Factor and impact on storage
  • Retention: Duration and storage implications

9. Configuring Retention Policies

Property Default Value Description
retention.ms -1 Time in milliseconds to retain log segments. -1 means infinite retention.
retention.bytes -1 Maximum size of the log segments in bytes. -1 means no limit.
log.roll.ms 604800000 Interval in milliseconds to roll log segments (default 7 days).
log.roll.hours 168 Alias for log.roll.ms, in hours (default 168 hours = 7 days).

Example: Setting Retention Policies

{"retention.ms": 604800000, "retention.bytes": 1073741824}
Note: Retention policies apply per topic and can be overridden at the topic level.
Warning: Setting retention.ms to 0 will cause logs to be deleted immediately.

10. Monitoring Resource Utilization

Resource Metric Monitoring Tool Threshold
CPU CPU usage percentage OS-level monitoring tools (e.g., top, htop) > 80% for extended periods
Memory Heap usage JVM GC logs, JConsole, VisualVM > 90% of max heap
Disk I/O Read/Write throughput iostat, dstat > 80% of disk bandwidth
Network Bytes in/out per second iftop, nethogs > 90% of network capacity

Example: Kafka Broker CPU Usage Monitoring

top -p <broker_pid>

Example: Memory Usage with JConsole

ManagementFactory.getMemoryMXBean().getHeapMemoryUsage()
Note: Use jstat or jconsole to monitor GC activity and heap usage for Kafka brokers.
Warning: High disk I/O can lead to broker lag and data loss if not addressed.

Was this article helpful?

Related Cheat Sheets