Configuring Broker Settings

1. Setting Broker ID

Property	Default	Description
broker.id	`-1`	Unique identifier for the broker within the cluster. Must be unique across all brokers in the cluster.

Example: Setting Broker ID in server.properties

broker.id=1

Note: If broker.id is set to -1, Kafka will auto-generate a unique ID for the broker.

2. Configuring Listeners

Property	Default Value	Description
listeners	`PLAINTEXT://localhost:9092`	Comma-separated list of listeners for the broker to listen on. Format: `<listener_name>://<host>:<port>`
advertised.listeners	`NULL`	Listeners to advertise to clients. If not set, uses the value of `listeners`.
listener.security.protocol.map	`PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL`	Maps listener names to security protocols. Format: `<listener_name>:<security_protocol>`

Example: Configuring Multiple Listeners

listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
advertised.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL

Example: Using SASL Authentication

listeners=SASL_PLAINTEXT://localhost:9092
advertised.listeners=SASL_PLAINTEXT://localhost:9092
listener.security.protocol.map=SASL_PLAINTEXT:SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN

3. Setting Log Directories

Property	Default Value	Description
log.dirs	`/tmp/kafka-logs`	Comma-separated list of directories under which to store log files

Example: Configuring Log Directories

log.dirs=/var/lib/kafka/logs,/mnt/kafka-logs

Note: Ensure that the directories specified in log.dirs are writable and have sufficient disk space for log retention.

4. Configuring Number of Threads

Property	Default Value	Description
num.io.threads	`8`	Number of I/O threads to use for handling network requests.
num.network.threads	`3`	Number of threads used for handling network requests.
num.replica.fetchers	`1`	Number of fetcher threads used for replicating data from leaders.
num.validator.fetchers	`1`	Number of fetcher threads used for validating messages.

Example: Setting I/O Threads

num.io.threads=12

Note: Increasing the number of I/O threads can improve throughput but may also increase memory usage and CPU overhead.

5. Setting Socket Buffer Sizes

Property	Default Value	Description
socket.send.buffer.bytes	`1024 * 1024`	Size of the TCP send buffer for sockets in bytes
socket.receive.buffer.bytes	`1024 * 1024`	Size of the TCP receive buffer for sockets in bytes

Example: Configuring Socket Buffer Sizes

socket.send.buffer.bytes=2097152
socket.receive.buffer.bytes=2097152

6. Configuring Max Message Size

Property	Default Value	Description
message.max.bytes	`1048588`	Maximum size of a message that can be sent to the broker. This is the maximum size of a single message in bytes.
replica.lag.time.max.ms	`30000`	Maximum time a replica can lag behind the leader before it is considered out of sync. This setting affects how quickly a replica is removed from the ISR (In-Sync Replicas) list.
log.message.max.bytes	`1048588`	Maximum size of a message that can be stored in the log. This setting ensures that messages do not exceed the maximum size allowed by Kafka's storage layer.

Example: Setting max message size in server.properties

message.max.bytes=10485760
log.message.max.bytes=10485760

Note: Ensure that message.max.bytes is set to a value greater than or equal to log.message.max.bytes. If not, Kafka will throw an error during startup.

7. Setting Default Replication Factor

Property	Default Value	Description
default.replication.factor	`1`	Default replication factor for auto-created topics

Example: Setting Default Replication Factor

{"default.replication.factor": 3}

Note: The replication factor must be less than or equal to the number of brokers in the cluster.

8. Configuring Min ISR

Property	Default Value	Description
min.insync.replicas	2	Minimum number of replicas that must acknowledge a write for the write to be considered successful. Ensures data durability.

Example: Setting min.insync.replicas

{"min.insync.replicas": 3}

Note: Setting min.insync.replicas to a value higher than the number of in-sync replicas will cause writes to be rejected until enough replicas are available.

9. Setting Unclean Leader Election

Property	Default Value	Description
unclean.leader.election.enable	`false`	Enables or disables unclean leader election. When set to `true`, a follower broker can be elected as the new leader even if it is not in sync with the current leader.

Note: Enabling unclean leader election can lead to data loss if the elected leader is not fully synchronized with the previous leader.

Example: Enabling Unclean Leader Election

{
  "unclean.leader.election.enable": true
}

10. Configuring Log Retention

Property	Default Value	Description
log.retention.hours	`168`	Retention period for log segments in hours (default 7 days)
log.retention.bytes	`-1`	Maximum size of the log before it is eligible for deletion in bytes (-1 = unlimited)
log.retention.ms	`-1`	Retention period in milliseconds (-1 = use log.retention.hours)
log.roll.hours	`168`	Interval in hours to roll log segments (default 7 days)
log.roll.ms	`-1`	Interval in milliseconds to roll log segments (-1 = use log.roll.hours)

Example: Setting Log Retention

# Retain logs for 30 days
log.retention.hours=720

# Retain up to 10GB of logs
log.retention.bytes=10737418240

# Roll log segments every 24 hours
log.roll.hours=24

Note: When both log.retention.hours and log.retention.bytes are set, Kafka will delete log segments when either condition is met.

11. Setting Log Segment Size

Property	Default Value	Description
log.segment.bytes	`1073741824`	Maximum size of a single log segment in bytes. When this limit is reached, Kafka rolls over to a new log segment.
log.roll.hours	`168`	Interval in hours to roll log segments regardless of size. This ensures logs are rotated even if they haven't reached the byte limit.
log.roll.ms	`604800000`	Interval in milliseconds to roll log segments. Overrides `log.roll.hours` if set.

Example: Configuring Log Segment Size

{
  "log.segment.bytes": 1073741824,
  "log.roll.hours": 168
}

Note: Setting log.segment.bytes too low can lead to excessive segment creation and increased metadata overhead. Setting it too high may cause longer recovery times in case of broker failures.

12. Configuring Log Cleanup

Property	Default Value	Description
log.cleaner.enable	`false`	Enables the log cleaner for compacted topics.
log.cleaner.threads	`1`	Number of threads to use for log cleaning.
log.cleaner.min.retention.ms	`1000 * 60 * 60` (1 hour)	Minimum time a message can be retained before being eligible for deletion.
log.cleaner.max.compaction.lag.ms	`9223372036854775807` (Long.MAX_VALUE)	Maximum time a message can be retained before compaction is forced.
log.cleaner.io.buffer.size	`512 * 1024`	Size of the I/O buffer used for cleaning logs.
log.cleaner.io.max.bytes.per.second	`1.7976931348623157E308` (Unlimited)	Maximum I/O bytes per second for log cleaning.

Example: Enabling Log Cleaning

{"log.cleaner.enable": true,"log.cleaner.threads": 2}

Note: Log cleaning is only effective for topics with cleanup.policy set to compact.

Warning: Setting log.cleaner.max.compaction.lag.ms too low can lead to excessive compaction and performance degradation.