Configuring Producer Settings

1. Setting Batch Size

Property	Default	Description
batch.size	`16384`	Maximum size of a batch in bytes. Larger batches are more efficient but increase latency.
enable.idempotence	`false`	Enables idempotent producer to ensure no duplicates. Requires `batch.size` <= 1MB.

Example: Configuring batch size

{
  "bootstrap.servers": "localhost:9092",
  "batch.size": 32768,
  "enable.idempotence": true
}

2. Configuring Linger Time

Property	Default Value	Description
linger.ms	`0`	Time in milliseconds the producer will wait to batch records before sending them. Increasing this value can improve throughput by batching more messages.

Example: Setting linger.ms to 5ms

{
  linger: 5
}

Note: Setting linger.ms to a higher value can increase throughput but may introduce latency.

3. Setting Buffer Memory

Property	Default Value	Description
buffer.memory	`33554432` (32 MB)	Maximum amount of memory in bytes that the producer can use to buffer records waiting to be sent to the broker.

Example: Configuring Buffer Memory

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("buffer.memory", "67108864"); // 64 MB

Note: If the producer runs out of buffer memory and cannot send records, it will block until space becomes available or the max.block.ms timeout is reached.

4. Configuring Compression

Property	Default Value	Description
compression.type	`none`	Compression type to use for messages. Valid values are `none`, `gzip`, `snappy`, `lz4`, and `zstd`.
compression.level	`0`	Compression level for `zstd` compression. Valid values are 1-22, where higher numbers mean better compression but slower performance.

Example: Configuring GZIP Compression

{
  "compression.type": "gzip",
  "batch.size": 10240,
  "linger.ms": 5
}

Note: Enabling compression can significantly reduce network bandwidth usage but may increase CPU overhead. gzip offers a good balance between compression ratio and performance.

5. Setting Max Request Size

Property	Default Value	Description
max.request.size	`1048576`	Maximum size of a request in bytes. This setting ensures that the producer does not send requests larger than this limit.

Example: Configuring max.request.size

{
  "max.request.size": 10485760
}

6. Setting Request Timeout

Property	Default Value	Description
request.timeout.ms	`30000`	Timeout for request to the Kafka broker. If a request exceeds this time, it will be retried or fail.
max.block.ms	`60000`	Maximum time to block while waiting for a producer request to complete. If exceeded, a `TimeoutException` is thrown.

Example: Configuring Request Timeout

{
  "request.timeout.ms": 30000,
  "max.block.ms": 60000
}

7. Configuring Delivery Timeout

Property	Default Value	Description
delivery.timeout.ms	`120000`	Maximum time to await delivery confirmation before considering the request failed. This includes retries.
max.block.ms	`60000`	Maximum time to block on send operations. If the timeout is exceeded, a `TimeoutException` is thrown.

Example: Setting Delivery Timeout

{
  "delivery.timeout.ms": 30000,
  "max.block.ms": 5000
}

Note: The delivery.timeout.ms setting should be higher than the request.timeout.ms to allow for retries and network latency.

8. Configuring Retries

Property	Default	Description
retries	`2147483647`	Number of retries for failed sends. Set to `0` to disable retries.
retry.backoff.ms	`100`	Backoff time in milliseconds between retries.
retry.backoff.max.ms	`1000`	Maximum backoff time in milliseconds between retries.
retry.jitter	`0.2`	Jitter factor to randomize retry backoff times (0.0 to 1.0).

Example: Producer with Retry Configuration

const producer = kafka.producer({
  retries: 5,
  retryBackoffMs: 200,
  retryBackoffMaxMs: 5000,
  retryJitter: 0.3
});

Note: Retries are applied only to transient errors like network issues or broker unavailability. Permanent errors (e.g., invalid message format) will not be retried.

9. Setting Idempotence

Property	Values	Description
enable.idempotence	`false` \| `true`	Enables idempotent producer behavior to ensure exactly-once delivery semantics.
transactional.id	`string`	Unique identifier for the transactional producer. Required when `enable.idempotence` is true.

Example: Idempotent Producer Configuration

{
  "bootstrap.servers": "localhost:9092",
  "enable.idempotence": true,
  "transactional.id": "producer-1"
}

Note: Idempotence requires enable.idempotence to be set to true and max.in.flight.requests.per.connection to be less than or equal to 5.

10. Configuring Max In-Flight Requests

Property	Default	Description
max.in.flight.requests.per.connection	`5`	Maximum number of unacknowledged requests the client will send on a single connection before blocking.

Example: Setting max.in.flight.requests.per.connection

{
  "max.in.flight.requests.per.connection": 10
}

Note: Setting this value to 1 ensures in-order delivery but reduces throughput. Higher values increase throughput but may lead to out-of-order messages.