Producing Messages
1. Creating Producer Instance
| Property | Description | Default Value | Example |
|---|---|---|---|
| bootstrap.servers | Comma-separated list of broker addresses | localhost:9092 | bootstrap.servers=broker1:9092,broker2:9092 |
| key.serializer | Serializer for message keys | org.apache.kafka.common.serialization.StringSerializer | key.serializer=org.apache.kafka.common.serialization.StringSerializer |
| value.serializer | Serializer for message values | org.apache.kafka.common.serialization.StringSerializer | value.serializer=org.apache.kafka.common.serialization.StringSerializer |
| acks | Number of acknowledgments required | 1 | acks=all |
Example: Creating a Kafka Producer in Java
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
2. Setting Producer Properties
| Property | Default Value | Description |
|---|---|---|
| bootstrap.servers | localhost:9092 | Comma-separated list of broker addresses to connect to |
| key.serializer | org.apache.kafka.common.serialization.StringSerializer | Serializer for message keys |
| value.serializer | org.apache.kafka.common.serialization.StringSerializer | Serializer for message values |
| acks | 1 | Number of acknowledgments required from the leader broker before considering a message sent |
| retries | 2147483647 | Number of retries for failed sends (0 = no retries) |
| batch.size | 16384 | Size in bytes of the batch to accumulate before sending |
| linger.ms | 0 | Time in milliseconds to wait for additional messages before sending a batch |
| enable.idempotence | false | Enables idempotent producer behavior to ensure no duplicates |
Example: Producer Configuration
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");
props.put("retries", 3);
props.put("enable.idempotence", true);
3. Configuring Bootstrap Servers
| Property | Default Value | Description |
|---|---|---|
| bootstrap.servers | localhost:9092 |
Comma-separated list of host/port pairs to use for establishing the initial connection to the Kafka cluster |
| client.id | producer-1 |
A unique identifier for the client instance |
| acks | 1 |
The number of acknowledgments the producer requires the leader to receive before considering a request complete |
Example: Producer Configuration with Bootstrap Servers
{
"bootstrap.servers": "broker1:9092,broker2:9092",
"client.id": "my-producer",
"acks": "all"
}
4. Configuring Serializers
| Serializer Type | Class Name | Description | Example Usage |
|---|---|---|---|
| StringSerializer | org.apache.kafka.common.serialization.StringSerializer |
Serializes string values to bytes. | 'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer' |
| ByteArraySerializer | org.apache.kafka.common.serialization.ByteArraySerializer |
Serializes byte arrays directly. | 'value.serializer': 'org.apache.kafka.common.serialization.ByteArraySerializer' |
| JsonSerializer | org.apache.kafka.common.serialization.JsonSerializer |
Serializes JSON objects to bytes. | 'value.serializer': 'org.apache.kafka.common.serialization.JsonSerializer' |
| AvroSerializer | io.confluent.kafka.serializers.KafkaAvroSerializer |
Serializes Avro objects to bytes using schema registry. | 'value.serializer': 'io.confluent.kafka.serializers.KafkaAvroSerializer' |
Example: Using JsonSerializer
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.JsonSerializer");
KafkaProducer<String, MyObject> producer = new KafkaProducer<>(props);
5. Sending Messages
| Property | Value | Description |
|---|---|---|
| topic | my-topic |
Name of the Kafka topic to send the message to |
| key | user-123 |
Partitioning key for the message |
| value | {"name": "John", "age": 30} |
Message payload (JSON string or binary) |
Example: Sending a JSON Message
const { Kafka } = require("kafkajs");
const kafka = new Kafka({
clientId: "my-app",
brokers: ["localhost:9092"]
});
const producer = kafka.producer();
async function sendMessage() {
await producer.connect();
await producer.send({
topic: "my-topic",
messages: [
{
key: "user-123",
value: JSON.stringify({ name: "John", age: 30 })
}
]
});
await producer.disconnect();
}
sendMessage().catch(console.error);
6. Using Synchronous Send
When producing messages synchronously in Kafka, the producer blocks until the message is acknowledged by the broker or an error occurs.
| Method | Description | Return Type |
|---|---|---|
| send() | Sends a message synchronously and waits for acknowledgment | RecordMetadata |
Example: Synchronous Send
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "value1");
try {
RecordMetadata metadata = producer.send(record).get();
System.out.println("Message sent to partition " + metadata.partition() + " with offset " + metadata.offset());
} catch (Exception e) {
System.err.println("Error sending message: " + e.getMessage());
}
producer.close();
Note: The
.get() call makes the operation synchronous, blocking until either success or failure.
7. Using Asynchronous Send
Asynchronous message sending in Kafka allows producers to send messages without blocking the calling thread, improving performance and throughput.
| Property | Description | Default Value | Notes |
|---|---|---|---|
| enable.idempotence | Enables idempotent producer behavior to ensure message ordering and avoid duplicates. | false | Required for exactly-once semantics when used with transactions. |
| max.in.flight.requests.per.connection | Maximum number of unacknowledged requests the client will send on a single connection. | 5 | Set to 1 for strict ordering; higher values improve throughput. |
| retries | Number of retries for failed messages before giving up. | 2147483647 | Set to 0 to disable retries. |
| acks | Controls how many acknowledgments the leader broker must receive before considering a request complete. | 1 | '0' = no ack, '1' = leader only, 'all' = all replicas. |
Example: Asynchronous Send with Callback
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("my-topic", "key1", "value1"), (metadata, exception) -> {
if (exception != null) {
System.err.println("Error sending message: " + exception.getMessage());
} else {
System.out.println("Message sent to partition " + metadata.partition() + " with offset " + metadata.offset());
}
});
producer.flush();
Note: The asynchronous send does not block the calling thread, but it still requires handling potential exceptions via callbacks.
Warning: If retries are enabled and
enable.idempotence is false, duplicate messages may occur during network failures or broker restarts.
Key Points
- Asynchronous sends improve throughput by allowing multiple messages to be sent concurrently.
- Use callbacks to handle success and error cases for each message.
- Idempotent producers ensure that messages are not duplicated even if retries occur.
8. Setting Message Key
| Property | Values | Description |
|---|---|---|
| key | null or byte[] |
Optional key for message routing. If set, messages with the same key are sent to the same partition. |
| keySerializer | Serializer<K> |
Serializer used to encode the key. Default is StringSerializer for string keys. |
Example: Setting Message Key
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("topic", "key1", "message value"));
producer.close();
9. Specifying Target Partition
| Property | Values | Description |
|---|---|---|
| partition | integer |
Explicitly specify the partition number to send the message to. If not set, Kafka will use the partitioner to determine the target partition. |
| partitioner | string |
Partitioner class to use for determining the target partition. Default is org.apache.kafka.clients.producer.RoundRobinPartitioner. |
| partitioner.class | string |
Class name of the partitioner to use. Can be set to org.apache.kafka.clients.producer.UniformStickyPartitioner or custom implementations. |
Example: Using a specific partition
ProducerRecord<String, String> record = new ProducerRecord<>(
"my-topic",
2, // Partition number
"key-1",
"message-value"
);
Note: When explicitly specifying a partition, the
partitioner property is ignored.10. Adding Message Headers
| Property | Description | Example |
|---|---|---|
| headers | Map of key-value pairs to attach to the message | |
Example: Adding Headers to a Message
const producer = new KafkaProducer({
brokers: ['localhost:9092'],
clientId: 'my-app'
});
const message = {
topic: 'orders',
messages: [
{
value: JSON.stringify({ orderId: '123', amount: 99.99 }),
headers: {
'user-id': '12345',
'event-type': 'purchase'
}
}
]
};
await producer.send(message);
11. Configuring Acknowledgments
| Ack Mode | Description | Use Case |
|---|---|---|
| 0 | Producer does not wait for acknowledgment from broker. | Best performance, highest risk of data loss. |
| 1 | Producer waits for leader to acknowledge the write. | Good balance of performance and durability. |
| all | Producer waits for full ISR (In-Sync Replicas) to acknowledge. | Maximum durability, lowest performance. |
Example: Setting Ack Mode in Producer Configuration
{
"acks": "all",
"retries": 3,
"timeout": 30000
}
12. Flushing Producer Buffer
| Property | Default Value | Description |
|---|---|---|
| buffer.memory | 33554432 |
Maximum memory the producer can use to buffer records |
| linger.ms | 0 |
Time to wait before sending a batch of records |
| max.block.ms | 60000 |
Maximum time to block when the buffer is full |
Example: Flushing Producer Buffer
const { Kafka } = require('kafkajs');
const kafka = new Kafka({
clientId: 'my-app',
brokers: ['localhost:9092']
});
const producer = kafka.producer();
await producer.connect();
// Send messages with flush
await producer.send({
topic: 'test-topic',
messages: [
{ value: 'Hello Kafka!' }
]
});
// Flush the producer buffer
await producer.flush();
console.log('All messages flushed');