Skip to main content

Producing Messages

1. Creating Producer Instance

Property Description Default Value Example
bootstrap.servers Comma-separated list of broker addresses localhost:9092 bootstrap.servers=broker1:9092,broker2:9092
key.serializer Serializer for message keys org.apache.kafka.common.serialization.StringSerializer key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer Serializer for message values org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer
acks Number of acknowledgments required 1 acks=all

Example: Creating a Kafka Producer in Java

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

2. Setting Producer Properties

Property Default Value Description
bootstrap.servers localhost:9092 Comma-separated list of broker addresses to connect to
key.serializer org.apache.kafka.common.serialization.StringSerializer Serializer for message keys
value.serializer org.apache.kafka.common.serialization.StringSerializer Serializer for message values
acks 1 Number of acknowledgments required from the leader broker before considering a message sent
retries 2147483647 Number of retries for failed sends (0 = no retries)
batch.size 16384 Size in bytes of the batch to accumulate before sending
linger.ms 0 Time in milliseconds to wait for additional messages before sending a batch
enable.idempotence false Enables idempotent producer behavior to ensure no duplicates

Example: Producer Configuration

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");
props.put("retries", 3);
props.put("enable.idempotence", true);

3. Configuring Bootstrap Servers

Property Default Value Description
bootstrap.servers localhost:9092 Comma-separated list of host/port pairs to use for establishing the initial connection to the Kafka cluster
client.id producer-1 A unique identifier for the client instance
acks 1 The number of acknowledgments the producer requires the leader to receive before considering a request complete

Example: Producer Configuration with Bootstrap Servers

{
  "bootstrap.servers": "broker1:9092,broker2:9092",
  "client.id": "my-producer",
  "acks": "all"
}

4. Configuring Serializers

Serializer Type Class Name Description Example Usage
StringSerializer org.apache.kafka.common.serialization.StringSerializer Serializes string values to bytes. 'key.serializer': 'org.apache.kafka.common.serialization.StringSerializer'
ByteArraySerializer org.apache.kafka.common.serialization.ByteArraySerializer Serializes byte arrays directly. 'value.serializer': 'org.apache.kafka.common.serialization.ByteArraySerializer'
JsonSerializer org.apache.kafka.common.serialization.JsonSerializer Serializes JSON objects to bytes. 'value.serializer': 'org.apache.kafka.common.serialization.JsonSerializer'
AvroSerializer io.confluent.kafka.serializers.KafkaAvroSerializer Serializes Avro objects to bytes using schema registry. 'value.serializer': 'io.confluent.kafka.serializers.KafkaAvroSerializer'

Example: Using JsonSerializer

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.JsonSerializer");

KafkaProducer<String, MyObject> producer = new KafkaProducer<>(props);

5. Sending Messages

Property Value Description
topic my-topic Name of the Kafka topic to send the message to
key user-123 Partitioning key for the message
value {"name": "John", "age": 30} Message payload (JSON string or binary)

Example: Sending a JSON Message

const { Kafka } = require("kafkajs");

const kafka = new Kafka({
  clientId: "my-app",
  brokers: ["localhost:9092"]
});

const producer = kafka.producer();

async function sendMessage() {
  await producer.connect();
  await producer.send({
    topic: "my-topic",
    messages: [
      {
        key: "user-123",
        value: JSON.stringify({ name: "John", age: 30 })
      }
    ]
  });
  await producer.disconnect();
}

sendMessage().catch(console.error);

6. Using Synchronous Send

When producing messages synchronously in Kafka, the producer blocks until the message is acknowledged by the broker or an error occurs.

Method Description Return Type
send() Sends a message synchronously and waits for acknowledgment RecordMetadata

Example: Synchronous Send

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "value1");
try {
    RecordMetadata metadata = producer.send(record).get();
    System.out.println("Message sent to partition " + metadata.partition() + " with offset " + metadata.offset());
} catch (Exception e) {
    System.err.println("Error sending message: " + e.getMessage());
}
producer.close();
Note: The .get() call makes the operation synchronous, blocking until either success or failure.

7. Using Asynchronous Send

Asynchronous message sending in Kafka allows producers to send messages without blocking the calling thread, improving performance and throughput.

Property Description Default Value Notes
enable.idempotence Enables idempotent producer behavior to ensure message ordering and avoid duplicates. false Required for exactly-once semantics when used with transactions.
max.in.flight.requests.per.connection Maximum number of unacknowledged requests the client will send on a single connection. 5 Set to 1 for strict ordering; higher values improve throughput.
retries Number of retries for failed messages before giving up. 2147483647 Set to 0 to disable retries.
acks Controls how many acknowledgments the leader broker must receive before considering a request complete. 1 '0' = no ack, '1' = leader only, 'all' = all replicas.

Example: Asynchronous Send with Callback

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("my-topic", "key1", "value1"), (metadata, exception) -> {
    if (exception != null) {
        System.err.println("Error sending message: " + exception.getMessage());
    } else {
        System.out.println("Message sent to partition " + metadata.partition() + " with offset " + metadata.offset());
    }
});
producer.flush();
Note: The asynchronous send does not block the calling thread, but it still requires handling potential exceptions via callbacks.
Warning: If retries are enabled and enable.idempotence is false, duplicate messages may occur during network failures or broker restarts.

Key Points

  • Asynchronous sends improve throughput by allowing multiple messages to be sent concurrently.
  • Use callbacks to handle success and error cases for each message.
  • Idempotent producers ensure that messages are not duplicated even if retries occur.

8. Setting Message Key

Property Values Description
key null or byte[] Optional key for message routing. If set, messages with the same key are sent to the same partition.
keySerializer Serializer<K> Serializer used to encode the key. Default is StringSerializer for string keys.

Example: Setting Message Key

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("topic", "key1", "message value"));
producer.close();

9. Specifying Target Partition

Property Values Description
partition integer Explicitly specify the partition number to send the message to. If not set, Kafka will use the partitioner to determine the target partition.
partitioner string Partitioner class to use for determining the target partition. Default is org.apache.kafka.clients.producer.RoundRobinPartitioner.
partitioner.class string Class name of the partitioner to use. Can be set to org.apache.kafka.clients.producer.UniformStickyPartitioner or custom implementations.

Example: Using a specific partition

ProducerRecord<String, String> record = new ProducerRecord<>(
    "my-topic",
    2, // Partition number
    "key-1",
    "message-value"
);
Note: When explicitly specifying a partition, the partitioner property is ignored.

10. Adding Message Headers

Property Description Example
headers Map of key-value pairs to attach to the message
{
  "user-id": "12345",
  "event-type": "purchase"
}

Example: Adding Headers to a Message

const producer = new KafkaProducer({
  brokers: ['localhost:9092'],
  clientId: 'my-app'
});

const message = {
  topic: 'orders',
  messages: [
    {
      value: JSON.stringify({ orderId: '123', amount: 99.99 }),
      headers: {
        'user-id': '12345',
        'event-type': 'purchase'
      }
    }
  ]
};

await producer.send(message);

11. Configuring Acknowledgments

Ack Mode Description Use Case
0 Producer does not wait for acknowledgment from broker. Best performance, highest risk of data loss.
1 Producer waits for leader to acknowledge the write. Good balance of performance and durability.
all Producer waits for full ISR (In-Sync Replicas) to acknowledge. Maximum durability, lowest performance.

Example: Setting Ack Mode in Producer Configuration

{
  "acks": "all",
  "retries": 3,
  "timeout": 30000
}

12. Flushing Producer Buffer

Property Default Value Description
buffer.memory 33554432 Maximum memory the producer can use to buffer records
linger.ms 0 Time to wait before sending a batch of records
max.block.ms 60000 Maximum time to block when the buffer is full

Example: Flushing Producer Buffer

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});

const producer = kafka.producer();

await producer.connect();

// Send messages with flush
await producer.send({
  topic: 'test-topic',
  messages: [
    { value: 'Hello Kafka!' }
  ]
});

// Flush the producer buffer
await producer.flush();

console.log('All messages flushed');

Was this article helpful?