Installing and Configuring Kafka
1. Downloading Kafka
| Version | Download URL | Notes |
|---|---|---|
| 3.7.0 | kafka_2.13-3.7.0.tgz | Latest stable release as of 2026 |
| 3.6.1 | kafka_2.13-3.6.1.tgz | Previous stable release |
Example: Download and Extract Kafka
$ wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
$ tar -xzf kafka_2.13-3.7.0.tgz
$ cd kafka_2.13-3.7.0Note: Kafka requires Java 17 or later to run.
2. Setting Up Java Requirements
| Requirement | Version | Notes |
|---|---|---|
| Java Runtime Environment (JRE) | Java 11 |
Minimum required version for Kafka 3.5+ |
| Java Development Kit (JDK) | Java 11 |
Recommended for development and testing |
Note: Kafka is built on Java and requires a compatible JRE or JDK to run. Ensure that the
java command is available in your system's PATH.Example: Verify Java Installation
$ java -version
openjdk version "11.0.20" 2023-07-18
OpenJDK Runtime Environment (build 11.0.20+8-jvmci-23.1-b05)
OpenJDK 64-Bit Server VM (build 11.0.20+8-jvmci-23.1-b05, mixed mode)3. Configuring Broker Properties
| Property | Default Value | Description |
|---|---|---|
| broker.id | -1 | Unique identifier for the broker. Must be unique across all brokers in the cluster. |
| listeners | PLAINTEXT://localhost:9092 | List of listeners for the broker to accept client connections. Format: PLAINTEXT://host:port |
| advertised.listeners | Same as listeners |
Address that will be advertised to clients. Useful when Kafka is behind a NAT or proxy. |
| log.dirs | /tmp/kafka-logs | Comma-separated list of directories where log segments are stored. |
| num.partitions | 1 | Default number of partitions for new topics. |
| default.replication.factor | 1 | Default replication factor for new topics. |
| log.retention.hours | 168 | Number of hours to retain log segments before deletion. |
| log.segment.bytes | 1073741824 | Maximum size of a log segment before rolling to a new one. |
Example: Setting up broker configuration
# Kafka broker configuration
broker.id=0
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092
log.dirs=/var/lib/kafka/data
num.partitions=3
default.replication.factor=2
log.retention.hours=1684. Setting Broker ID
| Property | Default Value | Description |
|---|---|---|
| broker.id | -1 |
Unique identifier for the Kafka broker. Must be unique within the cluster. |
Note: The broker ID must be a positive integer and cannot be changed after the broker has been initialized.
5. Configuring Log Directories
| Property | Default Value | Description |
|---|---|---|
| log.dirs | /tmp/kafka-logs |
Comma-separated list of directories for log segments to be stored. |
| log.retention.hours | 168 |
Number of hours to retain log files before deletion (default 7 days). |
| log.retention.bytes | -1 |
Maximum size of logs before deletion (default unlimited). |
| log.segment.bytes | 1073741824 |
Maximum size of a single log segment file (default 1GB). |
| log.roll.hours | 168 |
Interval in hours to roll log segments (default 7 days). |
Note: Ensure that each directory in
log.dirs has sufficient space and is mounted with appropriate I/O performance for optimal Kafka throughput.
6. Configuring Listeners
| Property | Default Value | Description |
|---|---|---|
| listeners | PLAINTEXT://localhost:9092 |
Comma-separated list of listeners to accept client connections, in the form listener.name://host:port. |
| advertised.listeners | PLAINTEXT://localhost:9092 |
Listeners to use for advertising the broker to clients. If not set, listeners is used. |
| listener.security.protocol.map | PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL |
Maps listener names to security protocols (e.g., PLAINTEXT, SSL, SASL_SSL). |
Example: Configuring SSL Listeners
listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
advertised.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit
Note: Ensure that the
advertised.listeners are accessible from client machines, especially in containerized or cloud environments.
7. Setting Up Cluster ID
Kafka uses a unique identifier to distinguish clusters in a distributed environment. The cluster ID is automatically generated when a Kafka broker is started for the first time.
| Configuration Property | Default Value | Description |
|---|---|---|
| cluster.id | Generated automatically | Unique identifier for the Kafka cluster |
Note: The cluster ID must be unique across all Kafka clusters in your ecosystem to prevent data corruption or misrouting of messages.
8. Starting ZooKeeper Server
Kafka uses Apache ZooKeeper for managing and coordinating its brokers. Before starting Kafka, you must have a running ZooKeeper instance.
| Configuration Property | Default Value | Description |
|---|---|---|
| zookeeper.connect | localhost:2181 |
Comma-separated list of ZooKeeper servers to connect to |
| zookeeper.timeout.ms | 6000 |
ZooKeeper session timeout in milliseconds |
Note: Ensure that the
zookeeper.properties file is correctly configured with the appropriate data directory and port settings before starting the server.
Warning: Kafka requires ZooKeeper to be running before it can start. If ZooKeeper is not available, Kafka will fail to start.
9. Starting Kafka Broker
| Property | Default Value | Description |
|---|---|---|
| broker.id | -1 |
Unique identifier for the broker in the cluster |
| listeners | PLAINTEXT://localhost:9092 |
Network listeners for the broker to accept connections |
| advertised.listeners | PLAINTEXT://localhost:9092 |
Address advertised to clients for connecting to the broker |
| log.dirs | /tmp/kafka-logs |
Directories where Kafka logs are stored |
| zookeeper.connect | localhost:2181 |
Connection string to the ZooKeeper ensemble |
Note: Ensure that the
zookeeper.connect property points to a running ZooKeeper instance.
Warning: The
broker.id must be unique within the Kafka cluster.
Startup Process Overview
- Kafka broker reads configuration from
server.properties - Binds to listeners specified in
listeners - Connects to ZooKeeper for cluster coordination
- Begins accepting consumer and producer requests
10. Using KRaft Mode
KRaft (Kafka Raft Metadata) mode is the modern way to run Kafka without requiring a separate ZooKeeper ensemble. It uses the Raft consensus protocol for managing metadata.
| Configuration Property | Default Value | Description |
|---|---|---|
| process.roles | broker |
Set to broker or controller in KRaft mode. A broker can be a controller too. |
| node.id | 0 |
Unique identifier for the broker or controller node. |
| controller.quorum.voters | |
List of controller voters in the format 1@host:port. |
| metadata.log.dir | /tmp/kafka-logs |
Directory where metadata logs are stored. |
| metadata.log.segments | 1000 |
Number of segments to keep before rolling over. |
Example: Minimal KRaft Configuration
node.id=1
process.roles=broker
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kafka-logs
Note: In KRaft mode, the
zookeeper.connect property is not used.
Warning: KRaft mode is still experimental in older Kafka versions. Ensure you're using Kafka 3.0+ for stable KRaft support.
| Feature | KRaft Support | Notes |
|---|---|---|
| Controller Election | Yes |
Uses Raft consensus for leader election. |
| ZooKeeper | No |
Not required in KRaft mode. |
| Metadata Storage | Yes |
Stores metadata in local log segments. |
11. Verifying Installation
| Command | Description | Output |
|---|---|---|
| kafka-version | Displays the version of Kafka installed | 2.8.0 |
| kafka-topics.sh --list --bootstrap-server localhost:9092 | List all topics in the Kafka cluster | Empty list or topic names |
Note: Ensure that the
KAFKA_HOME environment variable is set correctly.Warning: If
kafka-version returns an error, Kafka is not properly installed or the PATH is not configured.