Debugging and Troubleshooting

1. Enabling Debug Logging

Property	Values	Description
log4j.logger.org.apache.kafka	`DEBUG`	Enables debug logging for Kafka client and server components
log4j.logger.org.apache.kafka.clients	`DEBUG`	Enables debug logging for Kafka client operations
log4j.logger.org.apache.kafka.common.network	`DEBUG`	Enables debug logging for network communication

Example: Kafka Server Debug Logging Configuration

# log4j.properties
log4j.rootLogger=INFO, console

# Enable debug logging for Kafka components
log4j.logger.org.apache.kafka=DEBUG
log4j.logger.org.apache.kafka.clients=DEBUG
log4j.logger.org.apache.kafka.common.network=DEBUG

# Console appender
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

Note: Debug logging can significantly impact performance and should only be enabled temporarily for troubleshooting.

2. Using Broker Logs

Example: Log Configuration

# kafka-server-start.sh
log4j.rootLogger=INFO, stdout, file

# Log to console
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

# Log to file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=/var/log/kafka/server.log
log4j.appender.file.MaxFileSize=10MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

Log Level	Description	Use Case
TRACE	Most verbose logging, includes all internal operations	Debugging complex issues or performance bottlenecks
DEBUG	Includes internal state and operation details	Developing or testing new features
INFO	General operational information	Monitoring normal operation
WARN	Warning messages about potential issues	Identifying possible problems before they escalate
ERROR	Error messages about failures	Investigating system crashes or data loss

Note: Kafka uses Log4j 2.x for logging. Ensure the log configuration file is correctly loaded by checking that logs appear in the configured location.

Example: Log Rotation

# Log rotation settings
log4j.appender.file.MaxFileSize=100MB
log4j.appender.file.MaxBackupIndex=5

# Log pattern to include timestamp and thread
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %-5p %c{1}:%L - %m%n

Key Log Categories

Server Logs: General broker activity and errors
Consumer Logs: Consumer group management and offset commits
Producer Logs: Producer message sending and error handling
Controller Logs: Leader election and broker state changes

3. Checking Controller Logs

Example: Controller Log Configuration

# kafka-server-start.sh
log4j.logger.org.apache.kafka.controller=INFO, controller-file
log4j.appender.controller-file=org.apache.log4j.RollingFileAppender
log4j.appender.controller-file.File=/var/log/kafka/controller.log
log4j.appender.controller-file.MaxFileSize=100MB
log4j.appender.controller-file.MaxBackupIndex=10

Log Level	Description	Use Case
INFO	General operational information	Monitor normal controller behavior
DEBUG	Detailed debugging information	Investigate specific controller issues
TRACE	Most granular logging	Deep-dive analysis of controller operations

Note: Controller logs are essential for understanding leadership changes, partition reassignments, and ISR (In-Sync Replica) updates.

Key Log Fields to Monitor

leaderChange - Indicates when a broker becomes leader for a partition
partitionReassignment - Shows ongoing reassignment progress
isrChange - Logs when ISR sets change
controllerState - Tracks the current state of the controller

4. Analyzing State Change Logs

Log Type	Description	Key Fields	Usage
state.change.log	Tracks state changes in Kafka brokers	`broker.id`, `topic.name`, `partition.id`, `state`	Monitor cluster state transitions and failures
controller.log	Logs from the controller node	`event.type`, `timestamp`, `operation`	Understand leadership changes and partition reassignments

Example: Parsing State Change Logs

grep "state.change.log" /var/log/kafka/server.log | tail -n 10

Note: State change logs are critical for diagnosing partition reassignments and leader elections.

Key Points

State change logs capture all state transitions in the Kafka cluster
They are essential for understanding leadership changes and partition movements
Logs can be parsed to detect anomalies or failures in the cluster

5. Using Kafka Dump Log Tool

Example: Dumping a Kafka log segment

kafka-run-class.sh kafka.tools.DumpLogSegments --files /path/to/kafka/logs/topic-1/00000000000000000000.log

Option	Values	Description
--files	`path/to/log/file`	Path to the log file(s) to dump
--deep-iteration	`true` \| `false`	Whether to iterate through all messages in the log
--print-data	`true` \| `false`	Whether to print the actual message data
--index-skip-segment	`number`	Skip index segments up to the given number

Note: The DumpLogSegments tool is useful for inspecting the contents of Kafka log segments and diagnosing issues related to message storage or corruption.

6. Checking Topic Metadata

Property	Description	Example
topic.name	Name of the Kafka topic	`user-activity`
partitions	Number of partitions in the topic	`6`
replication.factor	Replication factor for topic	`3`
retention.ms	Time in milliseconds to retain messages	`604800000`
cleanup.policy	Policy for cleaning up old messages (delete or compact)	`delete`

Example: Describe Topic Metadata

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic user-activity

Example: List All Topics

kafka-topics.sh --bootstrap-server localhost:9092 --list

7. Analyzing Consumer Lag

Property	Description	Default Value	Notes
consumer.lag	Measures the delay between the latest message in a partition and the consumer's current position	0	Indicates how far behind the consumer is in processing messages
consumer.lag.ms	Time in milliseconds that the consumer is lagging behind	0	Used for monitoring and alerting on consumer performance

Example: Consumer Lag Monitoring Script

const { Kafka } = require("kafkajs");

const kafka = new Kafka({
  clientId: "lag-monitor",
  brokers: ["localhost:9092"]
});

const consumer = kafka.consumer({ groupId: "lag-group" });

async function analyzeLag() {
  await consumer.connect();
  await consumer.subscribe({ topic: "test-topic", fromBeginning: false });

  // Fetch consumer lag metrics
  const lag = await consumer.fetchLag();
  console.log("Consumer Lag:", lag);
}

analyzeLag().catch(console.error);

Note: Consumer lag can be an indicator of system bottlenecks or slow processing in consumer groups.

Warning: High consumer lag may lead to data loss or outdated data in real-time systems.

Command	Description	Usage
kafka-consumer-groups.sh	CLI tool to describe consumer groups and their lag	`kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group`
kafka-run-class.sh	Run Kafka tools via command line	`kafka-run-class.sh kafka.tools.ConsumerGroupCommand --bootstrap-server localhost:9092 --group my-group`

8. Using JMX Console

Example: Connecting to Kafka via JMX

# Start Kafka with JMX enabled
bin/kafka-server-start.sh \
  --config server.properties \
  -Dcom.sun.management.jmxremote=true \
  -Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.authenticate=false \
  -Dcom.sun.management.jmxremote.ssl=false

Property	Value	Description
com.sun.management.jmxremote	`true`	Enables JMX remote monitoring
com.sun.management.jmxremote.port	`9999`	JMX port to listen on
com.sun.management.jmxremote.authenticate	`false`	Disable authentication (for dev only)
com.sun.management.jmxremote.ssl	`false`	Disable SSL (for dev only)

Note: For production, always enable authentication and SSL for JMX connections.

Example: Monitoring Consumer Lag via JMX

// JMX MBean for consumer lag
ObjectName consumerLagName = new ObjectName(
  "kafka.consumer:type=consumer-lag,client-id=consumer-1");

// Get current lag value
long lag = (Long) mBeanServer.getAttribute(
  consumerLagName, "Lag");

Key JMX MBeans for Kafka

kafka.server:type=KafkaServer - Server metrics
kafka.consumer:type=consumer-lag - Consumer lag monitoring
kafka.producer:type=producer-metrics - Producer performance
kafka.network:type=SocketServer - Network connection stats

9. Tracing Request Path

Example: Kafka Producer Trace

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.TRACE_CONFIG, "true");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key1", "value1");
producer.send(record);
producer.close();

Configuration Property	Value	Description
trace.enable	`true`	Enables tracing for Kafka client operations
trace.level	`INFO`	Log level for trace messages (INFO, DEBUG)
trace.log.file	`/var/log/kafka/trace.log`	File path to write trace logs

Note: Tracing is useful for debugging latency issues in Kafka operations. It logs internal state changes and timing information.

Warning: Enabling trace logging can significantly impact performance and disk space usage in production environments.

10. Debugging Rebalance Issues

Issue	Description	Symptoms	Resolution
Consumer Group Rebalancing	Consumers in a group are repeatedly rebalancing, causing performance degradation.	High CPU usage Repeated consumer lag Slow message processing	Check for consumer lag using `kafka-consumer-groups.sh` Ensure consumers are not timing out due to long processing times Adjust `session.timeout.ms` and `heartbeat.interval.ms`
Unstable Consumer Group	Consumers in a group are frequently leaving and rejoining, causing rebalances.	Rebalancing occurs every few seconds High rebalance count in consumer group metrics	Verify network stability between consumers and brokers Check for resource constraints (CPU, memory) Ensure `max.poll.interval.ms` is set appropriately

Example: Consumer Lag Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group --describe

Note: Rebalancing is a normal part of Kafka consumer groups, but excessive rebalancing can indicate misconfiguration or performance issues.