Implementing Backup and Recovery

1. Understanding Backup Strategies

Strategy	Description	Use Case	Notes
Full Backup	Backs up entire Kafka cluster data including all topics, partitions, and configurations.	Disaster recovery, migration to new cluster	Time-consuming; requires significant storage space
Incremental Backup	Backs up only the data that has changed since last backup.	Regular backups with minimal overhead	Requires tracking of changes; more complex to implement
Snapshot Backup	Uses Kafka's built-in snapshot mechanism to capture state at a point in time.	Point-in-time recovery	Requires Kafka version 2.5+; leverages log compaction
Replica-Based Backup	Relies on Kafka's replication factor to maintain copies across brokers.	High availability, fault tolerance	Does not protect against cluster-level failures; only data loss within brokers

Example: Full Backup Command Using Kafka Tools

kafka-run-class.sh kafka.tools.BackupProducer \
  --broker-list "localhost:9092" \
  --topics "topic1,topic2" \
  --output-dir "/backup/kafka"

Note: Full backups are recommended for compliance and long-term archival purposes.

Warning: Incremental backups must be consistent with Kafka's internal state to avoid data inconsistency.

2. Backing Up Topic Data

Property	Description	Default Value	Notes
log.dirs	Directory where topic logs are stored	/tmp/kafka-logs	Can be a comma-separated list of directories
log.retention.hours	How long to retain logs before deletion	168 (7 days)	Configurable in hours or milliseconds
log.cleanup.policy	Policy for cleaning up logs (delete or compact)	delete	Can be set to `compact` or `delete`

Example: Configuring Log Retention

{
  "log.retention.hours": 168,
  "log.cleanup.policy": "delete",
  "log.retention.bytes": -1
}

Note: Kafka does not natively support full topic backups. You must implement external backup solutions using tools like confluent backup or custom scripts that replicate topic data to another storage system.

Backup Strategy Summary

Use log.dirs to define storage paths for topic logs
Configure log.retention.hours to control data lifecycle
Implement external tools or scripts for full topic backup and restore

3. Backing Up Consumer Offsets

Consumer offsets in Kafka are managed by the __consumer_offsets topic, which is automatically created and managed by Kafka. To back up consumer offsets, you can either:

Export the offsets using Kafka tools like kafka-run-class.sh with the org.apache.kafka.tools.ExportConsumerOffsets class.
Use the kafka-consumer-groups.sh command-line tool to describe and reset consumer groups.

Example: Exporting Consumer Offsets

kafka-run-class.sh org.apache.kafka.tools.ExportConsumerOffsets \
  --bootstrap-server localhost:9092 \
  --group my-group \
  --output-file consumer-offsets.json

Example: Importing Consumer Offsets

kafka-run-class.sh org.apache.kafka.tools.ImportConsumerOffsets \
  --bootstrap-server localhost:9092 \
  --group my-group \
  --input-file consumer-offsets.json

Option	Description	Notes
--bootstrap-server	Bootstrap server address	Required for connecting to Kafka cluster
--group	Consumer group ID	Used to identify the consumer group
--output-file	Output file path	JSON file to store offset data
--input-file	Input file path	JSON file to restore offset data

Note: Consumer offsets are stored in a compacted topic and are automatically managed by Kafka. Backing up consumer offsets is essential for recovery in case of consumer group failures or migrations.

4. Backing Up Schemas

Schema backups in Kafka are essential for maintaining data integrity and ensuring that schema evolution can be safely managed. While Kafka itself does not store schemas directly, it integrates with schema registries like Confluent Schema Registry to manage and store schemas.

Schema Registry API Endpoint	Description	Notes
/schemas	Retrieve all schemas	GET request to list all registered schemas
/schemas/ids/{id}	Get schema by ID	Retrieve a specific schema using its unique identifier
/subjects	List all subjects	GET request to list all schema subjects
/subjects/{subject}/versions	List schema versions for a subject	GET request to list all versions of a specific schema subject

Example: Fetching a Schema by ID

curl -X GET http://localhost:8081/schemas/ids/123

Example: Listing All Subjects

curl -X GET http://localhost:8081/subjects

Note: Schema Registry provides a REST API to manage schemas. Backups should include the schema definitions and their versions to support recovery in case of registry failures or data loss.

5. Backing Up Configuration

Configuration Type	Description	Backup Method	Notes
Broker Configs	Settings for individual Kafka brokers	Export to JSON/YAML files	Include `server.properties` and any custom configs
Topic Configs	Settings for topics like retention, partitions	Use `kafka-configs.sh` CLI	Can be backed up with `--describe` flag
Consumer Group Configs	Offsets and consumer group settings	Use `kafka-consumer-groups.sh`	Offsets stored in internal `__consumer_offsets` topic

Example: Backing Up Topic Configurations

kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name my-topic --describe --underlying

Note: Configuration backups should include all custom settings and be stored securely to ensure recovery in case of broker failures or configuration drift.

6. Implementing Point-in-Time Recovery

Point-in-Time Recovery (PITR) in Kafka allows you to restore a consumer group's offsets to a specific point in time, enabling recovery from data loss or corruption. This feature is particularly useful for maintaining data consistency and ensuring that consumers can reprocess messages from a known good state.

Configuration	Description	Default Value	Notes
log.flush.interval.messages	Number of messages after which the log is flushed to disk	`10000`	Controls how frequently messages are persisted to disk, affecting recovery speed
log.flush.interval.ms	Interval in milliseconds after which the log is flushed to disk	`900000`	Ensures data durability even if `log.flush.interval.messages` is not reached
offsets.retention.minutes	Retention period for consumer group offsets in minutes	`1440`	Defines how long offsets are retained before being eligible for deletion
offsets.retention.check.interval.ms	How often to check for expired offsets	`600000`	Frequency of offset retention cleanup cycles

Example: Configuring Point-in-Time Recovery

{log.flush.interval.messages: 10000,log.flush.interval.ms: 900000,offsets.retention.minutes: 1440}

Note: PITR relies on the availability of logs and consumer group metadata. Ensure that your Kafka cluster has sufficient disk space and that retention policies are configured to preserve logs long enough for recovery operations.

Warning: If offsets are not retained long enough, PITR may fail to restore consumers to a desired point in time. Adjust offsets.retention.minutes based on your recovery window requirements.

Key Considerations for PITR

Ensure log.flush.interval.ms is set to a reasonable value for durability
Configure offsets.retention.minutes to align with your recovery window
Monitor disk usage to prevent log retention from consuming excessive storage
Use kafka-consumer-groups.sh to inspect and reset consumer group offsets for PITR

7. Restoring from Backup

Example: Restore a Kafka Topic from Backup

kafka-topics.sh --create --topic restored-topic \
  --bootstrap-server localhost:9092 \
  --replication-factor 3 \
  --partitions 12

kafka-streams-application --restore-from-backup \
  --backup-path /backups/topic-backup-2025-01-15 \
  --topic restored-topic

Step	Description	Command
1. Prepare Topic	Create topic with same configuration as backup	`kafka-topics.sh --create --topic restored-topic --bootstrap-server localhost:9092 --replication-factor 3 --partitions 12`
2. Restore Data	Use backup tool to restore messages to topic	`kafka-streams-application --restore-from-backup --backup-path /backups/topic-backup-2025-01-15 --topic restored-topic`

Note: Ensure the topic configuration matches exactly to prevent data loss or misalignment.

Warning: Restoring to an existing topic may cause data inconsistency. Always verify backup integrity before restoring.

8. Recovering Consumer Groups

Property	Description	Default Value
group.id	Unique identifier for the consumer group	None
session.timeout.ms	Timeout for consumer session	45000
heartbeat.interval.ms	Interval between heartbeats	3000
auto.offset.reset	What to do when no offset is found	latest

Example: Consumer Group Recovery Configuration

const consumer = new KafkaConsumer({
  bootstrapServers: 'localhost:9092',
  groupId: 'my-group-id',
  sessionTimeout: 60000,
  heartbeatInterval: 5000,
  autoOffsetReset: 'earliest'
});

Note: When recovering a consumer group, ensure that the group.id remains consistent to maintain offset tracking.

Warning: If a consumer group is not properly recovered, it may lead to duplicate message processing or missed messages.

Recovery Steps

Stop all consumers in the group
Reset offsets to a known safe point or use earliest for reprocessing
Restart consumers with the same group.id
Monitor for any lag or errors in processing

9. Testing Recovery Procedures

Example: Basic Recovery Test Setup

# Create a test topic with multiple partitions
bin/kafka-topics.sh --create --topic test-recovery-topic \
  --partitions 3 --replication-factor 2 \
  --bootstrap-server localhost:9092

# Produce test messages
bin/kafka-console-producer.sh --topic test-recovery-topic \
  --bootstrap-server localhost:9092 <<EOF
Message 1
Message 2
Message 3
EOF

# Verify messages are produced
bin/kafka-console-consumer.sh --topic test-recovery-topic \
  --from-beginning --bootstrap-server localhost:9092

Recovery Test	Description	Command	Status
Topic Recreation	Verify topic can be recreated after deletion	`bin/kafka-topics.sh --delete --topic test-recovery-topic`	NEW
Consumer Group Recovery	Test consumer group state restoration	`bin/kafka-consumer-groups.sh --describe --group test-group`	DEPRECATED
Broker Failover	Simulate broker failure and recovery	`bin/kafka-server-stop.sh`	EXPERIMENTAL

Recovery Test Procedure

Stop one or more Kafka brokers
Verify consumer groups are rebalanced
Restart stopped brokers
Confirm data consistency across partitions
Validate consumer lag is minimal

Note: Recovery tests should be run in a staging environment before production deployment.

Warning: Ensure backup procedures are tested regularly to avoid data loss during actual failures.

10. Automating Backup Process

Example: Kafka Backup Script

#!/bin/bash
# Backup Kafka topics and configurations
BACKUP_DIR="/opt/kafka/backups"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p $BACKUP_DIR/$DATE

# Backup topic configurations
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --describe --all > $BACKUP_DIR/$DATE/topic_configs.txt

# Backup consumer groups
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all > $BACKUP_DIR/$DATE/consumer_groups.txt

# Backup topic data using kafka-run-class.sh
kafka-run-class.sh kafka.tools.Exporter --topic-list "my-topic" --output-dir $BACKUP_DIR/$DATE/topic_data

echo "Backup completed: $BACKUP_DIR/$DATE"

Component	Description	Automation Tool
kafka-configs.sh	Exports topic configurations and broker settings	`bash`
kafka-consumer-groups.sh	Exports consumer group metadata and offsets	`bash`
kafka-run-class.sh	Runs Kafka tools like Exporter for topic data backup	`bash`

Note: Ensure proper permissions and environment variables are set for the backup script to execute successfully.

Backup Automation Steps

Set up a scheduled job (e.g., cron) to run the backup script
Store backups in a secure, offsite location
Implement backup retention policies (e.g., keep last 30 backups)
Monitor backup success and failure logs