Skip to main content
Mapagam

56 SECTIONS · 1363 TOPICS

Kafka Roadmap

  1. Understanding Kafka Architecture

    1. 1. Understanding Broker Role
    2. Example: Broker Configuration
    3. 2. Understanding Topics and Partitions
    4. Example: Publishing to a Topic with Partitions
    5. Partitioning Strategy
    6. 3. Understanding Producers and Consumers
    7. Example: Producer Configuration
    8. Example: Consumer Configuration
    9. 4. Understanding Consumer Groups
    10. Example: Consumer Group Configuration
    11. Key Points About Consumer Groups
    12. Consumer Groups vs. Independent Consumers
    13. 5. Understanding Replication
    14. Example: Replication Configuration
    15. 6. Understanding Log Segments
    16. Example: Log Segment Configuration
    17. 7. Understanding Offsets
    18. Example: Fetching Offsets
    19. 8. Understanding Message Flow
    20. Example: Producer to Consumer Flow
    21. Message Flow Steps
    22. 9. Understanding ZooKeeper Role
    23. Example: ZooKeeper Ensemble Configuration
    24. 10. Understanding KRaft Architecture
    25. Example: KRaft Configuration
    26. KRaft vs ZooKeeper Architecture

  2. Installing and Configuring Kafka

    1. 1. Downloading Kafka
    2. Example: Download and Extract Kafka
    3. 2. Setting Up Java Requirements
    4. Example: Verify Java Installation
    5. 3. Configuring Broker Properties
    6. Example: Setting up broker configuration
    7. 4. Setting Broker ID
    8. Example: Setting Broker ID in server.properties
    9. 5. Configuring Log Directories
    10. Example: Setting up multiple log directories
    11. 6. Configuring Listeners
    12. Example: Configuring SSL Listeners
    13. 7. Setting Up Cluster ID
    14. Example: Manually Setting Cluster ID
    15. 8. Starting ZooKeeper Server
    16. Example: Starting ZooKeeper Server
    17. 9. Starting Kafka Broker
    18. Example: Start Kafka Broker
    19. Startup Process Overview
    20. 10. Using KRaft Mode
    21. Example: Minimal KRaft Configuration
    22. 11. Verifying Installation
    23. Example: Check Kafka version

  3. Managing Topics

    1. 1. Creating Topics
    2. Example: Create a topic with default configuration
    3. 2. Listing Topics
    4. 3. Describing Topics
    5. Example: Describe a Kafka Topic
    6. 4. Setting Partition Count
    7. Example: Creating a topic with 4 partitions
    8. Example: Changing partition count for an existing topic
    9. 5. Setting Replication Factor
    10. Example: Setting Replication Factor
    11. 6. Altering Topic Configuration
    12. Example: Altering topic configuration using CLI
    13. 7. Deleting Topics
    14. Example: Delete Topic via CLI
    15. 8. Configuring Topic Retention
    16. Example: Setting retention policy for a topic
    17. 9. Setting Cleanup Policy
    18. Example: Configuring Cleanup Policy for a Topic
    19. 10. Configuring Segment Settings
    20. Example: Setting Segment Size and Retention

  4. Working with Partitions

    1. 1. Understanding Partition Assignment
    2. Example: Consumer Group with Partition Assignment
    3. Key Concepts
    4. 2. Specifying Partition Key
    5. Example: Setting Partition Key
    6. 3. Viewing Partition Distribution
    7. Example: Using Kafka CLI to describe partitions
    8. 4. Adding Partitions
    9. Example: Adding Partitions to a Topic
    10. Key Considerations When Adding Partitions
    11. 5. Understanding Leader Election
    12. Example: Configuring Unclean Leader Election
    13. Leader Election Process
    14. 6. Understanding Log Segments
    15. Example: Configuring Log Segment Settings
    16. 7. Configuring Preferred Leader Election
    17. Example: Setting Preferred Leader Election Policy
    18. 8. Configuring Min In-Sync Replicas
    19. Example: Setting min.insync.replicas
    20. 9. Setting Partition Assignment Strategy
    21. Example: Configuring Partition Assignment Strategy
    22. 10. Manual Partition Reassignment
    23. Example: Reassigning partitions manually
    24. Example: Reassignment JSON structure
    25. Workflow for manual partition reassignment

  5. Producing Messages

    1. 1. Creating Producer Instance
    2. Example: Creating a Kafka Producer in Java
    3. 2. Setting Producer Properties
    4. Example: Producer Configuration
    5. 3. Configuring Bootstrap Servers
    6. Example: Producer Configuration with Bootstrap Servers
    7. 4. Configuring Serializers
    8. Example: Using JsonSerializer
    9. 5. Sending Messages
    10. Example: Sending a JSON Message
    11. 6. Using Synchronous Send
    12. Example: Synchronous Send
    13. 7. Using Asynchronous Send
    14. Example: Asynchronous Send with Callback
    15. Key Points
    16. 8. Setting Message Key
    17. Example: Setting Message Key
    18. 9. Specifying Target Partition
    19. Example: Using a specific partition
    20. 10. Adding Message Headers
    21. Example: Adding Headers to a Message
    22. 11. Configuring Acknowledgments
    23. Example: Setting Ack Mode in Producer Configuration
    24. 12. Flushing Producer Buffer
    25. Example: Flushing Producer Buffer

  6. Configuring Producer Settings

    1. 1. Setting Batch Size
    2. Example: Configuring batch size
    3. 2. Configuring Linger Time
    4. Example: Setting linger.ms to 5ms
    5. 3. Setting Buffer Memory
    6. Example: Configuring Buffer Memory
    7. 4. Configuring Compression
    8. Example: Configuring GZIP Compression
    9. 5. Setting Max Request Size
    10. Example: Configuring max.request.size
    11. 6. Setting Request Timeout
    12. Example: Configuring Request Timeout
    13. 7. Configuring Delivery Timeout
    14. Example: Setting Delivery Timeout
    15. 8. Configuring Retries
    16. Example: Producer with Retry Configuration
    17. 9. Setting Idempotence
    18. Example: Idempotent Producer Configuration
    19. 10. Configuring Max In-Flight Requests
    20. Example: Setting max.in.flight.requests.per.connection

  7. Consuming Messages

    1. 1. Creating Consumer Instance
    2. Example: Creating a Basic Consumer Instance
    3. 2. Setting Consumer Properties
    4. Example: Consumer Configuration
    5. 3. Configuring Bootstrap Servers
    6. Example: Setting Bootstrap Servers
    7. 4. Configuring Deserializers
    8. Example: Custom Deserializer Configuration
    9. 5. Setting Consumer Group
    10. Example: Consumer Group Configuration
    11. 6. Subscribing to Topics
    12. Example: Subscribing to Topics
    13. Example: Subscribing with Pattern
    14. 7. Polling Messages
    15. Example: Polling Messages
    16. 8. Processing Records
    17. Example: Basic Record Processing
    18. Example: Manual Offset Committing
    19. 9. Using Manual Assignment
    20. Example: Manual Partition Assignment
    21. 10. Seeking to Position
    22. Example: Seek to Specific Offset
    23. 11. Closing Consumer
    24. Example: Closing Consumer

  8. Configuring Consumer Settings

    1. 1. Setting Fetch Size
    2. Example: Configuring Fetch Size
    3. 2. Configuring Fetch Wait Time
    4. Example: Setting Fetch Wait Time
    5. 3. Setting Max Poll Records
    6. Example: Configuring max.poll.records
    7. 4. Setting Max Partition Fetch Bytes
    8. Example: Configuring max.partition.fetch.bytes
    9. 5. Configuring Partition Assignment Strategy
    10. Example: Setting Partition Assignment Strategy
    11. Assignment Strategy Comparison
    12. 6. Setting Auto Offset Reset
    13. Example: Configuring auto.offset.reset
    14. 7. Configuring Isolation Level
    15. Example: Setting Isolation Level
    16. 8. Setting Session Timeout
    17. Example: Configuring Session Timeout
    18. 9. Setting Heartbeat Interval
    19. Example: Configuring Heartbeat Interval
    20. 10. Configuring Max Poll Interval
    21. Example: Setting max.poll.interval.ms
    22. 11. Configuring Check CRCs
    23. Example: Enabling CRC Checksum Validation
    24. 12. Setting Client ID
    25. Example: Setting Client ID

  9. Managing Consumer Groups

    1. 1. Understanding Group Coordinator
    2. Example: Consumer Configuration with Group Coordinator
    3. 2. Understanding Rebalancing
    4. Example: Consumer Joining a Group
    5. Rebalancing Impact
    6. 3. Listing Consumer Groups
    7. 4. Describing Consumer Groups
    8. Example: Describe Consumer Group
    9. 5. Viewing Consumer Lag
    10. Example: Using Kafka Consumer Lag Metrics
    11. Key Metrics for Consumer Lag
    12. 6. Resetting Offsets
    13. Example: Reset Consumer Group Offsets to Beginning
    14. Example: Reset Offsets to Specific Datetime
    15. 7. Deleting Consumer Groups
    16. Example: Delete Consumer Group via CLI
    17. 8. Configuring Session Timeout
    18. Example: Consumer Configuration with Session Timeout
    19. 9. Setting Heartbeat Interval
    20. Example: Configuring Heartbeat Interval
    21. 10. Configuring Max Poll Interval
    22. Example: Setting max.poll.interval.ms

  10. Managing Offsets

    1. 1. Understanding Offset Commits
    2. Example: Manual Offset Commit
    3. 2. Enabling Auto Commit
    4. Example: Configuring Auto Commit
    5. 3. Setting Auto Commit Interval
    6. Example: Configuring Auto Commit Interval
    7. 4. Committing Synchronously
    8. Example: Committing Offsets Synchronously
    9. 5. Committing Asynchronously
    10. Example: Asynchronous Commit with Callback
    11. 6. Committing Specific Offsets
    12. Example: Committing Specific Offsets
    13. 7. Storing Offsets Externally
    14. Example: Manual Offset Commit to Database
    15. 8. Resetting to Earliest
    16. Example: Resetting Consumer Group to Earliest Offset
    17. 9. Resetting to Latest
    18. Example: Resetting Offsets to Latest
    19. 10. Resetting to Specific Offset
    20. Example: Reset to Specific Offset
    21. Example: Reset to Earliest Offset
    22. 11. Resetting by Timestamp
    23. Example: Resetting Offsets by Timestamp

  11. Working with Message Serialization

    1. 1. Understanding Serialization Concept
    2. Example: JSON Serialization in Kafka Producer
    3. 2. Using String Serializer
    4. Example: Producing String Messages
    5. 3. Using Byte Array Serializer
    6. Example: Publishing Byte Array Messages
    7. 4. Using Integer Serializer
    8. Example: Publishing Integer Messages
    9. Example: Consuming Integer Messages
    10. 5. Using Long Serializer
    11. Example: Serializing Long Values
    12. 6. Using JSON Serialization
    13. Example: Publishing JSON Messages
    14. Example: Consuming JSON Messages
    15. 7. Using Avro Serialization
    16. Example: Producer using Avro Serialization
    17. Example: Consumer using Avro Deserialization
    18. 8. Using Protobuf Serialization
    19. Example: Producer Configuration with Protobuf
    20. Example: Consumer Configuration with Protobuf
    21. 9. Creating Custom Serializers
    22. Example: Custom Serializer Implementation
    23. Example: Custom Deserializer Implementation
    24. 10. Handling Serialization Errors
    25. Example: Handling Serialization Errors in Producer
    26. 11. Configuring Schema Registry
    27. Example: Schema Registry Configuration in Producer

  12. Understanding Message Compression

    1. 1. Understanding Compression Benefits
    2. Example: Compression Ratio Comparison
    3. 2. Using GZIP Compression
    4. Example: Configuring GZIP Compression
    5. 3. Using Snappy Compression
    6. Example: Configuring Snappy Compression
    7. 4. Using LZ4 Compression
    8. Example: Configuring LZ4 Compression in Kafka Producer
    9. 5. Using ZSTD Compression
    10. Example: Producer Configuration with ZSTD
    11. 6. Configuring Compression Level
    12. Example: Configuring Compression Level in Producer
    13. Example: Configuring Compression Level in Consumer
    14. 7. Understanding Compression Ratio
    15. Example: Compression Ratio Calculation
    16. 8. Comparing Compression Performance
    17. Performance Trade-offs
    18. 9. Setting Producer Compression
    19. Example: Producer Configuration with Compression
    20. 10. Broker Compression Handling
    21. Example: Configuring Broker Compression
    22. 11. Monitoring Compression Metrics
    23. Example: Monitoring Compression Metrics with JMX

  13. Working with Message Headers

    1. 1. Understanding Headers Concept
    2. Example: Setting and Retrieving Headers
    3. 2. Adding Headers to Producer
    4. Example: Adding Headers to Producer Record
    5. 3. Reading Headers in Consumer
    6. Example: Reading Headers in a Kafka Consumer
    7. 4. Setting Custom Headers
    8. Example: Setting Custom Headers in Producer Record
    9. 5. Using Headers for Routing
    10. Example: Setting Routing Headers in Producer
    11. Example: Consuming with Header-Based Routing
    12. 6. Storing Metadata in Headers
    13. Example: Setting Headers in Producer
    14. Example: Reading Headers in Consumer
    15. 7. Filtering by Headers
    16. Example: Filtering Messages by Header
    17. Header Filtering Best Practices
    18. 8. Understanding Header Serialization
    19. Example: Setting and Reading Headers in Java
    20. 9. Propagating Headers in Streams
    21. Example: Setting Headers in a Stream Processor
    22. Key Points
    23. 10. Adding Trace IDs
    24. Example: Adding Trace ID to Kafka Message

  14. Understanding Message Timestamps

    1. 1. Understanding Timestamp Types
    2. Example: Setting Custom Timestamp in Producer
    3. 2. Using Create Time
    4. Example: Setting Create Time in Producer
    5. 3. Using Log Append Time
    6. Example: Configuring Log Append Time
    7. 4. Configuring Timestamp Type
    8. Example: Setting Timestamp Type in Producer Configuration
    9. 5. Setting Producer Timestamp
    10. Example: Setting Producer Timestamp
    11. 6. Understanding Timestamp Ordering
    12. Example: Setting Timestamp Type in Producer Configuration
    13. 7. Using Timestamps for Retention
    14. Example: Configuring Log Retention
    15. 8. Filtering by Timestamp
    16. Example: Filtering by Timestamp Range
    17. 9. Configuring Timestamp Difference
    18. Example: Setting Timestamp Difference Limit
    19. 10. Resetting Offsets by Timestamp
    20. Example: Resetting Offsets Using AdminClient

  15. Understanding Message Ordering

    1. 1. Understanding Ordering Guarantees
    2. Example: Producer Ordering in a Partition
    3. 2. Preserving Order with Single Partition
    4. Example: Producing messages to a single partition
    5. 3. Using Message Keys for Ordering
    6. Example: Producing Messages with Keys
    7. Example: Consuming Messages with Keys
    8. 4. Configuring Max In-Flight Requests
    9. Example: Setting max.in.flight.requests.per.connection
    10. 5. Enabling Idempotent Producer
    11. Example: Configuring Idempotent Producer
    12. 6. Understanding Ordering Across Partitions
    13. Example: Producer with Key-Based Partitioning
    14. 7. Implementing Total Ordering
    15. Example: Using Kafka's Total Ordering with Partitioning
    16. 8. Handling Out-of-Order Messages
    17. Example: Reordering Messages in a Consumer
    18. 9. Using Timestamps for Ordering
    19. Example: Setting Message Timestamps
    20. 10. Configuring Retries for Ordering
    21. Example: Configuring Retry Settings for Ordering

  16. Understanding Replication

    1. 1. Understanding Replication Concept
    2. Example: Configuring Replication Factor
    3. 2. Understanding Leader and Followers
    4. Example: Partition Leadership Change
    5. 3. Configuring Replication Factor
    6. Example: Setting Replication Factor at Topic Level
    7. Example: Configuring min.insync.replicas
    8. 4. Understanding In-Sync Replicas
    9. Example: Configuring In-Sync Replica Settings
    10. 5. Configuring Replica Lag
    11. Example: Configuring Replica Lag in server.properties
    12. 6. Understanding Leader Election
    13. Example: Configuring Unclean Leader Election
    14. Leader Election Process
    15. 7. Understanding Replica High Watermark
    16. Example: Replica High Watermark in Practice
    17. 8. Understanding Log End Offset
    18. Example: Fetching Log End Offset
    19. 9. Configuring Preferred Leader
    20. Example: Preferred Leader Configuration
    21. 10. Understanding Replica Fetcher
    22. Example: Configuring Replica Fetcher Settings
    23. 11. Manual Replica Reassignment
    24. Example: Reassigning Partitions
    25. Example: JSON file for topics to move
    26. Example: JSON file for reassignment

  17. Implementing Log Compaction

    1. 1. Understanding Compaction Concept
    2. Example: Compaction in Action
    3. 2. Enabling Log Compaction
    4. Example: Enabling Log Compaction on a Topic
    5. 3. Understanding Compaction Process
    6. Example: Compaction Configuration
    7. Compaction Process Overview
    8. 4. Understanding Tombstone Messages
    9. Example: Producing a Tombstone Message
    10. 5. Configuring Compaction Lag
    11. Example: Setting Compaction Lag in Kafka Configuration
    12. 6. Setting Delete Retention
    13. Example: Configuring Log Retention
    14. 7. Configuring Segment Size
    15. Example: Setting Segment Size
    16. 8. Setting Min Cleanable Ratio
    17. Example: Configuring min.cleanable.ratio
    18. 9. Configuring Compaction Thread
    19. Example: Configuring Compaction Thread Count
    20. 10. Setting Compaction Buffer
    21. Example: Configuring Compaction Buffer
    22. 11. Monitoring Compaction Progress
    23. Example: Check compaction status via Kafka Tool
    24. Compaction Monitoring Metrics

  18. Understanding Message Delivery Semantics

    1. 1. Understanding Delivery Guarantees
    2. Example: Producer Configuration for At-Least-Once Delivery
    3. Example: Producer Configuration for Exactly-Once Delivery
    4. 2. Understanding At-Most-Once Delivery
    5. Example: Producer Configuration for At-Most-Once
    6. 3. Understanding At-Least-Once Delivery
    7. Example: Producer Configuration for At-Least-Once
    8. 4. Implementing Exactly-Once Semantics
    9. Example: Idempotent Producer Configuration
    10. Example: Transactional Producer Usage
    11. 5. Configuring Producer Idempotence
    12. Example: Idempotent Producer Configuration
    13. 6. Using Transactional Producer
    14. Example: Initializing a Transactional Producer
    15. Example: Sending Messages in a Transaction
    16. 7. Setting Consumer Isolation Level
    17. Example: Configuring Isolation Level
    18. 8. Understanding Duplicate Messages
    19. Example: Idempotent Producer Configuration
    20. 9. Handling Message Ordering
    21. Example: Producer Ordering in a Partition
    22. 10. Implementing Idempotent Consumer
    23. Example: Idempotent Producer Configuration
    24. 11. Understanding Message Loss Scenarios
    25. Example: Producer with Idempotent Writes

  19. Implementing Transactions

    1. 1. Understanding Transactional Concept
    2. Example: Transactional Producer Setup
    3. 2. Enabling Transactional Producer
    4. Example: Configuring Transactional Producer
    5. 3. Initializing Transactions
    6. Example: Initializing Transactional Producer
    7. 4. Beginning Transaction
    8. Example: Beginning a Transaction
    9. 5. Sending Transactional Messages
    10. Example: Sending Transactional Messages
    11. 6. Sending Offsets in Transaction
    12. Example: Sending Offsets in Transaction
    13. 7. Committing Transaction
    14. Example: Committing a Transaction
    15. 8. Aborting Transaction
    16. Example: Aborting a Transaction
    17. 9. Configuring Transaction Timeout
    18. Example: Setting Transaction Timeout
    19. 10. Setting Read Committed
    20. Example: Configuring Read Committed in Producer
    21. Example: Configuring Read Committed in Consumer
    22. 11. Understanding Exactly-Once Semantics
    23. Example: Producer with Idempotent and Transactional Features
    24. Key Benefits of Exactly-Once Semantics

  20. Implementing Retry Mechanisms

    1. 1. Understanding Retry Behavior
    2. Example: Configuring Retry Behavior
    3. 2. Configuring Retry Count
    4. Example: Producer Retry Configuration
    5. 3. Setting Retry Backoff
    6. Example: Configuring Retry Backoff
    7. 4. Configuring Delivery Timeout
    8. Example: Setting Delivery Timeout
    9. 5. Understanding Retriable Exceptions
    10. 6. Handling Non-Retriable Errors
    11. Example: Handling Serialization Errors
    12. 7. Implementing Exponential Backoff
    13. Example: Retry Configuration with Exponential Backoff
    14. 8. Using Circuit Breaker Pattern
    15. Example: Circuit Breaker in Kafka Producer
    16. Summary of Circuit Breaker States
    17. 9. Configuring Request Timeout
    18. Example: Setting Request Timeout
    19. 10. Implementing Consumer Retry Logic
    20. Example: Consumer Retry Configuration
    21. Example: Handling Retry Logic in Consumer

  21. Configuring Broker Settings

    1. 1. Setting Broker ID
    2. Example: Setting Broker ID in server.properties
    3. 2. Configuring Listeners
    4. Example: Configuring Multiple Listeners
    5. Example: Using SASL Authentication
    6. 3. Setting Log Directories
    7. Example: Configuring Log Directories
    8. 4. Configuring Number of Threads
    9. Example: Setting I/O Threads
    10. 5. Setting Socket Buffer Sizes
    11. Example: Configuring Socket Buffer Sizes
    12. 6. Configuring Max Message Size
    13. Example: Setting max message size in server.properties
    14. 7. Setting Default Replication Factor
    15. Example: Setting Default Replication Factor
    16. 8. Configuring Min ISR
    17. Example: Setting min.insync.replicas
    18. 9. Setting Unclean Leader Election
    19. Example: Enabling Unclean Leader Election
    20. 10. Configuring Log Retention
    21. Example: Setting Log Retention
    22. 11. Setting Log Segment Size
    23. Example: Configuring Log Segment Size
    24. 12. Configuring Log Cleanup
    25. Example: Enabling Log Cleaning

  22. Using KRaft Consensus Protocol

    1. 1. Understanding KRaft Architecture
    2. Example: KRaft Controller Configuration
    3. KRaft Architecture Overview
    4. 2. Understanding Metadata Replication
    5. Example: Configuring Metadata Replication
    6. 3. Configuring Metadata Log
    7. Example: Configuring Metadata Log
    8. 4. Setting Controller Quorum Voters
    9. Example: Configuring Controller Quorum Voters
    10. 5. Formatting Storage
    11. Example: Formatting Kafka Storage Directories
    12. 6. Starting KRaft Controller
    13. Example: Starting KRaft Controller
    14. 7. Starting KRaft Broker
    15. Example: Starting KRaft Broker
    16. 8. Using Combined Mode
    17. Example: Combined Mode Configuration
    18. 9. Configuring Controller Listener
    19. Example: Controller Listener Configuration
    20. 10. Migrating from ZooKeeper
    21. Example: Enabling KRaft in server.properties
    22. Migration Steps Summary

  23. Managing Cluster Metadata

    1. 1. Understanding Metadata Topics
    2. Example: Consumer Group Offset Commit
    3. 2. Viewing Cluster Metadata
    4. Example: Describe a specific topic
    5. 3. Describing Cluster Configuration
    6. Example: Configuring Broker ID and Listeners
    7. 4. Managing Broker Configuration
    8. Example: Setting Broker Configuration
    9. 5. Understanding Controller Role
    10. Example: Controller Election Process
    11. 6. Monitoring Controller Metrics
    12. Example: Collecting Controller Metrics with JMX
    13. 7. Handling Metadata Updates
    14. Example: Configuring Metadata Refresh
    15. 8. Configuring Metadata Max Age
    16. Example: Setting metadata.max.age.ms
    17. 9. Understanding Partition Metadata
    18. Example: Partition Metadata in Topic Description
    19. 10. Debugging Metadata Issues
    20. Example: Checking Broker Metadata
    21. Workflow: Troubleshooting Metadata Sync

  24. Using Kafka Streams

    1. 1. Understanding Streams Concept
    2. Example: Basic Stream Processing
    3. 2. Creating Streams Application
    4. Example: Basic Kafka Streams Application Setup
    5. 3. Building Stream Topology
    6. Example: Basic Kafka Streams Topology
    7. Building a Stream Topology
    8. Key Concepts in Kafka Streams Topology
    9. 4. Transforming Streams
    10. Example: Basic Transformation with Map
    11. Example: Filtering Records
    12. Example: Using FlatMap
    13. Transformation Operations Summary
    14. 5. Aggregating Streams
    15. 6. Joining Streams
    16. Example: Inner Join
    17. Example: Left Join
    18. 7. Windowing Operations
    19. Example: Counting Events in a Tumbling Window
    20. 8. Using State Stores
    21. Example: Using a Persistent State Store
    22. Example: Using an In-Memory State Store
    23. 9. Processing Records
    24. Example: Processing Records with Kafka Streams
    25. Processing Records Summary
    26. 10. Using Processors API
    27. Example: Custom Processor Implementation
    28. Example: Registering Custom Processor
    29. Key Concepts
    30. 11. Starting Streams Application
    31. Example: Basic Kafka Streams Application
    32. 12. Handling Streams Exceptions
    33. Example: Custom Exception Handler
    34. Key Points

  25. Configuring Kafka Streams

    1. 1. Setting Application ID
    2. Example: Configuring Application ID
    3. 2. Configuring Bootstrap Servers
    4. Example: Setting Bootstrap Servers in Kafka Streams Configuration
    5. 3. Setting Default Serdes
    6. Example: Configuring Default Serdes
    7. 4. Configuring Commit Interval
    8. Example: Setting Commit Interval
    9. 5. Setting Cache Size
    10. Example: Configuring Cache Size
    11. 6. Configuring State Directory
    12. Example: Setting State Directory
    13. 7. Setting Processing Guarantee
    14. Example: Configuring Processing Guarantee
    15. 8. Configuring Replication Factor
    16. Example: Setting Replication Factor in Topic Creation
    17. 9. Setting Number of Threads
    18. Example: Configuring Number of Threads
    19. 10. Configuring Window Store Retention
    20. Example: Setting Window Store Retention

  26. Working with Time-Based Processing

    1. 1. Understanding Time Semantics
    2. Example: Using Event Time in Kafka Streams
    3. 2. Understanding Event Time
    4. Example: Setting Event Time in a Kafka Stream
    5. 3. Understanding Processing Time
    6. Example: Processing Time Usage
    7. 4. Understanding Ingestion Time
    8. Example: Ingestion Time in Producer Record
    9. 5. Configuring Timestamp Extractors
    10. Example: Using Custom Timestamp Extractor
    11. 6. Implementing Custom Timestampers
    12. Example: Custom Timestamp Extractor Implementation
    13. 7. Using Tumbling Windows
    14. Example: Tumbling Window for Counting Events
    15. 8. Using Hopping Windows
    16. Example: Hopping Window Configuration
    17. 9. Using Sliding Windows
    18. Example: Sliding Window Configuration in Kafka Streams
    19. 10. Using Session Windows
    20. Example: Session Windows in Kafka Streams
    21. 11. Configuring Grace Period
    22. Example: Consumer Graceful Shutdown Configuration
    23. Example: Producer Graceful Shutdown Configuration
    24. 12. Handling Out-of-Order Events
    25. Example: Using Watermarks in Kafka Streams

  27. Implementing Stream Joins

    1. 1. Understanding Join Types
    2. 2. Using Stream-Stream Join
    3. Example: Stream-Stream Join Configuration
    4. 3. Using Stream-Table Join
    5. Example: Stream-Table Join in Kafka Streams
    6. 4. Using Table-Table Join
    7. Example: Table-Table Join
    8. Key Considerations
    9. 5. Implementing Inner Join
    10. Example: Inner Join Implementation
    11. 6. Implementing Left Join
    12. Example: Left Join Configuration
    13. 7. Implementing Outer Join
    14. Example: Performing an Outer Join
    15. 8. Configuring Join Windows
    16. Example: Setting Join Window Duration
    17. 9. Setting Join Retention
    18. Example: Configuring Join Retention
    19. 10. Understanding Join Semantics
    20. Example: Stream-Stream Join
    21. 11. Optimizing Join Performance
    22. Example: Configuring Join Window Size
    23. Performance Tips

  28. Using Kafka Connect

    1. 1. Understanding Connect Framework
    2. Example: Source Connector Configuration
    3. Example: Sink Connector Configuration
    4. Key Features of Kafka Connect
    5. 2. Running Standalone Mode
    6. Example: Standalone Configuration File
    7. 3. Running Distributed Mode
    8. Example: Distributed Mode Configuration
    9. Steps to Start Distributed Mode
    10. Key Features of Distributed Mode
    11. 4. Creating Source Connectors
    12. Example: Creating a Source Connector
    13. 5. Creating Sink Connectors
    14. Example: Creating a Sink Connector
    15. 6. Configuring Connector Properties
    16. Example: Basic Connector Configuration
    17. Example: Connector with Transformations
    18. 7. Managing Connectors via REST API
    19. Example: List all connectors
    20. Example: Create a new connector
    21. Example: Update connector config
    22. 8. Listing Connectors
    23. Example: Listing Connectors via REST API
    24. 9. Checking Connector Status
    25. Example: Retrieve connector status via REST API
    26. Example: Sample response from connector status endpoint
    27. 10. Pausing and Resuming Connectors
    28. Example: Pausing a Connector
    29. Example: Resuming a Connector
    30. 11. Deleting Connectors
    31. Example: Delete a Connector

  29. Configuring Kafka Connect

    1. 1. Setting Bootstrap Servers
    2. Example: Configuring Bootstrap Servers
    3. 2. Configuring Key and Value Converters
    4. Example: Configuring JSON Converters
    5. Example: Configuring Avro Converters
    6. Key vs Value Converter Differences
    7. 3. Setting Offset Storage
    8. Example: Configure Offset Storage
    9. 4. Configuring Config Storage
    10. Example: Configuring Kafka-based Storage
    11. 5. Setting Status Storage
    12. Example: Configure Status Storage Topic
    13. 6. Configuring Replication Factor
    14. Example: Setting Replication Factor for a Topic
    15. 7. Setting Group ID
    16. Example: Configuring Group ID in Connect Worker Properties
    17. 8. Configuring Plugin Path
    18. Example: Setting Plugin Path
    19. 9. Setting REST Port
    20. Example: Configure REST Port
    21. 10. Configuring Internal Topics
    22. Example: Internal Topic Configuration

  30. Using Schema Registry

    1. 1. Understanding Schema Registry Role
    2. Example: Schema Registration
    3. 2. Installing Schema Registry
    4. Example: Installing Schema Registry on Ubuntu
    5. Installation Steps
    6. 3. Configuring Schema Registry URL
    7. Example: Producer Configuration
    8. Example: Consumer Configuration
    9. 4. Registering Schemas
    10. Example: Registering a new schema
    11. 5. Retrieving Schemas
    12. Example: Retrieve Schema by ID
    13. Example: Retrieve Schema Version
    14. Example: List All Subjects
    15. 6. Listing Subjects
    16. Example: List all subjects
    17. Example: List all versions of a subject
    18. Example: Get schema by ID
    19. 7. Deleting Schema Versions
    20. Example: Delete Schema Version
    21. 8. Setting Compatibility Mode
    22. Example: Configuring Compatibility Mode via REST API
    23. 9. Checking Compatibility
    24. Example: Checking Schema Compatibility
    25. 10. Exporting Schemas
    26. Example: Export Schema to File

  31. Managing Schema Evolution

    1. 1. Understanding Schema Compatibility
    2. Example: Schema Registry Configuration
    3. Schema Compatibility Rules
    4. 2. Implementing Backward Compatibility
    5. Example: Adding Fields to Schema
    6. 3. Implementing Forward Compatibility
    7. Example: Schema Registry Configuration
    8. 4. Using Full Compatibility
    9. Example: Schema Evolution with FULL Compatibility
    10. 5. Using Transitive Compatibility
    11. Example: Registering a Schema with Transitive Compatibility
    12. 6. Configuring Compatibility Level
    13. Example: Setting Compatibility Level in Schema Registry
    14. 7. Handling Schema Evolution Errors
    15. Example: Schema Registry Error Handling
    16. Common Schema Evolution Error Scenarios
    17. 8. Versioning Schemas
    18. Example: Schema Versioning in Avro
    19. Example: Adding a New Field to Schema
    20. 9. Using Schema References
    21. Example: Defining a Schema with References
    22. 10. Migrating Schema Formats
    23. Example: Converting Avro to JSON Schema

  32. Implementing Custom Partitioners

    1. 1. Understanding Partitioner Concept
    2. Example: Default Partitioner Behavior
    3. Key Points
    4. 2. Creating Custom Partitioner
    5. Example: Custom Partitioner Implementation
    6. 3. Implementing partition() Method
    7. Example: Custom Partitioner Implementation
    8. 4. Configuring Partitioner Class
    9. Example: Custom Partitioner Configuration
    10. 5. Using Hash-Based Partitioning
    11. Example: Hash-Based Partitioning Configuration
    12. Example: Producer with Key for Hash Partitioning
    13. 6. Implementing Round-Robin Partitioning
    14. Example: Producer Configuration with Round-Robin Partitioner
    15. 7. Creating Sticky Partitioner
    16. Example: Using Sticky Partitioner in Producer Configuration
    17. Key Features of Sticky Partitioner
    18. 8. Implementing Range Partitioner
    19. Example: Range Partitioner Implementation
    20. Key Features of Range Partitioner
    21. 9. Using Composite Key Partitioning
    22. Example: Custom Partitioner Implementation
    23. 10. Handling Null Keys
    24. Example: Custom Partitioner with Null Key Handling

  33. Implementing Custom Interceptors

    1. 1. Understanding Interceptor Concept
    2. Example: Basic Producer Interceptor
    3. Example: Basic Consumer Interceptor
    4. 2. Creating Producer Interceptor
    5. Example: Implementing a Kafka Producer Interceptor
    6. 3. Creating Consumer Interceptor
    7. Example: Basic Consumer Interceptor Implementation
    8. 4. Implementing onSend() Method
    9. Example: Modifying Record Before Sending
    10. 5. Implementing onConsume() Method
    11. Example: Implementing onConsume()
    12. 6. Configuring Interceptor Chain
    13. Example: Producer Interceptor Configuration
    14. Example: Consumer Interceptor Configuration
    15. 7. Adding Monitoring Metrics
    16. Example: Custom Interceptor with Metrics
    17. 8. Implementing Tracing
    18. Example: Basic Tracing Interceptor
    19. Tracing Interceptor Configuration
    20. 9. Adding Security Checks
    21. Example: Authentication Check
    22. Example: Authorization Check
    23. 10. Transforming Messages
    24. Example: Modifying Message Value
    25. Example: Adding Headers to Messages

  34. Using Admin API

    1. 1. Creating Admin Client
    2. Example: Creating Admin Client
    3. 2. Creating Topics
    4. Example: Create a topic with default configuration
    5. 3. Listing Topics
    6. Example: List all topics
    7. 4. Describing Topics
    8. Example: Describe a Topic
    9. Example: Describe Topics with Detailed Information
    10. 5. Deleting Topics
    11. Example: Delete a Kafka Topic
    12. 6. Creating Partitions
    13. Example: Creating Partitions via Admin Client
    14. 7. Altering Configs
    15. Example: Altering Topic Configs
    16. 8. Describing Configs
    17. Example: Describe Broker Configs
    18. Example: Describe Topic Configs
    19. 9. Describing Consumer Groups
    20. Example: Describe Consumer Group
    21. 10. Listing Consumer Groups
    22. Example: Listing Consumer Groups via Admin Client
    23. 11. Deleting Consumer Groups
    24. Example: Delete Consumer Group
    25. 12. Deleting Records
    26. Example: Deleting Records from a Topic

  35. Using Console Tools

    1. 1. Producing Messages
    2. Example: Produce a message to a Kafka topic
    3. 2. Consuming Messages
    4. Example: Consuming messages from a topic
    5. Example: Consuming messages with custom deserializer
    6. 3. Reading from Beginning
    7. Example: Read from Beginning
    8. 4. Specifying Consumer Group
    9. Example: Consumer Group Configuration
    10. 5. Setting Message Format
    11. Example: Setting JSON format for message consumption
    12. Example: Setting Avro format for message production
    13. 6. Using Key Deserializer
    14. Example: Configuring Key Deserializer
    15. 7. Setting Partition
    16. Example: Create topic with 3 partitions
    17. 8. Reading Specific Offset
    18. Example: Reading from a specific offset
    19. 9. Displaying Message Timestamp
    20. Example: Displaying Message Timestamps in Console Consumer
    21. 10. Using Timeout
    22. Example: Setting Timeout for Console Consumer

  36. Using Kafka CLI Tools

    1. 1. Managing Topics
    2. Example: Creating a Topic with Custom Configuration
    3. 2. Working with Consumer Groups
    4. Example: List consumer groups
    5. Example: Describe consumer group
    6. Example: Reset consumer group offsets to earliest
    7. 3. Managing ACLs
    8. Example: Create ACL for a user to read a topic
    9. Example: List all ACLs
    10. Example: Delete an ACL
    11. 4. Managing Configs
    12. Example: View Topic Configs
    13. Example: Executing a reassignment plan
    14. Example: Verifying a reassignment plan
    15. 6. Preferred Leader Election
    16. Example: Enable Preferred Replica Election
    17. 7. Verifying Consumer Group
    18. Example: Describe consumer groups
    19. 8. Verifying Producer
    20. Example: Send a message using the console producer
    21. 9. Dumping Log Segments
    22. Example: Dump log segments for a topic
    23. 10. Mirroring Data
    24. Example: Mirroring Topics
    25. 11. Running Performance Tests
    26. Example: Kafka Test Tools Configuration
    27. Performance Test Summary

  37. Implementing Security

    1. 1. Understanding Kafka Security Model
    2. Example: Enabling SSL/TLS Authentication
    3. Kafka Security Layers
    4. 2. Enabling SSL Encryption
    5. Example: Broker SSL Configuration
    6. 3. Configuring SSL Certificates
    7. Example: Basic SSL Configuration
    8. Example: Generating a Self-Signed Certificate
    9. Example: Creating a Truststore
    10. 4. Setting Keystore Password
    11. Example: Configuring Keystore Password
    12. 5. Configuring SSL Endpoint Identification
    13. Example: Kafka SSL Configuration
    14. 6. Enabling SASL Authentication
    15. Example: SASL Configuration in server.properties
    16. 7. Configuring SASL/PLAIN
    17. Example: Kafka Server Configuration
    18. Example: Client Configuration
    19. 8. Using SASL/SCRAM
    20. Example: Configuring SASL/SCRAM for Kafka Broker
    21. Example: Configuring SASL/SCRAM for Kafka Client
    22. 9. Configuring SASL/GSSAPI
    23. Example: Kafka Server Configuration for GSSAPI
    24. 10. Setting Up SASL/OAUTHBEARER
    25. Example: Producer Configuration
    26. Example: Consumer Configuration
    27. 11. Enabling Inter-Broker Security
    28. Example: Enabling SSL for Inter-Broker Communication

  38. Managing Access Control

    1. 1. Understanding ACL Model
    2. Example: ACL Configuration
    3. Key ACL Concepts
    4. 2. Enabling Authorization
    5. Example: Enable ACL authorization in Kafka
    6. 3. Creating ACLs
    7. Example: Create ACL for Topic Read Access
    8. Example: Create ACL for Consumer Group Access
    9. 4. Listing ACLs
    10. Example: Listing ACLs for a Specific Topic
    11. Example: Listing ACLs for a Specific User
    12. 5. Removing ACLs
    13. Example: Remove ACL for a Topic
    14. 6. Setting Principal Format
    15. Example: Configuring Principal Format
    16. 7. Configuring Super Users
    17. Example: Define super users in Kafka
    18. 8. Understanding ACL Operations
    19. Example: Creating an ACL to allow reading from a topic
    20. 9. Setting Resource Patterns
    21. Example: Configure Resource Pattern for Topics
    22. Example: Configure Resource Pattern for Consumer Groups
    23. 10. Using Wildcard ACLs
    24. Example: Define ACLs with Wildcards
    25. 11. Configuring ACL Binding
    26. Example: Configuring ACL Binding for a Topic
    27. Example: Configuring ACL Binding for a Consumer Group

  39. Configuring Quotas

    1. 1. Understanding Quotas Concept
    2. 2. Setting Producer Quotas
    3. Example: Configuring Producer Quotas in server.properties
    4. 3. Configuring Consumer Quotas
    5. Example: Consumer Quota Configuration
    6. 4. Setting Request Quotas
    7. Example: Configuring Request Quotas in server.properties
    8. 5. Configuring Client-Level Quotas
    9. Example: Configuring Client-Level Quotas in server.properties
    10. Example: Configuring User-Level Consumer Byte Rate
    11. 7. Using Default Quotas
    12. Example: Default Quota Configuration
    13. 8. Configuring Quota Window
    14. Example: Setting Quota Window Configuration
    15. 9. Monitoring Quota Violations
    16. Example: Configuring quota monitoring in Kafka
    17. Example: Using JConsole to monitor quota violations
    18. Quota Monitoring Best Practices
    19. 10. Dynamic Quota Configuration
    20. Example: Configure Producer Byte Rate
    21. Example: Configure Consumer Byte Rate

  40. Implementing Dead Letter Queues

    1. 1. Understanding DLQ Pattern
    2. Example: DLQ Configuration in Kafka
    3. Key Benefits of DLQ Pattern
    4. 2. Creating DLQ Topic
    5. Example: Create DLQ Topic
    6. 3. Routing Failed Messages
    7. Example: Configuring Dead Letter Topic
    8. Example: Consumer Configuration for DLQ
    9. 4. Configuring DLQ in Streams
    10. Example: Basic DLQ Configuration
    11. Key Points
    12. 5. Using DLQ in Connect
    13. Example: Configuring DLQ in a Kafka Connect Connector
    14. DLQ Configuration Summary
    15. 6. Adding Error Metadata
    16. Example: Adding Error Metadata to a Dead Letter Message
    17. 7. Monitoring DLQ
    18. Example: Monitoring DLQ with Prometheus and Grafana
    19. Example: Grafana Dashboard Setup
    20. 8. Reprocessing DLQ Messages
    21. Example: Reprocessing DLQ Messages
    22. 9. Setting DLQ Retention
    23. Example: Configuring DLQ Retention
    24. 10. Implementing Circuit Breaker
    25. Example: Basic Circuit Breaker Pattern with Kafka
    26. Key Concepts

  41. Handling Errors and Exceptions

    1. 1. Understanding Producer Exceptions
    2. Example: Handling SerializationException
    3. 2. Handling Timeout Exceptions
    4. Example: Configuring Timeout Settings
    5. 3. Dealing with Serialization Errors
    6. Example: Handling Serialization Errors in Producer
    7. Best Practices for Serialization Error Handling
    8. 4. Handling Authorization Failures
    9. Example: Configuring ACL-based Authorization
    10. Common Authorization Failure Scenarios
    11. Authorization vs Authentication
    12. 5. Managing Retriable Exceptions
    13. Example: Retry Logic with Exponential Backoff
    14. Best Practices for Retriable Exceptions
    15. 6. Handling Non-Retriable Errors
    16. Example: Handling Serialization Errors
    17. 7. Understanding Consumer Exceptions
    18. Example: Handling RetriableConsumerException
    19. 8. Handling Offset Out of Range
    20. Example: Configuring Offset Reset Behavior
    21. 9. Dealing with Rebalance Exceptions
    22. Example: Handling Rebalance Exceptions in Consumer
    23. 10. Configuring Error Handlers
    24. Example: Dead Letter Queue Configuration
    25. Example: Retryable Error Handler Setup

  42. Optimizing Producer Performance

    1. 1. Tuning Batch Size
    2. Example: Configuring Batch Size
    3. 2. Configuring Linger Time
    4. Example: Setting linger.ms to 5ms
    5. 3. Optimizing Compression
    6. Example: Producer Configuration with Compression
    7. 4. Setting Buffer Memory
    8. Example: Configuring Buffer Memory
    9. 5. Configuring Max In-Flight Requests
    10. Example: Setting max.in.flight.requests.per.connection
    11. 6. Using Idempotent Producer
    12. Example: Idempotent Producer Configuration
    13. 7. Optimizing Serialization
    14. Example: Using Avro with Kafka Producer
    15. 8. Tuning Request Timeout
    16. Example: Setting Request Timeout
    17. 9. Configuring Retries
    18. Example: Producer Configuration with Retry Settings
    19. 10. Monitoring Producer Metrics
    20. Example: Producer Metrics Configuration

  43. Optimizing Consumer Performance

    1. 1. Tuning Fetch Size
    2. Example: Configuring Fetch Size
    3. 2. Configuring Fetch Wait Time
    4. Example: Configuring fetch.wait.max.ms
    5. 3. Setting Max Poll Records
    6. Example: Configuring max.poll.records
    7. 4. Optimizing Partition Assignment
    8. Example: Configuring Partition Assignment Strategy
    9. 5. Using Parallel Processing
    10. Example: Consumer Group with Multiple Threads
    11. Best Practices for Parallel Processing
    12. 6. Configuring Prefetching
    13. Example: Configuring Prefetching Parameters
    14. 7. Optimizing Deserialization
    15. Example: Custom Deserializer
    16. 8. Tuning Commit Frequency
    17. Example: Configuring Auto Commit Interval
    18. 9. Using Asynchronous Commits
    19. Example: Asynchronous Commit Usage
    20. 10. Monitoring Consumer Metrics
    21. Example: Consumer Metrics Configuration

  44. Optimizing Broker Performance

    1. 1. Tuning Number of Partitions
    2. Example: Setting Partition Count for a Topic
    3. Best Practices for Partition Tuning
    4. 2. Optimizing Replica Fetcher Threads
    5. Example: Configuring Replica Fetcher Threads
    6. 3. Configuring Network Threads
    7. Example: Adjusting Network Thread Configuration
    8. 4. Tuning I/O Threads
    9. Example: Configuring I/O Threads
    10. 5. Optimizing Log Segment Size
    11. Example: Configuring Log Segment Size
    12. 6. Configuring Zero-Copy Transfer
    13. Example: Enabling Zero-Copy Transfer
    14. 7. Tuning Flush Settings
    15. Example: Configuring Flush Settings
    16. 8. Optimizing Page Cache
    17. Example: Configuring Log Flush Settings
    18. 9. Setting Socket Buffer Sizes
    19. Example: Configuring Socket Buffer Sizes
    20. 10. Monitoring Broker Metrics
    21. Example: Monitoring Kafka Broker Metrics with JMX

  45. Optimizing Consumer Rebalancing

    1. 1. Understanding Rebalance Protocol
    2. Example: Consumer Group Joining
    3. Rebalance Phases
    4. 2. Configuring Session Timeout
    5. Example: Consumer Configuration
    6. 3. Setting Heartbeat Interval
    7. Example: Configuring Heartbeat Interval
    8. 4. Configuring Max Poll Interval
    9. Example: Setting max.poll.interval.ms
    10. 5. Using Cooperative Rebalancing
    11. Example: Enabling Cooperative Rebalancing
    12. Benefits of Cooperative Rebalancing
    13. Cooperative vs. Classic Rebalancing
    14. 6. Implementing Static Membership
    15. Example: Configuring Static Membership
    16. 7. Minimizing Rebalance Overhead
    17. Example: Consumer Configuration
    18. 8. Using Incremental Rebalancing
    19. Example: Configuring Incremental Rebalancing
    20. 9. Handling Rebalance Listeners
    21. Example: Implementing Rebalance Listeners
    22. 10. Monitoring Rebalance Latency
    23. Example: Configuring Rebalance Monitoring
    24. Key Metrics to Monitor

  46. Monitoring Broker Metrics

    1. 1. Understanding JMX Metrics
    2. Example: Fetching JMX Metrics via JConsole
    3. 2. Monitoring Bytes In/Out Rate
    4. Example: Kafka Broker Metrics Configuration
    5. 3. Tracking Message Rate
    6. Example: Monitoring Message Rate via JMX
    7. 4. Monitoring Request Metrics
    8. Example: Configuring Request Metrics in Kafka
    9. 5. Checking Under-Replicated Partitions
    10. Example: Using Kafka Consumer Groups to Identify Under-Replicated Partitions
    11. 6. Monitoring ISR Shrink/Expand
    12. Example: ISR Shrink Alert Configuration
    13. 7. Tracking Leader Election Rate
    14. Example: Configuring Kafka to expose leader election rate
    15. 8. Monitoring Log Flush Latency
    16. Example: Configuring Log Flush Interval
    17. 9. Checking Network Processor Idle
    18. Example: Monitoring Idle Percent
    19. 10. Monitoring Request Queue Size
    20. Example: Monitoring Request Queue Size via JMX

  47. Monitoring Producer Metrics

    1. 1. Tracking Record Send Rate
    2. Example: Monitoring send rate with JMX
    3. 2. Monitoring Batch Size
    4. Example: Configuring Batch Size
    5. 3. Checking Compression Ratio
    6. Example: Configuring Compression
    7. 4. Monitoring Record Queue Time
    8. Example: Configuring Producer Metrics
    9. 5. Tracking Request Latency
    10. Example: Monitoring Request Latency with JMX
    11. 6. Monitoring Buffer Available Bytes
    12. Example: Configuring Buffer Size
    13. 7. Checking Waiting Threads
    14. Example: Monitoring Waiting Threads
    15. 8. Tracking Record Error Rate
    16. Example: Error Rate Monitoring
    17. 9. Monitoring Record Retry Rate
    18. Example: Configuring Retry Rate
    19. 10. Checking Metadata Age
    20. Example: Configuring Metadata Age

  48. Monitoring Consumer Metrics

    1. 1. Tracking Fetch Rate
    2. Example: Fetch Rate Monitoring
    3. 2. Monitoring Records Consumed
    4. Example: Consuming Records with Metrics
    5. 3. Checking Consumer Lag
    6. Example: Consumer Lag Monitoring with Kafka Consumer Groups
    7. 4. Monitoring Fetch Latency
    8. Example: Fetch Latency Configuration
    9. 5. Tracking Commit Rate
    10. Example: Commit Rate Monitoring
    11. 6. Monitoring Join Rate
    12. Example: Consumer Group Join Rate Monitoring
    13. 7. Checking Sync Rate
    14. Example: Monitoring Sync Rate
    15. 8. Tracking Heartbeat Response Time
    16. Example: Configuring Heartbeat Response Time Monitoring
    17. 9. Monitoring Last Poll Time
    18. Example: Consumer Configuration
    19. 10. Checking Assigned Partitions
    20. Example: Consumer Group Assigned Partitions

  49. Managing Cluster Operations

    1. 1. Adding Brokers to Cluster
    2. Example: Adding a New Broker Configuration
    3. Workflow: Adding a Broker to an Existing Cluster
    4. 2. Removing Brokers from Cluster
    5. Example: Decommissioning a Broker
    6. 3. Expanding Cluster
    7. Example: Adding a New Broker to Kafka Cluster
    8. Steps to Expand Kafka Cluster
    9. 4. Balancing Partition Load
    10. Example: Configuring Partition Assignment Strategy
    11. Example: Setting Replication Factor
    12. Partition Load Balancing Best Practices
    13. 5. Rolling Restart
    14. Example: Rolling Restart Script
    15. Steps to Perform a Rolling Restart
    16. Best Practices for Rolling Restart
    17. 6. Upgrading Kafka Version
    18. Example: Rolling Upgrade Process
    19. Upgrade Workflow
    20. Upgrade Considerations
    21. 7. Migrating to KRaft Mode
    22. Example: KRaft Controller Configuration
    23. Example: ZooKeeper Mode Configuration (Legacy)
    24. Migration Steps Overview
    25. 8. Handling Broker Failures
    26. Example: Monitoring Broker Health
    27. Recovery Steps Summary
    28. Broker Failure vs. Partition Reassignment
    29. 9. Managing Disk Space
    30. Example: Configuring Log Retention
    31. Example: Setting Up Disk Space Monitoring
    32. Comparison: Log Retention Policies
    33. 10. Configuring Rack Awareness
    34. Example: Setting up Rack Awareness
    35. Key Points

  50. Using MirrorMaker for Replication

    1. 1. Understanding MirrorMaker Architecture
    2. Example: MirrorMaker Configuration
    3. 2. Configuring MirrorMaker 2.0
    4. Example: Basic MirrorMaker 2.0 Configuration
    5. 3. Setting Source Cluster
    6. Example: Source Cluster Configuration
    7. 4. Setting Target Cluster
    8. Example: Target Cluster Configuration
    9. 5. Configuring Topic Replication
    10. Example: Topic Replication Configuration
    11. 6. Setting Replication Flow
    12. Example: Configuring Replication Flow
    13. 7. Configuring Consumer Groups Sync
    14. Example: Consumer Group Configuration
    15. 8. Setting ACLs Replication
    16. Example: Configure ACLs for Replication
    17. Example: Set Up Replication with ACLs
    18. 9. Configuring Heartbeat Connectors
    19. Example: Heartbeat Configuration
    20. 10. Monitoring Replication Lag
    21. Example: Configuring Lag Monitoring in MirrorMaker

  51. Implementing Multi-Datacenter Replication

    1. 1. Understanding Multi-DC Patterns
    2. Example: Active-Active Configuration
    3. 2. Using Cluster Linking
    4. Example: Configuring Cluster Linking
    5. Key Features of Cluster Linking
    6. 3. Configuring Geo-Replication
    7. Example: Basic Geo-Replication Setup
    8. 4. Setting Up Disaster Recovery
    9. Example: Basic disaster recovery configuration
    10. Disaster Recovery Setup Steps
    11. 5. Implementing Failover Strategy
    12. Example: Basic Kafka Failover Configuration
    13. Failover Strategy Workflow
    14. Failover vs. Recovery
    15. Key Failover Considerations
    16. 6. Configuring Conflict Resolution
    17. Example: Enabling Unclean Leader Election
    18. 7. Monitoring Cross-Cluster Lag
    19. Example: Using Kafka Manager to Monitor Lag
    20. 8. Using Replicator
    21. Example: Configuring Replicator
    22. Key Considerations
    23. 9. Configuring Network Latency Handling
    24. Example: Adjusting Replica Lag Time
    25. 10. Testing Failover Procedures
    26. Example: Simulating Broker Failure
    27. Failover Testing Workflow
    28. Failover Testing Best Practices

  52. Deploying Kafka on Cloud Platforms

    1. 1. Using Amazon MSK
    2. Example: Creating MSK Cluster via CLI
    3. MSK vs Self-Managed Kafka
    4. 2. Deploying on Confluent Cloud
    5. Example: Creating a Kafka Cluster on Confluent Cloud
    6. Example: Configuring Security with Confluent Cloud
    7. Comparison: Dedicated vs Serverless Clusters
    8. 3. Using Azure Event Hubs
    9. Example: Sending Events to Event Hubs
    10. Example: Reading Events from Event Hubs
    11. Key Differences from Kafka
    12. 4. Deploying on Google Cloud
    13. Example: Creating a Kafka Cluster on Google Cloud
    14. Deployment Summary
    15. 5. Configuring Cloud-Specific Settings
    16. Example: AWS Configuration
    17. Example: GCP Configuration
    18. Example: Azure Configuration
    19. 6. Managing Cloud Kafka Security
    20. Example: Configuring TLS for Kafka
    21. Key Security Practices
    22. 7. Scaling Cloud Kafka Clusters
    23. Example: Adding a New Broker to AWS MSK Cluster
    24. Scaling Considerations
    25. 8. Monitoring Cloud Deployments
    26. Example: Kafka Cloud Monitoring Setup
    27. Key Metrics to Monitor
    28. 9. Optimizing Cloud Costs
    29. Example: Cost-Optimized Kafka Configuration
    30. Summary of Cost Optimization Tips
    31. 10. Implementing Hybrid Cloud Setup
    32. Example: Kafka Cluster Configuration for Hybrid Cloud
    33. Hybrid Cloud Deployment Considerations

  53. Capacity Planning and Sizing

    1. 1. Estimating Message Throughput
    2. Example: Throughput Calculation
    3. 2. Calculating Partition Count
    4. Example: Estimating Partition Count
    5. Partition Count Best Practices
    6. 3. Sizing Broker Hardware
    7. Example: Broker Memory Configuration
    8. Key Considerations
    9. 4. Estimating Storage Requirements
    10. Example: Storage Calculation
    11. 5. Planning Network Bandwidth
    12. Example: Producer Bandwidth Calculation
    13. 6. Determining Replication Factor
    14. Example: Setting Replication Factor
    15. 7. Sizing Consumer Groups
    16. Example: Consumer Group Configuration
    17. Consumer Group Sizing Guidelines
    18. 8. Planning for Peak Load
    19. Example: Peak Load Calculation
    20. Peak Load Planning Steps
    21. Key Metrics for Peak Load
    22. 9. Configuring Retention Policies
    23. Example: Setting Retention Policies
    24. 10. Monitoring Resource Utilization
    25. Example: Kafka Broker CPU Usage Monitoring
    26. Example: Memory Usage with JConsole

  54. Testing Kafka Applications

    1. 1. Using EmbeddedKafka
    2. Example: Using EmbeddedKafka with JUnit 4
    3. Example: Using EmbeddedKafka with JUnit 5
    4. 2. Creating Test Producer
    5. Example: Basic Test Producer Setup
    6. Example: Sending a Test Message
    7. 3. Creating Test Consumer
    8. Example: Basic Test Consumer Setup
    9. 4. Testing Streams Topology
    10. Example: Basic Kafka Streams Test
    11. Key Concepts
    12. 5. Mocking External Dependencies
    13. Example: Using Mockito to Mock Kafka Producer
    14. Best Practices for Mocking Kafka
    15. 6. Testing Serialization
    16. Example: String Serialization
    17. Example: JSON Serialization
    18. 7. Testing Error Scenarios
    19. Example: Simulating Producer Failures
    20. Example: Testing Consumer Retry Logic
    21. 8. Integration Testing
    22. Example: Integration Test Setup with Embedded Kafka
    23. Best Practices for Integration Testing
    24. 9. Performance Testing
    25. Example: Measuring Throughput
    26. Example: Latency Measurement
    27. 10. Load Testing
    28. Example: Kafka Load Test Configuration
    29. Load Testing Best Practices

  55. Debugging and Troubleshooting

    1. 1. Enabling Debug Logging
    2. Example: Kafka Server Debug Logging Configuration
    3. 2. Using Broker Logs
    4. Example: Log Configuration
    5. Example: Log Rotation
    6. Key Log Categories
    7. 3. Checking Controller Logs
    8. Example: Controller Log Configuration
    9. Key Log Fields to Monitor
    10. 4. Analyzing State Change Logs
    11. Example: Parsing State Change Logs
    12. Key Points
    13. 5. Using Kafka Dump Log Tool
    14. Example: Dumping a Kafka log segment
    15. 6. Checking Topic Metadata
    16. Example: Describe Topic Metadata
    17. Example: List All Topics
    18. 7. Analyzing Consumer Lag
    19. Example: Consumer Lag Monitoring Script
    20. 8. Using JMX Console
    21. Example: Connecting to Kafka via JMX
    22. Example: Monitoring Consumer Lag via JMX
    23. Key JMX MBeans for Kafka
    24. 9. Tracing Request Path
    25. Example: Kafka Producer Trace
    26. 10. Debugging Rebalance Issues
    27. Example: Consumer Lag Check

  56. Implementing Backup and Recovery

    1. 1. Understanding Backup Strategies
    2. Example: Full Backup Command Using Kafka Tools
    3. 2. Backing Up Topic Data
    4. Example: Configuring Log Retention
    5. Backup Strategy Summary
    6. 3. Backing Up Consumer Offsets
    7. Example: Exporting Consumer Offsets
    8. Example: Importing Consumer Offsets
    9. 4. Backing Up Schemas
    10. Example: Fetching a Schema by ID
    11. Example: Listing All Subjects
    12. 5. Backing Up Configuration
    13. Example: Backing Up Topic Configurations
    14. 6. Implementing Point-in-Time Recovery
    15. Example: Configuring Point-in-Time Recovery
    16. Key Considerations for PITR
    17. 7. Restoring from Backup
    18. Example: Restore a Kafka Topic from Backup
    19. 8. Recovering Consumer Groups
    20. Example: Consumer Group Recovery Configuration
    21. Recovery Steps
    22. 9. Testing Recovery Procedures
    23. Example: Basic Recovery Test Setup
    24. Recovery Test Procedure
    25. 10. Automating Backup Process
    26. Example: Kafka Backup Script
    27. Backup Automation Steps