Configuring Quotas
1. Understanding Quotas Concept
| Quota Type | Description | Default Value | Notes |
|---|---|---|---|
| Producer Rate | Maximum number of messages a producer can send per second | Unlimited | Applies to all producers in the cluster |
| Consumer Rate | Maximum number of messages a consumer can fetch per second | Unlimited | Applies to all consumers in the cluster |
| Request Size | Maximum size of a single request in bytes | 1048576 (1 MB) | Prevents large requests from overwhelming the broker |
| Response Size | Maximum size of a single response in bytes | 1048576 (1 MB) | Prevents large responses from overwhelming the client |
Note: Quotas are enforced at the broker level and can be configured per user, client, or cluster-wide.
2. Setting Producer Quotas
| Property | Default Value | Description |
|---|---|---|
| producer.byte.rate | 1048576 | Maximum bytes per second that a producer can send to the broker |
| producer.request.rate | 1000 | Maximum number of requests per second that a producer can send to the broker |
| producer.message.rate | 10000 | Maximum number of messages per second that a producer can send to the broker |
Example: Configuring Producer Quotas in server.properties
producer.byte.rate=2097152
producer.request.rate=2000
producer.message.rate=15000
Note: These quotas can be overridden per user or per client ID using the
kafka-configs.sh command.
3. Configuring Consumer Quotas
| Property | Default Value | Description |
|---|---|---|
| consumer.byte.rate | 10485760 | Maximum bytes per second consumed by a consumer group |
| consumer.request.rate | 1000 | Maximum number of fetch requests per second per consumer |
| consumer.max.lag.time.ms | 300000 | Maximum time a consumer can lag behind the leader before being removed from the group |
Example: Consumer Quota Configuration
{
"quotas": {
"consumer": {
"byte.rate": 20971520,
"request.rate": 500
}
}
}
Note: Consumer quotas are enforced at the consumer group level and can be configured via
server.properties or dynamically through the AdminClient API.
Warning: Setting
consumer.byte.rate too low can cause consumers to fall behind and increase lag.
4. Setting Request Quotas
| Property | Default Value | Description |
|---|---|---|
| request.rate | 1000000 | Maximum number of requests per second per client |
| request.size | 1048576 | Maximum size of a single request in bytes |
| response.size | 1048576 | Maximum size of a single response in bytes |
Example: Configuring Request Quotas in server.properties
# Set request rate quota to 5000 requests per second
request.rate=5000
# Set maximum request size to 2MB
request.size=2097152
# Set maximum response size to 2MB
response.size=2097152
Note: Quota settings are applied per client and can be overridden by individual client configurations.
Warning: Setting quotas too low may cause request throttling or client disconnections.
5. Configuring Client-Level Quotas
| Property | Default Value | Description |
|---|---|---|
| client.quota.window.ms | 1000 |
Window size in milliseconds for quota calculations |
| client.quota.window.num | 5 |
Number of windows to maintain for quota calculations |
| client.quota.byte.rate | 1048576 |
Maximum bytes per second for client-level byte rate quota (1 MB/s) |
| client.quota.produce.rate | 1000 |
Maximum produce requests per second for client-level quota |
| client.quota.fetch.rate | 1000 |
Maximum fetch requests per second for client-level quota |
Example: Configuring Client-Level Quotas in server.properties
<code>client.quota.window.ms=1000
client.quota.window.num=5
client.quota.byte.rate=2097152
client.quota.produce.rate=2000
client.quota.fetch.rate=2000</code>
</pre>
</div>
</div>
<div class="note">
<strong>Note:</strong> These quotas are enforced per client ID and can be overridden at the user or IP level using <code>user.quota</code> or <code>ip.quota</code> settings.
</div>
<h2 id="6-setting-user-level-quotas">6. Setting User-Level Quotas<a href="#6-setting-user-level-quotas" class="toc__headingLink" aria-label="Link to this section"><svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"></path><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"></path></svg></a></h2>
<table>
<tbody>
<tr>
<th>Property</th>
<th>Default Value</th>
<th>Description</th>
</tr>
<tr>
<td class="property">quotas.user.<user>.producer_byte_rate</td>
<td>None</td>
<td>Maximum bytes per second produced by a user</td>
</tr>
<tr>
<td class="property">quotas.user.<user>.consumer_byte_rate</td>
<td>None</td>
<td>Maximum bytes per second consumed by a user</td>
</tr>
<tr>
<td class="property">quotas.user.<user>.request_percentage</td>
<td>None</td>
<td>Maximum percentage of requests a user can make</td>
</tr>
</tbody>
</table>
<div class="example">
<h2 id="example-configuring-user-level-producer-byte-rate">Example: Configuring User-Level Producer Byte Rate<a href="#example-configuring-user-level-producer-byte-rate" class="toc__headingLink" aria-label="Link to this section"><svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"></path><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"></path></svg></a></h2>
<div class="codeBlock" data-language="properties">
<pre><code>quotas.user.john.producer_byte_rate=10485760
Note: Replace
john with the actual username. These quotas are enforced per user and can be set to limit resource usage.
7. Using Default Quotas
| Quota Type | Default Value | Description |
|---|---|---|
| producer_byte_rate | 1048576 |
Maximum bytes per second produced by a client (1 MB/s) |
| consumer_byte_rate | 1048576 |
Maximum bytes per second consumed by a client (1 MB/s) |
| request_percentage | 10.0 |
Maximum percentage of requests per second from a client (10%) |
Example: Default Quota Configuration
{
"producer_byte_rate": 1048576,
"consumer_byte_rate": 1048576,
"request_percentage": 10.0
}
Note: Default quotas apply to all clients unless overridden by specific client or user quotas.
8. Configuring Quota Window
| Property | Default Value | Description |
|---|---|---|
| quota.window.ms | 1000 |
Window size in milliseconds for quota calculations. This determines how frequently quota limits are evaluated. |
| quota.window.num | 10 |
Number of quota windows to keep in memory for each client. This controls how long historical data is retained for quota calculations. |
Note: Adjusting
quota.window.ms affects how quickly quota limits are enforced. Smaller values provide more granular control but may increase overhead.
9. Monitoring Quota Violations
Example: Configuring quota monitoring in Kafka
# Enable quota monitoring via Kafka configuration
export KAFKA_OPTS="$KAFKA_OPTS -Dkafka.metrics.reporters=org.apache.kafka.common.metrics.JmxReporter"
# Start Kafka with metrics enabled
bin/kafka-server-start.sh config/server.properties
| Quota Type | Monitoring Method | Metrics |
|---|---|---|
| Producer Quota | Metrics via JMX | producer-byte-rate, producer-request-rate |
| Consumer Quota | Metrics via JMX | consumer-byte-rate, consumer-request-rate |
| Request Quota | Metrics via JMX | request-rate, request-size |
Note: Kafka's built-in JMX reporter exposes quota metrics under the
kafka.server domain.
Example: Using JConsole to monitor quota violations
Connect to Kafka broker using JConsole:
- Host: localhost
- Port: 9999 (JMX port)
- MBean: kafka.server:type=KafkaServer,name=BrokerTopicMetrics
Quota Monitoring Best Practices
- Enable JMX reporting for real-time metrics
- Set up alerts when
byte-rateexceeds configured limits - Use Prometheus + Grafana for advanced monitoring dashboards
10. Dynamic Quota Configuration
| Property | Default Value | Description |
|---|---|---|
| quota.window.ms | 1000 |
Window size in milliseconds for quota calculations |
| quota.window.max.revs | 1000 |
Maximum number of revisions to keep in the window |
| quota.producer.byte.rate | 1048576 |
Maximum bytes per second for producers (1 MB/s) |
| quota.consumer.byte.rate | 1048576 |
Maximum bytes per second for consumers (1 MB/s) |
| quota.request.rate | 1000 |
Maximum requests per second |
Note: Quota configurations can be applied at the broker, client, or user level. Changes take effect immediately.