Implementing Security
1. Understanding Kafka Security Model
| Component | Description | Security Type |
|---|---|---|
| Authentication | Verifies the identity of clients and brokers | Plaintext, SSL/TLS, SASL/PLAIN, SASL/SCRAM, SASL/GSSAPI |
| Authorization | Controls access to topics, groups, and clusters | ACLs (Access Control Lists), RBAC (Role-Based Access Control) |
| Encryption | Protects data in transit and at rest | SSL/TLS for transport, encryption at rest (Kafka Server 3.0+) |
Example: Enabling SSL/TLS Authentication
# server.properties
ssl.enabled.protocols=TLSv1.2
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=password
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=password
ssl.key.password=password
Note: Kafka supports both
SSL/TLS and SASL for authentication. For production, SSL/TLS with client certificates is recommended.
Kafka Security Layers
- Authentication: Ensures who is connecting to the cluster
- Authorization: Determines what actions a user can perform
- Encryption: Protects data in transit and at rest
2. Enabling SSL Encryption
SSL/TLS encryption in Kafka ensures secure communication between brokers, clients, and other components. It protects data in transit from eavesdropping and tampering.
| Configuration Property | Default Value | Description |
|---|---|---|
| ssl.enabled.protocols | TLSv1.2, TLSv1.3 | Protocols allowed for SSL/TLS connections |
| ssl.truststore.location | None | Path to the truststore file containing CA certificates |
| ssl.truststore.password | None | Password to access the truststore |
| ssl.keystore.location | None | Path to the keystore file containing the broker's private key and certificate |
| ssl.keystore.password | None | Password to access the keystore |
| ssl.key.password | None | Password for the private key in the keystore |
Example: Broker SSL Configuration
listeners=SSL://localhost:9093
ssl.enabled.protocols=TLSv1.2,TLSv1.3
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=changeit
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit
Note: Ensure that the truststore and keystore are properly secured and accessible only to authorized users.
3. Configuring SSL Certificates
Example: Basic SSL Configuration
{
<span class="property">ssl.enabled</span>: true,
<span class="property">ssl.keystore.location</span>: "/path/to/keystore.jks",
<span class="property">ssl.keystore.password</span>: "changeit",
<span class="property">ssl.key.password</span>: "changeit",
<span class="property">ssl.truststore.location</span>: "/path/to/truststore.jks",
<span class="property">ssl.truststore.password</span>: "changeit"
}| Property | Description | Default Value | Notes |
|---|---|---|---|
| ssl.enabled | Enables or disables SSL | false |
Set to true to enable SSL |
| ssl.keystore.location | Path to the keystore file | null |
Required when SSL is enabled |
| ssl.keystore.password | Password for the keystore | null |
Must match the keystore password |
| ssl.key.password | Password for the private key | null |
Must match the key password |
| ssl.truststore.location | Path to the truststore file | null |
Required for client authentication |
| ssl.truststore.password | Password for the truststore | null |
Must match the truststore password |
Note: Ensure that the keystore and truststore files are properly secured and accessible only to authorized users.
Warning: If
ssl.keystore.location is not set, Kafka will not be able to establish secure connections.
Example: Generating a Self-Signed Certificate
keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey -keyalg RSA -storepass changeit -keypass changeit -dname "CN=localhost, OU=Test, O=Test, L=Test, ST=Test, C=US" -ext "SAN:dns:localhost,ip:127.0.0.1"Example: Creating a Truststore
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert.pem -storepass changeit4. Setting Keystore Password
| Property | Default Value | Description |
|---|---|---|
| ssl.keystore.password | None | Password to access the keystore file. |
Note: The keystore password must match the password used when creating the keystore file.
5. Configuring SSL Endpoint Identification
| Property | Default Value | Description |
|---|---|---|
| ssl.endpoint.identification.algorithm | https |
Algorithm used to check the endpoint identification of SSL certificates. Set to empty string to disable. |
| ssl.truststore.location | None | Path to the truststore file containing trusted certificates. |
| ssl.truststore.password | None | Password to access the truststore. |
Example: Kafka SSL Configuration
ssl.endpoint.identification.algorithm=https
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=changeit
Note: The
ssl.endpoint.identification.algorithm property should be set to https or an empty string to enable or disable endpoint identification checks respectively.
Warning: Disabling endpoint identification can expose the system to man-in-the-middle attacks.
6. Enabling SASL Authentication
| Property | Values | Description |
|---|---|---|
| sasl.enabled | true |
Enables SASL authentication for Kafka brokers |
| sasl.mechanism | PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 |
Authentication mechanism to use |
| sasl.jaas.config | |
JAAS configuration for SASL |
Example: SASL Configuration in server.properties
sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled=true
Note: Ensure that
sasl.enabled.mechanisms includes the desired SASL mechanism and that sasl.mechanism.inter.broker.protocol is set to the same mechanism for inter-broker communication.
7. Configuring SASL/PLAIN
| Property | Default Value | Description |
|---|---|---|
| sasl.enabled | false |
Enables SASL authentication for Kafka brokers |
| sasl.mechanism | PLAIN |
SASL mechanism to use for authentication (e.g., PLAIN, SCRAM-SHA-256) |
| sasl.jaas.config | "" |
JAAS configuration for SASL mechanisms (e.g., PLAIN) |
Example: Kafka Server Configuration
sasl.enabled=true
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="admin" \
password="admin-secret";Example: Client Configuration
const kafka = new Kafka({
clientId: 'my-app',
brokers: ['localhost:9092'],
sasl: {
mechanism: 'plain',
username: 'admin',
password: 'admin-secret'
}
});
Note: Ensure that the
sasl.jaas.config is properly escaped in configuration files to avoid parsing errors.
8. Using SASL/SCRAM
SASL/SCRAM (Simple Authentication and Security Layer / Salted Challenge Response Authentication Mechanism) is a mechanism for securing Kafka clusters by providing strong authentication and encryption.
| Mechanism | Description | Configuration |
|---|---|---|
| SASL/SCRAM-256 | SCRAM with SHA-256 hashing algorithm. Provides strong authentication and is recommended for production environments. | sasl.mechanism=SCRAM-SHA-256 |
| SASL/SCRAM-512 | SCRAM with SHA-512 hashing algorithm. Offers even stronger security than SHA-256 but requires more computational resources. | sasl.mechanism=SCRAM-SHA-512 |
Example: Configuring SASL/SCRAM for Kafka Broker
sasl.enabled.mechanisms=SCRAM-SHA-256
sasl.mechanism=SCRAM-SHA-256
Example: Configuring SASL/SCRAM for Kafka Client
{"saslMechanism": "SCRAM-SHA-256",
"securityProtocol": "SASL_SSL",
"saslJaasConfig": "org.apache.kafka.common.security.scram.ScramLoginModule required username='user' password='password';"}
Note: Ensure that the Kafka cluster and clients are configured with matching SASL mechanisms and credentials.
Warning: SCRAM requires TLS to be enabled for secure transmission of credentials.
9. Configuring SASL/GSSAPI
| Property | Default Value | Description |
|---|---|---|
| sasl.enabled | false |
Enables SASL authentication for Kafka brokers |
| sasl.mechanism | PLAIN |
SASL mechanism to use (e.g., PLAIN, GSSAPI) |
| sasl.kerberos.service.name | kafka |
Service principal name for GSSAPI |
| sasl.kerberos.kinit.args | -kt /etc/security/keytabs/kafka.service.keytab kafka/localhost |
Arguments for kinit command to obtain Kerberos ticket |
Example: Kafka Server Configuration for GSSAPI
sasl.enabled=true
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
sasl.kerberos.kinit.args=-kt /etc/security/keytabs/kafka.service.keytab kafka/localhost
Note: Ensure that the Kerberos keytab file is accessible and has appropriate permissions for Kafka to read it.
Warning: GSSAPI requires proper Kerberos setup and configuration on both client and server.
10. Setting Up SASL/OAUTHBEARER
| Property | Value | Description |
|---|---|---|
| sasl.mechanism | OAUTHBEARER |
Authentication mechanism to use |
| sasl.jaas.config | org.apache.kafka.common.security.oauth.OAuthBearerLoginModule required; |
JAAS configuration for OAuthBearer login module |
| security.protocol | SASL_SSL |
Security protocol to use (SSL required for OAuth) |
Example: Producer Configuration
{
"bootstrap.servers": "localhost:9093",
"security.protocol": "SASL_SSL",
"sasl.mechanism": "OAUTHBEARER",
"sasl.jaas.config": "org.apache.kafka.common.security.oauth.OAuthBearerLoginModule required;"
}Example: Consumer Configuration
{
"bootstrap.servers": "localhost:9093",
"security.protocol": "SASL_SSL",
"sasl.mechanism": "OAUTHBEARER",
"sasl.jaas.config": "org.apache.kafka.common.security.oauth.OAuthBearerLoginModule required;",
"group.id": "my-group"
}
Note: Ensure the OAuth provider is configured to issue tokens compatible with Kafka's OAUTHBEARER implementation.
11. Enabling Inter-Broker Security
| Property | Default Value | Description |
|---|---|---|
| inter.broker.protocol.version | 3.5-IV2 |
Version of the protocol used for communication between brokers. Must be set to at least 3.5-IV2 to enable security features. |
| security.inter.broker.protocol | PLAINTEXT |
Security protocol for inter-broker communication. Options include PLAINTEXT, SASL_PLAINTEXT, SSL, and SASL_SSL. |
| sasl.mechanism.inter.broker.protocol | PLAIN |
Mechanism used for SASL authentication between brokers. Supported values: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. |
| ssl.keystore.location | |
Path to the SSL keystore file used for broker-to-broker encryption. |
| ssl.keystore.password | |
Password to access the SSL keystore. |
| ssl.truststore.location | |
Path to the SSL truststore file used for validating broker certificates. |
| ssl.truststore.password | |
Password to access the SSL truststore. |
Example: Enabling SSL for Inter-Broker Communication
inter.broker.protocol.version=3.5-IV2
security.inter.broker.protocol=SSL
ssl.keystore.location=/path/to/broker.keystore.jks
ssl.keystore.password=changeit
ssl.truststore.location=/path/to/broker.truststore.jks
ssl.truststore.password=changeit
Note: Ensure that all brokers in the cluster are configured consistently to use the same security protocol and certificates.