Organizations trust Apache Kafka to handle massive data streams, from customer transactions to operational metrics, which may contain personally identifiable information (PII): credit card credentials, home shipping addresses, individual health records, and more. Securing that data in motion is not optional, and the penalties for getting it wrong are heavy.
$9.48MAverage data breach cost in the U.S. (IBM Research, 2023)
€20M / 4%GDPR maximum fine, of annual global revenue, whichever is higher
204 daysAverage time to detect a breach (IBM Research, 2023)
Why streaming data is different
Many data security solutions were built for batch data sitting in databases or data lakes, not the unique demands of streaming data. Securing data in motion requires both specialized tools and specialized workflows. One misconfigured topic can cost more than your entire Kafka deployment is worth.
Out of the box, open source Kafka ships with real security gaps. This guide covers the key requirements for closing them: data encryption, access control, and security monitoring, along with the fundamental tradeoffs each involves. Part 1 lays out the three prerequisite risks every Kafka estate has to address. The gated sections that follow work through each in depth, from encryption in transit and at rest, to authentication and authorization, to audit logging and threat detection, and close with a readiness checklist you can score your own estate against.
This guide is for platform engineers, SREs, DevOps, and security teams responsible for securing data in real-time environments. If you are handling sensitive data, navigating compliance, or need visibility into who is accessing what, this is your playbook.
Out of the box, open source Kafka has certain security vulnerabilities. These can be addressed either by building your own in-house platform or by using third-party tools, but either way the gaps are the same. Three risks account for most of them.
Data flowing between a Kafka broker and connected clients is unencrypted. Adding encryption prevents attackers and unauthorized parties from intercepting information you would rather keep private. Depending on the distribution, Kafka may not encrypt data stored on disk either, which leaves stored data exposed if the underlying storage is compromised.
Kafka regulates user permissions via Access Control Lists (ACLs). While ACLs are a reasonable starting point for user management, they are not without flaws:
- Hard to use and maintain at scale.
- Lack key features such as auditing, scaling, and customization.
- Do not always integrate well with other technologies.
- Struggle to meet the needs of more complex, extensive Kafka environments.
Visibility into access attempts and security posture is critical for preventing threats before they escalate. Monitoring tools help identify anomalous broker behavior and performance; observability frameworks provide deeper context into system performance and health; and audit logs go further, capturing each action from every user, which simplifies root cause analysis, compliance, and forensic investigations.
The three gaps you have to close
- Encryption keeps data unreadable both on the wire and on disk, so that intercepted or exfiltrated data stays useless without the key.
- Access controls decide who can connect and what they can do, moving beyond raw ACLs to something maintainable at scale.
- Monitoring and auditing give you the trail to detect anomalies, prove compliance, and investigate incidents after the fact.
Together, these three cover the majority of Kafka's security gaps and are what a safe, compliant deployment is built on. The rest of this guide takes each in turn, along with the tradeoffs that come with every choice.
For most Kafka users, encryption is the primary concern. Even in private networks or a virtual private cloud (VPC), Kafka deployments are vulnerable to leaks from misconfigurations, permissive policies, or unauthorized access via stolen credentials. Encrypted data, if exfiltrated, remains unreadable without the correct key.
Kafka data needs encryption both in transit and at rest. At-rest encryption can occur client-side or on the broker. The key disadvantage of doing it on the broker is that the data is exposed temporarily in plaintext after in-transit decryption and before re-encryption to disk, which may not meet all requirements. Client-side encryption avoids that exposure but risks decentralized control, since clients may implement different standards, libraries, and algorithms that become difficult to track. Encrypting client traffic in middleware is another option.
Other types of metadata, such as logs or configurations, will also require additional setup for full compliance, especially if they are stored on disk.
Secure Sockets Layer (SSL) and its successor, Transport Layer Security (TLS), secure data moving between clients and brokers. Any communication begins with a TLS handshake between parties to authenticate server identity (optionally client identity with mTLS), provide certificates to prove legitimacy, and supply specific, one-time keys. Data is then encrypted, sent from one party to the next, and decrypted upon arrival.
Today, SSL is considered obsolete and insecure, and TLS is the preferred method of encryption in transit, with mTLS for the most secure setup. Depending on the client library you use, enabling TLS requires only minor changes to a broker's configuration files, along with generating and storing cryptographic certificates. Once configured correctly, unencrypted client connections are no longer permitted by the broker, so messages are protected from interception or alteration while in transit.
Kafka, like many data storage tools, does not encrypt the data it stores on disk. Your first line of defense should be file system- or disk-level encryption to protect Kafka's stored data. The ideal tool depends on your broker's runtime environment or cloud provider: Linux systems typically use Linux Unified Key Setup (LUKS) for full-disk encryption, while cloud vendors offer in-house tools for encrypting block storage, such as Amazon EBS, Azure Disk Encryption, and Google Cloud.
Beyond disk-level encryption, there are two forms of fine-grained encryption to consider:
Message-level
- The entire message is encrypted.
- Limits Kafka's ability to filter and route messages based on their contents.
- Excels at protecting sensitive data that must remain confidential throughout its lifecycle.
Field-level
- Only certain message fields containing sensitive data (such as passwords or PII) are encrypted.
- Leaves remaining fields (such as names or order numbers) unencrypted.
- Lets applications continue to use those fields to route messages to the right consumers.
Field-level vs message-level: what to use and when
| Feature / tradeoff | Field-level encryption | Message-level encryption |
|---|
| Granularity | Encrypts only sensitive fields (e.g. PII) | Encrypts the entire message payload |
| Routing & filtering | Preserved. Non-sensitive fields remain usable | Lost. Kafka cannot inspect encrypted contents |
| Security strength | Moderate (selective obfuscation) | High (all content hidden from intermediaries) |
| Performance overhead | Lower. Less data to encrypt | Higher. Entire payload to encrypt |
| Use case fit | Real-time processing, analytics, conditional routing | Highly sensitive data (health, finance, compliance) |
| Searchability & indexing | Easier to meet data minimization requirements | Disabled, entire content is opaque |
| Compliance flexibility | Selective control over what is exposed | Easier to guarantee data non-disclosure |
| Operational impact | Low disruption, more interoperable | Higher complexity, limited middleware visibility |
Beyond simply where and when to encrypt data, there are some unexpected logistical concerns. These are the tradeoffs that shape whether an encryption strategy holds up in production.
Compute consumption
Encryption and decryption require processing power in the form of additional CPU cycles, adding performance overhead. Architects must account for this performance impact on their deployments.
Securely storing and using encryption keys
Encrypted data is worthless if attackers can also steal the associated keys. Any security solution will therefore have to be compatible with a key management service (KMS), which helps automate the generation, validation, and storage of encryption keys.
Deleting encryption keys
Because of legislation like the GDPR's right to be forgotten, data needs to be removed upon request. One effective way to do this is to make encrypted data permanently inaccessible by deleting the encryption key associated with it. This is known as crypto shredding: once a key is deleted, any data encrypted with it becomes unreadable instantly across all systems, avoiding the slow and costly process of removing all the PII from downstream systems.
Certificate management
Because TLS certificates periodically expire, they have to be replaced. New certificates and keys can be generated, deployed to Kafka brokers and clients, and tested for reliability.
Encryption-induced lag
Field- and message-level encryption can introduce processing delays, negatively impacting low-latency use cases like fraud detection or real-time order routing. While horizontal scaling (adding partitions) might seem like a solution, encryption inherently consumes resources, leading to unavoidable lag. Teams must test their encryption under peak load conditions and monitor lag in production.
There is no right answer for everyone. Choosing the right solution for encryption at rest depends on where your data is stored. Teams with on-premise Kafka may want to use their operating system's full-disk encryption; teams with cloud-based infrastructure may instead rely on their storage types and the built-in encryption from their cloud providers.
While encryption protects against unwanted access by outsiders, it does not limit access to your Kafka broker. The next important step in securing Kafka is controlling who can connect (authentication) and what connected clients can do (authorization).
Authentication determines whether the client connecting to a broker is who it claims to be. The two most common ways of adding authentication to Kafka are mutual TLS (mTLS) and Simple Authentication and Security Layer (SASL).
mTLS
- An extension of TLS encryption where both the client and server present certificates to verify their identities, rather than just the server.
- Signing certificates must be obtained from a root certification authority trusted by both client and server.
- The broker's TLS config is updated to request client certificates; clients must supply one signed by a trusted authority.
- During the handshake, both sides validate each other's certificates, creating two-way trust.
SASL
- Integrates Kafka with authentication frameworks like Kerberos and LDAP, and also enables username and password authentication.
- When a broker is configured to use SASL, connecting clients must supply valid credentials.
- Producers and consumers must specify their SASL mechanism (like PLAIN, SCRAM, or GSSAPI) and credentials via configuration properties.
- The process varies by mechanism but is well documented.
Note that Kafka can use TLS and SASL simultaneously, providing two independent security checks. Clients need both a valid certificate and correct credentials. If an unauthorized party acquires a client certificate, access is still denied without the correct SASL credentials.
Kafka ships with Access Control Lists (ACLs) to manage basic authorization needs. While ACLs are useful, most large-scale Kafka deployments will require role-based access control (RBAC) to simplify permission management.
RBAC provides a structured approach to authorization by organizing around roles, which can be granted specific permissions. Platforms like Conduktor add fine-grained RBAC authorization to Kafka without requiring your DevOps team to build custom solutions or manually sync policies across your infrastructure. This makes RBAC worthwhile to most teams, unless they are running smaller or relatively simple Kafka environments.
Whether you use ACLs or RBAC, a few key concepts apply:
- Users represent authenticated identities. They can be individuals, applications, or services, and can be granted specific permissions through ACLs or role assignments.
- Groups allow related users to be managed collectively. Assigning multiple users to a group simplifies permission changes at scale. For example, a "DataScienceTeam" group could have read-only access to certain topics, ensuring every team member inherits those permissions.
- Resources represent Kafka entities like topics, consumer groups, and configuration settings. Controlling access to resources enables fine-grained authorization: certain roles might only need to read from specific topics, while others might be permitted to create or delete them.
RBAC policies combine these concepts to link users (or groups) to resources, defining which actions they can perform. A "Producer" role might have write-only access to certain topics, while a "Viewer" role has read-only access. When a new developer joins the data science team, assigning them to the "DataScienceTeam" group grants the permissions they require. While it is possible to implement your own Kafka authorization, this requires significant time and effort to build and maintain; tools like Conduktor make it easy to manage users and simplify group management.
The full authentication and authorization flow
To recap, clients authenticate using mTLS, SASL, or both. Then the broker applies RBAC policies to grant or deny access to specific topics and actions. TLS encrypts data transmitted over the network, and encryption at rest protects data stored on disk.
- Client connects. A producer or consumer initiates a connection to the Kafka broker.
- Authentication. The client presents credentials via SASL or a certificate via mTLS.
- Broker verification. The broker validates identity and checks for certificate trust.
- Authorization. The broker applies ACL or RBAC rules to permit or deny topic access.
- Access granted or denied. The outcome is decided based on credentials and access policies.
Monitoring and auditing are other key security operations. While Kafka does not include comprehensive audit tooling, it can produce detailed logs that can be imported by external systems for analysis and alerting.
Kafka uses Log4j to record security-related events. When its audit loggers are enabled, they capture detailed information on authentication attempts, authorization outcomes, and access to broker resources. These logs can be sent to any SIEM or log aggregator that supports Log4j.
With proper configuration and the right parsing tools, teams can spot issues like repeated authorization failures or unauthorized topic access in near real-time. Combined with existing security tooling, this creates a reliable audit trail for compliance and threat detection.
Kafka exposes operational metrics via Java Management Extensions (JMX), which can be integrated with performance monitoring tools like Prometheus and Grafana. Metrics such as throughput, latency, and partition usage help identify both performance and potential security issues.
For example, unusual spikes in errors or drops in consumer activity may point to misconfigurations or unauthorized access. Combined with your audit logs, these metrics provide a complete view of both the performance and security of your Kafka cluster.
A complete security monitoring stack layers four components, each feeding the one above it:
| Layer | What it does |
|---|
| Dashboards & alerts | Visualizes metrics, sets up alert thresholds, and surfaces suspicious behavior in real-time. |
| Log aggregator / SIEM | Parses and stores logs, correlates events, and enables threat detection and compliance checks. |
| Log framework (Log4j) | Captures structured audit logs from Kafka, including user actions and system events. |
| Kafka broker | Generates security-relevant logs: authentication attempts, authorization checks, topic access. |
Use this checklist to assess the security readiness of your Kafka environment across encryption, authentication, authorization, monitoring, and governance. Score each item: fully implemented and monitored (2 points), partially implemented or inconsistently applied (1 point), not implemented or unknown (0 points).
Data in transit (TLS)
- TLS enforced between all Kafka brokers and clients
- Valid CA-signed certificates in place for all endpoints
- Mutual TLS (mTLS) configured (if using client auth via certificates)
- Certificate expiration dates tracked and rotation process documented
Data at rest (disk encryption)
- Broker logs and topic data encrypted at disk level (LUKS, BitLocker, or cloud-native)
- Kafka metadata (e.g. ZooKeeper or KRaft configs) also encrypted
- OS-level encryption monitored and validated regularly
Application-level encryption
- Field-level encryption implemented for PII, credentials, and sensitive data
- Message-level encryption applied where routing/filtering isn't needed
- Encryption keys stored securely in a KMS or HSM
- Key rotation policies defined and tested
- Crypto-shredding or key-based deletion supported to comply with data erasure requirements
- Data masking or redaction enabled in user-facing tools
For brokers and clients
- SASL authentication enforced (PLAIN, SCRAM, GSSAPI, or OAuth)
- Mutual TLS enabled where applicable
- Authentication logs monitored for failed login attempts
- Expired credentials automatically reviewed and revoked
Certificate management
- TLS cert renewal process documented and automated
- Broker restarts scheduled for cert reloads
- Certificate storage secured and access-controlled
Access control policies
- ACLs or RBAC enforced across all Kafka resources (topics, consumer groups, etc.)
- Principle of least privilege enforced across producers, consumers, and admins
- Group-based access (e.g. "DataScienceTeam") used to simplify role management
- Access policies are centrally visible
RBAC
- Roles defined for producers, consumers, admins, and viewers
- Role assignments tracked and reviewed quarterly
- Changes to role assignments logged and auditable
Audit logging
- Authentication and authorization events captured in audit logs
- Logs shipped to a SIEM or log aggregator (e.g. Splunk, ELK, or Datadog)
- Alerts configured for suspicious activity (e.g. repeated failed logins, unauthorized topic access)
- Audit logs retained in compliance with data governance policies
Observability and metrics
- Kafka JMX metrics collected via Prometheus or similar
- Dashboards exist for broker health, topic throughput, and consumer lag
- Alerts in place for error spikes, abnormal latencies, or dropped connections
- Real-time anomaly detection to flag sudden behavior changes
Policies and documentation
- Data classification policies define what data types require encryption or access controls
- Kafka security policies formally documented and shared across teams
- Compliance obligations (e.g. GDPR, HIPAA, PCI DSS) mapped to Kafka security controls
- Regular audits scheduled to validate controls against policies
Resilience
- Disaster recovery plan documented and tested
- Kafka failover architecture validated (e.g. multi-AZ or multi-region)
- Switch between Kafka clusters during outages automated
- Backup and restore procedures tested regularly
- Chaos engineering used to simulate failure scenarios
Securing Kafka is not optional, but it does not have to be complex. Multiple layers are involved: encryption in transit and at rest, strong authentication, fine-grained authorization, continuous monitoring, and audit logging. None of them alone is sufficient. Together, they provide defense in depth and ensure compliance with both organizational policy and data protection laws.
Kafka does not include these capabilities by default. Most teams rely on a combination of manual configuration, custom tooling, and third-party integrations to fill the gaps. While this can work, it introduces operational overhead and does not scale easily. As Kafka becomes more central to sensitive, regulated, or real-time environments, standardizing its security becomes critical: teams need to centralize policy enforcement, automate auditing, and reduce blind spots in access and usage. These are no longer optional; they are part of what it means to run Kafka in production.
Securing Kafka involves many key activities, including encryption, RBAC, audit logging, and monitoring. Managing them independently can be time-consuming and error-prone. Conduktor offers a consolidated platform that brings these security and compliance functions together in a few important ways.
Centralized management
- A single interface for managing user authentication and RBAC policies.
- Makes it easier to create secure Kafka infrastructure and reduces the chance of misconfiguration.
Standardized encryption
- Conduktor Shield provides both field- and message-level encryption applied to sensitive payloads before they reach the broker.
- Removes a potential vulnerability, eliminates per-client encryption logic, and keeps data protected across its lifecycle.
Resilience and recovery
- Test Kafka infrastructure in simulated production environments to stay robust under real-world conditions.
- Fail over non-performing clusters so data remains accessible even if specific clusters go down.
Compliance and governance
- Detailed audit trails and access controls, with monitoring and alerting that make it simple to demonstrate compliance.
- Unified visibility helps identify and resolve security incidents from a central dashboard, instead of piecing together scattered logs and metrics.
Conduktor brings these layers together in one platform, enabling teams to secure Kafka without slowing it down. From encryption management to RBAC, real-time alerts, or data masking, it gives you a governed Kafka that is ready for scale, and frees DevOps teams from micromanaging Kafka security and compliance so they can focus on solving problems for your customers.
Ready to secure Kafka without slowing it down?
See how Conduktor brings encryption, RBAC, audit logging, and monitoring together in a single governed platform. Book a demo and we will walk through securing your estate end to end.
Book a demo
