Security for Apache Kafka in Conduktor Platform: A Deep Dive

Conduktor Platform has introduced loads of impressive and innovative new solutions for Apache Kafka, but when it comes to importance, none will be as relevant as the security solutions. With so much of the world's infrastructure now involving Kafka in some way, enterprises need every tool at their disposal to help keep data safe. In this blog, we'll be looking at the security features available for users of the Conduktor Platform.

How the Conduktor Platform provides security

We can divide security features into two areas: security for your Kafka, and security for the Platform itself. There are three key Platform security features: Role-based access control (RBAC), Single sign-on (SSO), and audit logs. Security for Kafka is provided through End-to-end encryption and Data masking.

RBAC (Role Based Access Control)

Role-based access control allows administrators to assign specific roles to individual users, specifying what each role can access.

In the Conduktor Platform, RBAC permissions can be set at the topic level, meaning different roles would have different levels of access to individual topics. You can assign multiple users to roles at once by creating groups.

Let's imagine a standard Kafka dev team. One or more people will have "Admin" roles, having complete power over Conduktor Platform and the different roles. A "Senior Developer" role would have edit access to everything in the Platform, but no admin permissions like deleting users or editing roles. "Junior Developer" is more limited, without edit access to mission-critical topics. "QA" could be given access to Conduktor Testing and Conduktor Monitoring but not other parts of the Platform. And a "Business User" role could have read-only access to everything with no edit permissions.

SSO (Single Sign On)

Single sign-on enables users to login to Conduktor Platform with a common set of credentials, for example, a Google or Microsoft account. The Conduktor Platform supports the LDAP and OpenId protocols for SSO. Once you've performed the necessary setup for these protocols, controlling access to the Platform is easy. Anyone with access to enterprise credentials will have access to Conduktor, and anyone you remove from your enterprise loses access. You'll have single, centralized control.

Having SSO is a necessary step in enabling RBAC, since it allows administrators to control registrations, instantly enabling or disabling users and assigning them to the correct roles.

Audit Logs

An audit log is a track of actions taken within a system. Applications have always had logs to enable developers to fix bugs, but the purpose of an audit log is instead to see what actions users are taking and who is logged in. Conduktor's audit logs are comprehensive, demonstrating everything a user does within the Kafka ecosystem. There are a number of reasons audit logs could be vital:

With Conduktor Platform, you can see every action taking place within your Kafka ecosystem ensuring compliance and enhancing troubleshooting efforts.

End-to-end encryption

Encryption is a challenge in Kafka. It doesn't exist as a standard feature of the technology. There are other security features like TLS, but TLS is used for transport layer security. It only protects your data as it's flowing across the network. TLS does not protect your data while on a Kafka client or broker. While there are workarounds - for example, you could encrypt the disks that your brokers are running on - they'll still have issues. Kafka is a distributed system moving messages from producers to consumers, meaning disk encryption only really works at one point. Anyone with access to the producers or consumers can see the unencrypted message. Until now, the only way to get faultless end-to-end encryption has been to make it yourself, with all the work that entails.

The Conduktor Proxy, available to Conduktor Platform subscribers, can transparently encrypt and decrypt data as it is written or read from Kafka. This requires no changes to clients that access the proxy but retains the information in Kafka in a format that is unreadable by illegitimate clients. Implementation could not be simpler - a single API call can configure encryption and decryption for all data that passes through your clusters, or for specific topics.

Encryption can be applied to entire messages or sensitive fields within them. Plus, integration with Schema Registry supports a wide variety of message types.

Data Masking

Even with encrypted messages, you don't necessarily want key holders to be able to view all the data contained within a message. For example, data protection regulations like GDPR & CCPA mean that personally identifiable information (PII) should not be viewable by just anyone. Conduktor's data masking enables you to add instant, field-based masking across your entire Kafka ecosystem, meaning personal information stays personal.

Access Control Lists

If you don't want to use RBAC and SSO, Kafka Access Control Lists (ACLs) are still supported by the Conduktor Platform. ACLs function similarly to RBAC, in that it determines what access an individual user has, but without the granularity of roles available through RBAC.

How Do I Get Started?

Role-based access controls and audit logs will be enabled automatically when you have an enterprise subscription to Conduktor Platform. Single sign-on will require setup to connect to your enterprise credentials; social logins are automatic. Please see our documentation for more info on setting up.

End-to-end encryption is run through the Conduktor Proxy, which is available on request to all enterprise subscribers: Please contact us directly to find out how to gain access. To enable end-to-end encryption with the proxy, you need to run only a single command that defines encrypted fields. For example:

docker-compose exec kafka-client curl \
    --silent \
    --request POST "conduktor-proxy:8888/tenant/1-1/feature/encryption" \
    --header 'Content-Type: application/json' \
    --data-raw '{
        "config": { 
            "topic": "encrypted_topic",
            "fields": [ { 
                "fieldName": "password",
                "keySecretId": "secret-key-password",
                "algorithm": { 
                    "type": "TINK/AES_GCM",
                    "kms": "TINK/KMS_INMEM" 
                }
            },
            { 
                "fieldName": "visa",
                "keySecretId": "secret-key-visaNumber",
                "algorithm": { 
                    "type": "TINK/AES_GCM",
                    "kms": "TINK/KMS_INMEM" 
                } 
            }] 
        },
        "direction": "REQUEST",
        "apiKeys": "PRODUCE"
    }'

A full demo of encryption is available here, for Platform license holders

Data Masking is enabled from within Conduktor Platform. You just need to click on the Data Masking option and then add the policies that you need. Please note that Data Masking is currently only available for Docker versions of the Platform, but will be coming soon to the web app.

Summary

Security features are rarely a focus but are highly important. Conduktor Platform ensures that you don't ever have to worry about the security of your Kafka or your Kafka tools. Our security just works, no real setup needed. If you want to see all of these features in action, start your free trial of Conduktor today.