Security for humans and applications in Apache Kafka

Allowing and denying access to a resource is a core requirement for any system and a major concern for any organization. This can be achieved by different methods including network access limitations, Access Control Lists(ACL), and Role-based access controls(RBAC). What is often forgotten is there are two different types of entities that need to be approved or denied; Humans and Applications. Inside of the human and application classification, there are also sub categories; trusted and untrusted.

In this article, we will discuss the two most viable ways to protect and provide access to resources inside of your Apache Kafka deployment. In addition, we'll discuss how your choice of trusted or untrusted entities impacts your auth strategy.

Kafka authentication strategies

Network barriers

Examples of network barriers include things like Firewalls, internal(non-routable) IPs, and air-gaping systems(Non-network connected). For a system like Kafka simply disconnecting from the network is simple but hard to scale for controlling user access. Because data distribution and sharing is a core use case for Kafka, firewall rules can quickly become complex and unmaintainable.

Access control lists

The most basic type of access control is Access Control Lists(ACLs). ACLs are a simple list of rules that govern access to individual objects by individual entities. An example of a simple ACL list in Kafka would be:

ANONYMOUS has NO ACCESS to topic="book_topic";

Bob has WRITE access to topic="book_topic";

Alice has READ access to topic="book_topic";

Mitch has READ AND WRITE access to topic="book_topic";

In Kafka, these lists can be far more specific than simply allowing READ or WRITE access. Other objects such as consumer group creation, topic configuration management, cluster-level administration rights, and broker configuration operations are also in play and should be considered when creating and managing authorizations for your Kafka environments.

Access control lists can quickly become complex, hard to debug, and extremely large. In essence, teams are creating a substitute for RBAC that is hard to manage and scale. In order to solve this problem proper Role-based access controls should be introduced.

Role-based access controls (RBAC)

Role-based access controls are an abstraction and tool to better organize ACL permissions. RBAC aims to solve the complexity and administration problems created by ACLs.

One of the core concepts of RBAC is that it allows you to create groups that are assigned permissions. Once you've created groups, you can simply add or remove users or applications to & from these groups to enable or disable access to objects within Kafka. This ability to define permissions once and then assign users based on the level of access they need allows teams to easily on-board and scale permissioning in an organization.

Integrating security on the control plane

When implementing RBAC and ACL security mechanisms for an application with a user interface, ensuring consistent security practices for each end-user becomes a challenge. To maintain the same level of security, each end-user on the UI must have their own specific Kafka broker and identity, which can be difficult to manage and maintain, especially for a UI with a large user base.

For a Kafka UI application, creating and managing individual clients for each user is cumbersome and resource-intensive. As the number of users grows, maintaining separate Kafka access for each user becomes increasingly impractical, leading to bottlenecks in performance and scalability.

Moreover, implementing distinct Kafka connections for each user can also hinder user experience, as it might require frequent updates and increase the complexity of managing user sessions. As a result, application developers need to strike a balance between maintaining a robust security posture and providing a seamless user experience.

Conduktor recommends a combination of both worlds: having the same RBAC source for both the control and data plane.

The group is responsible for managing access to resources, and it has an association with identity and access control. Role-based access control (RBAC) is a method for restricting access based on the roles of users. The data plane is represented by Kafka, which is responsible for data processing, while the control plane is responsible for managing the data plane. Both data and control planes have an association with RBAC.

Summary

RBAC should be considered the default choice for any organization no matter what stage of Kafka maturity they're at. The ability to define rules once and reuse them multiple times makes RBAC the obvious choice to reduce overhead and complexity when it comes to Kafka auth.

Conduktor sits on top of your existing Kafka ecosystem and helps you explore, manage, and govern Kafka data. Given that there is important customer data flowing through Conduktor, we implemented granular RBAC capabilities to define rules down to the topic level. You can also import your existing third-party auth source of truth directly into Conduktor.

Try Conduktor today with a 14-day free trial to see how it can help you accelerate your data streaming journey.