Back to blog
Apache Kafka is a highly available, and scalable data streaming platform that thousands of companies use to support their mission-critical applications. While it provides immense power to teams delivering real-time applications, there are limitations in Kafka that lead to time sinks for central platform and development teams. We're excited to launch Conduktor Gateway to provide central platform teams responsible for Kafka with enhanced functionality to help govern, secure, and enable Kafka self-service to teams across an organization.
At a fundamental level, Conduktor Gateway is the transport layer between your Kafka client applications and your Kafka clusters. This transport layer is enhanced with Conduktor Interceptors or plugins that you can install on Gateway.
Announcing Conduktor Marketplace
To ensure people can have easy access to both the Gateway and Interceptors we're announcing the release of Conduktor Marketplace! Our marketplace contains an open-source version of Conduktor Gateway along with pre-built Interceptors or plugins that help you enhance Kafka for free. You'll also be able to build Interceptors to introduce any functionality you feel is missing in your Kafka.
Apache Kafka Limitations and Gateway Interceptors
Conduktor has been used by developers and central platform teams to explore and troubleshoot issues in Kafka data. Our customers operating Kafka at scale have increasingly had pain points surrounding governance, testing, and cost. With the launch of Conduktor Marketplace, we're releasing our first batch of pre-built Interceptors that are going to help solve the most critical issues for teams using Kafka.
Enforcing Kafka best practices across the organization
Central platform teams often have the most Kafka expertise in an organization. When Kafka starts to spread to individual project teams these central platform teams are often responsible for enabling these teams with Kafka best practices.
Introducing Safeguard
Conduktor Safeguard protects your data and infrastructure from Kafka errors by enforcing rules that are centrally defined. Rules can be defined by the central platform team when they should be followed by the whole organization or defined by the individual domain teams when that expertise is important.
Safeguard Interceptors:
Manage cost and data resilience by ensuring that topic creation adheres to required minimum and maximum values for replication factor and partition count.
Reduce outages by ensuring that all records produced in Kafka have a schema set.
Ensure data quality in Kafka by defining topics as read-only.
Increasing the reliability of Kafka
Kafka is a resilient system so it is difficult to simulate and test application problems that could occur in production in your dev and testing environments.
Introducing Chaos
With Conduktor Chaos you can simulate incidents in your Kafka application's development or testing environments.
Chaos Interceptors:
Ensure that your Kafka applications can handle messages which don't arrive in the format that is expected. For example, how would your Kafka application react if it's expecting a message similar to
transferAmount:10, currency: poundsbut actually receivestransferAmount: 10, currency: poundsABC?Validate that your application behaves correctly when duplicate records are written to Kafka.
Gateway Enterprise
Our pre-built Interceptors in Conduktor Marketplace give you a preview of the kind of functionality that you can build into Kafka. Gateway Enterprise has Interceptors that are more in line with the needs of companies that are operating Kafka at scale.
Ensuring security and compliance in Kafka
Encrypting the data flowing through Kafka either requires you to implement encryption at the application layer before the data is sent or after it is received in Kafka. This either requires you to spend cycles on your development team to build encryption into all of your applications or time from your central platform teams to implement encryption in storage.
Introducing Encryption
Conduktor Encryption works after data is sent from your application before it reaches your Kafka brokers. You can encrypt messages down to the field level using KMS solutions from AWS, Azure, or GCP.
Reducing the cost of Kafka
If you're in a company where you have a large number of clusters for historical, technical, or other reasons, these clusters cost you money, whether they're physical or in the cloud.
Introducing Multi-tenancy
Similar to virtual machines, Conduktor Multi-tenancy enables you to create as many virtual clusters as you need on a single Kafka cluster, allowing you to mathematically reduce the number of clusters you need to pay and maintain.
Conclusion
This is just the start of what we're building with Conduktor Gateway. We'll continue to release more Interceptors based on feedback from our customers. If you have any ideas, please let us know.
Today, users have the choice of downloading the open-source version of Gateway from our Marketplace to start enhancing Kafka with our free Interceptors or contacting us to experience Gateway Enterprise.
We aim to accelerate Kafka projects delivery by making developers and organizations more efficient with Kafka.