Conduktor & Redpanda: Best of Breed Kafka Experience

Conduktor & Redpanda# 

Conduktor's mission is to provide simple, flexible and powerful tooling for Kafka developers and infrastructure. Our product is composed of two parts: a UI dedicated to developers efficiency and Conduktor Gateway, a Kafka proxy to embrace DevSecOps needs in organizations (enforcing best practices, end-to-end encryption auditability, multi-tenancy, seamless failover, and much more).

Redpanda is a simple and cost-efficient streaming data platform compatible with Kafka APIs, and it's source available! It's a single binary, there is no Zookeeper, and it's very performant.

Our mission definitely overlaps: Redpanda builds the best present and future of real-time data while Conduktor builds the best tooling for real-time data! What if we combine both?

Getting Started with Conduktor# 

At Conduktor, our focus is on the value we can bring to our users. We value simplicity, a quick feedback loop, and an easily deployable solution.

Conduktor runs on-premise, Docker or Kubernetes. As we want to provide the best experience out-of-the-box for our users discovering our potential, we want the simplest and fastest way to get started with Docker on your laptop.

We've been using Apache Kafka® for years, and we know how painful it can be to get started with Kafka, Apache ZooKeeper™, and Schema Registry. It's a lot of moving parts, it takes CPU and memory, it can be slow to start and lead to startup synchronization problems. It's definitely not the best experience for a first-time user.

Redpanda helps us to solve this problem. To start Conduktor with an embedded Kafka today, it's a one-liner:

1curl -L https://releases.conduktor.io/quick-start -o docker-compose.yml && docker compose up

/quick-start is a simple docker-compose.yml you can inspect. It starts Conduktor on http://localhost:8080 and Redpanda.

This gives us all we want for our users:

• The simplicity of a single command
• The speed of starting Conduktor to check its value against a local Kafka cluster
• The download size is minimal (Redpanda is just 130MB). It's important to us as we have users all around the world.
• The Schema Registry is embedded into Redpanda! No additional part to start here, we love it!
• No problem of startup synchronization between different moving parts (Kafka, Zookeeper, Schema Registry, etc.)

If you want to connect to your own Kafka cluster instead, you can use this other docker-compose flavor to start only Conduktor without embedded Kafka:

1curl -L https://releases.conduktor.io/console -o docker-compose.yml && docker compose up

Gateway: the Architect dream?# 

Conduktor Gateway is at its core a Kafka proxy: it listens and talks the Kafka protocol (my-kafka:9092).

Its role is to intercept the traffic from any applications and tooling —thinking they speak to the real Kafka cluster to do their business— and to seamlessly accept/deny/transform the requests/responses adding the layer of intelligence and control you want in your organization! It's totally transparent for the clients and for the real Kafka clusters, no code change, nothing. They're just communicating together through a smart layer on which you have control.

This way, you get infinite flexibility without having to change a single line of code in your applications or brokers, no need to upgrade them, no operational burden, no big rollout etc. The experience is seamless. The value is instant. It's a single endpoint to connect for your users and applications. Gateway takes care of all the complexity: networking, security, multi-tenancy, auditability, etc. We truly believe this is the best and the most efficient solution to build your internal Kafka platform to provide self-service to your teams and enforcing your rules and security at scale.

It's fully compatible with the whole Kafka protocol and all its APIs (produce, fetch, consumer groups, transactions, metadata, etc.). Redpanda is itself fully compatible with the Kafka protocol, so both work in synergy to have the best of the two worlds!

Getting Started with Redpanda# 

Redpanda is a cutting-edge streaming data platform designed to handle real-time data streams with exceptional performance and scalability. It's fully compatible with the Kafka protocol and delivers low-latency, high-throughput streaming data processing, making it ideal for building real-time applications and event-driven architectures.

We love Redpanda because it has a very small compute footprint, boots instantly (!), and comes with a built-in Schema Registry, an HTTP proxy, its own performance autotuner, and many other capabilities included. It's also free from other dependencies like JVM, ZooKeeper, or dedicated KRaft servers making any deployment easier than a standard Kafka distribution.

Redpanda comes with a free Community Edition you can host yourself and a fully managed Cloud solution with dedicated clusters.

Conduktor and Redpanda# 

Conduktor is fully compatible with Redpanda. We make sure that Conduktor Console (your Kafka UI, for users) and Conduktor Gateway (for applications and infrastructure) fully complement all the Redpanda benefits. We're even using it in our internal CI/CD, for the same reasons highlighted above!

Let's go through some use-cases where Redpanda users could benefit from synergies with Conduktor.

Multi-cloud and across networks# 

Many organizations are running on multiple Clouds and sometimes multiple Kafka providers like Confluent and Redpanda. It depends on the maturity of the company, its scale, and its governance model (or lack of it), and the desire to move from one provider to another for various reasons.

The more flexible your architecture, the more networking fancy things you'll have to build: VPCs, VPC peering, BGP routing, making sure DNS resolutions work, and so on. It's not always easy to get it right, and it's never easy to debug when something goes wrong. (i.e. a developer reporting that "My application can't connect", now what?)

Due to the fact that Conduktor Gateway is a Kafka proxy: it can be deployed anywhere, across VPC networks, and make the bridge between various parts of the business. It can take care of doing simple or complex Kafka routing: sending the traffic to the right cluster/topic and passing the traffic from the clusters to the users/applications. All of this can be configured at runtime without impacting any applications. If your traffic has multiple hops to traverse your networking topology, Conduktor Gateway is your Kafka router through all of this. Think of it as the nginx for the Kafka protocol (and not HTTP).

You can use Redpanda Cloud to manage your streaming infrastructure, and Conduktor Gateway to connect to them from anywhere, including your on-premises datacenter, or your other Cloud providers.

Introducing Redpanda in your existing Kafka infrastructure# 

Gateway is a layer on top of your Kafka infrastructure and providers. This allows you to centralize your security, audit, governance, routing, data encryption, etc., at scale, without impacting your applications and your infrastructure decisions.

In the end, you have fewer ties with your Kafka providers and more flexibility in these particular organizational domains. You can fully rely on your Kafka provider to provide you with their best performant streaming infrastructure and separate these other concerns if this is important to you.

This could be an opportunity to introduce Redpanda seamlessly in your organization, having the Gateway connected to it. You can start with one cluster and scale up as you go, without having to change anything in your applications or asking your developers to do anything, as you are in control.

What's next?# 

We're excited to work with Redpanda to continuously provide a better and amazing Data Streaming Platform experience for users and organizations trusting us. Both Redpanda and Conduktor care about the developer experience and the operational experience. The simpler, the better!

Conduktor Gateway makes your Kafka infrastructure more flexible, and we're excited to see how you'll use it with Redpanda! Join the Conduktor Community on Discord to talk about it with us and other Conduktor adepts.

If you have questions about Redpanda or want to share what you’re working on with like-minded folks, join the Redpanda Community on Slack.