# Conduktor - Full Content > Complete text content for LLM context. --- # About Conduktor Our Mission Conduktor makes your operational data ready to use, trusted, and AI-ready by connecting it to the right people at the right time. We unify access, governance, and observability for all data operations so teams can innovate faster, cut complexity, and operate with confidence. --- Leadership ## Led by our founders Driven by the belief that data streaming is crucial for organizations to achieve the potential of digital transformation. Conduktor was founded in 2020 by [Nicolas Orban](https://www.conduktor.io/authors/nicolas-orban) (CEO), [Stéphane Derosiaux](https://www.conduktor.io/authors/stephane-derosiaux) (CTO), and [Stéphane Maarek](https://www.conduktor.io/authors/stephane-maarek) (Chief Evangelist). Investors ## Backed by leading, global investors We benefit from the expertise and support of world-class investors known for backing category-defining companies. - [Accel](https://www.accel.com/) - [Frontline](https://frontline.vc/) - [RTP Global](https://rtp.vc/) - [M12](https://m12.vc/) - [Ansa](https://www.ansa.co/) Teamwork ## And Supported by Our People + Lexi We're system builders, problem-solvers and product crafters. Together, we're exploring what's possible in data streaming. assets/images/team/team-5.png assets/images/team/team-6.png assets/images/team/thelads.jpg assets/images/team/offsite1.jpg assets/images/team/offsite2.jpg assets/images/team/group.png assets/images/team/theteam.jpg Our Journey ## Highlights from our journey The journey from an idea to a full-blown operations platform for Kafka - **2026** — Conduktor leads the convergence of real-time data streaming and AI, making operational data instantly accessible for intelligent applications - **2025** — Conduktor expands U.S. operations and scales to serve Fortune 500 enterprises with mission-critical streaming infrastructure - **2024** — Conduktor [raises $30 million](https://techcrunch.com/2024/11/12/conduktor-aims-to-gatekeep-bad-data-out-of-companys-apps/) in a Series B round to accelerate expansion in the U.S. and product development - **2023** — Conduktor releases our [enterprise offering](https://www.conduktor.io/console) with a Kafka proxy, serving FTSE 500 companies - **2022** — Conduktor reaches a milestone of 100,000 developers using our [platform](https://www.conduktor.io/get-started?src=about) - **2021** — Conduktor [raises $20 million](https://venturebeat.com/technology/conduktor-which-brings-a-user-friendly-gui-to-apache-kafka-nabs-20m/) in a Series A round, led by Accel - **2020** — Conduktor is [founded](https://x.com/focusaurus/status/1204143097854095362). Inspired by our CTO's drive to solve the pain he was experiencing with Kafka. ## Our Values Six principles we live by every day. They shape how we build, ship, and support each other. - **Embrace the Pack** — At Conduktor, we function as a trusted wolf pack, moving swiftly toward collective goals while embracing each member's unique role - **Own It** — We build trust by turning words into action, embracing full responsibility for our commitments whether we succeed or fail - **Act with Urgency** — We leverage our agility to outpace competitors, empowering everyone to take action and continuously seek ways to accelerate progress - **Go the Extra Mile** — We exceed expectations through curiosity and daily improvements, creating compounding impact in a competitive market - **No Grit, No Glory** — We embrace the unglamorous work required to scale, staying humble while tackling difficult challenges that move the needle - **Take Smart Risks** — We create a safe environment for calculated risk-taking, knowing breakthrough innovation requires stepping beyond comfort zones Announcements ## Featured News October, 2025[Conduktor Joins AWS ISV Accelerate Program](https://www.businesswire.com/news/home/20251022066158/en/Conduktor-Joins-AWS-ISV-Accelerate-Program)BusinessWire November, 2024[Conduktor Raises $30 Million to Accelerate Data Streaming With Apache Kafka](https://techcrunch.com/2024/11/12/conduktor-aims-to-gatekeep-bad-data-out-of-companys-apps/)TechCrunch April, 2024[Conduktor Launches 2.0 Platform Enhancing Kafka Ecosystem Management](https://www.datanami.com/this-just-in/conduktor-launches-2-0-platform-enhancing-kafka-ecosystem-management-with-free-immediate-insights/)Datanami April, 2024[Conduktor Targets Kafka Developers With Conduktor 2.0](https://www.enterprisetimes.co.uk/2024/04/11/conduktor-targets-kafka-developers-with-conduktor-2-0/)Enterprise Times October, 2022[Data Streaming Utility Startup Conduktor Stakes Out A Leadership Position In Real-Time Big Data](https://www.forbes.com/sites/melaniefine/2022/10/19/data-streaming-utility-startup-conduktor-stakes-out-a-leadership-position-in-real-time-big-data/)Forbes September, 2022[Conduktor Says It Has an Easier Way for Developers to Work With Kafka](https://siliconangle.com/2022/09/22/conduktor-says-come-easier-way-developers-work-kafka/)SiliconANGLE November, 2021[Apache Kafka Startup Conduktor Nabs $20M Investment Led by Accel](https://siliconangle.com/2021/11/24/apache-kafka-startup-conduktor-nabs-20m-investment-led-accel/)SiliconANGLE --- Locations ## Our Offices We call both **London** and **New York** home. Two global hubs where our teams connect, create, and grow together. London, UK Uncommon 81-87 High Holborn London WC1V 6DF New York, USA 1 Pennsylvania Plaza, Suite 3910 New York NY 10119 Explore jobs at Conduktor [Open positions](https://www.conduktor.io/careers) Last updated: March 2026 Source: https://www.conduktor.io/about --- # Alternatives to Conduktor Playground We no longer host the Kafka Playground, but here are free alternatives to get started with Kafka. ## Coming from Apache Kafka Series — Learn Apache Kafka for Beginners v3? Keep reading! This is important. We've removed Conduktor Playground, so you'll need to update your playground.config to follow the course without interruption. ## Conduktor Free Kafka Playground has shut down We are sorry to announce that our Free Kafka Playground has been removed. But fear not, there are free alternatives. - **Option 1: Local Kafka** — Run a Kafka cluster & Conduktor locally on your laptop. One command if you have Docker installed. **Recommended.** - **Option 2: Aiven Cloud** — Connect to a free cloud provider. Requires more steps due to SSL security, but no local setup needed. **Note:** If you're already comfortable with Kafka, you can use any alternative: Confluent, Aiven, AWS MSK, Strimzi, Redpanda. ## Option 1: Run a local Kafka cluster Fastest and simplest — recommended for most users. **1. Run Conduktor (and Kafka) with Docker** Follow the course to install Conduktor UI with Docker (lecture #57) or follow our [Get Started guide](https://www.conduktor.io/get-started?src=alternatives-for-conduktor-playground). You will need [Docker](https://docs.docker.com/get-docker/) installed. ```bash curl -fL https://releases.conduktor.io/quick-start | docker compose -f - up ``` **2. Connect to the local Kafka cluster** After running the command, you'll have a local Kafka cluster on `localhost:19092`. No security configured, so you can interact directly: ```bash kafka-topics.sh --bootstrap-server localhost:19092 --list ``` You should see topics created by the Conduktor stack. Return to the course. **3. Destroy the local stack** When finished, run this in the same directory as your `docker-compose.yml`: ```bash docker compose down -v ``` ## Option 2: Create a cloud Kafka cluster Using Aiven's free tier — requires SSL configuration. **1. Create your Aiven account** Go to [console.aiven.io/signup](https://console.aiven.io/signup), register and verify your account. **2. Create your free Kafka cluster** Choose the Apache Kafka service. Use default options, but change authentication to **SASL** (toggle at the bottom). **3. Create kafka-client.config** The Aiven cluster uses SSL. Save this as `kafka-client.config`: ```bash bootstrap.servers= security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username='' password=''; ssl.truststore.location=certificate.pem ssl.truststore.type=PEM ssl.endpoint.identification.algorithm= ``` **4. Create certificate.pem** Find the CA certificate in Aiven connection details. Copy and save as `certificate.pem`. **5. Connect to the cluster** ```bash kafka-topics.sh --bootstrap-server --list --command-config kafka-client.config ``` **6. Connect Conduktor to Aiven** Add your cluster at [localhost:8080/settings/clusters](http://localhost:8080/settings/clusters). Test the connection — you're ready! ## You're ready to continue! We've shown how to connect Conduktor and the CLI to a Kafka cluster, whether local or cloud. Questions? Contact us at [support@conduktor.io](mailto:support@conduktor.io). Happy learning! Source: https://www.conduktor.io/alternatives-for-conduktor-playground --- # Learn Apache Kafka for Beginners Thank you for enrolling in this course! We're happy to have you here! Before we start learning some amazing stuff, let's get you set up in 3 easy steps! [Download the course code](https://www.conduktor.io/assets/kafka-for-beginners-code-20250813.zip) [Explore Kafkademy](https://www.conduktor.io/kafka) ### Step 1: Download the first part of the code Click [here to download part one](https://www.conduktor.io/assets/kafka-for-beginners-code-20250813.zip) and the download will start right away. ### Step 2: Get the second part on GitHub The second part lives on [GitHub](https://github.com/conduktor/kafka-beginners-course). You can grab it as a zip by [downloading it here](https://github.com/conduktor/kafka-beginners-course/archive/main.zip). Decompress both zips and keep them somewhere handy. You'll use them throughout the course. ### Step 3: You're ready to start! Both parts downloaded and unzipped? You're all set. Happy learning :) ## Keep learning Kafka, for free The course slides aren't distributed, but every concept, diagram, and worked example lives on Conduktor Kafkademy: our free, no-signup learning hub. It's the fastest way to go deeper the moment a lesson sparks a question. [Explore Kafkademy →](https://www.conduktor.io/kafka) ## The teams building on Kafka trust Conduktor The concepts you're learning here run in production at companies worldwide. When they need to govern, secure, and scale Kafka, they reach for Conduktor. > It's been a game changer for efficiency and adoption. > — Scot M, Platform Engineer, Virgin Australia > RBAC was the turning point. > — Anke Raich, Swiss Post > Conduktor easily integrated with our CI/CD pipelines. > — Taras Slipets, Staff Data Engineer, Flix Source: https://www.conduktor.io/apache-kafka-for-beginners --- # Conduktor Architecture: Console & Gateway How Conduktor fits the Kafka you already run A management plane and inline proxies in front of the clusters you already run. Self-hosted, Kafka-protocol compliant, nothing to migrate. [Talk to an architect](https://www.conduktor.io/contact?src=architecture) [Read the docs](https://docs.conduktor.io/guide/conduktor-in-production) CONSOLEGATEWAY ## Self-hosted, in your environment Runs inside your infrastructure: cloud, hybrid, on-prem, or air-gapped. Your data and credentials never leave your network. It connects to the Kafka, identity, and key systems you already run. ## Architecture **Console** is the management plane: control, governance, automation, and insights, reached through a UI, API, CLI, and MCP. The **data plane** runs two proxies on the wire: the **Gateway** for Kafka and a separate **Schema Registry Proxy** for schemas, routing, securing, and enforcing policy inline. Your clients keep speaking Kafka, seamlessly, with no SDK swap or rewrite. ## The four components Console is the management plane; the Gateway and Schema Registry proxies are the data plane. Observability handles metrics and alerting, and the CLI and MCP open everything to automation and AI. - **Console (management plane)** — Control, governance, automation, and insights: Web UI, API, RBAC, policies, audit, GitOps, Terraform. One container plus PostgreSQL; two or more for HA. - **Gateway (data plane)** — Two inline proxies, stateless and horizontally scaled. The Gateway (Kafka protocol) routes and enforces encryption, masking, data quality, quotas, and multi-tenancy; a separate Schema Registry Proxy handles schema authentication and authorization. - **Platform Observability** — Real-time metrics and self-service alerts for platform and app teams on their own topics and consumer groups. Built-in alerts and Insights flag Kafka best-practice risks. Bundled, or bring your own Prometheus or Mimir. - **CLI, MCP & automation** — Resources as code via the CLI and Terraform; AI agents via the built-in MCP server and Skills, all under the same RBAC. ## Where Gateway sits in your data path A transparent, Kafka-protocol proxy. Clients repoint their bootstrap server, no rewrite. Records pass through untouched unless a policy encrypts, masks, or rejects them. A separate Schema Registry Proxy fronts your registry the same way. One network hop, minimal latency, the same control model across any provider. See [how Gateway cuts Kafka DR from hours to minutes](https://www.conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes). - **No single point of failure** — Stateless: the Kafka cluster holds all state. Run several instances; clients reconnect to a healthy one automatically. A restart never drops traffic. [Load balancing & HA](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing) - **Scales horizontally** — CPU-bound: encryption, masking, and guardrails run on the wire. ~20–30 MB/s per instance (2 CPU / 4 GB), more on bigger nodes. Add instances as traffic and policies grow; three or more in production. [Scaling Gateway](https://docs.conduktor.io/guide/conduktor-in-production/system-requirements#scaling-gateway) - **Disaster-recovery failover** — Hot-switch traffic to a secondary Kafka cluster through the Gateway API, with no client changes. [Failover guide](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/failover) ## Where it runs, and what it needs Kubernetes with Helm (recommended), Docker, or AWS. Production needs an external PostgreSQL 13+, blob storage (S3, GCS, or Azure), and Kafka 2.7+. The public reference architecture covers HA, sizing, and a runnable example. [Reference architecturePublic, maintained repo: Helm HA topologies, S/M/L sizing, and a runnable K3D stack with Postgres HA, Vault KMS, and Keycloak OIDC.Open the repo →](https://github.com/conduktor/conduktor-reference-architecture) [Footprint at a glanceConsole: 1 container (2+ for HA) + PostgreSQL + blob storage. Gateway: stateless, 3+ instances. Built-in metrics, or your own Prometheus/Mimir.Component overview →](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts) [Technical requirementsFull hardware, dependency, and provider matrix in the docs.System requirements →](https://docs.conduktor.io/guide/conduktor-in-production/system-requirements) ## Plugs into your security stack Conduktor enforces controls using the identity and key systems you already run, not new ones it asks you to adopt. See [achieving data security for Kafka](https://www.conduktor.io/resources/ebooks/achieving-data-security-for-kafka). - **Your identity, your keys** — SSO over OIDC (Okta, Entra ID, Google, Keycloak, and more) or LDAP, with group mapping. Keys stay in your own Vault, AWS KMS, or Azure Key Vault. - **Encryption & masking on the wire** — Field-level or full-payload encryption at the proxy, across JSON, Avro, and Protobuf. Masking hides sensitive fields without touching the data in Kafka. - **RBAC & self-service** — Granular, prefix-aware RBAC across topics, groups, subjects, connectors, and clusters. Teams self-serve within the policies you set. - **Audit & multi-tenancy** — Audit logs in the vendor-neutral CloudEvents format. Virtual Clusters and Partner Zones isolate teams, environments, and partners on one cluster. Deep dives for architects Go deeper on the decisions behind the architecture. [Disaster recovery beyond replicationThe six things that keep Kafka running through a failover.Read more →](https://www.conduktor.io/blog/kafka-disaster-recovery-the-complete-strategy-beyond-replication) [Vendor-neutral KafkaWhy your architecture should not depend on a single provider, and how a proxy decouples it.Read more →](https://www.conduktor.io/blog/the-case-for-vendor-neutral-architecture) [Isolate tenants with Virtual ClustersReplace cluster-per-environment sprawl with isolation on one physical cluster.Read more →](https://www.conduktor.io/blog/how-to-isolate-non-prod-kafka-with-virtual-clusters) [Field-level encryption, no tech debtEncrypt at the proxy instead of rebuilding crypto in every app.Read more →](https://www.conduktor.io/blog/how-to-encrypt-data-in-kafka-without-piling-up-tech-debt) [Govern Kafka as codeTopics, policies, and access through Terraform and GitOps.Read more →](https://www.conduktor.io/blog/govern-kafka-as-code-terraform-x-conduktor) [MCP & Skills for AI agentsGive agents a governed, RBAC-scoped surface on your Kafka.Read more →](https://www.conduktor.io/blog/conduktor-mcp-and-skills-for-ai-agents) ‹ › ## Frequently asked questions **Is Gateway in the data path, and what happens if it fails?** Yes, it's an inline, stateless proxy. The Kafka cluster holds all state, so run several instances and clients reconnect on standard retry. A restart never drops traffic; run three or more in production. **Do I need to change my applications?** No. Gateway speaks the Kafka protocol. Point your bootstrap server at it and keep your standard clients: no SDK swap, no rewrite. Records pass through unless a policy applies. **Does my data ever leave my environment?** No. Conduktor is self-hosted; your messages, credentials, and keys stay in your network. Even the built-in MCP server runs locally, under your RBAC. **Which clouds and Kafka providers are supported?** AWS, Azure, GCP, or on-prem, in front of Confluent, Amazon MSK, Aiven, Redpanda, and open-source Apache Kafka (2.7+). Kubernetes with Helm recommended; Docker and AWS also supported. **Can we integrate with Vault or a KMS?** Yes. Keys live in HashiCorp Vault, AWS KMS, or Azure Key Vault, bring-your-own. Conduktor never holds them. **Can it run fully air-gapped?** Yes. Offline installs with local registries, updated without external network access. ## Planning your deployment? From sizing and high availability to compliance, our team will walk through the architecture and the trade-offs with you. [Talk to an architect](https://www.conduktor.io/contact?src=architecture) Source: https://www.conduktor.io/architecture --- # Áine O'Neill **Account Executive, Conduktor** Source: https://www.conduktor.io/authors/aine-oneill --- # Alasdair Ross **Engineering Manager** Alasdair leads engineering projects at Conduktor. His expertise spans from developing complex software solutions - including data pipelines, RESTful APIs, and cloud applications - to hands-on design and craftsmanship in furniture making. ## Articles by Alasdair Ross - [How to Pivot from Software Engineering to Event Streaming](https://www.conduktor.io/blog/from-engineer-to-event-streaming-expert-how-to-pivot-roles) (2025-04-16) — A practical guide to transitioning into Kafka and real-time data roles, with specific technologies, projects, and job search strategies. Source: https://www.conduktor.io/authors/alasdair-ross --- # Ana Nash **Account Manager** Ana is an Account Manager at Conduktor. Source: https://www.conduktor.io/authors/ana-nash --- # Andy Allison **VP of Product** ## Articles by Andy Allison - [Your Kafka Investment Is Hiding Costs and Opportunities](https://www.conduktor.io/blog/your-kafka-investment-is-hiding-costs-and-opportunities) (2026-02-20) — Your Kafka investment is hiding costs, unused capacity, and optimization opportunities that no dashboard can surface. AI changes what's possible to ask. - [Why Your Kafka Architecture Should Not Depend on Any Single Vendor](https://www.conduktor.io/blog/the-case-for-vendor-neutral-architecture) (2025-12-18) — IBM's Confluent acquisition shows why vendor-neutral Kafka matters. Decouple apps from infra to cut costs and stay flexible. Source: https://www.conduktor.io/authors/andy-allison --- # Augusto Leal **Sales Engineer** Source: https://www.conduktor.io/authors/augusto-leal --- # Aurelie Marcuzzo **Solutions Architect** Aurelie is a Solutions Architect at Conduktor, helping teams design and implement Kafka operations at scale. [LinkedIn profile](https://www.linkedin.com/in/aureliemarcuzzo/) Source: https://www.conduktor.io/authors/aurelie-marcuzzo --- # Casey O'Shea **Chief Technology Officer, Atyeti** Casey O'Shea is the Chief Technology Officer at Atyeti, where he leads the design and development of secure, real-time systems for modern enterprises. Source: https://www.conduktor.io/authors/casey-oshea --- # Christoph Schubert **Head of Product** Source: https://www.conduktor.io/authors/christoph-schubert --- # Chuck Larrieu Casias **Solutions Architect** ## Articles by Chuck Larrieu Casias - [Connect to AWS MSK Across VPCs Without Timeouts](https://www.conduktor.io/blog/kafka-msk-cross-network-connectivity) (2026-07-09) — Your AWS MSK bootstrap connects but every broker times out across a VPC or account boundary. Here's why, and the Kafka-aware proxy pattern that fixes it. - [The Surprising Cost of Kafka Partition Waste](https://www.conduktor.io/blog/the-surprising-cost-of-kafka-partition-waste) (2026-06-23) — Most Kafka clusters carry 40 to 70 percent partition waste. On managed Kafka it hits the invoice; on self-managed clusters it sets your broker count. - [Validate JSON in Kafka Without Migrating to Schema Registry](https://www.conduktor.io/blog/bounce-bad-kafka-data-with-json-schema-rules) (2025-09-04) — Block malformed JSON before it breaks Kafka pipelines. Conduktor Trust validates schema, types, and fields—no Schema Registry migration needed. - [Conduktor Gateway on Kubernetes with SNI Routing](https://www.conduktor.io/blog/deploy-conduktor-gateway-with-kubernetes-and-host-based-routing) (2024-11-05) — Step-by-step guide to deploying Conduktor Gateway on Kubernetes with host-based (SNI) routing, TLS certificates, and external Ingress access. Source: https://www.conduktor.io/authors/chuck-larrieu-casias --- # Dr. Stefan Brandner **Vorstand, SupplyOn AG** Source: https://www.conduktor.io/authors/dr-stefan-brandner --- # Emily Lewis **Senior Director, Marketing** Source: https://www.conduktor.io/authors/emily-lewis --- # Emma Humber **Software Engineer** ## Articles by Emma Humber - [Build a Kafka Streaming Platform That Scales Without a Bottleneck](https://www.conduktor.io/blog/5-practices-for-kafka-leaders-to-build-an-efficient-streaming-platform) (2023-06-08) — Centralized Kafka bottlenecks productivity. Decentralized sprawls costs. Build hybrid streaming platforms with multi-tenancy and self-service. Source: https://www.conduktor.io/authors/emma-humber --- # Eric Sammer **CEO and Founder, Decodable** Source: https://www.conduktor.io/authors/eric-sammer --- # Fabian Schmitzer **Sales Engineer** Fabian is a Sales Engineer at Conduktor. Source: https://www.conduktor.io/authors/fabian-schmitzer --- # Florent Ramière **Product Lead** ## Articles by Florent Ramière - [Crypto Shredding in Kafka: GDPR Compliance Without Deletion](https://www.conduktor.io/blog/crypto-shredding-in-kafka-a-cost-effective-way-to-ensure-compliance) (2025-03-18) — Crypto shredding destroys encryption keys to make Kafka data permanently unreadable. Instant GDPR compliance without topic deletion or costly reprocessing. - [Kafka Disaster Recovery: RTO, RPO, and Automated Failover](https://www.conduktor.io/blog/when-disaster-hits-kafka-are-you-ready) (2023-06-17) — Kafka disaster recovery: set RTO/RPO targets, implement multi-region failover, and automate switchover with Conduktor Gateway to eliminate manual errors. - [Kafka Encryption: Field-Level Without the DIY Tech Debt](https://www.conduktor.io/blog/how-to-encrypt-data-in-kafka-without-piling-up-tech-debt) (2023-06-05) — DIY Kafka encryption libraries become multi-language, multi-KMS nightmares. Proxy-layer field-level encryption eliminates tech debt across producers and cons... - [8 Ways to Cut Kafka Costs Without Sacrificing Performance](https://www.conduktor.io/blog/8-new-ways-to-drastically-reduce-your-kafka-costs) (2023-05-30) — Kafka costs exploding? Cut storage, networking, and partition expenses 10x with virtual clusters, compression enforcement, and cold storage. Source: https://www.conduktor.io/authors/florent-ramiere --- # Frances O'Rafferty **Senior Manager, Baringa** Senior Manager at Baringa with 15 years of experience in data and analytics, helping organizations unlock the power of their data. Source: https://www.conduktor.io/authors/frances-orafferty --- # Francesca Scantlebury **Head of People and Talent** ## Articles by Francesca Scantlebury - [How a University Lecturer in China Became Conduktor's Head of Sales Engineering](https://www.conduktor.io/blog/from-academia-to-sales-engineering-a-journey-into-streaming-data) (2025-02-20) — Christoph Schubert's journey from teaching in China to Confluent to Conduktor's Head of Sales Engineering, mastering Kafka and customer success. - [How Maria Went from Languages to Product Design at Conduktor](https://www.conduktor.io/blog/a-product-designer-s-journey-to-conduktor) (2025-02-17) — Maria's path from studying languages in Seoul to designing UX/UI at Conduktor shows that design careers don't require a traditional route. Source: https://www.conduktor.io/authors/francesca-scantlebury --- # François Teychené **Software Engineer** ## Articles by François Teychené - [How to deploy Conduktor Gateway?](https://www.conduktor.io/blog/how-to-deploy-conduktor-gateway) (2023-06-01) — Conduktor Gateway sits between Kafka clusters and apps. Review deployment options, networking patterns, and trade-offs. Source: https://www.conduktor.io/authors/francois-teychene --- # Howard Holton **COO, GigaOm** Howard Holton is COO at GigaOm responsible for the leading technology evangelism, client executive advocacy, and external delivery. Source: https://www.conduktor.io/authors/howard-holton --- # James White **Director of Product** ## Articles by James White - [What 200 Engineers Told Us at Current London 2025](https://www.conduktor.io/blog/current-london-2025-learnings-to-drive-our-roadmap) (2025-06-06) — Current London 2025 insights: data quality failures, migration complexity, developer bottlenecks, and Flink challenges shape Conduktor's roadmap priorities. - [Conduktor Trust Validates Streaming Data Before It Breaks Your AI](https://www.conduktor.io/blog/conduktor-launches-trust-to-guarantee-data-quality-at-the-source) (2025-05-21) — Conduktor Trust validates streaming data quality at ingestion—preventing AI drift, bad analytics, and compliance failures with proactive governance. - [Five Kafka Scaling Problems That Hit Enterprises Hardest](https://www.conduktor.io/blog/five-hidden-kafka-challenges-for-enterprises) (2025-04-15) — Schema registries won't save you. Real Kafka challenges at scale: zombie topics, unclear ownership, legacy integrations. - [SQL Over Kafka: Transforming Real-Time Data Into Instant Insights](https://www.conduktor.io/blog/sql-over-kafka-transforming-real-time-data-into-instant-insights) (2024-12-19) — Query Kafka directly with SQL using Conduktor. Access real-time insights, troubleshoot issues, and optimize operations without data duplication. - [Financial Services Data Platforms Are Fragmented, Siloed, and Complex. Here's How to Fix Them.](https://www.conduktor.io/blog/future-of-financial-services-lies-in-data-key-takeaways-fima-europe) (2024-11-26) — FIMA Europe 2024: data management beats AI as top priority. Break silos, shift quality upstream, expand real-time streaming. - [Building Kafka Tooling That Actually Scales: Security, Governance, and GitOps](https://www.conduktor.io/blog/real-time-data-with-kafka) (2024-07-18) — Build scalable Kafka tooling with GitOps and governance automation. Avoid common pitfalls in real-time data security, costs, and compliance. - [Why Kafka ACLs Break at Scale and What to Use Instead](https://www.conduktor.io/blog/kafka-encryption-and-security-best-practices-for-protecting-data) (2024-07-11) — Kafka's default security fails in production. ACLs become unmanageable, PII leaks, encryption is missing. Here's the fix. - [Automate Kafka Configuration at Scale with GitOps](https://www.conduktor.io/blog/kafka-gitops-for-efficient-scaling) (2024-07-04) — Complex infrastructure calls for automation. GitOps streamlines Kafka configuration and scales deployments efficiently. - [Conduktor 2.0: Free Debugging, Monitoring, and Encryption Tools for Kafka Teams](https://www.conduktor.io/blog/conduktor-2-0-the-new-standard-for-teams-working-with-kafka) (2024-04-11) — Conduktor 2.0 expands free tier with debugging, monitoring, and encryption tools—simplifying Kafka for developers, architects, and platform teams. - [Deploy Conduktor Console on AWS with CloudFormation in 10 Minutes](https://www.conduktor.io/blog/step-by-step-deploy-the-most-advanced-kafka-console-on-aws-in-minutes) (2024-02-19) — Deploy Conduktor Console on AWS in minutes using CloudFormation and ECS Fargate. Works with MSK, Confluent, Redpanda, and Kafka Connect. - [How to Scale Kafka Across Teams Without Losing Control](https://www.conduktor.io/blog/scaling-kafka-securely-for-your-organization) (2023-12-22) — Scale Kafka across teams with federated governance and self-service data platforms. Tackle access management, security, and onboarding bottlenecks. - [Kafka Proxies, Cost Virtualization, and Flink: Takeaways from Current 2023](https://www.conduktor.io/blog/the-future-of-data-streaming-current-2023-big-data-london) (2023-11-16) — Key insights from Current 2023 and Big Data London: Kafka proxies for encryption, cost optimization through virtualization, and Flink's serverless future. - [Four Ways to Test Apache Kafka Applications](https://www.conduktor.io/blog/testing-your-apache-kafka-data-with-confidence-2023) (2023-10-19) — Test Kafka components with Testcontainers, Streams test-utils, chaos engineering, and Conduktor Console. Ensure data quality and resilience at scale. - [Desktop vs Console: Why Centralized Kafka Management Wins](https://www.conduktor.io/blog/why-you-should-consider-migrating-to-conduktor-console) (2023-07-12) — Migrate from Desktop to Console: centralized deployment, RBAC, field-level masking, audit logs, SSO, monitoring—enterprise Kafka UI advantages. Source: https://www.conduktor.io/authors/james-white --- # Jean-Louis Boudart **Head of Customer Success Engineering** ## Articles by Jean-Louis Boudart - [Data Silos Cost $3.1 Trillion Annually. Kafka Alone Won't Fix Them.](https://www.conduktor.io/blog/break-your-data-silos-before-they-break-your-business) (2025-05-02) — Data silos cost $3.1 trillion annually. Break them with Kafka + Conduktor's self-service, discoverability, and granular access controls. - [How to Handle Large Records in Kafka Without Killing Performance](https://www.conduktor.io/blog/beyond-limits-produce-large-records-without-undermining-apache-kafka) (2023-06-19) — Bypass Kafka's 1MB limit without performance hits. Use S3 cold storage with claim check pattern for PDFs, images, and massive batches—seamlessly. Source: https://www.conduktor.io/authors/jean-louis-boudart --- # Johnathan Law **Senior Product Manager** ## Articles by Johnathan Law - [Why Schemaless Kafka Topics Break Production Systems](https://www.conduktor.io/blog/the-hidden-pitfalls-of-kafka-s-schemaless-data) (2025-07-24) — Kafka's schemaless flexibility creates data quality issues. Conduktor Trust validates schemaless topics, blocks bad data, and enforces custom rules. - [Fix Data Quality at the Source, Not After Ingestion](https://www.conduktor.io/blog/proactive-and-reactive-the-two-paths-towards-data-quality) (2025-06-12) — Low-quality data destroys AI and ML outputs. Fix data quality at the source—before problems travel downstream. Source: https://www.conduktor.io/authors/johnathan-law --- # Jonathan Deane **Account Executive, Conduktor** Source: https://www.conduktor.io/authors/jonathan-deane --- # Jordan Roomer **Account Executive** Jordan is an Account Executive at Conduktor, working with engineering teams running Kafka at scale. [LinkedIn profile](https://www.linkedin.com/in/jordanroomer/) Source: https://www.conduktor.io/authors/jordan-roomer --- # Jorge Ruiz **Senior Product Marketing Manager** ## Articles by Jorge Ruiz - [Chaos Test Kafka Without Breaking Production](https://www.conduktor.io/blog/chaos-test-kafka-and-build-resilient-streaming-environments) (2025-07-10) — Simulate broker failures, latency, and corrupted messages without breaking production. Chaos test Kafka with Conduktor's interceptors. - [Why Real-Time Data Sharing Matters](https://www.conduktor.io/blog/why-real-time-data-sharing-matters) (2025-06-26) — Real-time data sharing drives 3x ROI. Explore Kafka-powered use cases in finance, healthcare, logistics—secure partner data exchange with Conduktor. - [What Engineers Actually Struggle With in Kafka: QCon London 2025](https://www.conduktor.io/blog/engineers-pov-on-streaming-qcon-2025-takeaways) (2025-04-11) — What software engineers, architects, and platform teams told us about the real challenges of scaling Kafka—straight from the floor at QCon London 2025. - [Real-Time Data Streaming for Transport and Logistics with Apache Kafka](https://www.conduktor.io/blog/drive-speed-efficiency-and-revenue-growth-with-streaming-data-for-transport-and-logistics) (2024-10-28) — How transport and logistics use Kafka for route optimization, fleet tracking, and why data governance matters at scale. - [Kafka in Financial Services: Architecture Patterns for Compliance, Fraud Detection, and Real-Time Payments](https://www.conduktor.io/blog/how-to-streamline-your-kafka-architecture-and-drive-success-in-financial-services) (2024-10-21) — Financial services use Kafka for fraud detection, real-time payments, and compliance. Build secure streaming architectures. - [Reliable Data Streaming for Retail: Kafka Operations at Scale](https://www.conduktor.io/blog/how-retail-organizations-can-drive-reliable-accurate-data-streaming) (2024-10-15) — Learn how retail organizations can simplify Kafka operations, secure data streaming, and drive growth with Conduktor's enterprise data management platform. - [BYOC, Storage-Compute Separation, and Shift Left Are Reshaping Data Streaming](https://www.conduktor.io/blog/the-biggest-trends-in-data-streaming) (2024-10-11) — BYOC, decoupling compute from storage, and Shift Left strategies dominate data streaming. Insights from Aiven and Conduktor on AI-driven real-time data. Source: https://www.conduktor.io/authors/jorge-ruiz --- # Julien Chanaud **Product Manager** ## Articles by Julien Chanaud - [Map Kafka Costs to Teams and Projects with Chargeback](https://www.conduktor.io/blog/chargeback-attribute-map-kafka-costs-to-your-business) (2025-05-08) — Track Kafka costs by team, project, and environment—not just partitions. Conduktor Chargeback maps service accounts to real business units. - [Multi-Team Kafka Alerting with Granular Ownership and Webhooks](https://www.conduktor.io/blog/enterprise-ready-kafka-alerting-decentralized-control-webhooks-for-multiple-teams) (2025-01-28) — Conduktor's alerts system gives teams ownership over their Kafka notifications with webhook integrations and event-driven alerting. Source: https://www.conduktor.io/authors/julien-chanaud --- # Julien Maillard **Senior Architect, CDC Informatique** Julien est Senior Architect chez CDC (Caisse des Dépôts et Consignations) Informatique, où il pilote la stratégie Kafka et l'adoption du streaming de données à l'échelle de l'entreprise. Source: https://www.conduktor.io/authors/julien-maillard --- # Lorcan Cooke **Customer Success Engineer, Conduktor** Source: https://www.conduktor.io/authors/lorcan-cooke --- # Loretta Schwaebe **Account Executive, DACH, Conduktor** Source: https://www.conduktor.io/authors/loretta-schwaebe --- # Mansi Vaghela **Partner Solutions Architect, AWS** Source: https://www.conduktor.io/authors/mansi-vaghela --- # Manuel Calvé **Head of Partnerships** Manuel is the Head of Partnerships at Conduktor. Source: https://www.conduktor.io/authors/manuel-calve --- # Marcel Boldt **Solutions Architect** Marcel is a Solutions Architect at Conduktor. Source: https://www.conduktor.io/authors/marcel-boldt --- # Maria Jose Lopez Monzon **Scrum Master, SIX** Source: https://www.conduktor.io/authors/maria-jose-lopez-monzon --- # Matt Schumpert **Head of Product, Core, Redpanda** Source: https://www.conduktor.io/authors/matt-schumpert --- # Matt Searle **Engineering Director** Matt Searle is the Director of Engineering at Conduktor, leading and growing our teams of product and platform engineers to unlock the value of data streaming for our customers. Matt brings 25 years of technical experience as a principal engineer and leader in startups and large enterprises - with a career long focus on high-volume data problems and building and scaling up motivated high performing engineering teams. Matt has been working in depth with Apache Kafka since its beginnings, and is passionate about delivering capabilities for all enterprises to unlock the power of data streaming. ## Articles by Matt Searle - [Testing a Kafka Proxy: Taming Millions of Permutations](https://www.conduktor.io/blog/testing-a-kafka-proxy) (2026-06-25) — How we test the Conduktor Gateway Kafka proxy: ~9 million protocol, broker, and client permutations, tamed with pairwise testing and real Kafka clients. - [What's Really Inside Kafka's poll()?](https://www.conduktor.io/blog/kafka-consumer-poll-explained) (2026-06-15) — A Kafka consumer looks simple: a single connection to Kafka, then you poll(). It couldn't be so far from the truth. - [Self-Service Kafka Without the Chaos](https://www.conduktor.io/blog/who-needs-self-service-you-do) (2025-09-25) — Kafka self-service with guardrails: Conduktor enables developer autonomy while preventing data chaos, cost overruns, and compliance risks. - [Why ML Models Fail in Production: The Training Data Mismatch Problem](https://www.conduktor.io/blog/the-hidden-data-gap-that-wrecks-ai-projects) (2025-08-07) — Mismatched training and production data cripples AI models. Real story from a UK lender shows why Conduktor Trust prevents ML pipeline failures. - [How to Attribute Apache Kafka Costs with Chargeback for Accountability & Better Forecasting](https://www.conduktor.io/blog/tackling-apache-kafka-cost-attribution-with-chargeback) (2024-12-05) — Track Kafka costs by team, domain, or application. Attribute ingress/egress usage to optimize infra and forecast spending. Source: https://www.conduktor.io/authors/matt-searle --- # Matthew Scott **Head of Strategic Partnerships & Channels** ## Articles by Matthew Scott - [Conduktor Joins AWS ISV Accelerate Program to Simplify MSK Deployments](https://www.conduktor.io/blog/conduktor-joins-aws-isv-accelerate-program) (2025-10-22) — Conduktor joins AWS ISV Accelerate Program, enabling secure MSK deployments for SageMaker, Redshift—accelerating AI adoption at Swiss Post and beyond. - [Conduktor Partners with LHP to Bring Kafka Governance to Automotive and Aerospace Manufacturing](https://www.conduktor.io/blog/secure-and-govern-real-time-manufacturing-data-with-conduktor-and-lhp) (2025-08-18) — Conduktor partners with LHP to secure and govern streaming data for automotive and aerospace manufacturers using Kafka for IIoT, AI, and analytics. Source: https://www.conduktor.io/authors/matthew-scott --- # Mike Carley **Sales Engineer, Conduktor** Source: https://www.conduktor.io/authors/mike-carley --- # Mirja Hollman **Technical Product Owner, Flix** Source: https://www.conduktor.io/authors/mirja-hollman --- # Nicolas Orban **CEO** Co-founder and CEO of Conduktor. Passionate about making data streaming accessible to every organization. [LinkedIn profile](https://www.linkedin.com/in/nicolas-orban-64b660b8/) ## Articles by Nicolas Orban - [Conduktor Raises $30M Series B to Expand U.S. Operations and Accelerate Product Development](https://www.conduktor.io/blog/conduktor-secures-30-million-series-b-to-lead-the-future-of-streaming-data-management) (2024-11-12) — Conduktor raises $30M in Series B funding led by RTP Global to expand in the U.S. and meet rising demand for secure, real-time data streaming solutions. - [What We Shipped in 2023 and Why It Matters](https://www.conduktor.io/blog/conduktor-retrospective-2023) (2024-01-11) — Conduktor's 2023 journey: launching Console, introducing Gateway proxy, achieving SOC2 Type II, and evolving from developer tool to enterprise platform. - [Claudiu Danaila: From Agency Founder to Head of Design at Conduktor](https://www.conduktor.io/blog/meet-the-team-claudiu-danaila) (2023-08-23) — Claudiu Danaila shares his path from running a design agency to leading design at Conduktor, and why good design is a business differentiator. - [What a Series A Startup Can Actually Do About Work-Life Balance](https://www.conduktor.io/blog/promoting-work-life-balance-in-a-series-a-start-up) (2023-08-09) — How Conduktor built work-life balance at Series A through hybrid working, transparent feedback, and health benefits—without sacrificing startup velocity. - [Aurélie Marcuzzo: From Business Intelligence to Customer Success in a Startup](https://www.conduktor.io/blog/meet-the-team-aurelie-marcuzzo) (2023-06-27) — Meet Aurélie Marcuzzo, Customer Success at Conduktor. From France to London, she onboards customers, tests features, and gives users the love they need. - [What We Changed After Running Four Company Offsites](https://www.conduktor.io/blog/key-learnings-from-our-2023-offsite) (2023-05-01) — Conduktor's CEO Nicolas Orban shares key learnings from our 2023 offsite: quality over quantity, inclusive activities, and transparent communication. Source: https://www.conduktor.io/authors/nicolas-orban --- # Nicole Bouchard **Senior Product Manager** Nicole is a Senior Product Manager at Conduktor specializing in developer tools, infrastructure, and data platforms. Previously at Spotify. [LinkedIn profile](https://www.linkedin.com/in/nicolebouchard/) ## Articles by Nicole Bouchard - [Your Platform Team Can’t Fix Kafka Costs Alone](https://www.conduktor.io/blog/your-platform-team-cant-fix-kafka-costs-alone) (2026-05-19) — Kafka costs keep growing because most orgs lack the ownership model to act on them. Moving to the next stage of cost maturity is the engineering leader's call. - [Why Kafka Costs Keep Rising (Even When Usage Doesn’t)](https://www.conduktor.io/blog/a-better-conversation-about-kafka-costs) (2026-04-30) — Kafka costs are driven by structure and provisioning, not usage. A view from the field on where the savings actually hide. - [Introducing Schema Registry Proxy (Preview)](https://www.conduktor.io/blog/introducing-schema-registry-proxy) (2026-03-24) — Schema registry governance gets deferred because the tooling isn't there. We built Schema Registry Proxy to change that. - [Chaos Testing for Kafka Compliance: DR Evidence That Works](https://www.conduktor.io/blog/chaos-testing-for-kafka-compliance-dr-evidence-that-works) (2026-03-19) — Map chaos experiments to DORA, SOC 2, PCI-DSS, and GDPR requirements. Turn your Kafka resilience testing into the compliance evidence regulators expect. - [Kafka Chaos Tests: What They Teach You That Monitoring Can't](https://www.conduktor.io/blog/what-kafka-chaos-tests-actually-teach-you-that-monitoring-doesnt) (2026-03-12) — Monitoring tells you what broke. Chaos testing tells you what will. Five Kafka system behaviors that only surface under controlled failure. - [Chaos Engineering for Kafka: Test Recovery Before You Need It](https://www.conduktor.io/blog/chaos-engineering-for-kafka-testing-recovery-before-you-need-it) (2026-03-04) — Five tests that validate your Kafka disaster recovery plan before a real outage does. A practical guide for platform engineering teams. - [Kafka Disaster Recovery: The Complete Strategy Beyond Replication](https://www.conduktor.io/blog/kafka-disaster-recovery-the-complete-strategy-beyond-replication) (2026-02-11) — Three operational phases, six technical areas, and a failover runbook. The complete disaster recovery strategy most Kafka teams are missing. - [Why Kafka Platform Health Requires More Than Monitoring](https://www.conduktor.io/blog/why-kafka-platform-health-requires-more-than-monitoring) (2026-02-02) — Platform health differs from infrastructure health. Observability alone won't cut it. Here are the questions that reveal your Kafka platform's real state. - [Kafka Schema Governance: From Chaos to Confidence in 4 Steps](https://www.conduktor.io/blog/kafka-schema-governance-from-chaos-to-confidence) (2026-01-14) — Fix Kafka schema chaos with a 4-step governance framework: visibility, audit, accountability, enforcement. Used by Swiss Post across hundreds of topics. Source: https://www.conduktor.io/authors/nicole-bouchard --- # Paul Feldman **Regional Sales Director, East and Central** ## Articles by Paul Feldman - [Why Enterprise AI Struggles With ROI and Data Control](https://www.conduktor.io/blog/ai-expectations-realities-and-risks) (2025-04-17) — Enterprise leaders tackle AI ROI, GenAI governance, and data controls. Conduktor execs reveal why shift-left alone won't protect LLM inputs. - [What Enterprise Data Leaders Actually Worry About: Silos, Security, and Legacy Systems](https://www.conduktor.io/blog/from-promise-to-practice-insights-from-the-millennium-alliance-enterprise-ai-maturity-and-transformation-assembly) (2025-03-18) — Enterprise AI Assembly insights: breaking data silos across regions, federated governance, shift-left security, and mainframe modernization challenges. Source: https://www.conduktor.io/authors/paul-feldman --- # Poria Zarrinhonar **Renewals Account Manager, Conduktor** Source: https://www.conduktor.io/authors/poria-zarrinhonar --- # Quentin Packard **SVP of Sales** VP Sales at Conduktor, where he helps business leaders solve complex data challenges within their organization. Previously at HashiCorp and Splunk, he has driven global change in strategy and sales. As an Advisory Board Member at InspireCIO, Quentin has spent over four years guiding CIOs through the evolving tech landscape and fostering innovation. ## Articles by Quentin Packard - [AI Governance Starts at the Data Layer](https://www.conduktor.io/blog/secure-data-operations-in-the-time-of-ai) (2025-06-19) — AI agents now query live operational data directly. A control and audit layer brings AI data governance to every agent-to-data interaction. - [What Dash 2025 Revealed About AI Observability Security](https://www.conduktor.io/blog/48-hours-at-dash-security-ai-and-the-observability-paradox) (2025-06-18) — Dash 2025 revealed AI observability risks—from exposed PII to agent telemetry attacks. How Conduktor + Datadog secure LLMs at scale. - [Data Streaming Strategy for 2025: What Industry Experts Recommend](https://www.conduktor.io/blog/your-2025-data-streaming-strategy-blueprint-insights-from-industry-experts) (2025-01-28) — 2025 data streaming strategy: experts from Atyeti, GigaOm discuss data maturity, AI readiness, governance, and modernizing your tech stack. - [GenAI Adoption Lessons from Resolve.ai: Data Quality, Use Cases, and Build vs. Buy](https://www.conduktor.io/blog/taking-the-genai-bet-what-innovators-have-learned) (2025-01-02) — GenAI adoption insights from Resolve.ai CEO Spiros Xanthos: data quality strategies, strategic use cases, and buy vs. build decisions for AI success. - [Why Streaming Data Quality Requires Proactive Monitoring, Not Reactive Fixes](https://www.conduktor.io/blog/mastering-data-quality-in-streaming) (2024-11-18) — Master data quality in streaming with proactive monitoring, use-case checks, and real-time alerts. Build trust in data for AI and real-time decisions. Source: https://www.conduktor.io/authors/quentin-packard --- # Rehm Kharal **Account Executive, Conduktor** Source: https://www.conduktor.io/authors/rehm-kharal --- # Richa Khandelwal **Senior Engineering Leader, Airbnb** Source: https://www.conduktor.io/authors/richa-khandelwal --- # Robert Mitchell **Strategic Account Executive, Conduktor** With 20+ years of experience in enterprise software and infrastructure, Robert Mitchell helps organizations unlock business value through cloud, DevOps, and open-source technologies. Source: https://www.conduktor.io/authors/robert-mitchell --- # Romain Lecomte **Staff Software Engineer** ## Articles by Romain Lecomte - [Partner Zones: Share Kafka Data with External Partners Without Replicating Clusters](https://www.conduktor.io/blog/partner-zones-secure-kafka-data-sharing-without-duplication) (2025-02-07) — Share Kafka streams with external partners through isolated access zones—no cluster duplication or complex replication required. Source: https://www.conduktor.io/authors/romain-lecomte --- # Ron Kapoor **Technical Product Marketing Manager** Ron is Technical Product Marketer/GTM and Dev Rel at Conduktor, bringing experience from both financial services and startups. When he's not creating content or defining positioning and messaging for Conduktor's suite of products, he loves to engage with the community! [LinkedIn profile](https://www.linkedin.com/in/ron-kapoor/) ## Articles by Ron Kapoor - [We Deleted a Kafka Topic. Which App Broke?](https://www.conduktor.io/blog/we-deleted-a-kafka-topic-which-app-broke) (2026-09-10) — Understand your Kafka cluster for effective change management and troubleshooting. Stream Lineage in Conduktor Console maps it from your ACLs. - [How to Approach Multi-Tenancy in Kafka](https://www.conduktor.io/blog/how-to-do-multi-tenancy-in-kafka) (2026-08-05) — Kafka has no built-in tenants. There are three ways to add them: prefixes and ACLs, a cluster per tenant, or virtual clusters. Here's when each fits. - [Best Tools for Monitoring Kafka Consumer Lag (2026)](https://www.conduktor.io/blog/best-kafka-consumer-lag-monitoring-tools) (2026-07-30) — Burrow, KMinion, Datadog, CloudWatch, AKHQ, Kafbat and more, compared on lag detail, time lag, and what you can do when the alert fires. - [Kafka Migration Tools: What Each One Actually Moves](https://www.conduktor.io/blog/kafka-migration-tools) (2026-07-23) — Asking which tool migrates a Kafka cluster is like asking which tool builds a house. A migration is five jobs, and each layer of the stack has its own tools. - [Kafka Schema Registry: Open by Default](https://www.conduktor.io/blog/kafka-schema-registry-open-by-default) (2026-07-10) — By default, the Kafka schema registry is unguarded. How do you get true per-subject control without half-baked engineering workarounds? - [Kafka Guardrails Gate. Templates Guide.](https://www.conduktor.io/blog/kafka-guardrails-gate-templates-guide) (2026-06-23) — Enforcement tells Kafka teams what they can't do, never what good looks like. Configuration templates are the other half of self-service provisioning. - [Why Copy a Kafka Topic Just to Filter It?](https://www.conduktor.io/blog/why-copy-a-kafka-topic-just-to-filter-it) (2026-06-18) — You don't have to. Topic views, now GA in Conduktor Gateway, give each consumer a filtered, projected view of a Kafka topic at read-time without copies. - [What We Learned at Current 2026](https://www.conduktor.io/blog/what-we-learned-at-current-2026) (2026-05-27) — We were at Current 2026 in London last week. Between booth conversations and recorded interviews on the floor, here's what stood out. - [The Apache Kafka Security Playbook: Beyond TLS and ACLs](https://www.conduktor.io/blog/apache-kafka-security-playbook) (2026-05-11) — TLS only goes so far. The real Kafka security gaps: payload encryption, service accounts left rotten, ACLs that fall behind reality, and scattered audit logs. - [Set Up a Kafka Platform with an AI Agent](https://www.conduktor.io/blog/set-up-a-kafka-platform-with-an-ai-agent) (2026-03-20) — Install Kafka, configure governance, and onboard developers with one AI conversation. Open-source Conduktor skill for Claude Code, Cursor, and 30+ tools. - [Ground Control to On-Call: Kafka Debugging Has Landed](https://www.conduktor.io/blog/what-debugging-kafka-should-actually-look-like) (2026-02-17) — No more juggling 7 CLI tools. A walkthrough of debugging a Kafka consumer lag incident with Conduktor Console. - [Houston, We Have 7 CLI Tools and Zero Answers](https://www.conduktor.io/blog/houston-we-have-7-cli-tools-and-zero-answers) (2026-02-06) — You've accepted that debugging matters more than dashboards. Then you open a terminal and realize Kafka's native tools were never built for this. - [Why Every Kafka Incident Ends with "Restart It"](https://www.conduktor.io/blog/why-every-kafka-incident-ends-with-restart-it) (2026-02-04) — Teams invest in Kafka observability thinking they're prepared. Then an incident hits, and they realize dashboards don't tell you which room is on fire. - [IBM Acquires Confluent for $11B: What It Means for Kafka Teams](https://www.conduktor.io/blog/ibm-confluent-11-billion-says-streaming-isn-t-optional) (2025-12-10) — IBM's acquisition of Confluent validates streaming as critical infrastructure. Here's what changes for organizations running Kafka. Source: https://www.conduktor.io/authors/ron-kapoor --- # Sarah Polan **Chief Technical Officer, Redhat** Source: https://www.conduktor.io/authors/sarah-polan --- # Shadan Ashfaie **Enterprise PMO** Source: https://www.conduktor.io/authors/shadan-ashfaie --- # Shuchi Agarwal **Head of AI Execution, SMBC Group** Source: https://www.conduktor.io/authors/shuchi-agarwal --- # Siddhanth Lathar **Solutions Architect** ## Articles by Siddhanth Lathar - [AWS MSK Authentication: OIDC and mTLS Without Touching the Broker](https://www.conduktor.io/blog/kafka-msk-authentication-oidc-mtls) (2026-07-21) — AWS MSK clients can only use IAM, SCRAM, or ACM-bound mTLS. Here's how to use OIDC and mTLS with AWS MSK by terminating the scheme at a Kafka proxy. - [Connect Kafka Across VPCs Without Extra Peering](https://www.conduktor.io/blog/connect-kafka-across-vpcs-without-extra-peering) (2026-07-02) — Reaching Kafka across VPCs isn't a networking problem, it's an addressing one. Why peering, transit gateways, and per-broker NLBs miss it. - [FedRAMP High for Kafka Without Replatforming](https://www.conduktor.io/blog/fedramp-high-for-kafka-without-replatforming) (2026-06-08) — Your Kafka cluster isn't FedRAMP High and replatforming everything is a no-go? Encrypt and sign records at the proxy layer inside your own environment. Source: https://www.conduktor.io/authors/siddhanth-lathar --- # Sreeni Gadhar **VP of Global Data and Analytics, Marsh McLennan** Source: https://www.conduktor.io/authors/sreeni-gadhar --- # Stéphane Derosiaux **CPTO** Co-founder and CTO of Conduktor. 15+ years in software and data engineering, building large-scale streaming systems. [LinkedIn profile](https://www.linkedin.com/in/stephane-derosiaux/) ## Articles by Stéphane Derosiaux - [Designing Kafka Applications for Data Privacy Compliance](https://www.conduktor.io/blog/designing-kafka-applications-for-data-privacy-compliance) (2026-07-27) — Stop coding privacy compliance into every Kafka app. Put encryption, erasure, and access control in one enforcement layer between your apps and Kafka. - [Benchmarking the Conduktor Gateway: what a Kafka proxy really costs](https://www.conduktor.io/blog/conduktor-gateway-performance-benchmark) (2026-07-21) — Real numbers over SASL and TLS, the way you actually run Kafka in production: what a proxy in the hot path does to your throughput and your p99. - [Bring Your Own Encryption to Managed Kafka](https://www.conduktor.io/blog/kafka-provider-independence-gateway-encryption) (2026-07-18) — Encrypt Kafka at the proxy with keys that stay in your KMS: your provider can't read what you protect, and your encryption isn't tied to one vendor. - [Terraform Your Whole Kafka Platform](https://www.conduktor.io/blog/kafka-terraform-providers) (2026-07-13) — Kafka Terraform providers don't manage the same layer: the cluster, the topics, or the platform around them. Where Mongey, Confluent and Conduktor fit. - [One Kafka Control Layer for DORA, GDPR and IT-Grundschutz](https://www.conduktor.io/blog/dora-gdpr-it-grundschutz-kafka) (2026-07-11) — DORA, GDPR and IT-Grundschutz often map to the same Kafka controls: encryption, erasure, access, audit and resilience. Build one control layer instead of thr... - [SNI Routing vs Port Routing in a Kafka Proxy](https://www.conduktor.io/blog/kafka-proxy-sni-routing-vs-port-routing) (2026-07-08) — A Kafka proxy must give every client a routable address for every broker. Port-per-broker breaks at scale; SNI routing is the production answer, at a cost. - [What a Kafka Proxy Can Do: From Routing to Enforcement](https://www.conduktor.io/blog/what-a-kafka-proxy-can-do) (2026-06-30) — A Kafka proxy is not a reverse proxy. It speaks the Kafka wire protocol, so it can become the one place security, governance, and policy live. - [Kafka Cost Optimization: The Layer Above Broker Tuning](https://www.conduktor.io/blog/kafka-cost-optimization-above-broker-tuning) (2026-06-29) — Kafka cost optimization usually stops at broker tuning. The usage layer above (fan-out, partition waste, chargeback) is where the real savings live. - [Most Kafka Guardrails Don't Protect Your Data](https://www.conduktor.io/blog/kafka-guardrails-data-protection) (2026-06-27) — A Kafka guardrail in a wiki, CI, or Terraform only holds if the request goes through it. Your data is only as protected as what can reject it on the wire. - [If Kafka Credentials Leak, Does the Attacker Get Plaintext or Ciphertext?](https://www.conduktor.io/blog/kafka-credentials-leak-plaintext-or-ciphertext) (2026-06-12) — TLS, disk encryption, and BYOK each stop a different attacker. None of them stops a leaked Kafka credential from reading topics in plaintext. - [Good Kafka Is Boring Kafka](https://www.conduktor.io/blog/good-kafka-is-boring-kafka) (2026-06-11) — A regulated bank scaled Kafka from 40 to 160 apps without growing the platform team. The lever wasn't the tech, it was governed self-service. - [How to Isolate Non-Prod Kafka with Virtual Clusters](https://www.conduktor.io/blog/how-to-isolate-non-prod-kafka-with-virtual-clusters) (2026-06-09) — Hard network segregation between non-prod environments is expensive. How Kafka virtual clusters give security teams the same isolation on one cluster. - [Ownership → Accountability → Trust → Self-Service](https://www.conduktor.io/blog/no-kafka-data-platform-without-ownership) (2026-06-09) — Ownership is the first domino of a real Kafka data platform: accountability, trust, autonomy, self-service, scale, adoption. Why federated ownership makes ev... - [Kafka Partitions are the wrong ordering abstraction. Keys are.](https://www.conduktor.io/blog/kafka-partitions-are-the-wrong-ordering-abstraction-keys-are) (2026-06-01) — Confluent deprecated Parallel Consumer and points to Share Consumers. Different problem: in event-driven systems, the key is the real unit of work. - [HIPAA 2026 and Kafka: Securing ePHI Streams](https://www.conduktor.io/blog/kafka-hipaa-compliance) (2026-05-25) — The 2026 HIPAA Security Rule mandates encryption, MFA, and annual asset inventory. What it breaks in a typical Kafka deployment, and how to fix it. - [Rethinking Kafka Security for HIPAA 2026](https://www.conduktor.io/blog/rethinking-kafka-security-for-hipaa-2026) (2026-05-25) — Rethinking Kafka security for HIPAA 2026. Why streaming breaks the 'at rest' abstraction the rule was written around, and what to fix in the architecture. - [Kafka 4.3: OAuth assertions, share groups, tiered bootstrap](https://www.conduktor.io/blog/kafka-4-3-oauth-share-groups-tiered-bootstrap) (2026-05-23) — Apache Kafka 4.3 shipped May 22, 2026. The three KIPs that matter in production: OAuth client assertions, share groups, and tiered bootstrap. - [Conduktor MCP + Skills for AI Agents](https://www.conduktor.io/blog/conduktor-mcp-and-skills-for-ai-agents) (2026-05-17) — How Conduktor MCP and Skills let AI agents work with your live Kafka. Patterns for platform onboarding, dev team self-service, and on-demand insights. - [librdkafka vs Java Client: 10 Config Defaults That Differ](https://www.conduktor.io/blog/librdkafka-vs-java-client) (2026-05-05) — Python confluent-kafka and Java share config names but not behavior. 10 librdkafka differences tested on a KRaft cluster, with source code and real output. - [The Data Sharing Maturity Ladder](https://www.conduktor.io/blog/the-data-sharing-ladder) (2026-04-15) — Data sharing maturity has four levels: keep data streaming, expose it safely, create partner value, then package it as sellable data products. - [Kafka in Production: 11 Pitfalls You Should Avoid](https://www.conduktor.io/blog/kafka-production-pitfalls) (2026-03-30) — 11 Kafka behaviors we found helping customers, then validated in the source code. Silent safety bypasses, configs that lie, compression traps. - [Building a Data Lake with Kafka and S3 (Without Breaking It)](https://www.conduktor.io/blog/building-a-data-lake-with-kafka-and-s3) (2026-03-11) — One corrupt Parquet file froze a data lake for 6 hours. How Conduktor Gateway prevents bad data from reaching S3 with CEL validation rules. - [How Conduktor Gateway Reduces Kafka DR from Hours to Minutes](https://www.conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes) (2026-03-02) — Conduktor Gateway cuts Kafka DR coordination overhead. One API call switches all clients to the secondary cluster. Cut execution from hours to minutes. - [Stream Real-Time WebSocket Data to Apache Kafka with Our New Source Connector](https://www.conduktor.io/blog/kafka-connect-websocket-source-connector) (2026-02-24) — An open-source Kafka Connect connector for WebSocket streams. Auto-reconnect, auth support, and configurable buffering. Get started in 5 minutes. - [Enterprise Kafka: Control Without Bottlenecks](https://www.conduktor.io/blog/enterprise-kafka-management) (2026-02-09) — Enterprise Kafka management needs automated governance, not slow approval processes. Control multi-cluster environments without creating bottlenecks. - [AI for Kafka Operations](https://www.conduktor.io/blog/ai-kafka-operations) (2026-02-07) — AI for Kafka operations makes context instantly available. Reduce MTTR from hours to minutes with natural-language diagnostics and MCP. - [Capture OpenTelemetry Signals in Apache Kafka with Our New OTLP Source Connector](https://www.conduktor.io/blog/kafka-connect-opentelemetry-source-connector) (2026-02-06) — Stream traces, metrics, and logs into Kafka topics for durable storage, replay, and custom analytics. Open source, Apache License 2.0. - [Stop Calling Your Kafka Topics Data Products](https://www.conduktor.io/blog/kafka-data-products) (2026-02-03) — A Kafka topic is shared mutable state: anyone writes, anyone reads, nobody answers for it. A data product is the interface you wrap around it. - [Safeguard Kafka from Timestamp-Based Data Loss](https://www.conduktor.io/blog/safeguard-kafka-from-timestamp-based-data-loss) (2026-02-03) — Producer timestamps can trigger immediate retention deletion. Configure timestamp validation, use LogAppendTime, or enforce bounds at the gateway level. - [Kafka Latency: Measuring p99 & p999 Tails](https://www.conduktor.io/blog/kafka-latency) (2026-01-30) — Kafka latency averages hide tail problems. Optimize p99 and p999, not p50, and measure end-to-end from producer commit to consumer processing. - [Kafka DR: Why Replication Isn't the Hard Part](https://www.conduktor.io/blog/kafka-dr-why-replication-isnt-the-hard-part) (2026-01-27) — PagerDuty's 9-hour Kafka outage reveals the real DR problem - coordinating 47 services during crisis. Why replication alone won't save you. - [Kafka Policy Enforcement: Code Over Docs](https://www.conduktor.io/blog/kafka-policy-enforcement) (2026-01-21) — Kafka policies in Confluence are suggestions. CEL-based enforcement at the API layer prevents bad configs before they reach production. - [Ingest gRPC Streaming Data into Apache Kafka with Our New Source Connector](https://www.conduktor.io/blog/kafka-connect-grpc-source-connector) (2026-01-19) — Stream data from gRPC server endpoints directly into Kafka topics with our open-source Kafka Connect source connector. Zero custom code, just configuration. - [Kafka Compliance: GDPR, SOC2, HIPAA, DORA](https://www.conduktor.io/blog/kafka-security-compliance) (2026-01-14) — GDPR, SOC2, HIPAA, and DORA compliance for Kafka infrastructure. Generate audit evidence in minutes with continuous automated controls. - [Kafka Control Plane: API-Driven Management](https://www.conduktor.io/blog/kafka-control-plane) (2026-01-06) — Your Kafka data plane is fast and resilient. Your control plane—provisioning, access, config—probably runs on scripts and tribal knowledge. - [Automating Kafka Audit Evidence](https://www.conduktor.io/blog/kafka-audit-automation) (2025-12-30) — Manual Kafka audit prep means rebuilding the evidence before every audit. Generate compliance evidence continuously instead, as a byproduct of operations. - [Kafka Data Contracts: A Schema Is Not a Contract](https://www.conduktor.io/blog/kafka-data-contracts) (2025-12-23) — A schema tells you the shape of your data. A contract stops a producer from breaking every consumer downstream. The difference, and where to enforce it. - [Kafka Alerting: Fix Alert Fatigue](https://www.conduktor.io/blog/kafka-alerting) (2025-12-16) — Kafka alert fatigue is a design problem, not a sensitivity problem. Build tiered alerts where 95% of pages result in action, not silence. - [IBM's $11B Confluent Acquisition: What It Means for Kafka Users](https://www.conduktor.io/blog/ibm-confluent-acquisition-kafka-future) (2025-12-15) — Strategic analysis of IBM's Confluent acquisition. What changes for enterprise Kafka deployments and architecture decisions. - [Rolling Restarts Without Downtime](https://www.conduktor.io/blog/rolling-restarts-without-downtime) (2025-12-10) — Zero-downtime Kafka rolling restarts. Pre-flight checks, ISR verification, controlled shutdown, and KRaft controller order. - [Kafka Costs: Stop Overpaying by 30-50%](https://www.conduktor.io/blog/kafka-costs) (2025-12-09) — Teams overpay for Kafka 30-50% through over-provisioning and topic sprawl. Measure per-team usage and right-size clusters to cut costs. - [Testing Kafka Applications: Testcontainers, Embedded Kafka, and Mocks](https://www.conduktor.io/blog/testing-kafka-testcontainers-embedded-mocks) (2025-12-09) — Choose the right Kafka testing strategy. Testcontainers for integration tests, MockProducer for unit tests, TopologyTestDriver for Streams. - [A Practical Guide to Kafka Consumer Lag Alert Thresholds](https://www.conduktor.io/blog/kafka-consumer-lag-alerting-thresholds-guide) (2025-12-08) — Stop alert fatigue from consumer lag metrics. Offset vs time-based lag, per-workload thresholds, and rate-of-change detection. - [Why Enterprise AI Projects Stall on Data Access, Not Models](https://www.conduktor.io/blog/the-hidden-architecture-problem-behind-enterprise-ai) (2025-12-01) — Enterprise AI fails due to data access bottlenecks, multi-hop pipelines, and late governance. SMBC, Marsh McLennan leaders share real-world solutions. - [Kafka Encryption: Transit, Rest, End-to-End](https://www.conduktor.io/blog/kafka-encryption) (2025-11-25) — Kafka encryption has three layers: TLS in transit, disk encryption at rest, end-to-end for payload protection. Match each to your threat model. - [Kafka Tiered Storage: Infinite Retention Without Infinite Disks](https://www.conduktor.io/blog/kafka-tiered-storage-infinite-retention) (2025-11-21) — Configure Kafka tiered storage with S3 to reduce storage costs 3-9x. Broker config, common errors, and performance tradeoffs. - [Kafka Automation Platform: Self-Serve by Policy](https://www.conduktor.io/blog/kafka-automation-platform) (2025-11-18) — Manual Kafka provisioning turns platform teams into a ticket queue and makes developers wait. Policy-based automation gives the team their time back. - [Kafka Multi-Cloud: AWS, Azure, and GCP](https://www.conduktor.io/blog/kafka-multi-cloud-management) (2025-11-09) — Multi-cloud Kafka across AWS, Azure, and GCP needs unified management. One control plane for visibility, policies, and operations everywhere. - [Stop Over-Partitioning: You Don't Need 100 Partitions](https://www.conduktor.io/blog/stop-over-partitioning-kafka) (2025-11-08) — Over-partitioned Kafka topics waste memory, slow failovers, and complicate rebalancing. The actual formula for right-sizing partitions based on throughput. - [Kafka's Real Problem Is Governance, Not Speed](https://www.conduktor.io/blog/the-new-era-of-streaming) (2025-11-06) — Most Kafka clusters push less than 1MB/s. The challenge isn't throughput—it's building the control plane that makes streaming usable at enterprise scale. - [Why Every Streaming Vendor Is Building a Kafka Proxy](https://www.conduktor.io/blog/the-great-confluence-of-streaming) (2025-11-05) — Streaming proxies enable governed self-service and AI-ready data at scale. Why Kafka control planes from Confluent, Kong, and Conduktor are converging. - [Governed Kafka Self-Service](https://www.conduktor.io/blog/governed-kafka-self-service) (2025-11-02) — Self-service Kafka without governance is chaos. Policy-based automation gives developers autonomy while enforcing organizational standards. - [Kafka Offset Management: How Consumer Groups Track Progress](https://www.conduktor.io/blog/kafka-offset-management-consumer-commit-guide) (2025-11-01) — Understand Kafka offset commits, auto-commit pitfalls, CommitFailedException, and offset reset behavior. Stop losing messages. - [Kafka Access Management: Fix ACL Sprawl](https://www.conduktor.io/blog/kafka-access-management) (2025-10-28) — Kafka ACL sprawl makes permissions unknowable. Application-based access with periodic reviews replaces manual ACL-per-service-account chaos. - [5 Sessions Worth Your Time at Current New Orleans 2025](https://www.conduktor.io/blog/five-can-t-miss-sessions-at-current-new-orleans-2025) (2025-10-27) — Current New Orleans 2025 highlights: OpenAI's StreamLink, Kafka protocol deep-dive, Notion's event stack, KIP-1163 diskless topics, and JPMorgan's strategy. - [Kafka Security: Access Control That Scales](https://www.conduktor.io/blog/kafka-security) (2025-10-21) — Kafka security goes beyond TLS encryption. Scalable access control, ACL management, and provable permissions for production environments. - [Kafka Exactly-Once: When It Works and When It Doesn't](https://www.conduktor.io/blog/exactly-once-semantics-when-it-works) (2025-10-15) — Kafka's exactly-once works within cluster boundaries. Idempotent producers, transactions, and the hard limits you need to understand. - [Kafka Security Best Practices](https://www.conduktor.io/blog/kafka-security-best-practices) (2025-10-14) — Kafka security best practices fail without enforcement. Automate TLS, ACLs, and encryption policies so compliance is built-in, not bolted-on. - [Kafka Proxy: Beyond ACLs for Governance](https://www.conduktor.io/blog/kafka-proxy) (2025-10-05) — Kafka proxies enforce field-level filtering, data masking, and rate limiting that native ACLs can't. Apply governance at the wire level. - [Kafka Governance: Automated Guardrails Over Wiki Policies](https://www.conduktor.io/blog/kafka-governance) (2025-09-27) — Kafka governance fails when policies live in wikis. Automated guardrails enforce naming, retention, and schema rules before incidents reach production. - [Kafka Cluster Management: Beyond SSH](https://www.conduktor.io/blog/kafka-cluster-management) (2025-09-19) — Managing 10+ Kafka clusters via SSH and scripts wastes senior engineering time. Centralized tooling gives visibility and control at scale. - [Kafka Monitoring: 10 Metrics That Matter](https://www.conduktor.io/blog/apache-kafka-monitoring) (2025-09-12) — Stop drowning in 200+ Kafka broker metrics. Focus on the 10 signals that actually predict outages, consumer lag, and cluster instability. - [Kafka as a Database: When to Use Compacted Topics for State](https://www.conduktor.io/blog/kafka-compacted-topics-as-database) (2025-09-10) — Use Kafka compacted topics as a lightweight state store. Log compaction configuration, query limitations, and when to choose a real database instead. - [Kafka Observability: Beyond Monitoring](https://www.conduktor.io/blog/kafka-observability) (2025-09-05) — Kafka monitoring shows what's broken. Observability shows why. Combine metrics, traces, and context to diagnose root causes in minutes. - [Kafka Testing: Beyond Production](https://www.conduktor.io/blog/kafka-testing) (2025-08-29) — Kafka testing means validating schemas, configs, and consumer logic before production. Build confidence with contract tests and local clusters. - [Kafka Data Quality: A Schema Registry Isn't Enough](https://www.conduktor.io/blog/kafka-data-quality) (2025-08-26) — By the time a consumer reads a bad record, it is already durable, replicated, and wrong everywhere. Why Kafka data quality has to start at write time. - [Transactional Outbox: Database-Kafka Consistency](https://www.conduktor.io/blog/transactional-outbox-pattern-database-kafka) (2025-08-25) — Solve the dual-write problem with the transactional outbox pattern. PostgreSQL setup, Debezium CDC, and idempotent consumers. - [Kafka Performance: Measure Before You Tune](https://www.conduktor.io/blog/kafka-performance) (2025-08-19) — Kafka performance is 80% understanding your workload, 20% tuning configs. Measure bottlenecks before changing partition counts or batch sizes. - [Custom Partitioners: When Default Hashing Isn't Enough](https://www.conduktor.io/blog/kafka-custom-partitioners-beyond-default-hashing) (2025-08-13) — Build custom Kafka partitioners for geographic routing, priority lanes, and hot key distribution. Java implementation with production-tested patterns. - [The Real Problems with Apache Kafka: 10,000 Forum Posts Analyzed](https://www.conduktor.io/blog/apache-kafka-what-10-000-forum-posts-reveal) (2025-08-11) — 10,000+ Kafka forum posts reveal the truth: connector crashes, auth nightmares, schema failures, and cryptic errors plague production deployments. - [Bridging Operational and Analytical Data with Kafka](https://www.conduktor.io/blog/bridging-gap-between-operational-and-analytical-data) (2025-08-01) — Kafka bridges the operational-analytical data divide in real time. Eliminate batch ETL lag and deliver fresh streaming data to analytics and AI systems. - [Escaping Vendor Lock-In: Kafka Portability Strategies](https://www.conduktor.io/blog/kafka-vendor-lock-in-portability-strategies) (2025-07-08) — Avoid Kafka vendor lock-in with portable configurations and abstraction layers. Code examples for MSK, Confluent, and self-managed. - [Broker Won't Start: Common Kafka Startup Failures](https://www.conduktor.io/blog/broker-wont-start-common-failures-fixes) (2025-06-19) — Kafka broker startup failures with real error messages and fixes. Cluster ID mismatch, port conflicts, disk issues, and config errors. - [Kafka Network Partitions and Split-Brain Failures](https://www.conduktor.io/blog/kafka-network-partition-split-brain-failures) (2025-06-13) — Understand Kafka network partition failures, split-brain scenarios, and unclean leader election. ISR shrinkage and data loss prevention. - [Strimzi on Kubernetes: From Zero to Production Kafka](https://www.conduktor.io/blog/strimzi-kubernetes-zero-to-production) (2025-04-19) — Deploy production-grade Kafka on Kubernetes with Strimzi. Separate node pools, KRaft mode, and the configuration choices that matter. - [How to Monetize Real-Time Data](https://www.conduktor.io/blog/own-the-stream-power-the-model-monetizing-real-time-data-in-the-ai-era) (2025-04-15) — Data monetization starts with real-time streams you own, govern, and trust. Learn how companies turn first-party streaming data into revenue and AI advantage. - [Streaming at Scale: Don't Let Data Chaos Derail Your Growth](https://www.conduktor.io/blog/streaming-at-scale-don-t-let-data-chaos-derail-your-growth) (2025-03-27) — Scale streaming data with governance and quality controls. Shift left to prevent bad data, ensure AI success, and avoid architectural chaos. - [Kafka and Flink Are the Infrastructure for AI Agents](https://www.conduktor.io/blog/ai-agents-at-scale-the-critical-role-of-kafka-and-flink) (2025-03-25) — AI agents need real-time context to act autonomously. Discover how Kafka + Flink power adaptive, intelligent systems at Netflix and beyond. - [Audit Logging in Kafka: Who Did What and When](https://www.conduktor.io/blog/kafka-audit-logging-compliance-forensics) (2025-03-24) — Configure Kafka audit logging for SOC2, HIPAA, and PCI-DSS compliance. Authorizer logging and forensic analysis techniques. - [80% of Enterprise Data Sits Unused. Real-Time Activation Changes That.](https://www.conduktor.io/blog/your-dark-data-is-a-goldmine-if-you-activate-it-in-real-time) (2025-03-21) — Most enterprise data never drives a decision. Real-time data activation transforms idle data into competitive advantage. - [Avro Schema Evolution: Compatibility Guide](https://www.conduktor.io/blog/schema-evolution-avro-compatibility-guide) (2025-03-15) — Master Avro schema evolution with BACKWARD vs FORWARD compatibility modes. Compatibility matrix, curl commands for Schema Registry, and what breaks. - [Real-Time AI Requires Kafka and Guardrails](https://www.conduktor.io/blog/how-to-succeed-with-ai-kafka-and-ai-guardrails-are-non-negotiable) (2025-03-14) — Discover how Kafka and Flink power real-time AI agents, enabling autonomous decision-making, adaptive intelligence, and scalable data streaming. - [Dead Letter Topics: Handling Poison Pills](https://www.conduktor.io/blog/dead-letter-topics-handling-poison-pills) (2025-02-23) — Implement Kafka dead letter queues to isolate poison pill messages. Java error handling patterns, retry strategies, DLT monitoring, and reprocessing workflows. - [Kafka Isn't a Queue: Stop Designing It Like One](https://www.conduktor.io/blog/kafka-isnt-a-queue) (2025-02-21) — Queues delete messages on ack. Kafka keeps a replayable log. Why the difference drives your topic design, and what Kafka 4.2 share groups (KIP-932) change. - [Your First Kafka Producer: Python, Java, and Go Examples](https://www.conduktor.io/blog/first-kafka-producer-python-java-go) (2025-01-28) — Build your first Kafka producer in Python, Java, and Go. Working code examples, common connection errors, and essential configuration for reliable message de... - [Stop Building Kafka Encryption Libraries](https://www.conduktor.io/blog/stop-building-kafka-encryption-libraries) (2025-01-25) — Client-side encryption creates ungovernable sprawl. Proxy-layer encryption centralizes control and makes Kafka a dumb pipe. - [JVM Tuning for Kafka Brokers: G1GC vs ZGC in Production](https://www.conduktor.io/blog/kafka-jvm-tuning-g1gc-vs-zgc-production) (2024-12-10) — Configure G1GC and ZGC for Kafka brokers. Heap sizing, pause time targets, and when to switch collectors in production. - [Kafka Headers: Metadata Without Polluting Your Payload](https://www.conduktor.io/blog/kafka-headers-metadata-without-polluting-payload) (2024-10-10) — Use Kafka headers for distributed tracing, content-based routing, and audit metadata without modifying payloads. Java and Python code examples included. - [Kafka Team Structure: Platform Team vs Embedded Model](https://www.conduktor.io/blog/kafka-team-structure-platform-embedded) (2024-10-03) — Platform teams centralize expertise but bottleneck delivery. Embedded engineers move fast but create chaos. Here's how to choose. - [Field-Level Encryption in Kafka: Beyond TLS](https://www.conduktor.io/blog/kafka-field-level-encryption-beyond-tls) (2024-09-07) — TLS protects data in transit, but brokers still see plaintext. Field-level encryption protects PII at rest and enables crypto shredding. - [Kafka Streams vs ksqlDB: Choosing Right](https://www.conduktor.io/blog/kafka-streams-vs-ksqldb-choosing) (2024-09-02) — Choose between Kafka Streams and ksqlDB for stream processing. Use case comparison, team skills assessment, deployment models, and operational trade-offs. - [Field-Level Encryption in Kafka Without Code Changes Using Conduktor Gateway](https://www.conduktor.io/blog/zero-code-encryption-strategies-for-apache-kafka) (2024-08-28) — Zero-code Kafka encryption: field-level encryption/decryption with Conduktor Gateway—secure PII, role-based access, no application changes required. - [Apache Kafka SQL: Query Topics Like a Database in Real Time](https://www.conduktor.io/blog/apache-kafka-sql-topics) (2024-08-14) — Run SQL queries directly against Kafka topics — filter, join, and aggregate streaming data without moving it to a warehouse. Real-time SQL over Kafka. - [KRaft Explained: Kafka Without ZooKeeper](https://www.conduktor.io/blog/kraft-explained-kafka-without-zookeeper) (2024-07-27) — How KRaft replaces ZooKeeper in Kafka 4.0. Controller quorum architecture, broker configuration examples, and step-by-step migration from ZooKeeper. - [Kafka Topic Naming Conventions That Scale](https://www.conduktor.io/blog/kafka-topic-naming-conventions-scale) (2024-07-13) — Build a Kafka topic naming convention that scales. Structural patterns, ACL alignment, and common mistakes that break governance. - [Schema Registry Isn't Optional](https://www.conduktor.io/blog/schema-registry-isnt-optional) (2024-07-13) — The "we'll add it later" approach costs weeks of debugging and painful migrations. Schema Registry is day-one infrastructure. - [Event sourcing with Kafka: patterns and pitfalls](https://www.conduktor.io/blog/event-sourcing-kafka-patterns-pitfalls) (2024-07-03) — Build event-sourced systems on Kafka. Topic design, aggregate reconstruction, CQRS projections, and the gotchas that break production. - [PagerDuty's Kafka Outage: Lessons from 4.2 Million Rogue Producers](https://www.conduktor.io/blog/pagerduty-kafka-outage-lessons) (2024-06-28) — A code pattern created 4.2M Kafka producers per hour, crashed the cluster, and silenced alerts for 9+ hours. Dissecting the postmortem. - [When NOT to Use Kafka: 5 Scenarios Where Simpler Wins](https://www.conduktor.io/blog/when-not-to-use-kafka) (2024-05-21) — Kafka adds complexity you may not need. Five scenarios where PostgreSQL LISTEN/NOTIFY, Redis Streams, SQS, or simple webhooks outperform Kafka. - [Kafka Retention: When Messages Disappear and Why](https://www.conduktor.io/blog/kafka-retention-policies-when-messages-disappear) (2024-05-06) — Master Kafka retention—retention.ms, segment.ms, and why messages persist longer than expected. Debug commands included. - [Build Idempotent Kafka Consumers: Patterns That Actually Work](https://www.conduktor.io/blog/building-idempotent-consumers) (2024-04-26) — Handle duplicate Kafka messages gracefully. Database constraints, Redis lookups, and the deduplication patterns that scale. - [GDPR and Kafka: Right to Erasure](https://www.conduktor.io/blog/gdpr-kafka-right-to-erasure) (2024-04-24) — Handle GDPR Article 17 deletion requests in Kafka's append-only log. Crypto shredding implementation, tombstone patterns, and compliant retention strategies. - [What Kafka Users Actually Care About in 2024: Kafka Summit London Recap](https://www.conduktor.io/blog/kafka-summit-london-2024) (2024-03-29) — Kafka Summit London 2024 highlights: real-time analytics, GitOps self-service, SQL over Kafka, and the shift from batch to streaming data. - [When Kafka Exactly-Once Semantics Are Worth the Performance Cost](https://www.conduktor.io/blog/kafka-exactly-once-performance-cost) (2024-03-23) — Kafka exactly-once semantics add latency and complexity. When the cost is worth paying, and when at-least-once with idempotent consumers wins instead. - [Partition Count: The Decision You Can't Undo](https://www.conduktor.io/blog/partition-count-decision-guide) (2024-03-01) — How to choose Kafka partition count. Sizing formulas, overhead costs, and why decreasing partitions requires topic migration. - [Conduktor and Confluent Kafka Meetups Across the USA, February 2024](https://www.conduktor.io/blog/confluent-conduktor-usa-tour) (2024-02-08) — Join Conduktor's February 2024 USA tour in NYC, Boston, Bay Area, and Seattle. Learn about Kafka poison pills, schema compatibility, and data contracts. - [Kafka Platform: Build vs Buy Decision Framework](https://www.conduktor.io/blog/kafka-build-vs-buy-decision-framework) (2024-01-26) — Framework for evaluating self-managed Kafka vs managed services like Confluent Cloud and MSK. TCO analysis, team skills assessment, and decision criteria. - [Disk Full: Emergency Recovery When Kafka Runs Out of Space](https://www.conduktor.io/blog/disk-full-emergency-recovery) (2024-01-17) — Emergency runbook for Kafka disk full scenarios. Immediate triage commands, safe segment deletion, recovery steps, and retention tuning to prevent recurrence. - [Hierarchical Tagging for Kafka: Cost Attribution, Ownership, and Governance](https://www.conduktor.io/blog/kafka-hierarchical-tagging) (2024-01-16) — Transform data chaos with strategic tagging. Enable Data Mesh, ABAC security, and better analytics through hierarchical metadata organization. - [Multi-Region Kafka: Active-Active vs Active-Passive](https://www.conduktor.io/blog/multi-region-kafka-active-active-passive) (2024-01-11) — Compare Kafka DR patterns. MirrorMaker 2 setup, offset translation, conflict resolution, and when each architecture makes sense. - [SOC2 Type II Certification: One Year of Writing Policies and Proving We Follow Them](https://www.conduktor.io/blog/what-we-learned-from-soc2-type2-write-what-you-do-do-what-you-write) (2024-01-02) — SOC2 Type II journey: policies, pentesting, CVE management, agent monitoring, and compliance automation with Drata—lessons from Conduktor's certification. - [Real-Time SQL on Kafka with PySpark](https://www.conduktor.io/blog/getting-started-with-pyspark-and-kafka) (2023-09-17) — Run real-time SQL queries on Kafka with PySpark. Stream processing tutorial with JSON transformations, ChatGPT integration, and data generation. - [Why Conduktor Uses Redpanda for Local Kafka Development](https://www.conduktor.io/blog/conduktor-redpanda-best-breed-kafka-experience) (2023-08-10) — Conduktor uses Redpanda for local development because it starts instantly, weighs 130MB, includes Schema Registry, and eliminates ZooKeeper complexity. - [Conduktor Now Available on AWS Marketplace for MSK Users](https://www.conduktor.io/blog/conduktor-available-on-the-aws-marketplace-to-complement-msk) (2023-08-02) — Deploy Conduktor on AWS Marketplace for MSK with IAM, Glue Schema Registry support, and seamless ECS/EKS integration in minutes. - [Kafka Producer & Consumer Config: 5 Performance Settings That Matter](https://www.conduktor.io/blog/kafka-producer-consumer-performance-config) (2023-07-26) — Master Kafka performance: partition sizing, SerDe optimization, dual-write solutions, tombstones, and producer tuning for robust applications. - [What Kafka Users Actually Asked Us at Kafka Summit London 2023](https://www.conduktor.io/blog/kafka-summit-london-2023-feedback-from-our-booth) (2023-05-25) — Field-level encryption, multi-tenancy, topic-as-a-service, and RBAC dominated conversations at our booth. Here's what the community cares about most. - [Data Mesh Explained: Decentralized Data Ownership for Scalable Organizations](https://www.conduktor.io/blog/what-is-a-data-mesh) (2023-05-04) — Data mesh decentralizes ownership for agility and scale. Learn core principles, architecture, and how Kafka + Conduktor enable domain-driven data products. - [Kafka RBAC: Role-Based Access Control for Teams and Applications](https://www.conduktor.io/blog/security-for-humans-and-applications-in-apache-kafka) (2023-04-08) — Kafka RBAC with fine-grained permissions for humans and applications. Manage producer, consumer, and admin access without modifying broker-level ACLs. - [Data Integration: How Merging Datasets Drives Better Business Decisions](https://www.conduktor.io/blog/data-integration-the-what-why-and-benefits) (2023-03-09) — Master data integration for big data success. Explore ETL vs ELT processes, data quality benefits, and automation strategies for business intelligence. - [Change Data Capture Explained: How CDC Streams Database Changes in Real-Time](https://www.conduktor.io/blog/what-is-change-data-capture-anyways) (2023-03-09) — Change Data Capture (CDC) explained: real-time database replication, log-based streaming, push vs. pull methods, and Kafka CDC implementation patterns. - [Automate Kafka Topic Governance with Topic as a Service](https://www.conduktor.io/blog/topic-as-a-service-automate-governance-decisions-for-apache-kafka) (2023-01-31) — Automate Kafka topic governance and ACL management with Topic as a Service—eliminate bottlenecks, enforce standards, free your ops team. - [Kafka Schema Registry: How It Works & Why You Need It](https://www.conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it) (2022-11-24) — Schema Registry enforces data contracts in Kafka, preventing schema drift and producer-consumer mismatches. Covers serialization, Avro, Protobuf, evolution r... Source: https://www.conduktor.io/authors/stephane-derosiaux --- # Stéphane Maarek **Chief Evangelist** Co-founder and Chief Evangelist of Conduktor. AWS Hero, renowned instructor, and Apache Kafka expert with millions of students worldwide. [LinkedIn profile](https://www.linkedin.com/in/stephanemaarek/) Source: https://www.conduktor.io/authors/stephane-maarek --- # Stuart Mould **Product Operations Manager** ## Articles by Stuart Mould - [Automate Kafka Governance with Terraform and Conduktor](https://www.conduktor.io/blog/govern-kafka-as-code-terraform-x-conduktor) (2025-04-02) — Govern Kafka at scale with Terraform and Conduktor. Automate infrastructure, enforce policies, enable self-service—no chaos, no bottlenecks. - [Enforcing Kafka Data Quality at Scale](https://www.conduktor.io/blog/enforcing-kafka-data-quality-at-scale) (2024-07-05) — Ensure Kafka data quality at scale: prevent invalid messages, enforce schemas, automate checks without disrupting pipelines. - [Explore Kafka in Minutes with Conduktor's Embedded Cluster](https://www.conduktor.io/blog/a-new-getting-started-experience-with-kafka-and-conduktor) (2023-12-18) — Run Kafka locally with one command. Explore real-time data, encryption, masking, and safeguards without any setup. Source: https://www.conduktor.io/authors/stuart-mould --- # Taras Slipets **Staff Data Engineer, Flix** Source: https://www.conduktor.io/authors/taras-slipets --- # Thomas Pressler **Group IT – Infrastructure Technology Integration Solutions, BMW Group** Source: https://www.conduktor.io/authors/thomas-pressler --- # Tom Scott **Software Engineer** ## Articles by Tom Scott - [How Conduktor Gateway Prevents Kafka Outages Before They Happen](https://www.conduktor.io/blog/proactive-safeguarding-with-conduktor-gateway) (2023-05-17) — Prevent Kafka outages with proactive safeguarding via Conduktor Gateway. Automate best practices, enforce rules, and eliminate tech debt before it starts. - [Kafka Log Retention and Segments: How Storage Actually Works](https://www.conduktor.io/blog/kafka-log-retention-and-segments) (2022-07-19) — Deep dive into Kafka segments, indexing, and log retention policies—optimize storage, compaction, and broker performance for production clusters. Source: https://www.conduktor.io/authors/tom-scott --- # Viktor Gamov **Principal Developer Advocate, Confluent** Principal Developer Advocate at Confluent, Java Champion, and co-author of "Kafka in Action." A recognized expert in distributed systems and streaming technologies. Source: https://www.conduktor.io/authors/viktor-gamov --- # Vincent Gauthier **Account Executive** Vincent is an Account Executive at Conduktor, working with platform teams and architects operating Kafka across industries. [LinkedIn profile](https://www.linkedin.com/in/vincentgauthier/) Source: https://www.conduktor.io/authors/vincent-gauthier --- # William To **Content Marketing Manager** ## Articles by William To - [Federated Kafka Governance: Let Teams Self-Serve Without Losing Control](https://www.conduktor.io/blog/federated-governance-solve-the-speed-vs-control-dilemma) (2025-10-10) — Central teams set guardrails; domain teams self-serve within them. Federation delivers velocity, safety, and scale. - [Real-Time Data is the Bottleneck for AI: Lessons from Gartner D&A Summit 2025](https://www.conduktor.io/blog/unlocking-ai-s-full-potential-what-we-learned-at-the-gartner-d-a-summit-2025) (2025-03-11) — Gartner D&A Summit 2025: How Conduktor + Cloudera deliver trustworthy real-time data for AI—RAG, agentic AI, and streaming governance insights. Source: https://www.conduktor.io/authors/william-to --- # Zeke Dean **Sales Engineer, Conduktor** Source: https://www.conduktor.io/authors/zeke-dean --- # Authors Meet the team behind Conduktor's insights on data streaming, Apache Kafka, and platform engineering. --- - [Áine O'Neill](https://www.conduktor.io/authors/aine-oneill) — Account Executive, Conduktor - [Alasdair Ross](https://www.conduktor.io/authors/alasdair-ross) — Engineering Manager - [Ana Nash](https://www.conduktor.io/authors/ana-nash) — Account Manager - [Andy Allison](https://www.conduktor.io/authors/andy-allison) — VP of Product - [Augusto Leal](https://www.conduktor.io/authors/augusto-leal) — Sales Engineer - [Aurelie Marcuzzo](https://www.conduktor.io/authors/aurelie-marcuzzo) — Solutions Architect - [Casey O'Shea](https://www.conduktor.io/authors/casey-oshea) — Chief Technology Officer, Atyeti - [Christoph Schubert](https://www.conduktor.io/authors/christoph-schubert) — Head of Product - [Chuck Larrieu Casias](https://www.conduktor.io/authors/chuck-larrieu-casias) — Solutions Architect - [Dr. Stefan Brandner](https://www.conduktor.io/authors/dr-stefan-brandner) — Vorstand, SupplyOn AG - [Emily Lewis](https://www.conduktor.io/authors/emily-lewis) — Senior Director, Marketing - [Emma Humber](https://www.conduktor.io/authors/emma-humber) — Software Engineer - [Eric Sammer](https://www.conduktor.io/authors/eric-sammer) — CEO and Founder, Decodable - [Fabian Schmitzer](https://www.conduktor.io/authors/fabian-schmitzer) — Sales Engineer - [Florent Ramière](https://www.conduktor.io/authors/florent-ramiere) — Product Lead - [Frances O'Rafferty](https://www.conduktor.io/authors/frances-orafferty) — Senior Manager, Baringa - [Francesca Scantlebury](https://www.conduktor.io/authors/francesca-scantlebury) — Head of People and Talent - [François Teychené](https://www.conduktor.io/authors/francois-teychene) — Software Engineer - [Howard Holton](https://www.conduktor.io/authors/howard-holton) — COO, GigaOm - [James White](https://www.conduktor.io/authors/james-white) — Director of Product - [Jean-Louis Boudart](https://www.conduktor.io/authors/jean-louis-boudart) — Head of Customer Success Engineering - [Johnathan Law](https://www.conduktor.io/authors/johnathan-law) — Senior Product Manager - [Jonathan Deane](https://www.conduktor.io/authors/jonathan-deane) — Account Executive, Conduktor - [Jordan Roomer](https://www.conduktor.io/authors/jordan-roomer) — Account Executive - [Jorge Ruiz](https://www.conduktor.io/authors/jorge-ruiz) — Senior Product Marketing Manager - [Julien Chanaud](https://www.conduktor.io/authors/julien-chanaud) — Product Manager - [Julien Maillard](https://www.conduktor.io/authors/julien-maillard) — Senior Architect, CDC Informatique - [Lorcan Cooke](https://www.conduktor.io/authors/lorcan-cooke) — Customer Success Engineer, Conduktor - [Loretta Schwaebe](https://www.conduktor.io/authors/loretta-schwaebe) — Account Executive, DACH, Conduktor - [Mansi Vaghela](https://www.conduktor.io/authors/mansi-vaghela) — Partner Solutions Architect, AWS - [Manuel Calvé](https://www.conduktor.io/authors/manuel-calve) — Head of Partnerships - [Marcel Boldt](https://www.conduktor.io/authors/marcel-boldt) — Solutions Architect - [Maria Jose Lopez Monzon](https://www.conduktor.io/authors/maria-jose-lopez-monzon) — Scrum Master, SIX - [Matt Schumpert](https://www.conduktor.io/authors/matt-schumpert) — Head of Product, Core, Redpanda - [Matt Searle](https://www.conduktor.io/authors/matt-searle) — Engineering Director - [Matthew Scott](https://www.conduktor.io/authors/matthew-scott) — Head of Strategic Partnerships & Channels - [Mike Carley](https://www.conduktor.io/authors/mike-carley) — Sales Engineer, Conduktor - [Mirja Hollman](https://www.conduktor.io/authors/mirja-hollman) — Technical Product Owner, Flix - [Nicolas Orban](https://www.conduktor.io/authors/nicolas-orban) — CEO - [Nicole Bouchard](https://www.conduktor.io/authors/nicole-bouchard) — Senior Product Manager - [Paul Feldman](https://www.conduktor.io/authors/paul-feldman) — Regional Sales Director, East and Central - [Poria Zarrinhonar](https://www.conduktor.io/authors/poria-zarrinhonar) — Renewals Account Manager, Conduktor - [Quentin Packard](https://www.conduktor.io/authors/quentin-packard) — SVP of Sales - [Rehm Kharal](https://www.conduktor.io/authors/rehm-kharal) — Account Executive, Conduktor - [Richa Khandelwal](https://www.conduktor.io/authors/richa-khandelwal) — Senior Engineering Leader, Airbnb - [Robert Mitchell](https://www.conduktor.io/authors/robert-mitchell) — Strategic Account Executive, Conduktor - [Romain Lecomte](https://www.conduktor.io/authors/romain-lecomte) — Staff Software Engineer - [Ron Kapoor](https://www.conduktor.io/authors/ron-kapoor) — Technical Product Marketing Manager - [Sarah Polan](https://www.conduktor.io/authors/sarah-polan) — Chief Technical Officer, Redhat - [Shadan Ashfaie](https://www.conduktor.io/authors/shadan-ashfaie) — Enterprise PMO - [Shuchi Agarwal](https://www.conduktor.io/authors/shuchi-agarwal) — Head of AI Execution, SMBC Group - [Siddhanth Lathar](https://www.conduktor.io/authors/siddhanth-lathar) — Solutions Architect - [Sreeni Gadhar](https://www.conduktor.io/authors/sreeni-gadhar) — VP of Global Data and Analytics, Marsh McLennan - [Stéphane Derosiaux](https://www.conduktor.io/authors/stephane-derosiaux) — CPTO - [Stéphane Maarek](https://www.conduktor.io/authors/stephane-maarek) — Chief Evangelist - [Stuart Mould](https://www.conduktor.io/authors/stuart-mould) — Product Operations Manager - [Taras Slipets](https://www.conduktor.io/authors/taras-slipets) — Staff Data Engineer, Flix - [Thomas Pressler](https://www.conduktor.io/authors/thomas-pressler) — Group IT – Infrastructure Technology Integration Solutions, BMW Group - [Tom Scott](https://www.conduktor.io/authors/tom-scott) — Software Engineer - [Viktor Gamov](https://www.conduktor.io/authors/viktor-gamov) — Principal Developer Advocate, Confluent - [Vincent Gauthier](https://www.conduktor.io/authors/vincent-gauthier) — Account Executive - [William To](https://www.conduktor.io/authors/william-to) — Content Marketing Manager - [Zeke Dean](https://www.conduktor.io/authors/zeke-dean) — Sales Engineer, Conduktor Source: https://www.conduktor.io/authors --- # Meet Conduktor at AWS Summit Zurich Meet us at AWS Summit Zurich [Book a meeting →](#book) Conference 2 September 2026 Find us at Booth B19 Migration & Modernization cluster · Level 1 (Halls 3-4) Why meet us Get more out of your Kafka Whatever Kafka you run, Conduktor adds the governance, security, and cost control on top, with no changes to your producers or consumers. Cut Kafka costs 20 to 40% Consolidate clusters, right-size partitions, and attribute spend by team, so Kafka infrastructure becomes accountable spend instead of a mystery line item. Secure data at the field level Encrypt, mask, or tokenize sensitive fields before they hit Kafka. Meet compliance requirements without touching a single producer or consumer. One control plane for every team Give developers self-service access while platform teams keep governance and visibility. Conduktor Console and Conduktor Gateway work together across your Kafka. What to expect A focused 20 minutes with the right people, matched to what you actually care about. A live look at Conduktor Console and Gateway against your real Kafka setup. Straight answers on cost visibility, field-level security, and governance. A calendar invite with the booth spot pinned the second you book. Who you'll meet Loretta SchwaebeEnterprise Account Executive Fabian SchmitzerSales Engineer [Book a meeting →](#book) Trusted by enterprise teams Enterprises across Europe run Kafka with Conduktor 5×Kafka growth · 800+ users Swiss Post scaled Kafka without losing control Swiss Post grew to 800+ users and 5x more applications on Kafka using Conduktor's governance, RBAC, and self-service. Come hear how a Swiss team runs Kafka at scale. [Read the Swiss Post story →](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) Book your 20 minutes Pick a time that works for you Choose a slot and we'll send a calendar invite with the booth spot pinned. Where to find us Booth B19 Booth B19, Level 1 Migration & Modernization cluster · Halls 3-4 **How to find us:** head to the Migration & Modernization cluster on Level 1 and look for the green Conduktor booth at B19. Booked a meeting? Your host's number is in your confirmation, so message us if you're running late. Booth resources Conduktor's on-site booth resources Everything we're sharing at Booth B19. Take the one-pagers with you, and save your spot at our next Swiss event. [15OCT Next Swiss event Kafka Leadership Exchange · Bern An invite-only breakfast roundtable for streaming platform leaders, hosted at Swiss Post. View the event →](https://www.conduktor.io/kafka-leadership-exchange/bern) [Company Overview What Conduktor does and the outcomes teams get on Kafka. Download PDF →](https://www.conduktor.io/assets/Conduktor-Company-Overview.pdf) [Console Overview Monitoring, self-service, and governance for every Kafka team. Download PDF →](https://www.conduktor.io/assets/Conduktor-Console-Overview.pdf) [Gateway & Proxy Overview The Kafka proxy for routing, security, and data policies, with no app changes. Download PDF →](https://www.conduktor.io/assets/Conduktor-Gateway-Proxy-Overview.pdf) [Schema Registry Proxy Overview Authentication, RBAC, and audit for your Kafka schemas. Download PDF →](https://www.conduktor.io/assets/Conduktor-Schema-Registry-Proxy-Overview.pdf) Before you visit Take the Kafka Ops Assessment 10 questions on where your Kafka stack is bleeding time, risk, or budget. Get a personalized report, bring your score to Booth B19, and we'll walk through your biggest gaps live. STEP 01 Take the assessment Five minutes, ten questions. It scores you across Dev Tooling, Incident Response, Governance, Security, and Cost. STEP 02 Bring your score to B19 Show us your results and we'll pinpoint the fastest wins for your setup. STEP 03 Grab your swag Everyone who runs through their results with us at the booth walks away with Conduktor swag. [Take the assessment →](https://www.conduktor.io/kafka-assessment) Source: https://www.conduktor.io/aws-summit-zurich --- # Kafka Blog | Streaming Data Best Practices & Tutorials - [We Deleted a Kafka Topic. Which App Broke?](https://www.conduktor.io/blog/we-deleted-a-kafka-topic-which-app-broke) (2026-09-10) — Understand your Kafka cluster for effective change management and troubleshooting. Stream Lineage in Conduktor Console maps it from your ACLs. - [How to Approach Multi-Tenancy in Kafka](https://www.conduktor.io/blog/how-to-do-multi-tenancy-in-kafka) (2026-08-05) — Kafka has no built-in tenants. There are three ways to add them: prefixes and ACLs, a cluster per tenant, or virtual clusters. Here's when each fits. - [Best Tools for Monitoring Kafka Consumer Lag (2026)](https://www.conduktor.io/blog/best-kafka-consumer-lag-monitoring-tools) (2026-07-30) — Burrow, KMinion, Datadog, CloudWatch, AKHQ, Kafbat and more, compared on lag detail, time lag, and what you can do when the alert fires. - [Designing Kafka Applications for Data Privacy Compliance](https://www.conduktor.io/blog/designing-kafka-applications-for-data-privacy-compliance) (2026-07-27) — Stop coding privacy compliance into every Kafka app. Put encryption, erasure, and access control in one enforcement layer between your apps and Kafka. - [Kafka Migration Tools: What Each One Actually Moves](https://www.conduktor.io/blog/kafka-migration-tools) (2026-07-23) — Asking which tool migrates a Kafka cluster is like asking which tool builds a house. A migration is five jobs, and each layer of the stack has its own tools. - [Benchmarking the Conduktor Gateway: what a Kafka proxy really costs](https://www.conduktor.io/blog/conduktor-gateway-performance-benchmark) (2026-07-21) — Real numbers over SASL and TLS, the way you actually run Kafka in production: what a proxy in the hot path does to your throughput and your p99. - [AWS MSK Authentication: OIDC and mTLS Without Touching the Broker](https://www.conduktor.io/blog/kafka-msk-authentication-oidc-mtls) (2026-07-21) — AWS MSK clients can only use IAM, SCRAM, or ACM-bound mTLS. Here's how to use OIDC and mTLS with AWS MSK by terminating the scheme at a Kafka proxy. - [Bring Your Own Encryption to Managed Kafka](https://www.conduktor.io/blog/kafka-provider-independence-gateway-encryption) (2026-07-18) — Encrypt Kafka at the proxy with keys that stay in your KMS: your provider can't read what you protect, and your encryption isn't tied to one vendor. - [Terraform Your Whole Kafka Platform](https://www.conduktor.io/blog/kafka-terraform-providers) (2026-07-13) — Kafka Terraform providers don't manage the same layer: the cluster, the topics, or the platform around them. Where Mongey, Confluent and Conduktor fit. - [One Kafka Control Layer for DORA, GDPR and IT-Grundschutz](https://www.conduktor.io/blog/dora-gdpr-it-grundschutz-kafka) (2026-07-11) — DORA, GDPR and IT-Grundschutz often map to the same Kafka controls: encryption, erasure, access, audit and resilience. Build one control layer instead of thr... - [Kafka Schema Registry: Open by Default](https://www.conduktor.io/blog/kafka-schema-registry-open-by-default) (2026-07-10) — By default, the Kafka schema registry is unguarded. How do you get true per-subject control without half-baked engineering workarounds? - [Connect to AWS MSK Across VPCs Without Timeouts](https://www.conduktor.io/blog/kafka-msk-cross-network-connectivity) (2026-07-09) — Your AWS MSK bootstrap connects but every broker times out across a VPC or account boundary. Here's why, and the Kafka-aware proxy pattern that fixes it. - [SNI Routing vs Port Routing in a Kafka Proxy](https://www.conduktor.io/blog/kafka-proxy-sni-routing-vs-port-routing) (2026-07-08) — A Kafka proxy must give every client a routable address for every broker. Port-per-broker breaks at scale; SNI routing is the production answer, at a cost. - [Connect Kafka Across VPCs Without Extra Peering](https://www.conduktor.io/blog/connect-kafka-across-vpcs-without-extra-peering) (2026-07-02) — Reaching Kafka across VPCs isn't a networking problem, it's an addressing one. Why peering, transit gateways, and per-broker NLBs miss it. - [What a Kafka Proxy Can Do: From Routing to Enforcement](https://www.conduktor.io/blog/what-a-kafka-proxy-can-do) (2026-06-30) — A Kafka proxy is not a reverse proxy. It speaks the Kafka wire protocol, so it can become the one place security, governance, and policy live. - [Kafka Cost Optimization: The Layer Above Broker Tuning](https://www.conduktor.io/blog/kafka-cost-optimization-above-broker-tuning) (2026-06-29) — Kafka cost optimization usually stops at broker tuning. The usage layer above (fan-out, partition waste, chargeback) is where the real savings live. - [Most Kafka Guardrails Don't Protect Your Data](https://www.conduktor.io/blog/kafka-guardrails-data-protection) (2026-06-27) — A Kafka guardrail in a wiki, CI, or Terraform only holds if the request goes through it. Your data is only as protected as what can reject it on the wire. - [Testing a Kafka Proxy: Taming Millions of Permutations](https://www.conduktor.io/blog/testing-a-kafka-proxy) (2026-06-25) — How we test the Conduktor Gateway Kafka proxy: ~9 million protocol, broker, and client permutations, tamed with pairwise testing and real Kafka clients. - [Kafka Guardrails Gate. Templates Guide.](https://www.conduktor.io/blog/kafka-guardrails-gate-templates-guide) (2026-06-23) — Enforcement tells Kafka teams what they can't do, never what good looks like. Configuration templates are the other half of self-service provisioning. - [The Surprising Cost of Kafka Partition Waste](https://www.conduktor.io/blog/the-surprising-cost-of-kafka-partition-waste) (2026-06-23) — Most Kafka clusters carry 40 to 70 percent partition waste. On managed Kafka it hits the invoice; on self-managed clusters it sets your broker count. - [Why Copy a Kafka Topic Just to Filter It?](https://www.conduktor.io/blog/why-copy-a-kafka-topic-just-to-filter-it) (2026-06-18) — You don't have to. Topic views, now GA in Conduktor Gateway, give each consumer a filtered, projected view of a Kafka topic at read-time without copies. - [What's Really Inside Kafka's poll()?](https://www.conduktor.io/blog/kafka-consumer-poll-explained) (2026-06-15) — A Kafka consumer looks simple: a single connection to Kafka, then you poll(). It couldn't be so far from the truth. - [If Kafka Credentials Leak, Does the Attacker Get Plaintext or Ciphertext?](https://www.conduktor.io/blog/kafka-credentials-leak-plaintext-or-ciphertext) (2026-06-12) — TLS, disk encryption, and BYOK each stop a different attacker. None of them stops a leaked Kafka credential from reading topics in plaintext. - [Good Kafka Is Boring Kafka](https://www.conduktor.io/blog/good-kafka-is-boring-kafka) (2026-06-11) — A regulated bank scaled Kafka from 40 to 160 apps without growing the platform team. The lever wasn't the tech, it was governed self-service. - [How to Isolate Non-Prod Kafka with Virtual Clusters](https://www.conduktor.io/blog/how-to-isolate-non-prod-kafka-with-virtual-clusters) (2026-06-09) — Hard network segregation between non-prod environments is expensive. How Kafka virtual clusters give security teams the same isolation on one cluster. - [Ownership → Accountability → Trust → Self-Service](https://www.conduktor.io/blog/no-kafka-data-platform-without-ownership) (2026-06-09) — Ownership is the first domino of a real Kafka data platform: accountability, trust, autonomy, self-service, scale, adoption. Why federated ownership makes ev... - [FedRAMP High for Kafka Without Replatforming](https://www.conduktor.io/blog/fedramp-high-for-kafka-without-replatforming) (2026-06-08) — Your Kafka cluster isn't FedRAMP High and replatforming everything is a no-go? Encrypt and sign records at the proxy layer inside your own environment. - [Kafka Partitions are the wrong ordering abstraction. Keys are.](https://www.conduktor.io/blog/kafka-partitions-are-the-wrong-ordering-abstraction-keys-are) (2026-06-01) — Confluent deprecated Parallel Consumer and points to Share Consumers. Different problem: in event-driven systems, the key is the real unit of work. - [What We Learned at Current 2026](https://www.conduktor.io/blog/what-we-learned-at-current-2026) (2026-05-27) — We were at Current 2026 in London last week. Between booth conversations and recorded interviews on the floor, here's what stood out. - [HIPAA 2026 and Kafka: Securing ePHI Streams](https://www.conduktor.io/blog/kafka-hipaa-compliance) (2026-05-25) — The 2026 HIPAA Security Rule mandates encryption, MFA, and annual asset inventory. What it breaks in a typical Kafka deployment, and how to fix it. - [Rethinking Kafka Security for HIPAA 2026](https://www.conduktor.io/blog/rethinking-kafka-security-for-hipaa-2026) (2026-05-25) — Rethinking Kafka security for HIPAA 2026. Why streaming breaks the 'at rest' abstraction the rule was written around, and what to fix in the architecture. - [Kafka 4.3: OAuth assertions, share groups, tiered bootstrap](https://www.conduktor.io/blog/kafka-4-3-oauth-share-groups-tiered-bootstrap) (2026-05-23) — Apache Kafka 4.3 shipped May 22, 2026. The three KIPs that matter in production: OAuth client assertions, share groups, and tiered bootstrap. - [Your Platform Team Can’t Fix Kafka Costs Alone](https://www.conduktor.io/blog/your-platform-team-cant-fix-kafka-costs-alone) (2026-05-19) — Kafka costs keep growing because most orgs lack the ownership model to act on them. Moving to the next stage of cost maturity is the engineering leader's call. - [Conduktor MCP + Skills for AI Agents](https://www.conduktor.io/blog/conduktor-mcp-and-skills-for-ai-agents) (2026-05-17) — How Conduktor MCP and Skills let AI agents work with your live Kafka. Patterns for platform onboarding, dev team self-service, and on-demand insights. - [The Apache Kafka Security Playbook: Beyond TLS and ACLs](https://www.conduktor.io/blog/apache-kafka-security-playbook) (2026-05-11) — TLS only goes so far. The real Kafka security gaps: payload encryption, service accounts left rotten, ACLs that fall behind reality, and scattered audit logs. - [librdkafka vs Java Client: 10 Config Defaults That Differ](https://www.conduktor.io/blog/librdkafka-vs-java-client) (2026-05-05) — Python confluent-kafka and Java share config names but not behavior. 10 librdkafka differences tested on a KRaft cluster, with source code and real output. - [Why Kafka Costs Keep Rising (Even When Usage Doesn’t)](https://www.conduktor.io/blog/a-better-conversation-about-kafka-costs) (2026-04-30) — Kafka costs are driven by structure and provisioning, not usage. A view from the field on where the savings actually hide. - [The Data Sharing Maturity Ladder](https://www.conduktor.io/blog/the-data-sharing-ladder) (2026-04-15) — Data sharing maturity has four levels: keep data streaming, expose it safely, create partner value, then package it as sellable data products. - [Kafka in Production: 11 Pitfalls You Should Avoid](https://www.conduktor.io/blog/kafka-production-pitfalls) (2026-03-30) — 11 Kafka behaviors we found helping customers, then validated in the source code. Silent safety bypasses, configs that lie, compression traps. - [Introducing Schema Registry Proxy (Preview)](https://www.conduktor.io/blog/introducing-schema-registry-proxy) (2026-03-24) — Schema registry governance gets deferred because the tooling isn't there. We built Schema Registry Proxy to change that. - [Set Up a Kafka Platform with an AI Agent](https://www.conduktor.io/blog/set-up-a-kafka-platform-with-an-ai-agent) (2026-03-20) — Install Kafka, configure governance, and onboard developers with one AI conversation. Open-source Conduktor skill for Claude Code, Cursor, and 30+ tools. - [Chaos Testing for Kafka Compliance: DR Evidence That Works](https://www.conduktor.io/blog/chaos-testing-for-kafka-compliance-dr-evidence-that-works) (2026-03-19) — Map chaos experiments to DORA, SOC 2, PCI-DSS, and GDPR requirements. Turn your Kafka resilience testing into the compliance evidence regulators expect. - [Kafka Chaos Tests: What They Teach You That Monitoring Can't](https://www.conduktor.io/blog/what-kafka-chaos-tests-actually-teach-you-that-monitoring-doesnt) (2026-03-12) — Monitoring tells you what broke. Chaos testing tells you what will. Five Kafka system behaviors that only surface under controlled failure. - [Chaos Engineering for Kafka: Test Recovery Before You Need It](https://www.conduktor.io/blog/chaos-engineering-for-kafka-testing-recovery-before-you-need-it) (2026-03-04) — Five tests that validate your Kafka disaster recovery plan before a real outage does. A practical guide for platform engineering teams. - [How Conduktor Gateway Reduces Kafka DR from Hours to Minutes](https://www.conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes) (2026-03-02) — Conduktor Gateway cuts Kafka DR coordination overhead. One API call switches all clients to the secondary cluster. Cut execution from hours to minutes. - [Your Kafka Investment Is Hiding Costs and Opportunities](https://www.conduktor.io/blog/your-kafka-investment-is-hiding-costs-and-opportunities) (2026-02-20) — Your Kafka investment is hiding costs, unused capacity, and optimization opportunities that no dashboard can surface. AI changes what's possible to ask. - [Ground Control to On-Call: Kafka Debugging Has Landed](https://www.conduktor.io/blog/what-debugging-kafka-should-actually-look-like) (2026-02-17) — No more juggling 7 CLI tools. A walkthrough of debugging a Kafka consumer lag incident with Conduktor Console. - [Kafka Disaster Recovery: The Complete Strategy Beyond Replication](https://www.conduktor.io/blog/kafka-disaster-recovery-the-complete-strategy-beyond-replication) (2026-02-11) — Three operational phases, six technical areas, and a failover runbook. The complete disaster recovery strategy most Kafka teams are missing. - [Houston, We Have 7 CLI Tools and Zero Answers](https://www.conduktor.io/blog/houston-we-have-7-cli-tools-and-zero-answers) (2026-02-06) — You've accepted that debugging matters more than dashboards. Then you open a terminal and realize Kafka's native tools were never built for this. - [Why Every Kafka Incident Ends with "Restart It"](https://www.conduktor.io/blog/why-every-kafka-incident-ends-with-restart-it) (2026-02-04) — Teams invest in Kafka observability thinking they're prepared. Then an incident hits, and they realize dashboards don't tell you which room is on fire. - [Stop Calling Your Kafka Topics Data Products](https://www.conduktor.io/blog/kafka-data-products) (2026-02-03) — A Kafka topic is shared mutable state: anyone writes, anyone reads, nobody answers for it. A data product is the interface you wrap around it. - [Safeguard Kafka from Timestamp-Based Data Loss](https://www.conduktor.io/blog/safeguard-kafka-from-timestamp-based-data-loss) (2026-02-03) — Producer timestamps can trigger immediate retention deletion. Configure timestamp validation, use LogAppendTime, or enforce bounds at the gateway level. - [Why Kafka Platform Health Requires More Than Monitoring](https://www.conduktor.io/blog/why-kafka-platform-health-requires-more-than-monitoring) (2026-02-02) — Platform health differs from infrastructure health. Observability alone won't cut it. Here are the questions that reveal your Kafka platform's real state. - [Kafka Latency: Measuring p99 & p999 Tails](https://www.conduktor.io/blog/kafka-latency) (2026-01-30) — Kafka latency averages hide tail problems. Optimize p99 and p999, not p50, and measure end-to-end from producer commit to consumer processing. - [Kafka DR: Why Replication Isn't the Hard Part](https://www.conduktor.io/blog/kafka-dr-why-replication-isnt-the-hard-part) (2026-01-27) — PagerDuty's 9-hour Kafka outage reveals the real DR problem - coordinating 47 services during crisis. Why replication alone won't save you. - [Kafka Schema Governance: From Chaos to Confidence in 4 Steps](https://www.conduktor.io/blog/kafka-schema-governance-from-chaos-to-confidence) (2026-01-14) — Fix Kafka schema chaos with a 4-step governance framework: visibility, audit, accountability, enforcement. Used by Swiss Post across hundreds of topics. - [Automating Kafka Audit Evidence](https://www.conduktor.io/blog/kafka-audit-automation) (2025-12-30) — Manual Kafka audit prep means rebuilding the evidence before every audit. Generate compliance evidence continuously instead, as a byproduct of operations. - [Kafka Data Contracts: A Schema Is Not a Contract](https://www.conduktor.io/blog/kafka-data-contracts) (2025-12-23) — A schema tells you the shape of your data. A contract stops a producer from breaking every consumer downstream. The difference, and where to enforce it. - [Why Your Kafka Architecture Should Not Depend on Any Single Vendor](https://www.conduktor.io/blog/the-case-for-vendor-neutral-architecture) (2025-12-18) — IBM's Confluent acquisition shows why vendor-neutral Kafka matters. Decouple apps from infra to cut costs and stay flexible. - [IBM Acquires Confluent for $11B: What It Means for Kafka Teams](https://www.conduktor.io/blog/ibm-confluent-11-billion-says-streaming-isn-t-optional) (2025-12-10) — IBM's acquisition of Confluent validates streaming as critical infrastructure. Here's what changes for organizations running Kafka. - [Why Enterprise AI Projects Stall on Data Access, Not Models](https://www.conduktor.io/blog/the-hidden-architecture-problem-behind-enterprise-ai) (2025-12-01) — Enterprise AI fails due to data access bottlenecks, multi-hop pipelines, and late governance. SMBC, Marsh McLennan leaders share real-world solutions. - [Kafka Automation Platform: Self-Serve by Policy](https://www.conduktor.io/blog/kafka-automation-platform) (2025-11-18) — Manual Kafka provisioning turns platform teams into a ticket queue and makes developers wait. Policy-based automation gives the team their time back. - [Kafka's Real Problem Is Governance, Not Speed](https://www.conduktor.io/blog/the-new-era-of-streaming) (2025-11-06) — Most Kafka clusters push less than 1MB/s. The challenge isn't throughput—it's building the control plane that makes streaming usable at enterprise scale. - [Why Every Streaming Vendor Is Building a Kafka Proxy](https://www.conduktor.io/blog/the-great-confluence-of-streaming) (2025-11-05) — Streaming proxies enable governed self-service and AI-ready data at scale. Why Kafka control planes from Confluent, Kong, and Conduktor are converging. - [5 Sessions Worth Your Time at Current New Orleans 2025](https://www.conduktor.io/blog/five-can-t-miss-sessions-at-current-new-orleans-2025) (2025-10-27) — Current New Orleans 2025 highlights: OpenAI's StreamLink, Kafka protocol deep-dive, Notion's event stack, KIP-1163 diskless topics, and JPMorgan's strategy. - [Conduktor Joins AWS ISV Accelerate Program to Simplify MSK Deployments](https://www.conduktor.io/blog/conduktor-joins-aws-isv-accelerate-program) (2025-10-22) — Conduktor joins AWS ISV Accelerate Program, enabling secure MSK deployments for SageMaker, Redshift—accelerating AI adoption at Swiss Post and beyond. - [Kafka Security: Access Control That Scales](https://www.conduktor.io/blog/kafka-security) (2025-10-21) — Kafka security goes beyond TLS encryption. Scalable access control, ACL management, and provable permissions for production environments. - [Kafka Security Best Practices](https://www.conduktor.io/blog/kafka-security-best-practices) (2025-10-14) — Kafka security best practices fail without enforcement. Automate TLS, ACLs, and encryption policies so compliance is built-in, not bolted-on. - [Federated Kafka Governance: Let Teams Self-Serve Without Losing Control](https://www.conduktor.io/blog/federated-governance-solve-the-speed-vs-control-dilemma) (2025-10-10) — Central teams set guardrails; domain teams self-serve within them. Federation delivers velocity, safety, and scale. - [Self-Service Kafka Without the Chaos](https://www.conduktor.io/blog/who-needs-self-service-you-do) (2025-09-25) — Kafka self-service with guardrails: Conduktor enables developer autonomy while preventing data chaos, cost overruns, and compliance risks. - [Validate JSON in Kafka Without Migrating to Schema Registry](https://www.conduktor.io/blog/bounce-bad-kafka-data-with-json-schema-rules) (2025-09-04) — Block malformed JSON before it breaks Kafka pipelines. Conduktor Trust validates schema, types, and fields—no Schema Registry migration needed. - [Kafka Data Quality: A Schema Registry Isn't Enough](https://www.conduktor.io/blog/kafka-data-quality) (2025-08-26) — By the time a consumer reads a bad record, it is already durable, replicated, and wrong everywhere. Why Kafka data quality has to start at write time. - [Conduktor Partners with LHP to Bring Kafka Governance to Automotive and Aerospace Manufacturing](https://www.conduktor.io/blog/secure-and-govern-real-time-manufacturing-data-with-conduktor-and-lhp) (2025-08-18) — Conduktor partners with LHP to secure and govern streaming data for automotive and aerospace manufacturers using Kafka for IIoT, AI, and analytics. - [The Real Problems with Apache Kafka: 10,000 Forum Posts Analyzed](https://www.conduktor.io/blog/apache-kafka-what-10-000-forum-posts-reveal) (2025-08-11) — 10,000+ Kafka forum posts reveal the truth: connector crashes, auth nightmares, schema failures, and cryptic errors plague production deployments. - [Why ML Models Fail in Production: The Training Data Mismatch Problem](https://www.conduktor.io/blog/the-hidden-data-gap-that-wrecks-ai-projects) (2025-08-07) — Mismatched training and production data cripples AI models. Real story from a UK lender shows why Conduktor Trust prevents ML pipeline failures. - [Bridging Operational and Analytical Data with Kafka](https://www.conduktor.io/blog/bridging-gap-between-operational-and-analytical-data) (2025-08-01) — Kafka bridges the operational-analytical data divide in real time. Eliminate batch ETL lag and deliver fresh streaming data to analytics and AI systems. - [Why Schemaless Kafka Topics Break Production Systems](https://www.conduktor.io/blog/the-hidden-pitfalls-of-kafka-s-schemaless-data) (2025-07-24) — Kafka's schemaless flexibility creates data quality issues. Conduktor Trust validates schemaless topics, blocks bad data, and enforces custom rules. - [Chaos Test Kafka Without Breaking Production](https://www.conduktor.io/blog/chaos-test-kafka-and-build-resilient-streaming-environments) (2025-07-10) — Simulate broker failures, latency, and corrupted messages without breaking production. Chaos test Kafka with Conduktor's interceptors. - [Why Real-Time Data Sharing Matters](https://www.conduktor.io/blog/why-real-time-data-sharing-matters) (2025-06-26) — Real-time data sharing drives 3x ROI. Explore Kafka-powered use cases in finance, healthcare, logistics—secure partner data exchange with Conduktor. - [AI Governance Starts at the Data Layer](https://www.conduktor.io/blog/secure-data-operations-in-the-time-of-ai) (2025-06-19) — AI agents now query live operational data directly. A control and audit layer brings AI data governance to every agent-to-data interaction. - [What Dash 2025 Revealed About AI Observability Security](https://www.conduktor.io/blog/48-hours-at-dash-security-ai-and-the-observability-paradox) (2025-06-18) — Dash 2025 revealed AI observability risks—from exposed PII to agent telemetry attacks. How Conduktor + Datadog secure LLMs at scale. - [Fix Data Quality at the Source, Not After Ingestion](https://www.conduktor.io/blog/proactive-and-reactive-the-two-paths-towards-data-quality) (2025-06-12) — Low-quality data destroys AI and ML outputs. Fix data quality at the source—before problems travel downstream. - [What 200 Engineers Told Us at Current London 2025](https://www.conduktor.io/blog/current-london-2025-learnings-to-drive-our-roadmap) (2025-06-06) — Current London 2025 insights: data quality failures, migration complexity, developer bottlenecks, and Flink challenges shape Conduktor's roadmap priorities. - [Conduktor Trust Validates Streaming Data Before It Breaks Your AI](https://www.conduktor.io/blog/conduktor-launches-trust-to-guarantee-data-quality-at-the-source) (2025-05-21) — Conduktor Trust validates streaming data quality at ingestion—preventing AI drift, bad analytics, and compliance failures with proactive governance. - [Map Kafka Costs to Teams and Projects with Chargeback](https://www.conduktor.io/blog/chargeback-attribute-map-kafka-costs-to-your-business) (2025-05-08) — Track Kafka costs by team, project, and environment—not just partitions. Conduktor Chargeback maps service accounts to real business units. - [Data Silos Cost $3.1 Trillion Annually. Kafka Alone Won't Fix Them.](https://www.conduktor.io/blog/break-your-data-silos-before-they-break-your-business) (2025-05-02) — Data silos cost $3.1 trillion annually. Break them with Kafka + Conduktor's self-service, discoverability, and granular access controls. - [Why Enterprise AI Struggles With ROI and Data Control](https://www.conduktor.io/blog/ai-expectations-realities-and-risks) (2025-04-17) — Enterprise leaders tackle AI ROI, GenAI governance, and data controls. Conduktor execs reveal why shift-left alone won't protect LLM inputs. - [How to Pivot from Software Engineering to Event Streaming](https://www.conduktor.io/blog/from-engineer-to-event-streaming-expert-how-to-pivot-roles) (2025-04-16) — A practical guide to transitioning into Kafka and real-time data roles, with specific technologies, projects, and job search strategies. - [Five Kafka Scaling Problems That Hit Enterprises Hardest](https://www.conduktor.io/blog/five-hidden-kafka-challenges-for-enterprises) (2025-04-15) — Schema registries won't save you. Real Kafka challenges at scale: zombie topics, unclear ownership, legacy integrations. - [How to Monetize Real-Time Data](https://www.conduktor.io/blog/own-the-stream-power-the-model-monetizing-real-time-data-in-the-ai-era) (2025-04-15) — Data monetization starts with real-time streams you own, govern, and trust. Learn how companies turn first-party streaming data into revenue and AI advantage. - [What Engineers Actually Struggle With in Kafka: QCon London 2025](https://www.conduktor.io/blog/engineers-pov-on-streaming-qcon-2025-takeaways) (2025-04-11) — What software engineers, architects, and platform teams told us about the real challenges of scaling Kafka—straight from the floor at QCon London 2025. - [Automate Kafka Governance with Terraform and Conduktor](https://www.conduktor.io/blog/govern-kafka-as-code-terraform-x-conduktor) (2025-04-02) — Govern Kafka at scale with Terraform and Conduktor. Automate infrastructure, enforce policies, enable self-service—no chaos, no bottlenecks. - [Streaming at Scale: Don't Let Data Chaos Derail Your Growth](https://www.conduktor.io/blog/streaming-at-scale-don-t-let-data-chaos-derail-your-growth) (2025-03-27) — Scale streaming data with governance and quality controls. Shift left to prevent bad data, ensure AI success, and avoid architectural chaos. - [Kafka and Flink Are the Infrastructure for AI Agents](https://www.conduktor.io/blog/ai-agents-at-scale-the-critical-role-of-kafka-and-flink) (2025-03-25) — AI agents need real-time context to act autonomously. Discover how Kafka + Flink power adaptive, intelligent systems at Netflix and beyond. - [80% of Enterprise Data Sits Unused. Real-Time Activation Changes That.](https://www.conduktor.io/blog/your-dark-data-is-a-goldmine-if-you-activate-it-in-real-time) (2025-03-21) — Most enterprise data never drives a decision. Real-time data activation transforms idle data into competitive advantage. - [Crypto Shredding in Kafka: GDPR Compliance Without Deletion](https://www.conduktor.io/blog/crypto-shredding-in-kafka-a-cost-effective-way-to-ensure-compliance) (2025-03-18) — Crypto shredding destroys encryption keys to make Kafka data permanently unreadable. Instant GDPR compliance without topic deletion or costly reprocessing. - [What Enterprise Data Leaders Actually Worry About: Silos, Security, and Legacy Systems](https://www.conduktor.io/blog/from-promise-to-practice-insights-from-the-millennium-alliance-enterprise-ai-maturity-and-transformation-assembly) (2025-03-18) — Enterprise AI Assembly insights: breaking data silos across regions, federated governance, shift-left security, and mainframe modernization challenges. - [Real-Time AI Requires Kafka and Guardrails](https://www.conduktor.io/blog/how-to-succeed-with-ai-kafka-and-ai-guardrails-are-non-negotiable) (2025-03-14) — Discover how Kafka and Flink power real-time AI agents, enabling autonomous decision-making, adaptive intelligence, and scalable data streaming. - [Real-Time Data is the Bottleneck for AI: Lessons from Gartner D&A Summit 2025](https://www.conduktor.io/blog/unlocking-ai-s-full-potential-what-we-learned-at-the-gartner-d-a-summit-2025) (2025-03-11) — Gartner D&A Summit 2025: How Conduktor + Cloudera deliver trustworthy real-time data for AI—RAG, agentic AI, and streaming governance insights. - [Kafka Isn't a Queue: Stop Designing It Like One](https://www.conduktor.io/blog/kafka-isnt-a-queue) (2025-02-21) — Queues delete messages on ack. Kafka keeps a replayable log. Why the difference drives your topic design, and what Kafka 4.2 share groups (KIP-932) change. - [How a University Lecturer in China Became Conduktor's Head of Sales Engineering](https://www.conduktor.io/blog/from-academia-to-sales-engineering-a-journey-into-streaming-data) (2025-02-20) — Christoph Schubert's journey from teaching in China to Confluent to Conduktor's Head of Sales Engineering, mastering Kafka and customer success. - [How Maria Went from Languages to Product Design at Conduktor](https://www.conduktor.io/blog/a-product-designer-s-journey-to-conduktor) (2025-02-17) — Maria's path from studying languages in Seoul to designing UX/UI at Conduktor shows that design careers don't require a traditional route. - [Partner Zones: Share Kafka Data with External Partners Without Replicating Clusters](https://www.conduktor.io/blog/partner-zones-secure-kafka-data-sharing-without-duplication) (2025-02-07) — Share Kafka streams with external partners through isolated access zones—no cluster duplication or complex replication required. - [Multi-Team Kafka Alerting with Granular Ownership and Webhooks](https://www.conduktor.io/blog/enterprise-ready-kafka-alerting-decentralized-control-webhooks-for-multiple-teams) (2025-01-28) — Conduktor's alerts system gives teams ownership over their Kafka notifications with webhook integrations and event-driven alerting. - [Data Streaming Strategy for 2025: What Industry Experts Recommend](https://www.conduktor.io/blog/your-2025-data-streaming-strategy-blueprint-insights-from-industry-experts) (2025-01-28) — 2025 data streaming strategy: experts from Atyeti, GigaOm discuss data maturity, AI readiness, governance, and modernizing your tech stack. - [Stop Building Kafka Encryption Libraries](https://www.conduktor.io/blog/stop-building-kafka-encryption-libraries) (2025-01-25) — Client-side encryption creates ungovernable sprawl. Proxy-layer encryption centralizes control and makes Kafka a dumb pipe. - [GenAI Adoption Lessons from Resolve.ai: Data Quality, Use Cases, and Build vs. Buy](https://www.conduktor.io/blog/taking-the-genai-bet-what-innovators-have-learned) (2025-01-02) — GenAI adoption insights from Resolve.ai CEO Spiros Xanthos: data quality strategies, strategic use cases, and buy vs. build decisions for AI success. - [SQL Over Kafka: Transforming Real-Time Data Into Instant Insights](https://www.conduktor.io/blog/sql-over-kafka-transforming-real-time-data-into-instant-insights) (2024-12-19) — Query Kafka directly with SQL using Conduktor. Access real-time insights, troubleshoot issues, and optimize operations without data duplication. - [How to Attribute Apache Kafka Costs with Chargeback for Accountability & Better Forecasting](https://www.conduktor.io/blog/tackling-apache-kafka-cost-attribution-with-chargeback) (2024-12-05) — Track Kafka costs by team, domain, or application. Attribute ingress/egress usage to optimize infra and forecast spending. - [Financial Services Data Platforms Are Fragmented, Siloed, and Complex. Here's How to Fix Them.](https://www.conduktor.io/blog/future-of-financial-services-lies-in-data-key-takeaways-fima-europe) (2024-11-26) — FIMA Europe 2024: data management beats AI as top priority. Break silos, shift quality upstream, expand real-time streaming. - [Why Streaming Data Quality Requires Proactive Monitoring, Not Reactive Fixes](https://www.conduktor.io/blog/mastering-data-quality-in-streaming) (2024-11-18) — Master data quality in streaming with proactive monitoring, use-case checks, and real-time alerts. Build trust in data for AI and real-time decisions. - [Conduktor Raises $30M Series B to Expand U.S. Operations and Accelerate Product Development](https://www.conduktor.io/blog/conduktor-secures-30-million-series-b-to-lead-the-future-of-streaming-data-management) (2024-11-12) — Conduktor raises $30M in Series B funding led by RTP Global to expand in the U.S. and meet rising demand for secure, real-time data streaming solutions. - [Conduktor Gateway on Kubernetes with SNI Routing](https://www.conduktor.io/blog/deploy-conduktor-gateway-with-kubernetes-and-host-based-routing) (2024-11-05) — Step-by-step guide to deploying Conduktor Gateway on Kubernetes with host-based (SNI) routing, TLS certificates, and external Ingress access. - [Real-Time Data Streaming for Transport and Logistics with Apache Kafka](https://www.conduktor.io/blog/drive-speed-efficiency-and-revenue-growth-with-streaming-data-for-transport-and-logistics) (2024-10-28) — How transport and logistics use Kafka for route optimization, fleet tracking, and why data governance matters at scale. - [Kafka in Financial Services: Architecture Patterns for Compliance, Fraud Detection, and Real-Time Payments](https://www.conduktor.io/blog/how-to-streamline-your-kafka-architecture-and-drive-success-in-financial-services) (2024-10-21) — Financial services use Kafka for fraud detection, real-time payments, and compliance. Build secure streaming architectures. - [Reliable Data Streaming for Retail: Kafka Operations at Scale](https://www.conduktor.io/blog/how-retail-organizations-can-drive-reliable-accurate-data-streaming) (2024-10-15) — Learn how retail organizations can simplify Kafka operations, secure data streaming, and drive growth with Conduktor's enterprise data management platform. - [BYOC, Storage-Compute Separation, and Shift Left Are Reshaping Data Streaming](https://www.conduktor.io/blog/the-biggest-trends-in-data-streaming) (2024-10-11) — BYOC, decoupling compute from storage, and Shift Left strategies dominate data streaming. Insights from Aiven and Conduktor on AI-driven real-time data. - [Field-Level Encryption in Kafka Without Code Changes Using Conduktor Gateway](https://www.conduktor.io/blog/zero-code-encryption-strategies-for-apache-kafka) (2024-08-28) — Zero-code Kafka encryption: field-level encryption/decryption with Conduktor Gateway—secure PII, role-based access, no application changes required. - [Apache Kafka SQL: Query Topics Like a Database in Real Time](https://www.conduktor.io/blog/apache-kafka-sql-topics) (2024-08-14) — Run SQL queries directly against Kafka topics — filter, join, and aggregate streaming data without moving it to a warehouse. Real-time SQL over Kafka. - [Building Kafka Tooling That Actually Scales: Security, Governance, and GitOps](https://www.conduktor.io/blog/real-time-data-with-kafka) (2024-07-18) — Build scalable Kafka tooling with GitOps and governance automation. Avoid common pitfalls in real-time data security, costs, and compliance. - [Why Kafka ACLs Break at Scale and What to Use Instead](https://www.conduktor.io/blog/kafka-encryption-and-security-best-practices-for-protecting-data) (2024-07-11) — Kafka's default security fails in production. ACLs become unmanageable, PII leaks, encryption is missing. Here's the fix. - [Enforcing Kafka Data Quality at Scale](https://www.conduktor.io/blog/enforcing-kafka-data-quality-at-scale) (2024-07-05) — Ensure Kafka data quality at scale: prevent invalid messages, enforce schemas, automate checks without disrupting pipelines. - [Automate Kafka Configuration at Scale with GitOps](https://www.conduktor.io/blog/kafka-gitops-for-efficient-scaling) (2024-07-04) — Complex infrastructure calls for automation. GitOps streamlines Kafka configuration and scales deployments efficiently. - [Conduktor 2.0: Free Debugging, Monitoring, and Encryption Tools for Kafka Teams](https://www.conduktor.io/blog/conduktor-2-0-the-new-standard-for-teams-working-with-kafka) (2024-04-11) — Conduktor 2.0 expands free tier with debugging, monitoring, and encryption tools—simplifying Kafka for developers, architects, and platform teams. - [What Kafka Users Actually Care About in 2024: Kafka Summit London Recap](https://www.conduktor.io/blog/kafka-summit-london-2024) (2024-03-29) — Kafka Summit London 2024 highlights: real-time analytics, GitOps self-service, SQL over Kafka, and the shift from batch to streaming data. - [Deploy Conduktor Console on AWS with CloudFormation in 10 Minutes](https://www.conduktor.io/blog/step-by-step-deploy-the-most-advanced-kafka-console-on-aws-in-minutes) (2024-02-19) — Deploy Conduktor Console on AWS in minutes using CloudFormation and ECS Fargate. Works with MSK, Confluent, Redpanda, and Kafka Connect. - [Conduktor and Confluent Kafka Meetups Across the USA, February 2024](https://www.conduktor.io/blog/confluent-conduktor-usa-tour) (2024-02-08) — Join Conduktor's February 2024 USA tour in NYC, Boston, Bay Area, and Seattle. Learn about Kafka poison pills, schema compatibility, and data contracts. - [Hierarchical Tagging for Kafka: Cost Attribution, Ownership, and Governance](https://www.conduktor.io/blog/kafka-hierarchical-tagging) (2024-01-16) — Transform data chaos with strategic tagging. Enable Data Mesh, ABAC security, and better analytics through hierarchical metadata organization. - [What We Shipped in 2023 and Why It Matters](https://www.conduktor.io/blog/conduktor-retrospective-2023) (2024-01-11) — Conduktor's 2023 journey: launching Console, introducing Gateway proxy, achieving SOC2 Type II, and evolving from developer tool to enterprise platform. - [SOC2 Type II Certification: One Year of Writing Policies and Proving We Follow Them](https://www.conduktor.io/blog/what-we-learned-from-soc2-type2-write-what-you-do-do-what-you-write) (2024-01-02) — SOC2 Type II journey: policies, pentesting, CVE management, agent monitoring, and compliance automation with Drata—lessons from Conduktor's certification. - [How to Scale Kafka Across Teams Without Losing Control](https://www.conduktor.io/blog/scaling-kafka-securely-for-your-organization) (2023-12-22) — Scale Kafka across teams with federated governance and self-service data platforms. Tackle access management, security, and onboarding bottlenecks. - [Explore Kafka in Minutes with Conduktor's Embedded Cluster](https://www.conduktor.io/blog/a-new-getting-started-experience-with-kafka-and-conduktor) (2023-12-18) — Run Kafka locally with one command. Explore real-time data, encryption, masking, and safeguards without any setup. - [Kafka Proxies, Cost Virtualization, and Flink: Takeaways from Current 2023](https://www.conduktor.io/blog/the-future-of-data-streaming-current-2023-big-data-london) (2023-11-16) — Key insights from Current 2023 and Big Data London: Kafka proxies for encryption, cost optimization through virtualization, and Flink's serverless future. - [Four Ways to Test Apache Kafka Applications](https://www.conduktor.io/blog/testing-your-apache-kafka-data-with-confidence-2023) (2023-10-19) — Test Kafka components with Testcontainers, Streams test-utils, chaos engineering, and Conduktor Console. Ensure data quality and resilience at scale. - [Real-Time SQL on Kafka with PySpark](https://www.conduktor.io/blog/getting-started-with-pyspark-and-kafka) (2023-09-17) — Run real-time SQL queries on Kafka with PySpark. Stream processing tutorial with JSON transformations, ChatGPT integration, and data generation. - [Claudiu Danaila: From Agency Founder to Head of Design at Conduktor](https://www.conduktor.io/blog/meet-the-team-claudiu-danaila) (2023-08-23) — Claudiu Danaila shares his path from running a design agency to leading design at Conduktor, and why good design is a business differentiator. - [Why Conduktor Uses Redpanda for Local Kafka Development](https://www.conduktor.io/blog/conduktor-redpanda-best-breed-kafka-experience) (2023-08-10) — Conduktor uses Redpanda for local development because it starts instantly, weighs 130MB, includes Schema Registry, and eliminates ZooKeeper complexity. - [What a Series A Startup Can Actually Do About Work-Life Balance](https://www.conduktor.io/blog/promoting-work-life-balance-in-a-series-a-start-up) (2023-08-09) — How Conduktor built work-life balance at Series A through hybrid working, transparent feedback, and health benefits—without sacrificing startup velocity. - [Conduktor Now Available on AWS Marketplace for MSK Users](https://www.conduktor.io/blog/conduktor-available-on-the-aws-marketplace-to-complement-msk) (2023-08-02) — Deploy Conduktor on AWS Marketplace for MSK with IAM, Glue Schema Registry support, and seamless ECS/EKS integration in minutes. - [Kafka Producer & Consumer Config: 5 Performance Settings That Matter](https://www.conduktor.io/blog/kafka-producer-consumer-performance-config) (2023-07-26) — Master Kafka performance: partition sizing, SerDe optimization, dual-write solutions, tombstones, and producer tuning for robust applications. - [Desktop vs Console: Why Centralized Kafka Management Wins](https://www.conduktor.io/blog/why-you-should-consider-migrating-to-conduktor-console) (2023-07-12) — Migrate from Desktop to Console: centralized deployment, RBAC, field-level masking, audit logs, SSO, monitoring—enterprise Kafka UI advantages. - [Aurélie Marcuzzo: From Business Intelligence to Customer Success in a Startup](https://www.conduktor.io/blog/meet-the-team-aurelie-marcuzzo) (2023-06-27) — Meet Aurélie Marcuzzo, Customer Success at Conduktor. From France to London, she onboards customers, tests features, and gives users the love they need. - [How to Handle Large Records in Kafka Without Killing Performance](https://www.conduktor.io/blog/beyond-limits-produce-large-records-without-undermining-apache-kafka) (2023-06-19) — Bypass Kafka's 1MB limit without performance hits. Use S3 cold storage with claim check pattern for PDFs, images, and massive batches—seamlessly. - [Kafka Disaster Recovery: RTO, RPO, and Automated Failover](https://www.conduktor.io/blog/when-disaster-hits-kafka-are-you-ready) (2023-06-17) — Kafka disaster recovery: set RTO/RPO targets, implement multi-region failover, and automate switchover with Conduktor Gateway to eliminate manual errors. - [Build a Kafka Streaming Platform That Scales Without a Bottleneck](https://www.conduktor.io/blog/5-practices-for-kafka-leaders-to-build-an-efficient-streaming-platform) (2023-06-08) — Centralized Kafka bottlenecks productivity. Decentralized sprawls costs. Build hybrid streaming platforms with multi-tenancy and self-service. - [Kafka Encryption: Field-Level Without the DIY Tech Debt](https://www.conduktor.io/blog/how-to-encrypt-data-in-kafka-without-piling-up-tech-debt) (2023-06-05) — DIY Kafka encryption libraries become multi-language, multi-KMS nightmares. Proxy-layer field-level encryption eliminates tech debt across producers and cons... - [How to deploy Conduktor Gateway?](https://www.conduktor.io/blog/how-to-deploy-conduktor-gateway) (2023-06-01) — Conduktor Gateway sits between Kafka clusters and apps. Review deployment options, networking patterns, and trade-offs. - [8 Ways to Cut Kafka Costs Without Sacrificing Performance](https://www.conduktor.io/blog/8-new-ways-to-drastically-reduce-your-kafka-costs) (2023-05-30) — Kafka costs exploding? Cut storage, networking, and partition expenses 10x with virtual clusters, compression enforcement, and cold storage. - [What Kafka Users Actually Asked Us at Kafka Summit London 2023](https://www.conduktor.io/blog/kafka-summit-london-2023-feedback-from-our-booth) (2023-05-25) — Field-level encryption, multi-tenancy, topic-as-a-service, and RBAC dominated conversations at our booth. Here's what the community cares about most. - [How Conduktor Gateway Prevents Kafka Outages Before They Happen](https://www.conduktor.io/blog/proactive-safeguarding-with-conduktor-gateway) (2023-05-17) — Prevent Kafka outages with proactive safeguarding via Conduktor Gateway. Automate best practices, enforce rules, and eliminate tech debt before it starts. - [Data Mesh Explained: Decentralized Data Ownership for Scalable Organizations](https://www.conduktor.io/blog/what-is-a-data-mesh) (2023-05-04) — Data mesh decentralizes ownership for agility and scale. Learn core principles, architecture, and how Kafka + Conduktor enable domain-driven data products. - [What We Changed After Running Four Company Offsites](https://www.conduktor.io/blog/key-learnings-from-our-2023-offsite) (2023-05-01) — Conduktor's CEO Nicolas Orban shares key learnings from our 2023 offsite: quality over quantity, inclusive activities, and transparent communication. - [Kafka RBAC: Role-Based Access Control for Teams and Applications](https://www.conduktor.io/blog/security-for-humans-and-applications-in-apache-kafka) (2023-04-08) — Kafka RBAC with fine-grained permissions for humans and applications. Manage producer, consumer, and admin access without modifying broker-level ACLs. - [Data Integration: How Merging Datasets Drives Better Business Decisions](https://www.conduktor.io/blog/data-integration-the-what-why-and-benefits) (2023-03-09) — Master data integration for big data success. Explore ETL vs ELT processes, data quality benefits, and automation strategies for business intelligence. - [Change Data Capture Explained: How CDC Streams Database Changes in Real-Time](https://www.conduktor.io/blog/what-is-change-data-capture-anyways) (2023-03-09) — Change Data Capture (CDC) explained: real-time database replication, log-based streaming, push vs. pull methods, and Kafka CDC implementation patterns. - [Automate Kafka Topic Governance with Topic as a Service](https://www.conduktor.io/blog/topic-as-a-service-automate-governance-decisions-for-apache-kafka) (2023-01-31) — Automate Kafka topic governance and ACL management with Topic as a Service—eliminate bottlenecks, enforce standards, free your ops team. - [Kafka Schema Registry: How It Works & Why You Need It](https://www.conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it) (2022-11-24) — Schema Registry enforces data contracts in Kafka, preventing schema drift and producer-consumer mismatches. Covers serialization, Avro, Protobuf, evolution r... - [Kafka Log Retention and Segments: How Storage Actually Works](https://www.conduktor.io/blog/kafka-log-retention-and-segments) (2022-07-19) — Deep dive into Kafka segments, indexing, and log retention policies—optimize storage, compaction, and broker performance for production clusters. Source: https://www.conduktor.io/blog --- # No Open Roles Right Now We don't have any open positions at the moment, but we're always growing. Check back soon or reach out. We'd love to hear from you. **In the meantime:** [Careers](https://www.conduktor.io/careers) Learn about working at Conduktor [Contact Us](https://www.conduktor.io/contact?src=careers-open-roles) Get in touch [Blog](https://www.conduktor.io/blog) Latest articles Source: https://www.conduktor.io/careers/open-roles --- # Join a Team of Builders We're driven by a clear goal: helping businesses move faster and smarter by unlocking the full potential of their data streaming. [Open Roles](https://www.conduktor.io/careers/open-roles) How We Work Our team operates focused, fast, and efficient. We're a group of builders and problem-solvers, where accountability, innovation, and collaboration are fundamental to our success. --- ## Our Values Six principles we live by every day. They shape how we build, ship, and support each other. - **Embrace the Pack** — At Conduktor, we function as a trusted wolf pack, moving swiftly toward collective goals while embracing each member's unique role. - **Own It** — We build trust by turning words into action, embracing full responsibility for our commitments whether we succeed or fail. - **Act with Urgency** — We leverage our agility to outpace competitors, empowering everyone to take action and continuously seek ways to accelerate progress. - **Go the Extra Mile** — We exceed expectations through curiosity and daily improvements, creating compounding impact in a competitive market. - **No Grit, No Glory** — We embrace the unglamorous work required to scale, staying humble while tackling difficult challenges that move the needle. - **Take Smart Risks** — We create a safe environment for calculated risk-taking, knowing breakthrough innovation requires stepping beyond comfort zones. Employee Voices ## What Our Team Says > Aurélie Marcuzzo > — Senior Solution Architect, "Working at Conduktor has been such an inspiring journey. I am surrounded by intelligent people all looking in the same direction and wanting the best for our customers. The Customer Success team is also a fantastic environment for growth, learning new skills, and making a positive impact on our customers' lifecycle." > Dave Richardson > — Engineering Manager, "I came to Conduktor for the problems - Kafka infrastructure at scale is hard, you can't fake your way through it. But it's the people that have kept me here. I lead teams all over the globe, and we trust everyone to get on with it: hire good people, then stay out of their way. If you want to do serious engineering, come and do it here." > Badai Aqrandista > — Senior Technical Support Engineer, "As a Technical Support Engineer, I don't just work on tickets. Conduktor allows me to learn to fix issues like I am part of the wider Engineering team. Conduktor also pushes me to embrace AI in my work, and experiment with new things that can improve the products or help the company achieve our mission. This environment allows me to grow and learn as both a software and support engineer." > Georgina Walker > — People Partner, "From the moment I joined Conduktor, I felt welcomed and supported. I love being part of a team that values growth and collaboration. Working in the People team has been an exciting journey, offering countless opportunities to challenge myself and develop, both personally and professionally." A Note from Our Founders ## Thanks for Considering Conduktor We know choosing your next role is a big decision, and we're genuinely grateful you're taking the time to explore Conduktor. We started the company because we believed working with data shouldn't be painful. What began as a shared frustration with the complexity of tools like Apache Kafka has grown into a platform we're proud of: one that's helping teams all over the world work with data more confidently and effectively. Along the way, we've built a team we're incredibly proud of. We work fast, take bold swings, and stay deeply focused on the impact we're here to make. Everyone here takes ownership seriously. We do what we say we'll do, and when things don't go as planned, we learn and improve together. We're not afraid to challenge ourselves or roll up our sleeves when it counts. We know the hard, unglamorous work is often what moves the needle most. We're still early in our journey, and there's so much more to come. If you're someone who thrives in a fast-moving environment, loves solving real problems, and wants to be part of a humble and driven team, we'd love to meet you. **Nicolas & Stéphane** — *Co-founders* Hiring Process ## How We Hire A straightforward process designed to respect your time. 1 Show Us Your Skills You'll complete a challenge — like a presentation or problem-solving task — so we can see your strengths in action. 2 Meet the Team You'll connect with key people, including your future manager and someone from our Talent team. 3 No Endless Rounds We move quickly, avoid drawn-out processes, and focus on making thoughtful decisions. Perks & Benefits ## What We Offer Benefits vary by location. Select your region to see details. UK --- - **Private Medical Insurance** — Vitality - **Pension** — 3% contribution to Smart Pension - **Life Insurance** — Yes - **Income Protection** — Yes - **Office** — London, UK - **Office Benefits** — Weekly lunch, monthly socials & Cycle to Work scheme - **Holidays** — 25 PTO - **Perks** — ClassPass membership - **Work from Home** — 2 days/week - **Offsite** — Annual Company Offsite - **Swag** — New starter swags - **Referral Bonus** — £1K before and £2K after probation US --- - **Private Medical Insurance** — Justworks - **401K** — 401K contribution plan (employee only) - **Office** — New York, USA - **Office Benefits** — Monthly socials - **Holidays** — Unlimited PTO - **Perks** — ClassPass membership - **Work from Home** — 2 days/week - **Offsite** — Annual Company Offsite - **Swag** — New starter swags - **Referral Bonus** — $1K before and $2K after probation France --- - **Private Medical Insurance** — Alan Blue - **Holidays** — 25 PTO - 9 RTT, "Prime de vacances" in June - **Perks** — ClassPass membership - **Work from Home** — Yes - **Offsite** — Annual Company Offsite - **Swag** — New starter swags - **Referral Bonus** — €1K before and €2K after probation - **Co-working Space** — $60/month allowance Ready to Join Us? [View Open Roles](https://www.conduktor.io/careers/open-roles) Source: https://www.conduktor.io/careers --- # Conduktor Community Edition The modern Kafka UI and API. Free forever. Discover and explore topics, messages, and schemas in real time. Ship faster without CLI scripts or custom tooling. [Install Now](https://www.conduktor.io/get-started?src=community) [Read the Docs](https://docs.conduktor.io/) Used by developers at ## Let's Not Make Kafka Harder Than It Needs to Be Topics pile up, consumer groups fall behind, and schemas drift. Conduktor makes it easy for developers to get instant visibility into Kafka from day one so they spend less time investigating and more time shipping. - **Core Kafka Operations** — Create, inspect, and manage topics, schemas, connectors, and consumer groups across all your clusters. — #simplify-kafka-operations - **Advanced Data Exploration** — Debug by browsing, filtering, and tailing messages in real time. Produce, replay, and auto-decode Avro, Protobuf, and JSON Schema. — #accelerate-troubleshooting - **Real-Time Monitoring** — Catch issues before they become incidents. Monitor clusters, topics, and consumer lag in real time. — #monitor-what-matters - **Insights Built In** — Health scoring, risk detection for data loss and under-replication, cost optimization, and config recommendations. — #know-where-to-focus - **API, CLI & MCP** — Full programmatic access via REST API, CLI, and AI agent integration through MCP. — #community-automation - [SSO & Access Control](https://docs.conduktor.io/guide/conduktor-in-production/admin/user-access/configure-sso#configure-sso-in-console) — OIDC and LDAP authentication with granular RBAC, included free. ## Who It's For - **Solo Developers** — One interface for local, staging, and dev clusters. Stop switching between CLI tools. - **Startups & Small Teams** — Share a Kafka control plane with your team. SSO and RBAC included. - **Platform Prototyping** — Evaluate Conduktor before an org-wide rollout. Everything you need to prove value. - **Open Source Users** — Upgrade from CLI tools without losing access to your infrastructure. ## Running in 5 Minutes ### Quick Start (Docker) The classic [Conduktor Community](https://www.conduktor.io/community) install. Kafka UI and API, preconfigured with embedded Redpanda, Schema Registry, and sample data. Up and running in under 5 minutes. ```bash curl -L https://releases.conduktor.io/quick-start -o docker-compose.yml && docker compose up ``` Or install Console only and connect your own Kafka: ```bash curl -L https://releases.conduktor.io/console -o docker-compose.yml && docker compose up ``` Access Console at `localhost:8080` once all containers are healthy. ### AI Coding Assistant Recommended The [Conduktor skill](https://github.com/conduktor/skills) gives your AI coding assistant full agentic access to the Conduktor CLI and API. It works with all versions of [Conduktor Console](https://www.conduktor.io/console), so it can discover, configure, and operate your Kafka platform on your behalf. ```bash npx skills add conduktor/skills ``` Then ask your agent: *"install Conduktor and set it up so I can login."* Works with Claude Code, Cursor, VS Code Copilot, Gemini CLI, and 30+ other tools. [See what the skill can do →](https://www.conduktor.io/blog/set-up-a-kafka-platform-with-an-ai-agent) Core Kafka Operations ## Simplify Kafka operations One interface for clusters, topics, schemas, and connectors across Confluent, AWS MSK, Redpanda, or self-managed. No CLI required. - **Unified management** for topics, schemas, and connectors in one UI - **Multi-cluster support** without context switching - **Native integrations** with Confluent Schema Registry and Kafka Connect - **Manage consumer groups** including offsets, assignments, and membership [Learn more about core Kafka operations →](https://docs.conduktor.io/guide) Create Topics Manage Clusters Inspect Schemas Data Exploration ## Accelerate troubleshooting Instant visibility into Kafka data without custom tooling or code, reducing time-to-resolution. - **Debug faster** by browsing, filtering, and tailing messages in real-time - **Test without deploying** by producing messages directly to topics - **Replay and recover** by reprocessing messages without code changes - **Auto-decode formats** including Avro, Protobuf, and JSON Schema [Learn more about data exploration →](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics#topic-consume) Browse Messages Produce Messages Decode Formats Observability ## Monitor what matters Continuous monitoring across clusters, topics, and consumers so you can catch issues before they become incidents. - **Real-time monitoring** for clusters, topics, and consumer lag - **Schema evolution tracking** with version history and change comparison [Learn more about observability →](https://docs.conduktor.io/guide/monitor-brokers-apps) Insights ## Know where to focus Actionable recommendations for health, cost, and risk across your entire Kafka estate. - **Health optimization** with partition, retention, and config recommendations - **Risk scoring** for data loss and under-replication - **Cost optimization** to surface unused, inactive, or over-provisioned topics [Learn more about Insights →](https://www.conduktor.io/solutions/use-case/kafka-health-and-risks) Automation Programmatic access and modern AI agent integration. ## Reduce operational burden ### REST API & CLI Full programmatic access to Console capabilities. ```bash curl \ --request GET \ --url http://localhost:8080/public/iam/v2/group/project-a \ --header "Authorization: Bearer $ADMIN_API_KEY" | jq . # Response: { "apiVersion": "v2", "kind": "Group", "metadata": { "name": "project-a" }, "spec": { "displayName": "Project A", "permissions": [...] } } ``` ### MCP Integration Operate Kafka with AI agents via natural language. ```json { "mcpServers": { "conduktor-console": { "command": "npx", "args": [ "mcp-remote", "https:///api/mcp", "--header", "Authorization: Bearer ${CONDUKTOR_API_TOKEN}" ], "env": { "CONDUKTOR_API_TOKEN": "your-api-token" } } } } ``` ## What Our Users Say > Conduktor has transformed our Kafka management. Its intuitive interface and robust real-time monitoring tools have streamlined our operations, cutting troubleshooting time by over 30%. > — **[Domen Kirn Krefl](https://www.linkedin.com/in/domen-kirn-krefl-51a395144/)**, Backend Engineer, Greentube > Out of all the Kafka tools I've used, Conduktor is by far the easiest to implement and has the most thorough set of features. Conduktor makes it easy to navigate and troubleshoot issues. > — **Chuck**, Production Support Engineer, Fermenta Energy > Conduktor helped us visualize and understand our data streams without the complexity of manual CLI commands. The interface was intuitive, and it made interacting with Kafka much easier. > — **[Abdelaziz Cheffi](https://www.linkedin.com/in/abdelaziz-cheffi/)**, Data Analyst, Opalean ## Frequently Asked Questions **Is Community Edition really free?** Yes. Free forever, no credit card, no trial expiration. Community is a full-featured product purpose-built for developers and small teams, not a stripped-down enterprise trial. **What's the difference between Community Edition and Team Edition?** Community Edition includes everything you need to operate Kafka: topics, schemas, connectors, monitoring, Insights, API, and SSO for up to 50 users and 3 clusters. [Team Edition](https://www.conduktor.io/pricing#console-team-edition) adds federated ownership, topic policy enforcement, advanced Insights with chargeback, unlimited clusters and users, group-level RBAC, and unlimited audit logs. Buy online from $125/seat/month. **How is Community different from open-source Kafka tools?** Open-source tools give you basic CRUD operations. Community adds intelligent features on top: topic health insights, cost optimization, advanced data exploration with filtering and SQL queries, full REST API/CLI/MCP automation, and SSO. It's a professional-grade tool, not just a basic viewer. **Can I use Community in production?** You can, but Community is designed for development. Smaller teams do run it in production. Most enterprises need [Team Edition or Enterprise](https://www.conduktor.io/pricing#console-team-edition) for production workloads. **When should I consider upgrading?** When you need more users, more clusters, or production-grade features. Your data and configuration carry over seamlessly. **Which Kafka providers are supported?** Any Kafka 2.5+, including Confluent Platform, Amazon MSK, Redpanda, Aiven, and self-managed Apache Kafka. **Can I run it in an air-gapped environment?** Yes. The Docker images work without internet access. Deploy on-premises with full functionality. ## Get Started with Conduktor Community Self-hosted, free forever. Up and running in minutes. [Get Started Free](https://www.conduktor.io/get-started?src=community) [Read the Docs →](https://docs.conduktor.io/) Ready to go further? [Team Edition from $125/seat/month →](https://www.conduktor.io/pricing#console-team-edition) Source: https://www.conduktor.io/community --- # Delta Sharing Alternatives: When You Need Real-Time Data Sharing Delta Sharing is a strong choice for sharing tables and datasets with partners. But if your partners need live events, positions, or payments as they happen, keep reading. [Talk to Us](https://www.conduktor.io/contact/demo?src=compare-delta-sharing) ## Delta Sharing Is Good at What It Was Built For Delta Sharing is an open protocol created by Databricks for secure data sharing across organizations. A provider shares Delta Lake tables through their catalog, and recipients read them with the tools they already use: pandas, Apache Spark, Power BI, or their own Databricks workspace. No proprietary lock-in on the recipient side, no full-copy exports flying around as CSVs. For analytical sharing, that model works well. Curated datasets, ML training data, reporting extracts, data marketplace listings: if the partner's question is "give me the current state of this dataset so I can query it," Delta Sharing answers it cleanly. The searches for "Delta Sharing alternatives" usually come from a different place. The need shifts when what you share is a *stream*: payment events a partner must react to in seconds, vehicle or shipment positions updating continuously, inventory changes that drive a partner's replenishment logic. Shared tables can be read incrementally, but these are events a partner subscribes to with a stream-native client, not a dataset they query. - **What table sharing solves** — A partner needs a dataset: current state, queryable, refreshed on a schedule. Analytics, BI dashboards, model training, reconciliation. Delta Sharing and similar data exchange protocols were designed for exactly this. - **Where stream sharing fits better** — A partner needs to *react* to events as they occur: a payment authorized, a truck rerouted, a stock level crossing a threshold. Kafka-native consumption gives them a long-running subscription to every event, rather than reads over a shared table. ## Batch Table Sharing vs Real-Time Stream Sharing These are architectural characteristics, not a scorecard. The two models are built for different data shapes, and each is the right tool for its shape. | Dimension | Batch table sharing (Delta Sharing) | Real-time stream sharing (Conduktor Partner Zones) | |---|---|---| | Data shape | Tables and datasets: rows the recipient queries | Event streams: continuous records the recipient subscribes to | | Freshness | As fresh as the latest table version or refresh cycle | Continuous: partners receive events as they're produced | | Consumption model | Query snapshots, change feeds, or Spark streaming over shared tables | Kafka-native clients continuously consume records from shared topics | | Partner access | REST-based protocol via pandas, Spark, Power BI, or Databricks | Standard Kafka clients (Java, librdkafka, any language) | | Governance & masking | Catalog-governed table and view controls; row/column masking support depends on the sharing mode | Per-partner field-level masking and encryption applied at the wire, on the same source topic | | Partner-side infrastructure | A query engine or connector that speaks the protocol | Any Kafka client; no replica cluster or pipeline on either side | | Typical use cases | Analytics, BI, ML training, reporting, data marketplace products | Operational integration: payments, logistics tracking, live inventory, telemetry | ## When to Choose Which Start from the data shape and the partner's job, not from the tooling. If your case is analytical, Delta Sharing is a good answer and you should use it. - **Delta Sharing: partners query, they don't react** — Your partner loads data into a warehouse, a notebook, or a BI tool and runs queries. Scheduled freshness (hourly, daily) is fine. Curated datasets, training data, data marketplace products. - **Delta Sharing: both sides live in the lakehouse** — The recipient already works in Databricks or Spark, governance runs through a catalog, and the sharing relationship is fundamentally "here is a dataset." - **Delta Sharing: the data is naturally batch** — Monthly statements, end-of-day positions, aggregated metrics. Turning batch data into a stream adds complexity for no benefit. - **Stream sharing: partners react to events** — Payment authorizations, fraud signals, shipment positions, stock movements. A table refreshed every hour arrives too late to act on. - **Stream sharing: no export pipeline to build** — Partners consume from the live source of truth. No ETL job that dumps streams into tables, no replica cluster, no per-partner integration code. - **Stream sharing: different views of one stream** — One partner sees full records, another sees masked fields, both from the same topic, with access removed by deleting the partner's zone when a contract ends. Many organizations end up with both: streams for operational integration, tables for analytics. The same events that partners consume live can land in Delta tables for the analytical sharing relationship. ## How Conduktor Does Real-Time Data Sharing Conduktor Gateway is a wire-level proxy that sits in front of your existing streaming infrastructure. [Partner Zones](https://www.conduktor.io/solutions/use-case/data-sharing) are its answer to external sharing: partners consume live streams directly, under your rules, without any data being copied. - **No replica, no pipeline.** Partners read from your existing cluster through an isolated zone. There's no export job, no mirrored cluster, no secondary copy of the data to secure and reconcile. - **Per-reader masking and encryption.** Field-level masking and encryption are applied per partner at the proxy. Two partners consume the same topic and see different views of it, enforced before data leaves your organization. - **Standard clients on the partner side.** Partners connect with any standard Kafka client. No SDK, no API wrapper, no protocol translation layer to build or maintain. - **Centralized audit evidence.** Gateway audit and Partner Zone events centralize access and configuration activity, and per-zone traffic analytics track bytes, records, and consumer lag: the evidence trail compliance teams ask for before approving a data sharing or data monetization deal. - **Clean offboarding.** When a partnership ends, delete the Partner Zone and partner access is removed; credentials are time-limited, so keep TTLs tight for fast cutoff. No provider-side replica or mirrored cluster remains to clean up. Data a partner already consumed sits on their side, as with any sharing protocol. These guarantees hold when the Gateway is the only network path to the brokers; the proxy sits inline on the data path, so treat it as critical infrastructure (HA, capacity) like the cluster itself. This is the streaming counterpart to what Delta Sharing does for tables: share without copying, govern per recipient, keep control at the provider. It's part of a broader [data streaming platform](https://www.conduktor.io/product/data-streaming-platform) that covers governance, security, and self-service on the infrastructure you already run. ## Share live streams with partners, without copying data. Partner Zones give external consumers governed, real-time access to your streams: per-partner masking, centralized audit evidence, clean offboarding. See it on your own use case. [Talk to Us →](https://www.conduktor.io/contact/demo?src=compare-delta-sharing) ## Frequently Asked Questions **Is Delta Sharing real-time?** Delta Sharing shares tables, and recipients see data as of the latest shared table version. Providers can update tables frequently, and Spark can even read a shared table as a streaming source, so freshness can reach minutes. But the unit of sharing remains the table: it's a pull model over datasets, not a push model over events. For partners that must react to individual events in seconds, a stream-native approach fits better. **Can I use Delta Sharing and real-time stream sharing together?** Yes, and many organizations should. Delta Sharing covers the analytical relationship: curated datasets, reporting, ML training data. Partner Zones cover the operational relationship: live events partners react to. The same source events can feed both, so the two are complements, not competitors. **What do partners need to consume a shared stream?** A standard Kafka client and the credentials you issue for their Partner Zone. They point the client at your Conduktor Gateway endpoint and consume. No custom SDK, no integration library, no infrastructure on their side beyond the client they likely already run. **How is access controlled in real-time stream sharing?** Each partner gets an isolated zone with its own credentials, an explicit list of accessible topics, per-partner rate limits, and field-level masking or encryption rules. Access and configuration activity is logged centrally. When the relationship ends, delete the zone; credentials also expire by TTL, and since no provider-side copy was created, there's no replicated infrastructure to clean up. Source: https://www.conduktor.io/compare/delta-sharing-alternatives --- # Kafka Alternatives: What You're Actually Comparing When people search for "kafka alternatives," they're usually asking one of two things: is there a better messaging system, or is there a better Kafka distribution? The answers are pretty different. [Talk to Us](https://www.conduktor.io/contact/demo?src=compare-kafka-alternatives) ## Alternatives to Apache Kafka (Different Protocol) These are messaging systems with different architectures. Migrating from Kafka requires rewriting producers and consumers. - **Apache Pulsar** — Multi-tenant pub/sub with separate compute and storage layers. Stronger multi-tenancy than Kafka natively. Smaller ecosystem; steeper operational curve. Used by organizations with very large message volumes or complex multi-region topologies. - **AWS Kinesis** — Managed streaming on AWS. Fully serverless, no infrastructure to manage. Limited throughput (1MB/sec per shard, 1000 records/sec). Tight AWS lock-in. Good for AWS-native workloads with moderate throughput needs. - **Google Pub/Sub** — Google's managed pub/sub service. At-least-once delivery (no ordering guarantees). Serverless and fully managed. Works well for event-driven architectures on GCP with variable workloads. - **RabbitMQ** — Traditional message broker. Point-to-point and pub/sub patterns. Not designed for log retention or replay. Better for task queues than event streaming. Different use case from Kafka. ## Kafka-Compatible Alternatives (Same Protocol, Different Distribution) These are Kafka-compatible systems or managed Kafka services. Existing Kafka applications work without code changes. You're changing the broker distribution, not the protocol. - **Confluent Platform / Cloud** — The most feature-complete commercial Kafka distribution. Includes Schema Registry, ksqlDB, Connect, and enterprise tooling. Also offers WarpStream (S3-backed, acquired 2024) for cost-sensitive workloads. IBM announced acquisition of Confluent in December 2025 (pending regulatory approval). - **Amazon MSK** — Managed Kafka on AWS. Handles broker provisioning, patching, and replication. Standard Apache Kafka, no Confluent-specific features. Good for AWS-native teams that want managed infrastructure without vendor lock-in. - **Redpanda** — Kafka-compatible broker rewritten in C++. No JVM, no ZooKeeper, giving lower operational overhead and lower latency than standard Kafka. Growing ecosystem (220+ connectors via Redpanda Connect). Best for latency-sensitive or cost-sensitive workloads. - **Azure Event Hubs** — Kafka-compatible managed service on Azure. Supports the Kafka protocol. Existing Kafka applications connect with minimal config changes. Good for Azure-native organizations wanting managed infrastructure with Kafka compatibility. ## Where Conduktor Fits Conduktor is not an alternative to Kafka. It's the management and governance layer that runs on top of any Kafka distribution. - **Multi-Team Management** — Kafka brokers don't manage teams. Conduktor adds ownership models, self-service portals, and access controls so 50 teams can run on one Kafka platform without constant platform team intervention. - **Field-Level Encryption** — No Kafka distribution provides field-level payload encryption natively. Conduktor Gateway adds it transparently, without changing application code. - **Governance Automation** — Topic naming conventions, retention policies, and schema standards enforced as code, not documentation. Works with any Kafka broker. - **Unified Operations** — One Console for all Kafka clusters, regardless of distribution. See MSK, Confluent Cloud, and self-managed Kafka in one unified view. Most organizations don't need a Kafka alternative. They need better tooling to run Kafka they already have. Conduktor is that tooling. ## Frequently Asked Questions **Should we migrate from Kafka to Pulsar or Redpanda?** Migration from Kafka is a significant undertaking with real risk. Before migrating, identify what specific problem you're trying to solve. Multi-tenancy? Conduktor adds it to Kafka. Latency? Redpanda is worth evaluating. Fully managed? MSK or Confluent Cloud. **Is Redpanda really a drop-in Kafka replacement?** For most use cases, yes. The Kafka protocol is compatible. Edge cases exist around Kafka Streams, KRaft, and some connector configurations. Test your specific workloads before migrating. **Confluent vs. open-source Kafka — what's the real difference?** Confluent adds Schema Registry, ksqlDB, Confluent Control Center, and commercial support on top of Apache Kafka. Open-source Kafka is the same broker. The question is whether you need Confluent's ecosystem or prefer to use Conduktor + Apache Kafka for the management and governance layer. **Does Conduktor work with Confluent as the broker?** Yes. Conduktor Console and Conduktor Gateway work with Confluent Platform and Confluent Cloud. Some organizations use Conduktor as a vendor-neutral management layer even when running Confluent brokers, because it provides features Control Center doesn't, particularly for multi-team governance. **What's the best Kafka alternative for cost reduction?** Redpanda or MSK with tiered storage can reduce infrastructure costs. WarpStream (now part of Confluent Cloud) uses S3-backed storage to cut per-byte costs. Conduktor's [cost attribution and chargeback](https://www.conduktor.io/blog/chargeback-attribute-map-kafka-costs-to-your-business) features help identify waste regardless of which broker you run. ## See What Conduktor Adds to Your Kafka Most "Kafka alternative" searches end with "better Kafka tooling." 30-minute demo: multi-team governance, field-level encryption, and self-service on top of the Kafka infrastructure you already have. [Talk to Us](https://www.conduktor.io/contact/demo?src=compare-kafka-alternatives) [Get Started Free →](https://www.conduktor.io/get-started?src=compare-kafka-alternatives) Source: https://www.conduktor.io/compare/kafka-alternatives --- # Kafka Proxy Comparison: Which One Is Right for Your Team? Most Kafka proxy solutions solve one piece of the puzzle. Here's how the major options compare across the capabilities that matter for platform teams managing Kafka at scale. ## The Main Kafka Proxy Solutions in 2026 There are six major Kafka proxy options worth evaluating in 2026, ranging from routing-only proxies and API gateway extensions to open-source projects and purpose-built solutions. Here's a quick overview of each before we get into the detailed comparison. - **Conduktor Gateway** — Purpose-built Kafka proxy for platform teams. Layered multi-tenancy, DR readiness with chaos testing, composable best practice enforcement, and a growth path into data security, quality, and partner sharing. - **Confluent Gateway** — Routing and migration proxy included with Confluent Private Cloud or available as a Cloud add-on. Handles cluster switching and credential management. No multi-tenancy or policy enforcement at the proxy layer. - **Kroxylicious** — Open-source Kafka proxy backed by Red Hat. Community edition is GA but unsupported. Paid version via Red Hat's Kafka distribution is a Technology Preview, not recommended for production. - **Gravitee Kafka Gateway** — API management platform that added Kafka support in 2024. Topic aliasing, rate limiting, message filtering, protocol mediation. Kafka-native policies maturing rapidly. - **Aklivity Zilla** — Primarily a protocol translator (HTTP/MQTT/gRPC to Kafka). Zilla Plus adds virtual clusters with ACLs and quotas for Kafka proxy use cases. Seed-stage startup. - **Kong Event Gateway** — API management platform that added a native Kafka proxy (GA announced Q4 2025). Virtual clusters with ACLs, message-level encryption. Requires Kong Konnect SaaS control plane. New to Kafka proxies? Read our guide on [what a Kafka proxy is and why you need one](https://www.conduktor.io/kafka-proxy). ## Kafka Proxy Feature Comparison The table below compares each Kafka proxy across the capabilities that come up most in platform team evaluations: multi-tenancy, disaster recovery, policy enforcement, data security, and operational tooling. | Capability | Conduktor | Confluent GW | Kroxylicious | Gravitee | Zilla Plus | Kong Event GW | |---|---|---|---|---|---|---| | Multi-tenancy | ✓ Layered isolation | Network-level only | Prefix naming | Topic aliasing | ACL-based | ACL-based | | DR readiness | ✓ API-driven + chaos testing | Redeploy required | Via redeploy | Enterprise tier | DNS-based | — | | Best practice enforcement | ✓ Composable interceptors | Auth/routing policies | Custom filters | Rate limiting, governance rules | Rate limiting, quotas | ACLs, auth mediation | | Schema Registry governance | ✓ Auth, access control, audit | Not in current release | — | Validation only | Validation only | Validation only | | Data security | ✓ Field-level + masking + tokenization | Client-side (CSFLE, bypassable) | Record-level only | Message-level (AES-GCM) | Field-level + masking | Message-level, static keys | | Data quality | ✓ Schema + business rules at wire | Broker-level + Stream Governance (client-side) | JSON schema only | Governance rules | Schema validation | Schema validation | | External data sharing | ✓ Partner zones with isolation | — | — | Connectivity via mediation | Connectivity via translation | Connectivity via mediation | | Operational tooling | ✓ Terraform (GA), CLI, runtime API | YAML + CFK, restarts required | YAML only, redeploy | Terraform (tech preview), API | YAML (Data Platform early access) | Konnect UI, Terraform, API | | Self-managed deployment | ✓ | ✓ | ✓ | ✓ | ✓ (AWS) | Requires Konnect SaaS | | Any Kafka vendor | ✓ | Confluent + Apache Kafka only | ✓ | ✓ | ✓ | ✓ | | Enterprise readiness | ✓ Years of deployments | v1.1.0 (Oct 2025) | GA (unsupported) | GA | Seed-stage | GA announced Q4 2025 | | License | Commercial | CPC/Cloud license | Free (+eng cost) | $30k/yr (Starter) | ~$15k/yr | Konnect (custom) | ## Which Kafka Proxy Should You Use? The right choice depends on what you already run and how complex your Kafka governance needs are. Here's a quick decision guide. - **Use Conduktor Gateway if...** — You have multiple teams sharing Kafka and need layered isolation, DR readiness, best practice enforcement, and a growth path into data security and quality. The go-to for platform teams managing Kafka at scale. - **Use Confluent Gateway if...** — You're a Confluent-only shop that only needs routing and migration. No multi-tenancy or governance requirements. Willing to accept YAML-only config with restarts for changes. - **Use Kroxylicious if...** — You have a strong platform engineering team, simple requirements, and genuinely want to own the codebase. The open-source version is free but unsupported. Be prepared to build and maintain everything yourself. - **Use Gravitee if...** — You already run Gravitee for REST APIs and want to add Kafka governance under the same platform. Good if the unified developer portal is a primary requirement. Kafka depth is maturing but still shallower. - **Use Aklivity Zilla if...** — Your primary need is protocol translation for non-Kafka clients (MQTT, HTTP, gRPC). Zilla Plus adds some Kafka proxy capabilities, but limited enterprise adoption and tooling. - **Use Kong Event Gateway if...** — You already run Kong for REST APIs, want consolidation, and the SaaS control plane requirement isn't a concern. Kafka proxy depth is limited compared to a purpose-built solution. ## What makes Conduktor Gateway different Every proxy on this list can route traffic. Conduktor is the only one built to handle the full lifecycle of Kafka proxy needs, from day-one operations through governance and data protection. Here's where the approach diverges: - **Multi-tenancy that actually isolates.** Other proxies offer naming conventions or basic ACLs. Conduktor layers virtual clusters, ACLs, and traffic control policies together. The difference shows up when your tenth team onboards and one of them starts producing 10x the expected volume. - **DR you can prove works.** Switching clusters is the easy part. Conduktor adds chaos testing so you can validate that your applications will survive a failover before an actual outage forces you to find out. - **Composable best practice enforcement.** Individual policies like rate limiting are useful. Conduktor's interceptors stack per virtual cluster, so you can enforce different rules for different teams on the same infrastructure without custom development. - **Schema registry governance.** Every other proxy on this list either ignores the schema registry or uses it for validation. Conduktor puts a governance layer in front of it: modern authentication, per-subject access control, and audit logging on every operation. - **Wire-level data security.** Client-side enforcement means any team that skips the library skips the rules. Conduktor enforces encryption, tokenization, and signing at the proxy, covering every client regardless of language, framework, or who deployed it. - **External data sharing with isolation.** Exposing Kafka data to partners through protocol mediation is one thing. Conduktor adds dedicated partner zones with isolation and traffic control so an external consumer can't impact your internal infrastructure. [Explore Conduktor Gateway →](https://www.conduktor.io/gateway) ## Conduktor Gateway: built for platform teams, not just routing. Layered multi-tenancy, DR readiness, best practice enforcement, and a growth path into data security and quality. Works with any Kafka provider. [Talk to Us →](https://www.conduktor.io/contact/demo?src=compare-kafka-proxies) ## Frequently Asked Questions **Conduktor Gateway vs Confluent Gateway** Confluent Gateway is a routing proxy for cluster migration and DR. Conduktor Gateway is a purpose-built Kafka proxy with layered multi-tenancy, DR readiness with chaos testing, composable best practice enforcement, and a growth path into data security and quality. Confluent's encryption (CSFLE) is client-side and bypassable; Conduktor enforces at the wire. **Conduktor vs Kong Event Gateway and Gravitee Kafka Gateway** Kong and Gravitee are API management platforms that added Kafka support. Both offer basic Kafka policies but lack the depth of a purpose-built proxy: layered isolation, composable interceptors, and Schema Registry governance. Kong also requires a SaaS control plane. **Conduktor vs Kroxylicious** Kroxylicious is an open-source Kafka proxy backed by Red Hat. It's free but unsupported, and the paid version is a Technology Preview not recommended for production. Your team owns all engineering: custom filters, YAML config, and every upgrade. Most organizations exceed the cost of a commercial product within the first year. **Conduktor vs Aklivity Zilla** Zilla is primarily a protocol translator for non-Kafka clients. Zilla Plus adds some Kafka proxy features (virtual clusters, ACLs). If your primary need is protocol translation, Zilla is purpose-built for that. For Kafka-native proxy governance, Conduktor provides significantly more depth. **Does Conduktor Gateway work with any Kafka provider?** Yes. Confluent, MSK, Redpanda, Aiven, open-source Apache Kafka. Same governance layer across all of them. Source: https://www.conduktor.io/compare/kafka-proxies --- # Kafka UI Tools: Which One Is Right for Your Team? Most Kafka UI tools are built first for developers troubleshooting clusters. They work well locally; the trade-offs show up when multiple teams need ownership, access requests, auditability, and automation. [Talk to Us](https://www.conduktor.io/contact/demo?src=compare-kafka-ui-tools) [See the V/O/A framework →](https://www.conduktor.io/kafka-ui) ## The Main Kafka UI Tools in 2026 - **Kafbat UI** — Community-maintained successor to Provectus kafka-ui at github.com/kafbat/kafka-ui. Apache 2.0, active in 2026, with multi-cluster management, RBAC, data masking, Swagger API docs, and MCP server support. No ownership model or self-service workflows. - **AKHQ** — Apache 2.0 Kafka UI covering topic management, topic data, consumer groups, schemas, and Connect. AKHQ v0.25 rewrote ACLs for multi-cluster RBAC. No self-service workflows or ownership model. - **Redpanda Console** — Formerly Kowl, now part of the Redpanda platform. Source-available under BSL, Kafka-compatible, and optimized for Redpanda clusters; Kafka Connect support is community-supported by Redpanda. - **Conduktor Console** — Commercial Kafka UI for multi-team operations. RBAC, self-service workflows, multi-cluster support, ownership model, audit logs, APIs, CLI, and Terraform automation. Some data security controls, such as field-level encryption, come from Conduktor Gateway. ## Where They Differ - **Open-Source vs. Commercial** — Kafbat UI and AKHQ are Apache 2.0 open-source. Redpanda Console is source-available under BSL. Conduktor is commercial, with a free self-hosted Community Edition for core Console use. - **Access Controls** — Kafbat and AKHQ both have configurable RBAC. Redpanda Console supports ACL management and enterprise identity features for Redpanda environments. Conduktor adds team/application ownership and self-service access workflows. - **Schema Registry** — All major tools support Schema Registry browsing or management. Conduktor adds schema ownership tracking through application resources and supports both Confluent Schema Registry and AWS Glue. - **API & Automation** — Conduktor exposes REST APIs for Console and Conduktor Gateway resources, plus CLI and Terraform workflows. Kafbat includes Swagger UI for its API; AKHQ and Redpanda Console are less focused on external automation. ## Feature Comparison Table | Feature | Kafbat UI | AKHQ | Redpanda Console | Conduktor | |---|---|---|---|---| | Topic management | ✓ | ✓ | ✓ | ✓ | | Message browser | ✓ | ✓ | ✓ | ✓ | | Consumer group management | ✓ | ✓ | ✓ | ✓ | | Schema Registry | ✓ | ✓ | ✓ | ✓ | | Kafka Connect | ✓ | ✓ | ✓ | ✓ | | Multi-cluster | ✓ | ✓ | Limited | ✓ | | RBAC / Access Controls | Limited | Limited | Limited | ✓ | | Team Ownership Model | — | — | — | ✓ | | Self-Service Workflows | — | — | — | ✓ | | REST API | Limited / Swagger | — | — | ✓ | | Terraform Provider | — | — | — | ✓ | | Field-Level Encryption | — | — | — | ✓ (Conduktor Gateway) | | Audit Logging | — | — | — | ✓ | | Data Masking | ✓ | Limited | ✓ | ✓ | | Alerting (Slack, Teams, Email) | — | — | — | ✓ | | [Data Quality Monitoring](https://www.conduktor.io/console#drive-data-driven-decisions) | — | — | — | ✓ | | [Self-Healing Connectors](https://www.conduktor.io/console#self-healing-connectors) | — | — | — | ✓ | | [Cost Attribution / Chargeback](https://www.conduktor.io/blog/chargeback-attribute-map-kafka-costs-to-your-business) | — | — | — | ✓ | | [MCP / AI Integration](https://www.conduktor.io/mcp) | ✓ | — | — | ✓ | | License | Apache 2.0 | Apache 2.0 | BSL | Commercial | ## Which Kafka UI Should You Use? - **Use Kafbat UI if...** — You want a feature-rich open-source Kafka UI for local development or small-team use. You can manage RBAC and configuration yourself, and you don't need application ownership or self-service workflows. - **Use AKHQ if...** — You need a mature open-source option with multi-cluster RBAC and broad Kafka compatibility, and don't need self-service workflows or an ownership model. - **Use Redpanda Console if...** — You're running Redpanda and want the UI best aligned with that distribution. It also works with standard Kafka clusters, with Kafka Connect support handled by the community. - **Use Conduktor Console if...** — You have multiple teams on Kafka and need governed access requests, ownership, audit logs, multi-cluster governance, and API/CLI/Terraform automation. ## Frequently Asked Questions **Is Conduktor free to use?** Conduktor Community Edition is the free self-hosted tier for core Console use, currently up to 3 Kafka clusters and 50 users. Advanced governance, multi-team, and security features require a paid plan. **Can I migrate from another Kafka UI to Conduktor?** Yes. Conduktor connects to the same Kafka clusters and Schema Registry instances. There's no data migration; you're changing the management layer. **Does Conduktor work with Confluent Cloud?** Yes. Conduktor Console connects to Confluent Cloud as a standard Kafka cluster and can be used as a complementary management and governance interface. **What about open-source alternatives to Conduktor?** Kafbat UI and AKHQ are good open-source options for smaller deployments. Both have RBAC, but neither is built around application ownership, self-service access workflows, or platform-team automation. ## Conduktor Console: built for teams, not just individuals. RBAC, ownership model, self-service, audit logs, and API/CLI/Terraform automation for Kafka teams. Community Edition available. [Get Started Free](https://www.conduktor.io/get-started?src=compare-kafka-ui-tools) [Talk to Us →](https://www.conduktor.io/contact/demo?src=compare-kafka-ui-tools) Source: https://www.conduktor.io/compare/kafka-ui-tools --- # Conduktor Console *Visibility. Ownership. Autonomy.* The modern Kafka UI and API that your teams deserve. Platform teams enforce standards and give developers the autonomy to ship fast. [Talk to Us](https://www.conduktor.io/contact/demo?src=console) [Buy Team Edition →](https://www.conduktor.io/pricing#console-team-edition) ## Kafka Scales. Operations Don't. Kafka has become critical infrastructure, but managing it at scale is broken. Developers and platform engineers often face their own set of distinct problems in the organization. Console is the [Kafka governance platform](https://www.conduktor.io/kafka-data-governance) that solves both under one control plane. **For Developers** Visibility and discovery into their apps, topics, schemas, and resources. Instantly. **For Platform Teams** Governance that scales. An ownership model that enforces standards across your organization. Core Kafka Operations ## Simplify Kafka operations Let developers operate Kafka effectively. One interface for clusters, topics, schemas, and connectors across Confluent, AWS MSK, Redpanda, or self-managed. No CLI required. - **Unified management** for topics, schemas, and connectors in one UI - **Multi-cluster support** without context switching - **Native integrations** with Confluent Schema Registry and Kafka Connect - **Manage consumer groups** including offsets, assignments, and membership [Learn more about core Kafka operations →](https://docs.conduktor.io/guide) Create Topics Manage Clusters Inspect Schemas Data Exploration ## Accelerate troubleshooting Give developers instant visibility into Kafka data without custom tooling or code, reducing time-to-resolution. - **Debug in production** by browsing, filtering, and tailing messages in real-time - **Test without deploying** by producing messages directly to topics - **Replay and recover** by reprocessing messages without code changes - **Auto-decode formats** including Avro, Protobuf, and JSON Schema [Learn more about data exploration →](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics#topic-consume) Browse Messages Produce Messages Decode Formats Observability ## Prevent issues before production Let developers see what's really going on with continuous monitoring, proactive alerting, and historical trend analysis that integrates with your existing stack. - **Real-time monitoring** for clusters, topics, and consumer lag - **Proactive alerts** to Slack, Teams, PagerDuty, and webhooks - **Prometheus integration** for unified observability [Learn more about observability →](https://docs.conduktor.io/guide/monitor-brokers-apps) [Kafka Monitoring →](https://www.conduktor.io/product/kafka-monitoring) Monitor Clusters Alert Teams Assess Risk Security & Access Control ## Secure without slowing down Let platform teams meet compliance requirements with governed access, data masking, and complete audit trails. Without blocking developer productivity. - **SSO integration** with OIDC and LDAP - **Granular RBAC** at user, group, and resource levels - **Field-level masking** for PII and PHI data - **Complete audit trails** for compliance tracking - **Deploy anywhere** on-premises, air-gapped, or cloud. Your data stays where you decide [Learn more about security →](https://www.conduktor.io/kafka-security) Control Access Mask Data Audit Actions Insights ## Drive data-driven decisions Let platform teams create accountability by making governance gaps, cost attribution, and data quality visible across your entire Kafka estate. - **Health optimization** with partition, retention, and config recommendations - **Risk scoring** for data loss and under-replication - **Governance visibility** into standards compliance - **Cost attribution** by team and cluster for chargeback - **Data quality monitoring** with CEL-based validation rules [Learn more about Insights →](https://www.conduktor.io/solutions/use-case/kafka-health-and-risks) Optimize Health Track Usage Validate Quality Federated Ownership ## Automate guardrails at scale Give developers autonomy over their Kafka resources while platform teams maintain control through automated policy enforcement. - **Self-service provisioning with automatic guardrails** lets developers create topics, schemas, and connectors within safe, policy-validated boundaries. - **Application catalog** defines ownership patterns so every resource maps to an accountable team. - **Topic catalog** enables discovery across clusters while protecting sensitive resources. - **Approval workflows** let owners approve or reject access requests without ticket queues. [Learn more about federated ownership →](https://www.conduktor.io/federated-ownership) Enforce Policies Catalog Apps Catalog Topics Approve Requests Automation Infrastructure-as-code, self-healing connectors, and AI agents that operate Kafka through MCP and the CLI — all inside the same RBAC, through the tools your team already uses. ## Reduce operational burden ### REST API & CLI Full programmatic access to Console capabilities — and the operational surface coding agents use, via the Conduktor CLI and Skills, to act inside your RBAC. ```bash curl \ --request GET \ --url http://localhost:8080/public/iam/v2/group/project-a \ --header "Authorization: Bearer $ADMIN_API_KEY" | jq . # Response: { "apiVersion": "v2", "kind": "Group", "metadata": { "name": "project-a" }, "spec": { "displayName": "Project A", "permissions": [...] } } ``` ### Terraform Provider Manage Kafka resources as code with GitOps and CI/CD pipelines. ```terraform terraform { required_providers { conduktor = { source = "conduktor/conduktor" version = "~> 1.0" } } } provider "conduktor" { console_url = "http://localhost:8080" api_token = var.conduktor_api_token } resource "conduktor_group" "project_a" { name = "project-a" display_name = "Project A" permissions { resource_type = "TOPIC" name = "project-a-*" pattern_type = "PREFIXED" permissions = ["topicConsume", "topicProduce"] } } ``` ### Self-healing Connectors Automatic restarts keep pipelines running. ### MCP Integration Ask your Kafka questions in plain language. MCP runs read-only inside Console, so AI inherits each user's exact RBAC and reasons over live metadata, schemas, ownership, and monitoring. Smarter because Console holds the context, safer because Console holds the permissions. ```json { "mcpServers": { "conduktor-console": { "command": "npx", "args": [ "mcp-remote", "https:///api/mcp", "--header", "Authorization: Bearer ${CONDUKTOR_API_TOKEN}" ], "env": { "CONDUKTOR_API_TOKEN": "your-api-token" } } } } ``` [Explore Conduktor MCP →](https://www.conduktor.io/mcp) [Learn more about automation →](https://docs.conduktor.io/guide/conduktor-in-production/automate) Measurable Impact Real results from platform teams using Conduktor. - **80–90% faster onboarding** — Teams provision resources themselves instead of waiting in ticket queues. - **50–70% fewer incidents** — Schema validation catches breaking changes before they reach production. - **Weeks to hours** — Audit preparation with automated evidence generation and access logs. - **100+ teams, same headcount** — Self-service and automation let platform teams scale without hiring. ## Read more customer stories - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Bitvavo: DORA & MiCA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [Smart Farming: 5B Events per Day](https://www.conduktor.io/customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk) ## Ready to Try Console? Self-hosted license. From $125/seat/month, delivered by email. [Buy Team Edition](https://www.conduktor.io/pricing#console-team-edition) [Talk to Us →](https://www.conduktor.io/contact/demo?src=console) Solo developer or smaller team? [Explore Community Edition →](https://www.conduktor.io/community) ### Explore Console by Use Case [Kafka UI →](https://www.conduktor.io/kafka-ui) · [Kafka Monitoring →](https://www.conduktor.io/product/kafka-monitoring) Source: https://www.conduktor.io/console --- # Book a demo, and we'll cover with you: **One control layer across every provider.** Unify authentication, controls, and standards over Confluent, MSK, Aiven, Redpanda, or open-source, one model instead of one per cluster. **Encryption and masking on the wire.** Field-level encryption with your own KMS keys, enforced at the proxy with no code changes, so different consumers see different data. **Self-service with guardrails.** Teams provision topics in minutes within the limits you set, so you keep standards without becoming the bottleneck. **Governed access for AI.** MCP, the CLI, and agents discover and query every cluster, each inheriting the user's exact RBAC. ## You're in good hands **An investment fund** reclaimed ~30% of a 50+ TB cluster with an AI assistant on the Conduktor MCP, deferring a bigger-cluster purchase. [Read the story →](https://www.conduktor.io/customer-stories/investment-fund-kafka-rightsizing-for-ai) **CDC Informatique**, IT for Caisse des Dépôts (one of France's largest public financial institutions, under ACPR banking supervision), scaled Kafka 4× (40 → 160 apps) with governed self-service while keeping its platform team flat. [Read the story →](https://www.conduktor.io/customer-stories/cdc-informatique-governed-kafka-self-service) **Swiss Post** scaled to 800+ users and 5× more applications on governed Kafka. [Read the story →](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) [And many more →](https://www.conduktor.io/customer-stories) Source: https://www.conduktor.io/contact/demo --- # *Free Kafka Cost Analysis* Most Kafka estates we look at are overpaying by 25 to 40 percent. Get a free cost analysis with our field engineering team and find out what is recoverable in yours. ## What to Expect Part offline analysis, part working session (about 30 minutes live). We take a look at your data and come back with recommendations and a custom ROI model to size the impact of each change together. It is a teaching exercise, not a pitch. Recommendations stand on their own, with or without Conduktor. **Recommendations grounded in your data** Cluster consolidation, naming standards, topic right-sizing, guardrails, and team-level chargeback. Whatever your estate actually needs, prioritized by impact. **An ROI estimate built around your setup** We customize our internal ROI model to your org so you see the cost and the gain of each change before you commit. **A recommended sequence** Where to start, what to phase, and what to leave alone. **Direct time with the field team** Hands-on with the engineers who run this exercise across other estates. Bring the messy questions; you will get straight answers. What to Have Ready A recent Kafka or cloud bill Cluster list with labels (environment and what each one is used for) Topic config and usage data, or admin access Your platform owner in the session Don't have all this data? That's fine. We can show you how to gather what is missing as part of the session. Source: https://www.conduktor.io/contact/kafka-cost-analysis --- # Contact Us Product questions, pricing, trials, partnerships, or just curious about Kafka? We're here to help. Tell us what you're working on and we'll get back to you shortly. Source: https://www.conduktor.io/contact --- # Smart Farming: 10x Kafka Utilization on AWS > "As our IoT data streams increased, managing Kafka provisioning through a central team became a bottleneck. We needed to give our developers more autonomy to keep up with demand." - Solutions System Architect ## Executive Summary This agricultural equipment manufacturer transformed its Kafka operations on AWS with Conduktor, boosting developer productivity, eliminating provisioning bottlenecks, and laying the groundwork for scalable governance and security in their IoT-driven smart farming initiatives. - **10x** — Kafka utilization - **70%** — faster provisioning ## Challenge Based in the United States, this organization is a major producer of agricultural equipment, including tractors, combines, and forestry machinery. After acquiring a company specializing in IoT devices for farming equipment, they intended to integrate these technologies to power real-time data streaming across their smart agricultural machinery. However, as their Kafka usage on AWS expanded, several challenges emerged: - **Centralized bottlenecks** — A central platform team managed all Kafka provisioning, which created scalability concerns and slowed down delivery of new data-driven features - **Limited developer autonomy** — Developers lacked self-service capabilities to provision and manage Kafka resources independently, leading to delays and reduced agility - **Growing complexity** — As IoT data requirements grew, so did the need for strong governance and data security to maintain compliance and operational efficiency ## Solution To overcome these challenges, this organization deployed Conduktor to streamline Kafka operations and empower their development teams. > "With Conduktor, we've streamlined our Kafka operations on AWS, making it easier to manage multiple clusters while ensuring security and compliance." - Solutions System Architect Key improvements included: - **[Self-service Kafka management](https://docs.conduktor.io/guide/use-cases/self-service)** — Developers gained autonomy to manage Kafka resources without relying on the platform team, accelerating deployment of new use cases - **Scalable [governance](https://docs.conduktor.io/guide/insights/governance)** — Conduktor provided centralized governance tools, ensuring compliance and security as the organization scaled its data streaming operations - **Enhanced security** — The organization is evaluating [Conduktor Shield](https://docs.conduktor.io/guide/reference/data-security) to strengthen data protection and ensure regulatory compliance as their IoT-driven initiatives grow ## Results Since implementing Conduktor, this organization has seen transformative results: - **10x increase in real-time data utilization** — By empowering developers with self-service controls, the organization removed operational bottlenecks, enabling faster decision-making and innovation - **70% reduction in Kafka provisioning times** — By eliminating centralized bottlenecks, they significantly sped up Kafka resource deployment - **Streamlined Kafka operations on AWS** — Conduktor simplified the management of 10 Kubernetes clusters, ensuring smooth and efficient data streaming workflows - **Foundation for future growth** — With scalable governance and data security in place, the organization is positioned to fully leverage IoT data streaming for smart farming innovation > "Conduktor has transformed our Kafka operations on AWS, giving our teams the autonomy they need to innovate while setting the foundation for scalable governance and security as we grow." - Solutions System Architect ## Frequently Asked Questions **How does Conduktor work with Kafka on AWS?** Conduktor integrates seamlessly with Kafka running on AWS to provide centralized management, governance tools, and self-service capabilities while maintaining AWS security and compliance standards. **What is self-service Kafka management?** Self-service Kafka management allows developers to provision and manage Kafka resources independently without relying on platform teams, accelerating deployment of new use cases. **How does Conduktor Shield enhance data protection?** Conduktor Shield provides additional data protection capabilities for sensitive IoT data, ensuring regulatory compliance as data streaming operations scale. ## Related Customer Stories - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) — **Logistics** — 800+ users. "Conduktor gave us time back. Developers can now help themselves." - [Retail: 99% Faster Kafka Onboarding](https://www.conduktor.io/customer-stories/one-hour-kafka-onboarding-retail-streaming-operations) — **Retail** — 70% faster. "Developers get a working Kafka environment within an hour." - [FlixBus: Data Mesh for 50+ Teams](https://www.conduktor.io/customer-stories/flix) — **Transportation** — Self-service. "Conduktor easily integrated with our CI/CD pipelines." Source: https://www.conduktor.io/customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk --- # Bitvavo: DORA & MiCA Compliance for 1.5M Users ## Executive Summary [Bitvavo](https://bitvavo.com/en/markets), one of Europe's leading cryptocurrency trading platforms, serves over 1.5 million users with a secure and intuitive user experience. As their customer base grew, their [Confluent Cloud](https://www.confluent.io/confluent-cloud/) infrastructure expanded to handle thousands of Kafka topics and partitions. Scaling wasn't just about handling more data—it meant navigating complex operational and regulatory challenges. Conduktor enabled Bitvavo to streamline workflows, enhance oversight, and meet strict security standards while staying ahead of evolving regulatory demands. ## Challenge Bitvavo's engineers knew the high stakes: Kafka served as the backbone of their platform, powering real-time trading, customer interactions, and critical data exchanges. As the platform scaled, several challenges emerged: - **Limited visibility** — Developers often struggled to locate existing Kafka topics, leading to inefficiencies and unnecessary data duplication - **Compliance risks** — Sensitive information could flow through the cluster in non-compliant ways, making [GDPR](https://gdpr-info.eu/) compliance a daunting task - **Mounting regulatory requirements** — [DORA](https://www.eiopa.europa.eu/digital-operational-resilience-act-dora_en) and [MiCA](https://www.esma.europa.eu/esmas-activities/digital-finance-and-innovation/markets-crypto-assets-regulation-mica) regulations added new layers of operational and security demands with severe financial penalties for violations For Bitvavo, addressing these challenges was vital to maintaining operational resilience and safeguarding their business in an increasingly regulated market. ## Solution The search led them to Conduktor. From the first conversation, it was clear: this wasn't just a tool; it was a framework for resilience and growth. - **Enhanced compliance** — Conduktor's granular [Role-Based Access Control (RBAC)](https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac) and detailed audit trails offered the compliance backbone they needed. For the first time, they could enforce access rules aligned with GDPR's strict requirements. - **Data security** — [Field-level data masking](https://docs.conduktor.io/guide/reference/data-security) protected sensitive information, even in development environments. Engineers could collaborate securely without exposing unnecessary data—critical for DORA requirements. See the operating model behind MiCA and GDPR retention on Kafka in [When Kafka Becomes the Ledger](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger). - **Discoverability** — Teams gained visibility into existing [topics](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), eliminating data duplication and cutting costs. ## Results With compliance frameworks established and operational challenges resolved, Bitvavo is now focusing on future-proofing its infrastructure. A critical priority is meeting DORA's stringent failover requirements. Current manual processes during incidents can result in extended downtime—a risk that's untenable in a market where trust and uptime are paramount. Conduktor's automated cluster switching and failover capabilities allow for significantly faster Recovery Time Objectives (RTO). With over 100 team members actively using Conduktor, Bitvavo has achieved more than just compliance—they have established a resilient, efficient, and scalable foundation for sustained growth. ## Frequently Asked Questions **How does Conduktor help with DORA compliance?** Conduktor provides automated cluster switching and failover capabilities for faster Recovery Time Objectives (RTO), meeting DORA's stringent operational resilience requirements for financial institutions. **What security features does Conduktor offer for cryptocurrency platforms?** Conduktor offers granular Role-Based Access Control (RBAC), field-level data masking, detailed audit trails, and strict access rules—all essential for GDPR, DORA, and MiCA compliance. **How does data masking work in Conduktor?** Field-level data masking protects sensitive information even in development environments, allowing engineers to collaborate securely without exposing unnecessary data. ## Related Customer Stories - [Instant Payments for European Banks](https://www.conduktor.io/customer-stories/securing-kafka-banking-financial-services) — **Financial Services** — PCI DSS compliance. "Conduktor ensures we extract data's full value before it goes stale." - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) — **Transportation** — 800+ users. "RBAC was the turning point for our compliance." - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) — **Aviation** — End-to-end encryption. "The tools made our migration seamless." Source: https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica --- # CDC Informatique: 4x Kafka Growth, Flat Team > "A good sign, for us, is that we don't hear much about Kafka." - Julien Maillard, Senior Architect, CDC Informatique ## Executive Summary CDC Informatique runs the IT for Caisse des Dépôts, one of France's largest public financial institutions, under ACPR banking supervision: HDS and ISO 27001 certified, sensitive health, disability, and pension data in flight, a few hours of tolerated downtime a year. Kafka arrived in 2019 bundled inside a Cloudera Big Data distribution. Developers loved it, adoption took off, and by 2021 the platform underneath, never built for that scale, hit a wall: no naming standard, no ACL convention, no ownership model. CDC Informatique rebuilt Kafka as a governed, self-service *product*: an internal GitOps "resources as code" layer, plus [Conduktor](https://www.conduktor.io/console) on the human-access path and the [Gateway](https://www.conduktor.io/gateway) on the data path. In three years it scaled 4x, to ~300 internal users, on a platform team that barely grew. - **40 → 160** — applications in 3 years - **5** — people on the platform team - **~30 min** — to onboard a new pattern The full account is in [the webinar Julien recorded with us](https://www.conduktor.io/events/webinars/cdc-parcours-kafka-avec-conduktor) and its written companion, [Good Kafka Is Boring Kafka](https://www.conduktor.io/blog/good-kafka-is-boring-kafka). ## Challenges Kafka is async, durable, and easy to plug into, so it spread from a single ingestion pipeline into cross-application messaging faster than anyone set standards for it. > "We ran into real trouble with Kafka in 2021." - Julien Maillard - **A regulated, high-stakes environment**: online banking under ACPR supervision, HDS and ISO 27001 certified, handling health, disability, and pension data. Full on-premise, dedicated VLANs per cluster, no crossing of environments, integrity absolute and traceability non-negotiable. - **Governance that couldn't keep up**: resource creation was centralized and scattered across teams at once, with no naming convention, no ACL standard, no ownership model, no shared patterns. - **No usable interface**: most people who touch Kafka build Spring apps, write Python, or run QA. The bundled UI failed on security and performance, so every non-developer who needed to inspect a topic or test a message had to find one. - **A cluster at the mercy of its clients**: with no control point, any client could ship uncompressed data, set `acks=0` and silently drop messages, or over-partition a topic, and you only found out from a dashboard, after it hurt. ## Solution CDC Informatique paused, secured sponsorship across production, engineering, and leadership, and re-laid the foundations in a six-month replatforming: out went the Cloudera bundle, in came dedicated clusters owned by a single team. > "Nothing in life is easy, but we got the sponsorship, and we had the courage to do it." - Julien Maillard ### Govern first, then delegate > "The biggest advice I can give is to think about governance from the moment you stand up the platform, and to design it so you can delegate it to the project teams, inside a frame you can industrialize." - Julien Maillard CDC Informatique built an internal GitOps tool on a "resources as code" model. Each project team declares the desired state of its Kafka world in one Git file: topics, ACLs, schemas, connectors, Kafka Streams apps, service accounts. The tooling reconciles it, applies controls such as partition caps, and enforces logical isolation per application and environment through naming rules and ACLs. About 99% of requests are standard; the other 1% get a conversation with the platform team. Conduktor provides the same shape under its [self-service framework](https://www.conduktor.io/blog/governed-kafka-self-service): a team declares an application and what it owns, and the platform derives the ACLs and ownership boundaries. The key move was pushing access decisions [left, onto the project teams](https://www.conduktor.io/blog/no-kafka-data-platform-without-ownership), who alone know whether one domain is business-authorized to read another's data. Any team can now stand up a PostgreSQL CDC connector and an outbox pattern in about half an hour, with no central team in the path. ### A console for the people who aren't Kafka people > "The UI in the distribution didn't work for us, not on security, not on performance." - Julien Maillard [Conduktor Console](https://www.conduktor.io/console) gave the non-experts a way in: search and filter across topics, produce a test message without code, diff two Avro schema versions, replay events, all behind SSO and an audit trail, mirroring the isolation the platform already enforces. First-level troubleshooting moved to the project teams; the central team only takes the genuinely hard cases. > "Console is the product our users really fell for. Take it away now and the reaction would be... loud." - Julien Maillard ### A control point on the data path [Conduktor Gateway](https://www.conduktor.io/gateway) sits in front of the clusters as a Kafka-native proxy. The only change for clients is their `bootstrap.servers`, so it rolled into integration and pre-production first. From that control point the team surfaces bad client configs (a producer with no compression, say) and requires a fix before production, and runs [chaos testing](https://www.conduktor.io/blog/chaos-engineering-for-kafka-testing-recovery-before-you-need-it), duplicating messages on the consumer side to confirm a replayed payment event isn't charged twice. This is the control-plane operating model [When Kafka Becomes the Ledger](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger) describes in full. > "The Gateway proxy won me over immediately." - Julien Maillard ### Compliance that holds up to an auditor Kafka usage is classified on the DICP scale (availability, integrity, confidentiality, traceability). SSO ties access to the enterprise identity system, audit logs record who did what, and the GitOps isolation is reproduced in the Console. Because ownership sits with the project teams, access decisions are made by the people accountable for them, inside guardrails the central team can show an auditor. ## Results - **4x application growth on a flat team**: 40 to 160 production applications in three years, with ~50 more in flight. The team went from three to five people and has been flat for eighteen months. - **~300 internal users enabled**: developers, QA, and analysts work independently within the guardrails instead of queuing for the platform team. - **Compliance maintained at scale**: logical isolation, SSO, audit, and DICP classification held steady through 4x growth under ACPR supervision. - **Lower support load**: first-level troubleshooting shifted to the project teams, freeing the central team for higher-value work. ## Conclusion Designed to be delegated and industrialized, governance is what lets a Kafka platform scale faster than the team that runs it. Good Kafka, it turns out, is boring Kafka, and in a regulated bank that is the highest compliment. ## Frequently Asked Questions **How did CDC Informatique scale Kafka without growing the platform team?** CDC Informatique encoded governance (naming, ACLs, ownership, partition caps) into an internal GitOps frame and Conduktor's self-service model, so project teams provision and troubleshoot their own Kafka resources within guardrails. The platform absorbed 4x application growth (40 to 160 apps) while the team stayed flat at five people. **What makes Kafka self-service safe in a regulated bank?** Self-service is safe when the guardrails are built in, not bolted on. CDC Informatique enforces logical isolation per application and environment through naming rules and ACLs, ties access to SSO, captures everything in audit logs, and classifies Kafka usage on the DICP scale, so teams move autonomously while the platform stays compliant under ACPR supervision. **How does Conduktor Gateway enforce Kafka best practices?** Conduktor Gateway is a Kafka-native proxy that sits between clients and clusters; clients only change their bootstrap.servers. From that control point, CDC Informatique surfaces and blocks risky client configurations (missing compression, unsafe acks) in lower environments before they reach production, and runs chaos tests like message duplication to verify consumer idempotency. ## Related Customer Stories - [Swiss Post: 5x Kafka Growth with 800 Users](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) — **Transport & Logistics** · 60 → 300+ apps. "RBAC was the turning point for our compliance." - [Bitvavo: DORA & MiCA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) — **Financial Services** · 1.5M+ users. "Conduktor offered the compliance backbone we needed." - [Securing Instant Payments for Banks](https://www.conduktor.io/customer-stories/securing-kafka-banking-financial-services) — **Banking** · PCI DSS, VISA, MasterCard. "We deliver data to our clients instantly." Source: https://www.conduktor.io/customer-stories/cdc-informatique-governed-kafka-self-service --- # FlixBus: Data Mesh for 50+ Teams > "Conduktor easily integrated with our CI/CD pipelines to enhance data governance. It's given us the tools to centralize compliance standards across the business, while allowing team-specific autonomy to speed up our processes." - Taras Slipets, Staff Data Engineer at Flix ## Executive Summary Flix scales Kafka data-streaming operations from 25 to 50 teams with Conduktor—optimizing fleet tracking, route efficiency, and AI-driven customer experiences across Europe and North America. - **50** — teams - **2,300+** — topics - **9,500** — partitions ## Kafka at Flix Operating at the crossroads of a tech startup, e-commerce platform, and transportation company, data is the lifeblood of Flix. Kafka plays a critical role in managing vast streams of real-time data, from vehicle locations to customer feedback, enabling Flix to: - Dynamically modify bus schedules on-the-fly - Optimize routes by analyzing traffic patterns and vehicle data - Rapidly respond to customer feedback, enhancing service quality ## Challenge As Flix expanded, so did the complexity of its Kafka infrastructure. Within five years, Kafka utilization expanded to over 2,300 topics and 9,500 partitions across 10 domains and 50 teams. > "We looked for solutions to significantly simplify the process. We didn't want to write our own tool for that kind of synchronization." - Taras Slipets, Staff Data Engineer Managing a data operation of this size presented multiple challenges: - **Complex permissions** — Managing access controls across global teams while adhering to local data governance and privacy regulations - **Access bottlenecks** — No single interface for technical and non-technical users to independently access Kafka data - **Limited discoverability** — Analysts struggled to discover available data, view schemas, and filter streams without a platform for observability - **Cost management** — Without a unified view, proactively consolidating data pipelines to prevent runaway costs was challenging ## Solution At Flix, robust data security is paramount. The sensitive nature of payment transactions and personal information required for travel bookings made data protection a critical concern. Flix adopted a resource-centric approach grounded in data mesh principles, giving teams ownership of their data within a federated control framework. ### Data mesh approach - **Resource-focused permissions** — Permissions tied to resources, centralizing control over crucial assets - **Distributed governance** — Resource owners manage access to their respective resources - **Centralized framework** — All oversight remains under a cohesive framework ensuring consistency and compliance ### Modular data architecture with Conduktor Flix implemented a modular architecture using individual YAML files for each team's Kafka resources: - **Clear ownership** — Clearly identifies the team responsible for resources within a namespace - **Contained leakage** — Limits problems with overly lenient access settings to affected resources - **Auditable changes** — CI/CD tools track all changes with merge history - **Promotes collaboration** — Teams request access through pull requests; owners approve without centralized authority ### How it works Flix's federated Kafka security management operates through open-source technologies and structured CI/CD workflows: ![Flix architecture diagram](https://www.conduktor.io/assets/customer-stories/flix-architecture.png) 1. Each team has a Kafka resource-oriented YAML file defining the owner and permissions granted to other teams 2. Changes trigger CI/CD workflows that validate configurations through federated governance 3. Files are transformed into a format compatible with Kafka Security Manager (KSM), a Conduktor open-source project 4. The same files propagate permissions into Conduktor for interaction and troubleshooting This workflow ensures each team is responsible for their resources while permissions are uniformly applied: - **[Kafka ACLs](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/service-accounts-acls)** — Control application access, ensuring only authorized applications interact with specific topics - **[Conduktor RBAC](https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac)** — Provide team members with permissions to manage, monitor, and troubleshoot their Kafka resources ## Results Through collaboration with Flix, Conduktor developed a [self-service framework](https://docs.conduktor.io/guide/use-cases/self-service) enhancing collaboration between Platform and Application teams: - **50+ teams** working independently with Kafka - **2,300+ topics** managed across 10 domains - **Reduced reliance** on central teams with intuitive self-service controls - **Automated compliance** with configurable approval workflows - **Enforced best practices** on Kafka configurations, maintaining consistency ## Frequently Asked Questions **What is a data mesh approach to Kafka governance?** Data mesh is a decentralized approach where teams own their data within a federated control framework. Flix uses resource-focused permissions with distributed governance while maintaining centralized oversight for consistency. **How does Flix manage Kafka permissions across 50+ teams?** Flix uses individual YAML files for each team's Kafka resources, tracked in CI/CD with merge history. Teams request access through pull requests that resource owners approve without centralized authority. **What is Kafka Security Manager (KSM)?** KSM is a Conduktor open-source project that transforms YAML configurations into Kafka ACLs. Flix uses it alongside Conduktor RBAC to control both application and team access. ## Related Customer Stories - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) — **Transportation** — 800+ users. "Conduktor gave us time back. Developers can now help themselves." - [Virgin Australia Saves 300 Hours/Month](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) — **Aviation** — Self-service. "It's been a game changer for efficiency and adoption." - [OpenTag: Simplified Kafka for Gaming](https://www.conduktor.io/customer-stories/opentag) — **iGaming** — Developer onboarding. "Kafka becomes available to everyone, not just specialists." Source: https://www.conduktor.io/customer-stories/flix --- # German Automaker: 3 Kafka Platforms Into 1 > "Confluent Cloud without governance on top is unusable for our enterprise. I will never let a user touch the portal directly. Everyone goes through Conduktor." - IT Project Lead, Leading German Automotive Manufacturer ## Executive Summary Apache Kafka is critical infrastructure at this German automaker. The event-driven architecture behind its vehicle sales process runs on it, which means every car the company sells, at a dealership or online, flows through Kafka. But Kafka had grown without coordination: each division (R&D, sales, production) had its own IT department, its own budget, and its own Kafka platform, each staffed by roughly ten people all doing more or less the same work. Today, over 200 applications and 700+ active users run on a single multi-tenant Kafka platform operated by one team of ten, with Confluent Cloud underneath and [Conduktor Console](https://www.conduktor.io/console) as the governance layer and API for the entire organization. Nobody touches the Confluent portal. - **200+** — applications on one platform - **700+** — active Kafka users - **3 → 1** — platform teams consolidated **Why not just use Confluent's built-in tools?** Confluent Cloud provides the broker and the ecosystem. As this team found, that governance alone doesn't cover 700+ users spread across divisions. They needed resource-level Kafka RBAC, self-service provisioning with automated guardrails, topic and application catalogs, and cross-team approval workflows. [See how Console extends Confluent](https://www.conduktor.io/partners/confluent). ## Challenges Kafka is easy to plug into, so it spread across divisions faster than anyone set standards for it. > "It became the centrally used Kafka platform under the radar. I was a firefighter for a project outside the main management focus. I operated somewhat independently within my department." - IT Project Lead One IT project lead had spent four years running the platform that started in R&D and quietly became the company-wide default. He had neither the mandate nor the tooling to support it, but he knew the platform, and he had a plan for what it should become. - **Three platforms, three teams, no coordination**: at least three separate Kafka platforms operated independently, one per division, each with roughly ten people duplicating the same work. - **200,000 runaway partitions**: users routinely created topics with 100 partitions for workloads that didn't need them, because, as the project lead put it, "they just like the number 100. One and two zeros." With no guardrails to stop them, production ballooned to 100,000 partitions and staging reached 200,000. When a broker went down, recovery took ages, with "everyone holding hands and fingers crossed." - **Shared superusers**: application teams shared a single superuser per namespace, a model the project lead called "a huge security risk as well as compliance risk." - **No way to discover data**: there was no public topic catalog, so teams could only see their own data. Finding out what other teams had was, in his words, "just not present." - **Onboarding measured in days**: requests sat in what the project lead called a "horrible" ServiceNow queue for up to a week, and cross-team access required email chains and manual approval. "Super time consuming." The stakes were rising at the same time. The company was rebuilding its entire vehicle sales process on event-driven architecture, every division was moving onto the same infrastructure, and the platform had to work. ## Solution When central IT raised the question of consolidating Kafka company-wide, the project lead was ready. He pulled out the plan he had been sitting on and became the owner of the initiative: one Kafka platform, operated by one team, serving every division. None of the existing platforms was fit to become the central service, so the company moved to Confluent Cloud for the underlying infrastructure. A German IT consultancy introduced the team to Conduktor Console, which became the single interface for the entire organization. The project lead chose it in part because he didn't want to lock into any single Kafka provider: Conduktor gave him the flexibility to switch, with all resources managed in one place. The architecture works on two levels: - **Platform team (~10 people)**: uses the Conduktor CLI and API to [automate provisioning, enforce policies, and manage infrastructure](https://www.conduktor.io/console#simplify-kafka-operations) through GitOps pipelines. - **Developers (700+)**: use the Conduktor UI as their day-to-day interface for browsing the topic catalog, creating resources, configuring alerts, monitoring connectors, and requesting access to other teams' data. Confluent Cloud runs underneath, but no one interacts with it. Everything goes through Conduktor. ### From shared superusers to individual identity With Console's [SSO and RBAC](https://www.conduktor.io/console#secure-without-slowing-down), every user authenticates via the company's identity provider and lands in Conduktor with the right group membership automatically. During onboarding, a GitOps pipeline creates owner groups in Conduktor and maps them to identity provider groups via external group mapping, which syncs users from the company directory and assigns the appropriate roles. From there, team members extend access to colleagues themselves, without involving the platform team. Role-based access control gives the platform team control over what each user and team can do without becoming a bottleneck for every request. Users pushed back at first, because they weren't used to being controlled, but that control is exactly what made it safe to put every division on the same cluster. With thousands of IT staff and each division running its own identity infrastructure, SSO and RBAC were the first things that had to work. ### Developers discover, create, and collaborate without waiting Through Console's [federated ownership](https://www.conduktor.io/federated-ownership) model, every application is registered with a defined ownership group, so topics, connectors, and schemas all map to accountable teams. No more guessing who owns what. Developers open the Conduktor UI and browse data across the entire cluster: what topics exist, who owns them, what schemas are in use. They create topics, schemas, and connectors themselves, and resource policies enforce the boundaries automatically. A cluster-wide policy caps topics at a default maximum of six partitions. > "You are only allowed to use max six partitions. For anything more, you need to get in touch with us and argue why this is required." - IT Project Lead When a team needs another team's data, they request it in the Conduktor UI and the owning team approves or rejects it directly. The platform team is no longer in the middle. ### Teams go live in 30 minutes, not five days The flow from request to a team being live on the platform: 1. A team submits a request through Jira Service Management with their application name, ID, and technical details. 2. A platform engineer validates the OAuth client ID (the technical user credential for Kafka authentication). Because that credential is provisioned in a separate system, this is the one step that couldn't be automated. 3. A ticket status change triggers a GitOps pipeline that runs sanity checks, then uses the Conduktor CLI to create application instances, application groups, owner groups, and external group mappings. 4. The team logs in via SSO and manages their use case independently. > "Within 30 minutes, a user can sign on and manage their entire use case independently. No involvement from us. Which is awesome, really." - IT Project Lead Once onboarded, developers use the Conduktor UI for their day-to-day work. The platform team also rolled out Console's built-in [alerting and connector auto-restart](https://www.conduktor.io/console#prevent-issues-before-production), which teams configure directly in the UI. Before, every team was expected to build its own monitoring for Kafka Connect failures, and most never did. It was a blind spot. ## Results During the migration, the platform team forced every team to recreate its topics from scratch. No replicator, no cluster linking, no shortcuts. Unused and oversized topics turned up everywhere, because, as the project lead put it, "people tend not to know Kafka in and out." Combined with the six-partition guardrail, that drove a 10x reduction in partitions and a direct cut in Confluent Cloud infrastructure cost. - **10x** — partition reduction - **Days → minutes** — onboarding time - **Zero** — major incidents since adoption | Before | After | |---|---| | 100,000 production partitions | 10,000 | | 200,000 staging partitions | 26,000 | | Onboarding queues of 5 to 7 days | Under an hour, typically 30 minutes | | 3 separate platforms, 3 separate teams | 1 platform, 1 team | | No guardrails on topic creation | Default max 6 partitions per topic | | Cluster instability from ungoverned growth | Zero major incidents since adoption | - **Three platforms consolidated into one**: what previously required three separate teams now requires one, cutting the total cost of ownership for Kafka infrastructure significantly. In the project lead's words, "a huge cost saving." - **10x fewer partitions**: 100,000 to 10,000 in production and 200,000 to 26,000 in staging, with a policy in place to keep it that way. - **700+ users self-sufficient**: developers discover data, provision resources, configure alerting, and grant each other access without a ticket. - **Platform team off the firefighting line**: with less time spent on manual provisioning and incidents, the team now focuses on new features and onboarding the next wave of platforms. > "It's just a different world now." - IT Project Lead ## Conclusion The consolidation isn't finished. One more platform was migrated this year and another with 500 users is in progress. The project lead who started as a firefighter and a misfit now owns the initiative reshaping how the entire company moves data, and his goal goes beyond consolidation: he wants to spread Kafka adoption across the company, replacing inefficient, time-delayed batch jobs used for moving data from A to B with smart, low-maintenance, real-time pipelines. > "I don't want to commit to any Kafka provider. Conduktor gives me that flexibility. The way application teams can onboard and manage themselves? It's just nice. For me, it was the best decision." - IT Project Lead ## Frequently Asked Questions **How did Conduktor help reduce Kafka partition counts?** Resource policies enforced a default maximum of six partitions per topic. Combined with a migration that required teams to recreate topics from scratch, this drove a 10x reduction: 100,000 to 10,000 partitions in production. **How does Conduktor handle identity and access at enterprise scale?** Conduktor integrates with enterprise identity providers via SSO and external group mapping. Combined with RBAC, this replaced shared superuser accounts with individual, auditable access across 700+ users. **How fast can new teams get started with Kafka using Conduktor?** With a GitOps-automated onboarding pipeline driven by the Conduktor CLI, new teams go from request to a working Kafka environment in under an hour, typically 30 minutes, down from up to a week. **Can Conduktor support multi-division Kafka consolidation?** Yes. This manufacturer consolidated at least three separate Kafka platforms, each run by a different division, into one Conduktor-governed platform on Confluent Cloud, operated by a single team of ten. ## Related Customer Stories - [Swiss Post: 5x Kafka Growth with 800 Users](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) — **Transport & Logistics** · 60 → 300+ apps. "RBAC was the turning point for our compliance." - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) — **Aviation** · 2,000+ users. "The tools made our migration seamless." - [Retail: 99% Faster Kafka Onboarding](https://www.conduktor.io/customer-stories/one-hour-kafka-onboarding-retail-streaming-operations) — **Retail** · 3 weeks → 1 hour. "Developers get a working Kafka environment within an hour." Source: https://www.conduktor.io/customer-stories/german-automaker-unifies-kafka-event-driven-backbone --- # Swiss Post: 5x Kafka Growth with 800 Users > "With the same team, we managed to support this expansion and scalability—Conduktor plays one of the biggest roles in that. It helps us do a lot of things automatically, and that really supports my team." - Anke Raich ## Executive Summary Swiss Post focuses on its customers. With this in mind, it also wants to modernize its systems and technology landscape to become a data-driven company—from the generation of data in the physical world to its value-creating use. Swiss Post used Conduktor to expand data access throughout their organization and centralize control for their platform teams, and provide observability for data analysts and engineers. **Conduktor helped Swiss Post achieve the following results:** - Increased their Kafka applications from **60 to over 300** - Empowered **800+ internal users** across all departments with secured access to Kafka - **Decommissioned legacy technology** to improve governance, data quality, and reduce risks - Built a **foundation for AI** and predictive models ## Challenges Swiss Post is handling millions of deliveries daily. With real-time data critical to operations—from sorting facility automation to customer analytics—Swiss Post faced an inflection point in its modernization journey. The secure, trustworthy and sustainable handling of data, algorithms and technologies in line with customer requirements is an important success factor for corporate growth for Swiss Post and its digital business models. To mitigate the increasing costs associated with legacy middleware integration licenses, the team pivoted toward an open-source, scalable architecture with Kafka as the foundation. Under Anke Raich's leadership, the platform team set a clear mission: migrate all integrations to Kafka, eliminate technical debt, and establish governance that accelerates delivery—rather than blocking it. **As Kafka adoption grew, so did the risks:** - **High-stakes operations:** Kafka powers critical processes such as sorting instructions in real time. Any misconfigurations or outages could halt sorting hubs. - **Access bottlenecks:** Developers couldn't safely test or troubleshoot without the help of the platform team, risking delays or errors. - **Scaling pressure:** As their Kafka environment grew from 60 to 300+ applications, and from 300 to 1,100+ integrations, the same platform team now had to support 800+ users. One obstacle was a lack of visibility into Kafka clusters and operations. In order to perform root cause analysis and troubleshooting, engineers were forced into lengthy, awkward workarounds, slowing down crisis responses. > "Kafka was a black box for a lot of people. Nobody knew how it worked and what it did." - Anke Raich Lacking a centralized catalog of available data, analysts also were unable to discover and utilize data. They did not know which topic contained what information, and further, had no way to validate it easily. Teams had to rely on outdated or incorrect datasets, duplicated efforts, and generally left data unused. This lack of visibility also created problems for the data governance team. While validating data quality (such as schemas and formats) was fairly straightforward, ensuring semantic quality (essentially context and accuracy) was harder. Version control was also difficult—differences between producers and consumers had negative downstream effects. ## Solution By adopting Conduktor, Anke and her team at the IT department of Swiss Post were able to reduce operational overhead, improve monitoring and visibility, and implement [self-service abilities](https://docs.conduktor.io/guide/use-cases/self-service) with centralization and guardrails—leading to more autonomy and faster innovation. Conduktor also enabled Swiss Post's data analysts, data scientists, and developers to expand real-time data usage across the organization, unlocking new use cases and harnessing the true value of their real-time data. ### Regulatory compliance and full access control Swiss Post utilizes Conduktor's granular [role-based access controls (RBAC)](https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac) to define clear roles and meet strict internal and regulatory requirements. Developers are only authorized to access the topics that their applications produce to (or consume from); data analysts and data scientists have time‑bound access for investigation purposes; and finally, platform engineers have full control and streamlined oversight. This structure is not just operational, it satisfies Swiss Post's internal revision team and ensures alignment with Switzerland's data privacy law. > "RBAC was the turning point. Before Conduktor, we couldn't enforce access safely enough for our own expectations. Now it's built into our workflows and has become very important for our compliance, because responsible and trustworthy handling of our data is key for us." - Anke Raich ### Self-service and developer autonomy Conduktor helped Anke's team become an in-house service for providing platform stability, security, and automation across Swiss Post. Since adopting Conduktor, Anke and her team have remained the same size, even as their Kafka deployment increased from 60 to over 300 applications, while users grew to nearly 800. - **300+** — applications - **~800** This user base includes approximately a third of all data scientists at Swiss Post and hundreds of analysts; the former use Conduktor to explore, discover, and validate real-time data for AI projects, while the latter use it to more rapidly find and fix data errors and anomalies. Best of all, every user can work independently, operating within the guidelines set by Anke and the platform team—without having to consult or rely on them. > "Conduktor gave us time back. Developers can now help themselves, and our team can focus on governance instead of daily firefighting." - Anke Raich Conduktor also supports teams that are at the forefront of Swiss Post's digital innovation efforts, including the team responsible for predictive geo-routing, where algorithms adjust delivery routes based on real-time traffic patterns. ### Governance and observability Conduktor also serves as a real-time control tower, providing additional visibility into [data quality](https://docs.conduktor.io/guide/use-cases/observe-data-quality) and accelerating troubleshooting workflows. Anke and her team built a Grafana dashboard to monitor the percentage of data that fell within the bounds of valid schema. > "In the dashboard, you can choose the time frame to verify that the data is valid against the schema you defined. Any messages with less than 100% validity will be displayed. Then you can click on the message, which takes you through to Conduktor to drill down further—to investigate more deeply and to correlate any possible problems." - Anke Raich This additional visibility also facilitated Swiss Post's digital transformation. By modernizing their microservices architecture around Kafka and REST APIs (and utilizing Conduktor to provide governance, automation, and monitoring), Swiss Post saved a significant amount of money—but more importantly, they were able to gain reusability, security, and speed. "Before, nobody tracked CMP classification" Anke says, referring to data classification workflows for new APIs and topics. "Now we audit and enforce it automatically during design." Infrastructure management is another area where Anke and her team rely on Conduktor. "We have hundreds of [connectors](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect)" Anke says, "and the whole monitoring and start/restart process is done with Conduktor." ## Results - **5× increase in Kafka applications** — From 60 to over 300. Integrations grew from 300 to 1,100. The same platform team now supports all this and +1,000 topics in production. - **800+ internal users empowered** — Including developers, data scientists, and analysts with Conduktor. - **Hundreds of thousands saved** — By decommissioning legacy technology while improving transparency, data quality, and governance. - **Increased AI readiness** — Kafka now supports predictive models, with a third of their data scientists actively using Conduktor to explore, find, and validate data. - **Ensured compliance** — With Swiss Federal Act on Data Protection (FADP) by introducing role-based access controls that restrict data visibility to authorized teams. ## Conclusion With Conduktor, Swiss Post turned Kafka from a "black box" into a business advantage, providing security, governance, automation, and scale while maintaining usability. The platform now underpins mission-critical processes, accelerates real-time data access for hundreds of users, and lays a solid foundation for AI-driven innovation. "Conduktor makes our Kafka platform not just usable—but **governable and future-proof,**" Anke concludes. ## Frequently Asked Questions **How did Swiss Post scale Kafka from 60 to 300 applications?** Swiss Post used Conduktor's self-service platform with centralized governance to scale from 60 to 300+ Kafka applications while keeping the same platform team size. Developer autonomy combined with RBAC guardrails enabled rapid growth without bottlenecks. **What role does RBAC play in Kafka governance?** Role-Based Access Control (RBAC) was essential for Swiss Post's compliance with Swiss data privacy law. Conduktor's granular RBAC ensures developers only access topics their applications use, analysts get time-bound access, and platform engineers maintain full oversight. **How does Conduktor help with Kafka data quality?** Conduktor provides observability into data quality through schema validation monitoring. Swiss Post built Grafana dashboards to track schema validity percentages, with Conduktor enabling drill-down investigation of anomalies. ## Related Customer Stories - [FlixBus: Data Mesh for 50+ Teams](https://www.conduktor.io/customer-stories/flix) — **Transportation** — 2,300+ topics. "Conduktor easily integrated with our CI/CD pipelines to enhance data governance." - [Virgin Australia Saves 300 Hours/Month](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) — **Aviation** — 300 hr/mo saved. "It's been a game changer for efficiency and adoption." - [European Airline: 25 Kafka Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) — **Aviation** — 2,000+ users. "The tools made our migration seamless." Source: https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage --- # Investment Fund: AI-Assisted Kafka Rightsizing > "I did the whole rightsizing of my topics almost 100% with Claude and the Conduktor MCP." - The fund's platform team ## Executive Summary A multi-strategy investment fund runs Kafka on self-managed, on-premise Confluent: market data feeding trading desks, application logs in flight, hundreds of developers producing and consuming. - **14** — brokers - **50+ TB** — cluster, ~80% full before - **~30%** — reclaimed by rightsizing The cluster was oversized from the start, retention crept up, and topics accumulated faster than anyone pruned them. By early 2026, disk-space alerts were firing repeatedly, abnormal enough that the team stopped to look closely. An AI assistant working through the Conduktor [MCP](https://www.conduktor.io/mcp) did the analysis; the team applied the changes itself, rightsizing the handful of topics driving the growth and freeing about 30% of the 50+ TB cluster. Two things came out of it right away: a much clearer picture of how the cluster was actually used, and enough headroom to settle the running debate about buying a bigger cluster. That headroom matters now: AI is starting to push more of the firm's data through Kafka toward its data lake, and the cleared cluster is what that next wave will run on. ## Challenges Kafka here had the opposite problem of most teams: not runaway growth, but a system left untouched for too long. > "Our cluster size has been very static for the last seven years. We haven't added or removed any brokers." - **Oversized at birth, never re-measured.** The cluster was built big and grew bigger by accretion. Capacity headroom hid the cost of long retention and abandoned topics until the cluster neared 80% full. - **No clear health signal.** External professional services brought in for the version upgrade never answered the basic question: what should they actually track? They wanted concrete thresholds, like how many partitions and replicas per broker is too many and what good looks like, and got reassurance instead of numbers. - **A handful of topics drove almost all the weight.** Eight to ten large, heavily replicated topics, mostly telemetry and trading data, accounted for most of the storage, while only needing about a day of retention. - **Alerts said "full", not "why".** When a disk-space alert fired, the team could see which topic was large, but not whether one or two topics had suddenly grown. A point-in-time size is not a growth signal, and chasing the difference by hand each time was slow. - **A small team runs all of it.** A lean platform team carries Kafka for the whole firm. Any cleanup or sizing analysis had to fit into a calendar already full with a separate platform migration. ## Solution The missing piece was knowing what good looked like: how many partitions and replicas per broker is safe, and which topics were actually driving the storage. Conduktor put real numbers on both. They didn't add hardware or buy a bigger cluster; they measured what they had and cut the waste, with an AI assistant doing most of the legwork. ### Find the source of truth first The trust in Conduktor predates the rightsizing. During a major incident in April 2025, roughly 1,400 topics were deleted in a day, and when the servers came back the team couldn't see all its brokers through its usual monitoring. Conduktor showed the correct state of the cluster when it mattered most. > "On the day the incident happened, Datadog was not immediate enough. But Conduktor was prompt, giving us absolutely correct information." After that, Conduktor was the team's first stop. > "Conduktor is our main visualization tool. We don't even rely on Control Center 99% of the time." ### Rightsize with an AI assistant on the MCP The analysis ran through the [Conduktor MCP](https://www.conduktor.io/mcp), which exposes the platform's metadata and metrics to an AI assistant. Instead of hand-querying topic sizes, replication factors, and retention across thousands of topics, the team worked the problem in conversation: which topics drive storage, which retention settings are excessive, what to cut. The MCP is read-only by design, so nothing changed on the cluster on its own. It surfaced the picture and the plan; the team stayed in control and applied the changes: rightsizing the eight to ten topics driving the storage, and dropping retention on the telemetry and trading-data topics that never needed more than a day. ### Operational value, not a cost cut The fund treats Kafka as shared infrastructure rather than a charged-back service, so the case for cleanup was operational, not financial: more headroom, a smaller blast radius, less risk when a spike hits. The same goes for partition count. Running on-premise, the team first read partition waste as a managed-Kafka concern, something that only shows up on a cloud bill. But on a self-managed cluster the cost is just as real, in CPU, memory, open files, and recovery time, which is the case we make in [The Surprising Cost of Kafka Partition Waste](https://www.conduktor.io/blog/the-surprising-cost-of-kafka-partition-waste). ## Results - **~30% of a 50+ TB cluster reclaimed.** Fill dropped from near 80% to a comfortable level, on the same hardware. - **The "bigger cluster" debate, settled.** The reclaimed space took buying more hardware off the table for now, ending a running internal discussion about expanding the cluster. - **A clearer read on the cluster.** The same pass left the team understanding what actually drives storage, not just that it was filling. - **AI did the analysis, the team stayed in control.** The MCP surfaced what to rightsize; the team applied the changes by hand. - **A trusted source of truth.** Through a major broker incident and day-to-day operations, Conduktor is the tool the team reaches for first. ## A foundation for what's next Rightsizing did more than free space; it reset a seven-year-old cluster into something the team can grow on. The AI-driven demand is already taking shape, with Kafka increasingly the buffer in front of Snowflake. A cluster sitting near 80% full could not have absorbed that comfortably; a de-risked one can. The rightsizing was a one-off pass, and the team wants the visibility to become permanent. The plan, not yet in production, is to use the MCP to poll every topic on a schedule and correlate total cluster usage against per-topic sizes, so the next disk alert arrives with an answer attached: which topic, or topics, just grew. That turns a point-in-time size into a growth signal, the gap that made every past alert a manual investigation. Geography is the other front. The cluster is New York-centric while its consumers span the globe, from the New York desks to Hong Kong, so latency is the next thing to solve. Rather than stand up and run regional clusters, that load can move to the edge with [Conduktor Gateway](https://www.conduktor.io/gateway): a proxy that caches hot data close to remote readers, so a record crosses the ocean once instead of for every consumer, with access and encryption policy enforced right there at the edge. Two large Kafka upgrades come first: removing ZooKeeper, then moving to the next major Confluent version. After that, the fund sees governance as the clear next step, with a replicated lower environment of representative data already standing by to trial it. > "The days of us designing our own solutions are past. We don't have time to reinvent the wheel. The next step in Kafka for me is improving our governance, and that's where something like Conduktor fits." ## Frequently Asked Questions **How did the fund reclaim 30% of its Kafka cluster?** The platform team used Conduktor Console and the Conduktor MCP to see storage, replication, and retention across the whole cluster, identified the eight to ten heavily replicated topics driving most of the weight (mainly telemetry and trading data), and rightsized them, including cutting retention on topics that only needed about a day. The cluster went from near 80% full to a comfortable level on the same hardware. **What is the Conduktor MCP and how was it used here?** The Conduktor MCP exposes the platform's metadata and metrics to an AI assistant. The team ran the topic-rightsizing analysis in conversation, almost entirely autonomously, then applied the changes itself: the MCP is read-only, so it surfaces the picture and the plan but never changes anything on the cluster, and a human stays in the loop on every action. **Does cluster rightsizing only matter on the cloud?** No. This was a self-managed, on-premise cluster with no per-partition cloud billing. The value was operational: more headroom, a smaller blast radius when a spike hits, and shorter retention on the heaviest topics, rather than a line-item cost reduction. ## Related Customer Stories - [CDC Informatique: 4x Kafka Growth, Flat Team](https://www.conduktor.io/customer-stories/cdc-informatique-governed-kafka-self-service) — **Financial Services** · 40 → 160 apps. "A good sign, for us, is that we don't hear much about Kafka." - [Bitvavo: DORA & MiCA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) — **Financial Services** · 1.5M+ users. "Conduktor offered the compliance backbone we needed." - [Securing Instant Payments for Banks](https://www.conduktor.io/customer-stories/securing-kafka-banking-financial-services) — **Banking** · PCI DSS, VISA, MasterCard. "We deliver data to our clients instantly." Source: https://www.conduktor.io/customer-stories/investment-fund-kafka-rightsizing-for-ai --- # European Airline: 25 Kafka Clusters to Cloud > "Working with Conduktor and Confluent has been a game-changer for us. The tools they provide made our migration seamless, and the level of collaboration and support has been outstanding." - Software Engineer ## Executive Summary With Conduktor, this European airline securely migrated 25 Kafka clusters and 170 applications from their on-premises environment to Confluent Cloud over the course of nine months. They now have 2,000 self-service users across their organization with stringent security and governance standards. - **25** — Kafka clusters - **170** — applications - **2,000** — self-service users This airline serves over 100,000 passengers daily across more than 1,000 flights, generating massive streams of data critical for operations and customer experience. For the past seven years, this airline used Confluent to manage their on-prem Kafka clusters, leveraging point-to-point messaging and log ingestion to deliver personalized customer experiences. But as demand for real-time data surged and new use cases emerged, managing 25 Kafka clusters across three data centers became a major challenge. ## Challenge To address the growing complexity and operational burden, the airline decided to migrate its Kafka infrastructure to Confluent Cloud by the end of 2024. However: - **Strict security protocols** required end-to-end encryption before data could reach the cloud - **Scalable governance** was needed to enforce consistent security across their architecture post-migration ## Solution In just nine months, Conduktor helped the airline seamlessly migrate to Confluent Cloud, ensuring a more scalable, secure, and flexible data environment. ### Simplifying compliance with rigid security requirements Conduktor ensured full [encryption](https://docs.conduktor.io/guide/use-cases/encrypt-kafka) for Kafka data in motion and at rest, enabling the airline to complete their migration while adhering to strict security protocols—and laying the foundation for long-term compliance. ### Consolidating Kafka operations for improved efficiency Beyond migration, Conduktor empowers the airline to implement centralized [governance](https://docs.conduktor.io/guide/insights/governance) across their new Kafka architecture. Platform engineers can now monitor, troubleshoot, and secure streaming data in a single place. By standardizing these processes, the airline reduced duplication, eliminated data silos, and guaranteed data consistency and integrity. > "With Conduktor's platform and self-service features, we finally have the governance and control we need." - Software Engineer ### Driving operational reliability and agility With improved reliability and visibility for streaming data, the airline minimizes downtime and reduces failure rates. This results in shorter mean times to detection and resolution (MTTD/R), more reliable development cycles, and a deepened commitment to data-driven innovation. ### Empowering data innovation across the organization By adopting a "shift left" approach, the airline accelerates deployment of new use cases, giving users self-service access to streaming data while maintaining robust control. This balance of speed and governance drives faster time to value without sacrificing compliance. ## Results Together, Conduktor, Confluent, and this airline have set the stage for future growth. As they explore using Kafka for partner communication, enhancing customer experiences, and strengthening core operations, they're equipped to transform their business model. > "Kafka isn't widely used for external communications yet, but we know we'll need to open up Kafka for this in the future. Using Conduktor, we're preparing for that possibility. This is why we chose the full Conduktor package—to have all these options available as we continue to grow." - Software Engineer ## Frequently Asked Questions **How long does a Kafka cloud migration take with Conduktor?** This European airline migrated 25 Kafka clusters and 170 applications from on-premises to Confluent Cloud in just nine months with Conduktor's support. **What security features does Conduktor provide for cloud migrations?** Conduktor ensures full encryption for Kafka data in motion and at rest, enabling migrations that adhere to strict security protocols while establishing long-term compliance foundations. **How does Conduktor support Confluent Cloud?** Conduktor provides centralized governance, monitoring, troubleshooting, and security management across Confluent Cloud deployments, empowering 2,000+ self-service users in this case. ## Related Customer Stories - [Virgin Australia Saves 300 Hours/Month](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) — **Aviation** — Self-service. "It's been a game changer for efficiency and adoption." - [Bitvavo: DORA & MiCA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) — **Financial Services** — Security. "Conduktor offered the compliance backbone we needed." - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) — **Transportation** — Governance. "RBAC was the turning point for our compliance." Source: https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor --- # Retail: 99% Faster Kafka Onboarding > "Developers get a working Kafka environment within an hour of requesting it. Conduktor has become the hub for hundreds of users to interact with our Kafka resources." ## Executive Summary The product teams at this leading retailer used Conduktor to troubleshoot and stress-test Kafka infrastructure, create a hybrid governance model that balances autonomy and centralization, reduce onboarding time from three weeks to one hour, and increase adoption to nearly 350 Kafka users. With hundreds of stores and tens of thousands of employees nationwide, this Fortune 500 retailer relies heavily on real-time data to power everything from online checkout to inventory sync and in-store experiences. Streaming isn't optional; it's mission-critical. ## Challenge Apache Kafka sits at the center of that mission-critical system. But as Kafka adoption expanded across teams, so did the complexity: - **Slow onboarding** — New teams waited up to three weeks to get started with Kafka - **Operational complexity** — Deploying and maintaining Kafka connectors was difficult - **Revenue risk** — The checkout function directly impacted revenue, and any failure could mean lost sales > "We had some boilerplate configs and reference architectures, but they only go so far—there was a lot of trial and error." - Senior Software Engineer The retailer also launched an initiative to modernize its online checkout. Teams needed to test Kafka components under real-world conditions to ensure they wouldn't break during peak seasons. > "If customers have to go through the checkout process more than once, they'll get frustrated and give up. We're talking about a direct correlation to lost money." - Manager of Systems Engineering ## Solution ### One hour onboarding Before Conduktor, getting a team up and running on Kafka could take up to three weeks. Today, it takes one hour. Conduktor's model of centralized [governance](https://docs.conduktor.io/guide/insights/governance) and decentralized [self-service](https://docs.conduktor.io/guide/use-cases/self-service) dramatically reduces turnaround times while increasing autonomy for developer teams without compromising security or standardization. > "We had an intern who was brand new to the Kafka space. She went from having no knowledge of Kafka to having a working connector in a week—and it was one we had never used before." - Manager ### Chaos testing for checkout The checkout team now uses Conduktor for chaos testing to validate Kafka resilience before going live. > "They wanted a way to test without causing outages across different areas. Now they've used it to test new features while keeping our shared production service healthy." - Engineer > "For e-commerce modernization, the entire order processing mechanism hinges on Kafka availability. Instead of having to react, we now use Conduktor to be proactive." - Senior Software Engineer ### Seamless integrations Conduktor integrates with Azure, SSO for auditing and access control, and [Terraform/GitHub](https://docs.conduktor.io/guide/reference/self-service-reference) for programmatically provisioning and automating Kafka resources. ## Results - **99% faster onboarding** — From 3 weeks to 1 hour - **350+ Kafka users** — Including product owners and directors who monitor real-time operations during peak seasons - **Chaos testing in place** — Protecting the checkout experience and avoiding costly outages - **Faster delivery cycles** — Reliable workflows with clear visibility and fewer bottlenecks > "Conduktor definitely eases adoptability. It enhances our onboarding process and allows it to be a smooth transition from not knowing Kafka to having everything you need right here." - Manager of Systems Engineering ## What's Next The team is exploring disaster recovery and active-active Kafka architectures using Conduktor—so even in a failover scenario, apps can switch clusters without disruption. They're also exploring new [connectors](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) to stream real-time data into recommendation engines and personalization programs as part of AI initiatives, helping shoppers find relevant products and increasing store revenue. ## Frequently Asked Questions **How fast can new teams get started with Kafka using Conduktor?** With Conduktor's centralized governance and decentralized user experience, new teams can go from zero to a working Kafka environment in just one hour—down from three weeks. **What is chaos testing for Kafka?** Chaos testing validates Kafka resilience by intentionally testing failure scenarios. This retailer uses Conduktor to test new features while keeping shared production services healthy. **How does Conduktor integrate with CI/CD pipelines?** Conduktor integrates with Azure, SSO for auditing and access control, and Terraform/GitHub for programmatically provisioning and automating Kafka resources. ## Related Customer Stories - [OpenTag: Simplified Kafka for Gaming](https://www.conduktor.io/customer-stories/opentag) — **iGaming** — Fast onboarding. "Kafka becomes available to everyone, not just specialists." - [Virgin Australia Saves 300 Hours/Month](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) — **Aviation** — 300 hr/mo saved. "It's been a game changer for efficiency and adoption." - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) — **Logistics** — Self-service. "Conduktor gave us time back." Source: https://www.conduktor.io/customer-stories/one-hour-kafka-onboarding-retail-streaming-operations --- # OpenTag: Simplified Kafka for Gaming Platform > "Once we discovered Conduktor, our lives became a whole lot easier. Once someone adopts the platform, it becomes their go-to tool and Kafka becomes available to everyone, not just those with Kafka experience." - Rea Maor, CTO & Co-founder ## Executive Summary In 2020, OpenTag managed Kafka with Kafka CLI and found the experience challenging. After partnering with Conduktor, OpenTag transformed their complex, multi-tool and multi-module Kafka experience into a powerful, streamlined workflow. OpenTag's data strategy is built on Apache Kafka. Kafka is the backbone that drives the firm's platform, consisting of multiple microservices in different modules across several major domains. Kafka transforms OpenTag's business into a real-time operation, providing scale, flexibility, and incredible access to data. ## Challenge Kafka is powerful but presents challenges due to its complexity. Before Conduktor, OpenTag found Kafka complicated to work with—it was only understood by a select few. This created several problems: - **Uneven playing field** — Only specialists could work with Kafka effectively - **Increased DevOps workload** — Senior management spent too much time on Kafka operations - **Untapped potential** — Kafka's power remained inaccessible due to high entry barriers ## Solution Conduktor brings immense value to the entire department including DevOps, Ops, and senior management. It is now the go-to tool that gives all users the knowledge and access they need to use Kafka—bringing the entire team to the same level of understanding. Key benefits include: - **Reduced workload** — Less time wasted for DevOps and senior management managing Apache Kafka - **Full Kafka utilization** — The firm can now fully access and utilize Kafka's power - **Faster time-to-market** — Using Kafka's flexibility and scalability to roll into multiple markets quickly - **Universal compatibility** — Connects to every [Kafka cluster](https://docs.conduktor.io/guide/conduktor-in-production/admin/configure-clusters) and any version (Apache, MSK, Confluent) - **Fast onboarding** — Easy sandboxing and accelerated learning curve for new team members ## Results - **Reduced DevOps workload** — Team members across all departments can now self-serve - **Faster time-to-market** — Rapid expansion into new gaming markets - **Democratized Kafka access** — Kafka is now available to everyone, not just specialists ## Frequently Asked Questions **How does Conduktor simplify Kafka for non-specialists?** Conduktor provides an intuitive interface that gives all users the knowledge and access they need to use Kafka, eliminating the steep learning curve and enabling faster onboarding. **What Kafka providers does Conduktor support?** Conduktor connects to every Kafka cluster and any version including Apache Kafka, Amazon MSK, and Confluent Cloud with universal compatibility. **How does Conduktor reduce DevOps overhead?** By providing self-service access to Kafka for all team members, DevOps and senior management spend less time on Kafka operations and more on strategic initiatives. ## Related Customer Stories - [Retail: 99% Faster Kafka Onboarding](https://www.conduktor.io/customer-stories/one-hour-kafka-onboarding-retail-streaming-operations) — **Retail** — 1 hour onboarding. "An intern went from no Kafka knowledge to a working connector in a week." - [Virgin Australia Saves 300 Hours/Month](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) — **Aviation** — Self-service. "It's been a game changer for efficiency and adoption." - [FlixBus: Data Mesh for 50+ Teams](https://www.conduktor.io/customer-stories/flix) — **Transportation** — Team autonomy. "Conduktor easily integrated with our CI/CD pipelines." Source: https://www.conduktor.io/customer-stories/opentag --- # Instant Payments for European Banks > "Data has a short useful life, and Conduktor ensures we extract its full value before it goes stale. Now, we deliver data to our clients instantly, enhancing their opportunities for innovation." ## Executive Summary This organization processes millions of transactions monthly, acting as the central hub for payment traffic. To modernize, they adopted Kafka for real-time data movement. Conduktor helped them achieve compliance with PCI DSS, MasterCard, and VISA standards while democratizing access to Kafka across teams. ## Challenge Scaling Kafka for mission-critical workloads unveiled significant challenges: - **Compliance in a regulated environment** — Operating under strict regulations from MasterCard, VISA, and PCI DSS, the organization needed robust security controls, [encrypted data streams](https://docs.conduktor.io/guide/use-cases/encrypt-kafka), and tightly managed user access. Continuous auditing and governance consumed valuable resources. - **Kafka complexity** — Kafka's powerful capabilities came with a steep learning curve. Its complexity restricted access to a select group of specialists, creating bottlenecks that slowed development and hindered cross-team collaboration. - **Rising costs** — Expanding Kafka's ecosystem with Zookeeper, [Kafka Connect](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect), and [Schema Registry](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry). The organization faced mounting costs and struggled to maintain performance and security at scale. ## Solution The organization adopted Conduktor to simplify Kafka operations, improve security, and enhance visibility across their real-time data infrastructure. > "Initially, we were searching for a Kafka GUI that we could run on separate instances for development, testing and production, but found a platform that let us centrally manage our Kafka infrastructure." - Development Engineer ### Building a secure real-time data framework The organization built a secure framework by integrating Kafka with hardware security modules (HSMs) and encryption processes. Conduktor added centralized [Role-Based Access Control (RBAC)](https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac), Single Sign-On (SSO), and detailed audit logs, enabling compliance with VISA, MasterCard, and PCI DSS standards while reducing the overhead of manual governance. For the control-plane operating model behind payment data on Kafka, read [When Kafka Becomes the Ledger](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger). ### Democratizing access to Kafka Conduktor bridged the gap between Kafka's technical complexity and the organization's need for broader collaboration. Administrators now intuitively manage [topics](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), environments, and security settings, while non-technical users access and consume real-time data without mastering command-line tools. ### Empowering teams with ownership and visibility With visibility into Kafka operations, developers are now empowered to inspect topics, validate data accuracy, and monitor data flow. This reduced inter-team dependencies, streamlined debugging, and accelerated development cycles. ## Results Through Conduktor, the organization migrated legacy batch processes to real-time streaming. This transformation enabled them to provide up-to-date balance updates to banking customers and significantly improved operational agility. - **PCI DSS and card network compliance** — Full compliance with MasterCard, VISA, and PCI DSS standards - **Batch-to-realtime migration** — Legacy processes transformed to real-time streaming - **Accelerated time to market** — Reduced inter-team dependencies and streamlined development - **Organization-wide Kafka adoption** — Non-technical users can now access real-time data > "We're excited about the future of Kafka and how Conduktor will help us increase data use across the business. We've made several requests to the Conduktor team, and they've implemented our suggestions—so we're confident we chose the right Kafka management platform." ## Frequently Asked Questions **How does Conduktor help with PCI DSS compliance?** Conduktor provides centralized Role-Based Access Control (RBAC), Single Sign-On (SSO), and detailed audit logs that enable compliance with PCI DSS, VISA, and MasterCard standards. **What is batch-to-realtime migration?** Batch-to-realtime migration transforms legacy scheduled data processes into continuous streaming, enabling instant balance updates and real-time transaction visibility for banking customers. **How does Conduktor democratize Kafka access?** Conduktor bridges Kafka's technical complexity gap—administrators manage topics and security intuitively, while non-technical users access and consume real-time data without command-line knowledge. ## Related Customer Stories - [Bitvavo: DORA & MiCA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) — **Financial Services** — 1.5M users. "Conduktor offered the compliance backbone we needed." - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) — **Aviation** — Encryption. "The tools made our migration seamless." - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) — **Transportation** — RBAC. "RBAC was the turning point for our compliance." Source: https://www.conduktor.io/customer-stories/securing-kafka-banking-financial-services --- # Virgin Australia Saves 300 Hours/Month > "With Conduktor, it's so much easier for developers, analysts, and testers to get a handle on the data they need, how it all flows, and resolve issues quickly. It's been a game changer for efficiency and adoption." - Scot M, Platform Engineer ## Executive Summary By accelerating onboarding and simplifying workflows, Virgin Australia reclaimed up to 300 hours per month for their developers and testers. They also improved visibility and standardization across their Kafka environment, accelerating development cycles and data utilization. - **300 hr/mo** — saved ## Challenge To maintain its competitive edge, Virgin Australia embarked on an ambitious initiative to modernize and replatform its legacy systems, with Kafka at the core. However, the team soon encountered challenges: - **Limited visibility** — Teams lacked intuitive tools to effectively visualize and troubleshoot Kafka data flows - **Complex user access** — Provisioning users onto Confluent Cloud and controlling access for diverse user groups was time-consuming and inefficient - **Fragmented operations** — Without centralized tools, collaboration and operational efficiency were hindered To address these issues, Virgin Australia implemented Conduktor in 2022, empowering users to monitor, manage, and maintain their Kafka data flows. ### Growing pains As Kafka adoption grew, provisioning licenses and maintaining local installations became increasingly difficult for the platform team, creating bottlenecks for innovation. "Early on, we had to provision accounts, issue API keys, and manage individual licenses," Scot McPhee, a Platform Engineer at Virgin Australia, explains. "It was okay for developers but cumbersome for other users like testers and analysts." ## Solution To address these challenges and standardize their Kafka user experience, Virgin Australia upgraded to Conduktor at the end of 2024. > "The migration to Conduktor was pretty seamless and made user provisioning and access to Kafka significantly easier. Updates are now instantly available to everyone, and features like SSO integration or user group management ease our operations and increase Kafka adoption." - Scot M, Platform Engineer With Conduktor, Virgin Australia teams could now: - **Centralize deployment** — Automatic user updates, eliminating the need for individualized software installations - **Simplify [user management](https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac)** — Facilitating operations like onboarding and access provisioning - **Gain comprehensive visibility** — Powerful built-in tools and integrations with third-party observability solutions - **Prevent issues proactively** — Rapidly detecting, investigating, and debugging issues before they impact downstream applications - **Accelerate development cycles** — Enabling testers and developers to efficiently access and validate Kafka data ## Results The use of Conduktor has delivered quantifiable improvements: - **Increased efficiency** — Roughly 48 hours per week saved for testers and 20 hours per week for developers - **Widespread adoption** — A diverse range of users, including testers and analysts, now actively use Kafka - **Shortened investigations** — Teams use advanced tools and visualizations to better understand and troubleshoot their Kafka environment Beyond these measurable gains, Scot highlighted the "unquantifiable" value: "Being able to share data seamlessly and collaborate across teams without friction has saved countless hours. It's improved the quality of questions and insights we receive, which boosts overall productivity in ways that are hard to quantify." ## What's Next Virgin Australia utilizes Kafka to power several real-time use cases, such as streaming flight data to internal and external consumers, as well as ingesting booking information from third-party travel sites to update flyer profiles. With Kafka and Conduktor, the Virgin Australia team can support even more business verticals and streaming data applications, positioning it for future growth in a complex, competitive industry. ## Frequently Asked Questions **How much time did Virgin Australia save with Conduktor?** Virgin Australia reclaimed up to 300 hours per month—roughly 48 hours per week for testers and 20 hours per week for developers through simplified workflows and centralized deployment. **How does Conduktor improve Kafka onboarding?** Conduktor simplifies user management and provisioning, enabling testers, analysts, and developers to efficiently access and validate Kafka data without extensive technical knowledge. ## Related Customer Stories - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) — **Aviation** — 2,000+ users. "The tools made our migration seamless." - [Retail: 99% Faster Kafka Onboarding](https://www.conduktor.io/customer-stories/one-hour-kafka-onboarding-retail-streaming-operations) — **Retail** — 3 weeks to 1 hour. "Developers get a working Kafka environment within an hour." - [FlixBus: Data Mesh for 50+ Teams](https://www.conduktor.io/customer-stories/flix) — **Transportation** — 50+ teams. "Conduktor easily integrated with our CI/CD pipelines." Source: https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor --- # Customer Stories How enterprises across aviation, finance, retail, and logistics scale Kafka with Conduktor. ROI, scale, and lessons learned. Trusted by industry leaders - **Automotive** — German Automaker Unifies Three Kafka Platforms for 700+ Users — customer-stories/german-automaker-unifies-kafka-event-driven-backbone — Platform unification and migration · 200+ apps across R&D, production, and sales - **Financial Services** — Investment Fund Rightsizes Its Kafka Cluster with AI and the MCP — customer-stories/investment-fund-kafka-rightsizing-for-ai — 30% freed · AI-led analysis - **Financial Services** — CDC Informatique Scales Kafka 4x Under Banking Regulation — customer-stories/cdc-informatique-governed-kafka-self-service — 40 → 160 apps, flat team - **Aviation & Transport** — Virgin Australia Boosts Kafka Adoption and Efficiency — customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor — 300 hours/month saved - **Transport** — FlixBus: Data Mesh in Practice with Self-Service Kafka — customer-stories/flix — 50+ teams enabled - **Transport & Logistics** — Swiss Post Democratizes Kafka Usage, Growing to 5x More Apps — customer-stories/how-swiss-post-governs-democratizes-kafka-usage — 60 → 300+ apps, 800+ users - **Agriculture & IoT** — Accelerating Smart Farming Innovation with Amazon MSK — customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk — 10x utilization, 70% faster provisioning - **Aviation** — European Airline Secures Cloud Migration with End-to-End Encryption — customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor — 25 clusters, 170 apps, 2,000 users - **Banking** — Securing Instant Payments for Banks Across Europe — customer-stories/securing-kafka-banking-financial-services — PCI DSS, VISA, MasterCard compliant - **Financial Services** — Bitvavo Ensures DORA & MiCA Compliance for 1.5M+ Users — customer-stories/bitvavo-ensures-compliance-dora-mica — DORA, MiCA, GDPR compliance achieved - **iGaming** — OpenTag Simplifies Kafka for Gaming Platform — customer-stories/opentag — Reduced DevOps workload - **Retail** — Fortune 500 Retailer Transforms Kafka Operations — customer-stories/one-hour-kafka-onboarding-retail-streaming-operations — Onboarding: 3 weeks → 1 hour ## Want similar results? Explore how Conduktor fits your infrastructure, or get hands-on with a demo. - **Architecture** — See how Conduktor deploys in your environment: cloud, hybrid, or air-gapped. [Learn more](https://www.conduktor.io/architecture) - **Documentation** — Step-by-step guides to get started with Conduktor Console, Conduktor Gateway, and Testing. [Read docs](https://docs.conduktor.io) - **Pricing** — Start free and scale with modular, transparent pricing. [View pricing](https://www.conduktor.io/pricing) Had a good experience? [Share it on G2](https://www.g2.com/products/conduktor-for-apache-kafka/take_survey) ## Frequently Asked Questions **Can Conduktor run in air-gapped environments?** Yes. Conduktor supports full offline installations with local registries. You can install and update all components without external network access. **Which clouds are supported?** Conduktor runs on AWS, Azure, GCP, or on-prem. It integrates with managed Kafka services such as AWS MSK, Confluent Cloud, and Aiven, as well as open-source clusters. **How long does deployment take?** A simple setup is ready in minutes using Helm or Docker Compose. Production-grade installs, including HA, external identity providers, and security managers, complete in days. **Does Conduktor support SSO and granular access control?** Yes. Conduktor supports OIDC and LDAP synchronization. Access rules apply at the user, group, or application level and can be automated through APIs. **Can we integrate with Vault or KMS?** Yes. Conduktor integrates natively with HashiCorp Vault and all major KMS providers (AWS, Azure, GCP). Secrets can be injected through environment variables, mounted volumes, or API calls. # Ready to start? Get started for free or contact us to explore the right solution for your business. [Get Started Free](https://www.conduktor.io/get-started?src=customer-stories) [Talk to Us](https://www.conduktor.io/contact/demo?src=customer-stories) Source: https://www.conduktor.io/customer-stories --- # Advanced Kafka Virtualization Workshop ### Do More with the Kafka Infrastructure You Already Have Kafka environments tend to sprawl: more clusters, more partitions, duplicate topics, and replication pipelines that exist only to move data between them. Each one adds infrastructure cost and operational load, and most of it exists to solve isolation and access problems. In this live, hands-on workshop, you'll use Conduktor Gateway in a real AWS environment to isolate workloads, reduce partition waste, create topic views, and securely share Kafka data without changing application code. > **No AWS account needed.** We'll provide access to a live Conduktor environment on AWS so you can complete every exercise during the workshop. ### What You'll Learn Through a series of guided, hands-on labs, you'll learn how to: - Create isolated virtual Kafka clusters across one or more physical clusters - Hide internal naming conventions using clean topic aliases - Consolidate low-volume topics to reduce physical topic and partition counts - Create filtered and transformed topic views without duplicating data - Securely share selected Kafka data with external applications without provisioning another cluster or replicating records You'll leave with a practical understanding of how Kafka virtualization can reduce infrastructure overhead while giving teams more control over how workloads and data are accessed. > **Part 2 of our Kafka proxy workshop series.** It follows [Introduction to Kafka Proxy Capabilities](https://www.conduktor.io/events/webinars/introduction-to-kafka-proxy-capabilities), but each session stands alone, so there's no need to have attended the first. ### Who Should Attend This workshop is designed for: - Platform engineers operating Kafka environments - Architects designing streaming and data platforms - Developers building Kafka applications - Engineering leaders responsible for Kafka cost, governance, or infrastructure strategy Participants should have a basic understanding of Kafka concepts. No previous experience with Conduktor Gateway is required. ### Event Details **Date:** Thursday, October 15, 2026 **Time:** 11:00 AM ET **Duration:** 90 minutes **Format:** Live, instructor-led virtual workshop Register for this workshop Source: https://www.conduktor.io/events/webinars/advanced-kafka-virtualization --- # Building Trust in Real-Time Data for AI Poor data quality costs time, trust, and money—yet it's still treated like an afterthought. But in the age of AI, that risk just got a lot bigger. ## Garbage In, Disaster Out You've heard "garbage in, garbage out." With agentic AI, it's worse: **garbage in, disaster out**. Acting on stale or broken data leads to real-world consequences—automated decisions that breach policies, trigger compliance nightmares, or harm customer trust. In this webinar, see how **Conduktor Trust** enforces data quality in-stream—before bad data spreads. No retroactive cleanup. No downstream surprises. Just clean, compliant, AI-ready data the moment it hits your pipeline. ## What You'll Learn - Define precise rules to flag, fix, or block bad data as it's created - Catch silent data quality issues before they become big problems or corrupt AI outcomes - Guarantee trusted data flows into AI systems, event-driven architectures and real-time analytics **Plus:** Live Demo + Interactive Q&A ## Why You Can't Miss This AI will act on your data. Whether it's ready or not. If your data isn't trusted at the source, your AI outcomes will never be. Join us—and learn how to make your operational data AI-grade by default, not by chance. Source: https://www.conduktor.io/events/webinars/building-trust-in-real-time-data-for-ai-and-distributed-systems --- # Retour d'expérience : CDC Informatique partage son parcours Kafka avec Conduktor ### Un retour d'expérience client Kafka est devenu un pilier stratégique pour de nombreuses organisations. Mais à mesure que l'adoption croît, les équipes plateforme font face à de nouveaux défis : uniformité, gouvernance, self-service, sécurité et conformité. **La Caisse des Dépôts et Consignations (CDC)** est l'une des plus grandes institutions financières publiques en France, au service de l'intérêt général depuis plus de 200 ans. Elle gère des fonds d'épargne, finance des projets d'infrastructure, et accompagne la transformation numérique des territoires. Avec des dizaines d'équipes et de cas d'usage reposant sur Kafka, la filiale technologique de la CDC, CDC Informatique, a dû industrialiser sa plateforme pour répondre aux exigences de gouvernance, de sécurité et de scalabilité. Dans ce webinaire, **Julien Maillard**, Senior Architect chez CDC Informatique, partage le parcours de son organisation avec Kafka et Conduktor. ### Ce que vous apprendrez - **Le contexte CDC Informatique** — Comment Kafka s'inscrit dans l'architecture de la CDC Informatique, accompagné de contraintes de conformité. - **Les défis avant Conduktor** — Gestion des demandes, provisionnement des accès, éducation sur les bonnes pratiques, et enjeux de sécurité - **La transformation avec Conduktor** — Comment l'équipe plateforme utilise Conduktor au quotidien pour standardiser les workflows, améliorer l'expérience développeur, et compléter leur offre - **Les résultats concrets** — Impact sur le temps d'onboarding, la résolution d'incidents, et la posture sécurité/conformité ### À qui s'adresse ce webinaire - Équipes plateforme gérant Kafka à l'échelle - Architectes data et streaming - Responsables sécurité et conformité - Toute organisation cherchant à industrialiser sa plateforme Kafka Source: https://www.conduktor.io/events/webinars/cdc-parcours-kafka-avec-conduktor --- # Conduktor 2.0: The New Standard for Teams Working with Kafka We announced Conduktor 2.0, the new way for teams and organizations to work with real-time data at scale. In this webinar, we introduce the new Free Tier and our complete new Self-Serve GitOps approach, making it easier than ever to manage your Kafka infrastructure. Source: https://www.conduktor.io/events/webinars/conduktor-2-0-the-new-standard-for-teams-working-with-kafka --- # Data Mesh in Practice: Kafka for 50+ Teams with FlixBus Join us to hear from Taras Slipets, Staff Data Engineer at FlixBus, about the lessons learned from **building and maintaining a distributed self-service streaming platform** over the last 4 years. Flixbus are leaders in the global green passenger mobility market, servicing more than 400,000 daily connections. Real-time data is critical for decision making in this dynamic environment, facilitating fleet tracking, route optimization, delivering precise arrival times and leveraging ML/AI to optimize the customer experience. ## What We'll Cover - Kafka usage at FlixBus - Data mesh core principles - Naming conventions for domain-oriented ownership - Kafka's prefixed ACL pattern type - Lessons learned from enabling 10 domains and 50 internal teams - Self-service for resource management (architecture, CI/CD, permissions workflow) We'll also take a look at how Conduktor is committed to providing a scalable, vendor-agnostic solution for self-service on Kafka, facilitating collaboration between Platform Teams and Application Teams. Source: https://www.conduktor.io/events/webinars/data-mesh-in-practice-kafka-for-50-teams-with-flixbus --- # Data Strategies for 2025: Data Quality, Operations and AI Readiness Hybrid cloud, real-time systems, and AI are redefining enterprise data strategies for 2025. To thrive, businesses must focus on data quality, operational efficiency, and AI readiness. For streaming-first organizations, the stakes are higher than ever—data quality, compliance, and operational efficiency will define who thrives in the year ahead. Join Conduktor and leading industry experts Casey O'Shea, CTO at Atyeti, and Howard Holton, COO at GigaOm, for a 30-minute fireside chat to explore the critical strategies enterprises need to stay competitive in 2025 and beyond. ## What You'll Learn - **Simplify Your Tech Stack:** Discover how top organizations are cutting through complexity to streamline tools and improve collaboration across development, platform, and security teams - **Advance Your Data Maturity:** Understand how real-time, event-driven architectures are replacing traditional pipelines and powering AI and analytics at scale - **Master Hybrid Cloud Challenges:** Get expert advice on securing and governing data in hybrid cloud environments while maintaining performance and compliance ## Why Attend - Gain actionable insights from first-hand experience leading teams through complex data transformations - Participate in an interactive format designed to answer your most pressing questions - Be prepared to lead your organization through the challenges and opportunities of 2025 Source: https://www.conduktor.io/events/webinars/data-strategies-for-2025-data-quality-operations-and-ai-readiness --- # From Chaos to Control: Mastering Data Quality in Streaming Data is your most valuable asset—if you can trust it. Join us for a high-level discussion on why **proactive data quality** is the key to unlocking the full power of your Kafka pipelines, real-time streaming, and AI initiatives. **Frances O'Rafferty**, a leading voice in data and analytics from Baringa, and **Quentin Packard**, VP Sales at Conduktor, will take you through the big questions businesses need to be asking about data quality today. They'll explore how clean, reliable data can make or break your streaming strategy and what it takes to ensure your data works for you, not against you. ## What You'll Learn - How to stop firefighting and start managing **data quality proactively** in your streaming workflows - The real cost of bad data and how it impacts everything from AI models to decision-making - Why **data governance** is more than a compliance checkbox—it's essential to building trust across your organization This is a Streamline: Conversations in Data session for data leaders, strategists, and anyone looking to get more out of their data. No technical deep dives—just a straight-talking conversation on how to get ahead with trusted data. Source: https://www.conduktor.io/events/webinars/from-chaos-to-control-mastering-data-quality-in-streaming --- # GitOps & Kafka: Streamline Access Management with Console API Conduktor's Console is the most complete interface for visualizing your Kafka data and troubleshooting your streaming applications. It is deployed centrally for your teams, integrates with your identity provider and is designed to serve all actors that need to interact with Kafka data. As Kafka adoption grows in your organization, so does: - **User & Application Growth:** The number of teams and applications that require access, each with their own set of resource permissions - **The Platform Team Bottleneck:** The number of projects and new clusters you need to manage and onboard ## What You'll Learn In this webinar, we'll deep-dive into automating Kafka using the recently released Console API: - Bring a GitOps approach to effectively manage clusters, users, groups and permissions at scale - Explore several newly released features - Preview what's coming up in future releases Source: https://www.conduktor.io/events/webinars/gitops-kafka-streamline-access-management-with-console-api --- # Introduction to Kafka proxy capabilities (authentication, networking, virtualization) ### A Live Hands-On Workshop As Kafka adoption grows, platform teams hit friction beyond broker management: - Cross-network connectivity - Authentication for applications outside of AWS IAM (e.g. an external application that needs to authenticate with OIDC or mTLS with custom certificates) - Efficient multitenancy for non-production and developer sandbox environments This workshop explores all three problems and provides solutions using Conduktor Gateway, a flexible, Kafka-aware proxy. You'll work in a live AWS environment across three hands-on labs. You will leave knowing how to use Conduktor Gateway to solve these challenges within your Kafka environment. > **No AWS account needed.** You don't need your own AWS account to take part. We provide access to a live Conduktor environment on AWS so you can complete every lab during the session. ### What You'll Build - Route Kafka traffic through a single proxy endpoint. No per-broker networking config. - Onboard a new application without touching IAM configuration. - Run Dev and QA on a single cluster with full isolation between environments. Future workshops will cover additional capabilities like disaster recovery, field-level encryption, data quality enforcement, and protecting Kafka from poorly configured clients. ### Who Should Attend This workshop is designed for: - Platform engineers operating Kafka - Architects designing streaming platforms - Developers building Kafka applications Participants should have a basic understanding of Kafka concepts. ### Event Details **Date:** Thursday, July 23 **Time:** 10:00 AM ET **Duration:** 90 minutes Register for this workshop Source: https://www.conduktor.io/events/webinars/introduction-to-kafka-proxy-capabilities --- # Is Your Kafka Platform Actually Healthy? ## The Problem You enabled self-service Kafka. Teams are onboarding fast. But now you're flying blind. - Who owns that abandoned topic with 50GB of data? - Is that consumer group actually being used, or just forgotten? - Which teams are exceeding their quotas—and do they even know? Dashboards show real-time metrics. They don't show what's drifting, accumulating, or about to become a problem. ## What We'll Cover This webinar is for platform engineers who need to answer "is our Kafka platform healthy?" without digging through logs or chasing down team leads. You'll learn: - **Long-term signals over real-time noise** — How to identify health patterns that emerge over weeks, not seconds - **Ownership clarity** — Practical approaches to tracking who owns what, and what happens when they leave - **Usage drift detection** — Spotting inactive resources, runaway growth, and governance gaps before they escalate - **Self-service without chaos** — How centralized insights let you scale access without losing control We also demo Conduktor's Insights and show how these signals come together in practice. ## Who This Is For - Platform engineers running Kafka infrastructure - SRE and infrastructure teams supporting streaming workloads - Architects responsible for Kafka governance and reliability - Anyone tired of playing detective when something goes wrong Source: https://www.conduktor.io/events/webinars/is-your-kafka-platform-actually-healthy --- # Kafka Confidential: Data Encryption Simplified Discover the essentials of effective **encryption strategies** in this webinar. Join us to explore crucial considerations for implementing internal encryption and safeguarding data shared with third parties. ## What We'll Cover - The encryption landscape within Kafka systems - Encryption in transit vs. encryption at rest - Field-level & full payload encryption - Crypto shredding to meet legal obligations (GDPR, Right to be forgotten) - How to pinpoint vulnerability areas in your setup - Identifying and addressing potential risks We'll also guide you through contemporary solutions offered by both cloud providers and vendors, empowering you to make informed decisions for enhanced data security. This will also be the first time we demonstrate how to configure encryption via the Conduktor Console, showcasing the ease of setup regardless of which Kafka provider you are using. Source: https://www.conduktor.io/events/webinars/kafka-confidential-data-encryption-simplified --- # Kafka Disaster Recovery: Why It Fails in Practice Most Kafka teams believe they have disaster recovery covered. Replication is running, a backup cluster exists, and there is a runbook somewhere. But when a real incident happens, recovery often takes hours. In this session, Andy Allison and Christoph Schubert discuss why Kafka DR often breaks down in practice and what teams can do to make recovery predictable and repeatable. ## What You'll Learn - Why replication alone does not solve Kafka disaster recovery - The operational challenges behind Kafka failover - Common DR failure modes such as configuration drift and client coordination - How platform teams structure DR to recover in minutes instead of hours - Practical ways to evaluate your current Kafka DR strategy ## Related Resources - **Checklist:** [Kafka Disaster Recovery Readiness Checklist](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-readiness-checklist) - **Blog:** [Kafka Disaster Recovery: The Complete Strategy Beyond Replication](https://www.conduktor.io/blog/kafka-disaster-recovery-the-complete-strategy-beyond-replication) - **Blog:** [How Gateway Reduces Kafka DR from Hours to Minutes](https://www.conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes) ## Speakers Andy Allison — Head of Product & Go-To-Market, Conduktor Christoph Schubert — Head of Product, Conduktor Source: https://www.conduktor.io/events/webinars/kafka-disaster-recovery-why-it-fails-in-practice --- # Kafka Security 2.0: Securing Your Data with Encryption, RBAC and Audit This webinar will focus on practical implementations for addressing Kafka data security and access management needs via Conduktor. We'll recap why ACLs alone are not enough, and showcase how to strengthen your security posture through hands-on demonstrations. ## What We'll Cover - **ACLs:** Why Kafka ACLs are not enough - **RBAC:** Simplifying access management through Conduktor's RBAC system - **Audit:** Understanding who has done what in your Kafka environment - **Data Masking:** Masking sensitive (PII) and confidential data ## Encryption Deep-Dive - Field-level encryption - Key rotation - Crypto shredding - Encryption on fetch - Performance implications Source: https://www.conduktor.io/events/webinars/kafka-security-2-0-securing-your-data-with-encryption-rbac-and-audit --- # Kafka Security in 2025: Building Resilience Kafka security is no longer just about protection—it's about resilience. As streaming data becomes central to AI and operational decision-making, the cost of downtime, breaches, or misconfigurations has never been higher. According to a recent data streaming survey, security management, protection, and governance are the top strategic priorities for enterprises investing in streaming. It's not just a technical concern—it's now a board-level mandate. ## What We'll Cover - Real-world approaches to encryption - Kafka failover that doesn't disrupt your operations - Governance and observability built for speed This isn't about checking compliance boxes. It's about building Kafka infrastructure that holds under pressure and adapts when it doesn't. Source: https://www.conduktor.io/events/webinars/kafka-security-in-2025-building-resilience --- # Live Demo: Getting Started with Conduktor + Q&A Discover how Conduktor promotes better collaboration between Platform Teams, Architects, and Software Developers working with Apache Kafka. In this live demo and Q&A session, we walk through the key features of the Conduktor platform and answer your questions about getting started. Source: https://www.conduktor.io/events/webinars/live-demo-getting-started-with-conduktor-q-a --- # Make Your Most Valuable Data Safe for AI Today, [72 percent of organizations](https://newsroom.ibm.com/2025-05-06-ibm-study-ceos-double-down-on-ai-while-navigating-enterprise-hurdles) believe that their proprietary data is their secret weapon for maximizing the potential of generative AI. But valuable data remains locked away in on-prem databases, for a number of reasons. Organizations may lack the tools to identify and restrict access to sensitive data; they may struggle with fragile pipelines that require too much maintenance; or they may be concerned about data issues which can impact downstream applications. ## What You'll Learn - Why upstream changes can affect consumer applications—and how to prevent this - Why the lack of operational data can stall AI initiatives - How teams can use Conduktor Flow to bridge the gap between on-prem databases and cloud-based analytics, applications, and AI Learn how to make your operational data secure, AI-ready, and a competitive advantage (rather than a burden) with Conduktor Flow. Source: https://www.conduktor.io/events/webinars/make-your-most-valuable-data-safe-for-ai --- # Migrating from Conduktor Desktop to Scale We're Sunsetting Desktop. Meet Conduktor Scale! After years of helping you make the most of Kafka, Conduktor Desktop is stepping aside. Why? Because your data streaming needs have evolved—and so have we. We're saying goodbye to Desktop to focus on what matters most to our customers: team collaboration, rock-solid governance, and next-level scalability. ## What You'll Discover - **The key benefits** Conduktor Scale brings to your organization - **A simple, actionable walkthrough** for an effortless transition process - **Answers to your specific questions** during the live Q&A Get firsthand insights and connect with our team to ensure a smooth transition from Conduktor Desktop. Source: https://www.conduktor.io/events/webinars/migrating-from-conduktor-desktop-to-scale --- # The MSK Journey: Networking, Auth & Scale ## Watch the Webinar ## The Problems Nobody Warns You About If you're running Kafka on AWS, chances are you're using Amazon MSK. It solves infrastructure. But what happens next? As your Kafka usage grows, new challenges emerge — not at the infrastructure level, but above it: - Cross-account connectivity that breaks in unexpected ways - Authentication gaps (OIDC, mTLS) that slow teams down - Scaling inefficiencies across clusters and environments This webinar dives into the real-world problems teams face when operating Kafka on MSK, and how to solve them. ## What You'll Learn - Why cross-account MSK connectivity is harder than it looks - The hidden limitations of Kafka networking and how they impact your architecture - How to support modern authentication (OIDC, mTLS) in MSK environments - Ways to reduce cluster sprawl and improve efficiency - How to introduce centralized control without changing your applications ## Who This Is For - Platform engineers running Kafka infrastructure on AWS - SRE and infrastructure teams supporting MSK workloads - Architects responsible for Kafka governance and reliability - Anyone scaling Kafka across multiple accounts or environments Source: https://www.conduktor.io/events/webinars/msk-journey-networking-auth-scale --- # Rethinking Your Kafka Experience with Conduktor Gateway Apache Kafka is a highly available and scalable data streaming platform that thousands of companies use to support their mission-critical applications. Despite this, there are shortcomings in meeting Enterprise needs around governance, security, and providing developer autonomy without burdening central platform teams. Conduktor Gateway is a Kafka proxy that sits between your client applications and existing Kafka clusters. In this webinar, we'll take you on a guided tour of Conduktor Gateway. ## Maximize Your Kafka Investment Join us to learn how to maximize your Kafka investment: - How to create isolated environments for teams and domains - How to build serverless data transformations - Enforcing best practices to protect your clusters against bad actors - Augmenting your Kafka data with contextualized headers - Managing sensitive data to meet compliance and regulation standards - Handling large messages in Kafka by implementing the 'claim-check pattern' - Reducing Kafka costs for low throughput applications Source: https://www.conduktor.io/events/webinars/rethinking-your-kafka-experience-with-conduktor-gateway --- # Self-Service Kafka That Doesn't Sacrifice Control Unlock speed without losing oversight. Your teams shouldn't have to wait on platform teams to get Kafka resources. Delays slow innovation, frustrate engineers, and create bottlenecks—but giving up control isn't the answer. ## What You'll Learn - Enable fast, self-service Kafka provisioning without risking security - Enforce access controls and policies automatically, at scale - Monitor usage and maintain governance while teams move independently Stop slowing your teams down. Build a Kafka operating model that accelerates innovation safely, scales effortlessly, and keeps control in your hands. Source: https://www.conduktor.io/events/webinars/self-service-kafka-that-doesn-t-sacrifice-control --- # Simplifying Kafka on AWS MSK (EMEA) ### A Live Hands-On Workshop Amazon MSK handles the infrastructure, but as Kafka adoption grows, platform teams hit friction beyond broker management: - Cross-network connectivity - Authentication for applications outside of AWS IAM (e.g. an external application that needs to authenticate with OIDC or mTLS with custom certificates) - Efficient multitenancy for non-production and developer sandbox environments This workshop explores all three problems and provides solutions using Conduktor Gateway, a flexible, Kafka-aware proxy. You'll work in a live AWS environment across three hands-on labs. You will leave knowing how to use Conduktor Gateway to solve these challenges within AWS MSK. ### What You'll Build - Route Kafka traffic through a single proxy endpoint. No per-broker networking config. - Onboard a new application without touching IAM or MSK configuration. - Run Dev and QA on a single MSK cluster with full isolation between environments. Future workshops will cover additional capabilities like disaster recovery, field-level encryption, data quality enforcement, and protecting Kafka from poorly configured clients. ### Who Should Attend This workshop is designed for: - Platform engineers operating Kafka on AWS - Architects designing streaming platforms - Developers building Kafka applications - Teams currently using or evaluating **Amazon MSK** Participants should have a basic understanding of Kafka concepts. ### Event Details **Date:** Wednesday, May 13 **Time:** 3:00 PM CET **Duration:** 90 minutes Register for this workshop Source: https://www.conduktor.io/events/webinars/simplifying-kafka-on-aws-msk-emea --- # Simplifying Kafka on AWS MSK ### A Live Hands-On Workshop Amazon MSK handles the infrastructure, but as Kafka adoption grows, platform teams hit friction beyond broker management: - Cross-network connectivity - Authentication for applications outside of AWS IAM (e.g. an external application that needs to authenticate with OIDC or mTLS with custom certificates) - Efficient multitenancy for non-production and developer sandbox environments This workshop explores all three problems and provides solutions using Conduktor Gateway, a flexible, Kafka-aware proxy. You'll work in a live AWS environment across three hands-on labs. You will leave knowing how to use Conduktor Gateway to solve these challenges within AWS MSK. ### What You'll Build - Route Kafka traffic through a single proxy endpoint. No per-broker networking config. - Onboard a new application without touching IAM or MSK configuration. - Run Dev and QA on a single MSK cluster with full isolation between environments. Future workshops will cover additional capabilities like disaster recovery, field-level encryption, data quality enforcement, and protecting Kafka from poorly configured clients. ### Who Should Attend This workshop is designed for: - Platform engineers operating Kafka on AWS - Architects designing streaming platforms - Developers building Kafka applications - Teams currently using or evaluating **Amazon MSK** Participants should have a basic understanding of Kafka concepts. ### Event Details **Date:** Tuesday, April 28 **Time:** 2:00 PM ET **Duration:** 90 minutes Register for this workshop Source: https://www.conduktor.io/events/webinars/simplifying-kafka-on-aws-msk --- # Trailblazers in Tech: Women Shaping the Next Wave This event brings together trailblazing technologists, thought leaders, and rising stars for inspiring keynotes, interactive panels, and hands-on discussions designed to educate, empower, and connect. ## Why Attend - Learn from women transforming the tech industry at scale - Gain actionable strategies to grow your technical expertise and leadership presence - Connect with peers and mentors passionate about building a more inclusive future in technology - Be part of a global community that values teaching, learning, and lifting each other up ## Who Should Attend - Engineers, data scientists, and technologists eager to level up their skills - Leaders building inclusive teams and innovative solutions - Students and early-career professionals exploring their paths in tech - Allies who believe in championing women in technology Join us in celebrating women who code, lead, and inspire. Together, let's build a future where innovation thrives through inclusion. Source: https://www.conduktor.io/events/webinars/trailblazers-in-tech-women-shaping-the-next-wave --- # Unleashing the Power of Kafka Discover the latest innovations in Conduktor, the go-to solution trusted by hundreds of thousands of developers worldwide to simplify and manage Apache Kafka. Join us for an exclusive webinar as we showcase Conduktor's latest features from 2023 that empower DevOps teams to harness the full potential of Kafka. ## What We'll Cover - Enhanced Role-based access control (RBAC) features - Advanced Data Masking capabilities - Expansion of Monitoring - Free Kafka clusters in Conduktor - Seamless LDAP integration - Robust mTLS support - Preview of features on our upcoming roadmap Don't miss this opportunity to stay ahead of the curve and elevate your Kafka game with Conduktor. Source: https://www.conduktor.io/events/webinars/unleashing-the-power-of-kafka --- # Secure Kafka Data Sharing Without Duplication Real-time data isn't just a resource—it's **a competitive advantage** and **a revenue driver**. But securely sharing streaming data across partners is often **complex, expensive, and risky**—until now. Join us to see how **Conduktor Exchange** enables **secure, real-time Kafka data sharing**—without replication—turning your data into a **strategic, revenue-generating asset**. ## What You'll Learn - **Real-Time, Secure Data Sharing:** Eliminate duplication with data clean rooms and isolated [Partner Zones](https://www.conduktor.io/solutions/use-case/data-sharing) - **Cross-Platform Flexibility:** Share Kafka data with any partner, on any tech stack—without compatibility headaches - **Centralized Governance & Compliance:** Manage, audit, and enforce access policies across all shared data from a single platform ## Live Demo & Q&A See [Conduktor Console](https://www.conduktor.io/console) in action: - Seamless partner onboarding and secure data sharing setup - Managing access controls for each partner - Enforcing traffic policies and maintaining compliance Bring your toughest data-sharing questions—our experts are ready to help. ## Who Should Attend - Data Streaming Leaders & Strategists - Data Governance Professionals - Data Platform Engineers & Architects Whether you're starting your data-sharing strategy or scaling secure real-time data access, this session provides practical insights you can apply immediately. Source: https://www.conduktor.io/events/webinars/unlock-real-time-data-sharing-with-conduktor-exchange --- # Introduction to Conduktor Looking for a quick intro to Conduktor? In this short video, learn how leading organizations like BMW, Swiss Post, and Lufthansa are transforming their data streaming operations with Conduktor. Discover how Conduktor helps teams simplify operations, reduce costs, and unlock real-time value from their data. ## What You'll Learn - The benefits of a unified platform for managing Kafka at scale - How Conduktor improves collaboration and governance across teams - Why simplifying Kafka operations leads to time and cost savings - How data streaming supports AI, analytics, and real-time decision-making Explore how Conduktor helps you move faster, stay compliant, and make streaming data work for your business. Source: https://www.conduktor.io/events/webinars/unlock-the-power-of-streaming-data-a-10-minute-introduction-to-conduktor --- # Powering AI with Streaming Data: Accuracy, Security, Compliance AI is revolutionizing how organizations operate. Yet outdated, biased, or poor-quality data can derail AI applications, leading to errors, security risks, and compliance failures. In this webinar, discover how **streaming data** can keep your AI models **accurate, compliant, and free from hallucinations**. You'll also learn how **Conduktor** enables you to proactively enforce data quality on your streaming data to maximize AI's potential while mitigating risks. ## Key Topics - **Better AI with Streaming Data:** Use real-time contextual data in AI prompts to improve model performance and scalability - **Ensuring Data Quality & Freshness:** Ensure AI models are up-to-date and data ingested meets quality standards with Conduktor Trust - **Mitigating AI Risks:** Address security, compliance, and accuracy risks in GenAI pipelines **Interactive Q&A:** Get your questions answered by our experts during the live session. ## Who Should Watch - Data Streaming Leaders & Strategists - Data Governance Professionals - Data Platform Engineers & Architects - AI/ML Engineers and Decision Makers Whether you're building your first GenAI model or scaling an enterprise AI strategy, this session offers valuable insights on data streaming best practices. Source: https://www.conduktor.io/events/webinars/unlocking-the-potential-of-generative-ai-addressing-risks-with-streaming-data --- # Was ist neu bei Conduktor: Back-to-School-Webinar ### Neues Schuljahr, neuer Lehrplan Direkt vom Team, das Conduktor entwickelt: Christoph Schubert zeigt euch alles, was wir diesen Sommer ausgeliefert haben, plus einen ersten Blick auf das nächste Semester. **Das nehmt ihr mit:** - "Wer nutzt dieses Topic wirklich?" in Sekunden beantwortet, nicht in Slack-Threads - Kafka-Kosten pro Team mit Chargeback dem Controlling vorlegen - Teams Kafka im Self-Service nutzen lassen, während eure Guardrails halten Während alle anderen im Urlaub waren, haben wir die Hausaufgaben gemacht. Ihr betreibt Kafka als internen Service. Das heißt: Ihr solltet wissen, wer auf welches Topic schreibt, es gegenüber Auditoren belegen, es an Business Units weiterverrechnen, und dabei bitte nicht zum Bottleneck werden, welches die Entwicklungsteams ausbremst. Diese Session zeigt, was wir diesen Sommer dafür in die Produkt-Schultüte gepackt und fürs neue Schuljahr so geplant haben. ### Der Lehrplan - 🗺️ **Kapitel 1: Euren Estate live sehen.** Lineage zeigt jetzt echten Traffic statt nur statischer Verbindungen. "Wer nutzt dieses Topic wirklich?", beantwortet in Sekunden, nicht in Slack-Threads. - 📡 **Kapitel 2: Client-Telemetrie ohne Eingriff in den Client-Code.** Metriken im KIP-714-Stil werden am Gateway erfasst. So findet man den fehlkonfigurierten Consumer, bevor man um 2 Uhr Nachts deswegen geweckt wird. - 👥 **Kapitel 3: Gruppenarbeit, aber richtig.** Resource Templates, eine Public API für den Lifecycle von Consumer Groups und Connectors sowie Self-Service-Requests inklusive Rückmeldung zum Ergebnis. Die Teams arbeiten, die Guardrails halten. - 🧮 **Kapitel 4: Rechenweg zeigen.** Chargeback macht aus Shared-Cluster-Nutzung Kosten pro Team, die dem Controlling vorgelegt werden können. - 🔒 **Kapitel 5: Hausordnung am Eingang.** SASL/SCRAM für Gateway-managed Security, mehrere Listener, Schema Registry Proxy sowie zentrale Schema-Registry- und KMS-Verbindungen. - ⭐ **Zusatzpunkte: über Kafka hinaus.** Flink Statement Management für Confluent Cloud, dazu ein erster Blick auf Conduktor for Agents, REST Produce und Self-Service-Verschlüsselung. Zum Abschluss Sprechstunde: Live-Q&A mit dem Produktteam. > 🎓 **Kein Test. Kein Referat. Kamera darf aus bleiben.** Nur 45 Minuten zu dem, was ausgeliefert wurde und was als Nächstes kommt. ### Anwesenheitskontrolle: Wer sollte dabei sein? Kafka-Service- und Plattform-Owner, SRE- und Infrastructure-Engineers, Data-Platform-Leads sowie Architekten, die Kafka-Governance im großen Maßstab bewerten. Wir freuen uns auf bestehende Kunden, Nutzer der Community Edition und alle, die sich noch nicht sicher sind, wie sie ihre Kafka-Plattform skalieren wollen. ### Details zur Veranstaltung **Datum:** Dienstag, 15. September 2026 **Uhrzeit:** 11:00 Uhr CEST **Dauer:** 45 Minuten **Sprache:** Deutsch > **Lieber auf Englisch?** Dieselbe Session läuft auf Englisch um [16:00 Uhr CEST](https://www.conduktor.io/events/webinars/whats-new-back-to-school). Platz sichern Meldet euch unten an, und wir schicken euch den Einwahllink sowie eine Erinnerung vor der Session. Source: https://www.conduktor.io/events/webinars/whats-new-back-to-school-de --- # What's New in Conduktor: Back to School Webinar ### New term, new syllabus Straight from the team that builds Conduktor: Christoph Schubert and Ron Kapoor walk through everything we shipped this summer, plus a first look at next semester. **You'll walk away able to:** - Answer "who's actually using this topic?" in seconds, not Slack threads - Put per-team Kafka cost in front of finance with chargeback - Let teams self-serve Kafka while your guardrails hold While everyone was on holiday, we did the homework. You run Kafka as a service, which means you're expected to know who produces to what, prove it to an auditor, charge it back to a business unit, and never become the ticket queue that slows delivery down. This session covers what shipped this summer to help with all four, and where Conduktor goes next. ### On the syllabus - 🗺️ **Chapter 1: See your estate, live.** Lineage now shows real traffic, not just static wiring. "Who is actually using this topic?" answered in seconds, not in Slack threads. - 📡 **Chapter 2: Client telemetry without touching client code.** KIP-714-style metrics collected at the Gateway, so you spot the misconfigured consumer before it pages you at 2am. - 👥 **Chapter 3: Group work, done properly.** Resource templates, a public API for consumer group and connector lifecycle, and self-service requests with outcome notifications. Your teams do the work; your guardrails hold. - 🧮 **Chapter 4: Show your workings.** Chargeback turns shared-cluster usage into per-team cost you can actually put in front of finance. - 🔒 **Chapter 5: House rules at the door.** SASL/SCRAM for Gateway-managed security, multiple listeners, Schema Registry proxy, and centralized Schema Registry and KMS connections. - ⭐ **Extra credit: beyond Kafka.** Flink statement management for Confluent Cloud, plus a first look at Conduktor for Agents, REST produce and self-service encryption. We finish with office hours: live Q&A with the product team. > 🎓 **No pop quiz. No group project. Cameras optional.** Just 45 minutes on what shipped and what's next. ### Roll call: who should attend Kafka service and platform owners, SRE and infrastructure engineers, data platform leads, and architects evaluating how to govern Kafka at scale. All welcome: existing customers, Community Edition users, and anyone still comparing options. ### Event details **Date:** Tuesday, September 15, 2026 **Time:** 4:00 PM CEST **Duration:** 45 minutes **Language:** English > **Prefer German?** We're running the same session in German at [11:00 AM CEST](https://www.conduktor.io/events/webinars/whats-new-back-to-school-de). Save your seat Register below and we'll send your joining link, plus a reminder before the session. Source: https://www.conduktor.io/events/webinars/whats-new-back-to-school --- # Kafka Webinars: On-Demand & Live Sessions Watch free expert-led webinars on Kafka security, governance, AI readiness, and streaming operations. --- ## All Webinar Recordings - [The MSK Journey: Networking, Auth & Scale](https://www.conduktor.io/events/webinars/msk-journey-networking-auth-scale) — Watch Now - [Kafka Disaster Recovery: Why It Fails in Practice](https://www.conduktor.io/events/webinars/kafka-disaster-recovery-why-it-fails-in-practice) — Free to Watch - [Make Your Most Valuable Data Safe for AI](https://www.conduktor.io/events/webinars/make-your-most-valuable-data-safe-for-ai) — Free to Watch - [Building Trust in Real-Time Data for AI and Distributed Systems](https://www.conduktor.io/events/webinars/building-trust-in-real-time-data-for-ai-and-distributed-systems) — Free to Watch - [Powering AI with Streaming Data: Accuracy, Security, Compliance](https://www.conduktor.io/events/webinars/unlocking-the-potential-of-generative-ai-addressing-risks-with-streaming-data) — Free to Watch - [Data Strategies for 2025: Data Quality, Operations and AI Readiness](https://www.conduktor.io/events/webinars/data-strategies-for-2025-data-quality-operations-and-ai-readiness) — Free to Watch - [From Chaos to Control: Mastering Data Quality in Streaming](https://www.conduktor.io/events/webinars/from-chaos-to-control-mastering-data-quality-in-streaming) — Free to Watch - [Kafka Security in 2025: Building Resilience](https://www.conduktor.io/events/webinars/kafka-security-in-2025-building-resilience) — Free to Watch - [Kafka Confidential: Data Encryption Simplified](https://www.conduktor.io/events/webinars/kafka-confidential-data-encryption-simplified) — Free to Watch - [Kafka Security 2.0: Securing your data with Encryption, RBAC and Audit](https://www.conduktor.io/events/webinars/kafka-security-2-0-securing-your-data-with-encryption-rbac-and-audit) — Free to Watch - [Self-Service Kafka That Doesn't Sacrifice Control](https://www.conduktor.io/events/webinars/self-service-kafka-that-doesn-t-sacrifice-control) — Free to Watch - [Unleashing the power of Kafka](https://www.conduktor.io/events/webinars/unleashing-the-power-of-kafka) — Free to Watch - [Unlock the Power of Streaming Data: An Introduction to Conduktor](https://www.conduktor.io/events/webinars/unlock-the-power-of-streaming-data-a-10-minute-introduction-to-conduktor) — Free On-Demand - [Conduktor 2.0: The new standard for teams working with Kafka](https://www.conduktor.io/events/webinars/conduktor-2-0-the-new-standard-for-teams-working-with-kafka) — Free to Watch - [Live Demo: Getting Started with Conduktor + Q&A](https://www.conduktor.io/events/webinars/live-demo-getting-started-with-conduktor-q-a) — Free to Watch - [Rethinking your Kafka Experience with Conduktor Gateway](https://www.conduktor.io/events/webinars/rethinking-your-kafka-experience-with-conduktor-gateway) — Free to Watch - [Migrating from Conduktor Desktop to Scale](https://www.conduktor.io/events/webinars/migrating-from-conduktor-desktop-to-scale) — Free On-Demand - [Turn Streaming Data into An Asset: Secure Kafka Data Sharing Without Duplication](https://www.conduktor.io/events/webinars/unlock-real-time-data-sharing-with-conduktor-exchange) — Free On-Demand - [GitOps & Kafka: Streamline Access Management with Console API](https://www.conduktor.io/events/webinars/gitops-kafka-streamline-access-management-with-console-api) — Free to Watch - [Data Mesh in Practice: Kafka for 50 Teams with FlixBus](https://www.conduktor.io/events/webinars/data-mesh-in-practice-kafka-for-50-teams-with-flixbus) — Free to Watch - [Trailblazers in Tech: Women Shaping the Next Wave](https://www.conduktor.io/events/webinars/trailblazers-in-tech-women-shaping-the-next-wave) — Free to Watch Source: https://www.conduktor.io/events/webinars --- # Events & Webinars Join live webinars, watch on-demand recordings, and meet the Conduktor team at industry events. - **Workshop** — Advanced Kafka Virtualization: Virtual Clusters, Topic Views, and External Data Sharing — Do more with your existing Kafka. A live, hands-on workshop using Conduktor Gateway to isolate workloads, cut partition waste, and share data securely. — Oct 15, 2026 — events/webinars/advanced-kafka-virtualization — Register Now — upcoming - **Webinar** — Zurück auf die Schulbank: Was ist neu bei Conduktor — Ein 45-minütiges Live-Webinar zu den Releases dieses Sommers: seht, wer was besitzt, wer Zugriff hat und was Kafka pro Team kostet, plus ein Ausblick. — Sep 15, 2026 — events/webinars/whats-new-back-to-school-de — Jetzt anmelden — upcoming - **Webinar** — Class Is Back in Session: What's New in Conduktor — A live 45-minute webinar on what shipped this summer: see who owns what, who has access, and what Kafka costs per team, plus a first look at what's next. — Sep 15, 2026 — events/webinars/whats-new-back-to-school — Register Now — upcoming - [Conference](https://www.conduktor.io/aws-summit-zurich) — Conduktor at AWS Summit Zurich — Meet the Conduktor team at AWS Summit Zurich on 2 September 2026. Find us at Booth B19, or book 20 minutes to talk Kafka. — Sep 2, 2026 — Book a Meeting - **Workshop** — Introduction to Kafka proxy capabilities (authentication, networking, virtualization) — A live hands-on workshop exploring cross-network connectivity, authentication beyond AWS IAM, and efficient multitenancy for Kafka using Conduktor Gateway. — Jul 23, 2026 — events/webinars/introduction-to-kafka-proxy-capabilities — Register Now - [Meetup](https://luma.com/endxppwz) — Hot Topics: Kafka — NYC Meetup — Join the Conduktor team and the NYC data streaming community for an evening of Kafka talks and networking on June 30, 2026. — Jun 30, 2026 — Register - [Conference](https://www.conduktor.io/aws-summit-nyc) — Conduktor at AWS Summit NYC — Grab 20 minutes with the Conduktor team at AWS Summit New York on June 17, 2026. Book a time to chat in person. — Jun 17, 2026 — Chat With Us - **Webinar** — Retour d'expérience : CDC Informatique partage son parcours Kafka avec Conduktor — Découvrez comment CDC Informatique a transformé sa plateforme Kafka avec Conduktor. Gouvernance, self-service, sécurité : un retour d'expérience concret sur les défis et les résultats obtenus. — May 27, 2026 — events/webinars/cdc-parcours-kafka-avec-conduktor — Register Now - **Conference** — Conduktor at AWS Summit Hamburg 2026 — Meet the Conduktor team at AWS Summit Hamburg. Learn how Conduktor complements AWS MSK and join us for an exclusive executive dinner with DeepL. — May 20, 2026 — events/conferences/aws-summit-hamburg-2026 — Meet Us at the Summit - **Conference** — Conduktor at Current London 2026 — Meet the team, see the product, and connect beyond the booth at Current London 2026. — May 19, 2026 — events/conferences/current-london-2026 — Meet Us on Site - [Meetup](https://luma.com/lfn5s1fn) — Current London Happy Hour — Join Aiven, Conduktor, & OSO to kick things off the evening before Day 1 of Current London with a relaxed gathering of the Kafka community before the main event begins! — May 18, 2026 — Register Now - **Workshop** — Simplifying Kafka on AWS MSK (EMEA) — A live hands-on workshop exploring cross-network connectivity, authentication beyond AWS IAM, and efficient multitenancy for Kafka on AWS MSK using Conduktor Gateway. — May 13, 2026 — events/webinars/simplifying-kafka-on-aws-msk-emea — Register Now - [Meetup](https://luma.com/x1u1pn7y) — Bern Kafka Meetup — Join Conduktor, SPOUD, & Swiss Post for an evening of practical insights, real-world case studies, and great conversations with the Swiss data community. — May 7, 2026 — Register Now - **Workshop** — Simplifying Kafka on AWS MSK — A live hands-on workshop exploring cross-network connectivity, authentication beyond AWS IAM, and efficient multitenancy for Kafka on AWS MSK using Conduktor Gateway. — Apr 28, 2026 — events/webinars/simplifying-kafka-on-aws-msk — Register Now - **Webinar** — The MSK Journey: Networking, Auth & Scale — Dive into the real-world problems teams face when operating Kafka on MSK: cross-account connectivity, authentication gaps, and scaling inefficiencies. — Mar 26, 2026 — events/webinars/msk-journey-networking-auth-scale — Watch Now - **Podcast** — Kafka Disaster Recovery: Why It Fails in Practice — Replication is not the hardest part of Kafka disaster recovery. The operational complexity is. Learn why DR breaks down and how to make recovery predictable. — Mar 16, 2026 — events/webinars/kafka-disaster-recovery-why-it-fails-in-practice — Watch Now - **Conference** — Conduktor at QCon London 2026 — Meet the Conduktor team at QCon London Booth #2 for all things Kafka — Mar 15, 2026 — #meet-the-team-at-booth-2 — Schedule time to meet - **Webinar** — Is Your Kafka Platform Actually Healthy? — A proactive readiness check for platform teams enabling Kafka at scale. Learn how to use long-term signals to guide better decisions. — Jan 29, 2026 — events/webinars/is-your-kafka-platform-actually-healthy — Watch Now - **Webinar** — Trailblazers in Tech: Women Shaping the Next Wave — At Conduktor, we believe that technology becomes truly transformative when diverse voices are leading the conversation. That's why we're hosting Women in Tech Webinar— an event dedicated to celebrating women driving impact across engineering, data, analytics, and technology leadership. — Nov 6, 2025 — events/webinars/trailblazers-in-tech-women-shaping-the-next-wave — Watch Now - [Meetup](https://luma.com/683h13ij) — Real-Time London: Kafka, Flink & Beyond — Join us in London for an evening of real-time data expertise, networking, and innovation. — Oct 1, 2025 — Register Now - [Conference](https://attend.cloudera.com/evolve-forum-frankfurt-2025) — Cloudera Evolve Frankfurt — EVOLVE empowers businesses with strategies for navigating data management in the age of AI. — Sep 30, 2025 — Register Now - **Webinar** — Self-Service Kafka That Doesn't Sacrifice Control — Stop slowing your teams down. Build a Kafka operating model that accelerates innovation safely, scales effortlessly, and keeps control in your hands. — Sep 25, 2025 — events/webinars/self-service-kafka-that-doesn-t-sacrifice-control — Watch Now - [Conference](https://gdsgroup.com/events/physical-summit/retail-na-sep-25/) — GDS Retail Summit — Exploring the evolving role of AI and personalization in retail before and after purchase. — Sep 11, 2025 — Register Now - **Webinar** — Kafka Security in 2025: Building Resilience — Resilience is the new security standard. Learn how you can ensure resilience, security, and AI-readiness in your Kafka environment. — Jul 17, 2025 — events/webinars/kafka-security-in-2025-building-resilience — Watch Now - [Conference](https://www.tdwi-konferenz.de/en/program/conference-program) — Conduktor at TDWI Munich — Germany's largest meeting point for Data, Analytics, and AI. Visit our booth to explore how Conduktor unlocks the full potential of streaming data. — Jun 24, 2025 — Learn More - [Meetup](https://59xvt.share.hsforms.com/2OYynCsEEQE-ZbcA5oYSzDA) — Driving Decisions in Real Time – Automotive Forum & Dinner — Impulse für die Zukunft der Automobilbranche – präsentiert von Cloudera, Conduktor und SVA — Jun 23, 2025 — Register Now - [Meetup](https://www.redpanda.com/events/operationalizing-ai-agents-meetup-june2025) — Operationalizing AI Agents in Production — How to build resilient, scalable, and observable AI agent pipelines without sacrificing control, compliance, or performance. — Jun 17, 2025 — Register Now - [Conference](https://www.dashcon.io/) — Conduktor at DASH 2025 — Visit us at Booth 508—observability, security, and performance in cloud environments. — Jun 10, 2025 — Learn More - **Webinar** — Building Trust in Real-Time Data for AI — Meet Conduktor Trust, our new solution to enforce data quality in-stream—before bad data spreads. Stop chasing bad data downstream and ensure clean, compliant, AI-ready data the moment it hits your pipeline. — Jun 5, 2025 — events/webinars/building-trust-in-real-time-data-for-ai-and-distributed-systems — Watch Now - **Webinar** — Powering AI with Streaming Data: Accuracy, Security, Compliance — Learn how to proactively enforce data quality on your streaming data to maximize GenAI's potential while mitigating risks. — May 6, 2025 — events/webinars/unlocking-the-potential-of-generative-ai-addressing-risks-with-streaming-data — Watch Now - [Meetup](https://share.hsforms.com/1ZzkSuPh6Sdyss1358Gx6hg59xvt) — Kafka Meetup in Paris With Conduktor — Join us for an evening of talks and networking with the Kafka community in Paris. — Apr 29, 2025 — Register Now - [Conference](https://gdsgroup.com/events/physical-summit/data-analytics-na-apr-25/) — Conduktor at GDS Data & Analytics Insight Summit — Join our roundtable on the future of data streaming & real-time analytics. — Apr 28, 2025 — Learn More - **Webinar** — Introduction to Conduktor — Learn how data-driven companies are using Conduktor to improve data quality, ensure compliance, and unlock the full potential of Kafka. — Apr 11, 2025 — events/webinars/unlock-the-power-of-streaming-data-a-10-minute-introduction-to-conduktor — Watch Now - [Conference](https://share.hsforms.com/2dlz21GLpSD2RsUkIbaWKhQ59xvt) — Conduktor at QCon London — Meet the Conduktor team at QCon London for practical advice on real-time data management. — Apr 7, 2025 — Register Now - [Meetup](https://lu.ma/0av3pvoa) — NYC Data + AI Happy Hour — An evening of insights, networking, and collaboration with top minds in data engineering, AI, and streaming technologies. — Mar 20, 2025 — Register Now - [Webinar](https://share.hsforms.com/2uv17wJDUQlmxnw7--DTaOg59xvt) — Secure Kafka Data Sharing Without Duplication — Learn how to turn streaming data into a valuable asset with secure kafka sharing—without costly duplication. — Mar 18, 2025 — Watch Now - [Conference](https://share.hsforms.com/2fMqwEnwCT4abgzPlGPKiqA59xvt) — Conduktor at Gartner Data & Analytics Summit 2025 — Visit us at Booth #1202 in the AI & Data Science Village. — Mar 3, 2025 — Book Meeting - [Webinar](https://share.hsforms.com/28kl5YPcPTb6_9lkfzQVBLg59xvt) — Migrating from Conduktor Desktop to Scale — After years of helping you make the most of Kafka, Conduktor Desktop is stepping aside. Why? Because your data streaming needs have evolved—and so have we. — Feb 5, 2025 — Watch Now - **Webinar** — Data Strategies for 2025: Data Quality, Operations and AI Readiness — Hybrid cloud, real-time systems, and AI are redefining enterprise data strategies for 2025. To thrive, businesses must focus on data quality, operational efficiency, and AI readiness. — Jan 21, 2025 — events/webinars/data-strategies-for-2025-data-quality-operations-and-ai-readiness — Watch Now - **Webinar** — From Chaos to Control: Mastering Data Quality in Streaming — Our Streamline: Conversations in Data sessions are designed for data leaders, strategists, and anyone looking to get more out of their data. No technical deep dives—just a straight-talking conversation on how to get ahead with trusted data. — Nov 14, 2024 — events/webinars/from-chaos-to-control-mastering-data-quality-in-streaming — Watch Now - **Webinar** — GitOps & Kafka: Streamline Access Management with Console API — This webinar will deep-dive into automating Kafka using a GitOps approach — Aug 30, 2024 — events/webinars/gitops-kafka-streamline-access-management-with-console-api — Watch Now - **Webinar** — Conduktor 2.0: The New Standard for Teams Working with Kafka — We announced Conduktor 2.0, the new way for teams and organizations to work with real-time data at scale. We bring a new Free Tier and a complete new Self-Serve GitOps approach. — Mar 26, 2024 — events/webinars/conduktor-2-0-the-new-standard-for-teams-working-with-kafka — Watch Now - **Webinar** — Live Demo: Getting Started with Conduktor + Q&A — Join us to discover how Conduktor's platform promotes better collaboration between Platform Teams, Architects and Software Developers working with Apache Kafka. — Mar 13, 2024 — events/webinars/live-demo-getting-started-with-conduktor-q-a — Watch Now - **Webinar** — Data Mesh in Practice: Kafka for 50+ Teams with FlixBus — Lessons learned at Flixbus from building and maintaining a distributed self-service streaming platform — Feb 21, 2024 — events/webinars/data-mesh-in-practice-kafka-for-50-teams-with-flixbus — Watch Now - **Webinar** — Kafka Confidential: Data Encryption Simplified — Join us to explore crucial considerations for implementing internal encryption and safeguarding data shared with third parties. — Jan 24, 2024 — events/webinars/kafka-confidential-data-encryption-simplified — Watch Now - **Webinar** — Kafka Security 2.0: Securing Your Data with Encryption, RBAC and Audit — This webinar will focus on practical implementations for addressing Kafka data security and access management needs via Conduktor. We'll recap why ACLs alone are not enough. — Jul 19, 2023 — events/webinars/kafka-security-2-0-securing-your-data-with-encryption-rbac-and-audit — Watch Now - **Webinar** — Rethinking Your Kafka Experience with Conduktor Gateway — Apache Kafka is a highly available and scalable data streaming platform that thousands of companies use to support their mission-critical applications. — Jun 17, 2023 — events/webinars/rethinking-your-kafka-experience-with-conduktor-gateway — Watch Now - **Webinar** — Unleashing the Power of Kafka — Discover the latest innovations in Conduktor, the go-to solution trusted by hundreds of thousands of developers worldwide to simplify and manage Apache Kafka. — May 3, 2023 — events/webinars/unleashing-the-power-of-kafka — Watch Now - **Webinar** — Make Your Most Valuable Data Safe for AI — Organizations may lack the tools to identify and restrict access to sensitive data; they may struggle with fragile pipelines that require too much maintenance; or they may be concerned about data issues which can impact downstream applications. — events/webinars/make-your-most-valuable-data-safe-for-ai — Watch Now # Want to host a meetup or partner event? We sponsor and speak at leading data streaming conferences worldwide. [Contact Us](https://www.conduktor.io/contact/demo?src=events) Source: https://www.conduktor.io/events --- # Federated Ownership *Accelerate Revenue, Scale with the Same Team, and Cut Operational Costs* Automate guardrails, define ownership, and enable discovery. Developers get autonomy, platform teams maintain control, and every team creates data products they can trust. [Talk to Us](https://www.conduktor.io/contact/demo?src=federated-ownership) [Download Solution Brief](https://www.conduktor.io/assets/Conduktor-Federated-Ownership-Solution-Brief.pdf) ## Kafka Scales. Manual Processes Don't. The scripts, the Slack requests, the 'just ask the platform team'? They worked at 5 teams, but scaling them is a full-time job. Your platform team has better things to build. - **Misconfigurations cause downtime and erode reliability** — Without automated validation, broken configs slip through to production. Every hour of unplanned downtime directly impacts revenue and customer trust. - **Unclear ownership slows delivery and inflates platform costs** — Developers wait days for access because every request routes through the platform team. At scale, there's no system of record, and Kafka adoption stalls. - **Duplication inflates infrastructure costs and wastes engineering capacity** — Teams rebuild what already exists because there's no way to discover it. Redundant infrastructure and engineering effort compound to $100K+ per year in waste. Impact with Conduktor Based on results reported by Conduktor customers. - **75% Fewer Provisioning Tickets** — as developers self-serve instead of filing requests. Platform teams stop being a bottleneck for every topic, schema, and connector. - **$200K+ Annual Savings** — from reduced duplication, faster provisioning, and clear ownership. - **3,500+ Hours Saved Per Year** — across provisioning, access reviews, and ownership resolution. Time reclaimed goes back to building, not administrating. - **4x Faster Provisioning** — through automated guardrails that validate instantly, so teams don't wait on manual review. See How Teams Do It [FlixBus scaled Kafka to 50+ teams and 2,300+ topics with a data mesh architecture and federated governance.Read FlixBus's story →](https://www.conduktor.io/customer-stories/flix)[Swiss Post scaled Kafka to 800+ users and 5x more applications with governance, RBAC, and self-service.Read Swiss Post's story →](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage)[Virgin Australia saves 300 hours a month — faster onboarding, simplified workflows, and broader Kafka adoption.Read Virgin Australia's story →](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) How Federated Ownership Works Five steps to shift control to the right people. Automate GuardrailsCEL-based policies validate instantly. No manual review needed. Define OwnershipDeclare applications so every resource maps to an accountable team. Enable DiscoveryMake topics discoverable. Teams reuse existing data instead of rebuilding. Approve Cross-Team AccessData owners approve requests. Platform teams stay out of the loop. Create Trusted Data ProductsClear ownership, validated configs, governed access. Data products your org can depend on. Automate GuardrailsCEL-based policies validate instantly. No manual review needed. Define OwnershipDeclare applications so every resource maps to an accountable team. Enable DiscoveryMake topics discoverable. Teams reuse existing data instead of rebuilding. Approve Cross-Team AccessData owners approve requests. Platform teams stay out of the loop. Create Trusted Data ProductsClear ownership, validated configs, governed access. Data products your org can depend on. Automated Guardrails ## Self-Service Provisioning with Automated Guardrails Developers create topics, schemas, and connectors themselves. Policies validate instantly. No manual review needed. - **Naming conventions** enforced via regex patterns - **Partition and retention limits** enforced via min/max bounds - **Config restrictions** like `cleanup.policy`, `compression.type`, and `min.insync.replicas` - **Custom error messages** shown in Console, CLI, or GitOps when validation fails - **Compliant resources** created instantly in Kafka and tracked in Console [Learn how to configure policies →](https://docs.conduktor.io/guide/reference/self-service-reference#resourcepolicy) Ownership ## Application Catalog Define applications with ownership patterns so every resource maps to an accountable team. No more guessing who owns what. - **Ownership patterns** using literal names or prefixes like `orders-*` - **Permission modes** for full control (ALL) or read/write only (LIMITED) - **Service account binding** with one service account per application instance - **ACL generation** with Kafka ACLs auto-created from ownership definitions [Learn how to define applications →](https://docs.conduktor.io/guide/reference/self-service-reference#application) Discovery ## Topic Catalog Discover data across clusters. Control visibility to protect sensitive resources. Reuse existing topics instead of rebuilding. - **Full-text search** across topic names and descriptions - **Label-based filtering** by team, domain, or environment - **Public/private visibility** to control who can discover each topic - **Subscriber stats** showing which apps consume each topic [Learn how to use the topic catalog →](https://docs.conduktor.io/guide/use-cases/self-service#topic-catalog-page) Collaboration ## Approval Workflows Request access to other teams' resources for cross-team collaboration. Data owners approve or reject — not platform teams. Works with GitOps or direct approval. - **Owner group approval** so only members of the owning team can approve - **Granular grants** with separate permissions for users and service accounts - **GitOps or UI** to approve via pull request or directly in Console - **Full state history** tracking who requested, who reviewed, when, and why [Learn how approval workflows work →](https://docs.conduktor.io/guide/use-cases/self-service#application-instance-permissions) ## Why Conduktor? - **vs. Building Your Own** — No policy engine to build. No approval workflows to maintain. No catalog to keep in sync. Focus on your product, not your platform. - **vs. Scripts & Tickets** — Scripts break on edge cases. Tickets create bottlenecks. Automated guardrails validate instantly, whether you have 10 apps or 1,000. - **vs. Confluent Cloud** — Confluent offers fewer than 25 fixed RBAC roles with no self-service or approval workflows. Conduktor enables fine-grained, team-level permissions within guardrails. ## Frequently Asked Questions **How is this different from just using Terraform or IaC?** Terraform provisions resources but doesn't validate them against policies. Conduktor adds a governance layer — CEL-based policies validate configs at creation time, ownership is tracked automatically, and cross-team access has a proper workflow. You can use both together. **How do I prevent teams from creating inconsistent resources?** Conduktor's policy controls enforce naming conventions, partition limits, retention settings, and other standards automatically. Teams can only create resources that comply with your defined policies. **Can I still require approvals for sensitive operations?** Yes. Conduktor supports configurable approval workflows. You can require manual approval for production changes while allowing self-service for development environments. **Do I need to restructure my existing Kafka setup?** No. Conduktor works with your existing Kafka clusters — Confluent, AWS MSK, Redpanda, or self-managed. You define applications and ownership patterns on top of what you already have. **How does this integrate with GitOps and CI/CD?** Conduktor has a CLI and Terraform provider for managing resources as code. Teams define applications, topics, and permissions in YAML, commit to Git, and deploy through your existing pipelines. Policies validate at every step. # Ready to automate Kafka governance? See how federated ownership works in your environment. Automate guardrails, define ownership, and enable discovery — on the Kafka you already run, without adding a bottleneck. [Talk to Us](https://www.conduktor.io/contact/demo?src=federated-ownership) [Download Solution Brief](https://www.conduktor.io/assets/Conduktor-Federated-Ownership-Solution-Brief.pdf) Source: https://www.conduktor.io/federated-ownership --- # Get started with Gateway A free Kafka proxy that connects clients to clusters sitting in other VPCs, clouds, or private networks, with no changes to your brokers or credentials. Free forever. No trial, no credit card. [Read the docs →](https://docs.conduktor.io/guide/conduktor-in-production/manage-licenses/gateway-community-edition) ### Quickstart Run this on any machine with Docker. It clones the quickstart, prompts for your key, and brings up a working demo so you can watch clients reach Kafka across networks. ```bash bash <(curl -fsSL https://releases.conduktor.io/gateway-community-quickstart) ``` [View on GitHub →](https://github.com/conduktor/gateway-community-quickstart) ## When to Use Gateway Community Edition Three patterns where the proxy fits. Reach Kafka without adding peerings Every new client VPC would otherwise need its own peering with the cluster. Route them through one proxy in an already-peered VPC. The cluster sees one attachment, no matter how many client networks reach in. Reach private clusters from outside In situations where a cluster has no public endpoint, a proxy on the boundary lets external clients, partners, or cross-cloud workloads reach in. No cluster reconfiguration or per-client endpoints. Single egress point Each internal client opens its own connection to remote Kafka. One proxy funnels them all. One firewall rule, one TLS termination, and one audit trail. ## How It Works Three steps from unreachable cluster to working client. Drop in Update bootstrap.servers to the proxy's address. No client code changes. Existing SASL credentials. Translate The proxy rewrites broker addresses in Kafka's metadata responses, so clients see addresses they can actually reach. Route Each broker gets its own port on the proxy. Traffic routes deterministically per broker. SASL passes straight through. Without Gateway Community Edition With Gateway Community Edition Cluster-side attachments One peering or PrivateLink per client network attached to the cluster One attachment to the cluster, new networks attach to the proxy instead Exposing the cluster externally Per-broker NLB or PrivateLink endpoint, DNS, and TLS cert One ingress to the proxy, per-broker routing handled inside Adding network paths Rolling restart with reconfigured listeners, or impossible on managed Kafka New paths added on the proxy, the cluster stays untouched Egress for many internal clients Each client opens its own outbound path to firewall and audit Single egress chokepoint, one path to audit What You Don't Add The proxy adds reachability. Nothing else. No broker reconfiguration Adds new advertised addresses to clusters you can't change. Works with managed Kafka where listener config is locked. No new credentials SASL flows straight through. Existing API keys and OAuth tokens authenticate directly against Kafka. No new auth model Kafka ACLs and Confluent RBAC remain the source of truth. The proxy doesn't issue or store identities. Conduktor Gateway Ready to govern, secure, and control Kafka traffic? Everything in Community Edition, plus the governance and security controls teams need to run Kafka in production. The same proxy, fully unlocked. Encryption and field-level masking Schema validation and enforcement Virtual Clusters and multi-tenancy Interceptors for custom policies Disaster recovery and failover Partner data sharing [Get Gateway Enterprise →](https://www.conduktor.io/pricing#gateway) Get your Gateway key Free forever, no credit card. Your key and the install command arrive by email. ## Frequently Asked Questions **Why is Gateway Community Edition free?** We built it because customers kept asking for the reachability piece of Gateway without the full governance stack. Free isn't a trial. There's no time limit, no expiration, and no plan to put it behind a paywall. **What's the difference between Gateway Community Edition and Conduktor Gateway?** Gateway Community Edition is a license-restricted mode of Conduktor Gateway. It handles broker address translation and passes SASL credentials through to Kafka. Full Conduktor Gateway adds governance, encryption, masking, schema enforcement, virtual clusters, disaster recovery, and partner data sharing. [See Gateway →](https://www.conduktor.io/gateway) **What features are not included?** A Gateway Community Edition license excludes Interceptors, topic views, alias topics, Virtual Clusters, Gateway service accounts, Gateway groups, topic concentration, and failover. These all require a full Conduktor Gateway license. **Does the proxy terminate or store my Kafka credentials?** No. SASL flows straight through to Kafka in KAFKA_MANAGED mode. The broker authenticates the client, and Kafka ACLs or Confluent RBAC enforce authorization. **Which Kafka providers are supported?** Any Kafka 2.7+, including Confluent Cloud, Confluent Platform, AWS MSK, GCP Managed Service for Apache Kafka, Aiven, Redpanda, and self-managed Apache Kafka. SASL_PLAINTEXT and SASL_SSL only. **How does this compare to Grepplabs kafka-proxy, Kroxylicious, or AWS MSK Multi-VPC?** Grepplabs kafka-proxy is an open-source passthrough proxy with similar scope, without commercial support or operational tooling. Kroxylicious is a filter framework with broader scope that requires authoring and operating filters. AWS MSK Multi-VPC Private Connectivity is AWS-only and MSK-only. Gateway Community Edition is the right choice when you want a narrow-scope address translator that works across any Kafka, with an upgrade path to full Conduktor Gateway when you need governance, encryption, or schema enforcement. **Does it work across cloud providers?** Yes. Cross-cloud is a common use case. For example, on-prem or AWS clients reaching a Confluent Cloud cluster on GCP. **What level of support does Gateway Community Edition come with?** Same as [Console Community Edition](https://www.conduktor.io/pricing#console-community-edition): email support, plus docs and Slack as available. Last updated: June 2026 Source: https://www.conduktor.io/gateway/community-edition --- # Conduktor Gateway *Route. Protect. Transform.* A Kafka proxy that gives platform teams the controls they've always wanted without asking developers to change a single line of code. [Talk to Us](https://www.conduktor.io/contact/demo?src=gateway) [Download Product Overview](https://www.conduktor.io/assets/Conduktor-Gateway-Overview.pdf) ## Augment Your Infrastructure. Kafka is powerful, but it wasn't built for governance, security, or policy enforcement. A Kafka Gateway closes that gap, so platform teams can tailor Kafka to their requirements, not the other way around. **For Platform Teams** Scale Kafka across teams without multiplying infrastructure or complexity. **For Security & Compliance** Enforce encryption and data policies at the infrastructure layer, not in application code. ## What Is Conduktor Gateway? Conduktor Gateway is a [Kafka proxy](https://www.conduktor.io/kafka-proxy) that sits between your client applications and your Kafka brokers. It gives platform teams a single control point for security, routing, data quality, and multi-tenancy without changing application code. - **Native Kafka protocol** — Gateway speaks the Kafka wire protocol. Clients connect to it exactly like a broker. Just update `bootstrap.servers` and the proxy handles authentication, encryption, routing, and observability transparently. - **Infrastructure-layer control** — Add capabilities that aren't available natively, like field-level encryption, virtual clusters, SQL-based topic views, and chaos testing, without asking every application team to implement them individually. - **Zero application changes** — Every produce and consume request passes through the proxy. Policies are applied at the infrastructure layer, so platform teams can enforce standards across all teams from one place. **Just need to reach Kafka across networks?** Try [Gateway Community Edition](https://www.conduktor.io/gateway/community-edition), our free Kafka proxy. It connects clients to clusters sitting in other VPCs, clouds, or private networks, with no changes to your brokers or credentials. Gateway Use Cases [Disaster Recovery & FailoverAutomate cluster failover so applications recover in minutes, not hours. No code changes, no cross-team coordination, no manual runbooks. Test your recovery path with chaos engineering before you need it.See how →](https://www.conduktor.io/solutions/use-case/disaster-recovery-and-failover) [Data SharingShare real-time Kafka data with external partners through isolated Partner Zones. No replica clusters, no custom APIs. Standard Kafka protocol with built-in compliance controls and per-partner usage tracking.See how →](https://www.conduktor.io/solutions/use-case/data-sharing) [Kafka Cost OptimizationRecover 25 to 40 percent of your Kafka infrastructure bill through visibility, attribution, and architectural levers. Track costs by team, identify waste, and act without disruption.See how →](https://www.conduktor.io/solutions/use-case/kafka-cost-optimization) [Kafka MigrationMigrate to the cloud or switch providers with transparent traffic routing. Zero downtime cutover with no application changes or cross-team coordination.See how →](https://www.conduktor.io/solutions/use-case/kafka-migration) [Schema Registry GovernanceAdd authentication, access controls, and audit logging to your schema registry. A drop-in proxy that validates JWT/OIDC tokens, enforces per-subject permissions, and traces every operation.See how →](https://www.conduktor.io/schema-registry-proxy) Routing, Multi-tenancy & Performance Eliminate network barriers, deliver isolated Kafka environments without separate clusters, and optimize throughput at the proxy layer. Client Access Control & Routing ## Connect without code changes Remove network barriers to Kafka connectivity with Kafka proxy routing, centralize authentication through OIDC integration, and enforce fine-grained access control beyond what native ACLs provide. - **Cross-network connectivity** routing client connections through Gateway without client configuration changes - **Centralized authentication** integrating with external OIDC systems for unified access control across hybrid deployments - **Application audit trails** tracking every action across Kafka resources to meet SOC2, ISO 27001, and GDPR requirements [Learn more →](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/client-connections) Route Clients Centralize Authentication Audit Access # Gateway Configuration gateway: environment: GATEWAY_SECURITY_MODE: GATEWAY_MANAGED GATEWAY_SECURITY_PROTOCOL: SASL_PLAINTEXT # OIDC Provider Settings GATEWAY_OAUTH_JWKS_URL: "https://your-idp.com/.well-known/jwks.json" GATEWAY_OAUTH_EXPECTED_ISSUER: "https://your-idp.com" GATEWAY_OAUTH_EXPECTED_AUDIENCES: "kafka-gateway" GATEWAY_OAUTH_SUB_CLAIM_NAME: "sub" # Map OIDC identities to Gateway Service Accounts apiVersion: gateway/v2 kind: GatewayServiceAccount metadata: name: my-application spec: type: EXTERNAL externalNames: - "oauth-subject-id-from-token" # Value from 'sub' claim in JWT // Audit log event { "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "type": "APIKEYS_REQUEST", "time": "2024-10-15T14:30:45.123Z", "source": "//kafka/cluster/production", "authenticationPrincipal": "tenant-acme", "userName": "order-service", "connection": { "localAddress": "172.17.0.2:6969", "remoteAddress": "192.168.1.42:52341" }, "eventData": { "apiKeys": "PRODUCE", "topics": [ { "name": "orders", "partition": 0 }, { "name": "orders", "partition": 1 } ] }, "specVersion": "0.1.0" } Multi-tenancy & Virtualization ## Decouple tenants from clusters Deliver isolated Kafka environments to multiple teams without the cost of separate clusters, eliminate topic naming chaos, and dramatically reduce partition counts. - **Virtual clusters** deploying multiple logical Kafka clusters on a single physical cluster with isolated namespaces and ACL boundaries - **Topic concentration** consolidating low-volume topics into shared physical topics, reducing partition count by 90%+ - **Topic aliases** accessing topics with aliased names to rename without client changes or hide internal naming [Learn more →](https://docs.conduktor.io/guide/conduktor-concepts/virtual-clusters) Virtualize Clusters Concentrate Topics Alias Topics apiVersion: gateway/v2 kind: VirtualCluster metadata: name: payments-team spec: type: Standard aclEnabled: true superUsers: - payments-admin apiVersion: gateway/v2 kind: VirtualCluster metadata: name: orders-team spec: type: Standard aclEnabled: true superUsers: - orders-admin apiVersion: gateway/v2 kind: AliasTopic metadata: name: customers vCluster: partner-team spec: physicalName: internal-crm-customers apiVersion: gateway/v2 kind: AliasTopic metadata: name: orders vCluster: partner-team spec: physicalName: internal-billing-orders Performance & Efficiency ## Handle load efficiently Reduce broker load from repetitive reads, handle oversized payloads without impacting cluster performance, and provide filtered topic views without stream processing complexity. - **SQL topics** creating read-only virtual topics with SQL-based filtering and projections at the gateway layer - **Caching** serving frequently-accessed messages from Gateway cache to reduce broker load for high-frequency reads - **Large message offload** automatically offloading messages exceeding size thresholds to S3 or Azure Blob Storage [Learn more →](https://docs.conduktor.io/guide/use-cases/cache-data) Query Topics Cache Messages Offload Payloads apiVersion: gateway/v2 kind: Interceptor metadata: name: sql-filter-adults spec: pluginClass: io.conduktor.gateway.interceptor.VirtualSqlTopicPlugin priority: 100 config: virtualTopic: customers-adult statement: | SELECT firstName, lastName, email, country FROM customers WHERE age >= 18 AND country = 'US' schemaRegistryConfig: host: http://schema-registry:8081 apiVersion: gateway/v2 kind: Interceptor metadata: name: cache-high-traffic-topics spec: pluginClass: io.conduktor.gateway.interceptor.CacheInterceptorPlugin priority: 100 config: topic: "events.*" cacheConfig: type: IN_MEMORY inMemConfig: cacheSize: 1000 expireTimeMs: 60000 apiVersion: gateway/v2 kind: Interceptor metadata: name: offload-large-messages-s3 spec: pluginClass: io.conduktor.gateway.interceptor.LargeMessageHandlingPlugin priority: 100 config: topic: "media.*" minimumSizeInBytes: 1048576 localDiskDirectory: /tmp/kafka-offload s3Config: bucketName: kafka-large-messages region: us-east-1 Schema Registry Governance ## Govern your schema registry Add authentication, access control, and observability to your schema registry with a drop-in Kafka proxy that enforces governance on every operation. - **JWT/OIDC authentication** validating tokens from your existing identity provider before any request reaches the registry - **Per-subject access controls** enforcing Read and Write permissions with exact match, wildcard, and prefix matching - **Full observability** tracing every operation with OpenTelemetry, metering with Prometheus, and logging in structured JSON [Learn more →](https://www.conduktor.io/schema-registry-proxy) Configure Proxy Define Permissions Authenticate # docker-compose.yml schema-registry-proxy: ports: - "8080:8080" - "9464:9464" # Prometheus metrics environment: # Schema Registry backend CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081 # JWT authentication via OIDC provider AUTH_PROVIDER: jwt JWT_JWKS_URL: http://keycloak:8080/realms/kafka/protocol/openid-connect/certs JWT_SUBJECT_CLAIM_NAME: sub # Dynamic permissions via Kafka AUTH_USE_REACTIVE_CONFIG: true KAFKA_BOOTSTRAP_SERVERS: kafka:9092 # OpenTelemetry tracing OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317 # Permission format (Kafka topic: _conduktor_srp_commands) # Key: service account name # Value: permission entries # Admin — full access to all subjects { "permissions": [ { "subject": "*", "permissionType": "WRITE" } ]} # Payments team — prefix matching { "permissions": [ { "subject": "payments-*", "permissionType": "WRITE" }, { "subject": "orders-*", "permissionType": "READ" } ]} # Analytics — read-only, all subjects { "permissions": [ { "subject": "*", "permissionType": "READ" } ]} # Get a JWT token (client credentials flow) TOKEN=$(curl -s -X POST \ http://keycloak:8080/realms/kafka/protocol/openid-connect/token \ -d "grant_type=client_credentials" \ -d "client_id=payments-service" \ -d "client_secret=secret" | jq -r '.access_token') # Register a schema through the proxy curl -X POST http://localhost:8080/subjects/payments-value/versions \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"schema": "{\"type\":\"record\",\"name\":\"Payment\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}"}' # Response: {"id": 1} # Metrics: schema_registry_requests_total +1 # Trace: full span in Jaeger/OTLP Resilience & Disaster Recovery Prevent developer misconfigurations from destabilizing your platform, maintain operational continuity during failures, and validate fault tolerance through controlled testing. Application Resilience ## Prevent misconfigurations Eliminate noisy-neighbor problems and configuration drift with Kafka Gateway safeguards that enforce naming conventions, retention limits, quotas, and replication rules before issues reach production. - **Configuration policies** enforcing topic naming conventions, retention limits, replication rules, and schema requirements at provision and runtime - **Traffic control & quotas** enforcing bandwidth and rate limits per tenant or cluster to prevent noisy neighbors - **Producer & consumer policies** ensuring acks, compression, and idempotence requirements across all producers and controlling fetch behavior on consumers [Learn more →](https://docs.conduktor.io/guide/use-cases/apply-traffic-control-policies) Guard Configs Limit Traffic apiVersion: gateway/v2 kind: Interceptor metadata: name: topic-governance-policy spec: pluginClass: io.conduktor.gateway.interceptor.safeguard.CreateTopicPolicyPlugin priority: 100 config: namingConvention: value: "^[a-z]+-[a-z]+-[a-z]+$" action: BLOCK numPartition: min: 3 max: 12 action: OVERRIDE overrideValue: 6 replicationFactor: min: 3 max: 3 action: BLOCK retentionMs: min: 86400000 max: 604800000 action: OVERRIDE overrideValue: 259200000 apiVersion: gateway/v2 kind: Interceptor metadata: name: producer-rate-limit scope: vCluster: payments-team spec: pluginClass: io.conduktor.gateway.interceptor.safeguard.ProducerRateLimitingPolicyPlugin priority: 100 config: maximumBytesPerSecond: 10485760 action: BLOCK apiVersion: gateway/v2 kind: Interceptor metadata: name: consumer-rate-limit scope: vCluster: payments-team spec: pluginClass: io.conduktor.gateway.interceptor.safeguard.ConsumerRateLimitingPolicyPlugin priority: 100 config: maximumBytesPerSecond: 52428800 Disaster Recovery & Failover ## Migrate and recover with confidence Maintain operational continuity during cluster migrations and failover with transparent Kafka proxy redirection, and validate fault tolerance through controlled chaos testing. - **Cluster switching & failover** migrating applications between clusters or switching to failover infrastructure without application config changes or team coordination - **Chaos engineering** injecting latency, protocol errors, and message corruption via Gateway interceptors to test resilience without disrupting production [Learn more →](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/failover) Switch Clusters Failover Traffic Inject Chaos # Gateway cluster configuration config: main: bootstrap.servers: kafka-primary:9092 security.protocol: SASL_SSL sasl.mechanism: PLAIN failover: bootstrap.servers: kafka-secondary:9092 gateway.roles: failover # Switch from main → failover curl -X POST 'http://localhost:8888/gateway/v2/cluster-switching' \ -H 'Content-Type: application/json' \ -d '{"fromPhysicalCluster": "main", "toPhysicalCluster": "failover"}' # Chaos testing interceptor apiVersion: gateway/v2 kind: Interceptor metadata: name: chaos-broken-broker spec: pluginClass: io.conduktor.gateway.interceptor.chaos.SimulateBrokenBrokersPlugin priority: 100 config: rateInPercent: 100 errorMap: FETCH: UNKNOWN_SERVER_ERROR PRODUCE: CORRUPT_MESSAGE Data Protection Control message metadata, enforce data quality, protect sensitive fields, and verify message integrity at the proxy layer. Data Quality ## Stop bad data at the source Stop malformed and non-compliant data at the source rather than discovering quality issues after they've broken downstream systems. - **Data quality detection** evaluating data flowing through Gateway for compliance with validation rules and quality standards - **Data quality enforcement** blocking, redirecting, or marking messages that violate quality rules before they reach Kafka [Learn more →](https://docs.conduktor.io/guide/use-cases/enforce-data-quality) Define Quality Enforce Quality { "name": "myDataQualityProducerPlugin", "pluginClass": "io.conduktor.gateway.interceptor.safeguard.DataQualityProducerPlugin", "priority": 100, "config": { "statement": "SELECT x FROM orders WHERE amount_cents > 0 AND amount_cents < 1000000", "schemaRegistryConfig": { "host": "http://schema-registry:8081" }, "action": "BLOCK_WHOLE_BATCH", "deadLetterTopic": "dead-letter-topic", "addErrorHeader": false } } Data Confidentiality ## Encrypt consistently across apps Ensure consistent encryption across all applications with centralized KMS enforcement at the Kafka Gateway layer, and enable analytics on sensitive data through field-level tokenization. - **Centralized encryption** enforcing encryption with a standardized KMS connection to prevent config drift and enable crypto-shredding for GDPR compliance - **Multi-cloud KMS integration** connecting with Vault, AWS KMS, Azure Key Vault, GCP KMS, or Fortanix for PCI-DSS, HIPAA, and GDPR compliance - **Field-level encryption** encrypting sensitive fields with schema-based targeting while keeping non-sensitive data readable - **Tokenization** replacing sensitive values with tokens while preserving format for analytics [Learn more →](https://docs.conduktor.io/guide/use-cases/encrypt-kafka) · [Kafka Encryption →](https://www.conduktor.io/kafka-encryption) Encrypt Data Manage Keys apiVersion: gateway/v2 kind: Interceptor metadata: name: encrypt-pii-fields spec: pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin priority: 100 config: topic: "customers.*" recordValue: fields: - fieldName: ssn keySecretId: "vault-kms://vault:8200/transit/keys/pii-key" algorithm: AES256_GCM - fieldName: creditCard.number keySecretId: "vault-kms://vault:8200/transit/keys/payment-key" algorithm: AES256_GCM - fieldName: email keySecretId: "in-memory-kms://email-key" algorithm: AES128_GCM kmsConfig: vault: uri: http://vault:8200 type: TOKEN token: ${VAULT_TOKEN} apiVersion: gateway/v2 kind: Interceptor metadata: name: encrypt-with-kms spec: pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin priority: 100 config: topic: ".*" recordValue: fields: - fieldName: ssn keySecretId: "vault-kms://..." - fieldName: payment.cardNumber keySecretId: "aws-kms://..." - fieldName: healthRecord keySecretId: "azure-kms://..." kmsConfig: vault: uri: http://vault:8200 type: APP_ROLE aws: basicCredentials: accessKey: ${AWS_ACCESS_KEY} azure: tokenCredential: tenantId: ${AZURE_TENANT_ID} Data Integrity ## Verify message integrity Verify message authenticity and detect tampering to meet audit and chain-of-custody requirements. - **Cryptographic signing** signing messages at the gateway to detect tampering and verify message origin for compliance requirements [Learn more →](https://docs.conduktor.io/guide/reference/data-security#message-integrity-interceptor) Sign Messages apiVersion: gateway/v2 kind: Interceptor metadata: name: sign-messages spec: pluginClass: io.conduktor.gateway.interceptor.integrity.ProduceIntegrityPolicyPlugin priority: 100 config: topic: "transactions.*" secretKeyUri: "secret/data/signing-key#key" keyProviderConfig: vault: uri: https://vault:8200 type: TOKEN token: ${VAULT_TOKEN} Metadata Control ## Control message metadata Ensure that message metadata is consistent with enterprise standards without requiring application changes. - **Header & message handling** injecting, modifying, or removing message headers to transform and control metadata centrally [Learn more →](https://docs.conduktor.io/gateway) Inject Headers Remove Headers apiVersion: gateway/v2 kind: Interceptor metadata: name: inject-tracking-headers spec: pluginClass: io.conduktor.gateway.interceptor.DynamicHeaderInjectionPlugin priority: 100 config: topic: "orders.*" headers: X-Source-Region: "${GATEWAY_REGION}" X-Gateway-Timestamp: "${timestamp}" overrideIfExists: true apiVersion: gateway/v2 kind: Interceptor metadata: name: remove-internal-headers spec: pluginClass: io.conduktor.gateway.interceptor.safeguard.MessageHeaderRemovalPlugin priority: 100 config: topic: ".*" headerKeyRegex: "^X-Internal-.*" External Data Sharing Share Kafka data with external partners without overexposing your infrastructure, adapt data formats for partner consumption, and track usage for cost allocation. Partner Access & Data Sharing ## Share data securely Share Kafka data with external partners without overexposing your infrastructure, adapt data formats without dual-write patterns, and track usage for billing. - **Partner virtual clusters** provisioning isolated virtual clusters with topic mappings, rate limits, and service accounts for third parties - **Partner topic views** creating logical topic views with aliasing for partner-specific access without duplicating infrastructure - **Data format transformations** transforming message formats using SQL expressions for partner-specific schemas - **Partner chargeback** tracking consumption metrics by partner for usage attribution and cost allocation Sharing tables through a lakehouse instead? See how streaming compares in [Delta Sharing alternatives for real-time data](https://www.conduktor.io/compare/delta-sharing-alternatives). [Learn more →](https://www.conduktor.io/solutions/use-case/data-sharing) Track Usage Provision Tenants apiVersion: gateway/v2 kind: VirtualCluster metadata: name: partner-a spec: type: Partner aclEnabled: true superUsers: - partner-admin apiVersion: gateway/v2 kind: GatewayServiceAccount metadata: name: partner-admin vCluster: partner-a spec: type: LOCAL apiVersion: gateway/v2 kind: AliasTopic metadata: name: orders vCluster: partner-a spec: physicalName: internal-orders Kafka Proxy Comparison How does Conduktor Gateway stack up? Compare Conduktor Gateway against the field, feature by feature. Confluent Gateway Kong Gravitee Aklivity Zilla Kroxylicious [Compare Kafka proxies →](https://www.conduktor.io/compare/kafka-proxies) Measurable Impact Real results from platform teams using Gateway. - **25 clusters migrated** — European airline moved to Confluent Cloud in 9 months with zero downtime. - **PCI DSS compliant** — Payment processor achieved MasterCard and VISA certification with Gateway encryption. - **50+ teams, one Kafka proxy** — FlixBus scaled multi-tenancy without multiplying infrastructure. - **Zero code changes** — Encryption, routing, and policies applied at the proxy layer, not in applications. ## Read more customer stories - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) - [Banking: PCI DSS Compliance](https://www.conduktor.io/customer-stories/securing-kafka-banking-financial-services) - [FlixBus: 50+ Teams Data Mesh](https://www.conduktor.io/customer-stories/flix) ## Ready to Try Gateway? See how platform teams use our Kafka proxy to add encryption, multi-tenancy, and traffic control without changing application code. [Talk to Us](https://www.conduktor.io/contact/demo?src=gateway) ### Explore Gateway by Use Case [Kafka Security →](https://www.conduktor.io/kafka-security) · [Kafka Encryption →](https://www.conduktor.io/kafka-encryption) · [Kafka RBAC →](https://www.conduktor.io/kafka-rbac) · [Schema Registry Proxy →](https://www.conduktor.io/schema-registry-proxy) Source: https://www.conduktor.io/gateway --- # Get Started with Console One platform to explore, build, secure, and govern Kafka. Works with Confluent, MSK, Redpanda, or any Kafka 2.5+. ### Install it yourself Install [Conduktor Community](https://www.conduktor.io/community) on your own infrastructure. Docker Helm Kubernetes AWS CloudFormation ```bash curl -fL https://releases.conduktor.io/quick-start | docker compose -f - up ``` Access the Console at `localhost:8080` once it's running. Ships with embedded Redpanda, Schema Registry, and sample data. ```bash # full values & config in the docs → helm repo add conduktor https://helm.conduktor.io helm install console conduktor/console -f ``` Deploys Console with the official Helm chart. Requires an external PostgreSQL database; point it at your own Kafka. ```bash # full manifests & config in the docs → helm install console conduktor/console -f kubectl port-forward deployment/console 8080:8080 ``` Runs Console on your cluster. Expose it on `localhost:8080` with a port-forward or an ingress. ```bash # ECS Fargate + RDS — full steps in the docs → docker pull conduktor/conduktor-console: ``` Runs Console as a container on ECS Fargate, backed by an RDS PostgreSQL database. ```bash # 1-click launch from the AWS Marketplace listing → aws cloudformation deploy --template-file .yaml \ --stack-name --capabilities CAPABILITY_IAM ``` One-click deploy from the AWS Marketplace — Console and RDS, provisioned by a CloudFormation stack. [Full Docker install docs →](https://docs.conduktor.io/guide/get-started) ### Install with AI Recommended Let your coding agent do the setup. The [Conduktor skill](https://github.com/conduktor/skills) gives it agentic access to the CLI and API. ```bash npx skills add conduktor/skills ``` Then ask your assistant ✦Install Conduktor and set it up so I can login. [See what it can do →](https://www.conduktor.io/blog/set-up-a-kafka-platform-with-an-ai-agent) Conduktor Community is free to use. Your use of this product is governed by our [EULA](https://www.conduktor.io/legal/eula). Team Edition Ready for more clusters, more teams, more control? Everything in Community, plus the governance, automation, and Insights teams need to run Kafka at scale. Self-serve, no sales call. Unlimited clusters Federated ownership & approval workflows Topic policy enforcement Cost chargeback & advanced Insights Group-level RBAC Unlimited audit logs & alerts [Get Team Edition →](https://www.conduktor.io/pricing#console-team-edition) ## Watch Product Demos ### Kafka Lineage with Conduktor Console Walk the dependency map for a Kafka estate in Console: application and service account views, live traffic, drill-down by hops, and the consumers nobody registered. ### Kafka Stream Lineage End to End Retire a topic, find a hot path, and turn up an unregistered consumer across an airline's Kafka estate, using a dependency map built from ACLs, ownership, and live Gateway traffic. ### Why Is Effective Kafka Change Management So Tricky? Onboarding a team, retiring a topic, and chasing a hot connection all need the same three answers, and most teams get them by asking around. Why the dependency map is so hard to build. ### Block Bad Data with Data Quality Enforcement Reject malformed Kafka records at produce time with Gateway data quality policies: JSON Schema or CEL rules, plus a dead letter topic the producer can debug. ### Inside Gateway: ACLs, Audit Logs, and Metering Conduktor Gateway stores its state in Kafka topics. A look inside per-virtual-cluster ACLs, the audit log, interceptor configs, and per-partner byte metering. [View All Demos →](https://www.conduktor.io/product-demos) ## What Our Users Say > As a QA/Automation manager my focus is on seamless connectivity between the tools that make up the tech stack in my organization. Conduktor's UI is sleek, intuitive, and significantly enhances Kafka management by providing a visually rich and user-friendly experience. > — **[Gil Solomon](https://www.linkedin.com/in/gil-solomon-6324b0149/)**, QA Manager, Snappy > Conduktor has transformed our Kafka management. Its intuitive interface and robust real-time monitoring tools have streamlined our operations, cutting troubleshooting time by over 30%. The powerful debugging features are invaluable for our team. > — **[Domen Kirn Krefl](https://www.linkedin.com/in/domen-kirn-krefl-51a395144/)**, Backend Engineer, Greentube > We have been using Conduktor for about 1.5 years. Before Conduktor, we were trying to manage our Kafka clusters via cli/postman. It was very hard to use and open for errors. After we installed Conduktor our team quickly adopted it. The platform had a huge impact in our environment. > — **[Furkan Yusuf Pek](https://www.linkedin.com/in/furkan-yusuf-pek/)**, Data Analytics Team Lead, Ciceksepeti > Out of all the Kafka tools I've used, Conduktor is by far the easiest to implement and has the most thorough set of features. As a Production Support Engineer who depends on visibility on all components, Conduktor makes it easy to navigate and troubleshoot issues. > — **Chuck**, Production Support Engineer, Fermenta Energy > Conduktor helped us visualize and understand our data streams without the complexity of manual CLI commands. The interface was intuitive, and it made interacting with Kafka much easier. If you're using Kafka regularly, it's definitely worth trying! > — **[Abdelaziz Cheffi](https://www.linkedin.com/in/abdelaziz-cheffi/)**, Data Analyst, Opalean > Conduktor helps me daily with tracking all my Kafka topics, operations have been more smooth as we can detect quickly bottlenecks. I definitely recommend Conduktor to everyone, the effect on daily operations is huge. > — **[Charles Emmanuel](https://www.linkedin.com/in/cedias/)**, Data Scientist, Intelligent Locations Enjoying Conduktor? [Share your experience on G2](https://www.g2.com/products/conduktor-for-apache-kafka/take_survey) ## Frequently Asked Questions **What are the system requirements?** Console needs 2 CPU cores, 3 GB RAM, and 5 GB disk minimum. Conduktor Gateway, our Kafka proxy, needs 2 CPU cores and 4 GB RAM. Both require Kafka 2.5.0 or higher. **Is internet access required?** No, you can run Conduktor in air-gapped environments. The Docker images work without internet access. **What databases does Conduktor need?** An external PostgreSQL 13+ database for configuration, users, and permissions. For monitoring, you can optionally configure S3, GCS, or Azure blob storage. **How long does setup take?** The quick-start command deploys a complete stack in under 5 minutes. Production deployments typically take 1-2 hours depending on your infrastructure. **Is there a free tier?** Yes, Conduktor offers a free tier for individuals and small teams. See [pricing](https://www.conduktor.io/pricing) for details. **Which Kafka distributions are supported?** Conduktor works with any Kafka 2.5.0+, including Confluent Platform, Amazon MSK, Redpanda, Aiven, and self-managed Apache Kafka. Last updated: June 2026 Source: https://www.conduktor.io/get-started --- # Kafka Glossary | Complete Guide to Streaming Terms Kafka Glossary 185 deep-dives on Kafka, streaming, security, schemas, lakehouse, and AI. Search or scroll the A–Z list below. / Want hands-on learning? Try [Kafkademy](https://www.conduktor.io/kafka) for tutorials and practice. | Concept | What it means and why it matters | | --- | --- | | [Access Control for Streaming: Securing Kafka Topics and Consumer Groups](https://www.conduktor.io/glossary/access-control-for-streaming) | Implement fine-grained access control for Kafka and streaming platforms using ACLs, RBAC patterns, and enterprise authorization systems. | | [Agentic AI Pipelines: Streaming Data for Autonomous Agents](https://www.conduktor.io/glossary/agentic-ai-pipelines) | Build streaming data pipelines that power autonomous AI agents with real-time context, fresh vector embeddings, and robust governance. | | [AI Discovery and Monitoring: Tracking AI Assets Across the Enterprise](https://www.conduktor.io/glossary/ai-discovery-and-monitoring) | Build comprehensive visibility into AI models, pipelines, and data flows for effective governance, regulatory compliance, and MLOps operations. | | [Amazon MSK: Managed Kafka on AWS](https://www.conduktor.io/glossary/amazon-msk-managed-kafka-on-aws) | Amazon MSK simplifies Apache Kafka operations on AWS with fully managed clusters, automatic scaling, and seamless AWS service integrations. | | [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg) | Apache Iceberg delivers ACID transactions and schema evolution for data lakes, powering modern lakehouse architectures at petabyte scale. | | [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) | Apache Kafka powers real-time data systems with distributed event streaming, enabling high-throughput messaging and durable logs. | | [API Gateway Patterns for Data Platforms](https://www.conduktor.io/glossary/api-gateway-patterns-for-data-platforms) | Explore API gateway patterns for data platforms including routing, protocol translation, security, and Kafka integration strategies. | | [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) | Implement comprehensive audit logging in Kafka and streaming platforms to meet compliance requirements and enable security forensics. | | [Automated Data Quality Testing: A Practical Guide for Modern Data Pipelines](https://www.conduktor.io/glossary/automated-data-quality-testing) | Implement automated data quality testing for batch and streaming pipelines using validation frameworks and continuous quality monitoring. | | [Avro vs Protobuf vs JSON Schema: Kafka Serialization Compared (2026)](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema) | Compare Avro, Protobuf, and JSON Schema for data serialization, examining tradeoffs in performance, schema evolution, and compatibility. | | [Azure Event Hubs and Kafka Compatibility](https://www.conduktor.io/glossary/azure-event-hubs-and-kafka-compatibility) | Azure Event Hubs provides Kafka protocol compatibility, enabling seamless cloud migration and hybrid streaming architectures on Azure. | | [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) | Handle backpressure in streaming systems using throttling, buffering, and elastic scaling strategies for Kafka and Flink pipelines. | | [Building a Business Glossary for Data Governance](https://www.conduktor.io/glossary/building-a-business-glossary-for-data-governance) | Establish a comprehensive business glossary that bridges business terminology and technical data assets for unified data governance. | | [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) | Design a comprehensive data quality framework with validation rules, quality scorecards, and real-time monitoring for streaming data. | | [Building a Modern Data Lake on Cloud Storage](https://www.conduktor.io/glossary/building-a-modern-data-lake-on-cloud-storage) | Architect scalable data lakes on AWS S3, Azure Storage, and GCS with zone-based organization, Iceberg tables, and streaming integration. | | [Building and Managing Data Products](https://www.conduktor.io/glossary/building-and-managing-data-products) | Design and manage reusable data products with clear ownership, quality guarantees, and streaming integration using Kafka and Flink. | | [Building Recommendation Systems with Streaming Data](https://www.conduktor.io/glossary/building-recommendation-systems-with-streaming-data) | Build real-time recommendation systems using Kafka and Flink with feature stores, streaming data pipelines, and vector similarity search. | | [CDC for Microservices: Event-Driven Architectures](https://www.conduktor.io/glossary/cdc-for-microservices-event-driven-architectures) | Enable event-driven microservices with CDC, supporting CQRS, event sourcing, and reliable integration using Kafka and the outbox pattern. | | [CDC for Real-Time Data Warehousing](https://www.conduktor.io/glossary/cdc-for-real-time-data-warehousing) | Enable real-time data warehousing with CDC pipelines using Kafka, Debezium, and Flink for incremental data lake and warehouse loading. | | [Chaos Engineering for Kafka](https://www.conduktor.io/glossary/chaos-engineering-for-kafka) | Chaos engineering for Kafka: inject broker failures, latency spikes, and message corruption via proxy. Test resilience without infrastructure access or data ... | | [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems) | Apply chaos engineering to Kafka and Flink using failure injection, resilience testing, and automated experiments for fault tolerance. | | [CI/CD Best Practices for Streaming Applications](https://www.conduktor.io/glossary/cicd-best-practices-for-streaming-applications) | Implement CI/CD for Kafka and Flink applications with testing strategies, zero-downtime deployments, and state management best practices. | | [Clickstream Analytics with Kafka](https://www.conduktor.io/glossary/clickstream-analytics-with-kafka) | Build real-time clickstream analytics with Kafka for user behavior tracking, session analysis, and personalized experiences at scale. | | [CQRS and Event Sourcing with Kafka](https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka) | Implement CQRS and Event Sourcing with Kafka for scalable, auditable systems using event stores, projections, and materialized views. | | [Cross-AZ Traffic in Streaming: Managing Costs and Latency](https://www.conduktor.io/glossary/cross-az-traffic-streaming) | Optimize cross-AZ traffic costs in Kafka deployments using rack awareness, follower fetching, and tiered storage for cloud streaming. | | [Cross-Organization Data Sharing Patterns](https://www.conduktor.io/glossary/cross-organization-data-sharing-patterns) | Share data across organizations using event-driven patterns, Kafka multi-tenancy, Delta Sharing, and secure API gateways with governance. | | [Crypto Shredding for Kafka: GDPR-Compliant Data Deletion](https://www.conduktor.io/glossary/crypto-shredding-for-kafka) | Crypto shredding for Kafka: delete encryption keys to make data permanently unreadable. GDPR-compliant deletion without modifying immutable logs. | | [Dark Data Tax: The Hidden Costs of Unused Data](https://www.conduktor.io/glossary/dark-data-tax) | Identify and eliminate dark data costs in streaming platforms through usage tracking, lifecycle policies, and automated governance. | | [Data Access Control: RBAC and ABAC](https://www.conduktor.io/glossary/data-access-control-rbac-and-abac) | Implement RBAC and ABAC access control for Kafka using ACLs, OPA policies, and OAuth2 to secure streaming data with fine-grained permissions. | | [Data Classification and Tagging Strategies](https://www.conduktor.io/glossary/data-classification-and-tagging-strategies) | Classify streaming data using Kafka headers, Schema Registry metadata, and automated PII detection for compliance and security governance. | | [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) | Establish data contracts using Schema Registry, quality rules, and compatibility modes to prevent pipeline failures and enable safe evolution. | | [Data Drift in Streaming: Detecting and Managing Unexpected Changes](https://www.conduktor.io/glossary/data-drift-in-streaming) | Detect and manage data drift in streaming pipelines using statistical tests, schema validation, and automated monitoring for ML models. | | [Data Freshness Monitoring: SLA Management](https://www.conduktor.io/glossary/data-freshness-monitoring-sla-management) | Monitor data freshness and manage SLAs using consumer lag tracking, heartbeat metrics, and automated alerting for pipeline reliability. | | [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) | Define governance roles from data stewards to executives with clear responsibilities for streaming platforms, AI governance, and federated control. | | [Data Incident Management and Root Cause Analysis](https://www.conduktor.io/glossary/data-incident-management-and-root-cause-analysis) | Minimize MTTR with incident response strategies, Five Whys analysis, and automated monitoring for streaming and batch data pipelines. | | [Data Lake Zones: Bronze, Silver, Gold Architecture](https://www.conduktor.io/glossary/data-lake-zones-bronze-silver-gold-architecture) | Implement Medallion Architecture with Bronze, Silver, and Gold layers using Delta Lake and Iceberg for progressive data refinement. | | [Data Lineage: Tracking Data From Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption) | Track data lineage from source to consumption using OpenLineage, DataHub, and automated metadata collection for compliance and impact analysis. | | [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming) | Protect sensitive data in real-time Kafka streams using field masking, tokenization, and k-anonymity techniques with minimal latency impact. | | [Data Mesh Principles and Implementation](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation) | Implement Data Mesh with domain-owned data products, self-serve platforms, and federated governance using Kafka and streaming infrastructure. | | [Data Obesity: When Data Infrastructure Becomes Bloated](https://www.conduktor.io/glossary/data-obesity) | Combat data obesity in streaming platforms through payload optimization, tiered storage, and aggressive retention policies for cost reduction. | | [Data Pipeline Orchestration with Streaming](https://www.conduktor.io/glossary/data-pipeline-orchestration-with-streaming) | Orchestrate streaming pipelines using Kubernetes operators, Airflow for infrastructure management, and Kafka-native coordination patterns. | | [Data Product Governance: Building Trustworthy Data Assets](https://www.conduktor.io/glossary/data-product-governance) | Govern data products with clear ownership, quality SLAs, discoverability, and lifecycle management using contracts and governance platforms. | | [Data Quality Incidents: Detection, Response, and Prevention](https://www.conduktor.io/glossary/data-quality-incidents) | Manage data quality incidents with automated detection, severity classification, and prevention using contracts and governance policies. | | [Data Quality vs Data Observability: Key Differences](https://www.conduktor.io/glossary/data-quality-vs-data-observability-key-differences) | Compare data quality testing and observability monitoring. Build reliable pipelines using complementary approaches for catching known and unknown issues. | | [Data Versioning in Streaming: Managing Event History](https://www.conduktor.io/glossary/data-versioning-in-streaming) | Manage schema evolution and event versioning in Kafka and Flink. Maintain backward compatibility across real-time data platform deployments. | | [DataOps for Streaming: Operational Excellence in Real-Time Systems](https://www.conduktor.io/glossary/dataops-for-streaming) | Apply CI/CD, automated testing, and infrastructure as code to streaming platforms. Build reliable Kafka operations with DataOps principles. | | [dbt Tests and Data Quality Checks: Building Reliable Data Pipelines](https://www.conduktor.io/glossary/dbt-tests-and-data-quality-checks) | Implement comprehensive data quality checks with dbt generic and singular tests. Validate transformations using unit tests and streaming integration. | | [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling) | Handle failed messages systematically with Dead Letter Queues. Implement DLQ patterns in Kafka for resilient error handling without blocking pipelines. | | [Debezium CDC with Kafka: Setup & Examples](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) | Implement CDC with Debezium and Kafka: connector setup, transaction-log capture, error handling and production patterns for MySQL, Postgres and MongoDB. | | [Debezium vs Airbyte: CDC Approaches](https://www.conduktor.io/glossary/debezium-vs-airbyte) | Debezium vs Airbyte: log-based CDC via Kafka Connect vs polling-based and log-based connectors, streaming vs batch/micro-batch — when each fits your data int... | | [Delta Lake Deletion Vectors: Efficient Row-Level Deletes](https://www.conduktor.io/glossary/delta-lake-deletion-vectors-efficient-row-level-deletes) | Enable fast row-level deletes without rewriting files using Delta Lake deletion vectors. Improve performance and reduce storage costs dramatically. | | [Delta Lake Liquid Clustering: Modern Partitioning](https://www.conduktor.io/glossary/delta-lake-liquid-clustering-modern-partitioning) | Replace traditional partitioning with Delta Lake liquid clustering for better query performance and automatic maintenance without manual tuning. | | [Delta Lake Transaction Log: How It Works](https://www.conduktor.io/glossary/delta-lake-transaction-log-how-it-works) | Understand Delta Lake's transaction log mechanism enabling ACID transactions, time travel, and schema evolution for data lakes. | | [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters) | Implement backup, replication, and failover strategies for Kafka clusters. Plan RPO/RTO requirements for mission-critical streaming systems. | | [Diskless Kafka: Direct-to-S3 Topics](https://www.conduktor.io/glossary/diskless-kafka) | Diskless Kafka (KIP-1150) writes records straight to object storage, not replicated local disk. Covers leaderless writes and the batch coordinator. | | [Distributed Tracing for Kafka Applications](https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications) | Implement distributed tracing in Kafka applications using OpenTelemetry and Jaeger. Debug and monitor event-driven systems with end-to-end visibility. | | [E-Commerce Streaming Architecture Patterns](https://www.conduktor.io/glossary/e-commerce-streaming-architecture-patterns) | Build real-time e-commerce with streaming patterns for order processing, inventory management, fraud detection, and personalized recommendations. | | [Encryption at Rest and In Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka) | Configure TLS encryption for data in transit and volume encryption at rest in Kafka. Secure streaming data to meet compliance requirements. | | [Event Sourcing Patterns with Kafka](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka) | Implement event sourcing patterns with Kafka for audit trails and state reconstruction. Build immutable event stores for reliable system state management. | | [Event Streams: The Foundation of Real-Time Architectures](https://www.conduktor.io/glossary/event-stream-fundamentals) | Master event stream fundamentals including topics, partitions, offsets, and consumer groups. Build reliable Kafka streaming applications from scratch. | | [Event Time and Watermarks in Flink](https://www.conduktor.io/glossary/event-time-and-watermarks-in-flink) | Handle event time and watermarks in Apache Flink for accurate stream processing. Manage out-of-order events and late-arriving data effectively. | | [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture) | Build scalable, loosely-coupled systems with event-driven architecture. Apply EDA patterns and best practices using Kafka and event streaming. | | [Event-Driven Microservices Architecture](https://www.conduktor.io/glossary/event-driven-microservices-architecture) | Design microservices with event-driven architecture using Kafka. Build resilient, scalable systems with asynchronous messaging and event patterns. | | [Fine-Grained Access Control for Kafka](https://www.conduktor.io/glossary/fine-grained-access-control-for-kafka) | Implement fine-grained access control for Kafka beyond topic-level ACLs. Learn field-level filtering, dynamic policies, and proxy-based enforcement. | | [Flink DataStream API: Building Streaming Applications](https://www.conduktor.io/glossary/flink-datastream-api-building-streaming-applications) | Build streaming applications with Flink DataStream API. Process real-time data using transformations, windows, and stateful operators effectively. | | [Flink SQL and Table API for Stream Processing](https://www.conduktor.io/glossary/flink-sql-and-table-api-for-stream-processing) | Process streaming data with Flink SQL and Table API. Write SQL queries for real-time analytics and continuous table transformations. | | [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing) | Manage stateful stream processing with Flink's state backends and checkpointing. Enable fault tolerance and exactly-once processing guarantees. | | [Flink vs Kafka Streams: When to Choose](https://www.conduktor.io/glossary/flink-vs-kafka-streams) | Flink vs Kafka Streams: cluster runtime vs embedded library, checkpointing vs RocksDB changelog, event-time semantics, operational complexity — decide based ... | | [GDPR Compliance for Data Teams: Navigating Privacy in Modern Data Architectures](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams) | Implement GDPR compliance in streaming architectures with consent management, data deletion, encryption, and data subject rights for data teams. | | [Great Expectations: Data Testing Framework](https://www.conduktor.io/glossary/great-expectations-data-testing-framework) | Implement robust data quality testing with Great Expectations framework. Validate batch and streaming data using expectations and checkpoints. | | [Handling Late-Arriving Data in Streaming](https://www.conduktor.io/glossary/handling-late-arriving-data-in-streaming) | Handle late-arriving data in stream processing with watermarks, allowed lateness, and side outputs. Manage out-of-order events in Kafka and Flink. | | [Healthcare Data Streaming Use Cases](https://www.conduktor.io/glossary/healthcare-data-streaming-use-cases) | Transform healthcare with real-time data streaming for patient monitoring, device integration, clinical decision support, and secure data exchange. | | [High Value Assets: Protecting Critical Data in Streaming](https://www.conduktor.io/glossary/high-value-assets-streaming) | Identify, classify, and protect high-value data assets in streaming systems. Implement risk-based security controls and governance workflows. | | [Iceberg Catalog Management: REST, Hive, Glue, and Nessie](https://www.conduktor.io/glossary/iceberg-catalog-management-hive-glue-and-nessie) | Manage Apache Iceberg catalogs using Hive Metastore, AWS Glue, and Nessie. Configure catalog backends for lakehouse metadata management. | | [Iceberg Table Architecture: Metadata and Snapshots](https://www.conduktor.io/glossary/iceberg-table-architecture-metadata-and-snapshots) | Understand Apache Iceberg table architecture with metadata layers and snapshot isolation. Enable time travel and ACID transactions for data lakes. | | [Infrastructure as Code for Kafka Deployments](https://www.conduktor.io/glossary/infrastructure-as-code-for-kafka-deployments) | Manage Kafka infrastructure as code with Terraform, Kubernetes operators, and GitOps. Automate cluster provisioning and configuration management. | | [Integrating LLMs with Streaming Platforms](https://www.conduktor.io/glossary/integrating-llms-with-streaming-platforms) | Integrate Large Language Models with streaming platforms for real-time AI applications. Build LLM-powered event processing and enrichment pipelines. | | [Introduction to Confluent Cloud](https://www.conduktor.io/glossary/introduction-to-confluent-cloud) | Get started with Confluent Cloud for fully managed Kafka. Provision clusters, configure connectors, and build streaming applications in the cloud. | | [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams) | Build stream processing applications with Kafka Streams library. Process, transform, and aggregate real-time data using stateful operations. | | [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture) | Combine data lake flexibility with data warehouse performance using lakehouse architecture. Unify batch and streaming analytics on open table formats. | | [IoT Data Streaming Architectures](https://www.conduktor.io/glossary/iot-data-streaming-architectures) | Design IoT data streaming architectures for device ingestion, edge processing, and real-time analytics. Handle millions of concurrent device connections. | | [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) | Implement Kafka ACLs and authorization patterns for secure topic access. Configure fine-grained permissions and role-based access control. | | [Kafka Admin Operations and Maintenance](https://www.conduktor.io/glossary/kafka-admin-operations-and-maintenance) | Perform Kafka admin operations for cluster management, topic configuration, partition rebalancing, and performance tuning. Maintain production clusters. | | [Kafka Architecture Diagram & Components](https://www.conduktor.io/glossary/kafka-architecture-diagram) | Kafka architecture explained with diagrams: topics, partitions, brokers, producers, consumers and KRaft controllers, and how data flows through a cluster. | | [Kafka Authentication: SASL, SSL, and OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth) | Configure Kafka authentication with SASL, SSL/TLS, and OAuth 2.0. Secure broker connections and enforce client identity verification. | | [Kafka Backup vs Replication: Key Differences](https://www.conduktor.io/glossary/kafka-backup-vs-replication) | A Kafka backup is a point-in-time, immutable copy that protects against deletion and corruption, unlike replication, which only guards hardware. | | [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) | Right-size Kafka clusters with throughput, storage, memory, and network calculations for optimized production-scale streaming deployments. | | [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) | Essential Kafka metrics for broker health, producer throughput, consumer lag tracking with tools and alerting strategies for reliability. | | [Kafka Cluster: Architecture & How It Works](https://www.conduktor.io/glossary/kafka-cluster) | A Kafka cluster is a group of brokers that store and replicate topic partitions. Learn cluster architecture, KRaft controllers, replication and scaling. | | [Kafka Connect Single Message Transforms](https://www.conduktor.io/glossary/kafka-connect-single-message-transforms) | Transform data in Kafka Connect pipelines using built-in and custom SMTs for field masking, routing, and format conversion without code. | | [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines) | Build reliable data pipelines with Kafka Connect source/sink connectors, configuration patterns, and scaling strategies for data integration. | | [Kafka Consumer Groups: How They Work](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) | Kafka consumer groups distribute topic partitions across consumers for parallel processing. Learn partition assignment, rebalancing, offset commits, and grou... | | [Kafka Consumer Lag: Definition & Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) | Kafka consumer lag is the per-partition gap between log-end offset and committed offset. Covers calculation, causes, monitoring, alerts, and remediation. | | [Kafka Data Governance: Definition & Primitives](https://www.conduktor.io/glossary/kafka-data-governance) | Kafka data governance is the layer of policies — ownership, schema, RBAC, encryption, audit, quality — that lets multiple teams use Kafka safely. | | [Kafka Exactly-Once: Producers + Transactions](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) | Exactly-once semantics in Kafka: how idempotent producers and transactional APIs eliminate duplicates in streaming pipelines. Covers EOS config, trade-offs, ... | | [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained) | Kafka log compaction retains latest values per key by removing old records. Configuration, use cases for changelog topics and caches. | | [Kafka MirrorMaker 2: Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication) | Replicate Kafka topics across clusters with MirrorMaker 2 for disaster recovery, multi-region deployment, and active-active architectures. | | [Kafka Partitioning: 5 Strategies Compared](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices) | Compare Kafka partitioning strategies: key-based, round-robin, sticky, custom, and co-partitioning. Includes hot partition detection, ordering trade-offs, an... | | [Kafka Performance Tuning: Config Cheatsheet](https://www.conduktor.io/glossary/kafka-performance-tuning-guide) | Optimize Kafka throughput and latency with producer batching, broker tuning, consumer configuration, and OS-level performance optimizations. | | [Kafka Producers](https://www.conduktor.io/glossary/kafka-producers) | Write records to Kafka topics with control over serialization, partitioning, delivery guarantees, batching, and exactly-once semantics. | | [Kafka Producers and Consumers](https://www.conduktor.io/glossary/kafka-producers-and-consumers) | Kafka producers write records with delivery guarantees while consumers read using offset tracking, consumer groups for parallel processing. | | [Kafka Queues: Share Groups Explained](https://www.conduktor.io/glossary/kafka-share-groups) | Kafka share groups (KIP-932) bring queue semantics to Kafka: many consumers share a partition with per-record acks, decoupled from partition count. | | [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability) | Kafka replication with in-sync replicas ensures durability and automatic failover. Configure replication factor, min.insync.replicas for reliability. | | [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) | Secure Kafka with authentication, authorization, TLS encryption, ACLs, and Zero Trust principles for production streaming infrastructure. | | [Kafka Streams vs Apache Flink: When to Use What](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink) | Compare Kafka Streams and Apache Flink architectures, operational complexity, state management, and choose the right stream processing framework. | | [Kafka to Iceberg: Topics as Tables](https://www.conduktor.io/glossary/kafka-to-iceberg-topics) | Kafka to Iceberg turns a topic into a queryable Apache Iceberg table. Covers topic-native zero-copy vs copy-based sinks, schema mapping, exactly-once. | | [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) | Design Kafka topics with naming conventions, partition counts, replication factors, retention policies, and schema evolution for scalable systems. | | [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive) | Kafka transactions enable exactly-once semantics with two-phase commit, transaction coordinator, and atomic multi-partition writes for critical data. | | [Kafka vs Postgres: When to Use Each](https://www.conduktor.io/glossary/kafka-vs-postgres) | Kafka vs Postgres: distributed log with replay and fan-out versus an ACID store used as a queue via SKIP LOCKED. Decision framework and how they combine. | | [Kafka vs Pulsar: Architecture Compared](https://www.conduktor.io/glossary/kafka-vs-pulsar) | Kafka vs Pulsar: partitioned log vs segmented BookKeeper storage, single vs multi-layer architecture, subscription models, multi-tenancy — trade-offs for pla... | | [Kafka vs RabbitMQ: Streams vs Queues](https://www.conduktor.io/glossary/kafka-vs-rabbitmq) | Kafka vs RabbitMQ: log-based streaming vs message queue, pull vs push delivery, consumer groups vs competing consumers — when to use each and why they're oft... | | [ksqlDB for Real-Time Data Processing](https://www.conduktor.io/glossary/ksqldb-for-real-time-data-processing) | Build real-time stream processing with ksqlDB using SQL for filtering, joins, aggregations, and materialized views on Kafka topics without code. | | [Log Aggregation with Kafka](https://www.conduktor.io/glossary/log-aggregation-with-kafka) | Centralize logs from distributed systems with Kafka for real-time analysis, multi-consumer patterns, and integration with observability platforms. | | [Log-Based vs Query-Based CDC: Comparison](https://www.conduktor.io/glossary/log-based-vs-query-based-cdc-comparison) | Compare log-based CDC capturing from transaction logs vs query-based CDC polling tables. Latency, completeness, and operational trade-offs. | | [Low-Latency Pipelines: Achieving Millisecond Response Times](https://www.conduktor.io/glossary/low-latency-pipeline) | Build low-latency streaming pipelines with Kafka, Flink using fast serialization, tuned batching, and optimized network configurations. | | [Maintaining Iceberg Tables: Compaction and Cleanup](https://www.conduktor.io/glossary/maintaining-iceberg-tables-compaction-and-cleanup) | Maintain Apache Iceberg tables with compaction for query performance, snapshot expiration, orphan file cleanup, and metadata optimization. | | [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka) | Choose Kafka message serialization formats: Avro, Protobuf, JSON Schema with Schema Registry for type safety, evolution, and performance. | | [Metadata Management: Technical vs Business Metadata](https://www.conduktor.io/glossary/metadata-management-technical-vs-business-metadata) | Technical metadata describes schema and lineage while business metadata defines ownership and semantics for data governance and discovery. | | [Micro-Batching: Near-Real-Time Stream Processing](https://www.conduktor.io/glossary/micro-batching-streaming) | Micro-batching processes events in small time windows combining batch efficiency with near real-time latency for stream processing frameworks. | | [Migrating to Apache Iceberg from Hive or Parquet](https://www.conduktor.io/glossary/migrating-to-apache-iceberg-from-hive-or-parquet) | Migrate from Hive or Parquet to Apache Iceberg for ACID transactions, time travel, schema evolution with in-place or dual-write strategies. | | [Model Drift in Streaming: When ML Models Degrade in Real-Time](https://www.conduktor.io/glossary/model-drift-in-streaming) | Detect ML model drift in streaming pipelines by monitoring prediction accuracy, feature distribution, and concept drift for model retraining. | | [mTLS for Kafka: Mutual Authentication in Streaming](https://www.conduktor.io/glossary/mtls-for-kafka) | Implement mutual TLS authentication in Kafka using client certificates for strong two-way authentication without password management complexity. | | [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments) | Isolate tenants in shared Kafka clusters using topics, ACLs, quotas, and Virtual Clusters for secure, scalable multi-tenant platforms. | | [NewSQL Databases: Distributed SQL for Real-Time Applications](https://www.conduktor.io/glossary/newsql-databases-streaming) | NewSQL databases like CockroachDB and TiDB provide SQL with ACID transactions, horizontal scalability for real-time streaming workloads. | | [NoSQL Databases for Real-Time Streaming: Patterns and Integration](https://www.conduktor.io/glossary/nosql-databases-real-time) | Choose NoSQL databases like Cassandra, MongoDB, DynamoDB for low-latency writes, flexible schemas in real-time streaming applications. | | [On-Prem vs Hybrid Streaming: Multi-Environment Architecture Patterns](https://www.conduktor.io/glossary/on-prem-hybrid-streaming) | Deploy hybrid streaming architectures across on-premise and cloud with Kafka MirrorMaker, VPN connectivity, and multi-region replication. | | [Optimizing Delta Tables: OPTIMIZE and Z-ORDER](https://www.conduktor.io/glossary/optimizing-delta-tables-optimize-and-z-order) | Optimize Delta Lake tables with OPTIMIZE for compaction and Z-ORDER for data clustering to improve query performance and reduce storage costs. | | [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing) | Implement outbox pattern for reliable event publishing from databases to Kafka with transactional guarantees and CDC-based event sourcing. | | [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams) | Detect and mask PII in event streams using pattern matching, ML classifiers, and encryption at ingest for compliance and privacy protection. | | [PII Leakage Prevention: Protecting Personal Data in Streaming](https://www.conduktor.io/glossary/pii-leakage-prevention) | Prevent PII leakage in streaming data with data classification, field-level encryption, tokenization, and audit logging for compliance. | | [Policy Enforcement in Streaming: Automated Governance for Real-Time Data](https://www.conduktor.io/glossary/policy-enforcement-in-streaming) | Enforce data policies in streaming platforms with schema validation, ACLs, quotas, and automated governance rules for compliance and quality. | | [Quotas and Rate Limiting in Kafka](https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka) | Protect Kafka clusters with quotas limiting producer throughput, consumer bandwidth, and request rates per client ID for fair resource sharing. | | [Real-Time Analytics with Streaming Data](https://www.conduktor.io/glossary/real-time-analytics-with-streaming-data) | Real-time analytics with streaming data using Kafka, Flink, ksqlDB for aggregations, windowing, and low-latency dashboards on live events. | | [Real-Time Data Clean Room: Definition](https://www.conduktor.io/glossary/real-time-data-clean-room) | A real-time data clean room lets organizations collaborate on live data streams without exposing raw data. Covers architecture, use cases, and limits. | | [Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming) | Detect fraud in real-time with streaming analytics, rule engines, ML models on transaction patterns, and instant alerting for suspicious activity. | | [Real-Time Gaming Analytics with Streaming](https://www.conduktor.io/glossary/real-time-gaming-analytics-with-streaming) | Track player behavior, game events, and metrics in real-time with streaming analytics for matchmaking, leaderboards, and live optimization. | | [Real-Time ML Inference with Streaming Data](https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data) | Deploy ML models for real-time inference on streaming data with feature engineering, model serving, and online prediction in event-driven systems. | | [Real-Time ML Pipelines: Machine Learning on Streaming Data](https://www.conduktor.io/glossary/real-time-ml-pipelines) | Build ML systems that process streaming data with sub-second inference. Master feature engineering, online learning, and model serving patterns. | | [Real-Time Threat Detection: Security Monitoring for Streaming](https://www.conduktor.io/glossary/real-time-threat-detection) | Build threat detection for streaming platforms using anomaly detection, behavioral analysis, and SIEM integration to catch security breaches early. | | [Redpanda vs Kafka: Architecture, Trade-offs](https://www.conduktor.io/glossary/redpanda-vs-kafka) | Redpanda vs Kafka: thread-per-core vs multi-threaded JVM, Raft vs KRaft consensus, wire compatibility, BSL vs Apache 2.0 license — concrete trade-offs for pl... | | [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) | Deploy and manage Kafka on Kubernetes with StatefulSets, operators, and KRaft mode. Handle storage, networking, and scaling challenges in production. | | [Saga Pattern for Microservices Explained](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions) | The saga pattern keeps data consistent across microservices without locks. Compare choreography vs orchestration and handle compensation for failures. | | [Schema Evolution in Apache Iceberg](https://www.conduktor.io/glossary/schema-evolution-in-apache-iceberg) | Evolve Iceberg schemas without data rewrites. Add columns, rename fields, and promote types using column IDs and versioned metadata for lakehouses. | | [Schema Evolution: 8 Kafka Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) | Schema evolution in Kafka: backward, forward, and full compatibility modes explained with 8 best practices for safe schema changes in Avro, Protobuf, and JSO... | | [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) | Manage data schemas centrally to enforce compatibility rules, reduce message size with schema IDs, and govern evolution across producers and consumers. | | [Semantic Layer for Streaming: Business Meaning for Real-Time Data](https://www.conduktor.io/glossary/semantic-layer-for-streaming) | Apply semantic layers to streaming data. Provide business-friendly abstractions, unified metrics, and consistent definitions over technical event streams. | | [Session Windows in Stream Processing](https://www.conduktor.io/glossary/session-windows-in-stream-processing) | Group streaming events by activity patterns using session windows. Perfect for user analytics, IoT monitoring, and behavior-based fraud detection. | | [Shadow AI: Governing Unauthorized AI in the Enterprise](https://www.conduktor.io/glossary/shadow-ai-governance) | Detect and govern unauthorized AI models in your enterprise. Build frameworks to discover Shadow AI and enforce compliance before it becomes a risk. | | [SLAs for Streaming: Defining and Measuring Real-Time Guarantees](https://www.conduktor.io/glossary/sla-for-streaming) | Define and enforce SLAs for streaming platforms. Set targets for latency, throughput, availability, and durability with automated monitoring. | | [State Stores in Kafka Streams](https://www.conduktor.io/glossary/state-stores-in-kafka-streams) | Master state stores in Kafka Streams for aggregations, joins, and windowing. Handle fault tolerance, recovery, and RocksDB backend configuration. | | [Strangler Fig Pattern with Event Streaming](https://www.conduktor.io/glossary/strangler-fig-pattern-with-event-streaming) | Migrate legacy systems incrementally using the Strangler Fig Pattern with event streaming. Replace monoliths with microservices without downtime. | | [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns) | Combine and enrich real-time streams with joins. Master stream-to-stream, stream-to-table, and temporal joins in Kafka Streams and Flink. | | [Streaming Audit Logs: Traceability and Compliance for Real-Time Systems](https://www.conduktor.io/glossary/streaming-audit-logs) | Implement audit logging for Kafka to track all admin actions, data access, and configuration changes for compliance and security investigations. | | [Streaming Data in Financial Services](https://www.conduktor.io/glossary/streaming-data-in-financial-services) | Enable fraud detection, payment processing, and algorithmic trading with real-time streaming. Meet regulatory compliance in financial services. | | [Streaming Data Pipeline](https://www.conduktor.io/glossary/streaming-data-pipeline) | Build streaming pipelines with five core components: sources, ingestion, brokers, processing, and sinks for continuous real-time data flows. | | [Streaming Data Products](https://www.conduktor.io/glossary/streaming-data-product-asset) | Apply product thinking to event streams. Create discoverable, well-governed data products with clear ownership, quality standards, and SLAs. | | [Streaming ETL vs Traditional ETL](https://www.conduktor.io/glossary/streaming-etl-vs-traditional-etl) | Compare batch and streaming ETL architectures. Choose the right approach based on latency needs, data volume, and processing complexity. | | [Streaming Ingestion to Lakehouse: Building Real-Time Data Pipelines](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse) | Connect streaming platforms to lakehouse architectures. Design ingestion pipelines for unified batch and real-time analytics with Iceberg. | | [Streaming Maturity Model: Assessing Your Real-Time Data Capabilities](https://www.conduktor.io/glossary/streaming-maturity-model) | Assess your streaming maturity from experimental to enterprise-grade. Build a roadmap to advance governance, reliability, and scalability. | | [Streaming to Lakehouse Tables: Delta Lake, Iceberg, Hudi, and Paimon](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables) | Write streaming data to Iceberg, Delta Lake, and Hudi tables. Get ACID guarantees, schema evolution, and real-time queryability for lakehouses. | | [Streaming Total Cost of Ownership: Understanding the Full Picture](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership) | Calculate true TCO for streaming infrastructure. Optimize compute, storage, networking, and operational costs beyond monthly cloud bills. | | [Strimzi: Kafka Operator for Kubernetes](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes) | Deploy Kafka on Kubernetes using the Strimzi operator. Automate upgrades, scaling, and configuration with declarative CNCF patterns. | | [Supply Chain Visibility with Real-Time Streaming](https://www.conduktor.io/glossary/supply-chain-visibility-with-real-time-streaming) | Track inventory, shipments, and demand in real-time with streaming platforms. Build end-to-end supply chain visibility with Kafka and Flink. | | [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications) | Test streaming apps with unit tests, integration tests, and chaos experiments. Handle time semantics, state, and out-of-order events reliably. | | [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka) | Reduce Kafka storage costs by 3-9x with tiered storage. Move older segments to S3 while keeping recent data local for fast access. | | [Time Travel with Apache Iceberg](https://www.conduktor.io/glossary/time-travel-with-apache-iceberg) | Query historical Iceberg snapshots with time travel. Support audit compliance, debug data issues, and recover from mistakes with SQL syntax. | | [Trust Zones: Isolating Sensitive Data in Streaming Architectures](https://www.conduktor.io/glossary/trust-zones-streaming) | Design security zones for streaming platforms. Protect sensitive data through network isolation, access control, and compliance boundaries. | | [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) | Eliminate ZooKeeper with Kafka's KRaft mode. Simplify operations and improve scalability using Raft-based consensus for metadata management. | | [Using Kafka Headers Effectively](https://www.conduktor.io/glossary/using-kafka-headers-effectively) | Attach metadata to Kafka messages with headers. Enable routing, distributed tracing, and observability without modifying message payloads. | | [Vector Databases and Streaming Architectures](https://www.conduktor.io/glossary/vector-databases-and-streaming-architectures) | Integrate vector databases with streaming platforms for real-time similarity search, recommendations, and semantic AI workflows at scale. | | [Vector Embeddings in Streaming: Real-Time AI with Fresh Context](https://www.conduktor.io/glossary/vector-embeddings-in-streaming) | Generate and manage vector embeddings in streaming pipelines. Power RAG systems, semantic search, and AI apps with real-time embeddings. | | [Watermarks and Triggers in Stream Processing](https://www.conduktor.io/glossary/watermarks-and-triggers-in-stream-processing) | Master watermarks for event time tracking and triggers for result emission. Handle late data and timing in Flink and Kafka Streams correctly. | | [What is a Data Catalog? Modern Data Discovery](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery) | Enable data discovery with catalogs that index assets across databases, lakes, and streams. Help teams find, understand, and trust data. | | [What is a Kafka Broker? Role & Architecture](https://www.conduktor.io/glossary/kafka-brokers-explained) | Kafka brokers store partitions and serve client requests. Learn how brokers form clusters, elect leaders, replicate data and manage metadata with KRaft. | | [What is a Kafka Partition? How They Work](https://www.conduktor.io/glossary/kafka-partitions-explained) | Kafka partitions split topics into ordered, parallel logs stored across brokers. Learn how they enable scalability, ordering guarantees, and fault tolerance. | | [What is a Kafka Topic? Definition & Examples](https://www.conduktor.io/glossary/kafka-topics-explained) | Kafka topics are named, append-only logs where producers publish and consumers read records. Learn structure, retention, partitions, and naming conventions. | | [What is Apache Flink? Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) | Process streams with Apache Flink's stateful engine. Master exactly-once semantics, event time, and Kafka integration for real-time apps. | | [What is Change Data Capture? CDC Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) | Capture database changes in real-time with CDC. Stream INSERT, UPDATE, DELETE events using Debezium for data synchronization and analytics. | | [What is Data Observability? The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars) | Monitor data health with five observability pillars: freshness, volume, schema, distribution, and lineage. Detect and resolve quality issues fast. | | [What is Real-Time Data Streaming?](https://www.conduktor.io/glossary/what-is-real-time-data-streaming) | Build real-time data architectures with streaming fundamentals. Master event-driven patterns, Kafka, Flink, and continuous data processing. | | [Windowing in Apache Flink: Tumbling, Sliding, and Session Windows](https://www.conduktor.io/glossary/windowing-in-apache-flink-tumbling-sliding-and-session-windows) | Master Flink windowing with tumbling, sliding, and session windows. Aggregate streams by time with practical examples and best practices. | | [Zero Trust Architecture for Kafka](https://www.conduktor.io/glossary/zero-trust-architecture-for-kafka) | Zero trust for Kafka: continuous verification, mTLS, proxy-based authorization, and audit logging. Protect streaming data without implicit network trust. | | [Zero Trust for Streaming: Security Without Implicit Trust](https://www.conduktor.io/glossary/zero-trust-for-streaming) | Implement zero trust security for Kafka with continuous authentication, authorization, and encryption. Never trust, always verify access. | | [Zero-Copy Data Sharing: Eliminating Duplication in Modern Architectures](https://www.conduktor.io/glossary/zero-copy-data-sharing) | Share data without duplication using zero-copy patterns. Reduce storage costs and enable collaboration across streaming and lakehouse systems. | | [ZooKeeper to KRaft Migration](https://www.conduktor.io/glossary/zookeeper-to-kraft-migration) | Migrate Kafka from ZooKeeper to KRaft mode. Follow best practices for zero-downtime transition to Kafka's native consensus protocol. | No term matches that search. Try a broader query. **Using Conduktor in production?** Real reviews from engineers help others evaluate the platform. Two minutes of your time goes a long way. [Leave a G2 review →](https://www.g2.com/products/conduktor-for-apache-kafka/take_survey) Source: https://www.conduktor.io/glossary --- # Kafka on autopilot. Build more, run less. Conduktor is a Kafka Operations Platform bringing governance, security, and self-service to the clusters you already run. Every team and AI agent builds on it, while operations run themselves. [Install Console](https://www.conduktor.io/get-started?src=home)A Kafka UI for every team: visibility and self-service, plus Insights into the risk, cost, and governance of every cluster. [Install Gateway](https://www.conduktor.io/gateway/community-edition)A Kafka proxy that reaches clusters across VPCs and clouds, and enforces encryption, data masking, and policies on the wire, with no app changes. [Book a Demo](https://www.conduktor.io/contact/demo?src=home) Trusted by One control layer between your infrastructure and the teams that build on it. Conduktor sits in front of the infrastructure you already run, so every event is governed, secured, and validated before anything downstream uses it. It adds control without replacing what you have. ## How Conduktor Does It Conduktor delivers that control layer through two products: Console gives developers visibility and velocity, Gateway gives platform teams control on the wire, across any Kafka provider and infrastructure. [See full architecture →](https://www.conduktor.io/architecture) ## Conduktor Console ### Multi-cluster Kafka UI Enterprise UI to manage hundreds of clusters. Topics, schemas, connectors, producers, and consumers across Confluent, AWS MSK, Redpanda, or self-managed. ### Browse & Debug Messages Browse, filter, and replay messages in real-time. Troubleshoot and auto-decode Avro, Protobuf, and JSON Schema. ### Monitoring & Alerts Track hot topics, fan-out, dead streams, and consumer lag. Alerts to Slack, PagerDuty, Prometheus, and webhooks. ### Security & Compliance SSO (OIDC-based and LDAP), granular RBAC, role-based access, PII masking, and audit trails. ### Health & Cost Insights Health optimization for partitions, replication, naming, and configs. Cost attribution, data quality, and team accountability. ### Terraform, API & GitOps Terraform provider, REST API, GitOps workflows, and CI/CD integration. ```terraform terraform { required_providers { conduktor = { source = "conduktor/conduktor" version = "~> 1.0" } } } provider "conduktor" { console_url = "http://localhost:8080" api_token = var.conduktor_api_token } resource "conduktor_group" "project_a" { name = "project-a" display_name = "Project A" permissions { resource_type = "TOPIC" name = "project-a-*" pattern_type = "PREFIXED" permissions = ["topicConsume", "topicProduce"] } } ``` ## Conduktor Gateway ### Routing & Authentication Centralize identity with OIDC across all clusters. Route clients transparently without changing application code. ```yaml # Gateway Configuration gateway: environment: GATEWAY_SECURITY_MODE: GATEWAY_MANAGED GATEWAY_SECURITY_PROTOCOL: SASL_PLAINTEXT GATEWAY_OAUTH_JWKS_URL: "https://your-idp.com/.well-known/jwks.json" GATEWAY_OAUTH_EXPECTED_ISSUER: "https://your-idp.com" # Map OIDC identities to Service Accounts apiVersion: gateway/v2 kind: GatewayServiceAccount metadata: name: my-application spec: type: EXTERNAL externalNames: - "oauth-subject-id-from-token" ``` ### Resilience & Migration Switch clusters, failover traffic, and inject chaos, all at the proxy layer. Zero client changes required. ```yaml # Gateway cluster configuration config: main: bootstrap.servers: kafka-primary:9092 security.protocol: SASL_SSL sasl.mechanism: PLAIN failover: bootstrap.servers: kafka-secondary:9092 gateway.roles: failover # Switch from main → failover curl -X POST 'http://localhost:8888/gateway/v2/cluster-switching' \ -H 'Content-Type: application/json' \ -d '{"fromPhysicalCluster": "main", "toPhysicalCluster": "failover"}' ``` ### Multi-tenancy Virtual clusters, topic concentration, and aliasing. Isolate teams and workloads on shared infrastructure. ```yaml apiVersion: gateway/v2 kind: VirtualCluster metadata: name: payments-team spec: type: Standard aclEnabled: true superUsers: - payments-admin --- apiVersion: gateway/v2 kind: VirtualCluster metadata: name: orders-team spec: type: Standard aclEnabled: true superUsers: - orders-admin ``` ### Performance SQL queries on topics, message caching, and large payload offloading. Handle load without over-provisioning. ```yaml apiVersion: gateway/v2 kind: Interceptor metadata: name: sql-filter-adults spec: pluginClass: io.conduktor.gateway.interceptor.VirtualSqlTopicPlugin priority: 100 config: virtualTopic: customers-adult statement: | SELECT firstName, lastName, email FROM customers WHERE age >= 18 AND country = 'US' ``` ### Governance & Data Quality Config guardrails, rate limiting, and schema validation. Enforce standards before bad data hits Kafka. ### Security & Data Sharing Field-level encryption, key management, and partner tenants. Protect data while enabling controlled access. ```yaml apiVersion: gateway/v2 kind: Interceptor metadata: name: encrypt-pii-fields spec: pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin priority: 100 config: topic: "customers.*" schemaRegistryConfig: host: http://schema-registry:8081 kmsConfig: vault: uri: http://vault:8200 token: ${VAULT_TOKEN} recordValue: fields: - fieldName: ssn keySecretId: vault-kms://vault:8200/transit/keys/pii-key algorithm: AES256_GCM - fieldName: email keySecretId: vault-kms://vault:8200/transit/keys/email-key algorithm: AES256_GCM ``` What you have. What Conduktor adds. Your provider gives you Kafka to stream data. Conduktor adds what you need to maximize your platform: a set of controls across all of them, enforced on the wire. What you already run Your streaming platform Confluent, Amazon MSK, Aiven, or open-source Kafka: the transport and the security basics you've already invested in. **Kafka protocol**Start producing and consuming in seconds with standard clients. **Availability SLA**The uptime guarantee your provider commits to and operates for you. **Kafka ecosystem**Connectors, Schema Registry, stream processing, and replication. **RBAC & logs**The access control and audit logs your provider ships, scoped to that one platform. **Encryption in transit & at rest**TLS on the wire and disk encryption your provider manages for you. **Your operational expertise**The runbooks, SLAs, and provider relationship your team already owns and knows. What Conduktor adds, on top The cross-provider control layer A control plane and a proxy in front of your existing clusters. Nothing to migrate, no client changes, but you now have control over the wire: **One layer across every provider**Unify authentication, authorization, and controls over Confluent, MSK, Aiven, Redpanda, and open-source: one model to manage, not a different one per cluster. **Encryption & masking on the wire**Payload and field-level encryption with your own KMS keys, enforced at the proxy with no client changes and nothing to bypass. **Guardrails for self-service**Let teams provision on their own, within the limits you set: config, naming, and data-quality rules that catch misconfigured topics and bad data while reducing cost. **Many tenants on one cluster**Virtual clusters isolate teams and workloads on the infrastructure you already run, so you add tenants instead of clusters and the cost that comes with them. **Governed access for AI**AI, agents, and MCP discover and query every Kafka cluster the same way, each inheriting the user's exact RBAC. Now Point AI at Your Kafka Smarter AI, because you hold the context. Safer AI, because you hold the permissions. AI on Kafka is only useful if it's smart and only adoptable if it's safe. Conduktor is the self-hosted layer that delivers both, without ever taking ownership of your data. You hold the contextOwnership, schemas, lineage, policies, apps, and monitoring across every cluster. AI knows who owns what, what's sensitive, what changed, and what's allowed.You troubleshoot fasterMetadata, apps, infrastructure, and monitoring sit in one place, so an LLM correlates the right signals across lag, broken consumers, and schema breaks instead of guessing.You hold the permissionsEvery AI, MCP, and CLI action inherits the user's exact RBAC. Decide read-only versus read-write, per cluster and per team. AI runs inside the model you already trust.One surface, two toolsMCP runs read-only inside Console for safe exploration. The CLI and Skills give coding agents a larger, governed surface when it's time to act. [Explore Conduktor MCP](https://www.conduktor.io/mcp) [Set up with an AI agent](https://www.conduktor.io/get-started?src=home) Explore How Conduktor Helps Your Industry Platform teams use Conduktor to streamline Kafka operations and accelerate delivery. Explore our [customer stories](https://www.conduktor.io/customer-stories) or dive into the [blog](https://www.conduktor.io/blog). Financial Services Bitvavo achieved DORA, MiCA, and GDPR compliance for 1.5M+ users with Conduktor’s RBAC, data masking, and audit trails. [Read Story ›](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) 1.5M+USERS COMPLIANTDORACOMPLIANCE FRAMEWORK Transport & Logistics Swiss Post scaled Kafka to 800+ users and 5× more applications with governance, RBAC, and self-service. [Read Story ›](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) 800+MORE USERS5×MORE APPLICATIONS Agriculture & IoT 70% faster provisioning with Conduktor and Amazon MSK for IoT smart farming at scale. [Read Story ›](https://www.conduktor.io/customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk) 10×KAFKA UTILIZATION70%FASTER PROVISIONING # See what your existing Kafka can do with Conduktor on top Bring the platform you already run. We'll show you the governance, security, and AI-readiness you can unlock on top of it — without re-platforming a thing. [Talk to Us](https://www.conduktor.io/contact/demo?src=home) [Install Now](https://www.conduktor.io/get-started?src=home) Source: https://www.conduktor.io/ --- # Advanced Kafka Consumer tutorials with Java *Master advanced consumer patterns for production use cases* These tutorials cover advanced consumer patterns that go beyond basic polling. Use them when you need fine-grained control over offset management, partition assignment, or threading. **What you'll learn:** - How to use rebalance listeners for manual offset commits - How to use seek() and assign() for specific partition/offset access - How to run consumers in separate threads > **When to use these patterns** > The basic consumer code covers 90% of use cases. Use these advanced patterns only when you have specific requirements around offset management, replay scenarios, or multi-threaded architectures. ## Advanced tutorials | Tutorial | Use case | |----------|----------| | [Rebalance Listener](https://www.conduktor.io/kafka/java-consumer-rebalance-listener) | Manual offset commits, external offset storage | | [Seek and Assign](https://www.conduktor.io/kafka/java-consumer-seek-and-assign) | Replay from specific offset, read specific partitions | | [Consumer in Threads](https://www.conduktor.io/kafka/java-consumer-in-threads) | Background polling, concurrent processing | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-advanced-kafka-consumer-with-java) helps you debug advanced consumer scenarios by showing partition assignments, committed offsets, and consumer lag in real-time. ## Next steps - [Implement rebalance listeners](https://www.conduktor.io/kafka/java-consumer-rebalance-listener) for manual offset management - [Use seek and assign](https://www.conduktor.io/kafka/java-consumer-seek-and-assign) for replay scenarios - [Run consumers in threads](https://www.conduktor.io/kafka/java-consumer-in-threads) for concurrent processing Source: https://www.conduktor.io/kafka/advanced-kafka-consumer-with-java --- # Complete Kafka Consumer with Java *Build a complete Kafka consumer with consumer groups and graceful shutdown* This tutorial walks through building a Java consumer from scratch, including consumer groups, partition rebalancing, automatic offset commits, and graceful shutdown handling. **What you'll learn:** - How to configure and create a KafkaConsumer - How consumer groups and rebalancing work - How automatic offset committing works - How to implement graceful shutdown In IntelliJ IDEA, open the project created in the last section. > **Start Kafka!** > Before running your Java consumer application, make sure that you have [started Kafka](https://www.conduktor.io/kafka/how-to-start-kafka-with-conduktor). ## How to create a Kafka consumer in Java? These are the steps to create a Kafka consumer: 1. Create a Java Class `ConsumerDemo.java` 2. Create the consumer properties 3. Create a consumer 4. Subscribe the consumer to a specific topic 5. Create a Poll loop to receive data ### Create a Java class ConsumerDemo.java Create it as `io.conduktor.demos.kafka.ConsumerDemo` and insert the following content: ```java package io.conduktor.demos.kafka; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class ConsumerDemo { private static final Logger log = LoggerFactory.getLogger(ProducerDemo.class); public static void main(String[] args) { log.info("I am a Kafka Consumer"); } } ``` ### Create Kafka consumer properties Similar to the producer properties, Apache Kafka offers various different properties for creating a consumer as well. To know about each consumer property, visit [https://kafka.apache.org/documentation/#consumerconfigs](https://kafka.apache.org/documentation/#consumerconfigs) **Apache Kafka > Documentation > Configuration > Consumer Configs.** ![Screenshot showing Kafka Consumer properties and configurations from apache.kafka.org](https://www.conduktor.io/assets/kafka/image--54-.png) Here, we will list the required properties of a consumer: - **key.deserializer:** It is a Deserializer class for the key, which is used to implement the `org.apache.kafka.common.serialization.Deserializer` interface. - **value.deserializer:** A Deserializer class for value which implements the `org.apache.kafka.common.serialization.Deserializer` interface. - **bootstrap.servers:** It is a list of host and port pairs that are used to establish an initial connection with the Kafka cluster. It does not contain a full set of servers that a client requires. Only the servers which are required for bootstrapping are required. - **group.id:** It is a unique string that identifies the consumer of a consumer group. - **auto.offset.reset:** This property is required when no initial offset is present or if the current offset does not exist anymore on the server. There are the following values used to reset the offset values: - **earliest:** This offset variable automatically reset the value to its earliest offset. - **latest:** This offset variable reset the offset value to its latest offset. - **none:** If no previous offset is found for the previous group, it throws an exception to the consumer. These are some essential properties that are required to implement a consumer. Let's implement using IntelliJ IDEA. ```java String bootstrapServers = "127.0.0.1:9092"; String groupId = "my-fourth-application"; String topic = "demo_java"; // create consumer configs Properties properties = new Properties(); properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, groupId); properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); ``` > **Auto Offset Reset** > Auto Offset Reset is set to "earliest", which means on the first run of the application we will be reading all historical data in our topic. ### Create the Kafka consumer Create an object of `KafkaConsumer` leveraging our properties, as shown below: ```java // create consumer KafkaConsumer consumer = new KafkaConsumer<>(properties); ``` ### Subscribe the consumer to a specific topic To read the messages from a topic, we need to connect the consumer to the specified topic. Here, we use `Arrays.asList(),`as it allows our consumer to subscribe to multiple topics. Below code shows the implementation of subscription of the consumer to one topic: ```java // subscribe consumer to our topic(s) consumer.subscribe(Arrays.asList(topic)); ``` ### Poll for some new data The consumer reads data from Kafka through the polling method. The `poll` method returns the data that hasn't been fetched yet by the consumer subscribed to the partitions. The duration of the poll call for example `.poll(Duration.ofMillis(100))` is the amount of time to block on this call before returning an empty list in case no data was returned (also called long polling) ```java // poll for new data while(true){ ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord record : records){ log.info("Key: " + record.key() + ", Value: " + record.value()); log.info("Partition: " + record.partition() + ", Offset:" + record.offset()); } } ``` ### Run your Java Kafka consumer application Run the Java application. ![Screenshot showing how to run your Java Kafka Consumer Application in Intellij](https://www.conduktor.io/assets/kafka/image--55-.png) It will run displaying messages from the topic, `demo_java`, created in the last section. ``` ... [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Key: null, Value: hello world 4 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Partition: 2, Offset:13 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Key: null, Value: hello world 6 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Partition: 2, Offset:14 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Key: null, Value: hello world 8 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Partition: 2, Offset:15 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Key: id_2, Value: hello world 2 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Partition: 2, Offset:16 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Key: id_4, Value: hello world 4 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Partition: 2, Offset:17 [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Key: id_5, Value: hello world 5 ... ``` Consumer application runs in an infinite loop, use IntelliJ stop button to stop your application. ### Java consumers in a consumer group We have seen that consumers [can share reads in a Consumer Group](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets) in order to scale. We can achieve this [using the CLI](https://www.conduktor.io/kafka/kafka-consumers-in-group-cli-tutorial), and of course using Java (right below) ### Read messages from the beginning In the last section, we assigned a group id of `my-fourth-application` to the consumer. Stop the consumer application, and re-run it. You will notice there are no messages displayed in the console. This is due to the fact that Kafka keeps track of consumer reads (consumer groups offsets) and therefore the consumer will not encounter the same messages twice. If you change the group id to `my-fifth-application` as shown below. ```java String bootstrapServers = "127.0.0.1:9092"; String groupId = "my-fifth-application"; String topic = "demo_java"; ``` Re-start the application. It will display all the messages in the topic `demo_java` from the beginning. ![Screenshot showing all messages from the topic when the Kafka Consumer Java Application is restarted with a new Kafka Consumer Group ID](https://www.conduktor.io/assets/kafka/image--56-.png) ### Partition rebalance Moving partition ownership from one consumer to another is called a rebalance. Rebalances are important because they provide the consumer group with high availability and scalability. Currently, we only have one consumer in our group, and therefore that consumer reads from all the topic partitions. **To see rebalances in action, launch another consumer by running the same consumer application.** Make sure you have configured to run multiple instances of the application in project setting: ![Example showing how to edit configurations in order to handle partition rebalance of your Complete Kafka Java Consumer.](https://www.conduktor.io/assets/kafka/image--57-.png) Then, edit your configuration to allow for Multiple Instances ![Example of the dialog used to edit your complete Kafka Consumer configurations in Java taken from Intellij](https://www.conduktor.io/assets/kafka/image--58-.png) Then apply and okay to save your changes. Now you can run your consumer multiples times. **The first consumer displays the following information**: ``` [main] INFO ... Discovered group coordinator 127.0.0.1:9094 (id: 2147483644 rack: null) [main] INFO ... (Re-)joining group [main] INFO ... Request joining group due to: need to re-join with the given member-id [main] INFO ... (Re-)joining group [main] INFO ... Successfully joined group with generation Generation{generationId=3, memberId='consumer-my-fifth-application-1-c40fcb55-9893-4b03-a2dd-6acfe08b0c8d', protocol='range'} [main] INFO ... Finished assignment for group at generation 3: {consumer-my-fifth-application-1-c40fcb55-9893-4b03-a2dd-6acfe08b0c8d=Assignment(partitions=[demo_java-0, demo_java-1, demo_java-2])} [main] INFO ... Successfully synced group in generation Generation{generationId=3, memberId='consumer-my-fifth-application-1-c40fcb55-9893-4b03-a2dd-6acfe08b0c8d', protocol='range'} [main] INFO ... Notifying assignor about the new Assignment(partitions=[demo_java-0, demo_java-1, demo_java-2]) [main] INFO ... Adding newly assigned partitions: demo_java-0, demo_java-1, demo_java-2 [main] INFO ... Setting offset for partition demo_java-0 to the committed offset FetchPosition{offset=8, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[127.0.0.1:9092 (id: 1 rack: null)], epoch=0}} [main] INFO ... Setting offset for partition demo_java-1 to the committed offset FetchPosition{offset=8, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[127.0.0.1:9093 (id: 2 rack: null)], epoch=0}} [main] INFO ... Setting offset for partition demo_java-2 to the committed offset FetchPosition{offset=26, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[127.0.0.1:9094 (id: 3 rack: null)], epoch=0}} ``` As we see, the consumer is reading the latest consumer offsets and is assigned to the three partitions in the topic demo java `Assignment(partitions=[demo_java-0, demo_java-1, demo_java-2])` Launch a second consumer using the same process defined as above (do not stop the previous one) and notice the rebalance: **Second Consumer Log** ``` [main] INFO ... (Re-)joining group [main] INFO ... Successfully joined group with generation Generation{generationId=4, memberId='consumer-my-fifth-application-1-beb305ca-524d-466c-b8ce-9ef675382bc3', protocol='range'} [main] INFO ... Successfully synced group in generation Generation{generationId=4, memberId='consumer-my-fifth-application-1-beb305ca-524d-466c-b8ce-9ef675382bc3', protocol='range'} [main] INFO ... Notifying assignor about the new Assignment(partitions=[demo_java-0, demo_java-1]) [main] INFO ... Adding newly assigned partitions: demo_java-0, demo_java-1 [main] INFO ... Setting offset for partition demo_java-0 to the committed offset FetchPosition{offset=8, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[127.0.0.1:9092 (id: 1 rack: null)], epoch=0}} [main] INFO ... Setting offset for partition demo_java-1 to the committed offset FetchPosition{offset=8, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[127.0.0.1:9093 (id: 2 rack: null)], epoch=0}} ``` As you can see partitions `demo_java-0, demo_java-1` have been assigned to the consumer **First Consumer Log** ``` [main] INFO ... Request joining group due to: group is already rebalancing [main] INFO ... Revoke previously assigned partitions demo_java-0, demo_java-1, demo_java-2 [main] INFO ... (Re-)joining group [main] INFO ... Successfully joined group with generation Generation{generationId=4, memberId='consumer-my-fifth-application-1-c40fcb55-9893-4b03-a2dd-6acfe08b0c8d', protocol='range'} [main] INFO ... Finished assignment for group at generation 4: {consumer-my-fifth-application-1-c40fcb55-9893-4b03-a2dd-6acfe08b0c8d=Assignment(partitions=[demo_java-2]), consumer-my-fifth-application-1-beb305ca-524d-466c-b8ce-9ef675382bc3=Assignment(partitions=[demo_java-0, demo_java-1])} [main] INFO ... Successfully synced group in generation Generation{generationId=4, memberId='consumer-my-fifth-application-1-c40fcb55-9893-4b03-a2dd-6acfe08b0c8d', protocol='range'} [main] INFO ... Notifying assignor about the new Assignment(partitions=[demo_java-2]) [main] INFO ... Adding newly assigned partitions: demo_java-2 [main] INFO ... Setting offset for partition demo_java-2 to the committed offset FetchPosition{offset=26, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[127.0.0.1:9094 (id: 3 rack: null)], epoch=0}} ``` As you can see, the partitions assignment was revoked and after the rebalance, only `demo_java-2` was assigned to this consumer . Produce a few messages on the topic using the Kafka producer application that we developed in the last section. The two consumer applications will receive messages from their respective partitions only. ## Automatic offset committing strategy Using the Kafka Consumer Java API, offsets are committed regularly and automatically in order to enable at-least-once reading scenarios. You can get a refresher on [Consumer Offsets here](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets). By default, the property `enable.auto.commit=true` and therefore offsets are committed automatically with a frequency controlled by the config `auto.commit.interval.ms`. The process of committing the offsets happens when the `.poll()` function is called and the time between two calls to `.poll()` is greater than the setting `auto.commit.interval.ms` (5 seconds by default). This means that to be in an "at-least-once" processing use case (the most desirable one), you need to ensure all the messages in your consumer code are successfully processed before performing another `.poll()` call (which is the case in the sample code defined above). If this is not the case, then offsets could be committed before the messages are actually processed, therefore resulting in an "at-most once" processing pattern, possibly resulting in message skipping (which is undesirable). In that (rare) case, you have to disable `enable.auto.commit`, and most likely most processing to a separate thread, and then from time to time call `.commitSync()` or `.commitAsync()`with the correct offsets manually. This complicated use case is discussed in the [Kafka Consumer Documentation](https://kafka.apache.org/30/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html) under the section "Automatic Offset Committing". ![Diagram showing how Auto Offset Commit works in Kafka](https://www.conduktor.io/assets/kafka/Auto-Offest-Committing.png) ## Graceful shutdown of consumer Currently our consumer is running an infinite loop with `while(true)` but we can catch an Exception that happens when our consumer is shutting down. For this, we need to call `consumer.wakeup()` which will trigger a `WakeupException` next time the `.poll()` function is called. The `WakeupException` itself does not need to be handled, but then in a `finally{}` block we can call `consumer.close()` which will take care of: - committing the offsets if needed - close the connection to Kafka In order to call `consumer.wakeup()` we need to use a `ShutdownHook.` That ShutdownHook needs to be linked to the main thread in order to wait for all threads to complete before shutting down the program. The complete code is seen below: ```java package io.conduktor.demos.kafka; import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.ConsumerRecord; import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.common.errors.WakeupException; import org.apache.kafka.common.serialization.StringDeserializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.time.Duration; import java.util.Arrays; import java.util.Properties; public class ConsumerDemo { private static final Logger log = LoggerFactory.getLogger(ConsumerDemo.class); public static void main(String[] args) { log.info("I am a Kafka Consumer"); String bootstrapServers = "127.0.0.1:9092"; String groupId = "my-fifth-application"; String topic = "demo_java"; // create consumer configs Properties properties = new Properties(); properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, groupId); properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // create consumer KafkaConsumer consumer = new KafkaConsumer<>(properties); // get a reference to the current thread final Thread mainThread = Thread.currentThread(); // adding the shutdown hook Runtime.getRuntime().addShutdownHook(new Thread() { public void run() { log.info("Detected a shutdown, let's exit by calling consumer.wakeup()..."); consumer.wakeup(); // join the main thread to allow the execution of the code in the main thread try { mainThread.join(); } catch (InterruptedException e) { e.printStackTrace(); } } }); try { // subscribe consumer to our topic(s) consumer.subscribe(Arrays.asList(topic)); // poll for new data while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord record : records) { log.info("Key: " + record.key() + ", Value: " + record.value()); log.info("Partition: " + record.partition() + ", Offset:" + record.offset()); } } } catch (WakeupException e) { log.info("Wake up exception!"); // we ignore this as this is an expected exception when closing a consumer } catch (Exception e) { log.error("Unexpected exception", e); } finally { consumer.close(); // this will also commit the offsets if need be. log.info("The consumer is now gracefully closed."); } } } ``` The important part to look at is to have a try around our `while` loop and catch for the `WakeupException` that we will trigger from within the loop. ### Output of the graceful shutdown All in all, the output with a graceful shutdown is the following: ``` [Thread-0] INFO io.conduktor.demos.kafka.ConsumerDemo - Detected a shutdown, let's exit by calling consumer.wakeup()... [main] INFO io.conduktor.demos.kafka.ConsumerDemo - Wake up exception! [main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-my-fifth-application-1, groupId=my-fifth-application] Revoke previously assigned partitions demo_java-0, demo_java-1, demo_java-2 [main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-my-fifth-application-1, groupId=my-fifth-application] Member consumer-my-fifth-application-1-4498399b-0c5d-45a9-958f-8b31f124a3d9 sending LeaveGroup request to coordinator 127.0.0.1:9094 (id: 2147483644 rack: null) due to the consumer is being closed [main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-my-fifth-application-1, groupId=my-fifth-application] Resetting generation due to: consumer pro-actively leaving the group [main] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-my-fifth-application-1, groupId=my-fifth-application] Request joining group due to: consumer pro-actively leaving the group [main] INFO org.apache.kafka.common.metrics.Metrics - Metrics scheduler closed [main] INFO org.apache.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter [main] INFO org.apache.kafka.common.metrics.Metrics - Metrics reporters closed [main] INFO org.apache.kafka.common.utils.AppInfoParser - App info kafka.consumer for consumer-my-fifth-application-1 unregistered [main] INFO io.conduktor.demos.kafka.ConsumerDemo - The consumer is now gracefully closed. ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-complete-kafka-consumer-with-java) displays your consumer groups, partition assignments, and lag in real-time. Monitor rebalances and verify your consumer is reading from the expected partitions. ## Next steps - [Explore advanced consumers](https://www.conduktor.io/kafka/advanced-kafka-consumer-with-java) for rebalance listeners, seek/assign, and threading - [Tune consumer settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior) for production - [Understand delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers) for exactly-once processing Source: https://www.conduktor.io/kafka/complete-kafka-consumer-with-java --- # Complete Kafka producer with Java *Build a complete Kafka producer with callbacks and keys* This tutorial walks through building a Java producer from scratch, including callbacks to confirm delivery and message keys for partition routing. **What you'll learn:** - How to configure and create a KafkaProducer - How to send messages with callbacks - How to use message keys for ordering - How the sticky partitioner optimizes batching In IntelliJ IDEA, open the project created in the last section. > **Start Kafka!** > Before running your Java producer application, make sure that you have [started Kafka](https://www.conduktor.io/kafka/how-to-start-kafka-with-conduktor). ## How to create a Kafka producer in Java? There are four steps to create a Java producer: - Create a Java Class `ProducerDemo.java` - Create producer properties - Create the producer - Create a producer record - Send the data ### Create a Java class ProducerDemo.java Create it as `io.conduktor.demos.kafka.ProducerDemo` and insert the following content: ```java package io.conduktor.demos.kafka; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class ProducerDemo { private static final Logger log = LoggerFactory.getLogger(ProducerDemo.class); public static void main(String[] args) { log.info("I am a Kafka Producer"); } } ``` ### Create producer properties Apache Kafka offers various Kafka Properties which are used for creating a producer. To know about each property, visit the official site of Kafka - [https://kafka.apache.org/documentation](https://kafka.apache.org/documentation). Navigate to Kafka > Documentation > Configurations > Producer Configs. ![Screenshot from kafka.apache.org showing the Kafka producer properties and configurations available.](https://www.conduktor.io/assets/kafka/image--52-.png) The required properties that we have to specify are shown below: - **bootstrap.servers:** It is a list of the port pairs which are used for establishing an initial connection to the Kafka cluster. We use the bootstrap servers for making an initial connection to the cluster. This server is present in the host:port, host:port,... form. - **key.serializer:** It is a type of Serializer class of the key that is used to implement the `org.apache.kafka.common.serialization.Serializer` interface. - **value.serializer:** It is a type of Serializer class which implements the `org.apache.kafka.common.serialization.Serializer` interface. Now, let's see the implementation of the producer properties in the IntelliJ IDEA. ```java package io.conduktor.demos.kafka; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.common.serialization.StringSerializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.util.Properties; public class ProducerDemo { private static final Logger log = LoggerFactory.getLogger(ProducerDemo.class); public static void main(String[] args) { log.info("I am a Kafka Producer"); String bootstrapServers = "127.0.0.1:9092"; // create Producer properties Properties properties = new Properties(); properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); } } ``` ### Create the Kafka producer To create a Kafka producer, we just need to create an object of `KafkaProducer`. The object of KafkaProducer can be created as: ```java // create the producer KafkaProducer producer = new KafkaProducer<>(properties); ``` ### Create a producer record In order to send the data to Kafka, we need to create a `ProducerRecord`. Here, the producer specifies the topic name as well as the message value which is to be delivered to Kafka. The key is assumed to be `null` in this instance. A `ProducerRecord` can be created as: ```java // create a producer record ProducerRecord producerRecord = new ProducerRecord<>("first_topic", "hello world"); ``` ### Send the data Now, we are ready to send the data to Kafka. The producer just needs to invoke the object of the ProducerRecord as: ```java // send data - asynchronous producer.send(producerRecord); // flush data - synchronous producer.flush(); // flush and close producer producer.close(); ``` The data produced by a producer is asynchronous. Therefore, two additional functions, i.e., `flush()` and `close()` are required to ensure the producer is shut down after the message is sent to Kafka. The `flush()` will force all the data that was in `.send()` to be produced and `close()` stops the producer. If these functions are not executed, the data will never be sent to Kafka as the main Java thread will exit before the data are flushed. The complete code is shown below. ```java package io.conduktor.demos.kafka; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.common.serialization.StringSerializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.util.Properties; public class ProducerDemo { private static final Logger log = LoggerFactory.getLogger(ProducerDemo.class); public static void main(String[] args) { log.info("I am a Kafka Producer"); String bootstrapServers = "127.0.0.1:9092"; // create Producer properties Properties properties = new Properties(); properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); // create the producer KafkaProducer producer = new KafkaProducer<>(properties); // create a producer record ProducerRecord producerRecord = new ProducerRecord<>("demo_java", "hello world"); // send data - asynchronous producer.send(producerRecord); // flush data - synchronous producer.flush(); // flush and close producer producer.close(); } } ``` ### Run your Java Kafka producer application First, [create a Kafka topic](https://www.conduktor.io/kafka/kafka-topics-cli-tutorial) `demo_java` with 3 partitions: ```bash kafka-topics --bootstrap-server localhost:9092 --topic demo_java --create --partitions 3 --replication-factor 1 ``` To observe the output of our Java producer application, open the Kafka consumer CLI, `kafka-console-consumer` using the command: ```bash kafka-console-consumer --bootstrap-server localhost:9092 --topic demo_java ``` > **CLI Extensions** > Use CLI commands with appropriate extensions for your platform, e.g., `kafka-console-consumer.bat` for windows, `kafka-console-consumer.sh` for Linux Then let's run our Java Producer ![Screenshot from IntelliJ showing how to run the complete Kafka producer that we have programmed in Java.](https://www.conduktor.io/assets/kafka/image--53-.png) The log of our application should look like this: ``` [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.8.1 [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 8cb0a5e9d3441962 [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1641907903443 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {first_topic=LEADER_NOT_AVAILABLE} [kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Cluster ID: 2gpQgBYVQEeFycBwnOudWg [main] INFO org.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. [main] INFO org.apache.kafka.common.metrics.Metrics - Metrics scheduler closed [main] INFO org.apache.kafka.common.metrics.Metrics - Closing reporter org.apache.kafka.common.metrics.JmxReporter [main] INFO org.apache.kafka.common.metrics.Metrics - Metrics reporters closed [main] INFO org.apache.kafka.common.utils.AppInfoParser - App info kafka.producer for producer-1 unregistered BUILD SUCCESSFUL in 1s 2 actionable tasks: 2 executed ``` And our Kafka consumer should have successfully displayed the message ```bash kafka-console-consumer --bootstrap-server localhost:9092 --topic demo_java --from-beginning hello world ``` Success, we have sent our first message to Kafka using the Java Producer API. Let's explore some more functionalities. ## Java producer callbacks and sticky partitioner We saw how a producer sends data to Kafka. Now we'll learn more about the messages produced, e.g., whether the data was correctly produced, where it was produced, retrieving its offset and partition value, etc. For this, we need to implement a callback function. This function is implemented for asynchronously handling the request completion. That's why its return type will be void. This function will be implemented in the block where the producer sends data to Kafka. The callback function invoked by the producer is `onCompletion()`. Basically, this method requires two arguments: - **Metadata of the Record:** Metadata of the record e.g. information regarding the partition and its offsets. - **Exception:** Any exception thrown during the produce request Let's see the implementation of the Producer callback in the below code extract: ```java // send data - asynchronous producer.send(producerRecord, new Callback() { public void onCompletion(RecordMetadata recordMetadata, Exception e) { // executes every time a record is successfully sent or an exception is thrown if (e == null) { // the record was successfully sent log.info("Received new metadata. \n" + "Topic:" + recordMetadata.topic() + "\n" + "Partition: " + recordMetadata.partition() + "\n" + "Offset: " + recordMetadata.offset() + "\n" + "Timestamp: " + recordMetadata.timestamp()); } else { log.error("Error while producing", e); } } }); ``` If the exception value is equal to null, the logger will display the information, else the produce error will be displayed. When the above code is executed, we will come to know the topic name, partition number, timestamp, offset value where the message is sent as shown below. ``` [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 2 Offset: 1 Timestamp: 1641908650857 ``` If we run a loop to send 10 records ```java for (int i=0; i<10; i++ ) { // create a producer record ProducerRecord producerRecord = new ProducerRecord("demo_java", "hello world " + Integer.toString(i)); // send data - asynchronous producer.send(producerRecord, new Callback() { public void onCompletion(RecordMetadata recordMetadata, Exception e) { // executes every time a record is successfully sent or an exception is thrown if (e == null) { // the record was successfully sent log.info("Received new metadata. \n" + "Topic:" + recordMetadata.topic() + "\n" + "Partition: " + recordMetadata.partition() + "\n" + "Offset: " + recordMetadata.offset() + "\n" + "Timestamp: " + recordMetadata.timestamp()); } else { log.error("Error while producing", e); } } }); } ``` We see the following output: ``` [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 2 Offset: 2 Timestamp: 1641908746619 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 2 Offset: 3 Timestamp: 1641908746637 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 2 Offset: 4 Timestamp: 1641908746637 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 2 Offset: 5 Timestamp: 1641908746637 ``` **Surprisingly, even though the key is** **`null`** **we see all the records going to the same partition!** This is not a bug, this is a performance improvement feature. Since Kafka v2.4.0, the partitioner is a [Sticky Partitioner](https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner), which means the producer that receives messages sent in time close to each other will try to fill a batch into ONE partition before switching to creating a batch for another partition. To observe the round-robin feature of Kafka, we can add a `Thread.sleep(1000)` in between each iteration of the loop, which will force the batch to be sent and a new batch to be created for a different partition. ![Diagram showing the performance improvement delivered by the sticky partitioner in Kafka](https://www.conduktor.io/assets/kafka/Sticky-Partitioner-vs-Round-Robin--2-.png) ```java for (int i=0; i<10; i++ ) { // create a producer record ProducerRecord producerRecord = new ProducerRecord("demo_java", "hello world " + Integer.toString(i)); // send data - asynchronous producer.send(producerRecord, new Callback() { public void onCompletion(RecordMetadata recordMetadata, Exception e) { // executes every time a record is successfully sent or an exception is thrown if (e == null) { // the record was successfully sent log.info("Received new metadata. \n" + "Topic:" + recordMetadata.topic() + "\n" + "Partition: " + recordMetadata.partition() + "\n" + "Offset: " + recordMetadata.offset() + "\n" + "Timestamp: " + recordMetadata.timestamp()); } else { log.error("Error while producing", e); } } }); try { Thread.sleep(1000); } catch (InterruptedException e) { e.printStackTrace(); } } ``` We see now see the following output: ``` [kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Cluster ID: 2gpQgBYVQEeFycBwnOudWg [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 2 Offset: 12 Timestamp: 1641909170161 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 1 Offset: 0 Timestamp: 1641909171175 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 0 Offset: 0 Timestamp: 1641909172181 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 1 Offset: 1 Timestamp: 1641909173186 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoWithCallback - Received new metadata. Topic:demo_java Partition: 2 Offset: 13 ... ``` And we now observe that the data is going to different partitions in Kafka, thus achieving a form of "round-robin". ## Java producer with keys Keys become useful when a user wants to introduce ordering and ensure the messages that share the same key end up in the same partition. The `ProducerRecord` constructor accepts a key as shown below: ```java for (int i=0; i<10; i++ ) { // create a producer record String topic = "demo_java"; String value = "hello world " + Integer.toString(i); String key = "id_" + Integer.toString(i); ProducerRecord producerRecord = new ProducerRecord<>(topic, key, value); // send data - asynchronous producer.send(producerRecord, new Callback() { public void onCompletion(RecordMetadata recordMetadata, Exception e) { // executes every time a record is successfully sent or an exception is thrown if (e == null) { // the record was successfully sent log.info("Received new metadata. \n" + "Topic:" + recordMetadata.topic() + "\n" + "Key:" + producerRecord.key() + "\n" + "Partition: " + recordMetadata.partition() + "\n" + "Offset: " + recordMetadata.offset() + "\n" + "Timestamp: " + recordMetadata.timestamp()); } else { log.error("Error while producing", e); } } }); } ``` In the above snippet, we also added the following log line: ```java "Key:" + producerRecord.key() + "\n" + ``` Which will help us map the key to the partition. **Run the Java application.** It should show logs showing messages going to different partitions depending upon the key. ``` [kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Cluster ID: 2gpQgBYVQEeFycBwnOudWg [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_0 Partition: 1 Offset: 6 Timestamp: 1641909602395 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_8 Partition: 1 Offset: 7 Timestamp: 1641909602407 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_1 Partition: 0 Offset: 5 Timestamp: 1641909602406 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_3 Partition: 0 Offset: 6 Timestamp: 1641909602406 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_6 Partition: 0 Offset: 7 Timestamp: 1641909602407 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_2 Partition: 2 Offset: 21 Timestamp: 1641909602406 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_4 Partition: 2 Offset: 22 Timestamp: 1641909602406 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_5 Partition: 2 Offset: 23 Timestamp: 1641909602407 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_7 Partition: 2 Offset: 24 Timestamp: 1641909602407 [kafka-producer-network-thread | producer-1] INFO io.conduktor.demos.kafka.ProducerDemoKeys - Received new metadata. Topic:demo_java Key:id_9 Partition: 2 Offset: 25 Timestamp: 1641909602407 ``` ### Important things to note: - The messages are sent in batches by partition, which is why you see all the messages of Partition 1 first (first one that was sent), then message of Partition 0 then (second one that was sent) and then Partition 2 (last one) - If you re-run the code, you will find that the message with the same key will go to the same partition (try re-running the code and compare your outputs!) ## Conclusion In this complete tutorial, we have learned how the Kafka Producer API works, but also explored some of the interesting performance optimization the Kafka Producer has internally to efficiently send data to Kafka. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-complete-kafka-producer-with-java) lets you immediately see messages your producer sends. View message keys, values, partitions, and offsets to verify your producer is working correctly. ## Next steps - [Build a consumer](https://www.conduktor.io/kafka/complete-kafka-consumer-with-java) to read the messages you produced - [Learn about producer settings](https://www.conduktor.io/kafka/kafka-producer-acks-deep-dive) for durability - [Understand batching](https://www.conduktor.io/kafka/kafka-producer-batching) for performance Source: https://www.conduktor.io/kafka/complete-kafka-producer-with-java --- # Consumer auto offset reset behavior *Learn how to configure consumer offset reset behavior* When a consumer starts without committed offsets, or when committed offsets are invalid, Kafka needs to know where to start reading. The `auto.offset.reset` configuration controls this behavior and is critical for understanding data processing guarantees. **What you'll learn:** - The three auto offset reset options and when each is triggered - How to choose the right setting for your use case - Best practices for production and development environments - How to handle offset reset scenarios programmatically ## Auto offset reset options When a Kafka consumer starts and there are no committed offsets for its consumer group, or when the committed offset is no longer valid (e.g., because the data has been deleted), the consumer needs to decide where to start reading from. This behavior is controlled by the `auto.offset.reset` configuration. ### earliest ```properties auto.offset.reset=earliest ``` - Consumer will start reading from the beginning of the partition - Reads all available messages from the earliest available offset - Useful for reprocessing all historical data - **Use case**: Data migration, audit requirements, complete reprocessing ### latest (default) ```properties auto.offset.reset=latest ``` - Consumer will start reading from the end of the partition - Only processes new messages produced after the consumer starts - **Use case**: Real-time processing where historical data is not needed ### none ```properties auto.offset.reset=none ``` - Consumer throws an exception if no previous offset is found - Forces explicit offset management - **Use case**: Strict control over consumer behavior, prevents accidental data loss or reprocessing ### Decision guide ![Decision flowchart for choosing auto.offset.reset: need historical data leads to earliest, strict offset control leads to none, otherwise latest, each with its risk](https://www.conduktor.io/assets/kafka/diagrams/consumer-auto-offsets-reset-behavior.svg) ```mermaid flowchart TD Start["New consumer
or invalid offset"] --> Q1{"Need historical
data?"} Q1 -->|Yes| Earliest["earliest
Read from beginning"] Q1 -->|No| Q2{"Strict offset
control needed?"} Q2 -->|Yes| None["none
Throw exception"] Q2 -->|No| Latest["latest
Read new messages only"] Earliest --> Risk1["⚠️ Risk: May reprocess
large amounts of data"] Latest --> Risk2["⚠️ Risk: May miss
messages produced while down"] None --> Risk3["⚠️ Requires: Explicit
error handling in code"] ``` ## When auto offset reset is triggered The `auto.offset.reset` behavior is triggered in these scenarios: | Scenario | Description | Example | |----------|-------------|---------| | **New consumer group** | First time a consumer group subscribes to a topic | Deploying a new application | | **Invalid offset** | Committed offset no longer exists (data deleted due to retention) | Consumer offline longer than retention period | | **Offset out of range** | Committed offset is beyond the current log boundaries | Log truncation or corruption | ## Common scenarios ### Scenario 1: New consumer group ```java // First time this consumer group runs Properties props = new Properties(); props.put("group.id", "new-consumer-group"); props.put("auto.offset.reset", "earliest"); // Will read from beginning ``` ### Scenario 2: Data retention cleanup ```java // Consumer was offline for too long, committed offset expired // Behavior depends on auto.offset.reset setting Properties props = new Properties(); props.put("group.id", "existing-group"); props.put("auto.offset.reset", "latest"); // Will skip to latest ``` ## Best practices ### For production systems ```properties # Be explicit about offset reset behavior auto.offset.reset=latest # Enable offset commits enable.auto.commit=true auto.commit.interval.ms=5000 ``` ### For development/testing ```properties # Often want to reprocess data auto.offset.reset=earliest # May want manual control enable.auto.commit=false ``` ### For critical data processing ```properties # Prevent accidental data loss or reprocessing auto.offset.reset=none # Handle exceptions explicitly in code ``` ## Error handling example ```java Properties props = new Properties(); props.put("auto.offset.reset", "none"); KafkaConsumer consumer = new KafkaConsumer<>(props); try { consumer.subscribe(Arrays.asList("my-topic")); while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(1000)); // Process records } } catch (NoOffsetForPartitionException e) { // Handle case where no valid offset exists // Decide whether to seek to beginning or end consumer.seekToBeginning(consumer.assignment()); // or consumer.seekToEnd(consumer.assignment()); } ``` ## Offset management strategies ### Automatic offset management - Use `enable.auto.commit=true` - Set appropriate `auto.commit.interval.ms` - Choose suitable `auto.offset.reset` policy ### Manual offset management - Use `enable.auto.commit=false` - Call `commitSync()` or `commitAsync()` after processing - Handle offset reset scenarios explicitly ### External offset storage - Store offsets in external systems (database, file system) - Use `seek()` methods to position consumer - Implement custom offset management logic > **Data loss vs duplication** > - `auto.offset.reset=latest` can cause data loss if messages arrive while consumer is down > - `auto.offset.reset=earliest` can cause message duplication if consumer group is recreated > - `auto.offset.reset=none` requires explicit error handling but provides the most control ## Configuration recommendations | Use case | auto.offset.reset | enable.auto.commit | Notes | |----------|------------------|-------------------|-------| | High-throughput | `latest` | `true` | Accept potential data loss for speed | | Critical data | `none` | `false` | Manual control, handle exceptions | | Replay scenarios | `earliest` | `false` | Process all historical data | | Development | `earliest` | `true` | Easy testing with full data | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-consumer-auto-offsets-reset-behavior) lets you monitor consumer group offsets and lag in real-time. Identify when offset resets occur and track consumer position across partitions to validate your offset management strategy. ## Next steps - [Read from the closest replica](https://www.conduktor.io/kafka/consumer-read-from-closest-replica) to cut cross-datacenter latency - [Understand delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers) for reliable processing - [Configure consumer settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior) for optimal performance Source: https://www.conduktor.io/kafka/consumer-auto-offsets-reset-behavior --- # Consumer incremental rebalance and static group membership *Learn how to minimize rebalance disruption* Kafka's incremental cooperative rebalancing and static group membership features reduce the disruption caused by consumer group rebalances, improving overall system stability and performance. **What you'll learn:** - How incremental cooperative rebalancing reduces processing downtime - The multi-phase rebalancing process and how it differs from eager rebalancing - How to configure static group membership for stable consumer identities - Best practices for production deployments ## Traditional rebalancing problems ### Eager rebalancing (pre-Kafka 2.4) - **Stop-the-world**: All consumers stop processing during rebalance - **Complete reassignment**: All partitions are revoked and reassigned - **Processing downtime**: No messages processed during rebalance period - **Cascading rebalances**: One consumer failure affects entire group ### Performance impact ``` Consumer 1: [P0, P1, P2] → [ ] → [P0, P1] Consumer 2: [P3, P4, P5] → [ ] → [P2, P3] Consumer 3: [P6, P7, P8] → [ ] → [P4, P5, P6, P7, P8] All consumers stop processing during the transition ``` ### Eager versus incremental rebalancing comparison This diagram compares the impact of eager rebalancing versus incremental cooperative rebalancing: ![](https://www.conduktor.io/assets/kafka/increment-coop.png) ## Incremental cooperative rebalancing ### How it works (Kafka 2.4+) - **Minimal disruption**: Only affected partitions are reassigned - **Continued processing**: Unaffected partitions continue processing - **Gradual transition**: Rebalance happens in multiple phases - **Reduced downtime**: Significantly shorter processing interruptions ### Rebalancing phases The incremental rebalance happens in two distinct phases, minimizing disruption: ![](https://www.conduktor.io/assets/kafka/increment-phases.png) **Key improvements over eager rebalancing:** - ✅ Only ONE consumer stops ONE partition (P2) - ✅ Eight out of nine partitions never stop processing - ✅ Total downtime: ~100ms instead of several seconds - ✅ Cascading failures prevented ### Configuration ```properties # Enable incremental cooperative rebalancing (default in Kafka 2.4+) partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor # Or use range assignor with cooperative rebalancing partition.assignment.strategy=org.apache.kafka.clients.consumer.RangeAssignor ``` ## Static group membership ### Concept Static group membership allows consumers to maintain stable identities across restarts, preventing unnecessary rebalances during planned maintenance or brief outages. ### Benefits - **Fewer rebalances**: Consumer restarts don't trigger rebalances - **Stable assignments**: Partitions stay with the same consumer instance - **Faster recovery**: Consumers can resume processing from where they left off - **Operational efficiency**: Planned maintenance doesn't disrupt other consumers ### Configuration ```properties # Assign static member ID to consumer group.instance.id=consumer-instance-1 # Increase session timeout for planned restarts session.timeout.ms=300000 # 5 minutes # Adjust heartbeat interval accordingly heartbeat.interval.ms=100000 # ~1.7 minutes ``` ### Consumer lifecycle ```java Properties props = new Properties(); props.put("group.id", "my-consumer-group"); props.put("group.instance.id", "consumer-1"); // Static member ID props.put("session.timeout.ms", "300000"); // 5 minutes props.put("heartbeat.interval.ms", "100000"); // ~1.7 minutes KafkaConsumer consumer = new KafkaConsumer<>(props); ``` ## Use cases and benefits ### High-availability applications ```properties # Configuration for critical applications group.instance.id=${hostname}-${process.id} session.timeout.ms=300000 partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor # Allows planned restarts without affecting other consumers ``` ### Containerized environments ```yaml # Kubernetes deployment example apiVersion: apps/v1 kind: Deployment spec: template: spec: containers: - name: kafka-consumer env: - name: GROUP_INSTANCE_ID value: "consumer-${POD_NAME}" # Consumer will maintain identity across pod restarts ``` ### Stream processing applications - **State preservation**: Local state stores remain associated with specific consumers - **Reduced reprocessing**: Avoid recomputing state after rebalances - **Consistent partitioning**: Same consumer always processes same partitions ## Monitor and observe ### Key metrics - **Rebalance frequency**: Number of rebalances per time period - **Rebalance duration**: Time taken for rebalance completion - **Partition assignment stability**: How often partitions change owners - **Consumer lag during rebalance**: Processing delay during rebalances ### JMX metrics ``` # Rebalance metrics kafka.consumer:type=consumer-coordinator-metrics,client-id=* - rebalance-rate-per-hour - rebalance-latency-avg - rebalance-latency-max # Assignment metrics kafka.consumer:type=consumer-metrics,client-id=* - assigned-partitions ``` ## Configuration best practices ### For incremental rebalancing ```properties # Use cooperative assignors partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor # Optimize for stability session.timeout.ms=45000 heartbeat.interval.ms=15000 max.poll.interval.ms=300000 ``` ### For static group membership ```properties # Stable consumer identity group.instance.id=unique-consumer-id # Extended timeouts for planned restarts session.timeout.ms=600000 # 10 minutes heartbeat.interval.ms=200000 # ~3.3 minutes # Prevent accidental timeouts max.poll.interval.ms=900000 # 15 minutes ``` ### Combined configuration ```properties # Best of both worlds group.instance.id=consumer-${hostname} partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor session.timeout.ms=300000 heartbeat.interval.ms=100000 max.poll.interval.ms=600000 ``` ## Operational considerations ### Deployment strategies 1. **Rolling updates**: Use static group membership for zero-downtime deployments 2. **Blue-green**: Static IDs help maintain partition assignments 3. **Canary releases**: Incremental rebalancing minimizes impact on stable consumers ### Maintenance windows ```bash # Planned consumer restart with static membership # 1. Consumer stops gracefully # 2. Other consumers continue processing (no rebalance) # 3. Consumer restarts with same group.instance.id # 4. Resumes processing assigned partitions ``` ### Troubleshooting Common issues and solutions: - **Duplicate static IDs**: Ensure unique `group.instance.id` per consumer - **Long session timeouts**: Balance between stability and failure detection - **Assignment strategy conflicts**: Ensure all consumers use compatible assignors When migrating to incremental rebalancing and static membership, start with incremental rebalancing first, monitor rebalance behavior and performance, then gradually introduce static group membership and test failure scenarios thoroughly. > **Static member considerations** > - Static members that don't restart within `session.timeout.ms` will be removed from the group > - Ensure unique `group.instance.id` values to avoid conflicts > - Plan for scaling scenarios where static IDs need management ## Performance impact ### Before (eager rebalancing) ``` Rebalance triggered → All consumers stop → Complete reassignment → Resume processing Downtime: 10-30 seconds for entire consumer group ``` ### After (incremental + static) ``` Rebalance triggered → Only affected partitions stop → Minimal reassignment → Resume processing Downtime: 1-5 seconds for affected partitions only ``` ### Measurable improvements - **90% reduction** in processing downtime during rebalances - **50% fewer** unnecessary rebalances with static membership - **Improved throughput** due to reduced processing interruptions - **Better consumer utilization** with sticky partition assignments > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-consumer-incremental-rebalance-and-static-group-membership) visualizes consumer group rebalances in real-time, showing which partitions are being reassigned and the rebalance duration. Monitor consumer lag during rebalances to verify that incremental rebalancing is minimizing disruption as expected. ## Next steps - [Move on to Kafka administration](https://www.conduktor.io/kafka/kafka-administration) to operate and maintain your cluster - [Understand delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers) for reliable processing - [Configure consumer settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior) for optimal performance Source: https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership --- # Consumer read from closest replica *Learn how to optimize consumer reads with rack awareness* Kafka consumers can be configured to read from the closest replica rather than always reading from the leader, which can improve performance and reduce cross-datacenter network traffic in geographically distributed deployments. **What you'll learn:** - How replica reading works in Kafka - How to configure rack-aware consumers - Benefits and trade-offs of closest replica reading - Best practices for multi-region deployments ## How replica reading works By default, Kafka consumers always read from the partition leader. However, starting with Kafka 2.4, consumers can be configured to read from follower replicas that are "close" to the consumer. ### Default behavior (leader-only reads) ![Default leader-only reads: a consumer in US-West makes a cross-region read to the leader of partition 0 in US-East, which replicates to a follower in EU-Central](https://www.conduktor.io/assets/kafka/diagrams/consumer-read-from-closest-replica--1.svg) ```mermaid flowchart LR subgraph US-West["US-West (Consumer location)"] C["Consumer"] end subgraph US-East["US-East"] L["Leader
Partition 0"] end subgraph EU["EU-Central"] F["Follower
Replica"] end C -->|"Cross-region read"| L L -->|"Replicate"| F ``` - All consumers read from partition leaders - Followers only replicate data, never serve reads - Simple and consistent behavior - May result in cross-datacenter traffic ### Closest replica reading ![Closest replica reading: a consumer with client.rack=us-west does a local read from a us-west replica, which is replicated from the leader in US-East](https://www.conduktor.io/assets/kafka/diagrams/consumer-read-from-closest-replica--2.svg) ```mermaid flowchart LR subgraph US-West["US-West"] C1["Consumer
client.rack=us-west"] R1["Replica
broker.rack=us-west"] end subgraph US-East["US-East"] L["Leader"] end C1 -->|"Local read"| R1 L -->|"Replicate"| R1 ``` - Consumers can read from geographically closest replicas - Reduces network latency and cross-datacenter bandwidth - Requires proper rack awareness configuration - Maintains consistency guarantees ## Configuration ### Enable closest replica reading ```properties # Consumer configuration client.rack=us-west-2a # This tells Kafka which rack/availability zone the consumer is in # Kafka will prefer replicas in the same rack when available ``` ### Broker configuration for rack awareness ```properties # Broker configuration (server.properties) broker.rack=us-west-2a # Each broker should be configured with its rack/AZ # This enables Kafka to make intelligent replica placement decisions ``` ### Topic configuration When creating topics, consider replica placement: ```bash # Create topic with rack-aware replica assignment kafka-topics --bootstrap-server localhost:9092 \ --create --topic my-topic \ --partitions 6 \ --replication-factor 3 \ --config min.insync.replicas=2 ``` ## Benefits | Benefit | Description | |---------|-------------| | **Reduced latency** | Consumers read from local replicas instead of remote leaders | | **Cost savings** | Reduces expensive cross-region data transfer charges | | **Improved availability** | Continues reading even if cross-datacenter links are degraded | ## Consistency considerations ### Read-after-write consistency With closest replica reading, you may encounter scenarios where: - Producer writes to leader in datacenter A - Consumer reads from follower in datacenter B - Replication lag may cause temporary inconsistency ### Mitigation strategies ```properties # Ensure minimum in-sync replicas for writes min.insync.replicas=2 # Use appropriate acks setting acks=all # Configure consumer to handle potential inconsistencies ``` > **Replication lag impact** > When reading from follower replicas, consumers may see slightly stale data due to replication lag. Ensure your application can tolerate this eventual consistency model. ## Use cases ### Multi-region deployments ``` Region US-West: - Brokers with rack=us-west - Consumers with client.rack=us-west - Reads stay local to us-west replicas Region EU-Central: - Brokers with rack=eu-central - Consumers with client.rack=eu-central - Reads stay local to eu-central replicas ``` ### Availability zone optimization ``` AZ-1: broker.rack=az-1, client.rack=az-1 AZ-2: broker.rack=az-2, client.rack=az-2 AZ-3: broker.rack=az-3, client.rack=az-3 Each AZ reads from local replicas when possible ``` ## Configuration examples ### Cloud deployment (AWS) ```properties # Broker configuration broker.rack=${aws.availability.zone} # Consumer configuration client.rack=us-west-2a # Additional consumer settings for optimal performance fetch.min.bytes=1048576 fetch.max.wait.ms=500 ``` ### On-premises multi-datacenter ```properties # Broker configuration broker.rack=datacenter-1 # Consumer configuration client.rack=datacenter-1 # Network optimization socket.receive.buffer.bytes=65536 fetch.max.bytes=52428800 ``` ## Decision guide | Scenario | Recommendation | |----------|----------------| | Single datacenter | Not needed | | Multi-AZ within region | Optional, reduces AZ-to-AZ traffic | | Multi-region | Recommended for cost and latency | | Strong consistency required | Use leader-only reads | | Eventually consistent OK | Use closest replica | > **Gradual rollout** > Consider implementing closest replica reading gradually: > 1. Start with non-critical consumer groups > 2. Monitor metrics and consistency behavior > 3. Expand to more critical workloads as confidence builds ## Troubleshooting ### Common issues | Issue | Cause | Solution | |-------|-------|----------| | Still reading from leader | Missing rack configuration | Configure `client.rack` on consumer | | No local replicas | Insufficient replicas in rack | Add brokers or increase replication factor | | High replication lag | Follower falling behind | Monitor and tune replication | ### Verification steps ```bash # Check broker rack configuration kafka-configs --bootstrap-server localhost:9092 --describe --entity-type brokers # Monitor consumer metrics kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group your-group # Check topic replica distribution kafka-topics --bootstrap-server localhost:9092 --describe --topic your-topic ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-consumer-read-from-closest-replica) displays broker rack configuration and replica distribution across your cluster. Monitor which replicas consumers are reading from and track cross-datacenter traffic patterns. ## Next steps - [Configure incremental rebalancing](https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership) to reduce rebalance disruption - [Understand topic replication](https://www.conduktor.io/kafka/kafka-topic-replication) in depth - [Configure consumer settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior) for optimal performance Source: https://www.conduktor.io/kafka/consumer-read-from-closest-replica --- # Creating a Kafka Java Project using Gradle (build.gradle) *Set up a Gradle project with Kafka dependencies* Before writing Kafka producers and consumers, you need a Java project with the Kafka client library and logging dependencies configured. **What you'll learn:** - How to create a new Gradle project in IntelliJ - How to add Kafka client dependencies - How to configure logging with SLF4J - How to verify your setup works ## Create a Kafka Gradle project with build.gradle and set up dependencies In IntelliJ IDEA, create a new Java Gradle project (File > New > Project) ![Screenshot showing how to create a new project file for your Kafka Gradle Java Project in IntelliJ](https://www.conduktor.io/assets/kafka/image--46-.png) Then add your Gradle project attributes ![Intellij screenshot showing how to add attributes to your Kafka Gradle project in Java.](https://www.conduktor.io/assets/kafka/image--47-.png) The build tool Gradle contains a `**build.gradle**` file. The `build.gradle` is a default Gradle file that carries all the information regarding the Group and Version values . The user needs to define all the necessary project dependencies in the `build.gradle` file. Go to the `build.gradle` file. ![Screenshot showing how to define dependencies for your Kafka project in Gradle.](https://www.conduktor.io/assets/kafka/image--48-.png) Define the Kafka Dependencies in the `dependencies { ... }` block. Add a dependency for Kafka client and logging dependencies as shown below ```gradle dependencies { // https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients implementation 'org.apache.kafka:kafka-clients:2.8.1' // https://mvnrepository.com/artifact/org.slf4j/slf4j-api implementation 'org.slf4j:slf4j-api:1.7.32' // https://mvnrepository.com/artifact/org.slf4j/slf4j-simple implementation 'org.slf4j:slf4j-simple:1.7.32' testImplementation 'org.junit.jupiter:junit-jupiter-api:5.6.0' testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine' } ``` Load the Gradle changes with the menu from the right hand side to import the dependencies ![Screenshot showing how to load the Gradle changes to your Kafka project using the right hand menu.](https://www.conduktor.io/assets/kafka/image--49-.png) Now, we have set all the required dependencies. Let's try the _Simple Hello World_ example. ## Create your first class Create a java package say, `io.conduktor.demos.kafka.HelloWorld` ![Screenshot showing the New Java Class dialog for your Kafka Gradle project](https://www.conduktor.io/assets/kafka/image--50-.png) While creating the java package, follow the package naming conventions. Finally, create the sample application program as shown below. ```java package io.conduktor.demos.kafka; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class HelloWorld { private static final Logger log = LoggerFactory.getLogger(HelloWorld.class); public static void main(String[] args) { log.info("Hello World"); } } ``` Run the application (the play green button on line 9 in the screenshot below) and verify that it runs and prints the message, and exits with code `0`. This means that your Java application has run successfully. Expand the 'External Libraries' on the Project panel and verify that it displays the dependencies that we added for the project in `build.gradle` file ![Screenshot showing successful setup of your Kafka Gradle project in Java.](https://www.conduktor.io/assets/kafka/image--51-.png) We have created a sample Java project that includes all the needed dependencies. This will form the basis for creating Java producers and consumers next. > **See it in practice with Conduktor** > Once your project is set up, use [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-creating-a-kafka-java-project-using-gradle-build-gradle) to create topics, inspect messages, and test your Kafka applications during development. ## Next steps - [Create a Kafka producer](https://www.conduktor.io/kafka/complete-kafka-producer-with-java) to send your first messages - [Create a Kafka consumer](https://www.conduktor.io/kafka/complete-kafka-consumer-with-java) to read messages - [Set up with Maven](https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-maven-pom-xml) if you prefer that build tool Source: https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-gradle-build-gradle --- # Creating a Kafka Java Project using Maven (pom.xml) *Set up a Maven project with Kafka dependencies* Before writing Kafka producers and consumers, you need a Java project with the Kafka client library and logging dependencies configured. **What you'll learn:** - How to create a new Maven project in IntelliJ - How to add Kafka client dependencies to pom.xml - How to configure logging with SLF4J - How to verify your setup works ## Create a Maven project with pom.xml and set up dependencies In IntelliJ IDEA, create a new Java maven project (File > New > Project) ![Kafka Maven Java Project screenshot from IntelliJ IDEA when beginning a new Java Maven Project for interacting with a Kafka Cluster.](https://www.conduktor.io/assets/kafka/image--40-.png) Then add your Maven project attributes ![Screenshot showing the attributes for your Kafka Demo Java project in Maven ((pom.xl) via IntelliJ IDEA.](https://www.conduktor.io/assets/kafka/image--41-.png) The build tool Maven contains a `**pom.xml**` file. The `pom.xml` is a default XML file that carries all the information regarding the GroupID, ArtifactID, as well as the Version values. The user needs to define all the necessary project dependencies in the `pom.xml` file. Go to the `pom.xml` file. ![Defining the dependencies for Kafka Maven Client project in pom.xl xml file screenshot.](https://www.conduktor.io/assets/kafka/image--42-.png) Define the Kafka Dependencies. Create a `**...**` block within which we will define the required dependencies. Add a dependency for Kafka client as shown below ```xml ... org.apache.kafka kafka-clients 2.8.1 ``` If the version number appears red in color, it means the user missed to enable the '**Auto-Import**' option. If so, go to **View > Tool Windows > Maven**. A Maven Projects Window will appear on the right side of the screen. Click on the 'Refresh' button appearing right there. This will enable the missed Auto-Import Maven Projects. If the color changes to black, it means the missed dependency is downloaded. ![Screenshot showing the Maven Projects window on the right for our Kafka Maven project.](https://www.conduktor.io/assets/kafka/image--43-.png) Add another dependency for logging. This will enable us to print diagnostic logs while our application runs. ```xml org.slf4j slf4j-api 1.7.32 org.slf4j slf4j-simple 1.7.32 ``` Now, we have set all the required dependencies. Let's try the _Simple Hello World_ example. ## Create your first class Create a java package say, `io.conduktor.demos.kafka.HelloWorld` ![Screenshot showing how to create a new java class for our Kafka Maven project.](https://www.conduktor.io/assets/kafka/image--44-.png) While creating the java package, follow the package naming conventions. Finally, create the sample application program as shown below. ```java package io.conduktor.demos.kafka; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class HelloWorld { private static final Logger log = LoggerFactory.getLogger(HelloWorld.class); public static void main(String[] args) { log.info("Hello World"); } } ``` Run the application (the play green button on line 9 in the screenshot below) and verify that it runs and prints the message, and exits with code `0`. This means that your Java application has run successfully. Expand the 'External Libraries' on the Project panel and verify that it displays the dependencies that we added for the project in `pom.xml`. ![Screenshot from IntelliJ showing that we have created a sample Kafka Maven Java project that includes all required dependencies.](https://www.conduktor.io/assets/kafka/image--45-.png) All good! We have created a sample Java project that includes all the needed dependencies. This will form the basis for creating Java producers and consumers next. > **See it in practice with Conduktor** > Once your project is set up, use [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-creating-a-kafka-java-project-using-maven-pom-xml) to create topics, inspect messages, and test your Kafka applications during development. ## Next steps - [Set up with Gradle](https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-gradle-build-gradle) to see the alternative build tool - [Create a Kafka producer](https://www.conduktor.io/kafka/complete-kafka-producer-with-java) to send messages - [Create a Kafka consumer](https://www.conduktor.io/kafka/complete-kafka-consumer-with-java) to read messages Source: https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-maven-pom-xml --- # Delivery semantics for Kafka consumers *Learn how offset commit strategies affect message delivery guarantees* A consumer reading from a Kafka partition may choose when to commit offsets. That decision controls whether messages are skipped or read twice after a consumer restart. **What you'll learn:** - The three delivery semantics: at-most-once, at-least-once, exactly-once - How to implement each strategy - When to use each approach - Best practices for production systems ## Delivery semantics overview ![Comparison of the three delivery semantics: at-most-once commits the offset before processing so a failure loses the message, at-least-once processes before committing so a crash reprocesses the message, and exactly-once processes and commits in a single atomic transaction](https://www.conduktor.io/assets/kafka/diagrams/delivery-semantics-for-kafka-consumers.svg) ```mermaid flowchart TD subgraph AMO["At most once"] A1["Receive message"] --> A2["Commit offset"] A2 --> A3["Process message"] A3 -->|"If fails"| A4["Message LOST"] end subgraph ALO["At Least Once"] B1["Receive message"] --> B2["Process message"] B2 --> B3["Commit offset"] B3 -->|"If crash before commit"| B4["Message REPROCESSED"] end subgraph EO["Exactly Once"] C1["Receive message"] --> C2["Process + Commit
in transaction"] C2 -->|"Atomic"| C3["Message processed
exactly once"] end ``` ## At most once delivery In this case, offsets are committed as soon as a message batch is received after calling `poll()`. If the subsequent processing fails, the message will be lost. It will not be read again as the offsets of those messages have been committed already. ![Kafka Consumer Delivery Semantics At Most Once](https://www.conduktor.io/assets/kafka/Adv-Delivery-Semantics-for-Consumers-1-2x.png) ```java // At most once: commit before processing Properties props = new Properties(); props.put("enable.auto.commit", "true"); props.put("auto.commit.interval.ms", "0"); // Commit immediately while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); consumer.commitSync(); // Commit first for (ConsumerRecord record : records) { process(record); // Then process - if this fails, message is lost } } ``` **When to use:** - Non-critical data (metrics, logs) - When message loss is acceptable - When processing duplicates is more problematic than losing data ## At least once delivery (usually preferred) In at-least-once delivery, every event from the source system will reach its destination, but sometimes retries will cause duplicates. Here, offsets are committed after the message is processed. > **Idempotent Processing** > Make sure your processing is idempotent (i.e. processing again the messages won't impact your systems) ![Kafka Consumer Delivery Semantics At Least Once](https://www.conduktor.io/assets/kafka/Adv-Delivery-Semantics-for-Consumers-2-2x.png) ```java // At least once: commit after processing Properties props = new Properties(); props.put("enable.auto.commit", "false"); while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord record : records) { process(record); // Process first } consumer.commitSync(); // Then commit - if crash before, messages reprocessed } ``` **When to use:** - Most production applications - When data loss is unacceptable - When you can handle duplicate processing ### Implement idempotent consumers | Strategy | How it works | Example | |----------|--------------|---------| | **Unique ID check** | Track processed message IDs | Store ID in database before processing | | **Upsert operations** | Use insert-or-update logic | Database upsert with message key | | **Conditional writes** | Only write if not exists | Check-then-write with version check | ```java // Idempotent processing example void processIdempotently(ConsumerRecord record) { String messageId = record.key(); // Check if already processed if (processedIds.contains(messageId)) { log.info("Skipping duplicate: {}", messageId); return; } // Process the message doProcessing(record); // Mark as processed processedIds.add(messageId); } ``` ## Exactly once delivery Some applications require exactly-once semantics. Each message is delivered exactly once. This may be achieved in certain situations if Kafka and the consumer application cooperate: - Achievable for Kafka topic to Kafka topic workflows using the transactions API - For Kafka topic to External System workflows, use an idempotent consumer ```java // Exactly once with Kafka Streams Properties props = new Properties(); props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2); // Or with producer transactions producer.initTransactions(); try { producer.beginTransaction(); // ... produce messages ... producer.sendOffsetsToTransaction(offsets, groupId); producer.commitTransaction(); } catch (Exception e) { producer.abortTransaction(); } ``` **When to use:** - Financial transactions - Kafka Streams applications - Critical data pipelines where duplicates cause problems ## Summary comparison | Semantic | Commits when | Risk | Complexity | Use case | |----------|--------------|------|------------|----------| | At most once | Before processing | Data loss | Low | Metrics, logs | | At least once | After processing | Duplicates | Low | Most applications | | Exactly once | With transaction | None (if possible) | High | Financial, critical | > **Bottom Line** > For most applications, use 'At Least Once' processing and ensure transformations are idempotent. ## Automatic offset committing strategy By default, consumers are configured with `enable.auto.commit=true` which means that offsets will be committed automatically on a schedule. This provides at-least-once delivery semantics. ```properties # Default auto-commit settings enable.auto.commit=true auto.commit.interval.ms=5000 # Commit every 5 seconds ``` > **Auto-commit timing** > With auto-commit, offsets are committed periodically regardless of processing status. If your application crashes after auto-commit but before processing completes, you may lose messages. ## Manual offset committing strategy You can also choose to control when offsets are committed by setting `enable.auto.commit=false` and using the `commitSync()` or `commitAsync()` methods to manually commit offsets. ```java // Synchronous commit - blocks until complete consumer.commitSync(); // Asynchronous commit - non-blocking with callback consumer.commitAsync((offsets, exception) -> { if (exception != null) { log.error("Commit failed", exception); } }); ``` ### Commit strategies comparison | Strategy | Latency | Reliability | Use case | |----------|---------|-------------|----------| | `commitSync()` | Higher | Guaranteed | Critical data | | `commitAsync()` | Lower | Best effort | High throughput | | Batch + sync | Balanced | Guaranteed | Most applications | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-delivery-semantics-for-kafka-consumers) lets you monitor consumer offsets and lag per partition. Track commit progress and identify processing delays to validate your delivery semantics implementation. ## Next steps - [Configure consumer settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior) for optimal performance - [Configure auto offset reset](https://www.conduktor.io/kafka/consumer-auto-offsets-reset-behavior) for new consumers - [Write a Java consumer](https://www.conduktor.io/kafka/complete-kafka-consumer-with-java) with hands-on code Source: https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers --- # How to change a Kafka topic configuration using the CLI *Learn how to view and modify topic configurations using kafka-configs* While broker-level defaults work for most topics, some require customization. The `kafka-configs` CLI tool lets you override configurations for specific topics without affecting the cluster defaults. **What you'll learn:** - How to view current topic configurations - How to add configuration overrides with kafka-configs - How to remove configuration overrides - Common configurations that need per-topic tuning ## Common topic configuration overrides | Configuration | Default | Description | |---------------|---------|-------------| | `min.insync.replicas` | 1 | Minimum ISRs for acks=all | | `retention.ms` | 7 days | Data retention time | | `retention.bytes` | -1 (unlimited) | Maximum partition size | | `cleanup.policy` | delete | delete or compact | | `max.message.bytes` | 1 MB | Maximum message size | | `compression.type` | producer | Broker-side compression | The complete list of topic configurations can be found in the [Kafka documentation](https://kafka.apache.org/documentation/#topicconfigs). ## Override topic configuration defaults ![Broker default min.insync.replicas=1 overridden at the topic level to 2 with kafka-configs --add-config and reverted with --delete-config](https://www.conduktor.io/assets/kafka/diagrams/how-to-change-a-kafka-topic-configuration-using-the-cli.svg) ```mermaid flowchart LR subgraph Broker["Broker Default"] BD["min.insync.replicas=1"] end subgraph Topic["Topic Override"] TO["min.insync.replicas=2"] end BD -->|"kafka-configs --add-config"| TO TO -->|"kafka-configs --delete-config"| BD ``` The `kafka-configs` CLI tool allows you to set, view, and delete configurations for topics, brokers, and other entities without restarting the cluster. Use CLI commands with appropriate extensions for your platform (e.g., `kafka-configs.bat` for Windows, `kafka-configs.sh` for Linux). Let's change the `min.insync.replicas` configuration of a topic. The default broker value is `1`, but for production with replication factor of 3, a value of `2` is recommended. See [min.insync.replicas](https://www.conduktor.io/kafka/kafka-topic-configuration-min-insync-replicas) for details. Before running Kafka CLIs make sure that you have [started Kafka](https://www.conduktor.io/kafka/starting-kafka) successfully. First, create a topic named configured-topic with 3 partitions and a replication factor of 1, using Kafka topics CLI, `kafka-topics` ```bash kafka-topics --bootstrap-server localhost:9092 --create --topic configured-topic --partitions 3 --replication-factor 1 ``` Describe the topic to check if there are any configuration override set for this topic. ```bash kafka-topics --bootstrap-server localhost:9092 --describe --topic configured-topic ``` ``` Topic: configured-topic TopicId: CDU7SBxBQ1mzJGnuH68-cQ PartitionCount: 3 ReplicationFactor: 1 Configs: Topic: configured-topic Partition: 0 Leader: 2 Replicas: 2 Isr: 2 Topic: configured-topic Partition: 1 Leader: 3 Replicas: 3 Isr: 3 Topic: configured-topic Partition: 2 Leader: 1 Replicas: 1 Isr: 1 ``` There is no configuration override set. Set the `min.insync.replicas` value for the topic `configured-topic` to 2 ```bash kafka-configs --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name configured-topic --add-config min.insync.replicas=2 ``` And describe the topic again ```bash kafka-topics --bootstrap-server localhost:9092 --describe --topic configured-topic ``` ``` Topic: configured-topic TopicId: CDU7SBxBQ1mzJGnuH68-cQ PartitionCount: 3 ReplicationFactor: 1 Configs: min.insync.replicas=2 Topic: configured-topic Partition: 0 Leader: 2 Replicas: 2 Isr: 2 Topic: configured-topic Partition: 1 Leader: 3 Replicas: 3 Isr: 3 Topic: configured-topic Partition: 2 Leader: 1 Replicas: 1 Isr: 1 ``` Now, you can see there is a topic configuration override set (at the right side of the output) - `min.insync.replicas=2`. You can delete the configuration override by passing `--delete-config` in place of the `--add-config` flag. ```bash kafka-configs --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name configured-topic --delete-config min.insync.replicas ``` Describe the topic to make sure the configuration override has been removed. ## Other common configuration changes ### Set retention to 30 days ```bash kafka-configs --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name configured-topic \ --add-config retention.ms=2592000000 ``` ### Enable log compaction ```bash kafka-configs --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name configured-topic \ --add-config cleanup.policy=compact ``` ### Set multiple configurations at once ```bash kafka-configs --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name configured-topic \ --add-config min.insync.replicas=2,retention.ms=604800000 ``` Some configuration changes take effect immediately, while others only apply to new data. Test configuration changes in a non-production environment first. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-change-a-kafka-topic-configuration-using-the-cli) provides a visual interface for viewing and modifying topic configurations. See all overrides at a glance and make changes without memorizing CLI syntax. ## Next steps - [Send large messages](https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka) to raise size limits via config - [Configure min.insync.replicas](https://www.conduktor.io/kafka/kafka-topic-configuration-min-insync-replicas) for durability - [Set up log retention](https://www.conduktor.io/kafka/kafka-topic-configuration-log-retention) for data lifecycle Source: https://www.conduktor.io/kafka/how-to-change-a-kafka-topic-configuration-using-the-cli --- # How to install Apache Kafka on Linux without ZooKeeper (KRaft mode) *Install and run Kafka in KRaft mode on Linux* KRaft mode runs Kafka without ZooKeeper, simplifying deployment and reducing resource requirements. This guide walks you through setting up a single-node KRaft cluster. **What you'll learn:** - How to install Java 11 (required dependency) - How to generate a cluster ID and format storage - How to start Kafka in KRaft mode - How to configure your PATH for CLI access > **KRaft vs ZooKeeper** > KRaft mode became production-ready in Kafka 3.3. For maximum compatibility with older tutorials and tools, see the [Linux ZooKeeper installation guide](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux). ## Installation overview ![Linux Kafka KRaft install flow: install Java 11, download Kafka, generate cluster ID, format storage, start Kafka](https://www.conduktor.io/assets/kafka/diagrams/how-to-install-apache-kafka-on-linux-without-zookeeper-kraft-mode.svg) ```mermaid flowchart LR A[Install Java 11] --> B[Download Kafka] B --> C[Generate cluster ID] C --> D[Format storage] D --> E[Start Kafka] ``` ## Step 1: Install Java JDK 11 Kafka requires Java 11 or later. **For Ubuntu/Debian:** ```bash wget -O- https://apt.corretto.aws/corretto.key | sudo apt-key add - sudo add-apt-repository 'deb https://apt.corretto.aws stable main' sudo apt-get update sudo apt-get install -y java-11-amazon-corretto-jdk ``` **For other distributions:** Visit the [Amazon Corretto 11 Linux install page](https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/linux-info.html) for RPM, Alpine, and other packages. Verify the installation: ```bash java -version ``` Expected output: ``` openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment Corretto-11.0.10.9.1 (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM Corretto-11.0.10.9.1 (build 11.0.10+9-LTS, mixed mode) ``` ## Step 2: Download and extract Kafka ```bash wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.13-3.0.0.tgz tar -xzf kafka_2.13-3.0.0.tgz mv kafka_2.13-3.0.0 ~ ``` Or download manually from [kafka.apache.org/downloads](https://kafka.apache.org/downloads). ## Step 3: Generate cluster ID KRaft clusters require a unique identifier: ```bash ~/kafka_2.13-3.0.0/bin/kafka-storage.sh random-uuid ``` This returns a UUID like `76BLQI7sT_ql1mBfKsOk9Q`. Save this value. ## Step 4: Format storage Format the log directory using your cluster ID (replace `` with your generated UUID): ```bash ~/kafka_2.13-3.0.0/bin/kafka-storage.sh format \ -t \ -c ~/kafka_2.13-3.0.0/config/kraft/server.properties ``` This formats the directory specified in `log.dirs` (default: `/tmp/kraft-combined-logs`). ## Step 5: Start Kafka ```bash ~/kafka_2.13-3.0.0/bin/kafka-server-start.sh ~/kafka_2.13-3.0.0/config/kraft/server.properties ``` Keep this terminal window open. Kafka is now running at `localhost:9092` in KRaft mode. ## Step 6: Configure PATH Add Kafka binaries to your PATH for convenient access. Edit your shell configuration file (`~/.bashrc` or `~/.zshrc`): ```bash export PATH="$PATH:$HOME/kafka_2.13-3.0.0/bin" ``` Reload your shell: ```bash source ~/.bashrc ``` Verify the setup: ```bash kafka-topics.sh --version ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-install-apache-kafka-on-linux-without-zookeeper-kraft-mode) can connect to your KRaft cluster at `localhost:9092` for visual topic management. ## Next steps - [Install on Windows](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows) to set up Kafka on Windows via WSL2 - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to create topics and produce messages - [KRaft mode concepts](https://www.conduktor.io/kafka/kafka-kraft-mode) for deeper understanding Source: https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux-without-zookeeper-kraft-mode --- # How to install Apache Kafka on Linux *Install and run Kafka on Linux* This guide walks you through installing Apache Kafka with ZooKeeper on Linux. By the end, you'll have a working single-node Kafka cluster for development. **What you'll learn:** - How to install Java 11 (required dependency) - How to download and configure Kafka - How to start ZooKeeper and Kafka - How to set up your PATH for CLI access > **ZooKeeper vs KRaft** > This guide uses ZooKeeper mode for maximum compatibility. For KRaft mode (without ZooKeeper), see the [Linux KRaft installation guide](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux-without-zookeeper-kraft-mode). ## Installation overview ![Linux Kafka install flow: install Java 11, download Kafka, start ZooKeeper, start Kafka, configure PATH](https://www.conduktor.io/assets/kafka/diagrams/how-to-install-apache-kafka-on-linux.svg) ```mermaid flowchart LR A[Install Java 11] --> B[Download Kafka] B --> C[Start ZooKeeper] C --> D[Start Kafka] D --> E[Configure PATH] ``` ## Step 1: Install Java JDK 11 Kafka requires Java 11 or later. **For Ubuntu/Debian:** ```bash wget -O- https://apt.corretto.aws/corretto.key | sudo apt-key add - sudo add-apt-repository 'deb https://apt.corretto.aws stable main' sudo apt-get update sudo apt-get install -y java-11-amazon-corretto-jdk ``` **For other distributions:** Visit the [Amazon Corretto 11 Linux install page](https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/linux-info.html) for RPM, Alpine, and other packages. Verify the installation: ```bash java -version ``` Expected output: ``` openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment Corretto-11.0.10.9.1 (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM Corretto-11.0.10.9.1 (build 11.0.10+9-LTS, mixed mode) ``` ## Step 2: Download and extract Kafka ```bash wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.13-3.0.0.tgz tar -xzf kafka_2.13-3.0.0.tgz mv kafka_2.13-3.0.0 ~ ``` Or download manually from [kafka.apache.org/downloads](https://kafka.apache.org/downloads) (choose the latest binary with Scala 2.13). ## Step 3: Start ZooKeeper ZooKeeper has to be running before Kafka starts. Open a terminal and run: ```bash ~/kafka_2.13-3.0.0/bin/zookeeper-server-start.sh ~/kafka_2.13-3.0.0/config/zookeeper.properties ``` > Add the `-daemon` flag to run ZooKeeper in the background. Keep this terminal window open. ## Step 4: Start Kafka Open a **new terminal window** and run: ```bash ~/kafka_2.13-3.0.0/bin/kafka-server-start.sh ~/kafka_2.13-3.0.0/config/server.properties ``` Keep this terminal window open. Kafka is now running at `localhost:9092`. ## Step 5: Configure PATH Add Kafka binaries to your PATH for convenient access. Edit your shell configuration file (`~/.bashrc` or `~/.zshrc`): ```bash export PATH="$PATH:$HOME/kafka_2.13-3.0.0/bin" ``` Reload your shell: ```bash source ~/.bashrc ``` Verify the setup: ```bash kafka-topics.sh --version ``` ## Optional: Change data directories By default, Kafka stores data in `/tmp`, which may be cleared on restart. **Change ZooKeeper data directory:** Edit `~/kafka_2.13-3.0.0/config/zookeeper.properties`: ``` dataDir=/your/path/to/data/zookeeper ``` **Change Kafka data directory:** Edit `~/kafka_2.13-3.0.0/config/server.properties`: ``` log.dirs=/your/path/to/data/kafka ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-install-apache-kafka-on-linux) can connect to your local Kafka cluster at `localhost:9092` for visual topic management. ## Next steps - [Linux KRaft installation](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux-without-zookeeper-kraft-mode) for a ZooKeeper-free setup - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to create topics and produce messages - [Docker setup](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) for containerized Kafka Source: https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux --- # How to install Apache Kafka on Mac with Homebrew *Install and run Kafka on macOS with Homebrew* Homebrew provides a quick way to install Kafka on macOS. It handles Java dependencies automatically and places binaries in your PATH. **What you'll learn:** - How to install Kafka using Homebrew - How to start ZooKeeper and Kafka - How to locate configuration files - Differences between Intel and Apple Silicon paths > **Homebrew vs native installation** > Homebrew is convenient but provides less control over versions. For development with specific Kafka versions, consider the [native Mac installation](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac). ## Installation overview ![Mac Kafka Homebrew install flow: install Homebrew, install Kafka, start ZooKeeper, start Kafka](https://www.conduktor.io/assets/kafka/diagrams/how-to-install-apache-kafka-on-mac-with-homebrew.svg) ```mermaid flowchart LR A[Install Homebrew] --> B[Install Kafka] B --> C[Start ZooKeeper] C --> D[Start Kafka] ``` ## Step 1: Install Homebrew If you don't have Homebrew installed: ```bash /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` ## Step 2: Install Kafka Homebrew automatically installs Java as a dependency: ```bash brew install kafka ``` ## Installation paths Homebrew installs to different locations based on your chip architecture: | Component | Intel Mac | Apple Silicon Mac | |-----------|-----------|-------------------| | Binaries | `/usr/local/bin` | `/opt/homebrew/bin` | | Kafka config | `/usr/local/etc/kafka` | `/opt/homebrew/etc/kafka` | | ZooKeeper config | `/usr/local/etc/zookeeper` | `/opt/homebrew/etc/zookeeper` | | Kafka data | `/usr/local/var/lib/kafka-logs` | `/opt/homebrew/var/lib/kafka-logs` | > **Apple Silicon users** > Replace `/usr/local/` with `/opt/homebrew/` in all commands below if you have an M1, M2, or M3 Mac. ## Step 3: Start ZooKeeper ZooKeeper has to be running before Kafka starts: ```bash /usr/local/bin/zookeeper-server-start /usr/local/etc/zookeeper/zoo.cfg ``` Keep this terminal window open. ## Step 4: Start Kafka Open a **new terminal window** and start Kafka: ```bash /usr/local/bin/kafka-server-start /usr/local/etc/kafka/server.properties ``` Keep this terminal window open. Kafka is now running at `localhost:9092`. ## Optional: Change data directories **Change ZooKeeper data directory:** Edit `/usr/local/etc/zookeeper/zoo.cfg`: ``` dataDir=/your/path/to/data/zookeeper ``` **Change Kafka data directory:** Edit `/usr/local/etc/kafka/server.properties`: ``` log.dirs=/your/path/to/data/kafka ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-install-apache-kafka-on-mac-with-homebrew) can connect to your local Kafka cluster at `localhost:9092` for visual topic management. ## Next steps - [Install on Linux](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux) to set up Kafka on a Linux machine or server - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to create topics and produce messages - [Native Mac installation](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac) for more control over versions Source: https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-with-homebrew --- # How to install Apache Kafka on Mac without ZooKeeper (KRaft mode) *Install and run Kafka in KRaft mode on macOS* KRaft mode runs Kafka without ZooKeeper, simplifying deployment and reducing resource requirements. This guide walks you through setting up a single-node KRaft cluster. **What you'll learn:** - How to install Java 11 (required dependency) - How to generate a cluster ID and format storage - How to start Kafka in KRaft mode - How to configure your PATH for CLI access > **KRaft vs ZooKeeper** > KRaft mode became production-ready in Kafka 3.3. For maximum compatibility with older tutorials and tools, see the [Mac ZooKeeper installation guide](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac). ## Installation overview ![Mac Kafka KRaft install flow: install Java 11, download Kafka, generate cluster ID, format storage, start Kafka](https://www.conduktor.io/assets/kafka/diagrams/how-to-install-apache-kafka-on-mac-without-zookeeper-kraft-mode.svg) ```mermaid flowchart LR A[Install Java 11] --> B[Download Kafka] B --> C[Generate cluster ID] C --> D[Format storage] D --> E[Start Kafka] ``` ## Step 1: Install Java JDK 11 Kafka requires Java 11 or later. 1. Download [Amazon Corretto 11](https://corretto.aws/downloads/latest/amazon-corretto-11-x64-macos-jdk.pkg) (free OpenJDK distribution) 2. Double-click the downloaded `.pkg` file 3. Follow the installation wizard Verify the installation: ```bash java -version ``` Expected output: ``` openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment Corretto-11.0.10.9.1 (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM Corretto-11.0.10.9.1 (build 11.0.10+9-LTS, mixed mode) ``` ## Step 2: Download and extract Kafka Download from [kafka.apache.org/downloads](https://kafka.apache.org/downloads) (choose the latest binary with Scala 2.13) and extract: ```bash cd ~/Downloads tar -xzf kafka_2.13-3.0.0.tgz mv kafka_2.13-3.0.0 ~/ ``` ## Step 3: Generate cluster ID KRaft clusters require a unique identifier: ```bash ~/kafka_2.13-3.0.0/bin/kafka-storage.sh random-uuid ``` This returns a UUID like `76BLQI7sT_ql1mBfKsOk9Q`. Save this value. ## Step 4: Format storage Format the log directory using your cluster ID (replace `` with your generated UUID): ```bash ~/kafka_2.13-3.0.0/bin/kafka-storage.sh format \ -t \ -c ~/kafka_2.13-3.0.0/config/kraft/server.properties ``` This formats the directory specified in `log.dirs` (default: `/tmp/kraft-combined-logs`). ## Step 5: Start Kafka ```bash ~/kafka_2.13-3.0.0/bin/kafka-server-start.sh ~/kafka_2.13-3.0.0/config/kraft/server.properties ``` Keep this terminal window open. Kafka is now running at `localhost:9092` in KRaft mode. ## Step 6: Configure PATH Add Kafka binaries to your PATH for convenient access. Edit your shell configuration file (`~/.zshrc` for zsh or `~/.bashrc` for bash): ```bash export PATH="$PATH:$HOME/kafka_2.13-3.0.0/bin" ``` Reload your shell: ```bash source ~/.zshrc ``` Verify the setup: ```bash kafka-topics.sh --version ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-install-apache-kafka-on-mac-without-zookeeper-kraft-mode) can connect to your KRaft cluster at `localhost:9092` for visual topic management. ## Next steps - [Mac Homebrew installation](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-with-homebrew) for a one-command setup with managed services - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to create topics and produce messages - [KRaft mode concepts](https://www.conduktor.io/kafka/kafka-kraft-mode) for deeper understanding Source: https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-without-zookeeper-kraft-mode --- # How to install Apache Kafka on Mac *Install and run Kafka on macOS* This guide walks you through installing Apache Kafka with ZooKeeper on macOS. By the end, you'll have a working single-node Kafka cluster for development. **What you'll learn:** - How to install Java 11 (required dependency) - How to download and configure Kafka - How to start ZooKeeper and Kafka - How to set up your PATH for CLI access > **ZooKeeper vs KRaft** > This guide uses ZooKeeper mode for maximum compatibility. For KRaft mode (without ZooKeeper), see the [Mac KRaft installation guide](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-without-zookeeper-kraft-mode). ## Installation overview ![Mac Kafka install flow: install Java 11, download Kafka, start ZooKeeper, start Kafka, configure PATH](https://www.conduktor.io/assets/kafka/diagrams/how-to-install-apache-kafka-on-mac.svg) ```mermaid flowchart LR A[Install Java 11] --> B[Download Kafka] B --> C[Start ZooKeeper] C --> D[Start Kafka] D --> E[Configure PATH] ``` ## Step 1: Install Java JDK 11 Kafka requires Java 11 or later. 1. Download [Amazon Corretto 11](https://corretto.aws/downloads/latest/amazon-corretto-11-x64-macos-jdk.pkg) (free OpenJDK distribution) 2. Double-click the downloaded `.pkg` file 3. Follow the installation wizard Verify the installation: ```bash java -version ``` Expected output: ``` openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment Corretto-11.0.10.9.1 (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM Corretto-11.0.10.9.1 (build 11.0.10+9-LTS, mixed mode) ``` > If you have multiple Java versions installed, see [how to set the default Java version on Mac](https://stackoverflow.com/a/24657630). ## Step 2: Download and extract Kafka 1. Download Kafka from [kafka.apache.org/downloads](https://kafka.apache.org/downloads) (choose the latest binary with Scala 2.13) 2. Extract the archive to your home directory ```bash cd ~/Downloads tar -xzf kafka_2.13-3.0.0.tgz mv kafka_2.13-3.0.0 ~/ ``` ## Step 3: Start ZooKeeper ZooKeeper has to be running before Kafka starts. Open a terminal and run: ```bash ~/kafka_2.13-3.0.0/bin/zookeeper-server-start.sh ~/kafka_2.13-3.0.0/config/zookeeper.properties ``` > Add the `-daemon` flag to run ZooKeeper in the background. Keep this terminal window open. ## Step 4: Start Kafka Open a **new terminal window** and run: ```bash ~/kafka_2.13-3.0.0/bin/kafka-server-start.sh ~/kafka_2.13-3.0.0/config/server.properties ``` Keep this terminal window open. Kafka is now running at `localhost:9092`. ## Step 5: Configure PATH Add Kafka binaries to your PATH for convenient access. Edit your shell configuration file (`~/.zshrc` for zsh or `~/.bashrc` for bash): ```bash export PATH="$PATH:$HOME/kafka_2.13-3.0.0/bin" ``` Reload your shell: ```bash source ~/.zshrc ``` Verify the setup: ```bash kafka-topics.sh --version ``` ## Optional: Change data directories By default, Kafka stores data in `/tmp`, which may be cleared on restart. **Change ZooKeeper data directory:** Edit `~/kafka_2.13-3.0.0/config/zookeeper.properties`: ``` dataDir=/your/path/to/data/zookeeper ``` **Change Kafka data directory:** Edit `~/kafka_2.13-3.0.0/config/server.properties`: ``` log.dirs=/your/path/to/data/kafka ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-install-apache-kafka-on-mac) can connect to your local Kafka cluster at `localhost:9092` for visual topic management. ## Next steps - [Mac KRaft installation](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-without-zookeeper-kraft-mode) for a ZooKeeper-free setup - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to create topics and produce messages - [Mac Homebrew installation](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-with-homebrew) for an alternative setup Source: https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac --- # How to install Apache Kafka on Windows without ZooKeeper (KRaft mode) *Install and run Kafka in KRaft mode on Windows* KRaft mode runs Kafka without ZooKeeper, simplifying deployment and reducing resource requirements. This guide walks you through setting up a single-node KRaft cluster using WSL2. **What you'll learn:** - How to set up WSL2 on Windows - How to generate a cluster ID and format storage - How to start Kafka in KRaft mode - How to configure your PATH for CLI access > **Native Windows is not recommended** > Kafka has known issues when running directly on Windows due to missing POSIX features. Always use WSL2 or [Docker](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) for Windows installations. KRaft mode became production-ready in Kafka 3.3. For maximum compatibility with older tutorials and tools, see the [Windows ZooKeeper installation guide](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows). ## Installation overview ![Windows Kafka KRaft install flow: install WSL2, install Java 11, download Kafka, disable IPv6, generate cluster ID, format storage, start Kafka](https://www.conduktor.io/assets/kafka/diagrams/how-to-install-apache-kafka-on-windows-without-zookeeper-kraft-mode.svg) ```mermaid flowchart LR A[Install WSL2] --> B[Install Java 11] B --> C[Download Kafka] C --> D[Disable IPv6] D --> E[Generate cluster ID] E --> F[Format storage] F --> G[Start Kafka] ``` ## Step 1: Install WSL2 WSL2 provides a Linux environment on Windows without a virtual machine. Open **PowerShell as Administrator** and run: ```powershell wsl --install ``` This installs Ubuntu by default. Restart your computer when prompted. After restart, open Ubuntu from the Start menu and create a Linux username and password. > For troubleshooting WSL2 issues, see the [Microsoft troubleshooting guide](https://docs.microsoft.com/en-us/windows/wsl/troubleshooting). ## Step 2: Disable IPv6 on WSL2 WSL2 has a [networking issue](https://github.com/microsoft/WSL/issues/4851) that prevents external programs from connecting to Kafka. Disable IPv6 to fix this: ```bash sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1 sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1 ``` ## Step 3: Install Java JDK 11 In your WSL2 Ubuntu terminal: ```bash wget -O- https://apt.corretto.aws/corretto.key | sudo apt-key add - sudo add-apt-repository 'deb https://apt.corretto.aws stable main' sudo apt-get update sudo apt-get install -y java-11-amazon-corretto-jdk ``` Verify the installation: ```bash java -version ``` Expected output: ``` openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment Corretto-11.0.10.9.1 (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM Corretto-11.0.10.9.1 (build 11.0.10+9-LTS, mixed mode) ``` ## Step 4: Download and extract Kafka ```bash wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.13-3.0.0.tgz tar -xzf kafka_2.13-3.0.0.tgz mv kafka_2.13-3.0.0 ~ ``` Or download manually from [kafka.apache.org/downloads](https://kafka.apache.org/downloads). ## Step 5: Generate cluster ID KRaft clusters require a unique identifier: ```bash ~/kafka_2.13-3.0.0/bin/kafka-storage.sh random-uuid ``` This returns a UUID like `76BLQI7sT_ql1mBfKsOk9Q`. Save this value. ## Step 6: Format storage Format the log directory using your cluster ID (replace `` with your generated UUID): ```bash ~/kafka_2.13-3.0.0/bin/kafka-storage.sh format \ -t \ -c ~/kafka_2.13-3.0.0/config/kraft/server.properties ``` This formats the directory specified in `log.dirs` (default: `/tmp/kraft-combined-logs`). ## Step 7: Start Kafka ```bash ~/kafka_2.13-3.0.0/bin/kafka-server-start.sh ~/kafka_2.13-3.0.0/config/kraft/server.properties ``` Keep this terminal window open. Kafka is now running at `localhost:9092` in KRaft mode. ## Step 8: Configure PATH Add Kafka binaries to your PATH: ```bash echo 'export PATH="$PATH:$HOME/kafka_2.13-3.0.0/bin"' >> ~/.bashrc source ~/.bashrc ``` Verify the setup: ```bash kafka-topics.sh --version ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-install-apache-kafka-on-windows-without-zookeeper-kraft-mode) can connect to your WSL2 KRaft cluster at `localhost:9092` from Windows for visual topic management. ## Next steps - [Docker setup](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) for the most portable cross-platform Kafka - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to create topics and produce messages - [KRaft mode concepts](https://www.conduktor.io/kafka/kafka-kraft-mode) for deeper understanding Source: https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows-without-zookeeper-kraft-mode --- # How to install Apache Kafka on Windows *Install and run Kafka on Windows* This guide walks you through installing Apache Kafka with ZooKeeper on Windows using WSL2 (Windows Subsystem for Linux). By the end, you'll have a working single-node Kafka cluster for development. **What you'll learn:** - How to set up WSL2 on Windows - How to install Java 11 (required dependency) - How to download and configure Kafka - How to start ZooKeeper and Kafka > **Native Windows is not recommended** > Kafka has known issues when running directly on Windows due to missing POSIX features. Always use WSL2 or [Docker](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) for Windows installations. This guide uses ZooKeeper mode for maximum compatibility. For KRaft mode (without ZooKeeper), see the [Windows KRaft installation guide](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows-without-zookeeper-kraft-mode). ## Installation overview ![Windows Kafka install flow: install WSL2, install Java 11, download Kafka, disable IPv6, start ZooKeeper, start Kafka](https://www.conduktor.io/assets/kafka/diagrams/how-to-install-apache-kafka-on-windows.svg) ```mermaid flowchart LR A[Install WSL2] --> B[Install Java 11] B --> C[Download Kafka] C --> D[Disable IPv6] D --> E[Start ZooKeeper] E --> F[Start Kafka] ``` ## Step 1: Install WSL2 WSL2 provides a Linux environment on Windows without a virtual machine. Open **PowerShell as Administrator** and run: ```powershell wsl --install ``` This installs Ubuntu by default. Restart your computer when prompted. After restart, open Ubuntu from the Start menu and create a Linux username and password. > For troubleshooting WSL2 issues, see the [Microsoft troubleshooting guide](https://docs.microsoft.com/en-us/windows/wsl/troubleshooting). ## Step 2: Disable IPv6 on WSL2 WSL2 has a [networking issue](https://github.com/microsoft/WSL/issues/4851) that prevents external programs from connecting to Kafka. Disable IPv6 to fix this: ```bash sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1 sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1 ``` ## Step 3: Install Java JDK 11 In your WSL2 Ubuntu terminal: ```bash wget -O- https://apt.corretto.aws/corretto.key | sudo apt-key add - sudo add-apt-repository 'deb https://apt.corretto.aws stable main' sudo apt-get update sudo apt-get install -y java-11-amazon-corretto-jdk ``` Verify the installation: ```bash java -version ``` Expected output: ``` openjdk version "11.0.10" 2021-01-19 LTS OpenJDK Runtime Environment Corretto-11.0.10.9.1 (build 11.0.10+9-LTS) OpenJDK 64-Bit Server VM Corretto-11.0.10.9.1 (build 11.0.10+9-LTS, mixed mode) ``` ## Step 4: Download and extract Kafka ```bash wget https://archive.apache.org/dist/kafka/3.0.0/kafka_2.13-3.0.0.tgz tar -xzf kafka_2.13-3.0.0.tgz mv kafka_2.13-3.0.0 ~ ``` Or download manually from [kafka.apache.org/downloads](https://kafka.apache.org/downloads). ## Step 5: Start ZooKeeper ZooKeeper has to be running before Kafka starts. ```bash ~/kafka_2.13-3.0.0/bin/zookeeper-server-start.sh ~/kafka_2.13-3.0.0/config/zookeeper.properties ``` > Add the `-daemon` flag to run ZooKeeper in the background. Keep this terminal window open. ## Step 6: Start Kafka Open a **new WSL2 terminal window** and run: ```bash ~/kafka_2.13-3.0.0/bin/kafka-server-start.sh ~/kafka_2.13-3.0.0/config/server.properties ``` Keep this terminal window open. Kafka is now running at `localhost:9092`. ## Step 7: Configure PATH Add Kafka binaries to your PATH: ```bash echo 'export PATH="$PATH:$HOME/kafka_2.13-3.0.0/bin"' >> ~/.bashrc source ~/.bashrc ``` Verify the setup: ```bash kafka-topics.sh --version ``` ## Frequently asked questions **Can I run Kafka directly on Windows without WSL2?** Technically yes, but you will encounter issues. Kafka lacks support for certain Windows-specific behaviors, leading to problems when [deleting topics](https://issues.apache.org/jira/browse/KAFKA-8811) or during log segment rotation. **Can I use Windows Kafka binaries with WSL2 Kafka?** Yes. You can download Kafka binaries on Windows and use commands like `kafka-topics.bat` against your cluster running in WSL2. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-install-apache-kafka-on-windows) can connect to your WSL2 Kafka cluster at `localhost:9092` from Windows for visual topic management. ## Next steps - [Windows KRaft installation](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows-without-zookeeper-kraft-mode) for a ZooKeeper-free setup - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to create topics and produce messages - [Docker setup](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) for an alternative containerized setup Source: https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows --- # How to send large messages in Apache Kafka *Learn how to handle large messages in Kafka* Apache Kafka has default message size limits that can be configured to handle larger payloads, but there are important considerations and best practices to follow. **What you'll learn:** - Default message size limits across Kafka components - How to configure producers, brokers, and consumers for large messages - Performance implications of large messages - Alternative patterns for handling large payloads ## Default message size limits By default, Kafka has the following message size limits: | Component | Configuration | Default | |-----------|--------------|---------| | Producer | `max.request.size` | 1 MB | | Broker | `message.max.bytes` | 1 MB | | Topic | `max.message.bytes` | Inherits from broker | | Consumer | `max.partition.fetch.bytes` | 1 MB | ![Message size limits chained across Producer max.request.size, Broker message.max.bytes, Topic max.message.bytes, and Consumer max.partition.fetch.bytes, each stage at most the next](https://www.conduktor.io/assets/kafka/diagrams/how-to-send-large-messages-in-apache-kafka--1.svg) ```mermaid flowchart LR P["Producer
max.request.size"] -->|"≤ broker limit"| B["Broker
message.max.bytes"] B -->|"≤ topic limit"| T["Topic
max.message.bytes"] T -->|"≤ consumer limit"| C["Consumer
max.partition.fetch.bytes"] style P fill:#e1f5fe style C fill:#e8f5e9 ``` ## Configure Kafka for large messages To send messages larger than 1MB, you need to configure multiple components: ### Producer configuration ```properties # Set maximum request size for producer max.request.size=10485760 # 10MB # Increase buffer memory if needed buffer.memory=67108864 # 64MB ``` ### Broker configuration ```properties # Set maximum message size for broker message.max.bytes=10485760 # 10MB # Set maximum replica fetch size replica.fetch.max.bytes=10485760 # 10MB # Set maximum response size socket.receive.buffer.bytes=1048576 # 1MB socket.send.buffer.bytes=1048576 # 1MB ``` ### Topic configuration ```bash kafka-configs --bootstrap-server localhost:9092 \ --alter --entity-type topics --entity-name large-topic \ --add-config max.message.bytes=10485760 ``` ### Consumer configuration ```properties # Set maximum fetch size for consumer max.partition.fetch.bytes=10485760 # 10MB fetch.max.bytes=52428800 # 50MB ``` ## Performance implications Sending large messages in Kafka has several performance implications: | Area | Impact | Mitigation | |------|--------|------------| | Memory | Higher heap usage, GC pressure | Tune JVM heap sizes | | Network | More bandwidth, potential timeouts | Adjust buffer sizes | | Disk I/O | More operations, slower compaction | Use faster storage | | Throughput | Lower overall message rate | Enable compression | ## Alternative approaches Instead of sending large messages directly, consider these alternatives: ### 1. External storage pattern Store large payloads in external systems and send only references: ```json { "id": "message-123", "timestamp": "2023-01-01T00:00:00Z", "data_location": "s3://bucket/path/to/large-file.json", "metadata": { "size": 50000000, "checksum": "abc123" } } ``` **Benefits:** - Keeps Kafka messages small and fast - Allows for separate scaling of storage and messaging - Enables efficient caching strategies ### 2. Split messages Break large messages into smaller chunks: ```json { "message_id": "msg-123", "chunk_id": "chunk-1", "total_chunks": 5, "chunk_data": "...", "sequence": 1 } ``` **Benefits:** - Works within default Kafka limits - Allows for parallel processing - Provides better error recovery ### 3. Compression Enable compression to reduce message sizes: ```properties # Producer compression compression.type=snappy # or gzip, lz4, zstd ``` **Benefits:** - Reduces network bandwidth usage - Decreases storage requirements - Often improves throughput ## Best practices > **Recommendations for large messages** > 1. **Avoid large messages when possible** - Kafka is optimized for small, high-throughput messages > 2. **Use external storage** - Store large payloads externally and reference them in Kafka messages > 3. **Enable compression** - Always enable compression for large messages > 4. **Monitor memory usage** - Ensure adequate heap sizing for all components > 5. **Test thoroughly** - Verify performance impact in your specific environment ### Configuration checklist When configuring for large messages, ensure all these settings are aligned: - ✅ Producer `max.request.size` - ✅ Broker `message.max.bytes` - ✅ Topic `max.message.bytes` - ✅ Consumer `max.partition.fetch.bytes` - ✅ Consumer `fetch.max.bytes` - ✅ Broker `replica.fetch.max.bytes` ### Monitor large messages Monitor these metrics when working with large messages: - Memory usage on brokers, producers, and consumers - Network bandwidth utilization - Disk I/O patterns and latency - Garbage collection frequency and duration - Message throughput and latency ## Large message decision guide ![Decision tree for large payloads: compress first, else store externally (S3/blob), else chunk into split messages, else configure larger limits across all components](https://www.conduktor.io/assets/kafka/diagrams/how-to-send-large-messages-in-apache-kafka--2.svg) ```mermaid flowchart TD Start["Need to send
large payload?"] --> Q1{{"Can data be
compressed?"}} Q1 -->|"Yes"| Compress["Enable compression
(snappy, lz4, zstd)"] Q1 -->|"No"| Q2{{"Can data be
stored externally?"}} Compress --> Check{{"Still > 1MB?"}} Check -->|"No"| Done["Use default limits"] Check -->|"Yes"| Q2 Q2 -->|"Yes"| External["Use external storage pattern
(S3, blob storage)"] Q2 -->|"No"| Q3{{"Can data be
chunked?"}} Q3 -->|"Yes"| Chunk["Implement message splitting"] Q3 -->|"No"| Config["Configure larger limits
(all components)"] External --> Best["Best for large payloads"] Chunk --> Good["Good for moderate sizes"] Config --> Test["Test thoroughly"] ``` Large messages can significantly impact Kafka performance. Always test in a staging environment that mirrors your production setup before deploying large message configurations. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-send-large-messages-in-apache-kafka) lets you produce and consume messages while monitoring their sizes. Test your large message configurations and verify all component limits are aligned. ## Next steps - [Apply naming conventions](https://www.conduktor.io/kafka/kafka-topics-naming-convention) to keep large-payload topics organized - [Enable compression](https://www.conduktor.io/kafka/kafka-message-compression) to reduce message sizes - [Configure log segments](https://www.conduktor.io/kafka/kafka-topics-internals-segments-and-indexes) for storage optimization Source: https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka --- # How to start Kafka using Docker *Run Kafka on any operating system in five minutes* Docker provides the most portable way to run Kafka. It works identically on Mac, Windows, and Linux, and makes switching Kafka versions straightforward. **What you'll learn:** - How to run Kafka with Docker Compose - How to execute Kafka commands against a Docker cluster - How to manage your Docker-based Kafka environment ## Install Docker | Platform | Installation | |----------|--------------| | Mac | [Docker Desktop for Mac](https://docs.docker.com/desktop/mac/install/) | | Windows | [Docker Desktop for Windows](https://docs.docker.com/desktop/windows/install/) | | Linux | [Docker Engine](https://docs.docker.com/engine/install/ubuntu/) (choose your distro) | > Docker Desktop on Mac and Windows includes Docker Compose. On Linux, [install Docker Compose separately](https://docs.docker.com/compose/install/#install-compose). ## Run Kafka with Docker Compose Docker Compose runs multi-container applications. Each Kafka component (ZooKeeper, broker) runs in a separate container. **Step 1: Clone the repository** ```bash git clone https://github.com/conduktor/kafka-stack-docker-compose.git cd kafka-stack-docker-compose ``` **Step 2: Choose a configuration** | File | Configuration | |------|---------------| | `zk-single-kafka-single.yml` | One ZooKeeper, one broker (learning) | | `zk-single-kafka-multiple.yml` | One ZooKeeper, multiple brokers | | `zk-multiple-kafka-multiple.yml` | Multiple ZooKeeper, multiple brokers | **Step 3: Start the cluster** ```bash docker-compose -f zk-single-kafka-single.yml up -d ``` **Step 4: Verify the cluster** ```bash docker-compose -f zk-single-kafka-single.yml ps ``` Expected output: ``` Name Command State Ports ------------------------------------------------------------------------------------------- kafka1 /etc/confluent/docker/run Up 0.0.0.0:9092->9092/tcp, 0.0.0.0:9999->9999/tcp zoo1 /etc/confluent/docker/run Up 0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp ``` Kafka is available at `localhost:9092`. ## Run Kafka commands You have two options for running Kafka CLI commands: ### Option 1: Run inside the container ```bash docker exec -it kafka1 /bin/bash ``` Inside the container, run commands without the `.sh` extension: ```bash kafka-topics --version ``` ### Option 2: Run from your host machine Install the Kafka binaries on your system (skip the steps for starting ZooKeeper and Kafka): - [Mac](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac) (follow the whole document except starting Kafka and ZooKeeper) - [Linux](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux) (follow the whole document except starting Kafka and ZooKeeper) - [Windows](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows) (follow the whole document except starting Kafka and ZooKeeper) Then run commands with the `.sh` extension: ```bash kafka-topics.sh --bootstrap-server localhost:9092 --list ``` ## Stop and clean up **Stop containers** (preserves data): ```bash docker-compose -f zk-single-kafka-single.yml stop ``` **Remove containers and network** (removes data): ```bash docker-compose -f zk-single-kafka-single.yml down ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-how-to-start-kafka-using-docker) can connect to your Docker-based Kafka cluster at `localhost:9092` for visual management of topics and messages. ## Next steps - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to practice Kafka commands against your cluster - [Topics CLI tutorial](https://www.conduktor.io/kafka/kafka-topics-cli-tutorial) to create and manage your first topics - [Kafka fundamentals](https://www.conduktor.io/kafka/kafka-fundamentals) to understand core concepts Source: https://www.conduktor.io/kafka/how-to-start-kafka-using-docker --- # Start learning Kafka with Conduktor *Get a Kafka cluster running in minutes without local installation* Conduktor Console includes hosted Kafka clusters for learning and development. Create topics, produce messages, and explore Kafka concepts through a visual interface. ## Create a Kafka cluster 1. Go to [conduktor.io/get-started](https://conduktor.io/get-started) 2. Sign up or log in to Console 3. Create a new cluster from the dashboard 4. Your cluster is ready to use ## What you can do Once your cluster is running, use Console to: - **Create topics** and configure partitions and replication - **Produce messages** to topics and see them appear in real-time - **Consume messages** and browse topic contents - **Monitor consumer groups** and track offsets This hands-on exploration helps reinforce the concepts covered in the [fundamentals](https://www.conduktor.io/kafka/kafka-fundamentals) section. > Conduktor clusters are ideal for learning and prototyping. For production workloads, use a managed Kafka service or self-hosted cluster. ## Optional: Use CLI tools To follow the [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) with your Conduktor cluster, install the Kafka binaries locally: | Platform | Installation guide | |----------|-------------------| | Mac | [Install on Mac](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac) | | Linux | [Install on Linux](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux) | | Windows | [Install on Windows](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows) | Skip the steps for starting ZooKeeper and Kafka. You only need the binaries installed and `$PATH` configured. ## Next steps - [Install on Mac](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac) to run Kafka natively on macOS - [Docker setup](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) for local Kafka with version control - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to practice Kafka commands Source: https://www.conduktor.io/kafka/how-to-start-kafka-with-conduktor --- # Idempotent Kafka producer *Learn how idempotent producers prevent duplicate messages* Idempotent producers ensure that messages are not duplicated even when retries occur, providing exactly-once semantics for producer operations. This is essential for building reliable data pipelines where duplicates cause problems. **What you'll learn:** - How idempotent producers prevent duplicates - The mechanisms Kafka uses for deduplication - Configuration requirements and best practices - Limitations to be aware of ## What is producer idempotency? Producer idempotency means that sending the same message multiple times will result in exactly one copy of the message being written to the Kafka topic, even in the presence of failures and retries. ![Producer retries without idempotency create duplicates, while idempotency uses PID and sequence numbers to deduplicate on the broker](https://www.conduktor.io/assets/kafka/diagrams/idempotent-kafka-producer.svg) ```mermaid flowchart LR subgraph Without["Without idempotency"] P1["Producer"] -->|"Send"| B1["Broker"] B1 -->|"Network failure"| P1 P1 -->|"Retry"| B1 B1 --> D1["Duplicate!"] end subgraph With["With idempotency"] P2["Producer
PID=123, Seq=42"] -->|"Send"| B2["Broker"] B2 -->|"Network failure"| P2 P2 -->|"Retry (same Seq)"| B2 B2 -->|"Deduplicate"| OK["Single message"] end ``` ## Enable idempotent producers Idempotent producers are enabled by default in Kafka 3.0+. For older versions, enable explicitly: ```properties enable.idempotence=true ``` When idempotency is enabled, Kafka automatically sets these configurations: - `retries=Integer.MAX_VALUE` - `max.in.flight.requests.per.connection=5` - `acks=all` > **Default in Kafka 3.0+** > Idempotent producers are enabled by default in Kafka 3.0 and later versions. This provides better out-of-the-box reliability without requiring explicit configuration. ## How Kafka achieves idempotency Kafka uses two key mechanisms to ensure idempotency: ### 1. Producer ID (PID) Each producer instance gets a unique Producer ID from the broker: - Assigned when producer starts up - Valid for the lifetime of the producer session - Used to track message sequences ### 2. Sequence numbers Each message gets a sequence number per topic-partition: - Starts at 0 for each producer-topic-partition combination - Incremented for each message sent - Used by broker to detect duplicates ![Kafka Idempotent Producer Sequence Numbers](https://www.conduktor.io/assets/kafka/Adv-Idempotent-Producer-1.png) ## How deduplication works When a broker receives a message, it checks: | Scenario | Action | |----------|--------| | Expected sequence | Message is written normally | | Duplicate sequence | Message is discarded, success response sent | | Out-of-order sequence | `OutOfOrderSequenceException` thrown | ``` Broker state: Producer 123, Partition 0, Last sequence: 42 Incoming message: Sequence 43 ✅ Accept Incoming message: Sequence 42 ⚠️ Duplicate (ignore) Incoming message: Sequence 45 ❌ Out of order (reject) ``` ## Configuration requirements ### Required settings ```properties enable.idempotence=true acks=all # Automatically set retries=Integer.MAX_VALUE # Automatically set max.in.flight.requests.per.connection=5 # Max value for idempotency ``` ### Recommended production configuration ```properties # Complete idempotent producer configuration enable.idempotence=true acks=all retries=Integer.MAX_VALUE max.in.flight.requests.per.connection=5 delivery.timeout.ms=120000 compression.type=snappy batch.size=32768 linger.ms=5 ``` ## Performance implications | Configuration | Throughput | Latency | Duplicates | |---------------|------------|---------|------------| | No idempotency, retries=0 | Highest | Lowest | None (data loss possible) | | No idempotency, retries>0 | High | Medium | Possible | | Idempotent producer | Medium-High | Medium | None | ### Trade-offs **Benefits:** - Exactly-once semantics for producer - Simplified error handling - Better reliability **Costs:** - Memory overhead on broker for sequence state - Slightly higher latency for sequence checking - Max 5 in-flight requests per connection ## Error handling ### Retriable errors (automatic) Idempotent producers automatically retry: - `TimeoutException` - `RetriableException` - Network connectivity issues - Broker leadership changes ### Non-retriable errors (require handling) ```java try { producer.send(record).get(); } catch (OutOfOrderSequenceException e) { // Sequence numbers are wrong - producer is in bad state producer.close(); // Create new producer } catch (UnknownProducerIdException e) { // Producer ID expired - recreate producer producer.close(); } ``` ## Limitations > **Producer restarts** > If a producer application restarts, it will get a new Producer ID and sequence numbers reset to 0. This means potential duplicates across application restarts, even with idempotency enabled. | Limitation | Description | |------------|-------------| | **Session-based** | Idempotency only guaranteed within single producer session | | **Partition scope** | No deduplication across different partitions | | **Topic scope** | No deduplication across different topics | | **Memory** | Brokers maintain state per producer-partition | ## When to use idempotent producers | Use case | Recommendation | |----------|----------------| | Production applications | Always recommended | | Financial data | Essential | | Audit logs | Essential | | Metrics/logs | Recommended | | Development/testing | Optional | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-idempotent-kafka-producer) lets you monitor topic messages and verify no duplicates are written. Use the message browser to inspect message headers and validate your idempotent producer configuration. ## Next steps - [Configure message compression](https://www.conduktor.io/kafka/kafka-message-compression) to cut network and storage costs - [Optimize producer batching](https://www.conduktor.io/kafka/kafka-producer-batching) for throughput - [Configure producer retries](https://www.conduktor.io/kafka/kafka-producer-retries) for error handling Source: https://www.conduktor.io/kafka/idempotent-kafka-producer --- # Java consumer in threads *Run Kafka consumers in separate threads for advanced architectures* Running a consumer in a separate thread allows you to perform other tasks in the main thread, such as handling HTTP requests or running other business logic. **What you'll learn:** - How to run the consumer poll loop in a separate thread - How to implement graceful shutdown with threading - Thread synchronization patterns for consumers > **Advanced topic** > This is only recommended if you have experience with multi-threaded programming. Most use cases do not require running consumers in separate threads. The basic consumer pattern covers most scenarios. ## Consumer in threads sample code ```java package io.conduktor.demos.kafka; import org.apache.kafka.clients.consumer.*; import org.apache.kafka.common.errors.WakeupException; import org.apache.kafka.common.serialization.StringDeserializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.time.Duration; import java.util.Collections; import java.util.Date; import java.util.Properties; import java.util.concurrent.CountDownLatch; public class ConsumerDemoThreads { public static void main(String[] args) { ConsumerDemoWorker consumerDemoWorker = new ConsumerDemoWorker(); new Thread(consumerDemoWorker).start(); Runtime.getRuntime().addShutdownHook(new Thread(new ConsumerDemoCloser(consumerDemoWorker))); } private static class ConsumerDemoWorker implements Runnable { private static final Logger log = LoggerFactory.getLogger(ConsumerDemoWorker.class); private CountDownLatch countDownLatch; private Consumer consumer; @Override public void run() { countDownLatch = new CountDownLatch(1); final Properties properties = new Properties(); properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "my-sixth-application"); properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); consumer = new KafkaConsumer<>(properties); consumer.subscribe(Collections.singleton("demo_java")); final Duration pollTimeout = Duration.ofMillis(100); try { while (true) { final ConsumerRecords consumerRecords = consumer.poll(pollTimeout); for (final ConsumerRecord consumerRecord : consumerRecords) { log.info("Getting consumer record key: '" + consumerRecord.key() + "', value: '" + consumerRecord.value() + "', partition: " + consumerRecord.partition() + " and offset: " + consumerRecord.offset() + " at " + new Date(consumerRecord.timestamp())); } } } catch (WakeupException e) { log.info("Consumer poll woke up"); } finally { consumer.close(); countDownLatch.countDown(); } } void shutdown() throws InterruptedException { consumer.wakeup(); countDownLatch.await(); log.info("Consumer closed"); } } private static class ConsumerDemoCloser implements Runnable { private static final Logger log = LoggerFactory.getLogger(ConsumerDemoCloser.class); private final ConsumerDemoWorker consumerDemoWorker; ConsumerDemoCloser(final ConsumerDemoWorker consumerDemoWorker) { this.consumerDemoWorker = consumerDemoWorker; } @Override public void run() { try { consumerDemoWorker.shutdown(); } catch (InterruptedException e) { log.error("Error shutting down consumer", e); } } } } ``` ## Output ``` [Thread-0] Consumer poll woke up [Thread-0] Revoke previously assigned partitions demo_java-0 [Thread-0] Member consumer-my-sixth-application-1-fecba584-5838-418d-83b8-c88a528bc0e5 sending LeaveGroup request to coordinator 127.0.0.1:9094 (id: 2147483644 rack: null) due to the consumer is being closed [Thread-0] Resetting generation due to: consumer pro-actively leaving the group [Thread-0] Request joining group due to: consumer pro-actively leaving the group [Thread-0] Metrics scheduler closed [Thread-0] Closing reporter org.apache.kafka.common.metrics.JmxReporter [Thread-0] Metrics reporters closed [Thread-0] App info kafka.consumer for consumer-my-sixth-application-1 unregistered [Thread-1] Consumer closed ``` ## Next steps - [Dive into advanced topics](https://www.conduktor.io/kafka/kafka-topics-advanced) to move beyond client code into topic internals - [Tune consumer settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior) for production - [Understand delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers) for exactly-once processing Source: https://www.conduktor.io/kafka/java-consumer-in-threads --- # Java consumer rebalance listener *Implement rebalance listeners for manual offset management* When you manage offsets manually or store them externally, you need to commit offsets before partitions are revoked during a rebalance. The `ConsumerRebalanceListener` interface provides this capability. **What you'll learn:** - When consumer re-balances occur - How to implement a rebalance listener - How to commit offsets during partition revocation - Best practices for manual offset management ## When re-balances occur Consumer re-balances happen for the following events: - Number of partitions change for any of the subscribed topics. - A subscribed topic is created or deleted. - An existing member of the consumer group is shutdown or fails. - A new member is added to the consumer group. When any of these events are triggered, the provided listener will be invoked **twice**: first to indicate that the consumer's assignment has been revoked, and then again when the new assignment has been received. In that case, we have a chance to commit offsets and some cleanup work before our partition is revoked. This includes maybe closing database connections, etc.. If we handle this case gracefully, we will not process duplicate messages through re-balances. > Note that re-balances will only occur during an active call to `poll(Duration)`, so callbacks will also only be invoked during that time. You do not need to worry about threading because calls happen in the same consumer thread. ## When should you use a consumer rebalance? Excerpt from the [ConsumerRebalanceListener](https://kafka.apache.org/24/javadoc/index.html?org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html) documentation: One common use is saving offsets in a custom store. By saving offsets in the `onPartitionsRevoked(Collection)` call we can ensure that any time partition assignment changes the offset gets saved. Another use is flushing out any kind of cache of intermediate results the consumer may be keeping. For example, consider a case where the consumer is subscribed to a topic containing user page views, and the goal is to count the number of page views per user for each five-minute window. Let's say the topic is partitioned by the user id so that all events for a particular user go to a single consumer instance. The consumer can keep in memory a running tally of actions per user and only flush these out to a remote data store when its cache gets too big. However if a partition is reassigned, your consumer may want to automatically trigger a flush of this cache, before the new owner takes over consumption. Note that callbacks only serve as notification of an assignment change. They cannot be used to express acceptance of the change. Hence throwing an exception from a callback does not affect the assignment in any way, as it will be propagated all the way up to the `KafkaConsumer.poll(java.time.Duration)` call. If user captures the exception in the caller, the callback is still assumed successful and no further retries will be attempted. ## Consumer rebalance listeners example ### Consumer rebalance listener implementation In this example, we have created a `ConsumerRebalanceListener` that keeps track of how far we have been consuming in our Kafka topic partitions. ```java package io.conduktor.demos.kafka; import org.apache.kafka.clients.consumer.ConsumerRebalanceListener; import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.clients.consumer.OffsetAndMetadata; import org.apache.kafka.common.TopicPartition; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.util.Collection; import java.util.HashMap; import java.util.Map; public class ConsumerRebalanceListenerImpl implements ConsumerRebalanceListener { private static final Logger log = LoggerFactory.getLogger(ConsumerRebalanceListenerImpl.class); private KafkaConsumer consumer; private Map currentOffsets = new HashMap<>(); public ConsumerRebalanceListenerImpl(KafkaConsumer consumer) { this.consumer = consumer; } public void addOffsetToTrack(String topic, int partition, long offset){ currentOffsets.put( new TopicPartition(topic, partition), new OffsetAndMetadata(offset + 1, null)); } @Override public void onPartitionsRevoked(Collection partitions) { log.info("onPartitionsRevoked callback triggered"); log.info("Committing offsets: " + currentOffsets); consumer.commitSync(currentOffsets); } @Override public void onPartitionsAssigned(Collection partitions) { log.info("onPartitionsAssigned callback triggered"); } // this is used when we shut down our consumer gracefully public Map getCurrentOffsets() { return currentOffsets; } } ``` Important things to note: - We track internally in the class the offsets of how far we have consumers using `currentOffsets`. - In the function `addOffsetToTrack` we make sure to increment the offset by 1 in order to commit the position properly. - We use a synchronous `consumer.commitSync` call in `onPartitionsRevoked` to block until the offsets are successfully committed. ### Use the rebalance listener in consumer code Here's the implementation: ```java package io.conduktor.demos.kafka; import org.apache.kafka.clients.consumer.*; import org.apache.kafka.common.errors.WakeupException; import org.apache.kafka.common.serialization.StringDeserializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.time.Duration; import java.util.Arrays; import java.util.Properties; public class ConsumerDemoRebalanceListener { private static final Logger log = LoggerFactory.getLogger(ConsumerDemoRebalanceListener.class); public static void main(String[] args) { log.info("I am a Kafka Consumer with a Rebalance"); String bootstrapServers = "127.0.0.1:9092"; String groupId = "my-fifth-application"; String topic = "demo_java"; // create consumer configs Properties properties = new Properties(); properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, groupId); properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // we disable Auto Commit of offsets properties.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // create consumer KafkaConsumer consumer = new KafkaConsumer<>(properties); ConsumerRebalanceListenerImpl listener = new ConsumerRebalanceListenerImpl(consumer); // get a reference to the current thread final Thread mainThread = Thread.currentThread(); // adding the shutdown hook Runtime.getRuntime().addShutdownHook(new Thread() { public void run() { log.info("Detected a shutdown, let's exit by calling consumer.wakeup()..."); consumer.wakeup(); // join the main thread to allow the execution of the code in the main thread try { mainThread.join(); } catch (InterruptedException e) { e.printStackTrace(); } } }); try { // subscribe consumer to our topic(s) consumer.subscribe(Arrays.asList(topic), listener); // poll for new data while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord record : records) { log.info("Key: " + record.key() + ", Value: " + record.value()); log.info("Partition: " + record.partition() + ", Offset:" + record.offset()); // we track the offset we have been committed in the listener listener.addOffsetToTrack(record.topic(), record.partition(), record.offset()); } // We commitAsync as we have processed all data and we don't want to block until the next .poll() call consumer.commitAsync(); } } catch (WakeupException e) { log.info("Wake up exception!"); // we ignore this as this is an expected exception when closing a consumer } catch (Exception e) { log.error("Unexpected exception", e); } finally { try { consumer.commitSync(listener.getCurrentOffsets()); // we must commit the offsets synchronously here } finally { consumer.close(); log.info("The consumer is now gracefully closed."); } } } } ``` > - We disable auto commit, otherwise we wouldn't need a rebalance listener. > - On every message being successfully synchronously processed, we call `listener.addOffsetToTrack(record.topic(), record.partition(), record.offset());` which allows us to track how far we've been processing in our consumer. > - When we're done with a batch we call `consumer.commitAsync();` to commit offsets without blocking our consumer loop. > - On the consumer shutdown, we finally call again `consumer.commitSync(listener.getCurrentOffsets());` to commit one last time based on how far we've read before closing the consumer. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-java-consumer-rebalance-listener) displays consumer group rebalances, partition assignments, and committed offsets in real-time. Verify your rebalance listener commits offsets correctly before partitions are revoked. ## Next steps - [Seek and assign](https://www.conduktor.io/kafka/java-consumer-seek-and-assign) for reading specific offsets - [Consumer in threads](https://www.conduktor.io/kafka/java-consumer-in-threads) for multi-threaded consumers - [Incremental rebalancing](https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership) for smoother re-balances Source: https://www.conduktor.io/kafka/java-consumer-rebalance-listener --- # Java consumer seek and assign *Read from specific partitions and offsets using seek() and assign()* When you need to read from specific partitions or replay data from a particular offset, the `seek()` and `assign()` APIs bypass consumer groups entirely. **What you'll learn:** - When to use seek() and assign() instead of consumer groups - How to read from a specific partition and offset - How to implement a bounded consumer (read N messages) To use these API, make the following changes: - Remove the `group.id` from the consumer properties (we don't use consumer groups anymore). - Remove the subscription to the topic. - Use consumer `assign()` and `seek()` APIs. The code with these changes is shown in the snippet below. ```java package io.conduktor.demos.kafka; import java.time.Duration; import java.util.Arrays; import java.util.Properties; import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.ConsumerRecord; import org.apache.kafka.clients.consumer.ConsumerRecords; import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.common.TopicPartition; import org.apache.kafka.common.serialization.StringDeserializer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class ConsumerDemoAssignSeek { public static void main(String[] args) { Logger log = LoggerFactory.getLogger(ConsumerDemoAssignSeek.class.getName()); String bootstrapServers = "127.0.0.1:9092"; String topic = "demo_java"; // create consumer configs Properties properties = new Properties(); properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); properties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // create consumer KafkaConsumer consumer = new KafkaConsumer(properties); // assign and seek are mostly used to replay data or fetch a specific message // assign TopicPartition partitionToReadFrom = new TopicPartition(topic, 0); long offsetToReadFrom = 7L; consumer.assign(Arrays.asList(partitionToReadFrom)); // seek consumer.seek(partitionToReadFrom, offsetToReadFrom); int numberOfMessagesToRead = 5; boolean keepOnReading = true; int numberOfMessagesReadSoFar = 0; // poll for new data while(keepOnReading){ ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord record : records){ numberOfMessagesReadSoFar += 1; log.info("Key: " + record.key() + ", Value: " + record.value()); log.info("Partition: " + record.partition() + ", Offset:" + record.offset()); if (numberOfMessagesReadSoFar >= numberOfMessagesToRead){ keepOnReading = false; // to exit the while loop break; // to exit the for loop } } } log.info("Exiting the application"); } } ``` Here we want to read messages from the offset `7` of partition `0` of the topic `demo_java`. ## Partition offset Make sure the partition offset of partition `0` of the topic `demo_java` is at least 7. Produce a number of messages to the topic to achieve that. Run the application. The console of the application will display the selected messages. ``` [main] INFO Subscribed to partition(s): demo_java-0 [main] INFO Seeking to offset 7 for partition demo_java-0 [main] INFO Cluster ID: 8fresFx9R4Kod8UqUrixrg [main] INFO ConsumerDemoAssignSeek - Key: id_7, Value: hello world 7 [main] INFO ConsumerDemoAssignSeek - Partition: 0, Offset:7 [main] INFO ConsumerDemoAssignSeek - Key: id_8, Value: hello world 8 [main] INFO ConsumerDemoAssignSeek - Partition: 0, Offset:8 [main] INFO ConsumerDemoAssignSeek - Key: id_9, Value: hello world 9 [main] INFO ConsumerDemoAssignSeek - Partition: 0, Offset:9 [main] INFO ConsumerDemoAssignSeek - Key: id_0, Value: hello world 0 [main] INFO ConsumerDemoAssignSeek - Partition: 0, Offset:10 [main] INFO ConsumerDemoAssignSeek - Key: id_1, Value: hello world 1 [main] INFO ConsumerDemoAssignSeek - Partition: 0, Offset:11 [main] INFO ConsumerDemoAssignSeek - Exiting the application ``` ## Next steps - [Consumer in threads](https://www.conduktor.io/kafka/java-consumer-in-threads) for multi-threaded consumers - [Rebalance listener](https://www.conduktor.io/kafka/java-consumer-rebalance-listener) for manual offset management - [Consumer concepts](https://www.conduktor.io/kafka/kafka-consumers) for foundational understanding Source: https://www.conduktor.io/kafka/java-consumer-seek-and-assign --- # Java Kafka programming *Build Kafka producers and consumers with Java* Java is the official language for Kafka development, providing the most complete and up-to-date client library. This section walks through creating a project and building your first producer and consumer. **What you'll learn:** - How to set up a Java project with Kafka dependencies - How to create producers with callbacks and keys - How to create consumers with consumer groups - How to handle graceful shutdown ## Learning path | Step | Tutorial | Description | |------|----------|-------------| | 1 | Project setup | Choose [Maven](https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-maven-pom-xml) or [Gradle](https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-gradle-build-gradle) | | 2 | [Producer](https://www.conduktor.io/kafka/complete-kafka-producer-with-java) | Create producers with callbacks, keys, and batching | | 3 | [Consumer](https://www.conduktor.io/kafka/complete-kafka-consumer-with-java) | Create consumers with groups and graceful shutdown | | 4 | [Advanced](https://www.conduktor.io/kafka/advanced-kafka-consumer-with-java) | Rebalance listeners, seek/assign, threading | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-java-kafka-programming) helps you develop and debug Java applications by providing visual access to topics, consumer groups, and messages. Test your producer output and monitor consumer progress in real-time. ## Next steps - [Set up your project with Maven](https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-maven-pom-xml) to add the Kafka client dependencies - [Set up with Gradle](https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-gradle-build-gradle) if you prefer that build tool - [Build your first producer](https://www.conduktor.io/kafka/complete-kafka-producer-with-java) with callbacks and keys Source: https://www.conduktor.io/kafka/java-kafka-programming --- # Kafka administration *Master the essential skills for operating Kafka clusters in production* Administering Apache Kafka requires understanding cluster architecture, security, monitoring, and operational procedures. This section covers everything you need to run Kafka reliably at scale. **What you'll learn:** - How to plan and size Kafka cluster deployments - How to secure clusters with authentication and authorization - How to monitor cluster health and performance - How to perform upgrades and maintain availability ## Administration topics | Topic | Description | |-------|-------------| | [Cluster setup](https://www.conduktor.io/kafka/kafka-cluster-setup-overview) | Architecture, sizing, and deployment options | | [Security](https://www.conduktor.io/kafka/kafka-security) | Encryption, authentication, and ACLs | | [Monitoring](https://www.conduktor.io/kafka/kafka-monitoring-and-operations) | Metrics, alerting, and operations | | [Upgrades](https://www.conduktor.io/kafka/kafka-broker-and-client-upgrades) | Zero-downtime broker and client upgrades | | [Multi-cluster](https://www.conduktor.io/kafka/kafka-multi-cluster-and-mirrormaker) | Cross-datacenter replication with MirrorMaker | | [Advertised host](https://www.conduktor.io/kafka/kafka-advertised-host-setting) | Network configuration for client connectivity | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-administration) provides centralized management for all your Kafka clusters. Monitor broker health, manage topics, configure security, and troubleshoot issues from a single interface. ## Next steps - [Upgrade brokers and clients](https://www.conduktor.io/kafka/kafka-broker-and-client-upgrades) to keep your cluster current with zero downtime - [Secure your cluster](https://www.conduktor.io/kafka/kafka-security) with encryption, authentication, and ACLs - [Set up monitoring](https://www.conduktor.io/kafka/kafka-monitoring-and-operations) to track broker health and performance Source: https://www.conduktor.io/kafka/kafka-administration --- # Advanced Kafka concepts This section covers advanced topics for Kafka power users and administrators who want to understand the deeper aspects of Apache Kafka. ## Advanced topics Learn about advanced topic configurations, internals, and optimization strategies: - Topic configuration management - Segments and indexes - Log retention and compaction - Replication and partitioning strategies [Explore advanced topics](https://www.conduktor.io/kafka/kafka-topics-advanced). ## Advanced producers Master producer performance, reliability, and advanced configurations: - Acknowledgment settings explained - Retry mechanisms and idempotency - Compression and batching - Partitioning strategies [Explore advanced producers](https://www.conduktor.io/kafka/kafka-producers-advanced). ## Advanced consumers Understand consumer behavior, delivery semantics, and optimization: - Auto offset reset behavior - Delivery semantics (at-most-once, at-least-once, exactly-once) - Consumer poll and internal threads - Incremental rebalance and static group membership - Reading from closest replica [Explore advanced consumers](https://www.conduktor.io/kafka/kafka-consumers-advanced). ## Next steps - [Dive into advanced topics](https://www.conduktor.io/kafka/kafka-topics-advanced) for internals, retention, and compaction - [Master advanced producers](https://www.conduktor.io/kafka/kafka-producers-advanced) for acks, retries, and idempotency - [Explore advanced consumers](https://www.conduktor.io/kafka/kafka-consumers-advanced) for delivery semantics and rebalancing Source: https://www.conduktor.io/kafka/kafka-advanced-concepts --- # Kafka advertised host setting *Learn how to configure advertised listeners for client connectivity* The advertised listeners setting is one of the most important Kafka configurations. Mis-configuring it is a common cause of client connection failures. **What you'll learn:** - How Kafka clients discover and connect to brokers - Why advertised host configuration matters - How to configure advertised listeners for different network topologies - Common pitfalls and how to avoid them ## Connecting to a Kafka cluster When a client connects to a Kafka cluster, the broker responds with the advertised host that the client MUST use for all future communications - regardless of how the initial connection was made. ![Sequence diagram: a client connects to broker:9092, the broker replies that the client must use advertised.listeners for all future requests, and the client then connects to the advertised address; an unreachable advertised address causes connection failure](https://www.conduktor.io/assets/kafka/diagrams/kafka-advertised-host-setting.svg) ```mermaid sequenceDiagram participant C as Client participant B as Broker C->>B: Connect to broker:9092 B->>C: "Use advertised.listeners
for all future requests" C->>B: Connect to advertised address Note over C,B: If advertised address
is unreachable, connection fails ``` > **Common misconception** > Just because a Kafka hostname and port are reachable doesn't mean clients can establish a Kafka-protocol connection. The client has to be able to reach the advertised address, not just the initial bootstrap address. ## Setting advertised host as localhost If you set the Advertised Host as localhost, the Kafka client will successfully connect to the cluster if it is running on the same machine as the broker. ## Setting advertised host as public IP If you set the Advertised Host as the broker's Public IP, the Kafka client will successfully connect to the cluster as long as the Public IP doesn't change. It may change if the broker re-boots. If that happens, the connection cannot be re-established. ## Configure advertised host | Client location | Recommended setting | |-----------------|---------------------| | Same network as broker | Internal private IP or DNS hostname | | Public network | External public IP or DNS hostname | | Both internal and external | Configure multiple listeners | ### Example: Internal clients only ```properties listeners=PLAINTEXT://0.0.0.0:9092 advertised.listeners=PLAINTEXT://internal-broker1.company.com:9092 ``` ### Example: External clients only ```properties listeners=PLAINTEXT://0.0.0.0:9092 advertised.listeners=PLAINTEXT://public-broker1.example.com:9092 ``` ### Example: Both internal and external clients ```properties listeners=INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093 advertised.listeners=INTERNAL://internal-broker1.company.com:9092,EXTERNAL://public-broker1.example.com:9093 listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL ``` > **Use DNS hostnames** > Prefer DNS hostnames over IP addresses. IP addresses can change when instances reboot, but DNS can be updated to point to new addresses without client configuration changes. For an in-depth explanation of Kafka Listeners, see [Kafka Listeners Explained](https://rmoff.net/2018/08/02/kafka-listeners-explained/). > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-advertised-host-setting) displays broker listener configurations and helps diagnose connectivity issues. Test connections from different network locations to verify your advertised listener settings. ## Next steps - [Return to the administration overview](https://www.conduktor.io/kafka/kafka-administration) to revisit the full operations path - [Configure security](https://www.conduktor.io/kafka/kafka-security) for your listeners - [Monitor broker health](https://www.conduktor.io/kafka/kafka-monitoring-and-operations) to validate connectivity in production Source: https://www.conduktor.io/kafka/kafka-advertised-host-setting --- # Kafka broker and client upgrades *Learn how to upgrade Kafka clusters with zero downtime* Kafka's versioned API calls enable bi-directional compatibility between clients and brokers. This means you can upgrade brokers and clients independently without coordinated downtime. **What you'll learn:** - How Kafka client/broker version compatibility works - The rolling upgrade process for brokers - How to upgrade without data loss or downtime - When to upgrade clients vs brokers ## Kafka client version compatibility Since Kafka 0.10.2, clients and brokers have **bi-directional compatibility** because API calls are versioned. ![Compatibility diagram: client v2.1 and client v3.0 are each bi-directionally compatible with both broker v2.1 and broker v3.0](https://www.conduktor.io/assets/kafka/diagrams/kafka-broker-and-client-upgrades.svg) ```mermaid flowchart LR subgraph Clients["Clients"] C1["Client v2.1"] C2["Client v3.0"] end subgraph Brokers["Brokers"] B1["Broker v2.1"] B2["Broker v3.0"] end C1 <-->|"Compatible"| B1 & B2 C2 <-->|"Compatible"| B1 & B2 ``` | Scenario | Compatible? | |----------|-------------| | Old client (2.1) → New broker (3.0) | Yes | | New client (3.0) → Old broker (2.1) | Yes | This allows administrators to upgrade clients or brokers independently without coordinated downtime. > Always use the latest client library version if you can You can read more about client bi-directional compatibility here: [https://www.confluent.io/blog/upgrading-apache-kafka-clients-just-got-easier/](https://www.confluent.io/blog/upgrading-apache-kafka-clients-just-got-easier/) ## Kafka broker version upgrades Upgrading Kafka brokers is by far the most important operation that we have to do on our cluster. It is recommended to upgrade very often in order to keep up with bug fixes, new features, and performance improvements. The more we wait to upgrade a broker, the more painful it will be as there will be many versions jump. Instructions on how to upgrade Kafka are always [outlined in the documentation](https://kafka.apache.org/documentation/#upgrade). Regardless, the steps are usually pretty similar from version to version, as you'll see below. ### How to upgrade Kafka brokers without downtime? It is fairly simple to upgrade Kafka. There are five steps to upgrade with a rolling restart required at each step. In this example we are giving the steps to upgrade from version Kafka v2.1 to v3.0. 1. Set inter broker and log version to current Kafka version. If you are using 2.1 now, just put the following broker properties: `inter.broker.protocol.version=2.1` and `log.message.format.version=2.1` 2. One broker at a time: stop the broker, upgrade the binaries to 3.0, and start the broker. Wait for the cluster to stabilize (all the partitions are in-sync replicas (ISR). 3. At this stage, all your brokers are running the newer binaries of 3.0, but running with `inter.broker.protocol.version=2.1` and `log.message.format.version=2.1` 4. Ensure the cluster is stable in this form. You still have the option to downgrade here if you want to because the log message format has not been upgraded yet. 5. One broker at a time: stop the broker, change the inter broker protocol version for that broker with `inter.broker.protocol.version=3.0` and start that broker. Wait for the cluster to stabilize (all the partitions are in-sync replicas (ISR). 6. At the end of this stage, all the brokers are running with the new inter broker protocol version, but they are not using the new log message format yet. 7. Upgrade Kafka clients if documentation specifies it (all or most of them to avoid up and down conversion - which can affect performance) 8. One broker at a time: stop the broker, upgrade message protocol version `log.message.format.version=3.0` and start the broker. Wait for the cluster to stabilize (all the partitions are in-sync replicas (ISR). 9. At this stage, all brokers are upgraded to the latest binary, are running the latest inter-broker protocol and are using the latest log message format version. Congratulations! > **Monitor during upgrades** > Watch ISR counts, under-replicated partitions, and request latencies during rolling restarts. Wait for the cluster to stabilize before proceeding to the next broker. Refer to the documentation at: [https://kafka.apache.org/documentation/#upgrade](https://kafka.apache.org/documentation/#upgrade) ## Upgrade summary | Phase | Action | Rollback possible? | |-------|--------|-------------------| | 1 | Set version properties | Yes | | 2 | Upgrade binaries | Yes | | 3 | Upgrade inter.broker.protocol | Yes | | 4 | Upgrade clients | Yes | | 5 | Upgrade log.message.format | Limited | The longer you wait between upgrades, the more complex the upgrade becomes. Regular, incremental upgrades are easier than large version jumps. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-broker-and-client-upgrades) helps you monitor cluster health during upgrades. Track broker versions, ISR status, and partition health in real-time to ensure safe rolling restarts. ## Next steps - [Secure your cluster](https://www.conduktor.io/kafka/kafka-security) with encryption, authentication, and ACLs after upgrading - [Set up monitoring](https://www.conduktor.io/kafka/kafka-monitoring-and-operations) to watch ISR and latency during rolling restarts - [Review cluster setup](https://www.conduktor.io/kafka/kafka-cluster-setup-overview) for availability during upgrades Source: https://www.conduktor.io/kafka/kafka-broker-and-client-upgrades --- # Kafka brokers *Learn about Kafka brokers and clusters in eight minutes* Kafka brokers are the servers that store your data and serve client requests. Understanding how brokers work is essential for designing and operating Kafka clusters. **What you'll learn:** - What a Kafka broker is and how it runs - How brokers form a Kafka cluster - How topics and partitions are distributed across brokers - How clients discover and connect to brokers ## What is a Kafka broker? A single Kafka server is called a Kafka broker. That Kafka broker is a program that runs on the Java Virtual Machine (Java version 11+) and usually a server that is meant to be a Kafka broker will solely run the necessary program and nothing else. **Key responsibilities of a broker:** - Store data for topic partitions on disk - Handle read and write requests from producers and consumers - Manage partition replication - Coordinate with other brokers in the cluster ## What is Kafka cluster? An ensemble of Kafka brokers working together is called a Kafka cluster. Some clusters may contain one broker or others may contain three or potentially hundreds of brokers. Companies like Netflix and Uber run hundreds or thousands of Kafka brokers to handle their data. A broker in a cluster is identified by a unique numeric ID. In the figure below, the Kafka cluster is made up of three Kafka brokers. ![Diagram showing a Kafka cluster with multiple Kafka brokers.](https://www.conduktor.io/assets/kafka/Kafka-Brokers-1.png) > **Cluster sizing** > Start with three brokers for production workloads. This provides fault tolerance while keeping operations manageable. Scale up based on throughput requirements and storage needs. ## Kafka brokers and topics Kafka brokers store data in a directory on the server disk they run on. Each topic-partition receives its own sub-directory with the associated name of the topic. To achieve high throughput and scalability on topics, Kafka topics are partitioned. If there are multiple Kafka brokers in a cluster, then partitions for a given topic will be distributed among the brokers evenly, to achieve load balancing and scalability. ![3 Kafka Brokers with 2 Kafka Topics and topic partitions.](https://www.conduktor.io/assets/kafka/Kafka-Brokers-2.png) In the diagram above, there are two topics illustrated - *Topic-A* has three partitions. They are distributed evenly among the three available brokers in the cluster. Alternatively, there may be fewer (or more) partitions of a topic than the number of brokers in the cluster. *Topic-B*, in our case, has two partitions only. In this case, *Broker 103* does not contain any partition of *Topic-B*. There is no relationship between the broker ID and the partition ID - Kafka does a good job of distributing partitions evenly among the available brokers. In case the cluster becomes unbalanced due to an overload of a specific broker, it is possible for Kafka administrators to rebalance the cluster and move partitions. > **Partition limits** > Each broker has practical limits on the number of partitions it can handle (typically 2,000-4,000). Monitor partition counts as your cluster grows. ## How do clients connect to a Kafka cluster (bootstrap server)? A client that wants to send or receive messages from the Kafka cluster **may connect to any broker in the cluster.** Every broker in the cluster has metadata about all the other brokers and will help the client connect to them as well, and **therefore any broker in the cluster is also called a bootstrap server.** The bootstrap server will return metadata to the client that consists of a list of all the brokers in the cluster. Then, when required, the client will know which exact broker to connect to to send or receive data, and accurately find which brokers contain the relevant topic-partition. ![Connecting to a Kafka Cluster (Bootstrap Server) diagram. Process for connecting to a Kafka broker within an Apache Kafka cluster.](https://www.conduktor.io/assets/kafka/Kafka-Brokers-3.png) In practice, it is common for the Kafka client to reference at least two bootstrap servers in its connection URL, in the case one of them not being available, the other one should still respond to the connection request. That means that Kafka clients (and developers/DevOps) do not need to be aware of every single hostname of every single broker in a Kafka cluster, but only to be aware and reference two or three in the connection string for clients. ```properties # Example bootstrap server configuration bootstrap.servers=broker1.example.com:9092,broker2.example.com:9092 ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-brokers) provides a visual dashboard to monitor broker health, partition distribution, and cluster metrics. View which brokers are online, their disk usage, and how partitions are balanced across the cluster. ## Next steps - [Understand topic replication](https://www.conduktor.io/kafka/kafka-topic-replication) to see how brokers replicate data for fault tolerance - [Learn about KRaft mode](https://www.conduktor.io/kafka/kafka-kraft-mode) to understand modern broker coordination - [Explore cluster setup](https://www.conduktor.io/kafka/kafka-cluster-setup-overview) for production deployment guidance Source: https://www.conduktor.io/kafka/kafka-brokers --- # Kafka CLI tutorials *Master Kafka operations using the command line interface* The Kafka CLI provides powerful tools for managing topics, producing and consuming messages, and monitoring consumer groups. These tutorials walk you through essential CLI operations with practical examples. **What you'll learn:** - How to create, describe, and manage Kafka topics - How to produce and consume messages from the command line - How to work with consumer groups and monitor offsets - How to use Kafka Connect in standalone mode ## Available CLI tutorials | Tool | Purpose | Tutorial | |------|---------|----------| | `kafka-topics` | Create, list, describe, alter topics | [Topic management](https://www.conduktor.io/kafka/kafka-topics-cli-tutorial) | | `kafka-console-producer` | Send messages to topics | [Producer CLI](https://www.conduktor.io/kafka/kafka-producer-cli-tutorial) | | `kafka-console-consumer` | Read messages from topics | [Consumer CLI](https://www.conduktor.io/kafka/kafka-consumer-cli-tutorial) | | `kafka-console-consumer` | Demonstrate consumer groups | [Consumer groups](https://www.conduktor.io/kafka/kafka-consumers-in-group-cli-tutorial) | | `kafka-consumer-groups` | Manage and reset consumer groups | [Group management](https://www.conduktor.io/kafka/kafka-consumer-group-management-cli-tutorial) | | `connect-standalone` | Run Kafka Connect connectors | [Connect CLI](https://www.conduktor.io/kafka/kafka-connect-cli-tutorial) | CLI commands use platform-specific extensions: `.bat` for Windows, `.sh` for Mac and Linux. These tutorials show commands without extensions for brevity. > **See it in practice with Conduktor** > While the CLI is powerful for scripting and automation, [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-cli-tutorial) provides a visual interface for all these operations. Use both together for maximum productivity. ## Next steps - [Manage topics from the CLI](https://www.conduktor.io/kafka/kafka-topics-cli-tutorial) to create, list, and configure topics - [Produce messages](https://www.conduktor.io/kafka/kafka-producer-cli-tutorial) with the console producer - [Consume messages](https://www.conduktor.io/kafka/kafka-consumer-cli-tutorial) from your topics Source: https://www.conduktor.io/kafka/kafka-cli-tutorial --- # Kafka cluster setup overview *Learn how to plan and size Kafka clusters for production* A production Kafka deployment requires careful planning around broker count, availability zones, and infrastructure sizing. This guide covers architecture best practices and sizing considerations. **What you'll learn:** - Minimum requirements for production Kafka clusters - How to distribute brokers across availability zones - How to calculate storage and broker requirements - When to consider managed Kafka services ## Production cluster architecture At a bare minimum, a realistic Kafka cluster should have the following architecture: - To make the cluster highly available and performant, you want multiple brokers in different data centers (racks) to distribute your load. If you are setting up your cluster in AWS, this will be at least three different availability zones. - You also want a cluster of at least 3 ZooKeeper nodes (if using ZooKeeper, the alternative being KRaft mode) ![](https://www.conduktor.io/assets/kafka/zoo-brokers.png) All ZooKeeper nodes and Kafka brokers should be spread evenly across availability zones. In AWS, an Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region ## Kafka cluster setup gotchas Kafka cluster has many components for reasons of performance and high availability. It is important to remember a few important points while setting up a Kafka cluster. - It's not easy to set up a cluster. This is a huge effort and needs dedicated effort. This is why managed Kafka solutions are getting more and more popular ( [Amazon MSK](https://aws.amazon.com/msk/), [Confluent Cloud](https://www.confluent.io/confluent-cloud/), [Aiven](https://aiven.io), [CloudKarafka](https://www.cloudkarafka.com), [Instaclustr](https://www.instaclustr.com), [Upstash](https://upstash.com), etc...) - You want to isolate each Zookeeper & Broker on separate servers. It is not safe to host multiple Kafka components on the same machine as they will compete for resources and make the cluster fail easily in case of errors. - Monitoring needs to be implemented. As with any distributed system, to understand how the whole system is behaving, it is essential to have a good monitoring system implemented. - Operations have to be mastered. There are several operations that are required to be performed on a Kafka cluster while it is in operation. For example, upgrades, backups, etc. - You need a really good Kafka Admin. This role is essential and we need a dedicated Kafka administrator to manage the cluster. ## How many Kafka brokers do I need? The first factor is to understand how much disk space you're going to need to retain the messages based on your retention policy (one week of data by default) and your expected total throughput. If your whole cluster needs 50TB of storage and each broker can only have 10 TB of storage, you will need at least 5 brokers. To understand how much storage you need, you also need to take into account the replication factor. If for example, your write data at 2MB/s in your Kafka cluster across all topics, you need to retain 1 month of data at any time, and you have a replication factor of 3, then you need ``` Total Size in MB = 2MB * 3600 * 24 * 30 * 3 (replication factor) = 15552000MB Total Size in TB = 15.552 TB ``` The second factor is that you need to understand your network, CPU and memory requirements for your brokers so you size them appropriately and can handle peak load of throughput. This depends on how many producers and consumers you expect to have as each of these will consume a part of your broker network interface. All of these metrics need to be monitored (on top of system metrics such as OpenFileDescriptor, GC time, system load...) and you need to be ready to eventually add more brokers to handle the increased load in your cluster. ## Managed Kafka As an alternative, there are many vendors that offer Kafka as a fully managed service that can be deployed on the cloud of your choice. With a managed Kafka solution, you can focus on building your core apps. The security, uptime, and management of the Kafka cluster will be taken care of by the vendor at a price. | Provider | Notes | |----------|-------| | [Amazon MSK](https://aws.amazon.com/msk/) | AWS-native, integrates with AWS services | | [Confluent Cloud](https://www.confluent.io/confluent-cloud/) | Full Confluent ecosystem | | [Aiven](https://aiven.io) | Multi-cloud support | | [Upstash](https://upstash.com) | Serverless Kafka | Operating Kafka yourself requires dedicated expertise. Consider managed services if you lack experienced Kafka administrators or prefer to focus on application development. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-cluster-setup-overview) works with both self-managed and managed Kafka clusters. > Connect all your clusters to a single management interface regardless of where they run. ## Next steps - [Set up multi-cluster replication](https://www.conduktor.io/kafka/kafka-multi-cluster-and-mirrormaker) for disaster recovery and data locality - [Set up monitoring](https://www.conduktor.io/kafka/kafka-monitoring-and-operations) for your cluster - [Configure security](https://www.conduktor.io/kafka/kafka-security) for production Source: https://www.conduktor.io/kafka/kafka-cluster-setup-overview --- # Kafka Connect CLI tutorial *Learn how to use Kafka Connect in standalone mode* Kafka Connect provides a scalable way to move data between Kafka and external systems. This tutorial demonstrates running a connector in standalone mode using real-time Wikipedia changes. **What you'll learn:** - What Kafka Connect is and when to use it - How to configure a standalone connector - How to set up the required properties files - How to verify data flowing into Kafka > **Live data stream** > This tutorial uses the [Wikipedia recent changes stream](https://stream.wikimedia.org/v2/stream/recentchange) - a real-time feed of edits happening across Wikipedia. ## What is Kafka Connect? Kafka Connect is a framework for streaming data between Kafka and external systems using reusable connectors. Instead of writing custom producer/consumer code for common integrations, you can use pre-built connectors. ![Flowchart: source systems (database, S3, APIs) feed source connectors, which write to Kafka topics, which sink connectors read and deliver to sink systems (Elasticsearch, data warehouse, S3)](https://www.conduktor.io/assets/kafka/diagrams/kafka-connect-cli-tutorial.svg) ```mermaid flowchart LR subgraph Sources["Source Systems"] DB[(Database)] S3A[S3] API[APIs] end subgraph Connect["Kafka Connect"] SC["Source
Connectors"] SK["Sink
Connectors"] end subgraph Kafka["Kafka"] T[Topics] end subgraph Sinks["Sink Systems"] ES[Elasticsearch] DW[(Data Warehouse)] S3B[S3] end DB & S3A & API --> SC --> T --> SK --> ES & DW & S3B ``` | Connector type | Direction | Examples | |----------------|-----------|----------| | Source | External → Kafka | Debezium, JDBC, S3, MongoDB, Twitter | | Sink | Kafka → External | Elasticsearch, S3, JDBC, HDFS, Splunk | Find connectors on [Confluent Hub](https://www.confluent.io/hub/). ## How to use Kafka Connect in standalone mode? To use Kafka Connect in standalone mode, we need to provide the mandatory parameters: * Download a Kafka Connect connector, either from GitHub or Confluent Hub [Confluent Hub](https://www.confluent.io/hub/) * Create a configuration file for your connector * Use the `connect-standalone.sh` CLI to start the connector ### Example: Kafka Connect standalone with Wikipedia data Create the Kafka topic `wikipedia.recentchange` in Kafka with 3 partitions ``` kafka-topics --bootstrap-server localhost:9092 --topic wikipedia.recentchange --create --partitions 3 --replication-factor 1 ``` As well as the topic dead letter queue `wikipedia.dlq`, for catching any errors ``` kafka-topics --bootstrap-server localhost:9092 --topic wikipedia.dlq --create --partitions 3 --replication-factor 1 ``` Download the release JAR and configuration from [here](https://github.com/simplesteph/kafka-connect-sse/releases/download/v1.0/kafka-connect-sse.zip) and unzip the archive on your computer at `kafka_2.13-2.8.1/connectors/kafka-connect-sse`: ``` ~/kafka_2.13-2.8.1/connectors ls -R kafka-connect-sse ./kafka-connect-sse: connector.properties kafka-connect-sse-1.0-jar-with-dependencies.jar ``` Edit the configuration file `connectors/kafka-connect-sse/connector.properties` with the following properties: ``` name=sse-source-connector tasks.max=1 connector.class=com.github.cjmatta.kafka.connect.sse.ServerSentEventsSourceConnector topic=wikipedia.recentchange sse.uri=https://stream.wikimedia.org/v2/stream/recentchange errors.tollerance=all errors.deadletterqueue.topic.name=wikipedia.dlq ``` Look into your Kafka installation directory (where your `bin` and `config` folders are) Edit the content of the `config/connect-standalone.properties` file ``` key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=true offset.flush.interval.ms=10000 # EDIT BELOW IF NEEDED bootstrap.servers=localhost:9092 offset.storage.file.filename=/tmp/connect.offsets plugin.path=/Users/stephanemaarek/kafka_2.13-2.8.1/connectors ``` The last three lines are the most important to make everything work. In particular, the `plugin.path` config: this is where you indicate the folder where you store your Kafka connectors you have downloaded before. **This must be an absolute path (not relative, and no shortcut with `~`) to your `connectors` directory** If you fail this step, Kafka Connect will stop after starting it. Next, we can start our Kafka Connect standalone connector ``` connect-standalone ~/kafka_2.13-2.8.1/config/connect-standalone.properties ~/kafka_2.13-2.8.1/connectors/kafka-connect-sse/connector.properties ``` And as we can see, the data is flowing into our `wikipedia.recentchange` topic: ``` kafka-console-consumer --bootstrap-server localhost:9092 --topic wikipedia.recentchange ``` ## Standalone vs distributed mode | Mode | Use case | Scalability | |------|----------|-------------| | Standalone | Development, testing, single tasks | Single worker | | Distributed | Production, high availability | Multiple workers | Standalone mode runs on a single machine with no fault tolerance. For production deployments, use distributed mode with multiple workers. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-connect-cli-tutorial) provides a visual interface for managing Kafka Connect clusters, deploying connectors and monitoring connector health and throughput. ## Next steps - [Start programming with Kafka](https://www.conduktor.io/kafka/kafka-programming-tutorials) to build producers and consumers in code - [Monitor Kafka clusters](https://www.conduktor.io/kafka/kafka-monitoring-and-operations) including Connect workers - Explore [Confluent Hub](https://www.confluent.io/hub/) for more available connectors Source: https://www.conduktor.io/kafka/kafka-connect-cli-tutorial --- # Kafka consumer CLI tutorial *Learn how to consume messages from Kafka using kafka-console-consumer* The `kafka-console-consumer` CLI reads data from Kafka topics and outputs it to standard output. It's essential for testing, debugging, and verifying message delivery. **What you'll learn:** - How to consume messages from a topic - How to read historical vs real-time messages - How to display keys, timestamps, and metadata - Common consumer options and troubleshooting > Use CLI commands with appropriate extensions for your platform: for Windows - `kafka-console-consumer.bat`, for Mac and Linux - `kafka-console-consumer.sh`. ## How to consume data in a Kafka topic using the CLI? To consume data from a Kafka topic, we need to provide the mandatory parameters: * Find your Kafka hostname and port e.g., `localhost:9092` * Provide the mandatory parameters: topic name * If you need to read historical data, use the `--from-beginning` option * Otherwise, you will only be reading future data * Use the `kafka-console-consumer.sh` CLI > **Important recommendation: do NOT use the `--zookeeper` option** > Since Kafka v0.10, the consumer is leveraging a Kafka connection string, not Zookeeper. This is due to how consumer offsets are stored. Any tutorial you see using the `--zookeeper` option should be considered **outdated.** ### Example Make sure you have followed the [Kafka producer CLI tutorial](https://www.conduktor.io/kafka/kafka-producer-cli-tutorial) to create and send data to a Kafka topic first. **Consuming only the future messages of a Kafka topic** ``` kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic ``` **Consuming all historical messages and future ones in a Kafka topic** Use the `--from-beginning` option ``` kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --from-beginning ``` To exit the Kafka console consumer, use the keyboard combination `Ctrl+C`. ### Command output ``` Hello World My name is Conduktor I love Kafka ^CProcessed a total of 3 messages (<-- when doing a Ctrl+C to exit) ``` The Kafka console consumer will remain opened until you exit it, and will keep on displaying messages to the screen. It assumes all the messages coming in can be deserialized as text (`String`). By default, the Kafka console consumer does not show the key, or any partition information. If you do not see any output but you know your Kafka topic has data in it, don't forget to use the `--from-beginning` option. > **Important note on message ordering** > The order of messages is not total, it is per partition. As a topic may be created with more than one partition, the order is only guaranteed at the partition level. If you try with only one partition, you will see total ordering. ### Gotchas Here are the common mistakes and caveats with the `kafka-console-consumer.sh` command: Messages by default will not display the key or metadata information (see below for how to do it). * When you start a `kafka-console-consumer`, unless specifying the `--from-beginning` option, only future messages will be displayed and read * If the topic does not exist, the console consumer will automatically create it with default settings * You can consume multiple topics at a time with a comma-delimited list or a pattern * If a consumer group id is not specified, the `kafka-console-consumer` generates a random consumer group * If messages do not appear in order, remember that the order is at the partition level, not at the topic level ### Extra important options you can set (advanced) `--from-beginning` We won't repeat this one enough. To read all historical messages `--formatter` To display messages in a particular format (example below to display keys) `--consumer-property` To pass in any consumer property, such as the `allow.auto.create.topics` setting `--group` By default a random consumer group ID is chosen, but you can override it with this option. See the demo in the [Kafka consumers in group CLI tutorial](https://www.conduktor.io/kafka/kafka-consumers-in-group-cli-tutorial). `--max-messages` Number of messages to consume before exiting `--partition` If you want to only consume from a specific partition. ## How to consume a Kafka topic and show both the key and value using the Kafka console consumer CLI? By default, the console consumer will show only the value of the Kafka record. Using this command you can show both the key and value. Using the `formatter` `kafka.tools.DefaultMessageFormatter` and using the properties `print.timestamp=true` `print.key=true` `print.value=true`: ``` kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --formatter kafka.tools.DefaultMessageFormatter --property print.timestamp=true --property print.key=true --property print.value=true --from-beginning CreateTime:1641810588071 null hello CreateTime:1641823304170 name Stephane CreateTime:1641823301294 example key example value ``` More properties are available such as: * `print.partition` * `print.offset` * `print.headers` * `key.separator` * `line.separator` * `headers.separator` ## Quick reference | Operation | Command | |-----------|---------| | Consume new messages | `kafka-console-consumer --bootstrap-server localhost:9092 --topic NAME` | | Consume all messages | `... --from-beginning` | | Show keys | `... --property print.key=true` | | Show all metadata | `... --property print.timestamp=true --property print.key=true --property print.partition=true --property print.offset=true` | | Limit messages | `... --max-messages 10` | | Specific partition | `... --partition 0` | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-consumer-cli-tutorial) provides a visual interface for consuming messages with built-in deserialization, filtering, and search capabilities. ## Next steps - [Run consumers in a group](https://www.conduktor.io/kafka/kafka-consumers-in-group-cli-tutorial) to see partitions distribute across consumers - [Manage consumer group offsets](https://www.conduktor.io/kafka/kafka-consumer-group-management-cli-tutorial) with kafka-consumer-groups - [Understand consumer concepts](https://www.conduktor.io/kafka/kafka-consumers) in depth Source: https://www.conduktor.io/kafka/kafka-consumer-cli-tutorial --- # Kafka consumer group management CLI tutorial *Learn how to manage consumer groups and troubleshoot lag* Learn to work with consumer groups in Kafka using CLI. **What you'll learn:** - How to reset consumer group offsets to reprocess messages - How to list and describe consumer groups and their state - How to troubleshoot consumer lag and identify root causes - How to delete consumer groups and manage offsets The Kafka consumer groups CLI **kafka-consumer-groups** is used to manage consumer groups in Kafka. Make sure you have [started Kafka](https://www.conduktor.io/kafka/how-to-start-kafka-with-conduktor) beforehand. > Use CLI commands with appropriate extensions for your platform: for Windows - `kafka-consumer-groups.bat`, for Mac and Linux - `kafka-consumer-groups.sh`. ## How to reset a Kafka consumer group using the CLI? To reset a Kafka consumer groups, we need to: * Find your broker hostname and port e.g., `localhost:9092` * Understand the offset reset strategy (to earliest, to latest, to specific offset, shift by...) * Stop the running consumer groups (otherwise the command will fail) * Use the `kafka-consumer-groups.sh` CLI with the `--reset-offsets` option ## Example: Reset offsets to the earliest First, ensure that the consumers are stopped ("has no active members") ``` kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-first-application Consumer group 'my-first-application' has no active members. ``` Observe the current offsets for your consumer group (same command as above) ``` GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-first-application first_topic 0 3 3 0 - - - my-first-application first_topic 1 5 5 0 - - - my-first-application first_topic 2 6 6 0 - - - ``` We will reset the offsets to the earliest position in order to read the topic entirely again ``` kafka-consumer-groups --bootstrap-server localhost:9092 --group my-first-application --reset-offsets --to-earliest --execute --topic first_topic GROUP TOPIC PARTITION NEW-OFFSET my-first-application first_topic 0 0 my-first-application first_topic 1 0 my-first-application first_topic 2 0 ``` As you can see the new offsets for that consumer group for all partitions are 0, which means that upon restarting a consumer in that group, it will read from the beginning of each partition: ``` kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application third message fifth message seventh message tenth message first message fourth message eigth message hello world second message sixth message ninth message ``` **Note that messages are read in order for each partition, not across partitions (we have 3 partitions in this example) - we will never stop reminding you this.** ## Example: Reset offsets shift by Stop the running consumers to be able to reset offsets. Shift by allows you to rewind offsets by a specific value (negative to go back in messages and positive to advance in messages). ``` kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-first-application Consumer group 'my-first-application' has no active members. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-first-application first_topic 0 3 3 0 - - - my-first-application first_topic 1 5 5 0 - - - my-first-application first_topic 2 6 6 0 - - - ``` In this example we reset offsets by shifting by `-2` for the consumer group `my-first-application` subscribed on the topic `first_topic` ``` kafka-consumer-groups --bootstrap-server localhost:9092 --group my-first-application --reset-offsets --shift-by -2 --execute --topic first_topic GROUP TOPIC PARTITION NEW-OFFSET my-first-application first_topic 0 1 my-first-application first_topic 1 3 my-first-application first_topic 2 4 ``` As you can see the offsets decreased by 2 for each partition. Read messages from the topic first\_topic using Kafka console consumer CLI. It will only return last 2 messages from each partition of the topic. ``` kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application seventh message tenth message fourth message eigth message sixth message ninth message ``` ### Gotchas Here are the common mistakes and caveats with the `kafka-consumer-groups.sh` command: * **You cannot reset a consumer group if consumers are active in it.** * This command can be used to reprocess data for a consumer group (in case you have a bug fix) * This command be also be used to advance message consumption in Kafka (for example if a message is a poison pill, or if your consumer is too slow to catch up with the entire topic). ### Extra: Important options you can set (advanced) `--all-groups` Applies to all groups, use with caution `--all-topics` Consider all topics assigned to a group in the reset-offsets process, use with caution `--by-duration` Reset to offsets by duration `--dry-run` Only show the expected result, but does not actually run the command `--to-datetime`, `--by-period`, `--to-earliest`, `--to-latest`, `--shift-by`, `--from-file`, `--to-current` All the various options available to you to reset the offsets ## How to list all Kafka consumers in a consumer groups using the CLI? Listing all the Kafka consumer in a consumer group help you understand where the consumers are placed on your network, and how far they are into the topic consumption. ``` kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-first-application GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID my-first-application first_topic 0 3 3 0 consumer-my-first-application-1-0237b0a1-911d-45f1-891f-8fd7630a7593 /172.19.0.1 consumer-my-first-application-1 my-first-application first_topic 1 5 5 0 consumer-my-first-application-1-70ddc756-8dbc-45e4-b5b6-1e5a75db9e62 /172.19.0.1 consumer-my-first-application-1 my-first-application first_topic 2 6 6 0 consumer-my-first-application-1-a8ce2af3-97b3-4445-bba3-f4a5f6c4464d /172.19.0.1 consumer-my-first-application-1 ``` The `CONSUMER ID` represents the unique identifier of the consumer to the Kafka broker The `CLIENT ID` represents a client-side setting that you can optionally set to identify a consumer in your consumer groups (with the `client.id` consumer property) The `CURRENT-OFFSET` is the latest committed offset for that group The `LOG-END-OFFSET` represents the latest message offset available in the topic-partition for consumption The `LAG` is the difference of `LOG-END-OFFSET` and `CURRENT-OFFSET` and represents how far behind a consumer is to the tail of a topic. The `HOST` is the hostname / IP of the consumer client machine. ## How to list all Kafka consumers groups using the CLI? Listing the consumer groups help you understand which ones could be down, or not stable. **Listing consumer groups state** ``` kafka-consumer-groups --bootstrap-server localhost:9092 --list --state GROUP STATE my-first-application Stable ``` **Describe all consumer groups and state** (helpful for assignment strategy and coordinator ID) ``` kafka-consumer-groups --bootstrap-server localhost:9092 --describe --all-groups --state GROUP COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS my-first-application 127.0.0.1:9092 (1) range Stable 3 ``` You can instead of using `--all-groups` just specify the command for one group only. > **Note:** > As of Kafka >= 2.1, the "console-consumer-" groups will not appear in the output ## How to delete a consumer group in Kafka using the CLI? You may want to delete a consumer group in order to reset entirely the reading mechanism. For this, you can use the `delete` option: ``` $ kafka-consumer-groups --bootstrap-server localhost:9092 --delete --group my-first-application Deletion of requested consumer groups ('my-first-application') was successful. ``` Alternatively, if you want to only delete offsets for a specific topic (helpful when your consumer group is reading from multiple topics) you can use the following command: ``` kafka-consumer-groups --bootstrap-server localhost:9092 --delete-offsets --group my-first-application --topic first_topic Request succeed for deleting offsets with topic first_topic group my-first-application TOPIC PARTITION STATUS first_topic 0 Successful first_topic 1 Successful first_topic 2 Successful ``` ## Troubleshoot consumer lag Consumer lag occurs when consumers can't keep up with the rate of incoming messages. Understanding and resolving lag is critical for maintaining real-time data processing. ### Understand LAG **LAG** = `LOG-END-OFFSET` - `CURRENT-OFFSET` - **LAG = 0**: Consumer is fully caught up - **LAG > 0 and stable**: Consumer is behind but processing consistently - **LAG > 0 and growing**: Consumer is falling further behind (problem!) ### Consumer lag visualization This diagram shows how lag accumulates when a consumer cannot keep up with the producer: ![](https://www.conduktor.io/assets/kafka/consumer-lag.png) **Understanding the diagram:** - Producer has written seven messages (offset 0-6, LOG-END-OFFSET is 7) - Consumer has only processed three messages (CURRENT-OFFSET is 3) - LAG of 4 means four unprocessed messages are waiting - If LAG keeps growing, the consumer is too slow ### Check consumer lag ```bash kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-consumer-group GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG my-consumer-group orders 0 1500 2500 1000 my-consumer-group orders 1 2300 2350 50 my-consumer-group orders 2 900 5000 4100 ``` In this example: - Partition 0: 1,000 messages behind - Partition 1: 50 messages behind (acceptable) - Partition 2: 4,100 messages behind (critical!) ### Common causes and solutions #### 1. Slow message processing **Symptoms:** - LAG increases steadily over time - Consumer CPU usage is high - Processing time per message is high **Diagnosis:** ```bash # Check if lag is growing kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-consumer-group # Run again after 60 seconds and compare LAG values ``` **Solutions:** - Optimize consumer code (reduce processing time per message) - Add more consumer instances (up to number of partitions) - Increase consumer threads (`num.streams` in older API) - Use async processing where possible #### 2. Under-partitioned topic **Symptoms:** - Maximum consumers reached but still have lag - Cannot add more consumers to scale - Single partition has very high lag **Diagnosis:** ```bash # Check partition count kafka-topics --bootstrap-server localhost:9092 \ --describe --topic orders Topic: orders PartitionCount: 3 # Only 3 partitions! ``` **Solutions:** - Increase partition count (careful: can't decrease later) - Create new topic with more partitions and migrate - Consider if messages can be processed in parallel #### 3. Consumer rebalancing **Symptoms:** - Periodic spikes in lag - LAG increases then decreases repeatedly - Consumer logs show "Revoke" and "Assign" messages **Diagnosis:** Check consumer logs for rebalancing events: ``` [Consumer] Revoking previously assigned partitions [Consumer] partitions lost: [orders-0, orders-1] ``` **Solutions:** - Use [incremental cooperative rebalancing](https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership) - Increase `session.timeout.ms` (default: 10s → 45s) - Increase `max.poll.interval.ms` if processing takes long - Enable static group membership for planned restarts #### 4. Network or broker issues **Symptoms:** - Sudden spike in lag across all consumers - Intermittent connection errors - Broker CPU or disk I/O is saturated **Diagnosis:** ```bash # Check broker health kafka-broker-api-versions --bootstrap-server localhost:9092 # Monitor broker metrics (requires JMX) # Check: CPU, disk I/O, network throughput ``` **Solutions:** - Check network connectivity between consumers and brokers - Add more brokers if cluster is overloaded - Optimize broker configuration (buffer sizes, threads) - Check disk performance (especially if using spinning disks) ### Troubleshoot with this decision tree ![](https://www.conduktor.io/assets/kafka/consumer-lag-decide.png) ### Best practices **Monitor lag continuously:** - Set up alerts for LAG > threshold (e.g., 1000 messages) - Track lag growth rate, not just absolute value - Monitor per-partition lag, not just group average **Prevent lag:** - Start with enough partitions (2-3x expected consumers) - Use incremental rebalancing to minimize disruption - Optimize consumer processing before adding instances - Test consumer performance under load before production > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-consumer-group-management-cli-tutorial) provides real-time consumer lag monitoring with visual graphs, alerting thresholds, and historical trends. Identify lagging partitions at a glance and drill down to see consumer group details, rebalancing events, and processing rates. ## Next steps - [Run Kafka Connect from the CLI](https://www.conduktor.io/kafka/kafka-connect-cli-tutorial) to stream data in and out of Kafka - [Configure consumer group settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior) for optimal performance - [Learn about delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers) for reliable processing Source: https://www.conduktor.io/kafka/kafka-consumer-group-management-cli-tutorial --- # Kafka consumer groups and offsets *Learn how consumer groups enable parallel processing* Consumer groups allow multiple consumers to work together to process data from Kafka topics. Understanding consumer groups and offsets is essential for building scalable, fault-tolerant applications. **What you'll learn:** - How consumer groups coordinate partition assignment - How offsets track consumption progress - The different delivery semantics (at-most-once, at-least-once, exactly-once) - Best practices for consumer group configuration ## Kafka consumer groups We have seen that consumers can consume data from Kafka topics partitions individually, but for horizontal scalability purposes it is recommended to consume Kafka topics as a group. Consumers that are part of the same application and therefore performing the same "logical job" can be grouped together as a Kafka consumer group. A topic usually consists of many partitions. These partitions are a unit of parallelism for Kafka consumers. The benefit of leveraging a Kafka consumer group is that the consumers within the group will coordinate to split the work of reading from different partitions. ![Apache Kafka Consumer Group diagram showing how a consumer group reads messages from a Kafka topic with 5 partitions.](https://www.conduktor.io/assets/kafka/Consumer-Group-reading-from-topic-with-5-partitions.png) ### Kafka consumer group ID In order for indicating to Kafka consumers that they are part of the same specific group, we have to specify the consumer-side setting `group.id`. Kafka Consumers automatically use a `GroupCoordinator` and a `ConsumerCoordinator` to assign consumers to a partition and ensure the load balancing is achieved across all consumers in the same group. It is important to note that each topic partition is only assigned to one consumer within a consumer group, but a consumer from a consumer group can be assigned multiple partitions. ![Apache Kafka Consumer Group diagram showing how a consumer group reads messages from a Kafka topic with 5 partitions.](https://www.conduktor.io/assets/kafka/Consumer-Group-reading-from-topic-with-5-partitions.png) In the example above, **Consumer 1** of consumer group `consumer-group-application` has been assigned *Partition 0* and *Partition 1*, whereas **Consumer 2** is assigned *Partition 2* and *Partition 3*. Finally, **Consumer 3** is assigned *Partition 4*. Only **Consumer 1** receives messages from *Partition 0* and *Partition 1*, while only **Consumer 2** receives messages from *Partition 2* and *Partition 3*, with only **Consumer 3** receiving messages from *Partition 4*. Each of your applications (that may be composed of many consumers) reading from Kafka topics has to specify a different `group.id`. That means that multiple applications (consumer groups) can consume from the same topic at the same time: ![Diagram showing consumers within a consumer group reading messages from different topic partitions.](https://www.conduktor.io/assets/kafka/Kafka-Consumer-Groups-1.png) ### Consumer scaling limits If there are more consumers than the number of partitions of a topic, then some consumers will remain inactive as shown below. Usually, we have as many consumers in a consumer group as the number of partitions. If we want more consumers for higher throughput, we should create more partitions while creating the topic. Otherwise, some of the consumers may remain inactive. ![Diagram shows Consumer in a Kafka Consumer Group inactive when there are more consumers than partitions.](https://www.conduktor.io/assets/kafka/Kafka-Consumer-Groups-2.png) > **Maximum parallelism** > The maximum number of active consumers in a group equals the number of partitions. Adding more consumers than partitions results in idle consumers. ### Consumer group scaling formula | Partitions | Consumers | Result | |------------|-----------|--------| | 3 | 1 | 1 consumer handles all 3 partitions | | 3 | 2 | Each consumer handles 1-2 partitions | | 3 | 3 | Each consumer handles 1 partition (optimal) | | 3 | 5 | 2 consumers idle, 3 active | ## Kafka consumer offsets Kafka brokers use an internal topic named `__consumer_offsets` that keeps track of what messages a given **consumer group** last successfully processed. As we know, each message in a Kafka topic has a partition ID and an offset ID attached to it. Therefore, in order to "checkpoint" how far a consumer has been reading into a topic partition, the consumer will regularly **commit** the latest processed message, also known as **consumer offset**. In the figure below, a consumer from the consumer group has consumed messages up to offset `4262`, so the consumer offset is set to `4262`. ![Diagram showing how Kafka Consumers from a Consumer Group read messages from the last committed consumer offset.](https://www.conduktor.io/assets/kafka/Kafka-Consumer-Groups-3-2x.png) Most client libraries automatically commit offsets to Kafka for you on a periodic basis, and the responsible Kafka broker will ensure writing to the `__consumer_offsets` topic (therefore consumers do not write to that topic directly). The process of committing offsets is not done for every message consumed (because this would be inefficient), and instead is a periodic process. This also means that when a specific offset is committed, all previous messages that have a lower offset are also considered to be committed. ## Why use consumer offsets? Offsets are critical for many applications. If a Kafka client crashes, a rebalance occurs and the latest committed offset help the remaining Kafka consumers know where to restart reading and processing messages. In case a new consumer is added to a group, another consumer group rebalance happens and consumer offsets are yet again leveraged to notify consumers where to start reading data from. Therefore consumer offsets have to be committed regularly. ![Consumer offset recovery flow: before a crash, the consumer is reading a partition at offset 100 with the last committed offset at 95; after recovery, the restarted consumer resumes from offset 95 and reprocesses messages 95 to 100.](https://www.conduktor.io/assets/kafka/diagrams/kafka-consumer-groups-and-consumer-offsets.svg) ```mermaid flowchart LR subgraph Before["Before crash"] C1["Consumer"] -->|"Reading"| P["Partition
Offset: 100"] C1 -->|"Committed"| O1["Offset: 95"] end subgraph After["After recovery"] C2["Consumer
(restarted)"] -->|"Resume from"| O2["Offset: 95"] C2 -->|"Reprocess"| M["Messages 95-100"] end Before --> After ``` ### Delivery semantics for consumers By default, Java consumers automatically commit offsets (controlled by the `enable.auto.commit=true` property) every `auto.commit.interval.ms` (5 seconds by default) when `.poll()` is called. Details of that mechanism are discussed in [Delivery Semantics for Consumers](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers). A consumer may opt to commit offsets by itself (`enable.auto.commit=false`). Depending on when it chooses to commit offsets, there are delivery semantics available to the consumer. The three delivery semantics are explained below. ### At most once - Offsets are committed as soon as the message is received. - If the processing goes wrong, the message will be lost (it won't be read again). ### At least once (usually preferred) - Offsets are committed after the message is processed. - If the processing goes wrong, the message will be read again. - This can result in duplicate processing of messages. Therefore, it is best practice to make sure data processing is idempotent (i.e. processing the same message twice won't produce any undesirable effects) ### Exactly once - This can only be achieved for Kafka topic to Kafka topic workflows using the transactions API. The Kafka Streams API simplifies the usage of that API and enables exactly once using the setting `processing.guarantee=exactly_once_v2` (`exactly_once` on Kafka < 2.5) - For Kafka topic to External System workflows, to *effectively* achieve exactly once, you have to use an idempotent consumer. > **Recommended approach** > In practice, at least once with idempotent processing is the most desirable and widely implemented mechanism for Kafka consumers. | Semantic | Commits when | Risk | Use case | |----------|--------------|------|----------| | At most once | Before processing | Data loss | Metrics, logs | | At least once | After processing | Duplicates | Most applications | | Exactly once | With transaction | Complexity | Financial, critical | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-consumer-groups-and-consumer-offsets) provides a visual interface to manage Kafka consumer groups. View consumer group details, monitor lag per partition, track offset commits, and perform operations like resetting offsets or deleting consumer groups. ## Next steps - [Meet the brokers that run the cluster](https://www.conduktor.io/kafka/kafka-brokers) to see the servers that store partitions and serve clients - [Understand incremental rebalancing](https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership) to minimize disruption - [Explore delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers) in depth Source: https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets --- # Consumer poll and heartbeat settings *Learn how to tune Kafka consumer settings for optimal performance* Kafka consumers use sophisticated polling and heartbeat mechanisms to efficiently fetch data while maintaining group membership. Understanding these settings is essential for building high-performance, reliable consumer applications. **What you'll learn:** - How the consumer poll loop works internally - The relationship between heartbeat and poll threads - Key configuration settings and their impact - How to tune for throughput vs latency ## Consumer poll behavior Kafka consumers poll the Kafka broker to receive batches of data. Once the consumer is subscribed to Kafka topics, the poll loop handles all details of coordination, partition rebalances, heartbeats, and data fetching, leaving the developer with a clean API that simply returns available data from assigned partitions. > **Internal code optimization** > If the consumer successfully fetched some data from Kafka, it will start sending the next fetch requests ahead of time, so that while processing the current batch, there will be less waiting on the next `.poll()` call. ![Kafka Consumer Poll Behavior](https://www.conduktor.io/assets/kafka/Adv-Consumer-Poll-Behaviour-1.png) Polling allows consumers to control: - From where in the log they want to consume - How fast they want to consume - Ability to replay events ## Internal poll thread and heartbeat thread The way consumers maintain membership in a consumer group and ownership of partitions is by sending `heartbeats` to a Kafka broker designated as the **group coordinator**. ![Kafka Consumer Internal Threads](https://www.conduktor.io/assets/kafka/Adv-Consumer-Internal-Threads-1.png) ![Consumer internal threads: the poll thread running your code exchanges poll() calls and records with a broker, while a background heartbeat thread exchanges heartbeats and rebalance signals with the group coordinator](https://www.conduktor.io/assets/kafka/diagrams/kafka-consumer-important-settings-poll-and-internal-threads-behavior--1.svg) ```mermaid flowchart LR subgraph Consumer["Consumer Application"] PT["Poll Thread
(Your code)"] HT["Heartbeat Thread
(Background)"] end subgraph Kafka["Kafka Cluster"] GC["Group Coordinator"] B["Broker"] end PT -->|"poll()"| B HT -->|"heartbeats"| GC B -->|"records"| PT GC -->|"rebalance
signals"| HT ``` ### Kafka consumer heartbeat thread Heartbeats help to determine consumer liveliness: - As long as the consumer sends heartbeats at regular intervals, it is assumed to be alive and processing messages - If the consumer stops sending heartbeats long enough, its session will time out and trigger a rebalance Key configurations: - `heartbeat.interval.ms` (default: 3 seconds) - `session.timeout.ms` (Kafka v3.0+: 45 seconds) > This mechanism detects consumer application downtime or network failures. ### Kafka consumer poll thread Consumers poll brokers periodically using the `.poll()` method. Key configuration: - `max.poll.interval.ms` (default: 5 minutes) This controls the maximum time between calls to `poll()`. If this interval is exceeded, the consumer is considered failed and triggers a rebalance. ## Important consumer settings ### Poll behavior settings | Setting | Default | Description | |---------|---------|-------------| | `max.poll.records` | 500 | Maximum records returned in single `poll()` | | `fetch.min.bytes` | 1 | Minimum data to return for a fetch request | | `fetch.max.wait.ms` | 500ms | Maximum wait time if insufficient data | **max.poll.records** - Lower values can improve latency but may reduce throughput - Higher values improve throughput but may increase processing time per batch **fetch.min.bytes** - Setting higher values can improve throughput by reducing request overhead - May increase latency as consumer waits for more data ### Session and heartbeat settings | Setting | Default (Kafka 3.0+) | Description | |---------|---------------------|-------------| | `session.timeout.ms` | 45 seconds | Timeout for detecting consumer failures | | `heartbeat.interval.ms` | 3 seconds | Expected time between heartbeats | | `max.poll.interval.ms` | 5 minutes | Maximum delay between `poll()` calls | > **Heartbeat interval rule** > Set `heartbeat.interval.ms` to approximately 1/3 of `session.timeout.ms`. This ensures the consumer sends enough heartbeats within the session timeout. ## Performance tuning guidelines ### For high throughput ```properties # Maximize batch sizes and reduce overhead max.poll.records=1000 fetch.min.bytes=1048576 # 1MB fetch.max.wait.ms=1000 # Allow more time for processing large batches max.poll.interval.ms=300000 # 5 minutes ``` ### For low latency ```properties # Return data quickly, small batches max.poll.records=100 fetch.min.bytes=1 fetch.max.wait.ms=100 # Shorter timeouts for faster rebalances session.timeout.ms=10000 # 10 seconds heartbeat.interval.ms=3000 # 3 seconds ``` ### For long processing times ```properties # Accommodate slow processing without triggering rebalance max.poll.interval.ms=600000 # 10 minutes session.timeout.ms=60000 # 1 minute heartbeat.interval.ms=20000 # 20 seconds # Smaller batches to stay within poll interval max.poll.records=100 ``` ### Decision tree ![Tuning decision tree: throughput priority leads to a high-throughput config (max.poll.records=1000, fetch.min.bytes=1MB), latency leads to a low-latency config (max.poll.records=100, fetch.min.bytes=1), and processing time leads to a long-processing config (max.poll.interval.ms=10min, max.poll.records=100), all converging on monitoring consumer lag](https://www.conduktor.io/assets/kafka/diagrams/kafka-consumer-important-settings-poll-and-internal-threads-behavior--2.svg) ```mermaid flowchart TD Start["Tuning consumer"] --> Q1{"Priority?"} Q1 -->|"Throughput"| HT["High throughput config
max.poll.records=1000
fetch.min.bytes=1MB"] Q1 -->|"Latency"| LL["Low latency config
max.poll.records=100
fetch.min.bytes=1"] Q1 -->|"Processing time"| LP["Long processing config
max.poll.interval.ms=10min
max.poll.records=100"] HT --> Monitor["Monitor consumer lag"] LL --> Monitor LP --> Monitor ``` ## Best practices 1. **Tune `max.poll.records`** based on your processing time per message 2. **Set `max.poll.interval.ms`** higher than your worst-case processing time 3. **Monitor consumer lag** to ensure settings are appropriate 4. **Test rebalance behavior** under your expected load conditions 5. **Consider batch processing** patterns when setting poll configurations > **Avoid blocking in poll loop** > Never perform long-running operations in the thread that calls `poll()`. This can trigger unnecessary rebalances and degrade performance. Use separate worker threads for heavy processing. ## Common anti-patterns | Anti-pattern | Problem | Solution | |--------------|---------|----------| | Blocking in poll loop | Triggers rebalance | Use async processing | | `max.poll.interval.ms` too low | Constant rebalances | Increase or reduce batch size | | Ignoring heartbeat settings | Slow failure detection | Tune for your SLAs | | Same config for all consumers | Suboptimal performance | Tune per use case | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-consumer-important-settings-poll-and-internal-threads-behavior) displays real-time consumer lag and rebalance events. Monitor how your configuration changes affect consumer performance and identify optimal settings for your workload. ## Next steps - [Configure auto offset reset](https://www.conduktor.io/kafka/consumer-auto-offsets-reset-behavior) for new consumers - [Understand delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers) for reliable processing - [Implement incremental rebalancing](https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership) to reduce disruption Source: https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior --- # Advanced Kafka consumers Learn about advanced Kafka consumer configurations, behaviors and optimization strategies for production deployments. ## What you'll learn - Consumer delivery semantics and reliability guarantees - Poll behavior and internal thread management - Offset reset strategies and recovery scenarios - Geographic optimization with closest replica reading - Incremental rebalancing and static group membership - Performance tuning and operational best practices ## Topics covered - **[Delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers)**: at-most-once, at-least-once, and exactly-once delivery - **[Consumer important settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior)**: poll behavior and internal threads - **[Auto offset reset](https://www.conduktor.io/kafka/consumer-auto-offsets-reset-behavior)**: offset reset behavior and strategies - **[Closest replica](https://www.conduktor.io/kafka/consumer-read-from-closest-replica)**: reading from geographically closest replicas - **[Incremental rebalance](https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership)**: cooperative rebalancing and static membership ## Next steps - [Start with delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers) to understand at-most-once, at-least-once, and exactly-once - [Tune poll and thread settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior) for stable, performant consumption - [Configure offset reset behavior](https://www.conduktor.io/kafka/consumer-auto-offsets-reset-behavior) for recovery scenarios Source: https://www.conduktor.io/kafka/kafka-consumers-advanced --- # Kafka consumer groups CLI tutorial *Learn how Kafka consumer groups work through hands-on CLI practice* Consumer groups enable parallel message processing by distributing partitions among group members. This tutorial demonstrates consumer group behavior using the `kafka-console-consumer` CLI. **What you'll learn:** - How to create multiple consumers in the same group - How partitions are distributed among consumers - What happens when consumers join or leave a group - How offsets are committed and resumed > Use CLI commands with appropriate extensions for your platform: for Windows - `kafka-console-consumer.bat`, for Mac and Linux - `kafka-console-consumer.sh`. In addition, we will provide an optional consumer group parameter with `--group` flag. ## How to create consumers in a Kafka consumer group? To start consumers in a consumer group, do the following: 1. Create a topic with at least 2 partitions and send data to it 2. Create a first `kafka-console-consumer` and assign a group name with `--group` 3. Open a new terminal / shell window 4. Create a second `kafka-console-consumer` and use the same `--group` argument 5. Send data to the topic and you will see consumers sharing the reads If you need a refresh on how consumers in a consumer group work, [have a read here](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets). ### Create consumer group example You cannot have more consumers in a group than partitions in your Kafka topic, and therefore we first need to create a Kafka topic with a few partitions (in the example 3). ``` kafka-topics --bootstrap-server localhost:9092 --topic first_topic --create --partitions 3 --replication-factor 1 ``` Then launch a consumer in a consumer group, named `my-first-application` ``` kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application ``` Open a new terminal/shell window and launch a second consumer in the same consumer group `my-first-application` (note we're using the exact same command) ``` kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application ``` Open a new terminal/shell window and launch a third consumer in the same consumer group `my-first-application` ``` kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application ``` Each consumer in the consumer group `my-first-application` will get assigned a partition. Produce a few string messages in the topic. ``` $ kafka-console-producer --bootstrap-server localhost:9092 --topic first_topic >first message >second message >third message >fourth message ``` Each consumer will show only the messages produced on the partition that are assigned to it. If you stop a consumer, messages automatically get sent to the remaining consumers because consumers in a consumer group automatically perform a **consumer rebalance**. ### Stop all consumers **And keep on producing to the topic** ``` >eigth message >ninth message >tenth message ``` **Upon restart of a consumer in the group, the consumer will read from the latest committed offsets and read only the messages you've just produced** ``` $ kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application eigth message ninth message tenth message ``` You have seen how consumers work in consumer groups! ### Gotchas * If you consume in a consumer groups using the `--group` command, then if you try using the `--from-beginning` option afterwards with the same group, it will be ignored. Instead, you need to reset your consumer groups as [shown here](https://www.conduktor.io/kafka/kafka-consumer-group-management-cli-tutorial). * If you don't specify a --group option, the consumer group of the consumer will be a random consumer group such as console-consumer-11984 * If you see one consumer getting all the messages, that probably means that your topic was only created with 1 partition, which you can verify with the `kafka-topics --describe` command ## Consumer group behavior summary ![Flowchart: a producer writes to a topic with three partitions (0, 1, 2), and each partition is assigned to one consumer in the consumer group (partition 0 to consumer 1, partition 1 to consumer 2, partition 2 to consumer 3)](https://www.conduktor.io/assets/kafka/diagrams/kafka-consumers-in-group-cli-tutorial.svg) ```mermaid flowchart LR subgraph Producers P["Producer"] end subgraph Topic["Topic (3 partitions)"] P0["Partition 0"] P1["Partition 1"] P2["Partition 2"] end subgraph Group["Consumer Group"] C1["Consumer 1"] C2["Consumer 2"] C3["Consumer 3"] end P --> P0 & P1 & P2 P0 --> C1 P1 --> C2 P2 --> C3 ``` | Scenario | Result | |----------|--------| | 3 consumers, 3 partitions | Each consumer gets 1 partition | | 2 consumers, 3 partitions | One consumer gets 2 partitions | | 4 consumers, 3 partitions | One consumer sits idle | | Consumer leaves group | Partitions rebalanced to remaining consumers | | Consumer joins group | Rebalance distributes partitions | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-consumers-in-group-cli-tutorial) displays consumer group membership, partition assignments, and lag metrics in real-time. Monitor rebalances and track consumer health visually. ## Next steps - [Manage and reset consumer group offsets](https://www.conduktor.io/kafka/kafka-consumer-group-management-cli-tutorial) with kafka-consumer-groups - [Understand incremental rebalancing](https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership) for production - [Configure consumer settings](https://www.conduktor.io/kafka/kafka-consumer-important-settings-poll-and-internal-threads-behavior) for stability Source: https://www.conduktor.io/kafka/kafka-consumers-in-group-cli-tutorial --- # Kafka consumers *Learn how Kafka consumers read data from topics* Consumers are applications that read data from Kafka topics. Understanding how consumers work, including deserialization and the pull model, is essential for building reliable data processing applications. **What you'll learn:** - How consumers read messages from Kafka topics - How message deserialization works - The consumer pull model and its benefits - Best practices for message format compatibility ## Kafka consumers Once a topic has been created in Kafka and data has been placed in the topic, we can start to build applications that make use of this data stream. Applications that pull event data from one or more Kafka topics are known as Kafka consumers. Applications that read data from Kafka topics are known as consumers. Applications integrate a Kafka client library to read from Apache Kafka. Excellent client libraries exist for almost [all programming languages](https://www.conduktor.io/kafka/kafka-sdk-list) that are popular today including Python, Java, Go, and others. Consumers can read from one or more partitions at a time in Apache Kafka, and data is read in order **within each partition** as shown below. ![Kafka consumers in this diagram are reading messages from various Apache Kafka Brokers and Topics.](https://www.conduktor.io/assets/kafka/Kafka-Consumers-1.png) ## How consumers read data A consumer always reads data from a lower offset to a higher offset and cannot read data backwards (due to how Apache Kafka and clients are implemented). If the consumer consumes data from more than one partition, the message order is not guaranteed across multiple partitions because they are consumed simultaneously, but the message read order is still guaranteed within each individual partition. By default, Kafka consumers will only consume data that was produced after it first connected to Kafka. Which means that to read historical data in Kafka, one has to specify it as an input to the command, as we will see in the practice section. ### Consumer pull model Kafka consumers are also known to implement a "pull model". This means that Kafka consumers have to request data from Kafka brokers in order to get it (instead of having Kafka brokers continuously push data to consumers). This implementation was made so that consumers can control the speed at which the topics are being consumed. ![Sequence diagram of the Kafka consumer pull model: the consumer calls poll() to request messages, the broker returns a batch, the consumer processes it, then polls again for the next batch.](https://www.conduktor.io/assets/kafka/diagrams/kafka-consumers.svg) ```mermaid sequenceDiagram participant Consumer participant Broker Consumer->>Broker: poll() - request messages Broker-->>Consumer: Return batch of messages Note over Consumer: Process messages Consumer->>Broker: poll() - request more Broker-->>Consumer: Return next batch ``` **Benefits of the pull model:** - Consumers control their own consumption rate - Slow consumers don't affect broker performance - Consumers can batch process messages efficiently - Natural backpressure handling ## Kafka message deserializers > Consumed data has to be deserialized in the same format it was serialized in. As we have seen before, the data sent by the Kafka producers is [serialized](https://www.conduktor.io/kafka/kafka-producers). This means that the data received by the Kafka consumers has to be correctly deserialized in order to be useful within your application. Data being consumed has to be deserialized in the same format it was serialized in. For example: - if the producer serialized a `String` using `StringSerializer`, the consumer has to deserialize it using `StringDeserializer` - if the producer serialized an `Integer` using `IntegerSerializer`, the consumer has to deserialize it using `IntegerDeserializer` ![Kafka Consumers must use the same format for deserialization that was used by the producer when serializing the message. This diagram shows the deserialization process.](https://www.conduktor.io/assets/kafka/Kafka-Consumers-2.png) ### Serialization compatibility The serialization and deserialization format of a topic should not change during a topic lifecycle. If you intend to switch a topic data format (for example from JSON to Avro), it is considered best practice to create a new topic and migrate your applications to leverage that new topic. > **Poison pills** > Messages sent to a Kafka topic that do not respect the agreed-upon serialization format are called **poison pills**. [They are not fun to deal with.](https://www.slideshare.net/ConfluentInc/streaming-apps-and-poison-pills-handle-the-unexpected-with-kafka-streams-loic-divad-xebia-france-kafka-summit-sf-2019) Failure to correctly deserialize may cause crashes or inconsistent data being fed to the downstream processing applications. This can be tough to debug, so it is best to think about it as you're writing your code the first time. ### Handling deserialization errors | Strategy | When to use | |----------|-------------| | Fail fast | Development, testing | | Log and skip | Non-critical data, metrics | | Dead letter queue | Production, data recovery needed | | Schema validation | Prevent bad data at producer | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-consumers) lets you consume and browse messages from topics directly in the UI. View message keys, values, headers, and timestamps with automatic deserialization support for common formats. ## Next steps - [Scale with consumer groups and offsets](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets) to understand parallel consumption and progress tracking - [Explore delivery semantics](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers) for exactly-once processing - [Write a Java consumer](https://www.conduktor.io/kafka/complete-kafka-consumer-with-java) with hands-on code examples Source: https://www.conduktor.io/kafka/kafka-consumers --- # Kafka fundamentals Learn the core concepts that power Apache Kafka. This guide covers everything you need to understand how Kafka works under the hood. ## Learning path Work through these topics in order to build a solid foundation: ### 1. Core concepts - [**What are Kafka topics?**](https://www.conduktor.io/kafka/kafka-topics) - learn about topics, partitions, and offsets - [**Kafka producers**](https://www.conduktor.io/kafka/kafka-producers) - how applications send data to Kafka - [**Kafka consumers**](https://www.conduktor.io/kafka/kafka-consumers) - how applications read data from Kafka ### 2. Advanced concepts - [**Consumer groups and offsets**](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets) - understanding consumer coordination - [**Kafka brokers**](https://www.conduktor.io/kafka/kafka-brokers) - the servers that make up a Kafka cluster - [**Topic replication**](https://www.conduktor.io/kafka/kafka-topic-replication) - how Kafka ensures data durability ### 3. Infrastructure - [**ZooKeeper**](https://www.conduktor.io/kafka/zookeeper-with-kafka) - Kafka's coordination service (legacy) - [**KRaft mode**](https://www.conduktor.io/kafka/kafka-kraft-mode) - Kafka without ZooKeeper (modern approach) ## What you'll learn After completing this section, you'll have a deep understanding of: - **Data flow**: how messages move through Kafka - **Scalability**: how Kafka handles millions of messages - **Reliability**: how Kafka ensures no data loss - **Architecture**: how all components work together ## Next steps - [Start with Kafka topics](https://www.conduktor.io/kafka/kafka-topics) to understand the foundation of how Kafka organizes data - [Learn how producers write data](https://www.conduktor.io/kafka/kafka-producers) to see how events get into Kafka - [Learn how consumers read data](https://www.conduktor.io/kafka/kafka-consumers) to understand the other side of the pipeline Source: https://www.conduktor.io/kafka/kafka-fundamentals --- # Kafka KRaft mode *Learn about Kafka's new consensus protocol in eight minutes* KRaft (Kafka Raft) mode is Kafka's built-in consensus protocol that replaces ZooKeeper for cluster coordination. This architectural change simplifies Kafka deployments and enables better scalability. **What you'll learn:** - Why ZooKeeper is being removed from Kafka - How KRaft mode works - Benefits of running Kafka without ZooKeeper - When to use KRaft mode ## Why remove ZooKeeper from Kafka? The Kafka project undertook one of its greatest changes with the introduction of [KIP-500](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum) on August 1st 2019: the desire to remove ZooKeeper as a dependency to running Apache Kafka. Kafka scaling has hit a performance bottleneck with ZooKeeper, which means Kafka has the following limitations with ZooKeeper: - Kafka clusters only support a limited number of partitions (up to 200,000) - When a Kafka broker joins or leaves a cluster, a high number of leader elections have to happen which can overload ZooKeeper and slow down the cluster temporarily - Kafka clusters setup is difficult and depends on another component to setup - Kafka cluster metadata is sometimes out-of-sync from ZooKeeper - ZooKeeper security is lagging behind Kafka security ## Kafka KRaft mode It has been noted as part of KIP-500 that the metadata of Kafka itself is a log and that Kafka brokers should be able to consume that metadata log as an internal metadata topic. Kafka leverages itself! Removing ZooKeeper means that Kafka has to still act as a quorum to perform controller election and therefore the Kafka brokers implement the [Raft protocol](https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum) thus giving the name KRaft to the new Kafka Metadata Quorum mode. ![Diagram showing the difference between Kafka with Zookeeper and Kafka in KRaft mode with Quorum Controller.](https://www.conduktor.io/assets/kafka/Kafka-KRaft-Mode-1.png) ### Architecture comparison ![Architecture comparison: with ZooKeeper, a separate ZooKeeper ensemble coordinates all brokers; with KRaft, dedicated controllers form a quorum among themselves and coordinate the brokers directly with no external dependency.](https://www.conduktor.io/assets/kafka/diagrams/kafka-kraft-mode.svg) ```mermaid graph TB subgraph ZooKeeper["With ZooKeeper"] ZK["ZooKeeper Ensemble"] B1["Broker 1"] B2["Broker 2"] B3["Broker 3"] ZK --> B1 ZK --> B2 ZK --> B3 end subgraph KRaft["With KRaft"] C1["Controller 1"] C2["Controller 2"] C3["Controller 3"] KB1["Broker 1"] KB2["Broker 2"] C1 <--> C2 C2 <--> C3 C1 --> KB1 C2 --> KB2 end ``` ## Benefits of KRaft mode Without ZooKeeper, the following benefits are observed in Kafka: | Benefit | Description | |---------|-------------| | **Scale** | Ability to scale to millions of partitions | | **Simplicity** | Single process to start Kafka, easier to maintain and set up | | **Stability** | Improved stability, easier to monitor, support, and administer | | **Security** | Single security model for the whole system | | **Performance** | Faster controller shutdown and recovery time | > **Production ready since Kafka 3.3** > KRaft was officially released as production ready in Kafka version 3.3. For new deployments, KRaft is the recommended approach. Starting tutorials for [Windows](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows-without-zookeeper-kraft-mode), [Mac](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-without-zookeeper-kraft-mode) and [Linux](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux-without-zookeeper-kraft-mode). ## KRaft deployment modes KRaft supports two deployment modes: ### Combined mode Controllers and brokers run in the same process. Best for: - Development environments - Small clusters (3-5 nodes) - Simplified operations ```properties process.roles=broker,controller ``` ### Isolated mode Controllers and brokers run as separate processes. Best for: - Production environments - Large clusters - Maximum stability ```properties # On controller nodes process.roles=controller # On broker nodes process.roles=broker ``` ## Migration considerations | Current setup | Recommendation | |---------------|----------------| | New cluster | Use KRaft mode | | Kafka 2.x | Upgrade to 3.x, then migrate to KRaft | | Kafka 3.x with ZooKeeper | Migrate to KRaft using official migration tools | | Kafka 4.x | ZooKeeper not supported, must use KRaft | > **Migration planning** > Migrating from ZooKeeper to KRaft requires careful planning. Test thoroughly in non-production environments before migrating production clusters. ## More KRaft resources - [KIP-833: Mark KRaft as Production Ready](https://cwiki.apache.org/confluence/display/KAFKA/KIP-833%3A+Mark+KRaft+as+Production+Ready) - [Benefits of KRaft described in the Confluent blog](https://www.confluent.io/blog/kafka-without-zookeeper-a-sneak-peek/) > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-kraft-mode) supports both ZooKeeper-based and KRaft-based Kafka clusters. Monitor controller status, metadata synchronization, and cluster health regardless of your deployment mode. ## Next steps - [Get started with a running cluster](https://www.conduktor.io/kafka/starting-kafka) to put these concepts into practice hands-on - [Set up Kafka with Docker](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) using KRaft mode - [Install Kafka on Linux with KRaft](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux-without-zookeeper-kraft-mode) for a native setup Source: https://www.conduktor.io/kafka/kafka-kraft-mode --- # Kafka message compression *Learn how to choose and configure Kafka compression algorithms* Kafka producer compression reduces the size of messages before sending them to brokers, improving throughput and reducing network bandwidth at the cost of CPU overhead. **What you'll learn:** - How compression works in Kafka and when to use it - Characteristics of each compression algorithm (GZIP, LZ4, Snappy, ZSTD) - How to choose the right algorithm for your use case - How to monitor compression effectiveness ## Why use compression? Compression provides several benefits in Kafka deployments: - **Reduced network usage**: Smaller message sizes mean less data transmitted - **Lower storage costs**: Compressed messages take less disk space on brokers - **Improved throughput**: More messages can fit in each batch - **Better performance**: Particularly beneficial for text-heavy payloads like JSON or XML > **Compression is most effective with:** > - Text-based formats (JSON, XML, CSV) > - Repetitive data patterns > - Large message payloads (> 1KB) ## Compression algorithms Kafka supports four compression algorithms, each with different performance characteristics: ### GZIP General-purpose compression with high compression ratios but higher CPU usage. **Characteristics:** - Highest compression ratio (smallest message size) - Slowest compression/decompression speed - Good for scenarios where network bandwidth is limited - CPU-intensive **Configuration:** ```properties compression.type=gzip ``` ### LZ4 Fast compression with moderate compression ratios, good balance of speed and size reduction. **Characteristics:** - Very fast compression/decompression - Moderate compression ratio - Low CPU overhead - Good general-purpose choice **Configuration:** ```properties compression.type=lz4 ``` ### Snappy Fast compression optimized for speed with reasonable compression ratios. **Characteristics:** - Fast compression/decompression (slightly slower than LZ4) - Moderate compression ratio - Good CPU efficiency - Popular choice for high-throughput scenarios **Configuration:** ```properties compression.type=snappy ``` ### ZSTD (Zstandard) Modern compression algorithm offering excellent compression ratios with good performance. **Characteristics:** - Excellent compression ratio (better than LZ4/Snappy, approaches GZIP) - Good compression/decompression speed - Relatively new (available since Kafka 2.1) - Tunable compression levels **Configuration:** ```properties compression.type=zstd ``` ## Performance comparison ### Compression ratio comparison Based on typical JSON payloads: | Algorithm | Compression ratio | CPU usage | Speed | |-----------|------------------|-----------|-------| | **GZIP** | ~75% | High | Slow | | **LZ4** | ~65% | Low | Fast | | **Snappy**| ~68% | Low | Fast | | **ZSTD** | ~72% | Medium | Medium| ### Visual comparison: Compression trade-offs This comparison shows how different algorithms balance compression ratio and speed: | Algorithm | Compression ratio | Speed | CPU usage | Best use case | |-----------|------------------|-------|-----------|---------------| | **GZIP** | Highest (~75%) | Slowest | High | Network-constrained environments | | **ZSTD** | High (~72%) | Medium | Medium | **Balanced choice (RECOMMENDED)** | | **LZ4** | Medium (~65%) | **Very fast** | Very Low | High-throughput scenarios | | **Snappy** | Medium (~68%) | Very fast | Low | High-throughput scenarios | **Key insights:** - **ZSTD** offers the best balance: high compression with reasonable speed (ideal for most workloads) - **LZ4 and snappy** prioritize speed over compression (best for real-time/high-throughput) - **GZIP** maximizes compression at the cost of speed (best for bandwidth-limited scenarios) ### Throughput impact Compression affects producer throughput in different ways: **Positive impact:** - Faster network transmission due to smaller message sizes - Higher effective batch sizes (more messages per batch) - Reduced broker disk I/O **Negative impact:** - CPU overhead for compression on producer side - Additional latency for compression process ## Configuration and tuning ### Basic compression configuration ```properties # Enable compression for all messages compression.type=lz4 # Compression happens at batch level batch.size=16384 linger.ms=5 ``` ### Compression at different levels Compression can be configured at multiple levels: **Producer level** (affects all topics): ```java Properties props = new Properties(); props.put("compression.type", "lz4"); Producer producer = new KafkaProducer<>(props); ``` **Topic level** (broker-side configuration): ```bash kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type topics --entity-name my-topic \ --alter --add-config compression.type=snappy ``` ### Interaction with batching Compression works at the batch level, so batching configuration matters: ```properties # Larger batches = better compression ratios batch.size=32768 # 32KB batches linger.ms=10 # Wait up to 10ms to fill batch compression.type=lz4 # Fast compression for batches ``` ## CPU vs network trade-offs ### When to prioritize CPU savings Choose faster algorithms (LZ4, Snappy) when: - CPU resources are limited - High message throughput is required - Network bandwidth is abundant - Low latency is critical ### When to prioritize network savings Choose higher compression (GZIP, ZSTD) when: - Network bandwidth is expensive or limited - Messages are stored for long periods - CPU resources are abundant - Storage costs are a concern ## Choose the right algorithm ### Decision matrix **For high-throughput, low-latency scenarios:** ```properties compression.type=lz4 # Fast compression, low CPU overhead ``` **For network-constrained environments:** ```properties compression.type=gzip # Maximum compression, reduce network usage ``` **For balanced performance:** ```properties compression.type=snappy # Good compression ratio with reasonable speed ``` **For modern deployments (Kafka 2.1+):** ```properties compression.type=zstd # Excellent compression with good performance ``` ### Decision tree: Choose your compression algorithm Use this decision tree to select the optimal compression algorithm based on your requirements: ![](https://www.conduktor.io/assets/kafka/compress-decide.png) > **Quick recommendation:** For most production workloads, use **ZSTD** (Kafka 2.1+) for excellent compression with good performance, or **LZ4** if maximum throughput is critical. ## Monitor compression effectiveness ### Key metrics to track **Compression ratio:** ``` compression_ratio = uncompressed_size / compressed_size ``` **Network savings:** ``` network_savings = (1 - compressed_size / uncompressed_size) × 100% ``` **CPU impact:** Monitor CPU utilization on producer machines before and after enabling compression. ### JMX metrics ``` kafka.producer:type=producer-metrics,client-id= - compression-rate-avg: Average compression ratio - record-size-avg: Average record size before compression - batch-size-avg: Average batch size (after compression) ``` ## Best practices ### Production recommendations 1. **Start with LZ4**: Good balance of speed and compression for most use cases 2. **Test with your data**: Compression effectiveness varies by payload type 3. **Monitor CPU usage**: Ensure compression doesn't become a bottleneck 4. **Combine with batching**: Larger batches compress better 5. **Consider message format**: JSON compresses better than binary formats ### Configuration examples **High-throughput producer:** ```properties compression.type=lz4 batch.size=65536 # Large batches for better compression linger.ms=5 # Short linger time for low latency buffer.memory=67108864 # 64MB buffer for batching ``` **Network-optimized producer:** ```properties compression.type=gzip batch.size=32768 # Reasonable batch size linger.ms=10 # Allow time for batching buffer.memory=134217728 # 128MB buffer for larger compressed batches ``` **Balanced producer (recommended starting point):** ```properties compression.type=snappy batch.size=16384 # Default batch size linger.ms=5 # Low latency buffer.memory=33554432 # 32MB buffer ``` ## Common pitfalls ### Mistakes to avoid 1. **Over-compressing small messages**: Compression overhead may exceed benefits 2. **Ignoring CPU monitoring**: Compression can become a producer bottleneck 3. **Not testing with production data**: Compression ratios vary significantly by content 4. **Using GZIP for high-throughput**: May create CPU bottlenecks 5. **Forgetting about decompression**: Consumers also pay CPU cost for decompression ### Troubleshooting compression issues **Poor compression ratios:** - Check message format (binary data compresses poorly) - Verify batch sizes are adequate - Consider if data is already compressed **High CPU usage:** - Switch to faster algorithm (LZ4 instead of GZIP) - Monitor producer thread CPU utilization - Consider reducing compression level if using ZSTD **Increased latency:** - Reduce linger.ms to decrease batching delay - Use faster compression algorithm - Monitor end-to-end message latency Remember that consumers have to decompress messages, which also uses CPU. Consider the total system CPU cost, not just producer-side overhead. Always benchmark compression algorithms with your actual production data, as compression effectiveness varies significantly based on message format, size, and content patterns. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-message-compression) monitors compression metrics in real-time, showing compression ratios, batch sizes, and CPU impact. View producer performance dashboards to validate your compression configuration choices. ## Next steps - [Configure producer batching](https://www.conduktor.io/kafka/kafka-producer-batching) to maximize compression effectiveness - [Tune custom partitioners](https://www.conduktor.io/kafka/producer-default-partitioner-and-sticky-partitioner) for message distribution - [Understand producer acks](https://www.conduktor.io/kafka/kafka-producer-acks-deep-dive) for durability Source: https://www.conduktor.io/kafka/kafka-message-compression --- # Kafka monitoring and operations *Learn how to monitor Kafka clusters and master essential operations* Effective monitoring is critical for running Kafka reliably. This guide covers how Kafka exposes metrics, which metrics to prioritize, and the operational procedures you need to master. **What you'll learn:** - How Kafka exposes metrics via JMX - Key metrics to monitor for cluster health - Common monitoring tools and integrations - Essential operational procedures ## Kafka monitoring Kafka runs on the JVM and exposes all metrics via Java Management Extensions (JMX). You can collect these metrics using agents that attach to the Kafka process. ![](https://www.conduktor.io/assets/kafka/monitor-flow.png) | Tool | Type | Notes | |------|------|-------| | Prometheus | Open source | Popular with Grafana dashboards | | Datadog | SaaS | Built-in Kafka integration | | New Relic | SaaS | APM with Kafka support | | ELK Stack | Open source | Log aggregation + metrics | | Confluent Control Center | Commercial | Kafka-specific tooling | ### Kafka metrics to monitor | Metric | Description | Alert threshold | |--------|-------------|-----------------| | UnderReplicatedPartitions | Partitions where followers are behind leader | > 0 for extended periods | | OfflinePartitionsCount | Partitions with no available leader | > 0 (critical) | | ActiveControllerCount | Number of active controllers | != 1 (critical) | | RequestHandlerAvgIdlePercent | Thread pool utilization | < 20% | | RequestQueueSize | Pending requests | Growing over time | | NetworkProcessorAvgIdlePercent | Network thread utilization | < 30% | | LogFlushLatency | Time to flush to disk | > baseline | | FetchConsumerTotalTimeMs | Consumer request latency | > baseline | | ProduceTotalTimeMs | Producer request latency | > baseline | Metrics are exposed using JMX in Kafka, although java agents or various vendors can help collect or expose metrics on different ports (for example Prometheus, etc...) Additionally, on top of broker metrics, client metrics in general (Producer, Consumer, Kafka Streams, Kafka Connect...) are important to collect and monitor. This page is meant as an introduction, and more content will soon be created towards metrics and monitoring in Apache Kafka. ### References There are many metrics exposed by Kafka providing information about nearly every function. To learn more about them, these references are very helpful: - [https://kafka.apache.org/documentation/#monitoring](https://kafka.apache.org/documentation/#monitoring) - [https://docs.confluent.io/current/kafka/monitoring.html](https://docs.confluent.io/current/kafka/monitoring.html) - [https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/](https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/) ## Kafka cluster operations For the day-to-day operations of Kafka, there are a number of operations that one has to learn and master to be able to perform them safely. These include: - Rolling Restart of Brokers - Updating Configurations - Rebalancing Partitions - Increasing replication factor - Adding a Broker - Replacing a Broker - Removing a Broker - Upgrading a Kafka Cluster with zero downtime It is important to remember that managing your own cluster comes with all these responsibilities and more. Don't forget to monitor producer and consumer metrics. Client-side metrics often reveal problems before broker metrics do. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-monitoring-and-operations) provides built-in monitoring for broker health, partition status, consumer lag, and throughput metrics. Set up alerts without configuring JMX agents or external monitoring systems. > The [Insights dashboard](https://docs.conduktor.io/guide/insights) analyzes your cluster and identifies topics at risk of data loss, poor cluster efficiency or load imbalance. Monitor business-critical VIP topics and track governance metrics like schema adoption across your infrastructure. ## Next steps - [Plan your cluster setup](https://www.conduktor.io/kafka/kafka-cluster-setup-overview) to size and architect a production deployment - [Set up multi-cluster](https://www.conduktor.io/kafka/kafka-multi-cluster-and-mirrormaker) replication for disaster recovery - [Configure security](https://www.conduktor.io/kafka/kafka-security) for your cluster Source: https://www.conduktor.io/kafka/kafka-monitoring-and-operations --- # Kafka multi-cluster and MirrorMaker *Learn how to replicate data across Kafka clusters* Multi-cluster deployments are common for enterprises with global presence, disaster recovery requirements, or data locality needs. This guide covers cross-cluster replication architectures and tools. **What you'll learn:** - When you need multiple Kafka clusters - Cross-cluster mirroring tools and options - Active-active vs active-passive architectures - Trade-offs and considerations for each approach ## Why multiple clusters? Enterprises commonly need multiple Kafka clusters for: - Geographic data locality (reduce latency) - Disaster recovery (business continuity) - Regulatory compliance (data residency) - Environment isolation (dev/staging/prod) ## Cross-cluster mirroring Mirroring tools consume data from one cluster and produce to another. The mechanism is simple: a consumer + producer. ![](https://www.conduktor.io/assets/kafka/multi-cluster.png) | Tool | Provider | Notes | |------|----------|-------| | MirrorMaker 2 | Apache Kafka | Ships with Kafka, Kafka Connect based | | Confluent Replicator | Confluent | Commercial, additional features | | uReplicator | Uber | Open source, performance optimized | | Custom Flink | Netflix | Custom implementation | ## Active-active architecture This architecture is used when two or more data centers share some or all of the data and each data center is able to both produce and consume events. Producers produce to the same topic in both the clusters, and mirroring occurs between two topics in the clusters. ### Advantages The advantages of this architecture are: - Ability to serve users from a nearby data center, which typically has performance benefits - Redundancy and resilience. Since every data center has all the functionality, if one data center is unavailable you can direct users to a remaining data center. ### Disadvantages The main drawback of this architecture is the challenges in avoiding conflicts when data is read and updated asynchronously in multiple locations. Also, you need to handle where to read from using your consumers (usually using timestamps) as offsets are not necessarily synchronized across clusters (it depends on what you're using for the replication mechanism). ## Active-passive architecture In some cases, the only requirement for multiple clusters is to support some kind of disaster scenario or to enable faster reads locally by mirroring an entire cluster. Perhaps you have two clusters in the same data center. You use one cluster for all the applications, but you want a second cluster that contains (almost) all the events in the original cluster that you can use if the original cluster is completely unavailable. In this architecture, producers publish data to the active cluster only. The passive cluster receives no writes, it just receives the mirrored data from the active cluster by Mirror Maker. ### Advantages The advantages of this architecture are: - Simplicity in setup and the fact that it can be used in pretty much any use case - No need to worry about access to data, handling conflicts, and other architectural complexities. ### Disadvantages The disadvantages are the waste of a good cluster and the fact that it is currently not possible to perform cluster failover in Kafka without either losing data or having duplicate events. > **Offsets aren't always preserved** > Replicating may not necessarily preserve offsets, just data! Data at an offset in one cluster may not be the same as the data at the same offset in another cluster. As of Mirror Maker 2 in Kafka 2.7, you can use the setting `sync.group.offsets.enabled` ## Architecture comparison | Aspect | Active-Active | Active-Passive | |--------|--------------|----------------| | Complexity | High | Low | | Data availability | Both clusters | Primary only | | Failover | Instant | Manual switchover | | Conflict handling | Required | Not needed | | Resource usage | Higher | Lower (standby idle) | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-multi-cluster-and-mirrormaker) can connect to multiple Kafka clusters simultaneously. Monitor replication lag, compare topic configurations, and manage all your clusters from a single interface. ## Further reading - [MirrorMaker 2 README](https://github.com/apache/kafka/blob/trunk/connect/mirror/README.md) - [Uber's uReplicator](https://eng.uber.com/ureplicator/) - [Confluent Multi-DC Documentation](https://docs.confluent.io/platform/current/multi-dc-deployments/index.html) ## Next steps - [Configure advertised listeners](https://www.conduktor.io/kafka/kafka-advertised-host-setting) so clients can reach brokers across networks - [Set up monitoring](https://www.conduktor.io/kafka/kafka-monitoring-and-operations) for cross-cluster replication - [Configure security](https://www.conduktor.io/kafka/kafka-security) for inter-cluster traffic Source: https://www.conduktor.io/kafka/kafka-multi-cluster-and-mirrormaker --- # Kafka producer acks explained *Learn how producer acknowledgments affect data durability* The producer acknowledgment (acks) setting is one of the most critical configurations for balancing durability, performance, and availability in Kafka. Understanding each option helps you make the right trade-offs for your use case. Acks depend on how replicas and leaders are laid out across brokers, see the [Kafka architecture diagram](https://www.conduktor.io/glossary/kafka-architecture-diagram). **What you'll learn:** - The three acks settings and their guarantees - How acks interacts with replication - Performance implications of each setting - How to choose the right setting for your use case ## Acks overview Kafka producers only write data to the current leader broker for a partition and have to specify an acknowledgment (`acks`) level to determine when a message is considered successfully written. > **The default value of acks has changed with Kafka v3.0** > - If using Kafka = v3.0, `acks=all` ![Comparison of acks=0 fire and forget, acks=1 leader-only acknowledgment, and acks=all full replication acknowledgment flows](https://www.conduktor.io/assets/kafka/diagrams/kafka-producer-acks-deep-dive.svg) ```mermaid flowchart TB subgraph acks0["acks=0 (Fire and forget)"] P0["Producer"] -->|"Send"| B0["Leader"] P0 -->|"Don't wait"| Done0["Continue"] end subgraph acks1["acks=1 (Leader only)"] P1["Producer"] -->|"Send"| B1["Leader"] B1 -->|"ACK"| P1 B1 -.->|"Replicate async"| R1["Replica"] end subgraph acksAll["acks=all (Full replication)"] P2["Producer"] -->|"Send"| B2["Leader"] B2 -->|"Replicate"| R2["Replica"] R2 -->|"Confirm"| B2 B2 -->|"ACK"| P2 end ``` ## acks=0 (fire and forget) When `acks=0`, producers consider messages "written successfully" immediately after sending, without waiting for broker acceptance. ![Kafka Producer Acks Setting set to 0](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-1.png) **Characteristics:** - Highest possible throughput - Lowest latency - No durability guarantees - Potential for data loss if broker fails **Use cases:** - Metrics collection where some data loss is acceptable - High-volume logging where speed is more important than reliability - Scenarios where losing some messages won't impact business logic ## acks=1 (leader acknowledgment) When `acks=1`, producers consider messages "written successfully" when the leader broker acknowledges receipt. ![Kafka Producer Acks Setting set to 1](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-2.png) **Characteristics:** - Balanced throughput and durability - Leader response required before considering message sent - Replication happens asynchronously in background - Risk of data loss if leader fails before replication completes **Use cases:** - Most common configuration for general-purpose applications - Good balance between performance and reliability - Suitable when some data loss is acceptable but should be minimized ## acks=all (full acknowledgment) When `acks=all`, producers consider messages "written successfully" when all in-sync replicas (ISR) accept the message. ![Kafka Producer Acks Setting set to all](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-3.png) **Characteristics:** - Highest durability guarantees - Lower throughput due to replication requirements - Works with `min.insync.replicas` setting - No data loss as long as at least one ISR remains **Use cases:** - Critical business data that cannot be lost - Financial transactions - Audit logs and compliance data - Any scenario where data durability is paramount > **Popular Configuration** > `acks=all` and `min.insync.replicas=2` provides optimal data durability and allows withstanding the loss of one Kafka broker. ## Interaction with min.insync.replicas The `min.insync.replicas` setting works together with `acks=all` to control durability: | min.insync.replicas | Meaning | Broker failures tolerated (RF=3) | |---------------------|---------|----------------------------------| | 1 | Only leader needs to be in-sync | 2 | | 2 | Leader + 1 follower must acknowledge | 1 (recommended) | | 3 | Leader + 2 followers must acknowledge | 0 | ## Performance comparison | Setting | Throughput | Latency | Durability | Data loss risk | |---------|------------|---------|------------|----------------| | `acks=0` | Highest | Lowest | None | High | | `acks=1` | High | Low | Leader only | Medium | | `acks=all` | Lower | Higher | Full ISR | Low | ## Configuration examples ### High durability (recommended for critical data) ```properties acks=all min.insync.replicas=2 retries=Integer.MAX_VALUE enable.idempotence=true ``` ### Balanced performance and reliability ```properties acks=1 retries=Integer.MAX_VALUE ``` ### Maximum throughput (use with caution) ```properties acks=0 retries=0 ``` Each setting has different data loss scenarios: with `acks=0`, data is lost if the producer fails to send or the broker is down; with `acks=1`, data is lost if the leader fails before replication; with `acks=all`, data is only lost if all ISRs fail simultaneously. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-producer-acks-deep-dive) lets you view topic configurations including replication factor and min.insync.replicas. Monitor ISR status to ensure your acks settings provide the intended durability guarantees. ## Next steps - [Configure producer retries](https://www.conduktor.io/kafka/kafka-producer-retries) to recover from transient failures - [Enable idempotent producers](https://www.conduktor.io/kafka/idempotent-kafka-producer) for exactly-once semantics - [Optimize producer batching](https://www.conduktor.io/kafka/kafka-producer-batching) for throughput Source: https://www.conduktor.io/kafka/kafka-producer-acks-deep-dive --- # Kafka producer batching *Learn how to optimize producer throughput with batching* Kafka producer batching groups multiple messages together before sending them to brokers, dramatically improving throughput at the cost of slightly increased latency. Understanding batching is essential for building high-performance producer applications, and for keeping an eye on [tail latency (p99/p999)](https://www.conduktor.io/blog/kafka-latency) as batches grow. **What you'll learn:** - How producer batching works internally - Key configuration parameters and their effects - How to tune for throughput vs latency - Memory management and monitoring strategies ## How batching works Instead of sending messages individually, the Kafka producer accumulates messages in memory and sends them in batches: ![Producer messages accumulate in a batch buffer and are sent to the broker when batch.size is reached or linger.ms elapses](https://www.conduktor.io/assets/kafka/diagrams/kafka-producer-batching--1.svg) ```mermaid flowchart LR subgraph Producer["Producer"] M1["Message 1"] --> Buffer["Batch Buffer"] M2["Message 2"] --> Buffer M3["Message 3"] --> Buffer end Buffer -->|"When batch.size reached
OR linger.ms elapsed"| Broker["Broker"] Broker --> P0["Partition 0"] ``` 1. **Message accumulation**: Producer collects messages in memory buffers per partition 2. **Batch creation**: Messages are grouped into batches based on size or time limits 3. **Network transmission**: Complete batches are sent to brokers in single network requests 4. **Broker processing**: Brokers process entire batches more efficiently than individual messages ![Kafka Producer Batching](https://www.conduktor.io/assets/kafka/Adv-Kafka-Producer-Batching-1.png) > **Batching is automatic** > Kafka producers handle batching automatically - you only need to configure the parameters that control when batches are sent. ## Key batching parameters ### batch.size Controls the maximum size of a batch in bytes. ```properties # Default: 16384 (16KB) batch.size=32768 # 32KB batches ``` **Behavior:** - Larger batches improve throughput but increase memory usage - Batches are sent when they reach this size, regardless of time - Each partition has its own batch buffer - Messages spread across many partitions reduce per-partition batch efficiency — this is why the sticky partitioner was introduced ### linger.ms Controls how long to wait for additional messages before sending a batch. ```properties # Default: 0 (send immediately) linger.ms=10 # Wait up to 10ms for more messages ``` **Behavior:** - Adds artificial delay to allow batches to fill up - Improves throughput by creating larger batches - Increases end-to-end latency slightly ### buffer.memory Total memory available for batching across all partitions. ```properties # Default: 33554432 (32MB) buffer.memory=67108864 # 64MB total buffer ``` **Behavior:** - Shared across all partitions and topics - Producer blocks when buffer is full - Must accommodate batch.size × number of active partitions ## When batches are sent Batches are sent when ANY of these conditions are met: | Condition | Description | |-----------|-------------| | **Batch size reached** | Batch reaches `batch.size` bytes | | **Linger time elapsed** | `linger.ms` milliseconds have passed | | **Buffer full** | Producer needs space for new messages | | **Producer flush** | Explicit `flush()` call or producer close | ## Tune for throughput vs latency ### Decision guide ![Decision tree for tuning batching by priority: throughput, latency, or balanced settings, all leading to monitoring batch utilization and memory usage](https://www.conduktor.io/assets/kafka/diagrams/kafka-producer-batching--2.svg) ```mermaid flowchart TD Start["Tuning batching"] --> Q1{"Priority?"} Q1 -->|"Throughput"| HT["High throughput
batch.size=64KB
linger.ms=20ms"] Q1 -->|"Latency"| LL["Low latency
batch.size=16KB
linger.ms=0ms"] Q1 -->|"Balanced"| BL["Balanced
batch.size=32KB
linger.ms=5ms"] HT --> Monitor["Monitor batch utilization
and memory usage"] LL --> Monitor BL --> Monitor ``` ### High-throughput configuration ```properties # Optimize for maximum throughput batch.size=65536 # 64KB batches linger.ms=20 # Wait for full batches buffer.memory=134217728 # 128MB buffer compression.type=lz4 # Fast compression acks=1 # Balance durability/speed ``` ### Low-latency configuration ```properties # Optimize for minimum latency batch.size=16384 # Moderate batch size linger.ms=0 # Send immediately buffer.memory=33554432 # Default buffer compression.type=none # No compression delay acks=1 # Fast acknowledgment ``` ### Balanced configuration (recommended) ```properties # Balance throughput and latency batch.size=32768 # 32KB batches linger.ms=5 # Short wait time buffer.memory=67108864 # 64MB buffer compression.type=snappy # Fast compression acks=1 # Reasonable durability ``` ## Memory management ### Memory allocation formula ``` Available batches = buffer.memory / batch.size ``` For example, with defaults: - `buffer.memory=32MB`, `batch.size=16KB` - Available batches = 32MB / 16KB = 2,048 batches ### Memory pressure handling When buffer memory is exhausted: 1. `send()` calls block until memory available 2. After `max.block.ms`, throw `TimeoutException` 3. Existing batches complete and free memory > **Memory blocking** > When buffer memory is exhausted, the producer will block `send()` calls. Monitor `buffer-available-bytes` to prevent blocking in production systems. ## Monitor batching performance ### Key metrics | Metric | Description | Target | |--------|-------------|--------| | `batch-size-avg` | Average batch size | Close to `batch.size` | | `records-per-request-avg` | Messages per batch | Higher is better | | `request-latency-avg` | Batch send latency | Stable | | `buffer-available-bytes` | Free buffer memory | > 0 | ### JMX monitoring path ``` kafka.producer:type=producer-metrics,client-id= ``` ## Common issues and solutions | Issue | Cause | Solution | |-------|-------|----------| | Low batch utilization | `linger.ms` too low | Increase `linger.ms` | | High memory usage | Too many partitions | Reduce `batch.size` or increase `buffer.memory` | | Increased latency | `linger.ms` too high | Decrease `linger.ms` | | Producer blocking | Buffer exhausted | Increase `buffer.memory` or reduce send rate | The `batch.size` parameter is a target, not a strict limit. Batches can exceed this size if single messages are larger, and will be smaller if `linger.ms` timeout occurs first. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-producer-batching) lets you produce test messages and observe batching behavior. Monitor producer metrics to validate your batching configuration works as expected. ## Next steps - [Choose a partitioner](https://www.conduktor.io/kafka/producer-default-partitioner-and-sticky-partitioner) to control how batches map to partitions - [Configure compression](https://www.conduktor.io/kafka/kafka-message-compression) to reduce batch sizes - [Understand producer acks](https://www.conduktor.io/kafka/kafka-producer-acks-deep-dive) for durability Source: https://www.conduktor.io/kafka/kafka-producer-batching --- # Kafka producer CLI tutorial *Learn how to produce messages to Kafka using kafka-console-producer* The `kafka-console-producer` CLI reads data from standard input and publishes it to Kafka topics. It's essential for testing, debugging, and quick data injection. **What you'll learn:** - How to send messages to a Kafka topic - How to produce messages with keys - How to produce messages from a file - Important producer options and settings > Use CLI commands with appropriate extensions for your platform: for Windows - `kafka-console-producer.bat`, for Mac and Linux - `kafka-console-producer.sh`. ## How to produce a message into a Kafka topic using the CLI? To produce to a Kafka topic, we need to provide the mandatory parameters: * Find your Kafka hostname and port e.g., `localhost:9092` * If Kafka v2.5+, use the `--bootstrap-server` option * If older version of Kafka, use the `--broker-list` option * Provide the mandatory parameters: topic name * Use the `kafka-console-producer.sh` CLI as outlined below ### Example Make sure the topic first_topic is already created with 3 partitions and a replication factor of 1: ``` kafka-topics --bootstrap-server localhost:9092 --topic first_topic --create --partitions 3 --replication-factor 1 ``` Start producing to the topic: **Kafka v2.5+**: ``` kafka-console-producer --bootstrap-server localhost:9092 --topic first_topic ``` **Kafka v2.4 or less:** ``` kafka-console-producer --broker-list localhost:9092 --topic first_topic ``` To exit the Kafka console producer, use the keyboard combination `Ctrl+C`. ### Command output ``` > ``` After the producer is opened, you should see a `>` sign. Then any line of text you write afterwards will be sent to the Kafka topic (when pressing `Enter`) ``` $ kafka-console-producer --bootstrap-server localhost:9092 --topic first_topic >Hello World >My name is Conduktor >I love Kafka >^C (<- Ctrl + C is used to exit the producer) ``` ### Gotchas Here are the common mistakes and caveats with the `kafka-console-producer.sh` command: * Messages are sent with the `null` key by default (see below for more options) * **If the topic does not exist, it can be auto-created by Kafka:** A topic with the name provided should exist. If you specify a topic that does not exist yet, a new topic with the name provided will be created with the **default number of partitions and replication factor.** These are controlled by the broker-side settings (in your `config/server.properties` file), with the following defaults: ``` auto.create.topics.enable=true num.partitions=1 default.replication.factor=1 ``` ### Extra important options you can set (advanced) `--compression-codec` To enable message compression, default `gzip`, possible values `'none'`, `'gzip', 'snappy', 'lz4', or 'zstd'` `--producer-property` To pass in any producer property, such as the `acks=all` setting `--request-required-acks` An alternative to set the `acks` setting directly ## How to produce messages from a file with the Kafka console producer CLI? Example file `topic-input.txt` (make sure each message is on a new line) ``` Hello World My name is John ``` Produce messages to the topic from the file (see the end of the command) ``` kafka-console-producer --bootstrap-server localhost:9092 --topic first_topic < topic-input.txt ``` ## How to produce messages with key in the Kafka console producer CLI? By default messages sent to a Kafka topic will result in messages with `null` keys. We have to use the properties `parse.key` and `key.separator` to send the key alongside messages. In this example, the separator between the key and the value is: `:` ``` kafka-console-producer --bootstrap-server localhost:9092 --topic first_topic --property parse.key=true --property key.separator=: ``` Example input: ``` >example key:example value >name:John ``` Do not forget to always include your key/value separator otherwise you will get an exception. ## Quick reference | Operation | Command | |-----------|---------| | Basic produce | `kafka-console-producer --bootstrap-server localhost:9092 --topic NAME` | | Produce with keys | `... --property parse.key=true --property key.separator=:` | | With compression | `... --compression-codec snappy` | | With acks=all | `... --producer-property acks=all` | | From file | `... < filename.txt` | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-producer-cli-tutorial) provides a visual interface for producing messages with full control over headers, keys, and serialization formats. ## Next steps - [Consume messages](https://www.conduktor.io/kafka/kafka-consumer-cli-tutorial) you've just produced from the CLI - [Understand producers](https://www.conduktor.io/kafka/kafka-producers) including message keys and partitioning Source: https://www.conduktor.io/kafka/kafka-producer-cli-tutorial --- # Kafka producer retries *Learn how to configure producer retries for reliable message delivery* Kafka producers can automatically retry failed requests to improve reliability and handle transient failures in distributed systems. **What you'll learn:** - How producer retries work and when they're triggered - The difference between retriable and non-retriable errors - How to configure retry behavior for idempotent producers - Best practices for retry timeout and backoff settings ## Why retries matter In distributed systems, temporary failures are common: - Network connectivity issues - Broker leadership changes - Temporary resource constraints - Replication delays Without retries, these transient issues would result in lost messages. Retries provide resilience against such failures. ## Retry configuration ### Basic retry settings ```properties # Number of retry attempts (default varies by Kafka version) retries=2147483647 # Time to wait between retries (default: 100ms) retry.backoff.ms=100 # Maximum time to wait for acknowledgment (default: 30s) request.timeout.ms=30000 # Maximum time to deliver a message including retries (default: 2 minutes) delivery.timeout.ms=120000 ``` ### Kafka version differences **Kafka < 3.0:** - `retries=0` (no retries by default) - Must explicitly enable retries **Kafka >= 3.0:** - `retries=Integer.MAX_VALUE` (unlimited retries) - Retries enabled by default with idempotent producers ## Types of errors ### Retriable errors These errors can potentially be resolved by retrying: - **TimeoutException**: Request timed out - **NotEnoughReplicasException**: Not enough in-sync replicas - **NotEnoughReplicasAfterAppendException**: Replication issues - **RetriableException**: Generic retriable error - **LeaderNotAvailableException**: Leader election in progress - **NetworkException**: Network connectivity issues ### Non-retriable errors These errors indicate permanent failures that won't be resolved by retrying: - **RecordTooLargeException**: Message exceeds size limits - **SerializationException**: Message serialization failed - **OffsetMetadataTooLarge**: Offset metadata too large - **InvalidTopicException**: Topic doesn't exist or is invalid - **UnknownTopicOrPartitionException**: Topic or partition invalid - **AuthorizationException**: Authentication/authorization failure ### Error handling decision tree This decision tree helps you understand how to handle different types of producer errors: ![](https://www.conduktor.io/assets/kafka/error-retries.png) ### Retry decision flowchart This flowchart shows how the producer decides whether to retry a failed request: ![](https://www.conduktor.io/assets/kafka/retries-decision.png) > **Idempotent producers (Kafka 2.4+):** With `enable.idempotence=true` and `acks=all`, you get unlimited retries by default without risk of duplicates, making retry configuration much simpler. ## Retry backoff strategies ### Fixed backoff (default) Waits a fixed amount of time between retries: ```properties retry.backoff.ms=100 # Always wait 100ms between retries ``` **Pattern:** Wait → Retry → Wait → Retry → Wait → Retry ### Exponential backoff Not natively supported by Kafka producer, but can be implemented at the application level: ``` Attempt 1: Wait 100ms Attempt 2: Wait 200ms Attempt 3: Wait 400ms Attempt 4: Wait 800ms ``` ## Impact on message ordering ### With retries enabled Retries can affect message ordering within a partition: ``` Message A sent → Fails → Retry scheduled Message B sent → Succeeds immediately Message A retry → Succeeds Result: Message B appears before Message A in partition ``` ### Preserve order To maintain strict ordering, configure: ```properties # Limit in-flight requests to preserve order max.in.flight.requests.per.connection=1 # Or use idempotent producer (recommended) enable.idempotence=true max.in.flight.requests.per.connection=5 # Up to 5 with idempotency ``` ## Delivery timeout vs request timeout ### Request timeout Time to wait for a single request attempt: ```properties request.timeout.ms=30000 # 30 seconds per attempt ``` ### Delivery timeout Total time limit for delivering a message (including all retries): ```properties delivery.timeout.ms=120000 # 2 minutes total ``` **Relationship:** ``` delivery.timeout.ms >= request.timeout.ms + (retries × retry.backoff.ms) ``` ## Configuration examples ### High reliability (recommended) ```properties # Unlimited retries with delivery timeout retries=2147483647 delivery.timeout.ms=300000 # 5 minutes total request.timeout.ms=30000 # 30 seconds per attempt retry.backoff.ms=100 # 100ms between retries enable.idempotence=true # Preserve ordering and avoid duplicates ``` ### Fast failure ```properties # Limited retries for quick feedback retries=3 delivery.timeout.ms=10000 # 10 seconds total request.timeout.ms=5000 # 5 seconds per attempt retry.backoff.ms=100 # 100ms between retries ``` ### No retries (not recommended for production) ```properties retries=0 request.timeout.ms=30000 ``` ## Monitor retry behavior ### Key metrics to track - **retry-rate**: Rate of retry attempts - **retry-total**: Total number of retries - **error-rate**: Rate of failed requests (after all retries) - **request-latency**: Time taken for requests (including retries) ### JMX metrics ``` kafka.producer:type=producer-metrics,client-id= - retry-rate - retry-total - request-rate - request-latency-avg ``` ## Error handling strategies ### Synchronous error handling ```java Properties props = new Properties(); props.put("retries", 5); props.put("retry.backoff.ms", 100); Producer producer = new KafkaProducer<>(props); try { ProducerRecord record = new ProducerRecord<>("topic", "key", "value"); RecordMetadata metadata = producer.send(record).get(); System.out.println("Message sent to " + metadata.topic() + ":" + metadata.partition()); } catch (Exception e) { System.err.println("Failed after all retries: " + e.getMessage()); } ``` ### Asynchronous error handling ```java ProducerRecord record = new ProducerRecord<>("topic", "key", "value"); producer.send(record, (metadata, exception) -> { if (exception != null) { System.err.println("Failed after all retries: " + exception.getMessage()); } else { System.out.println("Message sent successfully"); } }); ``` ## Best practices ### Production recommendations 1. **Enable unlimited retries**: Set `retries=Integer.MAX_VALUE` 2. **Use delivery timeout**: Set `delivery.timeout.ms` to control total time 3. **Enable idempotency**: Prevents duplicates during retries 4. **Monitor retry metrics**: Track retry rates and error patterns 5. **Handle non-retriable errors**: Implement proper error handling for permanent failures ### Configuration checklist - ✅ `retries=Integer.MAX_VALUE` (unlimited retries) - ✅ `delivery.timeout.ms=120000` (reasonable total timeout) - ✅ `request.timeout.ms=30000` (reasonable per-request timeout) - ✅ `retry.backoff.ms=100` (reasonable delay between retries) - ✅ `enable.idempotence=true` (prevent duplicates) ### Common mistakes to avoid - Setting `retries=0` in production - Not handling non-retriable errors - Setting delivery timeout too low - Ignoring retry metrics and error rates Starting with Kafka 3.0, producers have sensible retry defaults: unlimited retries with idempotency enabled, a 2-minute delivery timeout, and proper error handling for most use cases. Note that retries can affect message ordering within partitions—use `enable.idempotence=true` or `max.in.flight.requests.per.connection=1` if strict ordering is required. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-producer-retries) displays producer retry metrics and error rates in real-time. Monitor retry attempts, successful retries, and failed messages to validate your retry configuration and identify patterns in transient versus permanent failures. ## Next steps - [Enable idempotent producers](https://www.conduktor.io/kafka/idempotent-kafka-producer) to avoid duplicates on retry - [Understand acknowledgment settings](https://www.conduktor.io/kafka/kafka-producer-acks-deep-dive) for delivery guarantees - [Optimize producer batching](https://www.conduktor.io/kafka/kafka-producer-batching) for throughput Source: https://www.conduktor.io/kafka/kafka-producer-retries --- # Advanced Kafka producers Master producer performance, reliability, and advanced configurations to optimize your Kafka producers. ## What you'll learn - Producer acknowledgment settings explained - Retry mechanisms and error handling - Idempotent producers for exactly-once semantics - Message compression strategies - Producer batching for performance - Partitioning strategies and custom partitioners - Advanced producer configuration options ## Next steps - [Understand producer acks](https://www.conduktor.io/kafka/kafka-producer-acks-deep-dive) to balance durability and latency - [Configure retries](https://www.conduktor.io/kafka/kafka-producer-retries) for resilient message delivery - [Enable idempotent producers](https://www.conduktor.io/kafka/idempotent-kafka-producer) for exactly-once semantics Source: https://www.conduktor.io/kafka/kafka-producers-advanced --- # Kafka producers *Learn how Kafka producers send data to topics* Producers are applications that write data to Kafka topics. Understanding how producers work, including message keys, serialization, and partitioning, is essential for building reliable data pipelines. **What you'll learn:** - How producers send messages to Kafka topics - How message keys affect partitioning and ordering - The structure of a Kafka message - How serialization converts data to bytes ## Kafka producers Once a topic has been created with Kafka, the next step is to send data into the topic. This is where Kafka producers come in. Applications that send data into topics are known as Kafka producers. Applications typically integrate a Kafka client library to write to Apache Kafka. Excellent client libraries exist for almost [all programming languages](https://www.conduktor.io/kafka/kafka-sdk-list) that are popular today including Python, Java, Go, and others. ![Apache Kafka Producers send data into Kafka. These messages are then routed by the broker to the relevant topics and partitions.](https://www.conduktor.io/assets/kafka/Kafka-Producers-1.png) A Kafka producer sends messages to a topic, and messages are distributed to partitions according to a mechanism such as key hashing (more on it below). For a message to be successfully written into a Kafka topic, a producer has to specify a level of acknowledgment (acks). This subject will be introduced in depth in the [topic replication section](https://www.conduktor.io/kafka/kafka-topic-replication). ## Message keys Each event message contains an optional key and a value. In case the key (`key=null`) is not specified by the producer, messages are distributed evenly across partitions in a topic. This means messages are sent in a round-robin fashion (partition *p0* then *p1* then *p2*, etc... then back to *p0* and so on...). **If a key is sent** (`key != null`)**, then all messages that share the same key will always be sent and stored in the same Kafka partition**. A key can be anything to identify a message - a string, numeric value, binary value, etc. Kafka message keys are commonly used when there is a need for message ordering for all messages sharing the same field. For example, in the scenario of tracking trucks in a fleet, we want data from trucks to be in order at the individual truck level. In that case, we can choose the key to be `truck_id`. In the example shown below, the data from the truck with id *truck\_id\_123* will always go to partition *p0.* ![Apache Kafka Producer sending trucking fleet data into 2 Kafka Brokers. ](https://www.conduktor.io/assets/kafka/Kafka-Producers-2.png) ### When to use message keys | Use case | Key recommendation | |----------|-------------------| | Order processing | Order ID (keep all events for an order together) | | User activity tracking | User ID (maintain user event sequence) | | IoT sensor data | Device ID (preserve per-device ordering) | | Log aggregation | No key needed (maximize throughput) | | Metrics collection | No key needed (even distribution) | > **Key selection rule of thumb** > Use a message key when you need ordering guarantees for related messages. Skip the key when maximum throughput is more important than ordering. ## Kafka message anatomy Kafka messages are created by the producer. A Kafka message consists of the following elements: ![Diagram showing how Kafka Producers structure a message created by the Apache Kafka Producer.](https://www.conduktor.io/assets/kafka/Kafka-Producers-3.png) Structure of a Kafka message: - **Key**. Key is optional in the Kafka message and it can be null. A key may be a string, number, or any object and then the key is serialized into binary format. - **Value**. The value represents the content of the message and can also be null. The value format is arbitrary and is then also serialized into binary format. - **Compression Type**. Kafka messages may be compressed. The compression type can be specified as part of the message. Options are `none`, `gzip`, `lz4`, `snappy`, and `zstd` - **Headers**. There can be a list of optional Kafka message headers in the form of key-value pairs. It is common to add headers to specify metadata about the message, especially for tracing. - **Partition + Offset**. Once a message is sent into a Kafka topic, it receives a partition number and an offset id. The combination of topic+partition+offset uniquely identifies the message - **Timestamp**. A timestamp is added either by the user or the system in the message. ## Kafka message serializers In many programming languages, the key and value are represented as objects, which greatly increases the code readability. However, Kafka brokers expect byte arrays as keys and values of messages. The process of transforming the producer's programmatic representation of the object to binary is **called message serialization**. As shown below, we have a message with an `Integer` key and a `String` value. Since the key is an integer, we have to use an `IntegerSerializer` to convert it into a byte array. For the value, since it is a string, we have to use a `StringSerializer`. ![Message serialization diagram showing how Apache Kafka Producers integer and string serializers.](https://www.conduktor.io/assets/kafka/Kafka-Producers-4.png) Message serialization As part of the Java Client SDK for Apache Kafka, [several serializers already exist](https://github.com/a0x8o/kafka/tree/master/clients/src/main/java/org/apache/kafka/common/serialization), such as string (which supersedes JSON), integer, float. Other serializers may have to be written by the users, but commonly distributed Kafka serializers exist and are efficiently written for formats such as [JSON-Schema](https://github.com/confluentinc/schema-registry/blob/master/json-schema-serializer/src/main/java/io/confluent/kafka/serializers/json/KafkaJsonSchemaSerializer.java), [Apache Avro](https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroSerializer.java) and [Protobuf](https://github.com/confluentinc/schema-registry/blob/master/protobuf-serializer/src/main/java/io/confluent/kafka/serializers/protobuf/KafkaProtobufSerializer.java), thanks to the Confluent schema registry. > If you are not using a JVM-based programming language for serialization and deserialization, ensure that your Kafka client library supports the data formats that you need! ### Common serialization formats | Format | Best for | Schema support | |--------|----------|---------------| | String/JSON | Flexibility, debugging | No built-in | | Avro | Schema evolution, compact | Schema Registry | | Protobuf | Performance, cross-language | Schema Registry | ## Kafka message key hashing A Kafka partitioner is a code logic that takes a record and determines to which partition to send it into. ![Kafka Producers use default partitioning logic to assign Kafka messages to the appropriate Apache Kafka Partition.](https://www.conduktor.io/assets/kafka/Kafka-Producers-5.png) Default partitioner In that effect, it is common for partitioners to use the Kafka message keys to route a message into a specific topic-partition. As a reminder, all messages with the same key will go to the same partition. > **Kafka key hashing** is the process of determining the mapping of a key to a partition. In the default Kafka partitioner, the keys are hashed using the **murmur2 algorithm,** with the formula below for the curious: ``` targetPartition = Math.abs(Utils.murmur2(keyBytes)) % (numPartitions - 1) ``` You can override the default partitioner via the producer property `partitioner.class`, although it is not advisable unless you know what you are doing. If you increase the number of partitions for a topic, the same key may hash to a different partition. This breaks ordering guarantees for existing keys. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-producers) lets you produce messages to topics directly from the UI. Test different keys, values, and headers to see how messages are distributed across partitions. ## Next steps - [Learn how consumers read data](https://www.conduktor.io/kafka/kafka-consumers) to understand the other side of the pipeline - [Explore producer batching](https://www.conduktor.io/kafka/kafka-producer-batching) to optimize throughput - [Configure producer acknowledgments](https://www.conduktor.io/kafka/kafka-producer-acks-deep-dive) for reliability Source: https://www.conduktor.io/kafka/kafka-producers --- # Kafka Java programming *Build Kafka applications with your preferred programming language* After learning the CLI, it's time to build real applications. This section covers how to write Kafka producers and consumers programmatically, with a focus on Java (the official SDK). **What you'll learn:** - How to choose a Kafka client library for your language - How to set up a Java project with Kafka dependencies - How to write producers with keys, callbacks, and batching - How to write consumers with consumer groups and graceful shutdown ## Available tutorials | Topic | Description | |-------|-------------| | [SDK list](https://www.conduktor.io/kafka/kafka-sdk-list) | Recommended client libraries by language | | [Java programming](https://www.conduktor.io/kafka/java-kafka-programming) | Complete Java producer and consumer tutorials | | [Advanced consumers](https://www.conduktor.io/kafka/advanced-kafka-consumer-with-java) | Rebalance listeners, seek/assign, threading | | [Kafka Streams](https://www.conduktor.io/kafka-streams) | Stateful stream processing: joins, aggregations, windowing, and production operations | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-programming-tutorials) helps you test your applications by producing and consuming messages, monitoring consumer groups, and inspecting message content during development. ## Next steps - [Choose a client library](https://www.conduktor.io/kafka/kafka-sdk-list) for your programming language - [Get the Java overview](https://www.conduktor.io/kafka/java-kafka-programming) to start the official SDK path - [Build a complete producer](https://www.conduktor.io/kafka/complete-kafka-producer-with-java) with keys, callbacks, and batching Source: https://www.conduktor.io/kafka/kafka-programming-tutorials --- # Kafka SDK list *Find the right Kafka client library for your programming language* Apache Kafka only officially provides a Java SDK. For other languages, community implementations are available with varying levels of maturity and feature support. **What you'll learn:** - How to evaluate Kafka client libraries - Recommended libraries by programming language - Key considerations: librdkafka, Schema Registry support, security > **Feedback?** > Something looks wrong? Send us an email at feedback [at] conduktor [dot] io --- ## How to choose a Kafka client library ![Decision flowchart for evaluating a Kafka client library: first decide pure (easier deployment, no native dependencies) versus librdkafka-based (high performance, build complexity); then check Schema Registry support (yes means verify Avro/JSON/Protobuf, no means custom serializers may be needed); then review security mechanisms; finally test performance](https://www.conduktor.io/assets/kafka/diagrams/kafka-sdk-list.svg) ```mermaid flowchart TD Start["Evaluate library"] --> Q1{{"Pure or librdkafka
based?"}} Q1 -->|"Pure"| Pure["Easier deployment
No native dependencies"] Q1 -->|"librdkafka"| Lib["High performance
Build complexity"] Pure --> Q2{{"Schema Registry
support?"}} Lib --> Q2 Q2 -->|"Yes"| Schema["Check Avro/JSON/Protobuf"] Q2 -->|"No"| NoSchema["May need custom serializers"] Schema --> Q3{{"Security mechanisms?"}} NoSchema --> Q3 Q3 --> Final["Test performance"] ``` | Consideration | What to check | |---------------|---------------| | Implementation type | Pure vs [librdkafka](https://github.com/edenhill/librdkafka) wrapper (affects deployment) | | Security | SSL, SASL (PLAIN, SCRAM, GSSAPI, OAUTHBEARER, MSK IAM) | | Schema Registry | Confluent Schema Registry support (Avro, JSON, Protobuf) | | Performance | Benchmark with realistic workloads | --- ## Kafka client libraries SDK list Here you can find a list of libraries that has been compiled to ensure you have some pointers on when to start with your implementation: * Java * [The official client library](https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients): low-level client * [The official Kafka Streams client library](https://mvnrepository.com/artifact/org.apache.kafka/kafka-streams): to create your Kafka Streams application * [Kafka for Spring Boot](https://spring.io/projects/spring-kafka): applies Spring concepts to Kafka development * [Spring Cloud Stream](https://spring.io/projects/spring-cloud-stream): bindings for Kafka Stream * [Akka Streams](https://doc.akka.io/docs/akka/current/stream/index.html) & [Alpakka Kafka](https://doc.akka.io/docs/alpakka-kafka/current/home.html) * Scala * [FS2 Kafka](https://fd4s.github.io/fs2-kafka/): Functional Kafka Producer and Consumer * [ZIO Kafka](https://github.com/zio/zio-kafka): Kafka Client for ZIO * [Kafka Streams Scala](https://mvnrepository.com/artifact/org.apache.kafka/kafka-streams-scala): the official kafka-streams library has Scala [since Kafka 2.0](https://issues.apache.org/jira/browse/KAFKA-6670) * [Alpakka Kafka](https://doc.akka.io/docs/alpakka-kafka/current/home.html) * C/C++ * [librdkafka](https://github.com/edenhill/librdkafka): low-level implementation of the Kafka client, many other higher level language libraries are a wrapper around this library. Supports all kind of security, and [most KIPs](https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#supported-kips) * [CPP Kafka](https://github.com/mfontanini/cppkafka): based on librdkafka * [Modern CPP Kafka](https://github.com/morganstanley/modern-cpp-kafka): based on librdkafka * Golang * [Confluent Kafka Go](https://github.com/confluentinc/confluent-kafka-go): wrapper of librdkafka, currently [no Schema Registry support](https://github.com/confluentinc/confluent-kafka-go/issues/69) * [Schema Registry Client for Go](https://github.com/riferrei/srclient): compatible with Confluent Kafka Go * [Segment's Kafka Go](https://github.com/segmentio/kafka-go): pure Go implementation of the Kafka Client, good support * [Franz Go](https://github.com/twmb/franz-go): Pure go implementation, supports most KIPs * others if you're curious: [Sarama](https://github.com/Shopify/sarama), [Goka](https://github.com/lovoo/goka) * Python * [Confluent Kafka Python](https://github.com/confluentinc/confluent-kafka-python): based on librdkafka, includes an Admin client, Avro support with the Confluent Schema Registry * [Kafka Python](https://github.com/dpkp/kafka-python): not active, use at your own risk * Javascript / Node.js * [KafkaJS](https://kafka.js.org/): no external dependencies, good performance, support for [Schema Registry](https://github.com/kafkajs/confluent-schema-registry) * [Blizzard Node rdkafka](https://github.com/Blizzard/node-rdkafka): Node.js wrapper for librdkafka, low maintenance activity * .NET / C# * [Confluent Kafka DotNet](https://github.com/confluentinc/confluent-kafka-dotnet): wrapper around librdkafka, full Schema Registry support (Avro, JSON, Protobuf) * Rust * [Rust rdkafka](https://github.com/fede1024/rust-rdkafka): Rust wrapper for librdkafka, good performance * [Rust Schema Registry Converter](https://github.com/gklijs/schema_registry_converter): compatible with Rust rdkafka * [Kafka Rust](https://github.com/kafka-rust/kafka-rust): pure Rust implementation, low maintenance activity * REST API * [Confluent REST Proxy](https://github.com/confluentinc/kafka-rest): REST API for Kafka. Deploy with a reverse proxy sidecar to implement the security mechanism you need. Historically needed, but now that client library support for Kafka is quite good, you migh not have a use case for it. * Kotlin * simply use the standard Java library * Haskell * [HW Kafka Client](https://github.com/haskell-works/hw-kafka-client): based on librdkafka * [HW Kafka Avro](https://github.com/haskell-works/hw-kafka-avro): support for Schema Registry * Ruby * [rdkafka-ruby](https://github.com/appsignal/rdkafka-ruby): based on librdkafka * [Ruby Kafka](https://github.com/zendesk/ruby-kafka): limited support for new API > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-sdk-list) works with applications built in any language. Use it to inspect messages, monitor consumer groups, and debug your Kafka applications regardless of which SDK you choose. ## Next steps - [Get the Java overview](https://www.conduktor.io/kafka/java-kafka-programming) to start with the official SDK - [Build a producer](https://www.conduktor.io/kafka/complete-kafka-producer-with-java) in Java - [Build a consumer](https://www.conduktor.io/kafka/complete-kafka-consumer-with-java) in Java Source: https://www.conduktor.io/kafka/kafka-sdk-list --- # Kafka security *Learn how to secure Kafka clusters with encryption, authentication, and authorization* By default, Kafka has no security - anyone can read from or write to any topic. For production deployments with sensitive data or shared clusters, you need encryption, authentication, and authorization. **What you'll learn:** - The three pillars of Kafka security (encryption, authentication, authorization) - How to implement SSL/TLS encryption - Authentication options (SSL, SASL/PLAIN, SASL/SCRAM, Kerberos) - How to configure Access Control Lists (ACLs) ## Kafka security overview ![](https://www.conduktor.io/assets/kafka/security-overview.png) | Component | Purpose | Options | |-----------|---------|---------| | Encryption | Protect data in transit | SSL/TLS | | Authentication | Verify identity | SSL certs, SASL (PLAIN, SCRAM, Kerberos, OAuth) | | Authorization | Control access | ACLs | ## Problems Kafka security is solving Kafka Security has three components: - **Encryption of data in-flight using SSL/TLS:** this allows your data to be encrypted between your producers and Kafka and your consumers and Kafka. This is a very common pattern everyone has used when going on the web. That's the "S" of HTTPS (that beautiful green lock you see everywhere on the web). - **Authentication using SSL or SASL:** this lets your producers and consumers authenticate to your Kafka cluster which verifies their identity. It's also a secure way to enable your clients to endorse an identity. - **Authorization using ACLs:** once your clients are authenticated, your Kafka brokers can run them against access control lists (ACLs) to determine whether or not a particular client would be authorized to write or read to some topic (or any Kafka resource in general). ## Encryption (SSL) Encryption solves the problem of the man in the middle (MITM) attack. That's because your packets, while being routed to your Kafka cluster, travel your network and hop from machines to machines. If your data is PLAINTEXT (by default in Kafka), any of these routers could read the content of the data you're sending: With encryption enabled and carefully setup SSL certificates, your data is now encrypted and securely transmitted over the network. With SSL, only the first and the final machine have the ability to decrypt the packet being sent. This encryption comes at a cost: CPU is now leveraged for both the Kafka Clients and the Kafka Brokers in order to encrypt and decrypt packets. SSL Security comes at the cost of performance, but it's low to negligible. If you're using Java JDK 11, you will get a substantial performance improvement on SSL > The encryption is only in-flight and the data still sits un-encrypted on your broker's disk. ## Authentication (SSL & SASL) There are two ways to authenticate your Kafka clients to your brokers: SSL and SASL. Let's go over both. ### SSL authentication SSL Auth is basically leveraging a capability from SSL called two ways authentication. The idea is to also issue certificates to your clients, signed by a certificate authority, which will allow your Kafka brokers to verify the identity of the clients. ### SASL authentication SASL stands for Simple Authorization Service Layer. The idea is that the authentication mechanism is separated from the Kafka protocol (which is a nice idea). It's very popular with Big Data systems and Big Data setups for example Hadoop would probably already leverage Kerberos. SASL takes many shapes and forms and the following are supported by Kafka - **SASL PLAINTEXT:** This is a classic username/password combination. These usernames and passwords have to be stored on the Kafka brokers in advance and each change needs to trigger a rolling restart. It's very annoying and not the recommended kind of security. If you use SASL/PLAINTEXT, make sure to also enable SSL encryption so that credentials aren't sent as PLAINTEXT on the network - **SASL SCRAM:** This is a username/password combination alongside a challenge (salt), which makes it more secure. On top of this, username and password hashes are stored in Zookeeper (and will be stored in a Kafka topic when Zookeeper is removed), which allows you to scale security without rebooting brokers. If you use SASL/SCRAM, make sure to enable SSL encryption so that credentials aren't sent as PLAINTEXT on the network - **SASL GSSAPI (Kerberos):** This is based on Kerberos ticket mechanism, a very secure way of providing authentication. Microsoft Active Directory is the most common implementation of Kerberos. SASL/GSSAPI is a great choice for big enterprises as it allows the companies to manage security from within their Kerberos Server. Additionally, communications are encrypted to SSL encryption is optional with SASL/GSSAPI. Needless to say, setting up Kafka with Kerberos is the most difficult option, but worth it in the end. - **SASL OAUTHBEARER**: This will allow you to leverage OAUTH2 token for authentication, read more in the [Kafka documentation](https://kafka.apache.org/documentation/#security_sasl_oauthbearer). Currently it is said [not to be ready for production](https://kafka.apache.org/documentation/#security_sasl_oauthbearer_security) due to the usage of Unsecured JSON Web Tokens. ## Authorization (ACLs) Once your Kafka clients are authenticated, Kafka needs to be able to decide what they can and cannot do. This is where Authorization comes in, controlled by Access Control Lists (ACLs). **ACLs are what you expect them to be: User A can('t) do Operation B on Resource C from Host D**. Rules now support prefixes thanks to the new [`AclAuthorizer`](https://kafka.apache.org/documentation/#security_authz). **ACL are great because they can help you prevent disasters.** For example, you may have a topic that needs to be writeable from only a subset of clients or hosts. You want to prevent your average user from writing anything to these topics, hence preventing any data corruption or deserialization errors. ACLs are also great if you have some sensitive data and you need to prove to regulators that only certain applications or users can access that data. To add ACLs, you can use the `kafka-acls` command ([documentation here](https://kafka.apache.org/documentation/#security_authz_cli)). It also even has some facilities and shortcuts to add producers or consumers. ```bash kafka-acls --topic test --producer --bootstrap-server localhost:9092 --add --allow-principal User:alice ``` The result being: ``` Adding ACLs for resource `Topic:test`: User:alice has Allow permission for operations: Describe from hosts: * User:alice has Allow permission for operations: Write from hosts: *Adding ACLs for resource `Cluster:kafka-cluster`: User:alice has Allow permission for operations: Create from hosts: * ``` > **ACLs & Zookeeper** > Do not use the option `--authorizer-properties zookeeper.connect=localhost:2181` with the `kafka-acls` command because Zookeeper is going away and now brokers directly support handling these admin API commands. Please note that for now, your ACLs are stored in Zookeeper. Therefore, it is important to secure Zookeeper and make sure only your Kafka brokers are allowed to write to Zookeeper (`zookeeper.set.acl=true`). Otherwise any user could come in and edit ACLs, hence defeating the point of security. ## Authentication options comparison | Method | Complexity | Best for | |--------|------------|----------| | SSL (mTLS) | Medium | Small deployments, certificate infrastructure exists | | SASL/PLAIN | Low | Development, testing | | SASL/SCRAM | Medium | Production without Kerberos | | SASL/GSSAPI (Kerberos) | High | Enterprise with Active Directory | | SASL/OAUTHBEARER | Medium | OAuth2 environments | When using SASL/PLAIN or SASL/SCRAM, always enable SSL encryption. Otherwise, credentials are sent in plain text over the network. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-security) supports all Kafka authentication methods and provides a visual interface for managing ACLs. Test your security configuration and audit access permissions across your cluster. ## Next steps - [Set up monitoring](https://www.conduktor.io/kafka/kafka-monitoring-and-operations) to track authentication failures and cluster health - [Plan your cluster setup](https://www.conduktor.io/kafka/kafka-cluster-setup-overview) to architect a secure production deployment - [Plan upgrades](https://www.conduktor.io/kafka/kafka-broker-and-client-upgrades) to enable security on existing clusters Source: https://www.conduktor.io/kafka/kafka-security --- # Kafka topic configuration: log compaction *Learn how log compaction retains only the latest value per key* Log compaction is Kafka's alternative cleanup policy that keeps only the most recent value for each key, making it ideal for changelog-style topics. This guide covers both the theory and hands-on practice of log compaction. **What you'll learn:** - The difference between delete and compact cleanup policies - How log compaction works internally - Log compaction guarantees and common misconceptions - How to configure and test log-compacted topics ## Kafka log cleanup policies Kafka stores messages for a set amount of time and purges messages older than the retention period. This expiration happens due to a policy called **`log.cleanup.policy`**. There are two cleanup policies: | Policy | Default for | Behavior | |--------|-------------|----------| | `delete` | User topics | Deletes events older than retention time | | `compact` | `__consumer_offsets` | Keeps only the most recent value per key | ![cleanup.policy=delete removes messages past the retention period, while cleanup.policy=compact keeps only the latest value per key, removing superseded versions such as an older key=A entry](https://www.conduktor.io/assets/kafka/diagrams/kafka-topic-configuration-log-compaction.svg) ```mermaid flowchart LR subgraph Delete["cleanup.policy=delete"] D1["msg1"] --> D2["msg2"] --> D3["msg3"] D1 -.->|"After retention"| Gone1["Deleted"] end subgraph Compact["cleanup.policy=compact"] C1["key=A, v1"] --> C2["key=B, v1"] --> C3["key=A, v2"] C1 -.->|"After compaction"| Gone2["Removed
(superseded)"] C3 -->|"Kept"| Latest["Latest for key=A"] end ``` ### Purpose of log cleanup Kafka was not initially meant to keep data forever (although now some people are going in that direction with large disks or [Tiered Storage](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage)), nor does it wait for all consumers to read a message before deleting it. By configuring a retention policy for each topic, it allows administrators to: - Control the size of the data on the disk and delete obsolete data - Limit maintenance work on the Kafka Cluster - Limit the amount of historical data a consumer may have to consume to catch up on the topic ## Kafka log compaction theory Kafka supports use cases by allowing the retention policy on a topic to be set to `compact`, **with the property to only retain at least the most recent value for each key in the partition**. It is very useful if we just require a SNAPSHOT instead of full history. ### Log compaction example We want to keep the most recent salary for our employees. We create a topic named `employee-salary` for the purpose. We don't want to know about the old salaries of the employees. ![Kafka Compaction diagram showing how log compaction works in a real world example.](https://www.conduktor.io/assets/kafka/Adv-Kafka-Topic-Log-Comp-2.png) Applications producing these events should contain both a key and a value. The key, in this case, will be employee id, and the value will be their salary. As the data comes in, it will be appended into segments of a partition. After compaction, a new segment is created with only the latest events for a key being retained. The older events for that key are deleted, the offset of the messages are kept intact. ## Log compaction guarantees There are some important guarantees that Kafka provides for messages produced on the log-compacted topics: - **Tail consumers see all messages**: Any consumer that is reading from the tail of a log, i.e., the most current data, will still see all the messages sent to the topic - **Ordering preserved**: Ordering of messages at the key level and partition level is kept, log compaction only removes some messages, but does not re-order them - **Offsets are immutable**: The offset of a message never changes. Offsets are just skipped if a message is missing - **Deleted records visible briefly**: Deleted records can still be seen by consumers for a period of `log.cleaner.delete.retention.ms` (default is 24 hours) ## Log compaction myth busting Let us look at some of the misconceptions around log compaction and clear them. **What log compaction does NOT do:** - **It doesn't prevent duplicate data**: De-duplication is done after a segment is committed. Your consumers will still read from the tail as soon as data arrives - **It doesn't prevent reading duplicates**: If a consumer re-starts, it may see duplicate data based on at-least-once semantics You also can't trigger log compaction using an API call—it happens in the background automatically if enabled. ## How log compaction works If compaction is enabled when Kafka starts, each broker will start a compaction manager thread and a number of compaction threads. These are responsible for performing the compaction tasks. ![Diagram showing the way Kafka Log Compaction is achieved using Cleaner Threads created by brokers when Log Compaction is enabled.](https://www.conduktor.io/assets/kafka/Adv-Kafka-Topic-Log-Comp-3.png) 1. Cleaner threads start with the oldest segment and check their contents. The active segments are left untouched 2. If the message it has just read is still the latest for a key, it copies over the message to a replacement segment. Otherwise it omits the message 3. Once the cleaner thread has copied over all the messages that still contain the latest value for their key, we swap the replacement segment for the original 4. At the end of the process, we are left with one message per key - the one with the latest value > **Ordering intact** > New segments are created/combined from old segments after the cleaner thread has done its work. The offsets are left untouched and the key-based ordering as well. ## Log compaction configurations | Configuration | Default | Description | |---------------|---------|-------------| | `log.cleaner.enable` | true | Enable/disable log compaction | | `log.cleaner.threads` | 1 | Background threads for log cleaning | | `log.segment.ms` | 7 days | Max time before closing active segment | | `log.segment.bytes` | 1GB | Max size of a segment | | `log.cleaner.delete.retention.ms` | 24 hours | How long tombstones are visible | | `log.cleaner.backoff.ms` | 15 seconds | Sleep time when no logs to clean | | `min.cleanable.dirty.ratio` | 0.5 | Minimum dirty ratio to trigger cleaning | ## Log compaction tombstones Compaction also allows for deletes. A message with a key and a `null` payload will be treated as a delete from the log. Such a record is sometimes referred to as a **tombstone**. This delete marker will cause any prior message with that key to be removed. Tombstones are themselves cleaned out of the log after `delete.retention.ms` to free up space. ## Log compaction practice > **Windows Users: you can't use log compaction if you're not using WSL2** > Windows has a long-standing bug (KAFKA-1194) that makes Kafka crash if you use log cleaning. The only way to recover from the crash is to manually delete the folders in `log.dirs` directory. ### Create a log-compacted topic Create a log-compacted topic named `employee-salary` with a single partition and a replication factor of 1: ```bash kafka-topics --bootstrap-server localhost:9092 --create --topic employee-salary \ --partitions 1 --replication-factor 1 \ --config cleanup.policy=compact \ --config min.cleanable.dirty.ratio=0.001 \ --config segment.ms=5000 ``` Configuration explanation: - `cleanup.policy=compact`: Enables log compaction - `min.cleanable.dirty.ratio=0.001`: Ensures log cleanup is triggered frequently (for testing) - `segment.ms=5000`: New segment every 5 seconds (compaction only happens on closed segments) Describe the topic to verify: ```bash kafka-topics --bootstrap-server localhost:9092 --describe --topic employee-salary ``` ### Produce messages with keys Start a Kafka console producer with key parsing enabled: ```bash kafka-console-producer --bootstrap-server localhost:9092 \ --topic employee-salary \ --property parse.key=true \ --property key.separator=, ``` Produce messages with duplicate keys: ``` Patrick,salary: 10000 Lucy,salary: 20000 Bob,salary: 20000 Patrick,salary: 25000 Lucy,salary: 30000 Patrick,salary: 30000 ``` Wait a minute, and produce a few more messages: ``` John,salary: 0 ``` ### Consume and verify compaction Start a consumer to read all messages: ```bash kafka-console-consumer --bootstrap-server localhost:9092 \ --topic employee-salary \ --from-beginning \ --property print.key=true \ --property key.separator=, ``` After compaction completes, you'll see only the unique keys with their latest values: ``` Bob,salary: 20000 Lucy,salary: 30000 Patrick,salary: 30000 John,salary: 0 ``` Log compaction will take place in the background automatically. We cannot trigger it explicitly. However, we can control how often it is triggered with the log compaction properties. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-topic-configuration-log-compaction) lets you browse compacted topics and see the latest value for each key. Monitor compaction progress and verify your cleanup policy configuration is working as expected. ## Next steps - [Configure min.insync.replicas](https://www.conduktor.io/kafka/kafka-topic-configuration-min-insync-replicas) for write durability - [Configure log retention](https://www.conduktor.io/kafka/kafka-topic-configuration-log-retention) for time-based cleanup - [Learn about log segments](https://www.conduktor.io/kafka/kafka-topics-internals-segments-and-indexes) for deeper understanding Source: https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction --- # Kafka topic configuration: log retention *Learn how to configure data retention policies in Kafka* Kafka log retention controls how long messages are stored before being deleted. Understanding retention configuration is essential for managing storage costs, compliance requirements, and consumer catch-up scenarios. **What you'll learn:** - How Kafka's delete cleanup policy works - How to configure retention by time and size - The relationship between broker and topic-level settings - Common retention patterns for different use cases ## Log retention overview Kafka stores messages for a set amount of time and purges messages older than the retention period. This expiration happens due to the `log.cleanup.policy=delete` policy (the default for user topics). ![Within a topic partition, segments past the retention time or size are deleted while segments within retention are kept, and the active segment is always kept](https://www.conduktor.io/assets/kafka/diagrams/kafka-topic-configuration-log-retention--1.svg) ```mermaid flowchart LR subgraph Topic["Topic Partition"] S1["Segment 1
Oldest"] S2["Segment 2"] S3["Segment 3"] S4["Active Segment
Newest"] end S1 -->|"Past retention
time/size"| Delete["Delete"] S2 -->|"Within retention"| Keep["Keep"] S3 --> Keep S4 -->|"Always kept"| Keep ``` ## Retention configuration options ### Retention by time The most common configuration for how long Kafka will retain messages is **by time**. | Setting | Default | Description | |---------|---------|-------------| | `log.retention.hours` | 168 (7 days) | Retention time in hours | | `log.retention.minutes` | - | Retention time in minutes | | `log.retention.ms` | - | Retention time in milliseconds | If more than one is specified, the smaller unit size will take precedence. > **Use milliseconds for consistency** > Because the Kafka CLI command only allows you to set the `ms` version of this parameter, we recommend using `retention.ms` across all your configurations. ### Retention by size Another way to expire messages is based on the total number of bytes of messages retained. | Setting | Default | Description | |---------|---------|-------------| | `log.retention.bytes` | -1 (unlimited) | Maximum size per partition | The default is **-1**, meaning that there is no limit and only a time limit is applied. This parameter is useful to set a to positive value if you want to keep the size of a log under a threshold. > **Broker-level vs Topic-level** > Kafka broker-level topic configurations are prefixed by `log.` and we can remove it to find the equivalent Kafka topic-level configuration. For example, `log.retention.ms` becomes `retention.ms` at the topic level. ## How retention is applied ![Diagram illustrating the process Kafka uses to delete old log segments based on the time or size rules configured for the Kafka Topic.](https://www.conduktor.io/assets/kafka/Adv-Kafka-Topic-Log-Comp-1.png) Retention by time is performed by examining the last modified time on each log segment file on disk. This is the time that the log segment was closed, and represents the timestamp of the last message in the file. If you have specified a value for both `log.retention.bytes` and `log.retention.hours`, messages may be removed when either criteria is met. > **Minimum guarantees, not hard limits** > These are minimum guarantees, not hard limits. The active segment does not count toward the byte limit, and the time limit can be much greater than expected if the segment is very big (few messages per day in a 1GB segment). ## Common retention patterns ### One week of retention (default) ```properties retention.ms=604800000 # 7 days in milliseconds retention.bytes=-1 # No size limit ``` ### Infinite time retention bounded by 500MB ```properties retention.ms=-1 # No time limit retention.bytes=524288000 # 500MB per partition ``` ### Short retention for high-volume topics ```properties retention.ms=86400000 # 24 hours retention.bytes=-1 # No size limit ``` ### Compliance: 90 days retention ```properties retention.ms=7776000000 # 90 days retention.bytes=-1 # No size limit ``` ## Configure retention To set these configurations using the CLI: ```bash kafka-configs --bootstrap-server localhost:9092 \ --alter --entity-type topics --entity-name configured-topic \ --add-config retention.ms=-1,retention.bytes=524288000 ``` ## Retention decision guide ![Decision tree for retention: set retention.ms for compliance, else retention.bytes for storage limits, else a longer retention.ms for historical data or a shorter one otherwise, always monitoring disk usage](https://www.conduktor.io/assets/kafka/diagrams/kafka-topic-configuration-log-retention--2.svg) ```mermaid flowchart TD Start["Configure retention"] --> Q1{"Compliance
requirements?"} Q1 -->|"Yes"| Compliance["Set retention.ms based
on compliance period"] Q1 -->|"No"| Q2{"Storage
constraints?"} Q2 -->|"Yes"| Size["Set retention.bytes
to limit size"] Q2 -->|"No"| Q3{"Need historical
data?"} Q3 -->|"Yes"| Long["retention.ms = longer
(30+ days)"] Q3 -->|"No"| Short["retention.ms = shorter
(1-7 days)"] Compliance --> Monitor["Monitor disk usage"] Size --> Monitor Long --> Monitor Short --> Monitor ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-topic-configuration-log-retention) displays topic retention settings and current storage usage. Monitor disk space consumption to validate your retention configuration meets both storage and compliance needs. > The [Insights dashboard](https://docs.conduktor.io/guide/insights/cost-control) identifies empty, stale, and tiny topics that may be consuming unnecessary storage, helping you make data-driven decisions about retention policies and topic cleanup. ## Next steps - [Understand log compaction](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction) as an alternative cleanup policy - [Learn about log segments](https://www.conduktor.io/kafka/kafka-topics-internals-segments-and-indexes) for deeper understanding - [Configure topics with CLI](https://www.conduktor.io/kafka/how-to-change-a-kafka-topic-configuration-using-the-cli) for more options Source: https://www.conduktor.io/kafka/kafka-topic-configuration-log-retention --- # Kafka topic configuration: minimum in-sync replicas *Learn how min.insync.replicas ensures data durability* The `min.insync.replicas` setting works with producer acknowledgments to control when writes are considered successful. Understanding this configuration is essential for balancing data durability with availability in production Kafka deployments. **What you'll learn:** - How min.insync.replicas works with producer acks - The relationship between replication factor, ISR, and availability - How to configure min.insync.replicas at topic and broker level - Common configuration patterns for production ## How min.insync.replicas works The `min.insync.replicas` setting specifies the minimum number of replicas that have to acknowledge a write for it to be considered successful when using `acks=all`. ![A producer with acks=all writes to the leader, which replicates to two ISR followers; with min.insync.replicas=2, the ACK is returned once two in-sync replicas confirm the write](https://www.conduktor.io/assets/kafka/diagrams/kafka-topic-configuration-min-insync-replicas.svg) ```mermaid flowchart LR P["Producer
acks=all"] -->|"Write"| L["Leader"] L -->|"Replicate"| F1["Follower 1
(ISR)"] L -->|"Replicate"| F2["Follower 2
(ISR)"] subgraph Check["min.insync.replicas=2"] L F1 end Check -->|"2 ISRs confirmed"| ACK["ACK to Producer"] ``` > **The default value of acks has changed with Kafka v3.0:** > - if using Kafka = v3.0, `acks=all` ## Producer acks review ### acks=0 When `acks=0` producers consider messages as "written successfully" the moment the message was sent without waiting for the broker to accept it at all. ![Illustration of Kafka Producer acks Setting set to 0. This 'fire-and-forget' approach is only useful for scenarios where it is OK to potentially lose messages or data.](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-1.png) If the broker goes offline or an exception happens, we won't know and will lose data. This is useful for data where it's okay to potentially lose messages, such as metrics collection, and produces the highest throughput setting because the network overhead is minimized. ### acks=1 When `acks=1` , producers consider messages as "written successfully" when the message was acknowledged by only the leader. ![Overview of process when the Kafka Producer acks Setting is set to 1. The message receipt is only acknowledged by the leader in the Kafka replication setup.](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-2.png) Leader response is requested, but replication is not a guarantee as it happens in the background. If an ack is not received, the producer may retry the request. If the leader broker goes offline unexpectedly but replicas haven't replicated the data yet, we have a data loss. ### acks=all When `acks=all`, producers consider messages as "written successfully" when the message is accepted by all in-sync replicas (ISR). ![Diagram showing process when the Kafka producer acks setting is set to 'all'. The message is acknowledged by all in-sync replicas.](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-3.png) The lead replica for a partition checks to see if there are enough in-sync replicas for safely writing the message (controlled by the broker setting `min.insync.replicas`). The request will be stored in a buffer until the leader observes that the follower replicas replicated the message, at which point a successful acknowledgement is sent back to the client. The`min.insync.replicas` can be configured both at the topic and the broker-level. The data is considered committed when it is written to all in-sync replicas - `min.insync.replicas.` A value of 2 implies that at least 2 brokers that are ISR (including leader) have to respond that they have the data. If you would like to be sure that committed data is written to more than one replica, you need to set the minimum number of in-sync replicas to a higher value. If a topic has three replicas and you set `min.insync.replicas` to `2`, then you can only write to a partition in the topic if at least two out of the three replicas are in-sync. When all three replicas are in-sync, everything proceeds normally. This is also true if one of the replicas becomes unavailable. However, if two out of three replicas are not available, the brokers will no longer accept produce requests. Instead, producers that attempt to send data will receive `NotEnoughReplicasException`. ![Diagram showing how Kafka Topic Replication, ISR and Producer acks settings combine to provide reliable message safety even when 2 out of 3 brokers fail.](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-4.png) ## Durability vs availability trade-offs For a topic replication factor of 3, topic data durability can withstand 2 brokers loss. As a general rule, for a replication factor of `N`, you can permanently lose up to `N-1` brokers and still recover your data. ### Availability matrix | Configuration | Broker failures tolerated | Use case | |---------------|--------------------------|----------| | RF=3, acks=all, min.insync=1 | 2 | Default, low durability | | RF=3, acks=all, min.insync=2 | 1 | **Recommended for production** | | RF=3, acks=all, min.insync=3 | 0 | Maximum durability, no fault tolerance | ### Availability rules - **Reads**: As long as one partition is up and considered an ISR, the topic will be available for reads - **Writes with acks=0 or acks=1**: As long as one partition is up and ISR, writes succeed - **Writes with acks=all**: Must have at least `min.insync.replicas` ISRs available **Formula**: With `acks=all`, `replication.factor=N`, and `min.insync.replicas=M`, you can tolerate `N-M` brokers going down for topic availability purposes. > **Popular Configuration** > `acks=all` and `min.insync.replicas=2` is the most popular option for data durability and availability and allows you to withstand at most the loss of **one** Kafka broker ## Configure min.insync.replicas at topic level > **CLI Extensions** > Use CLI commands with appropriate extensions for your platform, e.g., `kafka-configs.bat` for windows, `kafka-configs.sh` for Linux Before running Kafka CLIs make sure that you have [started Kafka](https://www.conduktor.io/kafka/starting-kafka) successfully. First, create a topic named configured-topic with 3 partitions and a replication factor of 1: ```bash kafka-topics --bootstrap-server localhost:9092 --create --topic configured-topic --partitions 3 --replication-factor 1 ``` Describe the topic to check if there are any configuration override set: ```bash kafka-topics --bootstrap-server localhost:9092 --describe --topic configured-topic ``` ``` Topic: configured-topic TopicId: CDU7SBxBQ1mzJGnuH68-cQ PartitionCount: 3 ReplicationFactor: 1 Configs: Topic: configured-topic Partition: 0 Leader: 2 Replicas: 2 Isr: 2 Topic: configured-topic Partition: 1 Leader: 3 Replicas: 3 Isr: 3 Topic: configured-topic Partition: 2 Leader: 1 Replicas: 1 Isr: 1 ``` Set the `min.insync.replicas` value for the topic `configured-topic` to 2: ```bash kafka-configs --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name configured-topic --add-config min.insync.replicas=2 ``` And describe the topic again: ```bash kafka-topics --bootstrap-server localhost:9092 --describe --topic configured-topic ``` ``` Topic: configured-topic TopicId: CDU7SBxBQ1mzJGnuH68-cQ PartitionCount: 3 ReplicationFactor: 1 Configs: min.insync.replicas=2 Topic: configured-topic Partition: 0 Leader: 2 Replicas: 2 Isr: 2 Topic: configured-topic Partition: 1 Leader: 3 Replicas: 3 Isr: 3 Topic: configured-topic Partition: 2 Leader: 1 Replicas: 1 Isr: 1 ``` Now, you can see there is a topic configuration override set (at the right side of the output) - `min.insync.replicas=2`. You can delete the configuration override by passing `--delete-config` in place of the `--add-config` flag: ```bash kafka-configs --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name configured-topic --delete-config min.insync.replicas ``` ## Configure min.insync.replicas at broker level ### Through a configuration file change The default value of this configuration is `1`. However, this can be changed at the broker level. Open the broker configuration file `config/server.properties` and append the following at the end of the file: ```properties min.insync.replicas=2 ``` Unlike using `kafka-configs` which can change configuration while the broker is running, this, however, requires a broker restart for the configuration change to take effect. ### Dynamic broker configuration change using kafka-configs CLI The `kafka-configs` CLI can also update broker configuration dynamically without requiring a broker restart: ```bash kafka-configs --bootstrap-server localhost:9092 --alter --entity-type brokers --entity-default --add-config min.insync.replicas=2 ``` Output: ``` Completed updating default config for brokers in the cluster. ``` Describe the dynamically updated configurations: ```bash kafka-configs --bootstrap-server localhost:9092 --describe --entity-type brokers --entity-default ``` ``` Default configs for brokers in the cluster are: min.insync.replicas=2 sensitive=false synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:min.insync.replicas=2} ``` Delete the dynamic configuration: ```bash kafka-configs --bootstrap-server localhost:9092 --alter --entity-type brokers --entity-default --delete-config min.insync.replicas ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-topic-configuration-min-insync-replicas) displays topic configurations including min.insync.replicas and replication status. Monitor ISR counts across partitions to ensure your durability settings are effective. > The [Insights dashboard](https://docs.conduktor.io/guide/insights/risk-analysis) automatically identifies topics at risk of data loss based on their replication factor and min.insync.replicas configuration, helping you prioritize remediation. ## Next steps - [Understand unclean leader election](https://www.conduktor.io/kafka/kafka-topic-configuration-unclean-leader-election) trade-offs - [Configure log retention](https://www.conduktor.io/kafka/kafka-topic-configuration-log-retention) for data lifecycle management - [Explore replication and partition decisions](https://www.conduktor.io/kafka/kafka-topics-choosing-the-replication-factor-and-partitions-count) for durability planning Source: https://www.conduktor.io/kafka/kafka-topic-configuration-min-insync-replicas --- # Kafka topic configuration: unclean leader election *Learn about unclean leader election and the availability vs durability trade-off* When no in-sync replicas (ISRs) are available, Kafka has to decide between staying unavailable or promoting an out-of-sync replica to leader. The `unclean.leader.election.enable` configuration controls this critical trade-off. **What you'll learn:** - What unclean leader election means - When unclean elections cause data loss - Use cases where enabling this setting makes sense - How to configure unclean leader election ## Clean vs unclean leader election When the leader for a partition becomes unavailable, one of the in-sync replicas (ISR) becomes the new leader. This is a "clean" election because committed data exists on all ISRs by definition. But what happens when no ISR exists except for the failed leader? When the leader fails and no ISR is available, the outcome depends on `unclean.leader.election`: ![When a leader fails with no ISR available, the false default waits for an ISR (data safe, availability impacted) while true promotes an out-of-sync replica (available immediately, potential data loss)](https://www.conduktor.io/assets/kafka/diagrams/kafka-topic-configuration-unclean-leader-election--1.svg) ```mermaid flowchart TD Fail["Leader fails
No ISR available"] --> Choice{{"unclean.leader.election
enabled?"}} Choice -->|"false (default)"| Wait["Wait for ISR
(Topic unavailable)"] Choice -->|"true"| Promote["Promote out-of-sync replica
(Potential data loss)"] Wait --> Safe["Data safe
Availability impacted"] Promote --> Available["Available immediately
Some data may be lost"] ``` | Option | Behavior | Risk | |--------|----------|------| | Wait for ISR (default) | Topic unavailable until ISR recovers | Availability loss | | Unclean election | Promote out-of-sync replica | Data loss | ## The availability vs durability trade-off **If we allow out-of-sync replicas to become leaders, we will have data loss and data inconsistencies.** If we don't allow them to become leaders, we face lower availability as we have to wait for the original leader to become available before the partition is back online. > **Dangerous setting** > Unclean leader election can cause permanent data loss. Understand the implications fully before enabling it. The default value of `false` is recommended for most use cases. ## When to enable unclean leader election ![Decision tree for unclean leader election: keep it disabled unless data loss is acceptable and availability is critical; enable it for metrics collection, log aggregation, and non-critical events, but keep it disabled for financial transactions, order processing, and audit logs](https://www.conduktor.io/assets/kafka/diagrams/kafka-topic-configuration-unclean-leader-election--2.svg) ```mermaid flowchart TD Start["Consider unclean
leader election?"] --> Q1{{"Is data loss
acceptable?"}} Q1 -->|"No"| Disable["Keep disabled
(default)"] Q1 -->|"Yes"| Q2{{"Is availability
critical?"}} Q2 -->|"No"| Disable Q2 -->|"Yes"| Enable["Consider enabling"] Enable --> Examples["Use cases:
- Metrics collection
- Log aggregation
- Non-critical events"] Disable --> Critical["Use cases:
- Financial transactions
- Order processing
- Audit logs"] ``` | Use case | Recommendation | Reason | |----------|---------------|--------| | Financial transactions | Disable | Data loss unacceptable | | Metrics/logs | Consider enabling | Availability more important | | Event streaming | Depends on criticality | Evaluate trade-off | | Audit compliance | Disable | Regulatory requirements | For an in-depth analysis, see [Kafka at Datadog: Unclean Leader Elections](https://www.datadoghq.com/blog/kafka-at-datadog/#unclean-leader-elections-to-enable-or-not-to-enable). ## Configure unclean leader election ### Enable for a specific topic ```bash kafka-configs --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name configured-topic \ --add-config unclean.leader.election.enable=true ``` ### Disable (restore default) ```bash kafka-configs --bootstrap-server localhost:9092 --alter \ --entity-type topics --entity-name configured-topic \ --delete-config unclean.leader.election.enable ``` ### Verify configuration ```bash kafka-configs --bootstrap-server localhost:9092 --describe \ --entity-type topics --entity-name configured-topic ``` Configure unclean leader election per topic rather than cluster-wide. This lets you enable it only for topics where availability is more important than durability. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-topic-configuration-unclean-leader-election) displays topic configurations including unclean leader election status. Monitor ISR counts and leader distribution to understand your replication health. ## Next steps - [Explore advanced producers](https://www.conduktor.io/kafka/kafka-producers-advanced) to build on these durability settings - [Configure min.insync.replicas](https://www.conduktor.io/kafka/kafka-topic-configuration-min-insync-replicas) for durability guarantees - [Set up monitoring](https://www.conduktor.io/kafka/kafka-monitoring-and-operations) for ISR health Source: https://www.conduktor.io/kafka/kafka-topic-configuration-unclean-leader-election --- # Kafka topic replication *Learn how Kafka replication provides fault tolerance* Replication copies each partition to multiple brokers so data survives broker failures. This section explains how leaders, followers, ISR and producer acknowledgments work together to keep your data safe. **What you'll learn:** - How replication protects against data loss - What leaders, followers, and in-sync replicas (ISR) are - How producer acknowledgment settings affect durability - Trade-offs between availability and consistency ## Kafka topic replication factor **One of the main reasons for Kafka's popularity, is the resilience it offers in the face of broker failures.** Machines fail, and often we cannot predict when that is going to happen or prevent it. Kafka is designed with replication as a core feature to withstand these failures while maintaining uptime and data accuracy. > **Kafka replication** helps prevent data loss by writing the same data to more than one broker. In Kafka, replication means that data is written down not just to one broker, but many. The replication factor is a topic setting and is specified at topic creation time. - A replication factor of `1` means no replication. It is mostly used for development purposes and should be avoided in test and production Kafka clusters - A replication factor of `3` is a commonly used replication factor as it provides the right balance between broker loss and replication overhead. In the cluster below consisting of three brokers, the replication factor is `2`. When a message is written down into *Partition 0* of *Topic-A* in *Broker 101*, it is also written down into *Broker 102* because it has *Partition 0* as a replica. ![Diagram showing a default Apache Kafka Replication Factor of 2 distributing messages across 3 different Kafka Brokers.](https://www.conduktor.io/assets/kafka/Kafka-Topic-Replication-1.png) Thanks to a replication factor of 2, we can withstand the failure of one broker. This means that if *Broker 102* failed, as you see below, *Broker 101 and 103* would still have the data. ![Diagram showing how Kafka Topic Replication provides resilience when a Kafka Broker fails.](https://www.conduktor.io/assets/kafka/Kafka-Topic-Replication-2.png) ### Replication factor guidelines | Environment | Replication factor | Rationale | |-------------|-------------------|-----------| | Development | 1 | Saves resources, data loss acceptable | | Testing | 2 | Basic fault tolerance | | Production | 3 | Industry standard, tolerates 2 failures | | Critical data | 5 | Maximum durability | ## What are Kafka partitions leader and replicas? For a given topic-partition, one Kafka broker is designated by the cluster to be responsible for sending and receiving data to clients. That broker is known as the leader broker of that topic partition. Any other broker that is storing replicated data for that partition is referred to as a replica. Therefore, each partition has one leader and multiple replicas. ## What are in-sync replicas (ISR)? An ISR is a replica that is up to date with the leader broker for a partition. Any replica that is not up to date with the leader is out of sync. ![Kafka Topic Replication relies on leader selection and the creation of in-sync replicas (ISR). This diagram shows the Kafka replication process across 3 brokers.](https://www.conduktor.io/assets/kafka/Kafka-Topic-Replication-3.png) Leaders and in-sync replicas Here we have *Broker 101* as *Partition 0* leader and *Broker 102* as the leader of *Partition 1*. *Broker 102* is a replica for *Partition 0* and *Broker 103* is a replica for *Partition 1*. If the leader broker were to fail, one of the replicas will be elected as the new partition leader by an election. ![Kafka replication flow across a 3-broker cluster: the producer writes to P0 on Broker 1 (leader of P0), Broker 1 replicates to Broker 2 (ISR for P0 and leader of P1), and the consumer reads from P0 on Broker 1. Broker 3 is the ISR for P1.](https://www.conduktor.io/assets/kafka/diagrams/kafka-topic-replication.svg) ```mermaid flowchart LR subgraph Cluster["Kafka Cluster"] B1["Broker 1
Leader P0"] B2["Broker 2
ISR P0
Leader P1"] B3["Broker 3
ISR P1"] end P["Producer"] -->|"Write to P0"| B1 B1 -->|"Replicate"| B2 C["Consumer"] -->|"Read from P0"| B1 ``` ## Kafka producers acks setting Kafka producers only write data to the current leader broker for a partition. Kafka producers also have to specify a level of acknowledgment `acks` to specify if the message has to be written to a minimum number of replicas before being considered a successful write. > The default value of `acks` has changed in Kafka v3.0: > - if using Kafka = v3.0, `acks=all` ### acks=0 When `acks=0` producers consider messages as "written successfully" the moment the message was sent without waiting for the broker to accept it at all. ![Illustration of Kafka Producer acks Setting set to 0. This 'fire-and-forget' approach is only useful for scenarios where it is OK to potentially lose messages or data.](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-1.png) If the broker goes offline or an exception happens, we won't know and will lose data. This is useful for data where it's okay to potentially lose messages, such as metrics collection, and produces the highest throughput setting because the network overhead is minimized. ### acks = 1 When `acks=1`, producers consider messages as "written successfully" when the message was acknowledged by only the leader. ![Overview of process when the Kafka producer acks setting is set to 1. The message receipt is only acknowledged by the leader in the Kafka replication setup.](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-2.png) Leader response is requested, but replication is not a guarantee as it happens in the background. If an ack is not received, the producer may retry the request. If the leader broker goes offline unexpectedly but replicas haven't replicated the data yet, we have a data loss. ### acks = all When `acks=all`, producers consider messages as "written successfully" when the message is accepted by all in-sync replicas (ISR). ![Diagram showing process when the Kafka producer acks setting is set to 'all'. The message is acknowledged by all in-sync replicas.](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-3.png) The lead replica for a partition checks to see if there are enough in-sync replicas for safely writing the message (controlled by the broker setting `min.insync.replicas`). The request will be stored in a buffer until the leader observes that the follower replicas replicated the message, at which point a successful acknowledgement is sent back to the client. The `min.insync.replicas` can be configured both at the topic and the broker-level. The data is considered committed when it is written to all in-sync replicas - `min.insync.replicas.` A value of 2 implies that at least 2 brokers that are ISR (including leader) have to respond that they have the data. If you would like to be sure that committed data is written to more than one replica, you need to set the minimum number of in-sync replicas to a higher value. If a topic has three replicas and you set `min.insync.replicas` to `2`, then you can only write to a partition in the topic if at least two out of the three replicas are in-sync. When all three replicas are in-sync, everything proceeds normally. This is also true if one of the replicas becomes unavailable. However, if two out of three replicas are not available, the brokers will no longer accept produce requests. Instead, producers that attempt to send data will receive `NotEnoughReplicasException`. ![Diagram showing how Kafka Topic Replication, ISR and Producer acks settings combine to provide reliable message safety even when 2 out of 3 brokers fail.](https://www.conduktor.io/assets/kafka/Adv-Producer-Acks-DD-4.png) ### Acks comparison | Setting | Durability | Throughput | Latency | Use case | |---------|-----------|------------|---------|----------| | `acks=0` | None | Highest | Lowest | Metrics, logs | | `acks=1` | Leader only | High | Low | Most applications | | `acks=all` | Full | Lower | Higher | Critical data | ## Kafka topic durability and availability For a topic replication factor of 3, topic data durability can withstand the loss of 2 brokers. As a general rule, for a replication factor of `N`, you can permanently lose up to `N-1` brokers and still recover your data. Regarding availability, it is a little bit more complicated... To illustrate, let's consider a replication factor of 3: - Reads: As long as one partition is up and considered an ISR, the topic will be available for reads - Writers: - `acks=0` & `acks=1`: as long as one partition is up and considered an ISR, the topic will be available for writes. - `acks=all`: - `min.insync.replicas=1` (default): the topic has to have at least 1 partition up as an ISR (that includes the reader) and so we can tolerate two brokers being down - `min.insync.replicas=2`: the topic has to have at least 2 ISR up, and therefore we can tolerate at most one broker being down (in the case of replication factor of 3), and we have the guarantee that for every write, the data will be at least written twice. - `min.insync.replicas=3`: this wouldn't make much sense for a corresponding replication factor of 3 and we couldn't tolerate any broker going down. - in summary, when `acks=all` with a `replication.factor=N` and `min.insync.replicas=M` we can tolerate `N-M` brokers going down for topic availability purposes > **Kafka topic replication settings** > `acks=all` and `min.insync.replicas=2` is the most popular option for data durability and availability and allows you to withstand at most the loss of **one** Kafka broker. ## Kafka consumers replicas fetching Kafka consumers read by default from the partition leader. ![Diagram showing how Kafka consumers typically read from the topic partition leader](https://www.conduktor.io/assets/kafka/Kafka-Topic-Partition-Leader--1-.png) But since Apache Kafka 2.4, it is possible to configure consumers to read from in-sync replicas instead (usually the closest). Reading from the closest in-sync replicas (ISR) may improve the request latency, and also decrease network costs, because in most cloud environments cross-data centers network requests incur charges. ![Diagram showing the Kafka Consumers replica fetching process in Apache Kafka](https://www.conduktor.io/assets/kafka/Kafka-Consumers-Replica-Fetching.png) ## Preferred leader The preferred leader is the designated leader broker for a partition at topic creation time (as opposed to being a replica). > **Leader election** is the process of deciding which broker is a leader at topic creation time is called a preferred leader election. When the preferred leader goes down, any partition that is an ISR (in-sync replica) is eligible to become a new leader (but not a preferred leader). Upon recovering the preferred leader broker and having its partition data back in sync, the preferred leader will regain leadership for that partition. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-topic-replication) displays replication status for all topics, showing which partitions are under-replicated and ISR status. Monitor leader distribution and identify imbalanced clusters at a glance. ## Next steps - [See how ZooKeeper coordinates brokers](https://www.conduktor.io/kafka/zookeeper-with-kafka) to understand the metadata layer behind replication and leader election - [Configure min in-sync replicas](https://www.conduktor.io/kafka/kafka-topic-configuration-min-insync-replicas) for write durability - [Choose replication factor and partitions](https://www.conduktor.io/kafka/kafka-topics-choosing-the-replication-factor-and-partitions-count) for your topics Source: https://www.conduktor.io/kafka/kafka-topic-replication --- # Advanced Kafka topics Learn about advanced topic configurations, internals, and optimization strategies for Apache Kafka topics. ## What you'll learn - How to configure topics using CLI tools - Topic internals including segments and indexes - Log retention and compaction strategies - Replication factor and partition count optimization - Handling large messages - Topic naming conventions and best practices ## Next steps - [Explore topic internals](https://www.conduktor.io/kafka/kafka-topics-internals-segments-and-indexes) to understand segments and indexes - [Change topic configuration](https://www.conduktor.io/kafka/how-to-change-a-kafka-topic-configuration-using-the-cli) using the CLI - [Configure log retention](https://www.conduktor.io/kafka/kafka-topic-configuration-log-retention) to manage storage Source: https://www.conduktor.io/kafka/kafka-topics-advanced --- # Kafka topics: choosing the replication factor and partition count *Learn how to choose optimal replication and partitioning settings* Replication factor and partition count directly affect performance, availability and scalability. Get them wrong and you hit throughput ceilings or lose data during failures. **What you'll learn:** - How to select the appropriate replication factor for your use case - How to calculate the right number of partitions for throughput requirements - Trade-offs between fault tolerance, performance, and resource usage - Production-ready configuration patterns ## Replication factor The replication factor determines how many copies of your data are maintained across the Kafka cluster. ### How replication works - Each topic partition has one **leader** and zero or more **followers** - All reads and writes go through the leader - Followers replicate data from the leader - If a leader fails, one of the followers becomes the new leader ### Replication architecture diagram This diagram shows how Kafka replicates data across brokers for fault tolerance: ![](https://www.conduktor.io/assets/kafka/replication-kaka.png) **Key concepts:** - **Replication factor 3** means one leader + two followers - **In-sync replicas (ISR)** are followers that are caught up with the leader - If leader fails, one ISR becomes the new leader - Producers can wait for ISR acknowledgment for durability (configure with `acks=all`) ### Choose replication factor **For production environments:** - **Minimum**: 3 (recommended) - **Common**: 3-5 depending on cluster size and requirements - **Maximum**: Generally not more than 5-7 **For development/testing:** - **Minimum**: 1 (acceptable for non-critical data) - **Recommended**: 2-3 for realistic testing ### Decision tree: Choose your replication factor Use this decision tree to determine the right replication factor for your topic: ![](https://www.conduktor.io/assets/kafka/replication-decide.png) > **Recommended starting point:** For most production workloads, start with replication factor 3 and `min.insync.replicas=2`. This provides a good balance between fault tolerance, performance, and resource usage. ### Replication factor considerations > **Fault tolerance formula** > With replication factor `N`, you can tolerate up to `N-1` broker failures while maintaining data availability. > - Replication factor 1: No fault tolerance (data loss if broker fails) > - Replication factor 3: Tolerates 2 broker failures > - Replication factor 5: Tolerates 4 broker failures **Trade-offs:** - **Higher replication factor**: - ✅ Better fault tolerance and availability - ✅ Higher data durability - ❌ Increased storage requirements - ❌ Higher network overhead - ❌ Increased replication lag - **Lower replication factor**: - ✅ Lower storage costs - ✅ Reduced network overhead - ❌ Reduced fault tolerance - ❌ Higher risk of data loss ## Partition count Partitions enable Kafka to scale and parallelize data processing across multiple brokers and consumers. ### How partitions work - Topics are divided into partitions - Each partition is an ordered, immutable sequence of messages - Partitions are distributed across brokers - Consumers can process partitions in parallel ### Choose partition count **Starting recommendations:** - **Small topics** (< 1GB/day): 1-3 partitions - **Medium topics** (1-10GB/day): 6-12 partitions - **Large topics** (> 10GB/day): 20+ partitions ### Decision tree: Choose your partition count Use this decision tree to calculate the optimal number of partitions: ![](https://www.conduktor.io/assets/kafka/partition-decide.png) > **Production recommendation:** Start with 6-12 partitions for most new topics, then scale based on observed throughput. Plan for 2-3x your expected peak load to accommodate growth. **Factors to consider:** ### 1. Throughput requirements ``` Partitions needed ≥ Target throughput / Single partition throughput ``` Example: If you need 100MB/s and each partition handles 10MB/s, you need at least 10 partitions. ### 2. Consumer parallelism - Maximum consumers in a consumer group = Number of partitions - More partitions = More potential parallelism - Fewer partitions = Less parallelism but simpler management ### 3. Broker distribution - Partitions should be evenly distributed across brokers - Each broker should handle a reasonable number of partitions - Avoid having too many partitions per broker (recommended: < 4000) ### Partition count considerations **Trade-offs:** - **More partitions**: - ✅ Higher potential throughput - ✅ Better parallelism for consumers - ✅ Better load distribution - ❌ More overhead (file handles, memory) - ❌ Longer leader election times - ❌ More complex partition management - **Fewer partitions**: - ✅ Lower resource overhead - ✅ Simpler management - ✅ Faster leader elections - ❌ Limited throughput potential - ❌ Less consumer parallelism ## Best practices ### Plan for growth > **Partition planning** > Start with enough partitions to handle 2-3x your expected peak throughput. It's easier to start with more partitions than to add them later. **Partitions cannot be decreased** after topic creation, so plan for growth: 1. Estimate peak throughput needs for the next 2-3 years 2. Add 50-100% buffer for unexpected growth 3. Consider data retention and storage requirements ### Production recommendations **Replication factor:** - Use replication factor 3 for most production workloads - Use replication factor 5 for critical data that cannot tolerate loss - Never use replication factor 1 in production **Partition count:** - Start with 6-12 partitions for most new topics - Scale up based on observed throughput requirements - Aim for 10-100MB per partition per day as a rough guideline ### Performance testing Before finalizing partition and replication settings: 1. **Test with realistic data volumes** 2. **Measure single partition throughput** 3. **Test consumer group scaling** 4. **Monitor resource usage** (CPU, memory, disk I/O, network) 5. **Test failure scenarios** (broker failures, network partitions) ### Monitor and adjust Key metrics to monitor: - **Throughput per partition** - **Consumer lag by partition** - **Broker resource utilization** - **Replication lag** - **Leader election frequency** > **Partition limits** > Be aware of cluster-wide partition limits: > - Each broker has limits on total partitions (typically 2000-4000) > - ZooKeeper has metadata overhead for each partition > - Too many partitions can impact cluster stability ## Common patterns ### High-throughput topics ```bash # Large topic with high replication for critical data kafka-topics --create \ --topic high-throughput-topic \ --partitions 24 \ --replication-factor 3 \ --bootstrap-server localhost:9092 ``` ### Low-volume, critical topics ```bash # Small topic with high replication for critical data kafka-topics --create \ --topic critical-events \ --partitions 3 \ --replication-factor 5 \ --bootstrap-server localhost:9092 ``` ### Development/testing topics ```bash # Simple topic for development kafka-topics --create \ --topic dev-topic \ --partitions 1 \ --replication-factor 1 \ --bootstrap-server localhost:9092 ``` > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-topics-choosing-the-replication-factor-and-partitions-count) provides a visual interface for creating and managing topics with guided configuration for replication factor and partition count. View real-time metrics for throughput per partition, consumer lag, and broker distribution to validate your configuration choices. > The [Insights dashboard](https://docs.conduktor.io/guide/insights/risk-analysis) analyzes cluster efficiency and load imbalance, identifying topics with sub-optimal partition allocation or uneven data distribution across partitions. ## Next steps - [Configure log retention](https://www.conduktor.io/kafka/kafka-topic-configuration-log-retention) to manage storage effectively - [Understand min in-sync replicas](https://www.conduktor.io/kafka/kafka-topic-configuration-min-insync-replicas) to configure write durability - [Explore topic configuration](https://www.conduktor.io/kafka/how-to-change-a-kafka-topic-configuration-using-the-cli) for additional settings Source: https://www.conduktor.io/kafka/kafka-topics-choosing-the-replication-factor-and-partitions-count --- # Kafka topics CLI tutorial *Learn how to manage Kafka topics using the kafka-topics CLI* The `kafka-topics` CLI is your primary tool for creating, listing, describing, altering, and deleting Kafka topics. Topics live inside brokers across the cluster; see the [Kafka architecture diagram](https://www.conduktor.io/glossary/kafka-architecture-diagram) for the full picture. **What you'll learn:** - How to create topics with specific partitions and replication factor - How to list and describe existing topics - How to alter topic partition count - How to delete topics > **Platform extensions** > Use CLI commands with appropriate extensions for your platform: `.bat` for Windows, `.sh` for Mac and Linux. ## How to create a Kafka topic? To create a Kafka topic, you need to provide: - Kafka hostname and port (e.g., `localhost:9092`) - Topic name - Number of partitions - Replication factor ### Example Creating a topic named `first_topic` with 3 partitions and replication factor of 1: For Kafka v2.2+: ```bash kafka-topics --bootstrap-server localhost:9092 --topic first_topic --create --partitions 3 --replication-factor 1 ``` ### Important gotchas - Cannot specify a replication factor greater than the number of brokers - No default values for partitions and replication factor - Topic name must contain only ASCII alphanumerics, '.', '_' and '-' ## How to list Kafka topics? Use `kafka-topics` with the `--list` option. ### Example For Kafka v2.2+: ```bash kafka-topics --bootstrap-server localhost:9092 --list ``` ## How to describe a Kafka topic? Use `kafka-topics` with the `--describe` option. ### Example For Kafka v2.2+: ```bash kafka-topics --bootstrap-server localhost:9092 --describe --topic first_topic ``` ## How to increase the number of partitions? Use `kafka-topics` with the `--alter` option. > **Caution**: Increasing the number of partitions in a Kafka topic is a DANGEROUS OPERATION if your applications are relying on key-based ordering. ### Example ```bash kafka-topics --bootstrap-server localhost:9092 --alter --topic first_topic --partitions 5 ``` ## How to delete a Kafka topic? Use `kafka-topics` with the `--delete` option. ### Example ```bash kafka-topics --bootstrap-server localhost:9092 --delete --topic first_topic ``` > Ensure `delete.topic.enable=true` is set on brokers for deletion to work properly. ## Quick reference | Operation | Command | |-----------|---------| | Create topic | `kafka-topics --bootstrap-server localhost:9092 --create --topic NAME --partitions N --replication-factor R` | | List topics | `kafka-topics --bootstrap-server localhost:9092 --list` | | Describe topic | `kafka-topics --bootstrap-server localhost:9092 --describe --topic NAME` | | Add partitions | `kafka-topics --bootstrap-server localhost:9092 --alter --topic NAME --partitions N` | | Delete topic | `kafka-topics --bootstrap-server localhost:9092 --delete --topic NAME` | > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-topics-cli-tutorial) provides a visual interface for topic management. Create, configure, and monitor topics without memorizing CLI syntax. ## Next steps - [Produce messages](https://www.conduktor.io/kafka/kafka-producer-cli-tutorial) to your new topic with the console producer - [Consume messages](https://www.conduktor.io/kafka/kafka-consumer-cli-tutorial) from topics - [Configure topic settings](https://www.conduktor.io/kafka/how-to-change-a-kafka-topic-configuration-using-the-cli) for production Source: https://www.conduktor.io/kafka/kafka-topics-cli-tutorial --- # Kafka topic internals: segments and indexes *Learn how Kafka stores data on disk with segments and indexes* Understanding Kafka's storage internals helps you troubleshoot issues, tune configurations, and make informed decisions about segment sizing and retention policies. **What you'll learn:** - How partitions are split into segments on disk - The role of offset and timestamp indexes - Segment configuration options and their impact - How to inspect Kafka's directory structure ## Kafka topic partitions and segments The basic storage unit of Kafka is a partition replica. When you create a topic, Kafka first decides how to allocate the partitions between brokers. It spreads replicas evenly among brokers. Kafka brokers split each partition into **segments**. Each segment is stored in a single data file on the disk attached to the broker. By default, each segment contains either 1 GB of data or a week of data, whichever limit is attained first. When the Kafka broker receives data for a partition, as the segment limit is reached, it will close the file and start a new one: ![Kafka Topic Internals Diagram showing how Kafka Topic Partitions are divided into Segments based on the number of offsets in the partition.](https://www.conduktor.io/assets/kafka/Adv-Kafka-Topic-Internals-1.png) Only one segment is ACTIVE at any point in time - the one data is being written to. A segment can only be deleted if it has been closed beforehand. ### Segment configuration | Configuration | Default | Description | |---------------|---------|-------------| | `log.segment.bytes` | 1 GB | Maximum size of a single segment | | `log.segment.ms` | 7 days | Time before closing segment if not full | > **Topic-level override** > These broker-level configurations can be overridden at the topic level using `segment.bytes` and `segment.ms`. See [log retention](https://www.conduktor.io/kafka/kafka-topic-configuration-log-retention) for more details. A Kafka broker keeps an open file handle to every segment in every partition - even inactive segments. This leads to a usually high number of open file handles, and the OS has to be tuned accordingly. ## Kafka topic segments and indexes Kafka allows consumers to start fetching messages from any available offset. To help brokers quickly locate the message for a given offset, Kafka maintains two indexes for each segment: | Index type | Purpose | Use case | |------------|---------|----------| | Offset to position | Maps offset to byte position in segment | Fast message lookup by offset | | Timestamp to offset | Maps timestamp to nearest offset | Time-based message seeking | ![Diagram showing how Topic Partitions are split into segments and how Kafka maintains two different index types for each segment in the partition, a position index and a timestamp index.](https://www.conduktor.io/assets/kafka/Adv-Kafka-Topic-Internals-2.png) ## Inspect the Kafka directory structure Kafka stores all of its data in a directory on the broker disk. This directory is specified using the property `log.dirs` in the broker's configuration file. For example, ```properties # A comma separated list of directories under which to store log files log.dirs=/tmp/kafka-logs ``` Explore the directory and notice that there is a folder for each topic partition. All the segments of the partition are located inside the partition directory. Here, the topic named `configured-topic` has three partitions, each having one directory - `configured-topic-0`, `configured-topic-1` and `configured-topic-2`. ![Kafka Storage Windows Screenshot showing where Kafka stores logs such as log.dirs and how the data is structure in Topic and Segment folders.](https://www.conduktor.io/assets/kafka/image--61-.png) Descend into a directory for a topic partition. Notice the indexes - time and offset for the segment and the segment file itself where the messages are stored. ![Kafka Internals Screenshot showing Kafka Logs in Windows and the two types of Index, timestamp and offset, for a segment within a Kafka Topic Partition.](https://www.conduktor.io/assets/kafka/image--62-.png) ## Considerations for segment configurations Let us review the configurations for segments and learn their importance. ### log.segment.bytes As messages are produced to the Kafka broker, they are appended to the current segment for the partition. Once the segment reaches the size specified by `log.segment.bytes` (default 1 GB), the segment is closed and a new one is opened. **Considerations:** - A smaller segment size means files have to be closed and allocated more often, reducing disk write efficiency - Once closed, segments become eligible for cleanup based on retention policy - Topics with low produce rates may need smaller segments to enable timely cleanup - Very small segments increase open file handles, risking "Too many open files" errors ### log.segment.ms Specifies the time after which a segment should be closed (default 1 week). Kafka closes a segment when either the size limit or time limit is reached, whichever comes first. **Considerations:** - Time-based limits can cause multiple segments to close simultaneously, impacting disk performance - Shorter times enable more frequent [log compaction](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction) > **File handle limits** > A Kafka broker keeps an open file handle to every segment in every partition. With many partitions and segments, this can exhaust OS file handle limits. Tune your OS `ulimit` settings accordingly. ## Segment sizing decision guide ![Decision tree for segment sizing: use larger segments for high message volume, smaller segments or a shorter segment.ms when frequent cleanup is needed, otherwise keep defaults, always monitoring disk usage and file handles](https://www.conduktor.io/assets/kafka/diagrams/kafka-topics-internals-segments-and-indexes.svg) ```mermaid flowchart TD Start["Configure segments"] --> Q1{{"High message
volume?"}} Q1 -->|"Yes"| Large["Use larger segments
(1GB default)"] Q1 -->|"No"| Q2{{"Need frequent
cleanup?"}} Q2 -->|"Yes"| Small["Use smaller segments
or shorter segment.ms"] Q2 -->|"No"| Default["Keep defaults"] Large --> Monitor["Monitor disk usage
and file handles"] Small --> Monitor Default --> Monitor ``` ## Next steps - [Change topic configuration](https://www.conduktor.io/kafka/how-to-change-a-kafka-topic-configuration-using-the-cli) to tune segment settings - [Configure log retention](https://www.conduktor.io/kafka/kafka-topic-configuration-log-retention) for time-based cleanup - [Understand log compaction](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction) for key-based retention Source: https://www.conduktor.io/kafka/kafka-topics-internals-segments-and-indexes --- # Kafka topic naming conventions *Learn how to design effective topic naming conventions* Well-designed topic naming conventions improve system organization, maintainability, and team productivity while preventing naming conflicts and confusion. **What you'll learn:** - Character restrictions and length limitations for topic names - Common naming patterns (kebab-case, snake_case, dot notation) - Organizational strategies for domains, teams, and environments - Anti-patterns to avoid and migration strategies ## Importance of naming conventions Good topic naming conventions provide multiple benefits: - **Organizational clarity**: Instantly understand topic purpose and ownership - **Conflict prevention**: Avoid naming collisions between teams and applications - **Operational efficiency**: Easier monitoring, alerting, and troubleshooting - **Automation support**: Enable automated tooling and deployment processes - **Scalability**: Support growth from small to large organizations > **Establish early** > Define naming conventions before your first production deployment. Changing topic names later requires data migration and application updates. ## Character restrictions and limitations Kafka topic names have specific character and length restrictions. ### Allowed characters Topic names can contain: - Lowercase letters (a-z) - Uppercase letters (A-Z) - Numbers (0-9) - Periods (.) - Underscores (_) - Hyphens (-) ### Restricted characters Topic names cannot contain: - Spaces ( ) - Forward slashes (/) - Backslashes (\) - Commas (,) - Control characters (ASCII 0-31) - Characters above ASCII 127 ### Length limitations | Limit | Value | Recommendation | |-------|-------|----------------| | Maximum | 255 characters | Hard Kafka limit | | Practical | < 100 characters | For readability | | Recommended | < 50 characters | For maintainability | | Minimum | 1 character | Not recommended | ### Special restrictions **Reserved names to avoid:** - Names starting with double underscores (`__`) - reserved for internal topics - Names containing only dots (`.` or `..`) - filesystem conflicts - Case-sensitive duplicates - avoid `Topic` and `topic` simultaneously **Examples:** ```bash # Valid topic names user-events order.processing.queue CustomerData_v2 analytics-2024 # Invalid topic names user events # Contains space order/processing # Contains slash __internal_topic # Reserved prefix user,events # Contains comma ``` ## Naming pattern decision guide ![Decision tree for naming patterns: scale the pattern by organization size from a simple name to domain context to full ownership, then pick a style of kebab-case for URL-friendly, snake_case for code-friendly, or dot notation for hierarchical](https://www.conduktor.io/assets/kafka/diagrams/kafka-topics-naming-convention.svg) ```mermaid flowchart TD Start["Choose naming pattern"] --> Q1{{"Organization
size?"}} Q1 -->|"Small (< 10)"| Simple["Simple pattern
user-events"] Q1 -->|"Medium (10-50)"| Domain["Add domain context
users.events"] Q1 -->|"Large (50+)"| Full["Include ownership
platform.users.events"] Simple --> Style{{"Style preference?"}} Domain --> Style Full --> Style Style -->|"URL-friendly"| Kebab["kebab-case"] Style -->|"Code-friendly"| Snake["snake_case"] Style -->|"Hierarchical"| Dot["dot.notation"] ``` ## Basic naming patterns ### Kebab case (recommended) Use lowercase with hyphens for separators: ```bash user-events order-processing payment-transactions inventory-updates ``` **Benefits:** - Easy to read and type - Consistent with URL patterns - Works well with command-line tools ### Snake case Use lowercase with underscores for separators: ```bash user_events order_processing payment_transactions inventory_updates ``` **Benefits:** - Familiar to developers - Compatible with variable naming - Clear word separation ### Dot notation Use dots to create hierarchical names: ```bash analytics.user.events commerce.order.created system.health.metrics audit.security.events ``` **Benefits:** - Creates natural hierarchy - Supports grouping and filtering - Good for enterprise organizations ## Organizational naming strategies ### Domain-based naming Organize topics by business domain: ```bash # User domain users.registration users.profile-updates users.authentication # Order domain orders.created orders.updated orders.cancelled # Payment domain payments.processed payments.failed payments.refunded ``` ### Service-based naming Organize topics by owning service: ```bash # User service user-service.events user-service.commands user-service.replies # Order service order-service.events order-service.commands order-service.replies # Payment service payment-service.events payment-service.commands payment-service.replies ``` ### Data type-based naming Organize topics by data characteristics: ```bash # Events (immutable facts) events.user-registered events.order-placed events.payment-completed # Commands (requests for action) commands.process-payment commands.update-inventory commands.send-notification # Snapshots (current state) snapshots.user-profiles snapshots.product-catalog snapshots.inventory-levels ``` ## Environment-specific naming ### Environment prefixes Include environment in topic names: ```bash # Development environment dev.user-events dev.order-processing dev.analytics-events # Staging environment staging.user-events staging.order-processing staging.analytics-events # Production environment prod.user-events prod.order-processing prod.analytics-events ``` ### Environment suffixes Place environment at the end: ```bash # Different environments user-events.dev user-events.staging user-events.prod order-processing.dev order-processing.staging order-processing.prod ``` ### Cluster separation (recommended) Use separate Kafka clusters for environments: ```bash # Development cluster user-events order-processing analytics-events # Production cluster (same names, different cluster) user-events order-processing analytics-events ``` **Benefits:** - Complete environment isolation - Simpler topic names - No cross-environment data leakage risk ## Versioning strategies ### Version suffixes Include version in topic names: ```bash user-events-v1 user-events-v2 user-events-v3 order-schema-v1_0 order-schema-v1_1 order-schema-v2_0 ``` ### Schema evolution (preferred) Use schema registries for versioning instead of topic names: ```bash # Topic names remain stable user-events order-events payment-events # Schema versions managed separately # user-events: schema v1, v2, v3 # order-events: schema v1, v2 ``` **Benefits:** - Stable topic names - Backward compatibility support - Consumer flexibility in version handling ## Team and ownership patterns ### Team prefixes Include team or department identifiers: ```bash # Marketing team marketing.campaign-events marketing.user-segments marketing.conversion-tracking # Engineering team engineering.deployment-events engineering.error-logs engineering.performance-metrics # Data team data.pipeline-status data.quality-metrics data.lineage-events ``` ### Application prefixes Identify owning applications: ```bash # E-commerce application ecommerce.user-sessions ecommerce.cart-events ecommerce.checkout-flow # Analytics platform analytics.page-views analytics.user-behavior analytics.conversion-funnel # Notification service notifications.email-queue notifications.sms-queue notifications.push-notifications ``` ## Content and purpose indicators ### Event type indicators Specify the type of events: ```bash # Domain events events.user-registered events.order-completed events.payment-processed # System events system.server-started system.connection-failed system.backup-completed # Business events business.revenue-target-met business.customer-milestone business.campaign-launched ``` ### Data lifecycle indicators Indicate data retention and lifecycle: ```bash # Short-term data temp.session-data cache.user-preferences short.click-streams # Long-term data archive.historical-orders permanent.audit-logs long.customer-profiles # Real-time data realtime.stock-prices live.user-activity streaming.sensor-data ``` ## Examples by use case ### E-commerce platform ```bash # User lifecycle users.registered users.profile-updated users.account-deleted # Product catalog products.created products.updated products.discontinued # Orders and checkout orders.cart-created orders.item-added orders.checkout-started orders.payment-completed orders.order-fulfilled # Analytics and tracking analytics.page-viewed analytics.product-clicked analytics.conversion-tracked ``` ### Financial services ```bash # Account management accounts.opened accounts.closed accounts.balance-updated # Transactions transactions.initiated transactions.authorized transactions.settled transactions.failed # Risk and compliance risk.fraud-detected compliance.kyc-completed audit.access-logged # Market data market.price-updated market.trade-executed market.order-book-changed ``` ### IoT platform ```bash # Device management devices.registered devices.status-updated devices.firmware-upgraded # Sensor data by type sensors.temperature sensors.humidity sensors.pressure sensors.vibration # Alerts and monitoring alerts.threshold-exceeded alerts.device-offline alerts.battery-low # Data processing processed.hourly-aggregates processed.daily-summaries processed.anomaly-detection ``` ## Anti-patterns to avoid ### Poor naming examples **Too generic:** ```bash data # What kind of data? events # What events? messages # What messages? queue # What queue purpose? ``` **Too long:** ```bash user-account-profile-management-service-events-for-customer-lifecycle-tracking ``` **Inconsistent casing:** ```bash UserEvents # Mixed with user-events ORDER_QUEUE # Mixed with order-queue Payment.Events # Mixed with payment.events ``` **Environment mixing:** ```bash prod-user-events-dev # Confusing environment indicators staging.prod.orders # Contradictory environments ``` **Version confusion:** ```bash user-events-old user-events-new user-events-latest user-events-backup ``` ### Common mistakes 1. **No consistent pattern**: Each team uses different conventions 2. **Too many hierarchies**: Complex nested naming that's hard to remember 3. **Abbreviations**: Cryptic shorthand that's not immediately clear 4. **Technical implementation details**: Including partition counts or replication in names 5. **Temporary names**: Using "temp" or "test" in production topic names ## Implementation guidelines ### Documentation Document your naming convention: ```markdown # Topic Naming Convention ## Pattern {domain}.{event-type}.{version} ## Examples - users.registered.v1 - orders.completed.v1 - payments.failed.v2 ## Environments - Development: Use dev cluster - Staging: Use staging cluster - Production: Use prod cluster ## Ownership - Format: {team-name}.{topic-name} - Example: platform.user-events ``` ### Validation tools Create tooling to validate topic names: ```bash #!/bin/bash # validate-topic-name.sh TOPIC_NAME=$1 PATTERN="^[a-z][a-z0-9\-\.]*[a-z0-9]$" if [[ $TOPIC_NAME =~ $PATTERN ]]; then echo "Valid topic name: $TOPIC_NAME" exit 0 else echo "Invalid topic name: $TOPIC_NAME" echo "Must match pattern: $PATTERN" exit 1 fi ``` ### Automation integration Integrate naming validation into CI/CD: ```yaml # Example GitLab CI validate-topic-names: script: - find . -name "topics.yaml" -exec validate-topics.sh {} \; rules: - if: $CI_MERGE_REQUEST_ID ``` ### Migration strategy For existing systems: 1. **Document current state**: Inventory all existing topic names 2. **Define target convention**: Establish new naming standards 3. **Create migration plan**: Plan topic renames and data migration 4. **Gradual migration**: Migrate topics incrementally 5. **Update applications**: Modify producer and consumer applications 6. **Monitor transition**: Ensure no data loss during migration ## Best practices summary ### Recommended approach 1. **Choose one pattern**: Stick to kebab-case, snake_case, or dot.notation consistently 2. **Use meaningful names**: Names should be self-documenting 3. **Include context**: Domain, purpose, and data type should be clear 4. **Separate environments**: Use different clusters rather than name prefixes 5. **Avoid versioning in names**: Use schema registries for version management 6. **Keep reasonable length**: Under 50 characters when possible 7. **Document conventions**: Make naming rules explicit and accessible 8. **Validate consistently**: Use tooling to enforce naming standards ### Organization-specific considerations **Small teams (< 10 people):** ```bash # Simple pattern user-events order-events payment-events ``` **Medium teams (10-50 people):** ```bash # Add domain context users.events orders.events payments.events ``` **Large organizations (50+ people):** ```bash # Include team ownership platform.users.events commerce.orders.events finance.payments.events ``` Changing topic names in production requires careful coordination, so plan naming conventions early and consider the migration cost when evaluating changes. Begin with simple naming conventions and add complexity only as your organization grows—it's easier to add structure than to remove it. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-topics-naming-convention) displays all your topics in an organized view. Use topic search and filtering to validate that your naming conventions enable easy discovery and organization. ## Next steps - [Choose replication and partition count](https://www.conduktor.io/kafka/kafka-topics-choosing-the-replication-factor-and-partitions-count) for new topics - [Change topic configuration](https://www.conduktor.io/kafka/how-to-change-a-kafka-topic-configuration-using-the-cli) using your naming conventions - [Configure log retention](https://www.conduktor.io/kafka/kafka-topic-configuration-log-retention) for your topics Source: https://www.conduktor.io/kafka/kafka-topics-naming-convention --- # Kafka topics *Learn about Kafka topics, partitions and offsets* Topics are the fundamental unit of organization in Apache Kafka. Understanding how topics, partitions, and offsets work together is essential for building effective data streaming applications. **What you'll learn:** - What Kafka topics are and how they organize data - How partitions enable scalability and parallelism - What offsets are and how they track message position - How messages are ordered within partitions ## Kafka topics, partitions and offsets [Beginner](#what-is-a-kafka-topic) [Advanced](https://www.conduktor.io/kafka/kafka-topics-advanced) ### What is a Kafka topic? Similar to how databases have tables to organize and segment datasets, Kafka uses the concept of topics to organize related messages. A topic is identified by its name. For example, we may have a topic called **logs** that may contain log messages from our application, and another topic called **purchases** that may contain purchase data from our application as it happens. ![A Kafka Cluster with 4 Topics shown in a diagram](https://www.conduktor.io/assets/kafka/Apache-Kafka-Cluster-with-4-topics.png) Kafka topics can contain any kind of message in any format, and the sequence of all these messages is called a data stream. > Unlike database tables, Kafka topics are **not query-able**. Instead, we have to create Kafka producers to send data to the topic and Kafka consumers to read the data from the topic in order. By default, data in Kafka topics is deleted after one week (also called the default message retention period), and this value is configurable. This mechanism of deleting old data ensures a Kafka cluster does not run out of disk space by recycling topics over time. ## What are Kafka partitions? Topics are broken down into a number of partitions. A single topic may have more than one partition, it is common to see topics with 100 partitions. The number of partitions of a topic is specified at the time of topic creation. Partitions are numbered starting from `0` to `N-1`, where `N` is the number of partitions. The figure below shows a topic with three partitions, with messages being appended to the end of each one. ![Kafka Topics are broken into partitions for improved fault tolerance. This diagram shows a Kafka Topic with 3 partitions and their respective offsets.](https://www.conduktor.io/assets/kafka/Kafka-Topics-1.png) The offset is an integer value that Kafka adds to each message as it is written into a partition. Each message in a given partition has a unique offset. > Kafka topics are **immutable**: once data is written to a partition, it cannot be changed. ### Why use partitions? Partitions serve two critical purposes: 1. **Scalability**: Data is distributed across multiple brokers, allowing the cluster to handle more data than a single server could 2. **Parallelism**: Multiple consumers can read from different partitions simultaneously, increasing throughput ![Diagram of a topic named orders with three partitions spread across brokers: Partition 0 on Broker 1 is read by Consumer 1, Partition 1 on Broker 2 by Consumer 2, and Partition 2 on Broker 3 by Consumer 3, each consumer reading its own partition in parallel.](https://www.conduktor.io/assets/kafka/diagrams/kafka-topics.svg) ```mermaid graph LR subgraph Topic["Topic: orders"] P0["Partition 0
Broker 1"] P1["Partition 1
Broker 2"] P2["Partition 2
Broker 3"] end C1["Consumer 1"] --> P0 C2["Consumer 2"] --> P1 C3["Consumer 3"] --> P2 ``` ### Kafka topic example ![Apache Kafka has many real world applications. This diagram shows how Apache Kafka can be used for fleet tracking in the transport industry.](https://www.conduktor.io/assets/kafka/Kafka-Topics-2.png) A traffic company wants to track its fleet of trucks. Each truck is fitted with a GPS locator that reports its position to Kafka. We can create a topic named - **trucks\_gps** to which the trucks publish their positions. Each truck may send a message to Kafka every 20 seconds, each message will contain the truck ID and the truck position (latitude and longitude). The topic may be split into a suitable number of partitions, say 10. There may be different consumers of the topic. For example, an application that displays truck locations on a dashboard or another application that sends notifications if an event of interest occurs. ## What are Kafka offsets? Apache Kafka offsets represent the position of a message within a Kafka partition. Offset numbering for every partition starts at `0` and is incremented for each message sent to a specific Kafka partition. This means that Kafka offsets only have a meaning for a specific partition, e.g., offset 3 in partition 0 doesn't represent the same data as offset 3 in partition 1. > **Kafka offset ordering**: if a topic has more than one partition, Kafka guarantees the order of messages within a partition, but there is no ordering of messages across partitions. Even though we know that messages in Kafka topics are deleted over time (as seen above), the offsets are not re-used. They continually are incremented in a never-ending sequence. ### Message ordering Here is how message ordering works: | Scenario | Ordering guarantee | |----------|-------------------| | Messages with same key | Ordered (same partition) | | Messages without key | Not ordered (round-robin) | | Messages across partitions | Not ordered | Once a topic is created, you can increase the partition count but cannot decrease it. Plan your partition count carefully, considering future growth. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-kafka-topics) provides a visual interface for creating and managing topics. Browse topic messages, view partition distribution, and monitor topic metrics in real-time. ## Next steps - [Learn how producers write to topics](https://www.conduktor.io/kafka/kafka-producers) to understand how data enters partitions - [Learn about Kafka consumers](https://www.conduktor.io/kafka/kafka-consumers) to understand how data is read from topics - [Explore topic configuration](https://www.conduktor.io/kafka/kafka-topics-advanced) for advanced settings like retention and compaction Source: https://www.conduktor.io/kafka/kafka-topics --- # Other advanced Kafka producer configurations *Learn advanced producer configuration options* Beyond basic producer settings, advanced configurations provide fine-grained control over producer behavior, performance, and reliability for production deployments. **What you'll learn:** - Buffer memory and timeout configurations - Connection and network settings - Monitoring and metrics configuration - Environment-specific tuning templates ## Configuration categories ![Advanced producer configuration categories grouped into memory management, timeouts, network, and reliability settings](https://www.conduktor.io/assets/kafka/diagrams/other-advanced-kafka-producer-configurations.svg) ```mermaid flowchart TD subgraph Memory["Memory Management"] BM["buffer.memory"] MBL["max.block.ms"] end subgraph Timeouts["Timeouts"] RT["request.timeout.ms"] DT["delivery.timeout.ms"] MA["metadata.max.age.ms"] end subgraph Network["Network"] CI["connections.max.idle.ms"] IF["max.in.flight.requests"] RB["reconnect.backoff.ms"] end subgraph Reliability["Reliability"] ID["enable.idempotence"] TID["transactional.id"] end ``` ## Buffer memory configuration ### buffer.memory Total memory allocated for producer buffering: ```properties # Default: 33554432 (32MB) buffer.memory=67108864 # 64MB total buffer ``` **Impact:** - **Too small**: Producer blocks frequently when buffer fills - **Too large**: Excessive memory usage, potential GC pressure - **Optimal size**: Balance between memory efficiency and producer throughput ### max.block.ms Time to block when buffer is full: ```properties # Default: 60000 (1 minute) max.block.ms=30000 # Block for 30 seconds maximum ``` ### Memory calculation ``` Required buffer = batch.size × active_partitions × safety_factor Example: - 100 active partitions - 32KB batch size - 2x safety factor = 32KB × 100 × 2 = 6.4MB minimum ``` ## Request timeout settings | Setting | Default | Description | |---------|---------|-------------| | `request.timeout.ms` | 30s | Timeout for individual requests | | `delivery.timeout.ms` | 2min | Overall timeout including retries | | `metadata.max.age.ms` | 5min | How long to cache topic metadata | ### Timeout relationship If `delivery.timeout.ms` is shorter than the worst-case retry sequence, the producer throws a `ConfigException` at startup: ```properties delivery.timeout.ms >= request.timeout.ms + (retries × retry.backoff.ms) ``` ## Connection configurations ### max.in.flight.requests.per.connection Maximum unacknowledged requests per connection: ```properties # Default: 5 max.in.flight.requests.per.connection=5 ``` **Impact on ordering:** | Setting | Ordering | Throughput | |---------|----------|------------| | 1 | Strict | Lower | | 5 (with idempotence) | Maintained | Higher | | > 5 (without idempotence) | May reorder | Highest | ## Configuration templates ### High-throughput configuration ```properties # Optimize for maximum throughput batch.size=65536 linger.ms=20 buffer.memory=134217728 compression.type=lz4 max.in.flight.requests.per.connection=5 request.timeout.ms=30000 delivery.timeout.ms=120000 enable.idempotence=true ``` ### Low-latency configuration ```properties # Optimize for minimum latency batch.size=16384 linger.ms=0 buffer.memory=33554432 compression.type=none max.in.flight.requests.per.connection=1 request.timeout.ms=10000 delivery.timeout.ms=30000 ``` ### High-reliability configuration ```properties # Optimize for maximum reliability acks=all retries=2147483647 enable.idempotence=true max.in.flight.requests.per.connection=5 delivery.timeout.ms=600000 request.timeout.ms=30000 buffer.memory=67108864 batch.size=32768 ``` ### Memory-constrained configuration ```properties # Minimize memory usage buffer.memory=16777216 # 16MB batch.size=8192 # 8KB max.request.size=1048576 # 1MB connections.max.idle.ms=60000 ``` ## Monitor your configuration ### Essential metrics to track | Metric | Description | Alert threshold | |--------|-------------|-----------------| | `buffer-available-bytes` | Available buffer space | < 10% | | `request-latency-avg` | Average request latency | > baseline | | `record-error-rate` | Rate of failed records | > 0 | | `record-retry-rate` | Rate of retried records | > baseline | ### Client identification ```properties # Always set for easier troubleshooting client.id=order-service-producer-01 ``` ## Common configuration mistakes | Mistake | Problem | Solution | |---------|---------|----------| | Insufficient `buffer.memory` | Frequent blocking | Increase buffer or reduce batch size | | Too high `linger.ms` | Unnecessary latency | Reduce to 0-20ms | | Mismatched timeouts | `delivery.timeout` too small | Ensure delivery > request + retries | | Missing `client.id` | Hard to troubleshoot | Always set meaningful client ID | | Wrong `max.in.flight` | Ordering issues | Use 5 with idempotence | Some configurations are interdependent. Enabling `enable.idempotence=true` automatically sets `retries=Integer.MAX_VALUE`, `max.in.flight.requests.per.connection=5`, and `acks=all`. Begin with default configurations and adjust one parameter at a time based on monitoring data—most applications work well with minimal configuration changes. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-other-advanced-kafka-producer-configurations) helps you test different producer configurations and observe their effects. Monitor producer metrics to validate your configuration changes work as expected. ## Next steps - [Move on to advanced consumers](https://www.conduktor.io/kafka/kafka-consumers-advanced) to tune the read side of your pipeline - [Configure producer batching](https://www.conduktor.io/kafka/kafka-producer-batching) for throughput optimization - [Enable compression](https://www.conduktor.io/kafka/kafka-message-compression) for network efficiency Source: https://www.conduktor.io/kafka/other-advanced-kafka-producer-configurations --- # Producer default partitioner and sticky partitioner *Learn how Kafka partitioners affect message distribution* Kafka producer partitioners determine which partition receives each message, affecting load distribution, ordering guarantees, and overall system performance. Understanding partitioning strategies helps you optimize for your specific use case. **What you'll learn:** - How partitioners route messages to partitions - The difference between sticky and round-robin partitioning - When to use key-based vs keyless partitioning - How to implement custom partitioners ## How partitioning works When a producer sends a message, the partitioner decides which partition receives it based on: 1. **Message key**: If present, used to determine partition assignment 2. **Partition specification**: Explicit partition number in ProducerRecord 3. **Partitioner logic**: Default or custom partitioning algorithm ![Partition routing decision: keyed messages use hash(key) % partitions for guaranteed ordering, keyless messages use the sticky partitioner on Kafka 2.4+ or round-robin on older versions](https://www.conduktor.io/assets/kafka/diagrams/producer-default-partitioner-and-sticky-partitioner--1.svg) ```mermaid flowchart TD M["Message"] --> Check{"Has key?"} Check -->|"Yes"| Hash["hash(key) % partitions"] Check -->|"No"| Sticky["Sticky partitioner
(Kafka 2.4+)"] Check -->|"No"| RR["Round-robin
(Kafka < 2.4)"] Hash --> P0["Same key → Same partition
(ordering guaranteed)"] Sticky --> P1["Fill current batch
then switch partition"] RR --> P2["Cycle through all partitions"] ``` ![Kafka Producer Partitioning](https://www.conduktor.io/assets/kafka/Sticky-Partitioner-vs-Round-Robin--2-.png) > **Partition assignment impact** > Partitioning affects message ordering, consumer distribution, load balancing, and data locality. Choose your partitioning strategy carefully based on your use case requirements. ## Default partitioner behavior ### With message keys When messages have keys, the default partitioner uses a hash-based approach: ```java partition = hash(key) % number_of_partitions ``` > **Hot keys** > If one key appears far more often than others (e.g., a single tenant generating most traffic), all those messages land on the same partition. This creates a "hot partition" that overwhelms one broker and one consumer. Monitor per-partition message rates and consider key redesign or a custom partitioner if distribution is uneven. **Characteristics:** - Same key always goes to same partition (within same topic configuration) - Provides ordering guarantees per key - Distributes load based on key distribution - Changes when partition count changes ### Without message keys For messages without keys, behavior depends on Kafka version: | Kafka version | Partitioner | Behavior | |---------------|-------------|----------| | < 2.4 | Round-robin | Cycles through partitions sequentially | | >= 2.4 | Sticky | Sticks to one partition until batch is full | ## Sticky partitioner The sticky partitioner, introduced in Kafka 2.4, improves performance for messages without keys. ### How sticky partitioner works 1. **Partition selection**: Choose random available partition 2. **Batch filling**: Send all messages to chosen partition until batch fills 3. **Partition switching**: Switch to different partition for next batch 4. **Repeat process**: Continue cycle for optimal batching ### Benefits of sticky partitioner | Benefit | Description | |---------|-------------| | **Improved throughput** | Better batch utilization (more messages per batch) | | **Fewer network requests** | Reduced overhead from larger batches | | **Better compression** | Larger batches compress more efficiently | | **Even distribution** | Over time, messages distribute evenly across partitions | ### Configuration Sticky partitioner is enabled by default in Kafka 2.4+: ```properties # Explicitly configure (usually not needed) partitioner.class=org.apache.kafka.clients.producer.internals.DefaultPartitioner ``` ## Round-robin partitioner The original default partitioner (pre-2.4) that cycles through partitions. ```properties partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner ``` > **Round-robin performance impact** > Round-robin partitioning creates smaller batches and reduces throughput compared to sticky partitioning for keyless messages. ## Partitioning strategies decision guide ![Decision tree for choosing a partition strategy based on ordering and custom routing needs: message keys, single partition, custom partitioner, or sticky partitioner, all leading to monitoring partition distribution](https://www.conduktor.io/assets/kafka/diagrams/producer-default-partitioner-and-sticky-partitioner--2.svg) ```mermaid flowchart TD Start["Choosing partition strategy"] --> Q1{"Need ordering
guarantees?"} Q1 -->|"Yes, per entity"| Key["Use message keys
(e.g., user_id, order_id)"] Q1 -->|"Yes, global"| Single["Use single partition
(limits throughput)"] Q1 -->|"No"| Q2{"Need custom
routing?"} Q2 -->|"Yes"| Custom["Implement custom partitioner"] Q2 -->|"No"| Sticky["Use sticky partitioner
(default)"] Key --> Monitor["Monitor partition distribution"] Custom --> Monitor Sticky --> Monitor ``` ## Custom partitioner implementation Create custom partitioners for specific business requirements: ```java public class CustomPartitioner implements Partitioner { @Override public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) { List partitions = cluster.partitionsForTopic(topic); int partitionCount = partitions.size(); if (key == null) { return ThreadLocalRandom.current().nextInt(partitionCount); } // Custom key-based partitioning logic String keyString = key.toString(); if (keyString.startsWith("priority-")) { return 0; // Route priority messages to partition 0 } // Use hash for other messages return Utils.toPositive(Utils.murmur2(keyBytes)) % partitionCount; } @Override public void configure(Map configs) { } @Override public void close() { } } ``` Register your custom partitioner: ```properties partitioner.class=com.example.CustomPartitioner ``` ## Partition count considerations > **Partition count changes** > Changing partition count breaks key-to-partition mapping and can disrupt ordering guarantees. Plan partition counts carefully and avoid frequent changes. ### Impact of adding partitions ``` Original: hash(key) % 3 partitions → key "user123" → partition 2 After adding: hash(key) % 6 partitions → key "user123" → partition 5 ``` **Consequences:** - Same key may go to different partition - Breaks ordering guarantees temporarily - May create temporary hotspots ## Best practices | Scenario | Recommendation | |----------|----------------| | Need ordering per entity | Use message keys | | Maximum throughput | Use sticky partitioner (no keys) | | Custom routing logic | Implement custom partitioner | | Even distribution | Monitor for hot keys | Always test custom partitioners with realistic data distributions and load patterns. Partition assignment affects performance significantly. > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-producer-default-partitioner-and-sticky-partitioner) displays partition distribution and message counts per partition. Monitor how your partitioning strategy affects load balance across your topic partitions. ## Next steps - [Tune advanced producer configs](https://www.conduktor.io/kafka/other-advanced-kafka-producer-configurations) for buffers, timeouts, and backpressure - [Configure producer batching](https://www.conduktor.io/kafka/kafka-producer-batching) to optimize throughput - [Understand message keys](https://www.conduktor.io/kafka/kafka-producers) for ordering Source: https://www.conduktor.io/kafka/producer-default-partitioner-and-sticky-partitioner --- # Set up Kafka for development *Get your first Kafka cluster running* This section guides you through setting up Apache Kafka for development and learning. Choose from multiple installation options based on your operating system and preferences. **What you'll learn:** - How to set up a Kafka cluster for development - Different installation methods and their trade-offs - How to verify your Kafka installation **Prerequisites:** - Basic command line knowledge - Java 11+ installed (for local installations) - Docker installed (for container-based setup) ## Choose your setup method ![Kafka setup decision tree: choose Docker Compose if you have Docker, otherwise pick by OS — Homebrew or native on Mac, native on Linux, WSL2 plus native on Windows](https://www.conduktor.io/assets/kafka/diagrams/starting-kafka.svg) ```mermaid flowchart TD A[Want to run Kafka?] --> B{Have Docker?} B -->|Yes| C[Docker Compose
Recommended] B -->|No| D{Operating System?} D -->|Mac| E{Prefer Homebrew?} D -->|Linux| F[Native install] D -->|Windows| G[WSL2 + Native] E -->|Yes| H[Homebrew install] E -->|No| I[Native install] C --> J[Ready in 5 min] F --> J G --> J H --> J I --> J ``` ## Installation options | Method | Best for | Time | Requirements | |--------|----------|------|--------------| | [Docker Compose](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) | Cross-platform, version switching | 5 min | Docker | | [Mac native](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac) | macOS development | 15 min | Java 11 | | [Mac Homebrew](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-with-homebrew) | Quick Mac setup | 10 min | Homebrew | | [Linux native](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux) | Linux development | 15 min | Java 11 | | [Windows WSL2](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows) | Windows development | 20 min | WSL2, Java 11 | > **Which method should I choose?** > For most users, Docker Compose is the best choice. It works on all operating systems, makes version switching straightforward, and keeps your system clean. Use native installation if you need direct access to Kafka binaries or prefer not to use Docker. ## ZooKeeper vs KRaft mode Kafka traditionally requires ZooKeeper for cluster coordination. Starting with Kafka 2.8, KRaft mode allows Kafka to run without ZooKeeper. | Mode | Status | Recommendation | |------|--------|----------------| | ZooKeeper | Stable, production-ready | Use for learning and production | | KRaft | Production-ready (Kafka 3.3+) | Use for simplified deployments | KRaft installation guides: - [Mac KRaft](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-mac-without-zookeeper-kraft-mode) - [Linux KRaft](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-linux-without-zookeeper-kraft-mode) - [Windows KRaft](https://www.conduktor.io/kafka/how-to-install-apache-kafka-on-windows-without-zookeeper-kraft-mode) > **See it in practice with Conduktor** > [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-starting-kafka) provides a visual interface for connecting to and managing your Kafka clusters once they're running. ## Next steps - [Conduktor Console](https://www.conduktor.io/kafka/how-to-start-kafka-with-conduktor) for a hosted cluster with no local setup - [Docker setup](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) for the fastest cross-platform local cluster - [CLI tutorials](https://www.conduktor.io/kafka/kafka-cli-tutorial) to interact with your cluster once it's running Source: https://www.conduktor.io/kafka/starting-kafka --- # What is Apache Kafka? *Learn about Apache Kafka and its ecosystem.* When it comes to data event streaming, Apache Kafka is the de facto standard. It is an open-source distributed system consisting of servers and clients. Apache Kafka is used primarily to build real-time data streaming pipelines. Apache Kafka is used by thousands of organizations for data pipelines, streaming analytics and data integration. For a visual overview of how the pieces fit together, see the [Kafka architecture diagram](https://www.conduktor.io/glossary/kafka-architecture-diagram). In this introductory section, you will learn about: 1. [what Apache Kafka is and where it came from](#data-integration-challenges) 1. [what the main components are](#definition-of-core-apache-kafka-concepts) 1. [what the Apache Kafka ecosystem is](#kafka-ecosystem) By the end, you will confidently understand Apache Kafka and its place in the data streaming world. Looking for a start on Apache Kafka? The Conduktor platform is the ultimate way to start learning Apache Kafka. [Try it now for free](https://www.conduktor.io/get-started) and find out how easy Kafka can be. ## Data integration challenges ### Context A typical organization has multiple sources of data with disparate data formats. Data integration involves combining data from these multiple sources into one unified view of their business. A typical business collects data through a variety of applications, e.g., accounting, billing, CRM, websites, etc. Each of these applications have their own processes for data input and update. In order to get a unified view of their business, engineers have to develop bespoke integrations between these different applications. These direct integrations can result in a complicated solution as shown below. ![Apache Kafka helps to solve many of the challenges associated with the integration of data from multiple different systems. This diagram shows how complex the flow of data can be when systems are not decoupled.](https://www.conduktor.io/assets/kafka/What-is-Apache-Kafka-Part-1---Data-Integration-Challenges.png) Each integration comes with difficulties around * **Protocol**: how the data is transported (TCP, HTTP, REST, FTP, JDBC…) * **Data format**: how the data is parsed (Binary, CSV, JSON, Avro…) * **Data schema & evolution**: how the data is shaped and may change #### Decoupling different data systems Apache Kafka allows us to decouple data streams and systems. With Apache Kafka as a data integration layer, data sources will publish their data to Apache Kafka and the target systems will source their data from Apache Kafka. This decouples source data streams and target systems allowing for a simplified data integration solution, as you can see in the diagram below. ![Apache Kafka provides an effective way for organizations to overcome data integration challenges by making it easy to decouple different systems](https://www.conduktor.io/assets/kafka/What-is-Apache-Kafka-Part-1---Decoupling-Different-Data-Systems.png) ### What is a data stream in Apache Kafka? A data stream is typically thought of as a potentially unbounded sequence of data. The name streaming is used because we are interested in the data being accessible as soon as it is produced. Each of the applications in an organization where data is created is a potential data stream creator. Data created as part of data streams are typically small. The data throughput to data streams is highly variable: some streams will receive tens of thousands of records per second, and some will receive one or two records per hour. Apache Kafka is used to store these data streams (also called topics), which then allows systems to perform stream processing - an act of performing continual calculations on a potentially endless and constantly evolving source of data. Once the stream is processed and stored in Apache Kafka, it may be transferred to another system, e.g., a database. ![Apache Kafka allows for data from various different business applications and sources to flow into a real-time data pipeline that can process millions of events per second.](https://www.conduktor.io/assets/kafka/What-is-Apache-Kafka-Part-1---Use-Cases-and-Applications.png) #### Examples of data streams The following are examples of some of data streams in real-world that companies process * **Log Analysis**. Modern applications include tens to thousands of microservices - all of which constantly produce logs. These logs are full of information that can be mined for business intelligence, failure prediction, and debugging. The challenge then is how to process these large volumes of log data being produced in one place. Companies push log data into a data stream to perform stream processing. * **Web Analytics**. Another common use for streaming data is web analytics. Modern web applications measure almost every user activity on their site, e.g., button clicks, page views. These actions add up fast. Stream processing allows companies to process data as it is generated and not hours later. ### Why should a company use Apache Kafka? As soon as a company has real-time data streaming needs, a streaming platform has to be put in place. Apache Kafka is one of the most popular data streaming processing platforms in the industry today, being used by more than 80% of the Fortune 100 companies. Kafka provides a simple message queue interface on top of its append-only log-structured storage medium. It stores a log of events. Data is distributed to multiple nodes. Kafka is highly scalable and fault-tolerant to node loss. Kafka has been deployed in sizes ranging from just one node to thousands of nodes. It is used extensively in production workloads in companies such as Netflix, Apple, Uber, Airbnb, in addition to LinkedIn. The creators of Kafka left LinkedIn to form their own company called Confluent to focus full-time on Kafka and its ecosystem. Apache Kafka is now an open-source project maintained by Confluent. ### Apache Kafka history Kafka was created at LinkedIn to service internal stream processing requirements that could not be met with traditional message queueing systems. Its first version was released in January 2011. Kafka quickly gained popularity and since then became one of the most popular projects of the Apache Foundation. The project is now mainly maintained by Confluent, with help from other companies such as IBM, Yelp, Netflix and so on. ### What are the use cases of Apache Kafka? The use cases of Apache Kafka are many. These include stream processing for different business applications. Apache Kafka makes up the storage mechanism for some of the prominent stream processing frameworks, e.g., Apache Flink, Samza. * Messaging systems * Activity Tracking * Gather metrics from many different locations, for example, IoT devices * Application logs analysis * De-coupling of system dependencies * Integration with Big Data technologies like Spark, Flink, Storm, Hadoop. * Event-sourcing store You can find a list of use cases at [https://kafka.apache.org/uses](https://kafka.apache.org/uses) ## Where is Apache Kafka not a great fit? Apache Kafka is a great fit for the use cases outlined above, but there are a few use cases when using Apache Kafka is either not possible or not recommended: * **Proxying millions of clients** for mobile apps or IoT: the Kafka protocol is not made for that, but some proxies exist to bridge the gap. * **A database with indexes:** Kafka is an event streaming log with no analytical capability built in and no complex query model. * **An embedded real-time technology for IoT:** there are lower level and lighter alternatives to perform these use cases on embedded systems. * **Work queues:** Kafka is made of topics, not queues (unlike RabbitMQ, ActiveMQ, SQS). Queues are meant to scale to millions of consumers and to delete messages once processed. In Kafka data is not deleted once processed and consumers cannot scale beyond the number of partitions in a topic. * **Kafka as a blockchain**: Kafka topics present some characteristics of a blockchain, where data is appended in a log, and Kafka topics can be immutable, but lack some key properties of blockchains such as the cryptographic verification of the data, as well as full history preservation. ### How is Kafka concretely being used within the industry? Apache Kafka is widely used in the industry. Some of the use cases are highlighted below. * **Uber** uses Kafka extensively in their real-time pricing pipeline. Kafka is the backbone through which a significant proportion of the events are communicated to the different stream processing calculations. The speed and flexibility of Kafka allows Uber to adjust their pricing models to the constantly evolving events in the real world (number of available drivers and their position, users and their position, weather event, other events), and bill users the right amount to manage offer and demand. * **Netflix** has integrated Kafka as the core component of its data platform. They refer to it internally as their Keystone data pipeline. As part of Netflix's Keystone, Kafka handles billions of events a day. Just to give an idea about the huge amount of data that Kafka can handle, Netflix sends about 5 hundred billion events and 1.3 petabytes of data per day into Kafka. Unknown to many, Kafka is at the core of lots of the services we enjoy on a daily basis from some of the world's largest tech companies such as Uber, Netflix, Airbnb, LinkedIn, Apple & Walmart. ## Definition of core Apache Kafka concepts Now that we've learned about Apache Kafka at a high level, let's start learning how to use the tool. In this part, we will cover the basics of Kafka topics, producers, and consumers. ![Apache Kafka Components. An overview of the relationship between Kafka clusters, Kafka topics, Kafka producers and Kafka Consumers.](https://www.conduktor.io/assets/kafka/Producer---Topic---Consumer.png) ### Kafka topics Kafka topics organize related events. For example, we may have a topic called **logs,** which contains logs from an application. Topics are roughly analogous to SQL tables. However, unlike SQL tables, Kafka topics are not queryable. Instead, we have to create Kafka producers and consumers to utilize the data. The data in the topics are stored in the key-value form in binary format. Read more in [Kafka Topics, Partitions & Offsets page](https://www.conduktor.io/kafka/kafka-topics). ### Kafka producers Once a topic is created in Kafka, the next step is to send data into the topic. Applications that send data into a topic are known as Kafka producers. There are many ways to produce events to Kafka, but applications typically integrate with Kafka client libraries in languages like Java, Python, Go, as well as many other languages. Note that Kafka producers are deployed outside Kafka and only interact with Apache Kafka by sending data directly into the Kafka topics. Read more in [Kafka producers](https://www.conduktor.io/kafka/kafka-producers). ### Kafka consumers Once a topic has been created and data produced into the topic, we can have applications that make use of the data stream. Applications that pull event data from one or more Kafka topics are known as Kafka consumers. There are many ways to consume events from Kafka, but applications typically integrate with Kafka client libraries in languages like Java, Python, Go, as well as many other languages. By default consumers only consume data that was produced after the consumer first connected to the topic. Note that Kafka consumers are deployed outside Kafka and only interact with Apache Kafka by reading data directly from Kafka topics. Read more in [Kafka consumers](https://www.conduktor.io/kafka/kafka-consumers). ## Kafka ecosystem A number of additional tools and libraries have been developed for Kafka over the years to expand its functionality. The sections below cover some of the most popular parts of the wider Kafka ecosystem. ### Kafka Streams Once we have produced data from external systems into Kafka, we may want to process them using stream processing applications. Stream processing applications make use of streaming data stores like Apache Kafka to provide real-time analytics. For example, let's assume we are having a Kafka topic named `twitter_tweets` that is a data streaming of all tweets on Twitter. From this topic, we may want to: * Filter only tweets that have over `10` likes or replies, to capture important tweets * Count the number of tweets received for each hashtag every `1` minute * Combine the two to get trending topics and hashtags in real-time! ![An illustrated example of how Apache Kafka and Kafka Streams can support stream processing applications for real-time analytics and other use cases.](https://www.conduktor.io/assets/kafka/Apache-Kafka---Kafka-Streams---Stream-Processing-Example.png) In order to perform topic-level transformation within Apache Kafka, we can use streaming libraries that are meant for this use case instead of writing very complicated producer & consumer code. In that case, we can use the Kafka Streams library, which is a stream processing framework that is released alongside Apache Kafka. Alternatives you may have heard of for Kafka Streams are Apache Spark, or Apache Flink. ### Kafka connect In order to get data into Apache Kafka, we have seen that we need to use Kafka producers. Over time, it has been noticed that many companies shared the same data source types (databases, systems, etc...) and so writing open-source standardized code could be helpful for the greater good. The same thinking goes for Kafka Consumers. Kafka Connect is a tool that allows us to integrate popular systems with Kafka. It allows us to re-use existing components to source data into Kafka and sink data out from Kafka into other data stores. Examples of popular Kafka Connectors include: * **Kafka Connect Source Connectors (producers)**: Databases (through the Debezium connector), JDBC, Couchbase, GoldenGate, SAP HANA, Blockchain, Cassandra, DynamoDB, FTP, IOT, MongoDB, MQTT, RethinkDB, Salesforce, Solr, SQS, Twitter, etc… * **Kafka Connect Sink Connectors (consumers):** S3, ElasticSearch, HDFS, JDBC, SAP HANA, DocumentDB, Cassandra, DynamoDB, HBase, MongoDB, Redis, Solr, Splunk, Twitter ![An overview of how Apache Kafka with Kafka Connect helps to stream data between sources and sinks.](https://www.conduktor.io/assets/kafka/Source-Connector----Sink-Connector.png) ### Schema registry The schema registry helps register data schemas in Apache Kafka and ensure that producers and consumers will be compatible with each other while evolving. It supports the Apache Avro, Protobuf and JSON-schema data formats. ### Data schemas **Data schemas** define for your data the expected fields, their names, and value types Without a schema registry, producers and consumers are at the risk of breaking when the data schema changes. ### KsqlDB ksqlDB is a stream processing database that provides a SQL-like interface to transform Kafka topics and perform common database-like operations such as joins, aggregates, filtering, and other forms of data manipulation on streaming data. Behind the scenes, the ksqlDB webserver translates the SQL commands into a series of Kafka Streams applications. ![An illustrated overview of how ksqlDB works with Apache Kafka](https://www.conduktor.io/assets/kafka/What-is-Apache-Kafka-Part-3---ksqlDB.png) ## Next steps - [Master the core concepts](https://www.conduktor.io/kafka/kafka-fundamentals) to build a solid foundation before diving deeper - [Understand Kafka topics](https://www.conduktor.io/kafka/kafka-topics) to learn how data is organized into partitions - [Learn how producers write data](https://www.conduktor.io/kafka/kafka-producers) to see how events get into Kafka Source: https://www.conduktor.io/kafka/what-is-apache-kafka --- # ZooKeeper with Kafka *Learn about ZooKeeper's role in Kafka in five minutes* ZooKeeper has been a critical component of Kafka clusters, handling metadata management and cluster coordination. While ZooKeeper is being phased out in favor of KRaft mode, understanding its role is important for managing existing deployments. **What you'll learn:** - What ZooKeeper does in a Kafka cluster - Why ZooKeeper is being removed from Kafka - Best practices for ZooKeeper with current Kafka versions > **ZooKeeper is being eliminated from Kafka** > 1. Kafka 0.x, 1.x & 2.x must use ZooKeeper > 2. Kafka 3.x can work without ZooKeeper (KRaft mode) and is production ready as of 3.3 > 3. Kafka 4.x will not have ZooKeeper > For new deployments, consider using [KRaft mode](https://www.conduktor.io/kafka/kafka-kraft-mode) instead. ## What is ZooKeeper in Kafka and what does it do? How do the Kafka brokers and clients keep track of all the Kafka brokers if there is more than one? The Kafka team decided to use ZooKeeper for this purpose. ZooKeeper is used for metadata management in the Kafka world. For example: - ZooKeeper keeps track of which brokers are part of the Kafka cluster - ZooKeeper is used by Kafka brokers to determine which broker is the leader of a given partition and topic and perform leader elections - ZooKeeper stores configurations for topics and permissions - ZooKeeper sends notifications to Kafka in case of changes (e.g. new topic, broker dies, broker comes up, delete topics, etc.) > **Consumer offsets**: ZooKeeper does NOT store consumer offsets with Kafka clients >= v0.10. Offsets are stored in the internal `__consumer_offsets` topic. ### ZooKeeper ensemble A ZooKeeper cluster is called an *ensemble*. It is recommended to operate the ensemble with an odd number of servers, e.g., 3, 5, 7, as a strict majority of ensemble members (a quorum) has to be working in order for ZooKeeper to respond to requests. ZooKeeper has a leader to handle writes, the rest of the servers are followers to handle reads. ![alt text](https://www.conduktor.io/assets/kafka/Zookeeper-with-Kafka-1.png) ### Size your ZooKeeper ensemble | Servers | Quorum needed | Failures tolerated | |---------|---------------|-------------------| | 1 | 1 | 0 | | 3 | 2 | 1 | | 5 | 3 | 2 | | 7 | 4 | 3 | ## Should you use ZooKeeper with Kafka brokers? For Kafka versions prior to 3.3, you have to use ZooKeeper in your production deployments. For Kafka 3.3 and later, KRaft mode is production ready and recommended for new deployments. ## Should you use ZooKeeper with Kafka clients? Over time, the Kafka clients and CLI have been migrated to use the brokers as a connection endpoint instead of ZooKeeper. This means that: - Since Kafka 0.10, consumers store offset in Kafka and should not connect to ZooKeeper as the option is deprecated - Since Kafka 2.2, the `kafka-topics.sh` CLI command references Kafka brokers and not ZooKeeper for topic management (creation, deletion, etc.) and the ZooKeeper CLI argument is deprecated. - All of the APIs and commands that were previously using ZooKeeper are migrated to use Kafka instead, so that when clusters are migrated to be without ZooKeeper, the change is invisible to clients. - ZooKeeper is also less secure than Kafka, and therefore ZooKeeper ports should only be opened to allow traffic from Kafka brokers, and not Kafka clients > **Never connect clients to ZooKeeper** > To be a great modern-day Kafka developer, never use ZooKeeper as a configuration in your Kafka clients, and other programs that connect to Kafka. Always use the bootstrap server configuration pointing to Kafka brokers. ```properties # Correct: Connect to Kafka brokers bootstrap.servers=broker1:9092,broker2:9092 # Wrong: Never use ZooKeeper connection for clients # zookeeper.connect=zk1:2181,zk2:2181 # DEPRECATED ``` ## Next steps - [Learn about KRaft mode](https://www.conduktor.io/kafka/kafka-kraft-mode) to see the future of Kafka without ZooKeeper - [Understand topic replication](https://www.conduktor.io/kafka/kafka-topic-replication) to see how brokers coordinate - [Set up Kafka with Docker](https://www.conduktor.io/kafka/how-to-start-kafka-using-docker) for development Source: https://www.conduktor.io/kafka/zookeeper-with-kafka --- # Kafka ACLs Access Control Lists are how Kafka decides who's allowed to do what. They work, they're native, and they scale badly. Here's how to configure them, the defaults that catch most teams out, and when it's time to layer RBAC on top. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-acl) [Read the Security Guide →](https://www.conduktor.io/kafka-security) ## What a Kafka ACL actually is An ACL (Access Control List) in Kafka is an allow rule. Each entry maps a principal (like `User:alice`) to an operation (`Read`, `Write`, `Create`, `Describe`, `Alter`) on a specific resource (`Topic`, `ConsumerGroup`, `Cluster`, `TransactionalId`). If a request doesn't match at least one allow rule, the broker denies it. ACLs live inside the Kafka cluster itself. They're stored in KRaft (or ZooKeeper on older deployments) and enforced by the broker on every request. ## Turning ACLs on ACLs are off by default. Without them, every authenticated principal can do anything. Turn them on in `broker.properties` before connecting any clients you actually care about: ```properties # broker.properties authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer super.users=User:admin allow.everyone.if.no.acl.found=false ``` The most common mistake here is leaving `allow.everyone.if.no.acl.found=true`. That flag makes Kafka fall back to "allow" when no ACL exists, which defeats the whole point. Set it to `false` so the broker denies any request that isn't explicitly permitted. ## Adding ACLs with the CLI Kafka ships with `kafka-acls.sh` for adding, listing, and removing rules: ```bash # Grant alice read access to a topic bin/kafka-acls.sh --bootstrap-server localhost:9092 \ --add --allow-principal User:alice \ --operation Read --topic customer-events # List all ACLs on a cluster bin/kafka-acls.sh --bootstrap-server localhost:9092 --list # Remove an ACL bin/kafka-acls.sh --bootstrap-server localhost:9092 \ --remove --allow-principal User:alice \ --operation Read --topic customer-events ``` Each grant is per-principal, per-operation, per-resource. No groups, no role inheritance, no bulk patterns beyond wildcard topic matching. ## Where ACLs stop working For small clusters with a handful of users, ACLs are fine. Past that, four things start to hurt: - **No groups.** Every ACL is per-principal. Onboarding ten new engineers means adding ten sets of ACLs one at a time. - **No role inheritance.** A "developer" role doesn't exist natively, so you reproduce the same permissions for every developer principal. - **No audit trail.** The broker doesn't log who added which ACL when, so when an auditor asks about change history, you don't have one. - **No cross-cluster view.** Listing ACLs across five clusters means running `kafka-acls.sh --list` five times and diffing the output. Most teams start with raw ACLs, scale past the point where they're manageable, and end up layering RBAC on top. The ACLs stay. They're still the enforcement layer at the broker. Something else just manages them. ## Managing Kafka ACLs at scale The four limits above all start to matter at roughly the same scale. Most teams don't notice how fast the entries multiply: 20 teams, 500 topics, three permissions per topic per team is **30,000 ACL entries** before you count service accounts, consumer groups, or non-prod environments. Each one is a separate ACL binding to create, review, and remove. Each cluster gets its own copy. In practice, teams hit three failure modes: 1. **The CLI script that becomes a load-bearing artefact.** Someone writes a bash wrapper around `kafka-acls.sh` that iterates over a YAML file of grants. It works. Then it lives in a Git repo nobody owns, runs from one engineer's laptop, and drifts from production whenever a manual ACL is added during an incident. 2. **The audit question with no answer.** When a compliance reviewer asks "who has produced to `payments-events` in the last 90 days, and who granted them access?", `kafka-acls.sh --list` answers half the first part and none of the second. Broker logs are not an audit trail. 3. **Off-boarding, principal by principal.** A developer leaves the team. Revoking their access means finding every principal they own, across every cluster, and running a `--remove` for each. Until someone finishes that sweep, their credentials still authorize requests at the broker. ### What scale looks like The fix has the same shape regardless of which platform you use to deliver it. Grants are expressed against **groups or roles**, not individual principals. Changes go through a **reviewable, audited interface**, not ad-hoc CLI on a laptop. And the same definition applies **across every cluster**, not one config per environment. Conduktor delivers this as a layer on top of native Kafka ACLs — the broker still enforces ACLs the way it always has; Conduktor manages them: ```yaml # payments-developers.yaml — grants the "payments-developers" group # read on any topic with the "payments-" prefix, across every connected # cluster, with an audit entry. apiVersion: v2 kind: Group metadata: name: payments-developers spec: displayName: Payments Developers externalGroups: - payments-developers permissions: - resourceType: TOPIC cluster: "*" name: payments- patternType: PREFIXED permissions: - topicConsume ``` Applied through the [Conduktor CLI](https://github.com/conduktor/ctl): ```bash conduktor apply -f payments-developers.yaml ``` A single declarative resource replaces dozens of `kafka-acls.sh` bindings. The same definition lives in Git, runs in CI, and is also available as a [Terraform provider](https://registry.terraform.io/providers/conduktor/conduktor/latest). ### `kafka-acls.sh` vs Conduktor vs Confluent RBAC | Concern | `kafka-acls.sh` | Conduktor | Confluent RBAC | |---|---|---|---| | Granularity | Per-principal, per-resource | Group / role with pattern matching | Role-based, Confluent-only | | Multi-cluster | One invocation per cluster | Single command across every connected cluster | Confluent Platform / Cloud only | | Change audit | None natively (Log4j only) | Audit history of direct changes; request/approval metadata for self-service access | Confluent audit logs | | Off-boarding | Manual `--remove` per principal | Revoke at the group / IdP level, all permissions follow | Tied to Confluent identity | | Works against any Kafka | Yes | Yes (MSK, Confluent, Redpanda, Aiven, self-managed) | No — Confluent only | | Self-service | None | Topic / access requests with approval workflow | Limited | Conduktor does not replace the broker's ACL enforcement; it replaces the spreadsheet and the bash wrapper around it. Underneath, the broker still sees standard Kafka ACLs. ## ACLs vs RBAC ACLs and RBAC aren't competing technologies; they stack. RBAC handles the *who* and *what role* at the identity layer (users, groups, roles), then translates those assignments into the ACLs the broker actually enforces. You get the usability of roles and groups, and the broker still enforces the low-level rules it's always enforced. For the full picture (how ACLs fit alongside encryption, authentication, and auditing), read the [Kafka Security Guide](https://www.conduktor.io/kafka-security) or the deeper [Apache Kafka Security Playbook](https://www.conduktor.io/blog/apache-kafka-security-playbook). For the RBAC side specifically, see [Kafka RBAC](https://www.conduktor.io/kafka-rbac). ## Frequently Asked Questions **Are Kafka ACLs enabled by default?** No. Out of the box, Kafka has no authorizer configured, so any authenticated principal can do anything. Set `authorizer.class.name` in `broker.properties` to turn them on. **What happens if I turn on ACLs without defining any?** With `allow.everyone.if.no.acl.found=true` (the old default), the broker allows everything. That defeats the point. With `allow.everyone.if.no.acl.found=false`, the broker denies everything, including your own admin user, so add a `super.users` entry first. **Can I use ACLs without authentication?** Technically yes, but it's pointless. The broker can't enforce a per-principal rule if no principal has been authenticated. ACLs assume TLS, SASL, or mTLS is already in place. **How do I audit ACL changes?** Kafka's Log4j authorizer logger captures ACL updates when configured, but there's no native history or query UI. Ship the log output to a SIEM (Splunk, ELK, Datadog) for searchable history. **Do ACLs apply to consumer groups?** Yes. Consumer group operations (join, leave, commit offsets) are a separate resource type. Read access to a topic doesn't include consumer-group access; both need explicit ACLs. **I have more questions.** [Drop us a line](https://www.conduktor.io/contact?src=kafka-acl) and we'll get back to you. ## Going beyond ACLs? See how the Conduktor [Kafka governance platform](https://www.conduktor.io/kafka-data-governance) layers RBAC, audit trails, and cross-cluster ACL management on top of your existing brokers, without replacing Kafka's native enforcement. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-acl) [Read the Security Guide →](https://www.conduktor.io/kafka-security) Source: https://www.conduktor.io/kafka-acl --- # Kafka Operations Assessment Score your Kafka operations. 10 questions, 3 minutes. A benchmark report that tells you where you're bleeding time, risk, or budget. Plus a 30-min debrief with a human who's seen hundreds of clusters. Step 1 · Who are you? [See sample report](#sample) On average, users find they're eligible for: $500K+in annual savings 20–40%infrastructure cost reduction 5,000+engineering hours / yr reclaimed ******[Step 2 · Try Q1](#) $area: developer tooling When a developer needs a new Kafka topic or schema, what does the process look like? [1]File a ticket, ops team handles everything [2]Submit a PR or IaC change and wait for review [3]Some self-service, but ops still reviews most [4]Fully self-service with automated guardrails [5] Unlock the value trapped in your Kafka stack. Specific fixes, ranked by impact. SAMPLE REPORT Your Kafka Operations Score 53/100 REACTIVE Your team is firefighting. Structure exists but key workflows are manual, enforcement is inconsistent, and most issues are caught after they cause damage. **Your weakest area is Governance & Ownership at 22%.** Pillar breakdown Developer Tooling50% · Needs Attention ** Can developers ship without waiting on ops? Incident Response45% · Needs Attention ** Who investigates, and how fast can you recover? Governance & Ownership22% · Needs Attention ** Who owns what, and how is access controlled? Security & Compliance100% · Strong ** How is sensitive data protected end-to-end? Cost & Infrastructure45% · Needs Attention ** Can you attribute Kafka spend to teams? Things people ask first What to expect. Will I get spammed? No. You get your results. A 30-min debrief with our team is optional. Confluent / MSK / Redpanda? All of them. One of the first questions profiles your stack so the cohort is apples-to-apples. My data? Anonymized into the benchmark. Identifiable answers stay yours. ← → Your Kafka Operations Score 0OUT OF 100 [Discuss Your Results With Our Team →](https://www.conduktor.io/contact/demo?src=kafka-assessment) Send me my report Drop your email and we'll send your full report, plus an option to book a 30-minute debrief with our team. Work email I agree to receive marketing communications from Conduktor Inc. Send me my report → By submitting this form, you acknowledge that your information will be processed in accordance with our [Privacy Policy](https://www.conduktor.io/legal/privacy-policy). Your findings Each finding traces to one of your answers. Pillar Breakdown Each pillar is scored 0–100 based on your answers. Higher scores reflect more mature practices. Recommendation Based on your assessment, this is where Conduktor delivers the most impact: View Your Answers Source: https://www.conduktor.io/kafka-assessment --- # Kafka Consumer Lag Monitoring The cheapest signal you have for a stuck Kafka consumer. Measure it, alert on it, and stop running a Prometheus stack just to read one number. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-consumer-lag) [Try Conduktor Console →](https://www.conduktor.io/get-started?src=kafka-consumer-lag) ## What Kafka consumer lag actually is Consumer lag is the number of messages a consumer group still has to read on a partition. For partition `p`, lag is `LogEndOffset(p) − CommittedOffset(group, p)`. It is reported per partition; the lag of a group is the sum across the partitions it owns. It's the most direct answer to "is my pipeline keeping up?". A flat non-zero lag is fine — the consumer is steady-state behind. Lag that climbs and never drains means the consumer is backed up. Zero lag is ambiguous: the consumer is either caught up, the topic is idle, or the consumer is dead. You can't tell from lag alone, so watch `LogEndOffset` rate alongside it. ![Conduktor Console showing offset lag and time lag for a consumer group, with consume rate and historical graphs](https://www.conduktor.io/assets/console/obs-monitoring.png) ## Measuring lag with the Kafka CLI Kafka ships `kafka-consumer-groups.sh`. Run it against any group to see the lag per partition: ```bash bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --describe --group payments-processor # GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID # payments-processor payments 0 15243 15243 0 consumer-1 /10.0.1.4 worker-1 # payments-processor payments 1 14817 18402 3585 consumer-2 /10.0.1.5 worker-2 # payments-processor payments 2 15102 15102 0 consumer-3 /10.0.1.6 worker-3 ``` That gives you one point in time. To watch lag, you either script the command on a cron, or you publish JMX metrics from the broker and consumer JVM and scrape them with Prometheus. Both work. Both are work you have to maintain. If you're weighing the full field first, we compared thirteen options, exporters, APM platforms, and Kafka UIs, in [Best Tools to Monitor Kafka Consumer Lag](https://www.conduktor.io/blog/best-kafka-consumer-lag-monitoring-tools). ## What a healthy consumer lag looks like There is no single threshold. Lag is meaningful relative to the consumer's throughput and the SLA you owe downstream. - **Frame lag as N seconds of traffic, not a raw message count.** If your consumer processes 5,000 msg/s and you accept 30 s of latency, your alert threshold is 150,000 messages. - **Rate of change matters more than absolute level.** A consumer doing 50k/s that suddenly drops to 0/s with lag climbing linearly is more urgent than a steady-state lag of 200k. - **Per-partition matters.** Group-level lag can look fine while one partition is stuck because of a poison pill or a slow downstream call. ## Monitoring consumer lag with Conduktor Console Conduktor Console connects to your Kafka clusters and reads consumer lag through the same admin API that `kafka-consumer-groups.sh` uses — no agents on your brokers, no consumer-side instrumentation. What you get on top of the raw number: 1. **A live view of lag per group and per partition** across every cluster you connect, with the lag trend over time rather than a single snapshot. 2. **Threshold-based alerts** routed to Slack, Microsoft Teams, email, or a webhook (the webhook covers PagerDuty, Opsgenie, and anything else with an HTTP endpoint). Set a threshold on offset lag (messages) or on time lag (seconds). Teams own their own group alerts; the platform team owns cluster-level ones. 3. **Ownership context.** Consumer groups registered through the application catalog can be mapped to the application and team that runs them, so the alert lands with the person who can fix it — not the platform on-call. For the broader monitoring picture (broker health, partition replication, connector failures), see [Kafka monitoring with Conduktor](https://www.conduktor.io/product/kafka-monitoring). For a step-by-step walkthrough of setting an alert, see the [alerting documentation](https://docs.conduktor.io/guide/monitor-brokers-apps). ## Conduktor vs Prometheus + Grafana for consumer lag Both options work. The right choice depends on what you already run, how many teams own consumers, and how much glue code you want to maintain. | Concern | Prometheus + Grafana | Conduktor Console | |---|---|---| | Setup | `kafka_exporter` (or similar) + Prometheus scrape config + Grafana dashboard. Add a JMX exporter on each consumer if you also want client-side `records-lag` | Connect cluster, lag is visible immediately | | Lag granularity | Per-group, per-partition (with the right exporter) | Per-group, per-partition | | Alerting | Prometheus AlertManager rules in PromQL | UI thresholds, per-group, owned by the team | | Ownership / per-team alerts | DIY with labels and routing trees | First-class: groups mapped to applications and owners | | Multi-cluster | Per-cluster scrape, federate or remote-write into Thanos / Cortex / a central Prometheus | Single dashboard across every connected cluster | | Works alongside the other | — | Yes — Conduktor exposes its own metrics endpoint to Prometheus | Conduktor does not replace Prometheus for infrastructure-level metrics (node CPU, disk, network). It replaces the part of your Prometheus setup that exists *only* to scrape Kafka consumer lag and render it in Grafana. ## Common consumer lag problems and where to look - **Lag climbing on one partition.** Check for a poison-pill message at the head of that partition, or for a key whose downstream call (DB write, HTTP request) is slow. - **Lag climbing across all partitions in a group.** Throughput problem. Either the consumer is under-provisioned, the downstream is slow, or a single consumer instance is doing all the work because of an uneven partition assignment. - **Lag holding at zero with no recent LogEndOffset movement.** The consumer looks caught up, but the topic may be idle and the consumer process may already be dead. Lag alone cannot distinguish the two — watch consumer group member count and time-since-last-commit alongside lag. - **Lag oscillating around a high value.** The consumer is keeping up on average but is bursty. Usually a downstream call with variable latency. Not always urgent, but worth a graph. For the concept-level primer, see the glossary entry on [consumer lag monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). For a deeper take on choosing thresholds, see [Kafka Consumer Lag Alerting Thresholds](https://www.conduktor.io/blog/kafka-consumer-lag-alerting-thresholds-guide). ## Frequently Asked Questions **What is Kafka consumer lag?** Consumer lag is the number of messages a consumer group still has to process on a partition. It equals the partition's log-end offset minus the group's last committed offset. Group-level lag is the sum across the partitions the group owns. **How do I check consumer lag from the command line?** Use `kafka-consumer-groups.sh --bootstrap-server --describe --group `. The output includes CURRENT-OFFSET, LOG-END-OFFSET, and LAG per partition. **What's a normal consumer lag threshold?** There isn't a universal one. Express the threshold in seconds-of-traffic relative to your consumer's throughput rather than a raw message count, then alert when lag exceeds the latency budget you owe downstream. **How does Conduktor monitor consumer lag?** Conduktor Console reads lag through the Kafka admin API the same way the CLI does, then renders it per group and per partition across every connected cluster. You set thresholds per group on either offset lag or time lag, and route alerts to Slack, Teams, email, or any webhook endpoint (which covers PagerDuty, Opsgenie, and similar). **Do I need to install agents on my brokers or consumers?** No. Conduktor connects to the cluster as a Kafka client and uses the standard admin API. There is nothing to install on the brokers or the consumer applications. **Can Conduktor work alongside Prometheus and Grafana?** Yes. Conduktor exposes its own metrics endpoint that Prometheus can scrape. Teams who already have Grafana dashboards keep them; Conduktor adds the per-team ownership and self-service alerting that raw metrics dashboards don't provide. **I have more questions.** [Drop us a line](https://www.conduktor.io/contact?src=kafka-consumer-lag) and we'll get back to you. ## Monitor Consumer Lag in Minutes Connect your cluster and see lag per group, per partition, across every environment. Alerts routed to the team that owns the consumer, not the platform on-call. [Try Conduktor Console](https://www.conduktor.io/get-started?src=kafka-consumer-lag) [Talk to Us →](https://www.conduktor.io/contact/demo?src=kafka-consumer-lag) Source: https://www.conduktor.io/kafka-consumer-lag --- # The Kafka Governance Platform Conduktor is the Kafka governance platform: schema policies, topic ownership, RBAC, field-level encryption, audit trails, and data quality rules — applied across every Kafka cluster you run, without changing your brokers. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-data-governance) [Try Conduktor Console →](https://www.conduktor.io/get-started?src=kafka-data-governance) ## What Kafka data governance covers Kafka data governance is the policy and control layer that keeps a multi-team Kafka platform safe to operate. For any given topic, it tells you who owns it, who can touch it, what shape the data has to take, and what happens to records that fail. It's not a single feature — it's the layer you build (or buy) once the brokers alone stop being enough. ![The six primitives of Kafka data governance — schema policy, topic ownership, access control, encryption & masking, audit & lineage, data quality — sitting as a layer above the Kafka brokers](https://www.conduktor.io/assets/images/glossary/kafka-data-governance-0.webp) A complete Kafka governance platform answers six questions: 1. **Schema policy.** What schemas are allowed on which topics, what evolution rules apply (backward, forward, full), and what happens when a producer breaks the contract. 2. **Topic ownership.** For every topic, which application and which team is accountable. No orphan topics, no unowned resources. 3. **Access control.** Who — human or service account — can produce, consume, create, or delete. Expressed as roles and groups, not raw ACLs. 4. **Encryption and masking.** Which fields are PII, which need to be encrypted at the field level, which need to be masked in lower environments, and which keys protect them. 5. **Audit and lineage.** Who did what, when, from where. Available in a queryable, exportable form, not just broker log lines. 6. **Data quality.** What validation rules a message must pass to be considered acceptable, and what to do with the ones that fail. A platform that only covers two or three of these is not a Kafka governance platform. It is a Kafka UI with some access control bolted on. ## Why Kafka makes governance hard Apache Kafka is a broker. It serves bytes. It does not, on its own, know what a "team" is, what a "topic owner" is, or what counts as a sensitive field. Every governance primitive above has to come from somewhere outside the broker: - **Schemas** live in a Schema Registry, which is a separate service most teams treat as an artifact store rather than a policy point. - **Ownership** is a Confluence page or a spreadsheet. - **Access** is `kafka-acls.sh` and a manually-maintained ACL inventory. - **Encryption** is "we have TLS", which protects the wire but not the payload sitting on disk. - **Audit** is brokers logging to local files, sometimes shipped, sometimes not. - **Data quality** is whatever the producer team decided to validate, applied inconsistently. Multiply that by 10 teams, 500 topics, three clusters, and an auditor asking "show me everyone who has access to topics containing PII", and the gap becomes visible. ## How Conduktor delivers governance as one platform Conduktor is built as a governance platform rather than a UI with policies attached. Three components share one identity, ownership, and policy model: - **Conduktor Console** — topic-level governance at the management plane - **Conduktor Gateway** — a [Kafka proxy](https://www.conduktor.io/kafka-proxy) enforcing wire-level controls at the data plane - **Schema Registry Proxy** — per-subject RBAC and audit at the schema layer | Governance primitive | Conduktor capability | |---|---| | Schema policy | [Schema Registry Proxy](https://www.conduktor.io/schema-registry-proxy) adds per-subject RBAC and audit on top of Confluent Schema Registry (AWS Glue coming); Gateway blocks breaking changes at produce time | | Topic ownership | Application catalog: topics registered against an application inherit an accountable team | | Access control | RBAC with roles, groups, and SSO/LDAP identity; ACLs managed for you at the broker | | Encryption | Field-level encryption for JSON, Avro, and Protobuf payloads via Conduktor Gateway, with KMS-backed keys (Vault, AWS KMS, Fortanix) | | Audit | Console audit logs cover management actions; Gateway audit events cover request-level traffic. Both are exportable to a SIEM | | Data quality | CEL-based validation rules at the gateway, with route-on-failure to a DLQ | All of this runs in front of any Kafka — AWS MSK, Confluent Cloud, self-managed, Redpanda, Aiven — without broker reconfiguration. ## Conduktor governance vs Confluent Control Center ![Conduktor Console policy editor showing a GDPR-compliant data masking rule for PII fields with risk level and resource scope](https://www.conduktor.io/assets/console/sec-mask.png) | Concern | Confluent Control Center | Conduktor | |---|---|---| | Cluster support | Confluent Platform and Confluent Cloud | Any Kafka (MSK, Confluent, self-managed, Redpanda, Aiven) | | Topic ownership model | Limited | Application catalog with accountable owners | | RBAC | Confluent RBAC, Confluent-only | Works against any cluster, SSO-integrated | | Field-level encryption | CSFLE (limited, schema-dependent) | Gateway encryption with KMS, any payload | | Data quality validation | Schema Registry only | CEL rules enforced by Conduktor Gateway, with DLQ routing | | Audit centralisation | Per-product logging | One audit trail across Console + Gateway | | Self-service for developers | Limited | First-class: topic and access requests with workflow approval | Both are credible enterprise tools. Conduktor is a better fit when the Kafka estate is heterogeneous, when governance has to extend to data masking and quality, and when the developer experience of self-service is part of the requirement. ## Where governance lives in the Conduktor architecture - **Conduktor Console** enforces policies at the management plane: schema rules, topic naming, ownership, RBAC, self-service approvals, audit. Read more on the [Console page](https://www.conduktor.io/console). - **Conduktor Gateway** is a Kafka proxy that enforces policies at the data plane: authentication, RBAC, field-level encryption, masking, data quality, traffic transformations. It sits between clients and brokers, so applications connect to the gateway instead of the cluster directly. Read more on the [Gateway page](https://www.conduktor.io/gateway). - **Conduktor Schema Registry Proxy** sits in front of your Schema Registry (Confluent SR today, AWS Glue coming) and adds per-subject RBAC, OIDC/JWT authentication, and audit without forking the registry itself. Read more on the [Schema Registry Proxy page](https://www.conduktor.io/schema-registry-proxy). - **Identity** is shared via your IdP. Console authenticates users through OIDC SSO or LDAP; Gateway and Schema Registry Proxy accept OIDC/OAuth tokens (LDAP via a federated IdP like Keycloak). Removing a user from the IdP cuts off every surface. For the underlying security primitives the governance layer relies on (encryption, authentication, ACLs vs RBAC, auditing), see the [Kafka Security Guide](https://www.conduktor.io/kafka-security). ## Frequently Asked Questions **What is Kafka data governance?** Kafka data governance is the set of policies and controls that determine, for every topic and message in a Kafka estate, who owns it, who can access it, what schema and quality rules apply, what data is sensitive, and how every action is audited. It is delivered by a governance platform on top of Kafka, not by the broker itself. **Is Apache Kafka enough on its own for data governance?** No. Kafka brokers serve bytes; they have no concept of teams, owners, sensitive fields, or audit retention. Governance requires a layer above Kafka — Schema Registry, an ownership model, RBAC, encryption, audit pipelines, and quality rules — coordinated through one platform. **How does Conduktor compare to Confluent for governance?** Both cover the basics. Conduktor extends governance to any Kafka cluster (not only Confluent), adds gateway-level encryption and data quality enforcement, and is built around an application/team ownership model that maps registered topics to an accountable owner. Confluent Control Center is tightly coupled to Confluent Platform and Confluent Cloud. **Do we need to change our brokers to run Conduktor governance?** No. Conduktor connects to any Kafka cluster as a client and runs the Gateway as a transparent proxy in front of it. No broker plugins, no broker reconfiguration. **Where does Schema Registry fit in?** Schema Registry stores schemas; out of the box it does not enforce per-subject access control or produce an audit trail. Conduktor's [Schema Registry Proxy](https://www.conduktor.io/schema-registry-proxy) is a drop-in in front of Confluent Schema Registry (with AWS Glue support coming) that adds OIDC/JWT authentication, per-subject RBAC, and audit logging. Schema content stays where it is — the proxy adds the governance layer around it. **I have more questions.** [Drop us a line](https://www.conduktor.io/contact?src=kafka-data-governance) and we'll get back to you. ## Run a Kafka Governance Platform Across Every Cluster One control plane for schema policy, ownership, RBAC, encryption, audit, and data quality. Works in front of MSK, Confluent Cloud, Redpanda, Aiven, or self-managed Kafka. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-data-governance) [Explore Console →](https://www.conduktor.io/console) Source: https://www.conduktor.io/kafka-data-governance --- # Kafka Encryption Beyond TLS Enabling TLS doesn't encrypt your Kafka data. It encrypts the connection. Message payloads remain readable by anyone with broker access. Conduktor adds field-level encryption: specific fields are encrypted before reaching the broker, decrypted only for authorized consumers. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-encryption) ## Three Layers of Kafka Encryption Kafka encryption has three distinct layers, each protecting against different threats. (For where encryption fits alongside authentication, authorization, and auditing, see the [Apache Kafka Security Playbook](https://www.conduktor.io/blog/apache-kafka-security-playbook).) - **Encryption in Transit (TLS)** — Protects data moving between clients and brokers. Prevents network eavesdropping. TLS 1.2/1.3 configured on broker listeners. mTLS for mutual authentication. Required baseline for any production deployment. - **Encryption at Rest (Disk)** — Protects data on broker disks from physical theft or unauthorized storage access. Managed by your cloud provider (EBS encryption on AWS, persistent disk encryption on GCP) or OS-level disk encryption. Doesn't protect against broker compromise. - **Field-Level Payload Encryption** — Encrypts the actual content of specific message fields before they're written to the broker. The broker stores and replicates ciphertext. Even with full broker access, sensitive fields are unreadable without the decryption key. This is where Conduktor focuses. ## Field-Level Encryption: How Conduktor Does It - **Encryption at the Kafka Proxy** — Conduktor Gateway intercepts produce requests and encrypts specified fields before writing to the broker. Algorithm is configurable (AES-128-GCM by default, AES-256-GCM and ChaCha20-Poly1305 also supported). No changes to producer application code required. - **Key Management Integration** — Encryption keys are stored in your KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS, HashiCorp Vault, or Fortanix). Conduktor never stores keys. It retrieves and caches them with a configurable TTL. - **Per-Consumer Decryption** — Decrypt requests are intercepted at consume time. Gateway checks consumer permissions, retrieves the decryption key from KMS, and delivers decrypted fields to authorized consumers only. - **Schema-Aware Field Selection** — Define which fields to encrypt in each topic using field path notation (e.g., payload.creditCardNumber, headers.ssn). Supports Avro, JSON Schema, and Protobuf. ## Encryption for Compliance Use Cases - **GDPR Right to Erasure** — Implement [crypto shredding](https://www.conduktor.io/blog/crypto-shredding-in-kafka-a-cost-effective-way-to-ensure-compliance): encrypt each user's data with a per-user key. To erase a user's data, destroy their key. Instant, verifiable data deletion without topic compaction or reprocessing. - **PII Protection** — Encrypt PII fields (names, emails, SSNs, credit card numbers) in Kafka messages. Downstream consumers that don't need the raw value receive the encrypted field or a masked version. - **Multi-Tenant Data Isolation** — Each tenant's data encrypted with a separate key. Even if one tenant's data is accessed, other tenants' data remains protected. Required for shared-cluster multi-tenancy. - **Regulated Industry Compliance** — HIPAA, PCI-DSS, and financial regulations require encryption of sensitive data at rest. Field-level encryption satisfies this requirement for Kafka-based pipelines. See [When Kafka Becomes the Ledger](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger) for the fintech operating model. ## Frequently Asked Questions **How is field-level encryption different from TLS?** TLS encrypts the connection between client and broker. Data is decrypted when it reaches the broker. Field-level encryption encrypts the message payload. Data remains encrypted in the broker's storage. TLS and field-level encryption are complementary, not alternatives. **Which KMS providers does Conduktor support?** AWS KMS, Azure Key Vault, GCP Cloud KMS, HashiCorp Vault, and Fortanix. Keys are retrieved at encryption/decryption time and cached with a configurable TTL. **Does this require changes to producer or consumer code?** No. Producers write to Conduktor Gateway instead of directly to Kafka. Encryption is applied transparently. Consumers that aren't authorized for decryption receive the encrypted ciphertext or an error, depending on policy configuration. **What happens if the KMS is unavailable?** Gateway caches keys with a configurable TTL, so short KMS outages don't immediately affect encryption or decryption. If the KMS is unreachable and the cache has expired, requests fail with an error rather than passing through unencrypted. For compliance use cases, this fail-closed behavior is correct. **Can I encrypt only specific topics?** Yes. Encryption policies are configured per topic or topic pattern. Most organizations start with topics containing PII or financial data and expand from there. ## See Kafka Encryption in Production Stop relying on TLS as your only Kafka encryption. 30-minute demo: configuring field-level encryption on a topic, integrating with your KMS, and verifying broker-side ciphertext. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-encryption) [Read the Encryption Guide →](https://www.conduktor.io/blog/how-to-encrypt-data-in-kafka-without-piling-up-tech-debt) Source: https://www.conduktor.io/kafka-encryption --- # Kafka Proxy: Beyond the Routing Layer A Kafka proxy should do more than route traffic. The right one gives your platform team three things: a central control point to connect clients to clusters, the isolation and tooling to govern Kafka across teams, and wire-level enforcement to protect your data without changing applications. Most proxies only deliver the first. ## What a Kafka proxy should give your team The 60-second version A Kafka proxy sits between your clients and your clusters. Every message passes through it. Think of a proxy as doing three jobs: **Connect** (abstract clients from clusters), **Govern** (let multiple teams share safely), and **Protect** (enforce security at the wire). Every proxy on the market handles Connect: stable endpoints, backend remapping, DR failover. That part is solved. Govern is where it gets interesting. Virtual clusters, layered isolation, best practice enforcement, operational tooling that doesn't require YAML and a redeploy for every policy change. Protect is where most proxies have nothing at all. Wire-level encryption, masking, data quality, schema registry governance, all covering every client without code changes. Routing-only proxies stop at Connect. API gateways bolt on some Govern. Open-source proxies give you building blocks but your team does the rest. Purpose-built proxies are the only option that ships all three. On this page [What is a Kafka proxy?](#what-is-a-kafka-proxy) [Stage 01: Connect](#connect) [Stage 02: Govern](#govern) [Stage 03: Protect](#protect) [Types of Kafka proxy solutions](#types-of-kafka-proxy-solutions) [Where Conduktor Gateway fits](#where-conduktor-gateway-fits) [FAQ](#frequently-asked-questions) What is a Kafka proxy? A Kafka proxy is a layer between your Kafka clients and your Kafka clusters. It intercepts the native Kafka protocol so clients connect to the proxy instead of directly to brokers, but from their perspective nothing changes: the same protocol, the same client libraries, no code modifications. Routing traffic is the floor. The interesting part is what else the proxy can do once it's sitting in the path of every message. We think about this in three stages: - **01. Connect** — Abstract clients from clusters. Route traffic, switch backends, fail over, all without changing applications. - **02. Govern** — Let multiple teams share Kafka safely. Isolate, enforce guardrails, and give platform teams operational tooling. - **03. Protect** — Enforce security and data quality at the wire. Every client covered, no bypass possible, no code changes needed. All proxies provide the Connect functionality. Where proxies actually differ is Govern and Protect, and whether any of it works once you have more than one team on a cluster. The rest of this page walks through each stage in detail. 01 Stage 01 — Connect Can I abstract my clients from my clusters? Every Kafka proxy you'll evaluate can route traffic through a stable endpoint. The part worth paying attention to is whether your platform team can remap, migrate, and fail over backend clusters without touching a single application. ### Client abstraction At a minimum, a useful proxy gives you: - **A stable endpoint.** Clients connect to the proxy, not directly to broker addresses. When brokers change, the proxy handles it. - **Backend remapping.** Point clients at a different cluster by changing the proxy configuration, not by redeploying every application. - **DR failover.** Switch traffic to a standby cluster centrally. The question is whether you can do this via an API call or whether it requires a redeploy of the proxy itself. If your proxy can't do these three things without client-side changes, your platform team is going to spend migration weekends coordinating application teams instead of just flipping a switch. ### DR readiness vs. DR switching All proxies can switch clusters. The hard part is knowing your applications will actually survive the switch. A proxy that takes DR seriously provides: - **Chaos testing.** Simulate broker failures, latency spikes, and leader elections against your live traffic to validate that applications handle failover correctly. - **API-driven switching.** Change the backend cluster without redeploying the proxy. In an emergency, the difference between an API call and a change management cycle matters. Without readiness validation, you're finding out whether DR works during an actual outage. That's not a DR strategy, it's a gamble. Go deeper: [Client abstraction →](https://www.conduktor.io/gateway#connect-without-code-changes) · [DR and Failover →](https://www.conduktor.io/gateway#migrate-and-recover-with-confidence) · [Chaos testing for Kafka →](https://www.conduktor.io/blog/chaos-engineering-for-kafka-testing-recovery-before-you-need-it) > DR is more than cluster switching. Our ebook [Kafka Disaster Recovery: Beyond Replication](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-beyond-replication) covers the full strategy, from replication topology to readiness validation and runbook design. 02 Stage 02 — Govern Can multiple teams share Kafka safely? Most proxies stop being useful right about here. They were built assuming one team runs one cluster, but once you have five teams on shared Kafka infrastructure, that assumption falls apart. Teams step on each other, the platform team becomes a bottleneck for every change, and eventually someone says "just give us our own cluster" because isolation doesn't exist. ### Multi-tenancy and layered isolation Real multi-tenancy means each team operates as if they have their own Kafka cluster, while actually sharing physical infrastructure. This requires multiple layers working together: - **Virtual clusters** that give each team their own logical environment with separate topic namespaces. - **ACLs** that control who can access what within each virtual cluster. - **Traffic control policies** that prevent one team's workload from impacting another's: quotas, rate limits, and resource boundaries. The distinction matters. Some proxies offer virtual clusters with basic ACLs, but that only provides access control, not isolation. When your tenth team onboards and one of them starts producing 10x the expected volume, access control alone won't protect the other nine: layered policies do. > If your proxy separates teams by naming conventions, it's not multi-tenancy, it's an honor system. ### Best practice enforcement As more teams use Kafka, inconsistency becomes the default. Without enforcement at the proxy: - Teams create topics with wildly different configurations (replication factor 1 in production, anyone?) - Naming conventions exist in a wiki but aren't enforced anywhere - Consumer groups proliferate without ownership - The platform team reviews every change manually because there's no automated guardrail A proxy with composable interceptors lets the platform team define the rules once and enforce them on every request automatically. Topic naming conventions, replication minimums, partition limits, rate limits, all applied per virtual cluster without requiring each team to know or care about the policies. ### Operational tooling Governance also means the proxy itself needs to be operable at scale. If every configuration change requires editing YAML and redeploying the proxy, your platform team is going to spend more time managing the proxy than managing Kafka. Production-ready operational tooling means: - **Terraform provider** so proxy configuration fits into the same IaC workflows as the rest of your infrastructure. - **CLI and REST API** for automation and scripting. - **Runtime configuration** so policy changes take effect without restarts or redeployment. If onboarding a new team requires a pull request, a YAML edit, a CI pipeline, and a proxy restart, you've traded one bottleneck for another. Go deeper: [Multi-tenancy →](https://www.conduktor.io/gateway#decouple-tenants-from-clusters) · [Best practice enforcement →](https://www.conduktor.io/gateway#prevent-misconfigurations) 03 Stage 03 — Protect Is my data protected at the wire? This isn't where most organizations start, but where many end up. The proxy already sits in the path of every message, so it's the obvious place to enforce security and data quality. No client changes, no library to install, no team that can accidentally (or deliberately) bypass the rules. If you pick a proxy that has these capabilities built in, you turn them on when you need them. If your proxy doesn't have them, you re-platform. ### Wire-level security The critical distinction is where enforcement happens. With client-side enforcement (like Confluent's CSFLE), each application must include a specific library. Any application that doesn't include it sends data unencrypted. With wire-level enforcement, the proxy handles it: - **Field-level encryption.** Encrypt specific fields within a message (just the PII, not the whole payload) with KMS integration. Every client is covered regardless of language or framework. - **Tokenization.** Replace sensitive values with non-reversible tokens at the proxy. The original data never reaches downstream consumers, but the token preserves referential integrity so joins and aggregations still work. - **Cryptographic signing.** Attach signatures to messages at the proxy to prove authenticity and detect tampering. Consumers can verify that a message hasn't been modified since it passed through the proxy. Ask any proxy vendor: "What happens when a new application gets deployed without the encryption library?" If the answer is "that data goes through unencrypted," your security has an exception for every team that doesn't follow the process. ### Data quality Bad data in Kafka is expensive to fix after the fact. A proxy can catch it at the source: - **Schema enforcement.** Validate that messages conform to their registered schema before they reach the broker. - **Business rule validation.** Enforce rules beyond schema compliance: field value ranges, required fields, format constraints. - **Blocking and routing.** Reject bad data at the wire or route it to a dead letter topic for investigation. Most schema registries validate schemas but not the data itself. A proxy that enforces data quality at the wire closes that gap. ### Schema registry governance Most organizations' schema registries are either wide open or locked behind basic auth. One accidental breaking change to a schema can cascade through downstream consumers and turn into a production incident. A dedicated schema registry proxy adds: - **Modern authentication.** JWT/OIDC token validation (Keycloak, Auth0, Okta, Azure AD) so your registry uses the same identity provider as the rest of your infrastructure. - **Per-subject access control.** Teams can only register or modify schemas for the subjects they own. Read and write permissions with wildcard and prefix matching. - **Audit logging.** Every schema operation is traced (OpenTelemetry), metered (Prometheus), and logged. When something breaks, you can trace it back to the schema change that caused it. ### External data sharing Sharing Kafka data with external partners goes beyond just exposing an endpoint. Without isolation at the proxy layer, an external consumer can impact your internal infrastructure through excessive polling, unbounded consumption, or unexpected traffic patterns. A proxy designed for external data sharing provides: - **Dedicated partner zones** with their own isolation and traffic control policies. - **Resource boundaries** that prevent a partner's workload from affecting internal teams. - **Access controls** scoped to exactly the topics and operations the partner needs, and nothing more. Go deeper: [Data security →](https://www.conduktor.io/kafka-security) · [Schema Registry Proxy →](https://www.conduktor.io/schema-registry-proxy) · [External data sharing →](https://www.conduktor.io/solutions/use-case/data-sharing) > Wire-level security covers more than encryption. Our ebook [Achieving Data Security for Kafka](https://www.conduktor.io/resources/ebooks/achieving-data-security-for-kafka) breaks down the full toolkit: encryption, masking, tokenization, and how to build a security layer that doesn't depend on every team getting it right. ## Types of Kafka proxy solutions A scan of the landscape, scored through the Connect / Govern / Protect lens. For a detailed feature-by-feature comparison with specific products, see our Kafka proxies comparison page. - **Routing-Only Proxies** — Built for Connect: cluster migration, DR failover, and client abstraction. Strong at routing, but no multi-tenancy, no policy enforcement, and no data security at the proxy layer. You'll outgrow it the moment you need Govern or Protect. *Examples: Confluent Gateway.* - **API Gateway Extensions** — API management platforms that added Kafka support. Good Connect story if you already run one for REST APIs. Limited Govern (topic aliasing instead of virtual clusters, basic policies instead of composable interceptors). Protect capabilities are maturing but typically shallower than a purpose-built proxy. *Examples: Kong Event Gateway, Gravitee Kafka Gateway.* - **Open-Source Proxies** — Deliver Connect for free, with building blocks for Govern. But your team owns all the engineering: custom filters, YAML configuration, every upgrade. No Protect capabilities out of the box. Most organizations exceed the cost of a commercial product within the first year. *Examples: Kroxylicious, Aklivity Zilla.* - **Purpose-Built Kafka Proxies** — Designed for all three stages. Connect, Govern, and Protect from a single product, with operational tooling that doesn't require your platform team to build and maintain it themselves. The only category that covers the full spectrum of proxy use cases out of the box. *Examples: Conduktor Gateway.* ## Where Conduktor Gateway fits Every capability in Conduktor Gateway maps back to one of the three stages. Works with any Kafka provider: Confluent, MSK, Redpanda, Aiven, and open-source Apache Kafka. Continue reading [Conduktor GatewayFull capability breakdown with architecture diagrams.](https://www.conduktor.io/gateway) [Schema Registry ProxyAuthentication, access control, and audit logging for your schema registry.](https://www.conduktor.io/schema-registry-proxy) ## Read more customer stories - [FlixBus: Self-Service for 50+ Teams](https://www.conduktor.io/customer-stories/flix) - [Swiss Post: 5x Kafka Growth with 800 Users](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Virgin Australia: 300 Hours/Month Saved](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) ## Frequently Asked Questions **What is a Kafka proxy?** A Kafka proxy is a layer between your Kafka clients and your Kafka clusters. It intercepts the native Kafka protocol so every message passes through it, giving platform teams a single control point for routing, multi-tenancy, security, governance, and operational management without changing applications. **What is the difference between a Kafka proxy and an API gateway?** An API gateway manages HTTP/REST traffic. Some API gateways have added Kafka support, but their Kafka proxy capabilities (layered multi-tenancy, composable policy enforcement, schema registry governance) are typically shallower than a purpose-built Kafka proxy. They're best suited when Kafka is one of many integration points, not the primary infrastructure. **What is the difference between a Kafka proxy and protocol translation?** Protocol translation converts non-Kafka protocols (HTTP, MQTT, gRPC) into Kafka traffic for clients that don't speak Kafka natively. A Kafka proxy governs traffic that's already on the Kafka protocol: routing, isolation, security, and best practices. They solve different problems and can be complementary. **How does a Kafka proxy differ from Kafka ACLs?** Kafka ACLs are built-in access control rules managed directly in Kafka. A proxy manages governance at a higher level, layering virtual clusters, traffic control, and composable policies on top. You manage rules in the proxy, and teams don't need to know or care about the underlying Kafka ACL configuration. **Does a Kafka proxy add latency?** Single-digit milliseconds, typically. The proxy operates at the Kafka protocol level, so the overhead is minimal. ## Ready to see a Kafka proxy built for platform teams? In a 30-minute demo, we'll walk through your current Kafka operations and show you where Conduktor Gateway closes the gaps in Connect, Govern, and Protect, without changing how your applications work. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-proxy) [Explore Gateway →](https://www.conduktor.io/gateway) Source: https://www.conduktor.io/kafka-proxy --- # Kafka RBAC Without the ACL Nightmare Native Kafka ACLs don't understand teams. They're per-broker, hard to audit at scale, and there's no central view of who has access to what. Conduktor adds a role-based control plane on top, without touching the broker. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-rbac) ## Why Native Kafka ACLs Don't Scale Native Kafka ACLs work fine for small deployments. They break down at scale. The case for Kafka RBAC becomes clear when you have 20 teams, 500 topics, and a compliance team asking "who has access to what?" ACLs are managed per-broker with no central view, there's no concept of roles or groups, and changes require broker admin access you don't want to hand out. RBAC is one slice of the broader [Kafka governance platform](https://www.conduktor.io/kafka-data-governance) Conduktor delivers; for the security architecture it sits in, see the [Apache Kafka Security Playbook](https://www.conduktor.io/blog/apache-kafka-security-playbook). - **Role-Based Permissions** — Define roles (producer, consumer, admin, read-only) and assign them to users, service accounts, or groups. Permissions apply across all topics in a namespace automatically. - **Team Namespacing** — Isolate teams into virtual clusters with dedicated namespaces. Team A can only see and access Team A's topics. Cross-team access requires explicit grants. - **Service Account Management** — Create service accounts for applications with scoped permissions. Rotate credentials without touching broker configuration. Audit service account activity independently. - **Centralized Access View** — See every user, every group, every service account, and what they can access, across all clusters, in one interface. Export for compliance audits. - **Self-Service Access Requests** — Developers request topic access through a portal. Platform teams approve or set policies for auto-approval. Full audit trail of every grant and revocation. - **Zero Broker Changes** — Conduktor Gateway, a Kafka proxy, enforces RBAC transparently. Applications connect to Gateway instead of directly to Kafka. No broker reconfiguration required. ## RBAC Architecture: How It Works - **Identity Integration** — Connect to your existing identity provider (LDAP or OIDC-based SSO). User roles sync automatically. When someone leaves the company, their Kafka access is revoked with their identity. - **Policy Enforcement at the Kafka Proxy** — Conduktor Gateway intercepts every client request and checks it against the RBAC policy engine. Unauthorized requests are rejected before reaching the broker, with a clear error message. - **Audit Everything** — Every access grant, revocation, and permission check is logged. Who requested access, who approved it, when it was used last. All queryable and exportable. - **Principle of Least Privilege** — Start with no access. Grant exactly what each team or service needs. Conduktor enforces boundaries. Applications can't accidentally (or intentionally) consume from topics they haven't been granted access to. ## Frequently Asked Questions **How is Conduktor RBAC different from native Kafka ACLs?** Native ACLs are low-level and per-resource. Conduktor RBAC adds role abstractions, team namespacing, a central control plane, and human-readable permissions on top. The underlying broker ACLs are managed by Conduktor automatically. **Does it work with existing Kafka deployments?** Yes. Conduktor Gateway works as a transparent proxy in front of any Kafka cluster (MSK, Confluent Cloud, on-prem, Redpanda). No broker changes required. **Can I give read-only access to certain topics?** Yes. Roles can be scoped to: produce only, consume only, read metadata, or full access. Permissions apply per topic, per topic pattern, or per namespace. **How do we handle service-to-service Kafka access?** Create service accounts in Conduktor with scoped permissions. Applications use mTLS or token-based auth to Conduktor Gateway. Credentials are rotated without changing broker configuration. **What happens when a developer leaves the team?** Revoke their role in Conduktor. Conduktor Gateway immediately stops authorizing their requests. If integrated with your IdP, this happens automatically on user deactivation. **Does RBAC cover Schema Registry subjects?** Yes. A dedicated Schema Registry proxy enforces access control on schema operations — who can register, update, or delete schemas per subject. It uses the same OIDC identity as the rest of Conduktor, so permissions stay consistent across Kafka and Schema Registry. ## See Kafka RBAC in Action Stop managing Kafka ACLs by hand. 30-minute demo: setting up team namespaces, assigning roles, and auditing access, all without touching a broker. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-rbac) [Read the RBAC Guide →](https://www.conduktor.io/blog/security-for-humans-and-applications-in-apache-kafka) Source: https://www.conduktor.io/kafka-rbac --- # Kafka Security: Beyond TLS Securing Apache Kafka means covering four pillars: encryption, authentication, authorization, and auditing. TLS handles one slice of one pillar. This guide walks through each one, the tradeoffs between approaches, and the configurations that actually work in production. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-security) [Get the Ebook →](https://www.conduktor.io/resources/ebooks/achieving-data-security-for-kafka) ## The Four Pillars of Kafka Security The 60-second version Kafka security is four pillars: encryption, authentication, authorization, auditing. TLS alone is one slice of one pillar. Encryption covers the wire (TLS), the payload (field-level, for PII), and disk (at rest). Each blocks a different attacker. Authentication is how brokers prove who's connecting. mTLS for services, SASL or OAuth for humans, SSO for the rest. Stop using shared service accounts. Authorization is what callers can do. Native ACLs work at small scale. RBAC scales past 10 teams. Plan the migration before you need it. Auditing is what you can prove after the fact. Ship broker logs to a SIEM or you can't reconstruct an incident. 68% of breaches involve the human element, not zero-days. Most Kafka security work is closing the defaults the cluster ships with. Kafka security comes down to four questions. Can an outsider read the data? Who is connecting to the broker? What are they allowed to do? What did they actually do? Those map to four pillars: encryption, authentication, authorization, auditing. Get any one wrong and the defaults Kafka ships with become your security posture. The four pillars are also what a [Kafka governance platform](https://www.conduktor.io/kafka-data-governance) has to enforce centrally once you operate Kafka across multiple teams. For the architectural walkthrough of each pillar with code, real authorizer log lines, and our recommendations, see the [Apache Kafka Security Playbook](https://www.conduktor.io/blog/apache-kafka-security-playbook). - **1. Encryption** — Protect data in transit, at rest, and at the field level. TLS alone is not enough. - **2. Authentication** — Verify every client connecting to a broker. mTLS, SASL, and SSO. - **3. Authorization** — Control which users and services can read, write, or admin each resource. - **4. Auditing** — Capture every action with user, timestamp, and payload context. Native Kafka covers parts of each pillar. It leaves real gaps in others. The rest of this page walks through them one by one, with the tradeoffs and the practices that hold up at scale. On this page [Pillar 01: Kafka Encryption](#kafka-encryption) [Pillar 02: Kafka Authentication](#kafka-authentication) [Pillar 03: Kafka Authorization](#kafka-authorization) [Pillar 04: Kafka Auditing & Monitoring](#kafka-auditing) [Kafka Security Best Practices](#kafka-security-best-practices) [How Conduktor Covers the Four Pillars](#how-conduktor-covers-the-four-pillars) [Security for Regulated Industries](#security-for-regulated-industries) [FAQ](#frequently-asked-questions) 01 Pillar 01 — Kafka Encryption Can an outsider read the data? Without encryption, anyone on the network or with disk access can read your Kafka messages. You end up needing encryption in three places: in traffic between clients and brokers, in whatever Kafka writes to disk, and inside the payload itself for anything sensitive. Each layer stops a different kind of attacker, and none of them substitutes for the others. ### Data in transit: TLS and mTLS Kafka supports TLS out of the box. Clients negotiate a TLS session with the broker, exchange certificates, and encrypt traffic over the wire. Once TLS is enforced on a listener, unencrypted connections get rejected. Mutual TLS (mTLS) extends this. Instead of only the server presenting a certificate, the client presents one too. The broker validates the client certificate against a trusted CA. You get encrypted traffic and authentication in a single handshake. A minimal broker config that enforces TLS 1.3 and requires client certificates: ```properties # broker.properties listeners=SSL://:9093 listener.security.protocol.map=SSL:SSL ssl.keystore.location=/var/kafka/kafka.server.keystore.jks ssl.keystore.password= ssl.truststore.location=/var/kafka/kafka.server.truststore.jks ssl.truststore.password= ssl.client.auth=required ssl.enabled.protocols=TLSv1.2,TLSv1.3 ssl.protocol=TLSv1.3 ``` Set `ssl.client.auth=required` to reject any client that can't present a valid certificate. Drop the PLAINTEXT listener entirely; there's no production reason to keep one. Common mistake Treating TLS as the whole of Kafka security. It isn't. TLS protects the connection. Once a message reaches the broker, it's decrypted and stored on disk in plaintext. Any process with broker access can read it: admins, monitoring agents, a compromised service. ### Data at rest: disk-level encryption Kafka doesn't encrypt the messages it stores on disk. That's on the OS or your cloud provider. Common options: - **Linux Unified Key Setup (LUKS)** for self-managed brokers - **Amazon EBS encryption** for AWS - **Azure Disk Encryption** for Azure - **Google Cloud persistent disk encryption** for GCP Disk-level encryption protects against stolen disks and misconfigured backups. It does nothing about a process that already has broker access; that process sees plaintext. ### Field-level and message-level encryption For sensitive data like credentials, payment details, or health records, disk-level encryption isn't enough. The payload itself needs to be encrypted before it reaches the broker. This is called **application-level encryption** and comes in two forms. **Message-level encryption** encrypts the entire payload. The broker sees only ciphertext. It cannot route or filter based on content. Maximum confidentiality, but Kafka loses some capabilities. **Field-level encryption** encrypts only sensitive fields, such as PII or payment data. Routing and filtering on non-sensitive fields still work. The broker never sees the protected values in plaintext. | Layer | TLS only | Disk-level | Field-level | Message-level | |---|---|---|---|---| | **On the wire** | Encrypted | Plaintext unless TLS is also enabled | Encrypted with TLS | Encrypted with TLS | | **On broker disk** | Plaintext | Encrypted by the OS or cloud | Ciphertext only | Ciphertext only | | **Readable by broker admins** | Yes | Yes, through a Kafka client | No | No | | **Routing and filtering** | Preserved | Preserved | Works on non-encrypted fields | Lost — the full payload is opaque | | **Typical use** | Baseline | Stored-data compliance | PII, PCI, HIPAA | End-to-end confidentiality | | **Overhead** | Low | Low | Moderate | Higher | Native Kafka gives you TLS. The other three layers are on you. Either you write encryption code in every producer and consumer, or you use a proxy layer that enforces it centrally. Go deeper: [full Kafka encryption guide →](https://www.conduktor.io/kafka-encryption) · [how Conduktor Gateway handles encryption across apps →](https://www.conduktor.io/gateway#encrypt-consistently-across-apps) 02 Pillar 02 — Kafka Authentication Who is connecting to the broker? Authentication verifies the identity of every client connecting to Kafka. Without it, anyone on the network can produce or consume messages. There are three common approaches. Certificates Mutual TLS Client and broker exchange certs signed by a trusted CA. Common for service-to-service traffic. Credentials / tickets SASL Pluggable mechanisms: PLAIN, SCRAM, GSSAPI (Kerberos), OAUTHBEARER. Choose per threat model. Human users SSO (OIDC / LDAP) Front the cluster with a proxy that terminates SSO and maps to a Kafka principal. > 68% of breaches involve the human element: errors, stolen credentials, social engineering. Not exotic exploits. ([Verizon 2024 DBIR](https://www.verizon.com/business/resources/reports/dbir/)) ### Mutual TLS (mTLS) Client and broker each present a certificate signed by a trusted CA. Once validated, the TLS session carries an authenticated identity. Common in service-to-service scenarios where certificate issuance is already part of the infrastructure. ### SASL (Simple Authentication and Security Layer) A plug-in framework. Kafka supports four SASL mechanisms: - **SASL/PLAIN** — username and password. Simple. Credentials travel with every connection, so use only with TLS. - **SASL/SCRAM** — challenge-response with hashed passwords. Credentials never travel in plaintext. - **SASL/GSSAPI (Kerberos)** — ticket-based auth against a KDC. Common in large enterprises with existing Kerberos. - **SASL/OAUTHBEARER** — token-based auth for OAuth2 and OpenID Connect providers. ### SSO: OIDC and LDAP SSO isn't native to Kafka, but it's what operators end up needing. Humans don't manage their own certificates. They log into Okta, Azure AD, or Google Workspace. A proxy or control plane in front of Kafka terminates the OIDC or LDAP handshake and maps the identity to a Kafka principal. A minimal SASL/SCRAM configuration: ```properties # broker.properties listener.security.protocol.map=SASL_SSL:SASL_SSL sasl.enabled.mechanisms=SCRAM-SHA-512 # client.properties security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=\ org.apache.kafka.common.security.scram.ScramLoginModule required \ username="alice" password="alice-secret"; ``` Kafka validates the SCRAM handshake against credentials stored in KRaft (or ZooKeeper on older clusters) and binds the connection to a principal like `User:alice`. Go deeper: [SSO for humans in Conduktor Console →](https://www.conduktor.io/console#secure-without-slowing-down) · [service auth translation at the wire in Conduktor Gateway →](https://www.conduktor.io/gateway) 03 Pillar 03 — Kafka Authorization What are they allowed to do? Authentication tells you who is connecting. Authorization decides what they're allowed to do. Kafka has two main approaches: native ACLs, and role-based access control layered on top. ### Access Control Lists (ACLs) ACLs are off by default. Turn them on in `broker.properties` before anything else, otherwise every authenticated principal can do everything: ```properties # broker.properties (KRaft) authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer super.users=User:admin allow.everyone.if.no.acl.found=false ``` On ZooKeeper-backed clusters, swap the authorizer for `kafka.security.authorizer.AclAuthorizer`. The other two lines are the same. With `allow.everyone.if.no.acl.found=false`, the broker denies any request that isn't explicitly permitted. Safer default than the alternative. ACLs are then managed through the `kafka-acls.sh` CLI. Each ACL is an allow rule mapping a principal to a resource and operation: ```bash bin/kafka-acls.sh --bootstrap-server localhost:9092 \ --add --allow-principal User:alice \ --operation Read --topic customer-events ``` ACLs are fine-grained but have practical limits at scale: - **No groups.** Add a new team member, add ACLs one at a time. - **No role inheritance.** Every principal's permissions are managed directly. - **No audit of changes.** Who added this ACL last Tuesday? Hard to answer. - **No centralized visibility.** Listing permissions across clusters means running the CLI per cluster. > For dozens of users and topics, ACLs work. For hundreds or thousands across multiple clusters, they become unmanageable. ### Role-Based Access Control (RBAC) RBAC adds a layer of indirection between users and permissions. Permissions attach to roles. Users join groups. Groups get roles. Change the role once, and everyone in the group picks up the new permissions. This makes onboarding and offboarding one step. A new data scientist joining `data-science-team` inherits every permission that team has. A leaver removed from the group loses access immediately. | | ACLs | RBAC | |---|---|---| | **Unit of control** | Per-resource, per-operation | Per-role, applied across resources | | **Groups** | Not supported | First-class | | **Role inheritance** | Not supported | Nested roles and hierarchies | | **Centralized visibility** | CLI per cluster | Unified UI or API | | **Change audit** | Limited, no native history | Full history of role and assignment changes | | **Scale** | Small deployments | Hundreds of users and topics | | **Standards alignment** | Kafka-specific | NIST RBAC | ACLs aren't going away. They're still the enforcement layer at the broker. RBAC sits on top to manage them. The combination is what you want: roles and groups for humans, ACLs for the broker. Go deeper: [Kafka ACL guide →](https://www.conduktor.io/kafka-acl) · [Kafka RBAC guide →](https://www.conduktor.io/kafka-rbac) · [group-based RBAC in Conduktor Console →](https://www.conduktor.io/console#secure-without-slowing-down) 04 Pillar 04 — Kafka Auditing & Monitoring What did they actually do? When an auditor asks "who consumed customer PII on March 3 at 2 PM?", the answer should take minutes, not weeks. Auditing captures every action with enough context to reconstruct what happened. ### What Kafka logs natively Kafka uses Log4j to emit authorization and authentication events. With the right configuration, you get records of: - Successful and failed authentication attempts - Authorization decisions (allow or deny) - Admin API calls (topic creation, ACL changes, config updates) ```properties # log4j.properties log4j.logger.kafka.authorizer.logger=INFO, authorizerAppender log4j.logger.kafka.request.logger=INFO, requestAppender ``` What you don't get natively: consumer-side activity (who actually read which message from which partition), full payload context, and cross-cluster correlation. Turning raw broker logs into a proper audit trail is a separate build. ### SIEM integration Kafka's Log4j output ships to any standard SIEM or log aggregator: - **Splunk** — via universal forwarders or HEC - **ELK / OpenSearch** — Filebeat or Logstash - **Datadog** — Datadog Agent with the Kafka integration - **Sumo Logic, New Relic** — via their Log4j appenders Once in the SIEM, detection rules do the actual security work: repeated auth failures, unusual IP patterns, privilege escalation, admin API spikes. ### Compliance reporting Auditors don't want raw logs. They want evidence: "show me every access to PCI data in Q3." That means: - **Retention policies** aligned with regulations (GDPR: case-by-case; PCI-DSS: at least one year; HIPAA: six years) - **Tamper-evidence** — signed or write-once audit logs - **Searchable reports** filtered by user, resource, time range, or action type - **Export formats** your auditors will accept, usually CSV, JSON, or PDF Native Kafka produces the raw events. Turning them into audit-ready evidence is work most teams end up doing themselves. Go deeper: [audit trails and SIEM integration in Console →](https://www.conduktor.io/console#secure-without-slowing-down) · [wire-level auth logging in Conduktor Gateway →](https://www.conduktor.io/gateway) ## Kafka Security Best Practices > All four pillars working together is what holds up in production. Lean on any one alone and there's a gap. 01Enforce TLS on every brokerNo unencrypted listeners in production. Disable PLAINTEXT on all ports. Set a minimum TLS version of 1.2. 02Use mTLS or SASL/SCRAMAvoid SASL/PLAIN outside local testing. For human users, front the cluster with SSO through a Kafka proxy. 03Rotate certificates automaticallyTrack expiry dates. Automate renewal and broker reloads. Expired certs cause outages. 04Prefer RBAC over raw ACLs at scaleKeep ACLs as the enforcement layer. Manage access through roles and groups. Audit role changes. 05Enforce least privilegeProducers get write-only on their own topics. Consumers get read-only. Admins are a small, named group with MFA. 06Encrypt PII at the field levelDisk encryption and TLS do not protect against privileged broker access. Encrypt sensitive fields with a KMS-managed key. 07Centralize audit logs in a SIEMRaw Log4j output on brokers is not an audit trail. Ship to Splunk, ELK, or Datadog with alerting rules. 08Monitor authentication failuresRepeated failed logins are often brute-force attempts. Alert on rate thresholds. 09Isolate environmentsSeparate clusters or virtual clusters for dev, staging, and prod. Do not share credentials across environments. 10Run quarterly security reviewsReview ACLs and RBAC assignments, check certificate expiry, verify SIEM ingestion, run a tabletop exercise. ## How Conduktor Covers the Four Pillars Conduktor splits the work across two products. Console handles people and policy: SSO, RBAC, ACL management, user audit trails. Conduktor Gateway handles data in motion: field-level encryption, tokenization, application-level audit. You can run one or both, which is why Conduktor Gateway works in front of Confluent Cloud or AWS MSK without bringing Console along. - **Encryption — Conduktor Gateway** — [Conduktor Gateway](https://www.conduktor.io/gateway#encrypt-consistently-across-apps) encrypts at the wire, in the payload, and in headers with eight algorithms including AES-GCM and ChaCha20-Poly1305. Schema-aware for Avro, JSON, and Protobuf. Crypto-shredding supports per-record keys for GDPR right-to-erasure. Any KMS: AWS, Azure, GCP, HashiCorp Vault, Fortanix. - **Authentication — Console + Conduktor Gateway** — [Console](https://www.conduktor.io/console#secure-without-slowing-down) terminates SSO (OIDC, LDAP) for humans. [Conduktor Gateway](https://www.conduktor.io/gateway#decouple-tenants-from-clusters) enforces mTLS, SASL/SCRAM, and OAUTHBEARER for workloads, validating OIDC claims at the proxy. One identity layer across every connection. - **Authorization — Console + Conduktor Gateway** — [Console](https://www.conduktor.io/console#secure-without-slowing-down) handles group-based RBAC and native ACL management. [Conduktor Gateway](https://www.conduktor.io/gateway#decouple-tenants-from-clusters) enforces virtual ACLs per tenant at the wire, with rate limiting and per-tenant isolation. - **Auditing — Console + Conduktor Gateway** — [Console](https://www.conduktor.io/console#secure-without-slowing-down) logs user and admin actions. [Conduktor Gateway](https://www.conduktor.io/gateway) logs application-level activity across produce, fetch, and admin APIs. [70+ event types](https://docs.conduktor.io/guide/conduktor-in-production/admin/audit-logs), with SIEM export to Splunk, ELK, and Datadog. Also in Conduktor Gateway **Tokenization**Format-preserving tokens via HashiCorp Vault Transform. Run analytics on protected fields without exposing real values. **Cryptographic signing**HMAC-SHA256 message signing and verification. Detect tampered payloads in flight. Chain-of-custody for regulated workloads. **Dynamic data masking**Mask fields per consumer at the wire. Support team sees masked card numbers. Fraud team sees the real data. **Data quality enforcement**CEL and SQL validation at the proxy. Block or redirect messages that violate schema or business rules before they reach consumers. **Virtual clusters**Prefix-based logical isolation. One physical cluster can serve many tenants with independent ACLs, quotas, and policies. **Schema Registry proxy**Extend the same authentication and authorization model to schema operations. Sensitive fields defined in the schema, enforced at the wire. Continue reading [Encrypt Kafka without tech debtThe difference between CSFLE and proxy-based encryption.](https://www.conduktor.io/blog/how-to-encrypt-data-in-kafka-without-piling-up-tech-debt) [Data Protection use caseThe business case, outcomes, and ROI for securing Kafka.](https://www.conduktor.io/solutions/use-case/data-protection) [Security best practices guideLong-form deep dive on encryption and auth configuration.](https://www.conduktor.io/blog/kafka-encryption-and-security-best-practices-for-protecting-data) Security for regulated industries The same four pillars, mapped to the frameworks auditors actually ask about. GDPRRight to erasureField-level encryption plus crypto shredding. Destroy the key, individual records become permanently unreadable. No topic deletion, no reprocessing. DORA / FinancialResilience for regulated workloadsAudit trails, access controls, and incident response for financial services. Bitvavo runs DORA-compliant Kafka with Conduktor in production. [The fintech operating model →](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger) HIPAAPHI protection in streamsEncrypt PHI at the field level before it reaches Kafka brokers. Healthcare teams use Conduktor to protect patient data in streaming pipelines. SOC 2Audit-ready evidenceFull audit logs of data access and admin actions. Evidence export for compliance reviews without manual log aggregation. PCI-DSSCardholder data, unchanged clientsTokenization and field-level encryption for cardholder data. Meet PCI requirements without rebuilding every producer and consumer. ISO 27001Documented controls, end-to-endEncryption, authentication, authorization, and audit trails in a single control plane. Fewer custom builds to justify to your auditor. ## Read more customer stories - [Bitvavo: DORA & MiCA Compliance for 1.5M Users](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [Instant Payments for European Banks](https://www.conduktor.io/customer-stories/securing-kafka-banking-financial-services) - [Swiss Post: 5x Kafka Growth with 800 Users](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) ## Frequently Asked Questions **Is Kafka secure by default?** No. Out of the box, Kafka accepts plaintext connections with no authentication. TLS, SASL, and ACLs are all opt-in. Most Kafka breaches come from misconfigured clusters, not exploited vulnerabilities. **What is the difference between Kafka SSL and TLS?** SSL is the older protocol. TLS is its successor. Modern Kafka uses TLS exclusively, though older docs and config parameters still say "SSL." When you see `security.protocol=SSL`, it is actually TLS under the hood. **How do I secure Kafka topics?** Start with TLS on every listener so traffic is encrypted. Add SASL or mTLS so the broker knows who's connecting. Then restrict access with ACLs, or with RBAC once you're past a handful of users. If you handle PII or payment data, encrypt those fields on top. **What is the difference between authentication and authorization?** Authentication verifies who you are. Authorization decides what you can do. You need both. Authenticating without authorizing means anyone who logs in can do anything. **How does RBAC differ from Kafka ACLs?** ACLs are per-resource allow rules with no concept of groups or role inheritance. RBAC organizes permissions around roles you assign to users or groups. A new team member inherits the group's permissions automatically; a leaver loses them immediately. **Do I need both TLS and field-level encryption?** For sensitive data, yes. TLS protects the wire. Field-level encryption protects the payload. Without field-level encryption, anyone with broker or disk access can read plaintext even when TLS is enforced. **How do I audit Kafka access?** Enable Kafka's Log4j audit loggers, ship the output to a SIEM (Splunk, ELK, Datadog), and set up detection rules for failed auth attempts, privilege escalation, and unusual access patterns. **Does Conduktor replace Kafka's native security?** No. It extends it. TLS, ACLs, and SASL still work. Conduktor adds field-level encryption, RBAC, SSO, data masking, and richer audit logging on top. **What are the main Kafka security vulnerabilities?** The usual suspects aren't CVEs in the broker; they're misconfigurations. Plaintext listeners left on. SASL/PLAIN without TLS. Shared service-account credentials. ACLs that nobody audits or revokes when someone leaves. Disk encryption but no field-level encryption, so any process with broker access reads plaintext PII. Almost every published Kafka incident traces back to one of these, not to a novel exploit. **What's the overhead of field-level encryption?** Measurable but usually small. For most workloads, field-level encryption adds single-digit milliseconds per message when implemented at a proxy, because only the protected fields go through the crypto path and the rest flows through untouched. Heavier schemes (full-payload encryption with chain-of-custody signing) cost more but still sit well under native Kafka's throughput ceiling for most deployments. Benchmark your own payload sizes before committing. **How do I migrate from ACLs to RBAC without downtime?** Run both in parallel during the transition. Model your existing ACLs as roles in the RBAC system, assign those roles to the right groups or users, and verify the effective permissions match. Once verified, flip applications over to the RBAC-managed identities one at a time. ACLs remain the enforcement layer at the broker throughout, so nothing breaks if you pause the migration mid-way. **Does Kafka support OAuth2?** Yes, through SASL/OAUTHBEARER. Kafka validates a bearer token from your OIDC provider (Okta, Azure AD, Keycloak, Google) and binds the connection to a principal. In practice, most teams front Kafka with a proxy that terminates OAuth at the edge so brokers don't need to be exposed to the identity provider directly. ## Ready to Secure Your Kafka Deployment? Production Kafka security takes more than TLS. In 30 minutes with our team, we'll walk through your current posture across the four pillars and show you what Conduktor would change. No edits in your producers or consumers. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-security) [Get the Ebook →](https://www.conduktor.io/resources/ebooks/achieving-data-security-for-kafka) Source: https://www.conduktor.io/kafka-security --- # Kafka Streams aggregations *Learn how to aggregate a stream, and why the output surprises people.* Aggregation is where Kafka Streams earns its keep: turning a firehose of events into running counts, sums, and rollups per key. You group records, then fold them into a result: a count of orders per customer, a sum of payments per account, the latest reading per sensor. The mechanics are three method calls. The surprise is the output shape: an aggregation does not emit one answer per key, it emits a *stream of revisions* to that answer, and in production a record cache decides how many of those revisions you actually see. Get that mental model right and the rest is detail. **What you'll learn:** - `groupByKey` vs `groupBy`, and the repartition topic the second one creates - `count`, `reduce`, and `aggregate`: when to reach for each - Why an aggregation emits a continuous stream of intermediate updates, not one final result - How `Materialized` controls the store and changelog behind the result ## Group first, then aggregate Every aggregation in the DSL starts by grouping a `KStream` into a `KGroupedStream`. You have two ways to do it, and the difference is not cosmetic: it decides whether Kafka Streams writes a repartition topic. ```java // Records are already keyed by what you want to group on → no repartition KGroupedStream byCustomer = orders.groupByKey(); // Group by something other than the current key → re-keys → repartition topic KGroupedStream byProduct = orders.groupBy((key, order) -> order.productId()); ``` `groupByKey()` keeps the existing record key. Records for the same key are already on the same partition (that is how they were produced), so Kafka Streams can aggregate them in place. No extra topic, no extra hop. `groupBy(...)` picks a *new* key. Records for the new key are scattered across partitions, so Kafka Streams must shuffle them: it writes every record back to an internal **repartition topic** keyed by the new value, then reads it back so each key lands on one partition. That is a full round-trip through the broker: extra produce, extra fetch, extra storage. > **Prefer `groupByKey` when you can.** If your records are already keyed correctly, don't re-key them just to read more naturally. A needless `groupBy` doubles the write volume for that branch of the topology and adds a topic to operate. When you *do* need it, name the repartition topic with `Grouped.as("...")` so it survives topology edits; see [evolving a topology](https://www.conduktor.io/kafka-streams/topology-evolution). Left unnamed, the repartition topic borrows the aggregation store's positional name (`KSTREAM-AGGREGATE-STATE-STORE-0000000007-repartition`), so one shifted index renames the store, its changelog, *and* the repartition topic in a single edit. ## count, reduce, aggregate Once grouped, you fold the records. The DSL gives you three operators, from least to most flexible. | Operator | What it does | When to use | |---|---|---| | `count()` | Counts records per key | "How many orders per customer" | | `reduce(...)` | Combines two values of the *same* type into one | Running max, latest-wins, sum of a numeric stream | | `aggregate(...)` | Folds records into a result of a *different* type | Anything `reduce` can't express: stats objects, lists, custom accumulators | `count` is the trivial case: ```java KTable ordersPerCustomer = orders.groupByKey().count(); ``` `reduce` takes two values of the same type and returns one of that type. Good for "keep the most expensive order seen per customer": ```java KTable biggestOrder = orders .groupByKey() .reduce((current, incoming) -> incoming.amount() > current.amount() ? incoming : current); ``` `aggregate` is the general fold. You supply an **initializer** (the starting value, called once per key) and an **adder** (how each new record updates the running result). The result type can differ from the input type, which is what makes it strictly more powerful than `reduce`: ```java KTable stats = orders .groupByKey() .aggregate( OrderStats::new, // initializer: empty accumulator (key, order, agg) -> agg.add(order), // adder: fold one record in Materialized.>as("order-stats") .withValueSerde(orderStatsSerde)); // store + serde for the new type ``` Two things to note. The aggregate value type is no longer your input type, so the default value serde no longer applies: you must declare a serde for the accumulator via `Materialized.withValueSerde(...)`, or the very first record written to the store throws a `StreamsException` with a `ClassCastException` at its root. It fails immediately on the store write, not on a later emit, and since Kafka 3.x the message spells out the fix: *"A serializer (StringSerializer) is not compatible to the actual value type (OrderStats). Change the default Serdes in StreamConfig or provide correct Serdes via method parameters."* (Serde mismatches are their own rabbit hole; see [serde errors](https://www.conduktor.io/kafka-streams/serdes).) Second, the initializer runs **once per key**, not once per record; the adder runs for every record. ## Aggregating several streams at once: cogroup When one aggregate is fed by *several* input streams (orders, refunds, and adjustments all folding into one balance per customer), don't N-way join them first. `cogroup` folds multiple grouped streams into a single result, each stream with its own adder, sharing one state store and one output `KTable`: ```java KTable balance = orders.groupByKey().cogroup(applyOrder) .cogroup(refunds.groupByKey(), applyRefund) .cogroup(adjustments.groupByKey(), applyAdjustment) .aggregate(Balance::new, Materialized.as("balance")); ``` The initializer runs once, each source keeps its own adder, and there's no intermediate join or second store. Available since Kafka 2.5 (KIP-150). ## The output is a stream of updates, not one answer Here is the part that catches almost everyone. An aggregation returns a `KTable`, and a `KTable` is a *changelog*: the latest value per key. Every time a record updates a key's running result, that updated result is a new entry in the changelog. So if you do this: ```java orders.groupByKey().count().toStream().to("order-counts"); ``` …and ten orders arrive for `customer-42`, you do not get one record `(customer-42, 10)` on `order-counts`. You get a sequence of *refinements*: `(customer-42, 1)`, `(customer-42, 2)`, … up to `(customer-42, 10)`: one output per input, each superseding the last. The aggregation continuously revises its answer as data flows in. There is no "the input is done now" moment, because the stream is unbounded. This trips up people who expect SQL `GROUP BY` semantics, where you get one row per group at the end. A streaming aggregation has no end. It emits an ever-improving estimate of the answer, forever. ### The record cache modulates how many updates you see ![Record cache coalesces 10 input updates for customer-42 into 2 emitted values between commit 1 and commit 2](https://www.conduktor.io/assets/kafka-streams/diagrams/aggregations-record-cache.svg) In production you usually see *fewer* updates than inputs, which makes the behavior look inconsistent if you don't know why. Kafka Streams puts a **record cache** in front of each store (sized by `statestore.cache.max.bytes`, formerly `cache.max.bytes.buffering`). Updates to the same key coalesce in that cache and are only forwarded downstream when the cache fills *or* on the next commit, governed by `commit.interval.ms`. So between commits, ten updates to `customer-42` may collapse into one or two emitted records: you see the value jump from 3 to 9, skipping the values in between. This is why the same topology behaves differently in two places: | Environment | Cache | What you observe | |---|---|---| | Production (default cache + commit interval) | On | Coalesced updates: intermediate values skipped | | Cache disabled (`statestore.cache.max.bytes=0`) | Off | Every single update emitted, one output per input | | [`TopologyTestDriver`](https://www.conduktor.io/kafka-streams/testing) in tests | Off | Every update emitted, your test sees more records than prod | > 🚫 *"I counted, and the aggregation emits one final result per key."* > It emits a continuous stream of intermediate results. The record cache and `commit.interval.ms` only decide how many of those intermediate results escape, they never turn the stream into a single final answer. The cache is an optimization for emission volume, **not** a correctness boundary. Never rely on it to deduplicate or to "wait for the final value." If you genuinely need one result per key per time bucket, a closed window with a single answer, that is a different mechanism entirely: you bound the aggregation in time with [windowing](https://www.conduktor.io/kafka-streams/windowing) and gate emission with [`suppress`](https://www.conduktor.io/kafka-streams/suppress-not-emitting). Those pages cover the trap in full; the one-line version is that `suppress(untilWindowCloses(...))` holds back every update until the window closes on stream-time, and gets it wrong in subtle ways on quiet streams. ## Deletes, tombstones, and the result table Because the result is a `KTable`, it speaks the table dialect, including deletes. When a key's aggregate is removed (for example a `filter()` on the result table dropping a row, or an adder returning `null` to delete the key), Kafka Streams emits a **tombstone**, a record with a `null` value, to signal "this key is gone." If you call `toStream()` on the result and forward it onward, those `null` values flow with it, and code that assumes every value is non-null will throw an NPE. One removal that does *not* tombstone: window retention expiry. When a window ages out, Kafka Streams drops the old store segments locally and forwards nothing downstream, so don't build on expiry tombstones that never come. This is the same table semantics that govern `filter` and `mapValues` on any `KTable`: tombstones are special and your value functions may not even run for them. The full treatment lives in [KStream vs KTable vs GlobalKTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable), worth reading before you push an aggregation result downstream. ## Rolling up to several time resolutions A dashboard often needs the same metric at more than one granularity, per minute, per hour, per day. There are three ways to build that, and the right one depends on whether your reads or your writes dominate. - **Independent stores from the raw stream.** Run a separate aggregation per resolution, each reading the source topic. Simple, but you re-read the raw stream once per resolution and maintain that many changelogs, the most expensive option, and rarely worth it. - **Cascade smallest to largest.** Aggregate to the finest bucket, then aggregate *that* result up to the next (minute → hour → day), each `KTable` feeding the next. Every resolution is pre-materialized and cheap to read, at the cost of a chain of stores and changelogs. Reach for this when reads are heavy and you want each resolution ready to serve. - **Keep only the finest store, roll up on read.** Materialize just the smallest bucket and sum the range at query time (via [interactive queries](https://www.conduktor.io/kafka-streams/state-store) or your serving layer). Cheapest to maintain, one store, one changelog, at the cost of work per read. Reach for this when writes dominate and reads are occasional. The decision is the usual streaming trade-off in miniature: pre-compute and store (read-heavy) versus store less and compute on demand (write-heavy). ## Where the result lives Every aggregation is stateful: the running result for each key has to be kept somewhere between records. That somewhere is a [state store](https://www.conduktor.io/kafka-streams/state-store), and you configure it with `Materialized`: ```java KTable counts = orders .groupByKey() .count(Materialized.as("orders-per-customer")); // named, queryable, stable changelog ``` Naming the store does three things at once: it gets a stable changelog topic (so a topology edit doesn't orphan the state), it becomes queryable from your service via interactive queries, and it shows up under a name you can recognize on the cluster instead of a positional one like `KSTREAM-AGGREGATE-STATE-STORE-0000000003`. Since Kafka 4.3 you can make this a guardrail: set `ensure.explicit.internal.resource.naming=true` and the app refuses to start with any unnamed internal topic or store. The store, its off-heap RocksDB memory, and the restore-on-restart cost are covered in [state stores](https://www.conduktor.io/kafka-streams/state-store). ## Frequently Asked Questions **How do I aggregate records in Kafka Streams?** Group the stream first with `groupByKey()` or `groupBy(...)` to get a `KGroupedStream`, then fold it with `count()`, `reduce(...)`, or `aggregate(...)`. The result is a `KTable` holding the running result per key, backed by a state store you configure with `Materialized`. **What is the difference between aggregate, reduce, and count?** `count()` just counts records per key; `reduce(...)` combines two values of the same type into one; `aggregate(...)` is the general fold that can produce a result of a different type via an initializer and an adder. Reach for `aggregate` whenever the output type differs from the input or `reduce` can't express it. **What is the difference between groupBy and groupByKey?** `groupByKey()` keeps the existing record key, so records are already co-located and no repartition is needed. `groupBy(...)` picks a new key, which forces a shuffle through an internal repartition topic, so prefer `groupByKey` and only re-key when you genuinely must group on a different field. **Why does my Kafka Streams aggregation emit so many intermediate updates?** An aggregation returns a `KTable`, which is a changelog, so it emits a revised result on every input record rather than one final answer. In production a record cache (`statestore.cache.max.bytes`) and `commit.interval.ms` coalesce some updates, so you see fewer than one output per input, but never a single final value. **How do I compute an average with Kafka Streams?** Use `aggregate(...)` with an accumulator that tracks both the running sum and the count, then derive the average with a `mapValues` on the resulting `KTable`. You can't do it with `reduce` because the accumulator type (sum and count) differs from the input record type. > **See it in practice with Conduktor** > An aggregation quietly creates internal topics, a changelog for the store, and a repartition topic if you used `groupBy`. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-aggregations) lets you see those topics, check the changelog's size and compaction, and watch consumer group lag, so you can tell whether a stalled aggregation is a restore in progress or a genuine backlog. ## Next steps - [Windowing](https://www.conduktor.io/kafka-streams/windowing), bound aggregations in time with tumbling, hopping, and session windows - [The suppress() trap](https://www.conduktor.io/kafka-streams/suppress-not-emitting), get one final result per window without losing data on quiet streams - [State stores](https://www.conduktor.io/kafka-streams/state-store), where aggregation results live, and why they bite in production Source: https://www.conduktor.io/kafka-streams/aggregations --- # Kafka Streams for AI agents *Learn how Kafka Streams becomes the memory layer for a multi-agent AI system.* Multi-agent AI is a distributed-systems problem. Once you stop wiring agents together with direct HTTP calls and put Kafka between them, the conversation becomes a durable, ordered log, and that log is exactly the kind of thing Kafka Streams was built to turn into queryable state. This page shows the pattern that keeps surfacing in production agent stacks: **Kafka Streams as the agents' context store**, materialized in the same data plane the agents already run on, with no external database and no Flink. **What you'll learn:** - Why an agent needs a context store, and why bolting on a separate database is a tax - How to materialize conversation memory into a `KTable` keyed by conversation - How to track turn-rate with a window store for quotas and "user is stuck" detection - How agents read that state in single-digit milliseconds via interactive queries ## Why an agent needs a context store An LLM's context window is finite; a conversation grows without bound. Something has to summarize, truncate, and re-inject the relevant history into each prompt, and that something needs the full conversation, fast, on every turn. The reflex is to bolt on a database: Postgres, DynamoDB, Redis. But in a Kafka-based agent system the conversation *already exists*, ordered and durable, on topics like `user.messages`, `subagent.responses`, and `agent.responses`. Re-ingesting it into a second store is a pipeline nobody budgets for and everybody ends up operating. Kafka Streams collapses that loop: it reads the conversation topics and materializes the memory in place, next to the bus, where it sees every turn the moment it lands. > **This is a real architecture, not a thought experiment.** It's the design behind Conduktor's multi-agent reference implementation, an orchestrator, specialized sub-agents, share-group tool workers, an audit tap, and a Kafka Streams context store with HTTP interactive queries. The wider argument is in [Kafka and Flink as the infrastructure for AI agents](https://www.conduktor.io/blog/ai-agents-at-scale-the-critical-role-of-kafka-and-flink). ## The conversation is already a log When agents communicate over Kafka, every message, a user turn, a sub-agent handoff, a final response, is an event on a topic. Key those events by `conversationId` and Kafka does two useful things for free: all of a conversation's turns land on the **same partition** (so they stay ordered), and a Streams app can [group them by key](https://www.conduktor.io/kafka-streams/aggregations) and fold them into a running view. ## Materialize conversation memory in a KTable ![Three agent conversation topics merge through Kafka Streams into a conversation-context KTable (RocksDB + compacted changelog); a parallel window store tracks turn rate; the orchestrator reads back via interactive query HTTP GET](https://www.conduktor.io/assets/kafka-streams/diagrams/ai-agents-context-store.svg) Merge the conversation topics, group by `conversationId`, and aggregate each conversation's turns into a single context object held in a [state store](https://www.conduktor.io/kafka-streams/state-store): ```java StreamsBuilder builder = new StreamsBuilder(); KStream turns = builder.stream("user.messages", Consumed.with(Serdes.String(), turnSerde)) .merge(builder.stream("subagent.responses", Consumed.with(Serdes.String(), turnSerde))) .merge(builder.stream("agent.responses", Consumed.with(Serdes.String(), turnSerde))); turns .groupByKey() // already keyed by conversationId .aggregate( ConversationContext::new, (conversationId, turn, ctx) -> ctx.append(turn), Materialized.>as("conversation-context") .withValueSerde(contextSerde)); ``` The result is a `KTable`, the full transcript per conversation, kept in RocksDB and backed by a compacted changelog (`-conversation-context-changelog`, auto-created with `cleanup.policy=compact`) so it survives restarts. The three input topics must be [co-partitioned](https://www.conduktor.io/kafka-streams/joins) on `conversationId` (same key, same partition count) so each conversation's turns converge on one task. One trap: Streams only enforces co-partitioning for joins, not for merged sources feeding an aggregation. With mismatched partition counts the app reaches RUNNING with no error or warning, and a conversation's turns silently split across tasks, leaving a partial transcript in the store. Verify the counts yourself. Because the store is named, its changelog is stable across [topology edits](https://www.conduktor.io/kafka-streams/topology-evolution). ## Track turn-rate with a window store The same app can answer operational questions about a conversation. A five-minute [tumbling window](https://www.conduktor.io/kafka-streams/windowing) counting user turns gives you both a rate limit and a "this user is looping / stuck" signal: ```java builder.stream("user.messages", Consumed.with(Serdes.String(), turnSerde)) .groupByKey() .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5))) .count(Materialized.as("turns-5m")); ``` One Streams app, two materialized views: the full transcript for prompt enrichment, and a windowed turn-count for quotas, both derived from the log the agents are already writing. ## Serve it to the agents with interactive queries A materialized store isn't only internal plumbing. With [interactive queries](https://www.conduktor.io/kafka-streams/state-store), the agent reads it directly over HTTP, no separate database, no network hop to an external system: ```java ReadOnlyKeyValueStore store = streams.store(StoreQueryParameters.fromNameAndType( "conversation-context", QueryableStoreTypes.keyValueStore())); // GET /conversation/{id}, enrich the next prompt with prior turns ConversationContext ctx = store.get(conversationId); ``` The orchestrator calls `GET /conversation/{id}` to fold prior turns into the next prompt, or `GET /conversation/{id}/activity` to enforce a quota from the windowed store. Both answer in single-digit milliseconds because the state lives in-process, next to the agent's data plane. The one operational detail to plan for: with more than one instance, the state is sharded across hosts, so a conversation may live on another instance. You ask `streams.queryMetadataForKey(...)` which instance owns the key and route the call there, the same multi-instance routing covered in [state stores](https://www.conduktor.io/kafka-streams/state-store). Add a replica and the view rebuilds itself from the log. ## When this fits, and when it doesn't Be honest about the boundary, the same way you would about adding Kafka at all. A single agent with modest traffic does not need this, a row in Postgres holds the conversation fine, and you skip the operational surface entirely. The context-store pattern earns its place when the agents *already* run on Kafka at scale, and you want their memory in the same governed, observable plane as the rest of the traffic rather than in a second system you have to keep in sync. That last part is the quiet payoff: when agent memory is a Kafka topic and a Streams store, it inherits everything Kafka already gives you, ordering, replay, audit, and access control, instead of needing its own. ## Frequently Asked Questions **Where does Kafka Streams fit in an AI agent architecture?** It serves as the agents' context store: a Streams app reads the conversation topics, aggregates each conversation into a `KTable` keyed by `conversationId`, and the agents read that state over interactive queries to enrich the next prompt. The memory lives in the same data plane as the agent traffic, with no external database. **Can Kafka Streams power real-time features for AI agents?** Yes, for stateful enrichment and operational signals derived from the conversation log. The same app that materializes the transcript can run a five-minute tumbling window counting user turns to drive both a rate limit and a "user is looping / stuck" signal, both from the log the agents already write. **How do agents read the context store in low latency?** Through interactive queries: the agent calls `streams.store(...)` and reads the value directly in-process, answering in single-digit milliseconds with no network hop to an external system. With more than one instance the state is sharded, so you ask `streams.queryMetadataForKey(...)` which instance owns the key and route the call there. **Why not just use Postgres or Redis for agent memory?** When agents communicate over Kafka, the conversation already exists as an ordered, durable log, so re-ingesting it into a second store is a pipeline you have to operate and keep in sync. A single agent with modest traffic is fine in Postgres; the Streams context store earns its place when the agents already run on Kafka at scale and you want their memory in the same governed, observable plane. **Do the conversation topics need to be co-partitioned?** Yes. The input topics (`user.messages`, `subagent.responses`, `agent.responses`) must share the same key (`conversationId`) and partition count so every turn of a conversation converges on one task and stays ordered. Streams does not enforce this for merged sources (only for joins): with mismatched partition counts the app runs without error and a conversation silently splits across tasks, so check the counts yourself. > **See it in practice with Conduktor** > Agent traffic is just Kafka topics, so it observes and governs like any other. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-ai-agents) shows the `user.messages` / `subagent.responses` / `agent.responses` topics, their consumer-group lag, and the changelog behind the context store, the full conversation trail in one place. And because the data plane is shared, [Gateway interceptors](https://www.conduktor.io/gateway) can redact sensitive fields or enforce quotas on agent topics with no change to the agent code. Conduktor also speaks [MCP](https://www.conduktor.io/mcp) for AI-driven Kafka operations. ## Next steps - [State stores & interactive queries](https://www.conduktor.io/kafka-streams/state-store), the mechanism that serves context to the agents - [Windowing](https://www.conduktor.io/kafka-streams/windowing), the tumbling windows behind turn-rate and quotas - [The future of Kafka Streams](https://www.conduktor.io/kafka-streams/future), remote state, new rebalance protocols, and the AI direction Source: https://www.conduktor.io/kafka-streams/ai-agents --- # Kafka Streams anti-patterns *Spot the eight that cause most production incidents.* Most Kafka Streams outages are not exotic. They trace back to a handful of design choices that look reasonable in a code review and fall over under load, during a rebalance, or six months later when someone edits the topology. This page is a map: each anti-pattern gets the symptom you'll see, why it's wrong, and a link to the page that fixes it properly. If you read one section, read the first. A blocking call inside the topology is the single most common way a Kafka Streams app goes from "fast" to "stalled" with no error in the logs. **What you'll learn:** - The eight anti-patterns behind most Streams production incidents - The symptom each one produces, so you can recognize it in your own app - Why each is wrong at the mechanism level, not just "best practice says so" - Where to go for the real fix ## 1. Synchronous external calls inside the topology **Symptom:** throughput collapses under load; one slow dependency stalls a whole partition; consumer lag climbs while CPU sits idle; with exactly-once, transactions time out and you see producer-fenced errors. A `map` or `process` that makes a blocking REST or database call per record is the most common Streams anti-pattern, and the most damaging. A stream thread processes records on one partition *sequentially*. Block it on a 50ms network round-trip and that partition's ceiling is ~20 records/second, no matter how many cores you have. It gets worse: - **It breaks ordering and back-pressure.** The thread can't move to the next record until the call returns; a single slow upstream caps the partition. - **It breaks exactly-once.** A transaction stays open across the blocking call. If the call is slow, the transaction can hit `transaction.timeout.ms` and abort, and a [task migration fences the producer](https://www.conduktor.io/kafka-streams/exactly-once-duplicates) mid-flight. - **It couples availability.** Your Streams app is now only as available as the slowest service it calls per record. **The fix:** bring the data *to* the stream instead of calling out per record. Load reference data as a [`KTable` or `GlobalKTable`](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable) and [join](https://www.conduktor.io/kafka-streams/joins) against it, a local lookup, no network. When you truly must call an external system, make it asynchronous and batched, and accept that you've left the comfortable part of the framework. > 🚫 *"It's just one quick REST call inside the `map` to enrich each event, it's fast enough."* ## 2. Hand-rolling joins and aggregations **Symptom:** a growing pile of bespoke code that keeps state in a plain `HashMap`, loses it on restart, double-counts after a rebalance, and silently corrupts on reprocessing. Implementing a join by stashing one side in a static map, or an aggregation by mutating a field in a processor, throws away everything the framework gives you: changelog-backed fault tolerance, correct partitioning, and recovery after failure. A `HashMap` in your app is not fault-tolerant, when the instance dies, the state is gone, and a rebalance moves the partition to an instance that never saw the data. **The fix:** use the DSL's [aggregations](https://www.conduktor.io/kafka-streams/aggregations) and [joins](https://www.conduktor.io/kafka-streams/joins). They're backed by a state store and a changelog topic, so state survives crashes and follows partitions. Drop to the [Processor API](https://www.conduktor.io/kafka-streams/processor-api) only for logic the DSL genuinely can't express, and even then, use a *real* state store, not a field. ## 3. Unbounded state with no TTL or windowing **Symptom:** disk fills up over weeks; restarts get slower and slower; RocksDB memory creeps until the container is OOMKilled; the changelog topic grows without bound. Aggregating on an ever-growing key space, user IDs, session IDs, request IDs, with a plain `KTable` keeps *every* key forever. There is no automatic expiry on a non-windowed aggregate. The state store, its changelog, and the restore time all grow with your key cardinality until something breaks. **The fix:** bound the state deliberately. Use [windowed aggregations](https://www.conduktor.io/kafka-streams/windowing) with a retention period so old windows are purged, or a windowed store, or a [Processor API punctuator](https://www.conduktor.io/kafka-streams/processor-api) that deletes expired keys on a timer. If state is genuinely unbounded by nature, plan for its size explicitly, see [state restore time](https://www.conduktor.io/kafka-streams/state-restore) and [RocksDB tuning](https://www.conduktor.io/kafka-streams/rocksdb-tuning). ## 4. Ignoring co-partitioning **Symptom:** a join that produces nothing, or drops records, with no exception in the logs. ![Co-partitioning: a join task pairs partition N of one side with partition N of the other. When orders and customers use different partitioners, key acct-42 lands on partition 2 in orders and partition 0 in customers, so no task sees both halves and the join emits zero rows with no error. After repartition re-keys both sides through the same Streams partitioner, acct-42 lands on partition 1 on both and task 1 joins them.](https://www.conduktor.io/assets/kafka-streams/diagrams/co-partitioning.svg) A `KStream`-`KStream` or `KStream`-`KTable` join requires both sides to be *co-partitioned*: same partition count **and** records placed by the same partitioner. You'd hope a partition-count mismatch fails fast with `TopologyException`, but Streams only runs that co-partitioning check when the topology contains at least one repartition topic (verified on Kafka 3.9 and 4.3). A plain join of two source topics with mismatched counts starts cleanly and silently misses matches on the extra partitions. A partitioner mismatch is just as quiet, if one topic was written by a producer using a different partitioning scheme, the same key lands on different partition numbers on each side, the join silently sees no match, and you get empty or partial output with nothing in the logs. **The fix:** ensure both sides share a partition count and partitioner, or re-key and let Streams repartition. This failure is subtle enough to deserve its own page, see [joins that drop data](https://www.conduktor.io/kafka-streams/join-troubleshooting). ## 5. Not naming your operators **Symptom:** a topology change you expected to be a rolling upgrade instead orphans all your state; the app refuses to start, or starts with an empty store, after you inserted one operator. Kafka Streams names internal stores, changelog topics, and repartition topics *positionally*, `KSTREAM-AGGREGATE-STATE-STORE-0000000005` and the like. Insert or reorder an operator and every downstream number shifts. The new topology looks for `...0000000006`, finds nothing, and your existing state under `...0000000005` is orphaned. **The fix:** name everything explicitly from day one, `Materialized.as(...)`, `Grouped.as(...)`, `Repartitioned.as(...)`, `StreamJoined.withName(...)`, `Named.as(...)`. Names you control don't shift when the graph changes. Since Kafka 4.3, building a topology with unnamed internal resources logs a WARN listing each one, and setting `ensure.explicit.internal.resource.naming=true` (KIP-1111) turns any unnamed operator into a hard startup failure. This is the foundation of safe upgrades, see [evolving a topology](https://www.conduktor.io/kafka-streams/topology-evolution). ## 6. One giant topology **Symptom:** a deploy of one unrelated feature forces a full-app rebalance and state restore; you can't scale or tune one part without affecting all of it; one poison record takes down the whole pipeline. Cramming every unrelated stream into a single `application.id` couples concerns that should be independent. The blast radius of any change, any rebalance, and any failure becomes the entire application. Independent workloads end up sharing thread pools, restore time, and failure domains for no reason. **The fix:** split unrelated pipelines into separate applications with their own `application.id`, deployed and scaled independently. Smaller topologies rebalance faster, restore less state, and isolate failures. Use sub-topologies (Streams splits the graph at repartition boundaries) within an app, and separate apps across unrelated domains, see [scaling](https://www.conduktor.io/kafka-streams/scaling). ## 7. Using exactly-once as a deduplication substitute **Symptom:** you turn on `exactly_once_v2` expecting all duplicates to vanish, and downstream consumers still see repeats. Exactly-once semantics make the *consume-process-produce cycle inside Kafka* atomic. They do **not** deduplicate records that were duplicated *before* they reached your topology, a producer that sent the same business event twice, an upstream retry, a re-ingested file. EOS is effect-once for the read-process-write loop, not a content-level dedup over your input. **The fix:** understand exactly what EOS covers and what it doesn't ([exactly-once, and why you still see duplicates](https://www.conduktor.io/kafka-streams/exactly-once-duplicates)), then deduplicate on a business key explicitly with a state store when you need content-level dedup ([deduplication](https://www.conduktor.io/kafka-streams/deduplication)). Layer them: idempotent producer upstream, EOS for the cycle, explicit dedup for cross-producer repeats. ## 8. In-memory stores for large state **Symptom:** steady-state performance is great, then a restart or rebalance takes minutes because the store rebuilds from scratch every time. An in-memory store (`Stores.inMemoryKeyValueStore`) is bounded by your heap and, crucially, holds nothing on disk. So on every restart it must replay its *entire* changelog from the beginning before processing a single record. A persistent RocksDB store keeps data on local disk and only catches up on the delta it missed. For large state, in-memory turns a 10-second restart into a multi-minute one. **The fix:** use the default persistent (RocksDB) store for anything but small, fast-churning state where you've accepted the full-restore cost. The trade-off is laid out in [state stores](https://www.conduktor.io/kafka-streams/state-store), and the restore mechanics in [state restore time](https://www.conduktor.io/kafka-streams/state-restore). ## The common thread Seven of these eight share a root cause: treating Kafka Streams like an ordinary application and ignoring that it's a *stateful, partitioned, fault-tolerant* one. State lives in stores backed by changelogs, work is pinned to partitions, and tasks move between instances on rebalance. Design with that model and most of these anti-patterns never appear. Fight it, block a thread, keep state in a field, ignore partitioning, and you'll meet them in production. ## Frequently Asked Questions **What are the most common Kafka Streams anti-patterns?** The ones behind most production incidents are synchronous external calls inside the topology, hand-rolling joins and aggregations in a plain `HashMap`, unbounded state with no TTL or windowing, ignoring co-partitioning, not naming operators, cramming everything into one giant topology, treating exactly-once as a dedup mechanism, and using in-memory stores for large state. **Should I call a REST API or database inside map()?** No. A stream thread processes one partition sequentially, so a blocking 50ms call caps that partition at roughly 20 records/second regardless of cores, breaks back-pressure, and can blow `transaction.timeout.ms` under exactly-once. Bring the data to the stream as a `KTable` or `GlobalKTable` and join against it instead. **What is the "ever-growing state" anti-pattern and how do I avoid it?** Aggregating on an unbounded key space with a plain `KTable` keeps every key forever, so disk, changelog, and restore time grow until something is OOMKilled. Bound it deliberately with windowed aggregations and a retention period, or a Processor API punctuator that deletes expired keys on a timer. **Why is naming your operators a production best practice?** Kafka Streams names internal stores, changelog, and repartition topics positionally, so inserting or reordering one operator shifts every downstream number and orphans your existing state. Naming everything explicitly with `Materialized.as(...)`, `Grouped.as(...)`, and `Named.as(...)` keeps those names stable across topology changes. **Does exactly-once deduplicate records duplicated before they reach my topology?** No. Exactly-once makes the consume-process-produce cycle inside Kafka atomic, but it does not remove duplicates a producer sent twice, an upstream retry, or a re-ingested file. For content-level dedup you need an explicit dedup step on a business key with a state store. > **See it in practice with Conduktor** > Several of these anti-patterns show up first as Kafka symptoms, not application errors: climbing consumer group lag (the blocking-call stall), a sprawling set of internal topics (unbounded state), or an empty changelog after a deploy (orphaned state from an unnamed operator). [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-anti-patterns) surfaces the lag, the changelog and repartition topics, and the partition assignment you need to catch these before they page you. ## Next steps - [Joins that drop data](https://www.conduktor.io/kafka-streams/join-troubleshooting), diagnosing the co-partitioning failure - [Evolving a topology safely](https://www.conduktor.io/kafka-streams/topology-evolution), why naming operators matters - [Deduplication in Kafka Streams](https://www.conduktor.io/kafka-streams/deduplication), content-level dedup the right way Source: https://www.conduktor.io/kafka-streams/anti-patterns --- # Kafka Streams architecture *Understand how a Kafka Streams app actually runs.* You write a topology, a graph of operators, and call `streams.start()`. What happens next decides how your app scales, how many threads it spins up, and how many internal topics quietly appear on your cluster. Kafka Streams turns that topology into a set of tasks, hands the tasks to threads, and runs the threads inside your own process. The single idea that makes all of this make sense: the **task** is the fixed unit of parallelism, and it's bounded by partition count. Get that, and rebalancing, scaling, and "why won't my app go faster" stop being mysteries. **What you'll learn:** - How a topology splits into sub-topologies and tasks - Why a task maps to exactly one partition of a sub-topology - How stream threads run tasks, and what the internal consumer/producer do - Why the topology graph, not you, decides which internal topics exist ## From topology to running code ![One Kafka Streams topology compiles into sub-topologies, tasks (one per partition), stream threads, and JVM instances](https://www.conduktor.io/assets/kafka-streams/diagrams/topology-task-thread-hierarchy.svg) When you build a topology with the DSL, you describe a directed graph: source nodes that read topics, processor nodes that transform records, sink nodes that write topics. `builder.build()` gives you a `Topology` object; `Topology.describe()` prints it. Reading that description is the fastest way to understand what your app will actually do, Matthias Sax's "Nuts and Bolts of Kafka Streams" deep dive (Current 2023) frames the whole library as exactly this: a graph compiled into runtime units. Worth treating the topology as the source of truth, not the DSL code you typed. Here is the rough shape of the runtime, top to bottom: ``` Topology one graph you define └─ Sub-topology a connected chunk, split at repartition boundaries └─ Task one per partition of the sub-topology's input └─ runs inside a Stream Thread └─ inside your JVM process (one of N instances) ``` Everything below "Topology" is created for you. You don't instantiate a task or schedule a thread by hand, you set `num.stream.threads`, run more instances, and Kafka Streams does the assignment. ## Sub-topologies: where the graph gets cut A topology isn't always one connected piece. Kafka Streams splits it into **sub-topologies** wherever data has to be *repartitioned*, that is, shuffled across the network because a key changed or a `repartition` was inserted. Why the cut matters: records inside one sub-topology flow directly, in-process, node to node, no network hop. Crossing from one sub-topology to the next means writing to an internal **repartition topic** and reading it back. That boundary is also where parallelism is re-decided, because the downstream sub-topology reads a different topic with its own partition count. A simple stateless pipeline (`stream → filter → mapValues → to`) is one sub-topology: same key throughout, no shuffle. The moment you re-key and then aggregate, `stream → selectKey → groupByKey → count`, Kafka Streams cuts the graph at the re-key, because the aggregation needs records grouped by the new key on the right partition. (Re-keying is where a lot of accidental cost hides; see [stateless operations](https://www.conduktor.io/kafka-streams/stateless-operations).) ## Tasks: the fixed unit of parallelism This is the part to internalize. A **task** is the processing for **one partition** of a sub-topology's input. If a sub-topology reads a topic with 6 partitions, Kafka Streams creates 6 tasks for it, `0_0` through `0_5`. Each task owns its partition's records, its own copy of any [state stores](https://www.conduktor.io/kafka-streams/state-store) in that sub-topology, and its own position in the log. The number of tasks is pinned to the partition count of the input to each sub-topology. It's recomputed at each rebalance, so it only moves if someone changes a topic, and that is not a scaling lever: add partitions under a stateful app and it shuts down with `Existing internal topic ...-changelog has invalid partitions` and tells you to run `StreamsResetter`, while a stateless app silently picks up the new tasks at the next rebalance. Either way, the partition count gives you the hard ceiling: > The maximum parallelism of a Kafka Streams app is the largest partition count among its sub-topologies. More threads or instances past that point sit idle. Two consequences people learn the hard way: - **A 6-partition input topic caps you at 6 tasks for that sub-topology.** Spin up 10 instances with 1 thread each and 4 of them have no task to run, they become hot standbys at best, idle at worst. - **A repartition topic with fewer partitions throttles everything downstream of it.** If you repartition into 3 partitions mid-pipeline, the downstream sub-topology has at most 3 tasks no matter how wide the source was. Scaling is its own topic with its own gotchas (uneven assignment, standbys, when to add partitions), see [scaling Kafka Streams](https://www.conduktor.io/kafka-streams/scaling). The thing to carry here: you scale by tasks, and tasks are pinned to partitions. ## Stream threads: who runs the tasks Tasks are *what* to do; **stream threads** are *who* does it. Each instance of your app runs `num.stream.threads` threads (default 1), and Kafka Streams distributes the available tasks across all threads across all instances. A thread runs its assigned tasks in a loop: poll records, push them through the topology, commit. One thread can run several tasks; it processes them in turn, not in parallel within the thread. So your real concurrency is `min(total tasks, total threads across all instances)`. | Knob | What it changes | Bounded by | |---|---|---| | `num.stream.threads` | Threads per instance | CPU cores on the instance | | Number of instances | Threads across the fleet | N/A | | **Tasks (derived)** | Actual parallel work | **Max partition count of a sub-topology** | You tune the first two. The third is the ceiling, and it's set by your topics, not your config. To go past it you repartition the input, which means changing the topic, not the app. See [choosing partition counts](https://www.conduktor.io/kafka/kafka-topics-choosing-the-replication-factor-and-partitions-count). > **Threads aren't the parallelism story, tasks are.** A common pattern is to crank `num.stream.threads` to 16 on a topic with 4 partitions and wonder why nothing got faster. Twelve of those threads have no task. Match threads to the tasks available on each instance, and add partitions if you genuinely need more parallel processing. ## The internal consumer and producer A Kafka Streams app is, underneath, an ordinary consumer group plus producers, you just don't write the consumer loop. - **Internal consumers** read your source topics, the repartition topics, and (during recovery) the changelog topics. They belong to one [consumer group](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets) named after your `application.id`. This is why a Streams app shows up in `kafka-consumer-groups.sh` like any other consumer, and why partition assignment and rebalancing apply to it directly. - **Internal producers** write to your sink topics, the repartition topics, and the changelog topics that back state stores. With [exactly-once](https://www.conduktor.io/kafka-streams/exactly-once) enabled, these become transactional producers. - A **restore consumer** is a separate, non-group consumer used only to replay changelogs when a task has to rebuild state on a new instance. Because the whole thing is a consumer group, the same operational levers you know from plain [consumers](https://www.conduktor.io/kafka/kafka-consumers), `max.poll.interval.ms`, `session.timeout.ms`, group coordination, govern your Streams app too. A thread that blocks too long in your processing code misses its poll deadline and triggers a rebalance, the same as any consumer. ## How the graph decides your internal topics You never run `kafka-topics --create` for these, but they appear on your cluster the first time the app starts, prefixed with your `application.id`: - **Repartition topics**, created at each sub-topology boundary where a re-key forces a shuffle. They carry data between sub-topologies. They are not compacted: `cleanup.policy=delete` with infinite retention, and Streams itself purges records once they're consumed. - **Changelog topics**, created for every state store that has logging enabled (the default; `withLoggingDisabled()` skips them), [log-compacted](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction) so a store can be rebuilt after a crash. Windowed stores get `compact,delete` with a retention derived from the window. (Covered in [state stores](https://www.conduktor.io/kafka-streams/state-store).) Two facts that bite later: 1. **The number and partitioning of these topics come straight from the topology graph.** A stateful app with several aggregations and joins can easily triple your topic and partition count on the cluster. Count them before you size the cluster. 2. **Their names are positional, derived from the operator's position in the graph.** An unnamed store or repartition gets a name like `KSTREAM-AGGREGATE-STATE-STORE-0000000001`. Insert or reorder an operator and the positions, and therefore the names, shift, which can orphan existing state on an upgrade. Name them explicitly from day one; one `Materialized.as("agg")` names the store, its changelog, and the related repartition topic. Kafka Streams 4.x warns at build time about unnamed internal resources, and `ensure.explicit.internal.resource.naming` (KIP-1111) turns that into a hard error. This is the entire reason topology changes are risky; see [evolving a topology](https://www.conduktor.io/kafka-streams/topology-evolution). ## Frequently Asked Questions **How does Kafka Streams work under the hood?** You define a topology (a graph of operators); Kafka Streams compiles it into sub-topologies, splits each into tasks bound to input partitions, and runs those tasks on stream threads inside your own JVM process. It is, underneath, an ordinary consumer group plus producers, you just don't write the consumer loop. **Does Kafka Streams need its own cluster to run?** No. Kafka Streams is a library that runs inside your application process, so there is no separate processing cluster, job scheduler, or resource manager to deploy. It only needs a Kafka cluster to read from and write to; you scale it by running more instances. **What is a stream task in Kafka Streams?** A task is the processing for exactly one partition of a sub-topology's input. It owns that partition's records, its own copy of any state stores in the sub-topology, and its own position in the log. The number of tasks equals the input partition count, so it only changes if partitions are added to a topic. **How do tasks, threads, and partitions relate in Kafka Streams?** A task maps to one input partition and is the fixed unit of parallelism; stream threads run the tasks (one thread can run several, in turn). Real concurrency is `min(total tasks, total threads across all instances)`, so the maximum parallelism is the largest partition count among your sub-topologies. **Why does my Kafka Streams app create extra topics on the cluster?** The topology graph decides them. Kafka Streams creates a repartition topic at each boundary where a re-key forces a shuffle, and a compacted changelog topic for every state store with logging enabled (the default). Their names and partition counts come straight from the graph and are prefixed with your `application.id`. > **See it in practice with Conduktor** > A Kafka Streams app is a consumer group plus a set of internal topics. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-architecture) lets you see the repartition and changelog topics your topology created, check their partition counts against what you expected, and watch the consumer group's partition assignment and lag, the signals that tell you whether tasks are balanced and whether a restore is still running after a deploy. ## Next steps - [Scaling Kafka Streams](https://www.conduktor.io/kafka-streams/scaling), turn the task model into real throughput - [Kafka Streams state stores](https://www.conduktor.io/kafka-streams/state-store), the local state each task owns - [Build your first Kafka Streams app](https://www.conduktor.io/kafka-streams/getting-started), a runnable topology in Java Source: https://www.conduktor.io/kafka-streams/architecture --- # Kafka Streams async processing *Understand the per-thread throughput wall external calls create, and the real ways around it.* A Kafka Streams app that just filters and routes records is limited by your network and your CPU. The moment you put a synchronous external call inside the topology, a database read, a REST call, a model inference, the math changes completely, and not in your favor. Throughput stops being about how big your box is and becomes a function of one number: how long that call takes. This is the throughput wall that sends people searching for "Kafka Streams async processing" at 2am, and the honest answer is more nuanced than "add threads". **What you'll learn:** - Why a stream thread processes records serially, and why that makes throughput = 1 / record-latency - Why adding `num.stream.threads` does nothing once you've hit the task ceiling - The three real fixes, partitions, co-located lookups, and true async processing, and when each one applies - What the async pattern has to solve to stay correct (per-key order, exactly-once, offset commits) ## How a stream thread actually spends its time The thing most people get wrong about Kafka Streams parallelism is invisible until you put a slow call in the middle of it. A [stream thread](https://www.conduktor.io/kafka-streams/architecture) processes records for its assigned tasks **serially, one record at a time**, depth-first through the topology. It picks a record, runs it all the way through every processor, `map`, `filter`, `process`, the store writes, the forwards to children, and only *then* picks up the next record. There is no concurrency *within* a task. One record, start to finish, then the next. That design is deliberate and it's what gives you per-key ordering for free. It's also why a blocking call is so destructive. If a processor calls out to DynamoDB and that call takes 10ms, the thread does nothing for those 10ms but wait. It can't get ahead on the next record, because that would break ordering. So the ceiling for that task is brutally simple: ``` throughput per thread ≈ 1 / per-record latency ``` A 10ms call → ~100 records/second per thread. That's the whole story for one thread. (Illustrative arithmetic, not a benchmark, your real number depends on the call, but the *shape* `1/latency` is exact.) It compounds. Add a second external call, say a 90ms model inference after the 10ms lookup, and the per-record latency is now ~100ms, so the same thread does **~10 records/second**. The slow call dominates, and they add up rather than overlap, because everything in a task runs in series. ``` record → [10ms DynamoDB read] → [90ms model call] → emit └──────────── thread idle ~100ms, doing nothing ─────────────┘ ``` ## Why you can't just add threads The instinct is to throw `num.stream.threads` at it. Here's why that hits a wall faster than you'd expect. Kafka Streams parallelism is capped by **partition count**, not by threads. Each partition maps to exactly one task, and a task runs on exactly one thread at a time. So with an 8-partition input topic you get at most 8 tasks, and at most 8 threads doing useful work, whether they live on one big box or eight small ones. Past 8 threads, the rest are idle: there's no task to give them. (Full mental model: [scaling and parallelism](https://www.conduktor.io/kafka-streams/scaling).) Put the two facts together and you get a hard ceiling. 8 partitions × ~9 records/second/thread ≈ **~72 records/second**, app-wide, no matter how much hardware you provision. You cannot buy your way out with a bigger instance, and you cannot buy your way out with more threads, because every thread past the eighth has nothing to do. > 🚫 *"It's slow because of the external call, bump `num.stream.threads` and it'll keep up."* > Threads only help up to the task count (one task per partition of each sub-topology, summed across sub-topologies; the bottleneck stage's useful parallelism is its own sub-topology's partition count). Once `num.stream.threads × instances` ≥ that, every extra thread is idle and the slow call still bottlenecks each active thread at `1/latency`. More threads add CPU cost and zero throughput. The lever is fewer/faster calls or more tasks, not more workers. This is the [blocking-call anti-pattern](https://www.conduktor.io/kafka-streams/anti-patterns) seen from the throughput side. The anti-patterns page warns it stalls a partition and can blow `transaction.timeout.ms`; this page is about the ceiling it imposes and how to lift it. ## Fix 1: more tasks, raise partitions or `repartition(N)` The standard, vendor-neutral lever is to give the slow stage **more tasks** so more threads run in parallel. Two ways: - **More input partitions.** 24 partitions instead of 8 raises the ceiling to 24 × your per-thread rate. This is the cleanest answer, *if you can do it*. On a [stateful app you cannot just bump partition count](https://www.conduktor.io/kafka-streams/scaling): changing the partition count changes `hash(key) % N`, so a key now hashes to a different partition than the one holding its state. The failure is loud, not subtle: on restart Streams sees the changelog topic still has the old partition count and dies with a `StreamsException` ("Existing internal topic ... has invalid partitions") that points you at the `StreamsResetter` tool, which means a reset and full reprocess. The silent-corruption risk is real only if you force past that check, or for stateful consumers outside Streams. Plan partitions for peak up front, or treat an increase as a full reprocess. - **`repartition(N)` before the slow stage.** If the work is stateless (an enrichment call that doesn't touch a store), you can re-partition into a wider internal topic and run the downstream sub-topology with more tasks than the input has partitions: ```java stream .repartition(Repartitioned.as("enrich-fanout") .withNumberOfPartitions(48)) // 48 tasks downstream, regardless of input width .mapValues(order -> enrichWithBlockingCall(order)); ``` This genuinely beats the input-partition ceiling for the downstream stage. It isn't free: you've added an internal topic, extra storage, extra produce/fetch, an extra network hop, and you've widened a [sub-topology boundary](https://www.conduktor.io/kafka-streams/architecture) that you now have to size and watch. But for "I need 6× the parallelism on one enrichment step", it's the legitimate, in-the-box answer, and it's where most teams should start. One footgun: `Repartitioned.as(...)` falls back to the default serdes, so without `default.key.serde`/`default.value.serde` configured this snippet fails at startup ("Failed to initialize key serdes for source node"); pass them explicitly with `Repartitioned.with(keySerde, valueSerde)`. ## Fix 2: don't make the call at all The fastest external call is the one you delete. A huge fraction of "RPC in the topology" cases are *enrichment*: for each event, look up a row and attach it. If that reference data fits in memory and is itself a Kafka topic (or can be one), you don't need a network call, you need a [join](https://www.conduktor.io/kafka-streams/joins). | Pattern | Per-record cost | Fits when | |---|---|---| | Synchronous RPC per record | A network round-trip (the `1/latency` wall) | Data is huge, or lives only in a remote system you can't mirror | | `KTable` join (co-partitioned) | A local store lookup, no network | Reference data is keyed the same as the stream and co-partitioned | | `GlobalKTable` join | A local lookup by *any* field, fully replicated to every instance | Reference data is small enough to replicate to every instance | A `GlobalKTable` is replicated in full to every instance, so any key is a local lookup with no co-partitioning requirement, ideal for smallish, slowly-changing dimension data (currency rates, feature flags, a product catalog). A co-partitioned `KTable` join scales to larger reference data. Either way the lookup is in-process and microsecond-cheap, and your throughput goes back to being bounded by CPU instead of by someone else's API. Replace the call before you parallelize it. See [joins](https://www.conduktor.io/kafka-streams/joins) for the co-partitioning contract that makes this work, and the silent ways it breaks if you get it wrong. This doesn't cover everything. A genuinely large dataset, a system of record you can't mirror into Kafka, or a model you have to call live, those you can't turn into a local lookup. That's where fix 3 comes in. ## Fix 3: the async processing pattern ![Serial processing sends one record at a time; async pattern puts calls for keys A, B, C in flight concurrently while preserving same-key order and draining before each commit](https://www.conduktor.io/assets/kafka-streams/diagrams/async-serial-vs-concurrent.svg) When the call is irreducible, you *must* hit that REST API or that model per record, and raising partitions isn't enough, there's a third option: process records of **different keys** within a single partition concurrently, while preserving same-key order. This is the "async processing" pattern. The intuition: within one partition, records for key `A` and key `B` are independent. There's no ordering requirement *between* them, only *within* each key. So you can have the call for `A` in flight at the same time as the call for `B`, and a third for `C`, using a small worker pool, instead of forcing all of them through one thread in single file. If a partition carries thousands of distinct keys (it usually does), you can have many calls outstanding at once and saturate the external system's concurrency instead of its latency. The ceiling moves from `1/latency` toward `concurrency/latency`. That sounds simple. It is not, and the hard parts are exactly the guarantees Kafka Streams gives you that you must not lose: - **Per-key ordering.** Records for the same key must still be processed in offset order, even though different keys are now interleaved out of order across the worker pool. The pattern needs per-key queues so `A`'s second record never overtakes `A`'s first. - **Exactly-once.** Under [EOS](https://www.conduktor.io/kafka-streams/exactly-once), the writes a record produces are part of a transaction. With calls in flight, you have records that have *started* but not *finished* when a commit comes around, the commit has to drain every in-flight record first, so a transaction never commits a half-processed record's offset. - **Offset-commit correctness.** Streams commits the offset of processed records. If record at offset 100 finishes before offset 95 (different keys, different call latencies), you cannot commit 100, a crash would skip 95 to 99. The committed offset has to track the lowest *un*-finished offset, not the highest finished one. - **Store writes and forwards.** A processor that writes to a state store or forwards downstream can't do that from an arbitrary worker thread, the store and the downstream are owned by the stream thread. So forwards and store writes have to be **buffered** on the worker and **replayed on the stream thread** when the record completes, in a sandboxed processor context. This is genuinely advanced, and today it's mostly delivered by a library rather than core Apache Kafka. **Responsive's Async Processor** is the notable implementation: a worker thread pool per stream thread, per-key queues to hold ordering, and a sandboxed `ProcessorContext` that buffers forwards and store writes for replay on the stream thread, solving exactly the four problems above. It's a [Processor API](https://www.conduktor.io/kafka-streams/processor-api) feature (not the DSL), and it's an SDK addition, not part of the Apache Kafka distribution. Responsive has stated an intent to contribute the pattern upstream; as of writing it is not in core Kafka, so version-gate any plan that depends on it. > **Async changes your commit and transaction settings.** Because a low commit interval forces frequent drains of every in-flight record, an aggressive `commit.interval.ms` cancels much of the async benefit, you spend your time waiting for in-flight calls to finish instead of starting new ones. And under [EOS](https://www.conduktor.io/kafka-streams/exactly-once) the default drops from 30000ms to 100ms, so an exactly-once app starts at the aggressive end without you touching anything. Implementations of this pattern recommend raising the commit interval and setting `transaction.timeout.ms` comfortably above it (commit interval plus headroom), so transactions don't time out while records are in flight. Treat the commit interval as a throughput knob here, not a default. ## Which fix, in what order Reach for these top to bottom. Most teams never need the third. | Fix | Reach for it when | Cost | |---|---|---| | **Co-located lookup** ([join](https://www.conduktor.io/kafka-streams/joins)) | The call is enrichment and the data can live in Kafka | Memory for the table; co-partitioning discipline | | **More tasks** (partitions / [`repartition(N)`](https://www.conduktor.io/kafka-streams/scaling)) | The work is stateless, or you can plan partitions up front | An internal topic + a sub-topology boundary to watch | | **Async processing** (library pattern) | The call is irreducible *and* more tasks still aren't enough | A dependency, Processor API only, careful commit/txn tuning | Try to delete the call first. If you can't, add tasks. Only when the call is truly unavoidable and tasks aren't enough do you take on the complexity, and the library dependency, of true async processing. ## Frequently Asked Questions **Why is my Kafka Streams app slow with external calls?** A stream thread processes its task's records serially, one at a time. A blocking call makes the thread wait per record, so throughput per thread is roughly 1 / call-latency, a 10ms call caps a thread near 100 records/second, regardless of CPU. Multiple calls per record add up rather than overlap. **How do I parallelize Kafka Streams beyond partition count?** Useful parallelism is normally capped at the partition count (one task per partition). To go wider on a stateless stage, insert `repartition(N)` to write to a wider internal topic and run more downstream tasks. Beyond that, the async processing pattern runs records of different keys concurrently within one partition. **Can Kafka Streams process records concurrently?** Not within a task by default, processing is single-threaded per task to preserve per-key ordering. Records of different keys in the same partition can be processed concurrently with the async processing pattern (e.g. Responsive's Async Processor), which keeps per-key order, exactly-once, and offset-commit correctness while calls are in flight. **Should I call a REST API from a Kafka Streams processor?** Avoid a synchronous one if you can. It caps throughput at 1/latency per thread and can blow `transaction.timeout.ms` under exactly-once. Prefer enriching via a co-partitioned KTable or GlobalKTable lookup; if you must call out, use the async pattern and raise the commit interval. **What is the Responsive Async Processor?** A Processor-API feature in the Responsive SDK that processes records of different keys concurrently using a per-stream-thread worker pool, per-key queues for ordering, and a sandboxed context that replays store writes and forwards on the stream thread. It is one implementation of the async pattern, not part of core Apache Kafka. > **See it in practice with Conduktor** > The throughput wall shows up as one unambiguous signal: [consumer group lag](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets) climbing while CPU sits idle. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-async-processing) shows per-partition lag and the consumer group's partition assignment, so you can tell a genuine throughput ceiling (uniform lag, more threads won't help) from a fix you can actually apply, and watch the internal topic a `repartition(N)` creates once you add one. ## Next steps - [Scaling & parallelism](https://www.conduktor.io/kafka-streams/scaling), the task ceiling this page works around, and the partition-increase trap - [Anti-patterns](https://www.conduktor.io/kafka-streams/anti-patterns), why a blocking call in the topology is the most damaging Streams mistake - [Joins](https://www.conduktor.io/kafka-streams/joins), replace an enrichment RPC with a co-partitioned KTable or GlobalKTable lookup Source: https://www.conduktor.io/kafka-streams/async-processing --- # Kafka Streams dead letter queue *Stop poison pills from killing your stream.* A single malformed record can take down a Kafka Streams application. Not degrade it: stop it. The default behavior on a record Streams can't deserialize is to throw, kill the `StreamThread`, and refuse to make progress until you intervene. Worse, because the bad record stays at the same offset, a naive restart reads it again and dies again: a poison-pill loop. Kafka Streams has *three* separate error-handling hooks, and each one fires at a different stage of the read-process-write cycle. Knowing which handler catches what (and where the historical gap was) is the difference between a stream that quarantines bad data and one that pages you at 3am. This page covers all three, the manual dead-letter-queue (DLQ) pattern that fills the gap, and the native support that recently landed. **What you'll learn:** - The three error handlers and exactly where each one fires - Why a poison pill causes a reprocessing loop, not a clean skip - The manual DLQ pattern: route the bad record and its error metadata to a `.DLT` topic - The native processing-exception handler and DLQ (KIP-1033 / KIP-1034), and the version to check ## Three handlers, three stages A Streams record flows through three stages where things break: deserializing the input, running your processing logic, and producing the output. There is a distinct handler for each. | Stage | Handler | Fires on | Default behavior | |---|---|---|---| | **Read** (deserialize) | `DeserializationExceptionHandler` | A record the configured serde can't deserialize (a poison pill) | `LogAndFailExceptionHandler`: log, then **kill the thread** | | **Process** (your logic) | `ProcessingExceptionHandler` (KIP-1033, Kafka 3.9+) | An exception thrown inside `map`, `filter`, a processor, etc. | Historically none: the exception propagated up and killed the thread. Since 3.9: `LogAndFailProcessingExceptionHandler` | | **Write** (produce) | `ProductionExceptionHandler` | The producer fails to write a record (e.g. `RecordTooLargeException`) | `DefaultProductionExceptionHandler`: fail the thread | Mixing these up wastes hours. A `RecordTooLargeException` on output is *not* caught by the deserialization handler. A `NullPointerException` in your `mapValues` was, until recently, not caught by *any* DSL handler. Match the handler to the stage. ### Read-side: the deserialization handler This is the one most people mean by "skip bad records." Kafka ships two implementations: ```properties # Kill the thread on a record the serde can't decode (the DEFAULT) deserialization.exception.handler=org.apache.kafka.streams.errors.LogAndFailExceptionHandler # Log the record and keep going, survive the poison pill deserialization.exception.handler=org.apache.kafka.streams.errors.LogAndContinueExceptionHandler ``` On Kafka 3.x the key is `default.deserialization.exception.handler`; 4.0 deprecated the `default.` prefix (KIP-1056). The old key still works on 4.x, but Kafka's own error messages now point you at the new one. `LogAndContinueExceptionHandler` is the quick fix that keeps the app alive. But read the next section before you reach for it: "continue" silently drops the record, and a dropped record is gone unless you also route it somewhere. > 🚫 *"A bad record just gets skipped, Streams handles it."* By default it does **not** skip. The default deserialization handler is `LogAndFail`, which stops the thread. Switching to `LogAndContinue` makes it skip, but skip means *discard with a log line*, with no copy of the record kept. If that record mattered, you've lost it and you'll find out downstream. ## The poison-pill reprocessing loop ![Offset 4711 is a poison pill: StreamThread reads it, throws DeserializationException, dies before commit, app restarts and reads 4711 again in an infinite loop](https://www.conduktor.io/assets/kafka-streams/diagrams/dlq-poison-pill-loop.svg) Here is the failure mode that surprises people running the default handler. A poison pill sits at, say, offset 4711. The `StreamThread` tries to deserialize it, throws, and dies, **before committing the offset**. Streams either replaces the thread or the app restarts, the consumer resumes from the last committed offset (4711), reads the same poison pill, and dies again. The app flaps in a loop, making zero progress, while the offset never advances past the bad record. This is why `LogAndFail` plus an automatic restarter is the worst of both worlds: it neither stops cleanly nor recovers. You either skip the record (`LogAndContinue`, ideally routing it to a DLQ) or you manually advance the offset past it. There is no version where the loop resolves itself. ## The uncaught-exception handler, and its gotcha Separate from the per-stage handlers, there is an application-level safety net for any exception that nothing else caught: ```java streams.setUncaughtExceptionHandler(exception -> { // Choose what happens after an uncaught exception kills a StreamThread return StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse.REPLACE_THREAD; // or .SHUTDOWN_CLIENT: shut down this instance // or .SHUTDOWN_APPLICATION: shut down every instance with the same application.id }); ``` - `REPLACE_THREAD`: spin up a fresh `StreamThread` and carry on. Fine for transient faults; a *loop* if the cause is a poison pill, because the new thread hits the same record. - `SHUTDOWN_CLIENT`: stop this instance only. - `SHUTDOWN_APPLICATION`: stop the whole application across all instances. The gotcha: this handler runs **once the `StreamThread` has hit a fatal error and stopped processing**. It is invoked on the dying thread as it shuts down, before any replacement is spawned. It is a notification-and-decision hook, not a `try/catch` around your code. It cannot resume the failed record or repair in-flight state; processing on that thread is over and the decision is only "what next." Don't mistake it for in-line error handling inside your topology. ## The historical gap: no handler for business logic Notice what the three stages *didn't* cover for most of Kafka Streams' life: an exception thrown by **your own processing logic**. A `NumberFormatException` parsing a field, a null dereference, a downstream call that throws: none of these were deserialization or production errors. There was no DSL-level hook for them. The exception propagated up, killed the thread, and you were back in restart-loop territory. The workaround the community settled on is the **manual DLQ pattern**: wrap your risky logic in a `try/catch` inside the processor, and on failure, route the offending record (plus enough metadata to debug it) to a dedicated dead-letter topic, conventionally named `.DLT`. ## The manual DLQ pattern The idea: never let a business-logic exception escape the processor. Catch it, emit the original record to a side topic with error context in the headers, and keep the main stream flowing. A `branch`/`split` keeps it readable in the DSL: ```java // Parse risky input; on failure, tag the record instead of throwing KStream parsed = builder.stream("orders", Consumed.with(Serdes.String(), Serdes.String())); Map> branches = parsed .mapValues(DeadLetterExample::tryParseOrTag) // returns OK: or ERR::: .split(Named.as("route-")) .branch((k, v) -> v.startsWith("ERR:"), Branched.as("dead")) .defaultBranch(Branched.as("ok")); // Good records continue down the topology branches.get("route-ok") .mapValues(v -> v.substring(3)) // strip the OK: tag .to("orders-validated", Produced.with(Serdes.String(), Serdes.String())); // Bad records: enrich with error headers, send to the dead-letter topic branches.get("route-dead") .process(() -> new Processor() { private ProcessorContext ctx; public void init(ProcessorContext context) { this.ctx = context; } public void process(Record rec) { String[] parts = rec.value().substring(4).split("::", 2); // message :: raw Record dead = new Record<>(rec.key(), parts.length > 1 ? parts[1] : "", rec.timestamp()); dead.headers() .add("dlq-error", parts[0].getBytes(StandardCharsets.UTF_8)) .add("dlq-source-topic", "orders".getBytes(StandardCharsets.UTF_8)) .add("dlq-ts", Long.toString(System.currentTimeMillis()).getBytes(StandardCharsets.UTF_8)); ctx.forward(dead); } }) .to("orders.DLT", Produced.with(Serdes.String(), Serdes.String())); ``` The headers are the point. A bare copy of the bad value in a `.DLT` topic tells you nothing six weeks later. Record the exception message, the source topic and partition/offset if you have them, and a timestamp, so the dead-letter topic is something you can triage and replay, not just a graveyard. For deserialization failures specifically (which happen *before* your code sees the record), the same routing belongs in a custom `DeserializationExceptionHandler`: it receives the original `ConsumerRecord` with the raw bytes, so it can ship them to the DLQ before telling Streams to keep going. Up to Kafka 4.1 that means producing inside `handle(...)` and returning `CONTINUE`; since 4.2 the method is `handleError(...)` returning `Response.resume()`, and the `Response` itself can carry dead-letter records for Streams to produce for you (KIP-1034). The `handle(...)` variants are deprecated in 4.3 but still functional. ### Retry tier vs dead-letter tier Not every failure is a poison pill. A transient error (a downstream timeout, a momentary dependency blip) usually succeeds on a retry; a malformed record never will. Routing both to the same place sends recoverable records to manual triage and dilutes the DLT's signal. A common refinement splits them into two topics: - **`.retry`** for transient failures. A small daemon consumes it, waits a backoff (1s, 2s, 4s, 8s…), and republishes to the original topic, carrying an attempt count in a header and giving up to the DLT after N attempts. Most failures self-heal with nobody looking. - **`.DLT`** for terminal/poison records. Quarantined for human inspection and explicit reinjection, never auto-retried. This retry-daemon-plus-DLT shape is exactly the one that shows up in [multi-agent AI systems on Kafka](https://www.conduktor.io/kafka-streams/ai-agents), where it matters even more: a non-deterministic worker (an LLM) can fail a record once and succeed on replay, so bounded automatic retries recover far more than they would with deterministic code. ## Native support: KIP-1033 and KIP-1034 The manual pattern was load-bearing for years because the framework had no built-in answer for processing errors. That changed recently. > **Version-gate this.** Native processing-error handling arrived through **KIP-1033** (a `ProcessingExceptionHandler` for exceptions thrown in user processing code), shipped in **Kafka 3.9**; native dead-letter-queue support arrived through **KIP-1034**, shipped in **Kafka 4.2**. Confirm your exact Kafka and Kafka Streams version before depending on either. On older versions the manual pattern above is still the way. Where it's available, **KIP-1033** gives the missing third handler: a `ProcessingExceptionHandler` invoked when your own logic throws, with the same `FAIL` / `CONTINUE` contract as the deserialization handler, so a business-logic exception no longer has to mean a dead thread. **KIP-1034** builds on the handler contracts so a failed record can be routed to a dead-letter topic through configuration (`errors.dead.letter.queue.topic.name`, unset by default) and the handler return value, rather than the hand-rolled `branch`-and-`process` plumbing. The manual pattern still works and still gives you the most control over headers and routing; the native path removes boilerplate once you're on a version that has it. ## Putting it together A robust Streams app usually layers these: - **Read**: a `DeserializationExceptionHandler` that routes poison pills to a DLQ, not bare `LogAndContinue` (so nothing is silently lost). - **Process**: `try/catch` in your processors routing to a `.DLT` topic with error headers, or the KIP-1033 `ProcessingExceptionHandler` if your version has it. - **Write**: a `ProductionExceptionHandler` for produce-side failures you've decided are non-fatal. - **Backstop**: a `StreamsUncaughtExceptionHandler` so a genuinely unexpected exception triggers a deliberate `REPLACE_THREAD` or shutdown, not an accidental flap. And whatever you skip, route it somewhere. A skipped record with no DLQ is a data-loss bug wearing a "handled" label. ## Frequently Asked Questions **How do I handle a bad record or poison pill in Kafka Streams?** The default deserialization handler is `LogAndFailExceptionHandler`, which kills the thread on an undecodable record. Switch to `LogAndContinue` to survive it, but pair it with routing to a dead-letter topic: "continue" silently discards the record with only a log line, so anything that mattered is lost. **What is the difference between LogAndContinue and LogAndFail in Kafka Streams?** `LogAndFailExceptionHandler` (the default) logs the bad record and stops the `StreamThread`; `LogAndContinueExceptionHandler` logs it and skips it so the app keeps running. Skipping discards the record, so route it to a DLQ if you can't afford to lose it. **How do I implement a dead letter queue in Kafka Streams?** Wrap risky logic in a `try/catch` inside a processor and, on failure, forward the offending record to a dedicated topic conventionally named `.DLT`, attaching the exception message, source topic/partition/offset, and a timestamp as headers. The headers are the point: a bare copy of the value tells you nothing weeks later when you try to triage or replay it. **What does the StreamsUncaughtExceptionHandler do and what are REPLACE_THREAD versus SHUTDOWN?** It's an application-level safety net, invoked on the dying `StreamThread` once a fatal error has stopped processing, that decides what happens next: `REPLACE_THREAD` starts a fresh thread, `SHUTDOWN_CLIENT` stops the instance, and `SHUTDOWN_APPLICATION` stops every instance sharing the `application.id`. It is a notification-and-decision hook, not a `try/catch` around your code, and `REPLACE_THREAD` becomes a loop if the cause is a poison pill. **What is the KIP-1033 processing exception handler in Kafka Streams?** KIP-1033 added a `ProcessingExceptionHandler` for exceptions thrown in your own processing code, with the same `FAIL`/`CONTINUE` contract as the deserialization handler, and shipped in Kafka 3.9; native DLQ support followed in KIP-1034, shipped in Kafka 4.2. Confirm your exact version before depending on either. On older versions the manual `branch`-and-`process` DLQ pattern is still the way. > **See it in practice with Conduktor** > A dead-letter topic is only useful if you can read it. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-dead-letter-queue) lets you browse your `.DLT` topics, read the error headers you attached to each bad record, and watch whether the count is growing, so quarantined records get triaged and replayed instead of piling up unseen. It also surfaces the consumer group lag that tells you a poison-pill loop is stalling the app in the first place. ## Next steps - [Kafka Streams serde errors](https://www.conduktor.io/kafka-streams/serdes): the deserialization failures that feed a DLQ - [Processor API & punctuators](https://www.conduktor.io/kafka-streams/processor-api): the low-level hooks the manual DLQ pattern uses - [Deduplication in Kafka Streams](https://www.conduktor.io/kafka-streams/deduplication): handling the duplicate records a reprocessing loop can create Source: https://www.conduktor.io/kafka-streams/dead-letter-queue --- # Kafka Streams deduplication *Build a deduplication operator Kafka Streams doesn't give you.* There is no `dedup()` in the Kafka Streams DSL. No `distinct()`, no `.deduplicate()`. This surprises people, because de-duplicating a stream feels like a primitive operation, and it's a question that comes up constantly. The reason there's no operator is that "duplicate" is a business decision: same key? same payload? same value within five minutes? Kafka can't guess, so it gives you the building blocks and you assemble the operator yourself. This page is that assembly. We'll cover why exactly-once doesn't do this for you, the canonical pattern (a `Processor` plus a windowed state store), a complete code sketch you can adapt, and the layered defense that's usually the right production answer. **What you'll learn:** - Why exactly-once and the idempotent producer don't de-duplicate business records - The canonical pattern: a stateful `Processor` keyed by a dedup id - Why the dedup store must be *windowed* so its state stays bounded - The layered defense: idempotent producer + dedup processor + idempotent sink ## Why exactly-once doesn't solve this The first reflex is "I'll turn on exactly-once and the duplicates go away." They won't, and it's worth being precise about why before writing any code. [Exactly-once](https://www.conduktor.io/kafka-streams/exactly-once) makes the consume → process → produce cycle inside Kafka atomic. It guarantees each *input record* is processed once even across retries and crashes. What it cannot do is recognize that two **different** input records (two distinct offsets) represent the same logical event. To Kafka, those are simply two records, and EOS faithfully processes both. The [idempotent producer](https://www.conduktor.io/kafka/idempotent-kafka-producer) has the same blind spot from the other direction. It de-duplicates retries *from a single producer session*, using a producer id and sequence numbers. It does nothing about the same event sent by two different producers, or the same event sent again after the producer restarted (new producer id, sequence reset), or a batch job that ran twice. So the duplicates that need this pattern are the ones that arrive as genuinely separate records: - An upstream service retried at the application layer and published the event twice. - A producer crashed and replayed from its own source, re-emitting events. - Two source systems feed the same topic and overlap. - A backfill or reprocessing job re-published historical events. None of these are "the cycle ran twice." They're "the data contains two copies." That's a stateful problem: to know a record is a duplicate, you have to *remember* the ones you've already seen. ## The canonical pattern: remember keys, emit on first sight The pattern is the same one a maintainer will point you to on the mailing list, because it's the only one that works without a remote lookup: 1. Pick a **dedup key**: the field that defines "same event." Often an event id or idempotency key carried in the payload, not the Kafka record key. 2. Keep a **state store** of keys you've already emitted. 3. On each record, check the store. **Not seen → forward it and record the key. Already seen → drop it.** 4. **Bound the store in time** so it doesn't grow forever. Steps 1–3 are straightforward. Step 4 is the one that separates a pattern that works in production from one that quietly fills your disk, and it's why this needs the [Processor API](https://www.conduktor.io/kafka-streams/processor-api), not the DSL. ### Why the store must be windowed ![WindowStore keeps event-A for 1 hour: a duplicate at t=20min is dropped, a copy at t=75min passes again after the entry is purged](https://www.conduktor.io/assets/kafka-streams/diagrams/deduplication-windowed-store.svg) If you back dedup with a plain `KeyValueStore`, you remember every key you've *ever* seen, forever. For a high-cardinality stream (order ids, click events, payment references) that store grows without bound. The backing changelog topic grows with it. Eventually the disk fills or the [state restore](https://www.conduktor.io/kafka-streams/state-store) after a rebalance takes minutes because there's so much to replay. Deduplication is almost always **time-bounded** in practice: "the same event won't legitimately reappear after N minutes/hours." So you only need to remember keys for that window. The right tool is a `WindowStore`, which retains entries only for a configured retention period and drops the rest: ```java import org.apache.kafka.streams.state.Stores; import org.apache.kafka.streams.state.WindowBytesStoreSupplier; import java.time.Duration; Duration dedupWindow = Duration.ofHours(1); // how long a key is "remembered" WindowBytesStoreSupplier supplier = Stores.persistentWindowStore( "dedup-store", dedupWindow, // retention period: entries older than this are purged dedupWindow, // window size false); // retainDuplicates = false: one entry per key per window ``` A window store gives you time-bounded state with purging handled by the store itself: you don't have to track expiry manually. The bound extends to the changelog: Kafka Streams creates a window store's changelog with `cleanup.policy=compact,delete` and `retention.ms` set to the store retention plus a 24-hour buffer (`windowstore.changelog.additional.retention.ms`), so the 1-hour store above gets a 25-hour changelog. A `KeyValueStore` changelog is compact-only and never expires by time. (You *can* purge a `KeyValueStore` by hand with a punctuator instead; the window store just makes it the store's job.) ### The Processor Here's a complete dedup processor using the modern `process()` API. It forwards a record the first time it sees the dedup key within the window, and drops it on any subsequent sighting: ```java import org.apache.kafka.streams.processor.api.Processor; import org.apache.kafka.streams.processor.api.ProcessorContext; import org.apache.kafka.streams.processor.api.Record; import org.apache.kafka.streams.state.WindowStore; import org.apache.kafka.streams.state.WindowStoreIterator; import java.time.Duration; public class DeduplicationProcessor implements Processor { private final Duration window; private final java.util.function.Function dedupKeyExtractor; private ProcessorContext context; private WindowStore seen; public DeduplicationProcessor(Duration window, java.util.function.Function dedupKeyExtractor) { this.window = window; this.dedupKeyExtractor = dedupKeyExtractor; } @Override public void init(ProcessorContext context) { this.context = context; this.seen = context.getStateStore("dedup-store"); } @Override public void process(Record record) { String dedupKey = dedupKeyExtractor.apply(record.value()); if (dedupKey == null) { context.forward(record); // no dedup id: pass through rather than drop return; } long now = record.timestamp(); long from = now - window.toMillis(); boolean alreadySeen = false; try (WindowStoreIterator it = seen.fetch(dedupKey, from, now)) { alreadySeen = it.hasNext(); } if (!alreadySeen) { seen.put(dedupKey, now, now); // record this sighting at its event time context.forward(record); // first time within the window → emit } // else: duplicate within the window → drop silently } } ``` Wiring it into a topology, registering the store so Kafka Streams creates its changelog: ```java import org.apache.kafka.common.serialization.Serdes; import org.apache.kafka.streams.StreamsBuilder; import org.apache.kafka.streams.state.StoreBuilder; import org.apache.kafka.streams.state.Stores; StoreBuilder> storeBuilder = Stores.windowStoreBuilder(supplier, Serdes.String(), Serdes.Long()); StreamsBuilder builder = new StreamsBuilder(); builder.addStateStore(storeBuilder); builder.stream("orders") .process(() -> new DeduplicationProcessor<>( Duration.ofHours(1), order -> order.eventId()), "dedup-store") .to("orders-deduplicated"); ``` Two details that matter: - **`fetch(key, from, now)`** scans the window for a prior sighting. If the iterator has any entry, you've seen this key inside the window: drop. Using event time (`record.timestamp()`) rather than wall-clock means the dedup window tracks *when events happened*, consistent with how the rest of Kafka Streams reasons about time. - The `WindowStore` purges entries older than its retention period on its own. If you'd rather use a plain `KeyValueStore`, schedule a punctuator to evict old keys yourself: ```java // Alternative eviction if you back dedup with a KeyValueStore instead of a WindowStore context.schedule(Duration.ofMinutes(5), PunctuationType.STREAM_TIME, timestamp -> { try (KeyValueIterator all = kvStore.all()) { while (all.hasNext()) { KeyValue entry = all.next(); if (timestamp - entry.value > window.toMillis()) { kvStore.delete(entry.key); } } } }); ``` Note `PunctuationType.STREAM_TIME`: it advances with record timestamps, so on an idle partition it won't fire: same stream-time behavior that trips up windowing and `suppress`. It also fires immediately on the first record processed, then on each interval boundary, so don't write eviction logic that assumes the first run comes only after five minutes. If you need eviction to run on a wall-clock cadence regardless of traffic, use `PunctuationType.WALL_CLOCK_TIME` instead, and accept that it evicts by processing time rather than event time. ## Layered defense: where dedup actually belongs A single dedup processor in the middle of a topology is rarely the whole answer. The robust production posture is **layers**, each catching what the previous one can't: | Layer | Catches | Doesn't catch | |---|---|---| | [Idempotent producer](https://www.conduktor.io/kafka/idempotent-kafka-producer) (`enable.idempotence=true`) | Retry duplicates within one producer session | Cross-producer, cross-restart, re-published events | | Dedup processor (this page) | Genuinely-duplicate records by business key, within a time window | Duplicates older than the window; non-Kafka side effects | | Idempotent sink (upsert keyed by business id) | Whatever slips past the above, at the final write | Nothing further: it's the last line | The idempotent producer is free correctness: turn it on always (it's the producer client default since Kafka 3.0; older clients don't get it even against a modern broker). The idempotent sink is often the *most* important layer, because it's the one that protects the system of record no matter what happened upstream. The dedup processor sits in the middle for cases where you need a clean de-duplicated *topic* (other consumers read it), or where the sink genuinely can't be made idempotent. > **Do you actually need a dedup processor?** Before adding stateful dedup, check whether a cheaper layer already covers you. If duplicates only come from producer retries, `enable.idempotence=true` is enough: no state store, no operator. If your only concern is the final write, an idempotent sink (an upsert keyed by a business id) is simpler and more robust than a mid-pipeline processor, and it survives duplicates the dedup window would have missed. Reach for the stateful dedup processor specifically when you need a de-duplicated *topic* that downstream consumers read, or when the same event genuinely arrives as separate records and the sink can't absorb them. A state store is real operational weight (disk, restore time, a changelog), so add it deliberately, not reflexively. ## Choosing the window and the key Two decisions determine whether this works: - **The dedup key.** It must be stable across the duplicate copies: an event id or idempotency key assigned at the *source*, carried in the payload. A key generated inside your pipeline (a fresh UUID, a timestamp) defeats the whole thing, because each copy gets a different one. If the upstream doesn't provide a stable id, that's the first thing to fix; dedup can't work without one. - **The window length.** Long enough to cover the realistic gap between duplicate arrivals (a producer retry storm, a backfill that overlaps live traffic), short enough to keep the store bounded. If duplicates can legitimately arrive days apart, a windowed in-stream dedup is the wrong tool: push the guarantee to an idempotent sink with a permanent unique constraint instead. ## Frequently Asked Questions **How do I deduplicate records in Kafka Streams?** There's no `dedup()` operator, so you build one with the Processor API: pick a stable dedup key from the payload, keep already-seen keys in a state store, and on each record forward it on first sight and drop it on any later sighting within a time window. Back the store with a `WindowStore` so old keys are purged and the state stays bounded. **Does exactly-once already deduplicate records for me?** No. Exactly-once makes the consume-process-produce cycle atomic so each *input record* is processed once, but it cannot tell that two distinct records at two offsets represent the same logical event. The idempotent producer has the same blind spot: it only dedupes retries within one producer session, not the same event from two producers or after a restart. **Why must the Kafka Streams dedup store be windowed?** A plain `KeyValueStore` remembers every key forever, so for a high-cardinality stream it grows without bound, its changelog grows with it, and state restore after a rebalance slows down. A `WindowStore` retains keys only for a configured retention period and drops the rest, which matches how dedup works in practice: the same event won't legitimately reappear after N minutes or hours. **How do I choose the dedup key for Kafka Streams deduplication?** Use a key that is stable across the duplicate copies: an event id or idempotency key assigned at the *source* and carried in the payload, not the Kafka record key. A key generated inside your pipeline (a fresh UUID or timestamp) defeats dedup because each copy gets a different one; if the upstream doesn't provide a stable id, fix that first. **Where should deduplication live in a Kafka Streams pipeline?** Use layers: the idempotent producer catches retry duplicates within one session, the dedup processor catches genuinely-duplicate records by business key within a window, and an idempotent sink (an upsert keyed by business id) is the last line that protects the system of record. Reach for the stateful dedup processor specifically when you need a de-duplicated *topic* that other consumers read, or when the sink can't be made idempotent. > **See it in practice with Conduktor** > A dedup processor is backed by a real state store with a real changelog topic, and the duplicates you're chasing live in real topics. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-deduplication) lets you inspect the input and output topics to confirm the processor is actually collapsing duplicates, watch the size and compaction of the dedup store's changelog, and monitor consumer group lag and restore progress after a rebalance, so you can tell whether the dedup window is sized right or quietly growing without bound. ## Next steps - [Kafka Streams Processor API](https://www.conduktor.io/kafka-streams/processor-api): `process()`, punctuators, and manual state store access - [Why you still see duplicates](https://www.conduktor.io/kafka-streams/exactly-once-duplicates): when EOS is on but duplicates persist - [State stores](https://www.conduktor.io/kafka-streams/state-store): RocksDB, changelogs, and bounding the state this pattern creates Source: https://www.conduktor.io/kafka-streams/deduplication --- # Kafka Streams exactly-once but still seeing duplicates *Diagnose duplicates that survive exactly_once_v2.* You set `processing.guarantee=exactly_once_v2`, redeployed, and there are still duplicate records downstream. This is the most-pasted Kafka Streams support thread in existence, and the frustrating part is that the flag is almost always doing exactly what it promises. The duplicates are real, but they're coming from somewhere the Kafka transaction was never able to reach. This page assumes you already know what exactly-once *is*. If not, start with [exactly-once](https://www.conduktor.io/kafka-streams/exactly-once). Here we go straight to the failure modes: the four ways duplicates slip past a correctly-configured EOS app, the alarming-but-usually-benign errors in your logs, and the upgrade hazard that produced *genuine* duplicate processing on real clusters. **What you'll learn:** - The four real causes of duplicates with EOS switched on, each with a fix - Why `InvalidProducerEpochException` and "task may be migrated out" are usually noise - The version-upgrade path that caused actual duplicate processing - The broker config EOS needs, and why `initTransactions` hangs without it > 🚫 *"exactly_once_v2 removes every duplicate record everywhere in my pipeline."* Exactly-once makes the **consume → process → produce cycle inside Kafka** atomic. That is the entire scope. It does not reach your database, your REST calls, your downstream consumers' config, or two genuinely-distinct records that happen to mean the same thing. Every cause below is a place that boundary doesn't cover. ![EOS transaction scope: the Kafka transaction boundary wraps only consume-process-produce inside Kafka; JDBC inserts, REST calls, and read_uncommitted downstream consumers are outside and can duplicate](https://www.conduktor.io/assets/kafka-streams/diagrams/eos-transaction-scope.svg) ## Cause 1: a downstream consumer is reading uncommitted records This is the single most common one, and it's not even a bug in your Streams app. When EOS is on, your app writes output inside a Kafka transaction. Records from an *aborted* transaction physically exist in the partition (they were written, then marked aborted) and they stay there until compaction or retention removes them. Whether a consumer *sees* them is decided entirely by that consumer's `isolation.level`: | `isolation.level` | What it reads | |---|---| | `read_uncommitted` (the consumer default) | Every record, including ones from aborted and in-flight transactions | | `read_committed` | Only records from committed transactions | The default for a plain Kafka consumer is `read_uncommitted`. So a *correctly* producing EOS Streams app can write a clean, de-duplicated output topic, and a downstream service reading that topic with default settings will still see the aborted attempts, which look exactly like duplicates. The fix is one line on the **downstream** consumer: ```properties isolation.level=read_committed ``` This applies to every reader of an EOS-produced topic: a second Streams app (it sets `read_committed` for you when *it* runs EOS, but not otherwise), a Kafka Connect sink, a microservice, an ad-hoc console consumer you're using to debug. If you're verifying "did EOS work?" with `kafka-console-consumer` and not passing `--isolation-level read_committed`, you are looking at the aborted records and concluding EOS is broken. It isn't; your consumer is. ## Cause 2: the external sink isn't idempotent EOS covers Kafka reads, Kafka writes, and Kafka offset commits, as one transaction. It cannot enroll anything that isn't Kafka. So if your `process()` or `foreach()` does this: ```java // Inside the topology: NOT covered by the Kafka transaction .foreach((key, order) -> jdbc.execute( "INSERT INTO orders(id, total) VALUES (?, ?)", order.id(), order.total())); ``` …then when a batch aborts and reprocesses (a crash, a rebalance, a transient broker error, all normal), that `INSERT` runs a second time. The Kafka side rolls back cleanly; the row in your database does not. You get a duplicate row with EOS reporting success, because from Kafka's point of view nothing went wrong. The same applies to every non-Kafka effect: an HTTP `POST` to a payment API, an email, a webhook, a write to a cache or search index. The transaction aborts; the side effect already happened. The fix is to make the *sink itself* absorb the duplicate, keyed by something deterministic from the record (not a generated UUID or a timestamp): ```sql -- Idempotent write: reprocessing the same record is a no-op INSERT INTO orders (id, total) VALUES (?, ?) ON CONFLICT (id) DO UPDATE SET total = EXCLUDED.total; ``` For a REST call, prefer an endpoint that takes an idempotency key (most payment and messaging providers support this) and derive that key from the record's business id. The general rule: **EOS is effect-once for Kafka; for everything else, you make the edge idempotent.** This is also why many teams skip EOS entirely and run at-least-once with an idempotent sink: same result at the boundary that matters, lower cost. That trade-off is covered on the [exactly-once](https://www.conduktor.io/kafka-streams/exactly-once) page. ## Cause 3: non-deterministic processing logic EOS quietly assumes that reprocessing the same input produces the same output. That assumption is what lets it abort an attempt and retry it as if the first never happened. Break the assumption and you get duplicates that look impossible: two *different* output records for one input. The usual culprits live inside the topology: - **Reading the wall clock**: `System.currentTimeMillis()`, `Instant.now()`. The retry sees a later time, emits a different value. - **Random or UUID generation**: `UUID.randomUUID()` as a record key or field. Every reprocess produces a new id, so a downstream dedup keyed on it sees two distinct records. - **External mutable state**: a lookup against a database or cache that changed between the first attempt and the retry. After an abort-and-retry, the first attempt's output is invisible to `read_committed` readers, *if it was deterministic*. If your logic produced value A on the first try and value B on the retry, only B is committed, but if A leaked through a non-Kafka side effect (cause 2) or you generated a fresh key, you now have two records in play. The fix is to make the logic deterministic: derive keys and timestamps from the *record* (event-time carried in the payload, a business id from the value), never from ambient state. ## Cause 4: the records are genuinely distinct; EOS was never going to help This is the case people miss because it's not a malfunction at all. If an upstream producer publishes the same logical event **twice** (two separate records, two offsets, perhaps because *its* producer retried without [idempotence](https://www.conduktor.io/kafka/idempotent-kafka-producer), or a batch job ran twice), Kafka Streams sees two distinct inputs and faithfully processes both. EOS guarantees each input is handled exactly once. It has no idea the two inputs are "the same event"; they aren't, as far as Kafka is concerned. Two committed outputs, both correct, both unwanted. No transaction setting fixes this, because nothing went wrong inside the cycle. De-duplicating by a *business key* (across producers, across restarts, across a time window) is a separate problem that needs a stateful dedup operator backed by a state store. That pattern is its own page: [deduplication](https://www.conduktor.io/kafka-streams/deduplication). The quick triage for cause 4 vs the others: consume the **input** topic with `--isolation-level read_committed` and look for the duplicate there. If it's already in the input, EOS on your app was never going to remove it: fix it upstream or add a dedup step. If the input is clean and only the output duplicates, you're looking at causes 1–3. ## The alarming errors that usually aren't a problem EOS apps log a family of exceptions that look catastrophic and are, most of the time, the system working as designed: - `InvalidProducerEpochException` - `ProducerFencedException` - `Producer attempted to produce with an old epoch` - `the producer is fenced, indicating the task may be migrated out` Here's the mechanism. Under `exactly_once_v2` the transactional producer lives per **stream thread**, not per task, and fencing works through consumer group metadata ([KIP-447](https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics)): every transactional offset commit carries the thread's group member and generation ids. After a rebalance moves tasks to another instance, a stale instance trying to commit a lingering transaction fails that generation check (and the transaction coordinator has since bumped the producer epoch), so the broker **fences** it and rejects the write. That rejection surfaces as one of the exceptions above. This is the fencing mechanism that *prevents* the zombie old instance from writing duplicate output. It firing means EOS is protecting you. Kafka Streams treats these as recoverable: the affected task is reset, the transaction is aborted, and processing resumes after the rebalance settles. You'll see the error, then normal operation. Tightened transaction handling in [KIP-890](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense) (recent Kafka releases) makes this fencing more robust against the edge cases where a hung or partially-committed transaction used to slip through. When is it a *real* problem? When it's not transient: a tight loop of fence-rebalance-fence that never settles. That's not a transaction bug; it's a rebalancing problem (sessions timing out, `max.poll.interval.ms` too low, pods churning) wearing an EOS error message. Diagnose it as a rebalance: check how often the group is rebalancing and why, not the transaction config. Repeated fencing is a symptom, not the disease. ## The upgrade hazard: when duplicates were real Everything above is "EOS is fine, look elsewhere." There is one class of exception worth flagging honestly: **specific version upgrades have caused genuine duplicate processing**, not the illusory kind. The notable case involved the migration from the original `exactly_once` (v1) to `exactly_once_v2`, and certain broker/client version combinations across the 2.x-to-3.x range. The migration to v2 is deliberately a **two-rolling-bounce** procedure precisely because v1 and v2 use incompatible transactional semantics, and during the transition a mishandled bounce could reprocess records. This isn't a reason to avoid v2 (v2 is correct and you want it), it's a reason to do the upgrade by the documented steps: > **Version note.** `exactly_once_v2` needs Kafka brokers on 2.5+ and a Streams app on 2.6+ (the literal `exactly_once_v2` config value arrived in 3.0; 2.6–2.8 spelled it `exactly_once_beta`). EOS v1 was removed in Kafka 4.0. Migrating an app from v1 to v2 is a **two-phase rolling upgrade**: first bounce every instance onto a 3.0+ build still running `exactly_once`, then bounce again switching to `exactly_once_v2`. Skipping the intermediate bounce, or mixing v1 and v2 instances in one group for an extended window, is the path that has produced real duplicates. Check the upgrade guide for your exact source and target versions before you start. If you saw a burst of duplicates *during* an upgrade and they stopped once every instance was on the same version, that was the migration, not your topology. If duplicates persist after the dust settles, you're back to causes 1–4. ## The broker config EOS depends on EOS doesn't run on the app alone. Transactions live in an internal topic, `__transaction_state`, and that topic has durability requirements that bite when a broker is down. The key setting is `transaction.state.log.min.isr` (broker config, default 2), alongside `transaction.state.log.replication.factor` (default 3). For transactions to commit, enough replicas of the transaction-state partitions must be in-sync. On a small or degraded cluster this is a real constraint: ```properties # Broker side: transaction state durability transaction.state.log.replication.factor=3 transaction.state.log.min.isr=2 ``` The symptom when it's wrong is unmistakable: a Streams app starting up with EOS **gets stuck on `initTransactions()`** and never begins processing. `initTransactions()` blocks until it can find the transaction coordinator, timing out after `max.block.ms` (60 seconds by default). Streams catches the timeout and retries on every loop while the app sits in **REBALANCING**, never reaching RUNNING; after `task.timeout.ms` (5 minutes by default) the thread gives up with `Task did not make progress within 300000 ms` and the app goes to ERROR. For those six minutes it looks like a hang, and a restart loop repeats it indefinitely. People read this as "Streams is stuck on startup" and look at their topology; the actual cause is a broker outage starving the transaction log. The practical implications: - **A single-broker or RF-1 dev cluster cannot run EOS** with default transaction-state settings: `__transaction_state` is never even created (internal topic creation fails until the cluster meets the replication factor), so `initTransactions` never finds a coordinator and times out. Either lower `transaction.state.log.replication.factor` for local dev or run a 3-broker setup. - **Losing one broker on a 3-broker cluster** with `min.isr=2` can stall transactions if it drops the relevant partition's ISR to 1. EOS trades some availability for its guarantee. - **A hung `initTransactions` on deploy** after a broker incident is a cluster-health problem, not an app problem. Check broker and `__transaction_state` health first. ## Frequently Asked Questions **Why am I still seeing duplicates with exactly_once_v2 enabled?** The flag is almost always working; exactly-once only makes the consume-process-produce cycle inside Kafka atomic. Duplicates come from outside that boundary: a downstream consumer reading uncommitted records, a non-idempotent external sink, non-deterministic processing logic, or genuinely distinct records an upstream producer published twice. **Why do my downstream consumers see records that look like duplicates?** A consumer's default `isolation.level` is `read_uncommitted`, so it reads records from aborted transactions that physically exist in the partition until compaction or retention removes them. Set `isolation.level=read_committed` on every reader of an EOS-produced topic, including any `kafka-console-consumer` you use to verify it. **What causes ProducerFencedException or InvalidProducerEpochException?** Under exactly_once_v2 the transactional producer is per stream thread, and fencing works through consumer group metadata: a stale instance that tries to commit a lingering transaction after a rebalance fails the group generation check and the broker fences it. This is the mechanism that prevents a zombie instance from writing duplicates: it firing means EOS is protecting you, and Kafka Streams treats it as recoverable. **Why does my Kafka Streams app hang on startup with exactly-once?** EOS stores transactions in the internal `__transaction_state` topic, and `initTransactions()` blocks until the transaction coordinator is found, timing out after `max.block.ms` (60s default). A single-broker or RF-1 dev cluster, or a broker outage that drops the ISR below `transaction.state.log.min.isr`, starves the transaction log: Streams retries while stuck in REBALANCING, then fails to ERROR after `task.timeout.ms` (5 minutes default), a cluster-health problem, not a topology one. **Does exactly-once deduplicate records produced twice by an upstream producer?** No. Two separate records with two offsets are distinct inputs as far as Kafka is concerned, and EOS faithfully processes each exactly once. De-duplicating by a business key needs a stateful dedup operator backed by a state store; triage it by consuming the input topic with `read_committed` to see if the duplicate is already there. > **See it in practice with Conduktor** > Most "EOS still duplicates" investigations are answered by looking at the cluster, not the code. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-exactly-once-duplicates) lets you consume a topic with `read_committed` to confirm whether duplicates are real or just aborted records a `read_uncommitted` reader is seeing, watch how often the app's consumer group is rebalancing (the source of repeated fencing errors), and check broker and internal-topic health when `initTransactions` hangs on startup: the signals that tell you whether the problem is your topology or the Kafka underneath it. ## Next steps - [Kafka Streams exactly-once](https://www.conduktor.io/kafka-streams/exactly-once): what `exactly_once_v2` actually guarantees, and its scope - [Deduplication patterns](https://www.conduktor.io/kafka-streams/deduplication): removing genuinely-duplicate records by business key - [Dead letter queues in Kafka Streams](https://www.conduktor.io/kafka-streams/dead-letter-queue): handling the bad records that abort batches in the first place Source: https://www.conduktor.io/kafka-streams/exactly-once-duplicates --- # Kafka Streams exactly-once *Understand what exactly-once actually guarantees (and what it doesn't).* Exactly-once is the feature people cite as the reason to use Kafka Streams, and also the one they most often misread. In Kafka Streams it is genuinely one config flag, and it genuinely works, for a precisely-scoped problem. The trouble starts when "exactly-once" is heard as "duplicates can never happen anywhere", because that is not what it means, and believing it leads to duplicates in production with the flag switched on. This page is the honest version: what the guarantee covers, what it deliberately does not, and why a lot of experienced teams choose at-least-once instead. **What you'll learn:** - What `exactly_once_v2` turns on, and what it builds upon - The exact scope of the guarantee: the consume-process-produce cycle inside Kafka - Why external side effects (databases, REST calls) are not covered - When the performance cost makes at-least-once the better call ## One flag Exactly-once in Kafka Streams is a single setting: ```properties processing.guarantee=exactly_once_v2 ``` That's it. The mode behind `exactly_once_v2` has been the recommended choice since Kafka 2.6, where it shipped under the name `exactly_once_beta` (KIP-732 renamed it in 3.0). It replaces the original `exactly_once` (v1), which needed one producer per input partition to achieve the same result. EOS v1 was deprecated in Kafka 3.0 and removed in Kafka 4.0, so on any supported cluster v2 is the one you want: same guarantee, dramatically lower overhead. > **Version note.** On Kafka 3.0+ clients, use `exactly_once_v2`. On 2.6–2.8 clients the same mode was named `exactly_once_beta`; setting `exactly_once_v2` there throws a `ConfigException`. The mode needs brokers on 2.5+. The older `exactly_once` (v1) was deprecated in 3.0 and removed in 4.0. On 2.6–3.x it still runs but is significantly more resource-hungry, and on 4.0+ `exactly_once_v2` is the only accepted value. There is no reason to choose v1. Behind that flag, Kafka Streams wires together three mechanisms you could assemble by hand but really shouldn't: - **The [idempotent producer](https://www.conduktor.io/kafka/idempotent-kafka-producer)**, so retries inside the cycle don't write a record twice to a Kafka partition. - **Kafka transactions**, so the records produced *and* the consumer offsets committed for a batch are written atomically, all-or-nothing. - **`read_committed` isolation** on the consume side, so the application only reads records from committed transactions, never from ones that were aborted and rolled back. ## What "exactly-once" actually means here ![EOS commit protocol: happy path commits input read, output produce, and offset advance atomically; crash before commit causes broker abort and clean replay from the same offset](https://www.conduktor.io/assets/kafka-streams/diagrams/eos-commit-protocol.svg) The guarantee has a precise boundary, and naming it correctly is the whole point of this page. Kafka Streams makes the **consume → process → produce cycle, inside Kafka, atomic.** For each batch, three things happen as one unit: 1. it reads input records, 2. it produces output records to Kafka topics (including internal changelog and repartition topics), 3. it commits the input offsets. Either all three commit together, or none do. If the application crashes mid-batch, the transaction aborts: the output records are marked aborted (they stay in the log physically; `read_committed` readers skip them client-side), the offsets are not advanced, and on restart the batch is reprocessed cleanly. The net effect (the records that land in the output topics and the state that ends up in your stores) is **as if each input record were processed exactly once.** That last phrase is the accurate one: **effect-once**, not "the bytes physically appear once and are never retried under the hood." Internally, records *are* re-read and reprocessed after a failure. What exactly-once removes is the *observable duplicate*: the aborted attempt is invisible to a correct reader, so downstream sees each result a single time. > 🚫 *"Exactly-once means a record can never be duplicated anywhere in my pipeline."* The guarantee is scoped to Kafka-to-Kafka processing within the Streams application. It says nothing about duplicates produced *outside* that boundary, and assuming otherwise is the single most common way people get burned by it. ## What it does not cover: external side effects The transaction is a *Kafka* transaction. It spans Kafka reads, Kafka writes, and Kafka offset commits. It cannot span anything that is not Kafka. So if your processing logic does any of the following, exactly-once does **not** make it exactly-once: - **Writing to a database.** A `INSERT` you issue inside a `process()` call is not part of the Kafka transaction. If the batch aborts and reprocesses, that insert runs again. You get a duplicate row unless the write is idempotent (an upsert keyed by something deterministic). - **Calling a REST API.** An HTTP `POST` to a payment provider, an email send, a webhook: none of these roll back when the Kafka transaction aborts. The side effect already happened. - **Anything non-deterministic in your logic.** Exactly-once assumes reprocessing the same input yields the same output. If your code reads the wall clock, calls a random generator, or depends on external mutable state, reprocessing after an abort can produce a *different* result, and now you have two different outputs for one input. This is why "I turned on `exactly_once_v2` and I still see duplicates" is one of the most common Kafka Streams support threads. The flag is usually working exactly as designed; the duplicates are coming from a non-idempotent sink, a downstream consumer reading `read_uncommitted`, or non-deterministic processing, none of which the Kafka transaction can govern. The full debugging checklist is in [why you still see duplicates](https://www.conduktor.io/kafka-streams/exactly-once-duplicates). ## Exactly-once is not physical de-duplication A related misconception: exactly-once will *not* collapse two genuinely-different input records that happen to mean the same thing. If the same logical event is produced **twice by an upstream producer** (two separate records, two offsets), Kafka Streams sees two distinct inputs and faithfully processes both. The transaction guarantees each is handled once; it does not know they are "the same". De-duplicating across producers, across restarts, or by a business key is a *different* problem, and it needs a different tool: a stateful dedup operator backed by a state store, an idempotent sink keyed by a business id, or both. See [deduplication](https://www.conduktor.io/kafka-streams/deduplication) for the pattern. Exactly-once de-duplicates the *cycle*; it does not de-duplicate your *data*. ## The cost, and why at-least-once is a fair choice Exactly-once is not free, and the trade-off is real enough that plenty of seasoned teams deliberately do not use it. | | At-least-once (default) | Exactly-once (`exactly_once_v2`) | |---|---|---| | Duplicates in output | Possible after a failure | None observable (within Kafka) | | Consume isolation | `read_uncommitted` | `read_committed` | | Commit cadence | Offsets committed periodically | Transaction per commit interval | | Latency | Lower | Higher: readers wait for commit | | Overhead | Minimal | Transaction coordination on every cycle | Two costs dominate. First, `read_committed` adds **end-to-end latency**: a downstream consumer cannot see records from a transaction until that transaction commits, so output appears in bursts aligned to the commit interval rather than continuously. Second, every cycle carries **transaction coordination** overhead (begin, produce, commit offsets, commit transaction) which throttles throughput compared to fire-and-forget at-least-once. One default softens the first cost: enabling exactly-once silently drops the default `commit.interval.ms` from 30000 ms to 100 ms, which keeps the bursts sub-second out of the box. It also means an exactly-once app commits 300x more often than an at-least-once one unless you tune the interval. The pragmatic alternative is **at-least-once plus an idempotent sink.** Run the default `at_least_once`, accept that a failure may reprocess a batch, and make the *final* write absorb the duplicate: an upsert keyed by a deterministic id, a `INSERT ... ON CONFLICT DO NOTHING`, or a downstream dedup keyed by a business id. This pushes the correctness guarantee to the edge of the system where it actually matters, avoids the per-cycle transaction tax, and is often simpler to reason about than convincing yourself every link in the chain honors the transaction. The broader framing (at-most-once, at-least-once, exactly-once and how offset commits decide which one you get) is in [delivery semantics for Kafka consumers](https://www.conduktor.io/kafka/delivery-semantics-for-kafka-consumers). Exactly-once is the strongest option, not the default one; choose it when the consume-process-produce cycle inside Kafka is genuinely where your duplicates come from, and reach for an idempotent sink when they come from anywhere else. ## Frequently Asked Questions **Is Kafka Streams exactly-once real?** Yes, but it is scoped: Kafka Streams makes the consume-process-produce cycle inside Kafka atomic, so the records that land in output topics and the state in your stores are as if each input were processed exactly once. It is effect-once: failed attempts are re-read internally but the aborted output is invisible to a correct reader, not "bytes never retried." **How do I enable exactly-once in Kafka Streams?** Set the single config `processing.guarantee=exactly_once_v2`. Behind it, Kafka Streams wires together the idempotent producer, Kafka transactions that commit output records and consumer offsets atomically, and `read_committed` isolation on the consume side. **What is the difference between exactly_once and exactly_once_v2?** `exactly_once_v2` gives the same guarantee with dramatically lower overhead, because the original `exactly_once` (v1) needed one producer per input partition. The v2 mode has been recommended since Kafka 2.6 (named `exactly_once_beta` until 3.0 renamed it); v1 was deprecated in 3.0 and removed in 4.0, so there is no reason to choose it. **Does Kafka Streams exactly-once cover writes to an external database?** No. The transaction is a Kafka transaction spanning Kafka reads, writes, and offset commits only: it cannot enroll a database insert, a REST call, or any non-Kafka side effect. If a batch aborts and reprocesses, those side effects run again unless the write is idempotent. **Should I use exactly-once or at-least-once with an idempotent sink?** Exactly-once adds end-to-end latency (`read_committed` readers wait for the commit) and per-cycle transaction coordination overhead. Many teams instead run at-least-once and make the final write absorb duplicates with an upsert keyed by a deterministic id, pushing correctness to the edge where it matters. Choose exactly-once when the Kafka-to-Kafka cycle is genuinely where duplicates come from. > **See it in practice with Conduktor** > Exactly-once runs on Kafka transactions and `read_committed` consumers, and both leave signals on the cluster. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-exactly-once) lets you inspect the topics a Streams app reads and writes, watch consumer group lag, and confirm processing is keeping up, so when output arrives in bursts or a transaction stalls, you can see whether it's the commit cadence or a genuine backlog. ## Next steps - [Why you still see duplicates](https://www.conduktor.io/kafka-streams/exactly-once-duplicates): the debugging checklist when exactly-once is on but duplicates persist - [Deduplication patterns](https://www.conduktor.io/kafka-streams/deduplication): de-duplicating data, which exactly-once does not do - [Idempotent Kafka producer](https://www.conduktor.io/kafka/idempotent-kafka-producer): the producer-level guarantee exactly-once builds on Source: https://www.conduktor.io/kafka-streams/exactly-once --- # The future of Kafka Streams *Read where the library is heading, grounded in real work.* Kafka Streams is a mature library, but the ground under it is moving. The themes that dominate recent Current, Kafka Summit, and Flink Forward talks aren't incremental config tweaks, they target the two things that have always made Streams hard to operate: local state that's expensive to restore, and rebalances that stop the world. At the same time, the broader Kafka project is growing new primitives (share groups, queues) that change *when* you'd reach for a Streams topology at all. This page is a vendor-neutral read on those directions: what's shipping, what's still proposal-stage, and what each one means for choosing Kafka Streams today. No product pitch, most of this is upstream Apache Kafka and an active open ecosystem. **What you'll learn:** - Why remote, object-storage-backed state stores are the hottest theme, and what they fix - How the new rebalance protocols cut the stop-the-world pain - When a share-group consumer fits better than a Streams topology - Where Streams sits in real-time ML and agentic-AI architectures ## Remote state stores: getting state off local disk This is the dominant theme, and it attacks the deepest operational pain in Kafka Streams. Today, state lives in [RocksDB on each instance's local disk](https://www.conduktor.io/kafka-streams/state-store), backed by a changelog topic. That design is fast in steady state but brittle in motion: an instance that starts with no local data must [replay its entire changelog before it can process](https://www.conduktor.io/kafka-streams/state-restore), which can mean minutes, or, for a multi-hundred-million-key store, far longer. Remote (or *disaggregated*) state stores invert this. State lives in shared, durable storage, typically object storage like S3, instead of on the instance. The payoff: - **Restores become near-instant.** A new or rebalanced instance reads from the shared store rather than replaying a changelog from offset zero. - **Instances become effectively stateless.** They hold a cache, not the system of record, so they can be killed, scaled, and rescheduled cheaply, which matters on Kubernetes and spot capacity. - **State size stops bounding the instance.** Local disk no longer caps how much state a node can be responsible for. This is real, not vaporware, because the Kafka Streams state store interface is **pluggable**, you can supply your own store implementation. Several open and commercial efforts build remote stores on that seam, and the conference circuit has been thick with talks on object-storage-backed and snapshot-capable stores. The trade-off is honest: you swap local-disk latency and changelog-replay restores for network round-trips to object storage and a caching layer to hide them. Whether that wins depends on your restore pain versus your latency budget. The convergence is the tell that this is a genuine industry direction, not one vendor's bet: stream processors are independently moving state to object storage. The same architectural pressure, make stateful stream processors elastic by getting state off the box, is reshaping the wider space, Kafka Streams included. > **What this means for choosing Streams today.** Remote state stores are emerging, not yet the default in Apache Kafka. If slow restores and stop-the-world rebalances are your main objection to Kafka Streams *right now*, evaluate the ecosystem stores on the pluggable interface, but build on the standard local-state model unless you've measured a restore problem you can't tune away with [standby replicas and persistent volumes](https://www.conduktor.io/kafka-streams/state-restore). Don't adopt a remote store speculatively; adopt it against a measured pain. ## New rebalance protocols: less stop-the-world The other historic pain is the [rebalance](https://www.conduktor.io/kafka-streams/rebalancing): when group membership changes, work pauses while assignments and state move. Stateful apps feel this hardest, because a reassigned task may have to restore its store before it can resume. Two upstream protocol changes are reshaping this: | Change | Status | What it does | |---|---|---| | **KIP-848**, next-gen consumer rebalance protocol | GA in Apache Kafka 4.0 | Moves assignment logic broker-side and makes rebalancing incremental, cutting the synchronization barrier that caused long, lock-step pauses across the whole group | | **KIP-1071**, Streams-specific rebalance protocol | GA in Kafka 4.2, but opt-in and a limited subset (sticky assignor only, offline migration, no static membership or topology updates) | Extends the broker-driven model to Kafka Streams' task assignment, so Streams gets the same reduced-pause behavior tuned for its tasks, standbys, and warm-up | The direction is consistent: less client-side coordination, fewer global stop-the-world pauses, smoother scaling. Combined with remote state (no restore to wait on) and existing tools like [warm-up replicas and standbys](https://www.conduktor.io/kafka-streams/rebalancing), the long-term trajectory is rebalances that are an operational non-event rather than a recurring incident. Version-gate any plan on this: KIP-848 is the one you can rely on today; KIP-1071 reached GA in 4.2 but is still opt-in and feature-limited, with the default switch not expected before Kafka 5.0. One concrete caveat: migrating an existing classic group to the streams protocol is not recommended in 4.2.0 due to a broker-side bug in the offline-migration path (KAFKA-20254); fresh streams groups are unaffected. Pilot it, but most apps still run the classic protocol, check the [Streams rebalance protocol docs](https://kafka.apache.org/documentation/streams/) for the version you actually run. ## Queues and share groups: a different tool Not every workload that touches Kafka should be a Streams topology. [KIP-932](https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka) introduces **share groups**, a consumer model that lets many consumers cooperatively process records from the same partitions without the strict one-consumer-per-partition limit, with per-record acknowledgement. In effect, it brings classic queue semantics to Kafka. That changes the decision tree. Reach for a **share-group consumer** when your problem is *competing-consumer work distribution*, a pool of workers draining a task queue, where you want parallelism beyond the partition count and per-message ack/redelivery, and you don't need stateful stream processing. Reach for a **Streams topology** when your problem is *stateful, ordered, per-key stream processing*, aggregations, joins, windowing, materialized views. Share groups are about throughput and work-spreading on a queue; Streams is about transforming and enriching keyed streams with managed state. They solve different problems, and having both in Kafka means fewer cases where you bend Streams into a job it was never the right shape for. Share groups went GA in Kafka 4.2, after early access in 4.0 and a preview in 4.1 that was explicitly not production-ready. They're still evolving fast (new ack types, adaptive batching, lag metrics all landed in 4.2), so check the release notes for the maturity level in your version before building on them. ## Kafka Streams in real-time ML and agentic AI Real-time AI architectures lean heavily on streaming, and most reference demos today are Flink-centric, Flink SQL feeding features, vector stores, and agent loops. That's a real pattern, and it's fair to say Flink currently owns the spotlight here. Streams' role is narrower and concrete: **stateful enrichment and real-time feature computation inside JVM applications.** If your service is already a Java application that owns its data and reacts to events, computing features or enriching events for a model with a Streams topology keeps that logic next to your business code, no separate cluster, no extra deployment surface. The honest framing: - Streams is a good fit for **real-time features and enrichment** embedded in a JVM service that's already part of your event-driven system. - It is **not** the tool for heavy SQL analytics across many heterogeneous sources, or for a dedicated streaming-ML platform a separate team operates, that's [Flink's territory](https://www.conduktor.io/kafka-streams/vs-flink). - The "agent" layer (orchestration, model calls) typically lives elsewhere; Streams contributes the stateful, low-latency data plane underneath it. The most concrete fit in a multi-agent system is a **context store**: a Streams app materializes the conversation into a queryable store that agents read over interactive queries, keeping their memory in the same data plane, no external database. That pattern has its own page: [Kafka Streams for AI agents](https://www.conduktor.io/kafka-streams/ai-agents). No hype here: Streams isn't becoming an AI framework. It stays what it is, a library for stateful stream processing, and that's exactly the slice of an AI architecture where it earns its place. ## State stores as a system of record A quieter but striking direction: using Kafka Streams' state stores not as a derived cache but as the *authoritative* store for a domain. Because a store is durable (via its changelog), partitioned, and queryable through [interactive queries](https://www.conduktor.io/kafka-streams/state-store), some teams build systems where the store *is* the source of truth, workflow engines, calculation engines, and orchestrators whose entire state lives in Streams rather than an external database. It's a legitimate pattern with real production deployments behind it, and it's a useful lens on what the library actually is: a way to keep durable, queryable, event-sourced state co-located with the code that mutates it. It also raises the operational bar, your state store is now load-bearing in a way a disposable cache never is, which makes the restore-time and rebalance themes above even more important. It's not the common case, but it shows where the model can go when you take its durability guarantees seriously. ## Frequently Asked Questions **What is the future of Kafka Streams?** The dominant directions are remote, object-storage-backed state stores that make restores near-instant and instances effectively stateless, plus new rebalance protocols (KIP-848, KIP-1071) that cut stop-the-world pauses. Alongside that, new Kafka primitives like share groups change when you'd reach for a Streams topology at all. **What is a remote or disaggregated state store?** It moves state off each instance's local disk into shared durable storage, typically object storage like S3, so a new or rebalanced instance reads from the shared store instead of replaying a changelog from offset zero. The trade-off is honest: you swap local-disk latency and changelog-replay restores for network round-trips and a caching layer to hide them. **Are remote state stores production ready?** They are emerging, not the default in Apache Kafka. The state store interface is pluggable, so several open and commercial efforts build on that seam, but you should build on the standard local-state model unless you've measured a restore problem you can't tune away with standby replicas and persistent volumes. **What is the KIP-1071 Streams rebalance protocol and which version has it?** It extends the broker-driven assignment model to Kafka Streams' tasks for reduced-pause rebalancing. It reached GA in Kafka 4.2 but is opt-in and a limited subset (sticky assignor only, offline migration, no static membership or topology updates), with the default switch not expected before Kafka 5.0, so pilot it, but most apps still run the classic protocol. **When should I use a share group (KIP-932) instead of a Streams topology?** Reach for a share-group consumer when your problem is competing-consumer work distribution, a pool of workers draining a task queue with per-record acknowledgement, where you want parallelism beyond the partition count and don't need stateful processing. Use a Streams topology for stateful, ordered, per-key processing: aggregations, joins, windowing, and materialized views. > **See it in practice with Conduktor** > Whatever direction state stores take, a Kafka Streams app remains a consumer group writing to changelog and repartition topics. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-future) gives you a stable vantage point across these shifts, watch consumer group lag, inspect the internal topics, and confirm partition assignment regardless of which rebalance protocol or store implementation runs underneath. ## Next steps - [Kafka Streams vs Flink vs ksqlDB](https://www.conduktor.io/kafka-streams/vs-flink), the honest, deployment-model comparison - [State restore time](https://www.conduktor.io/kafka-streams/state-restore), the problem remote stores are built to solve - [Slow rebalances](https://www.conduktor.io/kafka-streams/rebalancing), what the new protocols are fixing Source: https://www.conduktor.io/kafka-streams/future --- # Build your first Kafka Streams app *Get a real Kafka Streams app running.* The fastest way to understand Kafka Streams is to run one. This walks through WordCount, the canonical first app, as a complete Java program: the dependency, the minimal config, the topology, and how to start and (cleanly) stop it. Every piece here is the real thing, not pseudocode. By the end you'll have an app that reads a topic of lines, counts words continuously, and writes the running counts to an output topic, and you'll understand the few config choices that decide whether it behaves in production. **What you'll learn:** - The one dependency you need, in Maven and Gradle - The minimal configuration, and why `application.id` is the most important line - The WordCount topology, operator by operator - How to start the app, shut it down cleanly, and read its output ## The dependency Kafka Streams ships as a single JAR. Add it to a Java project, if you don't have one yet, start from [Maven](https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-maven-pom-xml) or [Gradle](https://www.conduktor.io/kafka/creating-a-kafka-java-project-using-gradle-build-gradle). ```xml org.apache.kafka kafka-streams 3.9.0 ``` Gradle: ```groovy implementation 'org.apache.kafka:kafka-streams:3.9.0' ``` `kafka-streams` pulls in `kafka-clients` transitively, so you don't add it separately. Add an SLF4J binding (such as `slf4j-simple`) so you see the library's logs, Kafka Streams is quiet by default, and during a [rebalance](https://www.conduktor.io/kafka-streams/rebalancing) or a state restore those logs are how you know what it's doing. ## The minimal configuration Four properties get a Streams app running. Two of them, the serdes and the offset reset, are convenience defaults; the other two are not optional. ```java Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-app"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass()); props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass()); props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); ``` - **`APPLICATION_ID_CONFIG`**, identifies your app. More on this below; it's the line that matters most. - **`BOOTSTRAP_SERVERS_CONFIG`**, your Kafka cluster. - **The default serdes**, how records are serialized and deserialized. We default both to `String` here; the moment a value isn't a string you override it per operator (a frequent source of [serde errors](https://www.conduktor.io/kafka-streams/serdes)). - **`AUTO_OFFSET_RESET_CONFIG=earliest`**, on first run there's no committed offset, and a plain consumer would default to `latest` and wait for new records. Kafka Streams already overrides that default to `earliest` for you (one of `StreamsConfig`'s built-in consumer overrides, in 3.9 and 4.x alike), so this line is documentation of intent rather than a behavior change. Still worth writing: it makes the first-run behavior explicit instead of buried in the library. ### application.id is doing three jobs at once `application.id` looks like a label. It isn't, it's an identity that Kafka Streams reuses for three different things: 1. The **consumer group id**. All instances sharing an `application.id` form one [consumer group](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets) and split the partitions between them. This is how the app scales. 2. The **prefix for every internal topic** it creates, the changelog and repartition topics behind your [state stores](https://www.conduktor.io/kafka-streams/state-store) are all named `-...`. 3. The **local state directory** name on disk, where RocksDB keeps its files. Because one string drives all three, choosing it carelessly has real consequences: > **Treat `application.id` as a namespace, and never reuse one by accident.** Two different apps pointed at the same `application.id` will join the same consumer group, fight over partitions, and write to each other's changelog and state directories, corrupting both. A copy-paste typo that collides with another app's id has caused exactly this in production. Make it unique and descriptive per app, and change it deliberately, knowing it resets the consumer group offsets and orphans the old internal topics. ## The topology ![WordCount topology: source to flatMapValues to filter to groupBy triggering repartition, then through an internal repartition topic to count with word-counts state store and changelog, then toStream and to words-output](https://www.conduktor.io/assets/kafka-streams/diagrams/wordcount-topology.svg) Now the actual logic. WordCount reads lines, splits them into words, counts each word, and writes the counts out. Here's the whole topology, this runs as-is on Apache Kafka 3.9.0: ```java StreamsBuilder builder = new StreamsBuilder(); builder.stream("words-input") .flatMapValues(line -> Arrays.asList(line.toLowerCase().split("\\W+"))) .filter((key, word) -> !word.isEmpty()) .groupBy((key, word) -> word) .count(Materialized.as("word-counts")) .toStream() .to("words-output", Produced.with(Serdes.String(), Serdes.Long())); Topology topology = builder.build(); ``` Reading it operator by operator: | Operator | What it does | |---|---| | `stream("words-input")` | Reads the input topic as a [KStream](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable) of lines | | `flatMapValues(... split ...)` | Splits each line into words, one input line becomes many word records | | `filter(word not empty)` | Drops the empty strings the regex split can produce | | `groupBy((k, word) -> word)` | Re-keys by the word itself, so all occurrences of a word group together | | `count(Materialized.as("word-counts"))` | Counts per word into a **named** state store, producing a `KTable` | | `toStream()` | Converts that running count table back into a stream of updates | | `to("words-output", Produced.with(...))` | Writes out, telling Kafka the value is now a `Long`, not a `String` | Two details worth pausing on. `groupBy` changes the key, which forces Kafka Streams to **repartition** the data through an internal topic so that every instance of a given word ends up on the same task, that's automatic, but it's why a stateful app creates topics behind your back. And the final `Produced.with(Serdes.String(), Serdes.Long())` is the explicit, safe choice rather than strictly mandatory here: `count()` sets a `Long` serde on its materialization and Kafka Streams propagates it through `toStream()` into `to()`, so this exact topology happens to work without it. But the moment an operator that loses type information sits between the aggregation and `to()` (a `map` or `mapValues`, say), propagation breaks, the default `String` serde kicks back in, and you get a `ClassCastException` wrapped in a `StreamsException` at runtime. Always state the serde at the sink. > **Naming the store (`Materialized.as("word-counts")`) is not cosmetic.** An unnamed store gets a *positional* name derived from its place in the topology, which shifts the moment you insert an operator upstream, orphaning the state. The name also anchors the internal topics: this topology creates `-word-counts-repartition` and `-word-counts-changelog`, all from that one string. Name your stores from day one. See [evolving a topology](https://www.conduktor.io/kafka-streams/topology-evolution). ## Start it, and shut it down cleanly A `Topology` is just a description. To run it, wrap it in a `KafkaStreams` instance and `start()`. The part people forget is the other half: stopping cleanly. ```java KafkaStreams streams = new KafkaStreams(topology, props); Runtime.getRuntime().addShutdownHook(new Thread(streams::close)); streams.start(); ``` `start()` kicks off the stream threads and returns immediately, the processing runs in the background. The shutdown hook is what makes it behave on `Ctrl+C`, a `SIGTERM`, or a Kubernetes pod eviction: `streams.close()` flushes state and commits offsets. It deliberately does *not* leave the consumer group: the member lingers until `session.timeout.ms` expires (45 seconds by default), so a quick restart doesn't trigger a rebalance. When an instance is going away for good, call `close(CloseOptions)` with `leaveGroup(true)` to remove it immediately. Skip the hook and a hard kill leaves uncommitted offsets and unflushed state, so the next start re-restores from the changelog and may reprocess records. The shutdown hook is one line. It saves you a slow, confusing restart later. ## Run it With a local Kafka cluster on `localhost:9092`, create the topics, start the app, and feed it some lines: ```bash # create the input and output topics kafka-topics.sh --create --topic words-input --bootstrap-server localhost:9092 kafka-topics.sh --create --topic words-output --bootstrap-server localhost:9092 # produce a few lines (type, then Ctrl+D) kafka-console-producer.sh --topic words-input --bootstrap-server localhost:9092 > the quick brown fox > the lazy dog ``` Then read the output. The values are `Long`, so the consumer needs the right deserializer and the keys printed: ```bash kafka-console-consumer.sh --topic words-output \ --bootstrap-server localhost:9092 \ --from-beginning \ --property print.key=true \ --value-deserializer org.apache.kafka.common.serialization.LongDeserializer ``` Don't panic when nothing appears for the first half-minute. With the default config the record cache holds updates and flushes them on commit, and `commit.interval.ms` defaults to 30 seconds, so the output topic stays empty until the first commit. Then everything arrives at once: ``` quick 1 the 2 brown 1 fox 1 lazy 1 dog 1 ``` Note `the` shows up once, as `2`. Both lines landed inside one commit interval, so the cache merged the intermediate `the 1` into the final count before emitting. That's still the `KTable` doing its job: each output record is the *current* count for that word, not a one-time final answer. Send more lines after the next commit and the counts keep climbing as new records. To watch every single update flow through, the `the 1` then `the 2` most tutorials show, which reflects `TopologyTestDriver` semantics, not real-broker defaults, set `statestore.cache.max.bytes` to `0`. (If you want a single final emission per window instead of running updates, that's windowing plus suppression; see [aggregations](https://www.conduktor.io/kafka-streams/aggregations).) That's a complete Kafka Streams app: a dependency, five config lines, a topology, and a clean shutdown. ## Frequently Asked Questions **What Maven dependency do I need for Kafka Streams?** Add `org.apache.kafka:kafka-streams` (this page uses version `3.9.0`). It pulls in `kafka-clients` transitively, so you don't add that separately; also add an SLF4J binding so you can see the library's logs during rebalances and state restores. **What is the application.id config and why does it matter?** `application.id` is an identity Kafka Streams reuses for three things: the consumer group id, the prefix for every internal topic it creates, and the local state directory on disk. Because one string drives all three, two apps sharing an `application.id` will fight over partitions and corrupt each other's state, so keep it unique per app. **Why does my Kafka Streams app read nothing on first run?** Usually not the offset reset: Kafka Streams overrides the consumer's `latest` default to `earliest`, so a fresh app reads the input topic from the beginning (a plain `KafkaConsumer` does not get this override). The likelier cause is the record cache plus the 30-second default `commit.interval.ms`: nothing reaches the output topic until the first commit. Wait out the interval, or set `statestore.cache.max.bytes` to `0` to see every update immediately. **Why do I need a shutdown hook in a Kafka Streams app?** `streams.start()` returns immediately and runs in the background, so a clean stop needs `streams.close()` wired to a shutdown hook. Without it, a hard kill leaves uncommitted offsets and unflushed state, forcing the next start to re-restore from the changelog and possibly reprocess records. **Why does my WordCount output show the same word with increasing counts?** An aggregation returns a `KTable`, so each output record is the current running count for that key, not a one-time final answer, `the` will appear as `1`, then `2`, and so on as more lines arrive. To collapse those updates into one result per time window instead of every update, you reach for windowing plus `suppress(untilWindowCloses)`; the record cache only reduces how many updates you see, it never yields a single final value. > **See it in practice with Conduktor** > Once your first app is running, it shows up on the cluster as a consumer group plus a set of internal topics. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-getting-started) lets you watch `words-output` fill up, find the `wordcount-app-...` repartition and changelog topics it created, and check the consumer group's lag, the fastest way to confirm the app is actually keeping up. ## Next steps - [Kafka Streams architecture](https://www.conduktor.io/kafka-streams/architecture), how a topology becomes tasks and threads - [Aggregations](https://www.conduktor.io/kafka-streams/aggregations), go beyond count with reduce and aggregate - [State stores](https://www.conduktor.io/kafka-streams/state-store), what `Materialized.as(...)` actually created Source: https://www.conduktor.io/kafka-streams/getting-started --- # Kafka Streams interactive queries *Serve your state store to the outside world.* A [state store](https://www.conduktor.io/kafka-streams/state-store) is a queryable view of your data living right inside your app. Interactive queries are how you read it (from a REST endpoint, another service, an [AI agent](https://www.conduktor.io/kafka-streams/ai-agents)) without standing up a separate database. The store page covers how state is *kept*. This page covers how you *serve* it, which is where the multi-instance problems live. The single fact that shapes everything here: state is **sharded across instances**, the same way partitions are. A key lives on exactly one instance, and querying the wrong one fails. Getting interactive queries right is mostly about finding the instance that owns the key. **What you'll learn:** - How to open a read-only handle on a store with `StoreQueryParameters` - Why a single-instance query breaks at two instances, and how to route to the owner - Why you keep hitting `InvalidStateStoreException`, and the two fixes - IQv1 vs IQv2 (KIP-796): the older API and the newer typed one ## Querying a store on one instance With a single instance, interactive queries are trivial. You ask the running `KafkaStreams` object for a read-only view of a named store and call `get`: ```java ReadOnlyKeyValueStore store = streams.store( StoreQueryParameters.fromNameAndType("word-counts", QueryableStoreTypes.keyValueStore())); Long count = store.get("kafka"); // point lookup KeyValueIterator all = store.all(); // full scan ``` The store type must match what you materialized. The common ones: | Store | `QueryableStoreTypes` factory | Reads | |---|---|---| | Key-value (`count`, `aggregate`, `reduce`) | `keyValueStore()` | `get(key)`, `range(from, to)`, `all()` | | [Windowed](https://www.conduktor.io/kafka-streams/windowing) | `windowStore()` | `fetch(key, timeFrom, timeTo)` | | Session | `sessionStore()` | `fetch(key)` | The store must be **named** when you create it (`Materialized.as("word-counts")`). An unnamed store gets a positional name that you can't reliably reference and that shifts when you [edit the topology](https://www.conduktor.io/kafka-streams/topology-evolution). Kafka 4.3 logs a startup WARN listing every unnamed store and internal topic, and `ensure.explicit.internal.resource.naming=true` turns unnamed resources into a hard failure. ## Enabling discovery across instances Run two instances and the picture changes. Each owns a subset of partitions, so each holds a *different slice* of the store. A query for `"kafka"` only works on whichever instance owns the partition `"kafka"` hashes to. The other instance has no idea what that key's value is. To make instances discoverable, set `application.server` to a host:port that the *other* instances (and your own query layer) can reach: ```properties application.server=10.0.1.42:8080 ``` This doesn't open any port for you. It's metadata Kafka Streams gossips through the consumer group so every instance knows every other instance's address. You still build the HTTP (or gRPC) endpoint at that address yourself. > 🚫 *"Interactive queries let me read any key from any instance, so I'll just query whichever one my load balancer hits."* That's the misconception that bites everyone first. An instance can only answer for the keys whose partitions it owns. Query a key the local instance doesn't own and you get an empty result or an exception, not the value sitting on the neighbor. You have to find the owner and route to it. ## Routing to the instance that owns the key ![queryMetadataForKey routes a single-key query directly to the owning instance; a keyspace-wide query fans out to all instances and merges results](https://www.conduktor.io/assets/kafka-streams/diagrams/interactive-queries-key-routing.svg) The store metadata tells you which instance owns a given key. You ask `queryMetadataForKey`, compare the returned host to your own `application.server`, and either read locally or make an HTTP call to the owner. ```java KeyQueryMetadata md = streams.queryMetadataForKey( "word-counts", "kafka", Serdes.String().serializer()); HostInfo owner = md.activeHost(); if (owner.equals(myHostInfo)) { return localStore.get("kafka"); // we own it } else { // RPC to http://owner.host():owner.port()/store/word-counts/kafka return httpClient.get(owner, "kafka"); } ``` For a query that has to span the whole keyspace (a dashboard total, an export), there's no single owner: you fan out. `streams.metadataForAllStreamsClients()` returns every instance and the partitions it holds; you call each one's endpoint and merge the results. (The `allMetadata()` from older tutorials is removed in Kafka 4.x, not just deprecated; code still calling it fails with `NoSuchMethodError`.) ```java for (StreamsMetadata instance : streams.metadataForAllStreamsClients()) { HostInfo host = instance.hostInfo(); // GET http://host/store/word-counts/all on every instance, then combine } ``` That fan-out REST layer (a controller that does a local read when it owns the key and an HTTP hop when it doesn't) is the part you write. Kafka Streams gives you the routing metadata; it does not give you the web server. ## Why you keep hitting InvalidStateStoreException This is the most-pasted interactive-queries error, and it has two distinct causes that need different fixes. ``` org.apache.kafka.streams.errors.InvalidStateStoreException: The state store, word-counts, may have migrated to another instance. ``` **Cause 1: the app isn't `RUNNING` yet, or a rebalance is in flight.** During startup, a [rebalance](https://www.conduktor.io/kafka-streams/rebalancing), or a [state restore](https://www.conduktor.io/kafka-streams/state-restore), the store is being (re)built and isn't available to query. The store handle you cached can also go stale across a rebalance. The exact message depends on the phase: before `start()` you get `StreamsNotStartedException`, during startup or a rebalance `streams.store()` says "Cannot get state store word-counts because the stream thread is STARTING, not RUNNING", and the "may have migrated" text above is what a stale handle throws. All are `InvalidStateStoreException` or its subclasses, so catch the parent. The fix is to treat the exception as *transient*: re-fetch the store handle and retry with backoff. Don't hold a `ReadOnlyKeyValueStore` reference for the life of the app: re-acquire it per query, or after any `REBALANCING` transition. **Cause 2: the key genuinely isn't on this instance.** You queried the local store for a key another instance owns. A plain `get()` here usually returns `null` silently rather than throwing, as long as the instance owns at least one partition of the store, so the miss is easy to mistake for a missing key. No retry will fix that; the value isn't here. The fix is the routing from the previous section: find the owner and call it. If you can tolerate slightly stale data during a rebalance, IQv1 also lets you query a store that's still restoring rather than waiting for `RUNNING`: ```java ReadOnlyKeyValueStore store = streams.store( StoreQueryParameters .fromNameAndType("word-counts", QueryableStoreTypes.keyValueStore()) .enableStaleStores()); // KIP-535: serve standby/restoring replicas ``` `enableStaleStores()` (KIP-535) lets a query hit a standby replica or a still-restoring active store, trading freshness for availability, useful when "an answer that's a few seconds behind" beats "an exception during every deploy." Pair it with [standby replicas](https://www.conduktor.io/kafka-streams/state-restore) (`num.standby.replicas`) so there's a warm copy to read from while the active restores. ## IQv1 vs IQv2 Everything above is **IQv1**: `streams.store(StoreQueryParameters...)` returning a typed `ReadOnly*Store`. It works, but each store type has its own interface and adding a new query shape means a new store type. **IQv2** (KIP-796) is the newer, typed query API: instead of fetching a store interface, you build a `Query` object and execute it through `streams.query(...)`, getting back a `StateQueryResult` with per-partition results and position info. ```java StateQueryRequest request = StateQueryRequest.inStore("word-counts") .withQuery(KeyQuery.withKey("kafka")); // KeyQuery returns the raw value StateQueryResult result = streams.query(request); ``` (`KeyQuery` unwraps to the plain value; use `TimestampedKeyQuery` if you want the `ValueAndTimestamp` wrapper.) IQv2 ships query types like `KeyQuery` (point lookup) and `RangeQuery` (scan), is extensible to custom queries, and exposes partition `Position` so you can reason about freshness. IQv1 is not deprecated and remains the most common code you'll see in the wild, so know both, but `query(...)` is the direction the project is moving, and it's the cleaner API for new code. ## Querying the whole pipeline, end to end Putting it together, a production interactive-query service is three layers: the Streams app materializing named stores; an HTTP endpoint per instance bound to its `application.server` address; and routing logic that either reads locally or fans out to the owners. This is exactly the shape behind the [AI-agent context store](https://www.conduktor.io/kafka-streams/ai-agents), where an orchestrator calls `GET /conversation/{id}` and the service routes to whichever instance holds that conversation's partition. ## Frequently Asked Questions **Why do I get InvalidStateStoreException?** Two causes. Either the app isn't `RUNNING` yet or a rebalance/restore is in flight (transient: re-fetch the store handle and retry with backoff), or the key lives on a different instance (not transient: route to the owner via `queryMetadataForKey`). `InvalidStateStoreException` and its subclasses (like `StreamsNotStartedException`) cover both, but the message varies: "Cannot get state store X because the stream thread is STARTING, not RUNNING" during startup or rebalance, "may have migrated to another instance" for stale handles and migrated stores. **Can I query a state store during a rebalance?** Not the active store by default: it may be migrating or restoring. You can opt into stale reads with `StoreQueryParameters.enableStaleStores()` (KIP-535) to query a standby or still-restoring replica, trading freshness for availability. Otherwise, retry until the app returns to `RUNNING`. **How do I query a state store on another instance?** Set `application.server` so instances discover each other, call `streams.queryMetadataForKey(store, key, serializer)` to get the owning `HostInfo`, and make an HTTP/gRPC call to that instance's endpoint. You build the REST layer; Kafka Streams only provides the routing metadata. **What's the difference between IQv1 and IQv2?** IQv1 (`streams.store(StoreQueryParameters...)`) returns a typed read-only store interface per store type. IQv2 (KIP-796) uses `streams.query(...)` with composable `Query` objects (`KeyQuery`, `RangeQuery`) and returns a `StateQueryResult` with partition position info. IQv1 isn't deprecated; IQv2 is the newer, more extensible API. **Do I need a database if I use interactive queries?** Often not. A materialized store is already a queryable, fault-tolerant view rebuilt from its changelog, so for read-your-own-derived-state cases it replaces a separate database. You add a database when you need cross-key transactions, ad-hoc SQL, or to serve data the Streams app didn't materialize. > **See it in practice with Conduktor** > Interactive queries read from local stores backed by changelog topics on the cluster. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-interactive-queries) lets you see those `-...-changelog` topics, check their compaction and size, and watch the restore consumer's lag, so when a query throws `InvalidStateStoreException` after a deploy, you can confirm whether a restore is still in progress instead of guessing. ## Next steps - [State stores](https://www.conduktor.io/kafka-streams/state-store): how the data you're querying is kept and made fault-tolerant - [Kafka Streams for AI agents](https://www.conduktor.io/kafka-streams/ai-agents): the context-store pattern that serves state over interactive queries - [Monitoring Kafka Streams](https://www.conduktor.io/kafka-streams/monitoring): the lag and restore signals that explain query failures Source: https://www.conduktor.io/kafka-streams/interactive-queries --- # Debugging a Kafka Streams join that produces nothing *Find out why your join drops records.* Your join compiles, the app starts clean, both input topics have data, and the output is empty, or it emits for some keys and silently swallows the rest. There is no exception in the logs. No "join failed" line. The DSL made the join a one-liner, and that one line is quietly producing nothing. The hard truth about Kafka Streams joins is that most of their failure modes are **silent**. A join carries a contract (co-partitioning, timing, key semantics) and breaking that contract usually drops records with no error at all. This page is the debug session: the handful of reasons a join produces nothing, how to confirm each one, and the fix. The join *types* themselves are covered in [joins](https://www.conduktor.io/kafka-streams/joins); this is the "why is mine broken" companion. **What you'll learn:** - Why broken co-partitioning is usually silent, and the one condition that makes it throw - Why a KStream-KTable join drops records when the table isn't ready yet - The null-key skip that makes a join "break after an upgrade" - A diagnostic order: what to inspect, in what sequence, when a join is empty > **Most join failures throw nothing.** A misconfigured join is far more likely to silently emit zero records than to raise an exception. Don't wait for an error to tell you something's wrong. If the output is empty or partial, walk the checklist below. ## 1. Not co-partitioned (silent more often than you'd think) ![Co-partitioning: a join task pairs partition N of one side with partition N of the other. When orders and customers use different partitioners, key acct-42 lands on partition 2 in orders and partition 0 in customers, so no task sees both halves and the join emits zero rows with no error. After repartition re-keys both sides through the same Streams partitioner, acct-42 lands on partition 1 on both and task 1 joins them.](https://www.conduktor.io/assets/kafka-streams/diagrams/co-partitioning.svg) Every join except the GlobalKTable join requires both sides to be **co-partitioned**: same partition count, *and* the same key landing on the same partition number on both topics. A join task owns one partition number from each side and matches keys within that partition only. If `acct-42` lands on partition 3 on one side and partition 7 on the other, the two halves never meet in the same task. The two ways to break co-partitioning fail differently, and both can be silent: | Violation | Detected? | Symptom | |---|---|---| | Partition-count mismatch | **Only if the topology has a repartition topic** | `TopologyException: ... Topics not co-partitioned` on the first rebalance; otherwise silent | | Partitioner mismatch (same count) | **No** | No exception. Wrong or empty results, indefinitely | **Count mismatch: loud only under one condition.** When the two input topics have different partition counts, Kafka Streams throws `TopologyException: Invalid topology: ... Topics not co-partitioned: [{topicA=2, topicB=3}]` during the first rebalance, not while building the topology. But the check lives inside repartition-topic setup, and it is skipped entirely when the topology contains no repartition topics (verified on Kafka 3.9 and 4.3). A bare join of two source topics, exactly the naive case, starts clean, runs as many tasks as the larger topic has partitions, and the extra partitions join against nothing. Any re-keying anywhere in the topology brings the check back for every join. The fix is the same either way: make the partition counts match, usually by repartitioning the smaller-partitioned side (see below), since you can't shrink a topic in place. **Partitioner mismatch: the silent one.** This is the failure that costs an afternoon. Both topics have, say, 12 partitions, so the count check passes. But they were *written* by different producers using different partitioning logic: a custom partitioner on one side and the default murmur2 hash on the other, or the same logical key serialized two different ways so the hash differs. Kafka Streams **cannot see how upstream producers partitioned the data**. It trusts that co-partitioning holds, finds no matching keys in the same partition, and emits wrong or empty output with no warning whatsoever. > 🚫 *"Both topics have the same partition count, so my join is co-partitioned."* > Same count is necessary, not sufficient. If the two sides were produced by different applications (different key serializers, or a custom partitioner on one side), keys can be mis-aligned even with matching counts, and the join is silently empty. Kafka Streams cannot detect producer-side partitioning, so it never warns you. **How to check it.** Two things, in order: 1. *Partition counts.* Compare the partition count of both input topics directly. Don't assume the absence of a `TopologyException` rules this out: the check only runs when the topology contains a repartition topic. 2. *Partitioning.* Harder, because it's a property of whoever produced the data. Ask: were both topics written by the same application with the same key serializer and default partitioner? If either side has a custom partitioner, or the two sides come from different producers, assume mis-alignment until proven otherwise. A quick empirical check: pick a key you *know* exists on both sides and confirm the records for it actually sit on the same partition number on each topic. **The fix.** When you can't guarantee upstream partitioning, repartition one (or both) sides explicitly before the join, forcing the data through Kafka Streams' own partitioner so co-partitioning is restored: ```java KStream aligned = orders.repartition(Repartitioned.with(Serdes.String(), orderSerde)); // now join `aligned` instead of `orders` ``` Name the repartition topic (`Repartitioned.as("...")`) so its name stays stable across topology edits. See [evolving a topology](https://www.conduktor.io/kafka-streams/topology-evolution). ## 2. KStream-KTable: the stream arrives before the table is populated A KStream-KTable join is **asymmetric**: only a record on the *stream* side triggers a lookup, and it looks up the table value *as it exists at that moment*. If the stream record for a key is processed **before** the table has been populated for that key, the lookup finds nothing and the record is **silently dropped**: no error, no log line, no output, not even a tick on the `dropped-records-total` metric. This shows up most on startup and after a restart. Both the stream topic and the table's source topic are consumed concurrently, and nothing guarantees the table side is read first. If the stream races ahead, early records miss a table that hasn't caught up. The symptom is distinctive: records are dropped *near the start* of processing (or just after a restart) and then the join works fine once the table has caught up, so a backfill or a cold start "loses the first few minutes." **The lever** is `max.task.idle.ms`, which controls how long a task will *wait* for a lagging input before processing what it already has. The current semantics (Kafka 3.0+, KIP-695) are a trichotomy, and the default is `0`: | `max.task.idle.ms` | Behavior | |---|---| | `-1` | Never idle: process whatever is locally buffered, even at the cost of out-of-order processing | | `0` (default) | Don't wait for *new* producer records, but do fetch data already sitting on the brokers before processing; idle only while caught up | | `> 0` | Additionally wait up to N ms for producers to send records when the task is caught up but an input is empty | ```properties # Let a task wait up to 5s for a lagging input so the table side can catch up max.task.idle.ms=5000 ``` When the idle deadline passes and the task processes anyway with an empty input, Kafka Streams increments the `enforced-processing-rate` / `enforced-processing-total` task metrics (DEBUG recording level) and logs, at **TRACE**: > *Continuing to process although some partitions are empty on the broker. There may be out-of-order processing for this task as a result. Partitions with local data: [...]. Partitions we gave up waiting for, with their corresponding deadlines: {...}. Configured max.task.idle.ms: ...* If that line appears (enable TRACE on the relevant logger) or the metric climbs, a task is giving up on a lagging input, exactly the condition behind dropped or out-of-order join results. This *reduces* the drop window; it does not eliminate it. A table side that is genuinely far behind still loses the race. If the table is large and slow to bootstrap, also consider whether the data belongs in a `GlobalKTable` (fully bootstrapped before processing starts) instead, covered in [KStream, KTable & GlobalKTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable). ## 3. Null keys are skipped (the join that "broke after an upgrade") Records with a **null key** have always been skipped before a join rather than processed: a null key cannot be co-partitioned or looked up. What did change in Kafka 2.7 (KAFKA-10277) is the KStream-GlobalKTable join: it started dropping records whose key mapper returns a null join key. That is the real "our join started dropping data after we upgraded" story, with no code change to explain it. **Kafka 3.7 (KIP-962) relaxed the null-key skip for left and outer joins** (the joiner is now called with `null` for the missing side), but **inner joins still drop null-key records**. So this is your cause mainly on an inner join, or on any join before 3.7. To confirm: check what fraction of your input has null keys. If it's meaningful and those records used to produce output (and you're on an inner join, or pre-3.7), this is your cause. The fix is to assign a key with `selectKey` *before* the join so the records survive and join on something real: ```java KStream keyed = events.selectKey((k, v) -> v.getAccountId()); // give null-keyed records a real key ``` ## 4. Stream-stream join: the window or grace is too tight A KStream-KStream join only matches records whose timestamps fall within the **join window** of each other. If your window is narrower than the real-world delay between the two events (a click that arrives 12 minutes after an impression, against a 10-minute window), the match never happens and both records pass through unjoined. With an inner join, that's silent: no output for the pair. Two timing factors to check: - **Window width.** Is the duration you pass to the `JoinWindows.ofTimeDifference...` factories (`ofTimeDifferenceWithNoGrace`, `ofTimeDifferenceAndGrace`) actually wide enough for how far apart your two events arrive in practice? Measure the real gap; don't guess. - **Grace.** Newer windowed operators default to **no grace** (`ofTimeDifferenceWithNoGrace`). There is no per-record lateness check in the join itself: grace extends how long each side's join store retains candidates, so once stream time on both sides moves past the window, a late record finds nothing left to join. No error, no drop log. If one side routinely arrives late, set grace deliberately. The stream-time and grace mechanics are the same ones that govern [windowing](https://www.conduktor.io/kafka-streams/windowing). If you genuinely need to catch pairs that arrive far apart, widen the window, but know that a wider window means both sides are buffered in state for longer, which is real memory and disk cost, not a free knob. ## 5. Foreign-key join: out-of-order updates and null foreign keys The KTable-KTable foreign-key join (joining on a field inside the value, since Kafka 2.4) has two edges that produce confusing-but-not-broken behavior: - **"Detected out-of-order KTable update" warnings.** The FK join routes updates through a pair of internal subscription and response topics so that a change on either side updates the result both ways. Under reordering, Kafka Streams logs the "Detected out-of-order KTable update" WARN, but the WARN is detection only: with default (non-versioned) stores the stale update is still applied, last write by offset wins. What actually protects the FK result is a hash check: stale subscription responses are discarded by comparing a hash of the current table value, logged only at TRACE and counted in `dropped-records-total`. Versioned stores (Kafka 3.5+, KIP-914) are what make a table skip stale updates outright. A flood of these warnings is still worth investigating as a sign of heavy reordering or a struggling FK join at scale. - **Null foreign-key semantics.** If the foreign-key extractor returns null for a record, there is nothing to join against. For a left join in particular, the result for a null foreign key has surprised people relative to how they read the docs, so don't assume; test the null-FK path explicitly with a record whose extracted key is null and assert what comes out. FK joins are powerful but heavier than same-key joins (those internal topics carry real traffic). Treat them as the tool for genuine foreign-key relationships, not a default. The full picture is in [joins](https://www.conduktor.io/kafka-streams/joins). ## How to diagnose, in order When a join produces nothing, resist guessing. Walk it top-down: 1. **Did the app throw on the first rebalance?** If you saw `TopologyException: ... Topics not co-partitioned`, it's a partition-count mismatch, and the message prints each topic's count. Fix counts and you're done. No exception does not clear you: the check is skipped when the topology has no repartition topics, so keep going. 2. **Compare partition counts** of both input topics. Equal is necessary but not sufficient. 3. **Question the partitioning.** Same producer, same key serializer, default partitioner on both sides? If not, suspect a silent partitioner mismatch and repartition one side. 4. **Is it empty everywhere, or only at the start / after restart?** Start-only drops on a KStream-KTable join point straight at the table-not-ready timing trap: tune `max.task.idle.ms`. 5. **Check for null keys** on the input, especially if the join "broke after an upgrade." 6. **For stream-stream joins, check timestamps and window width**: are the two events actually arriving within the window of each other? 7. **Inspect the internal topics.** A join with re-keying creates internal repartition topics; a foreign-key join creates subscription and response topics (`KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic` and `...-RESPONSE--topic` in 4.x, unless you name the join). Their existence, partition counts, and lag tell you whether records are even reaching the join, and whether one side is starving the other. Steps 2, 3, and 7 are all questions about *topics* (partition counts, partitioning, and the internal topics Kafka Streams created), which is why the cluster view matters here more than the application logs. ## Frequently Asked Questions **Why does my Kafka Streams join produce no output?** Most join failures are silent: no exception, just zero or partial records. Walk the checklist: confirm both sides are co-partitioned (equal partition counts and the same partitioner), check whether a KStream-KTable join is racing a not-yet-populated table, and check for null keys. The cause is usually a broken co-partitioning contract, not a bug. **What causes the "not co-partitioned" TopologyException?** The two join input topics have different partition counts. Kafka Streams checks this during the first rebalance, but only when the topology contains at least one repartition topic; a bare join of two source topics skips the check and starts cleanly. Fix it by making the counts match, usually by repartitioning the smaller-partitioned side, since you can't shrink a topic in place. **Why does my join return empty results even though both topics have the same partition count?** Equal counts are necessary, not sufficient. If the two sides were written by different producers (different key serializers, or a custom partitioner on one side), the same key can land on different partition numbers, so the join task never finds matches. Kafka Streams can't see producer-side partitioning, so it never warns you; repartition one side through Streams' own partitioner. **Why does my KStream-KTable join drop records at startup?** A KStream-KTable join looks up the table value as it exists the moment the stream record arrives. On startup both topics are read concurrently, so if the stream races ahead of a not-yet-populated table, early records find nothing and are silently dropped. Raise `max.task.idle.ms` so the task waits for the table side to catch up, or use a `GlobalKTable` which fully bootstraps first. **Why did my join start dropping records after an upgrade?** Null-key records have always been skipped before joins because a null key can't be co-partitioned. What changed in Kafka 2.7 (KAFKA-10277) is that KStream-GlobalKTable joins started dropping records whose mapped join key is null. Kafka 3.7 (KIP-962) relaxed the null-key skip for left and outer joins, but inner joins still drop null-key records. Assign a real key with `selectKey` before the join so those records survive. > **See it in practice with Conduktor** > A join's failure usually shows up at the topic level, not in your app logs. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-join-troubleshooting) lets you compare the partition counts of both join inputs side by side, inspect the internal repartition and subscription topics Kafka Streams created, and watch per-partition consumer group lag on each input, so you can tell whether records are reaching the join at all, and whether one side is starving the other. ## Next steps - [Joins](https://www.conduktor.io/kafka-streams/joins): the four join types and what each one needs from you - [KStream, KTable & GlobalKTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable): the stream-vs-table model behind every join - [Serde errors](https://www.conduktor.io/kafka-streams/serdes): the other silent way a join produces wrong or no output Source: https://www.conduktor.io/kafka-streams/join-troubleshooting --- # Kafka Streams joins *Learn the four join types and the rules that silently break them.* Joining is where Kafka Streams earns its keep: enrich an order stream with customer data, correlate clicks with impressions, stitch two changelogs into one view. The DSL makes the join itself a one-liner. What it does *not* make obvious is that every join carries a contract, co-partitioning, timing, key semantics, and breaking that contract usually fails silently, with no exception and no output. This page covers the four join types, what each one needs from you, and the three ways a join produces nothing while looking perfectly healthy. **What you'll learn:** - The four join types: KStream-KStream, KStream-KTable, KTable-KTable, GlobalKTable - Why co-partitioning is mandatory, and how a partitioner mismatch silently corrupts results - Why a KStream-KTable join drops records when the table isn't populated yet - When to reach for a foreign-key join or a GlobalKTable instead ## The join matrix Kafka Streams gives you four joins, and which one you pick is dictated by the *type* of each side, stream or table, not by taste. | Left | Right | Join semantics | Windowed? | Co-partition required? | |---|---|---|---|---| | KStream | KStream | Both sides are events; match within a time window | Yes | Yes | | KStream | KTable | Event looks up the latest table value for its key | No | Yes | | KTable | KTable | Two changelogs combined into one updated view | No | Yes (on key) | | KTable | KTable | Foreign-key: join on a non-key field | No | No (re-keyed internally) | | KStream / KTable | GlobalKTable | Lookup against a fully-replicated table by any field | No | **No** | The mental split: a **stream-stream** join asks "did these two things happen close together in time?" A **stream-table** join asks "what is the current state for this key?" A **table-table** join keeps a continuously-updated combined view. Get the question wrong and you've picked the wrong join, see [KStream, KTable & GlobalKTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable) for the underlying model. ## KStream-KStream: windowed event correlation Two event streams, matched when records with the same key arrive within a time window. This is the join for "click within 10 minutes of an impression" or "payment within an hour of an order". ```java KStream impressions = builder.stream("impressions"); KStream clicks = builder.stream("clicks"); KStream attributed = impressions.join( clicks, (impression, click) -> new Attribution(impression, click), JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(10)), StreamJoined.with(Serdes.String(), impressionSerde, clickSerde) ); ``` The join is symmetric and windowed: each side is buffered in a window store for the window duration, so a click can match an impression that arrived earlier *or* later within the window. That buffering is state, two more changelog-backed stores, sized by your window length and traffic. A wide window over high-volume topics is a real memory and disk cost, not a free lookup. `JoinWindows` controls the matching span; the grace period controls how long late records can still join after the window's nominal end. Newer windowed operations default to **no grace** (`ofTimeDifferenceWithNoGrace`), so a late event is refused by the join buffer and counted in the `dropped-records` metric unless you ask for tolerance explicitly. (It can still match on arrival if the other side's store hasn't expired the counterpart yet; it just can't match anything that arrives after it.) The interaction between stream-time, windows, and grace is the same one that bites windowing, covered in [windowing](https://www.conduktor.io/kafka-streams/windowing). A **self-join**, joining a stream with itself to correlate events on the same topic (matching a login with a later logout, say), is a special case. Since Kafka 3.4 (KIP-862), an inner self-join on the key is optimized to use a *single* state store instead of two, halving its state cost, when [topology optimization](https://www.conduktor.io/kafka-streams/topology-evolution) is enabled. The optimization covers inner, key-based self-joins only, not N-way ones. ## KStream-KTable: the enrichment lookup The most common join in production: a stream of events enriched with the current value from a table. An order stream joined against a `KTable` of customers, each order decorated with the customer's tier. ```java KStream orders = builder.stream("orders"); KTable customers = builder.table("customers"); KStream enriched = orders.join( customers, (order, customer) -> new EnrichedOrder(order, customer) ); ``` This join is **not** windowed and it is **asymmetric**: only a record on the *stream* side triggers a lookup. A new customer record updates the table silently; it does not re-emit past orders. That asymmetry is correct for enrichment, and it's also the source of the timing trap below. ## The timing trap: stream arrives before the table is ready Here is the failure that costs people an afternoon. A KStream-KTable join looks up the table value *as it exists when the stream record is processed*. If the stream record for a key is processed **before** the table has been populated for that key, the lookup finds nothing, and the record is **silently dropped**. No error. No log line. Not even a tick of the `dropped-records` metric. The join simply produced nothing for that record. This is most visible on startup and after a restart. Both the stream topic and the table's source topic are being consumed concurrently, and there is no guarantee the table side is read first. If your stream is ahead, early records miss a table that hasn't caught up yet. The lever is `max.task.idle.ms`. A task processes whichever buffered record has the lowest timestamp, and since Kafka 3.0 (KIP-695) the default (`0`) already pauses when one input has no buffered data but non-zero lag: it waits for records that exist on the broker to be fetched. Setting `max.task.idle.ms` above zero makes the task additionally wait for records that haven't been *produced* yet, giving the table side time to deliver records up to the same timestamp before the stream record is processed, improving timestamp synchronization across the two inputs. (`-1` disables idling entirely.) ```properties # Let a task wait up to 5s for a lagging input so the table side can catch up max.task.idle.ms=5000 ``` This reduces the drop window; it does not make it impossible (a table side that is genuinely far behind still loses the race). When a join "drops records on startup", this is usually why. The full diagnostic walkthrough lives in [joins that drop data](https://www.conduktor.io/kafka-streams/join-troubleshooting). ## Co-partitioning: the contract that breaks silently ![Co-partitioning: a join task pairs partition N of one side with partition N of the other. When orders and customers use different partitioners, key acct-42 lands on partition 2 in orders and partition 0 in customers, so no task sees both halves and the join emits zero rows with no error. After repartition re-keys both sides through the same Streams partitioner, acct-42 lands on partition 1 on both and task 1 joins them.](https://www.conduktor.io/assets/kafka-streams/diagrams/co-partitioning.svg) Every join except the GlobalKTable join requires the two sides to be **co-partitioned**. Co-partitioning means two things, and both must hold: 1. **Same partition count** on both input topics. 2. **Same partitioning**, the same key produces the same partition number on both sides. The reason is mechanical. A join task owns one partition number from each side and matches keys *within that partition only*. Records for key `acct-42` must land in the same partition on both topics, or the two halves of the join never meet in the same task. The two ways to break it fail very differently: | Violation | Detected? | Symptom | |---|---|---| | Partition-count mismatch | **Sometimes** | `TopologyException: ... not co-partitioned` at the first rebalance, but on 3.2+ only if the topology contains a repartition topic. Otherwise: silent wrong results | | Partitioner mismatch (same count) | **No** | No exception. Wrong or empty results, forever | A count mismatch *can* be the good failure: the check runs in the partition assignor during the first rebalance and kills the app with `TopologyException` "not co-partitioned" before it processes a single record. But since Kafka Streams 3.2, that check is skipped entirely when the topology contains no internal repartition topics, and the plain pre-keyed join above is exactly that case. The app starts, reports RUNNING, creates a task per partition of the larger topic, and joins only the overlapping partitions: the same silent wrong-results failure as a partitioner mismatch. Any operation that adds a repartition topic to the topology re-arms the check. The partitioner mismatch is the dangerous one. If both topics have 12 partitions but they were written by producers using *different* partitioning logic, a custom partitioner on one side, the default murmur2 hash on the other, or one topic keyed by a string and another by the same value serialized differently, Kafka Streams **cannot see how the upstream producers partitioned the data**. It assumes co-partitioning holds, finds no matches for most keys, and emits wrong or empty results with no warning whatsoever. > 🚫 *"Both topics have the same number of partitions, so the join is co-partitioned."* Same partition count is necessary, not sufficient. If the two sides were produced by different applications, with different key serializers or a custom partitioner on one side, the data can be mis-aligned even with matching counts, and you get a silently empty join. The fix when you can't guarantee upstream partitioning is to **repartition explicitly** before the join, forcing both sides through Kafka Streams' own partitioner: ```java KStream repartitioned = orders.repartition(Repartitioned.with(Serdes.String(), orderSerde)); ``` This writes through an internal repartition topic with a known partitioner, restoring the co-partitioning guarantee. Name the repartition topic (`Repartitioned.as("...")`) so it stays stable across topology edits, see [evolving a topology](https://www.conduktor.io/kafka-streams/topology-evolution). ## The null-key skip (a join that "broke after upgrade") Inner joins have always skipped records with a **null key**, since a null key cannot be co-partitioned or looked up. Kafka 2.7 extended that skip to the left-join paths (with a WARN log and a tick of the `dropped-records` metric), which is why teams upgrading from older versions hit this as "our join started dropping data after the upgrade." **Kafka 3.7 (KIP-962) then relaxed it for left and outer joins:** `leftJoin`/`outerJoin` (and left foreign-key joins) no longer drop null-key records, they call the `ValueJoiner` with `null` for the missing side, matching documented left/outer semantics. **Inner joins still drop null-key records**, since an inner match is impossible. On an inner join (or before 3.7), set a key with `selectKey` *before* the join so those records survive. ## KTable-KTable: combining two changelogs A table-table join keeps a continuously-maintained combined view: whenever either side updates for a key, the join output for that key is recomputed and re-emitted. Both sides must be co-partitioned on the key. ```java KTable accounts = builder.table("accounts"); KTable profiles = builder.table("profiles"); KTable view = accounts.join( profiles, (account, profile) -> new AccountView(account, profile) ); ``` Because the result is itself a KTable, a tombstone (null value) on either side flows through as a delete on the output. This is the right behavior for materialized views, and a footgun if you then `toStream()` the result and forget the values can be null. ## Foreign-key joins: joining tables on a non-key field The standard table-table join matches on the key. But often you need to join on a *field inside the value*, orders keyed by `orderId`, each carrying a `customerId`, joined to a customers table keyed by `customerId`. That's a **foreign-key join**, available for KTable-KTable since Kafka 2.4. ```java KTable orders = builder.table("orders"); // keyed by orderId KTable customers = builder.table("customers"); // keyed by customerId KTable enriched = orders.join( customers, order -> order.getCustomerId(), // extract the foreign key (order, customer) -> new EnrichedOrder(order, customer) ); ``` The foreign-key join does **not** require you to co-partition the inputs yourself, it re-keys internally using a pair of hidden subscription and response topics (they show up in your topic list as `-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION--topic` and `...-SUBSCRIPTION-RESPONSE--topic`) to route updates both ways (a change to a customer must update every order that references it). That machinery is powerful and not free: at scale the internal topics carry real traffic, and the join is heavier than a same-key join. It also has sharper edges, out-of-order update warnings and null-foreign-key left-join semantics that have surprised people. Treat it as the right tool for genuine foreign-key relationships, not a default. Since Kafka 4.0 (KIP-1104), the foreign-key extractor can read the record's **key as well as its value** (a `BiFunction` overload), so you no longer have to copy a key field into the value just to join on it. ## GlobalKTable: the lookup with no co-partitioning A `GlobalKTable` is replicated in full to **every** application instance, rather than partitioned across them. That single property removes the co-partitioning requirement: because every instance has the whole table locally, any record can be looked up regardless of how the stream side is partitioned. ```java KStream orders = builder.stream("orders"); GlobalKTable countries = builder.globalTable("countries"); KStream enriched = orders.join( countries, (orderKey, order) -> order.getCountryCode(), // map stream record to the table key (order, country) -> new EnrichedOrder(order, country) ); ``` Two things make GlobalKTable joins different from KTable joins: - **You supply a `KeyValueMapper`** that derives the table key from the stream record. You are not restricted to the stream's key, and the stream does **not** need to be re-keyed or co-partitioned. This is the idiomatic way to join on a field in the value without a foreign-key join's overhead. - **It is not time-synchronized.** A GlobalKTable is bootstrapped fully on startup and then updated continuously, but its updates are not aligned to the stream's timestamps the way a KTable's are. For small, slowly-changing reference data (country codes, currency tables, feature flags) that's fine; for large or fast-moving data, full replication to every instance is the cost you pay. Use a GlobalKTable when the right side is small enough to hold entirely in memory on every instance and you want to avoid co-partitioning gymnastics. Reach for a KTable when the data is large, partitioned, and you need timestamp-aligned semantics. ## Frequently Asked Questions **What are the join types available in Kafka Streams?** KStream-KStream (windowed event correlation), KStream-KTable (enrichment lookup of the latest table value), KTable-KTable (a continuously-updated combined view, on the key or on a foreign key), and joins against a GlobalKTable. Which one you use is dictated by the type of each side, stream or table, not by preference. **What is co-partitioning and why do joins require it?** Co-partitioning means both input topics have the same partition count and the same partitioning, so a given key lands in the same partition number on both sides. A join task owns one partition number from each side and matches keys within that partition only, so without it the two halves never meet. Every join except the GlobalKTable join requires it. **Why did my Kafka Streams join start dropping records after an upgrade?** Inner joins always skipped null-key records, and Kafka 2.7 extended that skip to left joins because a null key cannot be co-partitioned or looked up, which teams hit as "the join started dropping data." Kafka 3.7 (KIP-962) relaxed this for `leftJoin`/`outerJoin`, but inner joins still drop null-key records, so set a key with `selectKey` before an inner join. **Why does my KStream-KTable join produce nothing for some records?** A KStream-KTable join looks up the table value as it exists when the stream record is processed, and if the stream record arrives before the table is populated for that key, the lookup finds nothing and the record is silently dropped with no error. Setting `max.task.idle.ms` lets the task wait briefly for the table side to catch up, narrowing the drop window. **When should I use a foreign-key join or a GlobalKTable instead?** Use a KTable-KTable foreign-key join (since Kafka 2.4) to join on a field inside the value rather than the key, it re-keys internally and does not require you to co-partition. Use a GlobalKTable when the right side is small enough to replicate fully to every instance, which removes co-partitioning entirely and lets you look up by any field. > **See it in practice with Conduktor** > A join's co-partitioning contract is visible in your cluster: the partition counts of the two input topics, and the internal repartition and subscription topics Kafka Streams creates to satisfy it. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-joins) lets you inspect those topics, compare partition counts across both join sides, and watch consumer group lag on each input, the first things to check when a join produces nothing. ## Next steps - [Joins that drop data](https://www.conduktor.io/kafka-streams/join-troubleshooting), a diagnostic walkthrough for "my join produces nothing" - [KStream, KTable & GlobalKTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable), the stream-vs-table model joins are built on - [Windowing](https://www.conduktor.io/kafka-streams/windowing), grace, stream-time, and why stream-stream joins need them Source: https://www.conduktor.io/kafka-streams/joins --- # KStream vs KTable vs GlobalKTable *Get the core Kafka Streams mental model right.* Almost every confusing thing in Kafka Streams traces back to one question: is this data a stream of events, or a table of current state? The library gives you three abstractions for that, `KStream`, `KTable`, and `GlobalKTable`, and picking the wrong one is how you end up with joins that drop data, aggregations that double-count, and the occasional `NullPointerException` you can't explain. This is the single most useful concept in the library. Get it right and the rest of the DSL stops feeling arbitrary. **What you'll learn:** - KStream vs KTable: events versus a changelog, and why that distinction is everything - What a tombstone is, and the `NullPointerException` it sets you up for - When GlobalKTable earns its memory cost, and when it doesn't - A concrete rule for which abstraction to reach for ## KStream: an append-only log of events A `KStream` is an unbounded, append-only sequence of records. Every record is an independent fact that *happened*: a click, a payment, a sensor reading. Two records with the same key are two distinct events, not an update, if a customer places three orders, that's three records, and all three matter. ```java KStream orders = builder.stream("orders"); // key="customer-42" appears 3 times → 3 separate order events ``` This is the right model for anything event-shaped: transactions, page views, log lines, IoT readings. You filter them, transform them, route them, and aggregate them, but you never "overwrite" one. The log is the truth, and the order of records carries meaning. ## KTable: a changelog, where the latest value wins A `KTable` is the opposite framing. It models a table where each key has exactly one current value, and the underlying topic is read as a **changelog**: each new record for a key is an *update* (an upsert) that replaces the previous value. Three records for `customer-42` don't accumulate, the third one wins, and the first two are history. ```java KTable profiles = builder.table("user-profiles"); // key="customer-42" appears 3 times → table holds only the latest value ``` This is the right model for entity and reference data: a user's current profile, an account balance, a product's current price, a device's last-known status. You're not interested in every change as an event, you want *the current value per key*, and a `KTable` maintains exactly that, backed by a local [state store](https://www.conduktor.io/kafka-streams/state-store). ### The tombstone: a null value means delete Here is the part the 101 courses gloss over, and it bites people in production. In a changelog, you need a way to say "this key no longer exists." That's a **tombstone**: a record with a non-null key and a **null value**. When a `KTable` sees one, it removes that key from the table. Tombstones have semantics that will surprise you the first time: > 🚫 *"`mapValues` and `filter` run on every record in my KTable, so I'm safe to dereference the value."* They don't. On a `KTable`, `filter` and `mapValues` **skip your function for tombstones** and forward the `null` straight through, the delete has to propagate downstream, so the table can't just drop it. Your lambda never sees it. The trap springs when you call `toStream()`: ```java profiles .mapValues(profile -> profile.toUpperCase()) // skipped for tombstones, null passes through .toStream() .foreach((key, value) -> process(value.length())); // NullPointerException on a deleted key ``` `toStream()` faithfully surfaces those tombstones as records with `null` values, and your downstream `KStream` code, which assumed `mapValues` had already run, dereferences `null`. The fix is to filter nulls explicitly after `toStream()` when you don't care about deletes: ```java profiles .toStream() .filter((key, value) -> value != null) // drop tombstones before touching the value .foreach((key, value) -> process(value.length())); ``` If you *do* care about deletes (propagating them to a downstream system, for instance), keep the tombstone and handle `null` deliberately. Either way, the rule is: **on a stream derived from a table, assume nulls are real and decide what to do with them.** ## Stream-table duality KStream and KTable aren't separate worlds, they're two views of the same data, and you convert between them freely. This is *stream-table duality*, and it's the idea the whole DSL rests on. - A **table is a stream you've folded up.** Replay a changelog from the beginning, keeping only the latest value per key, and you've reconstructed the table. That's literally how a `KTable` is restored after a crash, by replaying its [changelog topic](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction). - A **stream is the changes a table emitted.** Call `toStream()` on a `KTable` and you get the sequence of updates (including tombstones) that produced it. Concretely: you `groupBy` + aggregate a `KStream` and the result is a `KTable` (a running view, see [aggregations](https://www.conduktor.io/kafka-streams/aggregations)); you `toStream()` that `KTable` to publish the changes back out. You move across the boundary constantly, often without thinking about it. ## GlobalKTable: replicated to every instance ![KTable shards keys across 3 instances holding partition subsets; GlobalKTable copies all partitions to every instance with no co-partitioning required](https://www.conduktor.io/assets/kafka-streams/diagrams/ktable-vs-globalktable.svg) A `KTable` is **partitioned**, each instance of your app holds only the keys for the partitions it owns. That's efficient, but it constrains joins: to join a `KStream` against a `KTable`, both sides must be [co-partitioned](https://www.conduktor.io/kafka-streams/joins) (same partition count, same partitioning, so matching keys land on the same task). And don't expect Kafka Streams to catch a violation for you: the famous `Topics not co-partitioned` error only fires when the topology contains an internal repartition topic. Without one, a mismatched join starts fine, reaches RUNNING, and silently misses matches (verified on 3.9 and 4.3). A `GlobalKTable` removes that constraint by making a different trade. It reads **all** partitions of its source topic into **every** instance of your app. Every instance holds a full, complete copy of the table. ```java GlobalKTable countries = builder.globalTable("country-reference"); ``` That full replication buys you two things: - **No co-partitioning required.** Because every instance has every key, a `KStream`-`GlobalKTable` join can look up *any* key locally. The stream side doesn't need to match the table's partition count or be re-keyed first. This is the usual reason people reach for it. - **Bootstrapped before processing, not time-synchronized.** A `GlobalKTable` is loaded fully on startup. The join is a pure point-in-time lookup against current state, it does **not** try to align record timestamps the way a `KStream`-`KTable` join does (best-effort ordering by default; an exact temporal lookup only with versioned state stores, KIP-914). That's a feature for stable reference data, a footgun if you expected event-time correctness. The cost is the part people underestimate. *Every instance stores the entire table.* A small lookup table (a few thousand currency codes, country names, feature flags) is free. A large one is not: > **A 30 GB GlobalKTable is 30 GB on every instance.** Run ten instances and you're holding ten full copies, memory and disk on each, plus the startup time to load all partitions before the app can process anything. Engineers regularly ask whether they can put a multi-gigabyte table in a `GlobalKTable`; the honest answer is that you *can*, but you're paying for it N times over and slowing every restart. For large lookup data, a partitioned `KTable` join (with proper co-partitioning) or an external store is usually the better call. One more limitation: you **can't re-key** a `GlobalKTable`. If you need to join on something other than its key, you supply a `KeyValueMapper` at join time that derives the join key from the stream's record, the table itself stays keyed as-is. ## Which one do I actually use? This is the question that sends people to the forums. A blunt rule covers most cases: | Your data is… | Use | Why | |---|---|---| | Events, each record is an independent fact | **KStream** | Order matters, nothing is overwritten | | Entity / reference data, partitioned, large | **KTable** | Latest value per key, scales with partitions | | Reference / lookup data, small, joined by any key | **GlobalKTable** | Full copy everywhere, no co-partitioning, fast local lookup | Sharper heuristics: - **"Did it happen" → KStream. "What is it now" → KTable.** Payments happen; an account balance *is*. If you'd ever want to replay each occurrence, it's a stream. - **Reach for `GlobalKTable` only when the table is small and you want to skip co-partitioning.** Currency rates, country codes, feature flags, small product catalogs, classic fits. The moment the table is large or grows unbounded, step back to a `KTable`. - **When unsure between KTable and GlobalKTable, default to KTable.** It scales with your partitions instead of replicating in full. You upgrade to `GlobalKTable` deliberately, to dodge a co-partitioning problem, not as a default. The "which do I use for X" confusion almost always dissolves once you classify the *data* as event-vs-entity first, and only then pick the abstraction. ## Frequently Asked Questions **What is the difference between a KStream and a KTable?** A `KStream` is an append-only log of events where each record is an independent fact, so two records with the same key are two separate events. A `KTable` is a changelog where each new record for a key is an upsert, so only the latest value per key is kept. **When should I use a KTable vs a GlobalKTable?** Use a `KTable` for partitioned entity data that scales with your partitions; use a `GlobalKTable` only when the lookup table is small and you want to skip co-partitioning, because it replicates the entire table to every instance. When unsure, default to `KTable` and upgrade to `GlobalKTable` deliberately. **What is stream-table duality?** Stream-table duality is the idea that a stream and a table are two views of the same data. A table is a stream folded up to the latest value per key, and a stream is the sequence of changes a table emits, which is why a `KTable` is restored by replaying its changelog and `toStream()` turns it back into updates. **Why does a GlobalKTable use so much memory?** Because a `GlobalKTable` reads all partitions of its source topic into every instance of your app, so a 30 GB table is 30 GB on each instance, plus the startup time to load it before processing begins. For large lookup data, a partitioned `KTable` join or an external store is usually the better call. **What is a tombstone in a KTable?** A tombstone is a record with a non-null key and a null value; it tells a `KTable` to delete that key. On a `KTable`, `filter` and `mapValues` skip your function for tombstones and forward the null through, so code that dereferences the value after `toStream()` can hit a `NullPointerException`. > **See it in practice with Conduktor** > A `KTable` is just a compacted Kafka topic read as a changelog, and a `GlobalKTable` reads all partitions of one. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-kstream-ktable-globalktable) lets you inspect those source topics, browse records to confirm a key's latest value, spot the tombstones (null-value records) that drive deletes, and check partition counts before you build a join that depends on co-partitioning. ## Next steps - [Kafka Streams joins](https://www.conduktor.io/kafka-streams/joins), co-partitioning, and the GlobalKTable join that skips it - [Aggregations](https://www.conduktor.io/kafka-streams/aggregations), turn a KStream into a KTable with count, reduce, and aggregate - [State stores](https://www.conduktor.io/kafka-streams/state-store), the local store and changelog that back every KTable Source: https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable --- # Kafka Streams monitoring *Learn which Kafka Streams metrics actually matter, and how to expose them.* A Kafka Streams app emits hundreds of metrics. Most dashboards copy the JVM template, heap, GC, thread count, and then go dark exactly when the app falls over. The signals that predict a Streams incident are different ones: consumer group lag, the app's own state, the thread-level ratios, and RocksDB's off-heap memory. None of those live on a stock JVM dashboard. This page is the shortlist: what to watch, why, and how to get each number out of the process. **What you'll learn:** - Why consumer lag and the app's `State` are your first two signals - The built-in metric groups worth alerting on (thread, task, processor-node, state-store) - Recording levels, and why the RocksDB metrics you want are off by default - JMX-to-Prometheus vs in-app Micrometer, and the sidecar trap ## Start where the app is blind: consumer lag Under the hood a Kafka Streams app is a [consumer group](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets). It reads input topics, and the gap between the latest offset on those partitions and the offset the app has committed is **consumer lag**, the single most honest measure of whether the app is keeping up. Lag is also where two of the most common Streams failures *show up first*: - A [rebalance](https://www.conduktor.io/kafka-streams/rebalancing) parks processing while partitions move; lag climbs on the affected partitions, then drains once the group stabilizes. - A [state restore](https://www.conduktor.io/kafka-streams/state-restore) after a deploy or failover blocks a task until its changelog has replayed; lag on that task's partitions stays pinned high for the whole restore. Watch lag **per partition**, not just the group total. A single hot or stuck partition (a skewed key, one task restoring) is invisible in the sum but obvious per-partition. The metric on the consumer is `records-lag-max` (in the `consumer-fetch-manager-metrics` group), but the cleaner source is the broker-side committed-offset gap, which is what an external lag monitor reads. > **Lag alone won't tell you *why*.** Rising lag means "behind," not "broken." Pair it with the app `State` and the rebalance/restore signals below, climbing lag during a `REBALANCING` state is a rebalance; climbing lag during `RUNNING` with a restore in progress is a restore; climbing lag during a healthy `RUNNING` state is a genuine throughput shortfall (scale out, see [scaling](https://www.conduktor.io/kafka-streams/scaling)). ## The app's own state: the StateListener Kafka Streams exposes a coarse lifecycle state (`CREATED`, `REBALANCING`, `RUNNING`, then `PENDING_SHUTDOWN`/`NOT_RUNNING` on a clean close, `PENDING_ERROR`/`ERROR` on a fatal failure), and you should treat transitions into and out of it as first-class alerts. Register a `StateListener`: ```java streams.setStateListener((newState, oldState) -> { // Export newState as a gauge; alert if REBALANCING persists too long, // or on any transition to ERROR. stateGauge.set(newState.ordinal()); if (newState == KafkaStreams.State.ERROR) { pageOncall("Streams app entered ERROR"); } }); ``` The actionable rule: a brief `REBALANCING` is normal (every deploy causes one). A `REBALANCING` state that *persists*, minutes, not seconds, means a rebalance storm or a long restore, and that is page-worthy. A transition to `ERROR` means the instance hit a fatal condition (all stream threads died, or the global stream thread died) and has shut itself down; nothing is being processed. There is also a per-thread state and, for restores specifically, a dedicated listener covered below. ## The built-in metric groups worth your attention ![Two-donut comparison: healthy state shows process as the largest lime slice with poll, commit, punctuate as smaller slices summing to 1.0; starvation state shows punctuate enlarged to half the donut, shrinking process to 20 percent with annotation showing punctuate steals from process](https://www.conduktor.io/assets/kafka-streams/diagrams/stream-thread-phase-ratios.svg) Streams publishes metrics through JMX in a handful of groups. You do not need all of them. These are the ones that earn dashboard space. | Group | Metric | What it tells you | |---|---|---| | `stream-thread-metrics` | `process-rate`, `poll-rate`, `commit-rate` | Throughput of each stream thread | | `stream-thread-metrics` | `process-latency-avg`, `commit-latency-avg`, `poll-latency-avg` | Where a thread spends its time | | `stream-thread-metrics` | `process-ratio`, `commit-ratio`, `poll-ratio`, `punctuate-ratio` | Fraction of the loop spent in each phase, the fastest read on health | | `stream-task-metrics` | `process-rate`, `commit-rate` | Per-task throughput (a task = one partition of a sub-topology) | | `stream-task-metrics` | `enforced-processing-rate` | How often Streams force-processed despite an empty input buffer | | `stream-processor-node-metrics` | `process-rate` | Throughput at a single operator, finds the slow node in a topology | | `stream-processor-node-metrics` | `record-e2e-latency-avg` / `-max` | End-to-end latency from source-record timestamp to this node (KIP-613) | | `stream-state-metrics` | `put-rate`, `get-rate`, `fetch-rate`, `flush-rate` | State store access rates | A few of these reward explanation: - **The phase ratios** (`process-ratio` / `commit-ratio` / `poll-ratio` / `punctuate-ratio`) sum to roughly 1.0 and tell you what the thread is *actually doing*. A thread spending most of its time in `commit-ratio` is over-committing (lower `commit.interval.ms` pressure or EOS overhead); high `punctuate-ratio` means a [punctuator](https://www.conduktor.io/kafka-streams/processor-api) is eating the loop, and because punctuators run *on* the stream thread, that directly starves processing. - **`enforced-processing-rate`** (a task metric, recorded at DEBUG) counts the times a multi-input task gave up waiting for a slow input and processed anyway after `max.task.idle.ms`. With the default `max.task.idle.ms=0` it fires every time one input buffer is empty, so a non-zero, climbing value is routine; the metric only becomes a signal once you raise the idle time, and then it correlates with out-of-order [join](https://www.conduktor.io/kafka-streams/join-troubleshooting) surprises. - **`record-e2e-latency`** (KIP-613) measures from the *record's timestamp* to when a node processes it. Useful, with a caveat: because the start point is the record timestamp, it inflates for late-arriving data and for records that sat buffered before a long commit interval, it is latency-since-the-event-happened, not latency-since-arrival. ### The state-store and RocksDB metrics For stateful apps, the state store is where memory and disk pressure build, and the [RocksDB](https://www.conduktor.io/kafka-streams/rocksdb-tuning) metrics are the only window into the off-heap world your JVM dashboard can't see. Two recording-level tiers matter here (the INFO gauges from KIP-607, the DEBUG statistics from KIP-471): | Metric | What it shows | Recording level | |---|---|---| | `size-all-mem-tables` | Off-heap write-buffer (memtable) memory in use | INFO | | `block-cache-usage` | Off-heap block-cache memory in use | INFO | | `total-sst-files-size` | On-disk size of the store's SST files | INFO | | `write-stall-duration-avg` / `-total` | Time writes were stalled by compaction backpressure | DEBUG (statistics) | | `block-cache-data-hit-ratio` | Read-cache effectiveness | DEBUG (statistics) | `block-cache-usage` and `size-all-mem-tables` are the numbers to sum across stores and compare against the container's memory limit, that comparison is what catches an impending `OOMKilled` (see the myth below). `total-sst-files-size` trending up without bound is the "disk fills silently" failure mode. A non-zero `write-stall-duration-total` means RocksDB is throttling your writes during compaction, a sign the write-buffer budget is undersized. Tuning all of these is its own topic: [RocksDB tuning](https://www.conduktor.io/kafka-streams/rocksdb-tuning). > 🚫 *"My JVM heap dashboard is green, so the Streams app is healthy."* That dashboard is measuring the wrong memory. RocksDB, block cache, memtables, index and filter blocks, lives **off-heap**, outside `-Xmx`, so the heap graph stays flat right up to an `OOMKilled` with no Java stack trace. And health for a Streams app isn't a heap number at all: it's lag draining, `State` at `RUNNING`, restores completing, and `block-cache-usage` / `size-all-mem-tables` staying inside the container budget. Build the dashboard around *those*, not heap and GC. ## Recording levels: most useful metrics are off by default Streams gates metric cost behind `metrics.recording.level`: ```properties # INFO (default): thread/task rates, record-e2e-latency, the INFO-level RocksDB memory & disk gauges # DEBUG: per-processor-node metrics, enforced-processing-rate, RocksDB stall/hit-ratio stats # TRACE: everything metrics.recording.level=DEBUG ``` The default is `INFO`. That gives you lag-adjacent thread/task rates, the INFO-level RocksDB gauges (`block-cache-usage`, `size-all-mem-tables`, `total-sst-files-size`), and, usefully, `record-e2e-latency`, which is the one processor-node metric reported at INFO rather than DEBUG. The rest of the per-processor-node metrics, `enforced-processing-rate`, and the RocksDB statistics (`write-stall-duration-avg`/`-total`, `*-hit-ratio`) are **DEBUG**, and the RocksDB statistics carry real overhead, so enable them deliberately when you're chasing a problem rather than leaving them on everywhere. One trap: the metric *names* are registered regardless of the recording level. At INFO, the DEBUG-tier metrics still show up as MBeans and in `metrics()`, they just read 0 or NaN forever. A dashboard can scrape a metric that silently never records, so verify the level, not just the metric's presence. ## Getting the numbers out: JMX vs Micrometer Streams metrics are JMX MBeans. Two ways to ship them to Prometheus, with a real trade-off: | Approach | How | The catch | |---|---|---| | **Prometheus JMX Exporter** | Run the exporter as a Java agent (`-javaagent`) or sidecar; it scrapes MBeans and exposes `/metrics` | Sidecar **cannot see custom metrics**, anything you register inside the app (your own `StateListener` gauge, business counters) isn't a JMX MBean it knows to scrape unless you also expose it via JMX | | **Micrometer in-app** | Bind `KafkaStreamsMetrics` to a Micrometer registry inside the process; expose `/actuator/prometheus` | One registry for Streams metrics *and* your custom gauges; the natural fit with [Spring Boot](https://www.conduktor.io/kafka-streams/spring-boot) | ```java // Micrometer: one registry for Streams metrics + your own StateListener gauge new KafkaStreamsMetrics(streams).bindTo(meterRegistry); ``` Run the JMX Exporter as a `-javaagent` (not a separate-process sidecar) if you go that route, same JVM, so it sees the MBeans. For most apps that already have a metrics registry, the in-app Micrometer binding is less fragile and keeps custom and built-in metrics in one place. ## The restore signal: StateRestoreListener When a task restores its store from the changelog, processing for that task is blocked, and from the outside it looks like a stuck partition with high lag. The `StateRestoreListener` makes the restore observable so you can tell "restoring" apart from "broken": ```java streams.setGlobalStateRestoreListener(new StateRestoreListener() { public void onRestoreStart(TopicPartition tp, String store, long start, long end) { log.info("Restoring {} for {}: {} records to replay", store, tp, end - start); } public void onBatchRestored(TopicPartition tp, String store, long offset, long n) { /* progress */ } public void onRestoreEnd(TopicPartition tp, String store, long total) { log.info("Restore complete for {}: {} records", store, total); } }); ``` If you'd rather watch the logs, raise the `org.apache.kafka.streams.processor.internals.StoreChangelogReader` logger to DEBUG, it narrates restore progress. Either way, a restore that never reaches `onRestoreEnd` is the thing to chase; that's covered in [state restore time](https://www.conduktor.io/kafka-streams/state-restore). ## Frequently Asked Questions **Which Kafka Streams metrics matter most?** Start with consumer-group lag per partition and the app `State` (via a StateListener). Then the thread-level phase ratios (process/commit/poll/punctuate), per-task `process-rate` and `enforced-processing-rate`, and for stateful apps the RocksDB gauges `block-cache-usage`, `size-all-mem-tables`, and `total-sst-files-size`. Heap and GC are the least useful signals for Streams. **How do I know a rebalance or restore is happening?** A rebalance shows as the app `State` going to `REBALANCING` and lag climbing then draining on the moved partitions. A restore shows as a task pinned with high lag while its `StateRestoreListener` fires `onRestoreStart` but not yet `onRestoreEnd`; raising the `StoreChangelogReader` logger to DEBUG narrates it in the logs. **JMX or Micrometer for Kafka Streams metrics?** Both read the same underlying metrics. The Prometheus JMX Exporter (run as a `-javaagent`, not a separate-process sidecar) needs no code but can't see custom metrics you register in-app. Micrometer binds `KafkaStreamsMetrics` inside the process, so your own gauges and the built-in metrics share one registry, the simpler choice if you already have a registry, and the default with Spring Boot. **How do I monitor RocksDB memory in Kafka Streams?** Watch the INFO-level gauges `block-cache-usage` and `size-all-mem-tables` (off-heap memory) and `total-sst-files-size` (disk), summed across all stores on the instance, against the container's limit, not the JVM heap, which never reflects RocksDB. Enable DEBUG statistics for `write-stall-duration-avg`/`-total` to catch compaction backpressure. **Why is my heap healthy but the container gets OOMKilled?** RocksDB's block cache, memtables, and index/filter blocks are off-heap and not bounded by `-Xmx`, so the heap graph stays flat while off-heap memory grows past the container limit and the Linux OOM killer terminates the process with no Java stack trace. Bound RocksDB with a shared cache and track `block-cache-usage` / `size-all-mem-tables`. > **See it in practice with Conduktor** > Most of the signals above are properties of the consumer group and the internal topics behind your app. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-monitoring) shows consumer-group lag per partition, the partition assignment across your instances, and the changelog and repartition topics Streams creates, so you can tell a rebalance from a restore from a genuine throughput shortfall without instrumenting anything inside the app. ## Next steps - [RocksDB tuning](https://www.conduktor.io/kafka-streams/rocksdb-tuning), bound the off-heap memory the gauges above are watching - [Slow rebalances](https://www.conduktor.io/kafka-streams/rebalancing), why the `REBALANCING` state persists, and how to fix it - [State restore time](https://www.conduktor.io/kafka-streams/state-restore), why a task stays pinned with high lag, and how to speed it up Source: https://www.conduktor.io/kafka-streams/monitoring --- # Kafka Streams Processor API and punctuators *Learn when to drop below the DSL, and how.* The Kafka Streams DSL, `map`, `filter`, `groupBy`, `join`, covers most stream processing cleanly. Then you hit a problem it can't express: emit a record only if a follow-up event *doesn't* arrive within 30 seconds, deduplicate on a business key, route different records to different topics by content, or flush an aggregate on a wall clock instead of on input. For those, you drop one layer down to the **Processor API**: the imperative core that the DSL itself is built on. This is not a lower-level dialect of the same thing. The Processor API hands you three capabilities the DSL deliberately hides: direct read/write access to a state store, explicit control over where each record goes (`context.forward()`), and a scheduled callback that fires on a timer (`context.schedule()` with a `Punctuator`). The cost is that you now own correctness, there is no operator doing the right thing for you. **What you'll learn:** - What `process()`, `Processor`, and `ProcessorContext` give you over the DSL - How to attach a state store and read and write it directly - How `context.forward()` routes records and how punctuators fire on a schedule - The `STREAM_TIME` vs `WALL_CLOCK_TIME` distinction that decides whether your timer ever fires ## Where the DSL ends The DSL is a closed set of operators. Each one is stateless or wires up its own state store and decides for you when to emit. That is exactly what you want until your requirement doesn't map onto any operator. Reach for the Processor API when you need: - **Custom windowing or timeouts**, "alert if a session hasn't closed in N minutes", "expire a pending order after a deadline". The DSL's windows close on stream-time; a timeout on a *quiet* key needs a wall-clock timer the DSL doesn't expose. - **Deduplication on a business key**, there's no `dedup()` operator. You keep seen-keys in a store and drop repeats yourself (see [deduplication](https://www.conduktor.io/kafka-streams/deduplication)). - **Content-based routing to many outputs**, `split()`/`branch()` handles fixed predicates; dynamic "send to the topic named in this field" wants `context.forward()` to a chosen child. - **Manual emit timing**, buffer updates and flush them on a timer rather than per-record or on cache eviction. If your logic *does* fit the DSL, stay there. The DSL gets co-partitioning, repartition topics, store naming, and exactly-once wiring right for free. The Processor API makes all of that your job. > **DSL or Processor API, the decision.** Default to the DSL. Drop to the Processor API for a *specific* operator that the DSL can't express, and use `process()` / `processValues()` to splice it into an otherwise-DSL topology, you do not have to rewrite the whole app in one style. Reserve a fully hand-built `Topology` for the rare case where the entire graph is custom. Mixing is the norm, not a compromise. ## process(), Processor, and ProcessorContext A `Processor` is a small object with three methods: `init`, `process`, and `close`. You splice it into a DSL pipeline with `process()` (or `processValues()` when you only touch the value and want to keep the key for repartition purposes): ```java KStream orders = builder.stream("orders"); orders.process( () -> new DedupProcessor(), // ProcessorSupplier: a fresh instance on every get() "dedup-store" // names of state stores this processor connects to ); ``` The `ProcessorContext` you receive in `init` is the seam to the runtime. Through it you reach the connected state stores, forward records downstream, schedule punctuators, read record metadata (`recordMetadata()` for topic/partition/offset), and commit (`context.commit()` *requests* a commit; the runtime decides when). Since Kafka 3.3 (KIP-820), `process()` returns a `KStream` and chains like any other DSL operator: whatever you `context.forward()` flows straight into the next step, so `.to(...)` works right after it. The real difference between the two: `process()` may change the key, so a downstream stateful operation triggers a repartition; `processValues()` fixes the key and avoids it. ## Connecting and using a state store A processor's store is not implicit. You build and register it, then connect it by name. The store is local, partitioned, and (if persistent) changelog-backed exactly like any DSL store, see [state stores](https://www.conduktor.io/kafka-streams/state-store). ```java // 1. Build the store and add it to the topology StoreBuilder> dedupStore = Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore("dedup-store"), Serdes.String(), Serdes.Long()); builder.addStateStore(dedupStore); // 2. In the processor, fetch it by name in init() public class DedupProcessor implements Processor { private ProcessorContext context; private KeyValueStore store; @Override public void init(ProcessorContext context) { this.context = context; this.store = context.getStateStore("dedup-store"); } @Override public void process(Record record) { Long seenAt = store.get(record.key()); if (seenAt != null) { return; // duplicate: drop it, emit nothing } store.put(record.key(), record.timestamp()); context.forward(record); // novel key: pass it on } @Override public void close() {} } ``` This unbounded store will grow forever, the missing piece is a punctuator that purges old keys. That's the next section, and it's also the difference between a toy and a production processor. ## context.forward() for custom routing In the DSL, where a record goes is implicit in the operator chain. In a processor you say it. `context.forward(record)` sends to all downstream children; you can also forward different records to different named children for content-based routing: ```java if (order.amount() > LARGE_THRESHOLD) { context.forward(record, "to-review"); } else { context.forward(record, "to-auto-approve"); } ``` You can forward zero records (a filter), one (a map), or many (a flatMap) per input, and you can forward from inside a punctuator, not only from `process()`. That last point is what makes timed emission possible. One caveat: `recordMetadata()` is empty during a punctuation, because there is no current record; guard it before use. ## context.schedule() and punctuators ![STREAM_TIME punctuators freeze when input stops; WALL_CLOCK_TIME fires at even intervals through idle gaps regardless of records](https://www.conduktor.io/assets/kafka-streams/diagrams/processor-punctuator-time.svg) A `Punctuator` is a callback you register with `context.schedule(interval, type, punctuator)`. It fires on a schedule independent of input records, which is the whole reason to use it, it lets a processor act when *nothing* is arriving. The `PunctuationType` decides what "schedule" means, and getting this wrong is the most common Processor API mistake. | `PunctuationType` | Fires when | Fires on idle input? | Use it for | |---|---|---|---| | `STREAM_TIME` | Stream-time advances past the interval; stream-time is the highest timestamp of records the task has processed | **No.** It freezes the moment input stops | Logic that must track event time (event-time windowing, watermark-like flushes) | | `WALL_CLOCK_TIME` | The system clock passes the interval, evaluated on each `poll()` loop, regardless of input | **Yes** | Timeouts, periodic flush, purging state on quiet streams | The trap: `STREAM_TIME` is driven only by records the task actually processes. Stream-time is the *maximum* timestamp seen so far, and it freezes entirely when input stops, so on an idle stream your `STREAM_TIME` punctuator **never fires**. This is the same mechanism that makes [suppress() emit nothing on idle input](https://www.conduktor.io/kafka-streams/suppress-not-emitting). (A single quiet partition does not stall it under default config: `max.task.idle.ms` defaults to 0, and a positive value only makes the task wait up to that bound.) If your job is "flush or time out even when the stream is silent", you need `WALL_CLOCK_TIME`. Here's the dedup processor completed with a wall-clock punctuator that expires keys older than a TTL, so the store stays bounded even if some keys never reappear: ```java @Override public void init(ProcessorContext context) { this.context = context; this.store = context.getStateStore("dedup-store"); // Fire on the wall clock so purging happens even when input is quiet. this.cancellable = context.schedule( Duration.ofMinutes(1), PunctuationType.WALL_CLOCK_TIME, this::purgeExpired); } private void purgeExpired(long now) { long cutoff = now - TTL_MS; try (KeyValueIterator all = store.all()) { while (all.hasNext()) { KeyValue entry = all.next(); if (entry.value < cutoff) { store.delete(entry.key); } } } } ``` Note `now` passed to the punctuator is wall-clock millis here; under `STREAM_TIME` it would be the current stream-time instead. Two scheduling surprises: the first `STREAM_TIME` fire happens on the first record processed, without waiting out the initial interval, and a stream-time jump across several intervals produces a single punctuation at the new time, not one call per missed boundary. The `schedule` call returns a `Cancellable`, hold onto it. > 🚫 *"The punctuator runs on its own timer thread, so a slow one is harmless."* A punctuator runs **inline on the stream thread**, in the same loop as `process()`, the two are never concurrent. So a slow punctuator doesn't just delay itself: it blocks record processing, holds up the consumer `poll()`, and delays commits. Iterate too long over a large store and you can blow `max.poll.interval.ms` and trigger a [rebalance](https://www.conduktor.io/kafka-streams/rebalancing). Keep punctuator work cheap, evict in bounded batches, and push heavy work off-thread rather than doing it in the callback. (This is also why wall-clock punctuation is "best effort": its granularity is bounded by how long a processing iteration takes.) ## Cancel punctuators in close() A scheduled punctuator is tied to the task: its punctuation queue is a per-task field, dropped with the task when it closes, so an uncancelled punctuator cannot outlive a [rebalance](https://www.conduktor.io/kafka-streams/rebalancing) or fire against a store the task no longer owns. Cancel in `close()` anyway: it is the mirror of `init()`, and it is the only way to stop a conditional or re-scheduled punctuator inside a live task: ```java @Override public void close() { if (cancellable != null) { cancellable.cancel(); } } ``` `close()` runs when the task is revoked or the app shuts down. Treat it as the mirror of `init()`: release schedules and any resources you opened, and do *not* close the state store yourself, the runtime owns its lifecycle. ## A note on manual commit Under the default at-least-once guarantee, `context.commit()` lets a processor *request* an offset commit at a logical boundary, for example after a punctuator has flushed a batch downstream. It is a request, not an immediate commit; the runtime still commits on its own interval. Under exactly-once it still works: `commit()` requests closing the current transaction earlier than `commit.interval.ms` (which defaults to 100ms under EOS), but the transaction boundary remains the runtime's, so there is even less reason to call it (see [exactly-once](https://www.conduktor.io/kafka-streams/exactly-once)). Don't reach for the Processor API *just* to control commits, reach for it for state, routing, or timers, and let the processing guarantee handle atomicity. ## Frequently Asked Questions **When should I use the Processor API instead of the DSL in Kafka Streams?** Default to the DSL and drop to the Processor API for a *specific* requirement it can't express: custom windowing or timeouts on quiet keys, deduplication on a business key, content-based routing to dynamic outputs, or buffering and flushing on a timer. The DSL gets co-partitioning, repartition topics, store naming, and exactly-once wiring right for free, so only take on that responsibility where you need the control. **Do I have to rewrite my whole Kafka Streams app to use the Processor API?** No. Splice a single processor into an otherwise-DSL topology with `process()` or `processValues()` and keep the rest in the DSL. Mixing the two is the norm; reserve a fully hand-built `Topology` for the rare case where the entire graph is custom. **How do I access a state store directly from a processor?** Build the store with a `StoreBuilder` and register it via `builder.addStateStore(...)`, then in the processor's `init()` fetch it by name with `context.getStateStore("store-name")`. The store is local, partitioned, and changelog-backed exactly like any DSL store; connect its name when you call `process(...)`. **How do I schedule a punctuator in Kafka Streams?** Call `context.schedule(interval, type, punctuator)` in `init()` and hold the returned `Cancellable` so you can cancel it in `close()`. A punctuator fires on a schedule independent of input records, which is what lets a processor act when nothing is arriving, for example purging expired keys from a state store. **What is the difference between STREAM_TIME and WALL_CLOCK_TIME punctuation?** `STREAM_TIME` advances only with records the task actually processes, it is the maximum timestamp seen so far, so it freezes when input stops and the punctuator never fires on an idle stream. `WALL_CLOCK_TIME` fires off the system clock on each `poll()` regardless of input, so use it for timeouts, periodic flushes, and purging state on quiet streams; both run inline on the stream thread, so keep the work cheap. > **See it in practice with Conduktor** > A Processor API app is still a consumer group writing to internal topics. When you add a manual state store, Kafka Streams creates its changelog topic behind the scenes. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-processor-api) lets you find that changelog, watch its size and compaction, and track the consumer group lag that tells you whether a punctuator-driven flush is keeping up or falling behind. ## Next steps - [Deduplication in Kafka Streams](https://www.conduktor.io/kafka-streams/deduplication), the canonical Processor API use case, end to end - [State stores](https://www.conduktor.io/kafka-streams/state-store), RocksDB, changelogs, and the memory that bites - [When suppress() emits nothing](https://www.conduktor.io/kafka-streams/suppress-not-emitting), the stream-time stall, the DSL version of the same trap Source: https://www.conduktor.io/kafka-streams/processor-api --- # Kafka Streams in Python? *Get the honest answer on Python stream processing for Kafka.* If you searched "Kafka Streams Python," you were probably hoping for a `pip install kafka-streams`. The honest answer first, so you don't waste an afternoon: **Kafka Streams is a JVM library: Java and Scala only. There is no official Python port, and there isn't going to be one.** It's part of Apache Kafka itself, written in Java, and it leans on JVM-specific machinery (RocksDB via JNI, the JVM consumer client) that doesn't transplant to Python. That's the bad news. The good news is that stream processing on Kafka from Python is absolutely doable: you just use a different tool than Kafka Streams. This page lays out the real options and where each one fits, without overselling any of them. **What you'll learn:** - Why there's no official Python Kafka Streams (and what that rules out) - When a plain Python consumer is all you need - The Python-native stream-processing libraries (Faust, Quix Streams, Bytewax) and their trade-offs - When to push the stateful work to a JVM app, Flink, or SQL instead > 🚫 *"There's a Python port of Kafka Streams, I just need to find the right package."* There isn't. Packages named to look like one are either thin client wrappers or unrelated projects. What exists instead is a set of *independent* Python stream-processing libraries (below) that solve similar problems with their own designs, not a port of the Kafka Streams DSL or its RocksDB-backed state model. ## Option A: a plain Python consumer (stateless / simple) If your job is stateless and per-record (read a message, transform it, maybe produce a result), you don't need a stream-processing framework at all in any language. The official `confluent-kafka` client (librdkafka-based) or `kafka-python` gives you a consumer and producer, and that's enough. This is the Python equivalent of choosing a [plain consumer over Kafka Streams](https://www.conduktor.io/kafka-streams/vs-consumer-api): the same logic applies regardless of language. ```python from confluent_kafka import Consumer, Producer consumer = Consumer({"bootstrap.servers": "localhost:9092", "group.id": "paid-orders", "auto.offset.reset": "earliest"}) producer = Producer({"bootstrap.servers": "localhost:9092"}) consumer.subscribe(["orders"]) while True: msg = consumer.poll(1.0) if msg is None or msg.error(): continue if b'"status":"PAID"' in msg.value(): producer.produce("paid-orders", key=msg.key(), value=msg.value()) ``` Where it stops: the moment you need aggregations, joins, windowing, or fault-tolerant local state, you'd be hand-building what a framework gives you. Don't reinvent it: pick one of the options below. ## Option B: Faust (Streams-like, but read the maintenance note) [Faust](https://faust-streaming.github.io/faust/) is the closest thing in spirit to Kafka Streams in Python: a library (not a cluster) where you define agents that process streams, with support for tables (stateful aggregations) backed by a changelog topic and a local store. If you want the Kafka-Streams *mental model* in Python, Faust is the nearest fit. The caveat you must know before adopting it: the **original Robinhood `faust` project was abandoned**, and the maintained line today is the community fork **`faust-streaming`**. It's alive, but it's community-maintained rather than backed by a vendor. Weigh that against the deep institutional backing Kafka Streams has inside Apache Kafka. For a long-lived production system, treat the maintenance model as a first-class part of the decision, not a footnote. ## Option C: Quix Streams and Bytewax (modern Python-native) Two newer libraries built for Python from the ground up, rather than mirroring a JVM API: - **[Quix Streams](https://github.com/quixio/quix-streams)**: a Python library for Kafka stream processing with a DataFrame-style API, stateful operations, and windowing. Kafka-centric, like Kafka Streams, but Python-native. - **[Bytewax](https://github.com/bytewax/bytewax)**: a Python stream-processing framework with a Rust core (built around a dataflow model). Sources and sinks go beyond Kafka, and it leans toward the data/ML ecosystem. Mind the maintenance signal, which differs sharply between them: Quix Streams is actively developed and commercially backed, while Bytewax became community-maintained in 2025 after its backing company wound down and the original team stepped back. Weigh that the same way you'd weigh Faust's. Both are younger and have smaller ecosystems than Kafka Streams or Flink, so check that the connectors, state guarantees, and operational story you need are actually there before committing: don't assume parity with a mature JVM engine. ## Option D: Flink (PyFlink), Flink SQL, or ksqlDB If you'd rather not run a Python library at all, push the processing to an engine that exposes a non-Java interface: - **PyFlink / Flink SQL**: Apache Flink offers a Python API and full SQL, and runs as a cluster a platform team operates. Good when you have heterogeneous sources, large scale, or want SQL-defined pipelines. See [Kafka Streams vs Flink](https://www.conduktor.io/kafka-streams/vs-flink) for the deployment-model trade-off (most of that comparison applies here too). - **ksqlDB**: SQL-over-Kafka, no Python or Java code at all. Be aware it's under lower active investment than it once was, with new SQL stream-processing momentum moving toward Flink SQL (covered in the [vs Flink](https://www.conduktor.io/kafka-streams/vs-flink) page). The trade-off versus a Python library: you operate a separate engine instead of embedding processing in your service. For SQL-shaped or very large workloads that's often the better deal. ## Option E: keep stateful processing on the JVM, consume results in Python A pattern worth naming because it sidesteps the whole problem: do the heavy stateful stream processing in a [Kafka Streams](https://www.conduktor.io/kafka-streams) app (JVM), write the results to an output topic, and let your Python services consume that topic with a plain consumer. Your Python code stays simple (it reads a derived, already-processed stream) and you keep Kafka Streams' mature state, exactly-once, and windowing where they're strongest. If your organization is already part-JVM, this is often the lowest-risk answer. ## Honest trade-offs | Option | State / joins / windows | Maturity & backing | Operate a separate engine? | Best for | |---|---|---|---|---| | **Plain consumer** (`confluent-kafka`) | No (DIY) | Mature, official client | No | Stateless, per-record work | | **Faust** (`faust-streaming` fork) | Yes | Community-maintained; original abandoned | No (library) | Streams-like model in Python, maintenance risk accepted | | **Quix Streams** | Yes | Actively developed, commercially backed | No (library) | Python-native, Kafka-centric processing | | **Bytewax** | Yes | Community-maintained since 2025 (company wound down) | No (library) | Python/ML dataflow, multi-source | | **PyFlink / Flink SQL** | Yes | Mature engine | Yes (cluster) | Heterogeneous sources, scale, SQL | | **ksqlDB** | Yes | Stable but low investment | Yes (server) | SQL-only transforms, no code | | **JVM Streams + Python consumer** | Yes (on JVM) | Mature (Kafka Streams) | No extra beyond Streams | Already part-JVM; keep Python simple | No single row is the universal answer. The split that decides it: stateless → plain consumer; stateful and you want it in Python → Faust / Quix / Bytewax (and check maturity); stateful, SQL-shaped, or huge → Flink/ksqlDB; already on the JVM → keep Streams and consume the output. ## Frequently Asked Questions **Is there a Python version of Kafka Streams?** No. Kafka Streams is a JVM (Java/Scala) library that's part of Apache Kafka, and there is no official Python port. Libraries like Faust, Quix Streams, and Bytewax are independent Python stream-processing tools inspired by similar ideas, not a port of the Kafka Streams API. **What is the Python equivalent of Kafka Streams?** The closest in spirit is Faust (specifically the maintained `faust-streaming` fork), which mirrors the library-and-tables model. Quix Streams and Bytewax are modern Python-native alternatives. For SQL or very large workloads, Flink SQL / PyFlink or ksqlDB are the usual answers instead. **Is Faust still maintained?** The original Robinhood `faust` project was abandoned. The actively maintained line today is the community fork `faust-streaming`. It works, but it's community-maintained rather than vendor-backed, so weigh the maintenance model before building a long-lived production system on it. **Can I do stateful stream processing in Python with Kafka?** Yes. Faust, Quix Streams, and Bytewax all support stateful operations like aggregations and windowing from Python. Alternatively you can run Flink (PyFlink/SQL) or ksqlDB, or keep the stateful work in a JVM Kafka Streams app and have Python consume the results. **Can I just use confluent-kafka or kafka-python for stream processing?** For stateless, per-record work (read, transform, produce), yes, a plain consumer and producer are enough. Once you need aggregations, joins, windowing, or fault-tolerant local state, you'd be rebuilding a framework by hand; reach for a stream-processing library or engine instead. > **See it in practice with Conduktor** > Whichever Python option you land on, it reads and writes Kafka, and shows up on the cluster as a consumer group with lag and offsets, plus any topics it creates for state. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-python) lets you watch that consumer group's lag, browse the input and output topics, and confirm records are flowing: the same observability whether the processing runs in a Python library, a JVM Streams app, or a Flink job. ## Next steps - [What is Kafka Streams?](https://www.conduktor.io/kafka-streams): why it's a JVM library, in depth - [Kafka Streams vs Flink vs ksqlDB](https://www.conduktor.io/kafka-streams/vs-flink): the non-JVM engine options, compared honestly - [Build your first Kafka Streams app](https://www.conduktor.io/kafka-streams/getting-started): if the JVM route is on the table Source: https://www.conduktor.io/kafka-streams/python --- # Why Kafka Streams rebalancing is slow *Diagnose and fix slow Kafka Streams rebalances.* A rebalance in a plain consumer group is cheap: partitions move, consumers resume, life goes on. A rebalance in a *stateful* Kafka Streams app can stall every instance for minutes while local state is rebuilt, and the lag piling up during that stall is what gets you paged. This is the single most common production complaint about Kafka Streams, and the official troubleshooting for it is thin. This page is about why that happens and the specific configs that make it stop. **What you'll learn:** - Why a stateful rebalance is effectively stop-the-world - The real symptoms: multi-minute stalls, `Running`↔`Rebalancing` loops, rebalance storms on Kubernetes - The configs that trigger rebalances you didn't ask for - The fix toolbox: cooperative rebalancing, warm-up replicas, standbys, static membership ## Why a stateful rebalance stops the world When the [consumer group protocol](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets) reassigns work, a stateless consumer just starts polling its new partitions. A Kafka Streams task can't. Each stateful task owns a [state store](https://www.conduktor.io/kafka-streams/state-store) (a local RocksDB database backed by a changelog topic) and it cannot process a single record until that store holds the right data for its partition. So when a task lands on an instance that has no local copy of its state, the instance must **restore** the store by replaying the changelog topic before processing resumes. With the classic *eager* rebalance protocol, the sequence is brutal: 1. Every instance revokes **all** of its tasks (and releases their stream threads). 2. The group leader computes a new assignment. 3. Each instance picks up its newly assigned tasks and restores any state it's missing. 4. Only then does processing resume, across the **whole** application. Steps 1 and 4 are the problem. Under eager rebalancing, one instance joining or leaving makes the entire app stop processing while everyone hands back their work and re-acquires it. If any reassigned task has a large store, step 3 dominates and the stall stretches from seconds into minutes. The restore itself (why a big store takes so long) is its own topic; see [state restore time](https://www.conduktor.io/kafka-streams/state-restore). The headline number people quote ("my rebalance took 40 minutes") is almost always restore time wearing a rebalance costume. ## What it actually looks like in production The symptoms are distinctive once you've seen them: - **Processing stalls for minutes, then catches up.** Consumer-group lag climbs in a straight line during the stall, then drains. A deploy or a single pod restart triggers it. - **A `Running` ↔ `Rebalancing` loop after a client upgrade.** The app never settles. State threads keep getting revoked. This usually means a mismatch in the rebalance protocol or assignor across instances during a rolling upgrade: half the group speaks one protocol, half speaks another. - **Rebalance storms on Kubernetes.** A team running 10 pods, 7 stream threads each, across ~180 partitions sees rebalances that take four to five minutes and seem to fire constantly. High thread-to-partition ratios make every reassignment touch more tasks, and spot-instance churn keeps the group membership moving. - **A rolling restart rebalances on every single pod.** You expected one settle at the end. Instead each pod that bounces triggers a full group rebalance. That last one surprises people, and it's not a bug. > **Kafka Streams opts out of clean leave-on-close, on purpose.** A normal consumer leaves the group on a graceful shutdown, triggering one prompt rebalance. Kafka Streams deliberately keeps a closing instance in the group: through Kafka 4.1 it forced `internal.leave.group.on.close=false`; since 4.2 that internal config is gone and `close()` defaults to `REMAIN_IN_GROUP` ([KIP-1153](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1153%3A+Rework+of+CloseOptions+in+KafkaStreams%3A+Replace+Class+with+Interface-Based+Approach)), with `KafkaStreams.close(CloseOptions.groupMembershipOperation(LEAVE_GROUP))` as the explicit opt-out for scale-in. The reasoning: a Streams instance is usually *coming back* (a rolling restart, a pod reschedule), and you don't want to reshuffle expensive state every time one bounces. The cost is that a shut-down instance stays "in" the group until its `session.timeout.ms` expires, so a rolling restart still rebalances, just on the session-timeout clock rather than instantly. Static membership (below) is the tool that turns this caveat into an advantage. ## The configs that trigger rebalances you didn't ask for Before tuning the rebalance *mechanism*, stop the *spurious* rebalances. Most "constant rebalancing" reports trace back to one of these: | Config | Default | How it bites | |---|---|---| | `max.poll.interval.ms` | 300000 (5 min) | If one `poll()`→`poll()` cycle takes longer than this (a slow punctuator, a blocking external call, a long restore on the same thread), the broker assumes the instance is dead and kicks it out. It rejoins → rebalance. A too-low value here is the classic self-inflicted rebalance loop. | | `session.timeout.ms` | 45000 (since 3.0) | The heartbeat deadline. Set too low for your network or GC pauses, a transient stall looks like death and ejects the instance. Set higher, and you tolerate blips at the cost of slower detection of *real* failures. | | `heartbeat.interval.ms` | 3000 | Should stay roughly ⅓ of `session.timeout.ms`. Rarely the root cause alone, but a mismatch makes the session timeout behave unexpectedly. | The trap is reaching for `session.timeout.ms` when the real culprit is `max.poll.interval.ms`. They fail differently: blowing the session timeout means heartbeats stopped (the instance was unreachable or paused); blowing the poll interval means heartbeats kept flowing but the application thread was stuck between polls. Check which one the logs name before you touch either. > 🚫 *"We're on cooperative rebalancing, so rebalances don't pause processing anymore."* Cooperative rebalancing reduces the pause dramatically, but it does not make it zero, least of all for stateful apps. A task that *moves* to a new instance still has to restore its state before it runs there, and that instance is still blocked on that task during the restore. Cooperative rebalancing's win is that tasks which *stay put* keep running; it does not abolish the cost of moving state. If you've adopted it and your stalls didn't vanish, restore time is your real problem, not the protocol. ## Fix #1: incremental cooperative rebalancing ![Eager rebalancing revokes all tasks on instances A, B, C simultaneously for 37,138 ms; cooperative rebalancing revokes only the one moving task on instance A while B and C keep running, cutting pause to 3,522 ms](https://www.conduktor.io/assets/kafka-streams/diagrams/rebalancing-eager-vs-cooperative.svg) The biggest single improvement, and you most likely already have it. The eager protocol's "revoke everything, then redistribute" dance is what made one membership change freeze the whole group. **Incremental cooperative rebalancing** ([KIP-429](https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol)) changes that: instances keep the tasks they're going to retain and only revoke the specific tasks that need to move. Most of the app keeps processing through the rebalance. This has been the **default in Kafka Streams since 2.4** via the `StreamsPartitionAssignor`. You don't enable it with a flag; you get it unless you've overridden the assignor or you're running a version old enough that you have other problems. (For the underlying consumer-protocol mechanics, see [incremental rebalancing and static membership](https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership).) How much does it matter? Confluent's published benchmark, a 10-instance stateful Streams app with RocksDB stores under a rolling bounce, measured total pause time dropping from **37,138 ms to 3,522 ms**, roughly a 10x reduction in stop-the-world time, just by moving from eager to cooperative ([confluent.io/blog](https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/)). The one operational catch: you cannot jump protocols in a single rolling bounce. Upgrading from a pre-2.4 eager deployment requires a documented two-rolling-restart upgrade path, because the group can't mix eager and cooperative members arbitrarily. That mismatch is, not coincidentally, a common cause of the `Running`↔`Rebalancing` loop after an upgrade. ## Fix #2: warm-up replicas (probing rebalances) Cooperative rebalancing stops *unrelated* tasks from pausing, but a task that genuinely needs to move to a new instance still blocks that instance while it restores. [KIP-441](https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams) makes that migration non-disruptive. When the assignor wants to move a stateful task, it does **not** hand the active task over immediately. It first places a **warm-up replica** on the destination instance, which restores the store in the background (off the hot path) while the original instance keeps serving the active task. Periodic *probing rebalances* check whether the warm-up has caught up. Only once it's within `acceptable.recovery.lag` of the changelog tail does the active task actually switch over, which is then near-instant. | Config | Default | What it controls | |---|---|---| | `acceptable.recovery.lag` | 10000 | How many records behind the changelog a warm-up store can be and still be treated as "caught up" and eligible to take over the active task. Lower = stricter (longer warm-up, faster handover); higher = looser. | | `max.warmup.replicas` | 2 | The cap on how many warm-up replicas the assignor will move at once. Raise it to migrate state faster when scaling out, at the cost of extra restore traffic and broker load. | | `probing.rebalance.interval.ms` | 600000 (10 min) | How often the assignor runs a probing rebalance to check warm-up progress and rebalance toward the balanced assignment. | The practical effect: scaling out or replacing an instance no longer means "lag spikes until the new node finishes restoring." The warm-up absorbs the restore cost in the background. The tradeoff is that probing rebalances are *extra* rebalances by design, and the app may run on a temporarily unbalanced assignment for several minutes until warm-ups complete. That is the correct behavior, even though it can look alarming on a dashboard. ## Fix #3: standby replicas `num.standby.replicas` (default `0`) tells Streams to keep *N* extra copies of each store continuously updated on other instances. These standbys tail the changelog in real time, so they're always close to current. Standbys are primarily a **crash** insurance: if an instance dies, a task can fail over to an instance that already holds a hot copy of its state, skipping most of the restore. They cost extra disk, memory, and changelog-consumption traffic on every instance: you're maintaining N+1 copies of your state. Crucially, standbys interact with warm-up replicas: a standby that's already hot is a much cheaper starting point for promotion than a cold instance restoring from zero. Set `num.standby.replicas=1` if your restore times are painful and you can afford the resource overhead. Just be clear about what they do and don't cover: the rolling-restart caveat is covered in depth in [state restore time](https://www.conduktor.io/kafka-streams/state-restore). ## Fix #4: static membership, do or don't Static membership ([group.instance.id](https://www.conduktor.io/kafka/consumer-incremental-rebalance-and-static-group-membership)) gives each instance a stable identity. When an instance with a static ID disconnects and reconnects within `session.timeout.ms`, the broker recognizes it as the *same* member and hands back its previous assignment **without a rebalance at all**. This is the direct antidote to the rolling-restart problem. A pod bounces, comes back with the same `group.instance.id` inside the session window, and reclaims its tasks and its local state: no reshuffle, no restore. ```properties # Each instance MUST get a UNIQUE, STABLE value. # On Kubernetes, the StatefulSet pod ordinal works well: group.instance.id=streams-app-${POD_NAME} session.timeout.ms=120000 # must exceed your worst-case restart-to-rejoin time ``` The honest tradeoff, and why it's a "do or don't" rather than an "always do": - **The IDs must be unique and stable.** Two instances with the same `group.instance.id` get fenced. Generate them from something durable (a StatefulSet ordinal), never a random UUID per process: a fresh UUID on every restart defeats the entire mechanism. - **It widens your real-failure detection window.** The same `session.timeout.ms` that lets a planned restart skip the rebalance also delays detecting a genuinely dead instance. Set it long enough to cover a normal restart but short enough that a true crash doesn't strand partitions for minutes. This is a real tension, and there's no universally correct value; pick it from your actual restart timings. - **Pair it with persistent local state.** Static membership lets the rejoining pod *reclaim* its tasks, but it only avoids a restore if the local store survived the restart. On ephemeral pods with no persistent volume, the state is gone and you restore anyway; see [state restore time](https://www.conduktor.io/kafka-streams/state-restore). Use it when restarts are frequent and predictable (Kubernetes, autoscaling). Skip it when your instances are genuinely ephemeral and identity churns, where it adds fragility for little gain. ## What's changing: KIP-848 and KIP-1071 The rebalance machinery itself is being rebuilt, and it's worth knowing where it's headed. - **KIP-848: the new consumer group protocol.** Moves rebalancing off the clients and onto the broker-side group coordinator, replacing the leader-driven "stop and recompute" model with incremental, coordinator-driven reconciliation. It went **GA in Kafka 4.0**. For plain consumers it removes whole classes of rebalance-storm behavior. Streams cannot ride it, though: the Streams main consumer hard-codes `group.protocol=classic`, which is exactly why KIP-1071 exists. - **KIP-1071: a Streams-specific rebalance protocol.** Built on KIP-848's foundation but aware of Streams concepts the plain consumer protocol can't see: tasks, standbys, warm-ups, and stateful assignment. This is the one that targets the pain on this page directly. It reached **GA in Kafka 4.2**, but read the fine print: it's **opt-in** on the client (`group.protocol=streams`; new 4.2 brokers ship the feature enabled) and ships a deliberate subset. Sticky assignment only (no warm-up or rack-aware placement yet), no static membership, no on-the-fly topology changes, and migration to or from the classic protocol is **offline only** (and broken for existing groups in 4.2.0 by a broker-side bug, [KAFKA-20254](https://issues.apache.org/jira/browse/KAFKA-20254)). Pilot it; don't default-migrate production yet. No default-switch has been announced, and client protocol defaults historically flip only at major releases, so most apps still run the classic protocol today. Until then, the four fixes above are what you have, and for most teams cooperative rebalancing plus warm-up replicas plus right-sized timeouts already turns a multi-minute stall into a tolerable blip. ## Frequently Asked Questions **Why is my Kafka Streams rebalance so slow?** The rebalance itself is fast; the slow part is `state restore`. A task that moves to a new instance must replay its changelog into a local RocksDB store before it can process, and a large store can stretch that from seconds into minutes. The "40-minute rebalance" people quote is almost always restore time wearing a rebalance costume. **Why does my Kafka Streams app keep rebalancing in a loop?** A `Running` ↔ `Rebalancing` loop after a client upgrade usually means a rebalance-protocol or assignor mismatch across instances during a rolling upgrade. Constant spurious rebalances more often trace to a too-low `max.poll.interval.ms` (a slow punctuator or blocking call exceeds it, so the broker ejects the instance and it rejoins). **What is cooperative rebalancing and does it stop the pause?** Incremental cooperative rebalancing (KIP-429, default in Streams since 2.4) lets instances keep the tasks they retain and only revoke tasks that must move, so most of the app keeps processing. It reduces the pause dramatically but does not make it zero: a task that moves still has to restore its state before it runs. **Should I use static membership to reduce rebalances?** Use it when restarts are frequent and predictable (Kubernetes, autoscaling): a stable `group.instance.id` lets a bouncing instance reclaim its tasks within `session.timeout.ms` with no rebalance. The IDs must be unique and stable (a StatefulSet ordinal, never a fresh UUID per process), and it widens your real-failure detection window. Pair it with persistent local state, or you still restore. **What is the KIP-1071 Streams rebalance protocol and can I use it?** KIP-1071 is a Streams-aware rebalance protocol built on KIP-848 that understands tasks, standbys, and warm-ups. It reached GA in Kafka 4.2 but is opt-in and ships a deliberate subset (sticky assignment only, no static membership, offline-only migration). Pilot it; don't default-migrate production. Most apps still run the classic protocol today. > **See it in practice with Conduktor** > A rebalance is, from the outside, your consumer group going quiet and lag climbing. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-rebalancing) shows your Streams app's consumer-group lag and partition assignment in real time, so you can see exactly when the group stalls, watch lag drain as restore completes, and confirm tasks landed where you expected, instead of inferring all of it from application logs. ## Next steps - [State restore time & standby replicas](https://www.conduktor.io/kafka-streams/state-restore): why the stall is actually restore, and how to shorten it - [Scaling Kafka Streams](https://www.conduktor.io/kafka-streams/scaling): threads, tasks, and the parallelism ceiling that shapes rebalances - [Kafka Streams architecture](https://www.conduktor.io/kafka-streams/architecture): tasks, threads, and how assignment works under the hood Source: https://www.conduktor.io/kafka-streams/rebalancing --- # Kafka Streams RocksDB tuning *Bound RocksDB memory and disk before it bounds your container.* The [state store](https://www.conduktor.io/kafka-streams/state-store) page told you RocksDB lives off-heap and that your container gets `OOMKilled` while the JVM heap looks fine. This page is the fix in full: where every byte of that off-heap memory goes, the arithmetic that makes it explode on an instance owning many partitions, the `RocksDBConfigSetter` that actually caps it, and the disk failure mode nobody warns you about until the volume is full. This is the deep version. If you haven't read why RocksDB memory is invisible to `-Xmx`, start with [state stores](https://www.conduktor.io/kafka-streams/state-store); here we assume you already know it's a native C++ library and pick up from there. **What you'll learn:** - Exactly which RocksDB structures consume off-heap memory, and which knob bounds each - The per-partition, per-store math that turns a "small" budget into a container OOM - A production `RocksDBConfigSetter` with a strict shared cache + write buffer manager - How disk fills up, and the native-library traps on Alpine and ARM ## Where the off-heap memory actually goes "RocksDB uses off-heap memory" is true but useless for tuning. You need to know the four structures that hold it, because each one has a different default and a different knob. | Structure | What it holds | Default (per store, per partition) | Bounded by | |---|---|---|---| | **Block cache** | Uncompressed data blocks read from SST files | 50 MiB (Streams override; RocksDB's own default is 32 MiB) | `LRUCache` capacity | | **Write buffer (memtable)** | In-memory writes not yet flushed to disk | 16 MiB × up to 3 buffers | `WriteBufferManager` | | **Index & filter blocks** | Per-SST-file index + bloom filters for lookups | grows with data on disk | folded into block cache *if* you opt in | | **Table readers / iterators** | Open file handles, pinned blocks during reads | grows with open files | `max.open.files` | Two of these are the usual culprits. The **block cache** is the big tunable knob: Kafka Streams overrides RocksDB's default (32 MiB since RocksDB 8.2) and gives each store its own fresh 50 MiB `LRUCache`. The **write buffers** are the silent one: every store gets its own 16 MiB memtable (up to three, so it can keep taking writes while flushing filled ones), and that allocation is *not* counted against the block cache unless you wire a `WriteBufferManager` to the same cache object. The index and filter blocks are the trap that grows over time. By default they live *outside* the block cache and expand as your data on disk grows: a store that was fine at 1 GiB of SST files can drift past its memory budget at 50 GiB purely from filter blocks. The fix is `setCacheIndexAndFilterBlocks(true)`, which moves them *into* the block cache so they count against (and are evicted by) your capacity limit. The trade-off: index/filter lookups can now miss the cache and re-read, so you pin the top-level index with `setPinTopLevelIndexAndFilter(true)` to keep the hot part resident. ## The math that gets you OOMKilled Here is the arithmetic that turns "50 MiB, that's nothing" into a dead container. RocksDB allocates **one instance per partition, per store**. The off-heap footprint of a single instance, with Streams defaults, is roughly: ``` block cache 50 MiB + write buffers 3 × 16 MiB = 48 MiB + index/filter (grows with on-disk data, unbounded by default) ≈ 98+ MiB per partition per store, before index/filter growth ``` Now multiply by what an instance actually owns. Tasks are pinned to partitions ([architecture](https://www.conduktor.io/kafka-streams/architecture)), so an instance assigned many partitions runs many RocksDB instances at once: | Instance owns | Stores in topology | RocksDB instances | Off-heap floor (~100 MiB each) | |---|---|---|---| | 10 partitions | 1 | 10 | ~1 GiB | | 20 partitions | 2 | 40 | ~4 GiB | | 30 partitions | 3 (e.g. an aggregation + two joins) | 90 | ~9 GiB | That ~9 GiB is *off-heap*, on top of your `-Xmx`, on top of the JVM's own overhead. Set a 4 GiB container limit with `-Xmx2g` thinking you've left 2 GiB of headroom, and the OOM killer fires the moment a rebalance hands this instance a fat slice of partitions. Windowed stores make it worse: a windowed store keeps multiple **segments** open (one RocksDB instance per active segment), so the partition count above is really partitions × open segments. > 🚫 *"My heap is healthy and GC is calm, so the app has enough memory."* > The heap graph cannot see RocksDB. Off-heap block cache, write buffers, and filter blocks live in native memory; they show up in container/cgroup memory metrics and `OOMKilled` events, never in your JVM heap dashboard. ## The fix: one shared cache for the whole instance ![Default per-store cache gives each partition its own 50 MiB block cache and 48 MiB write buffers multiplying off-heap by N; a shared LRUCache 512 MiB and WriteBufferManager 128 MiB cap total off-heap to a constant ceiling](https://www.conduktor.io/assets/kafka-streams/diagrams/rocksdb-cache-per-store-vs-shared.svg) The default model gives *every* store its own cache and its own write buffers, so memory scales with partition count. The fix is to give **all stores on an instance a single shared `LRUCache` and a single `WriteBufferManager`**, both with a strict capacity. Now the off-heap ceiling is a constant you choose, not a function of how many partitions got assigned. This `RocksDBConfigSetter` is the production-grade version of the snippet in the [state store](https://www.conduktor.io/kafka-streams/state-store) page: same idea, but with the strict-capacity flag, write buffers charged to the cache, and index/filter blocks folded in: ```java public class BoundedMemoryRocksDBConfig implements RocksDBConfigSetter { // ONE budget for ALL RocksDB instances on this app instance (not per store). // Size it from your container limit, leaving room for -Xmx + JVM overhead + page cache. private static final long TOTAL_OFF_HEAP_BYTES = 512L * 1024 * 1024; // 512 MiB private static final long TOTAL_MEMTABLE_BYTES = 128L * 1024 * 1024; // subset for write buffers // strictCapacityLimit=true => RocksDB throws rather than exceeding the cap. private static final Cache CACHE = new LRUCache(TOTAL_OFF_HEAP_BYTES, -1, /* strictCapacityLimit */ true); // WriteBufferManager charged against the SAME cache => memtables count toward the cap. private static final WriteBufferManager WRITE_BUFFER_MANAGER = new WriteBufferManager(TOTAL_MEMTABLE_BYTES, CACHE); @Override public void setConfig(String storeName, Options options, Map configs) { BlockBasedTableConfig table = (BlockBasedTableConfig) options.tableFormatConfig(); table.setBlockCache(CACHE); // Charge index + filter blocks to the cache so they can't grow unbounded. table.setCacheIndexAndFilterBlocks(true); table.setPinTopLevelIndexAndFilter(true); options.setWriteBufferManager(WRITE_BUFFER_MANAGER); options.setTableFormatConfig(table); } @Override public void close(String storeName, Options options) { // CACHE and WRITE_BUFFER_MANAGER are shared across all stores; never close them per store. } } ``` ```properties rocksdb.config.setter=com.example.BoundedMemoryRocksDBConfig ``` Three points that matter and are easy to get wrong: - **`strictCapacityLimit=true` makes the cap real, and loud.** Without it, the `LRUCache` capacity is advisory: RocksDB treats it as a hint and overshoots under load. With it, an allocation that would exceed the cap fails as a `RocksDBException` surfacing on the read or iteration that triggered it, which can kill the stream thread. That is the tradeoff: a loud failure you can alert on instead of the OOM killer. The official Kafka memory-management example deliberately passes `strictCapacityLimit=false` and reserves a high-priority share of the cache for index/filter blocks instead; pick strict when you prefer the exception over the silent overshoot. - **The `WriteBufferManager` must point at the same `Cache`.** That's what makes memtable memory count against the single budget. Pass a separate cache and you've just split your budget in two without realising it. - **Never `close()` the shared objects per store.** `close()` is called once per store; the cache and write buffer manager are `static` and shared, so closing them when one store shuts down would pull the rug out from every other store on the instance. The [Confluent "How to Tune RocksDB for Your Kafka Streams Application"](https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance/) blog walks the same shared-cache pattern and the individual `Options` knobs in more depth; it's the canonical reference and worth reading before you start moving numbers. ## Native memory fragmentation: reach for jemalloc Even with a strict cap, you may watch resident memory (RSS) creep above the sum of your configured limits and never come back down. That's usually not a leak: it's **allocator fragmentation**. RocksDB does a lot of variously-sized native allocations and frees; the default glibc `malloc` arena can hold onto freed pages rather than return them to the OS, and under a churning workload RSS ratchets upward. The standard mitigation is to swap the allocator for **jemalloc**, which fragments far less under this pattern: ```bash # In the container, point the dynamic linker at jemalloc before the JVM starts export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 # path is distro/arch-specific (arm64: /usr/lib/aarch64-linux-gnu/...) java -Xmx2g -jar your-streams-app.jar ``` This is not a tuning knob you turn for speed: it's a fix for the "RSS keeps growing and OOMs after a few days" failure that a strict cache cap alone doesn't solve. If your container memory climbs slowly over hours/days while the configured RocksDB budget is constant, fragmentation is the first thing to rule out. ## Verify real usage: don't trust the math The arithmetic above gives you a floor, not the truth. Measure the actual footprint two ways: **On disk**, point `du` at the state directory (`state.dir`, default `${java.io.tmpdir}/kafka-streams`, typically `/tmp/kafka-streams` on Linux). One subdirectory per task, named `/`. Streams itself logs a WARN at startup when `state.dir` resolves inside the OS temp dir, because the OS can clear it under your checkpoint files. Set it explicitly in production: ```bash du -sh /tmp/kafka-streams//* # per-task on-disk size du -sh /tmp/kafka-streams/ # total local state ``` **In memory**, watch the container's cgroup memory, not the JVM heap. On the host or via your orchestrator's metrics, the number to alert on is container memory working set approaching the limit: that's what the OOM killer reads. If working set tracks well above `-Xmx` + your configured RocksDB budget, you have either uncapped index/filter growth or allocator fragmentation, in that order of likelihood. Kafka Streams also surfaces RocksDB's own internals as metrics (KIP-607), reported on the state-store (`stream-state-metrics`) level. The size/usage ones are property-based and available at the **INFO** recording level (free to turn on); the ratio and stall ones come from RocksDB statistics (KIP-471) and need `metrics.recording.level=DEBUG`. Mind the trap: the DEBUG metric names are registered even at INFO, but they record nothing (0.0/NaN) until you raise the level. | Metric | Tells you | Level | |---|---|---| | `size-all-mem-tables`, `block-cache-usage`, `block-cache-pinned-usage` | Off-heap memory in use: sum them (plus `estimate-table-readers-mem`) for a live total | INFO | | `block-cache-capacity` | The per-store cache cap actually in effect: the quickest check that your shared-cache config took (50 MiB per store on defaults) | INFO | | `total-sst-files-size` | On-disk state size: your disk-fill early warning | INFO | | `write-stall-duration-avg`, `write-stall-duration-total` | RocksDB is back-pressuring writes; compaction can't keep up | DEBUG | | `memtable-hit-ratio`, `block-cache-data-hit-ratio` (plus `-index-` and `-filter-` variants) | Read efficiency: low ratios mean reads are going to disk | DEBUG | For production triage the cgroup number plus `du` still answer the two questions that matter most (*am I about to be OOMKilled* and *am I about to fill the disk*), but these metrics tell you *why* before it happens. ## When the disk fills up instead Memory gets the headlines; disk is the quieter outage. Local state on RocksDB grows from two directions: - **The data itself.** A store with no TTL grows forever. Windowed stores are bounded by retention; plain key-value aggregations are bounded only by your key space: an unbounded key space (per-request IDs, per-session keys with no cleanup) grows without limit until the volume is full. - **Compaction overhead.** RocksDB is an LSM tree: writes land in new SST files and a background **compaction** merges and rewrites them. During compaction the old and new files coexist, so peak disk usage is meaningfully higher than steady-state. Provision headroom above your resting size, not at it. When the disk fills, the symptom is a flush or compaction failing with an I/O error, the stream thread dying, and (because the changelog is intact on the broker) the task trying to restore elsewhere, moving the disk-fills problem to the next instance. The local store and the [changelog topic](https://www.conduktor.io/kafka-streams/state-store) are different things: the changelog is compacted and lives on the brokers (bounded by your key space, retained as the source of truth for [state restore](https://www.conduktor.io/kafka-streams/state-restore)); the local RocksDB files live on the instance's disk and carry compaction overhead on top. Size the broker side for compacted state, and the instance disk for state *plus* compaction headroom. ## The native-library traps: Alpine and ARM RocksDB ships as a precompiled native library in the `rocksdbjni` JAR that `kafka-streams` pulls in as a dependency. That bites in two environments. **Alpine / musl.** On Kafka 2.x-era `rocksdbjni` the bundled binary was linked against **glibc** only. Alpine Linux uses musl libc, so the app started, built its topology, and then died the instant it touched a state store with: ``` java.lang.UnsatisfiedLinkError: .../librocksdbjni... Error loading shared library ld-linux-x86-64.so.2 ``` Modern `rocksdbjni` (bundled since Kafka ~3.1) ships musl-linked natives alongside the glibc ones and detects musl automatically at load time, so current `kafka-streams` runs on Alpine out of the box. If you're pinned to an old Kafka version there is still no flag: the only fix is a glibc base image (`eclipse-temurin:21-jdk` on Debian/Ubuntu, not `:21-jdk-alpine`). On anything recent, verify your version's `rocksdbjni` ships a musl lib before ruling Alpine out; the crash that only appears at the first stateful operation is a Kafka 2.x story now. **Apple Silicon / ARM.** On M1/M2 (`arm64`) the bundled binary must match the architecture. Modern `kafka-streams` ships an `arm64` RocksDB and works on Apple Silicon out of the box. The historical failure was an older Kafka version with no ARM binary throwing `UnsatisfiedLinkError` on a Mac; the fix is to be on a current Kafka version rather than to hunt for a workaround. If you target `linux/arm64` containers (Graviton), confirm your Kafka version bundles the matching native lib; it does on recent releases, but it's worth verifying rather than assuming. > *RocksDB is the default persistent store: a stateful Streams app opens a `RocksDBTimestampedStore` per partition on startup, confirmed on macOS ARM with JDK 21. You don't opt into RocksDB; you opt out of it (to in-memory), and that has its own [restore cost](https://www.conduktor.io/kafka-streams/state-store).* ## Frequently Asked Questions **Why is my Kafka Streams pod getting OOMKilled when the JVM heap looks fine?** RocksDB is a native C++ library, so its block cache, write buffers, and filter blocks live off-heap, invisible to your heap dashboard but counted in container/cgroup memory and `OOMKilled` events. The fix is to bound RocksDB memory with a `RocksDBConfigSetter`, not to raise `-Xmx`. **Why does RocksDB ignore my -Xmx setting?** Because `-Xmx` only bounds the JVM heap, and RocksDB allocates off-heap (native) memory. Each store gets ~50 MiB block cache plus up to 48 MiB of write buffers per partition by default, and RocksDB runs one instance per partition per store, so an instance owning 30 partitions across 3 stores can sit ~9 GiB off-heap, on top of your heap. **How do I bound total RocksDB memory across all state stores?** Give every store on the instance a single shared `LRUCache` (with `strictCapacityLimit=true`) and a single `WriteBufferManager` pointed at that same cache, wired through a `RocksDBConfigSetter`. That turns the off-heap ceiling into a constant you choose instead of a function of partition count. Also set `setCacheIndexAndFilterBlocks(true)` so index/filter blocks count against the cap. **Why does my Kafka Streams memory keep growing slowly over days?** If the configured RocksDB budget is constant but RSS ratchets upward over hours or days, the usual cause is glibc malloc arena fragmentation, not a leak. Swap the allocator to jemalloc via `LD_PRELOAD` before the JVM starts; it fragments far less under RocksDB's churning allocation pattern. **Why do I get UnsatisfiedLinkError on Alpine?** You're on an old Kafka version. Kafka 2.x-era `rocksdbjni` bundled a glibc-only binary, and Alpine uses musl libc, so the app died the instant it touched a state store with `UnsatisfiedLinkError: ... ld-linux-x86-64.so.2`. Modern `rocksdbjni` (since Kafka ~3.1) ships musl-linked natives and detects musl automatically; on older versions the only fix is a glibc base image (e.g. `eclipse-temurin:21-jdk`, not `-alpine`). > **See it in practice with Conduktor** > RocksDB sizing is ultimately a question of how much state each task holds. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-rocksdb-tuning) lets you inspect the compacted changelog topics that back your stores: their size on the brokers is a direct proxy for how big each partition's local RocksDB will grow, and the consumer group lag tells you whether a restore is still rebuilding that state after a deploy. ## Next steps - [Kafka Streams state stores](https://www.conduktor.io/kafka-streams/state-store): the model RocksDB implements, and why memory is off-heap - [State restore time & standby replicas](https://www.conduktor.io/kafka-streams/state-restore): what happens when that RocksDB state has to be rebuilt - [Slow rebalances](https://www.conduktor.io/kafka-streams/rebalancing): why a rebalance hands one instance the partitions that blow its memory budget Source: https://www.conduktor.io/kafka-streams/rocksdb-tuning --- # Kafka Streams scaling *Make a Kafka Streams app go faster, and know when it can't.* The first instinct when a Kafka Streams app falls behind is to add threads or spin up more pods. Sometimes that works. Often it does nothing, because Kafka Streams parallelism is capped by something most config changes can't touch: your partition count. This page is the mental model for scaling Streams correctly, and the one trap (increasing partitions on a stateful app) that turns a scaling attempt into a data-corruption incident. The [architecture](https://www.conduktor.io/kafka-streams/architecture) page established that the **task** is the fixed unit of parallelism and that tasks are pinned to partitions. Here we turn that into operational decisions: how many threads, how many instances, what to do with the surplus, and why you can't just bump partitions to scale. **What you'll learn:** - Why total useful parallelism is capped by partition count, not by threads or instances - When extra threads and instances become idle standbys - How a narrow repartition topic throttles everything downstream - Why increasing input partitions on a stateful app breaks key→state mapping ## The ceiling: parallelism = max partitions of a sub-topology ![orders topic with 12 partitions creates 12 tasks; scenario A shows 1 instance with 12 threads all active; scenario B shows 14 instances with 12 active and 2 idle with no task](https://www.conduktor.io/assets/kafka-streams/diagrams/scaling-parallelism-ceiling.svg) One number governs how far a Kafka Streams app can scale: the **largest partition count among its sub-topologies**. That's the maximum number of tasks, and tasks are the only thing that runs in parallel. A quick reminder of the chain (full detail in [architecture](https://www.conduktor.io/kafka-streams/architecture)): ``` partitions of a sub-topology → tasks (one per partition) → run on stream threads → across instances ``` Your real concurrency is `min(total tasks, total stream threads across all instances)`. The first term is set by your **topics**; the second by your **config**. You can raise the second freely, but the moment it exceeds the first, the extra threads have no task to run and your throughput stops improving. A 12-partition input topic gives you 12 tasks for that sub-topology, and 12 is the hard ceiling for that stage, whether you run it on 1 instance with 12 threads or 12 instances with 1 thread each. Confluent's writeup on [scaling Streams with cooperative rebalancing](https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/) frames it the same way: you scale Streams by moving tasks onto more threads/instances, and the task count is fixed by partitions at startup. > 🚫 *"It's falling behind, add more threads (or more pods) and it'll catch up."* > Only up to the task count. Once `num.stream.threads × instances` ≥ the partition count of your busiest sub-topology, every additional thread or instance is idle (or a standby). Past the ceiling, more compute buys you nothing but cost: the fix is more partitions, not more workers, and that has consequences (below). ## Threads vs instances: two knobs, one ceiling You have two ways to add workers, and they trade off differently. | Knob | Scales | Cost / limit | Failure-domain effect | |---|---|---|---| | `num.stream.threads` | Threads within one instance | Bounded by CPU cores on that box | All threads die if the instance dies | | Number of instances | Threads across the fleet | Bounded by your orchestrator/budget | One instance dying loses only its share | `num.stream.threads` is the cheap lever: bump it to use the cores you already paid for on each box. But threads on one instance share that instance's fate, and its memory, which matters because each task carries its own [RocksDB state](https://www.conduktor.io/kafka-streams/rocksdb-tuning), so 8 threads each running stateful tasks multiply the off-heap footprint on a single box. Scaling out to more **instances** spreads both the failure domain and the memory, and lets [cooperative rebalancing](https://www.conduktor.io/kafka-streams/rebalancing) move tasks onto the new instance without a full stop-the-world. The rule of thumb: use threads to fill the cores on each instance, use instances to go wider than one box and to bound per-instance memory, and stop adding either once you hit the task ceiling. ## Past the ceiling: idle workers and standbys When you run more threads/instances than there are tasks, the surplus doesn't share the work: there's no work to share, because a partition is owned by exactly one task at a time. The surplus becomes one of two things: - **Idle**, if you have no standby replicas configured: the thread sits in the consumer group holding no active task, doing nothing but consuming a group slot. - **A standby replica**, if `num.standby.replicas > 0`: the surplus instance keeps a *warm copy* of another task's state by tailing its changelog, so if the active instance dies, failover is fast instead of a cold [state restore](https://www.conduktor.io/kafka-streams/state-restore). This is the one good reason to run more instances than tasks: turn the surplus into hot standbys. ```properties num.standby.replicas=1 ``` With `num.standby.replicas=1`, each task has one warm backup on another instance. That's not extra throughput (active and standby never process the same partition in parallel), it's faster recovery. So "more instances than tasks" is wasteful *unless* you've turned it into standby capacity, in which case you're buying availability, not speed. Be deliberate about which one you're paying for. ## The narrow-repartition throttle The ceiling isn't always your input topic. A [sub-topology boundary](https://www.conduktor.io/kafka-streams/architecture), created when you re-key and then aggregate or join, writes to an internal **repartition topic**, and *that* topic's partition count caps everything downstream of it. ``` input topic (24 partitions) → 24 tasks ──re-key──► repartition topic (?? partitions) → ?? tasks downstream ``` If the repartition topic has fewer partitions than the input, the downstream sub-topology is throttled to that smaller number no matter how wide the source was. By default Kafka Streams sizes a repartition topic to match its source, but you can override it with `Repartitioned.numberOfPartitions(...)`, and a too-small value silently caps downstream parallelism. When the *second half* of a topology lags while the first half keeps up, check the partition count of the repartition topic between them: that boundary is where parallelism is re-decided. ## Unbalanced assignment across instances Even at the right scale, work isn't always evenly spread. A few ways it goes lopsided: - **Tasks don't divide evenly by instances.** 10 tasks across 3 instances is 4/3/3: the instance with 4 does ~33% more work. With stateful tasks, that instance also holds ~33% more RocksDB state and memory. Pick an instance count that divides the task count cleanly when you can. - **Stateful and stateless tasks weigh differently.** The assignor balances *task count*, not load. An instance that happens to land several heavy stateful tasks (large stores, expensive aggregations) is hotter than one holding light stateless tasks, even with equal counts. - **A skewed key space defeats it entirely.** Partition assignment is even; *traffic* may not be. If 80% of your records carry a handful of hot keys that hash to two partitions, the two tasks owning them are saturated while the rest idle. No amount of scaling fixes skew: that's a partitioning/key-design problem upstream, not a Streams config. Watch per-instance and per-partition [consumer group lag](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets): uniform lag means you've hit the genuine throughput ceiling and need more partitions; lag concentrated on one instance or a few partitions means imbalance or key skew, and adding workers won't help. ## How to actually scale: size partitions up front Because parallelism is capped by partition count, the highest-leverage scaling decision happens before the app exists: **size your input topic partitions for your peak target, with headroom.** It's far cheaper to start at 24 partitions and run 6 today than to start at 6 and need 24 next quarter, because raising partition count on a live stateful app is the trap below, not a routine resize. A practical ordering when an app needs to scale: 1. **Raise `num.stream.threads`** to use the cores already on each instance, up to the task ceiling. Free, no rebalance of partitions across hosts. 2. **Scale out instances** toward `instance count ≤ task count`, letting cooperative rebalancing move tasks over. Spreads memory and failure domain. 3. **Convert the surplus to standbys** (`num.standby.replicas ≥ 1`) once instances exceed tasks: you're now buying recovery speed, not throughput. 4. **Only then add partitions**, and for a stateful app, treat it as a migration, not a config bump (next section). ## The partition-increase trap (stateful apps) This is the one that turns a scaling attempt into a corruption incident, so it gets its own section. Kafka routes a record to a partition by `hash(key) % partitionCount`. Your [state stores](https://www.conduktor.io/kafka-streams/state-store), and their changelog topics, are partitioned by that exact mapping: key `K` and all of its accumulated state live together on one partition because they hash to it. **Change the partition count and the modulus changes**, so `K` now hashes to a *different* partition than the one holding its state. The aggregation for `K` continues on the new partition starting from empty, while its real history sits stranded on the old partition's store. Kafka Streams has one guardrail here: at the first rebalance after the alter, it compares the new input partition count against the existing internal topics and crashes with a `StreamsException`: *"Existing internal topic `<appId>-...-changelog` has invalid partitions: expected: 6; actual: 3. Use org.apache.kafka.tools.StreamsResetter tool to clean up invalid topics before processing."* Loud, actionable, client shuts down. The corruption goes **silent** when nothing trips that check: changelog logging disabled (`withLoggingDisabled()`), the source-topic-as-changelog optimization, or an operator who "fixes" the startup error by manually widening the internal topics too. Then the app reaches RUNNING with no exception and produces wrong results: counts reset, joins miss, aggregates undercount. Verified on Kafka 4.3: with the changelog check bypassed, every key that moved partition restarted its count from empty (and going 3 → 6 partitions moved 7 of 8 sample keys, doubling does not keep keys in place). | | Stateless app | Stateful app | |---|---|---| | Increase input partitions | Mostly safe: more tasks, ordering per key preserved going forward | **Breaks key→state mapping: existing aggregations corrupt** | | What it costs | A rebalance | A state migration / reprocess, or wrong answers | For a stateful app, you do not "just add partitions." The correct paths: - **Reprocess from scratch** into a new, wider topic: create the topic at the target partition count, reset the app with the [application reset tool](https://www.conduktor.io/kafka-streams/topology-evolution) (input → earliest, internal topics deleted) so it rebuilds all state under the new mapping. One gap the tool itself warns about: it does *not* touch local state. Wipe the state directory or call `KafkaStreams#cleanUp()` before restarting, or the rebuild silently reuses stale RocksDB data. Viable when your input has full retention. - **Plan partition count for peak up front** so you never have to do this on a live stateful app: the reason "size partitions for your peak" is rule one of scaling Streams. The trap is the same if someone widens a *repartition* topic that backs stateful operators. Streams never resizes a repartition topic on its own (a fixed-size one actually insulates downstream state from input partition increases, since Streams owns the key→partition mapping into it), but an operator with alter rights on internal topics can. Either way, once the count changes, the key→partition mapping that your stores depend on has moved out from under them. ## Frequently Asked Questions **How do I scale a Kafka Streams application?** First raise `num.stream.threads` to use the cores already on each instance, then scale out to more instances (up to the task count) so cooperative rebalancing spreads memory and failure domain. But both are capped by partition count: once `threads × instances` reaches the partition count of your busiest sub-topology, more workers do nothing. **What is the maximum parallelism of a Kafka Streams app?** The largest partition count among its sub-topologies. Tasks are the only thing that runs in parallel, there's one task per partition, and real concurrency is `min(total tasks, total stream threads)`. A 12-partition topic gives a hard ceiling of 12 tasks for that stage, no matter how many threads or instances you add. **Why are some of my Kafka Streams instances idle?** Because you're running more threads/instances than there are tasks, and a partition is owned by exactly one task at a time. The surplus sits idle unless you set `num.standby.replicas > 0`, which turns it into warm standby copies for faster failover: availability, not throughput. **Can I add partitions to a running stateful Kafka Streams app?** Not safely. Kafka routes keys by `hash(key) % partitionCount`, and your state stores are partitioned by that exact mapping; changing the count sends a key to a different partition than the one holding its state. Streams usually catches the mismatch at the next rebalance and crashes with a StreamsException telling you to run the reset tool; if nothing trips that check (changelog logging disabled, internal topics widened by hand), aggregations and joins go silently wrong instead. The correct path is to reprocess into a new wider topic with the application reset tool, or size partitions for peak up front. **Why is one Kafka Streams instance lagging while the rest keep up?** Lag concentrated on one instance or a few partitions is imbalance or key skew, not a throughput ceiling; adding workers won't help. Uneven task division (10 tasks over 3 instances) or a few hot keys hashing to two partitions saturate those tasks. Uniform lag across all partitions is the real ceiling; that needs more partitions. > **See it in practice with Conduktor** > Scaling Streams is a question of tasks, partitions, and how evenly work is spread. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-scaling) shows the consumer group's partition assignment across instances and per-partition lag, so you can see at a glance whether you've hit the real throughput ceiling (uniform lag), have idle instances past the task count, or have lag piling up on a few skewed partitions that more workers won't fix. ## Next steps - [Kafka Streams architecture](https://www.conduktor.io/kafka-streams/architecture): the task and thread model this page scales - [Slow rebalances](https://www.conduktor.io/kafka-streams/rebalancing): what happens to tasks every time you add or remove an instance - [State restore time & standby replicas](https://www.conduktor.io/kafka-streams/state-restore): the cost of moving stateful tasks, and how standbys hide it Source: https://www.conduktor.io/kafka-streams/scaling --- # Kafka Streams serde errors *Diagnose and fix the two serde exceptions everyone hits.* Almost every Kafka Streams app throws a serde error before it ships. The two that fill Stack Overflow look unrelated but share a root cause: a serializer or deserializer was applied to a type it was never configured for. One fires when you change a value's type mid-topology and forget to tell Streams; the other fires when a plain deserializer meets bytes written by the Confluent Schema Registry. Both come with a stack trace people paste verbatim, and both have a precise fix once you understand how Kafka Streams *resolves* which serde to use for a given operator. This page walks the two errors, then the rule that prevents them. **What you'll learn:** - Why `ClassCastException: java.lang.Long cannot be cast to java.lang.String` shows up after a `map` or `aggregate` - What `Unknown magic byte!` actually means and how to fix it - How serde resolution works: the default config vs per-operator overrides - The principle: set the serde at every operator that changes a type ## Error 1: ClassCastException after a transform You write a topology, set sensible default serdes, and the app compiles. Then at runtime, the first record through an `aggregate` blows up: ``` Exception in thread "...-StreamThread-1" org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_0, processor=KSTREAM-AGGREGATE-0000000002 ... Caused by: org.apache.kafka.streams.errors.StreamsException: A serializer (org.apache.kafka.common.serialization.StringSerializer) is not compatible to the actual value type (value type: java.lang.Long). Change the default Serdes in StreamConfig or provide correct Serdes via method parameters. Caused by: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.String (java.lang.Long and java.lang.String are in module java.base ...) at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:29) ``` That is the 4.x wording, and it names both the offending type and the fix. On 2.8 and earlier the wrapper read `ClassCastException invoking Processor. Do the Processor's input types match the deserialized types?`, with the same `ClassCastException` underneath. Here is the trap. You configured a **default value serde** of `String` for the whole application: ```properties default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde ``` Then somewhere in the topology you changed the value type to one Streams cannot infer. An `aggregate` builds a running `Long` total. A `mapValues` turns a JSON string into an `Order` object. The moment Streams needs to *write* that new type, to a changelog topic, a repartition topic, or your output topic, it reaches for the only serde it knows: the default `StringSerde`. `StringSerializer.serialize` receives a `Long`, casts it to `String`, and throws. One operator gets a pass: `count()`. Streams knows its result is always a `Long`, so it materializes the store with `Serdes.Long()` and propagates that serde downstream through `toStream().to(...)`. The classic "count then to() blows up" example does not actually reproduce; it only fails if you force a wrong serde explicitly. `aggregate` gets no such inference, because its result type is whatever your initializer returns. The exception fires deep in the serializer, but the bug is the missing serde declaration at the operator that changed the type. ```java // Throws at runtime: aggregate() emits Long, default value serde is String builder.stream("orders") .groupByKey() .aggregate(() -> 0L, (key, value, agg) -> agg + 1) // value type is now Long, serde not inferred .toStream() .to("order-counts"); // store, changelog, and sink all reach for StringSerde ``` The fix is to set the serde explicitly on the operator that introduced the new type. Every stateful and re-keying operator accepts a `...with(...)` configuration object for exactly this: ```java // Tell each operator the real types builder.stream("orders", Consumed.with(Serdes.String(), Serdes.String())) .groupByKey(Grouped.with(Serdes.String(), Serdes.String())) .aggregate(() -> 0L, (key, value, agg) -> agg + 1, Materialized.with(Serdes.String(), Serdes.Long())) // store + changelog: Long values .toStream() .to("order-counts", Produced.with(Serdes.String(), Serdes.Long())); // output: Long values ``` `Consumed`, `Produced`, `Grouped`, and `Materialized` each carry a key serde and a value serde. Set them wherever the type differs from the default, and the `ClassCastException` is gone. > 🚫 *"I set the default serde once at the top, so the whole topology is covered."* The default serde only covers operators where the type matches the default. The instant an `aggregate` or a type-changing `mapValues` produces a type Streams cannot infer, the default is wrong for everything downstream of it, and you get the cast exception, not a "serde missing" message. ## How serde resolution works Kafka Streams picks a serde for every read and write using a simple precedence: | Source | Scope | How you set it | |---|---|---| | Per-operator serde | That one operator | `Consumed`, `Produced`, `Grouped`, `Materialized`, `StreamJoined`, `Repartitioned` `.with(...)` | | Default serde | Every operator without an explicit serde | `default.key.serde` / `default.value.serde` config | | (none configured) | N/A | `ConfigException` at startup: a serde is required | An explicit per-operator serde always wins over the default. There is no per-topic registry and no type inference from generics, the `` you write on `stream(...)` is a compile-time hint for the Java compiler, not a runtime instruction to Streams. At runtime, Streams only knows the default config and whatever you passed to each operator. This is why the type-change case bites: generics make the code *look* type-safe, but the serializer is chosen from config, not from the generic parameter. ## Error 2: Unknown magic byte! The second classic looks nothing like the first: ``` org.apache.kafka.common.errors.SerializationException: Unknown magic byte! at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(...) ``` That trace is from a Schema Registry 7.x client. On 8.x clients the wire-format parsing moved, so the top-level exception reads `SerializationException: Error deserializing schema ID` with `Unknown magic byte!` as the cause underneath. Same problem, same fix. This one is about *wire format*, not types. The Confluent Schema Registry serializers do not write a bare Avro (or Protobuf, or JSON Schema) payload. They prefix every record value with a small envelope: ``` byte 0 : magic byte (always 0x00) bytes 1-4 : 4-byte schema ID (big-endian int) bytes 5..end : the serialized payload ``` `Unknown magic byte!` means a Schema-Registry-aware deserializer read a record whose first byte was *not* `0x00`, so it could not find the schema ID it needs. There are two ways to land here: - **A plain serde wrote the data, a Schema Registry serde is reading it.** The bytes have no magic byte or schema ID, so the Avro deserializer rejects the very first byte. Common when one team produces with a `StringSerializer` and another consumes the topic expecting Avro. - **A Schema Registry serde wrote the data, a plain serde is reading it.** This is the mirror image and usually surfaces differently, a `StringDeserializer` happily turns the magic byte and schema ID into garbage characters at the front of your string, and you get corrupted-looking values instead of an exception. Same root cause: format mismatch. The fix is to use the matching serde on both sides and point it at the registry. For Avro `SpecificRecord`: ```java // Schema-Registry-aware Avro serde, configured with the registry URL final Map serdeConfig = Map.of("schema.registry.url", "http://schema-registry:8081"); final SpecificAvroSerde orderSerde = new SpecificAvroSerde<>(); orderSerde.configure(serdeConfig, false); // false = value serde (true for keys) builder.stream("orders", Consumed.with(Serdes.String(), orderSerde)) // matching deserializer on read .mapValues(order -> order.getTotal()) .to("order-totals", Produced.with(Serdes.String(), Serdes.Long())); ``` If you are using the lower-level consumer config instead of a DSL serde, the equivalent is a `KafkaAvroDeserializer` with `schema.registry.url` set. The rule is the same: the deserializer must speak the same wire format as the serializer that wrote the bytes. A frequent variant: the data is fine, but the consumer can't reach the registry, so it can't resolve the schema ID, check `schema.registry.url`, network reachability, and any auth the registry requires before assuming the records are bad. ## Windowed keys need a windowed serde There is a third place serdes catch people: windowed aggregations. When you window a `count` or `aggregate`, the key is no longer your plain key, it is a `Windowed` that wraps your key plus the window's time bounds. A plain `StringSerde` cannot serialize that. If you write a windowed result to a topic, or query a windowed store, you need a **windowed serde** that knows how to encode the key-plus-window: ```java import org.apache.kafka.streams.kstream.WindowedSerdes; builder.stream("clicks") .groupByKey() .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5))) .count() .toStream() // key is Windowed now, wrap the inner serde in a time-windowed serde .to("clicks-per-window", Produced.with( WindowedSerdes.timeWindowedSerdeFrom(String.class, 5 * 60 * 1000L), Serdes.Long())); ``` `WindowedSerdes.timeWindowedSerdeFrom(...)` builds a serde for the windowed key from the inner type and the window size. (Session windows have their own `sessionWindowedSerdeFrom`.) Forget it and you are back to a `ClassCastException`, this time `Windowed cannot be cast to String`. ## The principle: serde at every type change All three errors collapse into one rule: **Set the serde explicitly at every operator that introduces a type the default serde doesn't cover.** In practice that means: - An `aggregate` or `reduce` produces a value type the default doesn't cover → set `Materialized.with(...)` (covers the store *and* its changelog: store writes are serialized with the changelog topic name as the serde's topic argument, which is also the registry subject for Schema Registry serdes) and `Produced.with(...)` if you write the result out. `count()` is exempt: Streams fills in `Serdes.Long()` for you. - A `groupBy` / `groupByKey` before an aggregation → set `Grouped.with(...)` so the repartition topic is encoded correctly. - A `mapValues` / `map` that changes a type, followed by a stateful op or a `to(...)` → set the serde on the downstream operator that serializes. - A windowed aggregation written out or queried → use a windowed serde for the key. - Any topic carrying Schema Registry data → use the matching `SpecificAvroSerde` / Schema Registry serde with `schema.registry.url`. Reaching for a [poison-pill handler](https://www.conduktor.io/kafka-streams/dead-letter-queue) is *not* a substitute here. A serde misconfiguration is a deterministic bug in your topology, not a single bad record, skipping it would skip every record. Fix the serde; reserve dead-letter handling for genuinely malformed input. ## Frequently Asked Questions **What is a Serde in Kafka Streams?** A Serde bundles a serializer and a deserializer for one type, so Kafka Streams can read records into Java objects and write them back to bytes. Kafka Streams resolves a Serde for every read and write from either the per-operator setting or the `default.key.serde`/`default.value.serde` config. **What causes "Unknown magic byte!" in Kafka Streams?** A Schema-Registry-aware deserializer (like Avro) read a record whose first byte wasn't the expected `0x00` magic byte, so it couldn't find the schema ID. It usually means a plain serde wrote the data while a Schema Registry serde is reading it, or the reverse, fix it by using the matching serde on both sides and pointing it at `schema.registry.url`. **Why do I get a ClassCastException after a map, groupBy, or aggregate?** An operator changed the value type to one Streams can't infer (for example an `aggregate()` building a `Long` total) but the only Serde Streams knows downstream is the `String` default, so `StringSerializer.serialize` receives a `Long` and throws. Set the Serde explicitly on the operator that introduced the new type via `Materialized.with(...)`, `Grouped.with(...)`, or `Produced.with(...)`. **How do I set a Serde per operation versus as a default?** A per-operator Serde, passed through `Consumed`, `Produced`, `Grouped`, `Materialized`, `StreamJoined`, or `Repartitioned` `.with(...)`, always wins over the default. The `` generics on `stream(...)` are only a compile-time hint; at runtime Streams picks the serializer from config or the operator override, never from the generic parameter. **Why does a windowed aggregation need a windowed Serde?** After `windowedBy(...)`, the key becomes a `Windowed` wrapping your key plus the window bounds, which a plain `StringSerde` can't encode. Use `WindowedSerdes.timeWindowedSerdeFrom(...)` (or `sessionWindowedSerdeFrom` for session windows) when you write a windowed result out or query a windowed store, otherwise you get a `Windowed cannot be cast to String` error. > **See it in practice with Conduktor** > When a serde error fires, the fastest question to answer is "what's actually in the bytes?" [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-serdes) lets you inspect the records on the input, repartition, and changelog topics, including whether a value carries the Schema Registry magic byte and schema ID, or is plain text, so you can tell a type mismatch from a wire-format mismatch without attaching a debugger. ## Next steps - [Dead letter queues in Kafka Streams](https://www.conduktor.io/kafka-streams/dead-letter-queue), handling the bad records a serde can't decode - [KStream, KTable & GlobalKTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable), where types change and serdes matter - [Build your first Kafka Streams app](https://www.conduktor.io/kafka-streams/getting-started), configuring default and per-operator serdes from the start Source: https://www.conduktor.io/kafka-streams/serdes --- # Kafka Streams with Spring Boot *Wire Kafka Streams into Spring Boot the right way.* There are two completely different ways to run Kafka Streams under Spring Boot, and people constantly conflate them. One is **Spring for Apache Kafka** (`spring-kafka`): you write a real `StreamsBuilder` topology and Spring manages its lifecycle. The other is **Spring Cloud Stream**: you write `java.util.function.Function` beans and a binder wires them to topics. Same engine underneath, very different code and config. This page covers both, side by side, so you can tell which one a Stack Overflow answer is even talking about, and pick the right one for your service. **What you'll learn:** - The `spring-kafka` approach: `@EnableKafkaStreams`, `StreamsBuilderFactoryBean`, and `spring.kafka.streams.*` - The Spring Cloud Stream functional binder: `Function` beans and declarative bindings - When to pick which, and why mixing them up causes half the confusion - The lifecycle, error-handling, and testing details Spring hides from you ## Two libraries, one engine Both options run the exact same `org.apache.kafka:kafka-streams` library underneath, the topology, [state stores](https://www.conduktor.io/kafka-streams/state-store), changelog topics, and rebalancing all behave identically. What differs is how much Spring writes for you. | | Spring for Apache Kafka (`spring-kafka`) | Spring Cloud Stream (KStream binder) | |---|---|---| | You write | A `KStream<...>` topology with the DSL | A `Function` bean | | Topics wired by | You, in the topology (`stream(...)`, `to(...)`) | The binder, via `spring.cloud.stream.bindings.*` | | Config namespace | `spring.kafka.streams.*` | `spring.cloud.stream.kafka.streams.*` | | Lifecycle | `StreamsBuilderFactoryBean` | The binder | | Control you keep | Full, it's a normal topology | Less, bindings are declarative | | Good fit | One app, full control, interactive queries | Many small functions, event-driven microservices, multi-binder | Neither is "more correct." `spring-kafka` is closer to the metal and the better default when you want to *see* your topology. Spring Cloud Stream trades that visibility for declarative bindings and a uniform programming model across messaging systems. > **Versions.** Spring for Apache Kafka 4.0 (GA Nov 2025) and Spring Cloud Stream 5.x track the Kafka 4.x clients. If you're on Spring Boot 3.x you're on the earlier `spring-kafka` 3.x line, and Spring Cloud Stream 4.x stays there too (kafka-clients 3.8.x). The APIs below are stable across both, but check the [Spring Kafka Streams reference](https://docs.spring.io/spring-kafka/reference/streams.html) for your exact version. ## Approach 1: spring-kafka and @EnableKafkaStreams Add the dependency, annotate a config class with `@EnableKafkaStreams`, and Spring auto-configures a `StreamsBuilderFactoryBean` for you. You then declare one or more `@Bean public KStream<...>` methods that take the auto-configured `StreamsBuilder` and define the topology. Spring builds it and manages `start()`/`close()` for you. ```xml org.springframework.kafka spring-kafka ``` ```java @Configuration @EnableKafkaStreams public class StreamsConfig { @Bean public KStream pipeline(StreamsBuilder builder) { KStream stream = builder.stream("words-input"); stream .flatMapValues(line -> Arrays.asList(line.toLowerCase().split("\\W+"))) .filter((k, word) -> !word.isEmpty()) .groupBy((k, word) -> word) .count(Materialized.as("word-counts")) .toStream() .to("words-output", Produced.with(Serdes.String(), Serdes.Long())); return stream; } } ``` You do **not** create a `KafkaStreams` instance or call `start()` yourself, the factory bean does it after the context refreshes. Config goes in `application.yml` under `spring.kafka.streams`: ```properties spring.kafka.bootstrap-servers=localhost:9092 spring.kafka.streams.application-id=wordcount-app # default serdes have no default since Kafka 3.0 (KIP-741); without these the app fails at startup spring.kafka.streams.properties[default.key.serde]=org.apache.kafka.common.serialization.Serdes$StringSerde spring.kafka.streams.properties[default.value.serde]=org.apache.kafka.common.serialization.Serdes$StringSerde # anything not exposed as a typed property goes under .properties.* spring.kafka.streams.properties[processing.guarantee]=exactly_once_v2 spring.kafka.streams.properties[num.stream.threads]=2 ``` Raw Kafka Streams requires both `application.id` (the [identity that names your consumer group, internal topics, and state directory](https://www.conduktor.io/kafka-streams/getting-started)) and `bootstrap.servers`. Boot softens both, in opposite directions. `application-id` falls back to `spring.application.name`; the context only fails (with `InvalidConfigurationPropertyValueException`) when both are unset. And `bootstrap-servers` silently defaults to `localhost:9092`, so a missing broker config never fails fast: it just points your app at localhost, which is its own production footgun. ### Lifecycle and error handling The `StreamsBuilderFactoryBean` is where you hook the things a raw `KafkaStreams` exposes. Spring lets you customize it after auto-configuration: ```java @Bean public StreamsBuilderFactoryBeanConfigurer configurer() { return factoryBean -> { factoryBean.setStateListener((newState, oldState) -> log.info("State {} -> {}", oldState, newState)); factoryBean.setStreamsUncaughtExceptionHandler(ex -> StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse.REPLACE_THREAD); }; } ``` The `StateListener` is your signal for `REBALANCING → RUNNING` transitions and for the `ERROR` state. The uncaught-exception handler decides what happens when a stream thread dies, `REPLACE_THREAD`, `SHUTDOWN_CLIENT`, or `SHUTDOWN_APPLICATION`, the same decision covered in [dead letter queues](https://www.conduktor.io/kafka-streams/dead-letter-queue). Don't leave it on the default: since Kafka 2.8 (KIP-671) the default with no handler is `SHUTDOWN_CLIENT`, so one poison record halts the entire client (it transitions through `PENDING_ERROR` into `ERROR` and every thread stops). Set the handler deliberately, `REPLACE_THREAD` if you want to keep running, and alert on `ERROR` from the state listener. ### Reaching the KafkaStreams instance For [interactive queries](https://www.conduktor.io/kafka-streams/interactive-queries) you need the live `KafkaStreams` object. Spring exposes it through the factory bean: ```java @Autowired private StreamsBuilderFactoryBean factoryBean; public Long countFor(String word) { KafkaStreams streams = factoryBean.getKafkaStreams(); // null until the factory bean has started ReadOnlyKeyValueStore store = streams.store( StoreQueryParameters.fromNameAndType("word-counts", QueryableStoreTypes.keyValueStore())); return store.get(word); } ``` `getKafkaStreams()` returns `null` until the factory bean has started during context refresh, and `null` again after `stop()`, so endpoints can NPE during graceful shutdown too. The null check alone isn't the right gate either: a non-null instance still rejects `store(...)` with `InvalidStateStoreException` until the state reaches `RUNNING`. Guard on the state or the state listener; see the [interactive queries](https://www.conduktor.io/kafka-streams/interactive-queries) page for the full pattern. ## Approach 2: Spring Cloud Stream functional binder Spring Cloud Stream inverts the model. You don't touch `StreamsBuilder`. You write a bean of type `Function, KStream>` (or `Consumer>` for a sink), and the binder connects its input and output to topics declared in config. ```xml org.springframework.cloud spring-cloud-stream-binder-kafka-streams ``` ```java @Bean public Function, KStream> pipeline() { return input -> input .flatMapValues(line -> Arrays.asList(line.toLowerCase().split("\\W+"))) .filter((k, word) -> !word.isEmpty()) .groupBy((k, word) -> word) .count(Materialized.as("word-counts")) .toStream(); } ``` ```properties spring.cloud.function.definition=pipeline spring.cloud.stream.bindings.pipeline-in-0.destination=words-input spring.cloud.stream.bindings.pipeline-out-0.destination=words-output spring.cloud.stream.kafka.streams.binder.application-id=wordcount-app spring.cloud.stream.kafka.streams.binder.brokers=localhost:9092 ``` The binding names follow the convention `-in-` / `-out-`. The function name comes from `spring.cloud.function.definition`. This is the model Spring documents as current, the old annotation-based `@EnableBinding` / `@StreamListener` API was **removed in Spring Cloud Stream 4.0**, so any tutorial using those annotations is stale. Use the functional `Function`/`Consumer` style. > 🚫 *"I'll just add `@EnableKafkaStreams` to my Spring Cloud Stream app to make the state store work."* That line is the conflation in a nutshell. `@EnableKafkaStreams` is a `spring-kafka` annotation; it has no role in a Spring Cloud Stream binder app, where the binder owns the lifecycle. Mixing the two gives you two competing attempts to start Streams. Pick one stack and stay in it. ## Testing, same as plain Kafka Streams Spring changes how the topology is *wired*, not what it *is*. So the unit-testing story is unchanged: extract the topology and drive it with `TopologyTestDriver`, no broker and no Spring context required. That's the fast, deterministic test, covered in [testing Kafka Streams](https://www.conduktor.io/kafka-streams/testing). For an `@EnableKafkaStreams` app, refactor the topology into a plain method or `@Bean` that takes a `StreamsBuilder` so a test can build it in isolation: ```java StreamsBuilder builder = new StreamsBuilder(); new StreamsConfig().pipeline(builder); // your topology method Properties props = new Properties(); props.put("default.key.serde", Serdes.StringSerde.class); props.put("default.value.serde", Serdes.StringSerde.class); try (TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) { TestInputTopic in = driver.createInputTopic( "words-input", new StringSerializer(), new StringSerializer()); in.pipeInput("the quick brown the"); // assert on the output topic / state store } ``` No `application.id` or bootstrap config needed since Kafka 3.2. The default serde props are still required because the topology above relies on them, and default serdes have had no default since Kafka 3.0; use explicit serdes (`Consumed.with`, `Grouped.with`) in the topology and you can drop the props entirely. For an end-to-end test that exercises the real binder and Spring config, use `EmbeddedKafka` (from `spring-kafka-test`) or [Testcontainers](https://www.conduktor.io/kafka-streams/testing), slower, but it catches binding and serde wiring that `TopologyTestDriver` can't see. ## Which one should you pick? - **Reach for `spring-kafka` + `@EnableKafkaStreams`** when you have one app with one (or a few) topologies, you want the topology explicit in your code, and you need [interactive queries](https://www.conduktor.io/kafka-streams/interactive-queries) or fine control over the `StreamsBuilderFactoryBean` lifecycle. It's the smaller abstraction and the easier one to debug. - **Reach for Spring Cloud Stream** when you're building many small event-driven functions, you already use the framework for other binders (RabbitMQ, plain Kafka), and you value declarative bindings and a uniform model over seeing the topology spelled out. If you're unsure, start with `spring-kafka`. It's closer to the [getting-started](https://www.conduktor.io/kafka-streams/getting-started) mental model, and you can always move to the binder later. ## Frequently Asked Questions **What's the difference between @EnableKafkaStreams and Spring Cloud Stream?** `@EnableKafkaStreams` is part of `spring-kafka`: you write a real `StreamsBuilder` topology and Spring manages its lifecycle via `StreamsBuilderFactoryBean`. Spring Cloud Stream is a separate library where you write `Function` beans and a binder wires them to topics declaratively. They run the same engine but are not meant to be combined. **Why does my Spring Boot Kafka Streams app fail to start?** The most common cause since Kafka 3.0 is missing default serdes: set `default.key.serde` and `default.value.serde` under `spring.kafka.streams.properties`, or use explicit serdes everywhere, or startup throws a `ConfigException`. A missing `application-id` falls back to `spring.application.name`; Boot only throws `InvalidConfigurationPropertyValueException` when both are unset. `bootstrap-servers` never fails fast: it defaults to `localhost:9092`. **How do I access interactive queries in a Spring Boot Kafka Streams app?** With `spring-kafka`, autowire the `StreamsBuilderFactoryBean` and call `getKafkaStreams()` to get the live `KafkaStreams`, then `streams.store(...)`. It returns `null` until the factory bean has started, and even a non-null instance throws `InvalidStateStoreException` from `store(...)` until the state reaches `RUNNING`, so wait on the state listener. See the interactive queries page for multi-instance routing. **How do I test a Spring Boot Kafka Streams app?** Unit-test the topology with `TopologyTestDriver`, extract it into a method that takes a `StreamsBuilder` so no broker or Spring context is needed. For full binder/wiring coverage use `EmbeddedKafka` or Testcontainers, which start a real broker. **Can I put my whole topology in one @Bean?** Yes. A single `@Bean public KStream<...> pipeline(StreamsBuilder builder)` can hold the entire topology; the auto-configured `StreamsBuilderFactoryBean` builds and starts it. You can also split across multiple `@Bean` methods sharing the same injected `StreamsBuilder`. > **See it in practice with Conduktor** > Spring hides the wiring, not the runtime: under the hood your app is still a consumer group plus the changelog and repartition topics it creates. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-spring-boot) shows the `-...` internal topics, the consumer-group lag that tells you whether a restore is still running after a deploy, and the partition assignment across instances, the signals Spring's logs don't surface on their own. ## Next steps - [Build your first Kafka Streams app](https://www.conduktor.io/kafka-streams/getting-started), the plain-Java baseline Spring wraps, and why `application.id` matters - [Testing Kafka Streams](https://www.conduktor.io/kafka-streams/testing), `TopologyTestDriver` and when to reach for Testcontainers - [Interactive queries](https://www.conduktor.io/kafka-streams/interactive-queries), serve state-store data over REST from a Spring app Source: https://www.conduktor.io/kafka-streams/spring-boot --- # Why Kafka Streams state restore is slow *Understand and shorten Kafka Streams state restore.* When a Kafka Streams app comes back from a deploy and just sits there (consumer-group lag climbing, no records moving), it's almost always restoring state. Restore is the step where a task replays its changelog topic into a local store before it's allowed to process anything, and for a large store it can take minutes to hours. It's the mechanism hiding behind most "[rebalancing is slow](https://www.conduktor.io/kafka-streams/rebalancing)" reports, and it deserves its own treatment. **What you'll learn:** - What restore actually is, and why store size sets the clock - The rolling-restart caveat the docs underplay - Why persistent volumes are the single biggest lever, and the EBS/EFS traps - What standby and warm-up replicas really cover ## What restore is ![Restore lifecycle: a task enters RESTORING state while the restore consumer replays the changelog into RocksDB; processing is blocked until the task reaches RUNNING](https://www.conduktor.io/assets/kafka-streams/diagrams/state-restore-lifecycle.svg) Every stateful operation in Kafka Streams keeps its running state in a local [state store](https://www.conduktor.io/kafka-streams/state-store), by default a RocksDB database on the instance's disk. That store is durable not because the disk is, but because every write to it is *also* appended to a compacted Kafka topic: the **changelog**. The changelog is the source of truth. So when a task starts on an instance that doesn't already have the store's data on local disk (a fresh pod, a task that just moved here in a rebalance, a node with a wiped disk), Streams must rebuild the store before the task can process a record. It does that by **replaying the changelog from the beginning into the local store**: 1. The task is assigned but enters a `RESTORING` state, not `RUNNING`. 2. A restore consumer reads the changelog topic for that partition, from offset zero. 3. Each record is written into the local RocksDB store. 4. Once the store has caught up to the tail of the changelog, the task transitions to `RUNNING` and starts processing input. Processing for that task is blocked for the entire duration of steps 2–3. Because the changelog is [log-compacted](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction), it holds only the latest value per key, so its size tracks the size of your *state* rather than the total volume of updates ever made, which is the only reason restore is bounded at all. (Changelogs for windowed and session stores are the variant: they use `cleanup.policy=compact,delete` with a retention window sized to your window plus grace, so old windows age out by time rather than by compaction alone.) You don't have to guess whether a restore is running or how far along it is. Register a `StateRestoreListener` with `streams.setGlobalStateRestoreListener(...)`: it fires `onRestoreStart` / `onBatchRestored` / `onRestoreEnd` per store partition with the changelog offsets, so you can log or expose live restore progress. Since 4.3, restoration also runs on a dedicated state-updater thread with its own metrics group, `stream-state-updater-metrics` (`active-restoring-tasks`, `restore-records-rate`, `active-restore-ratio`), so progress is visible on a dashboard without writing a listener. For a deeper trace, raise the `org.apache.kafka.streams.processor.internals.StoreChangelogReader` logger to `DEBUG`. From outside the app, the same signal is consumer-group lag on the changelog topics: lag draining toward zero is the restore finishing. ## Why a large store means a long restore The math is unforgiving: restore time is proportional to how much data has to be read from the changelog and written into RocksDB. A store with a few thousand keys restores in seconds. A store with hundreds of millions of keys does not. A team running a key-value store with a few hundred million keys saw RocksDB rebuilds of **45 minutes to two hours** when bringing the app up cold in a disaster-recovery region, and during that window, nothing processes. That's not a misconfiguration; it's the floor set by the volume of the changelog and the write throughput of RocksDB on that hardware. You can shave it (more on that below), but you cannot make replaying a hundred-million-key changelog instantaneous on local disk. This is also why restore time, not the rebalance protocol, is usually the real villain when a "slow rebalance" is reported. The rebalance assigns the task in seconds; the restore that follows is what takes the hour. If you've already tuned [cooperative rebalancing](https://www.conduktor.io/kafka-streams/rebalancing) and the stalls persist, this page is where the fix lives. ## The rolling-restart caveat the docs underplay ![Standby replicas eliminate full restore after a crash but not during a rolling restart, when both the active instance and the standby are being bounced simultaneously](https://www.conduktor.io/assets/kafka-streams/diagrams/state-restore-standby-vs-rolling.svg) Here's the part that catches teams out. The standard answer to "restore is slow" is "add `num.standby.replicas`." That's good advice for a *crash*, and misleading for the most common operational event you'll actually run: a rolling restart. > 🚫 *"We have standby replicas, so restores during a deploy are instant."* Standby replicas keep a hot, continuously-updated copy of each store on another instance. If an instance **crashes**, a task can fail over to the standby's host and skip most of the restore, exactly what they're for. But a **rolling restart is not a crash.** When you redeploy, each instance leaves the group (or its session times out) and a [rebalance](https://www.conduktor.io/kafka-streams/rebalancing) reassigns tasks. The instance that comes back up rejoins as what the assignor sees as a candidate for tasks, and if it doesn't already hold the local state for the tasks it's given, **it restores them**. Worse, while a deploy is in flight, your standbys are *also* being bounced, so the hot copy you were counting on may not be where you need it at the moment a task needs to fail over. The net effect: a rolling restart can trigger restores even though you "have standbys," because standbys answer "what if a node dies," not "what if I redeploy every node in sequence." Two things actually address the rolling-restart case: - **[Warm-up replicas](https://www.conduktor.io/kafka-streams/rebalancing) (KIP-441).** When a task must move to a new instance, the assignor restores it in the background as a warm-up *before* handing over the active task, so the switchover is near-instant. This is the mechanism that makes a moved task non-disruptive, and it's covered in detail on the rebalancing page. - **Keeping the local state across the restart** so there's nothing to restore in the first place. This is the bigger lever, and it's mostly an infrastructure decision. ## The biggest lever: persistent local state Restore exists to rebuild state the instance doesn't have locally. The most effective fix is therefore to make sure the instance *still has it* after a restart. | Deployment shape | What happens on restart | Restore cost | |---|---|---| | **Ephemeral pods** (emptyDir, no volume) | Local RocksDB disk is wiped; instance comes back with nothing | Full restore from changelog, **every restart** | | **StatefulSet + persistent volume** | RocksDB files survive the pod restart on the attached disk | Restore only the *delta* missed while down, often seconds | On Kubernetes this is the difference between a `Deployment` with ephemeral storage and a `StatefulSet` with a `volumeClaimTemplate` backing a persistent disk. With a persistent volume, a pod that bounces and reattaches its disk finds its RocksDB store intact and only needs to catch up on the changelog records written while it was down. Combined with [static membership](https://www.conduktor.io/kafka-streams/rebalancing), the rejoining pod reclaims its old tasks *and* its old state, and skips both the rebalance and the restore. The traps live in *which* persistent storage you pick: - **Block storage (EBS-class) is the right default.** It behaves like a local disk, gives RocksDB the low-latency random I/O it wants, and survives a pod restart when reattached. The catch is that a block volume is typically bound to one availability zone, so a pod can only reattach it by landing in the same AZ, a constraint your scheduler has to respect. - **Network file storage (EFS/NFS-class) looks convenient and bites.** RocksDB is latency-sensitive and assumes local-disk semantics; running its files over a network filesystem adds latency to every store operation and degrades throughput, both at restore time and in steady state. Teams who put RocksDB on shared NFS-style storage also hit `Stale file handle` errors when the underlying handle is invalidated mid-operation, a failure mode you simply don't get on a local or block-attached disk. Avoid it for RocksDB state. Two more knobs matter at the edges: - **`state.cleanup.delay.ms`** (default 600000, 10 min) controls how long Streams waits before deleting the local state directory of a task that has migrated *away* from this instance. Leave it generous: if a task bounces back to this instance within the window, its state is still on disk and no restore is needed. Set it too aggressively and you throw away local state you'd have reused. - **`acceptable.recovery.lag`** (default 10000) decides how close a warm-up/recovering store must be to the changelog tail before it's treated as caught up and eligible to serve as the active task. It's the dial between "hand over sooner, slightly behind" and "wait for fully current." ## The exception: a hard crash under exactly-once Persistent volumes save you from restore on a *clean* restart. A non-graceful crash under [exactly-once](https://www.conduktor.io/kafka-streams/exactly-once) is the exception, and it catches people who assumed the disk would save them. Under `exactly_once_v2`, the write to the local RocksDB store and the commit of the changelog (and input offsets) are not one atomic transaction spanning RocksDB and Kafka. Through Kafka Streams 4.2, Streams reconciles them with a `.checkpoint` file that records "this local store is consistent up to changelog offset X", and under EOS that checkpoint is written only on a **graceful** shutdown. So after a `kill -9`, an OOMKill, or a node failure, there is no checkpoint, the local store may contain writes from a transaction that was later aborted, and Streams cannot tell clean state from dirty. Its only safe move is to **discard the local store and replay the whole changelog**, even though the RocksDB files are sitting right there on the persistent disk. Since 4.3 (KIP-1035) the `.checkpoint` file is gone and changelog offsets live inside the store itself, but the outcome after a hard crash is the same: the store is wiped and the full changelog replays. The practical consequence: **graceful shutdown is a performance feature under EOS.** Handle `SIGTERM`, give the process time to flush and write its checkpoint (on Kubernetes, a large enough `terminationGracePeriodSeconds`), and a redeploy avoids the wipe. A hard kill forfeits it and pays the full restore. > **This is changing.** KIP-1035 (managed changelog offsets) shipped in Kafka Streams 4.3, moving offset tracking out of the `.checkpoint` file and into the store itself, but on its own it does not stop the crash wipe. KIP-892 (transactional state stores, targeted for 4.4) is the piece that makes the local store transactional, so a crash rolls back cleanly instead of wiping. Until your app runs a Streams version with KIP-892 (it's a client-library feature, not a broker one), treat a hard crash under EOS as a full restore. ## Standbys and warm-ups, sized for restore Pulling the levers together for an app where restore is genuinely painful: ```properties # Keep a hot copy so a CRASH can fail over without a full restore num.standby.replicas=1 # Move more state in the background when scaling/replacing instances max.warmup.replicas=2 # How caught-up a recovering store must be before taking over the active task acceptable.recovery.lag=10000 # Don't discard local state too eagerly when a task moves away briefly state.cleanup.delay.ms=600000 ``` `num.standby.replicas=1` plus persistent volumes plus static membership covers the three events that cause restore: a crash (standby fails over), a rolling restart (persistent disk + static membership reclaims state), and a scale-out (warm-up restores in the background). What it costs is real: extra disk and memory per instance, plus the changelog-consumption traffic to keep every standby current, so reach for standbys when restore time is the bottleneck, not by default. ## Where this is heading: remote state stores The whole reason restore is slow is that state lives on *local* disk and must be reconstructed from the changelog whenever a task lands somewhere new. The ecosystem's clearest current direction is to break that assumption: **remote, object-storage-backed state stores** keep state off local disk so a task can attach to existing state almost immediately, making restore near-instant and instances effectively stateless. Kafka Streams' state-store interface is pluggable, so this is a live area rather than a thought experiment: it's the dominant theme across recent Current and Kafka Summit talks on Streams. We cover where it's going in [the future of Kafka Streams](https://www.conduktor.io/kafka-streams/future). ## Frequently Asked Questions **Why does my Kafka Streams app take so long to start up?** It's almost always restoring state: a task replays its changelog topic from offset zero into a local RocksDB store before it can process a record, and that time is proportional to store size. A store with hundreds of millions of keys can take 45 minutes to two hours to rebuild cold, and nothing processes during that window. **Do standby replicas make restores during a deploy instant?** No. Standby replicas (`num.standby.replicas`) cover a crash: a task fails over to a host that already holds a hot copy. A rolling restart is not a crash: each rejoining instance restores any tasks it doesn't hold locally, and your standbys are being bounced too. They answer "what if a node dies," not "what if I redeploy every node in sequence." **How do I avoid restore on every restart?** Keep the local state across the restart so there's nothing to rebuild. On Kubernetes that means a `StatefulSet` with a persistent volume (block storage / EBS-class) instead of ephemeral pods: the RocksDB files survive and only the delta missed while down is restored. Avoid network file storage (EFS/NFS) for RocksDB; it adds latency and triggers `Stale file handle` errors. **Why does a hard crash under exactly-once trigger a full restore even with a persistent disk?** Under `exactly_once_v2` the `.checkpoint` file that marks the local store consistent is written only on a graceful shutdown (through 4.2; since 4.3 offsets live inside the store, with the same crash behavior). After a `kill -9` or OOMKill the store may hold writes from an aborted transaction, and Streams can't tell clean from dirty, so it discards the local store and replays the whole changelog. Handle `SIGTERM` and allow time to flush. **How is local state rebuilt from the changelog?** Each write to a state store is also appended to a compacted Kafka topic (the changelog), which is the real source of truth. On restore, a restore consumer reads that topic from offset zero and writes each record back into RocksDB until it catches up to the tail; only then does the task move from `RESTORING` to `RUNNING`. > **See it in practice with Conduktor** > A restore is invisible from inside your app logs until you correlate it with the broker. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-state-restore) lets you watch the consumer-group lag that tells you a restore is still in progress after a deploy, inspect the changelog topics behind your stores to see their size and compaction, and confirm partition assignment, so "is it stuck or is it still restoring?" stops being a guess. ## Next steps - [Why rebalancing is slow](https://www.conduktor.io/kafka-streams/rebalancing): the event that triggers most restores, and how to tame it - [RocksDB tuning](https://www.conduktor.io/kafka-streams/rocksdb-tuning): memory, disk, and write throughput that bound restore speed - [Kafka Streams state stores](https://www.conduktor.io/kafka-streams/state-store): how changelogs and local stores work in the first place Source: https://www.conduktor.io/kafka-streams/state-restore --- # Kafka Streams state stores *Understand how Kafka Streams keeps state, and why it bites in production.* Every stateful operation in Kafka Streams (a count, an aggregation, a join) needs somewhere to keep its running state. That somewhere is a **state store**: a local, embedded key-value store that lives inside your application process, backed by a Kafka topic so it can survive a crash. State stores are where Kafka Streams stops being a tidy abstraction and starts having opinions about your memory, your disk, and your restart times. Understanding them is the difference between an app that scales and one that gets OOMKilled at 3am. **What you'll learn:** - What a state store is and where it lives - How the changelog topic makes local state fault-tolerant - Why RocksDB memory doesn't show up in your JVM heap - The restore trade-off between in-memory and persistent stores ## What a state store is When you call `count()`, `aggregate()`, or `reduce()`, Kafka Streams creates a state store to hold the result for each key. The store is: - **Local**: it lives on the instance (and the disk) of whichever task owns the partition. It is not a shared, remote database. - **Partitioned**: there is one store instance per partition of the input. A topic with 12 partitions means up to 12 independent stores, spread across your running instances. - **Embedded**, by default it's [RocksDB](https://rocksdb.org/), a fast embedded key-value engine written in C++, running *inside* your JVM process. Stateless operations (`map`, `filter`, `flatMap`) need no store. Stateful ones do, and that store is the thing you have to operate. ## The changelog topic: how local state survives a crash ![Every state-store write goes to both local RocksDB on disk and the changelog topic in Kafka; the changelog is the source of truth for recovery](https://www.conduktor.io/assets/kafka-streams/diagrams/state-store-changelog-write-path.svg) A local store on a single instance sounds fragile, if that instance dies, is the state gone? No. Every write to a state store is *also* written to a dedicated, compacted Kafka topic called the **changelog**. ``` your-app-id-KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog ``` This changelog is the source of truth. If a task moves to another instance (a rebalance) or an instance restarts with no local data, Kafka Streams **restores** the store by replaying its changelog from the beginning before processing resumes. Because the changelog is [log-compacted](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction), it only retains the latest value per key, so it stays roughly the size of your state, not the size of all updates ever made. > **These topics are created behind your back.** Kafka Streams auto-creates changelog (and [repartition](https://www.conduktor.io/kafka-streams/topology-evolution)) topics, named after your `application.id` and the operator's *position* in the topology. A stateful app with several aggregations and joins can quietly triple your topic and partition count. Know they exist before you go counting partitions on your cluster. This restore step is also the source of one of the biggest operational pains in Kafka Streams: a large store takes a long time to replay. We cover that in [state restore time](https://www.conduktor.io/kafka-streams/state-restore). ## The memory that doesn't show up in your heap Here is the surprise that puts Kafka Streams apps in the postmortem channel. RocksDB is a native C++ library. Its memory, block cache, write buffers (memtables), index and filter blocks, lives **off-heap**. Your `-Xmx` setting does not bound it. Your JVM heap metrics look perfectly healthy right up until the Linux OOM killer terminates the container, and what you get is not a Java `OutOfMemoryError` with a stack trace, it's a silent `OOMKilled`. > 🚫 *"My heap usage is flat at 60%, so the app has plenty of memory."* It gets worse with scale. There is one RocksDB instance **per partition** of each store, and each one carries its own default memory budget, a 50 MiB block cache plus write buffers (16 MiB each, up to three). An instance owning 20 partitions of a store is running 20 RocksDB databases, and their off-heap footprints add up far past anything your heap graph shows. The fix is to bound RocksDB explicitly with a shared cache across all the stores on an instance, via a `RocksDBConfigSetter`: ```java public class BoundedMemoryRocksDBConfig implements RocksDBConfigSetter { // One shared 64 MiB block cache + write-buffer budget for ALL stores on this instance private static final long TOTAL_OFF_HEAP = 64L * 1024 * 1024; private static final Cache CACHE = new LRUCache(TOTAL_OFF_HEAP); private static final WriteBufferManager WBM = new WriteBufferManager(TOTAL_OFF_HEAP, CACHE); @Override public void setConfig(String storeName, Options options, Map configs) { BlockBasedTableConfig table = (BlockBasedTableConfig) options.tableFormatConfig(); table.setBlockCache(CACHE); table.setCacheIndexAndFilterBlocks(true); options.setWriteBufferManager(WBM); options.setTableFormatConfig(table); } @Override public void close(String storeName, Options options) { /* shared objects: do not close per store */ } } ``` ```properties rocksdb.config.setter=com.example.BoundedMemoryRocksDBConfig ``` Sizing the off-heap budget, taming memory fragmentation with jemalloc, and the disk-fills-up failure mode are their own topic, see [RocksDB tuning](https://www.conduktor.io/kafka-streams/rocksdb-tuning). ## In-memory vs persistent: the restore trade-off Kafka Streams also offers in-memory stores (`Stores.inMemoryKeyValueStore(...)`). They sound faster, and for steady-state reads they are, but they hide a sharp edge. | | Persistent (RocksDB, default) | In-memory | |---|---|---| | Steady-state speed | Fast (off-heap, disk-backed) | Fastest | | Memory | Off-heap, can spill to disk | On-heap, bounded by `-Xmx` | | **Restart / failover** | Clean restart: restores only the missing *delta*. Failover to a fresh instance: full replay | **Restores the entire store from the changelog, every time** | That last row is the catch. A persistent store keeps its data on local disk, so after a clean restart it only needs to catch up on what it missed. An in-memory store starts empty on every restart and must replay the **whole** changelog before it can process a single record. For a large store, that turns a 10-second restart into a multi-minute one. Choose in-memory only for small, fast-changing state where you've accepted the restore cost. You materialize a store explicitly, and **name it**, like this: ```java .groupByKey() .count(Materialized.as("orders-per-customer")); // named store: stable changelog, queryable, survives topology edits ``` Naming matters for more than tidiness: an unnamed store gets a *positional* name that shifts if you edit the topology, orphaning its state. See [evolving a topology](https://www.conduktor.io/kafka-streams/topology-evolution). ## Querying state directly A materialized store isn't just internal plumbing, you can read it from your service with **interactive queries**, turning a Kafka topic into a queryable, real-time view without a separate database: ```java ReadOnlyKeyValueStore store = streams.store(StoreQueryParameters.fromNameAndType("orders-per-customer", QueryableStoreTypes.keyValueStore())); Long count = store.get("customer-42"); ``` With more than one instance, the state is sharded across hosts, so a given key may live on another instance. You ask `streams.queryMetadataForKey(...)` which instance owns the key, then route the call there. A common gotcha: a query that works with one instance returns `InvalidStateStoreException: the state store may have migrated to another instance` once you scale to two, that's the routing step you're missing, or a transient rebalance (retry, or serve stale reads with `StoreQueryParameters` enabled for it). ## When local state isn't enough The whole local-state model, fast, but tied to restore time and instance memory, is exactly what the ecosystem is now trying to move past. Remote, object-storage-backed state stores (the dominant theme across recent Current and Kafka Summit talks) keep state off local disk so restores become near-instant and instances become stateless. Kafka Streams' state store interface is pluggable, so this is an active, real area, not vaporware. We cover the direction in [the future of Kafka Streams](https://www.conduktor.io/kafka-streams/future). ## Frequently Asked Questions **What is a state store in Kafka Streams?** A state store is a local, embedded key-value store inside your application process that holds the running state of stateful operations like `count`, `aggregate`, and `reduce`. There is one store instance per input partition, and each is backed by a Kafka topic so it survives a crash. **What is the difference between RocksDB and in-memory state stores?** The default RocksDB store keeps data off-heap on local disk, so a clean restart only catches up on the missed delta. An in-memory store starts empty on every restart and must replay the whole changelog before processing, turning a fast restart into a multi-minute one for large state. **What is a changelog topic and how does it make state fault-tolerant?** The changelog is a dedicated, log-compacted Kafka topic that records every write to a state store, making it the source of truth. If a task moves or an instance restarts empty, Kafka Streams restores the store by replaying the changelog before resuming, and compaction keeps it roughly the size of the state rather than every update. **Why does my Kafka Streams pod get OOMKilled when the heap looks healthy?** RocksDB is a native C++ library whose block cache, write buffers, and index blocks live off-heap, so your `-Xmx` does not bound them and the JVM heap graph stays flat. With one RocksDB instance per partition, the off-heap footprints add up and the Linux OOM killer terminates the container with no Java `OutOfMemoryError`. **Can I plug in a custom or remote state store?** Yes, the state store interface is pluggable, which is why remote, object-storage-backed stores are an active area of development. The goal is to keep state off local disk so restores become near-instant and instances become effectively stateless. > **See it in practice with Conduktor** > The changelog and repartition topics behind your stores are real Kafka topics. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-state-store) lets you see them, check their size and compaction, and watch the consumer group lag that tells you whether a restore is still in progress after a deploy. ## Next steps - [State restore time & standby replicas](https://www.conduktor.io/kafka-streams/state-restore), why restarts are slow, and how to speed them up - [RocksDB tuning](https://www.conduktor.io/kafka-streams/rocksdb-tuning), bounding memory and disk for real - [Windowing](https://www.conduktor.io/kafka-streams/windowing), the most common source of unbounded state growth Source: https://www.conduktor.io/kafka-streams/state-store --- # Kafka Streams state store TTL *Learn why Kafka Streams state grows forever, and how to expire it safely.* Here is a fact that surprises people in production: **Kafka Streams has no built-in TTL for key-value state stores.** Every key you write to a plain `KeyValueStore` stays there forever, across restarts, across rebalances, until *you* delete it. There is no `expireAfter` setter, no eviction thread, no config flag. An app that aggregates by an unbounded key space (session IDs, request IDs, customer-then-churned IDs) grows its [state store](https://www.conduktor.io/kafka-streams/state-store) without limit until the disk fills or the container is `OOMKilled`. The obvious fixes, turn on RocksDB's own TTL, or call `store.delete()` in a punctuator, both have a sharp edge that brings the "deleted" data back. This page covers why, and the patterns that actually bound the store. **What you'll learn:** - Why windowed stores *are* bounded but plain key-value stores are not - The RocksDB-TTL trap, and how "expired" data resurrects on restore - Three real patterns to expire state, and why a tombstone is the load-bearing step - Why deleting from the local store is never enough on its own ## Why windowed stores are bounded and KV stores aren't ![WindowStore auto-expires via segment drop and compact-delete changelog; KeyValueStore grows forever with compacted-only changelog until a tombstone is written](https://www.conduktor.io/assets/kafka-streams/diagrams/state-ttl-windowed-vs-kv.svg) If you've used windowed aggregations, you may assume state expires automatically, and for *windowed* stores it does. A `WindowStore` (the store behind `TimeWindows` aggregations) has a retention period; segments older than retention are dropped wholesale, and the matching [changelog](https://www.conduktor.io/kafka-streams/state-store) topic uses `compact,delete` so old segments age out of Kafka too. (The changelog's `retention.ms` is the store retention plus `windowstore.changelog.additional.retention.ms`, 24 hours by default: a 2-hour store gets a 26-hour changelog.) Windowed state is self-limiting by design. A plain `KeyValueStore`, what you get from a non-windowed `count()`, `aggregate()`, or `reduce()`, or any store you build by hand, has **no such mechanism**. Its changelog is purely [log-compacted](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction): compaction keeps the latest value per key forever and only ever removes a key when it sees a *tombstone* (a record with a null value) for it. No tombstone, no removal. So both halves of the store, the local RocksDB copy and the changelog in Kafka, grow with the cardinality of your key space, not with anything time-bounded. That single asymmetry is the whole problem. The rest of this page is about how to retrofit expiry onto the unbounded case. ## The RocksDB TTL trap RocksDB, the engine behind the default store, has its own native TTL. It looks like exactly the knob you want. The first trap: you can't even reach the real one. RocksDB's per-value TTL lives in `TtlDB`, a wrapper database that Kafka Streams never opens (`RocksDBStore` calls plain `RocksDB.open`, with zero TTL handling as of Streams 4.3). What a [`RocksDBConfigSetter`](https://www.conduktor.io/kafka-streams/rocksdb-tuning) *can* set is `options.setTtl`, a column-family option that under Streams' default leveled compaction expires nothing: it only schedules old files for cascading compaction to the bottom level. Making that knob drop data at all means switching the store to FIFO compaction, which deletes whole files wholesale. And even real `TtlDB` semantics would still be a trap, for four reasons that compound: - **It's insertion-time, not access- or update-time.** `TtlDB` expires a value a fixed duration after it was *written*, regardless of how recently you read or updated the logical key. "Keep keys idle for 30 days" is not expressible. - **It's non-strict.** Expired data is only physically removed during compaction, so a `get()` or range scan can still return a value that is technically past its TTL. Your expiry logic becomes nondeterministic. - **It gives Streams no eviction callback.** When RocksDB drops an expired entry on compaction, Kafka Streams is never told. So Streams never writes a tombstone for that key. - **The config setter is global.** A `RocksDBConfigSetter` applies to *every* store on the instance, you can't TTL one store and not another without branching on `storeName`. The third point is the one that bites. Because Streams isn't notified, the **changelog still contains the key**, RocksDB quietly dropped it locally, but the source of truth in Kafka was never tombstoned. On the next [restore](https://www.conduktor.io/kafka-streams/state-restore) (a rebalance, a failover, a fresh instance), Streams replays that changelog and faithfully **re-inserts every "expired" key into the rebuilt store**. The data resurrects. > 🚫 *"I set a RocksDB TTL, so my Kafka Streams state is bounded now."* It isn't. At best, RocksDB's TTL trims the *local* copy on compaction but never tells Streams, so the changelog keeps every key, and the first restore replays them all straight back into the store. In Streams' default configuration it's worse: the reachable `setTtl` knob frees nothing locally either. You've added nondeterministic local reads without bounding anything durable. To bound the state you must remove keys in a way the changelog learns about: a tombstone. That's what the patterns below do. ## The patterns that actually work The throughline of every working pattern: **expiry must reach the changelog as a tombstone.** Deleting only the local RocksDB entry is cosmetic, the changelog still has the key, and restore brings it back. There are three solid approaches, in rough order of preference. ### Pattern A: re-key into a windowed store If your access pattern allows it, the cleanest answer is to not use an unbounded KV store at all. Re-key your data by entity into a `WindowStore` (or session store) with a retention equal to your desired TTL, and let Streams' built-in windowed-retention machinery age old windows out for free, both locally and in the changelog. No punctuator, no manual tombstones, no resurrection risk. This is the right default whenever "expire after N of inactivity/age" maps onto a window. When it doesn't, you need a true per-key idle timeout, or random-access lookups by key, fall through to B. ### Pattern B: a punctuator that scans, deletes, and tombstones The general-purpose pattern uses the [Processor API](https://www.conduktor.io/kafka-streams/processor-api): store a last-seen timestamp alongside each value, and schedule a **wall-clock** punctuator that finds stale keys and removes them. The critical detail is *how* you remove them. ```java public class TtlProcessor implements Processor { private KeyValueStore> store; private static final Duration TTL = Duration.ofDays(30); @Override public void init(ProcessorContext context) { // "entity-store" must be registered with Stores.timestampedKeyValueStoreBuilder, // otherwise this assignment to ValueAndTimestamp values fails at runtime. this.store = context.getStateStore("entity-store"); // WALL_CLOCK_TIME: fires on the poll cadence even when no records arrive. // STREAM_TIME would never fire on an idle partition, and idle is exactly // when you most need to expire stale state. context.schedule(Duration.ofMinutes(5), PunctuationType.WALL_CLOCK_TIME, this::expire); } private void expire(long now) { try (KeyValueIterator> it = store.all()) { while (it.hasNext()) { KeyValue> entry = it.next(); if (now - entry.value.timestamp() > TTL.toMillis()) { // store.delete() writes a NULL (tombstone) to the changelog. // That tombstone is what makes the deletion survive a restore. store.delete(entry.key); } } } } // process(...) updates store with ValueAndTimestamp.make(event, now) } ``` `store.delete(key)` on a changelogged store writes a null-valued record to the changelog. That null is the tombstone, and it is the entire point: on the next restore, replaying the changelog applies the tombstone and the key stays gone. A bare local removal that skipped the changelog would resurrect. Two operational landmines, both real: - **The punctuator runs *on* the stream thread and pauses processing while it runs.** A `store.all()` scan over millions of keys can block consumption for seconds, risk a poll-timeout-driven [rebalance](https://www.conduktor.io/kafka-streams/rebalancing), and grow linearly with state size. Mitigate by storing keys with a timestamp prefix and using `store.range(...)` to scan only the segment that could be expiring, or by time-boxing the punctuator (process for N milliseconds, remember the last key visited, resume on the next tick). Don't `store.all()` over a large store on a tight schedule. - This is **state expiry, unrelated to error handling.** Routing a *bad* record elsewhere is a [dead letter queue](https://www.conduktor.io/kafka-streams/dead-letter-queue); routing an *old* key out of the store is TTL. They share no machinery, don't conflate the two. ### Pattern C: tombstone the source topic for a KTable If the unbounded state is a `KTable` materialized from a source topic, the cleanest eviction is to **emit a tombstone to the source topic** (from an external job, or a punctuator producing to it). The KTable processes the null, deletes the key from its store, *and*, because the store's changelog mirrors those deletes and is compacted, the changelog compacts the key away too. One tombstone reclaims space in the local store, the changelog, and (eventually) the source topic. This is the only approach where the deletion propagates all the way back to the source of truth, so nothing downstream that re-reads the topic resurrects it either. (A related primitive for collapsing duplicate keys is covered in [deduplication](https://www.conduktor.io/kafka-streams/deduplication).) ## Why local deletion is never enough, restate the trap It's worth saying once more because it's the mistake everyone makes first. Three of the "obvious" ways to free state, RocksDB native TTL, a `Stores`-level in-memory eviction, or any code that drops the entry from the local map without going through the store's delete path, all leave the **changelog untouched**. The local copy shrinks; the durable copy doesn't. Then a restore replays the changelog and the state you "expired" comes straight back, often days later when an unrelated rebalance triggers the restore, which is the worst possible time to discover the bug. | Approach | Frees local store? | Tombstones changelog? | Survives restore? | |---|---|---|---| | RocksDB native TTL | Not under default compaction (FIFO only) | No | **No, data resurrects** | | `store.delete()` in a punctuator | Yes | Yes | Yes | | Tombstone to source topic (KTable) | Yes | Yes (compacts away) | Yes | | Windowed store + retention (Pattern A) | Yes (segment drop) | Yes (`compact,delete`) | Yes | The rule is one line: **to bound Kafka Streams state, the deletion must reach the changelog as a tombstone.** Everything that doesn't is theater. ## Frequently Asked Questions **Does Kafka Streams have a built-in TTL?** Not for key-value state stores. There is no `expireAfter` setting or eviction thread for a plain `KeyValueStore`, keys live forever until you delete them. Only *windowed* stores expire automatically, via their retention period. For non-windowed state you implement expiry yourself with a punctuator or by tombstoning. **Why does my Kafka Streams state store grow forever?** Because a plain key-value store has no time bound: its changelog is log-compacted, which keeps the latest value per key indefinitely and only removes a key when it sees a tombstone. If your keys come from an unbounded space (session IDs, request IDs) and you never tombstone them, both the local RocksDB store and the changelog grow with key cardinality until the disk fills. **How do I expire old keys in a Kafka Streams state store?** Three patterns: re-key into a windowed store and let retention drop old windows; or run a wall-clock punctuator that finds stale keys and calls `store.delete()` (which writes a tombstone to the changelog); or, for a KTable, emit a tombstone to the source topic. In every case the deletion must reach the changelog as a tombstone, or a restore brings the key back. **Does RocksDB's TTL work for Kafka Streams?** No, it's a trap twice over. RocksDB's real per-value TTL (`TtlDB`) isn't reachable from Streams at all, and the `options.setTtl` you can set via a `RocksDBConfigSetter` expires nothing under the default leveled compaction. Even where RocksDB TTL does apply, it's insertion-time-based, only evicts on compaction (so reads can still return expired data), and gives Streams no eviction callback: Streams never tombstones the changelog, so the next state restore replays every "expired" key and the data resurrects. Don't rely on it to bound state. **Why does deleted state come back after a restart?** Because the deletion only touched the local RocksDB copy and never wrote a tombstone to the changelog. The changelog is the source of truth: on restart or rebalance, Streams rebuilds the store by replaying it, re-inserting any key it still contains. Always delete through the store's delete path (`store.delete()`) so a tombstone is written. > **See it in practice with Conduktor** > Unbounded state is visible from outside the app: the [changelog topic](https://www.conduktor.io/kafka-streams/state-store) behind a store grows with it. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-state-ttl) lets you inspect that changelog topic's size and compaction settings, and watch the consumer-group lag that tells you whether a restore (the moment "expired" keys resurrect) is in progress, so you can confirm your TTL pattern is actually shrinking the durable state, not just the local copy. ## Next steps - [State stores](https://www.conduktor.io/kafka-streams/state-store), how the changelog makes local state durable, and why that's what TTL has to reach - [Processor API](https://www.conduktor.io/kafka-streams/processor-api), punctuators, `context.schedule()`, and time-boxing the expiry scan - [RocksDB tuning](https://www.conduktor.io/kafka-streams/rocksdb-tuning), bounding the off-heap memory and disk that unbounded state consumes Source: https://www.conduktor.io/kafka-streams/state-ttl --- # Kafka Streams stateless operations *Learn the stateless DSL, and the one that costs you a network round-trip.* Stateless operations are the workhorses of a Kafka Streams topology: transform a record, drop it, split the stream, recombine it. They keep no state, need no store, and survive a restart with nothing to recover. Most of your topology is probably stateless. But "stateless" doesn't mean "free." One distinction inside this group, whether an operator changes the record's *key*, decides whether Kafka Streams quietly inserts a repartition topic and a full network round-trip downstream. It's the cheapest performance win most people miss. **What you'll learn:** - The full stateless DSL: map, filter, flatMap, branch, merge, and friends - Which operators touch the key, and why that's the line that matters - Why `map`/`selectKey` can trigger a repartition while `mapValues`/`filter` never do - When stateless is enough, and when you've crossed into [stateful](https://www.conduktor.io/kafka-streams/aggregations) territory ## The stateless toolkit Every operator here takes a `KStream` and returns a `KStream`. No store, no changelog, no co-partitioning rules. They group into a few families. **Transform** ```java // map: change key AND/OR value, returns a new KeyValue KStream lengths = lines.map((k, v) -> KeyValue.pair(v.getUserId(), v.length())); // mapValues: change the value only, key untouched KStream sizes = lines.mapValues(v -> v.length()); ``` **Filter** ```java // filter: keep records matching the predicate KStream paid = orders.filter((k, order) -> order.isPaid()); // filterNot: the inverse, keep records that DON'T match KStream unpaid = orders.filterNot((k, order) -> order.isPaid()); ``` **Expand** ```java // flatMap: one record in, zero-or-many out, key and value free to change KStream words = lines.flatMap((k, line) -> Arrays.stream(line.split(" ")).map(w -> KeyValue.pair(w, w)).toList()); // flatMapValues: same, but key is preserved KStream tokens = lines.flatMapValues(line -> Arrays.asList(line.split(" "))); ``` **Re-key** ```java // selectKey: set a new key from the existing key/value, value untouched KStream byUser = clicks.selectKey((k, click) -> click.getUserId()); ``` **Split and recombine** ```java // split/branch: route records into named branches by predicate (modern API) Map> branches = orders.split(Named.as("by-")) .branch((k, o) -> o.getAmount() > 1000, Branched.as("large")) .branch((k, o) -> o.getAmount() > 0, Branched.as("small")) .defaultBranch(Branched.as("other")); // branches.get("by-large"), branches.get("by-small"), ... // merge: combine two streams of the same key/value type into one KStream all = largeOrders.merge(smallOrders); ``` **Observe (no transformation)** ```java // peek: side-effect per record (logging, metrics), record passes through unchanged orders.peek((k, o) -> log.debug("order {}", k)); // foreach: terminal side-effect, consumes the stream, returns nothing orders.foreach((k, o) -> log.info("processed {}", k)); ``` `peek` returns the stream so you can keep chaining; `foreach` is terminal. Neither is a place for a database call or a REST request, a blocking call here stalls the whole stream thread (risking a [rebalance](https://www.conduktor.io/kafka-streams/rebalancing) if it misses its poll deadline), and any external side effect it performs sits outside Kafka's [exactly-once](https://www.conduktor.io/kafka-streams/exactly-once) guarantee. ## The line that matters: does it touch the key? ![map marks the stream for repartition; the internal repartition topic only materializes when a downstream stateful operator like groupByKey or count consumes it - a fork shows path A to .to() with no topic created versus path B to groupByKey triggering KSTREAM-SINK to repartition topic to KSTREAM-SOURCE to aggregation store](https://www.conduktor.io/assets/kafka-streams/diagrams/stateless-repartition-trigger.svg) Here is the distinction that separates a cheap operator from one that can cost you an extra topic and a network hop. Kafka Streams partitions data by key. A stateful operation downstream, an aggregation, a join, needs all records for a given key on the *same* partition. So if an operator **changes the key**, Kafka Streams can no longer trust that the partitioning still matches the key. It **marks the stream for repartition**. Marking is not the same as doing it. The repartition, a write to an internal topic and a read back, is only **triggered** when a downstream operator actually depends on correct partitioning (a `groupByKey`, an aggregation, a join). If you re-key and then just `.to()` an output topic, no repartition topic is created; the mark goes unused. | Operator | Changes the key? | Marks for repartition? | |---|---|---| | `mapValues`, `flatMapValues` | No | No | | `filter`, `filterNot` | No | No | | `peek`, `foreach` | No | No | | `branch`/`split`, `merge` | No | No | | `map`, `flatMap` | Yes (can) | Yes | | `selectKey` | Yes | Yes | | `groupBy` | Yes (re-keys) | Yes | The pattern that bites: someone writes `map((k, v) -> KeyValue.pair(v.id(), transform(v)))` to change the value, and changes the key to `v.id()` only because `map` made them return a full `KeyValue`. The key change marks the stream. Add a `count()` downstream and Kafka Streams silently materializes a repartition topic, extra partitions on the cluster, an extra produce-and-consume per record, and an extra place for things to lag. The fix is almost always free: if you only need to change the value, use `mapValues` (or `flatMapValues`). The key is preserved, nothing is marked, no repartition can be triggered. > **Prefer `mapValues` over `map` whenever the key stays the same.** It's not a micro-optimization. A needless re-key followed by any stateful op adds a full internal topic and a network round-trip per record. Reach for `map`/`selectKey` only when you genuinely need a new key, for a join or an aggregation that groups by something other than the current key. > 🚫 *"Stateless operators are free, so it doesn't matter which one I use."* ## Reading whether a repartition will happen You don't have to guess. `topology.describe()` shows where a repartition happens, and the build log mentions auto-created repartition topics. When you re-key and feed a stateful operator, the description splits into two sub-topologies joined by an internal topic whose name ends in `-repartition` (a `KSTREAM-SINK` writes to it, a `KSTREAM-SOURCE` reads it back). That name is positional unless you set one, which is also why reordering operators is risky, see [evolving a topology](https://www.conduktor.io/kafka-streams/topology-evolution). If a repartition is genuinely required, name it explicitly with `Repartitioned.as("...")` (or `Grouped.as("...")` on the grouping step) so the topic name is stable across topology edits. An unnamed repartition topic gets a positional name that shifts the moment you insert an operator above it. ## Stateless vs stateful: where the easy half ends Everything on this page keeps no state. That's why these operators don't need a store, don't create a changelog, don't have co-partitioning requirements, and recover instantly, there's nothing to recover. The moment you `count`, `reduce`, `aggregate`, join two streams, or window anything, you've crossed into stateful processing: a local [state store](https://www.conduktor.io/kafka-streams/state-store), a changelog topic, restore-on-restart, and the memory and rebalance behaviour that come with it. That's a different operational world, covered in [aggregations](https://www.conduktor.io/kafka-streams/aggregations). The boundary between the two is exactly the re-key discussed above. `groupBy`/`groupByKey` is where a stateless stream becomes the input to a stateful operation, and where a marked-for-repartition stream finally pays for the repartition. Keeping the stateless part of your topology key-stable means that boundary is the *only* place you pay, not three operators earlier. ## Frequently Asked Questions **What is the difference between map and mapValues in Kafka Streams?** `map` can change both the key and the value and returns a full `KeyValue`; `mapValues` changes only the value and preserves the key. Because `map` may change the key, it marks the stream for repartition, while `mapValues` never does, so prefer `mapValues` whenever the key stays the same. **Which Kafka Streams operations trigger a repartition?** The operators that can change the key, `map`, `flatMap`, `selectKey`, and `groupBy`, mark the stream for repartition. Key-preserving operators like `mapValues`, `flatMapValues`, `filter`, `peek`, `branch`, and `merge` never do. **How do I split a stream into multiple branches?** Use `split()` with `branch()` to route records into named branches by predicate, optionally ending with `defaultBranch()`. It returns a `Map` of branch name to `KStream`, and you read each branch back by its name; `branch`/`split` does not change the key, so it triggers no repartition. **Does selectKey cause a repartition topic to be created?** `selectKey` marks the stream for repartition because it changes the key, but the repartition topic is only actually created when a downstream operator depends on correct partitioning, a `groupByKey`, an aggregation, or a join. Re-key and then just `.to()` an output topic and no repartition topic appears. **What are the stateless operations in the Kafka Streams DSL?** The stateless operators include `map`/`mapValues`, `filter`/`filterNot`, `flatMap`/`flatMapValues`, `selectKey`, `branch`/`split`, `merge`, and the observe-only `peek` and `foreach`. They keep no state, need no store or changelog, have no co-partitioning rules, and recover instantly because there is nothing to restore. > **See it in practice with Conduktor** > When a re-key triggers a repartition, a real internal topic appears on your cluster. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-stateless-operations) lets you spot those `-repartition` topics in the catalog, confirm whether your topology created ones you didn't intend, and watch their lag, a quick way to catch an accidental `map` that should have been `mapValues`. ## Next steps - [Kafka Streams aggregations](https://www.conduktor.io/kafka-streams/aggregations), where stateless ends and stateful begins - [KStream, KTable & GlobalKTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable), the streams and tables these operators act on - [Build your first Kafka Streams app](https://www.conduktor.io/kafka-streams/getting-started), a runnable topology in Java Source: https://www.conduktor.io/kafka-streams/stateless-operations --- # Why your Kafka Streams suppress() emits nothing *Debug a suppress() that never fires.* You wired up a windowed aggregation, added `suppress(Suppressed.untilWindowCloses(...))` to get one clean result per window instead of the noisy intermediate stream, deployed it, and the output topic is empty. No errors. No warnings. The aggregation is clearly running; the downstream topic just never receives a thing. In a test it might work; in production, or against a low-traffic partition, it goes silent. This is the single most-reported `suppress` problem, and it is almost never a bug in Kafka Streams. It is a direct consequence of how `untilWindowCloses` decides a window is finished. Once you internalize the one rule below, every variant of "suppress isn't emitting" becomes a checklist you can walk in a few minutes. **What you'll learn:** - The one rule that governs when `untilWindowCloses` fires, and why an idle partition breaks it - A root-cause checklist: idle streams, one lagging partition, test harnesses, buffer config - The dummy-future-event workaround, why it works, and why it's a smell - When to abandon `suppress` for a wall-clock punctuator instead ## The one rule: suppress fires on stream-time, not the clock ![suppress buffer fills with held results at 10:30 and 10:58; wall-clock passing 11:05 has no effect; only a record at 11:06 crossing the threshold flushes the buffer](https://www.conduktor.io/assets/kafka-streams/diagrams/suppress-buffer-stream-time.svg) `suppress(Suppressed.untilWindowCloses(...))` buffers every windowed result and releases it at exactly one moment: when the window *closes*. A window closes when **stream-time** passes the window's end plus its grace period. That is the whole mechanism, and every failure below is a way of stream-time not getting there. Stream-time is not your wall clock. It is Kafka Streams' internal sense of how far through the data a task has progressed, defined as the **maximum record timestamp the task has seen so far**. It is monotonic, it only moves forward, and, the part that bites, **it only advances when a new record arrives** carrying a timestamp later than the current maximum. No new record, no movement. The concept is covered in full in [windowing](https://www.conduktor.io/kafka-streams/windowing); here we use it as the lens for debugging. So for a window covering 10:00–11:00 with 5 minutes of grace, `suppress` releases that window's result the instant a record flows through the task with a timestamp ≥ 11:05. Not when your server clock reaches 11:05, when a *record stamped* 11:05-or-later is actually processed by that task. ```java events .groupByKey() .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofHours(1), Duration.ofMinutes(5))) .count() .suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded())) .toStream() .to("hourly-final"); // emits ONLY when stream-time crosses window-end + grace ``` Read that last comment as a warning, not a guarantee. If stream-time never crosses 11:05, `hourly-final` stays empty, forever, no matter how much real time passes. ## The root-cause checklist ### 1. Stream-time isn't advancing (the idle / low-traffic / sparse-key partition) This is the cause in the large majority of cases. The window for 10:00–11:00 can only close when a record with timestamp ≥ 11:05 arrives **on the same partition**. If the last event of the hour landed at 10:58 and the stream then went quiet, stream-time is frozen at 10:58. The window stays open. `suppress` keeps buffering and emits nothing. This bites hardest on: - **Low-traffic topics**, a few records an hour, with long gaps. The window that should have closed at 11:05 waits for the next record, which might be at 11:40, or tomorrow morning, or never. - **Sparse keys on quiet partitions**, the windowed *result* is per key, but the close decision is not: any record on the partition that pushes stream-time past window-end + grace closes *every* key's buffered window there. So a rarely-active key (a test tenant, a dormant account) is fine on a busy partition; other keys' traffic closes its windows too. It only goes silent when it sits on a partition where *all* keys go quiet, or alone on a low-traffic partition. - **End-of-day / weekend tails**, the last window before traffic dies is the one that never fires. You see results all day, then the final one is missing. The fix is conceptual first: there must be a *newer record, with a later timestamp, on the same partition* to push stream-time past `window-end + grace`. If your traffic naturally provides that, the result fires on its own, just later than wall-clock intuition suggests. If it doesn't, you've discovered that `suppress(untilWindowCloses)` is the wrong tool for a sparse stream, and you want the punctuator approach in the last section. > 🚫 *"suppress() is broken, it buffers forever and never emits."* > It is doing exactly what `untilWindowCloses` specifies: hold the result until stream-time crosses window-end + grace. On an idle or sparse partition, stream-time never gets there, so the window never closes. The behavior is correct; the stream is quiet. ### 2. One lagging partition holds back the whole task Stream-time is tracked **per task**. With a single input topic, each task owns exactly one partition of it, so a lagging partition stalls only the keys living on that partition: still "some keys emit, some don't", just clustered by partition. Tasks that own *multiple* partitions arise when a subtopology reads several source topics (joins, `merge()`). There, Kafka Streams chooses the *lowest-timestamp* buffered record to process next, to stay time-ordered, so a partition that is genuinely behind (heavy lag, a slow upstream producer, an offset reset on one partition only) anchors the task's notion of time and holds back windows fed by the faster partitions. Two limits on that holdback: it only applies when the lagging partition has data sitting on the broker. An idle, *empty* partition does not block under the default `max.task.idle.ms=0`; the task processes what it has. And the anchor is transient: the moment one fresh record from the straggler is processed, stream-time jumps to the maximum seen and every closeable window flushes at once. How to spot it: the symptom is partial, not total. Some keys emit, some don't, and the ones that don't tend to cluster. Check per-partition consumer lag on the input topic, an uneven lag profile, where one or two partitions trail the rest, is the tell. Even out the lag (fix the slow producer, let the lagging partition catch up, or address the offset reset) and the held-back windows close. ### 3. Your test passes but production doesn't (TopologyTestDriver) A `suppress` test that "works" while production stays silent usually means the test accidentally advances stream-time and production doesn't. With `TopologyTestDriver`, *you* control timestamps explicitly. If your test pipes a record with a later timestamp after the window's data, even incidentally, it pushes stream-time past the window end and the suppressed result pops out, leaving you convinced the topology is correct. To test `suppress` honestly, pipe one final record with a timestamp past `window-end + grace` to close the window, then assert on the output: ```java TestInputTopic in = driver.createInputTopic("events", new StringSerializer(), new StringSerializer()); // Records inside the 10:00–11:00 window in.pipeInput("k", "a", Instant.parse("2026-06-08T10:30:00Z")); in.pipeInput("k", "b", Instant.parse("2026-06-08T10:58:00Z")); // Nothing emitted yet, the window is still open. assertTrue(out.isEmpty()); // One record past window-end (11:00) + grace (5m) → stream-time crosses 11:05 → window closes in.pipeInput("k", "c", Instant.parse("2026-06-08T11:06:00Z")); assertFalse(out.isEmpty()); // NOW the suppressed result for 10:00–11:00 fires ``` One detail that surprises when correlating output: the emitted result carries the timestamp of the *last record that entered the window* (10:58 here), not the window end and not the 11:06 record that closed it. The lesson transfers straight to production: the thing that closes a window in a test is the same thing that closes it live, a later-timestamped record on that partition. `TopologyTestDriver` is also synchronous and applies no record cache, so it differs from production in other ways, a test that emits on every record won't reproduce production's cached, less-frequent emissions. ### 4. BufferConfig: unbounded vs bounded, and what happens when it fills `suppress` holds un-emitted results in an in-memory buffer until the window closes. That buffer is in heap, but it's backed by its own changelog topic (enabled by default, named `-KTABLE-SUPPRESS-STATE-STORE--changelog`, right next to the aggregation's `KSTREAM-AGGREGATE-STATE-STORE` changelog in a topic list), so a restart or rebalance rebuilds it from the changelog rather than dropping un-emitted results, at the cost of making the buffer part of [state restore](https://www.conduktor.io/kafka-streams/state-restore). A disk-spilling buffer has been proposed but never implemented, so a very large suppression buffer lives in memory. How you size that buffer changes the failure mode, but note up front: **a too-small buffer is a different problem from stream-time not advancing.** If your output is empty because the stream is idle (cases 1–3), no `BufferConfig` change will help; the window genuinely hasn't closed. `BufferConfig` matters when the *volume of open windows* is large, many keys, wide windows, long grace, so the buffer itself becomes the constraint: | BufferConfig | Behavior when full | Risk | |---|---|---| | `unbounded()` | Never "full", grows with open windows | Heap pressure / OOM if open-window count explodes | | `maxBytes(...)` / `maxRecords(...)` + `shutDownWhenFull()` | App **shuts down** when the cap is hit | Crash you'll see immediately, but it's a hard stop | | `maxBytes(...)` / `maxRecords(...)` + `emitEarlyWhenFull()` | Emits the *oldest* buffered records early to stay under the cap | **Breaks the "one final result per window" guarantee**, and only compiles with `untilTimeLimit`, not `untilWindowCloses` | ```java // "Strict" final-result semantics, bounded memory, fail loudly if it can't keep up: .suppress(Suppressed.untilWindowCloses( Suppressed.BufferConfig.maxBytes(64 * 1024 * 1024L).shutDownWhenFull())) ``` The trap here is reaching for `emitEarlyWhenFull()` to "fix" missing output. The type system blocks the worst version of this: `untilWindowCloses` accepts only a `StrictBufferConfig`, and `emitEarlyWhenFull()` returns an `EagerBufferConfig`, so the combination is a compile error. Using it means switching to `untilTimeLimit`, abandoning final-result semantics at the API level. It doesn't fix the idle-stream case at all, and where it *does* emit, it gives you partial windows, which is precisely what `untilWindowCloses` was supposed to prevent. Use `emitEarlyWhenFull()` only when you have explicitly accepted early emissions as a memory-safety valve, not as a remedy for silence. ## The dummy future-event workaround, and why it's a smell The folklore fix for the idle-partition case is to periodically inject a synthetic record with a *future* timestamp onto the input, on every partition, purely to drag stream-time forward and force open windows to close. It works, and the reason it works is now obvious: a record with a timestamp past `window-end + grace` is *exactly* what `suppress` is waiting for. Push stream-time forward artificially and the buffered results flush. But it's a smell, and worth naming as one: - The synthetic records are fake data flowing through your real topology. Your aggregation logic must recognize and skip them, or they pollute results. - You have to emit them on **every partition** (stream-time is per task), which means knowing the partition layout and producing accordingly. - The future timestamp can prematurely *expire* genuinely late records that would otherwise have landed within grace, you've moved the late-data cutoff forward by hand. - It's an out-of-band heartbeat bolted onto a data path that was never meant to carry one. If you find yourself building a heartbeat-injector, that's a signal that `suppress(untilWindowCloses)` doesn't fit a stream this sparse. The cleaner answer is below. ## When to use a wall-clock punctuator instead `suppress(untilWindowCloses)` ties emission to *data-driven* stream-time. When your stream is too sparse for that to be reliable, stop fighting it and drive emission from the *wall clock* instead, using the Processor API. A `WALL_CLOCK_TIME` punctuator fires on a fixed real-time schedule regardless of whether records are arriving, so it does not depend on stream-time advancing. You keep your windowed (or plain) aggregate in a state store, and on each punctuator tick you scan the store and emit any windows whose end-plus-grace is now in the past according to the wall clock. This trades a guarantee for liveness: you accept that "closed" is now decided by real time (and thus is approximate with respect to event-time) in exchange for results that actually come out on a quiet stream. ```java // In a Processor: emit finished windows on a real-time tick, independent of stream-time context.schedule(Duration.ofMinutes(1), PunctuationType.WALL_CLOCK_TIME, timestamp -> { // iterate the windowed store, forward + delete windows whose end+grace < now }); ``` This is a deliberate design choice, not a hack: it's the supported escape hatch for low-traffic windows. Note the contrast with `STREAM_TIME` punctuators, which fire on stream-time advancing and therefore suffer the *same* idle-partition silence as `suppress`. The full mechanics, `schedule`, `Punctuator`, store iteration, cancelling in `close()`, are in [the Processor API](https://www.conduktor.io/kafka-streams/processor-api). ## Frequently Asked Questions **Why is my Kafka Streams suppress() not emitting any records?** `suppress(untilWindowCloses)` releases a window only when stream-time crosses window-end plus grace, and stream-time advances only when a new record arrives with a later timestamp on the same partition. On an idle or low-traffic partition stream-time freezes, so the window never closes and the buffer never flushes, the behavior is correct, the stream is just quiet. **Why does suppress only emit when new records arrive?** Because it fires on stream-time, not your wall clock. Stream-time is the maximum record timestamp the task has seen, so a window covering 10:00–11:00 with 5m grace releases only when a record stamped ≥ 11:05 is actually processed, not when your server clock reaches 11:05. **Why does my suppress test pass but production emits nothing?** Your test almost certainly advances stream-time and production doesn't. With `TopologyTestDriver` you set timestamps explicitly, so piping a later-timestamped record after the window's data closes it. To test honestly, pipe one final record past `window-end + grace` and assert the output then appears. **Will a bigger BufferConfig fix my missing suppress output?** No, if the cause is an idle stream. `BufferConfig` only matters when the volume of open windows is large; it has no effect when the window simply hasn't closed because stream-time isn't advancing. And `emitEarlyWhenFull()` won't even compile with `untilWindowCloses`, it forces you onto `untilTimeLimit`, which gives partial windows, the exact thing final-result semantics were meant to prevent. **How do I emit window results on a low-traffic stream?** Stop using `suppress(untilWindowCloses)` and drive emission from the wall clock with a `WALL_CLOCK_TIME` punctuator in the Processor API. It fires on a fixed real-time schedule regardless of whether records arrive: keep the aggregate in a state store and on each tick scan it and emit windows whose end-plus-grace is now past. (A `STREAM_TIME` punctuator won't help, it suffers the same idle silence.) > **See it in practice with Conduktor** > When a `suppress` emits nothing, the first question is whether the window simply hasn't closed yet versus a real backlog. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-suppress-not-emitting) lets you watch per-partition consumer group lag on the input topic, to spot the one lagging partition holding a task back, and inspect the windowed aggregation's changelog topic to confirm state is being written even while the output stays quiet. ## Next steps - [Windowing](https://www.conduktor.io/kafka-streams/windowing), stream-time, grace, and why windows close when they do - [Aggregations](https://www.conduktor.io/kafka-streams/aggregations), the continuous-update model that `suppress` collapses into one result - [The Processor API](https://www.conduktor.io/kafka-streams/processor-api), wall-clock punctuators for emitting on sparse streams Source: https://www.conduktor.io/kafka-streams/suppress-not-emitting --- # Testing Kafka Streams applications *Learn how to test a Kafka Streams topology, and what your tests will lie to you about.* Kafka Streams is testable without a broker. `TopologyTestDriver` runs your real topology in-process, synchronously, in milliseconds, you pipe records in, read records out, no cluster, no rebalancing, no waiting. It is the right default for almost every Streams test. It also behaves *differently from production* in one specific, well-documented way, and if you don't know that, you'll write tests that pass on output prod will never produce. This page covers the fast path, the one trap that catches everyone, and how to turn `describe()` into a CI guardrail against the positional-name shifts that break stateful deploys. **What you'll learn:** - How `TopologyTestDriver` with `TestInputTopic`/`TestOutputTopic` works, and why it's synchronous - The caching gap that makes the test driver emit *more* updates than prod - How to test windowed and `suppress`-ed logic by advancing time deliberately - What the test driver can't catch, and when to reach for Testcontainers ## The fast path: TopologyTestDriver `TopologyTestDriver` takes a built `Topology` and the same `Properties` your app uses, then exposes typed input and output topics. The ergonomic `TestInputTopic`/`TestOutputTopic` API arrived in [KIP-470](https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements); use it rather than the older `pipeInput`/`readOutput` calls. ```java Topology topology = WordCountApp.buildTopology(); Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234"); // never contacted props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass()); props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass()); try (TopologyTestDriver driver = new TopologyTestDriver(topology, props)) { TestInputTopic in = driver.createInputTopic("in", new StringSerializer(), new StringSerializer()); TestOutputTopic out = driver.createOutputTopic("out", new StringDeserializer(), new LongDeserializer()); in.pipeInput("a", "hello world"); in.pipeInput("b", "hello kafka"); assertEquals(Map.of("hello", 2L, "world", 1L, "kafka", 1L), out.readKeyValuesToMap()); } ``` The `bootstrap.servers` value is never dialed, the driver simulates the broker, and since Kafka 2.8 ([KIP-680](https://cwiki.apache.org/confluence/display/KAFKA/KIP-680%3A+TopologyTestDriver+should+not+require+a+Properties+argument)) you can omit it and `application.id` entirely; the driver fills in dummy values, and passing your app's real `Properties` is just convenient. Everything runs on the calling thread: `pipeInput` processes the record through the entire topology *before it returns*, so there's nothing to await and no flakiness. For Avro, point the value serde at a [`MockSchemaRegistryClient`](https://docs.confluent.io/platform/current/schema-registry/index.html) via a `mock://` URL so `SpecificAvroSerde` resolves schemas in-memory instead of hitting a real registry, see [serde errors](https://www.conduktor.io/kafka-streams/serdes) for why a real Schema Registry wire format trips up tests. ## The caching gap: why tests see what prod won't Here is the difference that turns a green test suite into a false sense of safety. In production, a stateful operation does not emit a downstream record for every input. Kafka Streams has a **record cache** (sized by `cache.max.bytes.buffering`/`statestore.cache.max.bytes`) and a commit interval (`commit.interval.ms`). Updates to the same key are coalesced in the cache and only forwarded when the cache flushes or the commit fires. A key updated 100 times between flushes emits *one* downstream update with the latest value. `TopologyTestDriver` has **no record cache**, and it commits after every input record. So the same aggregation that emits one coalesced update in prod emits *one update per input record* under the test driver. > **Your test sees more intermediate updates than prod, and prod sees fewer than your test.** A windowed `count` fed three records for the same key emits three running values (`1`, `2`, `3`) in `TopologyTestDriver`, but in production the cache may collapse those into a single emitted `3`. If you assert on the exact stream of intermediate values, your test passes on behavior the cache hides in prod, and conversely, a prod consumer never sees the noisy interim records your test asserts on. Assert on *final, converged* values (read the whole output to a map; check the last value per key), not on every intermediate emission. This is not a bug in the test driver, it's the cost of being deterministic and synchronous. But it means a test is not a faithful simulation of emission timing. It tells you the topology *computes the right final answer*; it does not tell you *how many times* prod will emit along the way. That second question only has a real answer against a real broker. ## Testing windows and suppress: advance time on purpose The test driver gives you full control of record timestamps, and for windowed logic you need it. A window closes when *stream time*, the maximum record timestamp the task has seen, passes the window end plus the grace period. The driver never advances stream time on its own; it only moves when you pipe a record with a later timestamp. So a window stays open until you push it shut. This is exactly where `suppress(untilWindowCloses(...))` tests go wrong: the suppress buffer only emits the final result *after* the window closes, and the window only closes when a later-timestamped record arrives. Pipe three records into a window and read the output and you get nothing, not because suppress is broken, but because stream time never moved past the window. The fix in a test is to pipe one more record with a timestamp beyond the window + grace: ```java // topology windows with: .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1))) Instant t0 = Instant.parse("2026-01-01T00:00:00Z"); // three events inside a 1-minute tumbling window in.pipeInput("user-1", "a", t0); in.pipeInput("user-1", "b", t0.plusSeconds(10)); in.pipeInput("user-1", "c", t0.plusSeconds(20)); assertTrue(out.isEmpty(), "suppress holds the result until the window closes"); // advance stream time past window end (+grace) to flush the suppressed result in.pipeInput("user-1", "d", t0.plusSeconds(120)); KeyValue finalCount = out.readKeyValue(); assertEquals(3L, finalCount.value); // a, b, c, 'd' lands in the next window ``` The arithmetic only works because grace is zero: with `ofSizeAndGrace(Duration.ofMinutes(1), Duration.ofHours(24))` the record at `t0+120s` would not close the window, and `readKeyValue()` would fail on an empty topic. (The legacy `TimeWindows.of()` with its hidden 24-hour default grace is removed in Kafka 4.x, so modern code always states grace explicitly.) The same trick, explicitly stepping stream time forward, is how you test grace periods, session-window merges, and anything that fires on window close. If your suppress emits nothing in *production* (not just in a test), the cause is the same mechanism but a different trigger: a quiet partition that never advances stream time. That's its own deep dive, see [why suppress() emits nothing](https://www.conduktor.io/kafka-streams/suppress-not-emitting), and [windowing](https://www.conduktor.io/kafka-streams/windowing) covers stream time versus wall-clock in full. ## What the test driver won't catch `TopologyTestDriver` validates your topology's logic. It does not validate the things that depend on a real cluster: - **Missing source topics.** The driver creates input/output topics on demand. In production, reading from a topic that doesn't exist is a startup failure the driver never reproduces. - **Real serde and Schema Registry behavior.** A `mock://` registry isn't the real wire format, real subject-compatibility rules, or a registry that's momentarily down. The driver also won't surface a `ClassCastException` from a mismatched default serde the way a real run does under load. - **Rebalancing, restore, and standby replicas.** There's no consumer group, so [slow rebalances](https://www.conduktor.io/kafka-streams/rebalancing) and [state restore time](https://www.conduktor.io/kafka-streams/state-restore), the failures that actually page you, are invisible. The driver restores no changelog because there's no broker to hold one. - **The caching/commit timing** described above. For those, you need a real broker. [Testcontainers](https://java.testcontainers.org/modules/kafka/) spins up a throwaway Kafka in Docker for the test, so you can run the full app, produce to real topics, and assert on real consumed output, at the cost of seconds per test instead of milliseconds. Use it for a handful of integration tests (serde round-trips against a real registry, a real restart-and-restore) and keep the bulk of your coverage in the fast `TopologyTestDriver` tier. ## Snapshot the topology to guard deploys The most valuable test in a stateful Streams project isn't about output at all, it's a compatibility check. Internal store, changelog, and repartition topic names are *positional*: insert or reorder one operator and every downstream name shifts, orphaning the old state. (The full mechanism, and the naming discipline that prevents it, is in [topology evolution](https://www.conduktor.io/kafka-streams/topology-evolution).) You can catch that shift before it ships. `topology.describe()` renders the entire wiring, including the autogenerated names: ``` Sub-topology: 0 Source: KSTREAM-SOURCE-0000000000 (topics: [in]) Processor: KSTREAM-AGGREGATE-0000000002 (stores: [KSTREAM-AGGREGATE-STATE-STORE-0000000001]) Processor: KTABLE-TOSTREAM-0000000003 (stores: []) Sink: KSTREAM-SINK-0000000004 (topic: out) ``` (Trimmed for readability: the real output starts with a `Topologies:` header and adds `-->`/`<--` wiring lines after every node. Your committed snapshot will contain all of it.) Snapshot that string to a committed file and assert against it: ```java @Test void topologyMatchesSnapshot() throws Exception { String actual = MyApp.buildTopology().describe().toString(); String expected = Files.readString(Path.of("src/test/resources/topology.txt")); assertEquals(expected, actual); // fails the build if any internal name moved } ``` When a pull request shifts a `...0000000001` to `...0000000002`, this test fails with a diff, and a reviewer decides whether the change is a safe rolling deploy or needs the [application reset tool](https://www.conduktor.io/kafka-streams/topology-evolution). The decision moves into code review, where it's cheap, instead of into a restore failure at 3am, where it isn't. Kafka 4.3 adds a built-in, stricter version of this guardrail: set `ensure.explicit.internal.resource.naming=true` and the topology refuses to build at all if any internal store, changelog, or repartition topic is unnamed. ## Frequently Asked Questions **How do I unit test a Kafka Streams topology?** Use `TopologyTestDriver`, which runs your real topology in-process and synchronously with no broker. Feed records through a `TestInputTopic` and read results from a `TestOutputTopic`; `pipeInput` processes each record through the whole topology before it returns, so there's nothing to await and no flakiness. **What is TopologyTestDriver and how do I use it?** `TopologyTestDriver` takes a built `Topology` plus the same `Properties` your app uses and exposes typed input and output topics via `createInputTopic`/`createOutputTopic`. The `bootstrap.servers` value is never dialed, the driver simulates the broker, and since Kafka 2.8 you can omit it and `application.id` entirely, and the modern `TestInputTopic`/`TestOutputTopic` API from KIP-470 is preferred over the older `pipeInput`/`readOutput` calls. **Why does my Kafka Streams topology behave differently in TopologyTestDriver than in production?** The test driver has no record cache and commits after every input record, so a stateful operation emits one update per input, while production coalesces updates in the cache and emits fewer. Assert on *final, converged* values (read the whole output to a map, check the last value per key), not on every intermediate emission, or your test will pass on behavior the cache hides in prod. **How do I test windowed operations and suppress in Kafka Streams?** The driver only advances stream time when you pipe a record with a later timestamp, and a window closes when stream time passes the window end plus grace. So pipe your in-window events, then pipe one more record with a timestamp beyond window-end-plus-grace to push the window shut and flush a `suppress(untilWindowCloses(...))` result, otherwise the output is empty because stream time never moved. **When should I use Testcontainers instead of TopologyTestDriver?** Use Testcontainers (or EmbeddedKafka) for the things that need a real cluster: missing source topics, real Schema Registry wire format and compatibility, rebalancing, state restore, and the real caching/commit timing the driver can't reproduce. Keep those to a handful of integration tests at seconds each, and keep the bulk of coverage in the fast `TopologyTestDriver` tier. > **See it in practice with Conduktor** > Once a tested topology is deployed, [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-testing) shows the real changelog and repartition topics it created and the consumer group lag behind them, the production signals your test driver can't simulate. It's how you confirm that the behavior your tests proved on a final, converged result actually holds against a real broker, and that a restore finished after a deploy. Conduktor observes the Kafka your Streams app runs on; it doesn't replace your test suite. ## Next steps - [Topology evolution & the reset tool](https://www.conduktor.io/kafka-streams/topology-evolution), what the `describe()` snapshot protects you from - [Why suppress() emits nothing](https://www.conduktor.io/kafka-streams/suppress-not-emitting), the same time-advance trick, but in production - [Kafka Streams windowing](https://www.conduktor.io/kafka-streams/windowing), stream time vs wall-clock, in depth Source: https://www.conduktor.io/kafka-streams/testing --- # Evolving a Kafka Streams topology without losing state *Learn why a topology change breaks your app, and how to ship one safely.* You add one `filter` in the middle of a working Kafka Streams app, deploy it, and the app either refuses to start with a restore error or comes up with an aggregation that has silently reset to zero. Nothing in your diff touched state. The cause is almost never your code: it's that Kafka Streams names its internal topics by an operator's *position* in the topology, and you just shifted every position downstream of that `filter`. This is the single most under-documented way to break a stateful Streams app, and it's entirely preventable, but only if you do the right thing from the first commit, not after the incident. **What you'll learn:** - Why internal store, changelog, and repartition topic names are positional, and what shifts them - How to name every stateful operator so a refactor stops renaming your state - Which topology changes are safe rolling deploys, and which require a full reset - How to snapshot `Topology#describe()` and diff it in CI before a deploy ships ## Internal names are positional ![Two-panel comparison: Before inserting filter shows SOURCE-0000000000 to STATE-STORE-0000000001 to AGGREGATE-0000000002 to SINK-0000000003 with a populated changelog at -0000000001-changelog; After inserting filter shows SOURCE-0000000000 to filter-0000000001 to STATE-STORE-0000000002 to AGGREGATE-0000000003 to SINK-0000000004 with old changelog orphaned and new one starting from zero](https://www.conduktor.io/assets/kafka-streams/diagrams/topology-evolution-positional-names.svg) When you build a topology, Kafka Streams assigns every processor, store, and internal topic an autogenerated name. Call `topology.describe()` on a simple count-and-output app and you get this: real node names from the DSL, with the `Topologies:` header and the `-->`/`<--` edge lines trimmed for readability: ``` Sub-topology: 0 Source: KSTREAM-SOURCE-0000000000 (topics: [in]) Processor: KSTREAM-AGGREGATE-0000000002 (stores: [KSTREAM-AGGREGATE-STATE-STORE-0000000001]) Processor: KTABLE-TOSTREAM-0000000003 (stores: []) Sink: KSTREAM-SINK-0000000004 (topic: out) ``` Look at the numbers. `KSTREAM-SOURCE-0000000000`, then `-0000000001` for the store, `-0000000002` for the aggregate processor, and so on. They are assigned by a global counter as the builder walks your topology in declaration order. The store name `KSTREAM-AGGREGATE-STATE-STORE-0000000001` is what gets baked into the changelog topic name your cluster actually holds: ``` -KSTREAM-AGGREGATE-STATE-STORE-0000000001-changelog ``` The same positional scheme names repartition topics (`...-repartition`) and the internal stores behind joins. None of these names come from anything you wrote. They come from *where* in the graph the operator sits. Now insert one stateless operator upstream of the aggregate: ```java builder.stream("in") .filter((k, v) -> v != null) // <-- new operator, takes position 0000000001 .groupByKey() .count() // store is now 0000000002, was 0000000001 .toStream() .to("out"); ``` Every counter after the `filter` shifts by one. The aggregate's store becomes `KSTREAM-AGGREGATE-STATE-STORE-0000000002`, and its changelog becomes `...-STATE-STORE-0000000002-changelog`. On the next start, the app looks for a store named `...0000000002`, finds an empty changelog (the old `...0000000001-changelog` still holds all your data, now orphaned), and either restores nothing or fails consistency checks. Your running count is gone, sitting in a topic no operator references anymore. Reorder two branches and you get the same effect: positions move, names move, state is stranded. > 🚫 *"It's an internal refactor: I can rearrange my topology freely between deploys."* That assumption is exactly what strands state. A refactor that touches the *order or count* of operators rewrites the names Kafka Streams uses to find its own data. The DSL gives you no warning at compile time and, with stateless inserts, often none at startup either. The app just restarts its aggregations from scratch. It doesn't even reprocess old input to rebuild them: the consumer group's committed offsets survive the change (the group id is your unchanged `application.id`), so the new store only counts records that arrive after the redeploy. ## Name everything, from day one The fix is not to freeze your topology: it's to stop letting the position be the identity. Give every stateful piece an explicit, stable name, so its changelog/repartition/store name no longer depends on where it sits in the graph. Do this on the very first version, because retrofitting names onto an app that already has state in production is itself a breaking change (the names move from positional to explicit: see the reset section). | What | How to name it | What it stabilizes | |---|---|---| | Materialized store (count/aggregate/reduce) | `Materialized.as("orders-per-customer")` | The store name and its `-changelog` topic | | Repartition (after a re-key) | `Grouped.as("by-customer")` or `Repartitioned.as("by-customer")` | The `-repartition` topic name | | Stream-stream join | `StreamJoined.with(...).withName("o-p").withStoreName("o-p-store")` | The join's internal stores and changelogs | | Any operator (for `describe()` readability) | `Named.as("dedupe-step")` | The processor node name in the topology graph | Concretely: ```java builder.stream("in") .filter((k, v) -> v != null, Named.as("drop-nulls")) .groupByKey(Grouped.as("by-key")) .count(Materialized.as("event-count")) // changelog: -event-count-changelog .toStream() .to("out"); ``` Now the store is `event-count` and its changelog is `-event-count-changelog`, regardless of how many operators you add before or after it. You can insert a `filter`, add a branch, or reorder steps, and the count keeps finding its state. This is what [naming a state store](https://www.conduktor.io/kafka-streams/state-store) buys you beyond tidiness, and it's the foundation everything below depends on. Since Kafka 4.1 you can make this a hard rule: set `ensure.explicit.internal.resource.naming` to `true` ([KIP-1111](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1111%3A+Enforcing+Explicit+Naming+for+Kafka+Streams+Internal+Topics)) and building a topology with any auto-named internal topic or store throws a `TopologyException` listing the offenders. With the config off, recent versions still log a build-time WARN naming every unnamed changelog, so the signal is in your logs either way. The capability is old and stable: explicit operator naming landed in [KIP-307](https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL), and the [DSL topology naming guide](https://kafka.apache.org/documentation/streams/developer-guide/dsl-topology-naming.html) in the Apache docs is the canonical reference. Bill Bejeck's [Optimizing Kafka Streams Applications](https://www.confluent.io/blog/optimizing-apache-kafka-streams-applications/) walks through how naming interacts with topology optimization. Treat all three as required reading before your first stateful deploy. > **Naming alone doesn't let you move a store between operators.** A stable name keeps the changelog name fixed when *positions* shift around it. It does not migrate data if you change the operator's semantics: switching `count()` to `aggregate()`, or changing a join's window, produces state that is incompatible regardless of the name. Naming protects against accidental renames, not against genuinely different computations. ## Optimization can reshape the topology too Naming protects you from accidental position shifts in *your* code. One more thing can restructure the topology underneath you: the `topology.optimization` config. It defaults to `NO_OPTIMIZATION`, but when enabled it rewrites the logical plan before it becomes a physical topology: for example reusing a source topic directly as a `KTable`'s changelog (skipping a separate changelog topic), merging repartition topics that feed multiple downstream operators, and collapsing a stream-stream self-join onto a single store. Accepted values are `none`, `all`, or a comma-separated list of granular flags: `reuse.ktable.source.topics`, `merge.repartition.topics`, and `single.store.self.join`. The catch for *this* page: turning optimization on or off (or upgrading to a version that adds a new optimization) can change which internal topics exist and how they're wired, which is its own kind of topology change. Pin the specific flags you want in production rather than `all`, so a library upgrade doesn't silently restructure your internal topics out from under your running state. As always, verify the new plan with `topology.describe()` (see below) before you deploy. ## Which changes are safe, and which need a reset Not every topology change is dangerous. Group the changes you make into three buckets. **Safe as a rolling deploy** (names stable, state semantics unchanged): - Adding, removing, or editing a *stateless* operator (`map`, `filter`, `mapValues`, `peek`) when stores and repartitions are explicitly named. - Changing the logic *inside* a stateless lambda. - Adding a brand-new, independently named stateful sub-topology that reads existing topics: the old state is untouched; the new store restores from its own (initially empty) changelog. **Needs care (same name, different computation):** - Changing what a named stateful operator computes (count → sum, new window size, new join type). The changelog name is stable but its *contents* no longer mean what the new code expects. You usually need a new store name and a controlled rebuild. **Requires the reset tool** (the operator identity or input semantics changed and old state must be discarded): - You shipped without explicit names and now need to add them: the positional names are abandoned, so the old changelogs are orphaned and you must start clean. - You changed the meaning of the computation enough that replaying from the existing changelog would be wrong. - You want to reprocess input topics from the beginning under a new topology. For that last bucket, Kafka ships [`kafka-streams-application-reset`](https://kafka.apache.org/documentation/streams/developer-guide/app-reset-tool.html). It does three different things to three kinds of topic: ```bash kafka-streams-application-reset \ --application-id orders-aggregator \ --input-topics in \ --bootstrap-server broker:9092 ``` | Topic kind | What the tool does | |---|---| | Input topics (you pass `--input-topics`) | Resets the app's consumer offsets to **earliest** so it reprocesses from the start | | Intermediate topics (user-managed topics that are both an output and an input of the app, the old `through()` pattern, removed in Kafka 4.0; pass them via `--intermediate-topics`, deprecated in 4.x) | Skips to the end (seeks to the current offset) so old in-flight data isn't reprocessed | | Internal topics (changelog, repartition, including topics created by `.repartition()`) | **Deletes** them so they're recreated cleanly on next start | Two operational rules that the tool will not protect you from: 1. **Stop every instance first.** The tool resets offsets and deletes internal topics; if any instance is still running and committing, you get a corrupted, half-reset state. Scale the deployment to zero, run the tool, then scale back up. 2. **Triple-check the `--application-id`.** The application ID is the prefix for *all* of an app's internal topics and its consumer group. A typo that happens to match another app's ID will delete *that* app's changelogs and reset *its* offsets. There is no undo. This is the same footgun that makes a careless `application.id` dangerous when [building your first app](https://www.conduktor.io/kafka-streams/getting-started). After a reset, also wipe the local state directory (`state.dir`) on every instance, or each one will try to reuse stale local RocksDB data that no longer matches the deleted changelogs. ## Catch the break in CI, not in production You don't want to discover a positional shift from a 3am restore failure. The topology is fully describable before it ever runs: `topology.describe()` returns a `TopologyDescription` whose `toString()` is the untrimmed version of the block at the top of this page, `Topologies:` header and edge lines included. Snapshot it and diff it. ```java @Test void topologyMatchesSnapshot() { Topology topology = OrdersApp.buildTopology(); String actual = topology.describe().toString(); String expected = Files.readString(Path.of("src/test/resources/topology.txt")); assertEquals(expected, actual); // fails if any internal name or wiring changed } ``` Commit `topology.txt` alongside the code. When a pull request changes the topology, this test fails and prints a diff of the old vs new description, including any shifted `...0000000002` names. A reviewer then makes a deliberate call: is this a safe rolling change, or does it need the reset tool and a maintenance window? The point is that the decision happens in review, with the diff in front of you, instead of after the deploy when the state is already orphaned. This snapshot lives naturally inside your [Kafka Streams test suite](https://www.conduktor.io/kafka-streams/testing), which already runs the topology through `TopologyTestDriver`. The same discipline pays off when you scale. Repartition topic names and partition counts are part of the topology, so a change that alters them interacts with how tasks are assigned across instances. See [scaling and parallelism](https://www.conduktor.io/kafka-streams/scaling). ## Frequently Asked Questions **How do I change a Kafka Streams topology without losing state?** Give every stateful operator an explicit, stable name with `Materialized.as(...)`, `Grouped.as(...)`, or `StreamJoined.withName(...)` so its changelog and repartition topic names stop depending on the operator's position. With names fixed, you can insert, remove, or reorder operators around it and the store keeps finding its state. **Why does my Kafka Streams app fail to start or reset state after a code change?** Kafka Streams names internal stores, changelogs, and repartition topics by an operator's *position* in the topology, assigned by a global counter in declaration order. Insert or reorder one operator and every downstream name shifts, so the app looks for a renamed changelog, finds it empty, and either restores nothing or fails consistency checks: your old data is orphaned in a topic nothing references. **Which Kafka Streams topology changes are safe versus need a reset?** Safe rolling deploys: adding or editing a *stateless* operator, changing logic inside a stateless lambda, or adding a new independently named stateful sub-topology. A reset is required when you add explicit names to an app that shipped without them, change what a named operator computes, or want to reprocess input topics from the beginning. **When do I need the kafka-streams-application-reset tool?** Use it when an operator's identity or input semantics changed and the old state must be discarded: for example after retrofitting names onto a positional app, or to reprocess inputs from the start. It resets input-topic offsets to earliest, skips intermediate topics to the end, and deletes internal changelog/repartition topics; stop every instance first and triple-check the `--application-id`, as there is no undo. **How do I detect an incompatible topology change before deploying?** Snapshot `topology.describe().toString()` to a committed file and assert against it in a test, so any shifted internal name fails the build with a diff. A reviewer then decides in code review whether the change is a safe rolling deploy or needs the reset tool, instead of discovering it from a restore failure in production. > **See it in practice with Conduktor** > When a topology change orphans state, the evidence is in the topics. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-topology-evolution) shows the changelog and repartition topics your app created, so you can spot a stale `...-STATE-STORE-0000000001-changelog` that nothing reads anymore, confirm the new changelog is filling, and watch the consumer group lag that tells you whether a restore is still running after a deploy. Conduktor doesn't run your Streams app; it gives you the view of the internal topics and offsets the reset tool acts on. ## Next steps - [Test your Kafka Streams topology](https://www.conduktor.io/kafka-streams/testing): snapshot `describe()` and diff it in CI - [Kafka Streams state stores](https://www.conduktor.io/kafka-streams/state-store): why naming a store keeps its changelog stable - [Build your first Kafka Streams app](https://www.conduktor.io/kafka-streams/getting-started): where `application.id` becomes your topic prefix and group ID Source: https://www.conduktor.io/kafka-streams/topology-evolution --- # Kafka Streams vs the consumer API *Decide whether you need Kafka Streams or just a consumer.* Almost everyone starting with stream processing hits this fork: *do I reach for Kafka Streams, or is a plain [consumer](https://www.conduktor.io/kafka/kafka-consumers) enough?* The framing that trips people up is treating them as rivals. They aren't. Kafka Streams is built directly on top of the consumer and producer clients: it's not a different way to talk to Kafka, it's a higher-level processing layer sitting on the same transport. So the real question isn't "which transport is better." It's "how much of the processing machinery do I want to write myself?" This guide answers that honestly, with the same logic side by side so you can see exactly what the library buys you. **What you'll learn:** - Why Kafka Streams is built *on* the consumer, not instead of it - When a plain consumer is the right, simpler choice - When Kafka Streams earns its operational cost - The same job written both ways, line for line ## Kafka Streams sits on top of the consumer Open up a running Kafka Streams app and look at what it's doing on the wire: it polls partitions with a [consumer](https://www.conduktor.io/kafka/kafka-consumers), it commits offsets, it produces results with a producer, and its instances form a [consumer group](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets) that splits partitions between them. That's the consumer API: Streams didn't replace it, it wrapped it. What the library adds is everything *above* the poll loop: a topology you describe declaratively, local state that survives restarts, event-time windowing, and atomic read-process-write. You get those by configuring a topology instead of hand-coding them against `poll()`. > 🚫 *"We'll move off Kafka Streams to plain consumers to simplify, it's just a consumer anyway."* Dropping Streams for a raw consumer doesn't remove complexity from a stateful job: it relocates it into your code. The state store, the windowing, the exactly-once cycle, the rebalance handling: they don't disappear, you now own and maintain them by hand. Simplify by removing Streams only when the job is genuinely stateless. > **This is why "Kafka Streams vs consumer" is the wrong axis.** You are not choosing between two transports: both read and write Kafka the same way. You're choosing whether to write the processing layer yourself or let the library write it for you. Frame the decision as *plain consumer plus your own code* versus *Kafka Streams*, and the trade-off gets clear fast. ## When the plain consumer is enough Reach for a bare [consumer](https://www.conduktor.io/kafka/kafka-consumers) (and producer, if you emit results) when your work is simple and stateless. Specifically: - **Per-record consume-then-act, no memory between records.** Read a message, do one thing with it, move on. No counts, no running totals, nothing remembered across records. - **The "act" is not stream processing.** Write the record to a database, call an HTTP service, push to a queue, send an email. The interesting work happens *outside* Kafka, and you just need records delivered to your code. - **No joins, aggregations, or windows.** The moment you need to combine two streams or group by time, you're rebuilding the library. - **You want full control of the poll loop.** Custom commit timing, manual partition assignment, your own threading model, careful backpressure: the consumer hands you the raw controls; Streams hides them on purpose. A consumer that reads an `orders` topic and inserts each order into Postgres is a textbook plain-consumer job. Adding Kafka Streams there buys you nothing and costs you the Streams runtime and its [rebalance](https://www.conduktor.io/kafka-streams/rebalancing) behavior. A stateless topology creates no RocksDB store and no internal topics, but the moment anyone adds a stateful step, both appear and you operate them. ## When Kafka Streams pays off Reach for [Kafka Streams](https://www.conduktor.io/kafka-streams) when you need the things you'd otherwise reinvent (usually badly) on top of a consumer: - **Stateful operations.** Counts, sums, rollups, and joins backed by a local [state store](https://www.conduktor.io/kafka-streams/state-store) that survives a crash. Hand-rolling fault-tolerant local state on a raw consumer is a project in itself. - **Joins across streams.** Enriching an event stream against a reference table, or correlating two streams within a time window. (More on doing this without Streams below. Short version: you don't want to.) - **Event-time windowing.** Grouping by when events *happened*, with late-data handling, instead of when they arrived. - **Exactly-once across read-process-write.** One config flag makes the consume → transform → produce → commit cycle atomic ([exactly-once](https://www.conduktor.io/kafka-streams/exactly-once)). Replicating this by hand against the transactional producer API is error-prone. - **Scaling and fault tolerance you don't write.** Partitions, tasks, and state migrate between instances on their own. With a raw consumer you'd write the rebalance and state-handoff logic yourself. The pattern: if your processing is stateful, time-aware, or needs strong delivery guarantees, the consumer-plus-DIY route slowly converges on a worse version of Kafka Streams. Use the real one. ## The same job, both ways Here's a stateless filter (pass through only paid orders) written as a raw consumer and as a Kafka Streams topology. Both run on Apache Kafka 3.9.0. Plain consumer and producer: ```java try (var consumer = new KafkaConsumer(consumerProps); var producer = new KafkaProducer(producerProps)) { consumer.subscribe(List.of("orders")); while (true) { var records = consumer.poll(Duration.ofMillis(100)); for (var record : records) { if (record.value().contains("\"status\":\"PAID\"")) { producer.send(new ProducerRecord<>("paid-orders", record.key(), record.value())); } } producer.flush(); consumer.commitSync(); } } ``` Note the `producer.flush()` before the commit. `send()` is asynchronous, so committing offsets first means a crash can silently drop PAID records. That ordering bug is exactly the kind of detail you own on the raw clients. Kafka Streams: ```java StreamsBuilder builder = new StreamsBuilder(); builder.stream("orders") .filter((key, order) -> order.contains("\"status\":\"PAID\"")) .to("paid-orders"); KafkaStreams streams = new KafkaStreams(builder.build(), props); streams.start(); ``` For *this* job (one stateless filter) they're roughly even, and the consumer is arguably more transparent. That's the point: a one-line stateless transform is not a reason to adopt a framework. Now imagine the next requirement lands: *count paid orders per customer per hour, exactly once, and survive a restart without losing the counts.* On the consumer you start writing a state store, a flush-and-commit protocol, windowing, transactional sends, and rebalance-aware state handoff. On Streams it's a `groupByKey().windowedBy(...).count()` and one `processing.guarantee` flag. **That** is where the library stops being optional. ## Side by side | Dimension | Plain consumer (+ producer) | Kafka Streams | |---|---|---| | **What it is** | Low-level client: poll, process, commit | Processing library built *on* the consumer/producer | | **Transport to Kafka** | Consumer group, offsets | The same consumer group and offsets, underneath | | **State / aggregations** | You build and persist it yourself | Built-in [state stores](https://www.conduktor.io/kafka-streams/state-store) with changelog backup | | **Joins & windows** | Hand-rolled (hard, error-prone) | First-class [joins](https://www.conduktor.io/kafka-streams/joins) and [windowing](https://www.conduktor.io/kafka-streams/windowing) | | **Exactly-once read-process-write** | DIY with the transactional producer | One [config flag](https://www.conduktor.io/kafka-streams/exactly-once) | | **Scaling / failover** | You write rebalance & state handoff | Tasks and state migrate automatically | | **Control of the poll loop** | Full: commit timing, threading, assignment | Abstracted away on purpose | | **Operational surface** | Just a consumer group | + RocksDB, internal topics, [restore](https://www.conduktor.io/kafka-streams/state-restore) once stateful | | **Best for** | Stateless per-record work; act *outside* Kafka | Stateful, time-aware, exactly-once processing | Read the last two rows together: Streams adds power *and* operational surface. The moment your topology is stateful you take on RocksDB memory, auto-created changelog and repartition topics (named `--changelog` and `-repartition`), and rebalance/restore behavior. Pay that price for state, joins, and exactly-once, not for a filter, which creates none of it. ## Frequently Asked Questions **Is Kafka Streams just a consumer?** Under the hood, mostly yes: a Kafka Streams app uses the consumer and producer clients and forms a consumer group like any other. What it adds sits above the poll loop: a declarative topology, fault-tolerant local state, windowing, and exactly-once. So it's the consumer plus a processing layer, not a separate transport. **When do I need Kafka Streams over a consumer?** When your processing is stateful (counts, aggregations, joins), time-aware (windowing on event time), or needs exactly-once across the read-process-write cycle. Those are the parts you'd otherwise build by hand on top of a consumer. For simple, stateless, per-record work, a plain consumer is the lighter choice. **Does Kafka Streams replace the consumer API?** No. It's built on the consumer and producer APIs and uses them internally; it doesn't deprecate or replace them. Plenty of production systems keep using plain consumers for stateless work and reach for Kafka Streams only where state, joins, or exactly-once are needed. **Can I do joins with a plain consumer?** Technically yes, but you'd have to maintain the lookup data in your own state store, keep it co-partitioned with the stream, handle restarts and rebalances, and deal with timing, which is exactly what Kafka Streams joins do for you. For anything beyond a trivial in-memory lookup, a plain-consumer join is a maintenance trap. **Is a Kafka Streams app harder to operate than a consumer?** Yes, and that's the trade-off. A consumer is just a consumer group. A stateful Streams app adds a RocksDB state store, auto-created changelog and repartition topics, and rebalance/restore behavior you have to watch. A stateless one stays close to a plain consumer group, which is exactly why a plain consumer is enough there. > **See it in practice with Conduktor** > Both a plain consumer and a Kafka Streams app show up on the cluster the same way: as a consumer group with lag and committed offsets. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-vs-consumer-api) lets you watch that consumer group's lag, inspect committed offsets, and, for a stateful Streams app, see the extra changelog and repartition topics it creates. That side-by-side view is the quickest way to confirm what your processing is actually doing on Kafka, whichever approach you chose. ## Next steps - [What is Kafka Streams?](https://www.conduktor.io/kafka-streams): the library model, in depth - [Build your first Kafka Streams app](https://www.conduktor.io/kafka-streams/getting-started): a runnable WordCount in Java - [Kafka Streams state stores](https://www.conduktor.io/kafka-streams/state-store): the local state you'd otherwise hand-roll Source: https://www.conduktor.io/kafka-streams/vs-consumer-api --- # Kafka Streams vs Flink vs ksqlDB *Pick the right stream processor, by deployment model, not feature checklist.* Most "Kafka Streams vs Flink" comparisons hand you a feature grid and call it a decision. That grid is mostly noise: for the large majority of stream-processing problems, *either* tool does the job. The honest differentiator is the one most articles skip, the **deployment model** and the **team** that has to operate it. Get those two right and the feature list rarely matters. We sell neither Kafka Streams nor Flink, so there's no thumb on the scale here. This is the senior-architect version: the questions that actually decide it, and the ones that only look like they do. **What you'll learn:** - Why deployment model and team profile decide this, not features - A side-by-side of Kafka Streams, Flink, and ksqlDB - Straight answers to "is Kafka Streams dying?" and "is ksqlDB dead?" - A decision guide: when each one is the right call ## The question that actually decides it Kafka Streams is a **library**. You add `org.apache.kafka:kafka-streams` to your app, write a topology, and your service *becomes* the stream processor. No cluster to stand up, no job to submit. It [scales like any consumer group](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets), and its state lives in local stores backed by [changelog topics](https://www.conduktor.io/kafka-streams/state-store). The application team owns all of it, memory, restarts, rebalances. Flink is a **runtime**. You stand up a cluster (a JobManager and TaskManagers), submit a job to it, and that cluster schedules the work, checkpoints state to durable storage, and recovers on failure. It reads from and writes to anything, Kafka, a database, files, object storage. It scales well past your Kafka partition count. And it usually comes with an operator: a dedicated streaming or platform team that runs the cluster as shared infrastructure. That single split, *code you ship* versus *infrastructure you operate*, predicts almost everything else. It tells you who gets paged at 3am, how you scale, how you recover, and which team's roadmap the workload lives on. Start there. The feature comparison is a tiebreaker, not the decision. > **"For 80 to 90 percent of stream-processing use cases, either Kafka Streams or Flink will work."** This is a recurring line from Kafka maintainers, and it holds up in practice. Stateless transforms, windowed aggregations, enrichment joins, event-time handling, both engines do all of it. When both can do the job, you choose on operations and team, not capability. ## Side by side Read this as a map of *where each one is at home*, not a scoreboard. The last row is the one that matters. | Dimension | Kafka Streams | Apache Flink | ksqlDB | |---|---|---|---| | **Deployment model** | Library embedded in your app | Separate cluster (JobManager + TaskManagers) | A server you run (built on Kafka Streams) | | **What you deploy** | A JAR, a normal JVM service | A job submitted to a cluster | SQL statements to the ksqlDB server | | **Languages** | Java (JVM) | Java, Python, SQL (the dedicated Scala API was removed in Flink 2.0) | SQL only | | **Sources / sinks** | Kafka only, in and out | Any, Kafka, DBs, files, object stores | Kafka only | | **Scaling ceiling** | ≤ partition count of the input | Far beyond partition count (task parallelism) | ≤ partition count (it *is* Kafka Streams) | | **State recovery** | Replay the [changelog topic](https://www.conduktor.io/kafka-streams/state-store) | Restore from a [checkpoint](https://flink.apache.org/) in durable storage | Changelog replay (same as Streams) | | **Operational owner** | The application team | A dedicated streaming / platform team | Whoever runs the ksqlDB server | | **Batch + stream** | Streaming only | Unified batch and stream | Streaming only | | **When it wins** | In-app, stateful logic owned by app devs | Heterogeneous sources, huge scale, a platform team, SQL/Python | Fast SQL transforms when you've committed to the ksqlDB server | A few rows deserve a caveat the grid can't carry. **Scaling ceiling.** Kafka Streams parallelism tops out at the partition count of the busiest sub-topology, add instances past that and the extras sit idle (or host standby replicas, but only if you set `num.standby.replicas`; the default is 0). Flink decouples parallelism from partitions, so it scales past that ceiling. In practice most workloads never approach the limit, so this only decides the genuinely large jobs. See [scaling Kafka Streams](https://www.conduktor.io/kafka-streams/scaling). **State recovery.** Both engines are converging here (more below), but today the models differ in *who* holds the state and how a restart behaves. Streams replays a changelog into a local store; Flink restores from a checkpoint it wrote to S3/HDFS/GCS. The Streams model puts restore time on the app team's plate, the thing that turns a [rolling restart into a slow one](https://www.conduktor.io/kafka-streams/state-restore). **Event time.** A common myth is that Flink "has watermarks and Kafka Streams doesn't," implying weaker event-time support. Both handle event time fully. Kafka Streams deliberately uses *stream time* plus continuous refinement instead of watermark-driven triggers, a different design, not a missing feature. ## What about ksqlDB? ksqlDB is SQL layered on top of Kafka Streams. You write `SELECT … EMIT CHANGES`, the server compiles it to a Streams topology and runs it. So everything true of Kafka Streams' runtime, Kafka-only I/O, changelog-backed state, partition-bound scaling, is true of ksqlDB too. It trades the Java DSL for SQL and a server you operate. Be straight about its status: ksqlDB is under **lower active investment** than it once was, frequently described as effectively maintenance-mode, while Confluent steers new SQL stream-processing work toward Flink SQL. That's a real signal if you're starting fresh today. It is *not* a reason to rip out a working ksqlDB deployment, it still runs, and the Kafka Streams engine underneath it is very much alive. New SQL-first project: look hard at Flink SQL before committing to ksqlDB. Existing ksqlDB that works: leave it alone until you have a concrete reason to move. ## "Is Kafka Streams dying? Is ksqlDB dead?" The loaded questions, answered plainly. **Kafka Streams is not dying.** It's a core part of Apache Kafka and the default for in-app, stateful processing, fraud scoring, usage metering, materialized views, [event-driven microservices](https://www.conduktor.io/kafka-streams). Active KIPs are still landing (a [Streams-specific rebalance protocol](https://www.conduktor.io/kafka-streams/rebalancing), native dead-letter-queue and error-handling support). The confusion comes from conflating Kafka Streams with ksqlDB. They are not the same project, and Flink's rise does not retire the library, they serve different shapes of problem. **ksqlDB is not dead, but it is quiet.** See the section above: lower investment, the SQL momentum has moved to Flink. Treat it as stable-but-not-growing. **Flink and Kafka Streams coexist.** This is the part the "X killed Y" headlines miss. Plenty of organizations run both: Flink for cross-source pipelines and big shared jobs operated by a platform team, Kafka Streams for stateful logic that belongs *inside* a service the app team owns. They're complements far more often than competitors. > 🚫 *"Flink is the modern one, so we should migrate our Kafka Streams apps to it."* Migrating a working Kafka Streams microservice to a Flink cluster you don't yet operate trades a problem you've solved for one you haven't. Newer is not the axis. Deployment model and team are. ## They're converging anyway The sharpest practical difference today, *how state is stored and recovered*, is the one the industry is actively erasing. Both ecosystems are moving state off local disk and onto **object storage**: remote, disaggregated state stores so that restores become near-instant and instances become effectively stateless. On the Kafka Streams side the state-store interface is pluggable, and the remote-state work is being built on top of it by the ecosystem (vendors like Responsive ship object-storage-backed stores) rather than inside Apache Kafka itself. On the Flink side, Flink 2.0 ships disaggregated state through the ForSt backend, which keeps state on object storage but is still marked experimental. If that convergence lands the way it's trending, "changelog replay vs checkpoint restore" stops being a deciding factor, which pushes even more weight onto deployment model and team. We track the direction in [the future of Kafka Streams](https://www.conduktor.io/kafka-streams/future). ## A decision guide Choose the tool that fits your *deployment and team*, then confirm it covers your features (it almost always will). **Choose Kafka Streams when:** - The logic belongs inside an application your team already owns and deploys. - Your sources and sinks are Kafka, and you're a JVM shop. - You want stream processing to scale and deploy like any other microservice, no extra cluster. - The state is manageable, or you're prepared to own [restore time](https://www.conduktor.io/kafka-streams/state-restore) and [RocksDB memory](https://www.conduktor.io/kafka-streams/state-store). **Choose Flink when:** - You're pulling from non-Kafka sources, databases, files, object storage, multiple systems in one job. - You need to scale past your Kafka partition count, or run large shared jobs as infrastructure. - You have (or want) a dedicated streaming/platform team to operate a cluster. - Your authors live in SQL or Python, not just the JVM. - You want unified batch and streaming in one engine. **Choose ksqlDB when:** - You want SQL-defined streaming transforms *and* you've already committed to running the ksqlDB server. - For a new SQL-first project, weigh Flink SQL first, given where the investment is going. When two of these fit, default to the one whose operational model matches your team. A library workload forced onto a cluster (or a cross-source pipeline crammed into a library) fights you forever. ## Frequently Asked Questions **What is the difference between Kafka Streams and Apache Flink?** Kafka Streams is a library you embed in your application, so your service becomes the stream processor with no cluster to operate, and it reads and writes Kafka only. Flink is a runtime: you stand up a cluster, submit a job, and it reads from any source, scales past your partition count, and is usually run by a dedicated platform team. **Is Kafka Streams dying?** No. It is a core part of Apache Kafka and the default for in-app stateful processing, with active KIPs still landing such as a Streams-specific rebalance protocol and native error handling. The "dying" confusion usually comes from conflating Kafka Streams with ksqlDB, which are different projects. **When should I choose Kafka Streams over Flink?** Choose Kafka Streams when the logic belongs inside an application your team already owns, your sources and sinks are Kafka, you're a JVM shop, and you want stream processing to deploy and scale like any other microservice. Choose Flink for non-Kafka sources, scale beyond your partition count, SQL/Python authors, or a dedicated platform team. **Does Kafka Streams scale as large as Flink?** Kafka Streams parallelism tops out at the partition count of the busiest sub-topology, extra instances past that sit idle unless you configure standby replicas (num.standby.replicas defaults to 0). Flink decouples parallelism from partitions and scales further, but most workloads never approach the limit, so this only decides genuinely large jobs. **Is ksqlDB dead, and how does it compare?** ksqlDB is SQL layered on top of Kafka Streams, so it inherits Kafka-only I/O and partition-bound scaling. It is not dead but is under lower active investment and effectively maintenance-mode, with new SQL work steered toward Flink SQL, leave a working deployment alone, but weigh Flink SQL first for a new SQL-first project. > **See it in practice with Conduktor** > Whichever engine you pick, it runs on Kafka, and both Kafka Streams and Flink workloads show up there as consumer groups, lag, and topics. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-vs-flink) lets you watch consumer group lag, inspect partition assignment, and see the internal and changelog topics a job creates, so you can tell whether a Streams app or a Flink job is keeping up, independent of which framework owns the processing. One change on the horizon: with KIP-1071 (early access in Kafka 4.1), Streams apps move to a dedicated `streams` group type with its own `kafka-streams-groups.sh` tooling, instead of appearing as plain consumer groups. ## Next steps - [What is Kafka Streams?](https://www.conduktor.io/kafka-streams), the library model, in depth - [Kafka Streams state stores](https://www.conduktor.io/kafka-streams/state-store), the recovery model that drives the comparison - [The future of Kafka Streams](https://www.conduktor.io/kafka-streams/future), remote state and where both engines are headed Source: https://www.conduktor.io/kafka-streams/vs-flink --- # Kafka Streams vs ksqlDB *Understand the real difference, it's not what "vs" suggests.* The framing "Kafka Streams vs ksqlDB" is misleading before you read a single comparison, because at the engine level they aren't rivals. ksqlDB is built **on top of** Kafka Streams. Every ksqlDB query compiles down to a Kafka Streams topology and runs on the Streams runtime. So the honest question isn't "which engine is faster or more capable", it's "do I want to write SQL against a server someone operates, or embed a Java library in an app my team owns?" We sell neither, so there's no angle here. This is the senior-architect version: what the two things actually are, the status reality of ksqlDB that the reputational queries are really asking about, and when each is still the right call. **What you'll learn:** - Why ksqlDB *is* Kafka Streams, with SQL and a server bolted on - The real axis: SQL-on-a-server vs library-in-your-app - The honest status of ksqlDB, and where Confluent is steering SQL work - When to pick ksqlDB, when Kafka Streams, when Flink SQL ## ksqlDB is built on Kafka Streams Start with the fact that dissolves the "vs". You write a statement like this against a ksqlDB server: ```sql CREATE TABLE orders_per_customer AS SELECT customer_id, COUNT(*) AS order_count FROM orders GROUP BY customer_id EMIT CHANGES; ``` ksqlDB parses that, plans it, and **compiles it into a Kafka Streams topology**, the same `groupByKey().count()` shape you'd write by hand in the [Kafka Streams DSL](https://www.conduktor.io/kafka-streams), then runs it on the embedded Streams runtime inside the ksqlDB server. The result table is materialized in a [state store](https://www.conduktor.io/kafka-streams/state-store) backed by a compacted changelog topic named `<application.id>-<store-name>-changelog`, exactly as a hand-written Streams app would do it. That inheritance is the whole story. Everything true of the Kafka Streams runtime is true of ksqlDB underneath: - **Kafka-only I/O.** The compiled queries read and write Kafka topics only. Not databases, not files. ksqlDB the product can also manage Kafka Connect connectors (it can even run an embedded Connect worker inside the server) to bridge databases, but that data still transits through Kafka topics, unlike Flink, which reads external systems directly. - **Changelog-backed state.** The same local store plus [changelog](https://www.conduktor.io/kafka-streams/state-store) recovery model, and the same [restore time](https://www.conduktor.io/kafka-streams/state-restore) when a server restarts with a large table. - **Partition-bound scaling.** A ksqlDB query [scales like a consumer group](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets), capped at the partition count of its input, [exactly as Kafka Streams does](https://www.conduktor.io/kafka-streams/scaling). - **The same rebalance behavior.** Add or remove a ksqlDB server and the underlying Streams [rebalance](https://www.conduktor.io/kafka-streams/rebalancing) reassigns work. > **ksqlDB is not a separate stream-processing engine.** It is a SQL interface and a server runtime sitting on the Kafka Streams library. When you tune a ksqlDB query's performance, you are tuning Kafka Streams, RocksDB memory, changelog topics, partition count, the lot. The skills transfer directly. ## The real axis: SQL on a server vs a library in your app If the engine is the same, what actually differs? Two things, and they're the whole decision. **ksqlDB is SQL plus a server you operate.** You define streams and tables in SQL, submit them to a ksqlDB server cluster, and that cluster runs the compiled topologies as long-lived persistent queries. You operate the ksqlDB servers as standalone infrastructure, separate from the apps that produce and consume the data. The people writing the logic can be analysts or data engineers who know SQL and never touch the JVM. **Kafka Streams is a library you embed.** You add the `org.apache.kafka:kafka-streams` JAR to an application your team already builds and deploys, write a topology in Java, and your service *becomes* the processor. No extra cluster. The logic lives inside the app, owned by the app team, deployed on their pipeline. That split decides who writes the logic, who operates it, and how much custom behavior you can reach for. SQL is faster to write and read for the shapes it covers, filters, joins, windowed aggregations. The Java library gives you arbitrary code: call into a library, branch on complex logic, use the [Processor API](https://www.conduktor.io/kafka-streams/processor-api) and punctuators, implement a custom [deduplication](https://www.conduktor.io/kafka-streams/deduplication) processor or a [dead-letter-queue](https://www.conduktor.io/kafka-streams/dead-letter-queue) strategy that SQL can't express. ## Side by side Read this as a map of where each is at home. The last two rows are the ones that usually decide it. | Dimension | ksqlDB | Kafka Streams | |---|---|---| | **What it is** | SQL layer + server, built on Kafka Streams | A JVM library you embed in your app | | **Engine underneath** | Kafka Streams (compiled topologies) | Kafka Streams (you write it directly) | | **Language** | SQL only | Java (JVM; an official Scala DSL ships too) | | **Deployment** | A ksqlDB server cluster you run | A JAR inside a normal service, no extra cluster | | **Custom logic** | What SQL + UDFs express | Arbitrary code: Processor API, punctuators, any library | | **Sources / sinks** | Kafka only for queries; can manage Connect connectors | Kafka only | | **State & scaling** | Changelog-backed, ≤ partition count | Changelog-backed, ≤ partition count (identical) | | **Operational owner** | Whoever runs the ksqlDB servers | The application team | | **Momentum** | Low investment; SQL focus moved to Flink SQL | Core part of Apache Kafka, actively developed | The first eight rows describe a genuine tradeoff: SQL ergonomics and an analyst-friendly server, against full control inside an app you already own. The last row is different in kind, it's about where each project is *going*, not what it does. That deserves its own section, because it's what the "is ksqlDB dead" queries are really after. ## The status reality of ksqlDB Be factual about this, because it's a real and recurring question, and dodging it helps no one. ksqlDB is under **substantially lower active investment** than it once was. It is frequently described as effectively maintenance-mode: it still works, it still ships, but the SQL-for-stream-processing momentum inside Confluent, the company that builds it, has moved to **Flink SQL**. New streaming-SQL features and the roadmap energy land there now. That's not snark; it's where the commits and the product positioning point. Two concrete signals: ksqlDB now versions in lockstep with Confluent Platform releases instead of its old standalone 0.x scheme, and Confluent Cloud does not offer fully managed ksqlDB on Enterprise clusters. What that means in practice, without overstating it: - **It is not abandoned and not removed.** ksqlDB still runs in production at many organizations. The Kafka Streams engine underneath it is very much alive and actively developed, so the runtime isn't going stale even where the SQL layer is quiet. - **It is a real signal for *new* projects.** If you're choosing a streaming-SQL tool today with a multi-year horizon, building fresh on a low-investment layer is a risk you should weigh deliberately, look hard at Flink SQL first. - **It is not a reason to rip out a working deployment.** A ksqlDB cluster that does its job doesn't suddenly stop working because the roadmap cooled. Migrate when you have a concrete reason, a feature you need, a consolidation onto Flink, not because of a vibe. > 🚫 *"ksqlDB is maintenance-mode, so we should rewrite our queries in Kafka Streams to be safe."* Rewriting working ksqlDB SQL into hand-maintained Java topologies trades a quiet-but-functional layer for a pile of code your team now owns forever, and you land on the *same engine* you were already running. If you're going to spend a migration, spend it moving toward where the investment is (Flink SQL) only if you actually need cross-source pipelines or its roadmap; otherwise leave the working queries alone. ## When each is the right call Pick by who writes and operates the logic, then sanity-check the capability (SQL covers more than people expect, but not everything). **Choose ksqlDB when:** - Your authors live in SQL, analysts, data engineers, and the work is filters, joins, and windowed aggregations. - You've already committed to running a ksqlDB server cluster, or have one and it works. - You want quick, declarative streaming pipelines without standing up app code or a separate engine. **Choose Kafka Streams when:** - The logic belongs inside an application your team owns and deploys, with no extra server to operate. - You need custom code SQL can't express, arbitrary branching, library calls, the [Processor API](https://www.conduktor.io/kafka-streams/processor-api), bespoke error handling. - You're a JVM shop and want stream processing to deploy and [scale like any microservice](https://www.conduktor.io/kafka-streams/scaling). **Look at Flink SQL when:** - You want SQL streaming *and* you're starting fresh today, it's where the streaming-SQL investment is going. - You need non-Kafka sources, scale past your partition count, or a platform team operating a shared cluster. See [Kafka Streams vs Flink](https://www.conduktor.io/kafka-streams/vs-flink). The trap is treating this as ksqlDB *versus* Kafka Streams when they share an engine. The genuine fork is SQL-on-a-server vs library-in-an-app, and, for new SQL work, whether Flink SQL is the better bet than either. ## Frequently Asked Questions **Is ksqlDB just Kafka Streams under the hood?** Essentially, yes. ksqlDB compiles each SQL statement into a Kafka Streams topology and runs it on the Kafka Streams runtime embedded in the ksqlDB server. It adds a SQL language and a server you operate, but the processing engine, the changelog-backed state, and the partition-bound scaling are all Kafka Streams. **Is ksqlDB deprecated or in maintenance mode?** It is not formally deprecated and still ships and runs, but it is under much lower active investment, and is often described as effectively in maintenance mode. Confluent has moved its streaming-SQL focus to Flink SQL. Treat it as stable-but-quiet: fine to keep running, worth weighing carefully before betting a new project on it. **What is the difference between ksqlDB, Kafka Streams, and Flink SQL?** Kafka Streams is a JVM library you embed in your app. ksqlDB is SQL plus a server, built on top of Kafka Streams (Kafka-only, partition-bound). Flink SQL is SQL on Apache Flink, a separate cluster engine that reads many sources and scales past partition count, and is where new streaming-SQL investment is going. **Can I use SQL with Kafka Streams directly?** Not within the Kafka Streams library itself, its API is the Java DSL and Processor API, not SQL. To write SQL over the Kafka Streams engine you use ksqlDB, which compiles SQL to Streams topologies. For SQL on a different engine that reads beyond Kafka, use Flink SQL. **Should I migrate existing ksqlDB queries to Kafka Streams?** Usually not just because ksqlDB is quiet. You'd land on the same engine, having taken on Java code to maintain. Migrate only for a concrete reason, custom logic SQL can't express, or consolidating onto Flink SQL for cross-source pipelines. A working ksqlDB deployment doesn't need rescuing. > **See it in practice with Conduktor** > Whether you run ksqlDB queries or hand-written Kafka Streams apps, both surface on Kafka as consumer groups, changelog topics, and lag, because both *are* Kafka Streams underneath. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-vs-ksqldb) lets you watch consumer group lag, inspect the internal and changelog topics a query or app creates, and confirm partition assignment, so you can tell whether the processing is keeping up, independent of whether it was written in SQL or Java. ## Next steps - [Kafka Streams vs Flink](https://www.conduktor.io/kafka-streams/vs-flink), including where Flink SQL fits for new SQL work - [What is Kafka Streams?](https://www.conduktor.io/kafka-streams), the library that ksqlDB is built on - [The future of Kafka Streams](https://www.conduktor.io/kafka-streams/future), where the engine under both is headed Source: https://www.conduktor.io/kafka-streams/vs-ksqldb --- # Kafka Streams vs Spark *Pick between a per-record library and a cluster engine.* Most "Kafka Streams vs Spark" comparisons drown you in a feature grid and miss the one split that actually decides it. Kafka Streams is a **library** you embed in your own app to process Kafka data one record at a time. Spark Structured Streaming is a **distributed cluster engine** whose heritage is micro-batch processing, sitting inside a platform that also does batch, SQL, and machine learning at scale. Those are different tools for different shapes of problem, and the decision falls out of that, not from counting connectors. We sell neither, so there's no thumb on the scale. This is the senior-architect version: the deployment and latency models that actually decide it, and an honest account of where Spark wins outright. **What you'll learn:** - Why "library vs cluster engine" decides this, not a feature list - The per-record vs micro-batch latency model (and Spark's continuous mode) - A side-by-side of state, scaling, language, and sources - When Kafka Streams wins, and when Spark clearly wins ## Library vs cluster engine Kafka Streams is a `org.apache.kafka:kafka-streams` JAR on your classpath. You write a topology, your service [becomes the stream processor](https://www.conduktor.io/kafka-streams), and it [scales like a consumer group](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets), no separate cluster, no job to submit. It reads and writes Kafka, and only Kafka. The application team owns its memory, its [state](https://www.conduktor.io/kafka-streams/state-store), its restarts. Spark is a distributed compute engine. You stand up a cluster, a driver plus executors, on YARN, Kubernetes, or a managed platform, and submit a job that the cluster schedules across executors, checkpointing state to durable storage. It reads from and writes to almost anything: Kafka, object storage, files, JDBC databases, and more, often several in one job. It scales far past your Kafka partition count, and it usually comes with a data or platform team operating it as shared infrastructure. That single split, *code you ship inside an app* versus *jobs you submit to a cluster*, predicts most of the rest: who operates it, how it scales, what it can read, and which kind of workload it's built for. Start there; the feature comparison is a tiebreaker. ## The latency model: per-record vs micro-batch This is the difference people reach for first, and it's real, with a nuance worth getting right. **Kafka Streams processes one record at a time.** A record arrives, the topology runs, the result is emitted. There's no batching delay inherent to the model, but the defaults add two: Streams overrides the producer's `linger.ms` to 100, so out-of-the-box end-to-end latency sits around 100 ms (set it back to 0 and it's genuinely single-digit milliseconds), and cached KTable aggregations coalesce downstream updates until the commit interval (30 s by default) unless you disable caching. **Spark Structured Streaming's heritage is micro-batch.** It collects records into small batches, one per trigger (by default a new micro-batch starts as soon as the previous one finishes; a fixed interval is opt-in), and processes each batch as a tiny Spark job. That adds latency on the order of hundreds of milliseconds to seconds, in exchange for very high throughput and the full power of Spark's batch engine on each batch. > 🚫 *"Spark does real-time streaming, so its latency is the same as Kafka Streams."* Spark added a low-latency continuous processing mode (in 2.3) that drops per-record latency toward the millisecond range, but it's still **experimental**, gives only at-least-once (not exactly-once), and supports just map-like operations, no aggregations, so it narrows this gap for only a thin slice of jobs. Spark 4.1 shipped a separate Real-Time Mode, a third execution engine targeting sub-100 ms latency, currently limited to stateless queries. Micro-batch remains its center of gravity, the default, the best-supported path, and where its throughput and unified-batch strengths live. If single-record, millisecond-tail latency inside an application is the hard requirement, Kafka Streams meets it natively; with Spark you're choosing a mode and accepting its constraints. ## Side by side A map of where each is at home, not a scoreboard. The "when it wins" row is the one that matters. | Dimension | Kafka Streams | Spark Structured Streaming | |---|---|---| | **What it is** | A JVM library embedded in your app | A distributed cluster engine (driver + executors) | | **Processing model** | Per-record (event-at-a-time) | Micro-batch by default; low-latency continuous mode available | | **Deployment** | A JAR, a normal service, no extra cluster | A cluster you operate (YARN / K8s / managed) | | **Languages** | Java + Scala (official APIs); any JVM language | Scala, Python, Java, SQL + a large ML ecosystem | | **Sources / sinks** | Kafka only, in and out | Kafka, object storage, files, JDBC, many more | | **Scaling ceiling** | ≤ partition count of the input | Far beyond partition count (executor parallelism) | | **State & recovery** | Local store + [changelog](https://www.conduktor.io/kafka-streams/state-store) replay | State backend + [checkpoint](https://spark.apache.org/docs/latest/streaming/index.html) to durable storage | | **Batch + ML** | Streaming only | Unified batch, streaming, SQL, and ML in one engine | | **Operational owner** | The application team | A data / platform team | | **When it wins** | In-app stateful event processing, microservices | Heavy batch+stream unification, ML pipelines, multi-source ETL at scale | Two rows deserve a caveat the grid can't carry. **Scaling ceiling.** Kafka Streams parallelism tops out at the partition count of the busiest sub-topology, extra instances past that sit idle, or serve as standbys if you set `num.standby.replicas` above its default of 0. Spark decouples parallelism from input partitions and scales across executors, so it handles genuinely huge jobs and wide fan-out that Kafka Streams can't. Most in-app workloads never approach the Streams ceiling, so this decides the large, cluster-scale jobs. See [scaling Kafka Streams](https://www.conduktor.io/kafka-streams/scaling). **State & recovery.** Both checkpoint state durably, but the models differ in who holds it and how a restart behaves. Kafka Streams replays a changelog into a *local* store on the instance, which puts [restore time](https://www.conduktor.io/kafka-streams/state-restore) on the app team. Spark checkpoints to durable storage (HDFS/S3/DBFS) that the cluster manages, so recovery is the cluster's job, not the app's. ## When each wins Choose by deployment model and workload shape, then confirm the capability fits. **Choose Kafka Streams when:** - The processing belongs *inside* an application your team owns and deploys, an [event-driven microservice](https://www.conduktor.io/kafka-streams), not a data-platform job. - Your sources and sinks are Kafka, and you're a JVM shop. - You need per-record, low-millisecond latency without choosing a special engine mode. - You want stream processing to scale and deploy like any other service, no cluster to operate. - The state is manageable, or you're prepared to own [restore time](https://www.conduktor.io/kafka-streams/state-restore) and [RocksDB memory](https://www.conduktor.io/kafka-streams/rocksdb-tuning). **Choose Spark when:** - You're unifying **batch and streaming** in one engine, backfills and live processing sharing logic. - You're pulling from **non-Kafka sources**, object storage, files, JDBC, several systems in one job, for multi-source ETL. - You need **machine learning** alongside streaming (feature pipelines, MLlib, model scoring at scale). - You must **scale past your Kafka partition count** or run large shared jobs as platform infrastructure. - Your authors live in **Python, Scala, or SQL** rather than the JVM DSL. Be honest about the boundary: a heavy multi-source ETL or ML pipeline crammed into a Kafka-only library fights you forever, and so does an in-app, low-latency microservice forced onto a cluster you have to operate. Spark and Kafka Streams coexist in plenty of shops, Spark for the data-platform batch+stream+ML work, Kafka Streams for the stateful logic living inside services the app team owns. They're complements more often than rivals. ## Frequently Asked Questions **What is the difference between Kafka Streams and Spark Structured Streaming?** Kafka Streams is a JVM library you embed in your own application to process Kafka data per-record, scaling like a consumer group with no separate cluster. Spark Structured Streaming is a distributed cluster engine, micro-batch at its core, that reads many sources, scales past partition count, and unifies batch, streaming, SQL, and ML, operated as shared infrastructure. **Is Kafka Streams micro-batch?** No. Kafka Streams processes one record at a time (event-at-a-time), so there's no inherent batching delay. Spark Structured Streaming is the micro-batch engine by default, it groups records into small batches, one per trigger, though it also offers a low-latency continuous mode. **Which has lower latency, Kafka Streams or Spark?** Kafka Streams generally has lower and more predictable per-event latency because its model is per-record: low tens of milliseconds out of the box, single-digit once you drop Streams' default producer linger.ms of 100 to 0. Spark's default micro-batch adds latency on the order of hundreds of milliseconds; its continuous mode narrows the gap but micro-batch remains its primary, best-supported path. **When should I use Spark over Kafka Streams?** Choose Spark when you need to unify batch and streaming, read non-Kafka sources, run machine-learning pipelines, scale far past your Kafka partition count, or write in Python/Scala/SQL. Spark is a data-platform engine; Kafka Streams is for stateful logic embedded in a Kafka-native application your team owns. **Can Kafka Streams replace Spark for ETL?** Only for Kafka-to-Kafka, JVM-based, in-app transformations. Kafka Streams reads and writes Kafka only and is a library, not a multi-source ETL or batch engine. Multi-source ingestion, batch backfills, and ML feature pipelines at scale are Spark's territory, not Kafka Streams'. > **See it in practice with Conduktor** > Whichever engine you pick, the Kafka side of it shows up as consumer groups, lag, and topics. A Kafka Streams app is a consumer group named after its `application.id`, plus changelog topics. A Spark job reading Kafka appears as an auto-named consumer group too, but it doesn't commit offsets to Kafka by default (progress lives in its checkpoint), so set `kafka.group.id` and offset commits if you want Kafka-side lag. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-vs-spark) lets you watch consumer group lag, inspect partition assignment, and see the topics each one reads and writes, so you can tell whether a Streams app is keeping up with the partitions it owns, independent of which framework does the processing. ## Next steps - [Kafka Streams vs Flink](https://www.conduktor.io/kafka-streams/vs-flink), the other cluster-engine comparison, by deployment model - [What is Kafka Streams?](https://www.conduktor.io/kafka-streams), the library model, in depth - [Scaling Kafka Streams](https://www.conduktor.io/kafka-streams/scaling), why parallelism caps at your partition count Source: https://www.conduktor.io/kafka-streams/vs-spark --- # Kafka Streams windowing *Learn how windows work, and why they close when you don't expect.* A plain aggregation runs forever, the count of orders per customer has no end. Windowing carves that endless stream into bounded buckets of time: orders per customer *per hour*, errors per service *per five minutes*. You group as usual, then add a window, and Kafka Streams keeps a separate running result per key per bucket. The window *types* are the easy part, four of them, each a one-liner. The hard part, and the source of nearly every windowing bug filed against Kafka Streams, is *when a window closes*. It is not the wall clock. It is **stream-time**, and stream-time only moves when new records arrive. That single fact explains the idle-stream-never-emits problem, the flood of warnings when you reprocess, and why Kafka Streams refuses to implement watermarks at all. **What you'll learn:** - The four window types, tumbling, hopping, sliding, session, and when to use each - Grace periods, and why newer windowed operators default to *no* grace - Why windows close on stream-time, not wall-clock, the root of most windowing confusion - Why Kafka Streams rejects watermarks in favour of continuous refinement ## The four window types You window a grouped stream with `.windowedBy(...)`, then aggregate as normal. The result is keyed by a `Windowed`, the original key *plus* the window it belongs to. ### Tumbling, fixed, non-overlapping Adjacent, equal-size, gap-free buckets. Every record falls into exactly one window. The default choice for "per hour" / "per minute" rollups. ```java KTable, Long> hourly = events .groupByKey() .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofHours(1), Duration.ofMinutes(5))) .count(); ``` ### Hopping, fixed size, overlapping A window of size *S* that advances by a smaller step (the "hop" or "advance"). Windows overlap, so a record can land in several. Use it for sliding-style metrics on a fixed grid, "5-minute count, updated every minute." ```java TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofMinutes(1)) .advanceBy(Duration.ofMinutes(1)); // hop = 1 min → each record in up to 5 windows ``` A smaller hop means more overlapping windows, more keys, and more state. A hop equal to the size *is* a tumbling window. ### Sliding, windows defined by the data, not a grid `SlidingWindows` creates windows of a fixed size, but anchored on actual record timestamps rather than a fixed grid, so two records are aggregated together exactly when they fall within the window size of each other. Fewer, more meaningful windows than hopping for "events within *N* minutes of each other," at the cost of more complex semantics. ```java .windowedBy(SlidingWindows.ofTimeDifferenceAndGrace( Duration.ofMinutes(5), Duration.ofMinutes(1))) ``` ### Session, windows defined by gaps of inactivity A session window grows to include records while they keep arriving, and closes after an *inactivity gap* with no new records for that key. Sessions for the same key that come close enough together merge. The natural fit for user activity: a "browsing session" that ends when the user goes quiet. ```java .windowedBy(SessionWindows.ofInactivityGapAndGrace( Duration.ofMinutes(30), Duration.ofMinutes(5))) .count(); ``` | Window | Shape | Overlap | Use when | |---|---|---|---| | Tumbling | Fixed grid, size = step | No | Per-hour / per-minute rollups | | Hopping | Fixed grid, size > step | Yes | Sliding metric on a fixed grid | | Sliding | Anchored on record times | Yes | "Events within N of each other" | | Session | Grows until an inactivity gap | No | User sessions, bursty activity | ![The four Kafka Streams window types over one shared event stream: tumbling windows sit on a fixed grid with no overlap, hopping windows share that grid but overlap so one event lands in several, sliding windows are anchored on the record timestamps so events within five minutes of each other aggregate together, and session windows grow until a three-minute inactivity gap splits them.](https://www.conduktor.io/assets/kafka-streams/diagrams/windowing-window-types.svg) ## Grace: how late is too late Records arrive out of order. A record stamped 10:59 can show up at 11:02, after its 10:00–11:00 window looks finished. The **grace period** is how long Kafka Streams keeps a window open for stragglers after its end time. Within grace, a late record still updates its window; past grace, the record is dropped as expired. ```java TimeWindows.ofSizeAndGrace(Duration.ofHours(1), Duration.ofMinutes(10)); // window 10:00–11:00 accepts late records until stream-time reaches 11:10 ``` > **Set grace deliberately, don't rely on remembered defaults.** The current factory methods (`ofSizeAndGrace`, `ofTimeDifferenceAndGrace`, `ofInactivityGapAndGrace`) make grace an explicit, required argument, so you choose it, including choosing none, with `ofSizeWithNoGrace`. The default shifted across versions: the original `TimeWindows.of(size)` derived a large, retention-based grace (on the order of a day), while the newer explicit factories force the decision on you. Match grace to how late your data realistically arrives, and don't carry an old version's remembered default into a new one. ## Where the record timestamp comes from Stream-time is built from *record timestamps*, so it's worth knowing which timestamp that is. By default Kafka Streams uses the `FailOnInvalidTimestamp` extractor, which reads the timestamp Kafka embedded in the record, and that means the topic's timestamp type decides your time semantics: - `CreateTime` (the default) is the **producer's** timestamp → event-time semantics, with all the out-of-order and lateness concerns this page is about. - `LogAppendTime` is the **broker's** ingestion time. It's monotonic per partition, so records effectively *cannot* be late under `LogAppendTime`, there's no out-of-order to be late against. It also means you've thrown away true event time. You can override the extractor, via `default.timestamp.extractor` or per-stream with `Consumed.withTimestampExtractor(...)`: - `WallclockTimestampExtractor` ignores the record and returns `System.currentTimeMillis()`, pure processing-time, so windows close on the wall clock no matter what the data says. - A **custom `TimestampExtractor`** pulls event-time out of the message *payload*, the right move when the meaningful time is a field inside the event, not when it was produced or appended. - For invalid (negative) timestamps the built-ins differ: `FailOnInvalidTimestamp` throws, `LogAndSkipOnInvalidTimestamp` drops the record, and `UsePartitionTimeOnInvalidTimestamp` substitutes the last good timestamp from that partition. Pick the wrong extractor and every window below is bucketed by the wrong clock, so settle this before you reason about grace and stream-time. ## The thing that actually confuses everyone: stream-time Windows do not close because your clock reached 11:05. They close because **stream-time** passed the window's end plus its grace. Understanding stream-time is the whole game. Stream-time is Kafka Streams' internal notion of "how far through the data we are." It is defined as the **maximum record timestamp the task has seen so far**, and it is *monotonic*, it only moves forward, and it only moves when a new record arrives with a timestamp later than the current maximum. It is driven entirely by the data. It has nothing to do with wall-clock time. So a window for 10:00–11:00 with 5 minutes of grace closes at the moment a record arrives whose timestamp is ≥ 11:05. Not when your server's clock hits 11:05, when a *record* carrying that timestamp flows through the task. Stream-time is a property of the **task**, shared across every key that task processes, it is *not* tracked per key. So one busy key advancing stream-time also closes a quiet key's window; only when the task's whole input goes quiet does nothing close. (Per-key time tracking has been proposed upstream but never adopted, so per-task is the model you design around.) ![A 10:00 to 11:00 window with five minutes of grace closes only when a record arrives whose timestamp is at or past 11:05. Records at 10:02, 10:31, and 10:58 advance stream-time, then the stream goes quiet and stream-time freezes at 10:58, so the window stays open even as the wall clock passes 11:05; a record at 11:07 finally pushes stream-time past the threshold and the window closes and emits.](https://www.conduktor.io/assets/kafka-streams/diagrams/windowing-stream-time.svg) ### Why an idle stream never emits its final window This is the single most-reported windowing surprise, and it follows directly from the definition. If no new records arrive, stream-time does not advance. If stream-time does not advance, the window never reaches end + grace. If the window never closes, any logic waiting for it to close, most importantly [`suppress(untilWindowCloses(...))`](https://www.conduktor.io/kafka-streams/suppress-not-emitting), emits **nothing**. Picture a low-traffic partition: the last event of the hour arrives at 10:58, then the stream goes quiet. Stream-time is stuck at 10:58. The 10:00–11:00 window will not close until *the next record after 11:05* shows up, which on a quiet partition could be minutes or hours later, or never. The final result you were waiting for simply never fires. Teams hit this, conclude `suppress` is broken, and reach for the notorious workaround of injecting a dummy future-timestamped event just to shove stream-time forward. It is not broken. It is doing exactly what stream-time semantics require. The detailed symptom-to-cause checklist, sparse partitions, one lagging partition holding back a task, test harnesses that never advance time, buffer config, lives in [the suppress() trap](https://www.conduktor.io/kafka-streams/suppress-not-emitting). > 🚫 *"My windowed aggregation stopped emitting, so the app is stuck."* > More likely the stream went quiet and stream-time stopped advancing. Windows close on stream-time, and stream-time is the max record timestamp seen, no new records, no progress, no window close. ### Why reprocessing floods you with "skipping expired" warnings The same mechanism explains the opposite failure. When you reset an app and reprocess a topic from the beginning, the first records you read are *old*, hours or days in the past. The very first record sets stream-time to its timestamp; but as you race through history, stream-time leaps forward fast. A record from Tuesday read just after a record from Friday is now far older than `stream-time − grace`, so it is dropped as expired, and Kafka Streams logs a warning for each one. Replaying a large topic can therefore produce a torrent of "skipping record for expired window" messages. They are expected during a full reprocess, not a sign of corruption; the records being skipped are genuinely outside any open window given how quickly stream-time advanced. The warning itself is highly diagnostic, it includes the topic, partition, offset, record timestamp, window range, and current stream-time, and each drop increments the task-level `dropped-records-total` metric, which is worth alerting on in normal operation since expired drops are otherwise silent data loss. ## Why Kafka Streams doesn't use watermarks If you come from Flink, you expect *watermarks*: a special marker the system propagates to declare "no more events older than T will arrive," which triggers windows to fire. Kafka Streams deliberately does **not** do this. It uses stream-time plus **continuous refinement** instead. The reasoning, laid out in the Apache Kafka design (KIP-328) and Confluent's writing on the topic, is that watermarks force a single, up-front decision about completeness and a one-shot trigger: the window fires once, late data after the watermark is awkward to handle, and you must guess the watermark delay correctly. Kafka Streams takes the opposite stance, a windowed aggregation emits an *updated result every time the window changes* (modulated by the record cache, as covered in [aggregations](https://www.conduktor.io/kafka-streams/aggregations)), and the grace period defines exactly when it stops accepting changes. There is no watermark to tune and no single firing moment; the result is continuously correct-so-far and is finalized when grace expires. `suppress` is the opt-in layer you add *on top* when you want one emission per window instead of the continuous stream. The trade-off is real and worth stating plainly: continuous refinement means downstream consumers must tolerate seeing intermediate results, and getting a single clean result per window requires `suppress`, which reintroduces the stream-time dependency above. Neither model is free; they move the difficulty to different places. ## A complete windowed example Putting it together, count events per key per hour, with 5 minutes of grace, and write the closed-ish results to a topic. Note how the windowed key is unpacked back to the plain key on the way out: ```java events .groupByKey() .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofHours(1), Duration.ofMinutes(5))) .count(Materialized.as("events-per-hour")) // KTable, Long> .toStream() .to("hourly-counts", Produced.with( WindowedSerdes.timeWindowedSerdeFrom(String.class, Duration.ofHours(1).toMillis()), Serdes.Long())); // If you only want the original key downstream, map the Windowed back to K: // .toStream((Windowed wk, Long count) -> wk.key()) ``` Two practicalities. The result key is a `Windowed`, so serializing it needs a windowed serde, a common source of [serde errors](https://www.conduktor.io/kafka-streams/serdes) if you forget. The serde takes the window size because the serialized key only carries the window *start*; the end is reconstructed from the size at read time (the size-less `timeWindowedSerdeFrom(Class)` overload was deprecated by KIP-659 and removed in Kafka 4.0+). And without `suppress`, this emits the continuous stream of per-window updates described above, not one row per window. ## Frequently Asked Questions **What are the four window types in Kafka Streams?** Tumbling (fixed, non-overlapping buckets on a grid), hopping (fixed size that advances by a smaller step, so windows overlap), sliding (fixed size anchored on actual record timestamps), and session (grows while records keep arriving and closes after an inactivity gap). You apply one with `.windowedBy(...)` after grouping. **What is the difference between a tumbling and a hopping window?** A tumbling window has its size equal to its advance step, so every record lands in exactly one window with no overlap. A hopping window advances by a smaller step than its size, so windows overlap and a record can land in several, a hop equal to the size is just a tumbling window. **What is a grace period and how does it handle late records?** The grace period is how long Kafka Streams keeps a window open for out-of-order stragglers after its end time. Within grace a late record still updates its window; past grace the record is dropped as expired. Newer factory methods like `ofSizeAndGrace` make grace an explicit, required choice. **Why does my windowed aggregation stop emitting on a quiet stream?** Windows close on stream-time, not the wall clock, and stream-time is the maximum record timestamp the task has seen. If no new records arrive, stream-time does not advance, the window never reaches end-plus-grace, and anything waiting on the close, like `suppress(untilWindowCloses(...))`, emits nothing. **Why does reprocessing flood my logs with "skipping expired" warnings?** When you reset and replay a topic from the beginning, stream-time leaps forward as you race through old records. A record read just after a much newer one is now older than `stream-time − grace`, so it is dropped as expired and logged, expected during a full reprocess, not corruption. > **See it in practice with Conduktor** > A windowed aggregation creates a changelog topic whose keys carry the window boundaries, and it can grow large when windows or grace are wide. [Conduktor Console](https://www.conduktor.io/get-started?src=kafka-streams-windowing) lets you inspect that changelog topic, see its size and compaction, and watch the consumer group lag, useful for telling whether a window simply hasn't closed yet versus a real processing backlog. ## Next steps - [The suppress() trap](https://www.conduktor.io/kafka-streams/suppress-not-emitting), get one result per window, and why it emits nothing on quiet streams - [Aggregations](https://www.conduktor.io/kafka-streams/aggregations), count, reduce, and aggregate, and the continuous-update model - [Joins](https://www.conduktor.io/kafka-streams/joins), windowed stream-stream joins and the co-partitioning requirement Source: https://www.conduktor.io/kafka-streams/windowing --- # What is Kafka Streams? *Understand what Kafka Streams is, and what it isn't.* Kafka Streams is a Java library for processing data that lives in Apache Kafka. You add it as a dependency, write a topology, and your own application becomes the stream processor. There is no separate cluster to deploy, no job to submit, no scheduler to babysit: the processing runs inside your service, next to your business logic. That single design choice (*a library, not a platform*) explains almost everything about how Kafka Streams behaves in production, for better and for worse. This guide covers both. **What you'll learn:** - What Kafka Streams is and how it differs from the plain consumer API - Why being a library (not a cluster) shapes how you run it - What you can realistically build with it - When Kafka Streams is the right tool, and when it isn't ![A Kafka Streams application reads records from topics in a single Kafka cluster, processes them, and writes the results back into the same cluster, where other apps, other Kafka Streams applications, and dashboards or databases consume them](https://www.conduktor.io/assets/kafka-streams/diagrams/kafka-streams-dataflow.svg) ## A library, not a cluster Most stream processors (Flink, Spark Structured Streaming) are systems you stand up and submit jobs to. Kafka Streams inverts that. It is a `org.apache.kafka:kafka-streams` JAR on your classpath. Your app reads from topics, transforms records, and writes to topics, all through a fluent API: ```java StreamsBuilder builder = new StreamsBuilder(); builder.stream("orders") .filter((key, order) -> order.contains("\"status\":\"PAID\"")) .to("paid-orders"); KafkaStreams streams = new KafkaStreams(builder.build(), props); streams.start(); ``` That is a complete, runnable stream processor, with one catch in `props`: since Kafka 3.0, `default.key.serde` and `default.value.serde` have no default value, so set them (along with `application.id` and `bootstrap.servers`) or the app throws a `StreamsException` at startup: ```java Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "order-filter"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass()); props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass()); ``` It scales the same way any Kafka consumer scales: run more instances, and Kafka's [consumer group protocol](https://www.conduktor.io/kafka/kafka-consumer-groups-and-consumer-offsets) spreads the partitions across them. No resource manager, no cluster, no YARN. The trade-off is ownership. Because Kafka Streams runs inside your process, *you* own its memory, its state on local disk, its restarts, and its rebalances. A Flink operator hands those concerns to a cluster; a Kafka Streams developer carries them. > *"For 80 to 90 percent of stream-processing use cases, either Kafka Streams or Flink will work. The real question is the deployment model and who operates it."* > — paraphrasing a recurring theme from Kafka maintainers on the deployment-model trade-off ## More than the consumer API You could write all of this with a raw [Kafka consumer](https://www.conduktor.io/kafka/kafka-consumers) and producer. People do, and then slowly reinvent Kafka Streams badly. The library gives you, for free, things that are tedious and error-prone to hand-roll: - **Stateful operations**: aggregations, counts, and joins backed by a local store that survives restarts (covered in [state stores](https://www.conduktor.io/kafka-streams/state-store)). - **Event-time windowing**: group records by when they *happened*, not when they arrived ([windowing](https://www.conduktor.io/kafka-streams/windowing)). - **Exactly-once processing**: the read-process-write cycle made atomic with one config flag ([exactly-once](https://www.conduktor.io/kafka-streams/exactly-once)). - **Automatic scaling and fault tolerance**: partitions, tasks, and state move between instances without you writing rebalance code. The unit you work with is the **stream** (`KStream`, an unbounded log of events) and the **table** (`KTable`, the latest value per key). Understanding the difference between those two is the single most useful concept in the library. See [KStream vs KTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable). ## What you can build Kafka Streams fits a specific shape of problem: continuous, per-record processing of Kafka data, by the team that owns the application. - **Enrichment**: join an event stream against reference data (a `KTable` of users, products, accounts). - **Real-time aggregations**: counts, sums, and rollups per key and per time window (fraud scoring, usage metering, leaderboards). - **Materialized views**: turn a changelog into a queryable table you can read directly from your service via [interactive queries](https://www.conduktor.io/kafka-streams/state-store). - **Event-driven microservices**: services that react to events and emit new ones, with state and ordering handled for you. - **AI agent memory**: materialize a multi-agent conversation into a queryable context store that LLM agents read in real time ([Kafka Streams for AI agents](https://www.conduktor.io/kafka-streams/ai-agents)). ## When *not* to use Kafka Streams Being honest about the boundaries saves you a painful migration later: - **You're not a JVM shop.** Kafka Streams is Java/Scala only. If your team lives in Python or Go, a plain consumer or Flink's Python API is a better fit. - **You want someone else to operate the state.** Large local state (RocksDB) means slow restores and [rebalances](https://www.conduktor.io/kafka-streams/rebalancing) that can stall migrated tasks for minutes while their state rebuilds. If you don't want to own that, a managed cluster engine moves the burden off your team. - **Your sources aren't Kafka.** Kafka Streams reads and writes Kafka, full stop. Pulling from a database, a queue, and an HTTP API into one job is Flink's territory. - **It's a one-line stateless filter.** A single `filter` with no state is sometimes just a [consumer](https://www.conduktor.io/kafka/kafka-consumers) with three lines of code. Don't add a framework for it. > **Kafka Streams vs Flink vs ksqlDB.** This is the most common question newcomers ask, and most comparisons answer it dishonestly. We wrote a vendor-neutral one: [Kafka Streams vs Flink vs ksqlDB](https://www.conduktor.io/kafka-streams/vs-flink). ## What this guide covers This is a full course, built around the problems people actually hit, sourced from years of questions on the Confluent forum, Stack Overflow, and conference talks, not just the happy path. **Foundations:** [architecture](https://www.conduktor.io/kafka-streams/architecture) · [KStream, KTable & GlobalKTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable) · [stateless operations](https://www.conduktor.io/kafka-streams/stateless-operations) · [your first app](https://www.conduktor.io/kafka-streams/getting-started) · [aggregations](https://www.conduktor.io/kafka-streams/aggregations) · [state stores](https://www.conduktor.io/kafka-streams/state-store) · [windowing](https://www.conduktor.io/kafka-streams/windowing) · [joins](https://www.conduktor.io/kafka-streams/joins) · [exactly-once](https://www.conduktor.io/kafka-streams/exactly-once) **In production (where the bodies are buried):** [slow rebalances](https://www.conduktor.io/kafka-streams/rebalancing) · [state restore time](https://www.conduktor.io/kafka-streams/state-restore) · [RocksDB tuning](https://www.conduktor.io/kafka-streams/rocksdb-tuning) · [why you still see duplicates](https://www.conduktor.io/kafka-streams/exactly-once-duplicates) · [the suppress() trap](https://www.conduktor.io/kafka-streams/suppress-not-emitting) · [joins that drop data](https://www.conduktor.io/kafka-streams/join-troubleshooting) · [serde errors](https://www.conduktor.io/kafka-streams/serdes) · [evolving a topology](https://www.conduktor.io/kafka-streams/topology-evolution) · [dead letter queues](https://www.conduktor.io/kafka-streams/dead-letter-queue) · [deduplication](https://www.conduktor.io/kafka-streams/deduplication) · [scaling](https://www.conduktor.io/kafka-streams/scaling) · [testing](https://www.conduktor.io/kafka-streams/testing) ## Frequently Asked Questions **What is Kafka Streams used for?** Kafka Streams is a Java library for continuous, per-record processing of data in Apache Kafka: enrichment, real-time aggregations, materialized views, and event-driven microservices. The processing runs inside your own application, reading from topics and writing to topics, with no separate cluster to deploy. **Is Kafka Streams a database?** No. Kafka Streams is a stream-processing library, not a database. It can maintain local state (a `KTable` backed by a state store) and expose it for lookups, but the durable source of truth stays in Kafka topics, not in Kafka Streams. **What is the difference between Kafka and Kafka Streams?** Kafka is the distributed log that stores and moves the data; Kafka Streams is a client library that processes that data. They are complementary, not competitors: Kafka Streams reads from and writes to Kafka topics and runs as part of your application. **When should I use Kafka Streams instead of Kafka Connect?** Use Kafka Connect to move data between Kafka and external systems, and Kafka Streams to transform data already in Kafka. If your job is "get data in or out", that is Connect; if it is "filter, join, aggregate, or reshape records", that is Streams. **Does Kafka Streams replace the Kafka consumer and producer APIs?** It is built on top of them, not a replacement. Kafka Streams gives you stateful operations, event-time windowing, exactly-once, and automatic scaling for free, things that are tedious to hand-roll on the raw consumer and producer. For a one-line stateless filter, a plain consumer can still be simpler. > **See it in practice with Conduktor** > A Kafka Streams app is, under the hood, a consumer group plus a set of internal topics. [Conduktor Console](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) lets you watch its consumer group lag, inspect the changelog and repartition topics it creates, and confirm partition assignment, the signals you need when a Streams app misbehaves. One quick fingerprint: a Streams app's group reports its partition assignor as `stream`, which tells it apart from plain consumer groups at a glance. ## Next steps - [Kafka Streams architecture](https://www.conduktor.io/kafka-streams/architecture): topologies, tasks, and threads - [KStream vs KTable vs GlobalKTable](https://www.conduktor.io/kafka-streams/kstream-ktable-globalktable): the core mental model - [Build your first Kafka Streams app](https://www.conduktor.io/kafka-streams/getting-started): a runnable WordCount in Java Source: https://www.conduktor.io/kafka-streams --- # Kafka UI: Beyond the Browse Button A Kafka UI should do more than show you topics. The right one gives your team three things the CLI can't: visibility into what's happening, ownership over who's responsible, and autonomy to act without waiting on the platform team. Most Kafka UIs still only deliver the first. [Install Now](https://www.conduktor.io/get-started?src=kafka-ui) [Explore Console →](https://www.conduktor.io/console) ## What a Kafka UI should give your team The 60-second version A good Kafka UI answers three questions: what's happening, who owns it, can my team act? That's Visibility, Ownership, Autonomy. Every UI hits Visibility. Most stop there. Ownership needs real catalogs (topic, application, service account), RBAC with SSO, and audit trails shippable to your SIEM. Autonomy needs self-service within guardrails, plus a CLI and REST API so Kafka fits into the same CI/CD pipelines as the rest of your stack. Open-source tools (Kafbat, AKHQ, Redpanda Console) are solid at Visibility, thinner at Ownership, rarely do Autonomy at all. Conduktor Console does all three, through a UI, CLI, and REST API against the same cluster. A Kafka UI is a web interface for inspecting, managing, and operating Apache Kafka clusters without the CLI. It's where developers browse topics, read messages, debug consumer lag, and manage schemas. For platform teams, it's also where access, ownership, and compliance get enforced. The question that matters isn't "does it have a UI?" Every Kafka tool does. The question is whether the UI lets your team answer three things fast, without tripping over each other: - **01. Visibility** — See what's in Kafka right now: topics, messages, consumer lag, schemas, connector health, across every cluster. - **02. Ownership** — Know who owns which topic, which team is on the hook, who changed what. Provable to an auditor. - **03. Autonomy** — Developers move without filing tickets. Platform teams keep control through guardrails, not gatekeeping. Every Kafka UI claims Visibility. Most stop there. The real differentiation is how far each tool goes on Ownership and Autonomy, and whether it works past a single developer on a single cluster. On this page [Pillar 01: Visibility](#visibility) [Pillar 02: Ownership](#ownership) [Pillar 03: Autonomy](#autonomy) [Kafka UI tools today](#kafka-ui-tools-today) [Migrating off Kafka Tool / Offset Explorer / Kafka Manager / Provectus](#migrating-off-a-deprecated-tool) [Multi-cloud Kafka UI](#multi-cloud-kafka-ui) [Kafka UI vs CLI](#kafka-ui-vs-cli) [FAQ](#frequently-asked-questions) 01 Pillar 01 — Kafka UI Visibility What's happening in Kafka right now? Visibility is what every Kafka UI advertises. The floor is topic browsing and consumer group status. The ceiling, what separates a good UI from a great one, is whether your developers can debug a production issue in the UI alone, without SSH'ing into a broker or piping kcat output through jq. ### Core Kafka operations At a minimum, a usable UI shows you: - **Topics.** List, filter, inspect configuration, view partition layout, see replication status. - **Schemas.** Browse Schema Registry, compare versions, check compatibility mode. - **Connectors.** Deploy, monitor, and restart Kafka Connect tasks from the UI instead of curl-ing the REST API. - **Clusters.** Manage multiple Kafka clusters (self-managed, MSK, Confluent Cloud, Redpanda) from one place. If your UI can't do these four things in one view, your platform team ends up maintaining side-tools for the rest. That's how "one UI" turns into a graveyard of half-documented internal scripts. ### Data exploration (the troubleshooting test) The real test of Visibility is what happens when production breaks. Can a developer browse to the topic, filter messages by key or header, decode the Avro or Protobuf payload, and see the offending record in under a minute, without asking the platform team for help? A Kafka UI that passes this test needs: - Message browsing with offset and time-based navigation - Filtering by key, value fields (JQ expressions), headers, or timestamp - Native deserialization for Avro, JSON Schema, Protobuf, and raw bytes - The ability to produce test messages directly to a topic without deploying code - Replay and reprocess without writing one-off consumer scripts Most open-source UIs handle the basics here. What they don't give you is the loop *all the way back* to production. Once you find the bug, can you fix it in the UI, or do you need a platform team ticket to redeploy a connector with the right config? That part straddles Visibility and Autonomy. A debugging session end-to-end: topic lookup, filter messages by key, decode the payload, find the offending record. This is the troubleshooting test in practice — shown in Conduktor Console. ### Observability Visibility without alerting is just a dashboard. A production-grade Kafka UI shows you: - Consumer lag in real time, with alert thresholds - Broker health (CPU, disk, network) and partition distribution - Cluster-level trends: ingestion rate, message size, replication delay - Integration with your existing stack (Prometheus, Datadog, Slack, PagerDuty) The distinction here is observability that *drives action*, not observability that prints a pretty graph. If your UI shows you consumer lag is rising but there's no way to alert the right team before it becomes an incident, you're still operating reactively. Go deeper: [Visibility in Conduktor Console →](https://www.conduktor.io/console#simplify-kafka-operations) 02 Pillar 02 — Kafka UI Ownership Who is responsible for each resource? Ownership is where most Kafka UIs go quiet. Open-source tools were built on the assumption that one person, or maybe one team, operates the cluster. As soon as you have five teams sharing a Kafka deployment, that assumption breaks. The UI shows you a list of 400 topics and there's no way to tell which team owns each one. Fixing this needs more than a label field. ### Catalogs and team-to-resource mapping A real Ownership model goes beyond labels. The UI needs catalogs that map every resource to the team and application responsible for it, so any developer can answer "who owns this?" in seconds without opening a ticket: - **Topic catalog** showing every topic with its owner team, schema, data classification, and the apps consuming or producing it - **Application catalog** mapping each producer and consumer app to its owner team, with the topics it touches and its current state - **Service account catalog** for non-human principals (pipelines, connectors, batch jobs), so machine access isn't invisible - Consumer groups inheriting ownership from the applications that run them - Schemas with a subject owner separate from the topics that use them - Cross-cluster searches scoped to what your team owns and what you've been granted access to > If your UI shows every developer a flat list of everything, it's not a UI. It's a filesystem browser. An application catalog mapping producer and consumer apps to their owner teams — shown in Conduktor Console. ### Access controls and RBAC Ownership without enforcement is documentation. A Kafka UI that takes Ownership seriously lets platform teams: - Define roles at the user, group, and resource level - Grant scoped permissions per topic, cluster, and operation - Mask sensitive fields (PII, PHI) for users without data-access rights, while still letting them debug - Replace shared admin credentials with per-user accounts that produce audit trails The SSO layer matters here too. If your developers still use a different login for the Kafka UI than they do for everything else, your access list is going to drift the day someone joins or leaves. ### Insights and audit trails Ownership becomes real when it's auditable. The UI should give you: - Who created, modified, or deleted each topic, consumer group, and schema, with timestamps - Cost attribution per team or application (so you can hand a chargeback sheet to finance) - Data quality visibility: which topics have invalid payloads, which producers are out of spec - Long-retention audit logs that ship to your SIEM (Splunk, ELK, Datadog) for compliance If the UI can't produce these on demand, especially the "who changed what" record, you're going to fail a SOC 2 audit the first time one asks. An insights view across a cluster — health score, under-replicated topics, partition skew, and per-topic risk flags. Shown in Conduktor Console. Go deeper: [federated ownership in Console →](https://www.conduktor.io/console#federated-ownership) · [RBAC and audit trails →](https://www.conduktor.io/console#secure-without-slowing-down) 03 Pillar 03 — Kafka UI Autonomy Can my team act without waiting on the platform team? Autonomy is the hardest pillar, and it's the one that separates Kafka UIs designed for platform-scale teams from Kafka UIs designed for a developer's laptop. It answers one question: can a developer get their work done without a ticket sitting in the platform team's queue for three days? The right UI lets developers operate within policy. The policy is the platform team's way of delegating authority without giving up control. ### Self-service within guardrails Autonomy at scale means: - Developers create topics through the UI (or API, or Terraform) without waiting on ops - The platform team defines naming conventions, replication factors, retention policies, and partition limits that are enforced at creation time - Invalid configurations get rejected with clear error messages, not with a ticket bouncing back a day later - **Approval workflows route to the resource owner** via the ownership catalogs. Sensitive actions like deleting a prod topic flow to the team that owns it, not to a generic platform-team inbox. Routine requests auto-approve. The key invariant: the platform team doesn't approve each request. They approve the policy. Once the policy is in place, the UI enforces it automatically on every request. ### Independent troubleshooting Autonomy for debugging means developers resolve their own production issues without escalating. The UI gives them: - Message browsing, filtering, and decoding, so they can find the bad record themselves - Consumer group inspection, so they can see if their app is stuck, lagging, or rebalancing - Connector restart and reconfiguration, so they can recover from transient failures without waking the platform team - Safe production access: read-only for sensitive resources, write access to their own Most Kafka incidents are either a bad message or a misconfigured consumer. When developers can solve those themselves, your platform team stops being a bottleneck for every production issue. ### Automation that catches misconfigurations early Autonomy fails the first time a developer ships a topic that takes down the cluster. Prevention means the UI validates changes before they hit production: - Static analysis of topic configs against policy - Schema compatibility checks before a producer deploys - Impact analysis for deletions ("this topic has 12 active consumers, are you sure?") - Integration with CI/CD so changes go through review before they merge, not after they break A common misconception: a shell script that enforces naming conventions is a guardrail. It isn't. If the platform team is still approving every request, handling every exception, and checking that the script even ran, what you've built is a weekend project moonlighting as a production control. Real automated guardrails are enforced by the system, declared by the platform team, and largely invisible to the developer who just wants to ship. ### Kafka-as-code The deepest form of autonomy isn't a UI form. It's Kafka-as-code. A good Kafka UI treats the interface as one client of its own API, and exposes the same operations through a CLI and an infrastructure-as-code provider. Same RBAC, same guardrails, same audit trail. Developers provision Kafka resources the way they provision everything else: a YAML file in Git, a pull request, a CI pipeline that applies on merge. That matters for two reasons. First, CI/CD integration. Topic and schema creation shouldn't live in a different universe from the rest of your infrastructure. If your team provisions AWS resources through Terraform and deploys services through GitHub Actions, Kafka resources should flow through the same pipeline, reviewed and approved by the same tooling, audited in the same place. Second, scale. At 50 topics, clicking through a UI is fine. At 500, it isn't. Teams running Kafka at real scale need declarative resource files in a Git repo, automated linting in CI, and pull requests gating production changes. A UI that can't be driven from the outside caps your team at a size it won't stay at for long. What this looks like in practice: a topic defined in YAML, committed alongside the service that produces to it, rolled out by a pipeline that applies the YAML with a service account token. No tickets. No UI clicks. No platform team in the loop. The same policies that would reject a bad topic in the UI reject the pull request. The UI is the front door for interactive work. The CLI and API are the front door for everything else. A tool that only ships the first is one your platform team is going to outgrow. > Autonomy without validation is chaos with a better login screen. Autonomy without a CLI and an API is a ceiling. Go deeper: [self-service workflows in Console →](https://www.conduktor.io/console#federated-ownership) · [the `conduktor` CLI on GitHub →](https://github.com/conduktor/ctl) ## Kafka UI tools today A scan of the market (open-source and commercial), scored through the V/O/A lens. This is the short version. For a side-by-side feature table, see [Kafka UI Tools Compared](https://www.conduktor.io/compare/kafka-ui-tools). - **Kafbat UI** — Community-maintained continuation of Provectus kafka-ui, which Provectus paused in September 2023. Kafbat's fork is the active successor. Strong on Visibility (topics, messages, schemas, connectors). Limited Ownership (basic RBAC, no team namespacing). No Autonomy layer (no policy enforcement, no approval workflows). - **AKHQ** — Mature open-source tool. Solid Visibility across topics, consumer groups, schemas, Connect. Multi-cluster RBAC in recent versions. No ownership model or self-service workflows. - **Redpanda Console** — Polished UI, optimized for Redpanda but works with Apache Kafka. Good Visibility and UX. No ownership or autonomy layer for platform-scale use. - **Confluent Control Center** — First-party UI from Confluent, bundled with Confluent Platform. Deep integration with the Confluent stack including Schema Registry, ksqlDB, and Connect. RBAC ships in Confluent Platform Enterprise (not the Community edition). No catalog-based ownership model or automated guardrails for self-service. Best fit for teams standardized on Confluent. - **Kpow** — Commercial Kafka UI from Factor House. Strong on multi-distribution support and operational features. Decent Ownership tooling. Autonomy is more limited than dedicated platform-team products. - **Lenses.io** — Commercial Kafka management UI. Signature feature is SQL-based topic querying. RBAC available in paid tiers (none in Community). Topic approval workflows and environment-based namespace permissions, enforced through Lenses' own interfaces rather than at a proxy or wire layer. No native chargeback or cost attribution. Three older tools (Kafka Manager / CMAK, Offset Explorer, and Provectus kafka-ui) aren't actively developed anymore. They're covered in the [migration section below](#migrating-off-a-deprecated-tool). **Where to go from here:** every open-source Kafka UI delivers some version of Visibility. What separates them from each other, and from the commercial options, is how seriously they take Ownership and Autonomy. That's not the kind of thing a scorecard resolves. It depends on your team size, your compliance requirements, and how many clusters you run. [Kafka UI Tools Compared](https://www.conduktor.io/compare/kafka-ui-tools) walks through the feature-by-feature matrix if you want to go deep. Migrating off Kafka Tool, Offset Explorer, Kafka Manager, or Provectus A few of the most-searched Kafka UI tools aren't being actively developed. If your team is on any of these, you're on a deprecation timeline whether you've noticed or not: - **Kafka Tool → Offset Explorer.** Same tool, renamed a few years back. Desktop app, single-user, no team features. Still functional, still sold, but hasn't added anything material for teams in years. - **Kafka Manager / CMAK.** Yahoo's original Kafka UI. Still technically maintained (release 3.0.0.6 shipped in 2025 after a ~10-month gap), but the cadence is slow enough that most teams have moved on. If you're running it, treat it as a tool to plan off of rather than one to bet on. - **Provectus kafka-ui.** Provectus announced a pause on active development in September 2023. The project continues as a community fork at `github.com/kafbat/kafka-ui` (same project name, new org). If your docs still point to the old `provectus/kafka-ui` repo, update them, because the active work is happening on the Kafbat fork. ### What to evaluate when you migrate The mistake most teams make is migrating one-for-one. They replace Offset Explorer with Kafbat, and end up with the same "single user, no ownership" gap dressed in newer CSS. The migration is the moment to ask whether you need Visibility only, or whether your team has outgrown that. Three honest questions before you pick a replacement: 1. **How many teams touch Kafka today, versus a year ago?** If the number grew, you need an Ownership story that didn't exist in your old tool. 2. **How often does the platform team get pulled into a Kafka issue that wasn't theirs to begin with?** That's the Autonomy tax. Measure it, and pick a tool that reduces it. 3. **What did your auditor ask for last time?** If you can't produce a "who modified this topic on March 14" record, the tool has to change, not just the UI. Multi-cloud Kafka UI Most teams don't run Kafka on one provider anymore. A typical platform has MSK in one AWS account, Confluent Cloud for a different business unit, a self-managed cluster on Kubernetes, and maybe a Redpanda instance for a specific workload. Open-source Kafka UIs can connect to multiple clusters. What they usually *can't* do is: - Show a unified view of topics across clusters with ownership metadata - Apply the same RBAC policy to MSK, Confluent, and self-managed clusters from one place - Route a consumer lag alert to the team that owns the topic, regardless of which cluster it's on - Produce a single audit trail that spans all of them A multi-cloud Kafka UI is only a multi-cloud Kafka UI if it treats your clusters as an estate, not as separate apps sharing a login page. Kafka UI vs CLI, kcat, and kafka-console-consumer The honest answer: you still need the CLI for some things. `kafka-topics.sh` and `kafka-console-consumer` are irreplaceable when you're debugging at the protocol level, running one-off scripts, or working on a cluster that's in a degraded state where the UI can't reach it. A Kafka UI isn't a CLI replacement. It's a force multiplier for everyone who shouldn't *have* to use the CLI. Your senior Kafka engineers will keep using it for specialized work. Your application developers should not need to use it to figure out why their consumer group is lagging. The practical rule: UI for routine debugging, configuration, and monitoring. CLI for protocol-level investigation and emergency recovery. Any team insisting developers learn the CLI to do routine work is paying a tax that a decent UI eliminates. ## Where Conduktor Console fits Conduktor Console is a Kafka UI designed around all three pillars from the start, not bolted on after the fact. Every capability on the Console page maps to one of them. - **Visibility** — Core Kafka operations for topics, schemas, and connectors. Data exploration with message browsing, filtering, and decoding. Observability with real-time lag, alerting, and Prometheus integration. Across every cluster you run. [Explore Visibility →](https://www.conduktor.io/console#simplify-kafka-operations) - **Ownership** — SSO, RBAC, and field-level masking for people. Insights and chargeback for cost attribution. Federated ownership catalogs for topics, schemas, and applications. Audit trails shippable to your SIEM. [Explore Ownership →](https://www.conduktor.io/console#secure-without-slowing-down) - **Autonomy** — Self-service within platform-defined guardrails. Every operation exposed through CLI, REST API, and a Terraform provider, so Kafka fits into the same CI/CD pipelines as the rest of your infrastructure. Automated validation and self-healing connectors so developers recover without tickets. [Explore Autonomy →](https://www.conduktor.io/console#federated-ownership) ### The same operations, three surfaces Console exposes every resource through the UI, a CLI, and a REST API. Same RBAC, same guardrails, same audit trail across all three. Your team picks the surface that fits each workflow. - **UI.** Interactive work, debugging, and the 90% of everyday operations. - **CLI.** `conduktor apply -f ./resources/` applies topics, schemas, users, groups, RBAC policies, connectors, and data quality rules from YAML. Bulk operations, state tracking, dry-run mode, remote state in S3, GCS, or Azure Blob. Installable via Homebrew or Docker. Source at [github.com/conduktor/ctl](https://github.com/conduktor/ctl). - **REST API.** 20+ resource kinds, OpenAPI-documented, Bearer-token auth. Live at `/docs` on every deployment. - **Terraform provider.** `conduktor/conduktor` on the Terraform registry. Same resources, same RBAC, declarative IaC workflow. - **MCP server.** Read-only cluster context for AI assistants. List clusters, inspect topics and schemas, preview messages, surface insights, without handing an AI a platform-team credential. Continue reading [Kafka UI tools comparedSide-by-side feature comparison of Kafbat, AKHQ, Redpanda Console, and Conduktor.](https://www.conduktor.io/compare/kafka-ui-tools) [Conduktor ConsoleFull capability breakdown with videos and screenshots.](https://www.conduktor.io/console) [Kafka security guideThe four-pillar security framework that underpins Ownership.](https://www.conduktor.io/kafka-security) ## Read more customer stories - [FlixBus: Self-Service for 50+ Teams](https://www.conduktor.io/customer-stories/flix) - [Swiss Post: 5x Kafka Growth with 800 Users](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Virgin Australia: 300 Hours/Month Saved](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) ## Frequently Asked Questions **Is there a free Kafka UI?** Yes, several. Kafbat UI (the fork of Provectus kafka-ui) is the most feature-complete open-source option. AKHQ is a strong alternative, particularly if you already use Java-based tooling. Redpanda Console works with Apache Kafka, not just Redpanda. All three are fine for single-developer or single-team use. They stop being enough once you need team ownership, RBAC at scale, or self-service with guardrails. **What is the best Kafka UI?** The honest answer is "it depends on your team's stage." Solo developers and small teams usually do fine with Kafbat or AKHQ. Teams with 5+ developers sharing a cluster start running into ownership and access control gaps. Platform teams supporting 20+ teams need a tool with federated ownership and policy enforcement, which is where the open-source options tap out. **Does Confluent have a UI?** Yes, Confluent Control Center is their first-party Kafka UI. It's bundled with Confluent Platform and works best if you're fully on the Confluent stack. It's less commonly used with self-managed Apache Kafka, MSK, or Redpanda, which is the gap most multi-vendor teams are trying to close. **What happened to Provectus kafka-ui?** Development was paused by Provectus. The open-source community maintains a fork under the name Kafbat UI, which continues active development. If you're running Provectus kafka-ui today, plan a migration to Kafbat or an enterprise alternative. **Is Kafka Manager (CMAK) still maintained?** Not meaningfully. Yahoo's original Kafka Manager was renamed to CMAK and handed off; it hasn't seen a substantive release in years. If you're still on it, you're maintaining an orphan. **What's the difference between Kafka Tool and Offset Explorer?** They're the same product. The tool was renamed from Kafka Tool to Offset Explorer in 2020 and is still sold as a desktop application. It's single-user and not designed for team-scale use. **What's the difference between a Kafka UI and a CLI?** The CLI (kafka-topics.sh, kafka-console-consumer, kcat) works at the protocol level and is ideal for scripting, one-off debugging, and emergency recovery. A Kafka UI is a web interface that makes the same operations approachable for developers who don't specialize in Kafka. A good UI doesn't replace the CLI. It means 90% of your team doesn't have to learn it. **Does Conduktor Console run in Docker?** Yes. Console ships as container images and includes a docker-compose quickstart. See the [get-started guide](https://www.conduktor.io/get-started?src=kafka-ui) for the current setup. **Can I search Kafka messages by field value in the UI?** Yes, in Conduktor Console. You can filter by key, value, header, timestamp, and use JQ expressions against decoded JSON or Avro payloads. Most mature Kafka UIs support this; the depth varies. **Does Kafka have a native web UI?** No. Apache Kafka ships with CLI tools only. Every Kafka web UI (Kafbat, AKHQ, Conduktor Console, Confluent Control Center) is a separate project on top of Kafka. **Can one Kafka UI manage MSK, Confluent Cloud, and self-managed Kafka together?** The open-source options can connect to all three but show them as separate clusters with no unified ownership or policy layer. Conduktor Console treats them as one estate, with the same RBAC, ownership model, and audit trail across all of them. **I have more questions.** [Drop us a line](https://www.conduktor.io/contact?src=kafka-ui) and we'll get back to you. ## Ready to see a Kafka UI built for platform teams? In a 30-minute demo, we'll walk through your current Kafka operations and show you where Conduktor Console closes the gaps in visibility, ownership, and autonomy, without changing how your developers work. [Talk to Us](https://www.conduktor.io/contact/demo?src=kafka-ui) [Explore Console →](https://www.conduktor.io/console) Source: https://www.conduktor.io/kafka-ui --- # Learn Apache Kafka | Conduktor Kafkademy Learn Apache Kafka, *from scratch to production*. Free Apache Kafka tutorials, hands-on labs, and production playbooks. From your first topic to running brokers under load. Patient, opinionated, and built by [the team behind Conduktor](https://www.conduktor.io/about). Read in any order, skip what you already know. Where are you starting from? I'm new to KafkaStart from zero. We'll build the vocabulary. I've used it a bitSkip the basics, jump to CLI and code. I run it in productionPerformance, security, observability, on-call. [Start with the fundamentals →](https://www.conduktor.io/kafka/what-is-apache-kafka) [Open the hands-on track →](https://www.conduktor.io/kafka/how-to-start-kafka-with-conduktor) [Go advanced →](https://www.conduktor.io/kafka/kafka-topics-advanced) [Open the Kafka Options Explorer ↗](https://kafka-options-explorer.conduktor.io/) Lessons 101 Sign-up Not required Cost Free, forever 01 Understand it, do it, run it. Most Kafka courses pick one of these. Ours runs all three in parallel, because you don't really know Kafka until you've broken it on your laptop. 01 Learn : Build the mental model The vocabulary, the diagrams, the why. No setup, no Docker. Read on the train. ○ What is Apache Kafka? ○ Topics, partitions and offsets ○ Producers and consumers ○ Brokers and topic replication ○ KRaft and ZooKeeper 10 chapters, no install required [Start with the fundamentals →](https://www.conduktor.io/kafka/what-is-apache-kafka) 02 Practice : Get your hands dirty Install Kafka on your laptop. Drive it from the CLI, then from Java. ○ Install Kafka on Mac, Linux or Docker ○ Manage topics from the CLI ○ Drive producers and consumers via CLI ○ Build a Java producer and consumer ○ Wire up a Maven or Gradle project 24 hands-on guides across install, CLI and code [Open the hands-on track →](https://www.conduktor.io/kafka/how-to-start-kafka-with-conduktor) 03 Master : Run it like you mean it The settings that make or break an on-call shift. Tune, secure, monitor, scale. ○ Topic internals and log compaction ○ Producer acks, idempotence, batching ○ Consumer delivery semantics ○ Security, monitoring, multi-cluster ○ Min ISR and unclean leader election 31 deep dives on topics, producers, consumers, ops [Go advanced →](https://www.conduktor.io/kafka/kafka-topics-advanced) New track: 36 lessons Kafka Streams, end to end Go past producers and consumers. Build stateful stream processing (joins, aggregations, windowing), then run it in production, with the rebalancing, state restore, and exactly-once failures the official tutorials skip. The mental model: KStream vs KTable, state stores, windowing, joins The production playbook: slow rebalances, RocksDB memory, the suppress() trap, topology evolution Honest comparisons: Kafka Streams vs Flink vs ksqlDB [Start the Kafka Streams track →](https://www.conduktor.io/kafka-streams) Reference tool Kafka Options Explorer Compare Kafka configs across every version. Read KIPs, decode error codes, plan upgrades. Compare broker, producer, consumer and connect configs side by side Browse every KIP with summaries and status Generate upgrade reports between any two versions Look up error codes and wire protocol changes [Open the explorer →](https://kafka-options-explorer.conduktor.io/) Done with Kafkademy? Put what you learned into a real Kafka stack. Conduktor Console gives you topic management, message browsing, schema control and access policies, on top of any Kafka cluster. Free tier, no card needed. [Try Conduktor Console →](https://www.conduktor.io/get-started?src=kafka) Source: https://www.conduktor.io/kafka --- # Cookie Policy Updated March 11th, 2020 If you don’t wish to accept cookies from us, you should instruct your browser to refuse cookies from https://www.conduktor.io/, with the understanding that we may be unable to provide you with some of your desired content and services. ### What is a cookie? A cookie is a small piece of data that a website stores on your device when you visit, typically containing information about the website itself, a unique identifier that allows the site to recognize your web browser when you return, additional data that serves the purpose of the cookie, and the lifespan of the cookie itself. Cookies are used to enable certain features (eg. logging in), to track site usage (eg. analytics), to store your user settings (eg. timezone, notification preferences), and to personalize your content (eg. advertising, language). Cookies set by the website you are visiting are normally referred to as “first-party cookies”, and typically only track your activity on that particular site. Cookies set by other sites and companies (ie. third parties) are called “third-party cookies”, and can be used to track you on other websites that use the same third-party service. ### Types of cookies and how we use them #### Essential cookies Essential cookies are crucial to your experience of a website, enabling core features like user logins, account management, shopping carts and payment processing. We use essential cookies to enable certain functions on our website. #### Performance cookies Performance cookies are used in the tracking of how you use a website during your visit, without collecting personal information about you. Typically, this information is anonymous and aggregated with information tracked across all site users, to help companies understand visitor usage patterns, identify and diagnose problems or errors their users may encounter, and make better strategic decisions in improving their audience’s overall website experience. These cookies may be set by the website you’re visiting (first-party) or by third-party services. We use performance cookies on our site. #### Functionality cookies Functionality cookies are used in collecting information about your device and any settings you may configure on the website you’re visiting (like language and time zone settings). With this information, websites can provide you with customized, enhanced or optimized content and services. These cookies may be set by the website you’re visiting (first-party) or by third-party service. We use functionality cookies for selected features on our site. #### Targeting/advertising cookies Targeting/advertising cookies are used in determining what promotional content is more relevant and appropriate to you and your interests. Websites may use them to deliver targeted advertising or to limit the number of times you see an advertisement. This helps companies improve the effectiveness of their campaigns and the quality of content presented to you. These cookies may be set by the website you’re visiting (first-party) or by third-party services. Targeting/advertising cookies set by third-parties may be used to track you on other websites that use the same third-party service. We do not use this type of cookie on our site. ### Third-party cookies on our site We may employ third-party companies and individuals on our websites—for example, analytics providers and content partners. We grant these third parties access to selected information to perform specific tasks on our behalf. They may also set third-party cookies in order to deliver the services they are providing. Third-party cookies can be used to track you on other websites that use the same third-party service. As we have no control over third-party cookies, they are not covered by Conduktor's cookie policy. #### Our third-party privacy promise We review the privacy policies of all our third-party providers before enlisting their services to ensure their practices align with ours. We will never knowingly include third-party services that compromise or violate the privacy of our users. ### How you can control or opt out of cookies If you do not wish to accept cookies from us, you can instruct your browser to refuse cookies from our website. Most browsers are configured to accept cookies by default, but you can update these settings to either refuse cookies altogether, or to notify you when a website is trying to set or update a cookie. If you browse websites from multiple devices, you may need to update your settings on each individual device. Although some cookies can be blocked with little impact on your experience of a website, blocking all cookies may mean you are unable to access certain features and content across the sites you visit. This website is operated by Conduktor Inc. If you have any questions, complaints, or concerns about how your information is handled, or would like to exercise any of your rights for yourself or on behalf of someone else, please email us at #### support@conduktor.io Source: https://www.conduktor.io/legal/cookie --- # Data Processing Agreement (SaaS) Updated June 22, 2026 **DATA PROCESSING AGREEMENT — CONDUKTOR IVY HOSTED (SAAS) PLATFORM** This Data Processing Agreement ("**DPA**") sets out the terms and conditions under which Conduktor Inc. ("**Processor**" or "**Conduktor**") Processes Personal Data on behalf of the Customer ("**Controller**") in connection with Conduktor's hosted, cloud-based "Ivy" platform and related services (the "SaaS Services" as defined in the Agreement, referred to in this DPA as the "Services"), as governed by the [End User License Agreement: SaaS Services](https://www.conduktor.io/legal/eula-saas) or other applicable agreement between the parties (the "Agreement"). Unless otherwise defined in this DPA, capitalized terms have the meanings given to them in the Agreement. In the event of any conflict between this DPA and the Agreement, the terms of this DPA prevail solely with respect to the Processing of Personal Data and Covered Data, and shall not modify or supersede any commercial terms of the Agreement (including fees, remedies, indemnities, or limitation of liability), except to the extent directly required to comply with Applicable Data Protection Laws. **1. DEFINITIONS** Capitalized terms used but not defined within this DPA have the meaning set forth in the Agreement. The following terms used in this DPA are defined as follows: "**Applicable Data Protection Laws**" means all applicable laws, rules, regulations, and governmental requirements relating to the privacy, confidentiality, or security of Personal Data, including the GDPR, the UK GDPR, Swiss Data Protection Laws, and US State Privacy Laws, as they may be amended or updated from time to time. "**Controller Affiliate**" means an affiliate of Controller who is a beneficiary to the Agreement. "**Covered Data**" means Personal Data that is: (a) provided by or on behalf of Controller to Processor in connection with the Services; or (b) obtained, developed, produced, or otherwise Processed by Processor, or its agents or Sub-processors, for purposes of providing the Services. "**Data Subject**" means a natural person whose Personal Data is Processed, including (where protected under Applicable Data Protection Laws) a household or consumer. For clarity, the Services are intended for use by Controller's Authorized Users and other business personnel, and Covered Data is expected to relate to such individuals and any other individuals whose Personal Data is contained in data Controller submits to the Services. "**Deidentified Data**" means data created using Covered Data that cannot reasonably be linked to such Covered Data, directly or indirectly. "**EEA**" means the European Economic Area, including the European Union ("EU"). "**GDPR**" means Regulation (EU) 2016/679 (the "EU GDPR") or, where applicable, the "UK GDPR" as it forms part of the law of England and Wales, Scotland and Northern Ireland by virtue of section 3 of the UK European Union (Withdrawal) Act 2018, or, where applicable, the equivalent provision under Swiss Data Protection Laws. "**Member State**" means a member state of the EEA, being a member state of the European Union, Iceland, Norway, or Liechtenstein. "**Personal Data**" means any data or information that: (a) is linked or reasonably linkable to an identified or identifiable natural person; or (b) is otherwise "personal data," "personal information," "personally identifiable information," or similarly defined data under Applicable Data Protection Laws. "**Processing**" means any operation or set of operations performed on Personal Data, whether or not by automated means. "Process," "Processes," and "Processed" are interpreted accordingly. "**Security Incident**" means a confirmed or reasonably suspected breach of security leading to the accidental or unlawful destruction, loss, alteration, unauthorized disclosure of, or unauthorized access to (including unauthorized internal access to) Covered Data. "**Services**" means the SaaS Services (as defined in the Agreement), being the hosted Ivy platform and related services provided by Processor pursuant to the Agreement. "**Standard Contractual Clauses**" or "**SCCs**" means Module Two (controller to processor) and/or Module Three (processor to processor) of the Standard Contractual Clauses annexed to Commission Implementing Decision (EU) 2021/914. "**Sub-processor**" means an entity appointed by Processor to Process Covered Data on its behalf. "**UK**" means the United Kingdom. "**US State Privacy Laws**" means, to the extent applicable, US state laws relating to data protection, privacy, and the Processing of Personal Data, including the California Consumer Privacy Act as amended by the California Privacy Rights Act (collectively, the "CCPA/CPRA"), the Virginia Consumer Data Protection Act, the Colorado Privacy Act, the Connecticut Data Privacy Act, the Utah Consumer Privacy Act, and other comparable US state privacy laws in force from time to time. **2. INTERACTION WITH THE AGREEMENT** 2.1 This DPA is incorporated into and forms an integral part of the Agreement. This DPA supplements and (in case of contradictions) supersedes the Agreement solely with respect to the Processing of Covered Data and Personal Data, and shall not modify or supersede any commercial terms of the Agreement (including fees, remedies, indemnities, or limitation of liability), except to the extent directly required to comply with Applicable Data Protection Laws. 2.2 Any Processing operation as described in clause 4 (Details of Data Processing) and Schedule 1 to this DPA will be subject to this DPA. 2.3 Controller Affiliates will be beneficiaries under this DPA and, through Controller, be entitled to enforce all rights in relation to Covered Data provided by the respective Affiliate. Controller will ensure that all obligations under this DPA are passed on to the respective Controller Affiliate. 2.4 Controller warrants that it is duly mandated by any Controller Affiliates on whose behalf Processor Processes Covered Data to: (a) enforce the terms of this DPA on behalf of Controller Affiliates and act on their behalf in the administration and conduct of any claims arising in connection with this DPA; and (b) receive and respond to any notices or communications under this DPA on their behalf. 2.5 Controller will be the only point of contact for all communication between Controller Affiliates and Processor. **3. ROLE OF THE PARTIES** The parties acknowledge and agree that, in respect of the hosted Services: (a) for the purposes of the GDPR, UK GDPR, and Swiss Data Protection Laws, Processor acts as "processor" or "sub-processor," determined by the function of Controller: where Controller acts as a controller, Processor acts as a processor; where Controller acts as a processor on behalf of another controller, Processor acts as a sub-processor; (b) for the purposes of the US State Privacy Laws, Processor acts as a "service provider" or "processor" (as defined in the applicable US State Privacy Laws) in performing its obligations under the Agreement and this DPA; and (c) Controller is the controller (or business) with respect to Covered Data it submits to or Processes through the Services, and determines the purposes and means of Processing of such Covered Data, and is responsible for the accuracy, quality, and legality of Covered Data and the means by which it was acquired (including providing appropriate notices and obtaining consents where required), and for establishing a lawful basis for the Processing. **Hosted Processing acknowledgement.** Controller acknowledges that, unlike Conduktor's on-premises offering, the Services are hosted by Processor and that Processor will Process Covered Data submitted to the Services on Controller's behalf as described in Schedule 1. Controller has full control over what Personal Data it inputs into or uploads to the Services. Controller is responsible for ensuring that access to, and use of, the Services (and the submission of Covered Data) is limited to Authorized Users and other permitted business users under the Agreement, and that Controller does not knowingly submit Covered Data relating to individuals who are not within Controller's permitted use case except as otherwise permitted by Applicable Data Protection Laws. **4. DETAILS OF DATA PROCESSING** 4.1 The details of the Processing of Personal Data under the Agreement and this DPA (such as subject matter, nature and purpose of the Processing, categories of Personal Data and Data Subjects) are described in the Agreement and in Schedule 1 to this DPA. 4.2 Covered Data will only be Processed on behalf of and under the documented instructions of Controller and in accordance with Applicable Data Protection Laws. Processor shall Process Covered Data only as necessary to perform the Services. The Agreement and this DPA generally constitute Controller's instructions for the Processing of Covered Data. Controller may issue further written instructions in accordance with this DPA. Processor will inform Controller if, in its opinion, an instruction infringes Applicable Data Protection Laws. Without limiting the foregoing, Processor is prohibited from: - selling Covered Data or otherwise making Covered Data available to any third party for monetary or other valuable consideration; - sharing Covered Data with any third party for cross-context behavioral advertising; - retaining, using, or disclosing Covered Data for any purpose other than the business purposes specified in the Agreement or as otherwise permitted by Applicable Data Protection Laws; - retaining, using, or disclosing Covered Data outside of the direct business relationship between the parties; and - except as otherwise permitted by Applicable Data Protection Laws, combining Covered Data with Personal Data that Processor receives from or on behalf of another person, or collects from its own interaction with the Data Subject. 4.3 Processor certifies that it understands the restrictions in clause 4.2 and will comply with them. 4.4 Processor will limit access to Covered Data to personnel who have a business need to access it, and will ensure that such personnel are subject to obligations of confidentiality at least as protective of the Covered Data as the terms of this DPA and the Agreement. 4.5 Processor may, without prejudice to clause 11, Process Covered Data anywhere that Processor or its Sub-processors maintain facilities, subject to clause 5 of this DPA. The Services are hosted in the EEA (Ireland) as described in Schedule 1; certain Sub-processors are located in the United States as described in Schedule 5. 4.6 Processor will provide Controller with information reasonably necessary, taking into account the nature of the Processing and information reasonably available to Processor, to enable Controller to conduct and document any data protection impact assessments or transfer impact assessments required under Applicable Data Protection Laws, subject to Processor's confidentiality and security obligations. Processor will notify Controller promptly if it determines that it can no longer meet its obligations under Applicable Data Protection Laws. 4.7 Controller has the right to take reasonable and appropriate steps to ensure that Processor uses Covered Data in a manner consistent with Controller's obligations under Applicable Data Protection Laws, and to stop and remediate unauthorized use of Covered Data. **5. SUB-PROCESSORS** 5.1 Controller grants Processor general authorization to engage Sub-processors, subject to clause 5.2, as well as Processor's current Sub-processors listed in Schedule 5. 5.2 Processor will enter into a written agreement with each Sub-processor imposing data protection obligations that, in substance, are no less protective of Covered Data than Processor's obligations under this DPA. Processor remains fully liable to Controller for the performance of each Sub-processor's obligations to the same extent as if performed by Processor itself. Engagement of any Sub-processor does not relieve Processor of any of its obligations under this DPA. 5.3 Processor will provide Controller with at least fifteen (15) days' notice of any proposed changes to the Sub-processors it uses to Process Covered Data. Controller may object to a new Sub-processor (including when exercising its right to object under clause 9(a) of the SCCs, if applicable) by providing written notice within ten (10) days after Processor's notice (an "Objection"). If Controller does not object within the Objection period, consent will be assumed. If Controller objects, the parties will work together in good faith to find a mutually acceptable resolution. If no resolution is reached within a reasonable timeframe, either party may, as its sole and exclusive remedy, terminate the portion of the Agreement relating to the affected Services by written notice. During any such Objection period, Processor may suspend the affected portion of the Services. **6. DATA SUBJECT RIGHTS REQUESTS** 6.1 As between the parties, Controller has sole discretion and responsibility in responding to the rights asserted by any individual in relation to Covered Data under Applicable Data Protection Laws (each, a "Data Subject Request"). 6.2 Processor will promptly forward to Controller, without undue delay, any Data Subject Request it or any Sub-processor receives, and may advise the individual to submit the request directly to Controller. 6.3 Processor will provide Controller with reasonable assistance, taking into account the nature of the Processing and information reasonably available to Processor, as necessary for Controller to fulfil its obligation to respond to Data Subject Requests, including requests to access, correct, delete, or port Covered Data, or to opt out of its sale or sharing (to the extent applicable), subject to Processor's confidentiality and security obligations. **7. SECURITY AND AUDITS** 7.1 Processor will implement and maintain appropriate technical, administrative, and organizational measures designed to ensure the security of Covered Data, including protection against unauthorized or unlawful Processing and against accidental loss, destruction, or damage. When assessing the appropriate level of security, account will be taken of the nature, scope, context, and purpose of the Processing and the risks presented, in particular from accidental or unlawful destruction, loss, alteration, unauthorized disclosure of, or access to Covered Data. 7.2 Processor will implement and maintain, as a minimum standard, the measures set out in Schedule 2. Processor shall regularly review and update its technical and organizational measures to ensure they remain effective and appropriate in light of technological developments, evolving industry standards, and changes in the nature, scope, context, and purposes of Processing, and will notify Controller of any material adverse changes to such measures. 7.3 Controller has the right to audit Processor's compliance with this DPA. The parties agree that all such audits will be conducted: (a) upon reasonable written notice to Processor; (b) only once per year (except where required by a supervisory authority or following a Security Incident); and (c) only during Processor's normal business hours. 7.4 To conduct such audits, Controller may engage a third-party auditor who is suitably qualified, independent, and bound by appropriate confidentiality obligations. Controller must submit a detailed proposed audit plan at least two weeks in advance, and the parties will work cooperatively to agree a final audit plan. Audits must be conducted subject to the agreed plan and Processor's health, safety, security, and other relevant policies. Controller will promptly notify Processor of any non-compliance discovered. 7.5 Controller bears the costs of any audit it initiates, unless the audit reveals material non-compliance with this DPA. Upon request, Processor will provide documentation reasonably evidencing implementation of its technical and organizational measures. If the requested audit scope is addressed in a certification or third-party audit report (such as SOC 2 or ISO 27001) produced within twelve (12) months of the request, and Processor confirms no known material changes in the controls covered, Controller agrees to accept those findings in lieu of an on-site audit of the covered controls. 7.6 Processor will audit its Sub-processors on a regular basis and will, upon Controller's request, confirm their compliance with Applicable Data Protection Laws and their contractual obligations. **8. SECURITY INCIDENTS** 8.1 Processor will notify Controller in writing without undue delay and, in any event, no later than seventy-two (72) hours after becoming aware of any Security Incident, and will reasonably cooperate in any obligation of Controller under Applicable Data Protection Laws to make notifications to individuals or supervisory authorities. Processor will take reasonable steps to contain, investigate, and mitigate the Security Incident, and will provide Controller with timely information including the nature of the Security Incident, the measures taken to mitigate or contain it, and the status of the investigation. Processor's notification of or response to a Security Incident will not be construed as an acknowledgement of fault or liability. 8.2 Processor will provide reasonable assistance with Controller's investigation of a Security Incident and with any notification obligation of Controller under Applicable Data Protection Laws. **9. DELETION AND RETURN** 9.1 Processor will, within thirty (30) days of the termination or expiry of the Agreement: (a) if requested by Controller within that period, return a copy of all Covered Data or provide self-service functionality allowing Controller to do the same; and (b) delete all other copies of Covered Data Processed by Processor or any Sub-processor, except to the extent retention is required by applicable law, in which case Processor will protect the confidentiality of such Covered Data and Process it only as necessary for the purpose of such required retention. Covered Data submitted to the Services is otherwise subject to the default and maximum retention periods described in Schedule 1. **10. CONTRACT PERIOD** 10.1 This DPA becomes effective in parallel with the Agreement and, notwithstanding any termination of the Agreement, remains in effect until, and automatically expires upon, Processor's deletion of all Covered Data as described in this DPA. **11. STANDARD CONTRACTUAL CLAUSES AND INTERNATIONAL TRANSFERS** 11.1 The parties agree that the terms of the SCCs, Module Two (controller to processor) and Module Three (processor to processor), as further specified in Schedule 3, are incorporated by reference and deemed executed by the parties, and apply to any transfers of Covered Data falling within the scope of the GDPR from Controller (as data exporter) to Processor (as data importer). 11.2 To the extent applicable, the jurisdiction-specific addenda set out in Schedule 3 (including the UK Addendum and Swiss Addendum) are also incorporated by reference and deemed executed by the parties, and apply to transfers of Covered Data falling within the scope of Applicable Data Protection Laws in the listed jurisdiction(s). 11.3 Processor will provide Controller reasonable support, taking into account the nature of the Processing and information reasonably available to Processor, to enable Controller's compliance with the requirements imposed on international transfers of Covered Data, and will, upon request, provide information reasonably necessary for Controller to complete a transfer impact assessment ("TIA"), subject to Processor's confidentiality and security obligations. 11.4 Processor agrees to implement the supplementary measures set forth in Schedule 4 in order to enable Controller's compliance with requirements imposed on international transfers of Covered Data under Applicable Data Protection Laws. 11.5 Where Processor relies on the EU-US Data Privacy Framework (or its UK Extension or Swiss-US framework) as a transfer mechanism for any onward transfer to a US Sub-processor, it will maintain such certification or, where required under Applicable Data Protection Laws, ensure an alternative valid transfer mechanism applies. **12. US STATE PRIVACY LAW TERMS** 12.1 This clause applies to Processing of Covered Data subject to US State Privacy Laws. Processor acts as a "service provider" or "processor" and Processes Covered Data solely on Controller's behalf for the limited and specified business purposes set out in the Agreement and Schedule 1. 12.2 Processor will not: (a) sell or share Covered Data (as "sell" and "share" are defined under the CCPA/CPRA); (b) retain, use, or disclose Covered Data for any purpose other than the business purposes specified, including outside the direct business relationship between the parties; or (c) combine Covered Data with Personal Data received from or on behalf of another person, or collected from Processor's own interaction with the Data Subject, except as permitted by US State Privacy Laws. 12.3 Processor certifies that it understands and will comply with the restrictions in this clause 12. Controller may take reasonable and appropriate steps to ensure that Processor uses Covered Data consistently with Controller's obligations under US State Privacy Laws, and to stop and remediate any unauthorized use. 12.4 Processor will notify Controller if it determines it can no longer meet its obligations under US State Privacy Laws, and will assist Controller in responding to verifiable consumer rights requests as set out in clause 6. **13. DEIDENTIFIED DATA** If Processor receives Deidentified Data from or on behalf of Controller, then Processor will: - take reasonable measures to ensure the information cannot be associated with a Data Subject; - publicly commit to Process the Deidentified Data solely in deidentified form and not to attempt to reidentify the information; and - contractually obligate any recipients of the Deidentified Data to comply with the foregoing requirements and Applicable Data Protection Laws. **14. GENERAL** 14.1 The parties certify that they understand the requirements in this DPA and will comply with them. 14.2 The parties agree to negotiate in good faith any amendments to this DPA as may be required in connection with changes in Applicable Data Protection Laws. 14.3 This DPA and the Agreement set forth the entire agreement between the parties with respect to the subject matter hereof. **SCHEDULE 1: DETAILS OF PROCESSING** **A. List of Parties** The parties are set out in the preamble to this DPA. With regard to any transfers of Covered Data falling within the scope of the GDPR from Controller to Processor, additional information regarding the data exporter and data importer is set out below. **Data Exporter:** each of the Controller and/or Controller Affiliates operating in the countries comprising the EEA, UK, and/or Switzerland and/or, to the extent agreed by the parties, Controller and/or Controller Affiliates in any other country to the extent the GDPR applies. The data exporter's contact person, position, and contact details (and, if appointed, its data protection officer and/or representative) are included in the Agreement or will be disclosed to Processor upon request. The activities relevant to the data transfer are defined by the Agreement, and the data exporter decides on the scope of the Processing in connection with the Services as further described in section B below. **Data Importer:** Conduktor Inc., the Processor. The data importer's activities relevant to the data transfer are: hosting, storing, and Processing Personal Data submitted by the data exporter through the Ivy platform on behalf of the data exporter in connection with providing the Services, as further described in section B below and in the Agreement. The data importer's contact details are included in the Agreement or will be disclosed to Controller upon request. **B. Description of Processing** **Hosting model.** The Services are hosted on Amazon Web Services in the eu-west-1 (Ireland) region. The Controller has full control over what Personal Data it submits to the Services. Personal Data is held in a managed PostgreSQL database with tenant isolation enforced at the application layer. **Categories of Data Subjects:** Authorized Users and other personnel of Controller and Controller Affiliates; and, to the extent Controller submits such data, Controller's customers and their personnel, and other individuals whose Personal Data is contained in data submitted to the Services. **Categories of Personal Data:** - **Account and identity data:** email address, first and last name, profile picture, organization name, authentication identifiers, and session data. - **Access and audit data:** audit-log records of mutations, including actor email address, request identifier, user agent, requested path, status code, and duration; and last-seen timestamps. - **Customer-submitted data:** any Personal Data contained within streams, catalog entries, projects, schema versions, expositions, and associated metrics that Controller chooses to submit to the Services. Conduktor does not control and has limited visibility into the content of customer-submitted payload data. **Special categories of Personal Data:** None are required by the Services. Controller must not submit special-category Personal Data unless separately agreed in writing; if Controller chooses to submit such data, Controller is solely responsible for ensuring an appropriate lawful basis and any additional safeguards. **Nature and purpose of Processing:** hosting, storage, transmission, access management, authentication, audit logging, and maintenance of the data Controller submits to the Services, in order to provide the Services. **Frequency of Processing:** continuous, for the duration of the Agreement. **Retention / Storage limitation:** - Audit-log records: per workspace, default seven (7) days, maximum two (2) years (configurable per workspace). - Customer-submitted payload data (streams): default seven (7) days, maximum one (1) year (configurable per stream). - Sessions, invitation links, and device codes: short-lived; auto-expire. - Operational logs (used to diagnose incidents, not customer data): retained two (2) to four (4) weeks, then automatically deleted. - Otherwise, where Personal Data is not deleted on Controller's request during the term, the duration of Processing corresponds to the duration of this DPA as defined in clause 10. **Competent Supervisory Authority:** where the data exporter is established in an EU Member State, the supervisory authority of that Member State; where the data exporter is not established in an EU Member State but falls within the territorial scope of Article 3(2) GDPR and has appointed a representative under Article 27(1), the supervisory authority of the Member State where the representative is established; otherwise, the supervisory authority of Ireland (the Data Protection Commission). **SCHEDULE 2: TECHNICAL AND ORGANIZATIONAL MEASURES** Processor has implemented the following technical and organizational measures (including relevant certifications) to ensure an appropriate level of security, taking into account the nature, scope, context, and purpose of the Processing and the risks to the rights and freedoms of natural persons. These measures reflect the hosted Ivy architecture. - **Encryption at rest:** Customer data is stored in a managed relational database encrypted at rest. Production environments use a customer-managed encryption key to support key rotation, revocation, and audit drills. - **Encryption in transit:** TLS is enforced for all network traffic to and within the Services, using certificates managed by the cloud provider's certificate manager. - **Session and credential security:** Application sessions use sealed cookies (authenticated encryption: AES-256-GCM with HKDF), marked httpOnly and Secure, with origin-binding prefixes on administrative surfaces. API and machine tokens are stored only in hashed form and never in clear text. Secrets are held in a dedicated secrets manager and never in client-side storage. - **Access controls:** Logical access to systems and data is granted on a least-privilege, need-to-know basis using unique identifiers, with periodic review and prompt revocation on role change or termination. Cloud access uses short-lived, federated credentials (OIDC) with no long-lived static credentials in continuous integration. - **Tenant isolation:** Every query enforces a tenant identifier at the application layer, and bearer tokens cannot reach internal-only administrative interfaces. - **Audit logging and monitoring:** System and application event logging records user access and system activity for routine review, including an immutable per-tenant audit log of mutations. - **Vulnerability and patch management:** Vulnerability assessment, patch management, and threat-protection processes are used to identify, assess, mitigate, and protect against security threats and malicious code. - **Change management:** Documented procedures are used to test, approve, and monitor changes to technology and information assets, including infrastructure-as-code review. - **Network security:** Network controls (including load-balancer, firewall, and traffic-inspection mechanisms) protect systems from intrusion and limit the scope of any successful attack. - **Incident management:** Documented incident and problem-management procedures allow Processor to investigate, respond to, mitigate, and notify of events affecting its technology and information assets. - **Physical and environmental security:** Hosting is provided by a major cloud infrastructure provider whose data centers implement physical and environmental safeguards, including access control, monitoring, and protection against environmental hazards. - **Governance:** Organizational management and dedicated staff are responsible for the development, implementation, and maintenance of Processor's information security program, including periodic risk assessment and review. - **Resilience and recovery:** Business-continuity and disaster-recovery procedures are designed to maintain or restore the Services following foreseeable emergency situations. - **Secure disposal:** Operational procedures provide for secure disposal of systems and media so that information is rendered undecipherable or unrecoverable prior to disposal or release from Processor's possession. **SCHEDULE 3: STANDARD CONTRACTUAL CLAUSES AND ADDENDA** **Part A: EU Standard Contractual Clauses** The SCCs apply to any Processing of Covered Data subject to the EU GDPR. For the purposes of the SCCs: - Module Two applies to Processing under clause 3(a) of this DPA where Controller is a controller, and Module Three applies where Controller is a processor on behalf of another controller. - Clause 7 of the SCCs (Docking Clause) does not apply. - Clause 9(a), Option 2 (General written authorization) is selected; the time period to be specified is determined in clause 5.3 of this DPA. - The option in Clause 11(a) of the SCCs (Independent dispute resolution body) does not apply. - With regard to Clause 17 (Governing law), Option 1 applies and the governing law is the law of the Republic of Ireland. - In Clause 18 (Choice of forum and jurisdiction), the parties submit to the jurisdiction of the courts of the Republic of Ireland. - For Annex I of the SCCs, Schedule 1 of this DPA contains the specifications regarding the parties, the description of transfer, and the competent supervisory authority. - For Annex II of the SCCs, Schedule 2 of this DPA contains the technical and organizational measures. - For Annex III of the SCCs, the list of Sub-processors is determined by clause 5.1 and Schedule 5 of this DPA. **Part B: UK Addendum** This UK Addendum applies to any Processing of Covered Data subject to the UK GDPR or to both the UK GDPR and the EU GDPR. As used herein, "Approved Addendum" means the template addendum, version B.1.0, issued by the UK Information Commissioner under s.119A(1) of the Data Protection Act 2018 and laid before the UK Parliament on 2 February 2022, as it may be revised under Section 18 of the Mandatory Clauses; and "Mandatory Clauses" means Part 2 of the Approved Addendum. With respect to any transfers of Covered Data falling within the scope of the UK GDPR from Controller (as data exporter) to Processor (as data importer): (a) the Approved Addendum forms part of this DPA and the SCCs are read and interpreted in light of its provisions; (b) the parties are as specified in Schedule 1; (c) the selected Modules and Clauses are as specified in Part A above as amended by the Mandatory Clauses; (d) Tables 1 to 3 of the Approved Addendum are completed by reference to Schedules 1, 2, and 5 of this DPA; (e) Processor (as data importer) may end the DPA, to the extent the Approved Addendum applies, in accordance with Clause 19 of the Mandatory Clauses; and (f) Clause 16 of the Mandatory Clauses does not apply. **Part C: Swiss Addendum** This Swiss Addendum applies to any Processing of Covered Data subject to Swiss Data Protection Laws or to both Swiss Data Protection Laws and the GDPR. "Swiss Data Protection Laws" means the Swiss Federal Act on Data Protection and its implementing ordinance, as revised from time to time. This Addendum is read and interpreted in light of Swiss Data Protection Laws so that it provides the appropriate safeguards required by Article 46 GDPR and/or Article 6(2)(a) of the Swiss Data Protection Laws, as applicable, and will not be interpreted in a way that conflicts with rights and obligations under Swiss Data Protection Laws. In the event of conflict between this Addendum and the SCCs or related agreements, the provisions providing the most protection to Data Subjects prevail. To the extent Processing is exclusively subject to Swiss Data Protection Laws, the SCCs are amended so that: (a) references to the GDPR are replaced by references to Swiss Data Protection Laws; (b) references to the "EU," "Union," and "Member State" are replaced with "Switzerland"; (c) the competent supervisory authority is the Swiss Federal Data Protection and Information Commissioner; (d) Clause 17 is replaced so the SCCs are governed by the laws of Switzerland; (e) Clause 18 is replaced so disputes are resolved by the courts of Switzerland and a Data Subject may bring proceedings in the place of habitual residence; and (f) until the entry into force of the revised Swiss Data Protection Laws, the SCCs also protect the data of legal entities. Where Processing is subject to both Swiss Data Protection Laws and the GDPR, the DPA and SCCs apply as-is and additionally as amended above, except that Clause 17 is not replaced. Controller warrants that it and/or Controller Affiliates have made any notifications to the Commissioner required under Swiss Data Protection Laws. **Part D: US Transfers** For onward transfers of Covered Data to Sub-processors located in the United States, Processor may rely on one or more of the following, as applicable: the EU-US Data Privacy Framework (and its UK Extension and Swiss-US framework) where the relevant Sub-processor is certified; the SCCs together with the supplementary measures in Schedule 4; or another valid transfer mechanism under Applicable Data Protection Laws. **SCHEDULE 4: ADDITIONAL SUPPLEMENTARY MEASURES** Processor commits to implementing the following supplementary measures, based on guidance from EU supervisory authorities, to enhance the protection of Covered Data in connection with Processing in a third country. **Technical measures** - **Encryption in transit.** Personal Data transmitted between the parties, between Processor data centers, and to and from Sub-processors uses strong, state-of-the-art transport encryption with trustworthy public-key infrastructure and protections against active and passive attacks, including testing for software vulnerabilities. Where transport encryption alone is insufficient, application-layer encryption is applied. - **Encryption at rest.** Personal Data at rest is stored using strong encryption. Encryption algorithms and parameters conform to the state of the art and are robust against cryptanalysis, taking into account the resources available to public authorities and the period during which confidentiality must be preserved. Keys are reliably generated, administered, stored, and revoked by Processor or a trusted entity. **Organizational measures** - Adoption of internal policies for the governance of transfers, with clear allocation of responsibilities, reporting channels, and standard operating procedures for handling formal or informal requests from public authorities to access data, including notification to senior legal management and to Controller, and procedural steps to challenge disproportionate or unlawful requests. - Specific, periodically updated training for personnel responsible for managing public-authority access requests, including the requirements of EU law as to such access. - Publication of transparency reports or summaries regarding governmental access requests, insofar as permitted by local law. - Strict, granular, need-to-know access and confidentiality policies, data minimization (including restricted rather than full access for support cases), regular audits, and disciplinary enforcement. - Regular review of the suitability of the implemented measures and identification of additional or alternative solutions to maintain an essentially equivalent level of protection. **Contractual measures** - Processor declares that it has not purposefully created back doors or similar programming, has not purposefully changed its business processes to facilitate access to Personal Data or systems, and is not required by national law or government policy to create or maintain back doors, facilitate access, or hand over encryption keys. - Processor will verify the validity of the information provided for any TIA questionnaire on a regular basis and notify Controller of any changes without delay; Clause 14(e) of the SCCs remains unaffected. - In case of any order to disclose or grant access to Personal Data, Processor will inform the requesting public authority of the incompatibility of the order with the safeguards in the Article 46 GDPR transfer tool and the resulting conflict of obligations, and will challenge the order where lawful grounds exist. - The parties commit to reasonably assist Data Subjects in exercising their rights and seeking redress, and Processor commits to fairly compensate Data Subjects for material and non-material damage suffered because of a disclosure of their Personal Data in violation of these commitments. **SCHEDULE 5: SUB-PROCESSORS** Processor engages the following Sub-processors to Process Covered Data in connection with the hosted Ivy Services. Conduktor affiliates are listed for completeness. | Sub-processor | Location | Description of Processing | Data location | |---|---|---|---| | Amazon Web Services | United States (entity); hosting in eu-west-1 (Ireland) | Cloud hosting and infrastructure: compute, managed PostgreSQL database, load balancing, secrets management, object storage, container registry, DNS, and monitoring | Ireland (EEA) | | Vercel Inc. | United States | Hosting of the frontend application and documentation site; proxies API and authentication traffic server-side (HTTP traffic only, no persistent storage) | United States | | WorkOS, Inc. | United States | Authentication: login, sessions, organizations, and directory (email, name, profile picture, organization name) | United States | | Plus Five Five, Inc. (Resend) | United States | Transactional email, including workspace invitations (recipient email, invitation link) | United States | | Twilio Inc. (Segment) | United States | Product analytics (event names and properties); consent-gated browser analytics and server-side events | United States | | GitHub, Inc. | United States | Source code management and CI/CD (OIDC federation into hosting; no long-lived credentials) | United States | | Conduktor Inc. | 224 W 35th St Ste 500, #2947, New York, NY 10001, US | Conduktor affiliate, operation and support of the Services | United States | | Conduktor UK Ltd | 9th Floor, 107 Cheapside, London, EC2V 6DN, UK | Conduktor affiliate, operation and support of the Services | United Kingdom | | Conduktor France SAS | 3 Boulevard de Sebastopol, 75001 Paris, France | Conduktor affiliate, operation and support of the Services | France (EEA) | *The current list of Sub-processors, including any updates, is maintained by Processor and made available to Controller in accordance with clause 5.3.* Source: https://www.conduktor.io/legal/dpa-saas --- # Data Processing Addendum Updated November 14th, 2022 **DATA PROCESSING ADDENDUM** This Data Processing Agreement (‘’DPA’’) is attached to the End User License Agreement (available at [EULA](https://www.conduktor.io/legal/eula) and forms an inseparable part of the Agreement entered into by Conduktor and the Customer. This DPA shall set out the terms and conditions for the processing of Personal Data by Conduktor on behalf of the Customer under the Agreement. **DEFINITIONS** 1. Capitalized terms used but not defined within this DPA will have the meaning set forth in the Agreement. The following capitalized terms used in this DPA will be defined as follows: "**Applicable Data Protection Laws**" means all applicable laws, rules, regulations, and governmental requirements relating to the privacy, confidentiality, or security of Personal Data, as they may be amended or otherwise updated from time to time. "**Controller Affiliate**" means an affiliate of Controller who is a beneficiary to the Agreement. "**Covered Data**" means Personal Data that is: (a) provided by or on behalf of Controller to Processor in connection with the Services; or (b) obtained, developed, produced or otherwise Processed by Processor, or its agents or subcontractors, for purposes of providing the Services. "**Data Subject**" means a natural person whose Personal Data is Processed. "**Deidentified Data**" means data created using Covered Data that cannot reasonably be linked to such Covered Data, directly or indirectly. "**EEA**" means the European Economic Area including the European Union ("**EU**"). "**GDPR**" means Regulation (EU) 2016/679 (the "**EU GDPR**") or, where applicable, the "**UK GDPR**" as it forms part of the law of England and Wales, Scotland and Northern Ireland by virtue of section 3 of the UK European Union (Withdrawal) Act 2018 or, where applicable, the equivalent provision under Swiss data protection law. "**Member State**" means a member state of the EEA, being a member state of the European Union, Iceland, Norway, or Liechtenstein. "**Personal Data**" means any data or information that: (a) is linked or reasonably linkable to an identified or identifiable natural person; or (b) is otherwise "personal data," "personal information," "personally identifiable information," or similarly defined data or information under Applicable Data Protection Laws. "**Processing**" means any operation or set of operations which is performed on Personal Data or on sets of Personal Data, whether or not by automated means. "**Process**", "**Processes**" and "**Processed**" will be interpreted accordingly. "**Security Incident**" means a confirmed or reasonably suspected breach of security leading to the accidental or unlawful destruction, loss, alteration, unauthorized disclosure of, or unauthorized access to (including unauthorized internal access to), Covered Data. "**Services**" means the services to be provided by Processor pursuant to the Agreement. "**Standard Contractual Clauses**" or "**SCCs**" means Module Two (controller to processor) and/or Module Three (processor to processor) of the Standard Contractual Clauses annexed to Commission Implementing Decision (EU) 2021/914. "**Sub-processor**" means an entity appointed by Processor to Process Covered Data on its behalf. "**UK**" means the United Kingdom. "**US Data Protection Laws**" means, to the extent applicable, federal and state laws relating to data protection, the Processing of Personal Data, privacy and/or data protection in force from time to time in the United States. **INTERACTION WITH THE AGREEMENT** 1. This DPA is incorporated into and forms an integral part of the Agreement. This DPA supplements and (in case of contradictions) supersedes the Agreement with respect to any Processing of Covered Data. 2. Any Processing operation as described in clause 4 (Details of Data Processing) and Schedule 1\*\* to this DPA will be subject to this DPA. 3. Controller Affiliates will be beneficiaries under this DPA and, through Controller (see clauses 2.4 and 2.5), be entitled to enforce all rights in relation to Covered Data provided by the respective Affiliate. Controller will ensure that all obligations under this DPA will be passed on to the respective Controller Affiliate. 4. Controller warrants that it is duly mandated by any Controller Affiliates on whose behalf Processor Processes Covered Data in accordance with this DPA to (a) enforce the terms of this DPA on behalf of Controller Affiliates, and to act on behalf of Controller Affiliates in the administration and conduct of any claims arising in connection with this DPA; and (b) receive and respond to any notices or communications under this DPA on behalf of Controller Affiliates. 5. Controller will be the only point of contact for all communication between Controller Affiliates and Processor. **ROLE OF THE PARTIES** The Parties acknowledge and agree that: 1. for the purposes of the GDPR, Processor acts as "processor" or "sub-processor" (as defined in the GDPR). Processor's function as processor or sub-processor will be determined by the function of Controller: 1. Where Controller acts as a controller, Processor acts as a processor. 2. Where Controller acts as a processor on behalf of another controller, Processor acts as a sub-processor. 2. for the purposes of the US Data Protection Laws, Processor will act as a "service provider" or "processor" (as defined in US Data Protection Laws), as applicable, in its performance of its obligations pursuant to the Agreement and this DPA. 1. **DETAILS OF DATA PROCESSING** 1. The details of the Processing of Personal Data under the Agreement and this DPA (such as subject matter, nature and purpose of the Processing, categories of Personal Data and Data Subjects) are described in the Agreement and in Schedule 1\*\* to this DPA. 2. Covered Data will only be Processed on behalf of and under the instructions of Controller and in accordance with Applicable Data Protection Laws. The Agreement and this DPA will generally constitute instructions for the Processing of Covered Data. Controller may issue further written instructions in accordance with this DPA. Without limiting the foregoing, Processor is prohibited from: 1. selling Covered Data or otherwise making Covered Data available to any third party for monetary or other valuable consideration; 2. sharing Covered Data with any third party for cross-context behavioral advertising; 3. retaining, using, or disclosing Covered Data for any purpose other than for the business purposes specified in the Agreement or as otherwise permitted by Applicable Data Protection Laws; 4. retaining, using, or disclosing Covered Data outside of the direct business relationship between the Parties; and 5. except as otherwise permitted by Applicable Data Protection Laws, combining Covered Data with Personal Data that Processor receives from or on behalf of another person or persons, or collects from its own interaction with the Data Subject. 3. Processor will limit access to Covered Data to personnel who have a business need to have access to such Covered Data, and will ensure that such personnel are subject to obligations at least as protective of the Covered Data as the terms of this DPA and the Agreement. 4. Processor may (without prejudice to clause 11) Process Covered Data anywhere that Processor or its Sub-processors maintain facilities, subject to clause 5 of this DPA. 5. Processor will provide Controller with information to enable Controller to conduct and document any data protection assessments required under Applicable Data Protection Laws. In addition, Processor will notify Controller promptly if Processor determines that it can no longer meet its obligations under Applicable Data Protection Laws. 6. Controller will have the right to take reasonable and appropriate steps to ensure that Processor uses Covered Data in a manner consistent with Controller's obligations under Applicable Data Protection Laws. 2. **SUB-PROCESSORS** 1. Controller grants Processor the general authorisation to engage Sub-processors, subject to clause 5.2, as well as Processor's current Sub-processors listed in Schedule 5. 2. Processor will enter into a written agreement with each Sub-processor imposing data protection obligations that, in substance, are no less protective of Covered Data than Processor's obligations under this DPA.old 3. Processor will provide Controller with at least fifteen (15) days' notice of any proposed changes to the Sub-processors it uses to Process Covered Data. Controller may object to Processor's use of a new Sub-processor (including when exercising its right to object under clause 9(a) of the SCCs if applicable) by providing Processor with written notice of the objection within ten (10) days after Processor has provided notice to Controller of such proposed change (an "**Objection**"). If Controller does not object to the engagement within the Objection period, consent regarding the engagement will be assumed. In the event Controller objects to Processor's use of a new Sub-processor, Controller and Processor will work together in good faith to find a mutually acceptable resolution to address such Objection. If the Parties are unable to reach a mutually acceptable resolution within a reasonable timeframe, either Party may, as its sole and exclusive remedy, terminate the portion of the Agreement relating to the Services affected by such change by providing written notice to the other Party. During any such Objection period, Processor may suspend the affected portion of the Services. 3. **DATA SUBJECT RIGHTS REQUESTS** 1. As between the Parties, Controller will have sole discretion and responsibility in responding to the rights asserted by any individual in relation to Covered Data under Applicable Data Protection Laws (each, a "**Data Subject Request**"). 2. Processor will promptly forward to Controller without undue delay any Data Subject Request received by Processor or any Sub-processor and may advise the individual to submit their request directly to Controller. 3. Processor will provide Controller with reasonable assistance as necessary for Controller to fulfil its obligation under Applicable Data Protection Laws to respond to Data Subject Requests, including if applicable, Controller's obligation to respond to requests for exercising the rights set out in Applicable Data Protection Laws. 4. **SECURITY AND AUDITS** 1. Processor will implement and maintain appropriate technical and organizational data protection and security measures designed to ensure security of Covered Data, including, without limitation, protection against unauthorized or unlawful Processing and against accidental loss, destruction, or damage of or to it. When assessing the appropriate level of security, account will be taken in particular of the nature, scope, context and purpose of the Processing as well as the risks that are presented by the Processing, in particular from accidental or unlawful destruction, loss, alteration, unauthorized disclosure of, or access to Covered Data. 2. Processor will implement and maintain as a minimum standard the measures set out in Schedule 2. 3. Controller will have the right to audit Processor's compliance with this DPA. The Parties agree that all such audits will be conducted: 1. upon reasonable written notice to Processor; 2. only once per year; and 3. only during Processor's normal business hours. 4. To conduct such audits, Controller may engage a third-party auditor subject to such auditor complying with the requirements under clause 7.3 and provided that such auditor is suitably qualified and independent. 5. To request an audit, Controller must submit a detailed proposed audit plan to Processor at least two weeks in advance of the proposed audit date. Processor will review the proposed audit plan and work cooperatively with Controller to agree on a final audit plan. All such audits must be conducted subject to the agreed final audit plan and Processor's health and safety or other relevant policies. 6. Controller will promptly notify Processor of any non-compliance discovered during an audit. 7. Controller will bear the costs for any audit initiated by Controller, unless the audit reveals material non-compliance with the requirements of this DPA. 8. Upon request, Processor will provide to Controller documentation reasonably evidencing the implementation of the technical and organizational data security measures in accordance with industry standards. Processor may, in its discretion, provide data protection compliance certifications issued by a commonly accepted certification issuer which has been audited by a data security expert, or by a publicly certified auditing company. If the requested audit scope is addressed in such a certification produced by a qualified third-party auditor within twelve (12) months of Controller's audit request and Processor confirms there are no known material changes in the controls audited, Controller agrees to accept those findings in lieu of requesting an audit of the controls covered by the report 9. Processor will audit its Sub-processors on a regular basis and will, upon Controller's request, confirm their compliance with Applicable Data Protection Laws and the Sub-processors' contractual obligations. 5. **SECURITY INCIDENTS** Processor will notify Controller in writing without undue delay after becoming aware of any Security Incident, and reasonably cooperate in any obligation of Controller under Applicable Data Protection Laws to make any notifications, such as to individuals or supervisory authorities. Processor will take reasonable steps to contain, investigate, and mitigate any Security Incident, and will send Controller timely information about the Security Incident, including, but not limited to, the nature of the Security Incident, the measures taken to mitigate or contain the Security Incident, and the status of the investigation. Processor's notification of or response to a Security Incident under this clause 8 will not be construed as an acknowledgement by Processor of any fault or liability with respect to the Security Incident. Processor will provide reasonable assistance with Controller's investigation of the possible Security Incident and any notification obligation of Controller under Applicable Data Protection Laws, such as in relation to individuals or supervisory authorities. **DELETION AND RETURN** Processor will, within thirty (30) days of the date of termination or expiry of the Agreement (a) if requested to do so by Controller within that period, return a copy of all Covered Data or provide a self-service functionality allowing Controller to do the same; and (b) delete all other copies of Covered Data Processed by Processor or any Sub-processors. **CONTRACT PERIOD** This DPA shall become effective in parallel with the Agreement and, notwithstanding any termination of the Agreement, will remain in effect until, and automatically expire upon, Processor's deletion of all Covered Data as described in this DPA. **STANDARD CONTRACTUAL CLAUSES** 1. The Parties agree that the terms of the Standard Contractual Clauses Module Two (Controller to Processor) and Module Three (Processor to Processor), as further specified in Schedule 3 of this DPA, are hereby incorporated by reference and will be deemed to have been executed by the Parties and apply to any transfers of Covered Data falling within the scope of the GDPR from Controller (as data exporter) to Processor (as data importer). 2. To the extent applicable, the jurisdiction-specific addenda to the Standard Contractual Clauses set out in Schedule 3 are also incorporated herein by reference and will be deemed to have been executed by the Parties and apply to any transfers of Covered Data falling within the scope of Applicable Data Protection Laws in the listed jurisdiction(s) from Controller (as data exporter) to Processor (as data importer). 3. Processor will provide Controller reasonable support to enable Controller's compliance with the requirements imposed on international transfers of Covered Data. Processor will, upon Controller's request, provide information to Controller which is reasonably necessary for Controller to complete a transfer impact assessment ("**TIA**") under Applicable Data Protection Laws. 4. Processor further agrees to implement the supplementary measures agreed upon and set forth in Schedule 4 of this DPA in order to enable Controller's compliance with requirements imposed on international transfers of Covered Data under Applicable Data Protection Laws. **DEIDENTIFIED DATA** If Processor receives Deidentified Data from or on behalf of Controller, then Processor will: 1. take reasonable measures to ensure the information cannot be associated with a Data Subject. 2. publicly commit to Process the Deidentified Data solely in deidentified form and not to attempt to reidentify the information. 3. contractually obligate any recipients of the Deidentified Data to comply with the foregoing requirements and Applicable Data Protection Laws. 1. **GENERAL** 1. The Parties hereby certify that they understand the requirements in this DPA and will comply with them. 2. The Parties agree to negotiate in good faith any amendments to this DPA as may be required in connection with changes in Applicable Data Protection Laws. 3. This DPA and the Agreement set forth the entire agreement between the Parties with respect to the subject matter hereof. **DETAILS OF PROCESSING** **List of Parties** The Parties are set out in the preamble to this DPA. With regard to any transfers of Covered Data falling within the scope of the GDPR from Controller to Processor, additional information regarding the data exporter and data importer is set out below. **Data Exporter** The data exporter is: each of the Controller and/or Controller Affiliates operating in the countries which comprise the European Economic Area, UK and/or Switzerland and/or – to the extent agreed by the Parties – Controller and/or Controller Affiliates in any other country to the extent the GDPR applies. The data exporter's contact person's name, position and contact details as well as (if appointed) the data protection officer's name and contact details and (if relevant) the representative's contact details are included in the Agreement or will be disclosed to Processor upon request. The activities relevant to the data transfer under these Clauses are defined by the Agreement and the data exporter who decides on the scope of the Processing of Personal Data in connection with the Services further described in section B of this Schedule 1. **Data Importer** The data importer is: the Processor. The data importer's contact person and contact details are included in the Agreement or will be disclosed to Controller upon request. The data importer's activities relevant to the data transfer under these Clauses are as follows: the data importer Processes Personal Data provided by the data exporter on behalf of the data exporter in connection with providing the Services to the data exporter as further described in section B of this Schedule 1 and in the Agreement. **Description of Processing** The Customer has full control of what personal data will be processed by uploading such personal data into the cloud services. Conduktor has no visibility to such personal data provided and uploaded by the Customer. **Categories of Data Subjects** The categories of Data Subjects whose Personal Data are Processed: _Employees of Controller and Controller Affiliates, as well as Controller's customers and their employees._ **Categories of Personal Data** The Processed categories of Personal Data are: _name, phone number, email address, address data, name of employer_. **Special categories of Personal Data (if applicable)** The Processed Personal Data includes the following special categories of data: _No special categories of Personal Data are processed._ **Frequency of the Processing** The Processing is performed _continuously_. **Subject matter and nature of the Processing** The subject matter of the Processing is: _Hosting, storing and maintenance for the data Customer has input to the cloud services_. **Storage Limitation** The period during which the Personal Data will be Processed, or, if that is not possible, the criteria used to determine that period: _if Personal Data is not deleted upon request by Controller during the term of the Agreement, the duration of Processing corresponds to the duration of this DPA as defined in clause 9 of the DPA_ **Sub-processor (if applicable)** For Processing by sub-processors, specify subject matter, nature, and duration of the Processing: _Hosting, storing and maintenance for the data Customer has input to the cloud services_. **Competent Supervisory Authority** Identify the competent supervisory authority/ies in accordance with clause 13 of the SCCs Where the data exporter is established in an EU Member State: _The supervisory authority of the country in which the data exporter established is the competent authority_. Where the data exporter is not established in an EU Member State, but falls within the territorial scope of application of the GDPR in accordance with its Article 3(2) and has appointed a representative pursuant to Article 27(1) of the GDPR: The competent supervisory authority is the one of the Member State in which the representative is established. Where the data exporter is not established in an EU Member State, but falls within the territorial scope of application of the GDPR in accordance with its Article 3(2) without, however, having to appoint a representative pursuant to Article 27(2) of the GDPR: The competent supervisory authority is the supervisory authority of _Ireland_ **TECHNICAL AND ORGANIZATIONAL MEASURES** Processor has implemented the following technical and organizational measures (including any relevant certifications) to ensure an appropriate level of security, taking into account the nature, scope, context, and purpose of the processing, as well as the risks for the rights and freedoms of natural persons: 1. Organizational management and dedicated staff responsible for the development, implementation, and maintenance of Processor's information security program. 2. Audit and risk assessment procedures for the purposes of periodic review and assessment of risks to Processor's organization, monitoring and maintaining compliance with Processor's policies and procedures, and reporting the condition of its information security and compliance to internal senior management. 3. Utilization of commercially available and industry standard encryption technologies for Covered Data that is: 1. being transmitted by Processor over public networks (i.e., the Internet) or when transmitted wirelessly; or 2. at rest. 4. Data security controls which include at a minimum, but may not be limited to, logical segregation of data, logical access controls designed to manage electronic access to data and system functionality based on authority levels and job functions, (e.g., granting access on a need-to-know and least privilege basis, use of unique IDs and passwords for all users, periodic review, and revoking/changing access promptly when employment terminates or changes in job functions occur). 5. Password controls designed to manage and control password strength, expiration and usage including prohibiting users from sharing passwords and requiring that Processor's passwords that are assigned to its employees: (i) be at least eight (8) characters in length, (ii) not be stored in readable format on Processor's computer systems; (iii) must have defined complexity; (iv) must have a history threshold to prevent reuse of recent passwords; and (v) newly issued passwords must be changed after first use. 6. System audit or event logging and related monitoring procedures to proactively record user access and system activity for routine review. 7. Physical and environmental security of data center, server room facilities and other areas containing Personal Data designed to: (i) protect information assets from unauthorized physical access, (ii) manage, monitor, and log movement of persons into and out of Processor facilities, and (iii) guard against environmental hazards such as heat, fire, and water damage. 8. Operational procedures and controls to provide for configuration, monitoring and maintenance of technology and information systems according to prescribed internal and adopted industry standards, including secure disposal of systems and media to render all information or data contained therein as undecipherable or unrecoverable prior to final disposal or release from Processor's possession. 9. Change management procedures and tracking mechanisms designed to test, approve, and monitor all changes to Processor's technology and information assets. 10. Incident / problem management procedures design to allow Processor to investigate, respond to, mitigate, and notify of events related to Processor's technology and information assets. 11. Network security controls that provide for the use of firewall systems, and intrusion detection systems and other traffic and event correlation procedures designed to protect systems from intrusion and limit the scope of any successful attack. 12. Vulnerability assessment, patch management and threat protection technologies and scheduled monitoring procedures designed to identify, assess, mitigate, and protect against identified security threats, viruses, and other malicious code. 13. Business resiliency/continuity and disaster recovery procedures designed to maintain service and/or recovery from foreseeable emergency situations or disasters. 18 **STANDARD CONTRACTUAL CLAUSES** 1. **EU SCCS** The Standard Contractual Clauses will apply to any Processing of Covered Data that is subject to the GDPR. For the purposes of the Standard Contractual Clauses: 1. Module Two will apply in the case of the Processing under clause 3(a)(i) of the DPA and Module Three will apply in the case of Processing under clause 3(a)(ii) of the DPA. 2. Clause 7 of the Standard Contractual Clauses (Docking Clause) does not apply. 3. Clause 9(a) option 2 (General written authorization) is selected, and the time period to be specified is determined in clause 5.3 of the DPA. 4. The option in Clause 11(a) of the Standard Contractual Clauses (Independent dispute resolution body) does not apply. 5. With regard to Clause 17 of the Standard Contractual Clauses (Governing law), the Parties agree that, option 1 will apply and the governing law will be the law of the Republic of Ireland. 6. In Clause 18 of the Standard Contractual Clauses (Choice of forum and jurisdiction), the Parties submit themselves to the jurisdiction of the courts of the Republic of Ireland. 7. For the Purpose of Annex I of the Standard Contractual Clauses, Schedule 1 of the DPA contains the specifications regarding the parties, the description of transfer, and the competent supervisory authority 8. For the Purpose of Annex II of the Standard Contractual Clauses, Schedule 2 of the DPA contains the technical and organizational measures. 9. The specifications for Annex III of the Standard Contractual Clauses, are determined by clause 5.1 of the DPA. The Sub-processor's contact person's name, position and contact details will be provided by Processor upon request. **UK ADDENDUM** This UK Addendum will apply to any Processing of Covered Data that is subject to the UK GDPR or to both the UK GDPR and the GDPR. 1. As used in this UK Addendum: "**Approved Addendum**" means the template addendum, version B.1.0 issued by the UK Information Commissioner under S119A(1) Data Protection Act 2018 and laid before the UK Parliament on 2 February 2022, as it may be revised according to Section 18 of the Mandatory Clauses. "**Mandatory Clauses**" means "Part 2: Mandatory Clauses" of the Approved Addendum. 1. With respect to any transfers of Covered Data falling within the scope of the UK GDPR from Controller (as data exporter) to Processor (as data importer): 1. the Approved Addendum as further specified in this Schedule 5 will form part of this DPA, and the Standard Contractual Clauses will be read and interpreted in light of the provisions of the Approved Addendum, to the extent necessary according to Clause 12 lit. 1 of the Mandatory Clauses; 2. In deviation to Table 1 of the Approved Addendum and in accordance with Clause 17 of the Mandatory Clauses, the parties are further specified in Schedule 1A. of this DPA. 3. The selected Modules and Clauses to be determined according to Table 2 of the Approved Addendum are further specified in this Schedule as amended by the Mandatory Clauses. 4. Annex 1 A and B of Table 3 to the Approved Addendum are specified by Schedule 1 of this DPA, Annex II of the Approved Addendum is further specified by Schedule 2 of this DPA, and Annex III of the Approved Addendum is further specified by Schedule 1B.10 of this DPA. 5. Processor (as data importer) may end this DPA, to the extent the Approved Addendum applies, in accordance with clause ‎19 of the Mandatory Clauses; 6. Clause 16 of the Mandatory Clauses will not apply. **SWISS ADDENDUM** This Swiss Addendum will apply to any Processing of Covered Data that is subject to Swiss Data Protection Laws (as defined below) or to both Swiss Data Protection Laws and the GDPR. **Interpretation of this Addendum** 1. Where this Addendum uses terms that are defined in the Standard Contractual Clauses, those terms will have the same meaning as in the Standard Contractual Clauses. In addition, the following terms have the following meanings: This AddendumThis Addendum to the ClausesClausesThe Standard Contractual Clauses as further specified in this ScheduleSwiss Data Protection LawsThe Swiss Federal Act on Data Protection of 19 June 1992 and the Swiss Ordinance to the Swiss Federal Act on Data Protection of 14 June 1993, and any new or revised version of these laws that may enter into force from time to time. 1. This Addendum will be read and interpreted in the light of the provisions of Swiss Data Protection Laws, and so that if fulfils the intention for it to provide the appropriate safeguards as required by Article 46 GDPR and/or Article 6(2)(a) of the Swiss Data Protection Laws, as the case may be. 2. This Addendum will not be interpreted in a way that conflicts with rights and obligations provided for in Swiss Data Protection Laws. 3. Any references to legislation (or specific provisions of legislation) means that legislation (or specific provision) as it may change over time. This includes where that legislation (or specific provision) has been consolidated, re-enacted and/or replaced after this Addendum has been entered into. 1. **Hierarchy** In the event of a conflict or inconsistency between this Addendum and the provisions of the Clauses or other related agreements between the Parties, existing at the time this Addendum is agreed or entered into thereafter, the provisions which provide the most protection to Data Subjects will prevail. **Incorporation of the Clauses** 1. In relation to any Processing of Personal Data subject to Swiss Data Protection Laws or to both Swiss Data Protection Laws and the GDPR, this Addendum amends the DPA the Standard Contractual Clauses to the extent necessary so they operate: 1. for transfers made by the data exporter to the data importer, to the extent that Swiss Data Protection Laws or Swiss Data Protection Laws and the GDPR apply to the data exporter's Processing when making that transfer; and 2. to provide appropriate safeguards for the transfers in accordance with Article 46 of the GDPR and/or Article 6(2)(a) of the Swiss Data Protection Laws, as the case may be. 2. To the extent that any Processing of Personal Data is exclusively subject to Swiss Data Protection Laws, the amendments to the DPA including the SCCs, as further specified in this Schedule and as required by clause 3.1 of this Swiss Addendum, include (without limitation): 1. References to the "Clauses" or the "SCCs" mean this Swiss Addendum as it amends the SCCs. 2. Clause 6 Description of the transfer(s) is replaced with: "_The details of the transfer(s), and in particular the categories of Personal Data that are transferred and the purpose(s) for which they are transferred, are those specified in Schedule 1 of this DPA where Swiss Data Protection Laws apply to the data exporter's Processing when making that transfer_." 1. References to "Regulation (EU) 2016/679" or "that Regulation" or "GDPR" are replaced by "Swiss Data Protection Laws" and references to specific Article(s) of "Regulation (EU) 2016/679" or "GDPR" are replaced with the equivalent Article or Section of Swiss Data Protection Laws extent applicable. 2. References to Regulation (EU) 2018/1725 are removed. 3. References to the "European Union", "Union", "EU" and "EU Member State" are all replaced with "Switzerland". 4. Clause 13(a) and Part C of Annex I are not used; the "competent supervisory authority" is the Federal Data Protection and Information Commissioner (the "FDPIC") insofar as the transfers are governed by Swiss Data Protection Laws; 5. Clause 17 is replaced to state "_These Clauses are governed by the laws of Switzerland insofar as the transfers are governed by Swiss Data Protection Laws_". 1. Clause 18 is replaced to state: "_Any dispute arising from these Clauses relating to Swiss Data Protection Laws will be resolved by the courts of Switzerland. A Data Subject may also bring legal proceedings against the data exporter and/or data importer before the courts of Switzerland in which he/she has his/her habitual residence. The Parties agree to submit themselves to the jurisdiction of such courts_." Until the entry into force of the revised Swiss Data Protection Laws, the Clauses will also protect Personal Data of legal entities and legal entities will receive the same protection under the Clauses as natural persons. 1. To the extent that any Processing of Personal Data is subject to both Swiss Data Protection Laws and the GDPR, the DPA including the Clauses as further specified in this Schedule will apply (i) as is and (ii) additionally, to the extent that a transfer is subject to Swiss Data Protection Laws, as amended by clauses 3.1 and 3.3 of this Swiss Addendum, with the sole exception that Clause 17 of the SCCs will not be replaced as stipulated under clause 3.3(b)(vii) of this Swiss Addendum. 2. Controller warrants that it and/or Controller Affiliates have made any notifications to the FDPIC which are required under Swiss Data Protection Laws. **ADDITIONAL SUPPLEMENTARY MEASURES** Processor further commits to implementing supplementary measures based on guidance provided by EU supervisory authorities in order to enhance the protection of Covered Data in relation to the Processing in a third country, as described in this Schedule 4. **Additional Technical Measures** **Encryption** FORMCHECKBOX The Personal Data is transmitted (between the Parties and by Processor between data centers as well as to a sub-processor and back) using strong encryption. Hereby, it is ensured that the encryption protocols employed are state-of-the-art and provide effective protection against active and passive attacks with resources known to be available to the public authorities of this third country, the parties involved in the communication agree on a trustworthy public-key certification authority or infrastructure, specific protective and state-of-the-art measures are used against active and passive attacks on the sending and receiving systems providing transport encryption, including tests for software vulnerabilities and possible backdoors, in case the transport encryption does not provide appropriate security by itself due to experience with vulnerabilities of the infrastructure or the software used, Personal Data is also encrypted end-to-end on the application layer using state-of-the-art encryption methods, the encryption algorithm and its parameterization (e.g., key length, operating mode, if applicable) conform to the state-of-the-art and can be considered robust against cryptanalysis performed by the public authorities when data is transiting to this third country taking into account the resources and technical capabilities (e.g., computing power for brute-force attacks) available to them , the strength of the encryption takes into account the specific time period during which the confidentiality of the encrypted Personal Data must be preserved, the encryption algorithm is implemented correctly and by properly maintained software without known vulnerabilities the conformity of which to the specification of the algorithm chosen has been verified, e.g., by certification, the keys are reliably managed (generated, administered, stored, if relevant, linked to the identity of the intended recipient, and revoked), by Controller or by an entity trusted by Controller under a jurisdiction offering an essentially equivalent level of protection. In accordance with the requirements outlined in the previous paragraph, the Parties agree to implement strong end-to-end content encryption (between the Parties and by Processor between data centers as well as to a sub-processor and back). The Personal Data at rest is stored by Processor using strong encryption. _The encryption algorithm and its parameterization (e.g., key length, operating mode, if applicable) conform to the state-of-the-art and can be considered robust against cryptanalysis performed by the public authorities in the recipient country taking into account the resources and technical capabilities (e.g., computing power for brute-force attacks) available to them. The strength of the encryption and key length takes into account the specific time period during which the confidentiality of the encrypted Personal Data must be preserved. The encryption algorithm is implemented correctly and by properly maintained software without known vulnerabilities the conformity of which to the specification of the algorithm chosen has been verified, e.g., by certification. The keys are reliably managed (generated, administered, stored, if relevant, linked to the identity of an intended recipient, and revoked)._ **Additional Organizational Measures** 1. Internal policies for governance of transfers especially with groups of enterprises Adoption of adequate internal policies with clear allocation of responsibilities for data transfers, reporting channels and standard operating procedures for cases of formal or informal requests from public authorities to access the data. _Especially in case of transfers among groups of enterprises, these policies may include, among others, the appointment of a specific team, composed of experts on IT, data protection and privacy laws, to deal with requests that involve Personal Data transferred from the EEA; the notification to the senior legal and corporate management and to Controller upon receipt of such requests; the procedural steps to challenge disproportionate or unlawful requests and the provision of transparent information to Data Subjects._ Development of specific training procedures for personnel in charge of managing requests for access to Personal Data from public authorities, which should be periodically updated to reflect new legislative and jurisprudential developments in the third country and in the EEA. _The training procedures should include the requirements of EU law as to access by public authorities to Personal Data, in particular as following from Article 52(1) of the Charter of Fundamental Rights. Awareness of personnel should be raised in particular by means of assessment of practical examples of public authorities' data access requests and by applying the standard following from Article 52(1) of the Charter of Fundamental Rights to such practical examples. Such training should take into account the particular situation of the Processor, e.g. legislation and regulations of the third country to which Processor is subject to, and should be developed where possible in cooperation with Controller._ **Transparency and accountability measures** Regular publication of transparency reports or summaries regarding governmental requests for access to data and the kind of reply provided, insofar publication is allowed by local law. 3.3 **Organizational methods and data minimization measures** Already existing organizational requirements under the accountability principle, such as the adoption of strict and granular data access and confidentiality policies and best practices, based on a strict need-to-know principle, monitored with regular audits and enforced through disciplinary measures. Data minimization should be considered in this regard, in order to limit the exposure of Personal Data to unauthorized access. For example, in some cases it might not be necessary to transfer certain data (e.g. in case of remote access to EEA data, such as in support cases, when restricted access is granted instead of full access; or when the provision of a service only requires the transfer of a limited set of data, and not an entire database). Development and implementation of best practices by both Parties to appropriately and timely involve and provide access of information to their respective data protection officers, if existent, and to their legal and internal auditing services on matters related to international transfers of Personal Data transfers. **Others** Adoption and regular review by Processor of internal policies to assess the suitability of the implemented complementary measures and identify and implement additional or alternative solutions when necessary, to ensure that an essentially equivalent level of protection to that guaranteed within the EEA of the Personal Data transferred is maintained. **Additional Contractual Measures** **Transparency obligations** Processer declares that (1) it has not purposefully created back doors or similar programming that could be used to access the system and/or Personal Data, (2) it has not purposefully created or changed its business processes in a manner that facilitates access to Personal Data or systems, and (3) that national law or government policy does not require Processor to create or maintain back doors or to facilitate access to Personal Data or systems or for Processor to be in possession or to hand over the encryption key. Processor will verify the validity of the information provided for the TIA questionnaire on a regular basis and provide notice to Controller in case of any changes without delay. Clause 14(e) SCC will remain unaffected. **Obligations to take specific actions** In case of any order to disclose or to grant access to the Personal Data, Processor commits to inform the requesting public authority of the incompatibility of the order with the safeguards contained in the Article 46 GDPR transfer tool and the resulting conflict of obligations for Processor. **Empowering Data Subjects to exercise their rights** The Parties commit to reasonably assist the Data Subject in exercising his/her rights in the third country jurisdiction through ad hoc redress mechanisms and legal counselling. The Parties commit to reasonably assist the Data Subject to seek information and an effective redress in the EU (_e.g._, by lodging a claim with a competent supervisory authority and/or judicial authority in the EU). Processor commits to fairly compensate the Data Subject for any material and non-material damage suffered because of the disclosure of his/her Personal Data transferred under the chosen transfer tool in violation of the commitments it contains. **SUB-PROCESSORS** Name of Sub-processor Address of Sub-processor Description of Processing Conduktor Inc. 154 W 14th St, New York, NY 10011, USA Conduktor affiliate Conduktor UK Ltd 81-87 High Holborn, London, WC1V 6DF, United Kingdom Conduktor affiliate Conduktor France SaS 3 Boulevard de Sebastopol, 75001, Paris, France Conduktor affiliate Conduktor Ireland Technology Ltd 70 Sir John Rogerson's Quay, D02 R296, Dublin 2,Dublin, Ireland Hosting This website is operated by Conduktor Inc. If you have any questions, complaints, or concerns about how your information is handled, or would like to exercise any of your rights for yourself or on behalf of someone else, please email us at #### support@conduktor.io Source: https://www.conduktor.io/legal/dpa --- # EULA Enterprise Updated November 14th, 2022 This End User License Agreement (“**Agreement**”) is a legal agreement between you (either an individual or an entity) (“**Customer**”) and Conduktor Inc., a Delaware corporation (“**Conduktor**”), regarding proprietary software made available by Conduktor (“**Conduktor Software**”). This Agreement applies to Enterprise Licenses of the Conduktor Software. BY SIGNING AN ORDER FORM, OR BY USING THE CONDUKTOR SOFTWARE, CUSTOMER EXPRESSLY ACCEPTS AND AGREES TO THE TERMS OF THIS AGREEMENT. IF YOU ARE AN INDIVIDUAL AGREEING TO THE TERMS OF THIS AGREEMENT ON BEHALF OF AN ENTITY, SUCH AS YOUR EMPLOYER, YOU REPRESENT THAT YOU HAVE THE LEGAL AUTHORITY TO BIND THAT ENTITY AND “CUSTOMER” SHALL REFER HEREIN TO SUCH ENTITY. IF YOU DO NOT HAVE SUCH AUTHORITY, OR IF YOU DO NOT AGREE WITH THE TERMS OF THIS AGREEMENT, YOU MUST NOT ACCEPT THIS AGREEMENT AND MAY NOT USE THE SOFTWARE. ## **LICENSE** 1. License Terms. Subject to the terms of this Agreement, Conduktor grants to Customer a limited, non-exclusive, non-sublicensable, non-transferable license during the Term (as defined in Section 9.1) to use the Conduktor Software solely for use by Named Users (as defined below) to support Customer’s internal business operations. Subject to the restrictions set forth in this Agreement, Customer may permit its third-party service providers to use the Conduktor Software to provide outsourced services to Customer, and Customer will be solely responsible for such service provider’s compliance with this Agreement. “**Named User**” means any unique licensed user of the Conduktor Software who is authorized by the Customer to use the Conduktor Software. Named Users with the same domain (i.e., from the same company/customer) should be under the same subscription. 2. Additional Restrictions and Limitations. Customer shall not, and shall not permit or encourage any of Customer’s Named Users, or any employee, contractor, consultant, or third party to: 3. sublicense, sell, distribute or pledge the Conduktor Software or any of the rights herein; 4. use the Conduktor Software other than as expressly permitted by this Agreement, or use the Conduktor Software for third-party training, software-as-a-service, time-sharing or service bureau use; 5. provide access to the Conduktor Software to anyone other than an Named User; 6. use any third party software, including any open source software, in conjunction with the Conduktor Software, unless Customer ensures that such use does not cause the Conduktor Software to become subject to any third party license applicable to such third party software or require the public disclosure or distribution of the Conduktor Software or the licensing of the Conduktor Software for the purpose of making derivative works; 7. modify, translate, decrypt, create derivative works based on, disassemble, decompile or reverse engineer any portions of the Conduktor Software, or otherwise attempt to gain access to the source code to such Conduktor Software (or the underlying ideas, algorithms, structure or organization of the object code in the Conduktor Software). The foregoing restriction is inapplicable to the extent prohibited by applicable law; provided that, in the event that Customer intends to disassemble, decompile or reverse engineer such Conduktor Software, Customer shall first provide Conduktor with written notice thereof. 8. bypass or breach any security device or protection used by the Conduktor Software or access or use the Conduktor Software other than by a Named User through the use of valid access credentials; 9. input, upload, transmit, or otherwise provide to or through the Conduktor Software any information or materials that are unlawful or injurious, or that contain, transmit, or activate any harmful code; or 10. use the Conduktor Software in any manner or for any purpose that infringes, misappropriates, or otherwise violates any intellectual property rights or other right of any person, or that violates any applicable law. 3. Suspension of Access; Reservation of Rights. Conduktor reserves the right to modify or suspend Customer’s access to or use of the Conduktor Software in its sole discretion, without prior notice and without liability to Conduktor in the event: (a) Conduktor reasonably believes that Customer’s access to or use of the Conduktor Software may result in liability to Conduktor or adversely affect the integrity, functionality or usability of the Conduktor Software, including without limitation, interfering with the access to or use of the Conduktor Software by other users; (b) Customer is in breach of this Agreement, including without limitation, a failure to pay any amounts due under this Agreement; or (c) for scheduled maintenance. Conduktor shall not be responsible for any damages, liabilities or losses of Customer in connection with any suspension of the Conduktor Software that result from the foregoing. Conduktor reserves all rights not expressly granted in this Agreement. No rights are granted by implication. 4. Number of Users. At the time of the initial purchase of the Conduktor Software, the Customer may subscribe for licenses for any number of Named Users. Following the initial purchase Customer may subscribe for additional licenses for new Named Users, which licenses will be subject to the terms of this Agreement. Any and all employees, contractors, and other end users of Customer may only use the Conduktor Software as Named Users of Customer. Customer is permitted to reassign up to ten percent (10%) of the total number of purchased licenses for Named Users to new Named Users each month. Reassignment requests should be sent by email to [support@conduktor.io](mailto:support@conduktor.io). 5. Number of Clusters. Where a product limits the number of Kafka clusters (“**Clusters**”), this limit shall be referenced on an order form. Creating multiple organisations to create multiple Clusters shall not be allowed. 6. Number of Tests. Where a product limits the number of test scenarios that can be triggered manually inside the Conduktor Software or through an automated means (“**Test**”), this limit shall be referenced on an order form. Where a test suite results in multiple test scenarios, it is the scenarios that count towards this limit. 7. Reseller Activities. Customer may acquire licenses to use the Conduktor Software for purposes of transferring such licenses to other end users, either directly or indirectly, through other business intermediaries by entering into a separate written agreement with Conduktor that sets forth the terms of such engagement. 8. Delivery of Materials. The Conduktor Software, and any versions, updates or maintenance releases of any component thereof, will be delivered only through an electronic transfer. 9. Support and Maintenance. All support requests should be sent by email to [support@conduktor.io](mailto:support@conduktor.io), mentioning the company name, contact email, and a case description. The time taken to respond to a support request (“**Target Initial Response Time**”) for receipt of an initial reply to a reported Issue is within 48 business hours. The Target Initial Response Time shall be measured by the elapsed time between Conduktor’s receipt of a reported issue and the time Conduktor begins to address such issue by responding and initiating communication with Customer about the issue. The actual time required to fully resolve the issue, if full resolution occurs, maybe longer than the Target Initial Response Time. Customer understands and agrees that full resolution of an issue is not guaranteed and may not occur. 10. Preview Releases. Conduktor may invite you to try our beta services or features under development (“Beta Services or Features”) at no charge. You may accept or decline any such invitation at your sole discretion. Beta Services or Features will be clearly designated as beta, pilot, limited release, developer preview, non-production, evaluation or by a description of similar import. Beta Services or Features are for evaluation purposes only and not for production use, and are not considered Conduktor Software under this Agreement, are not supported, and may be subject to additional terms. Conduktor may discontinue Beta Services or Features at any time in Conduktor's sole discretion. Conduktor is undertaking no obligation to release a generally available version of Beta Service or Feature. We will have no liability for any harm or damage arising out of or in connection with a Beta Service or Feature. ### **USER AND PERFORMANCE DATA.** Conduktor may from time to time use and process data about Customer’s use of the Conduktor Software and Third Party Software for the purposes of creating statistics and analytics data. Conduktor may use such data for its own business purposes, including to maintain and improve the Conduktor Software, Third Party Software and other services and to monitor and analyse its activities in connection with the performance of such services. To the extent the Customers inputs anu personal data into the cloud services and the processing is subject to the EU General Data Protection Regulation (2016/679, the “**GDPR”**), Conduktor and the Customer acknowledge that the Customer acts as a data controller and Conduktor is the data processor processing personal data together with its subprocessors on behalf of the Customer for the purpose of providing the services. All such processing of the Costomer’s personal data by Conduktor is subject to the data processing addendum found at [DPA](https://www.conduktor.io/legal/dpa). ### **OWNERSHIP.** Customer acknowledges that Conduktor or its licensors retain all proprietary rights, title and interest, including all intellectual property rights, in and to the Conduktor Software and any changes, corrections, bug fixes, enhancements, updates and other modifications thereto (collectively, “**Modifications**”), and as between the parties all such rights shall vest in Conduktor. Customer acknowledges that the licenses granted in Section 1 do not include the right to prepare any Modifications of the Conduktor Software. ### **FINANCIAL TERMS**. 1. Customer shall pay Conduktor any fees or payments net 14-days from Conduktor’s invoice. Conduktor may charge Customer an additional 1.5% per month for all fees that are not paid on time. 2. Conduktor may increase annual recurring fees upon providing written notice at least sixty (60) days before the end of the then current Term. 3. Fees payable to Conduktor are exclusive of all applicable sales, use, value-added, goods and services, consumption, withholding, excise and any other similar taxes or government charges (“**Taxes**”). Customer shall pay Conduktor such applicable Taxes listed on the relevant invoice, which may be in addition to the total fees due and listed on an applicable order form. 4. Except as otherwise provided herein, all fees paid by Customer are non-refundable and no right of setoff exists. 5. **FEEDBACK.** Customer is under no duty to provide any suggestions, enhancement requests, or other feedback regarding the Conduktor Software (“**Feedback**”). If Customer chooses to offer Feedback to Conduktor, Customer hereby grants Conduktor a perpetual, irrevocable, non-exclusive, worldwide, fully-paid, sub-licensable, assignable license to incorporate into the Conduktor Software or otherwise use any Feedback Conduktor receives from Customer. Customer also irrevocably waives in favor of Conduktor any moral rights which Customer may have in such Feedback pursuant to applicable copyright law. Conduktor acknowledges that any Feedback is provided on an “as-is” basis with no warranties of any kind. Feedback shall constitute Confidential Information of Conduktor for purposes of Section 6. **CONFIDENTIALITY** 1. Nondisclosure and Limited Use. Each party shall retain in confidence the non-public information and know-how disclosed or made available by the other party pursuant to this Agreement which is either designated in writing as proprietary and/or confidential, if disclosed in writing, or if disclosed orally, is designated in writing (which may be via email) as confidential within thirty (30) days of the oral disclosure or should reasonably be understood to be confidential by the recipient (the “**Confidential Information**”). The Conduktor Software, Modifications, and the terms and conditions of this Agreement shall be Conduktor’s Confidential Information. Each party agrees to: (a) maintain the confidentiality of the other party’s Confidential Information; (b) refrain from using the other party’s Confidential Information except for the purpose of performing its obligations under this Agreement; and (c) not disclose such Confidential Information to any third party except to employees and subcontractors as is reasonably required in connection with the exercise of its rights and obligations under this Agreement (and only subject to binding written use and disclosure restrictions at least as protective as those set forth herein). Each party shall immediately notify the other party of any unauthorized disclosure or use of any Confidential Information and assist the other party in remedying such unauthorized use or disclosure by taking such steps as are reasonably requested by such other party. The foregoing obligations will not apply to Confidential Information of the other party which is: (i) already publicly known without breach of this Agreement; (ii) discovered or created by the receiving party without use of, or reference to, the Confidential Information of the disclosing party, as shown in records of the receiving party; (iii) otherwise known to the receiving party through no wrongful conduct of the receiving party, or (iv) required to be disclosed by law or court order; provided that the receiving party shall provide prompt notice thereof and reasonable assistance to the disclosing party to enable the disclosing party to seek a protective order or otherwise prevent or restrict such disclosure. Moreover, either party hereto may disclose any Confidential Information hereunder to such party’s agents, attorneys and other representatives (and only subject to confidentiality obligations at least as protective as those set forth herein) or any court of competent jurisdiction as reasonably required to resolve any dispute between the parties hereto. 2. Remedies. Any breach or threatened breach of this Section may cause irreparable injury to the disclosing party and, in addition to any other remedies that may be available, in law, in equity or otherwise, the disclosing party shall be entitled to seek injunctive relief against the threatened breach of this Agreement or the continuation of any such breach by the receiving party, without the necessity of proving actual damages or posting any bond, in addition to any other rights or remedies provided by law. ### **INDEMNITY** 1. **Conduktor shall, at its own expense, defend or at its option, settle, any claim or action brought against Customer by a third party to the extent it is based on a claim that the Conduktor Software, as may be updated, from time to time, by Conduktor and used in accordance with this Agreement, infringes any patent, copyright, or any trade secret of such third party. Furthermore, Conduktor will indemnify and hold Customer harmless from and against damages, costs, and fees reasonably incurred (including reasonable attorneys' fees) that are attributable exclusively to such claim or action and which are assessed against Customer in a final judgment. Conduktor’s obligations to defend, settle, or indemnify Customer are subject to (a) Customer promptly notifying Conduktor in writing of such claim; (b) Conduktor having the exclusive right to control such defense and/or settlement; and (c) Customer providing reasonable assistance (at Conduktor's expense) in the defense thereof. Customer shall not settle any claim, action or proceeding without Conduktor’s prior written approval.** 2. **CONDUKTOR SHALL NOT DEFEND, INDEMNIFY, OR HOLD CUSTOMER HARMLESS FOR ANY CLAIM IF: (A) CUSTOMER MADE MODIFICATIONS TO THE CONDUKTOR SOFTWARE OR PORTIONS THEREOF; (B) SUCH CLAIM WOULD HAVE BEEN AVOIDED BY USE OF THE THEN CURRENT RELEASE OF THE CONDUKTOR SOFTWARE MADE AVAILABLE TO CUSTOMER; (C) CUSTOMER CONTINUED ITS ALLEGEDLY INFRINGING ACTIVITY AFTER BEING PROVIDED WITH MODIFICATIONS THAT WOULD HAVE AVOIDED THE ALLEGED INFRINGEMENT; OR (D) SUCH CLAIM IS BASED ON CUSTOMER’S OUTPUT.** 3. **IF CONDUKTOR DEFENDS OR SETTLES AN INFRINGEMENT CLAIM ARISING UNDER SECTION 7.1 ABOVE, CONDUKTOR’S LIABILITY AND CUSTOMER’S SOLE AND EXCLUSIVE REMEDY (IN ADDITION TO THE “INDEMNITY”) SHALL BE FOR CONDUKTOR AT ITS OWN EXPENSE, TO EITHER (A) REPAIR, REPLACE OR MODIFY THE AFFECTED CONDUKTOR SOFTWARE OR (B) ALTERNATIVELY, PROCURE FOR CUSTOMER THE RIGHT TO CONTINUE TO USE THE AFFECTED CONDUKTOR SOFTWARE. IF THE FOREGOING REMEDIES ARE NOT COMMERCIALLY FEASIBLE (IN THE REASONABLE OPINION OF CONDUKTOR), CONDUKTOR MAY CANCEL THE APPLICABLE ORDER AND, AS APPLICABLE, FOR THE AFFECTED CONDUKTOR SOFTWARE, REFUND THE LICENSE FEES PAID TO CONDUKTOR BY CUSTOMER FOR THE AFFECTED CONDUKTOR SOFTWARE.** ### **DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITY** 1. Disclaimer of Any Warranties. CONDUKTOR MAKES NO WARRANTIES, EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH RESPECT TO THE CONDUKTOR SOFTWARE OR ANY OTHER MATERIALS PROVIDED HEREUNDER. CONDUKTOR SPECIFICALLY DISCLAIMS ALL OTHER WARRANTIES, EXPRESS AND IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT AND THOSE ARISING FROM A COURSE OF DEALING OR USAGE OR TRADE, AND ALL SUCH WARRANTIES ARE HEREBY EXCLUDED TO THE FULLEST EXTENT PERMITTED BY LAW. THE CONDUKTOR SOFTWARE IS PROVIDED ON AN “AS IS” AND “AS AVAILABLE” BASIS. 2. Limitation of Liability\*\*.\*\* IN NO EVENT SHALL CONDUKTOR’S LIABILITY ARISING UNDER THIS AGREEMENT EXCEED THE AMOUNT PAID OR PAYABLE BY CUSTOMER DURING THE TWELVE (12) MONTHS IMMEDIATELY PRIOR TO THE EVENT GIVING RISE TO SUCH LIABILITY. CONDUKTOR WILL NOT BE LIABLE TO CUSTOMER FOR ANY CONSEQUENTIAL, INCIDENTAL, SPECIAL, INDIRECT, PUNITIVE OR EXEMPLARY DAMAGES, INCLUDING WITHOUT LIMITATION LOST PROFITS, BUSINESS, CONTRACTS, REVENUE, GOODWILL, PRODUCTION, ANTICIPATED SAVINGS, LOSS OF DATA, OR COSTS OF PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES, OR FOR ANY CLAIM OR DEMAND BY ANY OTHER PARTY, HOWEVER CAUSED AND (TO THE FULLEST EXTENT PERMITTED BY LAW) UNDER ANY THEORY OF LIABILITY (INCLUDING NEGLIGENCE) EVEN IF CONDUKTOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. CUSTOMER ACKNOWLEDGES THAT THESE LIMITATIONS SHALL APPLY NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED REMEDY. ### **TERM AND TERMINATION** 1. Term. Unless terminated earlier in accordance with Section 9.2 below, this Agreement and the license granted hereunder will be effective as of the date Customer purchases the Conduktor Software (the “**Effective Date**”) and shall continue for a term of one year from the Effective Date (together with each renewal term, the “**Term**”). Thereafter, the license shall automatically renew for successive one-year terms at the then current list price (except as otherwise provided for in a separate written agreement) unless Customer notifies Conduktor in writing, such notice to be sent by email to [support@conduktor.io](mailto:support@conduktor.io), at least sixty (60) days prior to the end of the current term of Customer’s intention to terminate the license. 2. Termination. The license granted hereunder shall immediately terminate if Customer breaches Section 1 of this Agreement, regardless of whether Conduktor notifies Customer of such termination. In addition, Conduktor shall have the right to terminate this Agreement and the license granted herein upon written notice in the event the Customer fails to perform or observe any material term or condition of this Agreement and such default has not been cured no later than ten (10) days after written notice of such default to the Customer. Conduktor may also terminate this Agreement immediately if the Customer: (a) terminates or suspends its business; (b) becomes subject to any bankruptcy or insolvency proceeding under Federal or state statute; (c) becomes insolvent or subject to direct control by a trustee, receiver or similar authority; or (d) has wound up or liquidated, voluntarily or otherwise. 3. Effect of Termination. This Section 9.3 and Sections 3, 6, 7 (solely with respect to claims arising during the Term), 8 and 10 survive any termination or expiration of this Agreement. No other provisions of this Agreement survive the expiration or earlier termination of this Agreement. All of Customer’s rights in the Conduktor Software will terminate immediately upon termination of this Agreement. Customer shall not be entitled to any refund of fees paid. No later than five (5) days after termination of this Agreement, Customer shall return to Conduktor or, upon Conduktor’s request, destroy or render inaccessible, at Customer’s sole expense, all Confidential Information of Conduktor and materials containing any Confidential Information of Conduktor, and discontinue use of and uninstall the Conduktor Software, including all copies thereof. Nothing contained herein shall limit any other remedies that Conduktor may have for the default of Customer under this Agreement nor relieve Customer of any of its obligations incurred prior to such termination. ## **MISCELLANEOUS** 1. Publicity. Conduktor may identify Customer as a user of the Conduktor Software and may use Customer’s name, logo, and other trademarks in Conduktor’s customer list, press releases, blog posts, advertisements, and website (and all use thereof and goodwill arising therefrom shall inure to the sole and exclusive benefit of Customer). Customer may deny Conduktor Inc. this right at any time by submitting a written request via email to [support@conduktor.io](mailto:support@conduktor.io), requesting to be excluded from Conduktor Software promotional material. Requests made after Purchasing may take thirty (30) calendar days to process. Otherwise, neither party may use the name, logo, or other trademarks of the other party for any purpose without the other party’s prior written approval. 2. Force Majeure. In no event shall Conduktor be liable to Customer, or be deemed to have breached this Agreement, for any failure or delay in performing its obligations under this Agreement, if and to the extent such failure or delay is caused by any circumstances beyond Conduktor’s reasonable control, including but not limited to acts of God, flood, fire, earthquake, explosion, war, terrorism, invasion, riot or other civil unrest, strikes, labor stoppages or slowdowns or other industrial disturbances, or passage of law or any action taken by a governmental or public authority, including imposing an embargo. 3. Assignment. Customer shall not assign or otherwise transfer this Agreement or any rights or obligations hereunder, in whole or in part, whether by operation of law or otherwise, to any third party without Conduktor’s prior written consent. Any purported transfer, assignment or delegation without such prior written consent will be null and void and of no force or effect. Conduktor shall have the right to assign this Agreement to any successor to its business or assets to which this Agreement relates, whether by merger, sale of assets, sale of stock, reorganization or otherwise. Subject to this Section, this Agreement shall be binding upon and inure to the benefit of the parties hereto, and their respective successors and permitted assigns. 4. Entire Agreement; Modification; Waiver. This Agreement represents the entire agreement between the parties, and supersedes all prior agreements and understandings, written or oral, with respect to the matters covered by this Agreement, and is not intended to confer upon any third party any rights or remedies hereunder. Customer acknowledges that it has not entered in this Agreement based on any representations other than those contained herein. No modification of or amendment to this Agreement, nor any waiver of any rights under this Agreement, shall be effective unless in writing and signed by both parties. The waiver of one breach or default or any delay in exercising any rights shall not constitute a waiver of any subsequent breach or default. 5. Third Party Software, Services, and Webites. 1. Conduktor makes available certain third party open source software (“**Third Party Software**”). The Third Party Software shall be subject to the applicable open source license(s) and not this Agreement, and is provided by Conduktor at no charge. To the extent the terms of open source licenses applicable to Third Party Software prohibit any of the restrictions in this Agreement, such restrictions will not apply to such Third Party Software. To the extent the terms of open source licenses applicable to Third Party Software require Conduktor to make an offer to provide source code or related information in connection with the Third Party Software, such offer is made. 2. The Conduktor Software may contain links to other websites, services or software provided by third parties (“**Third-Party Content**”). Links from the Conduktor Software to Third-Party Content are provided for Customer convenience only and are not investigated, monitored or checked for accuracy, appropriateness, or completeness by Conduktor, and Conduktor is not responsible for any Third-Party Content accessed through the Conduktor Software, including without limitation the content, accuracy, offensiveness, opinions, reliability, privacy practices or other policies of or contained in the Third-Party Content. Inclusion of, linking to, or permitting the use or installation of any Third-Party Content does not imply approval or endorsement thereof by Conduktor. If Customer decides to access or install any Third-Party Content, Customer does so at its own risk. Customer should review the applicable terms and policies, including privacy and data gathering practices, of any Third-Party Content that you use or install. Conduktor reserves the right to change, modify or delete links to Third-Party Content without notice. 6. Governing Law. This Agreement shall in all respects be governed by the laws of the State of New York without reference to its principles of conflicts of laws. The parties hereby agree that all disputes arising out of this Agreement shall be subject to the exclusive jurisdiction of and venue in the federal and state courts within the City and County of New York, New York. Customer hereby consents to the personal and exclusive jurisdiction and venue of these courts. The parties hereby disclaim and exclude the application hereto of the United Nations Convention on Contracts for the International Sale of Goods. 7. Arbitration. Read this section carefully because it requires the parties to arbitrate their disputes and limits the manner in which you can seek relief from Conduktor. For any dispute with Conduktor, you agree to first contact us at [support@conduktor.io](mailto:support@conduktor.io) and attempt to resolve the dispute with us informally. In the unlikely event that Conduktor has not been able to resolve a dispute it has with you after sixty (60) days, we each agree to resolve any claim, dispute, or controversy (excluding any claims for injunctive or other equitable relief as provided below) arising out of or in connection with or relating to this Agreement, or the breach or alleged breach thereof (collectively, “**Claims**”), by binding arbitration by JAMS, under the Optional Expedited Arbitration Procedures then in effect for JAMS, except as provided herein. JAMS may be contacted at [www.jamsadr.com](http://www.jamsadr.com/). The arbitration will be conducted in New York, New York, unless you and Conduktor agree otherwise. If you are using the Conduktor Software for commercial purposes, each party will be responsible for paying any JAMS filing, administrative and arbitrator fees in accordance with JAMS rules, and the award rendered by the arbitrator shall include costs of arbitration, reasonable attorneys’ fees and reasonable costs for expert and other witnesses. If you are an individual using the Conduktor Software for non-commercial purposes: (i) JAMS may require you to pay a fee for the initiation of your case, unless you apply for and successfully obtain a fee waiver from JAMS; (ii) the award rendered by the arbitrator may include your costs of arbitration, your reasonable attorney’s fees, and your reasonable costs for expert and other witnesses; and (iii) you may sue in a small claims court of competent jurisdiction without first engaging in arbitration, but this does not absolve you of your commitment to engage in the informal dispute resolution process. Any judgment on the award rendered by the arbitrator may be entered in any court of competent jurisdiction. Nothing in this Section shall be deemed as preventing Conduktor from seeking injunctive or other equitable relief from the courts as necessary to prevent the actual or threatened infringement, misappropriation, or violation of our data security, intellectual property rights or other proprietary rights. 8. Class Action/Jury Trial Waiver. WITH RESPECT TO ALL PERSONS AND ENTITIES, REGARDLESS OF WHETHER THEY HAVE OBTAINED OR USED THE CONDUKTOR SOFTWARE FOR PERSONAL, COMMERCIAL OR OTHER PURPOSES, ALL CLAIMS MUST BE BROUGHT IN THE PARTIES’ INDIVIDUAL CAPACITY, AND NOT AS A PLAINTIFF OR CLASS MEMBER IN ANY PURPORTED CLASS ACTION, COLLECTIVE ACTION, PRIVATE ATTORNEY GENERAL ACTION OR OTHER REPRESENTATIVE PROCEEDING. THIS WAIVER APPLIES TO CLASS ARBITRATION, AND, UNLESS WE AGREE OTHERWISE, THE ARBITRATOR MAY NOT CONSOLIDATE MORE THAN ONE PERSON’S CLAIMS. YOU AGREE THAT, BY ENTERING INTO THIS AGREEMENT, YOU AND CONDUKTOR ARE EACH WAIVING THE RIGHT TO A TRIAL BY JURY OR TO PARTICIPATE IN A CLASS ACTION, COLLECTIVE ACTION, PRIVATE ATTORNEY GENERAL ACTION, OR OTHER REPRESENTATIVE PROCEEDING OF ANY KIND. 9. Severability. If any provision of this Agreement is held invalid or unenforceable under applicable law by a court of competent jurisdiction, it shall be replaced with the valid provision that most closely reflects the intent of the parties and the remaining provisions of the Agreement will remain in full force and effect. 10. Relationship of the Parties. Nothing in this Agreement is to be construed as creating an agency, partnership, or joint venture relationship between the parties hereto. Neither party shall have any right or authority to assume or create any obligations or to make any representations or warranties on behalf of any other party, whether express or implied, or to bind the other party in any respect whatsoever. 11. Notices. All notices permitted or required under this Agreement shall be in writing and shall be deemed to have been given when delivered in person (including by overnight courier), or three (3) business days after being mailed by first class, registered or certified mail, postage prepaid, to the address of the party specified in this Agreement or such other address as either party may specify in writing, or when sent if by email. It is your responsibility to keep your email address(es) current. You will be deemed to have received any email sent to the email address then associated with your account when we send the email, whether or not you actually receive the email. Notices made by you to Conduktor should be sent via email to [legal@conduktor.io](mailto:legal@conduktor.io). 12. U.S. Government Restricted Rights. If Conduktor Software is being licensed by the U.S. Government, the Conduktor Software is deemed to be “commercial computer software” and “commercial computer documentation” developed exclusively at private expense, and (a) if acquired by or on behalf of a civilian agency, shall be subject solely to the terms of this computer software license as specified in 48 C.F.R. 12.212 of the Federal Acquisition Regulations and its successors; and (b) if acquired by or on behalf of units of the Department of Defense (“**DOD**”) shall be subject to the terms of this commercial computer software license as specified in 48 C.F.R. 227.7202-2, DOD FAR Supplement and its successors. 13. Export Law Assurances. Customer understands that the Conduktor Software is subject to export control laws and regulations. Customer may not download or otherwise export or re-export the Conduktor Software or any underlying information or technology except in full compliance with all applicable laws and regulations, in particular, but without limitation, United States export control laws. None of the Conduktor Software or any underlying information or technology may be downloaded or otherwise exported or re- exported: (a) into (or to a national or resident of) any country to which the United States has embargoed goods; or (b) to anyone on the U.S. Treasury Department’s list of specially designated nationals or the U.S. Commerce Department’s list of prohibited countries or debarred or denied persons or entities. Customer hereby agrees to the foregoing and represents and warrants that customer is not located in, under control of, or a national or resident of any such country or on any such list. 14. Construction. The titles and section headings used in this Agreement are for ease of reference only and shall not be used in the interpretation or construction of this Agreement. No rule of construction resolving any ambiguity in favor of the non-drafting party shall be applied hereto. The word “including”, when used herein, is illustrative rather than exclusive and means “including, without limitation.” This website is operated by Conduktor Inc. If you have any questions, complaints, or concerns about how your information is handled, or would like to exercise any of your rights for yourself or on behalf of someone else, please email us at #### support@conduktor.io Source: https://www.conduktor.io/legal/eula-enterprise --- # EULA: SaaS Services Updated June 22, 2026 **END USER LICENSE AGREEMENT — SAAS SERVICES** This End User License Agreement ("**Agreement**") is a legal agreement between you (either an individual or an entity) ("**Customer**") and Conduktor Inc., a Delaware corporation ("**Conduktor**"), and governs your access to and use of Conduktor's hosted, cloud-based data platform services made available by Conduktor under a free license, including the "Ivy" hosted platform and any associated web applications, application programming interfaces, and documentation (collectively, the "**SaaS Services**"). The SaaS Services are intended for commercial and internal business use only. This Agreement applies to free, self-service access to the SaaS Services. Conduktor's on-premises and downloadable software products are not covered by this Agreement and are governed by separate terms. Access to and use of the SaaS Services is conditioned upon your compliance with this Agreement and all applicable local, state, national and international laws, rules and regulations. Your access to or use of the SaaS Services indicates your acceptance of this Agreement. We may modify this Agreement (including any policies we reference) at any time by posting a revised version on the Conduktor website or by otherwise notifying you in accordance with the Notices section (Section 10.7). The modified Agreement will become effective upon posting or, if we notify you by email, as stated in the email message. By continuing to access or use the SaaS Services after the effective date of any modifications to this Agreement, you agree to be bound by the modified Agreement. We last modified this Agreement on the date listed at the beginning of this Agreement. IF YOU ARE AN INDIVIDUAL AGREEING TO THE TERMS OF THIS AGREEMENT ON BEHALF OF AN ENTITY, SUCH AS YOUR EMPLOYER, YOU REPRESENT THAT YOU HAVE THE LEGAL AUTHORITY TO BIND THAT ENTITY AND "CUSTOMER" SHALL REFER HEREIN TO SUCH ENTITY. IF YOU ARE AN INDIVIDUAL AGREEING TO THIS AGREEMENT IN YOUR INDIVIDUAL CAPACITY (AND NOT ON BEHALF OF AN ENTITY), YOU ACKNOWLEDGE AND AGREE THAT YOU MAY ACCESS AND USE THE SAAS SERVICES ONLY FOR COMMERCIAL AND INTERNAL BUSINESS PURPOSES (AND NOT FOR PERSONAL, FAMILY, HOUSEHOLD, CONSUMER, OR OTHER NON-COMMERCIAL PURPOSES). IF YOU DO NOT HAVE SUCH AUTHORITY, OR IF YOU DO NOT AGREE WITH THE TERMS OF THIS AGREEMENT, YOU MUST NOT ACCEPT THIS AGREEMENT AND MAY NOT ACCESS OR USE THE SAAS SERVICES. **THE SAAS SERVICES ARE PROVIDED FREE OF CHARGE, STRICTLY ON AN "AS IS" AND "AS AVAILABLE" BASIS, WITHOUT ANY SUPPORT, WITHOUT ANY WARRANTY OF ANY KIND, AND WITHOUT ANY SERVICE LEVEL, UPTIME, OR AVAILABILITY COMMITMENT. SEE SECTION 8.** **1. ACCESS AND LICENSE** 1.1 **Access Grant.** Subject to the terms of this Agreement, Conduktor grants to Customer a limited, non-exclusive, non-sublicensable, non-transferable, revocable right during the applicable Term (as defined below) to access and use the SaaS Services solely for Customer's internal business operations, and subject to any usage limits made available by Conduktor. For clarity, the SaaS Services may not be used for personal, family, household, consumer, or other non-commercial purposes. "**Authorized User**" means any unique individual user who is authorized by Customer to access or use the SaaS Services under Customer's account. Customer is solely responsible for all activity occurring under its account and for the acts and omissions of its Authorized Users. 1.2 **Additional Restrictions and Limitations.** Customer shall not, and shall not permit or encourage any of Customer's Authorized Users, or any employee, contractor, consultant, or third party to: (a) access or use the SaaS Services for personal, family, household, consumer, or other non-commercial purposes; (b) sublicense, sell, resell, rent, lease, distribute, or pledge the SaaS Services or any of the rights under this Agreement, or make the SaaS Services available to any third party except as expressly permitted; (c) use the SaaS Services to provide hosting, managed-service, time-sharing, service-bureau, or outsourced services to any third party, or otherwise for the benefit of any person other than Customer; (d) provide access to the SaaS Services to anyone other than an Authorized User; any access to or use of the SaaS Services by a person who is not an Authorized User is a material breach of this Agreement; (e) use any third-party or open-source software in conjunction with the SaaS Services in a manner that would subject the SaaS Services to any open-source or third-party license requiring its public disclosure, distribution, or licensing for derivative works; (f) modify, translate, decrypt, create derivative works from, disassemble, decompile, or reverse-engineer the SaaS Services or any software underlying the SaaS Services, or otherwise attempt to access any source code or underlying ideas, algorithms, structure, or organization (except to the extent applicable law prohibits this restriction); (g) bypass or breach any security device or protection used by the SaaS Services; access or use the SaaS Services other than as an Authorized User with valid credentials; share, transfer, or disclose any credentials; permit any other person to use an Authorized User account or credentials; or use shared logins or shared accounts; (h) access or use the SaaS Services to build, train, or improve any competing or similar product or service, or to benchmark or conduct competitive analysis, or to copy any features, functions, or graphics of the SaaS Services; (i) input, upload, transmit, store, or otherwise provide to or through the SaaS Services any unlawful, infringing, or injurious information or materials, or any harmful, malicious, or disabling code; (j) interfere with or disrupt the integrity, performance, or operation of the SaaS Services or the data contained therein, or attempt to gain unauthorized access to the SaaS Services or their related systems or networks, including by means of any automated scraping, crawling, denial-of-service, or load-testing activity; or (k) use the SaaS Services in any manner or for any purpose that infringes, misappropriates, or otherwise violates any intellectual property right or other right of any person, or that violates any applicable law. 1.3 **Suspension; Reservation of Rights.** Conduktor reserves the right to restrict, suspend, disable, throttle, or terminate Customer's access to the SaaS Services (in whole or in part) at any time, in its sole discretion, and without liability to Customer, and may provide notice where practicable; provided that Conduktor may take such action immediately and without prior notice in the event: (a) Conduktor reasonably believes that Customer's use of the SaaS Services may result in liability to Conduktor or any third party, may involve use for personal, family, household, consumer, or other non-commercial purposes, or may adversely affect the integrity, functionality, security, or availability of the SaaS Services or any other customer's use; or (b) Customer is in breach of this Agreement. Conduktor is not responsible for any damages, liabilities, or losses of Customer resulting from any such restriction, suspension, disablement, throttling, or termination. Conduktor reserves all rights not expressly granted in this Agreement. No rights are granted by implication, estoppel, or otherwise. 1.4 **Usage Limits.** The SaaS Services are subject to usage limits as specified in the applicable product documentation or on the Conduktor website (including limits on the number of clusters, users, calls, throughput, storage, or other metrics), which usage limits are incorporated by reference into, and form part of, this Agreement as binding contractual restrictions. Customer shall not exceed, or attempt to circumvent, any such usage limits, including by creating multiple accounts or organizations, and any such exceeding or circumvention is a material breach of this Agreement. Conduktor may modify, introduce, or reduce usage limits at any time in its sole discretion. 1.5 **Changes to the SaaS Services.** Conduktor may, at any time and in its sole discretion, modify, update, enhance, suspend, or discontinue the SaaS Services or any feature, function, or component thereof, in whole or in part, with or without notice, and without any liability to Customer. Conduktor is under no obligation to maintain, continue, or make available the SaaS Services or any feature thereof, or to retain or preserve any Customer Data following discontinuation. Customer acknowledges that the SaaS Services are provided on an evolving basis and that backward compatibility is not guaranteed. 1.6 **No Support or Maintenance.** Conduktor is not obligated to provide any support, maintenance, updates, upgrades, error correction, or other services for the SaaS Services. Customer may access community support resources (such as documentation, forums, or community channels) to the extent Conduktor makes them available, but Conduktor provides no guarantee of response, resolution, availability, or correction. Any assistance Conduktor may elect to provide is at its sole discretion, on an "as is" basis, and may be discontinued at any time. 1.7 **No Service Levels or Availability Commitment.** Conduktor makes no commitment as to the uptime, availability, response time, reliability, performance, or accessibility of the SaaS Services. The SaaS Services may be unavailable, interrupted, suspended, degraded, or discontinued at any time, including for maintenance, updates, capacity, security, or any other reason or for no reason, in each case without notice and without liability to Customer. No service level agreement, availability target, or credit applies to this Agreement. 1.8 **Customer Data; Backups.** Customer is solely responsible for its Customer Data, including its accuracy, quality, legality, and the means by which Customer acquired it, and for maintaining its own backups. "**Customer Data**" means any data, content, or materials that Customer or its Authorized Users input, upload, transmit, or otherwise provide to or through the SaaS Services. Conduktor has no obligation to store, retain, back up, or preserve Customer Data, and Customer acknowledges that Customer Data may be deleted, lost, corrupted, or rendered inaccessible at any time. Conduktor will have no liability of any kind for any loss, deletion, corruption, or inaccessibility of Customer Data. To the extent Conduktor Processes (i.e., performs any operation or set of operations on, whether or not by automated means) any personal data contained in Customer Data on Customer's behalf, such Processing is governed by the Data Processing Agreement (available at [DPA: SaaS](https://www.conduktor.io/legal/dpa-saas)) made available by Conduktor, which is incorporated by reference. "Process," "Processes," and "Processed" are interpreted accordingly. Default and maximum retention periods for Customer Data are as described in the applicable product documentation and Conduktor's Privacy Policy (available at [Privacy Policy](https://www.conduktor.io/legal/privacy-policy)). 1.9 **Preview Releases.** Conduktor may make available pre-release versions of the SaaS Services or features under development ("**Beta Features**") at no charge. Beta Features will be clearly designated as beta, pilot, limited release, developer preview, non-production, evaluation, or by a description of similar import. Beta Features are for evaluation purposes only and not for production use, are not supported, and may be subject to additional terms. Conduktor may discontinue Beta Features at any time in its sole discretion and is under no obligation to release a generally available version of any Beta Feature. Conduktor will have no liability for any harm or damage arising out of or in connection with any Beta Feature. **2. USER, PERFORMANCE, AND PERSONAL DATA** 2.1 **Telemetry Data.** Certain features of the SaaS Services collect and transmit technical telemetry and usage data to Conduktor ("**Telemetry Data**"). Telemetry Data includes information such as feature usage patterns, error logs, performance metrics, and deployment or configuration data. Conduktor will Process Telemetry Data to: (a) maintain, improve, and secure the SaaS Services; (b) monitor product performance and diagnose technical issues; and (c) generate aggregated and anonymized product analytics. Conduktor will not sell Telemetry Data to third parties. 2.2 **Account Registration Data.** Customer must provide a valid email address and other registration details as part of the account registration process. Conduktor may use such data to administer Customer's account and to communicate with Customer regarding the SaaS Services, security matters, and material product changes. It is Customer's responsibility to keep such information current. For account registration data, Conduktor acts as an independent data controller and will Process such data in accordance with applicable data protection laws and Conduktor's Privacy Policy. 2.3 **Personal Data in Customer Data.** Customer is solely responsible for any personal data it inputs into, uploads to, or otherwise Processes using the SaaS Services, including for establishing and maintaining a lawful basis for such Processing, providing all required notices, and obtaining all required consents. To the extent Conduktor Processes personal data contained in Customer Data on Customer's behalf, Conduktor acts as a processor (or, where Customer is itself a processor, as a sub-processor) and such Processing is governed by the Data Processing Agreement. Conduktor's collection and use of personal data in its capacity as a controller is described in Conduktor's Privacy Policy. 2.4 **Aggregated and Deidentified Data.** Notwithstanding anything to the contrary, Conduktor may collect, generate, and use aggregated, anonymized, and deidentified data derived from the operation and use of the SaaS Services (which does not identify Customer, any Authorized User, or any individual) for any lawful business purpose, including to operate, improve, and develop the SaaS Services and Conduktor's other products and services. **3. OWNERSHIP** Customer acknowledges that Conduktor or its licensors retain all proprietary rights, title and interest, including all intellectual property rights, in and to the SaaS Services and any changes, corrections, bug fixes, enhancements, updates, and other modifications to them made by or on behalf of Conduktor (collectively, "**Modifications**"), and as between the parties all such rights will vest in Conduktor. For clarity, "Modifications" do not include, and Conduktor does not claim any ownership interest in, Customer Data or Customer's independent configurations, scripts, integrations, connectors, or other materials that Customer develops that do not incorporate Conduktor Confidential Information or Conduktor code. Customer grants Conduktor a non-exclusive, worldwide, royalty-free right to host, copy, transmit, display, and Process Customer Data solely as necessary to provide and operate the SaaS Services and as otherwise permitted under this Agreement. Customer acknowledges that the rights granted in Section 1 are limited in scope as set out in that Section. **4. FINANCIAL TERMS** 4.1 The SaaS Services made available under this Agreement are provided at no charge. Conduktor reserves the right to introduce fees for features, capacity, or usage tiers not covered by this Agreement; any such fees will be subject to a separate agreement. 4.2 To the extent any fees become payable in connection with use of the SaaS Services, such fees are exclusive of all applicable taxes and government charges, which Customer will be responsible for paying. **5. FEEDBACK** Customer is under no duty to provide any suggestions, enhancement requests, or other feedback regarding the SaaS Services ("**Feedback**"). If Customer chooses to offer Feedback to Conduktor, Customer grants Conduktor a perpetual, irrevocable, non-exclusive, worldwide, fully-paid, sub-licensable, assignable license to incorporate into the SaaS Services or otherwise use any Feedback Conduktor receives from Customer. Conduktor acknowledges that any Feedback is provided on an "as-is" basis with no warranties of any kind. Feedback will constitute Confidential Information of Conduktor for purposes of Section 6. **6. CONFIDENTIALITY** 6.1 **Nondisclosure and Limited Use.** "**Confidential Information**" means non-public information disclosed by one party to the other under this Agreement that is: (a) marked as confidential or proprietary in writing; (b) if disclosed orally, confirmed as confidential in writing (including email) within thirty (30) days; or (c) information that a reasonable recipient would understand to be confidential. The SaaS Services and Modifications are Conduktor's Confidential Information. Each party agrees to: (a) keep the other party's Confidential Information confidential; (b) use it only to perform its obligations or exercise its rights under this Agreement; and (c) not disclose it to any third party, except to employees and subcontractors who need access and are bound by confidentiality obligations at least as protective as this Section. Each party must promptly notify the other of any unauthorized disclosure or use and take reasonable steps to help remedy it. These confidentiality obligations do not apply to information that: (i) is or becomes publicly available without breach of this Agreement; (ii) the receiving party independently developed without reference to the disclosing party's Confidential Information; (iii) the receiving party lawfully obtained from another source without confidentiality restrictions; or (iv) must be disclosed by law or court order, provided the receiving party gives prompt notice and reasonable assistance so the disclosing party can seek a protective order. Either party may also disclose Confidential Information to its agents, attorneys, and other representatives (subject to confidentiality obligations at least as protective as this Section) or to any court as reasonably required to resolve a dispute under this Agreement. 6.2 **Remedies.** A breach or threatened breach of this Section may cause harm that cannot be adequately remedied by monetary damages alone. In such cases, the disclosing party may seek injunctive or other equitable relief without having to prove actual damages or post a bond, in addition to any other remedies available at law. **7. INDEMNITY** **No Conduktor Indemnity.** Customer acknowledges and agrees that, given that the SaaS Services are provided free of charge and on an "as is" basis, Conduktor provides no indemnity of any kind, including no defense or indemnity against any claim that the SaaS Services infringe, misappropriate, or otherwise violate any intellectual property or other right of any third party. Customer assumes all risk in this regard. **Customer Indemnity.** Customer will defend, indemnify, and hold harmless Conduktor, its affiliates, and each of their respective licensors, officers, directors, employees, and agents from and against any and all third-party claims, demands, suits, or proceedings, and all related damages, losses, liabilities, penalties, costs, and expenses (including reasonable legal fees) arising out of or relating to: (a) any allegation that Customer Data, or Customer's processing, transmission, storage, or other handling of Customer Data, violates applicable laws or infringes, misappropriates, or otherwise violates any third-party right; (b) Customer's breach of this Agreement, including any breach of the restrictions in Section 1 or of Customer's export controls and sanctions obligations, or any use of the SaaS Services in violation of applicable laws; (c) Customer's systems, networks, infrastructure, configurations, credentials, or access controls (including any security incident, unauthorized access, or failure to maintain appropriate security measures in Customer's environment); or (d) Customer's use of the SaaS Services in combination with any third-party products, services, software, systems, or data, to the extent the claim would not have arisen but for such combination. **Indemnity Procedures.** Conduktor will promptly notify Customer in writing of any claim for which it seeks indemnification; provided that any failure to provide prompt notice will not relieve Customer of its obligations except to the extent Customer is materially prejudiced by such failure. Customer will have sole control of the defense and settlement of the claim and will pay all defense costs and any settlement amounts or damages awarded; provided that Customer may not settle any claim in a manner that admits fault or liability on behalf of, or imposes any obligation on, Conduktor or any of its affiliates without Conduktor's prior written consent (not to be unreasonably withheld or delayed). Conduktor will provide reasonable cooperation in the defense of the claim at Customer's expense. **8. DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITY** **DISCLAIMER OF ALL WARRANTIES. THE SAAS SERVICES AND ALL OTHER MATERIALS, INFORMATION, AND OUTPUTS PROVIDED OR MADE AVAILABLE BY CONDUKTOR ARE PROVIDED STRICTLY ON AN "AS IS" AND "AS AVAILABLE" BASIS, WITH ALL FAULTS AND WITHOUT WARRANTY OF ANY KIND. CONDUKTOR MAKES NO WARRANTIES, REPRESENTATIONS, OR CONDITIONS, EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH RESPECT TO THE SAAS SERVICES OR ANY OTHER MATERIALS PROVIDED HEREUNDER, AND SPECIFICALLY DISCLAIMS ALL WARRANTIES, REPRESENTATIONS, AND CONDITIONS, EXPRESS AND IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, AND NONINFRINGEMENT, AND ANY WARRANTIES ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE. WITHOUT LIMITING THE FOREGOING, CONDUKTOR DOES NOT WARRANT THAT THE SAAS SERVICES WILL BE UNINTERRUPTED, AVAILABLE, TIMELY, SECURE, ERROR-FREE, OR FREE OF HARMFUL COMPONENTS, THAT ANY DATA WILL BE ACCURATE, PRESERVED, OR NOT LOST, OR THAT THE SAAS SERVICES WILL MEET CUSTOMER'S REQUIREMENTS. ALL SUCH WARRANTIES ARE EXCLUDED TO THE FULLEST EXTENT PERMITTED BY LAW. CUSTOMER'S USE OF THE SAAS SERVICES IS ENTIRELY AT CUSTOMER'S OWN RISK.** **EXCLUSION OF DAMAGES. TO THE FULLEST EXTENT PERMITTED BY LAW, CONDUKTOR WILL NOT BE LIABLE TO CUSTOMER OR ANY THIRD PARTY FOR ANY CONSEQUENTIAL, INCIDENTAL, SPECIAL, INDIRECT, PUNITIVE, OR EXEMPLARY DAMAGES, OR FOR ANY LOST PROFITS, BUSINESS, CONTRACTS, REVENUE, GOODWILL, PRODUCTION, ANTICIPATED SAVINGS, LOSS OR CORRUPTION OF OR INABILITY TO ACCESS DATA, OR COSTS OF PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES, OR FOR ANY CLAIM OR DEMAND BY ANY OTHER PARTY, HOWEVER CAUSED AND (TO THE FULLEST EXTENT PERMITTED BY LAW) UNDER ANY THEORY OF LIABILITY (INCLUDING NEGLIGENCE), WHETHER OR NOT CONDUKTOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.** **CAP ON LIABILITY. TO THE FULLEST EXTENT PERMITTED BY LAW, CONDUKTOR'S TOTAL CUMULATIVE LIABILITY ARISING OUT OF OR RELATING TO THIS AGREEMENT OR THE SAAS SERVICES, FROM ALL CAUSES OF ACTION AND UNDER ALL THEORIES OF LIABILITY, WILL NOT EXCEED ONE HUNDRED U.S. DOLLARS (US$100). CUSTOMER ACKNOWLEDGES THAT THE SAAS SERVICES ARE PROVIDED FREE OF CHARGE, THAT THE FOREGOING DISCLAIMERS AND LIMITATIONS REFLECT A REASONABLE ALLOCATION OF RISK AND ARE AN ESSENTIAL BASIS OF THE BARGAIN, AND THAT THESE LIMITATIONS WILL APPLY NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED REMEDY.** **EXCEPTIONS. NOTHING IN THIS AGREEMENT EXCLUDES OR LIMITS LIABILITY THAT CANNOT BE EXCLUDED OR LIMITED UNDER APPLICABLE LAW, INCLUDING LIABILITY FOR FRAUD OR FRAUDULENT MISREPRESENTATION, OR FOR DEATH OR PERSONAL INJURY CAUSED BY A PARTY'S GROSS NEGLIGENCE OR WILLFUL MISCONDUCT. WHERE LIABILITY CANNOT LAWFULLY BE EXCLUDED BUT MAY BE LIMITED, CONDUKTOR'S LIABILITY IS LIMITED TO THE GREATEST EXTENT PERMITTED BY LAW. SOME JURISDICTIONS DO NOT ALLOW CERTAIN OF THE FOREGOING EXCLUSIONS OR LIMITATIONS, SO SOME OF THEM MAY NOT APPLY TO CUSTOMER, IN WHICH CASE SUCH EXCLUSIONS AND LIMITATIONS WILL APPLY TO THE MAXIMUM EXTENT PERMITTED BY LAW.** **9. TERM AND TERMINATION** 9.1 **Term.** This Agreement is effective as of the date Customer first accesses or uses the SaaS Services or first accepts this Agreement (the "**Effective Date**") and continues until terminated in accordance with this Section 9. The period during which Customer is authorized to access and use the SaaS Services under this Agreement is referred to as the "**Term**". For the avoidance of doubt, this Agreement does not obligate Conduktor to continue providing, maintaining, updating, or supporting the SaaS Services. 9.2 **Termination.** Conduktor may suspend or terminate this Agreement and Customer's access to the SaaS Services at any time, for any reason or no reason, in its sole discretion, with or without notice, and without liability to Customer. The license and Customer's right to access the SaaS Services will immediately terminate if Customer breaches Section 1 of this Agreement, regardless of whether Conduktor notifies Customer. In addition, Conduktor may terminate this Agreement upon written notice if Customer fails to comply with any material term of this Agreement and does not cure the default within ten (10) days after written notice. Conduktor may also terminate this Agreement immediately if Customer: (a) terminates or suspends its business; (b) becomes subject to any bankruptcy or insolvency proceeding; (c) becomes insolvent or subject to direct control by a trustee, receiver, or similar authority; or (d) is wound up or liquidated, voluntarily or otherwise. Customer may terminate this Agreement at any time by ceasing all use of the SaaS Services and closing its account. 9.3 **Effect of Termination or Expiry.** The provisions of this Agreement that by their nature extend beyond termination or expiry will survive, including Sections 2 through 8 and 10. Upon termination or expiry, all of Customer's rights to access and use the SaaS Services will terminate immediately, and Customer must immediately discontinue all use of the SaaS Services. Conduktor may, but is not obligated to, make Customer Data available for export for a limited period following termination, after which Conduktor may delete or render inaccessible all Customer Data without liability. Conduktor will have no obligation or liability with respect to the deletion of or failure to retain any Customer Data. Nothing in this Section limits any other remedies Conduktor may have, or relieves Customer of obligations that arose before termination or expiry. **10. MISCELLANEOUS** 10.1 **Assignment.** Customer may not assign or transfer this Agreement, or any rights or obligations under it, without Conduktor's prior written consent. Any attempted assignment without that consent will have no effect. Conduktor may assign this Agreement to a successor to its business or relevant assets, whether by merger, sale, reorganization, or otherwise. Subject to this Section, this Agreement binds and benefits both parties and their respective successors and permitted assigns. 10.2 **Entire Agreement; Modification; Waiver.** This Agreement, together with any documents incorporated by reference (including the Data Processing Agreement and Privacy Policy), is the complete agreement between the parties on its subject matter and supersedes all prior agreements and understandings, whether written or oral. No third party gains any rights under this Agreement. Customer acknowledges that it has not entered into this Agreement based on any representations other than those contained in it. Except as described in the preamble (regarding Conduktor's right to modify this Agreement by posting or notice), no amendment or waiver will be effective unless in writing and signed by both parties. Waiving or delaying enforcement of any right on one occasion does not waive that right on any future occasion. 10.3 **Third Party Software and Services.** The SaaS Services may incorporate or interoperate with certain third-party open-source software and third-party services. Such third-party software and services are subject to their own applicable licenses and terms and not this Agreement. To the extent the terms of open-source licenses applicable to such software prohibit any of the restrictions in this Agreement, such restrictions will not apply to such software. Conduktor is not responsible for, and disclaims all liability arising from, any third-party software or services. 10.4 **Governing Law.** This Agreement is governed by the laws of the State of New York (without regard to conflict-of-law rules). Any disputes arising under this Agreement will be resolved exclusively in the state or federal courts located in the State of New York, and Customer consents to the exclusive jurisdiction of those courts. The United Nations Convention on Contracts for the International Sale of Goods does not apply to this Agreement. 10.5 **Severability.** If any provision of this Agreement is held invalid or unenforceable under applicable law by a court of competent jurisdiction, it will be replaced with the valid provision that most closely reflects the intent of the parties, and the remaining provisions of this Agreement will remain in full force and effect. 10.6 **Relationship of the Parties.** Nothing in this Agreement is to be construed as creating an agency, partnership, or joint venture relationship between the parties. Neither party has the right or authority to create obligations, make representations, or bind the other party in any way. 10.7 **Notices.** All notices under this Agreement must be in writing and will be considered delivered: (a) when delivered in person or by overnight courier; (b) three (3) business days after being mailed by first class, registered, or certified mail (postage prepaid) to the address specified in this Agreement or another address either party provides in writing; or (c) when sent by email. It is Customer's responsibility to keep its email address current. Customer will be deemed to have received any email sent to the email address then associated with Customer's account when Conduktor sends the email, whether or not Customer actually receives the email. Notices to Conduktor should be sent via email to [legal@conduktor.io](mailto:legal@conduktor.io). 10.8 **U.S. Government Restricted Rights.** If the SaaS Services are being licensed by the U.S. Government, the SaaS Services are deemed to be "commercial computer software" and "commercial computer documentation" developed exclusively at private expense, and (a) if acquired by or on behalf of a civilian agency, will be subject solely to the terms of this software license as specified in 48 C.F.R. 12.212 of the Federal Acquisition Regulations and its successors; and (b) if acquired by or on behalf of units of the Department of Defense ("DOD"), will be subject to the terms of this commercial computer software license as specified in 48 C.F.R. 227.7202-2, DOD FAR Supplement and its successors. 10.9 **Export Law Assurances.** Customer understands that the SaaS Services are subject to export control laws and regulations. Customer may not access, use, or export or re-export the SaaS Services or any underlying information or technology except in full compliance with all applicable laws and regulations, in particular United States export control laws. The SaaS Services may not be accessed, used, exported, or re-exported: (a) into (or to a national or resident of) any country to which the United States has embargoed goods; or (b) to anyone on the U.S. Treasury Department's list of specially designated nationals or the U.S. Commerce Department's list of prohibited countries or debarred or denied persons or entities. Customer represents and warrants that it is not located in, under the control of, or a national or resident of any such country or on any such list. 10.10 **Construction.** Section headings are for convenience only and do not affect interpretation. Neither party will be treated as the drafter of this Agreement for purposes of resolving any ambiguity. The word "including" means "including, without limitation." 10.11 **Force Majeure.** In no event will Conduktor be liable to Customer, or be considered to have breached this Agreement, for any failure or delay in performing its obligations under this Agreement, if and to the extent such failure or delay is caused by any circumstances beyond Conduktor's reasonable control, including acts of God, flood, fire, earthquake, explosion, war, terrorism, invasion, riot or other civil unrest, strikes, labor stoppages or slowdowns or other industrial disturbances, pandemic or epidemic, failure of third-party hosting, network, or telecommunications services, or passage of law or any action taken by a governmental or public authority, including imposing an embargo. **11. PUBLICITY RIGHTS** Conduktor may identify Customer as a user of the SaaS Services and may use Customer's name, logo, and other trademarks in Conduktor's customer list, press releases, blog posts, advertisements, and website (and any goodwill arising from such use belongs exclusively to Customer); provided that such rights apply only where Customer is an entity or other organization account (and not an individual), and Conduktor will not use the name, likeness, or other personal identifiers of any individual Customer or any Authorized User for promotional purposes without that individual's express consent. Customer can deny Conduktor this right at any time by submitting a written request via email to [support@conduktor.io](mailto:support@conduktor.io), requesting to be excluded from Conduktor promotional material. Requests may take thirty (30) calendar days to process, and such withdrawal will apply prospectively only. Following processing of such request, Conduktor will use commercially reasonable efforts to cease new uses and to remove Customer's name, logo, and other trademarks from Conduktor-controlled websites and other Conduktor-controlled marketing materials within such timeframe; provided that Conduktor will not be required to recall, retract, or delete any materials already published or printed prior to processing the request, or any materials not under Conduktor's control (including third-party posts or publications and paid advertisements already placed). Neither party may otherwise use the name, logo, or other trademarks of the other party for any purpose without the other party's prior written approval. This website is operated by Conduktor Inc. If you have any questions, complaints, or concerns about how your information is handled, or would like to exercise any of your rights for yourself or on behalf of someone else, please email us at #### support@conduktor.io Source: https://www.conduktor.io/legal/eula-saas --- # EULA Updated June 5th, 2026 This End User License Agreement ("**Agreement**") is a legal agreement between you (either an individual or an entity) ("**Customer**") and Conduktor Inc., a Delaware corporation ("**Conduktor**"), and governs your access to and use of the Conduktor software products made available by Conduktor for download under a free license (collectively, "**Conduktor Software**"). Certain Conduktor Software is licensed on a perpetual basis and certain Conduktor Software is licensed for a fixed term, in each case as specified in the applicable product documentation or on the Conduktor website. This Agreement applies to Free Licenses of the Conduktor Software. Conduktor's hosted or cloud-based services ("**SaaS Services**") are not covered by this Agreement and are governed by separate terms available at our [SaaS Services EULA](https://www.conduktor.io/legal/eula-saas). Use of and access to the Conduktor Software is conditioned upon your compliance with this Agreement and all applicable local, state, national and international laws, rules and regulations. Your use of the Conduktor Software indicates your acceptance of this Agreement. We may modify this Agreement (including any policies we reference) at any time by posting a revised version on the Conduktor website or by otherwise notifying you in accordance with Section 10.7. The modified Agreement will become effective upon posting or, if we notify you by email, as stated in the email message. By continuing to use the Conduktor Software after the effective date of any modifications to this Agreement, you agree to be bound by the modified Agreement. We last modified this Agreement on the date listed at the beginning of this agreement. IF YOU ARE AN INDIVIDUAL AGREEING TO THE TERMS OF THIS AGREEMENT ON BEHALF OF AN ENTITY, SUCH AS YOUR EMPLOYER, YOU REPRESENT THAT YOU HAVE THE LEGAL AUTHORITY TO BIND THAT ENTITY AND "CUSTOMER" SHALL REFER HEREIN TO SUCH ENTITY. IF YOU DO NOT HAVE SUCH AUTHORITY, OR IF YOU DO NOT AGREE WITH THE TERMS OF THIS AGREEMENT, YOU MUST NOT ACCEPT THIS AGREEMENT AND MAY NOT USE THE SOFTWARE. **1\. LICENSE** 1.1 **License Grant.** Subject to the terms of this Agreement, Conduktor grants to Customer a limited, non-exclusive, non-sublicensable, non-transferable license during the applicable Term (as defined below) to download, install, and use the Conduktor Software on Customer's own infrastructure, solely for Customer's internal business operations. Subject to the restrictions set forth in Section 1.5 below, Customer may permit its third-party service providers to use the Conduktor Software to provide outsourced services to Customer, and Customer will be solely responsible for such service provider's compliance with this Agreement. "**Named User**" means any unique licensed user of the Conduktor Software who is authorized by the Customer to access or use the Conduktor Software. 1.2 **Additional Restrictions and Limitations.** Customer shall not, and shall not permit or encourage any of Customer's Named Users, or any employee, contractor, consultant, or third party to: (a) sublicense, sell, distribute or pledge the Conduktor Software or any of the rights under this Agreement; (b) use the Conduktor Software other than as expressly permitted by this Agreement, or for third-party training, SaaS, hosting, managed-service, time-sharing, or service bureau use (but using third-party infrastructure providers solely to host Customer's own internal deployment is permitted); (c) provide access to the Conduktor Software to anyone other than a Named User; any access to or use of the Conduktor Software by a person who is not a Named User is a material breach of this Agreement; (d) use any third-party or open-source software in conjunction with the Conduktor Software in a manner that would subject the Conduktor Software to any open-source or third-party license requiring its public disclosure, distribution, or licensing for derivative works; (e) modify, translate, decrypt, create derivative works from, disassemble, decompile, or reverse-engineer the Conduktor Software, or otherwise attempt to access its source code or underlying ideas, algorithms, structure, or organization (except to the extent applicable law prohibits this restriction); (f) bypass or breach any security device or protection used by the Conduktor Software, or access or use it other than as a Named User with valid credentials; share, transfer, or disclose any Named User credentials; permit any other person to use a Named User account or credentials; or use shared logins or shared accounts; (g) input, upload, transmit, or otherwise provide to or through the Conduktor Software any unlawful or injurious information or materials, or any harmful code; (h) use the Conduktor Software in any manner or for any purpose that infringes, misappropriates, or otherwise violates any intellectual property rights or other right of any person, or that violates any applicable law. 1.3 **License Restrictions; Reservation of Rights.** Conduktor reserves the right to restrict, disable, or revoke Customer's license to use the Conduktor Software in its sole discretion, and without liability to Customer, and may provide notice where practicable; provided that Conduktor may take such action immediately and without prior notice in the event: (a) Conduktor reasonably believes that Customer's use of the Conduktor Software may result in liability to Conduktor or adversely affect the integrity, functionality, or security of the Conduktor Software; or (b) Customer is in breach of this Agreement. Conduktor is not responsible for any damages, liabilities, or losses of Customer resulting from any such restriction, disablement, or revocation. Conduktor reserves all rights not expressly granted in this Agreement. No rights are granted by implication. 1.4 **Usage Limits.** The Conduktor Software is subject to usage limits as specified in the applicable product documentation or on the Conduktor website (including limits on the number of clusters, users, or other metrics), which usage limits are incorporated by reference into, and form part of, this Agreement as binding contractual restrictions for the applicable Conduktor Software. Customer shall not exceed, or attempt to circumvent, any such usage limits, including by creating multiple accounts or organizations or deploying multiple instances of the Conduktor Software, and any such exceeding or circumvention is a material breach of this Agreement. 1.5 **Number of Users.** Where a product limits the number of Named Users, Customer shall not exceed the Named User limit specified in the product, applicable product documentation or on the Conduktor website. Use of the Conduktor Software by more Named Users than permitted under this Agreement is available only under a separate paid license agreement with Conduktor. The right of a Named User to use the Conduktor Software may not be assigned to a new user of the Customer, except if an existing Named User's relationship with the Customer has been severed. 1.6 **Delivery of Materials.** The Conduktor Software, and any versions, updates, or maintenance releases of any component thereof, will be made available for download by Customer from the Conduktor website or other distribution channel designated by Conduktor. Customer is solely responsible for downloading, installing, and configuring the Conduktor Software on Customer's infrastructure. Customer will install (a) updates designated by Conduktor as security, critical or hotfix updates within 30 days after being made available, and (b) all other updates within 90 days after being made available (or such other period specified by Conduktor in writing for a particular update). If Customer fails to timely install an update, (i) Conduktor may decline to provide support for the affected issue(s) to the extent they are caused by or would have been avoided by installing the update, and may require Customer to first install the applicable update as a condition to providing support; (ii) Conduktor will have no responsibility for any vulnerability, non-conformity, error, outage or other issue to the extent caused by Customer's failure to install the update; and (iii) to the extent permitted by law, any warranty, compatibility or indemnity obligations under this Agreement will not apply to claims or issues that would have been avoided by use of the then-current update made available to Customer. 1.7 **Support and Maintenance.** Conduktor is not obligated to provide maintenance or support services for the Conduktor Software licensed under this Agreement. Customer may access community support resources (such as documentation, forums, or community channels) to the extent Conduktor makes them available, but Conduktor provides no guarantee of response or resolution. In the event that Conduktor, in its sole discretion, elects to make available any updates or maintenance releases of the Conduktor Software for download, such updates or maintenance releases will be considered Conduktor Software under this Agreement. Customer is solely responsible for determining whether to install any such updates. Where Conduktor identifies a material security vulnerability in the Conduktor Software, Conduktor will use reasonable efforts to publish a security advisory and, where available, a remedial update, within a reasonable timeframe. 1.8 **Preview Releases.** Conduktor may invite Customer to try pre-release versions of the Conduktor Software or features under development ("**Beta Features**") at no charge. Customer may accept or decline any such invitation at Customer's sole discretion. Beta Features will be clearly designated as beta, pilot, limited release, developer preview, non-production, evaluation, or by a description of similar import. Beta Features are for evaluation purposes only and not for production use, and are not considered Conduktor Software under this Agreement, are not supported, and may be subject to additional terms. Conduktor may discontinue Beta Features at any time in Conduktor's sole discretion. Conduktor is under no obligation to release a generally available version of any Beta Feature. Conduktor will have no liability for any harm or damage arising out of or in connection with any Beta Feature. **2\. USER AND PERFORMANCE DATA** 2.1 **Telemetry Data.** Certain features of the Conduktor Software are configured to collect and transmit technical telemetry and usage data to Conduktor's servers ("**Telemetry Data**"). Telemetry Data includes information such as feature usage patterns, error logs, performance metrics, and deployment configuration data. Telemetry Data does not intentionally include Customer's business data, message content, or the personal data of Customer's end users. Conduktor will process Telemetry Data solely for the following purposes: (a) maintaining, improving, and securing the Conduktor Software; (b) monitoring product performance and diagnosing technical issues; and (c) generating aggregated and anonymized product analytics. Conduktor will not sell Telemetry Data to third parties. 2.2 **License Registration Data.** Where the Conduktor Software is licensed for a fixed term, Customer must provide a valid email address as part of the license registration process. Conduktor may use such email address to communicate with Customer regarding license expiry, renewal, and material product updates directly related to Customer's deployment. Customer may opt out of non-essential communications at any time by contacting [support@conduktor.io](mailto:support@conduktor.io). It is Customer's responsibility to keep such email address current. 2.3 **Personal Data.** Conduktor does not, under this Agreement, access, receive, or otherwise process any personal data contained in Customer's use of its on-prem deployment of the Conduktor Software, and Customer is solely responsible for any personal data it inputs into, uploads to, or otherwise processes using the Conduktor Software in its on-prem environment. Conduktor does not act as a data processor for Customer in connection with such on-prem processing. Notwithstanding the foregoing, Conduktor may process License Registration Data as described above. For purposes of License Registration Data, Conduktor acts as an independent data controller and will process such data in accordance with applicable data protection laws. **3\. OWNERSHIP** Customer acknowledges that Conduktor or its licensors retain all proprietary rights, title and interest, including all intellectual property rights, in and to the Conduktor Software and any changes, corrections, bug fixes, enhancements, updates and other modifications to it made by or on behalf of Conduktor (collectively, "**Modifications**"), and as between the parties all such rights will vest in Conduktor. For clarity, "Modifications" do not include, and Conduktor does not claim any ownership interest in, Customer's data or Customer's independent configurations, scripts, integrations, connectors or other materials that Customer develops that do not incorporate Conduktor Confidential Information or Conduktor code. Customer acknowledges that the licenses granted in Section 1 are limited in scope as set out in that Section. **4\. FINANCIAL TERMS** 4.1 The Conduktor Software licensed under this Agreement is provided at no charge. Conduktor reserves the right to introduce fees for features or usage tiers not covered by this Agreement; any such fees will be subject to a separate agreement. 4.2 To the extent any fees become payable in connection with use of the Conduktor Software, such fees are exclusive of all applicable taxes and government charges, which Customer will be responsible for paying. **5\. FEEDBACK** Customer is under no duty to provide any suggestions, enhancement requests, or other feedback regarding the Conduktor Software ("**Feedback**"). If Customer chooses to offer Feedback to Conduktor, Customer grants Conduktor a perpetual, irrevocable, non-exclusive, worldwide, fully-paid, sub-licensable, assignable license to incorporate into the Conduktor Software or otherwise use any Feedback Conduktor receives from Customer. Conduktor acknowledges that any Feedback is provided on an "as-is" basis with no warranties of any kind. Feedback will constitute Confidential Information of Conduktor for purposes of Section 6. **6\. CONFIDENTIALITY** 6.1 **Nondisclosure and Limited Use.** "**Confidential Information**" means non-public information disclosed by one party to the other under this Agreement that is: (a) marked as confidential or proprietary in writing; (b) if disclosed orally, confirmed as confidential in writing (including email) within thirty (30) days; or (c) information that a reasonable recipient would understand to be confidential. The Conduktor Software and Modifications are Conduktor's Confidential Information. Each party agrees to: (a) keep the other party's Confidential Information confidential; (b) use it only to perform its obligations under this Agreement; and (c) not disclose it to any third party, except to employees and subcontractors who need access and are bound by confidentiality obligations at least as protective as this Section. Each party must promptly notify the other of any unauthorized disclosure or use and take reasonable steps to help remedy it. These confidentiality obligations do not apply to information that: (i) is or becomes publicly available without breach of this Agreement; (ii) the receiving party independently developed without reference to the disclosing party's Confidential Information; (iii) the receiving party lawfully obtained from another source without confidentiality restrictions; or (iv) must be disclosed by law or court order, provided the receiving party gives prompt notice and reasonable assistance so the disclosing party can seek a protective order. Either party may also disclose Confidential Information to its agents, attorneys, and other representatives (subject to confidentiality obligations at least as protective as this Section) or to any court as reasonably required to resolve a dispute under this Agreement. 6.2 **Remedies.** A breach or threatened breach of this Section may cause harm that cannot be adequately remedied by monetary damages alone. In such cases, the disclosing party may seek injunctive or other equitable relief without having to prove actual damages or post a bond, in addition to any other remedies available at law. **7\. INDEMNITY** 7.1 If a third party brings a claim against Customer alleging that the Conduktor Software (as updated by Conduktor and used in accordance with this Agreement) infringes a patent, copyright, or trade secret, Conduktor will, at its own expense, defend or settle the claim. Conduktor will also cover damages, costs, and reasonable attorneys' fees awarded against Customer in a final judgment arising from such a claim. These obligations are subject to: (i) Customer promptly notifying Conduktor in writing of the claim; (ii) Conduktor having exclusive control over the defense and any settlement; and (iii) Customer providing reasonable assistance (at Conduktor's expense). Customer may not settle any such claim without Conduktor's prior written approval. 7.2 Conduktor's indemnity obligations under Section 7.1 do not apply if the claim arises because: (a) Customer modified the Conduktor Software; (b) Customer did not use the most current release of the Conduktor Software made available by Conduktor, and the current release would have avoided the claim; (c) Customer continued the allegedly infringing activity after Conduktor provided a modification that would have avoided it; (d) the claim is based on Customer's output; or (e) the claim arises from the combination of the Conduktor Software with any software, service, technology, content, data, or materials not provided by Conduktor, if the claim would not have arisen but for such combination. 7.3 If Conduktor defends or settles an infringement claim under Section 7.1, Conduktor's sole obligation (in addition to the indemnity) and Customer's exclusive remedy will be for Conduktor, at its own expense, to either: (a) repair, replace, or modify the affected Conduktor Software so it is no longer infringing; or (b) obtain for Customer the right to continue using the affected Conduktor Software. If neither option is commercially feasible (in Conduktor's reasonable opinion), Conduktor may terminate this Agreement and the license for the affected Conduktor Software. 7.4 **Customer Indemnity.** Customer will defend, indemnify and hold harmless Conduktor, its Affiliates, and each of their respective licensors, officers, employees and agents from and against any and all third-party claims, demands, suits or proceedings, and all related damages, losses, liabilities, penalties, costs and expenses (including reasonable legal fees) arising out of or relating to: any allegation that Customer Data, or Customer's processing, transmission, storage or other handling of Customer Data, violates applicable Laws or infringes, misappropriates or otherwise violates any third-party rights; Customer's breach of this Agreement, including any breach of the Restrictions or Customer Obligations, any breach of Customer's export controls and sanctions obligations, or any use of the Conduktor Software in violation of applicable Laws; Customer's systems, networks, infrastructure, configurations, or access controls (including any security incident, unauthorised access, or failure to maintain appropriate security measures in Customer's environment); or Customer's output, or Customer's use of the Conduktor Software in combination with any third-party products, services, software, systems or data, to the extent the claim would not have arisen but for such output or combination. This Customer indemnity is in addition to, and does not limit or narrow, Conduktor's indemnification obligations. Indemnity Procedures. Conduktor will promptly notify Customer in writing of any claim for which Conduktor seeks indemnification under this Customer indemnity; provided that any failure to provide prompt notice will not relieve Customer of its obligations except to the extent Customer is materially prejudiced by such failure. Customer will have sole control of the defence and settlement of the claim and will pay all defence costs and any settlement amounts or damages awarded; provided that Customer may not settle any claim in a manner that admits fault or liability on behalf of, or imposes any obligation on, Conduktor or any of its Affiliates without Conduktor's prior written consent (not to be unreasonably withheld or delayed). Conduktor will provide reasonable cooperation in the defence of the claim at Customer's expense. **8\. DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITY** 8.1 **Disclaimer of Any Warranties.** CONDUKTOR MAKES NO WARRANTIES, EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH RESPECT TO THE CONDUKTOR SOFTWARE OR ANY OTHER MATERIALS PROVIDED HEREUNDER. CONDUKTOR SPECIFICALLY DISCLAIMS ALL OTHER WARRANTIES, EXPRESS AND IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT AND THOSE ARISING FROM A COURSE OF DEALING OR USAGE OR TRADE, AND ALL SUCH WARRANTIES ARE HEREBY EXCLUDED TO THE FULLEST EXTENT PERMITTED BY LAW. THE CONDUKTOR SOFTWARE IS PROVIDED ON AN "AS IS" BASIS. 8.2 **Limitation of Liability.** IN NO EVENT SHALL CONDUKTOR'S LIABILITY ARISING UNDER THIS AGREEMENT EXCEED $500. CONDUKTOR WILL NOT BE LIABLE TO CUSTOMER FOR ANY CONSEQUENTIAL, INCIDENTAL, SPECIAL, INDIRECT, PUNITIVE OR EXEMPLARY DAMAGES, INCLUDING WITHOUT LIMITATION LOST PROFITS, BUSINESS, CONTRACTS, REVENUE, GOODWILL, PRODUCTION, ANTICIPATED SAVINGS, LOSS OF DATA, OR COSTS OF PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES, OR FOR ANY CLAIM OR DEMAND BY ANY OTHER PARTY, HOWEVER CAUSED AND (TO THE FULLEST EXTENT PERMITTED BY LAW) UNDER ANY THEORY OF LIABILITY (INCLUDING NEGLIGENCE) EVEN IF CONDUKTOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. CUSTOMER ACKNOWLEDGES THAT THESE LIMITATIONS SHALL APPLY NOTWITHSTANDING ANY FAILURE OF ESSENTIAL PURPOSE OF ANY LIMITED REMEDY. **9\. TERM AND TERMINATION** 9.1 **Term.** This Agreement will be effective as of the date Customer first downloads or installs any Conduktor Software (the "**Effective Date**"). (a) Perpetual License. Where the applicable product documentation or the Conduktor website specifies that Conduktor Software is licensed on a perpetual basis, the license for such Conduktor Software will continue in perpetuity unless terminated in accordance with Section 9.2 below. (b) Fixed-Term License. Where the applicable product documentation or the Conduktor website specifies that Conduktor Software is licensed for a fixed term, the license for such Conduktor Software will be valid for one (1) year from the date of license registration (each such period, a "**License Term**"). Upon expiry of a License Term, Customer must apply for a new license in accordance with the process specified in the applicable product documentation or on the Conduktor website in order to continue using such Conduktor Software. There is no automatic renewal. If Customer does not obtain a new license before the expiry of the then-current License Term, Customer's right to use the applicable Conduktor Software will cease upon such expiry. The period during which Customer is authorized to use any Conduktor Software under this Agreement (whether perpetual or for a fixed term) is referred to as the "**Term**". For the avoidance of doubt, neither a perpetual license nor a fixed-term license obligates Conduktor to continue distributing, maintaining, updating, or supporting the Conduktor Software. 9.2 **Termination.** In respect of Conduktor Software licensed for a fixed term, the license will automatically terminate upon expiry of the applicable License Term if Customer has not obtained a new license in accordance with Section 9.1(b). The license will immediately terminate if Customer breaches Section 1 of this Agreement, regardless of whether Conduktor notifies Customer of such termination. In addition, Conduktor may terminate this Agreement and the license upon written notice if Customer fails to comply with any material term of this Agreement and does not cure the default within ten (10) days after written notice of such default to the Customer. Conduktor may also terminate this Agreement immediately if the Customer: (a) terminates or suspends its business; (b) becomes subject to any bankruptcy or insolvency proceeding under Federal or state statute; (c) becomes insolvent or subject to direct control by a trustee, receiver or similar authority; or (d) has wound up or liquidated, voluntarily or otherwise. 9.3 **Effect of Termination or Expiry.** The provisions of this Agreement that by their nature extend beyond the termination or expiry of a license will survive such termination or expiry. Upon termination of this Agreement or expiry of an applicable License Term, all of Customer's rights in the affected Conduktor Software will terminate immediately, and Customer must immediately discontinue all use of the affected Conduktor Software. No later than five (5) days after such termination or expiry, Customer must destroy, delete, or render inaccessible, at Customer's sole expense, all Confidential Information of Conduktor relating to the affected Conduktor Software and materials containing any such Confidential Information, and uninstall the affected Conduktor Software, including all copies thereof. For the avoidance of doubt, expiry of a License Term for one Conduktor Software product does not affect Customer's rights in any other Conduktor Software licensed under this Agreement. Nothing in this Section limits any other remedies Conduktor may have for Customer's default under this Agreement, or relieves Customer of obligations that arose before termination or expiry. **10\. MISCELLANEOUS** 10.1 **Assignment.** Customer may not assign or transfer this Agreement, or any rights or obligations under it, without Conduktor's prior written consent. Any attempted assignment without that consent will have no effect. Conduktor may assign this Agreement to a successor to its business or relevant assets, whether by merger, sale, reorganization, or otherwise. Subject to this Section, this Agreement binds and benefits both parties and their respective successors and permitted assigns. 10.2 **Entire Agreement; Modification; Waiver.** This Agreement is the complete agreement between the parties on its subject matter and supersedes all prior agreements and understandings, whether written or oral. No third party gains any rights under this Agreement. Customer acknowledges that it has not entered into this Agreement based on any representations other than those contained in it. Except as described in the preamble (regarding Conduktor's right to modify this Agreement by posting or notice), no amendment or waiver will be effective unless in writing and signed by both parties. Waiving or delaying enforcement of any right on one occasion does not waive that right on any future occasion. 10.3 **Third Party Software.** Conduktor also makes available certain third party open source software ("**Third Party Software**"). The Third Party Software is subject to the applicable open source license(s) and not this Agreement, and is provided by Conduktor at no charge. To the extent the terms of open source licenses applicable to Third Party Software prohibit any of the restrictions in this Agreement, such restrictions will not apply to such Third Party Software. To the extent the terms of open source licenses applicable to Third Party Software require Conduktor to make an offer to provide source code or related information in connection with the Third Party Software, such offer is made. 10.4 **Governing Law.** This Agreement is governed by the laws of the State of New York (without regard to conflict-of-law rules). Any disputes arising under this Agreement will be resolved exclusively in the federal or state courts located within the City and County of New York, New York, and Customer consents to the jurisdiction of those courts. The United Nations Convention on Contracts for the International Sale of Goods does not apply to this Agreement. 10.5 **Severability.** If any provision of this Agreement is held invalid or unenforceable under applicable law by a court of competent jurisdiction, it will be replaced with the valid provision that most closely reflects the intent of the parties and the remaining provisions of the Agreement will remain in full force and effect. 10.6 **Relationship of the Parties.** Nothing in this Agreement is to be construed as creating an agency, partnership, or joint venture relationship between the parties. Neither party has the right or authority to create obligations, make representations, or bind the other party in any way. 10.7 **Notices.** All notices under this Agreement must be in writing and will be considered delivered: (a) when delivered in person or by overnight courier; (b) three (3) business days after being mailed by first class, registered, or certified mail (postage prepaid) to the address specified in this Agreement or another address either party provides in writing; or (c) when sent by email. It is Customer's responsibility to keep its email address(es) current. Customer will be deemed to have received any email sent to the email address then associated with Customer's account when Conduktor sends the email, whether or not Customer actually receives the email. Notices to Conduktor should be sent via email to [legal@conduktor.io](mailto:legal@conduktor.io). 10.8 **U.S. Government Restricted Rights.** If Conduktor Software is being licensed by the U.S. Government, the Conduktor Software is deemed to be "commercial computer software" and "commercial computer documentation" developed exclusively at private expense, and (a) if acquired by or on behalf of a civilian agency, will be subject solely to the terms of this computer software license as specified in 48 C.F.R. 12.212 of the Federal Acquisition Regulations and its successors; and (b) if acquired by or on behalf of units of the Department of Defense ("DOD") will be subject to the terms of this commercial computer software license as specified in 48 C.F.R. 227.7202-2, DOD FAR Supplement and its successors. 10.9 **Export Law Assurances.** Customer understands that the Conduktor Software is subject to export control laws and regulations. Customer may not download or otherwise export or re-export the Conduktor Software or any underlying information or technology except in full compliance with all applicable laws and regulations, in particular, but without limitation, United States export control laws. None of the Conduktor Software or any underlying information or technology may be downloaded or otherwise exported or re-exported: (a) into (or to a national or resident of) any country to which the United States has embargoed goods; or (b) to anyone on the U.S. Treasury Department's list of specially designated nationals or the U.S. Commerce Department's list of prohibited countries or debarred or denied persons or entities. Customer agrees to the foregoing and represents and warrants that Customer is not located in, under control of, or a national or resident of any such country or on any such list. 10.10 **Construction.** Section headings are for convenience only and do not affect interpretation. Neither party will be treated as the drafter of this Agreement for purposes of resolving any ambiguity. The word "including" means "including, without limitation." 10.11 **Force Majeure.** In no event will Conduktor be liable to Customer, or be considered to have breached this Agreement, for any failure or delay in performing its obligations under this Agreement, if and to the extent such failure or delay is caused by any circumstances beyond Conduktor's reasonable control, including but not limited to acts of God, flood, fire, earthquake, explosion, war, terrorism, invasion, riot or other civil unrest, strikes, labor stoppages or slowdowns or other industrial disturbances, or passage of law or any action taken by a governmental or public authority, including imposing an embargo. **11\. PUBLICITY RIGHTS** Conduktor may identify Customer as a user of the Conduktor Software and may use Customer's name, logo, and other trademarks in Conduktor's customer list, press releases, blog posts, advertisements, and website (and any goodwill arising from such use belongs exclusively to Customer). Customer can deny Conduktor this right at any time by submitting a written request via email to [support@conduktor.io](mailto:support@conduktor.io), requesting to be excluded from Conduktor promotional material. Requests may take thirty (30) calendar days to process, and such withdrawal will apply prospectively only. Following processing of such request, Conduktor will use commercially reasonable efforts to cease new uses and to remove Customer's name, logo and other trademarks from Conduktor-controlled websites and other Conduktor-controlled marketing materials within such timeframe; provided that Conduktor will not be required to recall, retract, or delete any materials already published or printed prior to processing the request, or any materials not under Conduktor's control (including third-party posts or publications and paid advertisements already placed). Neither party may otherwise use the name, logo, or other trademarks of the other party for any purpose without the other party's prior written approval. This website is operated by Conduktor Inc. If you have any questions, complaints, or concerns about how your information is handled, or would like to exercise any of your rights for yourself or on behalf of someone else, please email us at #### support@conduktor.io Source: https://www.conduktor.io/legal/eula --- # On-Premise Customer Agreement Updated June 9th, 2026 This On-Premise Customer Agreement (“**Agreement**”) is entered into between Conduktor Inc., a Delaware corporation having its principal offices at 224 W 35th Street, Ste 500 #2947, New York, NY 10001 ("**Conduktor**"), and the entity or individual who purchases or otherwise obtains access to the Conduktor Software ("**Customer**"). Customer agrees to be bound by this Agreement by any of the following: (a) completing a purchase through Conduktor's online platform; (b) clicking a button or checking a box indicating acceptance; (c) executing or referencing an Order that incorporates this Agreement; (d) issuing a purchase order that references this Agreement or an Order governed by it; (e) subscribing to the Conduktor Software through a third-party marketplace under a listing that references this Agreement; or (f) accessing or using the Conduktor Software. The earliest of these events is the "Effective Date". If you are executing this Agreement on behalf of an entity, you represent and warrant that you have authority to bind that entity to the terms of this Agreement. This Agreement governs all Orders placed by Customer under it. Each Order incorporates and is subject to the terms of this Agreement. Certain capitalised terms are defined in Section 1. Others are defined contextually. **1\. DEFINITIONS** | **Affiliate** | An entity that directly or indirectly controls, is controlled by, or is under common ownership or control with a party, where "ownership" means the beneficial ownership of 50% or more of an entity’s voting equity securities or other equivalent voting interests and "control" means the power to direct the management or affairs of an entity. | |----|:---| | **Authorised User** | Any individual employee, contractor or agent of Customer or its Affiliate who is permitted by Customer to use the Conduktor Software under this Agreement. | | **Business Day** | A day between Monday and Friday, excluding US public holidays. | | **Conduktor Software** | Conduktor’s proprietary software products, including Conduktor Console and Conduktor Gateway, together with any APIs, scripts, apps and other code provided to Customer by Conduktor, as identified in the relevant Order and as modified from time to time. | | **Confidential Information** | All non-public information disclosed by or on behalf of one party (the “Disclosing Party”) to the other party (the “Receiving Party”) under or in connection with this Agreement, whether orally, in writing or in any other form, that is designated as confidential or that a reasonable person would understand to be confidential given the nature of the information and the circumstances of disclosure. Confidential Information includes, without limitation, business plans, product roadmaps, technical data, source code, pricing, customer lists and the terms and conditions of this Agreement. | | **Customer Data** | Any data, content or materials that Customer or its Authorised Users process, transmit, store or otherwise handle through the Conduktor Software within Customer’s own environment. Customer Data does not include Usage Data. | | **Documentation** | Conduktor’s usage guidelines and standard technical documentation for the Conduktor Software, currently available at https://docs.conduktor.io. | | **Excluded Claims** | As defined in Section 10.3. | | **Fees** | The fees applicable to Customer’s use of the Conduktor Software as set out in the relevant Order. | | **Laws** | All applicable local, state, federal and international laws, regulations and conventions, including those relating to data privacy, data transfer, international communications and export of technical or personal data. | | **Order** | Any online purchase confirmation or other written order for access to the Conduktor Software, whether executed by the parties or completed through Conduktor’s online platform, that expressly incorporates or references this Agreement. Multiple Orders may be placed under this Agreement. | | **Performance Warranty** | The warranty set out in Section 8.1. | | **Privacy Policy** | Conduktor’s Privacy Policy, the current version of which is at https://www.conduktor.io/legal/privacy-policy/. | | **Regular Business Hours** | Hours between 09:00 and 17:00 US Eastern time on Business Days. | | **Subscription Term** | The term for Customer’s use of the Conduktor Software as identified in an Order, together with any renewal terms. | | **Usage Data** | Data such as volumes, frequencies, bounce rates, error reporting and software performance data generated by Customer’s use of the Conduktor Software, including technical telemetry and usage data such as feature usage patterns, logs, and performance and reliability metrics. Usage Data does not include Customer Data and does not intentionally include message content or Customer’s business data processed through the Conduktor Software. | **2\. LICENCE** 2.1 **Permitted Use.** During the Subscription Term, Conduktor grants Customer and its Authorised Users a limited, non-exclusive, non-sublicensable, non-transferable licence to use the Conduktor Software solely for Customer's internal business purposes, in accordance with the Documentation, the Privacy Policy and this Agreement, including any usage limits stated in an Order. 2.2 **Usage Metrics**. The commercial metric governing Customer's subscription is identified in the applicable Order. Where the Order identifies the licensed product as Conduktor Console, the subscription is licensed by the number of Authorised Users. Where the Order identifies the licensed product as Conduktor Gateway, the subscription is licensed by the number of Kafka clusters connected to or proxied through the Gateway. The maximum permitted number of Authorised Users or clusters (as applicable) is set out in the Order. Customer may add Authorised Users or clusters beyond the limits in the Order by placing a new Order or upgrading through the Conduktor online platform, subject to payment of any additional Fees. Creating multiple organisations to circumvent usage limits is not permitted. 2.3 **Credentials and Responsibility.** Customer is responsible for keeping Authorised User login credentials confidential, for its Authorised Users' compliance with this Agreement and for any actions taken through their accounts. Subject to this Agreement, Customer may permit its third-party service providers to use the Conduktor Software solely to provide outsourced services back to Customer; Customer remains solely responsible for such service providers' compliance with this Agreement. 2.4 **Administrators.** Customer may designate certain Authorised Users as administrators with control over Customer's account, as described in the Documentation. Customer is fully responsible for its choice of administrators and any actions they take. 2.5 **Customer Affiliates.** Customer's Affiliates may use the Conduktor Software in accordance with this Agreement and any Order. Customer and each of its Affiliates are jointly and severally liable for the acts and omissions of such Affiliates and their Authorised Users in connection with this Agreement. Only Customer may bring any claim against Conduktor on behalf of its Affiliates. 2.6 **Restrictions.** Customer shall not, and shall not permit any Authorised User or third party to (a) sublicense, sell, distribute or pledge the Conduktor Software or any rights under this Agreement; (b) use the Conduktor Software for third-party training, software-as-a-service, time-sharing or service bureau purposes; (c) provide access to the Conduktor Software to anyone other than an Authorised User; (d) use any third-party software (including open-source software) in conjunction with the Conduktor Software in a way that subjects the Conduktor Software to any third-party licence, or that requires public disclosure or distribution of the Conduktor Software; (e) modify, translate, create derivative works from, disassemble, decompile or reverse engineer the Conduktor Software, or otherwise attempt to obtain its source code, except to the extent required by applicable law, provided Customer first gives Conduktor prior written notice; (f) input or transmit through the Conduktor Software any information or materials that are unlawful, injurious or that contain harmful code; or (g) use the Conduktor Software in any manner that infringes intellectual property rights or violates applicable Laws. 2.7 **Reseller.** Customer may only acquire licences for resale to end users under a separate written reseller agreement with Conduktor. 2.8 **Delivery.** The Conduktor Software and any updates or maintenance releases will be delivered by electronic means only. **3\. SUPPORT** 3.1 **Support.** During the Subscription Term, Conduktor will provide support services in accordance with Schedule 1 (Support Terms) to this Agreement. 3.2 **Channels.** All support requests must be submitted through the in-software support functionality where available, or via email to support@conduktor.io. Support is provided in English during Regular Business Hours. 3.3 **Excluded Support.** Conduktor is not obligated to provide support where: (a) the Conduktor Software has been modified by anyone other than Conduktor; (b) Customer has changed its operating environment in a way that adversely affects the Conduktor Software without following Conduktor's prerequisites; (c) Customer is using the Conduktor Software in a manner for which it was not designed or as not authorised under this Agreement; (d) the issue arises from third-party software, hardware or networks not supplied by Conduktor; or (e) the issue results from internet connectivity problems or a force majeure event. 3.4 **Updates.** Conduktor will make updates to the Conduktor Software available to Customer via the Conduktor customer portal as and when generally released. Customer is responsible for installing updates. Conduktor will have no responsibility for any vulnerability, non-conformity, error, outage or other issue to the extent caused by Customer’s failure to install the update; and, to the extent permitted by law, any warranty, compatibility or indemnity obligations under this Agreement will not apply to claims or issues that would have been avoided by use of the then-current update made available to Customer. Conduktor warrants upward compatibility of updates and that updates will not result in a material reduction in the functionality or performance of the Conduktor Software during the Subscription Term. 3.5 **Beta and Trial Features.** Conduktor may invite Customer to try beta services or features under development ("Beta Features") at no charge. Beta Features are for evaluation only, are not covered by this Agreement's warranties, service levels or indemnities, and may be discontinued at any time in Conduktor's sole discretion. Conduktor's total liability for Beta Features will not exceed USD 500. Beta Features are clearly designated as beta, pilot, limited release, developer preview, non-production, evaluation or by a description of similar import. **4\. COMMERCIAL TERMS** 4.1 **Subscription Term and Renewal.** Unless otherwise specified in the applicable Order, each Subscription Term will automatically renew for successive periods equal to the expiring Subscription Term unless either party gives the other written notice of non-renewal at least 60 days before the end of the then-current Subscription Term. Upon any renewal, Conduktor may increase the Fees to Conduktor’s then-current list price for the applicable Conduktor Software (and any applicable support or other charges) as of the renewal start date, unless the applicable Order expressly sets different renewal pricing. 4.2 **Fees and Payment.** Fees are as set out in each Order and, except as otherwise provided in this Agreement, are non-cancellable and non-refundable and no right of set-off exists. Fees are invoiced on the schedule set out in the Order and are due within 30 days of the invoice date. Conduktor may charge Customer 1.5% per month on all Fees not paid on time. 4.3 **Payment Disputes.** Customer may dispute an invoice in good faith by written notice to Conduktor within 30 days of the invoice date, identifying the disputed amount and the reason. Customer shall pay all undisputed amounts by the due date. Unless prohibited by applicable law, Customer shall also pay the disputed amount when due and the parties will work in good faith to resolve the dispute within 30 days of the dispute notice. If Conduktor determines that Customer has overpaid, Conduktor will, at Customer’s election, promptly refund the overpaid amount or issue a credit against the next invoice (and, where required by applicable law, include interest). Conduktor will not charge late fees in respect of amounts genuinely in dispute, provided Customer cooperates diligently to resolve the dispute. 4.4 **Taxes.** All Fees are exclusive of all applicable sales, use, value-added, goods and services, withholding, excise or similar taxes or government charges ("Taxes"). Customer shall pay Conduktor any applicable Taxes listed on the relevant invoice, which are in addition to the Fees. 4.5 **Order Precedence.** In the event of conflict between this Agreement and an Order, the Order prevails, except that no Order may modify or override Sections 7 (Ownership and Feedback), 10 (Limitation of Liability), 6 (Confidentiality) or 12.7 (Governing Law and Jurisdiction) without a signed written amendment to this Agreement executed by authorised representatives of both parties. **5\. CUSTOMER DATA AND USAGE DATA** 5.1 **On-Premise Deployment.** The Conduktor Software is deployed and operated within Customer's own environment. Conduktor does not access, store, transmit or process Customer Data in the ordinary course of providing the Conduktor Software. Customer Data remains at all times within Customer's environment and under Customer's control. For clarity, in connection with Customer’s on-premise deployment, Conduktor does not access, receive, or otherwise process any personal data contained in Customer Data in the ordinary course of providing the Conduktor Software, and Conduktor does not act as a processor of such personal data for Customer. Conduktor has no licence or right in Customer Data, except that where Customer voluntarily transmits Customer Data to Conduktor in connection with a support request under Section 3, Conduktor may use such Customer Data solely as necessary to respond to that request, Customer will not provide (and will ensure its Authorised Users do not provide) personal data in any support request or related materials, except to the extent strictly necessary, and in that case Customer will minimise such personal data and where practicable anonymise or redact it prior to transmission. 5.2 **Usage Data.** Customer acknowledges and agrees that Conduktor may collect and use Usage Data for analytical purposes and to operate, maintain and improve the Conduktor Software including for maintaining, improving, and securing the Conduktor Software and monitoring product performance and diagnosing technical issues. Conduktor may use Usage Data for product improvement, benchmarking, analytics and other internal business purposes. Usage Data does not include Customer Data. To the extent the Conduktor Software is configured to collect and transmit Usage Data (including technical telemetry) to Conduktor’s servers, such Usage Data will be limited to technical and operational information relating to use of the Conduktor Software and will not intentionally include Customer’s business data, message content, or the personal data of Customer’s end users. Conduktor may generate aggregated and/or anonymised analytics from Usage Data. Conduktor will not sell Usage Data to third parties. 5.3 **Software Security.** Conduktor maintains reasonable technical and organisational measures relating to the development, build and distribution of the Conduktor Software, including secure software development practices, vulnerability identification and remediation, and adherence to internationally recognised security frameworks such as SOC 2 at the corporate level. Conduktor's security obligations relate to the Conduktor Software itself and to the processing of Usage Data; they do not extend to securing Customer Data in Customer's environment. 5.4 **Customer Obligations.** Customer is responsible for: (a) the content, accuracy and lawfulness of Customer Data; (b) securing the environment in which the Conduktor Software is deployed, including network, infrastructure, access controls and credentials of Authorised Users; and (c) complying with applicable Laws in using the Conduktor Software. **6\. CONFIDENTIALITY** 6.1 **Obligations.** Each party will, during the term of this Agreement and thereafter, keep confidential all Confidential Information of the other party and will not use it for any purpose other than performing its obligations or exercising its rights under this Agreement, nor disclose it to any third party without the prior written consent of the other party (except to professional advisors or as required by law or a regulatory authority). The Conduktor Software and the terms and conditions of this Agreement are Conduktor's Confidential Information. Each party may disclose Confidential Information to its employees, agents, contractors and other representatives who have a legitimate need to know and are bound by confidentiality obligations no less protective than this Section 6. 6.2 **Exceptions.** These obligations do not apply to information that: (i) is or becomes publicly known other than through a breach of this Agreement; (ii) the Receiving Party created independently without reference to the Disclosing Party's Confidential Information; (iii) was already lawfully known to the Receiving Party at the time of disclosure; or (iv) is required to be disclosed by law or court order, provided the Receiving Party gives prompt prior written notice and reasonable assistance to enable the Disclosing Party to seek a protective order. 6.3 **Remedies.** Any breach or threatened breach of this Section may cause irreparable harm. The Disclosing Party is entitled to seek injunctive or other equitable relief without the requirement to prove actual damages or post bond, in addition to any other remedies available at law or in equity. **7\. OWNERSHIP AND FEEDBACK** 7.1 Conduktor and its licensors retain all intellectual property and other rights in the Conduktor Software, Documentation and related Conduktor technology, including any modifications or improvements. No rights or licences are granted to Customer except as expressly set out in this Agreement. 7.2 Customer is under no obligation to provide any suggestions, enhancement requests or other feedback ("Feedback"). If Customer chooses to provide Feedback, Customer grants Conduktor a perpetual, irrevocable, non-exclusive, worldwide, fully paid, sublicensable and assignable licence to incorporate or otherwise use the Feedback without restriction or obligation. Customer irrevocably waives, to the fullest extent permitted by applicable law, any moral rights in the Feedback. 7.3 Conduktor acknowledges that Feedback is provided on an "as-is" basis with no warranties. Feedback constitutes Conduktor's Confidential Information. **8\. WARRANTIES** 8.1 **Performance Warranty.** Conduktor warrants to Customer that, when used in accordance with this Agreement and the Documentation, the then-current, unmodified Conduktor Software will perform materially in accordance with the Documentation during the Subscription Term. 8.2 **Warranty Remedy.** If Conduktor breaches the Performance Warranty and Customer makes a reasonably detailed warranty claim within 30 days of discovering the issue, Conduktor will use reasonable efforts to correct the non-conformity. If Conduktor cannot do so within 60 days of Customer's warranty claim, either party may terminate the affected Order. Conduktor will then refund to Customer any prepaid, unused Fees for the terminated portion of the Subscription Term. These procedures are Customer's exclusive remedy and Conduktor's entire liability for breach of the Performance Warranty. The Performance Warranty does not apply to issues caused by: (a) misuse, use not in accordance with this Agreement or the Documentation, or unauthorised modifications; (b) third-party software, hardware or systems; or (c) Beta Features. 8.3 **Disclaimer.** EXCEPT AS EXPRESSLY PROVIDED IN SECTION 8.1, THE CONDUKTOR SOFTWARE AND ALL RELATED SERVICES ARE PROVIDED "AS IS" AND "AS AVAILABLE". CONDUKTOR MAKES NO OTHER WARRANTIES, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, AND SPECIFICALLY DISCLAIMS ALL IMPLIED WARRANTIES, INCLUDING WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT, TO THE FULLEST EXTENT PERMITTED BY LAW. **9\. INDEMNIFICATION** 9.1 **Conduktor Indemnity.** Conduktor will, at its own expense, defend or settle any claim brought against Customer by a third party to the extent it is based on an allegation that the Conduktor Software, as updated from time to time and used in accordance with this Agreement, infringes any patent, copyright or trade secret of that third party. Conduktor will indemnify Customer for damages, costs and reasonable attorneys' fees assessed against Customer in a final judgment exclusively attributable to such claim. These obligations are conditional on: (a) Customer promptly notifying Conduktor in writing of the claim; (b) Conduktor having exclusive control of the defence and any settlement; and (c) Customer providing reasonable assistance at Conduktor's expense. Customer shall not settle any claim without Conduktor's prior written approval. 9.2 **Exclusions.** Conduktor shall not defend, indemnify or hold Customer harmless for any claim if: (a) Customer made modifications to the Conduktor Software; (b) such claim would have been avoided by use of the then-current release of the Conduktor Software made available to Customer; (c) Customer continued the allegedly infringing activity after being provided with modifications that would have avoided the infringement; or (d) such claim is based on Customer's output. 9.3 **Remediation.** If Conduktor defends or settles an infringement claim under Section 9.1, Conduktor's sole obligation (in addition to the indemnity) is, at its option: (a) repair, replace or modify the affected Conduktor Software; or (b) procure for Customer the right to continue using the affected Conduktor Software. If neither remedy is commercially feasible in Conduktor's reasonable opinion, Conduktor may cancel the applicable Order and refund the licence Fees paid for the affected Conduktor Software. 9.4 **Customer Indemnity.** Customer will defend, indemnify and hold harmless Conduktor, its Affiliates, and each of their respective licensors, officers, employees and agents from and against any and all third-party claims, demands, suits or proceedings, and all related damages, losses, liabilities, penalties, costs and expenses (including reasonable legal fees) arising out of or relating to: any allegation that Customer Data, or Customer’s processing, transmission, storage or other handling of Customer Data, violates applicable Laws or infringes, misappropriates or otherwise violates any third-party rights; Customer’s breach of this Agreement, including any breach of the Restrictions or Customer Obligations, any breach of Customer’s export controls and sanctions obligations, or any use of the Conduktor Software in violation of applicable Laws; Customer’s systems, networks, infrastructure, configurations, or access controls (including any security incident, unauthorised access, or failure to maintain appropriate security measures in Customer’s environment); or Customer’s output, or Customer’s use of the Conduktor Software in combination with any third-party products, services, software, systems or data, to the extent the claim would not have arisen but for such output or combination. This Customer indemnity is in addition to, and does not limit or narrow, Conduktor’s indemnification obligations. Indemnity Procedures. Conduktor will promptly notify Customer in writing of any claim for which Conduktor seeks indemnification under this Customer indemnity; provided that any failure to provide prompt notice will not relieve Customer of its obligations except to the extent Customer is materially prejudiced by such failure. Customer will have sole control of the defence and settlement of the claim and will pay all defence costs and any settlement amounts or damages awarded; provided that Customer may not settle any claim in a manner that admits fault or liability on behalf of, or imposes any obligation on, Conduktor or any of its Affiliates without Conduktor’s prior written consent (not to be unreasonably withheld or delayed). Conduktor will provide reasonable cooperation in the defence of the claim at Customer’s expense. **10\. LIMITATION OF LIABILITY** 10.1 **Consequential Damages Waiver.** Except for Excluded Claims, neither party (nor its suppliers) will have any liability arising out of or related to this Agreement for any loss of use, lost data, lost profits, revenues or goodwill, interruption of business, or any indirect, special, incidental, reliance or consequential damages of any kind, even if informed of their possibility in advance. 10.2 **Liability Cap.** Except for Excluded Claims, each party's (and its suppliers') entire liability arising out of or related to this Agreement will not exceed in aggregate the amounts paid or payable by Customer to Conduktor during the 12 months immediately prior to the event giving rise to such liability. 10.3 **Excluded Claims.** "Excluded Claims" means: (a) Customer's breach of Section 2.6 (Restrictions) or Section 5.4 (Customer Obligations); (b) either party's breach of Section 6 (Confidentiality); (c) amounts payable under Conduktor's indemnification obligations in Section 9; (d) either party's wilful misconduct; or (e) Conduktor's acts or omissions resulting in death or personal injury. 10.4 **Failure of Essential Purpose.** The waivers and limitations in this Section 10 apply regardless of the form of action, whether in contract, tort (including negligence), strict liability or otherwise, and survive and apply even if any limited remedy in this Agreement fails of its essential purpose. **11\. TERM AND TERMINATION** 11.1 **Term.** This Agreement starts on the Effective Date and continues until the expiration or termination of all Subscription Terms. 11.2 **Termination for Cause.** Either party may terminate this Agreement (including all Orders) by written notice if the other party: (a) fails to cure a material breach of this Agreement (including failure to pay Fees) within 30 days after written notice of such breach; (b) ceases operations without a successor; or (c) becomes subject to any bankruptcy, insolvency, receivership or similar proceeding that is not dismissed within 60 days. Notwithstanding the foregoing, any breach by Customer of Section 2.6 (Restrictions) will entitle Conduktor to terminate this Agreement immediately upon written notice. 11.3 **Effect of Termination.** On expiration or termination of this Agreement or an Order, all of Customer's rights to use the Conduktor Software cease immediately. Where Customer terminates for Conduktor's material breach under Section 11.2(a), Conduktor will refund Customer a pro-rata portion of any prepaid Fees covering the remainder of the applicable Subscription Term after the effective date of termination. In all other termination scenarios, Fees are non-refundable. Within five days of termination, Customer must return or, at Conduktor's election, destroy or render inaccessible all of Conduktor's Confidential Information, discontinue use of the Conduktor Software and uninstall all copies. Nothing in this Section limits any other remedies Conduktor may have, or relieves Customer of obligations incurred prior to termination. 11.4 **Survival.** The following Sections survive expiration or termination of this Agreement: 1, 2.6, 4.2, 4.3, 5.4, 6, 7, 8.3, 9, 10, 11.3, 11.4 and 12. Exercising a remedy under this Agreement does not limit other remedies a party may have. **12\. GENERAL TERMS** 12.1 **Publicity.** Conduktor may identify Customer as a user of the Conduktor Software and may use Customer's name, logo and other trademarks in Conduktor's customer list, press releases, blog posts, advertisements and website (and all goodwill arising therefrom inures to the sole and exclusive benefit of Customer). Customer may withdraw this right at any time by emailing support@conduktor.io; requests will be processed within 10 Business Days, and such withdrawal will apply prospectively only. Following processing of such request, Conduktor will use commercially reasonable efforts to cease new uses and to remove Customer's name, logo and other trademarks from Conduktor-controlled websites and other Conduktor-controlled marketing materials within such timeframe; provided that Conduktor will not be required to recall, retract, or delete any materials already published or printed prior to processing the request, or any materials not under Conduktor's control (including third-party posts or publications and paid advertisements already placed). Neither party may otherwise use the other's name, logo or trademarks without prior written consent. 12.2 **Assignment**. Customer may not assign or transfer this Agreement or any of its rights or obligations without Conduktor's prior written consent. Any purported assignment without consent is null and void. Conduktor may assign this Agreement to a successor to its business by merger, asset sale, sale of stock or reorganisation without consent. This Agreement binds the parties and their permitted successors and assigns. 12.3 **Subcontractors.** Conduktor may use subcontractors and permit them to exercise Conduktor's rights under this Agreement. Conduktor remains responsible for subcontractors' compliance with this Agreement and for its overall performance. 12.4 **Force Majeure.** Neither party will be liable for any failure or delay in performance caused by circumstances beyond its reasonable control, including acts of God, floods, fires, earthquakes, pandemics, wars, terrorism, civil unrest, strikes, labour stoppages or governmental action. The affected party will promptly notify the other in writing when such an event causes a delay or failure and when it ceases to do so. If such an event continues for more than three consecutive calendar months, either party may terminate this Agreement by written notice. This Section does not excuse either party's obligation to make payments due under this Agreement. 12.5 **Anti-Corruption and Trade Compliance.** Each party warrants that it will comply with all applicable anti-corruption, anti-money-laundering, economic sanctions, export controls and other international trade laws and regulations. Each party represents that it (and, in Customer's case, its Authorised Users) is not on any government prohibited, denied, unverified-party, sanctions, debarment or exclusion list. Customer will immediately discontinue use of the Conduktor Software and remove Authorised Users' access if Customer or any Authorised User is placed on any such list. Either party may terminate this Agreement immediately upon written notice if the other party breaches this Section. 12.6 **Export Controls.** Customer may not export or re-export the Conduktor Software or any underlying information or technology except in full compliance with applicable Laws. The Conduktor Software may not be exported: (a) into any country to which the United States has embargoed goods; or (b) to any person on the US Treasury Department's list of specially designated nationals or the US Commerce Department's list of prohibited parties. Customer represents and warrants that it is not located in, under control of, or a national or resident of any such country or on any such list. 12.7 **Governing Law and Jurisdiction.** This Agreement is governed by the laws of the State of New York, without reference to its conflict-of-law principles. Both parties submit to the exclusive jurisdiction of the state and federal courts sitting in the City and County of New York in relation to any dispute arising out of or in connection with this Agreement. The UN Convention on Contracts for the International Sale of Goods is excluded. 12.8 **Notices.** Notices under this Agreement must be in writing and are deemed given: (a) upon personal delivery or overnight courier; (b) three Business Days after first-class, registered or certified mail, postage prepaid; or (c) when sent by email to the address on record. Customer is responsible for keeping its contact details current. Notices to Conduktor must be sent to legal@conduktor.io. 12.9 **Insurance.** During the Subscription Term, Conduktor will maintain industry-standard commercial insurance coverage. 12.10 **Entire Agreement and Amendments.** This Agreement (including all Orders and the Privacy Policy) is the entire agreement between the parties regarding its subject matter and supersedes all prior or contemporaneous agreements. Any amendments must be in writing and signed by authorised representatives of both parties. Conduktor may update the Privacy Policy with notice to Customer, but updates will not materially decrease Conduktor's overall obligations during a Subscription Term. Customer purchase orders or vendor management portal terms have no legal effect on this Agreement and are expressly rejected by Conduktor. 12.11 **Waivers and Severability.** Waivers must be in writing signed by the waiving party's authorised representative and cannot be implied from conduct. If any provision of this Agreement is found invalid, illegal or unenforceable, it will be limited to the minimum extent necessary so the remainder of the Agreement remains in full force. 12.12 **Relationship of the Parties.** The parties are independent contractors. Nothing in this Agreement creates an agency, partnership or joint venture. Neither party has authority to make representations or create obligations on behalf of the other. 12.13 **Class Action and Jury Trial Waiver.** ALL CLAIMS MUST BE BROUGHT IN THE PARTIES' INDIVIDUAL CAPACITY, AND NOT AS A PLAINTIFF OR CLASS MEMBER IN ANY CLASS ACTION, COLLECTIVE ACTION, PRIVATE ATTORNEY GENERAL ACTION OR OTHER REPRESENTATIVE PROCEEDING. UNLESS THE PARTIES AGREE OTHERWISE, CLAIMS MAY NOT BE CONSOLIDATED. EACH PARTY WAIVES THE RIGHT TO A JURY TRIAL OR TO PARTICIPATE IN ANY CLASS OR REPRESENTATIVE ACTION OF ANY KIND. 12.14 **US Government Restricted Rights.** If the Conduktor Software is licensed by the US Government, it is deemed to be "commercial computer software" and "commercial computer documentation" developed exclusively at private expense. Civilian agencies: subject to 48 C.F.R. 12.212 of the Federal Acquisition Regulations and its successors. Department of Defense: subject to 48 C.F.R. 227.7202-2, DOD FAR Supplement and its successors. 12.15 **Third-Party Software.** Conduktor makes certain third-party open-source software available alongside the Conduktor Software. Such third-party software is subject to its applicable open-source licences and not this Agreement, and is provided at no charge. The Conduktor Software may contain links to third-party websites, services or software, provided for convenience only. Conduktor is not responsible for any such third-party content. Customer accesses third-party content at its own risk. 12.16 **Construction.** Section headings are for convenience only. "Including" and similar terms are to be construed without limitation. No rule of construction resolving ambiguity against the drafting party applies. **SCHEDULE 1 — SUPPORT TERMS** 1. **Scope.** During the Subscription Term, Conduktor will provide support services to Customer in accordance with this Schedule. Support is available during Regular Business Hours only. All support is provided in English. 2. **Submitting Requests.** Support requests must be submitted through the in-software support functionality where available, or via email to support@conduktor.io. Requests submitted outside Regular Business Hours will be addressed when Regular Business Hours resume. 3. **Response Times.** Conduktor will use reasonable efforts to provide an initial response to all support requests within 48 Business Hours of receipt. An initial response is an acknowledgement of receipt and the beginning of work on the issue. Full resolution time may be longer and is not guaranteed. 4. **Updates.** Updates are provided in accordance with Section 3.4 of the Agreement. Conduktor will provide support under this Schedule for each update for one year from the date of its commercial release. 5. **Excluded Support.** The exclusions set out in Section 3.3 of the Agreement apply to support provided under this Schedule. Source: https://www.conduktor.io/legal/on-premise-customer-agreement --- # Privacy Policy Updated August 30th, 2020 Conduktor Inc. (“Conduktor,” “we,” “our,” or “us”) understand that privacy is important for visitors to our website (our “Site”) and users of our products and services (our “Products”).This Privacy Policy ("Policy") explains how we collect, use, and share your personal information, and how you can exercise your privacy rights. If you are unable to access this Policy due to a disability or any physical or mental impairment, please contact us using the contact details in the How to Contact Us section and we will arrange to supply you with the information you need in an alternative format that you can access. You can also access a printable version of this Policy. This Privacy Policy covers personal information we process: (1) When you visit our Site, and in the usual course of business, such as in connection with our events, sales, and marketing activities (collectively, "Visitors"); and (2) When you register to or otherwise use Conduktor Products whether as an entity, individual, as an applicant, team member or employee of our customers (collectively, "Users"). This Privacy Policy does not apply to Site or Products that display or link to different privacy policies. This Policy also does not apply where Conduktor processes personal information of our customers on their behalf; our collection and processing of such personal information is governed by our agreements with those customers. We recommend that you read this Policy in its entirety to ensure you are fully informed. #### 1\. WHO WE ARE Conduktor is a company headquartered in 154 W 14th Street, 6th floor, New York, NY 10011, United States. Conduktor’s Products help organizations and developers work with Apache Kafka. #### 2\. PERSONAL INFORMATION WE COLLECT The information we collect depends on the context of your interactions with Conduktor and the choices you make (including your privacy settings), the Products and features you use, your location, and applicable law. **(i) Information You Provide to Us** **WHEN YOU VISIT OUR SITE** Certain parts of our Site may ask you to provide personal information voluntarily. We collect and store personal information that you directly provide us through our Site, through interactions on social media, participation in a survey or promotion, application for a job, when downloading white papers on our Site, and at events. We will collect certain personal information so that we may fulfill your request or keep in touch with them in connection with our sales and marketing activities (always in accordance with your marketing preferences). Information we collect includes: - **Contact information**: your name, email address, phone number, mailing address, billing address, and usernames; - **Demographic data**: your job title, company name, city, state, and country; - **Payment information**: your credit card numbers and other payment information. Our third-party payment processors (like Stripe) collect this information if you purchase our Products, such as online training, cloud services, or software subscriptions; and - **Content and communications**: any data you enter into any ‘free text’ boxes on our forms; comments in our chat forms, and communications with us on social media and other platforms, through phone, or messaging services. **WHEN YOU USE OUR PRODUCTS** If you are a User, you or your employer, may provide certain personal information to us through the use of Conduktor Products - for example, when you register for a Conduktor account to access and use our Products, when you consult with customer support, or send us an email or communicate with us in any way. Personal information we collect may include: - **Business contact information**: such as your name, job title, organization, phone number, email address and country. - **Marketing information**: such as your contact preferences. - **Account log-in credentials**: such as your email or username and password when you sign up for an account with us and the unique User/ team ID assigned to you in our systems. - **Troubleshooting and support data**: such as personal information you provide or we otherwise access in connection with support queries we receive from Users. This may include, for example, contact or authentication data, the content of your chats and other communications with Conduktor, and the product or service you are using related to your help inquiry. - **Computer Information**: such as the internet protocol (IP) addresses of computers on which our Product is installed. - **Brokers and cluster metadata**: such as ID information of the brokers and clusters with which you are using our Products. **(ii) Information We Collect Automatically.** **WHEN YOU VISIT OUR SITE** When you visit our Site or interact with our emails, we gather certain technical information from your browser or device automatically and store it in log files. In some (but not all) countries, including countries in the European Economic Area ("EEA"), UK and Switzerland, this information is considered personal information under data protection laws. To the extent that this automatically collected data includes, or is linked to, personal information, we will treat the data in accordance with this Privacy Policy. This information we collect includes: - **Geolocation data**: Such as your geolocation data when you visit our Site and use our Products. For example, we may derive your general location using an IP address. - **Identifiers and device information**. Such as your device’s operating system, device identifier, customer ID and other device information, user agent string, Internet Protocol (IP) address, access times, browser type, and log data detailing your interactions with our Site (e.g., number of clicks, pages viewed, information searched for), and the website you visited before and after coming to our Site (i.e., referrer header) are logged automatically. When you submit a support ticket, we receive data from the user agent string of your browser, which includes device information, browser information, OS information, city, state, country, and IP address. - **Information collected through social media and other platforms**. We receive information about you when you engage with us through various third-party platforms (e.g., Google group, Facebook, LinkedIn, GitHub, Twitter, Instagram, etc.), or by sharing content from our Site on Facebook, Twitter, or LinkedIn. The data we exchange with these third-party platforms may depend upon your privacy settings with these platforms. You should review and consider adjusting your privacy settings on third-party websites and services before engaging. Do not provide us with any sensitive personal data through these platforms. We are not responsible for the data protection and use practices of these third-party platforms. Please see their privacy policies to learn how they use your information. For more information please see the “Our Use of Cookies and Other Technologies” and “Product telemetry data” sections below. **WHEN YOU USE OUR PRODUCTS** When you use our Products, we collect certain technical information from your browser or device automatically. This information includes: - **Geolocation data**: Depending on your device settings, we collect geolocation data when you use our Products. For example, we may derive your general location using an IP address. - **Product telemetry data**: Such as telemetry data and other data about your usage, including your IP Address and other unique identifiers in combination with information about the version of our software you are running and how it is configured, to collect and aggregate certain diagnostic and analytics information. We may combine this information with information you provide when you download our Products or engage with customer support, including the name of your organization or company. You can turn off the collection of telemetry data as described in the “Choices” section below. - **Inferences**. We infer new information about you and your company from data we collect, including using automated means to generate information about your likely preferences, your service and product needs or other characteristics. For example, we infer your city, state, and country location based on your IP address. - **Information collected through social media and other platforms**. We receive information about you when you engage with us through various third-party platforms (e.g., Google group, Facebook, LinkedIn, GitHub, Twitter, Instagram, etc.), or by sharing content from our Site on Facebook, Twitter, or LinkedIn. The data we exchange with these third-party platforms may depend upon your privacy settings with these platforms. You should review and consider adjusting your privacy settings on third-party websites and services before engaging. Do not provide us with any sensitive personal data through these platforms. We are not responsible for the data protection and use practices of these third-party platforms. Please see their privacy policies to learn how they use your information. **(iii) Information We Collect from Third-party Sources.** We may occasionally receive your personal information from third party sources including our marketing and research partners. Users of our Products may also provide your personal information when they identify you as a billing, support, or technical contact, or when inviting you to use our Products or attend our events. If you integrate or link a third-party service with our Site or Products, we may receive personal information about you from that third-party service based upon the settings and permissions you've established with such third party service, and that third party service's privacy practices. The types of information we collect from third parties may include contact information, demographic data, and content and communications. We use this information to maintain and improve the accuracy of the records we hold about you, identify new customers, and provide a more tailored advertising experience. We may combine this information with other information we collect about you through our Site and Products. When you are asked to provide personal information, you may decline. But if you choose not to provide information that is necessary to provide certain Products, those Products may not be available or function correctly. 3. **OUR USE OF COOKIES AND SIMILAR TECHNOLOGIES** Our Site and related online services use cookies and similar technologies to enable certain functionality and help collect information about your visit. We use the following technologies, over which you have certain options: “**Cookies**” are small text files, typically containing a unique string of letters and numbers, stored on your hard drive by a Site. When you return to the Site using the same browser, the Site can read the cookie and thereby gather information about your usage over time. Among other things, we use cookies and other technology to see which areas and features are popular and to count visits, which helps us to improve our Site, our Products, and your experience. Most Web browsers are set to accept cookies by default. If you prefer, you can usually set your browser to remove cookies and to reject cookies. If you choose to remove or reject cookies, this could affect certain features or services of our Site or other Products. **Web Beacons**. We may collect information using Web beacons. Web beacons are electronic images that may be used on our Site, Products, or in our emails. We use Web beacons to deliver cookies, count visits, understand usage and campaign effectiveness, and to tell if an email has been opened and acted upon. **Hotjar**. We use Hotjar in order to better understand our users’ needs and to optimize this service and experience. Hotjar is a technology service that helps us better understand our users’ experience (e.g. how much time they spend on which pages, which links they choose to click, what users do and don’t like, etc.) and this enables us to build and maintain our service with user feedback. Hotjar uses cookies and other technologies to collect data on our users’ behavior and their devices. This includes a device's IP address (processed during your session and stored in a de-identified form), device screen size, device type (unique device identifiers), browser information, geographic location (country only), and the preferred language used to display our website. Hotjar stores this information on our behalf in a pseudonymized user profile. Hotjar is contractually forbidden to sell any of the data collected on our behalf. For further details, please see the ‘about Hotjar’ section of [Hotjar’s support site.](https://help.hotjar.com/hc/en-us/requests/new) **Google Analytics**: **Do Not Track**. There are many ways through which web browser signals and other similar mechanisms (for example, “Do Not Track”) can indicate your choice to disable tracking, and, while we and others give you choices described in this Privacy Policy, we do not currently honor these mechanisms. #### 4\. OUR USE OF PERSONAL INFORMATION Conduktor processes personal information for the following business purposes, and if you are a resident in the EEA or UK, on the legal basis identified below: - **Providing our website and Conduktor Products**: In reliance on our legitimate interest, we process your personal information to operate and administer our website, and to provide, operate, and maintain the Conduktor Products; - **Communicating with you about the Conduktor Products**: We may send you service, technical and other administrative or technical email, messages and other types of notifications(such as distribution and product updates and product patches and fixes) in reliance on our legitimate interests in administering the Conduktor Products and providing certain features. These communications are considered part of the Conduktor Products and in most cases you cannot opt-out of them. If an opt-out is available, you will find that option within the communication itself; - **Providing** **necessary** **functionality**: We process your personal information in reliance on our legitimate interest to provide you with the necessary functionality required during your use of our website and Conduktor Products; - **Transactional considerations**: We process your personal information to complete transactions, and send you related information, including purchase confirmations and invoices, to perform our contract with you and to the extent necessary in reliance on our legitimate interest; - **Handling** **contact and support requests**: If you fill out a “Contact Us” web form or request support as a User, or if you contact us by other means including via a phone call, we process your personal information to perform our contract with you and/or (if we have not entered into a contract with you) to the extent it is necessary for our legitimate interest in fulfilling your requests and communicating with you; - **Administering** **Events**: We process your personal information to plan and host events or webinars for which you have registered or that you attend, including sending related communications to you, billing, registration and to connect you with other event attendees, to perform of our contract with you, or to the extent necessary for our legitimate interests in fulfilling your requests to attend any such events; - **Developing and improving our website and services**: We process your personal information to analyze trends and to track your usage of and interactions with our website. marketing activities and Conduktor Products to the extent it is necessary for our legitimate interest in developing, improving and troubleshooting our website, marketing activities and the Conduktor Products and providing you with more relevant content and service offerings, or where necessary, in reliance on your consent; - **Sending** **marketing** **communications**: We will process your personal information for marketing purposes in accordance with your preferences, such as to communicate with you via email, SMS or telephone about services, features, surveys, newsletters, promotions or events we think may be of interest to you and/or to provide other news or information about Conduktor and/or our select partners, in each case in reliance on our legitimate interest in conducting direct marketing or where necessary with your consent. Please see the "Your Privacy Rights" section below, to learn how you can control the processing of your personal information by Conduktor for marketing purposes; - **Displaying personalized advertisements and content**: We process your personal information to conduct marketing research, advertise to you, provide personalized information about us on and off our website and to provide other personalized content based upon your activities and interests to the extent it is necessary for our legitimate interest in supporting our marketing activities or advertising the Conduktor Products or, where necessary, to the extent you have provided your prior consent (please see the "Your Privacy Rights" section, below, to learn how you can control how the processing of your personal information for personalized advertising purposes); - **Promoting the security of our website and service**s: To the extent necessary for our legitimate interests in promoting the safety and security of our website and services, we use your personal information to investigate and prevent fraudulent transactions, unauthorized access to the website, Conduktor Products, and other illegal activities; - **For** **our business purposes**, such as data analysis, audits, fraud monitoring and prevention, developing new products and features, enhancing, improving or modifying our products and services, identifying usage trends and expanding our business activities in reliance on our legitimate interests; - **Complying with legal obligations**: We process your personal information when cooperating with public and government authorities, courts or regulators in accordance with our legal obligations under applicable laws to the extent this requires the processing or disclosure of personal information to protect our rights or is necessary for our legitimate interest in protecting against misuse or abuse of our website, the Conduktor Products, protecting personal property or safety, pursuing remedies available to us and limiting our damages, complying with judicial proceedings, court orders or legal processes or to respond to lawful requests; - **Reviewing compliance with applicable usage terms**: We process your personal information to review compliance with our contract with you or your organization (where applicable) to the extent that it is in our legitimate interest to ensure adherence to the relevant terms; and - **Other** **purposes**: We will process your personal information for other purposes about which we notify you in advance, or for which we receive your consent. In general, we collect and process personal information about you, as necessary to provide the Products you use, operate our Site and business, meet our contractual and legal obligations, protect the security of our systems and our customers, or fulfil other legitimate interests as described in this Privacy Policy and in our notices to you. #### 5\. SHARING OF PERSONAL INFORMATION We may share or disclose your personal information to the following categories of recipients: - **Partners and other similar parties**. We may share your personal information with our sponsors and co-sponsors of events, whose information we believe may be relevant to you. We may also share your personal information with research and advisory firms whose reports are offered through our Site (subject to applicable laws). - **Third-party vendors and other service providers**. We may share your personal information with our third-party vendors and service providers that we use to provide services such as payment processing, reselling services, support ticket portals, secure transfer software, cloud hosting, video conference services, marketing automation platforms, project management tools, registration services, learning management services, collaboration and communication tools, data backup services, and professional services. - **Business transfers**. We may share your personal information with other parties in connection with a company transaction, such as a merger, sale of company assets or shares, reorganization, financing, change of control or acquisition of all or a portion of our business or assets by another company or third party, or in the event of a bankruptcy, dissolution, or related or similar proceedings. - **Compliance with laws**. We may share your personal information as required by law or subpoena or if we reasonably believe that such action is necessary to comply with the law or the reasonable requests of law enforcement. - **Legal rights**. We may share your personal information to enforce our Terms of Use or other agreements, to protect the security or integrity of our Site and Products, or to exercise or protect the rights, property, or personal safety of Conduktor, our customers, users, or others. - **Advertising partners**. We may partner with third party advertising networks, exchanges and social media platforms to display advertising on our website or to manage and service advertising on other sites and we may share personal information with them for this purpose. Please see the section above "Our Use of Cookies and Similar Tracking Technologies" for further information. #### 6\. **YOUR CHOICES ABOUT PERSONAL INFORMATION** Depending on your location, your jurisdiction, and subject to applicable law, you may have the rights below with regard to the personal information we control about you. We will respond to your requests within the appropriate timeline under applicable law. **Privacy Rights (EEA residents & California Consumers)**. EEA residents and California consumers are afforded certain privacy rights by relevant privacy laws, such as the General Data Protection Regulation ("GDPR") and the California Consumer Privacy Act ("CCPA"), respectively. Please see below for applicable rights under each legal regime. **European Data Protection Rights (EEA Residents)** If the processing of personal data about you is subject to European Union data protection law, you have certain rights with respect to that data: - You may request access to, and correction or erasure of, your personal information at any time. - You can object to processing of your personal information, ask us to restrict processing of your personal information or request portability of your personal information. - If the processing of personal information is based on your consent, you have a right to withdraw consent at any time for future processing. Withdrawing your consent will not affect the lawfulness of any processing we conducted prior to your withdrawal, nor will it affect processing of your personal information conducted in reliance on lawful processing grounds other than consent. - You have the right to complain to a data protection authority about our collection and use of your personal information but we encourage you to first contact us with any questions or concerns. For more information, please contact your local data protection authority. Contact details for data protection authorities in the European Economic Area (“EEA“) are available at [https://ec.europa.eu/justice/article-29/structure/data-protection-authorities/index\_en.htm](https://ec.europa.eu/justice/article-29/structure/data-protection-authorities/index_en.htm) - Conduktor does not engage in any automated decision making with User personal information. You may exercise the rights above by using this form or by emailing us at [legal@conduktor.io](mailto:legal@conduktor.io). **California Privacy Rights (California Consumers)** The California Consumer Privacy Act (“CCPA”), provides California residents with the following rights: - You may request access to, or for a copy of the personal information we have collected, used, disclosed, and sold about you over the past twelve (12) months. Please see the section below "Additional Disclosures for California Consumers" for more information on our general practices. - You may also request that we delete certain personal information we have collected from you. - You have a right not to receive discriminatory treatment for the exercise of your CCPA privacy rights. - You may also have the right to opt out of the sale of your personal information. In consideration of the right, Conduktor does not sell information as the terms “sale” or “sell” are traditionally understood (i.e. for money). However, making a California resident’s personal information (including IP addresses, cookies IDs, and mobile IDs) available to third parties as described above may broadly be considered a “sale” under the CCPA, given its broad definition of "sale." The definition is so broad it could include the sharing of personal information with third parties in exchange for something of value, even if no money changes hands. As discussed in this Privacy Policy, our advertising and analytics providers may collect your IP address, cookie ID, and mobile ID when you use our website, and such vendors may further share your information to provide similar advertising or analytics services to their other customers. If you are a California resident seeking to exercise your CCPA rights, or if you are an authorized agent wishing to exercise CCPA rights on behalf of someone else, please email us at [legal@conduktor.io](mailto:legal@conduktor.io). Please note that to protect your personal information, we will verify your identity by a method appropriate to the type of request you are making. We may also request that your authorized agent have written permission from you to make requests on your behalf, and we may also need to verify your authorized agent's identity to protect your personal information. #### 7\. ADDITIONAL DISCLOSURES FOR CALIFORNIA CONSUMERS California law requires us to provide California consumers with some additional information regarding how we collect, use, and share your "personal information" (as defined in the CCPA). Throughout this Policy, we discuss in detail the specific pieces of information we collect from you or your device and discuss how we use and share such information. As described in detail in section 2 above we may collect the following categories of information about you through the Conduktor Product or when you visit our site: - Identifiers, such as name and contact information; - Device identifiers, such as IP address; - Internet or other network or device activity, such as browsing history or app usage; - Geolocation information; and other information that identifies or can be reasonably associated with your device; - Commercial information, such as transaction data; - Professional or other employment related information, such as where you work and your title; and - Financial information, such as credit card and payment data. The sources from which we collect personal information are described in Section 2 ("Personal Information We Collect") of this Privacy Policy The business and commercial purposes for which we collect this information are described in section 4 ("Our Use of Personal Information") of this Privacy Policy. The categories of third parties to whom we "disclose" this information for a business purpose are described in section 5 ("Sharing of Personal Information") of this Privacy Policy. The CCPA defines ‘sale’ very broadly. It includes the sharing of California Information in exchange for anything of value. According to this broad definition, in the year before this section was last updated, we may have sold the following categories of personal information to third parties: identifiers (like your IP Address) and other electronic network activity (such as browsing history). We may have sold these categories of personal information to the third parties listed in section 5 ("Sharing of Personal Information") of this Privacy Policy. #### 8\. HOW WE STORE AND PROTECT PERSONAL INFORMATION **Storage and processing**. Your information collected through the Site and our Products may be stored and processed in any country in which Conduktor or its service providers maintain facilities. Our processing locations are chosen in order to operate efficiently, to improve performance, and to create redundancies in order to protect the data in the event of an outage or other problem. We take steps designed to ensure that the data we collect under this Privacy Policy is processed according to the provisions of this Privacy Policy and applicable law wherever the data is located. **International data transfers**. When we transfer personal information from the European Economic Area, or the United Kingdom to the United States or other countries which have not been determined by the European Commission to have laws that provide an adequate level of data protection, we use legal mechanisms, including contracts, designed to help ensure your rights and protections. **Keeping your information safe**. Conduktor cares about the security of your information and takes reasonable and appropriate technical and organizational measures designed to prevent loss, misuse, and unauthorized access, disclosure, alteration, and destruction of personal information. However, no security system is impenetrable, and we cannot guarantee the security of our systems or your information. For this reason, you should be mindful of the information you provide to us. **Lawful basis for processing personal information (EEA only)**. If you are located in the European Economic Area (EEA), Conduktor Inc., is the data controller of your information. For more details on the legal bases used for the processing of your personal information, please see section 4 "Our Use of Personal Information". Our legal basis for collecting and using the personal information above will depend on the personal information concerned and the specific context in which we collect it. However, we will normally collect personal information only where we have your consent to do so, where we need the personal information to perform a contract with you, or where the processing is in our legitimate interests and not overridden by your data protection interests or fundamental rights and freedoms. In some cases, we may also have a legal obligation to collect personal information from you. If we ask you to provide personal information to comply with a legal requirement or to perform a contract with you, we will indicate this at the relevant time and advise you whether the provision of your personal information is mandatory or not (as well as the possible consequences, if any, if you do not provide your personal information). Similarly, if we collect and use your personal information in reliance on our legitimate interests (or those of a third party), we will indicate to you at the relevant time what those legitimate interests are. If you have questions about the legal basis for processing or want to find out more, please contact us using the details at the end of this Privacy Policy in the section "How to Contact Us".\] **Retention**. We retain personal information for as long as we have an ongoing legitimate business need to do so. For example, we retain your account information for as long as your account is active or as needed to provide you with Products you have requested or authorized, including maintaining and improving the performance of the Products and protecting system security. We also retain personal data as needed to maintain appropriate business and financial records, protect our legal interests, resolve disputes, or comply with legal or regulatory requirements. When we have no ongoing legitimate business need to process your personal information, we will either delete or anonymize it, or if this is not possible (for example, because your personal information has been stored in backup archives), then we will store your personal information using appropriate security measures and take appropriate steps designed to isolate it from any further processing until deletion is possible. #### 9\. CHANGES TO OUR PRIVACY POLICY Conduktor may modify or update this Privacy Policy from time to time to reflect the changes in our business and practices. Please review this page periodically. If we make any changes to this Privacy Policy, we will notify you by changing the "Last Updated" date above. If we make any material changes, we will provide you with additional notice or obtain consent as may be required by applicable law. This website is operated by Conduktor Inc. If you have any questions, complaints, or concerns about how your information is handled, or would like to exercise any of your rights for yourself or on behalf of someone else, please email us at #### support@conduktor.io Source: https://www.conduktor.io/legal/privacy-policy --- # Terms of Service Updated June 21, 2024 #### **1\. Website Terms of Service Terms (“Terms”)** By accessing the website at [https://www.conduktor.io/](https://www.conduktor.io/) or using the Conduktor software, you are agreeing to be bound by these Terms, all applicable laws and regulations, and agree that you are responsible for compliance with any applicable local laws. If you do not agree with any of these Terms, you are prohibited from accessing the website or using the Conduktor software . The materials contained in this website are protected by applicable copyright and trademark law. Please read these Terms of Service carefully before you start to use the website. By using the website, you accept and agree to be bound by these Terms and our [Privacy Policy](#legal/privacy-policy), incorporated herein by reference. If you do not agree to these Terms or the Privacy Policy, you must not access or use the website. Any use of the Conduktor software is also subject to the End User License Agreement, found at \[INSERT\], and/or an Enterprise agreement. #### **2\. Changes to the Terms** We may revise and update these Terms from time to time in our sole discretion. All changes are effective immediately when we post them. Your continued use of the website following the posting of revised Terms means that you accept and agree to the changes. You are expected to check this page so you are aware of any changes, as they are binding on you. #### **3\. Accessing the website and account security** We reserve the right to withdraw or amend this website, and any service or material we provide on the website, in our sole discretion without notice. We will not be liable if for any reason all or any part of the website is unavailable at any time or for any period. From time to time, we may restrict user access, including registered user access, to some parts of the website or the entire website. You are responsible for both: Making all arrangements necessary for you to have access to the website. Ensuring that all persons who access the website through your internet connection are aware of these Terms and comply with them. To access the website or some of the resources it offers, you may be asked to provide certain registration details or other information. It is a condition of your use of the website that all the information you provide on the website is correct, current, and complete. You agree that all information you provide to register with this website or otherwise, including, but not limited to, through the use of any interactive features on the website, is governed by our [Privacy Policy](#legal/privacy-policy), and you consent to all actions we take with respect to your information consistent with our Privacy Policy. If you choose, or are provided with, a username, password, or any other piece of information as part of our security procedures, you must treat such information as confidential, and you must not disclose it to any other person or entity. You also acknowledge that your account is personal to you and agree not to provide any other person with access to this website or portions of it using your username, password, or other security information. You agree to notify us immediately of any unauthorized access to or use of your username or password or any other breach of security. You also agree to ensure that you exit from your account at the end of each session. You should use particular caution when accessing your account from a public or shared computer so that others are not able to view or record your password or other personal information. We have the right to disable any username, password, or other identifier, whether chosen by you or provided by us, at any time if, in our opinion, you have violated any provision of these Terms. #### **4\. Intellectual Property Rights** The website and its entire contents, features, and functionality (including but not limited to all information, software, text, displays, images, video, and audio, and the design, selection, and arrangement thereof) are owned by the Company, its licensors, or other providers of such material and are protected by United States and international copyright, trademark, patent, trade secret, and other intellectual property or proprietary rights laws. These Terms permit you to use the Website for your personal, non-commercial use only. You must not reproduce, distribute, modify, create derivative works of, publicly display, publicly perform, republish, download, store, or transmit any of the material on our Website, except as follows: Your computer may temporarily store copies of such materials in RAM incidental to your accessing and viewing those materials. You may store files that are automatically cached by your Web browser for display enhancement purposes. You may print or download one copy of a reasonable number of pages of the website for your own personal, non-commercial use and not for further reproduction, publication, or distribution. If we provide desktop, mobile, or other applications for download, you may download a single copy to your computer or mobile device solely for your own personal, non-commercial use, provided you agree to be bound by our End User License Agreement for such applications. You must not: Modify copies of any materials from this site. Delete or alter any copyright, trademark, or other proprietary rights notices from copies of materials from this site. You must not access or use for any commercial purposes any part of the website or any services or materials available through the website. If you print, copy, modify, download, or otherwise use or provide any other person with access to any part of the website in breach of the Terms, your right to use the website will stop immediately and you must, at our option, return or destroy any copies of the materials you have made. No right, title, or interest in or to the website or any content on the website is transferred to you, and all rights not expressly granted are reserved by the Conduktor. Any use of the website not expressly permitted by these Terms is a breach of these Terms and may violate copyright, trademark, and other laws. Conduktor’s names, trademarks and logos name, and all related names, logos, product and service names, designs, and slogans are trademarks of Conduktor or its affiliates or licensors. You must not use such marks without the prior written permission of Conduktor. All other names, logos, product and service names, designs, and slogans on this website are the trademarks of their respective owners. #### **5\. Prohibited Uses** You may use the website only for lawful purposes and in accordance with these Terms. You agree not to use the website: In any way that violates any applicable federal, state, local, or international law or regulation (including, without limitation, any laws regarding the export of data or software to and from the US or other countries). For the purpose of exploiting, harming, or attempting to exploit or harm minors in any way by exposing them to inappropriate content, asking for personally identifiable information, or otherwise. To impersonate or attempt to impersonate Conduktor, a Conduktor employee, another user, or any other person or entity (including, without limitation, by using email addresses associated with any of the foregoing). To engage in any other conduct that restricts or inhibits anyone's use or enjoyment of the Website, or which, as determined by us, may harm the Conduktor or users of the website, or expose them to liability. Additionally, you agree not to: Use the website in any manner that could disable, overburden, damage, or impair the site or interfere with any other party's use of the website, including their ability to engage in real time activities through the website. Use any robot, spider, or other automatic device, process, or means to access the website for any purpose, including monitoring or copying any of the material on the website. Use any manual process to monitor or copy any of the material on the website, or for any other purpose not expressly authorized in these Terms, without our prior written consent. Use any device, software, or routine that interferes with the proper working of the Website. Introduce any viruses, Trojan horses, worms, logic bombs, or other material that is malicious or technologically harmful. Attempt to gain unauthorized access to, interfere with, damage, or disrupt any parts of the website, the server on which the website is stored, or any server, computer, or database connected to the website. Attack the website via a denial-of-service attack or a distributed denial-of-service attack. Otherwise attempt to interfere with the proper working of the Website. #### 6\. Links Conduktor has not reviewed all of the sites linked to its website and is not responsible for the contents of any such linked site. The inclusion of any link does not imply endorsement by Conduktor of the site. Use of any such linked website is at the user’s own risk. #### 7\. Information About You and Your Visits to the Website All information we collect on this Website is subject to our [Privacy Policy](#legal/privacy-policy). By using the Website, you consent to all actions taken by us with respect to your information in compliance with the Privacy Policy. #### 8\. Disclaimer of Warranties You understand that we cannot and do not guarantee or warrant that files available for downloading from the internet or the Website will be free of viruses or other destructive code. You are responsible for implementing sufficient procedures and checkpoints to satisfy your particular requirements for anti-virus protection and accuracy of data input and output, and for maintaining a means external to our site for any reconstruction of any lost data. TO THE FULLEST EXTENT PROVIDED BY LAW, WE WILL NOT BE LIABLE FOR ANY LOSS OR DAMAGE CAUSED BY A DISTRIBUTED DENIAL-OF-SERVICE ATTACK, VIRUSES, OR OTHER TECHNOLOGICALLY HARMFUL MATERIAL THAT MAY INFECT YOUR COMPUTER EQUIPMENT, COMPUTER PROGRAMS, DATA, OR OTHER PROPRIETARY MATERIAL DUE TO YOUR USE OF THE WEBSITE OR ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE OR YOUR DOWNLOADING OF ANY MATERIAL POSTED ON IT, OR ON ANY WEBSITE LINKED TO IT. YOUR USE OF THE WEBSITE, ITS CONTENT, AND ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE IS AT YOUR OWN RISK. THE WEBSITE, ITS CONTENT, AND ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE ARE PROVIDED ON AN "AS IS" AND "AS AVAILABLE" BASIS, WITHOUT ANY WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED. NEITHER THE COMPANY NOR ANY PERSON ASSOCIATED WITH THE COMPANY MAKES ANY WARRANTY OR REPRESENTATION WITH RESPECT TO THE COMPLETENESS, SECURITY, RELIABILITY, QUALITY, ACCURACY, OR AVAILABILITY OF THE WEBSITE. WITHOUT LIMITING THE FOREGOING, NEITHER CONDUKTOR NOR ANYONE ASSOCIATED WITH CONDUKTOR REPRESENTS OR WARRANTS THAT THE WEBSITE, ITS CONTENT, OR ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE WILL BE ACCURATE, RELIABLE, ERROR-FREE, OR UNINTERRUPTED, THAT DEFECTS WILL BE CORRECTED, THAT OUR SITE OR THE SERVER THAT MAKES IT AVAILABLE ARE FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS, OR THAT THE WEBSITE OR ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE WILL OTHERWISE MEET YOUR NEEDS OR EXPECTATIONS. TO THE FULLEST EXTENT PROVIDED BY LAW, CONDUKTOR HEREBY DISCLAIMS ALL WARRANTIES OF ANY KIND, WHETHER EXPRESS OR IMPLIED, STATUTORY, OR OTHERWISE, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT, AND FITNESS FOR PARTICULAR PURPOSE. THE FOREGOING DOES NOT AFFECT ANY WARRANTIES THAT CANNOT BE EXCLUDED OR LIMITED UNDER APPLICABLE LAW. #### 9\. Limitation on Liability TO THE FULLEST EXTENT PROVIDED BY LAW, IN NO EVENT WILL CONDUKTOR, ITS AFFILIATES, OR THEIR LICENSORS, SERVICE PROVIDERS, EMPLOYEES, AGENTS, OFFICERS, OR DIRECTORS BE LIABLE FOR DAMAGES OF ANY KIND, UNDER ANY LEGAL THEORY, ARISING OUT OF OR IN CONNECTION WITH YOUR USE, OR INABILITY TO USE, THE WEBSITE, ANY WEBSITES LINKED TO IT, ANY CONTENT ON THE WEBSITE OR SUCH OTHER WEBSITES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, CONSEQUENTIAL, OR PUNITIVE DAMAGES, INCLUDING BUT NOT LIMITED TO, PERSONAL INJURY, PAIN AND SUFFERING, EMOTIONAL DISTRESS, LOSS OF REVENUE, LOSS OF PROFITS, LOSS OF BUSINESS OR ANTICIPATED SAVINGS, LOSS OF USE, LOSS OF GOODWILL, LOSS OF DATA, AND WHETHER CAUSED BY TORT (INCLUDING NEGLIGENCE), BREACH OF CONTRACT, OR OTHERWISE, EVEN IF FORESEEABLE. #### 10\. Indemnification You agree to defend, indemnify, and hold harmless Conduktor, its affiliates, licensors, and service providers, and its and their respective officers, directors, employees, contractors, agents, licensors, suppliers, successors, and assigns from and against any claims, liabilities, damages, judgments, awards, losses, costs, expenses, or fees (including reasonable attorneys' fees) arising out of or relating to your violation of these Terms or your use of the website, any use of the Website's content, services, and products other than as expressly authorized in these Terms, or your use of any information obtained from the website. #### 11\. Governing Law and Jurisdiction All matters relating to the website and these Terms, and any dispute or claim arising therefrom or related thereto (in each case, including non-contractual disputes or claims), shall be governed by and construed in accordance with the internal laws of the State of New York without giving effect to any choice or conflict of law provision or rule (whether of the State of New York or any other jurisdiction). All Any legal suit, action, or proceeding arising out of, or related to, these Terms or the website shall be instituted exclusively in the federal courts of the United States or the courts of the State of New York although we retain the right to bring any suit, action, or proceeding against you for breach of these Terms in your country of residence or any other relevant country. You waive any and all objections to the exercise of jurisdiction over you by such courts and to venue in such courts. At Company's sole discretion, it may require you to submit any disputes arising from these Terms or use of the website, including disputes arising from or concerning their interpretation, violation, invalidity, non-performance, or termination, to final and binding arbitration under the Rules of Arbitration of the American Arbitration Association applying the laws of the State of New York. #### 12\. Waiver and Severability No waiver by Conduktor of any term or condition set out in these Terms shall be deemed a further or continuing waiver of such term or condition or a waiver of any other term or condition, and any failure of the Conduktor to assert a right or provision under these Terms shall not constitute a waiver of such right or provision. If any provision of these Terms is held by a court or other tribunal of competent jurisdiction to be invalid, illegal, or unenforceable for any reason, such provision shall be eliminated or limited to the minimum extent such that the remaining provisions of the Terms will continue in full force and effect. #### 13\. Entire Agreement The Terms, our Privacy Policy, and the End User License Agreement or Enterprise agreement constitute the sole and entire agreement between you and Conduktor regarding the website and supersede all prior and contemporaneous understandings, agreements, representations, and warranties, both written and oral, regarding the website. #### 14\. Your Comments and Concerns This website is operated by Conduktor Inc. All feedback, comments, requests for technical support, and other communications relating to the website should be directed to: support@conduktor.io This website is operated by Conduktor Inc. If you have any questions, complaints, or concerns about how your information is handled, or would like to exercise any of your rights for yourself or on behalf of someone else, please email us at #### support@conduktor.io Source: https://www.conduktor.io/legal/terms-of-service --- # Legal [#### Privacy Policy Updated August 30th, 2020](#legal/privacy-policy) [#### Terms of Service Updated June 21, 2024](#legal/terms-of-service) [#### Cookie Policy Updated March 11th, 2020](#legal/cookie) [#### Data Processing Addendum Updated November 14th, 2022](#legal/dpa) [#### EULA Updated November 14th, 2022](#legal/eula) [#### EULA Enterprise Updated November 14th, 2022](#legal/eula-enterprise) This website is operated by Conduktor Inc. If you have any questions, complaints, or concerns about how your information is handled, or would like to exercise any of your rights for yourself or on behalf of someone else, please email us at #### support@conduktor.io Source: https://www.conduktor.io/legal --- # Conduktor MCP | AI-Powered Kafka Expertise Conduktor MCP *Turn natural language into Kafka expertise.* Build with confidence. Faster incident resolution, governance insights across every dimension, and full visibility into your Kafka investment. conduktor-mcp — cluster audit > Cross-reference my cluster health, schema coverage, and replication risks. What should I fix this sprint? Cluster Health: 88% — 19 topics, 108 partitions, 5 schemas P0 — Fix now payment-events is the only topic with RF=1. If that broker goes down, you lose payment data. P1 — This sprint payroll-employee producing 40 msg/sec with zero consumers. ~3.4M messages/day going unread. ──────────────────────────────────────── P0 Fix: RF→3 on payment-events P1 Fix: Investigate payroll consumers ## LLM Reasoning Meets Live Kafka Data MCP combines AI reasoning with your live infrastructure and deep Kafka domain knowledge, unlocking capabilities that weren't possible before. **For Developers** Understand how pipelines work, trace dependencies, and build new services with full context across clusters, schemas, and consumer groups. **For Platform Engineers** Reason across every dimension of your Kafka infrastructure at once, without missing context or building ad hoc views. **For Engineering Leaders** Know what your Kafka investment is hiding. Surface cost savings, uncover untapped capacity, and turn insights into opportunities. ## Prompt Engineering for Kafka MCP connects clusters, topics, schemas, consumer groups, and configurations in a single conversation, so you can understand how your applications and infrastructure interact and build with confidence, without losing a second. - **Incident Investigation** — *"Orders-processor has been lagging for 2 hours. Check schema versions, topic configs, and partition assignments. What's the likely root cause?"* - **Configuration Audit** — *"Review my topic configurations against best practices for a 50k msg/sec workload. Flag anything suboptimal."* - **Compliance Evidence** — *"All topics containing PII, their retention, encryption, and consumer access. Format for our quarterly audit."* - **Onboarding and Discovery** — *"How does the order fulfillment pipeline work? Trace the data flow from orders-raw through all downstream consumers."* - **Migration Planning** — *"I need to migrate user-events to a new schema. What consumers depend on it? Safest migration path?"* - **Cost and Opportunity Discovery** — *"Which topics have retention policies that don't match their actual consumption patterns?"* - **MTTR from hours to minutes** — Incident investigation that would take an engineer hours of cross-referencing happens in a single conversation. - **New engineers productive in days** — Onboarding no longer depends on your senior engineers' availability. MCP explains your infrastructure in context. - **Audit prep in minutes, not weeks** — Compliance evidence generated on demand, formatted for your auditors, from live data. - **Every engineer, a Kafka expert** — Deep Kafka knowledge available to every team member, not just your most senior SREs. ## How It Works Architecture ## Your Data Never Leaves Your Network MCP is built into [Conduktor Console](https://www.conduktor.io/console). Connect Claude Code, Cursor, or any MCP-compatible client to your Console URL, and your Kafka data never leaves your infrastructure. - **Built into Console**: No separate deployment. MCP ships with your existing Conduktor Console instance - **No data exfiltration**: Kafka metadata is processed locally, never leaves your network - **Token-based auth**: Console Personal Access Tokens inherit user permissions - **Authenticated on every request**: Token validity checked on every call. Revoke a token and access stops immediately Setup ## Three Steps to Get Started MCP uses Console's existing permissions model. No new access model to configure. A developer's AI assistant sees exactly what that developer already sees in Console. 1. **Generate a token**: Create a Personal Access Token in Console 2. **Configure your AI assistant**: Point Claude (or any MCP-compatible client) at your Console URL 3. **Start asking questions**: Natural language queries against your live Kafka infrastructure [Read the documentation →](https://docs.conduktor.io/guide/conduktor-in-production/automate/mcp) Generate Token Configure Claude Start Querying // claude_desktop_config.json { "mcpServers": { "conduktor-console": { "command": "npx", "args": [ "mcp-remote", "https://console.acme-corp.com/api/mcp", "--header", "Authorization: Bearer ${CONDUKTOR_API_TOKEN}" ], "env": { "CONDUKTOR_API_TOKEN": "your-personal-access-token" } } } } ## Cost Discovery and Optimization MCP explores your entire Kafka footprint to surface hidden costs no single dashboard can show, then synthesizes findings into prioritized optimization plans across every cluster, team, and dimension. - **Discover Hidden Costs** — Dashboards show predefined categories. MCP explores everything and follows the thread. *"Walk through my entire Kafka footprint and flag anything misconfigured, underutilized, or disproportionately expensive. What am I missing?"* - **Compare Across Clusters** — No UI compares clusters side by side. MCP reasons across your full infrastructure at once. *"Compare my production and staging clusters. Are they proportionally sized, or am I overpaying for environments that don't need the same capacity?"* - **Quantify the Impact** — Knowing the problem isn't enough. MCP projects what fixing it is worth. *"If I right-size the top 20 over-provisioned topics and reduce retention where consumers read within minutes, what's the projected savings?"* - **Prioritize by Team** — Turn findings into ownership. MCP attributes waste by team and ranks the highest-impact optimizations first. *"Generate a cost optimization report by team: who owns the most waste, what should they fix first, and what's the estimated impact?"* ## Why Your Security Team Will Approve This MCP doesn't create new access. It makes existing access more useful. Same permissions, same network boundary. - **Your Network, Your Data** — MCP runs inside your Console instance. Kafka data never leaves your infrastructure to reach external services. - **Console Permissions** — Personal Access Tokens inherit Console RBAC. No new access model to configure. AI sees what the user already sees in Console. - **Validated on Every Request** — Token validity checked on every MCP call. Revoke a compromised token and access stops immediately. - **Configurable Access** — Disable MCP entirely with a single feature flag in Console. One config change and the endpoint doesn't exist. ## Frequently Asked Questions **Where does my data go?** The MCP server runs inside your Console instance, in your environment. Kafka metadata flows to the AI assistant on the user's machine. Nothing is sent to external AI training systems. Your data never leaves your network. **Who can use MCP?** Any Console user who can create a Personal Access Token. Tokens inherit the creator's Console permissions, so MCP access is scoped to exactly what that user can already see in Console. **Can we disable MCP entirely?** Yes. Set `enable_mcp: false` and the endpoint doesn't exist. **What if a token is compromised?** Revoke it in Console like any other token. The MCP server validates token validity on every request. ## What's Next MCP is a platform, not a point feature. Today it surfaces intelligence. Tomorrow it acts on it, safely. - **Topic Creation Assistance** — AI-guided topic creation with best-practice defaults based on your workload. - **Schema Evolution Workflows** — Validate schema changes against consumers before deploying, with AI-powered compatibility analysis. - **Automated Remediation** — AI-recommended fixes with approval gates. Resolve common issues without context-switching. ## See What Your Kafka Infrastructure Can Tell You Book a demo and we'll show you MCP against a live environment. Incident investigation, compliance audits, cost discovery, and the questions you haven't thought to ask yet. [Talk to Us](https://www.conduktor.io/contact/demo?src=mcp) [Read Documentation](https://docs.conduktor.io/guide/conduktor-in-production/automate/mcp) Source: https://www.conduktor.io/mcp --- # Amazon MSK + Conduktor *MSK handles infrastructure. Conduktor handles the enterprise layer.* Whether you're using MSK today or migrating to it, Conduktor adds Kafka governance, encryption, and resilience at the protocol level — no application changes required. [Talk to Us](https://www.conduktor.io/contact/demo?src=partners-aws) [Download Solution Brief](https://www.conduktor.io/assets/Conduktor-AWS-MSK-Solution-Brief.pdf) Trusted by ## MSK solves infrastructure. These challenges live above it. ### Data protection gaps MSK encrypts data at rest and in transit, but any service with topic access reads the full payload, including PII and financial records. ### Operational governance gaps MSK provisions brokers, but has no guardrails for topic creation, client configurations, or data quality enforcement. ### Resilience and scale gaps MSK provides single-region high availability, but multi-region failover and multi-tenancy require solutions above the broker. ## What Changes When You Add Conduktor - **For Leadership** — Kafka investment scales across business units without multiplying clusters. Multi-region resilience protects revenue without a second operations team. - **For Platform Teams** — Encryption, access, and data quality enforced once at the gateway instead of rebuilt per project. Client guardrails catch misconfigurations before they cause outages. - **For Development Teams** — Same Kafka clients, same code, and same workflows with no application changes required. Isolated environments on demand without waiting for dedicated infrastructure. ## How Conduktor Gateway Complements MSK ### Field-level data protection Encrypt, tokenize, and crypto shred sensitive fields before data reaches MSK brokers, with per-consumer decryption controls and native AWS KMS integration. ### Real-time data quality Validate every message before it enters MSK, enforcing schema compliance and business rules with the ability to block, route, or flag bad data. ### Client governance Block unsafe client configurations, detect connection storms, and give operators immediate feedback at the protocol level. ### Cross-network connectivity Kafka-protocol-aware routing that rewrites broker metadata, giving clients in any topology access to MSK through a single entry point. ### Disaster recovery & resilience testing Single-command failover that redirects all client traffic in seconds, plus built-in chaos testing to validate resilience without production risk. ### Virtual clusters & infrastructure efficiency Logically isolated virtual clusters with independent namespaces and access controls, plus S3 payload offloading and caching for broker efficiency. ## Operate MSK at Scale with Console [Console](https://www.conduktor.io/console) complements your existing AWS tooling to enable [operational efficiency](https://www.conduktor.io/solutions/use-case/operational-efficiency) for Kafka teams. - [Unified Operations](https://www.conduktor.io/console#simplify-kafka-operations) — Manage topics, schemas, connectors, and consumer groups across all MSK clusters from one interface. Works alongside the AWS Console and CLI. - [Federated Ownership](https://www.conduktor.io/console#scale-ownership-across-teams) — Developers discover, provision, and own resources within automated guardrails, catalogs, and approval workflows. Adds Kafka-level ownership on top of IAM. - [Visibility & Troubleshooting](https://www.conduktor.io/console#accelerate-troubleshooting) — Browse and tail messages in production, monitor consumer lag, and route alerts to Slack, Teams, or PagerDuty. Adds application-level visibility alongside CloudWatch. - [Cost Attribution & Insights](https://www.conduktor.io/console#drive-data-driven-decisions) — Track per-team Kafka usage for chargeback and surface health and risk recommendations across clusters. Complements AWS Cost Explorer with per-team breakdowns. ## Built for the AWS Ecosystem Conduktor integrates natively with AWS services for authentication, encryption, storage, and deployment, so it fits into your existing infrastructure without additional tooling. - **MSK IAM Authentication** — Native support for AWS_MSK_IAM SASL mechanism. Conduktor Gateway inherits IAM roles from ECS, EKS, and EC2. - **AWS KMS** — Field-level encryption keys managed in KMS. IAM policies control which consumers decrypt which fields. CloudTrail logs every operation. - **Amazon S3** — Large payload offloading via the claim check pattern. Payloads never consume broker storage or network bandwidth. - **ECS / EKS Deployment** — Deploy Conduktor Gateway as ECS tasks or EKS pods within your VPC. Native container orchestration. - **AWS Marketplace** — Available on AWS Marketplace for simplified procurement. Streamlines vendor onboarding and purchasing. - **Glue Schema Registry** — Native support for AWS Glue Schema Registry for validation and evolution. ## Results with Amazon MSK + Conduktor Based on results reported by Conduktor customers. - **$500K** — + — First-Year ROI — Consolidation, faster migration, and reduced operational overhead. - **20-40** — % — Infrastructure Cost Reduction — Virtual clusters and consolidation eliminate cluster sprawl. - **Up to 95** — % — Faster Disaster Recovery — Single-command failover vs. manual coordination across teams. ## Read more customer stories - [Smart Farming: 10x Kafka Utilization on AWS](https://www.conduktor.io/customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk) - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Virgin Australia Saves 300 Hours/Month](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) ## Frequently Asked Questions **Does Conduktor work with all MSK cluster types?** Yes. Conduktor connects to MSK Serverless, MSK Provisioned (Standard and Express brokers), and self-managed Kafka on EC2. You can manage all cluster types from a single interface. **How does Conduktor Gateway work at the protocol level?** [Conduktor Gateway](https://www.conduktor.io/gateway) speaks the Kafka protocol natively. Applications connect to Conduktor Gateway instead of directly to MSK brokers using the same client libraries, same code, just a different address. Conduktor Gateway intercepts, transforms, and governs traffic without any application code changes. **How does Conduktor integrate with AWS IAM?** Conduktor Gateway natively supports the AWS_MSK_IAM SASL mechanism and inherits IAM roles from ECS, EKS, and EC2 through the AWS Default Credentials Provider Chain. No separate credential management required. **Can Conduktor help with migration to MSK?** Yes. [Conduktor Gateway](https://www.conduktor.io/gateway) enables zero-downtime migration by sitting between applications and brokers. Switch from one cluster to another with a single command while applications continue running unchanged. This works for migrations from self-managed Kafka or other platforms to MSK. **Can I use AWS Glue Schema Registry with Conduktor?** Yes. Conduktor integrates natively with AWS Glue Schema Registry for schema validation, evolution, and compatibility checks across Avro, JSON, and Protobuf data formats. **Is Conduktor available on AWS Marketplace?** Yes. Conduktor is available on AWS Marketplace for simplified procurement. Marketplace availability streamlines vendor onboarding and lets you leverage your existing AWS private purchasing agreements. **How do I deploy Conduktor on AWS?** Conduktor runs as a Docker container. Deploy [Conduktor Gateway](https://www.conduktor.io/gateway) and [Conduktor Console](https://www.conduktor.io/console) within your VPC on ECS, EKS, Fargate, or EC2 depending on your infrastructure preferences. See [Get Started](https://www.conduktor.io/get-started?src=partners-aws) for setup guides. **What does Conduktor Console add beyond CloudWatch monitoring?** CloudWatch and Prometheus provide infrastructure-level metrics. [Conduktor Console](https://www.conduktor.io/console) adds the operational layer above: which teams own which topics, how resources relate across clusters, self-service provisioning with policy guardrails, approval workflows for cross-team access, and per-team cost attribution. It gives platform teams a single operational layer that scales with the organization. **How is Conduktor different from Confluent?** Conduktor is infrastructure-agnostic. It works across MSK, [Confluent](https://www.conduktor.io/partners/confluent), Redpanda, and open-source Kafka. Rather than replacing your Kafka infrastructure, Conduktor adds an enterprise governance layer on top. If you're evaluating MSK as an alternative to Confluent, Conduktor fills the governance gap that MSK wasn't designed to address. **Can I use Conduktor across multiple Kafka platforms or clouds?** Yes. Conduktor is designed for multi-cloud and multi-Kafka environments. You can manage MSK alongside Confluent Cloud, Redpanda, Aiven, or self-managed Kafka clusters from a single [Conduktor Console](https://www.conduktor.io/console) instance. [Conduktor Gateway](https://www.conduktor.io/gateway) can front any Kafka-compatible cluster, so your governance policies, encryption rules, and data quality controls apply consistently regardless of which Kafka platform or cloud provider sits underneath. Teams get one consistent interface and one set of policies, even if your infrastructure spans AWS, GCP, and on-prem. ## Running Kafka on AWS? Whether you're using MSK today or migrating to it, our team can help you design the right governance architecture for your workloads. [Talk to Us](https://www.conduktor.io/contact/demo?src=partners-aws) [Download Solution Brief](https://www.conduktor.io/assets/Conduktor-AWS-MSK-Solution-Brief.pdf) Source: https://www.conduktor.io/partners/aws --- # Cloudera + Conduktor Unlock the full value of your streaming data for AI, analytics, and partner collaboration. [Let us show you](https://www.conduktor.io/contact?src=partners-cloudera) Trusted by ## Managing complex Kafka environments creates chaos without the right controls. ### High operational complexity Managing complex Kafka environments requires manual effort and creates bottlenecks. ### Missed opportunities The inability to securely share real-time data locks out partner collaboration. ### Poor data quality Low-quality, inconsistent data introduces inaccuracies to analytics and AI models. ## Why Cloudera + Conduktor - **Ship faster** — Self-service Kafka access removes bottlenecks so teams can deliver new use cases in days - **Monetize data** — Share streaming data with partners securely, without duplication or infrastructure exposure - **Trust the data** — Validate and govern data at the source so downstream apps and AI models stay reliable - **Secure Data Sharing** — Share real-time data with partners without exposing infrastructure or duplicating streams. - **Cost Attribution** — Know which teams and applications drive Kafka costs for accurate chargeback. - **Business Continuity** — Automatic failover and DR capabilities keep streaming apps running. - **AI-Ready Pipelines** — Clean, validated data at the source protects AI models from garbage in. ## Streaming-enabled Lakehouse Architecture ![Streaming-enabled Lakehouse Architecture with Cloudera and Conduktor Gateway](https://www.conduktor.io/assets/images/cloudera-architecture.png) ## Results with Cloudera + Conduktor Teams use Cloudera and Conduktor to simplify Kafka operations and unlock streaming data for AI, analytics, and new revenue. - **15** — x — Faster Time to Market — New teams and use cases go live in days instead of months. - **10** — x — Platform Adoption — Scale to thousands of users while maintaining governance and control. - **Near 0** — Downtime — Automatic failover keeps real-time applications running during DR events. ## Read more customer stories - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [FlixBus: Data Mesh for 50+ Teams](https://www.conduktor.io/customer-stories/flix) - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) ## Frequently Asked Questions **How does Conduktor integrate with Cloudera?** Conduktor connects to Kafka clusters running on Cloudera Data Platform. It adds a governance and self-service layer on top of your existing infrastructure without requiring changes to your Kafka deployment. **Can Conduktor help with streaming lakehouse architectures?** Yes. Conduktor Gateway, a Kafka proxy, validates and transforms data in-flight, ensuring clean data flows into your lakehouse. This protects downstream analytics and AI workloads from data quality issues. **How does Conduktor enable secure data sharing with partners?** Conduktor Gateway lets you expose specific topics to external partners without revealing your internal infrastructure. You control exactly what data partners can access, with full audit trails. **Does Conduktor support disaster recovery for Kafka?** Yes. Conduktor provides automatic failover capabilities that keep streaming applications running during DR events, with near-zero downtime for real-time workloads. **How does Conduktor help with Kafka cost attribution?** Conduktor tracks which teams and applications produce and consume data. This visibility enables accurate chargeback and helps identify optimization opportunities. ## Building on Cloudera? Whether you're scaling your streaming lakehouse, enabling secure data sharing, or preparing your pipelines for AI, our team can help you get there faster. [Talk to an expert](https://www.conduktor.io/contact?src=partners-cloudera) Source: https://www.conduktor.io/partners/cloudera --- # Confluent + Conduktor Make your Confluent investment work harder with enterprise governance, data quality enforcement, and wire-level security. [Book a demo](https://www.conduktor.io/contact/demo?src=partners-confluent) Trusted by ## How Conduktor extends Confluent ### Wire-level enforcement Policies enforced at the protocol level-clients can't bypass them. ### Encryption without a schema Encrypt any payload at the wire-plain JSON included. ### Data quality enforcement Catch bad data before it breaks downstream systems. ### Self-service with guardrails Onboard teams without bottlenecks-or bad configurations. ### RBAC you shape to your org Compose permission sets instead of picking from a fixed list of roles. ### Client configuration enforcement Stop bad client configurations at the wire. ### Developer-first experience A UI developers actually want to use. ### Multi-environment governance One interface for Cloud, Platform, and hybrid. ### External data sharing Share data with partners without replication. ## Why Confluent + Conduktor - **Wire-level security** — Encryption, masking, and policies enforced at the protocol level. Clients can't bypass-even with standard serializers. - **Encryption without schemas** — Full payload or field-level, on plain JSON as much as on Avro. No schema subject or data contract to register first. - **Roles you define** — Compose permission sets per user or group, instead of picking from a fixed catalog of predefined roles. - **No artificial limits** — No caps on API keys, role bindings, or service accounts. Your microservices architecture won't hit ceilings. - **GitOps your governance** — Terraform provider for RBAC, policies, data quality, encryption, self-service-not just Kafka resources. - **Developer experience** — A UI developers actually want to use-not an ops console with a steep learning curve. - **Wire-Level Enforcement** — Policies enforced at the protocol level. Clients can't bypass-unlike client-side rules. - **Encryption, Schema Optional** — Full payload or field-level, on plain JSON or Avro. 5 KMS backends, zero code changes, selective decryption by role. - **Data Quality** — Validate payloads with or without schemas. CEL rules, dead-letter routing, alerts. - **Client Enforcement** — Require compression, acks, idempotence at the wire. No code changes. - **Self-Service Catalog** — Request/approve workflows with CEL-based resource policies and guardrails. - **Developer-First UI** — Explore messages visually, self-serve with guardrails, modern interface developers love. - **GitOps Everything** — Terraform provider for clusters, RBAC, policies, data quality, self-service-not just Kafka resources. - **Virtual Clusters** — Logical multi-tenancy on shared infrastructure. No cluster proliferation. - **Composable RBAC** — No fixed role catalog. Pick the operations, scope them by prefix or wildcard, assign to users or IdP groups. ## No artificial limits Confluent Cloud enforces quota limits that require support escalation at scale. Conduktor removes the ceiling. [Confluent Cloud limits](https://docs.confluent.io/cloud/current/quotas/service-quotas.html) API keys to 1,000 per org, role bindings to 500-25,000 per cluster, and service accounts to 1,000 per org. Conduktor has no such limits. | Resource | Confluent Cloud | Conduktor | |----------|-----------------|-----------| | **API keys** | 1,000 per org, 50–2,000 per cluster | Unlimited | | **Role bindings** | 500 per cluster (25K on Dedicated) | Unlimited | | **Service accounts** | 1,000 per org | Unlimited | > **Example:** A single Kafka Streams application creates ~6 role bindings. At 500 per cluster (Standard/Enterprise), you hit the ceiling at ~80 applications. Even Dedicated clusters cap at ~4,000. A microservices architecture with 1,000+ services? You'll exhaust API keys and service accounts. Conduktor provides virtual clusters with unlimited service accounts and RBAC bindings. ## Results with Confluent + Conduktor - [$380K](https://www.conduktor.io/year) — Saved — A major payroll provider avoided infrastructure costs by encrypting at the wire, not duplicating data. - **3** — days — To production — New teams go from Kafka request to producing events-down from weeks. ## Read more customer stories - [Bitvavo: DORA Compliance for Crypto Exchange](https://www.conduktor.io/customer-stories/how-bitvavo-secures-real-time-crypto-data-with-conduktor) - [FlixBus: Scaling Event-Driven Architecture](https://www.conduktor.io/customer-stories/how-flix-uses-conduktor-to-scale-their-event-driven-architecture) - [Swiss Post: Governing Kafka at Scale](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) ## Frequently Asked Questions **Does Conduktor work with both Confluent Cloud and Confluent Platform?** Yes. Conduktor connects to Confluent Cloud (Dedicated, Standard, Basic) and self-managed Confluent Platform clusters. Manage all cluster types from a single interface. **What's the difference between wire-level and client-side enforcement?** Confluent Stream Governance data quality rules run in client serializers-if a producer uses a standard Kafka serializer, those rules don't apply. Stream processors like ksqlDB and Flink bypass them entirely. Conduktor Gateway intercepts at the wire: every message passes through, no bypass possible, works with any client. **Does Conduktor encryption require Schema Registry?** No. Confluent CSFLE does: a registered schema subject, a tag on the field, an encryption rule in the data contract, and only string and bytes fields in Avro, JSON Schema or Protobuf. Conduktor Gateway encrypts at the wire, so plain JSON with no schema anywhere behaves like Avro. Fields are picked from a list in the interceptor config, or from schema tags if you have them. **Can I encrypt a whole message, not just fields?** Yes. Full payload encryption covers the record key, value or headers-what you want when the whole message is sensitive or the payload shape varies. You can layer both: field-level first, full payload as a fallback for anything left. Confluent CSFLE is field-scoped by design. **How does Conduktor RBAC compare to Confluent's roles?** Confluent RBAC starts from predefined roles you can't create or modify, with bindings that are delete-and-recreate. Conduktor Console has no catalog: you pick the operations per resource (topics, consumer groups, subjects, connectors, clusters) and per Console feature, scoped by name, prefix or wildcard, on one cluster or all. Console RBAC covers actions through Conduktor; Self-service handles the broker side via Kafka ACLs or Confluent role bindings. **How does selective decryption work?** Conduktor Gateway encrypts fields at the wire, then selectively decrypts based on consumer identity. AI team with decrypt permission sees salary: $95,000. Support team without permission sees: XXXXX. Same topic, same data, different views. Zero producer code changes. **What can I manage with Terraform?** Conduktor's Terraform provider covers more than just Kafka resources: cluster connections, RBAC policies, topic policies, data quality rules, self-service catalog, applications, and Conduktor Gateway interceptors. GitOps your entire governance layer, not just topics. **How does data quality validation work without Schema Registry?** Conduktor Gateway validates payloads using CEL expressions and field-level rules-independent of Schema Registry. You can enforce formats, ranges, and required fields on JSON payloads without schemas. This enables gradual migration from schema-less to schema-ful architectures. **What client configurations can Conduktor enforce?** Compression type (require GZIP/LZ4/ZSTD), acks mode (enforce acks=-1), idempotence, offset commit rate limits, connection rate limits, and client ID naming conventions. All enforced at the wire level-no application code changes. **What are the limits on Conduktor vs Confluent Cloud?** Confluent Cloud limits: 1,000 API keys per org, 500 role bindings per cluster (25K on Dedicated), 1,000 service accounts per org. Conduktor has no such limits-virtual clusters support unlimited service accounts and RBAC bindings. **What are virtual clusters?** Logical isolation on shared infrastructure. Each team gets their own namespace with separate service accounts, topic prefixes, and rate limits-without spinning up separate Kafka clusters. Reduces cost while maintaining isolation. **How is Console different from Control Center?** Control Center is ops-centric with a steep learning curve. Console is developer-first: explore messages visually, self-serve topic creation with guardrails, schema browser with compatibility tracking. Developers actually want to use it. **Can I use Confluent Schema Registry with Conduktor?** Yes. Conduktor integrates natively with Confluent Schema Registry for schema validation, evolution tracking, and compatibility enforcement. Data quality rules can work alongside or independent of Schema Registry. **Does Conduktor require changes to my Confluent setup?** No. Conduktor connects via standard Kafka protocols. No configuration changes to your Confluent clusters. **Is Conduktor a replacement for Confluent?** No. Conduktor is complementary. Confluent handles your Kafka infrastructure; Conduktor adds enterprise governance, data quality, and security capabilities on top. **How do I encrypt Kafka messages without code changes?** Conduktor Gateway encrypts fields at the wire level using your KMS (AWS, Azure, GCP, HashiCorp Vault, Fortanix). Producers send plaintext; Gateway encrypts before the message hits the broker. Consumers with permission see decrypted data; others see masked values. Zero application code changes required. ## Running Kafka on Confluent? Whether you're using Confluent Cloud, Confluent Platform, or a hybrid setup, our team can help you design the right governance architecture for your workloads. See our [architecture overview](https://www.conduktor.io/architecture) for deployment options. [Book a Confluent + Conduktor demo](https://www.conduktor.io/contact/demo?src=partners-confluent) Source: https://www.conduktor.io/partners/confluent --- # Kafka to Snowflake with Conduktor Clean data, consistent schemas, and compliant routing before anything reaches Snowflake. [Let us show you](https://www.conduktor.io/contact/demo?src=partners-snowflake) Trusted by ## Kafka to Snowflake pipelines fail in hidden and inconsistent ways. ### Ingestion chaos Multiple ingestion tools fail differently. When a table stops updating, engineers spend 2–4 hours isolating which layer failed. ### Schema drift Schema changes break pipelines unpredictably. One schema change, four different outcomes. ### Multi-region risk Multi-region deployments increase risk and cost. Tracking 400+ topics across 6 regions manually doesn't scale. ### Ownership gaps No clear owner between Kafka and Snowflake. Kafka is Platform, Snowflake is Data: who owns the gap? ## Why Conduktor + Snowflake - **One control point** — Conduktor Gateway, a Kafka proxy, sits between producers and Kafka. No code changes required. - **Consistent behavior** — Every message passes through the same validation, transformation, and routing logic. - **Tool-agnostic** — Fivetran, Airbyte, Kafka Connect, Snowpipe: all inherit the same governance rules. - **Data Quality at Ingestion** — Validate messages against Schema Registry at produce time. Bad data gets rejected before it reaches Kafka. - **Schema Normalization** — Enforce canonical schemas, rename fields, normalize values in-flight. Snowflake tables stay stable as producers evolve. - **Regional Routing** — Route data to the correct region automatically. Invalid routes get rejected with full audit trails. - **Pipeline Visibility** — See producer activity, validation rates, and connector state end to end. Find failures in seconds, not hours. - **Cost Attribution** — Tag every message with application, team, and environment. Know exactly who drives costs and duplicate traffic. ## Outcomes Snowflake handles analytics and scale. Conduktor governs everything upstream. - **Faster incident resolution** — Debug in minutes, not hours. Failures surface with clear producer and policy context. - **Consistent data quality** — Same schema change, same result across every connector. No more silent data loss. - **Lower ingestion costs** — Identify waste early. Remove noisy or misrouted traffic before Snowflake sees it. - **Automated compliance** — Routing logs provide concrete evidence for GDPR Article 44 and internal audits. - **Fewer escalations** — Shared visibility ends ownership debates and shortens handoffs between teams. ## Read more customer stories - [Bitvavo: DORA & MiCA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [Virgin Australia Saves 300 Hours/Month](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) ## Frequently Asked Questions **Does Conduktor work with Kafka Connect for Snowflake?** Yes. Conduktor Gateway sits upstream of Kafka Connect and validates data before it reaches Kafka. This means your Snowflake Sink Connector receives clean, schema-compliant data. **How does Conduktor handle schema changes in Kafka to Snowflake pipelines?** Conduktor validates messages against Schema Registry at produce time. When a producer sends incompatible data, Conduktor rejects it immediately instead of letting it propagate to Snowflake. **Can Conduktor help with GDPR compliance for Snowflake data?** Yes. Conduktor Gateway enforces regional routing rules, ensuring EU data stays in EU regions. Every routing decision is logged with timestamps for audit evidence. **Does Conduktor work with Fivetran and Airbyte?** Yes. Conduktor is tool-agnostic. Whether you use Kafka Connect, Fivetran, Airbyte, or Snowpipe, all data passes through the same validation and transformation rules. **How does Conduktor reduce Snowflake ingestion costs?** Conduktor identifies and blocks duplicate, malformed, or misrouted traffic before it reaches Kafka. This reduces the volume of data that flows into Snowflake, lowering compute and storage costs. ## Streaming data to Snowflake? Whether you're troubleshooting ingestion failures, enforcing schema governance, or optimizing multi-region pipelines, our team can help you build reliable Kafka-to-Snowflake workflows. [Talk to an expert](https://www.conduktor.io/contact?src=partners-snowflake) Source: https://www.conduktor.io/partners/snowflake --- # Partner with Conduktor Join a global community of partners to build data streaming solutions and skills, market your company, and co-sell with Conduktor. [Become a Partner](https://www.conduktor.io/contact?src=partners) --- ## Partner Benefits Grow together with support from our Business Development team. - **Product training** — Deep-dive sessions on Conduktor platform capabilities - **Customer enablement materials** — Resources from Kafka experts to support your customers - **Co-marketing & branding** — Joint initiatives to amplify your reach - **Early access to beta features** — Get ahead with new capabilities before general availability - **Partner directory listing** — Visibility in our partner ecosystem --- ## Technology Partners - **[Amazon Web Services](https://www.conduktor.io/partners/aws)** — Simplify Kafka at scale with Amazon MSK + Conduktor - **[Cloudera](https://www.conduktor.io/partners/cloudera)** — Unlock the potential of your Kafka data with Cloudera + Conduktor - **[Snowflake](https://www.conduktor.io/partners/snowflake)** — Simplify Kafka to Snowflake pipelines with Conduktor --- ## Integrations Conduktor integrates with security, ops, communication, and monitoring tools across the ecosystem—including Splunk, Datadog, Kubernetes, and more. --- ## Become a Partner Interested in becoming a Conduktor partner? Tell us about yourself to start your journey. Questions? Reach out to us at **partnerships@conduktor.io** [Contact Us](https://www.conduktor.io/contact?src=partners) Source: https://www.conduktor.io/partners --- # Conduktor ROI Calculator | What Is Conduktor Worth to Your Enterprise Conduktor ROI Calculator Tell us about your Kafka platform and see how we can help, quantified in hours and dollars per year. Powered by data from 6,500+ customer engagements IndustryiYour industry tailors the estimate to your sector. It shapes which use cases are most relevant to you and the risk and compliance assumptions behind the numbers. Developers working with KafkaiThis is the main driver of scale. We multiply time savings across your developers, and ask how many projects they work across below, so the estimate reflects your organization's size. 50+ 500 1000 1500 2000 2500+ Drag or tap Average developers per team≈ – teams building on Kafka Choose your industry and set your team size to see your estimate. Where the value comes from Each square is 1% of your estimated annual value. Time back …engineering hours back per year Time your teams lose today to debugging, waiting on requests, and manual operations. The productivity and efficiency value is these hours priced at a benchmark engineering cost.iHours are valued at a staff cost of about $120K/yr (~$58/hr) for developers and $140K/yr (~$67/hr) for platform engineers, including salary, benefits, taxes, and overhead. [See the use cases behind your estimate](#how-we-can-help) How we can help Ranked by estimated annual value for your platform. Expand each to see what changes and why. Loading… See the other ways we can help Estimated annual value… [Talk to us about your estimate](https://www.conduktor.io/contact/demo?src=pricing-roi) Share estimate Source: https://www.conduktor.io/pricing/roi --- # Conduktor Pricing Model | Console and Gateway Plans Conduktor Pricing Start free and scale when you're ready. Pay only for the Console seats and Gateway clusters you use. See what Conduktor is worth to your enterprise Estimate your annual value in under a minute, from benchmarks across 6,500+ customer engagements. [Calculate your ROI](https://www.conduktor.io/pricing/roi) Choose your productConsoleGatewaySchema Registry Proxy Console The modern self-hosted Kafka UI and API. Smarter AI because you hold the context, safer AI because you hold the permissions. Community Built for developers and teams to move faster with Kafka. Troubleshoot issues, explore data, and monitor streams through one interface. Free [Install Free Edition](https://www.conduktor.io/get-started?src=pricing) Up to 50 users and 3 clusters Insights (topic health, cost, VIP topics) Full Kafka operations Message reprocessing Advanced data explorer API, CLI, and Terraform provider Agentic workflows via MCP and Skills Real-time metrics and consumer lag SSO/LDAP authentication Email support, plus docs and Slack Team Editionbuy online Unlock team governance, Insights, and automation: federated ownership with approval workflows and cost chargeback. Purchased online. Monthly Annual $125$1,200/seat/month [Buy Now](https://buy.stripe.com/9B6eVe8Fn1dgczd0Ef1ZS00) Everything in Community Federated ownership with approval workflows Unlimited audit logs Unlimited clusters Topic policies enforcement Advanced Insights chargeback Unlimited custom alerts Group-level RBAC DLQ reprocessing and message replay Applications catalog Unlimited data masking Connector auto-restart Standard Support: priority response, business hours Already an online customer? [Manage subscription →](https://billing.stripe.com/p/login/9B6eVe8Fn1dgczd0Ef1ZS00) Licensing or pricing questions? [Contact us →](https://www.conduktor.io/contact?src=pricing) Enterprise For teams that need volume pricing, dedicated support, and a custom contract. [Contact Us](https://www.conduktor.io/contact?src=pricing) Everything in Team Edition Volume pricing Dedicated customer success Custom contract and SLA Silver Support included, Gold (24/7) available Gateway license bundling Gateway Kafka proxy for routing, resilience, multi-tenancy, security, and data sharing. Community A Kafka proxy that connects clients to clusters sitting in other VPCs, clouds, or private networks, with no changes to your brokers or credentials. A license-restricted mode of Conduktor Gateway, scoped to network access only. Free [Install Free Edition](https://www.conduktor.io/gateway/community-edition#request-license) Reach Kafka across networks without adding peerings Reach private clusters from outside (GCP Managed Kafka, Confluent Cloud) Single egress point for many internal clients Authentication and Authorisation transparently delegated to your Kafka provider Works with any Kafka 2.7+ Deploys via Docker or Kubernetes Enterprise Proxy layer for routing, resilience, multi-tenancy, security, and data sharing. Per-cluster pricing (3 cluster minimum). Per-cluster (3 cluster min.) [Contact Us](https://www.conduktor.io/contact?src=pricing) Everything in Community Kafka Proxy and Schema Registry Proxy SQL Topics, Caching, and Large Message Handling Centralized Authentication and Audit Trails Virtual Clusters and Multi-tenancy Topic Concentration and Aliases Partner Zones and Secure Data Sharing Dedicated Partner Credentials Internal Naming Protection ADD-ONS + DR & Failover Resilience and continuity across Kafka clusters. Requires Enterprise Gateway. Cluster Switching and Transparent Failover Zero-Downtime Migrations Chaos Engineering (Latency Injection, Broker Failures, Network Partitions) Protect Guardrails, data security, and quality enforcement. Requires Enterprise Gateway. Configuration Guardrails and Traffic Control Producer and Consumer Policies Data Quality Detection and Enforcement Field-Level and Full Payload Encryption Multi-Cloud KMS Integration (Vault, AWS, Azure, GCP) Tokenization and Crypto-Shredding (GDPR) Cryptographic Signing Schema Registry Proxy Add authentication, authorization, and metrics collection to the schema registry you already run. No client changes. Enterprise A proxy that sits in front of any Confluent-API-compatible schema registry. Per-subject permissions, using the identities you already run. Per-cluster pricing. Per-cluster [Contact Us](https://www.conduktor.io/contact?src=pricing) Per-subject read and write permissions OAuth2/OIDC authentication (Keycloak, Auth0, Okta, Entra ID) Kafka-native permissions from Kafka ACLs or Console ownership OpenTelemetry traces and Prometheus metrics Works with any Confluent-API-compatible schema registry Single container, runs on-premises, in the cloud, or air-gapped ## Frequently Asked Questions **How is Conduktor priced?** Console is available in three tiers: Community (free), Team Edition ($1,200/seat/year or $125/seat/month), and Enterprise (contact sales). Conduktor Gateway uses per-cluster pricing with a 3-cluster minimum. Disaster Recovery & Failover and Protect are optional add-ons to Enterprise Gateway. **What's the difference between Community, Team Edition, and Enterprise?** Community is the free edition for developers and small teams, with core Kafka operations for up to 50 users and 3 clusters. Team Edition adds team governance, advanced Insights, federated ownership, and unlimited clusters. Buy online, license delivered by email. Enterprise adds volume pricing, dedicated customer success, custom contracts, SLA, and priority support. **Are any Console tiers cloud or SaaS?** No. Every Console tier (Community, Team Edition, and Enterprise) is self-hosted. You deploy Console on your own infrastructure: Kubernetes, Docker, or cloud VMs. The tiers differ in what features are unlocked and how you purchase. Community is free, Team Edition is a self-serve online purchase, Enterprise is a custom contract with dedicated support. **What's included with Team Edition?** A self-hosted Console license covering your seat count, with access to all paid features including Insights, ownership, governance, and automation. Includes Standard Support: email support for license, billing, and product configuration questions, plus public docs, release notes, and community Slack. Production uptime SLA and dedicated support come with Enterprise. **What's the difference between Console and Conduktor Gateway?** Console is our self-hosted platform for Kafka operations, monitoring, and governance: you point it at your clusters to observe and manage them. Conduktor Gateway is a proxy that sits in front of your brokers to enforce things in the data path: traffic control, encryption, data quality, multi-tenancy, and secure data sharing. They are independent products, and many teams run both. Gateway Community Edition, the free tier, is a license-restricted mode that connects clients to clusters across networks only, with no governance or encryption. [See Gateway Community Edition →](https://www.conduktor.io/gateway/community-edition) **Do I need Conduktor Gateway if I have Console?** No, Console and Conduktor Gateway are independent products. Many teams start with Console for operations and monitoring, then add Conduktor Gateway when they need enforcement capabilities like encryption, data quality rules, or partner data sharing. **What is the Schema Registry Proxy?** A proxy that sits in front of the schema registry you already run and adds access control, metrics, and policy enforcement, with no client changes. It checks per-subject read and write permissions on every request, using identities from your OAuth2/OIDC provider, Kafka ACLs, or Console ownership. Priced per cluster. [Learn more →](https://www.conduktor.io/schema-registry-proxy) **What Kafka providers and deployment models are supported?** Both Console and Conduktor Gateway work with any Kafka provider: Confluent, AWS MSK, Redpanda, and more. Both are deployed on-premise or on Kubernetes; there is no cloud-hosted dependency. Metrics can be exported via Prometheus, and all features are accessible through the OpenAPI and CLI. **Can I operate Kafka with AI and agents?** Yes. Console embeds a read-only MCP server and the CLI handles changes, so an LLM can troubleshoot incidents, enrich metadata, generate topic descriptions from schemas, clean up environments, and surface cost, using the ownership, schemas, lineage, and monitoring context Conduktor already holds. [See the MCP server →](https://www.conduktor.io/mcp) **How do you keep AI from breaking production?** Every AI, MCP, or CLI action inherits the exact RBAC permissions of the human using it. You decide what's read-only versus read-write, per cluster and per team, so AI works inside the governance you already trust. **What type of support is provided?** Three tiers. **Community** users get public docs, the community Slack, and best-effort email support. **Team Edition** customers get Standard Support: email support (Mon–Fri, business hours) for license, billing, and configuration questions. **Enterprise** customers get Silver Support by default: priority response during EMEA/US business hours and a dedicated support team. Gold Support (24/7 coverage for critical incidents) is available as an upgrade. ## Still have questions? Have a complex setup or additional questions around pricing? Contact our sales team for help and to get more information on Conduktor pricing. [Contact Us](https://www.conduktor.io/contact?src=pricing) [Not sure what you need? Get your score →](https://www.conduktor.io/kafka-assessment) Source: https://www.conduktor.io/pricing --- # The Data Streaming Platform for Kafka Operations Conduktor is the operational layer for Apache Kafka — the control plane your brokers don't come with, on any distribution you already run. Management, governance, and security for organizations running Kafka seriously. [Talk to Us](https://www.conduktor.io/contact/demo?src=product-data-streaming-platform) [Get Started Free](https://www.conduktor.io/get-started?src=product-data-streaming-platform) ## What Is a Data Streaming Platform? A data streaming platform is the infrastructure and tooling that organizations use to build, operate, and govern real-time data pipelines. Apache Kafka is the most widely adopted streaming backbone. But Kafka itself doesn't provide the operational layer: multi-team access controls, field-level security, governance automation, or the developer experience that modern platform engineering requires. Conduktor is the operational platform that sits on top of Kafka to provide these capabilities, from multi-team governance to [real-time data sharing with external partners](https://www.conduktor.io/solutions/use-case/data-sharing). - **Conduktor Console** — The management UI and API for your Kafka estate. Topic management, consumer group monitoring, schema registry, connector management, and developer self-service portal, across all clusters. - **Conduktor Gateway** — A transparent Kafka proxy that enforces field-level encryption, RBAC, data masking, schema validation, and routing policies. Applications connect to Gateway; Gateway enforces your policies. - **conduktor-ctl CLI** — Command-line interface for Kafka-as-Code. Apply YAML manifests for topics, ACLs, schemas, and policies. Integrate with CI/CD pipelines for automated Kafka governance. ## The Full Kafka Operations Stack - **Data Ingestion** — Manage Kafka Connect connectors: deploy, monitor, restart from the UI. Works with any connector in the Kafka Connect ecosystem. Self-service deployment with platform team guardrails. - **Data Visibility** — Browse and inspect messages in any topic. Search by field, filter by time range, deserialize any format. Full topic lifecycle visibility without CLI tools. - **Data Security** — Field-level encryption, RBAC, mTLS, data masking, and audit logging. Enforced at the Conduktor Gateway layer. Every producer and consumer inherits your security policies automatically. - **Data Governance** — Naming conventions, retention policies, schema standards, ownership model. Automated enforcement, not wiki documentation that developers ignore. - **Data Operations** — Consumer lag alerts, broker health monitoring, partition management, offset reset. The operational tools your team needs to run Kafka reliably at scale. - **Data Sharing** — Secure data sharing between teams and with external partners. Virtual clusters with isolated namespaces and controlled access. Partner zones for external data consumers. ## Data Streaming Platform Comparisons - **Apache Kafka** — The distributed streaming backbone. Conduktor extends Kafka with management, security, and governance tooling. It's not an alternative to Kafka. - **Confluent Platform** — Confluent's commercial Kafka distribution with its own management tooling. Conduktor is cloud and distribution-agnostic (Confluent, MSK, self-managed Kafka, Redpanda). - **AWS MSK** — Amazon's managed Kafka service. Conduktor works as the management and governance layer on top of MSK, adding capabilities MSK doesn't provide natively. - **Azure Event Hubs** — Kafka-compatible event streaming on Azure. Conduktor connects to Event Hubs via the Kafka protocol for management and governance. Evaluating table-sharing protocols like Delta Sharing for partner data exchange? See how they compare with streaming in [Delta Sharing alternatives for real-time data sharing](https://www.conduktor.io/compare/delta-sharing-alternatives). ## Read more customer stories - [Smart Farming: 10x Kafka Utilization on AWS](https://www.conduktor.io/customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk) - [FlixBus: Data Mesh for 50+ Teams](https://www.conduktor.io/customer-stories/flix) - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) ## Frequently Asked Questions **What makes Conduktor a "data streaming platform" vs. just a Kafka UI?** A UI is read-only visibility. A data streaming platform is the full operational layer: management, governance, security enforcement, developer self-service, GitOps integration, and programmatic API. Conduktor provides all of these. **Does Conduktor work with non-Kafka streaming systems?** Conduktor is purpose-built for Apache Kafka and Kafka-compatible brokers (Confluent, MSK, Redpanda, Aiven). It doesn't currently support non-Kafka systems like Pulsar or Kinesis. **Can Conduktor replace a cloud provider's Kafka service?** No. Conduktor runs on top of Kafka, not instead of it. It provides the management and governance layer regardless of which Kafka distribution or cloud service you use. **How does Conduktor handle multi-cloud Kafka deployments?** Console connects to Kafka clusters across any cloud provider. Platform teams manage all clusters from one interface. Conduktor Gateway can route traffic between clusters for failover or migration scenarios. ## See the Full Conduktor Platform Kafka is your data backbone. Conduktor is your operations platform. 30-minute demo covering Console, Conduktor Gateway, and CLI — and how they work together for platform teams at scale. [Talk to Us](https://www.conduktor.io/contact/demo?src=product-data-streaming-platform) [Get Started Free →](https://www.conduktor.io/get-started?src=product-data-streaming-platform) Source: https://www.conduktor.io/product/data-streaming-platform --- # Kafka Monitoring, Ready in Minutes Consumer lag and broker health visible out of the box. Connect your cluster and you have real metrics within minutes — no custom exporters, no dashboard templates. Works alongside Prometheus and Grafana if you already have them. [Talk to Us](https://www.conduktor.io/contact/demo?src=product-kafka-monitoring) [Get Started Free](https://www.conduktor.io/get-started?src=product-kafka-monitoring) ## What You Actually Need to Monitor in Kafka "Is the cluster up?" is the easy question. The hard ones are: which consumer group is falling behind, which broker just lost a replica, which connector task silently stopped processing. Most Kafka monitoring setups answer the first question and leave you guessing on the rest. - **Consumer Lag Monitoring** — See consumer group lag across all partitions in real time. Identify which groups are behind, by how much, and since when. Set alerts on lag thresholds before backlogs become incidents. - **Broker & Cluster Health** — Monitor broker metrics: under-replicated partitions, offline partition count, under-min-ISR partitions, disk usage, and message throughput. Detect broker issues before they cascade. - **Topic Throughput** — Bytes in/out, message rate, and total message count per topic. Identify hot partitions and throughput bottlenecks. Compare metrics across time windows. - **Connector Monitoring** — Track connector and task status and failed task counts. Know when a connector task fails without grepping through logs. Alert on task failures before they cause data gaps. - **Schema Registry Health** — Monitor schema registry availability and schema registration rates. Detect schema compatibility errors before they cause producer failures. - **Multi-Cluster Dashboard** — Aggregate metrics across all Kafka clusters (on-prem and cloud) in one view. Platform teams see the full estate; developers see their clusters. ## Alerting Before Problems Become Incidents - **Consumer Lag Alerts** — Set thresholds per consumer group or topic. Get notified via Slack, Teams, email, or webhook when lag exceeds your SLA. Teams manage their own thresholds. - **Under-Replicated Partition Alerts** — Immediate notification when partitions lose replicas. The first sign of broker trouble. Catch it before data is at risk or producers start failing. - **Offline Partition Alerts** — Alert when any partition loses its leader. Offline partitions block producers and consumers immediately. Catch broker failures before they cascade to applications. - **Connector Failure Alerts** — Alert when connector tasks fail. Each failed task stops data flowing through that connector. Set thresholds per connector and get notified before the gap becomes a problem. ## Enterprise Alerting for Multi-Team Kafka Conduktor supports decentralized alert ownership. Each team manages alerts for their own topics and consumer groups without platform team intervention. Platform teams set baseline cluster-level alerts; product teams own application-level alerts. ## Frequently Asked Questions **What metrics does Conduktor expose?** Consumer group lag (per partition and per group), broker metrics (under-replicated partitions, offline partitions, under-min-ISR partitions, disk usage, message throughput), topic metrics (bytes in/out, message rate, message count), and connector task status and failure counts. **Does Conduktor replace Prometheus + Grafana?** Conduktor provides out-of-the-box Kafka monitoring without Prometheus setup. If you already have Prometheus/Grafana, Conduktor complements it. It adds the operational context (ownership, self-service, alerts) that raw metrics dashboards lack. **How do we get notified about consumer lag?** Set a lag threshold on any consumer group. When lag exceeds the threshold for a configured time window, Conduktor sends alerts to Slack, Teams, email, or webhook. Teams manage their own alert thresholds. **Can we monitor multiple Kafka clusters in one dashboard?** Yes. Conduktor Console connects to multiple clusters simultaneously. The unified dashboard shows cross-cluster consumer lag, broker health, and topic metrics across every environment. **How is this different from Confluent Control Center?** Confluent Control Center is designed for Confluent Platform and Confluent Cloud. Conduktor works with any Kafka cluster (MSK, self-managed, Redpanda, Aiven) and adds multi-team features (ownership, access controls, self-service) that Control Center doesn't provide. ## Read more customer stories - [Virgin Australia Saves 300 Hours/Month](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) - [Swiss Post: 5x Kafka Growth with 800 Users](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Smart Farming: 10x Kafka Utilization on AWS](https://www.conduktor.io/customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk) ## Set Up Kafka Monitoring in Minutes Stop finding out about Kafka problems from angry Slack messages. Connect your cluster and see consumer lag, broker metrics, and alerts without writing a single Prometheus query. [Get Started Free](https://www.conduktor.io/get-started?src=product-kafka-monitoring) [Talk to Us →](https://www.conduktor.io/contact/demo?src=product-kafka-monitoring) Source: https://www.conduktor.io/product/kafka-monitoring --- # Application Catalog and Ownership The application catalog shows every application registered with your Kafka cluster, split by environment so permissions and resource policies map cleanly to your teams. Drill into any service to see exactly what it owns, topics, consumer groups, and connectors, with resource policies enforcing naming conventions, partition counts, and retention. Define the same policies as code and apply them with the Conduktor CLI. ## What You'll Learn - See every application and what it owns - Delegate permissions by team and environment - Enforce resource policies in the UI or as code Source: https://www.conduktor.io/product-demos/application-catalog-and-ownership --- # Block Bad Data with Data Quality Enforcement Bad data gets rejected at produce time, and the producer debugs it themselves. A malformed record is blocked by a Conduktor Gateway data quality policy and routed to a dead letter topic the producing partner can read, with every validation error in the record headers. No Schema Registry required. ## What You'll Learn - Block records that violate a data quality policy before they ever reach the topic - Give producers access to a dead letter topic where each rejected record carries its validation errors in the headers - Define rules with the JSON Schema specification or CEL business rules Source: https://www.conduktor.io/product-demos/block-bad-data-with-data-quality-enforcement --- # Column-Level Security in Kafka: Selective Decryption Same topic, same records, but each application sees only the fields it's allowed to see. Decryption interceptors in Conduktor Gateway are scoped to service accounts or groups: in this demo, one app can decrypt email but not last name, while the analytics group gets last name but not email. ## What You'll Learn - Scope a decryption interceptor to a single service account, decrypting only the fields it needs - Apply the same pattern to a group of applications with one interceptor - Get column-level security for Kafka, enforced at the proxy instead of in every consumer Source: https://www.conduktor.io/product-demos/column-level-security-selective-decryption --- # Conduktor Console: Full Product Walkthrough A complete, end-to-end tour of Conduktor Console. From cluster health and access control to application ownership, data quality, topic discovery, live tail, self-service access workflows, and estate-wide Insights, see how every team gets visibility and the autonomy to move. ## What You'll Learn - Navigate cluster health, ownership, and access control - Discover, create, and inspect topics with live tail - Surface risk, cost, and governance gaps with Insights Source: https://www.conduktor.io/product-demos/conduktor-console-full-product-walkthrough --- # Configuration Templates Walkthrough The full tour of configuration templates in Conduktor Console. We start from the YAML the platform team publishes, then show what a developer sees when they create a topic or a connector from a vetted template instead of a blank form. ## What You'll Learn - How the platform team writes and publishes a template with the CLI or API - What pre-fills the Create Topic form: partitions, replication, cleanup policy, retention, naming, and labels - How connector templates take a JDBC sink from dozens of properties down to a few decisions - Why placeholders keep your naming convention enforced at creation Source: https://www.conduktor.io/product-demos/configuration-templates-walkthrough --- # Connector Templates in Action A JDBC sink connector has dozens of properties, which is where self-service provisioning stalls and the tickets start. In this demo we show how a connector template codifies the handful of settings that actually need Kafka Connect expertise, so a developer supplies only the part they know. ## What You'll Learn - How a template fixes the settings that need Connect experience, like `upsert` keyed on the record key, `auto.create`, and `auto.evolve` - How the developer starts from a vetted connector class instead of a blank form - What's left for the developer to fill in: the source topic and where the data is going Source: https://www.conduktor.io/product-demos/connector-templates-in-action --- # Discover and Create Kafka Topics with Built-In Guardrails The topic catalog lets you discover what data already exists, descriptions, owners, fields, and tags, before building anything new. Create topics on the fly with resource policies, guardrails, and configuration recommendations validating inline. No requests, no waiting, and no need to be a Kafka expert to follow best practices. ## What You'll Learn - Discover existing topics before building new ones - Create topics with guardrails and config recommendations - Lower the barrier to entry for new and seasoned developers Source: https://www.conduktor.io/product-demos/discover-and-create-kafka-topics --- # Filter Kafka Messages with Live Tail Click into any Kafka topic and get instant visibility: producer, consumer, and record metrics up top, and a live tail of records flowing through in real time. Filter by time, partition, key, value format, or message content to pinpoint the exact record you need, then drill in to see its payload automatically decoded. ## What You'll Learn - Tail records on a topic live and filter by time, partition, key, value format, and content - Drill into a matched record and read its auto-decoded payload - See topic configuration, consumer group state and lag, the active schema, ACLs, and alerts, all in one view Source: https://www.conduktor.io/product-demos/filter-kafka-messages-with-live-tail --- # Kafka Connect Resilience Managing Kafka Connect at scale is complex—connectors fail, pipelines drift, and visibility drops fast. With Conduktor Console, you can monitor, manage, and recover Kafka connectors effortlessly. Built-in automation ensures continuous reliability and faster incident recovery. ## What You'll Learn - Deploy and manage Kafka connectors without CLI commands - Auto-restart failed tasks and trigger real-time alerts - Keep pipelines resilient and auditable with integrated monitoring Source: https://www.conduktor.io/product-demos/how-conduktor-makes-kafka-connect-resilient --- # How Swiss Post Scaled Kafka with Conduktor With Conduktor, Swiss Post scaled their Kafka estate from 60 to over 300 applications with the same platform team, supporting 800 internal users across development and data science. Through RBAC governance, centralized visibility, and self-service provisioning, they shipped faster and put their AI initiatives on a secure, scalable, and compliant foundation. ## What You'll Learn - Scale application count without growing the platform team - Support hundreds of users with RBAC and self-service - Build a secure, compliant base for AI initiatives Source: https://www.conduktor.io/product-demos/how-swiss-post-scaled-kafka-with-conduktor --- # Inside Conduktor Gateway: ACLs, Audit Logs, and Usage Metering Conduktor Gateway keeps its state where you'd want it: in Kafka topics. This under-the-hood look opens the internal topics behind a data-sharing setup, from per-virtual-cluster ACLs to the byte metering that powers usage-based chargeback. ## What You'll Learn - See per-tenant ACLs stored inside Gateway's internal topics, scoped to each virtual cluster - Trace policy violations and authorization events through the audit log, including safeguard events - Read every interceptor's configuration and the per-partner throughput metering used for chargeback or data monetization Source: https://www.conduktor.io/product-demos/inside-conduktor-gateway-acls-audit-logs-metering --- # Cluster Health and Access Control The moment you log in, Console gives you a bird's-eye view of cluster health and the metrics around your brokers, consumers, and producers. Platform owners set the rules: view every member at a glance, integrate single sign-on with external group mapping or local users, and apply granular role and resource access control out of the box. ## What You'll Learn - Read cluster health and key metrics at a glance - Manage members with SSO or local users - Apply role and resource access control out of the box Source: https://www.conduktor.io/product-demos/kafka-cluster-health-and-access-control --- # Kafka Field-Level Encryption End to End Encrypt sensitive fields in your Kafka records without changing a single line of client code. Conduktor Gateway sits between your applications and your cluster, encrypts PII on produce using keys from your KMS, and decrypts on consume based on who is asking. The full walkthrough, from the architecture diagram to schema-tag encryption. ## What You'll Learn - How Gateway encrypts fields before data reaches the broker, with keys managed centrally in your KMS - How to rotate keys in HashiCorp Vault and instantly revoke access to data encrypted with old key versions - How to scope decryption to service accounts or groups, so each app sees only the fields it's allowed to see - How application teams tag PII in their Avro schema and Gateway encrypts tagged fields automatically Source: https://www.conduktor.io/product-demos/kafka-field-level-encryption-end-to-end --- # Kafka Field-Level Encryption in 5 Minutes Only some fields in a Kafka record are sensitive. In this demo we encrypt those fields with a Conduktor Gateway interceptor, leave the rest readable, and never touch the client code. We also look at what's actually written to the underlying Kafka topic. ## What You'll Learn - Configure an interceptor to encrypt named fields on produce, with keys from HashiCorp Vault - Add a decrypt interceptor so authorized consumers get clear text back on consume - See what's stored in Kafka: ciphertext plus metadata headers that Gateway strips before consumers see them Source: https://www.conduktor.io/product-demos/kafka-field-level-encryption-in-5-minutes --- # Kafka Insights: Risk, Cost, and Governance Most teams are flying blind on their Kafka estate. Insights makes governance gaps, team ownership, and performance visible across everything you run. Catch under-replication and load imbalance before they become incidents, clean up wasted spend from unused topics, prioritize high-touch VIP topics, and know where standards are slipping before an audit tells you first. ## What You'll Learn - Spot risk like under-replication and load imbalance early - Cut costs by finding unused and inactive topics - Track governance and standards across your estate Source: https://www.conduktor.io/product-demos/kafka-insights-risk-cost-and-governance --- # Kafka Lineage with Conduktor Console Stream Lineage turns the producers, consumers, and topics you already run into a map you can walk. Switch between application instances and the raw service accounts behind them, toggle topics on and off, and follow a path out as many hops as you need. The interesting part is what the map catches: a service account reading production data that was never registered as an application. It shows up on the graph the moment traffic does. ## What You'll Learn - Read the estate through application instance and service account views - Use the Topics and Live toggles to narrow the graph to real traffic - Drill down hop by hop from any producer or consumer - Spot unregistered service accounts touching production data Source: https://www.conduktor.io/product-demos/kafka-lineage-with-conduktor-console --- # Kafka Stream Lineage End to End Every change to Kafka is a change to something you can't see. Stream Lineage builds the missing dependency map from three things you already have: the ACLs on your cluster, your ownership model, and live traffic through the Gateway. This walkthrough uses an airline estate to do the three jobs that map unlocks, retiring a topic, tracing a hot path, and turning up a consumer nobody registered. ## What You'll Learn - How lineage is assembled from ACLs, ownership, and live Gateway traffic - Confirm a topic is safe to retire before you touch it - Trace a hot path back to the applications driving it - Find consumers reading production data outside your ownership model Source: https://www.conduktor.io/product-demos/kafka-stream-lineage-end-to-end --- # Key Rotation for Encrypted Kafka Data Your KMS owns your keys, so that's where rotation happens. In this demo we rotate an encryption key in HashiCorp Vault, produce new data with the new key version, then disable the old version and watch the old records become undecryptable instantly. The same mechanism powers crypto shredding for GDPR deletion requests. ## What You'll Learn - Rotate a key in Vault and see new produces pick up the new key version - Disable an old key version to make data encrypted with it unreadable, with no re-encryption - Re-enable a key version to restore access, showing rotation is reversible until you delete Source: https://www.conduktor.io/product-demos/key-rotation-for-encrypted-kafka-data --- # Query Kafka Topics with SQL Kafka was never built for easy querying—until now. With Conduktor Console, teams can index Kafka topics into live, queryable tables and analyze real-time event streams using plain SQL. No custom pipelines. No CLI. Just fast, trusted insights from streaming data. ## What You'll Learn - Query Kafka topics instantly with SQL - Replay event history for debugging or analytics - Keep pipelines reliable with built-in monitoring and circuit breakers Source: https://www.conduktor.io/product-demos/query-your-kafka-topics-like-a-sql-database --- # Schema-Tag Encryption: App Teams Tag PII, the Platform Enforces It The platform team doesn't know which fields are sensitive. The teams producing the data do. With schema-based encryption, application teams tag fields as PII or GDPR directly in their Avro schema, and Conduktor Gateway encrypts anything carrying those tags automatically. ## What You'll Learn - Tag sensitive fields like name, password, and salary directly in an Avro schema - Configure one interceptor that encrypts any field tagged PII or GDPR, with no field lists to maintain - Keep encrypted records schema-compatible, and decrypt the full record for authorized consumers Source: https://www.conduktor.io/product-demos/schema-tag-encryption --- # Self-Service Topic Access Requests and Approvals Need access to another team's data? Request the permissions you need with a reason attached, and the data owner reviews and approves it directly, not a ticket queue. Approvals happen in the UI or with a config file Console provides, so cross-team autonomy stays fast and ownership stays clear at audit time. ## What You'll Learn - Request topic access with a reason attached - Let data owners approve in the UI or with config - Keep ownership clear for audits Source: https://www.conduktor.io/product-demos/self-service-topic-access-requests-and-approvals --- # External Data Sharing End to End Share Kafka topics with companies you don't control, without giving anyone direct access to your cluster. Payment processor Oscorp receives credit card transactions from one external partner and shares filtered results with another, all through Conduktor Gateway. The full walkthrough, from the architecture to what's stored on disk. ## What You'll Learn - Isolate each partner in their own virtual cluster with friendly topic names, layered on top of network isolation and ACLs - Enforce producer standards, filter what each consumer sees with SQL topic views, and encrypt and mask sensitive fields - Block malformed records with data quality policies and route them to a dead letter topic the producer can debug Source: https://www.conduktor.io/product-demos/share-kafka-data-externally-end-to-end --- # Share Kafka Topics with Partners or Other Teams One company produces, another consumes, and neither ever touches your Kafka cluster. An external partner gets its own isolated virtual cluster with friendly topic names, the Gateway rejects their produce until it meets your standards, and a second partner receives only the records you chose to share. Shown with external partners; the same configuration isolates internal teams. ## What You'll Learn - Give each partner an isolated virtual cluster where your internal topics appear under friendly alias names - Enforce producer standards like LZ4 compression and idempotence at the proxy, with the fix spelled out in the error - Filter what each consumer sees with a SQL topic view at read time, and watch a record flow end to end Source: https://www.conduktor.io/product-demos/share-kafka-topics-with-partners-or-other-teams --- # Topic Templates in Action A topic template is a YAML file the platform team publishes through the CLI or API. In this demo we take a high-throughput template and show everything it pre-fills in the Create Topic form, including the 24-partition capacity call that's painful to change after the fact. ## What You'll Learn - How a template seeds partitions, replication factor, cleanup policy, retention, and `min.insync.replicas` - How a naming convention and governance labels come pre-filled, with placeholders the developer replaces - Why the partition count is the one decision worth getting right at creation Source: https://www.conduktor.io/product-demos/topic-templates-in-action --- # Topic Views in Action Give one team a filtered, projected view of a Kafka topic without copying it. In this demo we build a topic view in Conduktor Gateway, consume it like any other topic, and show the raw topic stays untouched. ## What You'll Learn - Define a topic view with a SQL filter and projection - Consume the view by name, with only the rows and fields you chose - Confirm the raw topic is unchanged, plus a look at header filters, REGEXP, and nested fields Source: https://www.conduktor.io/product-demos/topic-views-in-action --- # Why Configuration Templates? Once you federate Kafka ownership and put guardrails in place, the next bottleneck is expertise. Not every team has a Kafka SME, and the ones who do have their knowledge stuck in wikis and word of mouth. This short walkthrough covers the idea behind configuration templates, and why guardrails alone don't get you there. ## What You'll Learn - Why enforcement gates what teams can't do, but never shows them what good looks like - How configuration templates codify your platform team's expertise once - Where templates and policies work together for self-service provisioning Source: https://www.conduktor.io/product-demos/why-configuration-templates --- # Why Is Effective Kafka Change Management So Tricky? Onboarding a new team, retiring a topic, chasing a connection that suddenly went hot: three different jobs that all stall on the same three questions. Who produces to this topic? Who consumes from it? What breaks if it changes? Most teams answer by asking around. This short talk covers why the dependency map is genuinely hard to build, and the four approaches teams reach for when they try. ## What You'll Learn - The three questions behind every Kafka change - Why tribal knowledge stops scaling as the estate grows - The four ways teams try to build a dependency map, and where each falls short Source: https://www.conduktor.io/product-demos/why-kafka-change-management-is-tricky --- # Why Topic Views? As Kafka usage grows, different teams and consumers need their own representation of the same topic. This short walkthrough covers where each tool fits, and when a read-time topic view is the simpler answer. ## What You'll Learn - When a Kafka Connect SMT, Kafka Streams, or Flink is the right tool - Why a per-consumer slice of an existing topic falls between them - How topic views give a filtered, projected view at read time, with no copy Source: https://www.conduktor.io/product-demos/why-topic-views --- # Why We Built Conduktor Console Kafka has become critical infrastructure, but operating it at scale is broken. Developers wrestle with cryptic CLI tools and manual approvals, while platform teams turn into full-time gatekeepers. That is why we built Console: one UI and API across every cluster, team, and environment, giving every team visibility, ownership, and the autonomy to move without waiting. ## What You'll Learn - Why Kafka operations break down as usage scales - How Console gives teams visibility, ownership, and autonomy - What a single UI and API across every cluster unlocks Source: https://www.conduktor.io/product-demos/why-we-built-conduktor-console --- # Product Demos Demos to highlight what Conduktor can do. - **Kafka Lineage with Conduktor Console** — Walk the dependency map for a Kafka estate in Console: application and service account views, live traffic, drill-down by hops, and the consumers nobody registered. — product-demos/kafka-lineage-with-conduktor-console — 2026-09-04 — nmxzgQHbUqo — kafka lineage with conduktor console walk the dependency map for a kafka estate in console application and service account views live traffic drill down by hops and the consumers nobody registered stream lineage dependency map service accounts application instances live traffic hops shadow consumers acls visibility console acl acls access control list — Console - **Kafka Stream Lineage End to End** — Retire a topic, find a hot path, and turn up an unregistered consumer across an airline's Kafka estate, using a dependency map built from ACLs, ownership, and live Gateway traffic. — product-demos/kafka-stream-lineage-end-to-end — 2026-09-04 — WvthWDaFYVE — kafka stream lineage end to end retire a topic find a hot path and turn up an unregistered consumer across an airline s kafka estate using a dependency map built from acls ownership and live gateway traffic stream lineage dependency map acls ownership gateway traffic topic retirement hot path shadow consumers visibility console acl acls access control list gateway proxy — Console - **Why Is Effective Kafka Change Management So Tricky?** — Onboarding a team, retiring a topic, and chasing a hot connection all need the same three answers, and most teams get them by asking around. Why the dependency map is so hard to build. — product-demos/why-kafka-change-management-is-tricky — 2026-09-04 — UaZZrC0c7gQ — why is effective kafka change management so tricky onboarding a team retiring a topic and chasing a hot connection all need the same three answers and most teams get them by asking around why the dependency map is so hard to build change management dependency map stream lineage topic retirement onboarding hot path tribal knowledge visibility console — Console - **Block Bad Data with Data Quality Enforcement** — Reject malformed Kafka records at produce time with Gateway data quality policies: JSON Schema or CEL rules, plus a dead letter topic the producer can debug. — product-demos/block-bad-data-with-data-quality-enforcement — 2026-07-28 — O8EMazCUksg — block bad data with data quality enforcement reject malformed kafka records at produce time with gateway data quality policies json schema or cel rules plus a dead letter topic the producer can debug data quality dead letter topic json schema cel validation produce time partner business rules data quality gateway gateway proxy — Gateway - **Inside Gateway: ACLs, Audit Logs, and Metering** — Conduktor Gateway stores its state in Kafka topics. A look inside per-virtual-cluster ACLs, the audit log, interceptor configs, and per-partner byte metering. — product-demos/inside-conduktor-gateway-acls-audit-logs-metering — 2026-07-28 — C9C35FLoDc8 — inside gateway acls audit logs and metering conduktor gateway stores its state in kafka topics a look inside per virtual cluster acls the audit log interceptor configs and per partner byte metering internal topics acl audit log interceptors observability metering chargeback monetization virtual clusters safeguard governance gateway acl acls access control list gateway proxy — Gateway - **External Data Sharing End to End** — Share Kafka topics with companies you don't control: network isolation, per-partner virtual clusters, traffic policies, topic views, encryption, and data quality. — product-demos/share-kafka-data-externally-end-to-end — 2026-07-28 — mE_5BVe411A — external data sharing end to end share kafka topics with companies you don t control network isolation per partner virtual clusters traffic policies topic views encryption and data quality external data sharing virtual clusters network isolation dmz traffic control policies topic views alias topics encryption masking data quality partner kms data sharing gateway pii personally identifiable information masking data masking gateway proxy encryption encrypt encrypted — Gateway - **Share Kafka Topics with Partners or Other Teams** — One company produces, another consumes, and neither touches your Kafka cluster: virtual clusters, alias topics, produce policies, and SQL topic views in action. — product-demos/share-kafka-topics-with-partners-or-other-teams — 2026-07-28 — MWgsC-V-cs8 — share kafka topics with partners or other teams one company produces another consumes and neither touches your kafka cluster virtual clusters alias topics produce policies and sql topic views in action external data sharing virtual clusters alias topics topic views produce policies compression idempotence sql filter partner multi tenancy data sharing gateway multi tenancy multi tenant multitenancy tenant isolation gateway proxy — Gateway - **Column-Level Security in Kafka** — Scope decryption to service accounts or groups with Conduktor Gateway: same topic, same records, but each app sees only the fields it's allowed to see. — product-demos/column-level-security-selective-decryption — 2026-07-27 — QVkHpTdZnLo — column level security in kafka scope decryption to service accounts or groups with conduktor gateway same topic same records but each app sees only the fields it s allowed to see column level security selective decryption service accounts interceptor pii access control proxy encryption gateway rbac role based access control access control pii personally identifiable information masking data masking gateway proxy encryption encrypt encrypted — Gateway - **Kafka Field-Level Encryption End to End** — Encrypt sensitive fields in Kafka records with Conduktor Gateway: encrypt on produce, rotate keys in Vault, and control which apps decrypt which fields. — product-demos/kafka-field-level-encryption-end-to-end — 2026-07-27 — qWLNev-9vZQ — kafka field level encryption end to end encrypt sensitive fields in kafka records with conduktor gateway encrypt on produce rotate keys in vault and control which apps decrypt which fields field level encryption kms vault envelope encryption key rotation schema tags pii gdpr column level security interceptor proxy encryption gateway pii personally identifiable information masking data masking gateway proxy encryption encrypt encrypted — Gateway - **Kafka Field-Level Encryption in 5 Minutes** — Encrypt specific fields in Kafka records with a Gateway interceptor: no client changes, no encryption libraries, keys managed in your KMS. — product-demos/kafka-field-level-encryption-in-5-minutes — 2026-07-27 — SgZZo3cAM4w — kafka field level encryption in 5 minutes encrypt specific fields in kafka records with a gateway interceptor no client changes no encryption libraries keys managed in your kms field level encryption kms vault interceptor pii ciphertext headers proxy encryption gateway pii personally identifiable information masking data masking gateway proxy encryption encrypt encrypted — Gateway - **Key Rotation for Encrypted Kafka Data** — Rotate Kafka encryption keys in HashiCorp Vault and instantly revoke access to old records, without re-encrypting topics or touching the data. — product-demos/key-rotation-for-encrypted-kafka-data — 2026-07-27 — IzvNeXVyIZ8 — key rotation for encrypted kafka data rotate kafka encryption keys in hashicorp vault and instantly revoke access to old records without re encrypting topics or touching the data key rotation vault kms crypto shredding gdpr key versions proxy encryption gateway gateway proxy encryption encrypt encrypted — Gateway - **Schema-Tag Encryption** — Application teams tag PII fields in their Avro schema and Conduktor Gateway encrypts anything carrying those tags, with no per-topic field lists. — product-demos/schema-tag-encryption — 2026-07-27 — NiHeqnB6MOU — schema tag encryption application teams tag pii fields in their avro schema and conduktor gateway encrypts anything carrying those tags with no per topic field lists schema tags avro pii gdpr schema registry interceptor platform team proxy encryption gateway pii personally identifiable information masking data masking gateway proxy encryption encrypt encrypted — Gateway - **Configuration Templates Walkthrough** — A full walkthrough of Kafka topic and connector templates in Conduktor Console, from the YAML spec to a developer's pre-filled create form. — product-demos/configuration-templates-walkthrough — 2026-06-23 — L9uq_qGaAzE — configuration templates walkthrough a full walkthrough of kafka topic and connector templates in conduktor console from the yaml spec to a developer s pre filled create form topic templates connector templates jdbc sink guardrails yaml cli platform team kafka expertise governance self service console self service selfservice — Console - **Connector Templates in Action** — A JDBC sink connector template takes dozens of properties down to the few decisions that need Connect expertise, plus the source and destination. — product-demos/connector-templates-in-action — 2026-06-23 — 0xVCOh9IUEU — connector templates in action a jdbc sink connector template takes dozens of properties down to the few decisions that need connect expertise plus the source and destination connector templates jdbc sink kafka connect upsert auto evolve connector wizard self service console self service selfservice — Console - **Topic Templates in Action** — A high-throughput topic template pre-fills the Create Topic form in Conduktor Console: partitions, replication, cleanup policy, retention, and labels. — product-demos/topic-templates-in-action — 2026-06-23 — 4phShkrzC6E — topic templates in action a high throughput topic template pre fills the create topic form in conduktor console partitions replication cleanup policy retention and labels topic templates yaml partitions replication factor retention naming convention guardrails governance self service console self service selfservice — Console - **Why Configuration Templates?** — Federate Kafka ownership and the next bottleneck is expertise. Configuration templates let the platform team codify what good looks like once. — product-demos/why-configuration-templates — 2026-06-23 — tlmRnd3qmXk — why configuration templates federate kafka ownership and the next bottleneck is expertise configuration templates let the platform team codify what good looks like once configuration templates platform team kafka expertise guardrails best practices governance self service console self service selfservice — Console - **Filter Kafka Messages with Live Tail** — Tail a Kafka topic in real time and filter by partition, key, value format, or message content, then drill into the auto-decoded payload of any record. — product-demos/filter-kafka-messages-with-live-tail — 2026-06-22 — jK1En3V7lIU — filter kafka messages with live tail tail a kafka topic in real time and filter by partition key value format or message content then drill into the auto decoded payload of any record live tail message filtering troubleshooting debugging consumer groups consumer lag schema acl partitions visibility console acl acls access control list — Console - **Topic Views in Action** — Give one team a filtered, projected view of a Kafka topic at read time: no copy, raw topic untouched. Built in Conduktor Gateway. — product-demos/topic-views-in-action — 2026-06-19 — c6PgwliYpsw — topic views in action give one team a filtered projected view of a kafka topic at read time no copy raw topic untouched built in conduktor gateway topic views sql filtering projection nested json read time no copy proxy gateway gateway proxy — Gateway - **Why Topic Views?** — When to reach for an SMT, Kafka Streams, or Flink to reshape a Kafka topic, and when a read-time topic view is the simpler fit for one consumer. — product-demos/why-topic-views — 2026-06-19 — 4T0JRMkMSXQ — why topic views when to reach for an smt kafka streams or flink to reshape a kafka topic and when a read time topic view is the simpler fit for one consumer topic views smt kafka streams flink read time projection proxy gateway gateway proxy — Gateway - **Why We Built Conduktor Console** — Why we built Console to give every team visibility, ownership, and autonomy across every Kafka cluster, team, and environment. — product-demos/why-we-built-conduktor-console — 2026-06-02 — j3uuo56H4hs — why we built conduktor console why we built console to give every team visibility ownership and autonomy across every kafka cluster team and environment visibility ownership autonomy cost attribution platform team gatekeepers self service cost governance console self service selfservice cost costs tco spend — Console - **Conduktor Console: Full Product Walkthrough** — A complete tour of Conduktor Console, from cluster health and ownership to data quality, topic management, live tail, access workflows, and Insights. — product-demos/conduktor-console-full-product-walkthrough — 2026-06-01 — Q8tWAkrj4KQ — conduktor console full product walkthrough a complete tour of conduktor console from cluster health and ownership to data quality topic management live tail access workflows and insights cluster health sso application catalog data quality topic catalog live tail consumer lag insights access requests self service visibility cost governance console self service selfservice cost costs tco spend — Console - **Cluster Health and Access Control** — Get a bird's-eye view of cluster health and metrics, then manage members, single sign-on, and granular access control out of the box. — product-demos/kafka-cluster-health-and-access-control — 2026-05-31 — wHHL7w9Akig — cluster health and access control get a bird s eye view of cluster health and metrics then manage members single sign on and granular access control out of the box cluster health brokers sso single sign on rbac access control permissions governance visibility console rbac role based access control access control — Console - **Kafka Insights: Risk, Cost, and Governance** — Surface under-replication, wasted spend, VIP topics, and governance gaps across your entire Kafka estate before an audit or incident finds them first. — product-demos/kafka-insights-risk-cost-and-governance — 2026-05-30 — 0TjEy8cN0XQ — kafka insights risk cost and governance surface under replication wasted spend vip topics and governance gaps across your entire kafka estate before an audit or incident finds them first insights risk analysis under replication cost savings waste vip topics schema audit cost visibility governance console cost costs tco spend — Console - **Application Catalog and Ownership** — See every application on your Kafka cluster, what each one owns, and the resource policies that enforce naming, partitions, and retention. — product-demos/application-catalog-and-ownership — 2026-05-29 — kN3oHeeCbrU — application catalog and ownership see every application on your kafka cluster what each one owns and the resource policies that enforce naming partitions and retention application catalog ownership resource policies naming conventions service accounts permissions groups self service governance console self service selfservice — Console - **Discover and Create Kafka Topics with Built-In Guardrails** — Discover existing topics and create new ones on the fly, with resource policies, guardrails, and configuration recommendations built in. — product-demos/discover-and-create-kafka-topics — 2026-05-28 — uu5I1olJNBo — discover and create kafka topics with built in guardrails discover existing topics and create new ones on the fly with resource policies guardrails and configuration recommendations built in topic catalog topic creation guardrails discovery resource policies self service console self service selfservice — Console - **Self-Service Topic Access Requests and Approvals** — Request access to another team's data with a reason attached and let the data owner approve it in the UI, no ticket queue required. — product-demos/self-service-topic-access-requests-and-approvals — 2026-05-27 — __TjXtndy8E — self service topic access requests and approvals request access to another team s data with a reason attached and let the data owner approve it in the ui no ticket queue required access requests approvals topic catalog data owner audit permissions governance self service console self service selfservice — Console - **How Swiss Post Scaled Kafka with Conduktor** — How Swiss Post scaled from 60 to over 300 applications and 800 users on the same platform team, with RBAC governance and self-service provisioning. — product-demos/how-swiss-post-scaled-kafka-with-conduktor — 2026-05-23 — PkJQ_3-tDXg — how swiss post scaled kafka with conduktor how swiss post scaled from 60 to over 300 applications and 800 users on the same platform team with rbac governance and self service provisioning swiss post customer story rbac scaling platform team self service resource provisioning self service governance console rbac role based access control access control self service selfservice — Console - **Query Kafka Topics with SQL** — Index Kafka topics into live, queryable tables and analyze real-time event streams using plain SQL. — product-demos/query-your-kafka-topics-like-a-sql-database — 2025-09-18 — ZQBPI-mJQgY — query kafka topics with sql index kafka topics into live queryable tables and analyze real time event streams using plain sql sql postgres indexing queryable event history replay debugging visibility console — Console - **Kafka Connect Resilience** — Monitor, manage, and recover Kafka connectors effortlessly. Built-in automation ensures continuous reliability. — product-demos/how-conduktor-makes-kafka-connect-resilient — 2025-09-10 — ezY5lsM0HMs — kafka connect resilience monitor manage and recover kafka connectors effortlessly built in automation ensures continuous reliability kafka connect connectors auto restart alerts monitoring sink source webhook slack self service visibility console self service selfservice — Console Source: https://www.conduktor.io/product-demos --- # Label-based Chargeback, Confluent/Aiven service accounts, and IAM auth ## Attribute costs by team or project with labels Chargeback (now GA in Scale Plus) supports user-defined labels. Group usage by team, environment, or project instead of raw service accounts. Add metadata labels to service accounts and get cost breakdowns that align with your org structure. ![Chargeback](https://www.conduktor.io/assets/images/releases/apr-2025-1.webp) ## Manage Confluent Cloud and Aiven service accounts via API View, manage, and label Confluent Cloud and Aiven service accounts through the API with full metadata support. Group accounts by function, team, or project. ## Alias shared topics in Partner Zones Add aliases to shared topics in Partner Zones to hide internal naming conventions. Control what external teams see without replicating or renaming topics internally. ## Authenticate with existing IAM and OAuth flows Conduktor Gateway, a Kafka proxy, supports Delegated OAuth and MSK IAM. Point apps to Conduktor Gateway and keep existing auth flows in place. ![Gateway Authentication](https://www.conduktor.io/assets/images/releases/apr-2025-2.png) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1330). Source: https://www.conduktor.io/product-releases/april-2025 --- # CLI for connectors, self-service Connect, and Console HA ## Manage connectors via CLI Deploy and manage Kafka Connect resources using infrastructure as code. ```yaml --- apiVersion: kafka/v2 kind: Connector metadata: connectCluster: kafka-connect name: click.my-connector labels: conduktor.io/auto-restart-enabled: true conduktor.io/auto-restart-frequency: 600 spec: config: connector.class: 'org.apache.kafka.connect.tools.MockSourceConnector' tasks.max: '1' topic: click.pageviews ``` ## Grant connector ownership to application teams Application teams can manage their connectors through self-service by adding connector ownership to application instances. ```yaml --- apiVersion: self-service/v1 kind: ApplicationInstance metadata: application: "clickstream-app" name: "clickstream-dev" spec: cluster: "shadow-it" serviceAccount: "sa-clicko" resources: - type: CONNECTOR connectCluster: shadow-connect patternType: PREFIXED name: "click." ``` ## View connector status over time The Kafka Connect UI now shows connector status graphs over time and connector previews on the cluster selection screen. ![Kafka Connect UI](https://www.conduktor.io/assets/images/releases/aug-2024-1.png) ## Deploy multiple Console instances for HA Multiple Console instances can run in parallel. Only one instance handles Kafka data indexing, so cluster impact stays minimal. --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1260). Source: https://www.conduktor.io/product-releases/august-2024 --- # Application group lifecycle, JSON schema validation, and multi-cluster Partner Zones ## Manage application group membership and permissions Self-service application groups now have full lifecycle management: - **Members tab**: Add and manage group members - **Resource access tab**: Set granular permissions across topics, consumer groups, subjects, and connectors - **External mapping**: Align application groups with user groups from identity providers Regex-based group assignment works for both user groups and application groups. ![](https://www.conduktor.io/assets/images/releases/aug-2025-1.png) ## See all resources associated with an application Application instances now include a resources tab showing all associated resources, dependencies, and relationships. ## Use Confluent subject naming strategies Support for Confluent subject strategies directly in the produce page. Reference and organize schemas without forcing a naming convention. ## Mark violations instead of blocking them Trust policies can mark violating messages with a special header instead of blocking them. Downstream systems see flagged data and can audit, route, or act on it. ## Validate JSON messages against schemas JSON Schema Rules validate the structure and content of every JSON message: - Check fields, types, and structure against your defined schema - Log, tag, or block violations - No Schema Registry required Learn more in [our blog](https://www.conduktor.io/blog/bounce-bad-kafka-data-with-json-schema-rules). ![](https://www.conduktor.io/assets/images/releases/aug-2025-2.webp) ## Use pre-built regex patterns for rules A regex library provides pre-built examples when creating data quality rules. ## Create Partner Zones across multiple clusters Partner Zones (now GA) work across any Kafka cluster configured in Conduktor Gateway. Create isolated environments for external partners without spinning up multiple proxies. ![](https://www.conduktor.io/assets/images/releases/aug-2025-3.png) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1370). Source: https://www.conduktor.io/product-releases/august-2025 --- # Live lineage, Flink SQL, and Chargeback ## See live traffic in Stream Lineage Stream Lineage now shows live traffic. Each connection reports its current throughput, and the graph updates as traffic changes. Access-based lineage tells you what an application is allowed to read; live traffic tells you what it actually reads. That's how you spot a permission nobody uses, or confirm a consumer has really been cut over. Traffic data comes from Gateway, so it needs a cluster running the observability Interceptor. Lineage is also faster: graphs with thousands of topics and service accounts draw in a couple of seconds and stay responsive as you pan and zoom. [Explore Stream Lineage →](https://docs.conduktor.io/guide/conduktor-concepts/stream-lineage) ![Stream Lineage in Live mode, showing service accounts connected to production topics with per-connection throughput labelled in KiB/s](https://www.conduktor.io/assets/images/releases/august-2026-live-lineage.png) ## Flink SQL on Confluent Cloud (preview) Console now includes a Flink SQL workbench. List the statements running across your Confluent environments, write and run new ones with a catalog browser alongside the editor, watch results stream in live, export to CSV, and stop or delete statements without leaving Console. It respects the permissions you already have: Console parses each statement's SQL and checks every topic it touches against that user's own Kafka permissions. Flink access follows your existing RBAC. This preview covers Confluent Cloud Flink only. [Read the release notes →](https://docs.conduktor.io/guide/release-notes) ![The Flink SQL workbench in Console, with the environment catalog on the left, a running INSERT INTO statement in the editor, and its status and SQL shown below](https://www.conduktor.io/assets/images/releases/august-2026-flink-sql.png) ## Chargeback and alerts, owned by the teams that use them Application owners can see Chargeback scoped to their own application and delegate that visibility to their team. The same applies to alerts and service accounts, which owners manage for their own resources without platform-wide permissions. Platform teams get reusable permission templates, plus a resource policy on application instance permissions to bound what can be delegated. It's all opt-in and off by default. [Learn about self-service →](https://docs.conduktor.io/guide/conduktor-concepts/self-service) ## Client metrics: KIP-714 telemetry without touching your clients Gateway now terminates KIP-714 client telemetry and forwards it to an OpenTelemetry collector, enriched with the identity Gateway already knows: `client_id`, `principal`, and `vcluster`. Kafka clients from 3.7.0 onward push these metrics by default, so for most fleets you get producer and consumer visibility without touching client code or configuring KIP-714 on your brokers. [Configure client metrics →](https://docs.conduktor.io/guide/reference/client-metrics) ## SCRAM credentials for Gateway service accounts Local Gateway service accounts support long-lived SCRAM-SHA-512 and SCRAM-SHA-256 credentials, with no expiry to rotate. Gateway derives a verifier from the password and stores only that. [Manage Gateway service accounts →](https://docs.conduktor.io/guide/conduktor-concepts/gateway-service-accounts) ## Incremental fetch sessions are generally available KIP-227 incremental fetch sessions are generally available and on by default. A consumer opens a session with one full Fetch request, then sends only the partitions that changed on each later request while the broker tracks the rest. Fetch requests get smaller, consumers reading hundreds of partitions get faster, and there's nothing to configure. [Read about incremental fetch sessions →](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#incremental-fetch-sessions) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/guide/release-notes). Source: https://www.conduktor.io/product-releases/august-2026 --- # SQL on Kafka, cluster health dashboard, and shareable filters ## Query Kafka data with SQL SQL on Kafka lets teams extract insights from real-time and historical data streams without duplicating data to external systems. Conduktor handles RBAC enforcement, controls which topics are indexed, and transforms Kafka data into a columnar format for querying. Use cases by role: - **Data engineers**: Explore raw streams, troubleshoot pipelines, debug messages with masking and access controls - **Business teams**: Analyze orders and customer events on-demand without permanent storage - **Ops teams**: Query metadata (timestamps, offsets, compression) to optimize performance and identify duplicates Read more in [Transforming Real-Time Data Into Instant Insights](https://www.conduktor.io/blog/sql-over-kafka-transforming-real-time-data-into-instant-insights). ![](https://www.conduktor.io/assets/images/releases/dec-2024-1.webp) ## View cluster health at a glance The redesigned homepage shows Kafka cluster health, indexing module status, and recently viewed resources. ![](https://www.conduktor.io/assets/images/releases/dec-2024-2.png) ## View consumer groups by topic or member The Consumer Group page organizes data by subscribed topics or active members. Drill down to individual member or topic-partition assignments. ![](https://www.conduktor.io/assets/images/releases/dec-2024-3.png) ## Control which topics appear in the catalog Set `catalogVisibility: PUBLIC` or `PRIVATE` on topics to control what appears in the topic catalog. ```yaml apiVersion: kafka/v2 kind: Topic metadata: cluster: shadow-it name: click.event-stream.avro catalogVisibility: PUBLIC # or PRIVATE spec: ... ``` ## Share consume page filters with teammates Filters in the Topic Consume view are shareable. Save filters as Private or Organization-wide to ensure everyone works with the same data subset. ![](https://www.conduktor.io/assets/images/releases/dec-2024-4.png) ## Export audit logs to Kafka Publish audit log events to a Kafka topic in CloudEvents format. Consume the stream to trigger actions (like creating ServiceNow tickets) for specific events. ```json { "source": "//kafka/kafkacluster/production/topic/website-orders", "data": { "eventType": "Kafka.Topic.Create", "metadata": { "name": "website-orders", "cluster": "production" } }, "datacontenttype": "application/json", "id": "ad85122c-0041-421e-b04b-6bc2ec901e08", "time": "2024-10-10T07:52:07.483140Z", "type": "AuditLogEventType(Kafka,Topic,Create)", "specversion": "1.0" } ``` ## Deploy Console with multi-host PostgreSQL Console's database now supports multi-host PostgreSQL for high availability. --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1300). Source: https://www.conduktor.io/product-releases/december-2024 --- # Track storage costs and query Kafka with AI ## Track storage and partition costs with Chargeback Chargeback now tracks storage (byte-hours) and partitions (partition-hours)—the two dominant cost drivers in modern Kafka deployments. You can attribute costs per topic, view daily or monthly aggregates, apply cluster-specific rates, and filter by total, storage-only, or partition-only spend. Works directly with Console's metadata indexer—no Conduktor Gateway required. [Use Chargeback without Gateway →](https://docs.conduktor.io/guide/use-cases/chargeback-without-gateway) ## Query Kafka clusters through AI assistants > **Preview**: This feature is subject to change as we continue development. Console now provides MCP (Model Context Protocol) tools for exploring and analyzing your Kafka clusters with AI assistants like Claude. Tools cover clusters, topics, schemas, consumer groups, and Insights data. [Configure Console MCP →](https://docs.conduktor.io/guide/conduktor-in-production/automate/mcp) ## Monitor Console usage and license utilization Track active users, self-service applications, access requests, partner zones, and shared topics across three time periods: last 30 days, current month, and last 12 months. Includes monthly trends graph and CSV export for offline analysis. Historical data builds up over time as Console collects usage metrics. [Learn about usage data →](https://docs.conduktor.io/guide/conduktor-in-production/manage-licenses) ![Usage insights dashboard](https://www.conduktor.io/assets/images/releases/december-2025-usage-insights.png) ## Detect data loss risk from ISR configuration Insights now factors in both replication factor and `min.insync.replicas` when assessing data loss risk, not just replication factor alone. ![Data loss risk detection](https://www.conduktor.io/assets/images/releases/december-2025-data-loss-detection.png) ## Encrypt Kafka messages with Fortanix KMS Conduktor Gateway supports field-level and full payload encryption using Fortanix Data Security Manager (DSM) for key management. This integration enables secure encryption of Kafka messages with keys managed by Fortanix. [Configure Fortanix KMS →](https://docs.conduktor.io/guide/reference/data-security) ## Deploy Console with a custom context path Console now supports context path configuration for deployments like `https://company.com/conduktor`. Useful for restricted Kubernetes environments with ingress controllers. [Deploy with a context path →](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console/kubernetes) ## Quality of life improvements - **Indexing performance**: Refactored topic and consumer group indexing to reduce load on Kafka clusters and improve CPU/memory usage - **Insights collection**: 2-3x faster data collection through various optimizations - **Insights refresh**: Default frequency changed from 5 to 15 minutes, freeing up system resources - **ksqlDB auth**: Server configuration now accepts authorization HTTP header for SSL + basic auth - **Self-service RBAC**: Operations now happen in a single transaction, preventing incomplete object creation in CI/CD - **CLI dry-run**: Now correctly reports `NotChanged` when there are no actual changes --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/guide/release-notes). Source: https://www.conduktor.io/product-releases/december-2025 --- # Community Edition and Kafka Connect governance ## Get started with Community Edition We've made some changes to our Community Edition. Community Edition provides a streamlined Kafka management experience at no cost: connect clusters, browse topics, consume messages, and explore schemas. If you're currently using the free license, changes apply when you upgrade to Console v1.43.0. [Learn about Community Edition →](https://docs.conduktor.io/guide/conduktor-in-production/manage-licenses/community-edition) ## Manage Kafka Connect resource policies in the UI Kafka Connect connector resource policies can now be created and managed directly in the Console UI. Define naming conventions, configuration standards, and governance rules for your connectors without leaving the browser. ## Kafka Connect now enforces RBAC The Kafka Connect UI now respects RBAC permissions. Actions a user isn't permitted to perform are hidden from the interface. This ensures connector management follows the same access controls as the rest of your Kafka platform. ## Deprecation notice: Self-service topic policies Self-service topic policies will be deprecated in Console v1.46.0. Convert your existing topic policies to resource policies as soon as possible. Resource policies have full feature parity with the legacy topic policies. Use the Console UI to migrate: go to **Settings** > **Policies migration**. ## Quality of life improvements - **New format types**: Produce and consume topic messages with Boolean, Short, Int, and UUID format types - **Improved error handling**: Topic consume page returns more explicit errors and includes a retry option - **Audit log tracking**: New `Kafka.ConsumerGroup.Create` event type tracks consumer group creation in the audit log - **Kafka Connect auto-restart API**: Public API endpoint to manage connector auto-restart settings - **Navigation cleanup**: MCP navigation moved to the Settings page to reduce main menu clutter --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/guide/release-notes). Source: https://www.conduktor.io/product-releases/february-2026 --- # Chargeback, granular alerts, and Partner Zones ## Assign Kafka costs to teams with Chargeback Track Kafka resource costs and allocate them to specific teams, departments, or projects based on actual data usage. Available for Scale Plus customers. ![Chargeback](https://www.conduktor.io/assets/images/releases/jan-2025-1.webp) ## Configure alerts by owner and delivery channel Alerts can be owned by specific users, teams, or applications. Deliver alerts via Slack, Teams, or webhooks, customized per team. Event-driven notifications trigger on consumer group state changes, topic updates, or audit events. End-users can configure their own alerts. ![Alerts](https://www.conduktor.io/assets/images/releases/jan-2025-2.webp) ## Share Kafka data with partners via Partner Zones Partner Zones create isolated environments for external partners to access selected Kafka topics. No data duplication or additional infrastructure. - Dedicated zones with strict access controls - Per-topic permissions - Encryption and masking policies ![Partner Zones](https://www.conduktor.io/assets/images/releases/jan-2025-3.webp) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1310). Source: https://www.conduktor.io/product-releases/january-2025 --- # Data quality insights, cluster-wide policies, and tokenization ## Understand data quality health at a glance The new Data Quality Insights dashboard provides a unified view of your data quality posture. A composite health score combines coverage (topics with policies) and enforcement (policies blocking violations) to help you identify whether you need more policies or stricter enforcement. [Understand data quality →](https://docs.conduktor.io/guide/conduktor-concepts/data-quality-policies) ![Data Quality Insights dashboard](https://www.conduktor.io/assets/images/releases/january-2026-data-quality.png) ## Enforce resource policies across entire clusters Resource policies can now be applied at the cluster level, not just per application. Attach governance rules, naming conventions, replication standards, or schema restrictions to a cluster and they automatically apply to all resources within it. This enables centralized policy enforcement before applications are even provisioned—no per-app configuration required. [Configure resource policies →](https://docs.conduktor.io/guide/reference/self-service-reference) ![Cluster-wide resource policies](https://www.conduktor.io/assets/images/releases/january-2026-resource-policies.png) ## Tokenize sensitive data with HashiCorp Vault Conduktor Gateway now supports tokenization through HashiCorp Vault's Transform Secrets Engine. Unlike standard encryption, tokenization produces deterministic output—the same input always generates the same token. This enables analytics on protected data: GROUP BY customer IDs, match patterns across transactions, or join records across systems without exposing sensitive values. [Configure Vault tokenization →](https://docs.conduktor.io/guide/reference/data-security#vault-kms) ## Drill down to topic-level costs in Chargeback Chargeback now breaks down storage and partition costs at the topic level. Filter by cluster or application instance to identify exactly which topics drive spend, with daily or monthly aggregation. Works with Console's metadata indexer—no Conduktor Gateway required. [Use Chargeback without Gateway →](https://docs.conduktor.io/guide/use-cases/chargeback-without-gateway) ## Quality of life improvements - **Insights filtering**: Filter all Insights data by label or topic type; click any label to apply it as a filter - **Insights sorting**: Tables in risk analysis, VIP topics, and governance sections now support column sorting and search - **Topic classification**: Topic type labels shown alongside custom labels for easier identification - **Filtered exports**: CSV exports include a `-filtered` suffix when filters are applied - **Metrics auth**: Optional basic auth for metric scraping endpoints via `CDK_MONITORING_BASICAUTH_EMAIL` and `CDK_MONITORING_BASICAUTH_PASSWORD` - **Audit log redaction**: Passwords, tokens, keys, and secrets now redacted in audit logs - **Webhook secrets**: Environment variables supported in webhook alert configurations using `{{env.CDK_WEBHOOK_SECRET_*}}` syntax --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/guide/release-notes). Source: https://www.conduktor.io/product-releases/january-2026 --- # Cluster connections via CLI, shareable message URLs, and topic catalog details ## Manage cluster connections via CLI Deploy Console cluster connections using infrastructure as code with KafkaCluster, KafkaConnectCluster, and KsqlDBCluster resources. ```yaml --- apiVersion: console/v2 kind: KafkaCluster metadata: name: cloud-kafka spec: displayName: "Cloud Kafka" icon: "kafka" color: "#000000" bootstrapServers: "34.140.204.135:12092" properties: sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret"; security.protocol: SASL_SSL sasl.mechanism: PLAIN schemaRegistry: url: http://34.140.204.135/registry/ security: type: BasicAuth username: superUser password: superUser ``` See the [Console Resources Reference](https://docs.conduktor.io/platform/reference/resource-reference/console/) for full definitions. ## Generate API tokens on startup The `conduktor login` command generates an API token using admin credentials, enabling full IaC deployments without UI access. See [the docs](https://docs.conduktor.io/guide/conduktor-in-production/automate/cli-automation). ## Link directly to individual messages Individual messages have unique URLs. Share a link to a specific Kafka message for review or investigation. The standalone message page shows key, value, metadata, and headers. Switch between JSON and table view, or filter with jq. ![Shareable Message Page](https://www.conduktor.io/assets/images/releases/jul-2024-ii-1.png) ## View large messages without performance issues Messages larger than 100KB load on a separate page instead of the Consume page, avoiding browser performance issues. ## Document topics in the catalog The Topic Details page shows contextual documentation about topics. Control whether descriptions can be edited in the UI using annotations. See the [Topic Resource documentation](https://docs.conduktor.io/guide/reference/kafka-reference). ![Topic Catalog](https://www.conduktor.io/assets/images/releases/jul-2024-ii-2.png) ## View user last login dates The Settings > Users page shows a 'Last login' column. Login events are also captured in the Audit Log. ![User Last Activity](https://www.conduktor.io/assets/images/releases/jul-2024-ii-3.png) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1251). Source: https://www.conduktor.io/product-releases/july-2024-ii --- # Application groups, schema subjects in self-service, and group provisioning ## Define team access with application groups Application Groups represent teams that use your application. Set group permissions to restrict or delegate access: a support team with read-only production access, a DevOps team with privileges across environments, or developers who maintain the product. ```yaml apiVersion: self-service/v1 kind: "ApplicationGroup" metadata: application: "clickstream-app" name: "clickstream-support" spec: title: Support Clickstream description: | Members of the Support Group are allowed: Read access on all the resources Can restart owned connectors Can reset offsets permissions: - appInstance: clickstream-app-dev resourceType: TOPIC patternType: "LITERAL" name: "*" permissions: ["topicViewConfig", "topicConsume"] - appInstance: clickstream-app-dev resourceType: GROUP patternType: "LITERAL" name: "*" permissions: ["consumerGroupCreate", "consumerGroupReset", "consumerGroupView"] members: - user1@company.org - user2@company.org externalGroups: - GP-COMPANY-CLICKSTREAM-SUPPORT ``` ## Manage schema subjects through self-service Schema subjects are now supported in self-service. Application teams can automate schema deployments. ```yaml apiVersion: v1 kind: Subject metadata: cluster: shadow-it name: myPrefix.topic-value spec: schemaFile: schemas/topic.avsc format: AVRO compatibility: FORWARD_TRANSITIVE ``` ## Automate user and group provisioning Create groups and set permissions through infrastructure as code. Declare Group resources with User assignments for automated on/off-boarding. ```yaml apiVersion: iam/v2 kind: "Group" metadata: name: "devops" spec: displayName: "DevOps team" description: "Making devs happy" permissions: - resourceType: PLATFORM permissions: ["userManage"] - type: CLUSTER name: "my-cluster" permissions: ["clusterViewBroker"] - type: TOPIC cluster: "my-cluster" patternType: "LITERAL" name: "*" permissions: ["topicViewConfig", "topicConsume"] members: - bob@conduktor.io - jane@conduktor.io externalGroups: - GP-COMPANY-DEVOPS ``` ## Filter topics in the catalog by labels Filter and discover topics using business metadata labels. Request access by adding the generated snippet to a PR for owner approval. --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1240). Source: https://www.conduktor.io/product-releases/july-2024 --- # Email alerts, consumer group labels, and header transformations ## Route alerts via email Email is now available as an alerting channel. For organizations that can't use Slack or Teams due to security restrictions, email routing enables alerts through internal messaging systems. Customize email content with variables: owner, description, trigger value, and cluster. Dynamic fields make alerts readable for stakeholders unfamiliar with Kafka. ![](https://www.conduktor.io/assets/images/releases/jul-2025-1.webp) ## Define custom webhook payloads Webhook bodies are now fully customizable. Define your own structure to match third-party tool schemas or integrate with proprietary systems. ![](https://www.conduktor.io/assets/images/releases/jul-2025-2.png) ## Label consumer groups Consumer groups support labels for ownership, environment, and purpose. Labels appear on the consumer group list, inside topics, and at the member level. This is part of bringing labels to every object in Conduktor: topics, applications, service accounts, and now consumer groups. ![](https://www.conduktor.io/assets/images/releases/jul-2025-3.png) ## Edit Trust policies from the UI Edit policies and rules directly from the interface. After creating a rule, the system prompts you to configure a corresponding policy. Updated charting shows violations over time and supports threshold-based alerts. ## Require Schema IDs in messages Trust can verify that messages contain a Schema ID, preventing poison pill attacks where malformed messages crash consumers. ![](https://www.conduktor.io/assets/images/releases/jul-2025-4.png) ## Add and remove headers when sharing data Transform headers within Partner Zones: - Add metadata (timestamp, source, service account) to inbound partner messages - Strip sensitive or internal metadata before sharing externally ![](https://www.conduktor.io/assets/images/releases/jul-2025-5.png) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1360). Source: https://www.conduktor.io/product-releases/july-2025 --- # Topic management via IaC, topic policies, and topic catalog ## Manage topics as code Application teams can manage their topic lifecycle through infrastructure as code using the [Topic object](https://docs.conduktor.io/guide/reference/kafka-reference). ![Self-service](https://www.conduktor.io/assets/images/releases/jun-2024-1.png) ## Enforce topic configuration with policies Platform teams can restrict topic configurations with [Topic Policies](https://docs.conduktor.io/platform/reference/resource-reference/self-service/#topic-policy): enforce naming standards, limit partition counts, require specific replication factors, and prevent expensive configurations. ## Discover topics through the catalog The Topic Catalog shows topics associated with applications across all clusters, searchable by name and labels. See ownership and business metadata at a glance. ![Topic Catalog](https://www.conduktor.io/assets/images/releases/jun-2024-3.png) ## Customize Consume page columns Choose which columns to display in the Consume page. Show or hide metadata based on what you need. ## View Schema ID and header count per message The message viewer shows schemaId and header count for each message. ## Sort connectors by task count The Kafka Connect list can be sorted by task count. --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1230). Source: https://www.conduktor.io/product-releases/june-2024 --- # Application view, self-service policies, and topic-level lag alerts ## Find applications and see their data flows A centralized application view shows the applications you own or depend on, who's responsible, and how applications connect (two degrees of data flow visibility). Drill into any app to see Kafka resources, access rights, and recent changes. ![Application View](https://www.conduktor.io/assets/images/releases/jun-2025-1.png) ## Enforce naming conventions and permission rules Platform teams can now impose guardrails on how application teams manage resources: - Enforce naming conventions - Lock Create/Delete permissions - Require metadata and external ticket references - Prevent individual user permissions (group-based only) ![Self-Service Policies](https://www.conduktor.io/assets/images/releases/jun-2025-2.png) ## Alert on consumer group lag per topic Consumer group lag alerts can now target a specific topic. When multiple teams share the same consumer group, topic-scoped alerts pinpoint which topic is slow. ![Precision Alerting](https://www.conduktor.io/assets/images/releases/jun-2025-3.webp) ## Export user activity timestamps for PCI-DSS compliance Console exposes each user's last activity timestamp via the API. PCI-DSS Requirement 8.1.4 requires removing inactive accounts within 90 days; this enables automated cleanup workflows. ## Filter connectors by label Display and filter connectors by labels: ownership, environment, data domain. Turn a cluttered connector list into an organized, searchable view. --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1350). Source: https://www.conduktor.io/product-releases/june-2025 --- # Configuration Templates, Topic Views GA, and Stream Lineage ## Configuration Templates: Scale Teams Without Scaling Expertise Platform teams can define approved templates for topics and connectors that include the right defaults, guardrails, and naming conventions. Application teams can then provision topics or connectors directly from these templates and get a validated configuration instantly, without needing to understand every partition, retention, or connector setting. Because the rules are embedded in the template, misconfigurations are caught before deployment instead of after they cause issues in production. Platform teams spend less time handling tickets, and application teams move faster with fewer ways to get it wrong. [Explore Configuration Templates →](https://docs.conduktor.io/guide/reference/self-service-reference#topictemplate) ## Topic Views Are Now Generally Available Topic Views, introduced in preview last month, are now generally available. They let consumers work with a filtered, projected view of a Kafka topic without duplicating data or running a separate stream processing job. They are lightweight, non-materialized views over data you already have. This release also expands filtering capabilities. You can now filter on record keys, headers, and metadata in addition to values, compare fields against each other, and use type-aware matching for numeric, boolean, and time-based fields. Unsupported SQL is validated and rejected when the view is created. [Explore Topic Views →](https://docs.conduktor.io/guide/conduktor-concepts/logical-topics#topic-views) ## Stream Lineage: See Who Can Access What From Day One Stream Lineage gives you a connected map of which applications and teams can produce to and consume from topics across your Kafka environment. It is built from access data alone, giving you immediate visibility into who can access what as soon as you connect Conduktor. No live traffic or extra setup required. It also lets different business units surface relationships across their respective domains, so each team can see its own dependencies without losing the bigger picture. This static view is the first step in our lineage story, with a live view to follow in a future release. Even today, it answers a key question that comes up before migrations, access changes, and incidents: what depends on this topic? [Explore Stream Lineage →](https://docs.conduktor.io/guide/conduktor-concepts/stream-lineage) ![Stream Lineage graph showing how applications like billing-prod, checkout-prod, and reporting-prod connect to shared Kafka topics across an environment](https://www.conduktor.io/assets/images/releases/june-2026-lineage.png) ## Public API and CLI: Automate Everything, Integrate Anywhere Platform teams can now manage consumer groups and connectors directly through the API and CLI. This includes describing consumer groups, previewing and resetting offsets, stopping and restarting connectors, pausing and resuming connector tasks, and managing offsets. This brings full Kafka lifecycle automation into GitOps pipelines, Backstage, and internal tooling, so day-to-day operations no longer require opening a browser. [Browse the API and CLI reference →](https://docs.conduktor.io/guide/conduktor-in-production/automate#conduktor-automation-cli-terraform-and-rest-api) ## Chargeback for Connect: Attribute Connector Costs to the Teams That Own Them Kafka Connect can be expensive, especially with fully managed services, and those costs have been hard to attribute to specific teams. Chargeback now assigns Connect costs to the application teams that own the connectors, using a default task-hour rate per cluster with optional overrides per connector class. Connect spend now appears alongside storage, partitions, ingress, and egress across cluster, application, and label views, and in CSV exports. [Learn about Chargeback →](https://docs.conduktor.io/guide/use-cases/chargeback) ## Multi-Listener Gateway Is Now Generally Available Multiple listeners on a single Conduktor Gateway are now generally available, each with its own security protocol and network configuration. This lets you serve different access patterns through one Gateway: an internal listener with one set of policies, an external listener for partners or customers with another, all backed by the same Kafka clusters. This is now the recommended approach for complex networking setups and replaces the legacy global network configuration, which is now deprecated. [Configure Gateway Listeners →](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listeners) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/guide/release-notes). Source: https://www.conduktor.io/product-releases/june-2026 --- # Partner Zones, Terraform provider, and crypto shredding ## Share Kafka data with partners without duplication Partner Zones create isolated environments where external partners access selected Kafka topics. No topic replication or additional clusters required. Create a Partner Zone in under 60 seconds: 1. Name the zone, select a service account, and choose a Gateway 2. Select topics and set permissions (read/write) 3. Configure throughput and bandwidth limits 4. Share credentials with your partner ![Partner Zones](https://www.conduktor.io/assets/images/releases/mar-2025-1.png) Learn more on the [data sharing use case page](https://www.conduktor.io/solutions/use-case/data-sharing). ## Deploy Conduktor configuration with Terraform The Conduktor Terraform provider provisions users, groups, permissions, and policies. Deploy Conduktor alongside your infrastructure, reduce configuration drift, and automate access control updates. Self-service resources are also supported through Terraform: delegate topic creation to application teams with guardrails in place. ![Terraform](https://www.conduktor.io/assets/images/releases/mar-2025-2.gif) ## Delete encrypted data by deleting keys Crypto shredding makes encrypted data permanently unreadable by deleting the encryption key. This provides GDPR compliance for data deletion requests without expensive per-message deletion. Conduktor's approach stores user keys in Kafka, making them easy to delete when a deletion request arrives. ![Crypto Shredding](https://www.conduktor.io/assets/images/releases/mar-2025-3.png) Learn more in [this blog post](https://www.conduktor.io/blog/crypto-shredding-in-kafka-a-cost-effective-way-to-ensure-compliance). --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1320). Source: https://www.conduktor.io/product-releases/march-2025 --- # RBAC for Insights, Schema Registry Proxy (Preview), and cryptographic signing ## Schema Registry Proxy (Preview) Now in Preview, Schema Registry Proxy sits between Kafka clients and a Confluent Schema Registry, adding authentication and authorization as a transparent layer. Producers and consumers keep working with just a URL change. Authentication supports any OAuth2/OIDC provider via JWT. Authorization rules are set per subject, controlling read and write access per service account with wildcard patterns. Permissions are managed in Console and applied in real time over Kafka, so access changes take effect without a restart. ## Insights for every team, not just admins Insights now respects RBAC permissions, opening access to non-admin users for the first time. Platform teams can give developers, SREs, and team leads direct visibility into the health and cost of the topics they own. Cluster-level permissions provide full visibility across a cluster, while topic-level permissions scope results to the specific topics that user can access. [Learn about Insights →](https://docs.conduktor.io/guide/insights) ## Dedicated pages and deeper filtering for Insights Insights now splits into four dedicated pages: Overview, Risk Analysis, Cost Control, and VIP Topics. The Overview page surfaces top-level signals, with each card linking through to its detailed view. New filtering options allow scoping by application or by topic prefix pattern (e.g., `orders-*`). Combined with RBAC, each user sees a view shaped by the topics and teams they work with. [Explore Insights →](https://docs.conduktor.io/guide/insights) ![Filtering Insights by topic prefix pattern and topic type](https://www.conduktor.io/assets/images/releases/march-2026-insights-filtering.png) ## Verify data integrity with cryptographic signing A new Conduktor Gateway interceptor signs records on produce and verifies them on consume, detecting if data was modified in flight or at rest. Conduktor Gateway applies signatures transparently with no client code changes. Records that fail verification are dropped before reaching the consumer. Signing keys are managed in HashiCorp Vault with built-in versioning and rotation. [Configure integrity signing →](https://docs.conduktor.io/guide/reference/data-security#message-integrity-interceptor) ## Federated ownership improvements - **Transactional ID support**: Applications using exactly-once processing can now manage transactional ID ownership and permissions through self-service, eliminating manual admin intervention - **Assignable permissions for access requests**: Requesting and approving access is now an assignable permission scoped to each app instance, keeping the app owner as the decision-maker - **Prefixed topic access**: Teams can request access to an entire family of topics with a single request, removing friction from cross-team data sharing at scale [Learn about federated ownership →](https://docs.conduktor.io/guide/conduktor-concepts/self-service) ## Confluent Server topic configurations Initial support for Confluent Server-specific topic configurations gives Confluent customers better visibility into their cluster setup directly within Console. [Configure clusters →](https://docs.conduktor.io/guide/conduktor-in-production/admin/configure-clusters) ## Label insights (Preview) A new page under Settings shows how labels are used across your Kafka resources—coverage rates, most-used labels, and per-label breakdowns by resource type. Coverage is tracked across clusters, topics, consumer groups, connectors, applications, and service accounts. Filter by cluster or resource type to focus on specific areas. [Learn about insights →](https://docs.conduktor.io/guide/insights) ![Label insights page](https://www.conduktor.io/assets/images/releases/march-2026-label-insights.png) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/guide/release-notes). Source: https://www.conduktor.io/product-releases/march-2026 --- # Introducing Conduktor Trust: in-stream data quality enforcement Bad data (missing fields, broken formats, out-of-range values) breaks downstream systems, corrupts AI outputs, and causes customer-facing issues. Traditional tools fix data after the damage. Data contracts push responsibility to producers with fragmented enforcement. **Conduktor Trust** enforces data quality in-stream, before bad data reaches your pipelines. ## How Trust works 1. **Define rules**: Use CEL (Common Expression Language) to specify structure, completeness, and conformance 2. **Apply policies**: Attach rules to specific topics and control where enforcement applies 3. **Act on violations**: Log violations for monitoring or block bad messages instantly 4. **Track patterns**: View violations over time to identify recurring issues ## What Trust prevents - **Data quality issues reaching downstream**: Catch problems at the source - **Bad data in AI models**: Feed only trusted data to models and analytics - **Compliance gaps**: Apply consistent quality policies across teams - **Reactive cleanup work**: Fix issues before they spread --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1340). Source: https://www.conduktor.io/product-releases/may-2025 --- # Chargeback, Topic Views, and ApplicationGroup delegation ## Chargeback: see where your Kafka bill comes from Chargeback is now at v1.0, refined from earlier releases based on customer feedback into an accounting view of Kafka spend. It is built to answer the question every platform lead eventually gets: which applications drive the cost, and which teams own them. Spend rolls up by application, service account, or any label, like team, department or business unit, so reports line up with how the business is structured rather than how the infrastructure is shaped. The bigger shift in this release is that Chargeback no longer needs Conduktor Gateway to give a complete picture. Confluent Cloud customers used to see storage costs only. Now Chargeback pulls ingress and egress directly from the Confluent Metrics API, joins them with Console's storage and partition data, and maps anonymous service account usage back to the applications, teams and people who actually own it. [Learn about Chargeback →](https://docs.conduktor.io/guide/use-cases/chargeback) ![Chargeback dashboard showing Kafka spend broken down by application instance with storage, partition, ingress and egress costs](https://www.conduktor.io/assets/images/releases/may-2026-chargeback.png) ## Topic Views: filtered slices of a Kafka topic, without copying data (preview) Topic Views give consumers a filtered, projected view of a Kafka topic without duplicating or modifying the underlying data. Define a view once, and downstream teams can subscribe to exactly the slice of data they need, without spinning up new pipelines or maintaining parallel topics. The initial preview supports SQL filtering and projection on JSON records, with CEL and schema-aware formats coming next. Topic Views replace the SQL Topic Plugin and are the foundation for richer topic-level transformations going forward. [Explore Topic Views →](https://docs.conduktor.io/guide/conduktor-concepts/logical-topics#topic-views-tech-preview) ## One Conduktor Gateway, many client populations (preview) Conduktor Gateway can now expose multiple listeners on a single instance, each with its own security protocol and network configuration. That means a permissive internal listener for in-cluster apps can sit alongside a strict external listener for partners or customers, all backed by the same Kafka clusters. This release also extends multi-listener with per-listener mTLS, so client certificate requirements can be tightened on external traffic without disrupting internal services. [Configure multiple listeners →](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/multiple-listeners) ## Delegate federated ownership to the right teams Application owner groups previously held a fixed bundle of permissions. Console 1.45.0 unbundles those into individual permissions on ApplicationGroups, so platform teams can hand off specific responsibilities, like requesting access to topics, approving access requests, managing API keys or managing service accounts, to the people who should be doing them, and nothing more. Existing access is preserved automatically on upgrade, so current users keep working without interruption while platform admins fine-tune access on their own timeline. [Configure ApplicationGroup permissions →](https://docs.conduktor.io/guide/reference/self-service-reference#instance-permissions-reference) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/guide/release-notes). Source: https://www.conduktor.io/product-releases/may-2026 --- # Federated OAuth login and Confluent Cloud topic sizes ## Log into Console with existing OAuth tokens Console now acts as an OAuth 2.0 Resource Server, validating access tokens from your external OIDC Identity Provider. Users authenticated with your organization's IdP are automatically logged into Console—no separate login required. This supports organizations that require centralized authentication management and provides a seamless single sign-on experience across applications. [Configure delegated JWT authentication →](https://docs.conduktor.io/guide/conduktor-in-production/admin/user-access/delegated-jwt-auth) ## View Confluent Cloud topic storage sizes Confluent Cloud topic storage sizes (bytes) are now visible throughout Console, including in Insights. This helps teams understand storage consumption and make informed decisions about retention and cost. Requires a Confluent Cloud API key with the `MetricsViewer` role. ## Alert on license expiry Platform teams can monitor days until license expiry using the `console_license_days_until_expiry` metric and set up alerts before expiration. See the [metrics reference](https://docs.conduktor.io/guide/reference/metric-reference) for all available metrics. ## Quality of life improvements - **Insights dashboard**: Enhanced usability and navigation - **CSV exports**: Zip compression support for easier file management - **Labels filter**: Searchable dropdown for faster label selection - **Topic Catalog**: Clickable links to applications, instances, and owner groups - **Database config**: New `database.options` and `kafka_sql.database.options` for passing JDBC/Hikari options directly. [See environment variables →](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console/env-variables) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/guide/release-notes). Source: https://www.conduktor.io/product-releases/november-2025 --- # Kafka Insights, data quality observability, and Partner Zone chargeback ## Get operational recommendations from Kafka Insights Insights monitors your Kafka environment and surfaces issues that need attention: misconfigured topics, wasted resources, and business-critical datasets. Works out of the box with your existing clusters. What Insights covers: - **Risk analysis**: Identify misconfigured topics before they cause outages or data loss. [Learn more →](https://docs.conduktor.io/guide/insights/risk-analysis) - **Cost control**: Find underutilized or overprovisioned resources. [Learn more →](https://docs.conduktor.io/guide/insights/cost-control) - **VIP topics**: Protect business-critical topics with targeted alerting. [Learn more →](https://docs.conduktor.io/guide/insights/vip-topics) - **Governance**: Track schema adoption and self-service usage. [Learn more →](https://docs.conduktor.io/guide/insights/governance) Future releases will add per-topic optimization, data quality rule coverage, and encryption recommendations. ![Conduktor Insights](https://www.conduktor.io/assets/images/releases/oct-2025-1.png) ## Use Confluent Cloud Identity Pools Conduktor integrates with Confluent Cloud Identity Pools. View role bindings, ACLs, and authentication methods (OIDC or mTLS) for each service account. ## Monitor data quality violations across all topics Data quality observability evaluates all topics in your Kafka environment. Developers see which quality issues occur, how frequently, and whether they represent real problems. **Fix at the source:** Identify common issues and fix them in your code, schemas, or pipelines. Many teams resolve quality problems without enforcement. **Decide on enforcement:** For business-critical topics, understand whether enforcement would block real problems or create overhead. [Learn about data quality policies →](https://docs.conduktor.io/guide/conduktor-concepts/data-quality-policies) ![Data Quality Observability](https://www.conduktor.io/assets/images/releases/oct-2025-2.png) ## Track Partner Zone usage and revenue Organizations using Exchange to share data externally can now track usage, costs, and revenue per Partner Zone. Chargeback provides near-real-time visibility into partner traffic. Teams can visualize consumption patterns, get rollups of shared data, and calculate ROI for individual Partner Zones. [Learn about Partner Zones →](https://docs.conduktor.io/guide/conduktor-concepts/partner-zones) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/guide/release-notes#console-1-39-0). Source: https://www.conduktor.io/product-releases/october-2025 --- # Confluent Cloud RBAC, policy migration, and rule diagnostics ## Create Confluent Cloud RBAC bindings through self-service Self-service applications targeting Confluent Cloud now create native RBAC role bindings instead of Kafka ACLs. Permissions appear directly in Confluent Cloud without manual role assignments. [Learn about self-service →](https://docs.conduktor.io/guide/conduktor-concepts/self-service) ![Confluent Cloud RBAC](https://www.conduktor.io/assets/images/releases/sep-2025-1.png) ## Migrate legacy topic policies to CEL Convert legacy topic policies to CEL-based resource policies. Migration creates new policies with descriptions of their origin. [Learn about Gateway policies →](https://docs.conduktor.io/guide/conduktor-in-production/admin/gw-policies) ## See exactly where and why rules fail Rule testing now highlights errors directly in the editor with the error path and reason. Hover over the icon to see what went wrong. ![Rule validation](https://www.conduktor.io/assets/images/releases/sep-2025-2.png) ## Add custom violation messages to rules Attach custom messages to rules so violations explain themselves. When a message fails, users see why in plain language. Example: Instead of "Schema validation failed," show "This event is missing the required `user_id` field." [Learn about data quality policies →](https://docs.conduktor.io/guide/conduktor-concepts/data-quality-policies) ## Handle different decryption failure types Conduktor Gateway now distinguishes between decryption error types: - **Retryable errors** (temporary KMS or Schema Registry outage): throttled rather than failing fast - **Fatal errors** (misconfiguration): flagged clearly, not lost in retries - **Key not found** (crypto-shredding): surfaced explicitly to distinguish recoverable vs permanent inaccessibility ## Download Partner Zone CA certificates directly Download the CA certificate directly from the Partner Zone page instead of navigating multiple screens to gather connection details. [Learn about Partner Zones →](https://docs.conduktor.io/guide/conduktor-concepts/partner-zones) --- For a full list of changes, read the [complete release notes](https://docs.conduktor.io/changelog/#console-1380). Source: https://www.conduktor.io/product-releases/september-2025 --- # Conduktor ### 2026 - [Live lineage, Flink SQL, and Chargeback](https://www.conduktor.io/product-releases/august-2026) — August 2026 - [Configuration Templates, Topic Views GA, and Stream Lineage](https://www.conduktor.io/product-releases/june-2026) — June 2026 - [Chargeback, Topic Views, and ApplicationGroup delegation](https://www.conduktor.io/product-releases/may-2026) — May 2026 - [RBAC for Insights, Schema Registry Proxy (Preview), and cryptographic signing](https://www.conduktor.io/product-releases/march-2026) — March 2026 - [Community Edition and Kafka Connect governance](https://www.conduktor.io/product-releases/february-2026) — February 2026 - [Data quality insights, cluster-wide policies, and tokenization](https://www.conduktor.io/product-releases/january-2026) — January 2026 ### 2025 - [Track storage costs and query Kafka with AI](https://www.conduktor.io/product-releases/december-2025) — December 2025 - [Federated OAuth login and Confluent Cloud topic sizes](https://www.conduktor.io/product-releases/november-2025) — November 2025 - [Kafka Insights, data quality observability, and Partner Zone chargeback](https://www.conduktor.io/product-releases/october-2025) — October 2025 - [Confluent Cloud RBAC, policy migration, and rule diagnostics](https://www.conduktor.io/product-releases/september-2025) — September 2025 - [Application group lifecycle, JSON schema validation, and multi-cluster Partner Zones](https://www.conduktor.io/product-releases/august-2025) — August 2025 - [Email alerts, consumer group labels, and header transformations](https://www.conduktor.io/product-releases/july-2025) — July 2025 - [Application view, self-service policies, and topic-level lag alerts](https://www.conduktor.io/product-releases/june-2025) — June 2025 - [Introducing Conduktor Trust: in-stream data quality enforcement](https://www.conduktor.io/product-releases/may-2025) — May 2025 - [Label-based Chargeback, Confluent/Aiven service accounts, and IAM auth](https://www.conduktor.io/product-releases/april-2025) — April 2025 - [Partner Zones, Terraform provider, and crypto shredding](https://www.conduktor.io/product-releases/march-2025) — March 2025 - [Chargeback, granular alerts, and Partner Zones](https://www.conduktor.io/product-releases/january-2025) — January 2025 ### 2024 - [SQL on Kafka, cluster health dashboard, and shareable filters](https://www.conduktor.io/product-releases/december-2024) — December 2024 - [CLI for connectors, self-service Connect, and Console HA](https://www.conduktor.io/product-releases/august-2024) — August 2024 - [Cluster connections via CLI, shareable message URLs, and topic catalog details](https://www.conduktor.io/product-releases/july-2024-ii) — July 2024 - [Application groups, schema subjects in self-service, and group provisioning](https://www.conduktor.io/product-releases/july-2024) — July 2024 - [Topic management via IaC, topic policies, and topic catalog](https://www.conduktor.io/product-releases/june-2024) — June 2024 Source: https://www.conduktor.io/product-releases --- # Achieving Data Security for Kafka What's in this ebook [01 The prerequisites The three security gaps in open source Kafka](#wp-prerequisites) [02 Encrypting Kafka data In transit, at rest, and field vs message-level](#wp-encryption) [03 Encryption tradeoffs Keys, certificates, compute, and latency](#wp-encryption-notes) [04 Auth and authorization Who can connect, and what they can do](#wp-authn-authz) [05 Monitoring Kafka Audit logging, metrics, and threat detection](#wp-monitoring) [06 Readiness checklist Score your estate across five dimensions](#wp-checklist) [07 Bringing it together Defense in depth for a governed Kafka](#wp-conclusion) Executive summary Organizations trust Apache Kafka to handle massive data streams, from customer transactions to operational metrics, which may contain personally identifiable information (PII): credit card credentials, home shipping addresses, individual health records, and more. Securing that data in motion is not optional, and the penalties for getting it wrong are heavy. $9.48MAverage data breach cost in the U.S. (IBM Research, 2023) €20M / 4%GDPR maximum fine, of annual global revenue, whichever is higher 204 daysAverage time to detect a breach (IBM Research, 2023) Why streaming data is different Many data security solutions were built for batch data sitting in databases or data lakes, not the unique demands of streaming data. Securing data in motion requires both specialized tools and specialized workflows. One misconfigured topic can cost more than your entire Kafka deployment is worth. **Out of the box, open source Kafka ships with real security gaps.** This guide covers the key requirements for closing them: data encryption, access control, and security monitoring, along with the fundamental tradeoffs each involves. Part 1 lays out the three prerequisite risks every Kafka estate has to address. The gated sections that follow work through each in depth, from encryption in transit and at rest, to authentication and authorization, to audit logging and threat detection, and close with a readiness checklist you can score your own estate against. > This guide is for platform engineers, SREs, DevOps, and security teams responsible for securing data in real-time environments. If you are handling sensitive data, navigating compliance, or need visibility into who is accessing what, this is your playbook. Part 1 The prerequisites for Kafka data security Out of the box, open source Kafka has certain security vulnerabilities. These can be addressed either by building your own in-house platform or by using third-party tools, but either way the gaps are the same. Three risks account for most of them. Encryption Data flowing between a Kafka broker and connected clients is unencrypted. Adding encryption prevents attackers and unauthorized parties from intercepting information you would rather keep private. Depending on the distribution, Kafka may not encrypt data stored on disk either, which leaves stored data exposed if the underlying storage is compromised. Access controls Kafka regulates user permissions via Access Control Lists (ACLs). While ACLs are a reasonable starting point for user management, they are not without flaws: Hard to use and maintain at scale. Lack key features such as auditing, scaling, and customization. Do not always integrate well with other technologies. Struggle to meet the needs of more complex, extensive Kafka environments. Monitoring, observability, and auditing Visibility into access attempts and security posture is critical for preventing threats before they escalate. Monitoring tools help identify anomalous broker behavior and performance; observability frameworks provide deeper context into system performance and health; and audit logs go further, capturing each action from every user, which simplifies root cause analysis, compliance, and forensic investigations. The three gaps you have to close **Encryption** keeps data unreadable both on the wire and on disk, so that intercepted or exfiltrated data stays useless without the key. **Access controls** decide who can connect and what they can do, moving beyond raw ACLs to something maintainable at scale. **Monitoring and auditing** give you the trail to detect anomalies, prove compliance, and investigate incidents after the fact. Together, these three cover the majority of Kafka's security gaps and are what a safe, compliant deployment is built on. The rest of this guide takes each in turn, along with the tradeoffs that come with every choice. Part 2 Encrypting Kafka data For most Kafka users, encryption is the primary concern. Even in private networks or a virtual private cloud (VPC), Kafka deployments are vulnerable to leaks from misconfigurations, permissive policies, or unauthorized access via stolen credentials. Encrypted data, if exfiltrated, remains unreadable without the correct key. Kafka data needs encryption both in transit and at rest. At-rest encryption can occur client-side or on the broker. The key disadvantage of doing it on the broker is that the data is exposed temporarily in plaintext after in-transit decryption and before re-encryption to disk, which may not meet all requirements. Client-side encryption avoids that exposure but risks decentralized control, since clients may implement different standards, libraries, and algorithms that become difficult to track. Encrypting client traffic in middleware is another option. > Other types of metadata, such as logs or configurations, will also require additional setup for full compliance, especially if they are stored on disk. Data in transit (SSL/TLS) Secure Sockets Layer (SSL) and its successor, Transport Layer Security (TLS), secure data moving between clients and brokers. Any communication begins with a TLS handshake between parties to authenticate server identity (optionally client identity with mTLS), provide certificates to prove legitimacy, and supply specific, one-time keys. Data is then encrypted, sent from one party to the next, and decrypted upon arrival. Today, SSL is considered obsolete and insecure, and TLS is the preferred method of encryption in transit, with mTLS for the most secure setup. Depending on the client library you use, enabling TLS requires only minor changes to a broker's configuration files, along with generating and storing cryptographic certificates. Once configured correctly, unencrypted client connections are no longer permitted by the broker, so messages are protected from interception or alteration while in transit. Data at rest (storage encryption) Kafka, like many data storage tools, does not encrypt the data it stores on disk. Your first line of defense should be file system- or disk-level encryption to protect Kafka's stored data. The ideal tool depends on your broker's runtime environment or cloud provider: Linux systems typically use Linux Unified Key Setup (LUKS) for full-disk encryption, while cloud vendors offer in-house tools for encrypting block storage, such as Amazon EBS, Azure Disk Encryption, and Google Cloud. Beyond disk-level encryption, there are two forms of fine-grained encryption to consider: Message-level The entire message is encrypted. Limits Kafka's ability to filter and route messages based on their contents. Excels at protecting sensitive data that must remain confidential throughout its lifecycle. Field-level Only certain message fields containing sensitive data (such as passwords or PII) are encrypted. Leaves remaining fields (such as names or order numbers) unencrypted. Lets applications continue to use those fields to route messages to the right consumers. Field-level vs message-level: what to use and when Feature / tradeoff Field-level encryption Message-level encryption **Granularity** Encrypts only sensitive fields (e.g. PII) Encrypts the entire message payload **Routing & filtering** Preserved. Non-sensitive fields remain usable Lost. Kafka cannot inspect encrypted contents **Security strength** Moderate (selective obfuscation) High (all content hidden from intermediaries) **Performance overhead** Lower. Less data to encrypt Higher. Entire payload to encrypt **Use case fit** Real-time processing, analytics, conditional routing Highly sensitive data (health, finance, compliance) **Searchability & indexing** Easier to meet data minimization requirements Disabled, entire content is opaque **Compliance flexibility** Selective control over what is exposed Easier to guarantee data non-disclosure **Operational impact** Low disruption, more interoperable Higher complexity, limited middleware visibility Part 3 Other considerations for encryption Beyond simply where and when to encrypt data, there are some unexpected logistical concerns. These are the tradeoffs that shape whether an encryption strategy holds up in production. Compute consumption Encryption and decryption require processing power in the form of additional CPU cycles, adding performance overhead. Architects must account for this performance impact on their deployments. Securely storing and using encryption keys Encrypted data is worthless if attackers can also steal the associated keys. Any security solution will therefore have to be compatible with a key management service (KMS), which helps automate the generation, validation, and storage of encryption keys. Deleting encryption keys Because of legislation like the GDPR's right to be forgotten, data needs to be removed upon request. One effective way to do this is to make encrypted data permanently inaccessible by deleting the encryption key associated with it. This is known as crypto shredding: once a key is deleted, any data encrypted with it becomes unreadable instantly across all systems, avoiding the slow and costly process of removing all the PII from downstream systems. Certificate management Because TLS certificates periodically expire, they have to be replaced. New certificates and keys can be generated, deployed to Kafka brokers and clients, and tested for reliability. Encryption-induced lag Field- and message-level encryption can introduce processing delays, negatively impacting low-latency use cases like fraud detection or real-time order routing. While horizontal scaling (adding partitions) might seem like a solution, encryption inherently consumes resources, leading to unavoidable lag. Teams must test their encryption under peak load conditions and monitor lag in production. > There is no right answer for everyone. Choosing the right solution for encryption at rest depends on where your data is stored. Teams with on-premise Kafka may want to use their operating system's full-disk encryption; teams with cloud-based infrastructure may instead rely on their storage types and the built-in encryption from their cloud providers. Part 4 Authentication and authorization in Kafka While encryption protects against unwanted access by outsiders, it does not limit access to your Kafka broker. The next important step in securing Kafka is controlling who can connect (authentication) and what connected clients can do (authorization). Authentication Authentication determines whether the client connecting to a broker is who it claims to be. The two most common ways of adding authentication to Kafka are mutual TLS (mTLS) and Simple Authentication and Security Layer (SASL). mTLS An extension of TLS encryption where both the client and server present certificates to verify their identities, rather than just the server. Signing certificates must be obtained from a root certification authority trusted by both client and server. The broker's TLS config is updated to request client certificates; clients must supply one signed by a trusted authority. During the handshake, both sides validate each other's certificates, creating two-way trust. SASL Integrates Kafka with authentication frameworks like Kerberos and LDAP, and also enables username and password authentication. When a broker is configured to use SASL, connecting clients must supply valid credentials. Producers and consumers must specify their SASL mechanism (like PLAIN, SCRAM, or GSSAPI) and credentials via configuration properties. The process varies by mechanism but is well documented. Note that Kafka can use TLS and SASL simultaneously, providing two independent security checks. Clients need both a valid certificate and correct credentials. If an unauthorized party acquires a client certificate, access is still denied without the correct SASL credentials. Authorization Kafka ships with Access Control Lists (ACLs) to manage basic authorization needs. While ACLs are useful, most large-scale Kafka deployments will require role-based access control (RBAC) to simplify permission management. RBAC provides a structured approach to authorization by organizing around roles, which can be granted specific permissions. Platforms like Conduktor add fine-grained RBAC authorization to Kafka without requiring your DevOps team to build custom solutions or manually sync policies across your infrastructure. This makes RBAC worthwhile to most teams, unless they are running smaller or relatively simple Kafka environments. Whether you use ACLs or RBAC, a few key concepts apply: **Users** represent authenticated identities. They can be individuals, applications, or services, and can be granted specific permissions through ACLs or role assignments. **Groups** allow related users to be managed collectively. Assigning multiple users to a group simplifies permission changes at scale. For example, a "DataScienceTeam" group could have read-only access to certain topics, ensuring every team member inherits those permissions. **Resources** represent Kafka entities like topics, consumer groups, and configuration settings. Controlling access to resources enables fine-grained authorization: certain roles might only need to read from specific topics, while others might be permitted to create or delete them. RBAC policies combine these concepts to link users (or groups) to resources, defining which actions they can perform. A "Producer" role might have write-only access to certain topics, while a "Viewer" role has read-only access. When a new developer joins the data science team, assigning them to the "DataScienceTeam" group grants the permissions they require. While it is possible to implement your own Kafka authorization, this requires significant time and effort to build and maintain; tools like Conduktor make it easy to manage users and simplify group management. The full authentication and authorization flow To recap, clients authenticate using mTLS, SASL, or both. Then the broker applies RBAC policies to grant or deny access to specific topics and actions. TLS encrypts data transmitted over the network, and encryption at rest protects data stored on disk. **Client connects.** A producer or consumer initiates a connection to the Kafka broker. **Authentication.** The client presents credentials via SASL or a certificate via mTLS. **Broker verification.** The broker validates identity and checks for certificate trust. **Authorization.** The broker applies ACL or RBAC rules to permit or deny topic access. **Access granted or denied.** The outcome is decided based on credentials and access policies. Part 5 Monitoring Kafka for security and compliance Monitoring and auditing are other key security operations. While Kafka does not include comprehensive audit tooling, it can produce detailed logs that can be imported by external systems for analysis and alerting. Audit logging Kafka uses Log4j to record security-related events. When its audit loggers are enabled, they capture detailed information on authentication attempts, authorization outcomes, and access to broker resources. These logs can be sent to any SIEM or log aggregator that supports Log4j. With proper configuration and the right parsing tools, teams can spot issues like repeated authorization failures or unauthorized topic access in near real-time. Combined with existing security tooling, this creates a reliable audit trail for compliance and threat detection. Monitoring tools and metrics Kafka exposes operational metrics via Java Management Extensions (JMX), which can be integrated with performance monitoring tools like Prometheus and Grafana. Metrics such as throughput, latency, and partition usage help identify both performance and potential security issues. For example, unusual spikes in errors or drops in consumer activity may point to misconfigurations or unauthorized access. Combined with your audit logs, these metrics provide a complete view of both the performance and security of your Kafka cluster. A complete security monitoring stack layers four components, each feeding the one above it: Layer What it does **Dashboards & alerts** Visualizes metrics, sets up alert thresholds, and surfaces suspicious behavior in real-time. **Log aggregator / SIEM** Parses and stores logs, correlates events, and enables threat detection and compliance checks. **Log framework (Log4j)** Captures structured audit logs from Kafka, including user actions and system events. **Kafka broker** Generates security-relevant logs: authentication attempts, authorization checks, topic access. Part 6 Kafka security readiness checklist Use this checklist to assess the security readiness of your Kafka environment across encryption, authentication, authorization, monitoring, and governance. Score each item: fully implemented and monitored (2 points), partially implemented or inconsistently applied (1 point), not implemented or unknown (0 points). Encryption Data in transit (TLS) TLS enforced between all Kafka brokers and clients Valid CA-signed certificates in place for all endpoints Mutual TLS (mTLS) configured (if using client auth via certificates) Certificate expiration dates tracked and rotation process documented Data at rest (disk encryption) Broker logs and topic data encrypted at disk level (LUKS, BitLocker, or cloud-native) Kafka metadata (e.g. ZooKeeper or KRaft configs) also encrypted OS-level encryption monitored and validated regularly Application-level encryption Field-level encryption implemented for PII, credentials, and sensitive data Message-level encryption applied where routing/filtering isn't needed Encryption keys stored securely in a KMS or HSM Key rotation policies defined and tested Crypto-shredding or key-based deletion supported to comply with data erasure requirements Data masking or redaction enabled in user-facing tools Authentication For brokers and clients SASL authentication enforced (PLAIN, SCRAM, GSSAPI, or OAuth) Mutual TLS enabled where applicable Authentication logs monitored for failed login attempts Expired credentials automatically reviewed and revoked Certificate management TLS cert renewal process documented and automated Broker restarts scheduled for cert reloads Certificate storage secured and access-controlled Authorization Access control policies ACLs or RBAC enforced across all Kafka resources (topics, consumer groups, etc.) Principle of least privilege enforced across producers, consumers, and admins Group-based access (e.g. "DataScienceTeam") used to simplify role management Access policies are centrally visible RBAC Roles defined for producers, consumers, admins, and viewers Role assignments tracked and reviewed quarterly Changes to role assignments logged and auditable Monitoring and auditing Audit logging Authentication and authorization events captured in audit logs Logs shipped to a SIEM or log aggregator (e.g. Splunk, ELK, or Datadog) Alerts configured for suspicious activity (e.g. repeated failed logins, unauthorized topic access) Audit logs retained in compliance with data governance policies Observability and metrics Kafka JMX metrics collected via Prometheus or similar Dashboards exist for broker health, topic throughput, and consumer lag Alerts in place for error spikes, abnormal latencies, or dropped connections Real-time anomaly detection to flag sudden behavior changes Governance and compliance Policies and documentation Data classification policies define what data types require encryption or access controls Kafka security policies formally documented and shared across teams Compliance obligations (e.g. GDPR, HIPAA, PCI DSS) mapped to Kafka security controls Regular audits scheduled to validate controls against policies Resilience Disaster recovery plan documented and tested Kafka failover architecture validated (e.g. multi-AZ or multi-region) Switch between Kafka clusters during outages automated Backup and restore procedures tested regularly Chaos engineering used to simulate failure scenarios Conclusion Bringing it together Securing Kafka is not optional, but it does not have to be complex. Multiple layers are involved: encryption in transit and at rest, strong authentication, fine-grained authorization, continuous monitoring, and audit logging. None of them alone is sufficient. Together, they provide defense in depth and ensure compliance with both organizational policy and data protection laws. Kafka does not include these capabilities by default. Most teams rely on a combination of manual configuration, custom tooling, and third-party integrations to fill the gaps. While this can work, it introduces operational overhead and does not scale easily. As Kafka becomes more central to sensitive, regulated, or real-time environments, standardizing its security becomes critical: teams need to centralize policy enforcement, automate auditing, and reduce blind spots in access and usage. These are no longer optional; they are part of what it means to run Kafka in production. Streamlining Kafka security with Conduktor Securing Kafka involves many key activities, including encryption, RBAC, audit logging, and monitoring. Managing them independently can be time-consuming and error-prone. Conduktor offers a consolidated platform that brings these security and compliance functions together in a few important ways. Centralized management A single interface for managing user authentication and RBAC policies. Makes it easier to create secure Kafka infrastructure and reduces the chance of misconfiguration. Standardized encryption Conduktor Shield provides both field- and message-level encryption applied to sensitive payloads before they reach the broker. Removes a potential vulnerability, eliminates per-client encryption logic, and keeps data protected across its lifecycle. Resilience and recovery Test Kafka infrastructure in simulated production environments to stay robust under real-world conditions. Fail over non-performing clusters so data remains accessible even if specific clusters go down. Compliance and governance Detailed audit trails and access controls, with monitoring and alerting that make it simple to demonstrate compliance. Unified visibility helps identify and resolve security incidents from a central dashboard, instead of piecing together scattered logs and metrics. Conduktor brings these layers together in one platform, enabling teams to secure Kafka without slowing it down. From encryption management to RBAC, real-time alerts, or data masking, it gives you a governed Kafka that is ready for scale, and frees DevOps teams from micromanaging Kafka security and compliance so they can focus on solving problems for your customers. Related resources [Governing Kafka Data for the AI Era](https://www.conduktor.io/resources/ebooks/governing-kafka-data-for-the-ai-era) [Federating Data Governance](https://www.conduktor.io/resources/ebooks/federating-data-governance-scaling-kafka-without-losing-control) [Conduktor Gateway](https://www.conduktor.io/gateway) Ready to secure Kafka without slowing it down? See how Conduktor brings encryption, RBAC, audit logging, and monitoring together in a single governed platform. Book a demo and we will walk through securing your estate end to end. [Book a demo](https://www.conduktor.io/contact/demo?src=resources-ebooks-achieving-data-security-for-kafka) Get the full ebook Source: https://www.conduktor.io/resources/ebooks/achieving-data-security-for-kafka --- # AI Can't Wait: Real-Time Data Quality What's in this ebook [01 The stakes are higher How AI raises the cost of bad data](#wp-stakes) [02 Data quality challenges What undermines modern AI pipelines](#wp-challenges) [03 Enforcing at scale Dead letter queues and shifting left](#wp-enforce) [04 An AI-ready strategy A six-step enforcement framework](#wp-strategy) [05 The business payoff What upstream quality returns](#wp-payoff) Executive summary The evolution of AI into increasingly autonomous, context-aware systems is forcing organizations to rethink how they govern and validate data. No matter how powerful your models are, poor data leads to poor outcomes. Hallucinations, biased decisions, regulatory violations, and degraded user experiences are just a few of the high-cost consequences when low-quality data enters AI pipelines. These risks are amplified in today's real-time, distributed environments. Unlike traditional batch systems, which ingest historical data at preset intervals, modern AI architectures (event-driven microservices, autonomous agents) act on streaming data the moment it arrives. In batch systems, you have hours or days to find and fix problems. In real-time AI systems, data quality issues must be resolved in milliseconds. There is no room for ambiguity, delay, or inconsistency. What the research shows In a 2025 Confluent survey of more than 4,000 technology executives, 68% cited data quality inconsistencies as a major challenge, while 67% struggled with uncertainty around data timeliness and trustworthiness. As pressure mounts to operationalize AI across the business, these issues have become a critical barrier to growth. **This guide offers a strategic blueprint for overcoming that barrier.** With AI systems making decisions in real time, enforcement of data quality must occur upstream, directly within the streaming infrastructure. That means shifting from passive monitoring to proactive validation, at the exact point where data enters your environment. For organizations scaling RAG, LLMs, or agentic systems, this approach provides the confidence and control required to move fast without breaking trust. **In short: better data beats bigger models.** Part 1 makes the case for why the cost of bad data rises as AI takes over more decisions. The sections that follow map the common quality challenges in AI pipelines, the enforcement approaches that work at production scale, a six-step strategy for building an AI-ready data foundation, and the business payoff of shifting quality left. Part 1 The stakes are higher: how AI raises the cost of bad data > "Garbage in, disaster out" is Conduktor CPTO Stephane Derosiaux's spin on the old maxim "garbage in, garbage out." As AI systems become more autonomous and embedded in real-time decision-making, the impact of poor quality data compounds rapidly. **Small errors at the point of ingestion can now cascade across entire architectures**, triggering faulty predictions, broken automations, and major downstream consequences. This is especially true in agentic and retrieval-augmented generation (RAG) pipelines. These systems depend on accurate, timely, and context-rich data to reason, plan, and act. When inputs are incomplete, inconsistent, or out of range, models can drift, hallucinate, and even make non-compliant decisions. The consequences are often invisible at first, but the effects surface quickly in the form of customer churn, degraded model performance, and regulatory fines. Real-time streaming systems increase this sensitivity even further. Because bad data moves fast, and there is no buffer to catch it, these systems require trust in the moment. The 1-10-100 rule captures this well: every dollar spent fixing an issue at the point of ingestion can save ten during transformation and a hundred at the point of consumption. $1to fix at the point of ingestion $10to fix during transformation $100to fix at the point of consumption Real-world example A publicly traded games technology company lost an estimated $110 million in revenue after ingesting corrupted data that degraded the accuracy of the audience-targeting product game developers relied on to monetize their apps. When AI decisions fuel revenue streams, data quality is not a background concern. It is a frontline risk. At the same time, the AI model landscape is leveling. Access to foundational models is more democratized than ever, with open-source options and APIs reducing the technical gap between competitors. In this environment, the differentiator isn't the model, it's the data feeding it. Organizations with better data pipelines, higher trust thresholds, and more consistent enforcement will deliver more accurate, compliant, and explainable results. **Better data unlocks your competitive edge.** Why the cost of bad data is rising **Autonomy removes the safety net.** As AI makes and supports decisions automatically, a small error at ingestion cascades across the whole architecture before anyone can intervene. **Real time removes the buffer.** Batch systems give you hours to catch problems. Streaming systems require trust in the moment, so quality has to be enforced as data arrives. **The model is no longer the moat.** With foundational models commoditized, the data feeding them is the competitive edge. Better data unlocks it. Part 2 Common data quality challenges in AI pipelines Real-time AI systems demand real-time trust. This means more than just accuracy. It means data that is: **Governed**, with clear ownership and policy enforcement. **Consistent**, across formats, schemas, and domains. **Auditable**, so every field can be traced back to its source. **Trusted** at the point of ingestion, not just after it lands in storage. Yet most organizations still rely on downstream data quality checks: batch validation jobs, SQL-based assertions, or dashboards that alert teams hours or days after a problem has already propagated. Without upstream enforcement, data issues become entrenched, harder to detect, and more expensive to fix. The most common issues that undermine modern AI Issue Consequence for AI **Inaccurate metadata** Breaks temporal context and disrupts the sequencing of agentic workflows. **Broken or missing schemas, invalid messages, inconsistent payloads** Cause retrieval and prompt failures, especially in RAG and LLM pipelines. **Missing fields** Lead to hallucinations, misrouted decisions, or failed automations. **Inconsistent formats or units** Degrade model performance and increase the risk of drift. **Duplicate records** Reinforce bias patterns and inflate perceived signal strength. **Out-of-range values** Trigger erratic behaviors or invalid inferences. **Unclassified or missing PII** Increases exposure to compliance violations. **Data quality debt** When left unresolved, compounds over time and drags down both model accuracy and engineering velocity. **Log-and-skip or crash-and-retry patterns** May seem harmless at first, but quickly become brittle and unscalable, eroding trust in model outputs. Left unchecked, these issues flow straight through to the business. The same quality problems that break context in agents, fail prompt inputs, reinforce biases, and produce hallucinations and model drift surface downstream as three familiar outcomes: **customer churn, lost revenue, and a compliance nightmare.** The challenge of decentralized teams Streaming-native environments like Kafka offer power and flexibility. But too often, that comes at the cost of consistency. When teams are left to enforce quality individually, controls become fragmented. One team may validate headers; another may skip them entirely. Some pipelines may have rigorous schema checks, while others rely on tribal knowledge or unlogged assumptions. The result is a patchwork of protections that make it **difficult to see what data is trustworthy, where violations occur, or who is responsible.** Without centralized enforcement, platform teams are forced to react to issues after the fact. And in AI pipelines, late detection almost always means real-world consequences. Why batch-based no longer works Traditional approaches to data quality (like enforcing rules between bronze and silver tiers in a medallion architecture) were never designed for real-time decision-making. They **introduce latency, assume schema rigidity, and evaluate data too far downstream.** By the time a violation is detected, the bad data has already landed, been processed, and potentially been used by AI systems. These workflows also struggle to handle schemaless or semi-structured data, which is increasingly common in streaming pipelines. Waiting for batch windows or relying on periodic SQL checks simply can't provide the protection or confidence AI systems require. > To keep pace with AI, data quality must move upstream. Validation must happen in-stream, the moment data is produced, regardless of schema or format. Anything less introduces unacceptable risk. Part 3 Enforcing Kafka data quality at production scale Kafka was designed for flexibility and speed. It allows producers to write nearly anything into a topic, structured or unstructured, clean or malformed. This design choice is what makes Kafka so powerful, but it's also what makes it risky for AI pipelines. Without controls at the point of ingestion, poor-quality data can flood your system before anyone has a chance to intervene. At production scale, quality failures of any size and origin impact teams and tools. Some may be minor, stemming from random invalid messages or records. Larger ones may be the unintended results of routine actions, such as missing schemas or misformatted messages. Small, random errors Slowly and subtly skew AI pipelines over time. Create drift and reduce accuracy. Hard to notice until model performance degrades. Larger, structural errors More likely to break downstream pipelines outright. Can take down entire AI workflows. Often the byproduct of routine actions like schema changes. One solution is a dead letter queue (DLQ). Instead of blocking all data because of some quality issues, DLQs allow compliant data to continue flowing downstream, while rerouting non-compliant data to a queue for review and reconstruction. Once fixed, DLQs then release the repaired data for reprocessing. Consider "shifting left" for AI-ready data Another strategy is shifting data quality left, upstream and closer to the source. By validating data as it enters the stream, this approach easily supports high-volume, real-time systems that rely heavily on the speed of ingestion and the timeliness of data. This includes schema validation, metadata checks, field enforcement, and business-specific rules, such as age ranges, email formats, or PII tags. By shifting left, teams can run these checks in-stream, before the data is stored or used by downstream applications. Instead of waiting for an alert to ping, pulling up a dashboard, and reacting to a violation, teams can automate these actions, removing humans from the loop and freeing up systems to respond more quickly, efficiently, and at any day or hour. In fact, this speed is what enables confidence in RAG pipelines, agentic systems, and live AI products where freshness and accuracy are non-negotiable. By embedding quality into the stream itself, organizations can prevent drift, reduce retraining, and **build reliable, reusable data products** that work across teams, partners, and platforms. Part 4 Building an AI-ready data strategy Delivering trustworthy, real-time AI outcomes requires more than reactive monitoring or occasional cleanup. It demands a deliberate, upstream approach to data quality that aligns technical enforcement with business risk and opportunity. This is how leading organizations are preparing for the next wave of AI. **With this foundation, better data becomes a force multiplier for every AI initiative.** Your teams will build faster, with greater confidence, and fewer surprises. 1. Audit current pipelines Start by identifying where real-time decisions are being made across the organization. This step is about mapping the critical dependencies between streaming data, AI models, and business operations. The goal is to uncover which pipelines directly influence model outputs, automation logic, customer experiences, or compliance-sensitive decisions. A thorough audit should combine technical inspection with operational context. **Inventory AI and ML use cases.** List every AI system that uses real-time or near-real-time data: recommendation engines, RAG pipelines, agentic systems or co-pilots, fraud detection models, predictive maintenance or operational forecasting tools. Include models in production as well as those in staging or active experimentation. Capture which use cases are mission-critical, customer-facing, or compliance-sensitive. **Map the streaming stack.** Identify Kafka topics, microservices, data ingestion jobs, and connectors associated with each use case. Document how data flows from source systems (IoT devices, APIs, event producers) into Kafka and onward to consuming systems. Highlight whether data is raw or transformed, and whether enforcement policies are currently applied at any stage. **Trace data-to-decision flows.** Pinpoint which models trigger user actions, financial outcomes, compliance responses, or customer experiences, anything where AI systems make or support decisions automatically. Map upstream data flows to downstream decision points so you can quantify the potential business impact of poor quality. **Assess pipeline complexity.** Note areas where ownership is unclear or distributed across multiple teams. Flag pipelines that include schemaless or semi-structured data, which may lack guardrails. Highlight high-velocity or high-volume pipelines, which carry greater operational and reputational risk. 2. Identify quality gaps Once your real-time pipelines and AI dependencies are mapped, the next step is to identify where quality enforcement is missing, inconsistent, or ineffective. This is where many organizations uncover a harsh reality: the tools and techniques they've relied on in batch environments do not translate to modern, distributed AI systems. The objective is to surface the specific weak points in your architecture that allow poor data to flow unchecked into critical decision systems. Batch jobs Validation occurs too late, after ingestion into Kafka or after downstream use. Quality checks only applied during nightly or hourly batch processes. Data is "cleaned" post-hoc rather than blocked at the source. Schemaless data Topics accepting JSON or loosely structured data without schema enforcement. Inconsistent fields or unexpected data types. Bypassed validation due to a lack of contracts or assumptions of flexibility. Violation patterns Frequent null or missing fields. Out-of-range values or inconsistent formats. Duplicated events or messages across partitions. Silent failures in prompt construction or retrieval pipelines. Alerting and ownership Quality issues detected only through dashboards or periodic reports. Retry logic used without root cause evaluation. No real-time alerts for policy violations or broken data contracts. No clearly defined owner for data quality at the Kafka topic level. Not all pipelines require the same level of enforcement. Focus first on those that feed agentic AI systems, RAG workflows, or real-time personalization engines. These systems are particularly sensitive to issues like missing metadata, inconsistent formats, or invalid references. Fixing quality here improves accuracy, reduces hallucinations, and decreases the need for costly retraining or intervention. Target domains that influence revenue, compliance, or user trust. 3. Engage platform and AI teams early Data quality enforcement is a shared operational responsibility. Platform engineers understand how and where to apply validation logic efficiently. More importantly, AI teams know what quality expectations their use cases rely on, and can help define the quality rules to enforce. The earlier these groups collaborate, the more effective your enforcement strategy will be. Begin by identifying key stakeholders across data platform, ML engineering, MLOps, and application teams. Host a working session focused on a shared goal: defining what "trusted data" means in the context of your AI systems. This should include: **What quality attributes** matter most (freshness, completeness, PII tagging, referential integrity). **Where enforcement** makes the biggest impact (on ingress to Kafka, before model scoring, before triggering agents). **What actions** should be taken when violations occur (block, alert, enrich, route). **How to escalate, monitor, and refine** enforcement rules over time. Ensure each team understands both the technical levers and the model performance consequences. This alignment helps avoid blind spots, for example, when platform teams assume data is usable if it's "valid JSON," or when AI teams assume data has already been cleaned upstream. Finally, provide both groups with access to a shared observability layer. When platform and AI teams can jointly monitor policy violations and enforcement outcomes in real time, they can collaborate more effectively and iterate faster. 4. Implement shift-left enforcement Catch bad data before it enters your system by validating at the point of ingestion. Shift-left enforcement involves moving checks upstream, directly into Kafka streams, so that poor-quality data is blocked, flagged, or enriched in real time. Start by identifying which Kafka topics feed your AI and ML systems, then define and enforce quality rules such as: Schema compliance (for example, Avro). Required fields and format checks. Valid value ranges. PII detection or structural integrity (headers, payload shape). Choose enforcement actions based on severity. Critical violations should be blocked outright; others can trigger alerts or be routed for enrichment. Start with high-impact streams and scale from there. The goal is to create a pipeline that filters out risk before it reaches your models. 5. Adopt a data mesh mindset As real-time systems scale, centralized quality control becomes a bottleneck. A data mesh approach assigns ownership to the teams closest to the data while maintaining platform-wide consistency. Define clear ownership of Kafka topics by domain. Empower domain teams to write and manage validation rules using a shared enforcement platform. This gives them control over: Schema definitions and required fields. Business-specific logic and constraints. Localized decisions on enforcement actions. Meanwhile, governance teams maintain shared policy templates, monitor violations across domains, and ensure alignment with compliance standards. Platform teams follow centralized guidelines, applying rules to their Kafka topics to address the root causes of the data quality issues triggering alerts. This model promotes autonomy without sacrificing control. Each team is accountable for the trustworthiness of the data it produces, creating stronger, cleaner streams across the board. 6. Monitor outcomes To sustain progress and prove value, data quality efforts must be tracked just like any other core capability. This means going beyond basic error counts or uptime metrics and measuring how quality enforcement impacts AI system performance, engineering velocity, and business results. Start by establishing baseline metrics before implementation, then continuously monitor how upstream enforcement changes key indicators over time. Technical performance Operational efficiency Business impact Reduction in data quality violations over time Time saved debugging or triaging downstream data issues Cost savings from reduced remediation and compliance effort Volume and type of blocked, flagged, or enriched messages Decrease in retraining frequency or emergency patching Improved decision quality in AI-driven workflows Model accuracy improvements (precision, recall, F1 score) Mean time to resolution (MTTR) for data-related incidents Reduced customer churn or error-induced fallout Reduction in false positives or hallucinations Increased confidence among application teams consuming Kafka topics Higher adoption of AI features across the enterprise Fewer retry cycles or failed downstream jobs Greater reuse of data products across teams or use cases Improved auditability and regulatory alignment Fewer data pipeline incidents and support escalations More consistent delivery of AI initiatives on budget and schedule Regular reporting across these dimensions establishes credibility with stakeholders and reinforces the business case for upstream, shift-left enforcement. Over time, these metrics help data leaders refine their quality standards, close remaining gaps, and expand best practices across the organization. Conclusion The business payoff Shifting data quality left, especially in real-time pipelines, produces measurable returns. Instead of chasing issues downstream or wasting time on fragile fixes, teams can proactively deliver value with data they trust. **Quality enforcement at the source turns data from a risk into a strategic asset.** Organizations that prioritize upstream quality are better positioned to scale AI safely and efficiently. They reduce operational drag, increase confidence in model outputs, and accelerate delivery across teams. Your teams can unlock a wide range of competitive advantages. Model and product impact Increased model accuracy, precision, and relevance. Fewer hallucinations or retrieval errors in RAG and agentic systems. Stronger alignment between user inputs and AI responses. Greater confidence in outputs, from end users to executives. Customer and compliance benefits Lower risk of customer-facing errors or broken automations. Faster resolution of data-related incidents. Improved auditability and compliance with internal and external policies. Fewer data silos and blind spots that slow down delivery. Operational and strategic ROI **Faster experimentation.** AI experimentation cycles run faster with fewer retrainings. **Reusable data products.** More trusted, reusable data products across teams and domains. **Proven cost savings.** Reduced remediation, downtime, and manual QA. **Stronger partnerships and adoption.** Confidence in shared streams strengthens data partnerships and drives higher adoption of AI capabilities across the business. In practice: a national postal service A major national postal service, delivering over a billion letters and hundreds of millions of parcels a year, built predictive models that ingest real-time Kafka data. By adopting Conduktor, it improved data governance and quality while building a foundation for AI readiness. Today, nearly one-third of its data scientists use Conduktor to explore, find, validate, and monitor data for everything from analytics to AI. Related resources [Governing Kafka Data for the AI Era](https://www.conduktor.io/resources/ebooks/governing-kafka-data-for-the-ai-era) [The 2025 Data Streaming & AI Report](https://www.conduktor.io/resources/ebooks/the-2025-data-streaming-ai-survey-report) [Achieving Data Security for Kafka](https://www.conduktor.io/resources/ebooks/achieving-data-security-for-kafka) Enforce data quality where it matters, at the source. See how Conduktor validates, blocks, and routes data in-stream, so your RAG, LLM, and agentic systems act on data you can trust in the moment. [Book a demo](https://www.conduktor.io/contact/demo?src=resources-ebooks-ai-can-t-wait-why-real-time-systems-demand-upstream-data-quality) Get the full ebook Source: https://www.conduktor.io/resources/ebooks/ai-can-t-wait-why-real-time-systems-demand-upstream-data-quality --- # AWS MSK: Govern, Secure & Scale Kafka What's in this brief [01 What Conduktor changes Impact by role, measured results, and what AWS MSK delivers vs. what Conduktor adds](#wp-changes) [02 What Gateway adds Six capabilities enforced at the protocol level, no code changes](#wp-gateway) [03 What Console adds Visibility, ownership, and cost attribution on top of AWS tooling](#wp-console) [04 Built for AWS Native integrations across AWS IAM, KMS, S3, CloudWatch, ECS/EKS, and Marketplace](#wp-ecosystem) Executive summary AWS MSK handles the infrastructure. It provisions, patches, and scales your Kafka brokers so your team doesn't have to. But the layer above the brokers, where encryption, governance, resilience, and multi-tenancy live, is still yours to build. Conduktor is the enterprise data and control plane for Apache Kafka. It sits between your applications and your brokers and adds encryption, governance, and resilience at the protocol level. It's trusted by hundreds of organizations, including Fortune 500 financial institutions, airlines, and SaaS platforms. Whether you're already running on AWS MSK or migrating to it, Conduktor Gateway is platform-agnostic, which reduces risk by giving you a consistent layer of control across mixed Kafka and cloud environments. This brief covers what changes when you add Conduktor to AWS MSK, the six Gateway capabilities that complement the brokers, how Console gives teams visibility and ownership, and the native AWS integrations that make it fit your existing infrastructure. Part 1 What changes when you add Conduktor Conduktor Gateway intercepts all Kafka traffic at the protocol level. Applications connect the same way they always have, just to a different address. Nothing in your application code changes, but a great deal changes in what your platform can guarantee. Impact by role For leadership For platform teams For development teams Regulated workloads move to production instead of stalling in compliance reviews. Encryption, access, and data quality are enforced once, not rebuilt per project. Same Kafka clients, same code, same workflows, with no application changes. Kafka investment scales across business units without unclear infrastructure growth. DR failover becomes routine, not a crisis coordination exercise. Isolated environments on demand, without waiting for dedicated infrastructure. AWS MSK migrations accelerate without re-architecting applications. Guardrails catch misconfigurations before they cause outages. Bad data is rejected at the source before it cascades to downstream systems. Measured results Based on results reported by Conduktor customers running on AWS MSK. Result What drives it $500K+ first-year value Consolidation, faster migration, and reduced operational overhead 20–40% lower infrastructure cost Virtual clusters and consolidation that eliminate cluster sprawl Up to 95% faster recovery Single-command failover instead of manual coordination across teams 4,000+ virtual clusters Running on standard infrastructure What AWS MSK delivers vs. what Conduktor adds What AWS MSK delivers What Conduktor adds Encryption at rest and in transit Field-level encryption and tokenization inside messages AWS IAM authentication and Kafka ACLs Application-level access controls with per-consumer visibility Single-region high availability Multi-region failover in seconds, with no application changes One cluster per environment Virtual clusters for multi-tenancy on shared infrastructure Cluster-level billing with tags Per-topic and per-team cost attribution with self-service guardrails The short version **AWS MSK handles infrastructure; Conduktor handles the enterprise layer.** Brokers are managed for you. Encryption within messages, governance, resilience, and multi-tenancy are what Conduktor adds on top. **Nothing changes in your applications.** Clients connect to the same Kafka they always did, just through Conduktor, so adoption doesn't require a rewrite. **The results are measurable.** Customers report 20–40% lower infrastructure cost, up to 95% faster recovery, and $500K+ in first-year value. Part 2 How Conduktor Gateway complements AWS MSK Gateway speaks the Kafka protocol natively, sitting between your applications and AWS MSK brokers. It intercepts and governs traffic at the protocol level, so every capability below is enforced centrally with no application code changes. 1. Field-level data protection AWS MSK encrypts data at rest and in transit. But any service with topic access reads the full payload, including PII and financial records. **Gateway adds** field-level encryption, tokenization, and crypto shredding, with per-consumer decryption controls and native AWS KMS integration. Encrypt customer.ssn while customer.region stays readable, and one policy at the gateway replaces a custom crypto implementation in every project. 2. Real-time data quality A single malformed message can crash consumers, break pipelines, and spread incorrect results before anyone notices. **Gateway adds** protocol-level validation before messages enter AWS MSK, enforcing schema compliance and business rules with the ability to block, route, or flag bad data. Adoption is gradual: start by monitoring, then move to blocking, with no big-bang enforcement. 3. Client best practices Kafka's native quotas cover throughput but can't enforce acknowledgments, compression, or commit frequency, which are the most dangerous misconfigurations at scale. **Gateway adds** protocol-level guardrails that block unsafe configurations, detect connection and rebalance storms, and give operators immediate feedback. 4. Cross-network connectivity AWS networking handles IP routing but doesn't understand Kafka's broker discovery protocol. Clients need every broker individually routable, which breaks across VPCs and accounts. **Gateway adds** Kafka-protocol-aware routing that rewrites broker metadata, giving clients in any topology (hybrid cloud, multi-VPC, on-prem) access through a single entry point. For the full walkthrough, see [Connect to AWS MSK Across VPCs Without Timeouts](https://www.conduktor.io/blog/kafka-msk-cross-network-connectivity). 5. Disaster recovery and failover AWS MSK Replicator syncs data across clusters, but switching hundreds of applications to a backup is the hard part, and most DR plans are never validated. **Gateway adds** single-command failover that redirects all client traffic in seconds, plus built-in chaos testing via protocol-level fault injection to validate resilience without production risk. 6. Virtual clusters and infrastructure efficiency Dedicating AWS MSK clusters per team and environment creates sprawl. Large messages and high-read scenarios add unnecessary broker load. **Gateway adds** logically isolated virtual clusters with independent namespaces and access controls, plus S3 payload offloading and caching for broker efficiency. Key points **One enforcement point, six capabilities.** Data protection, data quality, client guardrails, connectivity, DR, and multi-tenancy are all enforced at the gateway rather than rebuilt per application. **No code changes.** Every capability is applied at the protocol level, so applications connect exactly as they do to AWS MSK today. **AWS MSK stays untouched.** Fault injection, routing, and encryption happen in the proxy layer, not on your brokers. Part 3 How Conduktor Console completes the picture If Gateway governs traffic, Console gives your teams the visibility and ownership to operate Kafka at scale without bottlenecking on the platform team. Capability What it does **Unified operations** Manage topics, schemas, connectors, and consumer groups across all AWS MSK clusters from one interface. Works alongside the AWS Console and CLI. **Federated ownership** Developers discover, provision, and own resources within automated guardrails, catalogs, and approval workflows. Adds Kafka-level ownership on top of AWS IAM. **Visibility and troubleshooting** Browse and tail messages in production, monitor consumer lag, and route alerts to Slack, Teams, or PagerDuty. Adds application-level visibility alongside CloudWatch. **Cost attribution and insights** Track per-team Kafka usage for chargeback and surface health and risk recommendations across clusters. Complements AWS Cost Explorer with per-team breakdowns. Part 4 Built for the AWS ecosystem Conduktor integrates natively with AWS services for authentication, encryption, storage, and deployment, so it fits into your existing infrastructure without additional tooling. Integration What it provides **AWS MSK IAM authentication** Native support for the AWS_MSK_IAM SASL mechanism. Gateway inherits AWS IAM roles from ECS, EKS, and EC2. **AWS KMS** Field-level encryption keys managed in KMS. AWS IAM policies control which consumers decrypt which fields, and CloudTrail logs every operation. **Amazon S3** Large payload offloading via the claim-check pattern. Payloads never consume broker storage or network bandwidth. **ECS / EKS deployment** Deploy Gateway as ECS tasks or EKS pods within your VPC, using native container orchestration. **AWS Glue Schema Registry** Native support for validation and evolution across Avro, JSON, and Protobuf. **AWS Marketplace** Available for simplified procurement, streamlining vendor onboarding and purchasing against existing agreements. Related resources [Kafka Disaster Recovery Beyond Replication](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-beyond-replication) [Where Kafka Costs Hide: A Field Guide](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) [Conduktor + Amazon MSK](https://www.conduktor.io/partners/aws) See how Conduktor complements your AWS MSK investment Book a technical demo tailored to your environment and use cases. We'll walk through the capabilities that map to your workloads and show what changes when you add Conduktor to AWS MSK. [Book a technical demo](https://www.conduktor.io/contact/demo?src=resources-ebooks-aws-msk-solution-brief) Keep reading the full brief Source: https://www.conduktor.io/resources/ebooks/aws-msk-solution-brief --- # Federating Kafka Data Governance at Scale What's in this ebook [01 The governance dilemma Why control-vs-autonomy is a false choice](#wp-dilemma) [02 What federation is Centralized oversight plus local autonomy](#wp-federation) [03 Federation and data mesh Why a mesh cannot work without it](#wp-mesh) [04 Federation in practice Where to draw the line on control](#wp-practice) [05 Supercharging innovation How guardrails accelerate teams](#wp-innovation) [06 How Conduktor helps Operationalizing federation with tooling](#wp-conduktor) [07 Federation at scale A national postal operator's results](#wp-casestudy) [08 Where to go next Choosing the third path](#wp-conclusion) Executive summary Modern data environments force a difficult choice on every team that runs Kafka at scale: optimize for platform stability or for developer speed. Lean too far toward stability and the platform team becomes a bottleneck, throttling developer velocity. Lean too far toward autonomy and the estate descends into operational chaos, with inconsistent data, security gaps, and no shared understanding of schema, ownership, or data freshness. **This ebook argues that the dilemma is false. There is a third path: federated governance.** Just as federal governments devolve some powers to states and municipalities while retaining authority over shared concerns, platform teams can grant developers and data teams a measure of independence within safe, centrally defined boundaries. The result is platform stability and developer autonomy at the same time, without compromising security, data quality, or observability. Who this is for Platform leads, Kafka architects, and data teams who feel the tension between control and speed, and who want a practical operating model for governing a growing data estate without growing the platform team in proportion. The rest of the ebook builds the model. Part 1 lays out the dilemma and why both extremes produce the same failure. The parts that follow define federation, place it inside the data mesh paradigm, show where to draw the line between central control and local autonomy in practice, and explain how federation supercharges innovation rather than slowing it. A closing section shows how Conduktor operationalizes federation, followed by results from a national postal operator that scaled Kafka fivefold without adding platform headcount. Part 1 The governance dilemma Modern data environments are brutal to manage, for both rapidly growing startups and established enterprises alike. Regardless of team size or infrastructure scale, companies soon encounter the same choice: should they optimize for platform stability or developer speed? Lean too far toward stability If companies lean too far toward stability, routing every request through the platform team, then platform engineers are quickly overwhelmed by tickets and troubleshooting. Their team becomes a bottleneck, slowing innovation, stalling growth, and eventually burning out their engineers. Developers are blocked from accessing data, creating a topic, or deploying a connector, unable to complete projects and deliver value. The centralized model looks orderly on paper. Every configuration change flows through a merge request, a review, a delivery pipeline, and a synchronization process before it reaches the clusters. But the manual review step in the middle is where the queue forms, and across dozens of clusters it becomes the single constraint on how fast anything moves. Lean too far toward autonomy If a company goes too far the other way, favoring developer autonomy over everything else, it encounters operational chaos. Without standards for naming data infrastructure, updating schemas, or setting retention periods, developers invent their own, creating conflict and confusion. Inconsistent or malformed data spreads downstream, breaking applications, analytics, and AI, while security gaps appear as developers relax access controls for the sake of agility. In both approaches, operational and analytical data estates (and the teams behind them) end up disconnected and siloed, lacking a shared understanding of details such as schema, KPIs, data freshness, and ownership. Operational teams cannot persist analytics, maintain schema, or predict how their data will affect downstream applications. Analytical teams cannot find or access the data they need, or suddenly hit breaking changes when their applications ingest operational data. > **The same destination from opposite directions.** Both extremes are unsuitable, and they bring about the same result: your data environment becomes a liability rather than an asset. Too much control Platform team becomes a ticket queue Developer velocity throttled; projects stall Engineers burn out on troubleshooting Innovation and growth slow to the team's pace Too much autonomy No shared standards for naming, schema, retention Malformed data spreads downstream and breaks things Security gaps open as controls are relaxed Operational and analytical estates drift into silos The dilemma is the wrong frame **Stability and speed are not opposite ends of one dial.** Treating them as a single trade-off guarantees one side loses, and both failure modes end in the same place. **Both extremes turn data into a liability.** Whether through a bottlenecked platform team or an ungoverned free-for-all, the estate stops being an asset the business can rely on. **There is a third path.** The rest of this ebook develops federated governance: centralized oversight and local autonomy at the same time. Part 2 What federation is, and how it reconciles control and autonomy The stability-versus-speed dichotomy is false. Instead, companies can federate their data governance, getting the best of both worlds: centralized oversight alongside local autonomy. Without federation, organizations face spiraling costs, failed AI initiatives, and compliance gaps. With federation, they can build a model for teams to collaborate effectively, encouraging developers to move fast within safe boundaries while platform teams maintain stability without becoming bottlenecks. The political-science analogy The concept of federation originates in political science. The federal model, in use in Brazil, the United States, and Germany, divides powers between the federal (national) and local (state and municipal) governments. The federal government sets broad rules for defense, finance, and foreign relations, while states and cities define local laws for sanitation, healthcare, or transportation. All of it is rounded off by a constitution, which serves as a shared legal and operational framework. Platform team = national government Safeguards the overall data environment Defines security and encryption policy Owns disaster recovery and cross-region failover Sets naming, partitioning, and discovery standards Application teams = states Configure the details of their own environments Experiment with retention policies within limits Onboard temporary users such as consultants Monitor their own metrics and performance Application teams and end users, which include data analysts, AI and machine learning engineers, application developers, and data engineers, operate like states. They have the independence to configure their environments, experiment with retention, onboard temporary users, and monitor metrics. Depending on the company, platform teams may govern the entire business or be divided among subsidiaries in different industries or regional units. Either way, they are charged with safeguarding the overall data environment and defining key policies for security and encryption, disaster recovery, and naming conventions. Governance needs a technical framework Just as federal governments require a legal framework of courts and constitutions to function, federated data governance requires a technical framework. In practice, that framework is a solution that handles the technical details by: **Establishing safe boundaries** that data owners, developers, analysts, and engineers can work within. **Decentralizing ownership** by assigning owners to data domains and empowering them to approve or reject access requests and to manage controls for quality, schema evolution, and retention. **Providing insight** on Kafka usage and expenses, guardrail violations, expired or unused RBAC permissions, and performance and operational metrics. > **Federation = shared guardrails + local autonomy.** A central team defines global guardrails for security and compliance, cost attribution, catalog and discovery, and resilience and DR. Local teams (finance, analytics, product, AI/ML) keep autonomy over the things closest to their work: retention and namespaces, schema evolution, access control and self-service, performance tuning. Part 3 Where federation fits into the data mesh paradigm Like federation, data mesh is a decentralized approach to working with data, dividing entire data environments into domains and owners that operate autonomously but with a high level of compatibility. Domains such as AI and ML, applications, analytics, executives, finance, and customer success each own their data, all connected through a shared platform team at the center. In fact, federation is indispensable to the concept of a data mesh. Without it, data mesh simply recreates all the problems associated with the excessive devolution of power to local teams, including insecure or non-compliant data (and a heightened risk of leaks or hacks) and the operational chaos that arises from a lack of uniform governance. What a well-executed mesh delivers Executed correctly, data meshes carry a set of compounding benefits: **Parallelism.** Teams can scale, manage, and update their domain without impacting others. This also improves resilience, as issues in one domain no longer spill over into others. **Improved data quality.** Devolving domain management to the owning team, which is closer to and better understands the data, raises quality. Central teams cannot know the intricacies of every environment and domain in a company. **Interoperability.** Domains collaborate seamlessly, exchanging and understanding data regardless of where it lives or who owns it, without worrying about incompatibilities. Lacking interoperability, a data environment is not a mesh so much as a siloed mess. **Simplified data products.** With higher-quality data and better collaboration, teams build and release reusable products such as APIs, event streams, and curated data sets. Reusability removes redundancy and accelerates delivery, because teams reuse shared products instead of building from scratch. **Speed.** Because domains work independently, teams move products from concept to production without going through a central team. **Tighter integration of operational and analytical estates.** A properly implemented mesh lets real-time operational data flow into analytics, AI, and applications, producing more timely, trustworthy outputs and shortening decision cycles. **Accountability and transparency.** Clear guidelines plus supporting technology make it easier to stay within safe boundaries for operations, compliance, and security, and easier to trace the root cause when an issue arises. The mesh depends on the guardrails **Data mesh without federation is just decentralization.** The benefits above only materialize when local autonomy sits inside centrally defined, enforced boundaries. **Interoperability is the load-bearing benefit.** Without shared standards for naming, schema, and discovery, domains cannot exchange data and the mesh degrades into silos. Part 4 How federation looks in practice To federate data governance successfully, organizations must know where to draw the line between control and autonomy. Some operational concerns are too granular or business-specific for platform teams and must be devolved to data owners and application teams; others are too critical to leave to chance. The key is to balance these concerns. The specifics vary by company, but the basics stay similar across organizations and industries. What local teams own Generally, data owners manage the details closest to their work. They can name their topics however they like, so long as they follow approved naming conventions that prevent conflicts. Local teams define data retention policies, provided they stay within restrictions set centrally, and they onboard their own users. Because they understand their workloads and environments best, local teams also own performance management: determining acceptable latency thresholds, tuning consumer lag alerts, and adding or removing connectors. They manage schema versioning and upgrades, while leaving larger concerns such as backwards compatibility to the platform team. What platform teams own Platform teams have authority over systemic concerns that affect the entire enterprise. They oversee disaster recovery, cross-region replication, and failover; perform chaos testing and global performance monitoring; and define shared governance standards for naming, partitioning, and discovering topics. This way, platform teams prevent confusion and silos while ensuring interoperability. Importantly, federation acknowledges that not all environments are created equal. A team ingesting telemetry from a factory assembly line has different requirements and risk tolerances than a team analyzing event data from a software application. Each vertical needs the flexibility to operate on its own terms, but within a consistent enterprise framework. With the right technology, the flexibility federation provides removes friction. If a product team needs to grant temporary Kafka access to an external consultant or contractor, self-service controls let them do so safely, quickly, and without involving the platform team. The result is faster collaboration, tighter control, and less operational overhead. Part 5 How federation supercharges innovation Federation does not just improve data governance; it also boosts productivity and innovation. By balancing autonomy with oversight, it ensures that data is a competitive advantage rather than a source of friction. Clear rules make teams faster At its core, federation takes a systematic approach to collaboration. Faced with clear, non-negotiable rules, teams understand how to operate and what to expect. One vital concern is ensuring that new technologies or infrastructure remain compatible with the larger enterprise environment. A simple policy requiring all data connectors to remain backwards compatible ensures that updates never break existing data flows, so teams can experiment freely, deploying and scaling new products as needed, as long as their choices do not conflict with existing infrastructure. Some centralization removes confusion around operations and brings consistency and predictability. For instance, a platform team can mandate that schema updates cannot drop fields, instead retaining outdated fields with values of NULL or 0. By implementing this rule, platform teams reduce the frequency of application issues, saving application and downstream teams from wasting hours on restoring service. Discoverability, templates, and parallelism When data is more secure, compliant, and discoverable, it is also more usable. Federated governance lets owners build catalogs, helping other users quickly find the data they need and provision their own accounts without waiting on other teams, while data owners retain final authority to allow or deny access. Templating workflows for common components, such as connectors, topic configurations, or integrations, makes it easier for users to work independently and takes the guesswork and risk out of the process. Less technical roles, such as data analysts or business leaders, can use these preset sequences to access data without involving platform teams or data owners, and thus move faster. Because of their parallelism, federated data meshes are also far easier to scale. The dynamic is similar to a microservices architecture, where multiple interlinked yet independent nodes scale as needed. Each node is a different data domain that can scale at its own pace and more easily accommodate fluctuations in user traffic or demand. And federation encourages autonomy without sacrificing accountability: as data owners, domain teams manage access, quality, and compliance for their local environment, while still operating inside the boundaries set by platform teams. > *"Being able to share data seamlessly and collaborate across teams without friction has saved countless hours. It's improved the quality of questions and insights we receive, which boosts overall productivity."* > — Platform engineer at a major commercial airline Guardrails are an accelerator, not a brake **Non-negotiable rules reduce decisions, not options.** Teams that know the boundaries spend less time seeking approval and more time building. **Templates and catalogs push work outward.** Self-service discovery and preset workflows let non-specialist roles access data without a ticket. **Parallelism is what makes it scale.** Independent domains grow at their own pace, the way microservices do, without a central chokepoint. Part 6 How Conduktor handles federation As an operating model, federation is nearly impossible to execute manually, especially through a patchwork of technologies, scripts, and unwritten rules. An ad hoc approach cannot scale in large data environments: governance breaks down, visibility becomes nonexistent, and operations devolve into chaos. Any team that seeks to operationalize federation needs tooling to enforce boundaries, enable autonomy, and keep up with growth in users, traffic, and data domains. Conduktor is a governed data hub that sits on top of data streaming infrastructure, between source systems and consumers such as applications, lakehouses, business intelligence tools, and AI agents. It gives platform and local teams exactly what they need: guardrails to work safely, self-service to access data quickly, and shared visibility for collaboration. Governance tier Conduktor enables… **Central platform team** Defines global guardrails for encryption, schema, partition limits, and naming conventions **Local domain teams** Self-service, access control, connector deployment, and schema actions within bounds **Shared infrastructure** Unified observability, Insight Center, audit trails, disaster recovery, and data catalog and discovery The core challenges Conduktor solves Kafka as a black box Many platform teams have no observability into their data infrastructure and cannot understand what is happening. Conduktor provides visibility into topics, clusters, and domains through dashboards and monitoring. Security and compliance gaps Conduktor builds encryption, auditing, and resilience directly into the data streaming layer. Every interaction can be traced, every user action logged, and every environment held to preset rules. Governance impeding innovation Platform teams use pre-defined policies and templates to automate repetitive tasks. Routine actions such as onboarding users, deploying connectors, or setting retention become self-service workflows. Environments that cannot scale or integrate As a platform-agnostic solution, Conduktor prevents vendor lock-in. It supports on-prem, cloud-based, and hybrid deployments and a wide range of products. Fragmented operational data In a federated model, every team needs seamless access to insights. But when data is scattered across dashboards and tools, insights are hard to extract and become a blocker: platform teams cannot determine which policies to create, application teams cannot prioritize which components to optimize, and architects are unclear on the returns (or losses) of their governance efforts. > **On the roadmap.** Conduktor is planning Insight Center, an AI-powered feature that answers governance questions through natural-language queries. A platform team could ask for the five data domains with the highest monthly spend and their owners, or identify which topics are missing encryption for sensitive data or violating retention policies, rather than sifting through SQL and aggregations. This feature helps central teams get granular visibility and accountability into how they operationalize governance. Part 7 Federation at scale: a national postal operator A national postal and logistics operator handles more than a billion letters, tens of billions of customer assets, and hundreds of millions of parcels every year. As real-time data became central to everything from sorting-facility automation to customer analytics, the operator faced a familiar challenge: scaling Kafka and modernizing systems without ceding governance or stifling developer autonomy. As Kafka became increasingly important, the platform team struggled with visibility, access management, and compliance. Developers could not self-serve, analysts could not discover or access data, and Kafka had effectively become a black box. Using Conduktor, the operator implemented federated governance, balancing autonomy with oversight across the data environment. It set centralized guardrails for compliance and reliability while enabling autonomy through self-service and role-based access controls (RBAC). Developers, data scientists, and analysts could now independently discover, explore, and validate data without waiting for the platform team's help or approval. Conduktor also facilitated monitoring and observability for the platform team, giving it a unified view of Kafka clusters and schema health. > *"Being able to share data seamlessly and collaborate across teams without friction has saved countless hours. It's improved the quality of questions and insights we receive, which boosts overall productivity."* > — Platform team lead at a national postal operator The results 5x growth in Kafka applications, from 60 to over 300 1,100+ integrations, up from around 300 800+ internal users given secured access to Kafka 0 increase in platform headcount A fivefold increase in Kafka applications and a threefold increase in integrations. 800+ internal users across all departments empowered with secured access to Kafka. Zero increase in platform headcount, achieved through automation and self-service. Compliance with key legislation, such as the national data-protection act, via strict RBAC controls. Higher data quality and AI readiness, as teams could govern and reuse data with confidence. What the numbers show **Scale and headcount decoupled.** The estate grew fivefold in applications and more than threefold in integrations with no growth in the platform team. **Autonomy and compliance coexisted.** Self-service and RBAC let hundreds of users move independently while meeting statutory data-protection requirements. Conclusion Choosing the third path By federating data governance, enterprises can opt out of the developer-autonomy-or-platform-stability dilemma and choose to have both. By decentralizing ownership while standardizing shared safeguards, organizations move faster, stay compliant with rules and policies, and take a systematic approach to innovation. As a governance hub, Conduktor empowers organizations to take this third path and to facilitate self-service, monitoring, and management, with the help of automation and AI-powered insights. Whether an organization is new to Kafka or an established enterprise with a massive data environment, Conduktor can help its teams operate with accountability, independence, and confidence, turning Kafka from an expensive black box into a governed, valuable business asset. Related resources [The Journey to Self-Service](https://www.conduktor.io/resources/ebooks/the-journey-to-self-service) [Governing Kafka Data for the AI Era](https://www.conduktor.io/resources/ebooks/governing-kafka-data-for-the-ai-era) [Conduktor Console](https://www.conduktor.io/console) Ready to operationalize federation at scale? See how Conduktor gives platform teams centralized guardrails and gives developers self-service autonomy, so your estate can grow without your team growing with it. Book a demo and we will walk through your environment together. [Book a demo](https://www.conduktor.io/contact/demo?src=resources-ebooks-federating-data-governance-scaling-kafka-without-losing-control) Get the full ebook Source: https://www.conduktor.io/resources/ebooks/federating-data-governance-scaling-kafka-without-losing-control --- # Governing Kafka Data for the AI Era What's in this whitepaper [01 Kafka as AI's layer Why streaming became the default transport for GenAI](#wp-transport) [02 Data quality risks The quality failures that break AI systems](#wp-quality) [03 Security risks Exfiltration, prompt extraction, and injection](#wp-security) [04 Compliance & accuracy Regulatory exposure and the cost of getting it wrong](#wp-compliance) [05 A control plane How Conduktor governs data before it reaches AI](#wp-conduktor) [06 Next steps Consolidating governance into one management layer](#wp-conclusion) Executive summary AI is becoming part of the core operating fabric of modern enterprises. As Generative AI (GenAI) enters production, it increasingly relies on fresh and sensitive operational data, and Kafka has become the default transport layer for that data. But Kafka provides only basic security, and organizations building AI on top of it face governance challenges the platform was never designed to solve. Who this is for Engineering and platform teams building GenAI, RAG, and real-time inference workflows on Kafka, and the leaders responsible for the data quality, security, and compliance risk those workflows introduce. **The core argument of this paper: streaming data makes AI accurate and current, but poorly managed streaming data turns a GenAI investment into a liability. The three risks that matter most are data quality, security, and compliance, and all three have to be addressed at the source, before data reaches a model.** 89%of IT leaders see data streaming easing AI adoption 96%of executives think adopting GenAI makes a breach likely within three years 77%of Americans don't trust businesses to use AI responsibly Part 1 explains why Kafka became AI's transport layer and what streaming architecture buys an AI program. The gated sections then walk through the three risk categories in turn, the accuracy failures they produce, and how a control plane over Kafka lets engineering teams govern what enters models and where context comes from, without slowing them down. Part 1 Kafka as AI's transport layer AI is fast becoming integral to how organizations operate. Wharton and GBK Collective report that weekly usage of GenAI grew from 37% in 2023 to 72% in 2024, powering use cases at the heart of organizations' missions. As models move into production, they increasingly rely on fresh, sensitive operational data, and Kafka has become the default way to move that data at speed. The rapid adoption of GenAI exposes a critical challenge: ensuring data quality and freshness. Errors, biases, and outdated information in training data can lead to misleading outputs with significant consequences. Streaming data offers a solution to that challenge. What streaming architecture buys an AI program Investing in a streaming architecture lets organizations feed models current, trusted context and monitor how that data is used as it flows. Four capabilities stand out. Improve and scale training Keep AI systems up to date through incremental updates via Retrieval-Augmented Generation (RAG). Support fine-tuning techniques like Low Rank Adaptation (LoRA), which updates specific model layers while preserving the base model's knowledge. Supplement periodic batch retraining on curated datasets with real-time context. Provide fresh data LLMs train on static sets; RAG lets you feed them real-time data for more accurate, trusted answers. Powers use cases from chatbots to content personalization to fraud detection. Real-time streams incorporate the latest information so models generate reliable, context-aware outputs. Enable compliance monitoring Automated data classification detects PII and financial data based on configurable patterns. Retention policies enforce GDPR's right-to-deletion by purging data after specified periods. Audit trail generation logs data access patterns required by SOX, HIPAA, or GDPR. A rule can block messages containing European email addresses from model training unless consent flags are present. Add AI safeguards Tools like Llama Guard, NeMo, and OWASP AI add pre- and post-processing steps for prompts and responses. These steps run inside the streaming architecture, promoting safety and quality in AI applications. Guardrails travel with the data rather than living in each application. The RAG-plus-streaming pattern The dominant pattern for putting fresh data in front of a model combines RAG with streaming. Operational data (payments, sales orders, system logs) is converted into vectors by an embedding model and written to a vector database. When a user asks a question, the query is embedded, matched against that database, and the retrieved context is combined with the query in the prompt sent to the LLM. > A user asks "show me unusual transactions in our checkout system in the last three hours." The query is embedded, matched against streaming operational data in the vector database, and the retrieved context is passed to the LLM, which answers with the specific transactions and users involved. The answer is only as good, and as safe, as the data that reached it. In short, real-time data streams improve AI model accuracy by providing current information, but they introduce operational complexity. When poorly managed, streaming data brings inconsistent data quality, inadequate access controls, and limited audit trails. AI applications compound these concerns: models can inadvertently leak training data patterns, and prompt-based attacks can exploit AI systems to reveal information they weren't intended to share. IBM found that 96% of executives think adopting GenAI "makes a security breach likely in their organization within the next three years." Why the transport layer is where governance belongs **Streaming is what makes AI current.** RAG and fresh operational context are the difference between a model that reasons about today and one stuck on its training cutoff. **Kafka wasn't built for this risk profile.** It provides basic security, but AI amplifies the consequences of weak controls because models leak, and attackers prompt. **If the data is wrong or exposed when it reaches the model, nothing downstream can fix it.** The controls have to sit on the stream itself. The rest of this paper is about what those controls look like. Part 2 Data quality issues that break AI systems When data quality breaks down, so does AI. Incomplete or inconsistent inputs don't just create noise, they create risk. Each common data quality issue maps to a specific failure mode in AI systems, and each failure mode carries a real business consequence. Data quality issue Consequence for AI Inaccurate metadata or missing timestamps Broken temporal context in streaming agent systems, which means missed SLAs and failed autonomous decisions Missing schemas and inconsistent formats Broken prompt inputs and retrieval chains, leading to output failures that slow AI adoption across teams Duplicate or redundant records Inflated signal strength that reinforces biases and triggers faulty automations, damaging customer experience and increasing regulatory exposure Missing or incomplete fields Models hallucinate or guess, driving bad recommendations, false information, lost trust, and higher churn Out-of-range values Model logic thrown off, causing erratic AI behavior and decisions that break trust with users and systems These failures roll up into three business impacts that leadership recognizes immediately: **customer churn**, **lost revenue**, and **compliance exposure**. As AI applications drive increasingly critical decisions and processes, ensuring models produce accurate, high-quality outputs becomes a business requirement, not an engineering nicety. Some of that responsibility falls on LLM builders, and companies like OpenAI and Anthropic are actively developing techniques to minimize issues, but your organization can materially improve model outputs by investing in data quality and freshness at the source. Performance and reliability To deliver business value, LLMs must produce accurate, meaningful, contextually appropriate outputs, which requires clean, consistent, well-labeled, up-to-date data. Removing errors and obsolescence from training data minimizes the risk of misleading or biased results. That in turn minimizes the legal, financial, regulatory, and reputational risks associated with GenAI. Scalability and cost-efficiency Well-prepared data lets LLMs learn more efficiently, spending less time on irrelevant patterns and more on meaningful relationships. That reduces the need for costly computing power. Amid increasing scrutiny of GenAI ROI, focusing on data quality achieves high performance while keeping costs in check. You can use streaming data to keep LLMs up to date, accurate, and relevant. At the same time, it's crucial to be aware of the risks of poorly managed streams. If your streaming architecture doesn't provide the necessary level of security and control, your organization's GenAI investments can quickly turn into a liability with financial, reputational, regulatory, and legal consequences. Data quality is an upstream problem **Every quality issue has a named AI failure mode.** Missing timestamps break agents, missing schemas break retrieval, duplicates reinforce bias, gaps drive hallucination. **The business impact is churn, lost revenue, and compliance exposure.** These are the terms leadership already tracks. **Better data is also cheaper AI.** Clean inputs cut wasted compute, which matters as GenAI ROI comes under scrutiny. Part 3 Security vulnerabilities in Kafka-based AI pipelines Streaming pipelines often lack built-in security features, making them particularly vulnerable to persistent or stored attacks where malicious instructions are embedded into the data flow. Apache Kafka, the event streaming platform used by 80% of the Fortune 100, isn't secure out of the box. Implementing appropriate security mechanisms such as access control lists (ACLs) is cumbersome, time-consuming to manage, and prone to error. Kafka also falls short on data encryption at the field level, increasing the risk of exposing sensitive information. RAG applications add another layer of complexity. RAG augments LLMs by tapping into additional data sources, often unique or proprietary to an organization, such as CRM systems or product documentation. Without sufficient safeguards, an attacker could prompt a RAG application to leak sensitive information from its external knowledge bases. Three risks stand out. Data exfiltration Without strong security and control over your streaming data infrastructure, sensitive information could be accessed and transferred to external parties, either maliciously or accidentally. > A misconfigured AI assistant at a financial planning firm. An internal user asks the assistant to retrieve examples of large deposits within the past two days. The assistant returns them, complete with names, account numbers, and transaction amounts. The user can then export that data to a transcript and send it outside the organization via email, copy-paste, or screenshot. Prompt extraction To surface hidden information or context, such as customer metadata, transformation rules, and business-specific guidance, malicious parties may try to reverse engineer prompts. Known as prompt extraction, this method hijacks the programming of AI systems to compromise data security. Consider a logistics support chatbot that ingests real-time shipment data via Kafka. Its owner may instruct it to prioritize updates from a specific Kafka topic for priority shipments, refer to shipments by customer name and destination, and apologize for delays over three hours. A hacker asks targeted questions, why the chatbot apologizes for delays, or which Kafka topic it consumes from. Without proper governance, the chatbot can respond with its own instructions, exposing the exact topic name it draws from, its escalation and personnel protocols, and even customer data. Prompt injection Attackers may also insert malicious inputs into the prompt to twist LLM behavior and override its original programming. This can cause the model to hallucinate or violate compliance regulations, and some injections persist across sessions, letting attackers exfiltrate data later or inflict long-running damage. If a chatbot ingests unfiltered data from Kafka topics, an attacker could inject malicious prompts into the data stream itself, embedding instructions in a "customer-feedback" topic that tell the AI to ignore its safety guidelines. Without content filtering at the Kafka level, these poisoned prompts could affect the AI's behavior across all subsequent interactions and leak sensitive business information. Attackers can also inject indirectly, hiding harmful links inside unstructured data such as uploaded PDFs or forum posts; if the chatbot scans the link, it could activate malware that reaches unsecured Kafka infrastructure and damages user permissions or data sharing. What leaves the door open ACLs are cumbersome, manual, and error-prone. No native field-level encryption. Unfiltered topics ingested straight into prompts. No content filtering at the Kafka level. RAG knowledge bases exposed through the model. What each attack costs **Exfiltration:** named customer and account data leaves the organization. **Prompt extraction:** topic names, escalation logic, and personnel info exposed. **Prompt injection:** poisoned behavior persists across sessions and interactions. The stream is the attack surface **Kafka is not secure out of the box, and 80% of the Fortune 100 run it.** The gap between adoption and built-in security is where these attacks live. **AI turns data access into data leakage.** A model that can read sensitive fields is a model that can be prompted to reveal them. **Filtering and encryption have to happen before the topic reaches the model.** Content filtering at the Kafka level is what stops a poisoned message from ever entering a prompt. Part 4 Compliance and accuracy risks Compliance risks Because AI remains an emerging technology, best practices around compliance and regulation are still evolving. Any AI legislation has to address several key concerns. **The legality of data.** Is data protected from malicious parties? Was it obtained through legal means, and is personally identifiable information (PII) encrypted? Without a way to monitor lineage and consent, it's difficult to discern whether data was collected with permission, especially for sensitive data like health information. **The need for data access logs.** Without fine-grained controls and access logs, it's impossible to trace who accessed which data, when, and for how long, all prerequisites for investigations and audits. **The possibility of compromised or unvetted external datasets.** Data is only as good as its source. Unvetted, inconsistent, unreliable data will contaminate the training process and lead to untrustworthy results. **The lack of transparency around decision-making.** Many LLMs are known as "black boxes." Without clear explanations for actions like blocking or flagging suspicious credit card transactions, AI may frustrate or confuse the people it serves. Many obligations for organizations using AI go into effect in 2025, so it's paramount that organizations take their transparency requirements seriously. Beyond compliance, explainability and transparency are critical for gaining customers' trust. Gallup reports 77% of Americans say they don't trust businesses to use AI responsibly, but respondents also said that being clear about how AI is used and what data sources it pulls from would help alleviate their concerns. Transparency is especially hard with streaming systems, which handle massive volumes of data transiting at high speed. Apache Kafka doesn't provide native support for auditing, so without additional tooling it's hard to maintain a detailed log of how, where, or when data was used for model training or downstream decision-making. Kafka's distributed nature makes aggregating data lineage and user actions across multiple clusters even more complex. Accuracy risks The propensity of AI applications to hallucinate, producing inaccurate or nonsensical outputs with no factual basis in their training data, is well-documented, and it carries real-world liability. Attorneys sanctioned over a fabricated brief Lawyers used ChatGPT to write a legal brief and only discovered the LLM had cited nonexistent cases after submitting it to a judge. The lawyers and their firm were sanctioned and fined $5,000. An airline forced to honor a made-up fare A chatbot promised a passenger a discount that didn't align with the airline's policy. A court made the airline honor the fare and pay $812.02 in damages. Data quality, or the lack of it, is a major factor in hallucinations. Because of its real-time nature, streaming data can suffer from noise, errors, and incomplete information. Without a robust approach to data quality, GenAI accuracy will suffer. While hallucinations often result from AI prioritizing a user's request over accurate information, bad actors can also use prompt injection, context injection, or data poisoning to make a model produce misinformation. The latter two techniques affect RAG applications in particular: an attacker who gains access to a company's knowledge bases can introduce inaccurate information or make the model behave maliciously, for example feeding positive reviews about a competitor's product into a database so a chatbot recommends the competitor instead. > Streaming architectures propagate data in real time, which heightens the risk of errors. An incorrect answer to a RAG query can be repeatedly propagated throughout the system, leading to significant operational disruptions or reputational damage. Implementing safeguards that ensure streaming data meets your standards before it enters GenAI pipelines is essential. Compliance and accuracy are the same problem seen twice **Both require knowing where data came from and where it went.** Lineage, consent, and audit logs answer the compliance question and the "why did the model say that" question at once. **Hallucination is a data-quality symptom with a legal price.** Fabricated cases and made-up fares are quality failures that reached a user. **Real-time propagation raises the stakes.** A wrong or poisoned answer doesn't stay contained; it spreads at streaming speed. Part 5 A control plane for AI data risks Left unaddressed, these risks can expose your organization to significant consequences. With the right control layer, you can scale your streaming data efforts for GenAI so your models reap the benefits of real-time data without the risks. Conduktor is an intelligent data hub that unifies, optimizes, and accelerates operational data utilization. By enabling data accessibility, enforcing governance policies across your entire data infrastructure, and preventing tainted data from contaminating your systems, Conduktor supports data quality, security, and compliance, and trustworthy, reliable AI outputs. By intercepting data before it enters your system, Conduktor applies a "shift left" approach that addresses data quality and controls earlier in the pipeline, reducing downstream risk. Secure sensitive data in GenAI pipelines **Data masking and redaction.** Obfuscate personal and sensitive data before it enters AI applications so it can't be leaked or exfiltrated by bad actors or careless employees. You can also mask or redact sensitive data before using it to train models. **Server-side filtering.** Remove sensitive data from payloads altogether. **Multi-layer encryption.** Protect sensitive customer and business data as they pass through various systems. Conduktor enables field-level encryption, both at rest and in transit, securing sensitive data such as PII or financial fields without encrypting the entire payload. That improves usability and efficiency because unencrypted fields remain accessible for processing, filtering, and analytics. Before Encryption in Kafka is fragmented and inconsistent. Each application must implement its own encryption logic. The result is duplicate effort, misconfigurations, and compliance gaps. After Conduktor secures data before it reaches Kafka. Centralized policies let teams apply and update encryption in one place, with no code changes. Governed, AI-ready data flows into model training, real-time inference, and RAG pipelines. Proactively enforce data quality and detect anomalies Conduktor's "shift left" approach ensures data is protected and compliant at the source to enhance its utility for AI and prevent downstream issues ranging from hallucinations and toxicity to data breaches. Developers can focus on innovation without compromising quality and security. **Block invalid or costly configurations** with guardrails that maintain consistent policy enforcement, without adding management overhead. **Stop bad data at the source** with a middleware approach that solves quality issues at scale. **Reduce costs** by addressing low-quality data before it travels downstream and impacts your applications. **Speed up decision-making** by making data immediately available for analytics once it enters the pipeline. **Monitor data quality in real time** to address issues before they affect downstream applications. **Identify and address security breaches** and operational issues immediately with real-time, customizable alerts. Securely share data Whether it's ingesting real-time knowledge from external sources or streaming data to external plugins and vendors, sharing data with partners is an inevitable part of the AI process. Historically, sharing Kafka data meant replicating data and topics across clusters and instances, which requires manual work from internal and partner teams, drives up costs, and risks exposing sensitive data. Instead, you can use Conduktor to: Enable partners to directly access your Kafka data via Partner Zones, removing the need for replication and reducing data sharing expenses. Simplify and automate sharing workflows by connecting partners directly with clusters, maintaining a unified data source across organizations. Monitor partner usage and fulfill compliance requirements with quotas, traffic policies, and audit trails. Regulate tenant access so partners only ingest the data they are approved for. Centrally enforce controls By configuring permissions centrally, you can ensure users only see and access the data they're authorized to see, without managing permissions for each individual application. Conduktor enables you to: Define granular permissions for different users and groups via role-based access controls (RBAC). Deploy security updates once from a single, centralized console, and instantly apply them across all your applications and infrastructure. Balance autonomy and stability, letting developers move quickly with self-service while staying within parameters defined by the platform team. Facilitate trust Gaining the trust of your users and stakeholders is critical for GenAI success. Transparency in how data is used, stored, and protected must be at the heart of every decision; without it, adoption will stall. You can use Conduktor to: Track every action with user and resource audit logs that show who did what, when, and whether permission was granted. Mitigate potential bias by tracking data provenance. Ensure ethical handling of sensitive inputs by maintaining transparent, secure access to training data. One layer, three risks addressed **Security:** masking, redaction, server-side filtering, and field-level encryption keep sensitive data out of models and attackers' reach. **Quality:** guardrails and shift-left enforcement stop bad data at the source before it reaches a prompt. **Compliance:** RBAC, audit logs, and data provenance make the pipeline explainable and auditable across clusters. Conclusion Next steps for secure AI data pipelines Managing Kafka security through multiple point solutions creates operational complexity and potential security gaps. Conduktor consolidates these capabilities into a single management layer. This reduces operational overhead, empowers teams to innovate, and promotes compliance and security. Conduktor is deployed between your client applications and your existing Kafka infrastructure. It integrates with all major managed service providers, including Confluent, Amazon MSK, Redpanda, Aiven, and Microsoft Azure, and directly with Kafka. With no-code integration and multi-layer policies, Conduktor makes it easy to assess and enforce security without touching application code. > **The takeaway.** AI moves fast, and your data needs to keep up with guardrails. Streaming data is what makes GenAI current and accurate, but data quality, security, and compliance have to be enforced on the stream itself, before anything reaches a model. A control plane over Kafka is how engineering teams stay in charge of what enters models and where context comes from, without slowing down. Related resources [AI Can't Wait: Upstream Data Quality](https://www.conduktor.io/resources/ebooks/ai-can-t-wait-why-real-time-systems-demand-upstream-data-quality) [Achieving Data Security for Kafka](https://www.conduktor.io/resources/ebooks/achieving-data-security-for-kafka) [Federating Data Governance](https://www.conduktor.io/resources/ebooks/federating-data-governance-scaling-kafka-without-losing-control) Run secure, large-scale AI with confidence See how Conduktor gives your teams a single control plane over Kafka to secure sensitive data, enforce quality at the source, and govern every AI data flow. Book a demo and we'll walk through it against your own pipelines. [Book a demo](https://www.conduktor.io/contact/demo?src=resources-ebooks-governing-kafka-data-for-the-ai-era) Get the full whitepaper Source: https://www.conduktor.io/resources/ebooks/governing-kafka-data-for-the-ai-era --- # Kafka at Scale: 5 Challenges & Solutions What's in this ebook [01 Poor quality data Bad data at ingest breaks everything downstream](#wp-quality) [02 Slow manual workflows Ticket-driven access turns platform teams into bottlenecks](#wp-workflows) [03 A team disconnect Operational and analytical teams speak different languages](#wp-disconnect) [04 Zombie costs Forgotten topics and clusters quietly drive up the bill](#wp-zombie) [05 Fragmented tech Bridging legacy and cloud without brittle dependencies](#wp-legacy) [06 Operationalizing Kafka Turning Kafka into a true business asset](#wp-conclusion) Executive summary Apache Kafka has become indispensable infrastructure for real-time data, powering everything from customer-facing applications to analytics to AI across every industry. But operating Kafka at enterprise scale is a different problem from adopting it, and the difference is where most organizations get caught out. Through conversations with current customers and prospects, architects, product managers, tech leads, and executives across sectors ranging from logistics to finance, Conduktor has seen the same five obstacles surface again and again. They are not exotic edge cases. They are the predictable consequences of running Kafka without the governance, self-service, cost visibility, and integration layer it does not ship with. 5Recurring challenges leaders hit at scale 700Connectors deployed by one team with self-service 2,000Topics synced across those connectors **The through-line: technology is only as good as the people and processes around it.** Without the right practices and governance, teams struggle with Kafka, fail to tap its full power, and can end up worse off than before. This ebook walks through each challenge, what it looks like in the field, and the benefits organizations realize when they solve it. Part 1 covers poor quality data, the challenge that undermines every downstream initiative. The four that follow are gated: slow and overly manual workflows, the disconnect between operational and analytical teams, mystery costs and zombie infrastructure, and the gap between legacy and cloud technologies. A closing section shows how Conduktor addresses all five. Part 1 Poor quality data For organizations, every major initiative, including AI, analytics, compliance, and digital transformation, requires trustworthy, accurate data to execute. Unfortunately, many Kafka pipelines ingest poor quality data at scale, introducing inconsistent, misformatted, and missing data into downstream systems, with disastrous results. This produces low-quality outputs, reduces confidence among both internal and external users, impacts mission-critical applications, and can lead to outages or disruptions. What the challenge looks like The failure mode is rarely a single broken pipeline. It is the slow erosion of trust that happens when nobody owns data quality and nothing blocks bad data at the door. **Blurry ownership.** At one national postal operator, "blurry borders" around team responsibilities for data quality, schema governance, and data retention left key duties unattended. Owners were nominally responsible for their data meeting specific standards, but due to a lack of accountability and ill-defined requirements, the task was neglected. Downstream teams stopped trusting the data. **No way to block bad data.** A technical lead at a logistics company described how misformatted data fields at ingestion would immediately break something on the consumer side, usually something powering a business-critical function. With no way to block the intake of that data, developers and end users saw only broken applications and blamed Kafka. **Schema registries are not enough.** Even organizations using schema registries, the traditional solution, hit major issues, specifically a lack of visibility and validation. Data scientists at one retailer could only identify issues after they appeared and impacted KPIs, because they had no way to monitor messages within Kafka itself. > Without a way to block the intake of bad data, developers and end users see only broken applications, and Kafka takes the blame for a governance gap. What solving it delivers By implementing clear ownership, in-stream observability, and automatic enforcement, teams can stop low-quality data from entering applications in the first place. That shift produces three concrete outcomes. The payoff of blocking bad data at the door **Improved trust.** High quality data creates high quality outputs, true for everything from AI to analytics. That increases trust among colleagues and customers alike, who can use results and applications with confidence. **Reduced and reallocated time and money.** Preempting bad data saves the hours spent on reactive approaches like migrations or lift-and-shift. Employees pivot to other work, increasing productivity and innovation. **Meeting budgets, KPIs, and SLAs.** Uptime and accuracy are two SLAs and KPIs directly threatened by low-quality data. Blocking it up front improves application reliability, meets key goals, and avoids associated financial penalties. Part 2 Slow, overly manual workflows Developers, analysts, and product teams need data now. If they have to wait, blocked by overly manual processes and unclear governance, the business suffers. Many organizations lack the systems, standards, and processes to help internal users access data rapidly and safely. What the challenge looks like Platform engineers are responsible for performance, stability, and security, so they build in guardrails to reduce risk. But without automation or self-service, those guardrails become bottlenecks. When developers and analysts wait too long, productivity drops and inter-team tension builds. The manual path At one national postal operator, requesting operational data to be persisted for consumption took weeks. Tickets were filed with the platform team. The platform team created and configured connectors and S3 buckets. Platform teams then had to validate connector permissions before anyone could consume data. The result: an overly manual, frustrating process for everyone involved. The self-service path One platform team at a multinational retailer built a one-click system to move data to end users for analysis. With it, they deployed **700 connectors**. They synced **2,000 topics** across those connectors. Guardrails stayed in place; the bottleneck did not. What solving it delivers By implementing automation, standardization, and self-service, an organization helps analysts and developers move faster while reducing the workload on platform teams. Benefits of self-service with guardrails **Faster time-to-value and time-to-market.** Analysts and developers moving faster means more rapid results. Products and features get created, tested, and released more quickly, and decision makers access insights to stay ahead of the competition. **Scale operations without chaos.** Self-service with guardrails lets data usage grow without throwing operations into disarray. **Improved business agility.** Reducing platform friction increases developer throughput and lets the business pivot faster on market trends, competitor weaknesses, and changing economic conditions. **Reduced overhead and employee turnover.** Stressed, overworked employees leave. Automating self-service with guardrails reduces administrative overhead, so platform engineers focus on more fulfilling work and stay longer. **Clear ownership and procedures.** Consistent, repeatable workflows save time and money. Teams stop reinventing the wheel and focus on the high-value work they were hired to do. Part 3 A disconnect between teams Organizations continue to struggle with silos, specifically the separation between operational and analytical systems, and the teams that run them. Operational data such as credit card payments, delivery vehicle movements, or IoT sensor data remains locked within technologies such as Kafka or application databases, while analytics teams work in data lakes such as Snowflake, BigQuery, or Databricks. What the challenge looks like The divide is technical and human at the same time. Developers and analysts may not share the same governance, procedures, or even understanding of data, which breaks down trust between them. Analysts lack a way to explore and access streaming data, and lack knowledge of how it is ingested, leading them to blame developers for not building better, more consistent pipelines. Developers, in turn, resent analysts for constant requests for access, quality enforcement, and discoverability. Analysts cannot find or get timely data, forcing them to work with stale or incomplete data and risk skewing their results. Without the right tools, analysts and data scientists may not even know about missing data in the first place, an unknown unknown that further exacerbates flaws in analytics, AI, and decision making. To overcome these barriers, organizations need a common domain for different data systems and products: essentially a shared data language and structure that all teams and business units are familiar with and participate in. This also clearly defines owners for specific components, such as pipelines or topics, and controls, such as data quality or encryption. What solving it delivers At an organizational level, implementing shared technologies and bridging the breakdown between operational and analytical systems, and teams, brings key benefits. Benefits of bridging the operational-analytical divide **Faster, better insights.** A company that can use more of its data generates high-volume, high-quality insights more rapidly than competitors, with positive implications for innovation, organizational agility, and revenue. **Higher trust and cohesion.** Removing the obstacles between operational and data teams improves morale and cooperation. Clear accountability builds more efficient workflows and removes old cycles of recrimination and blame. **Better governance for AI, and more accurate results.** AI is only as good as its inputs. Ensuring governance and access maintains the integrity, security, and relevance of training data, leading to more reliable, precise outputs. **Decreased risk.** Improving compliance and auditing, in the form of automatic policy enforcement, RBAC, and user logs, reduces the possibility of violating key legislation such as HIPAA or GDPR. Part 4 Mystery costs and zombie infrastructure Kafka usage rarely grows in a logical, planned manner. Instead, it grows organically, with topics and schemas being created, partitioned, used, and abandoned, often without documentation or visibility. When migrations or re-factoring initiatives occur, these excess topics, schemas, and partitions might be forgotten entirely. The problem becomes even more entrenched if teams fail to establish clear ownership and naming standards at the time of creation. What the challenge looks like While some topics, partitions, and schemas may be revived, many remain dormant, seeing minimal usage even as they continue to generate costs. This zombie infrastructure often accumulates in legacy or non-production environments, where it remains unnoticed by busy teams. Without any way to identify and remove it, teams waste valuable resources and drive up expenditures, without even knowing why. How teams respond Automation and observability go a long way toward resolving this. The teams that get ahead of it apply two moves. Automated cleanup and tagging One team created a seven-day cleanup policy in their dev environment, which was neither production nor customer facing, deleting all unused assets after that period. They used improved monitoring to tag usage by environment, project, and team, ensuring better visibility for financial planning. Standardized guardrails To keep expenses down, one organization limited self-service users to 10 partitions each. Any additional partitions triggered a request that had to be manually approved. Standardizing and implementing policies across an environment keeps drift in check. What solving it delivers Ultimately, these solutions result in the same thing for organizations: realizing operational and financial efficiencies. Benefits of removing zombie infrastructure **Decreased cloud and platform spend.** More detailed visibility into financial data helps leaders understand where money is wasted and operate more efficiently, and improves budgeting and forecasting so resources are allocated more effectively. **Increased reliability and responsiveness for business-critical applications.** Cutting unused, excessive infrastructure boosts performance, with lower latency and fewer incidents arising from clutter or misconfigured assets, so applications and products become faster and more reliable. **Lowered risk.** Zombie infrastructure may contain sensitive data and legacy configurations, making it doubly vulnerable to hacks and leaks. Removing it, or even updating retention policies, reduces organizational exposure and improves compliance postures. **Higher team velocity and shorter time-to-market.** Streamlining Kafka architectures means developers deal with fewer irrelevant assets, so teams onboard and work faster, cutting development cycles. Part 5 Fragmented legacy and cloud technologies Kafka is not limited to ingesting streaming data. It has moved beyond that niche, becoming a central nervous system that bridges the divide between legacy systems and modern microservices architectures. What the challenge looks like Otherwise obsolete technologies such as Managed Queues (MQ), Managed File Transfer (MFT), and SOAP/XML remain within digital environments for a number of reasons. They may power mission-critical applications or complex workflows, or simply be hidden away from view. Whatever the case, these technologies often create fragile, undocumented dependencies, and removing them could lead to outages or worse. Kafka is the solution, serving as a strategic integration layer with its rich ecosystem of source and sink connectors. Because it can stream data asynchronously, delivering events from one producer to multiple consumers, it frees up data environments from the limits of point-to-point transmission. This flexibility is ideal for mixed architectures, allowing interoperability without the need to create, configure, and maintain custom connectors or pipelines. > Even so, using Kafka as a strategic backbone presents challenges. Without the right systems and standards, teams duplicate efforts, misuse data, or struggle to find and trust the streams they need. The absence of clear ownership and access policies can lead to bottlenecks or accidental exposure of sensitive data. What solving it delivers Assuming organizations can standardize, manage, and govern their Kafka infrastructure, they can fill the chasm between obsolete, holdover technologies and newer ones. Benefits of Kafka as a governed integration layer **Future-proofing your environment.** Outdated systems hold organizations back from moving forward competitively. Using Kafka as the connective tissue lets teams progress without abandoning vital, if older, infrastructure. **Migrate and upgrade without disruptions.** Streaming data between systems via Kafka buys time for teams to migrate on their own terms, with minimal impact on business-critical or customer-facing applications. **Unify data across platforms and clouds, and remove vendor lock-in.** Sharing data via Kafka enables it to move seamlessly across clouds, platforms, technologies, and ecosystems, removing silos by provider and freeing teams to work with the most optimal solutions. **Increase efficiency and ROI via reuse and standardization.** Using Kafka as connective tissue means teams stop building redundant pipelines per business unit or use case, and instead consume data as needed, centralizing governance and speeding up developer productivity. Conclusion Operationalizing Kafka at scale with Conduktor Apache Kafka has become essential infrastructure for modern digital environments, and the organizations that run them. But while Kafka moves data exceptionally well, it does not include features for managing that data securely, efficiently, or at scale. Kafka also lacks native tools for governance, cost control, access management, and collaboration, making it risky and expensive for enterprises to use without additional control. Conduktor fills that gap. As a data hub built specifically for Kafka, Conduktor helps organizations transform Kafka from a technical enabler into a true business asset, making it secure, scalable, and aligned to the goals of the enterprise. Each of the five challenges maps to a capability the platform provides. Challenge How Conduktor addresses it **Poor quality data** In-stream observability and validation to improve data quality and prevent downstream issues. **Slow, manual workflows** Standardized self-service abilities, so developers and analysts access data quickly without compromising governance. **A disconnect between teams** Centralized catalogs and metadata tagging that make Kafka data easier to find, understand, and use across teams. **Mystery costs and zombie infrastructure** Identifies unused topics, excessive partitions, and long-retention policies that quietly accumulate, giving platform teams visibility and levers to clean up without disrupting active pipelines. **Fragmented legacy and cloud tech** Enables Kafka to serve as a strategic integration layer across cloud and on-prem environments, replacing brittle point-to-point connections and reducing vendor lock-in through standardized, Kafka-native data sharing. The business impact Accelerate time-to-value and reduce platform and cloud costs. Strengthen compliance and unlock more value from real-time data. Move faster, stay secure, and scale Kafka without losing control. If you are ready to bring clarity, control, and confidence to your Kafka, we would love to show you how Conduktor can help. Related resources [Where Kafka Costs Hide: A Field Guide](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) [The Journey to Self-Service](https://www.conduktor.io/resources/ebooks/the-journey-to-self-service) [Kafka Disaster Recovery Beyond Replication](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-beyond-replication) Ready to operationalize Kafka at scale? See how Conduktor brings governance, self-service, cost control, and integration to Kafka in one platform. Book a demo and we will walk through the challenges that apply to your estate and show you how to solve them. [Book a demo](https://www.conduktor.io/contact/demo?src=resources-ebooks-kafka-at-scale-five-critical-challenges-and-how-to-solve-them) Get the full ebook Source: https://www.conduktor.io/resources/ebooks/kafka-at-scale-five-critical-challenges-and-how-to-solve-them --- # Kafka Disaster Recovery Beyond Replication What's in this paper [Executive summary The coordination gap, what this guide covers, and who it's for](#wp-summary) [01 The real DR problem Outage anatomy, scaling math, and why it matters beyond engineering](#wp-problem) [02 The complete DR strategy Three phases, six technical areas, and a failover runbook](#wp-strategy) [03 Proving it works Chaos testing methodology and compliance evidence](#wp-proving) [04 Making Kafka DR boring The centralized gateway pattern and measured recovery times](#wp-boring) [05 DR readiness assessment Six questions to bring to your team this week](#wp-assessment) Executive summary Most organizations running Kafka in production have some version of disaster recovery in place. However, it's often limited to having a secondary cluster in another region, replication running, and a runbook documented somewhere. If someone asks at a planning meeting whether Kafka DR is covered, the answer is yes but that answer is usually incomplete. It's incomplete because while replication gets the data to a secondary cluster, it doesn't get 50 services pointed at that cluster at 3 AM when half the team is asleep and nobody is sure who can authorize the change. The tasks invoved in assessing the failure and executing the DR runbook typically take 20-40 minutes, but that doesn't take into account the coordination needed: reaching people, getting approvals, sequencing deployments, and verifying each service actually switched. The coordination effort scales linearly with the number of critical applications you run on Kafka and that's where the hours go. This guide covers six technical areas a DR strategy needs beyond data replication (security and identity parity, topic and schema configuration, data protection continuity, observability, client switching, and testing), organized across three operational phases. We spend a fair amount of time on chaos testing as a way to measure your actual recovery time rather than estimate it, partly because it works, and partly because DORA, SOC 2, PCI-DSS, and GDPR all now require measured evidence of recovery capability. This ebook is designed for platform engineers, SREs, and infrastructure architects who own Kafka in production and want to improve their kafka disaster recovery process. If you want to skip ahead to a quick assessment, the DR Readiness Self-Assessment at the end works as a standalone gap analysis. Part 1 The real disaster recovery problem Data replication is not enough If you run Kafka in production, you probably have some version of disaster recovery in place. Two clusters in different regions, MirrorMaker 2 or Confluent Cluster Linking replicating topics and consumer offsets. Maybe a runbook in Confluence, last updated sometime before the person who wrote it changed teams. The secondary cluster has the data and the replication lag is acceptable, so if someone asked you at a planning meeting whether Kafka DR was covered, you would say yes. Copying data to a secondary cluster is necessary but nowhere near sufficient. As one platform architect put it after watching a failover drill go sideways: "Just copying data is useless if your clients can't access the data." The secondary cluster can have a perfect replica of every topic, every consumer offset, every schema, and none of that matters if your 47 services are still pointing at the dead primary and switching them requires a Slack message that starts with "Can everyone please update their bootstrap servers." Data availability is the solved part. Coordination is where failovers actually break down. Anatomy of a real outage On August 28, 2025, [PagerDuty's Kafka cluster failed](https://www.conduktor.io/blog/pagerduty-kafka-outage-lessons). The incident management platform that thousands of companies rely on to tell them when things break could not tell anyone that it was broken. The root cause was a code pattern that created a new KafkaProducer instance for every API request instead of reusing a shared one. At scale, this generated 4.2 million producer instances per hour. Each instance allocated buffer memory, opened broker connections, and registered for metadata tracking. The brokers drowned in overhead, JVM garbage collection spiraled, and the cluster collapsed. At peak, 95% of events were rejected, and the outage lasted over nine hours. PagerDuty runs one of the most critical notification systems on the internet and despite their best efforts and skilled engineers, they still had a major incident. Even organizations with functional replication, correct configurations, and a healthy secondary cluster experience prolonged outages, because the human and organizational response cannot keep pace with the failure. What the first three hours actually look like The timeline of a Kafka outage follows a painfully predictable pattern. The specifics change, but the shape stays the same. **T+0 — Monitoring fires** Assuming your monitoring does not depend on the infrastructure that just failed. PagerDuty's own status page updates were delayed during their outage because their alerting automation also relied on the broken Kafka cluster. This is more common than anyone admits. **T+5 min — Check Slack** Three people are awake, and none of them wrote the runbook. **T+15 min — Read the runbook** It says "switch traffic to the secondary cluster." But you have 47 services, each with bootstrap servers configured in environment variables, Kubernetes secrets, or config files scattered across five repositories. "Switch traffic" is dozens of actions owned by different teams and deployed through different pipelines. **T+30 min — Find the approver** The person who normally approves production changes is on vacation, and their backup is in a different time zone, asleep. Even when you reach the right person, they need context on the failure scope before they will authorize a failover, rightly so, because a premature failover can itself cause data loss. **T+60 min — Coordinate deployments** Deploy pipelines are slow and some services are owned by teams that will not be awake until morning. A few critical services turn out to have hardcoded bootstrap servers outside any config management system. The developer who wrote that code left six months ago. **T+90 min — Discover credential mismatch** Credentials on the secondary cluster do not match the primary. Three services authenticate successfully but cannot write to the topics they need because ACLs were never synced. **T+180 min — Finally recovered** Configs pushed, services restarted, things working. Your "5-minute RTO" was actually 180 minutes, and that is assuming nothing else went wrong. Why recovery time scales with your organization The timeline above is not the failure by specific teams or people, but the predictable outcome of a specific structural problem: recovery time scales linearly with the number of critical applications, not with infrastructure quality. The infrastructure work is typically a fixed-cost effort regardless of the infrastructure. Assessing the failure scope (single broker, partial failure, or full regional outage?) takes 5-15 minutes and executing the DR runbook (stopping replication, converting mirror topics, and verifying data integrity) takes another 15-25 minutes. That adds up to 20-40 minutes regardless of whether you have 5 critical applications or 50. The part that scales linearly is per-project coordination. For each critical application, the platform team needs to assess business impact, brief leadership, reach the on-call engineer, wait for acknowledgment, coordinate deployment sequencing (producers before consumers, dependency chains), then verify each application actually switched and is processing correctly. In practice, that runs 15-45 minutes per critical project. This is also a best case scenario because it assumes things go smoothly. Issues such as credential mismatches between clusters add investigation time; connection pool issues in Java services require restarts that were not in the plan. DNS-based failover, which seems like it should help, runs into a fundamental problem: Kafka clients cache broker connections and do not re-resolve DNS on existing sessions and the JVM caches DNS aggressively by default. This might mean that when you change the DNS record, half your services are still talking to the dead cluster. The infrastructure switch takes seconds, but everything around it takes hours and the driving factor is the size and complexity of your organization, not the quality of your technology. As one DR consultant who has worked through dozens of these incidents put it: "It's not really a technical problem. It's an organizational problem." Why this matters beyond engineering Technical teams measure outages in RTO and RPO, but the rest of the organization measures them in terms of business image. Revenue $1.25M lost in a 3-hour outage for a mid-size e-commerce company processing $10M/day. Organizations with 15-20 critical Kafka services face $150K-$500K/hour total exposure. Trust Trust takes years to build and hours to destroy. Your next outage is your competitor's sales opportunity. How many customers start evaluating alternatives the same week? Compliance DORA, SOC 2, PCI-DSS, and GDPR all require *tested* recovery capability. "We have a plan but haven't tested it" is a finding. A 3-hour recovery against a 15-minute target is also a finding. Human cost Repeated 3 AM pages burn out the engineers you need most. The people who can fix it under pressure are the same people who get paged every time. DR that depends on heroics is a retention risk. **Revenue.** A mid-size e-commerce company processing $10M per day loses roughly $1.25M in a 3-hour outage, before accounting for abandoned carts that never come back. For financial services, three hours of blocked transactions means regulatory gaps and customers who move their money elsewhere. The impact also multiplies across services, because when infrastructure fails it takes down every application that depends on it, not just one. Organizations with 15-20 critical Kafka-dependent services face exposure in the range of $150K-$500K per hour of total outage, depending on how many of those services are customer-facing. Calculate your DR exposure A rough calculation: take your number of critical Kafka-dependent projects, multiply by the estimated hourly business impact per project ($3K-$10K for internal systems, $10K-$25K for production and SLA-bound services, $25K-$100K for revenue-critical systems), and multiply by your honest estimate of recovery time in hours. That is your total exposure per incident. Now multiply by the probability of a major incident in any given year. Most organizations estimate 10-25%. That is your annualized DR risk, and the baseline for any investment conversation. **Trust.** PagerDuty published a transparent postmortem after their August 2025 incident, which was the right call, but how many of their customers started evaluating alternatives that week? Trust takes years to build and hours to destroy. **Compliance.** Regulatory frameworks have moved beyond "do you have a plan?" to "show me when you last tested it." DORA mandates periodically tested ICT continuity plans. SOC 2 Type II requires recovery infrastructure to be maintained and tested, with effectiveness assessed over time. PCI-DSS requires incident response plans to be tested at least annually. GDPR requires regularly testing the effectiveness of security measures. If your answer to an auditor is "we have a plan but have not tested it," that is a finding. If you tested it and it took three hours against a 15-minute target, that is also a finding. **Human cost.** Repeated 3 AM incidents burn out the engineers you need most, because the people who understand the system well enough to fix it under pressure are the same people who get paged every time. As one platform engineering lead observed: "Do we really want to page all our engineers at 3 AM repeatedly? They're just going to burn out." Disaster recovery that depends on heroics is a retention risk, not a strategy. Kafka disaster recovery is almost a victim of Kafka's own success and stability. The clusters run so reliably that the insurance policy rarely gets tested, like health insurance when you are healthy: easy to undervalue, impossible to replace when you finally need it. The difference is that when Kafka disaster recovery fails, the cost is organizational, financial, regulatory, and reputational all at once. Key points **The gap is coordination, not replication.** The infrastructure switch takes seconds. Reaching people, getting approvals, updating dozens of services, and discovering credential mismatches at 3 AM is what takes hours. **Recovery time scales with project count.** Infrastructure work is a fixed 20-40 minutes. Per-project coordination adds 15-45 minutes each. At 20 critical projects, coordination alone exceeds 10 hours. **The PagerDuty incident is the pattern.** Even well-resourced teams with functional replication experience prolonged outages when the organizational response cannot keep pace. **Business impact compounds across four dimensions:** revenue ($150K-$500K/hour for organizations with 15-20 critical services), customer trust, compliance exposure (DORA, SOC 2, PCI-DSS, GDPR all require tested recovery), and engineer burnout from repeated 3 AM incidents. Part 2 The complete disaster recovery strategy Three phases of disaster recovery readiness Disaster recovery is a discipline with three distinct phases, each requiring different work. 01 Prepare Define RTO/RPO per workload Choose an active-passive or active-active replication strategy Set up cross-cluster replication Map dependencies Identify Wave 1 applications 02 Validate Run chaos tests Test monitoring and decision chain Pre-stage DR configurations Measure actual RTO 03 Execute Switch clients via stable endpoint Enforce security during cutover Validate Wave 1 first Communicate to stakeholders Defining your Wave 1 **Which services must recover first?** **Revenue-critical services:** payments, checkout, pricing, order processing. **Authentication and authorization:** if users cannot log in, nothing else matters.. **Critical ETL and data pipelines:** downstream systems that break without upstream data.. **Regulatory:** audit logging, compliance reporting, transaction recording. If Wave 1 services recover in minutes, you have bought yourself time for everything else, so keep the Wave 1 list short and explicit. A good rule of thumb is if it has more than a dozen services on it, you have not prioritized enough. Be sure to evaluate what your critical services depend on and include them in your Wave 1. For example, a payment service that depends on a master data service means both are Wave 1. Test Wave 1 recovery in every drill since Wave 1 recovery time is the number your auditor will ask for. Six areas to verify The six technical areas below are the specific things your DR design review should check to ensure you're fully prepared. Most organizations will have invested heavily in one or two, but the ones that recover in minutes have addressed all six. Area 1: Security and identity parity Teams reliably replicate data but forget the security posture around it. Auth credentials are often cluster-specific and ACLs may exist on the primary but were never provisioned on the DR cluster. During failover, applications authenticate but cannot authorize or get broader access than intended, creating compliance exposure at the worst possible moment. **What to check:** Are TLS certificates and custom CAs deployed to both clusters? Do all authentication methods (mTLS, SASL, OAuth) work on the secondary? Are ACLs and RBAC rules synced automatically, or do they require a manual step that someone could miss? Are cluster-specific credentials (e.g., Confluent Cloud API keys are tied to individual clusters) provisioned for both? Are audit logs available from the secondary cluster for post-failover compliance review? If clients authenticate directly to Kafka, every client needs credentials for both clusters. A proxy layer that decouples client identity from cluster credentials means only the proxy needs dual-cluster access, a simplification that pays off well beyond DR. **What we have seen:** A platform team discovered during their first DR drill that ACLs on their secondary cluster were six months stale. Three services authenticated successfully but could not write to the topics they needed. The root cause was a missing step in their infrastructure-as-code pipeline that synced ACLs to the primary but not the secondary. Area 2: Topic and schema configuration parity Replication tools move data, not configuration, so topic-level settings tend to drift between clusters over time. Additionally, the schema registry is often a separate system entirely and schemas may not be replicated or compatibility rules may differ. That means that after a failover, consumers can fail on schema mismatches, topics might compact unexpectedly, or quotas may throttle catch-up traffic right when you need maximum throughput. **Common sources of configuration drift:** Retention policies or compaction settings changed on primary but not mirrored Dead letter topics that exist on primary but were never created on the secondary Quotas tuned for production traffic on primary, left at defaults on standby Schema Registry compatibility rules that differ between clusters, letting the wrong schema version through Topic-level overrides (e.g., max.message.bytes, segment.ms) applied ad hoc and never documented This kind of drift is very difficult to audit and track, so often they only surface under pressure. Area 3: Data protection and compliance continuity Encryption and masking policies are often applied at the application or infrastructure layer rather than within Kafka itself. When clients fail over to a different cluster or region, those policies do not automatically follow, especially if they are enforced by infrastructure co-located with the failed primary. This creates several risks that compound during failover: **Field-level encryption keys** may be stored in a regional KMS or Vault instance that the secondary cluster cannot reach, causing consumers to receive encrypted payloads they cannot decrypt **Dynamic masking rules** enforced on the primary side may not be replicated, exposing raw PII to consumers that normally only see masked data **Data residency requirements** may restrict certain topics to specific regions. Failing over those topics to a cluster in a different jurisdiction creates a compliance violation, not just a technical issue **Audit trail continuity** breaks if the logging infrastructure that records who accessed what data is tied to the primary cluster's region or network Verify that every data protection policy applied in your primary region has an equivalent enforcement mechanism in your DR region and test that enforcement during drills. Area 4: Observability and operational readiness Monitoring is often configured per-cluster, so if your dashboards, alerts, and on-call routing depend on the same infrastructure that failed, you are flying blind during failover. Separately, teams rarely plan for the burst of catch-up traffic that hits the secondary after the switch, which can cause cascading failures on what was supposed to be the rescue. **Three things to verify:** *1. Monitoring independence.* Does your Kafka metrics pipeline (JMX exporters, Prometheus scraping, Grafana dashboards) survive the failure it is supposed to detect? If it runs on the same cluster or network path as production traffic, you lose visibility exactly when you need it most. *2. Capacity headroom.* Plan secondary cluster capacity for at least 1.5x normal load to absorb the reconnection burst. Every client reconnecting simultaneously generates a spike that can overwhelm a standby cluster provisioned for steady-state traffic. *3. Producer timeout alignment.* Most teams estimate their Recovery Point Objective (RPO), the maximum amount of data they can afford to lose, by looking at cross-cluster replication lag. If the secondary is 30 seconds behind the primary, they assume 30 seconds of data loss in a failover. But replication lag only accounts for data that made it to the primary in the first place. Producers buffer unsent records in memory, and when brokers go down, that buffer fills up and eventually both blocked and buffered records fail with timeout exceptions. The default delivery.timeout.ms is two minutes, meaning producers will silently discard data after two minutes of broker unavailability. If your detection-and-decision window is longer than that (and for most organizations, it is) you are losing data on the producer side before anyone has even decided to fail over. Review your producer timeout settings: if delivery.timeout.ms is shorter than your expected detection-plus-decision window, your actual data loss window is wider than your replication lag suggests. **What we have seen:** A team's delivery.timeout.ms was set to the default two minutes, but their detection-and-decision window averaged eight minutes. During their first chaos experiment, they discovered that producers had been silently discarding data for six minutes before anyone decided to fail over. Their actual data loss was 4x greater than their replication lag alone would have predicted. Area 5: Client switching and traffic routing There is no mechanism built into Kafka to redirect running clients during failover. Every workaround, whether centralized Kubernetes operators, DNS-based service discovery, or custom wrapper code, either requires centralized control of all clients (which most organizations do not have at scale) or bespoke code in every application. **Why common approaches fall short:** **Per-service config updates** require touching environment variables, K8s secrets, or config files across many repositories and deploy pipelines. This is the approach described in most runbooks, and it is why recovery takes hours. **DNS-based failover** seems like it should work, but Kafka clients cache broker connections and do not re-resolve DNS on existing sessions. The JVM caches DNS aggressively by default, leading to an situation where you change the record but half your services keep talking to the dead cluster. **Client-side rebootstrap** (KIP-899, Kafka 3.8+) is an opt-in feature that lets clients fall back to bootstrap servers when all discovered brokers fail. Older clients cannot do this and require restarts. Factor your client version distribution into your RTO estimates. What good looks like: a stable endpoint that clients connect to once, with cluster switching handled behind it, so that failover becomes a single operational decision rather than a per-service coordination exercise. **What we have seen:** A logistics company attempted DNS-based failover across 40 services. After updating the DNS record, 60% of services picked up the change within 10 minutes. The remaining 40% (mostly Java services with aggressive JVM DNS caching) stayed connected to the dead cluster for over an hour. The platform team spent more time debugging DNS propagation than they would have spent on manual config updates. Area 6: Testing and continuous validation The most commonly skipped step, because it is disruptive, expensive, and politically difficult. Under the traditional model, each team participating in a full DR test spends 4-8 hours on pre-test coordination meetings, configuration changes, test execution, verification, and the retrospective. The platform team spends 1-3 hours coordinating each team's participation. At 20 critical projects, a single test costs 100-200+ person-hours, which is why most organizations test annually at best. **What annual-or-never testing actually means:** Configuration drift accumulates unchecked for 12+ months between tests Runbooks decay as team structures, tooling, and approval chains change New services get added to Kafka without being added to the DR plan The first real test happens during the actual disaster Injecting failures at the proxy layer (simulating broker errors, auth failures, latency) without touching production Kafka lowers the barrier significantly. Per-team test participation can drop from hours to under an hour, and teams can pre-validate application resilience through chaos testing in lower environments so that the live DR test becomes a confirmation rather than a discovery exercise. Organizations using this approach move from annual to quarterly testing while spending less total time. Choosing your architecture: active-passive vs. active-active Multi-region Kafka falls into two patterns, each with different tradeoffs. This is a per-workload decision, not a global one. For a deeper technical treatment, see [Multi-Region Kafka: Active-Active vs Active-Passive](https://www.conduktor.io/blog/multi-region-kafka-active-active-passive). **Active-passive** runs one primary cluster that handles all production traffic while a secondary cluster receives replicated data and waits. When the primary fails, you switch traffic to the secondary and consumers resume from replicated offsets. Simpler during normal operations, but harder during failover because switching requires config changes across all services (unless centralized behind a proxy). Best when the application cannot handle conflicts, a single source of truth is required, or the team lacks operational maturity for active-active. **Active-active** runs both clusters simultaneously, with regional producers writing to their local cluster and MirrorMaker replicating bidirectionally. Both clusters have local topics and replicated topics from the other region. Failover for regional traffic is near-instant since it is already flowing, but complexity is higher at all times. Best when users are geographically distributed, data can partition by region or entity, and the application is designed for eventual consistency. A key consideration: MirrorMaker 2 offset sync lags by the checkpoint interval (default 60 seconds), so after failover, consumers may reprocess up to 60 seconds of messages. Active-active also introduces write conflicts that MirrorMaker does not resolve; the application must handle this through regional authority, last-writer-wins, or idempotent event design. Aspect Active-passive Active-active **Recovery Point Objective (RPO)** Replication lag (seconds to minutes) Near-zero for regional data **Recovery Time Objective (RTO)** Minutes to hours (depends on switching mechanism) Seconds (traffic already flows) **Complexity** Lower normally, higher during failover Higher always **Consistency** Strong (single source of truth) Eventual (conflicts possible) For most organizations starting their DR journey, active-passive with centralized switching is the right starting point. It provides strong consistency guarantees and is operationally simpler to reason about. Active-active makes sense when you have users distributed across regions and can design your data model around regional partitioning. The failover runbook If preparation and validation are done well, actual failover should be a calm, sequenced procedure rather than an improvisation. This page should be simple enough that the person on call at 3 AM can execute it without the person who designed it. 01 Detect Confirm incident scope Single broker, partial, or full outage? 02 Decide Approve cutover Keep approval chain short 03 Switch Route via stable endpoint One action, not dozens 04 Validate Wave 1 apps first Don't wait for 100% 05 Stabilize Monitor and adjust Communicate status **Planned vs. unplanned failover:** For planned failovers (DR drills), you can pause producers, wait for replication to catch up, execute the switch, and resume. For unplanned failovers (the primary is dead), you verify secondary health, check replication lag (this is your data loss window), and execute the switch. There is no "migrate consumers first" because the primary is already gone. Accept the replication lag as data loss and cut over. **Failback.** Switching back to the original primary after the incident resolves is often harder than the initial failover, and most teams do not plan for it until they need it. Re-establishing replication in the reverse direction takes time and verification. Data written to the secondary during the incident needs to be reconciled (which cluster is now authoritative?). The secondary may be running at reduced capacity if it was provisioned as a standby rather than a full peer, and a rushed failback under pressure introduces the same coordination risks you just experienced during failover. Failback deserves its own tested runbook. If you have never practiced it, treat it as a separate project. Reserve human judgment for the *decision* to fail over, then let tooling handle the *act* of failing over. Systems do not stress at 3 AM, forget steps, or need approval chains. Key points **Three phases: Prepare, Validate, Execute.** Replication covers roughly one-third of the work. The other two-thirds is security parity, config sync, observability, client switching, and testing. **Six areas to verify.** Security and identity, topic and schema config, data protection, observability, client switching, and testing. Most teams have addressed one or two. Recovery in minutes requires all six. **Wave 1 drives your RTO.** Keep it short (under a dozen services), verify dependencies between Wave 1 services, and test Wave 1 recovery in every drill. **Active-passive is the right starting point** for most organizations. Active-active makes sense when users are geographically distributed and the data model supports regional partitioning. **The failover runbook should fit on one page.** Detect, Decide, Switch, Validate, Stabilize. Reserve human judgment for the decision; let tooling handle execution. Part 3 Proving it works The gap between the plan and reality An untested disaster recovery plan is a set of assumptions about how your system, your organization, and your people will behave under conditions none of them have experienced. Each untested assumption multiplies recovery time, and the gap between what your plan says and what actually happens is invisible until something breaks for real. **Configuration drift is silent.** Topic configs, ACLs, schemas, and quotas drift between primary and secondary clusters over weeks and months. Nobody introduces the drift deliberately, it just accumulates through routine changes that get applied to the primary but not the secondary. Without testing, you discover the drift during the real failover, alongside everything else going wrong. **Runbooks decay.** Written by someone who left six months ago, reviewed by nobody since. A runbook that assumes a toolchain, team structure, or approval chain that no longer exists gives the team false confidence they're prepared for how to respond to incidents. **Compounding assumptions.** Teams assume producer retry logic works under real broker failure conditions, that consumer group rebalancing completes within the RTO window, that monitoring fires when the infrastructure it monitors is the thing that just failed. Each assumption is reasonable in isolation but together, they create an RTO gap that only surfaces under pressure. **The "green dashboard" trap.** Everything on your monitoring dashboard can be green while your DR plan is broken. Monitoring shows averages over minutes or hours, but DR failures happen in the 30-120 seconds between a broker going down and consumer groups stabilizing. That transient window is where data loss, duplicate processing, and cascading failures occur and most monitoring tools sample too infrequently to catch any of it. If your RTO target is 15 minutes but your drill takes 90, that is a major gap, not a minor disagreement. The only way to discover it before a real incident is to test. **The economics of not testing.** DR testing is expensive under the traditional model. Each team participating in a full DR test spends 4-8 hours on pre-test coordination meetings, configuration updates, test execution, verification, and retrospectives. The platform team spends 1-3 hours coordinating each team's participation. At 20 critical projects, a single test costs 100-200+ person-hours before accounting for management and SRE involvement. Multiply by the coordination overhead (management attention, cross-team synchronization, schedule disruption) and the true cost is higher still. This is why most organizations test annually at best, or not at all, and why the first real test often happens during the actual disaster. What chaos testing reveals Chaos engineering means deliberately injecting controlled failures to discover how your system actually behaves, rather than how you think it behaves. State what you expect, inject the failure, observe what happens, document the delta. Kafka makes this harder than stateless services. A web service either returns a 200 or a 500, but a Kafka consumer might silently fall behind, double-process messages, or get stuck in an infinite rebalance loop without producing an immediate, obvious error. Stateful partitions, consumer group coordination, and offset management create failure modes that do not exist in request/response systems. The failure modes that matter for DR go beyond broker availability. Partial infrastructure outages trigger leader elections where producers and consumers see transient errors, and applications that do not retry correctly lose messages or stall. Consumer crashes and rebalances cause duplicate message processing; if the consumer logic is not idempotent, the consequences are concrete: duplicate payments, incorrect stock levels, double-counted orders. Slow brokers during garbage collection or disk contention introduce latency that breaks stream processing joins silently, and invalid schema changes during registry failover stop consumers entirely. These are common Kafka behaviors that most applications have never been tested against. Injecting failures without touching production Traditional chaos tools make this difficult. Killing broker pods, partitioning networks, or running destructive scripts on hosts requires infrastructure access, risks real data loss, and is hard to scope to a single workload. For DR validation the goal is to exercise application behavior under Kafka failure modes, not to prove that the operator correctly terminates a pod. A protocol-aware Kafka proxy changes the shape of the problem. The proxy sits between clients and brokers, intercepts Kafka wire-protocol requests, and fabricates broker responses at the protocol layer: error codes, response delays, duplicated records, corrupted payloads, invalid schema IDs. Brokers stay untouched. Experiments scope to a single virtual cluster, topic, or consumer group, so you can target one workload without affecting the rest of the estate. The practical result is that each scenario in the next section maps to a pre-built interceptor: broken brokers, leader election errors, latency, slow brokers, slow producers and consumers, invalid schema IDs, message corruption, and duplicate messages. Failure type, rate, scope, and duration are declarative configuration rather than code, which turns chaos testing from a quarterly special project into a routine activity. No other Kafka proxy on the market ships this breadth of pre-built chaos interceptors, which is the reason Conduktor Gateway stands out as the tool of choice for Kafka chaos testing. Five failure scenarios to test Scenario What to inject What it reveals Common first finding Broker unavailability Produce/fetch errors at 10-25% Whether clients reconnect cleanly or get stuck Retry storms from aggressive configs Leader election storms Leader errors at 30% Whether alerting distinguishes brief elections from sustained crises Rebalance takes 2-5x longer than expected Latency spikes 500ms-2s delays at 50% Whether timeout configs match RTO expectations delivery.timeout.ms set lower than detection time Schema registry failures Invalid schema IDs on fetch Whether consumers degrade gracefully or crash No fallback when registry is unreachable Duplicate messages Duplicated delivery Whether deduplication and idempotency actually work Payments double-charged, events counted twice **Your first experiment.** Start with broker unavailability at 10% on a non-critical topic in staging. Define your hypothesis: "If 10% of produce requests return errors, our producer retries successfully and no messages are lost." Set abort criteria before starting (consumer lag exceeding 5 minutes or producer error rate exceeding 50% means you stop immediately). Run for 15-30 minutes and document what you expected versus what happened. Then graduate: increase the failure rate, move to a more critical topic, try it in pre-production, and eventually run it as a game day with the full on-call team participating. Time investment for the first experiment: about 2 hours for setup, execution, and documentation. Compare that to the hours you will spend during an unvalidated failover. From resilience testing to compliance evidence Chaos testing is not just an engineering exercise. A single well-documented experiment produces the five artifacts auditors care about: a timestamped test execution record, measured recovery metrics, a gap analysis (expected versus observed), a remediation record, and re-test confirmation. Most teams treat compliance testing and resilience testing as separate workstreams with separate schedules, documentation, and stakeholders, when they are the same workstream. Document your chaos experiments properly and your next audit preparation is already done. Framework What it requires for DR Recommended testing cadence **DORA** Periodically tested ICT continuity plans, at least yearly plus after substantive changes Quarterly experiments + annual game day **SOC 2 Type II** Recovery infrastructure maintained and tested; effectiveness assessed over time Quarterly (showing pattern, not just one test) **PCI-DSS** Incident response plan tested at least annually Annual minimum; isolate payment pipeline specifically **GDPR** Regularly testing effectiveness of security measures On infrastructure change + quarterly The common thread: every framework requires tested and documented recovery capability, not just planned capability. The distinction matters at audit time. For an example in production, see [how Bitvavo built DORA and MiCA compliance evidence from their Kafka operations](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica). The 15-minute experiment report Document each chaos experiment with six fields. This template produces audit-ready evidence and engineering insights from the same activity with minimal effort and maximum utility. Field What to write **1. Hypothesis** "If we inject X failure at Y% rate, we expect Z behavior." **2. Test configuration** What was injected, at what rate, scoped to which topics/services, for how long. **3. Observed behavior** What actually happened. Consumer lag, error rates, rebalance timing, message counts. **4. Delta from expected** Where observed behavior diverged from the hypothesis. **5. Remediation** What configuration, code, or process changes were made as a result. **6. Re-test results** Evidence that the fix worked. Test the decision chain, not just the infrastructure The most important thing to test is the gap between "alert fires" and "someone calls the failover API." This is usually the single longest phase of a real incident, and almost nobody measures it. Test whether your monitoring survives the failure it is supposed to detect. If your Kafka metrics pipeline (JMX exporters, Prometheus scraping, Grafana dashboards) runs on the same cluster or network path as production traffic, you lose visibility exactly when you need it most. Test whether the person on call at 3 AM can actually authorize a failover or whether they need an approval chain that takes 30 minutes, and whether they know who to call, where the runbook lives, and what the approval process is. The game day tip: run a chaos experiment without telling the on-call team in advance. Clear it with management first, start the clock when the fault is injected, and measure the actual time from detection to decision to execution. Then compare that number to the estimate in your DR plan. The results are often sobering. Teams that estimated a 10-minute detection-to-decision window frequently measure 25-40 minutes when tested without warning. The gap comes down to the difference between a rehearsed scenario and the reality of being woken up at 3 AM, finding the right Slack channel, reaching the right person, explaining the situation, and getting authorization. Key points **An untested plan is a hypothesis.** Configuration drift, runbook decay, and compounding assumptions create an RTO gap that only surfaces under pressure. The "green dashboard" tells you nothing about recovery capability. **Chaos testing reveals what monitoring cannot.** Producer timeouts silently dropping data, consumer rebalances taking 2-5x longer than expected, and error handlers swallowing failures are the three findings that surprise teams most. **Test the decision chain, not just infrastructure.** The gap between "alert fires" and "someone calls the failover API" is usually the longest phase. Teams estimating 10 minutes frequently measure 25-40. **Resilience testing is compliance testing.** DORA, SOC 2, PCI-DSS, and GDPR all require documented, measured recovery capability. A single well-documented chaos experiment produces the five artifacts auditors need. **Traditional DR testing costs 100-200+ person-hours** per test at 20 projects. Proxy-layer chaos cuts that to 40-60 hours and makes quarterly testing viable. Part 4 Making Kafka DR boring The architectural shift Every DR challenge described above shares a common structural root: there is no single point of control for Kafka client connectivity. Producers and consumers connect directly to brokers, with credentials, bootstrap servers, and configuration distributed across services, repositories, and teams. When something needs to change, there is no single place to make it happen. The pattern that solves this is a centralized proxy that speaks the Kafka protocol. All clients connect to a single stable endpoint instead of directly to brokers, and the proxy handles cluster routing, credential management, and policy enforcement. Cluster switching happens behind it. This pattern addresses multiple DR areas simultaneously. Client switching (Area 5) becomes a single API call. Security decoupling (Area 1) means only the proxy needs credentials for both clusters. Continuous validation (Area 6) becomes possible through proxy-layer fault injection, simulating broker errors without touching production Kafka. **[Conduktor Gateway](https://www.conduktor.io/gateway)** is the implementation of this pattern. It speaks the Kafka protocol natively, with no client code changes and no special SDK. Applications connect to Gateway the same way they would connect to any Kafka cluster. Gateway is configured with both your primary and secondary clusters, routes all traffic to the primary during normal operation, and failover is one API call: Gateway closes all existing TCP connections, forcing clients to reconnect to the new cluster. It works with any Kafka-compatible system, whether it's cloud-based on on-prem: self-hosted Apache Kafka, AWS MSK, Confluent, Aiven, Redpanda, WarpStream. What this changes for recovery time Phase Without Gateway With Gateway Infrastructure assessment 5-15 min 5-15 min (same) DR runbook execution 15-25 min 15-25 min (same) Per-project coordination 15-45 min x N projects ~1 min App team action work 75-150 min per team 10-20 min per team (observe + verify) Gateway does not solve detection or replication, which remain your responsibility. The fixed costs of infrastructure assessment and runbook execution also remain. What it eliminates is the per-project coordination and action work that dominates RTO at scale, so that recovery time barely grows with project count instead of scaling linearly. Total recovery time by number of critical projects Detecting failures, replicating data, and deciding when to pull the trigger are still hard. The scramble afterward is what goes away. From 3-hour recovery to 15 minutes Consider a composite drawn from patterns we have seen across organizations adopting this approach. A mid-size financial services company with 60+ services on Kafka across two regions, processing payment transactions and regulatory reporting through Kafka pipelines. They started where most organizations start: active-passive setup with MirrorMaker 2, replication running, runbook documented, tested once two years ago. The platform team estimated their RTO at 30 minutes. When they ran their first structured chaos experiment, the findings were typical. ACLs had drifted between clusters. Three services had hardcoded bootstrap servers that were not in the runbook. Their monitoring dashboard depended on the primary cluster's metrics pipeline, and credential management differed between clusters. Their measured RTO was over 3 hours, driven almost entirely by per-project coordination. At 30 minutes of platform coordination per critical project across 18 projects, plus 2 hours of app team action work per team, the coordination and execution dwarfed the 20-minute infrastructure runbook. Annual DR testing was consuming 100+ person-hours per test, so they tested once a year at best, which meant drift accumulated unchecked between tests. After adopting a centralized gateway pattern, they moved to quarterly chaos testing with proxy-layer fault injection, dropping per-team test participation from 6 hours to under 2 hours and documenting experiments for their upcoming DORA compliance audit. The result: sub-15-minute measured recovery time. Infrastructure assessment plus runbook execution plus roughly one minute per project for verification. Quarterly chaos experiments generating both engineering insights and compliance evidence at a fraction of the previous testing cost. The last failover drill was, in one engineer's words, "boring," with one person making one API call and validating in minutes. The real investment was the decision to stop treating DR as a checkbox and start treating it as an operational discipline. The gateway made that discipline economically viable at their scale. The economics of DR testing at scale Traditional DR testing With Conduktor Gateway **Per-team time per test** 4-8 hours 1.5-2.5 hours **Platform coordination per team** 1-3 hours 25-40 minutes **Total person-hours at 20 projects** 100-200+ hours 40-60 hours **Achievable testing frequency** Annual Quarterly Continuous validation The same architecture that simplifies failover also enables the chaos testing methodology from Part 3. Proxy-layer fault injection lets you simulate broker errors, latency, leader elections, and schema failures without touching production Kafka, scoped to specific workloads. The goal is that when Kafka does fail (and it will, because distributed systems fail) recovery is a calm, sequenced, well-practiced procedure. Boring, in the best possible way. For a step-by-step walkthrough of this pattern in engineering terms, see [How Gateway Reduces Kafka DR from Hours to Minutes](https://www.conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes). For the product view with configuration, customer outcomes, and FAQs, see the [Disaster Recovery and Failover use case page](https://www.conduktor.io/solutions/use-case/disaster-recovery-and-failover). Key points **The root cause of slow recovery is fragmented client connectivity.** No single point of control means failover requires touching every service individually. **A centralized gateway eliminates per-project coordination.** One switch, all applications fail over together. Recovery time barely grows with project count instead of scaling linearly. **The fixed costs remain.** Detection, replication, and the decision to fail over are still your responsibility. What goes away is the scramble afterward. **Measured results: sub-15-minute recovery.** Organizations using this pattern report infrastructure assessment plus runbook execution plus roughly one minute per project for verification. **The same architecture enables continuous validation.** Proxy-layer fault injection lets you run chaos experiments without touching production Kafka, making quarterly testing economically viable. Assessment Where do you stand? Bring this to your next design review or DR audit. For each of the six areas, ask three questions: Have you addressed it? Have you tested it? When was the last time you verified it? **Signs you are on track:** Wave 1 applications are defined and explicitly documented. Quarterly chaos experiments are running against critical workloads. Your measured RTO matches your targets (not your estimates, your measurements). Compliance evidence builds automatically from your testing cadence. **Signs you have work to do:** The runbook was last updated more than six months ago, there is no chaos testing cadence, your RTO is an estimate rather than a measurement, the security posture on your secondary cluster has not been verified this quarter, or the person who designed your DR architecture has left the team and nobody has reviewed the plan since. Most teams fall somewhere in between. The [Disaster Recovery Readiness Checklist](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-readiness-checklist) scores your current state across all six technical areas and gives you a structured way to identify where the gaps are and which ones matter most for your risk profile. Six questions to bring to your team this week 1. When was the last time we tested failover end-to-end? Not reviewed the runbook, but actually executed it? 2. Can the person on call at 3 AM authorize a failover without an approval chain? Do they know they can? 3. Do we know the actual rebalance time for our largest consumer group under failure conditions? 4. Have we verified that ACLs, schemas, and credentials on the secondary cluster match production today, not six months ago? 5. If we had to fail over right now, how many teams would we need to coordinate with? How long would that take? 6. Is our delivery.timeout.ms longer than our expected detection-and-decision window? Kafka disaster recovery is a solvable problem. The infrastructure piece, replication, is mature and well-understood. The organizational piece, coordination under pressure, is where most teams get stuck, but the pattern for fixing it is clear: centralize client connectivity, validate continuously, and treat recovery as a measured capability rather than an untested plan. None of this requires rearchitecting your Kafka deployment or rewriting your applications. It requires deciding that DR is an operational discipline worth investing in, then working through the six areas methodically. Start with the gaps that carry the most risk for your organization, test your assumptions with a real chaos experiment, and use the results to build the case for whatever comes next. The teams that recover in minutes got there incrementally, not all at once. Published by Conduktor. For more on Kafka disaster recovery, visit [conduktor.io](https://conduktor.io). Keep reading the full guide The complete DR strategy, chaos testing methodology, failover runbook, and readiness assessment. Source: https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-beyond-replication --- # Kafka Disaster Recovery Readiness Checklist What's in this checklist [01 The three phases Prepare, validate, and execute: the readiness framework](#wp-phases) [02 The full checklist 24 items across the six technical areas, with a scoring guide](#wp-checklist) [03 The failover runbook A simple five-step sequence anyone on call can follow](#wp-runbook) [04 Signs and pitfalls How to tell you're ready, and the common blind spots](#wp-signs) Executive summary Most Kafka teams treat replication as synonymous with disaster recovery, when in reality it only covers one piece of the puzzle. Replication moves data, but it doesn't automatically switch clients, preserve security and governance, keep schemas compatible, or guarantee your on-call team can execute a failover under pressure. Teams that recover in minutes instead of hours have gone beyond replication and addressed six technical areas, and centralized how those areas are managed. This checklist distills those areas into a practical review tool you can bring to your next design review, disaster recovery audit, or game day planning session. It's the companion to our full guide, [Kafka Disaster Recovery Beyond Replication](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-beyond-replication). Use the framework below to frame the work, then run the checklist itself to find your gaps. Part 1 Three phases of readiness Disaster recovery readiness breaks into three phases, each requiring different work. Get all three right and a real failover becomes a calm, sequenced procedure instead of a 3 AM scramble. 01 Prepare Define RTO/RPO per domain and workload Set up cross-cluster replication (MirrorMaker, Cluster Linking) Identify Wave 1 applications for priority recovery Choose per service: standby (active/passive) or both serving (active/active) Map dependencies: apps, topics, schemas, ACLs, certs 02 Validate Run chaos tests in staging, then production game days Measure actual failover time during drills Test the monitoring and decision chain, not just infra Verify the runbook is current and executable by on-call Pre-stage configs in both regions and check for drift 03 Execute Switch clients through a stable endpoint, not per-app Validate Wave 1 apps first, then the long tail Enforce security and governance during cutover Communicate to stakeholders and compliance teams Plan failback once stable; reversing the switch needs its own runbook Before you start **Replication is one-third of the job.** Getting data to a secondary cluster is necessary but nowhere near sufficient; the other two-thirds is security parity, config sync, observability, client switching, and testing. **A plan you haven't tested is a hypothesis.** The only way to know your real recovery time is to measure it in a drill. **The full 24-item checklist is below.** Score yourself against the six areas and focus first where a failure would be unrecoverable. Part 2 The readiness checklist Six technical areas to verify in your disaster recovery design review. Work through each item and mark where you stand. Scoring guide If you can check **20+ items** on this list, you're well ahead of most organizations. **Fewer than 12** means significant gaps remain. Focus first on the areas where a failure would be unrecoverable: client switching, security parity, and testing. Security & identity parity **TLS and certificate strategy.** Covers both clusters, including custom CAs and cert renewal processes. **Authentication parity across regions.** mTLS, SASL, or OAuth configured identically; cluster-specific credentials pre-provisioned. **ACL/RBAC provisioned on secondary.** Least privilege maintained; no broader access on failover than on primary. **Audit logs on secondary.** Available, retained, and accessible during and after failover for compliance evidence. Topic & schema configuration parity **Topic configs match across clusters.** Retention policies, compaction strategies, and quota limits verified in sync. **Schema Registry replicated.** Schemas and compatibility rules match; consumers won't fail on schema mismatches after failover. **Configuration drift detection.** An automated process catches and alerts on divergence between primary and secondary. **Consumer offset sync configured.** Offset translation strategy defined; the reprocessing window is understood and acceptable. Data protection & compliance continuity **Encryption parity verified.** Field-level and payload-level encryption applied consistently across both clusters. **Data masking rules travel with the data.** Policies enforced regardless of which cluster serves traffic, not tied to co-located infrastructure. **PII handling consistent across regions.** Regulated data treated identically, with no exposure risk during the failover window. **Region-specific compliance requirements met.** The secondary site satisfies data residency, sovereignty, and regulatory obligations for all workloads. Observability & operational readiness **Monitoring independent of failed infrastructure.** Dashboards, alerts, and on-call routing don't depend on the region or cluster that went down. **Replication lag tracked continuously.** Cross-cluster lag is your effective RPO metric; alerting thresholds are set and tested. **Secondary cluster sized for catch-up load.** At least 1.5x steady-state capacity to absorb the reconnection burst after failover. **Producer delivery timeouts reviewed.** delivery.timeout.ms exceeds your expected detection + decision window; the 2-minute default is often too short. Client switching & traffic routing **Failover from a single control point.** One action switches all clients, with no per-app config changes, restarts, or cross-team coordination. **Client broker discovery method documented.** How clients find brokers today, and how that changes during failover, is clearly understood. **Failover authority defined.** Clear ownership of who can trigger a switch, with an approval chain short enough for a 3 AM incident. **Kafka client versions audited.** Clients on 3.8+ support rebootstrap (KIP-899); older clients get stuck and require restarts. Testing & continuous validation **Game days include full regional loss.** Partial failure tests aren't enough; validate the worst-case scenario your plan is designed for. **Chaos tests run quarterly at minimum.** Simulate broker loss, region loss, and auth failures for mission-critical workloads. **Named runbook owner with a review cadence.** Runbooks are versioned, reviewed on a schedule, and updated after every incident or drill. **On-call can execute without the author.** The person at 3 AM should be able to follow the runbook without the person who wrote it. Part 3 The failover runbook No heroics. The sequence should be simple enough that the person on call at 3 AM can execute it without the person who designed it. 01 Detect Confirm incident scope Broker failure, partial, or full regional outage? 02 Decide Choose recovery mode and approve cutover Keep the approval chain short 03 Switch Route clients via a stable endpoint One action, not dozens 04 Validate Wave 1 apps first, then the long tail Verify end-to-end data flow 05 Stabilize Monitoring, backpressure, comms Then plan your failback Part 4 How to know you're ready Use these indicators to assess where your disaster recovery program stands today, and where the common blind spots are. ✅ Signs you're on track **Failover time is measured, not estimated.** You've run a drill and have a real number to compare against your RTO target. **Game days run quarterly or more often.** Chaos testing is routine, not an annual event, and runbooks are updated afterward. **Failover is a single operational action.** One decision, one action, all clients move; no per-app coordination. **Security posture survives failover.** Encryption, RBAC, masking, and audit logging are enforced identically on both clusters. **The on-call engineer can execute the runbook.** Anyone on rotation can follow it at 3 AM without calling the author. **Failback has its own tested runbook.** Reversing the switch is often harder than the initial cutover; it's been planned and validated separately. 🚫 Common pitfalls to avoid **Treating replication as the entire DR strategy.** Data arrives at the secondary but clients can't connect, authenticate, or find the right schemas. **Monitoring that depends on what failed.** If dashboards and alerts are in the same region as the primary, you're flying blind during failover. **Failover that requires per-app coordination.** If switching 50 services means 50 config changes across 10 teams, your RTO scales with your org chart. **Producer timeouts shorter than your decision window.** The default delivery.timeout.ms of 2 minutes means producers silently drop data before anyone decides to fail over. **Configuration drift between clusters.** Topic settings, schemas, quotas, and ACLs diverge silently over time, so the secondary behaves differently by failover. **Untested or stale runbooks.** If the last drill was a year ago or the author has left, the runbook is a liability, not an asset. Related resources [Kafka Disaster Recovery Beyond Replication](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-beyond-replication) [The Kafka Debugging Playbook](https://www.conduktor.io/resources/ebooks/the-kafka-debugging-playbook) [Conduktor Gateway](https://www.conduktor.io/gateway) Ready to close the gaps? Book a 45-minute disaster recovery workshop with our team to review your Kafka estate and build a first-pass wave plan. [Book a DR workshop](https://www.conduktor.io/contact/demo?src=resources-ebooks-kafka-disaster-recovery-readiness-checklist) Get the full checklist Source: https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-readiness-checklist --- # The 2025 Data Streaming & AI Report What's in this report [01 The state of streaming Who we surveyed, and how fast adoption is growing](#wp-state) [02 AI raises the stakes How teams feed AI, the tools they use, and the quality issues that follow](#wp-ai) [03 Where scaling breaks The biggest challenges, the governance gap, and ingestion pain points](#wp-scaling) [04 Governance & ROI Why fragmentation hurts competitiveness, and the case for a control plane](#wp-governance) Executive summary Conduktor is the governance hub for streaming data and AI. We commissioned this report to understand where organizations are succeeding, and struggling, with turning streaming data into governed pipelines that improve analytics and decision-making, accelerate AI initiatives, and increase collaboration with external partners. One key conclusion: while most respondents report greater use of data streaming, particularly to feed AI systems with fresh data from multiple sources, the proliferation of streaming platforms creates big problems. Those problems include losing data when consolidating platforms and challenges integrating between systems. In the race to get ahead in AI, organizations cannot ignore the need for strong governance, and risk corporate-wide chaos when they do. 93%use multi-platform data streaming architectures 43%already use data streaming for training or running AI 86%struggle with integration across streaming platforms 98%are concerned about losing data when consolidating Part 1 The state of streaming The report is based on independent research from Pureprofile with 200 senior IT and data executives across Europe and the US. Respondents were drawn from companies with an annual revenue of $50 million or more and over 500 employees, operating in financial services, manufacturing, retail, telecoms, transportation and logistics, and utilities. 200senior IT & data executives surveyed 500+employees (company size) $50M+annual revenue Adoption is accelerating, and so is fragmentation The core tension revealed by our research is that while streaming adoption is accelerating, so too are the complexity and fragmentation that emerge as organizations adopt more platforms. Use multi-platform architectures93% Use more platforms than two years ago77% Consolidated down to one platform16% Use fewer platforms than two years ago7% Larger organizations in particular are consolidating platforms to reduce architectural complexity, lower costs, and cut governance overhead. But for most, the direction of travel is more platforms, not fewer, as teams scale streaming to address new use cases. The picture so far **Streaming is now multi-platform by default.** 93% run more than one platform, and most are adding rather than consolidating. **Fragmentation is the side effect.** Each new platform brings its own governance model, integration surface, and cost profile. **AI is the accelerant.** The rest of this report looks at how teams are feeding AI with streaming data, where it breaks, and what closes the gap. Part 2 AI raises the stakes The adoption of AI is now firmly at the heart of strategic IT and business planning, and data streaming has become a vital element of it, channeling relevant information from multiple sources to decision-makers and models. How organizations use streaming today Automating workflows83% Making real-time decisions51% Training or running AI43% The top three areas where respondents say they've benefited most from streaming are enhancing customer experience with faster, personalized services; improving real-time decision-making; and detecting and preventing fraud or security threats. But integration is where it strains Despite those benefits, the increase in the number of platforms has introduced chaos. **86%** of respondents struggle with integration across different data streaming platforms (29% call it a big issue, 57% an issue). The main reason is that each platform has its own governance model, access controls, encryption standards, auditing formats, schema enforcement, and retention policies. Integration becomes painful not just technically, but at the policy level too. Asked which data streaming capabilities need the most improvement, respondents named **integration with AI/ML platforms** first, ahead of connectors to mainstream applications and security and governance. Even so, 80% rate their current AI/ML integration as "good," 9% "excellent," and 11% "average", a confidence that the operational findings later in this report complicate. The AI tooling landscape Respondents use a wide range of tools for data preparation, training, and inference. Google Cloud Vertex AI66% Microsoft Azure Machine Learning50% Amazon SageMaker Data Wrangler42% AWS Glue DataBrew38% AWS SageMaker37% That breadth of tooling helps explain the quality problems teams report when using streaming data for AI. The top five quality issues were: Inconsistent data formats or schemas Duplicate events causing skewed results Missing or incomplete data Low-quality data affecting model accuracy Irrelevant or outdated data Part 3 Where scaling breaks The findings expose the gap between AI ambitions and operational reality. Respondents rate their integration as good, yet name real-time processing and data quality as their biggest obstacles when scaling data infrastructure to support AI. Data privacy and security concerns72% High infrastructure costs59% Lack of real-time processing58% Data quality issues56% Integration with existing systems49% Talent and expertise shortages7% The governance gap Organizations are focusing on a handful of success indicators, such as data latency and freshness, while ignoring a key fact: **they don't have unified governance across different platforms and technologies.** They need uniform access control, schema enforcement, and retention policies, not one set for each team or environment. Respondents report using a wide range of data lakes and warehouses, which compounds the problem: **Data lakes:** Amazon S3 and/or Lake Formation, Databricks Delta Lake, and Google Cloud Platform. **Data warehouses:** Google BigQuery, Amazon Redshift, and (joint) Azure Synapse Analytics / IBM Db2 Warehouse. Ingestion and transformation pain When moving data from streaming systems into lakes and warehouses, teams reach for a mix of approaches: Custom pipelines (Spark or Flink)73% Kafka Connect or similar tools69% Managed services (Firehose, Snowpipe)50% Micro-batching before loading49% ELT / ETL tools (Fivetran, Airbyte)28% The top three pain points during ingestion and transformation were **time efficiency** (collecting, connecting, and analyzing data in a centralized way), **schema changes and rising data complexity**, and **parallel architectures** that add complexity and require extra resources to manage. Part 4 Governance & ROI Chaos hurts competitiveness The pain points above paint a picture of potential chaos for organizations using streaming to feed AI. The old adage of "garbage in, garbage out" is only amplified as the speed and volume of ingested data spiral. Examples of chaos include: **Poor quality data.** Pipelines ingest inconsistent, misformatted, and missing data at scale, reducing confidence and trust in data by teams and customers. **Mystery costs.** Topics and schemas are created, used, and abandoned without documentation or visibility, and unused assets accumulate unseen, wasting resources. **Slow, overly manual workflows.** Teams lack the systems and standards to let developers access data rapidly and safely, creating productivity bottlenecks. Without the right governance in place, the business encounters increased time to market for new products, inaccurate outputs for real-time AI use cases like fraud detection, and degraded customer experiences. The difficulty of measuring ROI The data streaming and AI gold rush is encouraging significant investment, but questions remain about how return on investment is measured. Platform metrics such as data latency matter, but they don't reveal ROI on their own. Operational success metrics (top 3) Business success metrics (top 3) End-to-end latency (ingestion to insight) Revenue from real-time data services or products Data freshness (event creation to availability) Customer satisfaction / NPS from real-time experiences System uptime and failure recovery times Real-time decision-making accuracy or speed Operational KPIs help deliver the business metrics, but without attribution it's difficult to link streaming success to indicators such as revenue or customer churn. The need for a unified control plane Platform and security teams need to work together, unifying fragmented deployments into a single, cohesive environment. This is where [Conduktor](https://www.conduktor.io/) comes in. It's platform-agnostic, unifying governance across streaming Kafka environments, with federation built in to devolve responsibilities to developer and data teams while still letting platform teams implement guardrails. Conduktor is also bringing clarity and granularity to cost tracking, which simplifies cost attribution, ROI assessment, and forecasting. The bottom line **Adoption is outpacing governance.** Streaming is the connective tissue between operational systems, analytics, and AI, but most organizations lack the governance, integration, and observability to make it work together. **Fragmentation has a cost.** Without a common control plane, the growth of streaming platforms turns into chaos, wasted investment, and lost market opportunities. **A control plane closes the gap.** Conduktor turns fragmented Kafka deployments into governed, AI-ready data platforms. Related resources [Governing Kafka Data for the AI Era](https://www.conduktor.io/resources/ebooks/governing-kafka-data-for-the-ai-era) [Federating Data Governance](https://www.conduktor.io/resources/ebooks/federating-data-governance-scaling-kafka-without-losing-control) [Where Kafka Costs Hide: A Field Guide](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) Turn fragmented streaming into governed, AI-ready data See how Conduktor unifies governance, quality, and cost visibility across your streaming estate, with no application changes. [Book a demo](https://www.conduktor.io/contact/demo?src=resources-ebooks-the-2025-data-streaming-ai-survey-report) Get the full report Source: https://www.conduktor.io/resources/ebooks/the-2025-data-streaming-ai-survey-report --- # The Journey to Self-Service The six phases [01 Assess & discover Measure usage, ownership, and where the bottlenecks are](#wp-assess) [02 Planning & standards Ownership, naming, tagging, and resource policies](#wp-planning) [03 Stakeholder alignment Buy-in, platform-team reframing, and enablement](#wp-stakeholder) [04 Implementation Portal, guardrails, pilot, and legacy topic cleanup](#wp-implementation) [05 Change management Training, support, feedback, and adoption metrics](#wp-change) [06 Optimization & maturity Consolidation, automation, and a center of excellence](#wp-optimize) Introduction Platform teams become bottlenecks when every Kafka request requires manual approval. Transitioning to self-service isn't as simple as flipping a switch; it's a process. This checklist walks through the key considerations and steps to transform your Kafka environment, organized across six phases. A note on sequencing: phases 3–5 can happen in parallel. Stakeholder alignment, in particular, is an ongoing effort, and in organizations with larger or more complicated environments, implementation is often staggered across teams so it doesn't disrupt operations. 01Assess & discover 02Planning & standards 03Stakeholder alignment 04Implementation 05Change management 06Optimization & maturity Phase 01 Assess & discover You can't fix what you can't see. Start by measuring how Kafka is actually used, who owns what, and where the delays come from. **Conduct usage analysis.** Calculate partitions divided by throughput across all clusters over 90–180 days to identify underutilization. **Map topic ownership.** Identify the service accounts with write permissions on each topic and trace them back to application teams. **Evaluate Conway's Law tension.** Map your Kafka architecture against your org structure to find mismatches between infrastructure and business units. **Audit cluster sprawl.** Document all clusters and determine which were created for business reasons versus actual capacity needs. **Measure current bottlenecks.** Track time-to-provision for topics and clusters, and count platform-team tickets for routine requests. How to use this checklist **Self-service is a journey, not a destination.** Start small, learn as you go, and iterate based on what works for your organization. **Phase 1 is your baseline.** Complete the assessment above before designing standards, so your decisions are grounded in real usage data. **The full six-phase checklist follows**, through planning, stakeholder alignment, implementation, change management, and long-term maturity, plus success indicators and the pitfalls to avoid. Phase 02 Planning & standards Self-service without standards just moves the chaos closer to the developer. Set the rules of the road before you open the gates. **Define a topic ownership model.** Establish that the producer application/team owns the topic, with a clear process for ownership disputes. **Create naming conventions.** Design topic naming standards (e.g., ..) and conventions for consumer groups. **Implement a tagging strategy.** Define required tags (owner, cost-center, environment, criticality) and build a searchable taxonomy. **Set resource policies.** Define partition limits, retention policies by topic type, and tiered service levels (dev/staging/production). **Address "Gordian knot" topics.** Identify monolithic topics and decide whether to use topic filtering (SQL/CEL) or split them, based on throughput requirements. Phase 03 Stakeholder alignment The hardest part of self-service is rarely technical. Bring leadership and the platform team along before you migrate anything. **Secure executive buy-in.** Present the cost-savings case from better utilization and demonstrate business-velocity improvements. **Address platform-team concerns.** Reframe their role from gatekeeper to enabler, focusing on high-value work like failover, security, and observability. **Plan the internal campaign.** Develop a communication plan with clear migration deadlines and identify champions in each business unit. **Prepare training materials.** Create user guides, video tutorials, and FAQ documentation for development teams. Phase 04 Implementation Now stand up the platform and prove it on a real workload before scaling out. **Configure the self-service platform.** Set up the portal and tools with RBAC policies, permissions, and approval workflows. **Enable visibility and governance.** Implement topic discovery, usage monitoring, and dashboards for ownership and utilization. **Build safety rails.** Configure automated policy enforcement, quota management, and validation for naming conventions. **Start with a pilot application.** Migrate the first application end-to-end, document lessons learned, and refine the playbook. **Handle legacy topics.** Contact owners of non-compliant topics, provide a migration path, and set a deadline for compliance. **Address orphaned topics.** Identify topics with unclear ownership and establish a decommissioning process for unused ones. Phase 05 Change management Adoption is a capability-building exercise, not a tool rollout. Support people through the shift. **Deliver hands-on training.** Conduct workshops for common self-service tasks. **Document as much as possible.** Create quick-start guides and repo templates. **Establish support channels.** Set up a dedicated channel (Slack/forum) and schedule office hours with the platform team. **Gather and act on feedback.** Survey users about pain points and iterate on process improvements. **Track adoption metrics.** Monitor teams using self-service, time-to-provision improvements, and the reduction in platform-team tickets. Phase 06 Optimization & maturity Once self-service is live, keep tightening the loop, and free the platform team to work on what matters. **Optimize cluster utilization.** Review partition usage quarterly and consolidate underutilized clusters using virtual clusters where appropriate. **Enhance multi-tenancy.** Review isolation requirements and implement advanced quota management for better resource sharing. **Automate governance.** Implement automatic compliance scanning and policy-as-code enforcement. **Build a center of excellence.** Document patterns and anti-patterns, and share lessons across teams. **Evolve the platform team's focus.** Engineers shift to strategic activities like security, disaster recovery, and architecture guidance. Scorecard How to know it's working ✅ Success indicators New topics are provisioned in minutes, not days. Platform-team tickets drop 60%+ for routine requests. Development teams ship features faster. Cluster utilization improves measurably. You can answer "who owns this topic?" instantly. Data becomes more discoverable, and utilization rates increase. 🚫 Common pitfalls to avoid **Enforcing perfect standards before starting.** Start imperfect and improve iteratively. **Not addressing platform-team concerns early.** Engage them so they don't become obstacles. **Migrating everything at once.** Start with a pilot, learn, then scale. **Insufficient training.** Self-service is capability building, not just tool deployment. **Standards without enforcement.** Automate validation; don't publish guidelines and hope. Next steps Self-service is a journey, not a destination. Start small, learn as you go, and iterate based on what works for your organization. Complete your assessment using Phase 1 of this checklist. Identify your pilot application for the first migration. Join the conversation at upcoming Conduktor events. Related resources [Federating Data Governance](https://www.conduktor.io/resources/ebooks/federating-data-governance-scaling-kafka-without-losing-control) [Kafka at Scale: 5 Critical Challenges](https://www.conduktor.io/resources/ebooks/kafka-at-scale-five-critical-challenges-and-how-to-solve-them) [Conduktor Console](https://www.conduktor.io/console) Turn your platform team from gatekeeper to enabler Connect with the Conduktor team to discuss your specific challenges and learn from organizations that have successfully rolled out Kafka self-service. [Book a demo](https://www.conduktor.io/contact/demo?src=resources-ebooks-the-journey-to-self-service) Get the full checklist Source: https://www.conduktor.io/resources/ebooks/the-journey-to-self-service --- # The Kafka Debugging Playbook: Six Phases from Alert to Root Cause What's in this paper [01 The Problem Why Kafka monitoring and Kafka troubleshooting are different capabilities](#wp-problem) [02 The Framework Detect, Scope, Inspect, Trace, Resolve, Prevent](#wp-framework) [03 Incident 1: Poison Pill One bad message takes down an entire consumer group](#incident-1) [04 Incident 2: Dead Sink Task A connector says "Running" while a task is dead](#incident-2) [05 Incident 3: New Enum Value BACKWARD compatibility passes, old consumers still break](#incident-3) [06 The Pattern Three incidents compared. Where the diagnostic chain breaks and why.](#the-pattern) **This whitepaper is the culmination of our three-part blog series on Kafka debugging.** If you want the practitioner-level walkthroughs first, start here: [Why Every Kafka Incident Ends with "Restart It"](https://www.conduktor.io/blog/why-every-kafka-incident-ends-with-restart-it) [Houston, We Have 7 CLI Tools and Zero Answers](https://www.conduktor.io/blog/houston-we-have-7-cli-tools-and-zero-answers) [Ground Control to On-Call: Kafka Debugging Has Landed](https://www.conduktor.io/blog/what-debugging-kafka-should-actually-look-like) The Problem Most Kafka teams can detect an incident in seconds and resolve it in hours The alert fires. The Kafka monitoring dashboard confirms the symptom. Then the investigation stalls. Because Kafka observability tools and Kafka troubleshooting tools are completely different capabilities, and most teams running Kafka in production only have the first one. Monitoring tells you *that* consumer lag is growing. Debugging tells you *why*: which partition is stuck, what message is at the stuck offset, whether the schema changed, whether a connector task died. These are the questions that actually resolve a Kafka incident and reduce MTTR. The native Kafka CLI tools and dashboards make every one of them painful. Here's what a typical investigation looks like with the tools Kafka ships: | Question | Tool | The catch | |---|---|---| | Which consumer group is lagging? | `kafka-consumer-groups.sh --describe` | Raw offset numbers. No timestamps, no growth rate, no history. | | Which instance owns the stuck partition? | `kafka-consumer-groups.sh --members --verbose` | Same tool, different flags, completely different output format. You cross-reference by hand. | | What's in the stuck messages? | `kafka-console-consumer.sh` | Binary garbage if the topic uses Avro. Need a separate tool from a separate distribution. | | Did the schema change? | `curl` (Schema Registry REST API) | No CLI. Two API calls, pipe to files, run `diff`. Compatibility check needs escaped JSON nobody types correctly. | | Is the connector healthy? | `curl` (Connect REST API) | No CLI. Different API, different port. A "RUNNING" connector can have failed tasks buried in the response. | | What happened an hour ago? | Nothing | The CLI shows the present. If you didn't already have Prometheus and Grafana set up, there's no history. | Six questions, five different tools, no shared context between any of them. You carry the thread in your head. These tools were built in 2011 when Kafka was an internal LinkedIn project. The assumptions were reasonable then. They're not anymore. | 2011 assumption | 2026 reality | |---|---| | One team runs Kafka | Dozens of teams produce and consume | | Kafka admins debug Kafka | DevOps, SREs, and application developers debug Kafka | | Debugging = cluster operations | Debugging = understanding application data | | Plain text messages | Avro, Protobuf, JSON Schema with evolution | | Few topics, few consumer groups | Hundreds of topics, complex dependency graphs | The result is predictable. When the cost of a proper investigation exceeds the cost of restarting the consumer, the consumer gets restarted. The symptom clears. The root cause goes unidentified. And the same class of incident repeats. The Framework Every Kafka investigation follows the same six phases Regardless of whether the root cause is a poison pill, a dead connector task, a schema mismatch, or a slow consumer, the shape of the investigation is the same. The details change. The sequence of questions doesn't. 1 Detect What's wrong? Where do I start? 2 Scope How bad is it? What's affected? 3 Inspect What does the data look like? What's at the stuck offset? 4 Trace Where did it come from? What changed upstream? 5 Resolve How do I fix it safely? What's the blast radius? 6 Prevent Why did this happen? How do I stop it repeating? Why six phases and not three or ten? Because each one produces a specific output that the next phase needs as input: | Phase | Output | Feeds into | |---|---|---| | Detect | The consumer group and topic that are affected | Scope (where to look) | | Scope | Which partitions, the group state, acute vs chronic | Inspect (what to look at) | | Inspect | The actual data at the stuck offset | Trace (which direction to investigate) | | Trace | The upstream cause (schema, connector, producer) | Resolve (what to fix and how) | | Resolve | The immediate fix, applied safely | Prevent (what to change so it doesn't repeat) | | Prevent | Alert tuning, config changes, code fixes | Next incident starts from a better baseline | Skip Scope and you don't know which partition matters, so Inspect searches blind. Skip Inspect and you don't know whether the problem is in the data or the infrastructure, so Trace chases the wrong branch. Each phase narrows the search space for the next. The sequence isn't arbitrary. Here's what that looks like in practice. The whitepaper walks through three real incidents, each branching at a different phase... Incident 1 The poison pill: a single message takes down an entire consumer group The setup A JDBC source connector reads new rows from a PostgreSQL orders table and produces Avro-encoded messages to an 8-partition Kafka topic. Downstream, three consumer instances in the order-processing-v2 group process each order: payment API calls, inventory updates, then commit. Auto-commit is off. The consumer commits offsets after successful processing, which is standard for anything doing real work per message. The pipeline. PostgreSQL feeds through a JDBC source connector into an 8-partition Avro topic. Three consumer instances split the partitions. Partitions 6 and 7 (assigned to instance 3) are the problem. Sunday night, a refund order with a negative amount value lands in the database. The connector writes it to partition 7 like any other row. Monday morning, instance 3 picks up the message. The payment logic doesn't handle negative amounts. The API rejects it, the error handler treats it as fatal, the consumer crashes. Because auto-commit is off and the consumer commits after successful processing, the offset for partition 7 never advances. On restart, the consumer pulls the same message and crashes again. This is a poison pill: a message that a consumer can't process, creating an infinite crash loop. Why this becomes group-wide disruption Here's where the Kafka mechanics matter. Every time instance 3 crashes, the consumer group coordinator detects the lost heartbeat and triggers a rebalance. This group uses the default eager rebalancing protocol (RangeAssignor), which means each rebalance **revokes all partitions from all consumers** and reassigns them from scratch. During the rebalance window, no consumer in the group processes anything. Even the two healthy instances are paused. Instance 3 crashes every 30-60 seconds (startup time plus one failed poll cycle). Each crash triggers a rebalance. Each rebalance pauses the entire group for several seconds. The result: lag builds on partition 7 (because the offset never advances), and lag spills across the other partitions (because the healthy consumers keep getting interrupted). **Cooperative rebalancing** (CooperativeStickyAssignor) or **static group membership** (group.instance.id) would reduce the blast radius. Only the affected partitions get reassigned, and the healthy consumers keep processing. But most consumer groups in the wild still use the default eager protocol. If yours does, a single stuck consumer becomes group-wide disruption. The cascade **Refund order with negative amount hits the database** JDBC connector writes it to partition 7 like any other row **Instance 3 processes the message. Payment API rejects negative amount** Error handler treats it as fatal. Consumer crashes without committing offset. **Restart pulls the same message. Crashes again. Poison pill loop.** Offset never advances because commit happens after processing, and processing never succeeds. **Each crash triggers a full consumer group rebalance** Eager protocol revokes all partitions from all instances. Healthy consumers pause during every rebalance. **Alert fires: consumer lag exceeds threshold across the group** The symptom is group-wide lag. The cause is one message on one partition. The alert says "consumer lag is high." That's accurate but useless for diagnosis. Here's how the six phases close the gap between the alert and the root cause. Phase 1: Detect The alert arrives. Where do you start? The alert lands in Slack: consumer lag on order-processing-v2 for the orders topic has crossed the threshold. The on-call engineer sees the consumer group name and the topic. The alert itself isn't the interesting part. The transition to investigation is. Most teams hit a handoff gap at this exact moment. The alert came from the monitoring stack (Prometheus, Datadog, PagerDuty). The investigation needs to happen somewhere else entirely. The engineer opens a terminal, types kafka-consumer-groups.sh --describe, and they're already in a different context from where the alert was. The alert tells you where to look, but the tool that can answer "why" lives somewhere else. With native tooling Open a terminal. SSH to a machine with the Kafka CLI. Run kafka-consumer-groups.sh --describe --group order-processing-v2. Hope you have the right bootstrap server. The investigation starts 2-5 minutes after the alert, in a completely different environment. With Conduktor Console The [alert](https://docs.conduktor.io/guide/monitor-brokers-apps/alerts) links directly to the [consumer group view](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/consumer-groups). Click through. The investigation starts in the same place the alert pointed to, with zero context switch. Alert detail in Console. Consumer group, topic, metric, and delivery status all in one view. The engineer clicks through from here to start investigating. Every minute between alert and investigation is a minute where the engineer is navigating tooling, not diagnosing the problem. Phase 2: Scope How bad is it, and what's actually affected? Most teams skip this phase. It's the most expensive one to skip. Without scoping, you don't know if the problem is one partition or all eight. You don't know if lag is growing or recovering. You don't know if the consumer group is stable or mid-rebalance. Any action you take without this information is a guess. Scoping answers three questions: First: which partitions are affected? The consumer group has eight partitions. You need per-partition lag, not aggregate lag. An aggregate number of 12,400 could mean one partition with 12,400 lag and seven partitions at zero, or it could mean all eight partitions slowly falling behind. Completely different problems. Different root causes. With native tooling kafka-consumer-groups.sh --describe gives you per-partition lag as raw offset numbers. No timestamps, no growth rate. To see which consumer instance owns which partition, run a *second* command: --members --verbose. Different output format. You're the join engine. With Conduktor Console One table: partition, assigned member, current offset, end offset, offset lag, time lag. Partition 7 stands out immediately: lag of 12,400 and growing, assigned to instance 3. In this case, per-partition lag reveals the pattern instantly. Partition 7 has massive lag and it's growing. Partition 6 has elevated but stable lag. It's also assigned to instance 3, so it's falling behind because instance 3 keeps crashing. The other six partitions show brief spikes that recover: the signature of repeated rebalances disrupting otherwise healthy consumers. Second: what's the consumer group state? It reads "Rebalancing." Not "Stable." Something is triggering repeated rejoin cycles. If you'd jumped straight from the alert to restarting the consumer, you wouldn't know the group is mid-rebalance, and your restart would trigger yet another one. Making things worse, not better. Third: is this acute or chronic? The lag graph shows a spike that started 30 minutes ago. But zoom out and there's a second story underneath. Lag had been creeping upward for days, well below the alert threshold but visible in hindsight. The spike is the poison pill. The drift is something else. Consumer group detail for order-processing-v2. Per-partition lag and the lag graph over time. **Remember the drift.** It has nothing to do with the poison pill. It's a separate, chronic problem: the consumer group gradually falling behind as throughput increases. It comes back in Phase 6. For now, the spike is the priority. After Scope, the investigation has narrowed from "consumer lag is high" to "partition 7 is stuck, assigned to instance 3, group is rebalancing, and there's a chronic drift underneath." That's the search space for the next phase. **Without Scope:** You open the topic and start browsing messages across all eight partitions, looking for something that looks "wrong." You don't know which partition matters, so you're searching blind. Or worse, you see the rebalance spikes on the healthy partitions and assume the problem is group-wide. Twenty minutes into investigating broker throughput when the answer is one bad message on one partition. Phase 3: Inspect What's at the stuck offset? Scope told you partition 7 is stuck and gave you the committed offset. Now look at the message at that offset. This is where you go from "something is stuck" to "here is the thing that is stuck." Open the orders topic. Filter to partition 7. Jump to the committed offset. With native tooling The topic uses Avro. kafka-console-consumer.sh shows binary garbage: ??orders?abc-123PENDING?L????? You need kafka-avro-console-consumer from the Confluent distribution. A separate binary with its own Schema Registry URL, deserializer config, and authentication flags. You can technically jump to a specific offset (--offset 48293 --partition 7 --max-messages 1), but you're stitching the command together from the offset you read in a different tool, for a partition you identified in yet another tool, with flags you may not have memorized. There's no filtering, no browsing, no way to see surrounding messages for context. With Conduktor Console Filter to partition 7, jump to offset. Auto-decoded Avro. Every field readable. The message is right there. Partition 7 at the stuck offset. The poison pill: ord-98765, amount -42.50, status REFUND_PENDING. The messages before and after it are normal positive amounts. There it is. orderId: "ord-98765", amount: -42.50, status: "REFUND_PENDING". A negative amount. The messages before and after it are normal: positive amounts, standard order data. One record with a value the consumer's payment logic doesn't handle. The consumer isn't slow or overwhelmed. It's stuck on a single message it physically can't process. The error handler treats the failure as fatal (rather than routing the message to a dead letter topic and moving on), so the consumer crashes. Auto-commit is off, commit happens after processing, and processing never succeeds, so the offset for partition 7 never advances. On restart, the consumer fetches from the last committed offset. Same message. Same crash. Textbook poison pill. But finding the bad message isn't enough. You need to know: is this message even valid? Did the schema change to allow something it shouldn't? Or is this valid data that the consumer should have handled? **Skip Inspect and here's what happens:** you see the consumer crash-looping and jump to checking the connector, the schema, the broker health. Everything looks fine because everything *is* fine. The problem is in the data, not the infrastructure. You waste 30 minutes ruling out causes that were never in play, because you never looked at the actual message. Phase 4: Trace Where did it come from, and what changed? You have the bad message. Now trace backward: is the data valid against the schema? Did the schema change? Is the connector that produced it healthy? Check the schema Open the orders-value schema subject. Two versions registered. Diff them side by side: Schema subject orders-value. The amount field is a double. Compatibility mode is BACKWARD. Version 2 added an optional notes field with a default value. The amount field is untouched, a double in both versions. Avro constrains types, not value ranges. A double accepts any 64-bit floating point number, positive or negative. The schema has no way to say "amount must be positive." A negative amount is perfectly valid. The gap is in the consumer's application logic, not the data contract. The two versions are backward-compatible (adding an optional field with a default is always backward-compatible in Avro). Ruled out in ten seconds. With native tooling Two curl calls to the Schema Registry REST API. Download both versions, pipe to files, run diff. The compatibility check endpoint exists but requires a POST with escaped JSON that nobody types correctly on the first try. Most teams skip it. With Conduktor Console Open the [schema subject](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry). Select two versions. Side-by-side diff shows the change immediately. Compatibility check is one click. Check the connector The orders topic is fed by a JDBC source connector. One task, running, no errors. The connector faithfully produced what was in the database, including the refund order with a negative amount. The connector isn't the problem. One thing to understand about Kafka Connect: it reports status at two levels. The connector level and the task level. A connector can show RUNNING while one of its tasks is FAILED. In this case, the single task is healthy. But if you'd only checked the connector-level status (which is all you get from a quick curl without expanding the task list), you'd miss a failed task entirely. Incident 2 is built around exactly this trap. So: a single message with a negative amount that the consumer's payment logic can't process. The schema allows it. The connector produced it faithfully. The consumer is the only component that can't handle this data. **Skip Trace** and you find the bad message, skip the offset past it, and call it done. The consumer unblocks. But you never checked whether the schema changed or the connector is healthy. If the schema started allowing something new, more bad messages are coming. If the connector has a problem, you're back here in an hour. Trace is what turns a quick fix into an actual root cause. Phase 5: Resolve Skip the poison pill. Verify the fix. The consumer is stuck and every crash disrupts the entire group. The immediate goal is to unblock it. Stop the consumer group. Open the offset reset panel. Select partition 7, advance the offset by one past the blocking message. An offset reset changes the committed offset for a consumer group on a specific partition. It doesn't move or delete the message. The poison pill is still sitting at its original offset in partition 7. What changes is where the consumer starts reading next time it polls. You're telling it "skip this one, start from the next." Get this wrong and you cause real damage. Reset the wrong partition, or reset all partitions when you meant one, and you've either skipped messages (data loss) or forced reprocessing (duplicates). The difference between a safe reset and a dangerous one is whether you can see exactly what will change before you execute. With native tooling kafka-consumer-groups.sh --reset-offsets --to-offset 48294 --topic orders:7 --group order-processing-v2 --dry-run The --dry-run shows you the target offset but not what you're skipping. How many messages? What's in them? You won't know until after --execute. Also: the consumer group must be stopped first, and the CLI won't stop it for you. That's a separate operation. With Conduktor Console Select partition 7. Strategy: shift by 1. Preview shows exactly what changes: current offset, target offset, one message affected. Execute only after reviewing. Multiple strategies available: shift by N, to timestamp, to earliest/latest, to specific offset. Offset reset with preview. Partition 7, shift by 1. The preview confirms: one message skipped, from offset 48293 to 48294. Execute the reset. Restart the consumer. It picks up from offset 48,294 and processes normally. The rebalance storm stops. Lag on the other partitions clears within seconds as the healthy consumers catch up without interruption. Before closing the incident, note the skipped message for the post-mortem: the offset, the payload, the reason it failed. In a more mature setup, the consumer itself would have dead letter topic handling: messages that fail processing N times get automatically produced to a DLT instead of blocking the partition. This group doesn't have that yet, which is precisely why the poison pill created a crash loop instead of being routed aside. Produce a test message with a negative amount to verify the failure mode is reproducible. This confirms the eventual code fix (adding validation for negative amounts) will cover the right edge case. Phase 6: Prevent Find the pattern before it fires again Remember the lag drift from Phase 2? The slow upward trend hiding underneath the spike? That drift had nothing to do with the poison pill. The consumer group was gradually falling behind as throughput increased. Processing time per message was creeping up as the order volume grew, and the group didn't have enough instances to keep up. Nobody noticed because the drift was below the alert threshold. If the drift had continued for another week, the same alert would have fired for a completely different reason. The engineer would have been looking for a stuck message or a dead connector when the real problem was just inadequate consumer throughput. The poison pill was the acute failure. The drift was the chronic one hiding underneath it. Three things need to happen after this incident. The consumer needs validation for negative amounts: either handle them (route to a dead letter topic, log and skip) or fail gracefully instead of crash-looping. That's application code, not Kafka configuration. The consumer group also needs dead letter topic handling. Any consumer doing real work per message should route unprocessable messages to a DLT after N retries. That turns a poison pill from "group-wide outage" into "one message flagged for review, processing continues." And then there's the drift. Tune the alert threshold against the historical baseline so it fires on behavior change (lag growing faster than the trend), not when a static number is crossed. Scale the consumer group. Push the lag metrics into Prometheus and Grafana so the team's existing dashboards show what this investigation surfaced. There's a fourth angle that most teams don't consider until after the first poison pill: data quality at the pipeline level. Instead of fixing consumers to tolerate bad data, prevent bad data from reaching them. [Conduktor's data quality policies](https://docs.conduktor.io/guide/use-cases/observe-data-quality) let you define rules like "amount must be positive" on the orders topic. In observe mode (Console only), violations are flagged and tracked in a dashboard after production. You'd see the negative-amount message show up as a violation, and you'd know about it before the consumer crashed. In [enforce mode](https://docs.conduktor.io/guide/use-cases/enforce-data-quality) (with [Conduktor Gateway](https://conduktor.io/gateway)), the message never reaches the topic. Gateway intercepts it before production and either blocks it (the producer gets an error) or marks it with a header that downstream consumers can use to route it. The poison pill would have been stopped at the gate. If this prevention layer had been in place before Sunday night, the team would have caught the drift mid-week and scaled the consumer group before the poison pill ever arrived. And with data quality enforcement on the topic, the negative-amount message would have been blocked or flagged before any consumer saw it. What happened Root cause: one message with a negative amount that the consumer's payment logic couldn't handle. Symptom: lag across the entire consumer group. The distance between those two facts (one bad record and group-wide disruption) is what made this hard to diagnose with seven CLI tools and straightforward to diagnose by following the chain. The investigation took minutes. It didn't require tribal knowledge about which CLI flags to use or which REST API endpoint returns task status. Just a sequence of questions, each answer pointing to the next. Incident 2 The invisible failure: a sink connector task that died while the connector said "running" The setup The orders topic also feeds an Elasticsearch sink connector (es-orders-sink) that indexes every order for the customer support team's search interface. The connector has three tasks. The Connect framework assigns each task a subset of the topic's eight partitions: task 0 gets partitions [0, 1, 2], task 1 gets [3, 4, 5], task 2 gets [6, 7]. Each task maintains its own consumer position on its assigned partitions. The pipeline. Everything upstream is healthy. Task 1 lost its connection to Elasticsearch during a node replacement. The connector still reports "Running." Wednesday afternoon, the Elasticsearch cluster undergoes a brief network partition during a node replacement. Task 1's connection drops. The task encounters an unrecoverable ElasticsearchException and transitions to FAILED. Tasks 0 and 2 are connected to different Elasticsearch nodes and continue indexing normally. Phase 1: Detect A different consumer group, the same alert pattern The alert fires: consumer lag on the connect-es-orders-sink consumer group is growing. This isn't the order-processing-v2 group from Incident 1. It's the sink connector's internal consumer group. Every Kafka Connect sink connector creates its own consumer group (named connect-{connector-name} by default) to track offsets on the partitions it reads from. The on-call engineer might not immediately recognize this as a connector issue. The alert looks identical to any other consumer lag alert. The framework starts the same way regardless. Phase 2: Scope Lag on three partitions. The rest are fine. Per-partition lag: partitions 3, 4, and 5 are growing steadily. The other five partitions show zero or near-zero lag. Consumer group state: Stable. No rebalances. Two things stand out compared to Incident 1. The group is Stable. No crash loops, no rebalances. Whatever is wrong isn't causing a cascade. And lag is growing because the end offset keeps advancing (producers are still writing to all eight partitions) but the committed offset on partitions 3-5 has stopped moving. Something is consuming from five partitions but not the other three. In Incident 1, the rebalancing state pointed toward a crashing consumer. Here, a Stable group with selective partition lag points toward something that has silently stopped. Different signal, different direction. With native tooling kafka-consumer-groups.sh --describe --group connect-es-orders-sink shows per-partition lag, but not which Connect task owns which partition. You'd need to cross-reference against the Connect REST API to map partitions to tasks, and the Connect API doesn't expose partition assignments directly. You'd have to infer it from task count and Connect's round-robin distribution logic. With Conduktor Console Per-partition lag with member assignment in one table. Partitions 3, 4, 5 are assigned to the same consumer instance (task 1's consumer thread). The lag graph shows the inflection point: 45 minutes ago, those three partitions stopped advancing. Phase 3: Inspect Messages are still flowing. The consumer isn't reading them. Open the orders topic. Filter to partition 3. Fresh messages arriving every few seconds. Producers are healthy. The most recent message timestamp is seconds ago. This rules out a producer-side problem. The data is there. Something just isn't consuming it. Now check the latest committed offset on partition 3 for the connect-es-orders-sink group. It hasn't moved in 45 minutes. Meanwhile, the end offset has advanced by thousands. The gap between committed offset and end offset is the growing lag. Compare this to Incident 1, where the stuck offset pointed to a specific message the consumer couldn't process. Here, the message at the stuck offset is perfectly normal. The consumer just stopped reading. That's the signal to look at the consumer component itself, not the data. **Skip Inspect** and you go straight to the connector. But what if the sink connector were choking on a specific record with a field Elasticsearch can't index? You'd restart the task, it would hit the same message, and fail again. Thirty seconds of checking the data tells you whether you're dealing with a data problem or an infrastructure problem. Phase 4: Trace The connector is "RUNNING." The task is not. The consumer group is owned by the Elasticsearch sink connector. Open the connector in the [Kafka Connect view](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect). Connector status: RUNNING. And here's the trap. The connector-level status only reports whether the connector *instance* (the Java object that generates task configurations) is alive. It says nothing about whether the tasks are healthy. A connector with three tasks can show RUNNING while one, two, or all three tasks have failed. The connector and its tasks have independent lifecycles. $ curl -s localhost:8083/connectors/es-orders-sink/status | jq . { "name": "es-orders-sink", "connector": { "state": "RUNNING", # ← looks fine "worker_id": "kafka-connect:8083" }, "tasks": [ { "id": 0, "state": "RUNNING", "worker_id": "kafka-connect:8083" }, { "id": 1, "state": "FAILED", "worker_id": "kafka-connect:8083", "trace": "org.apache.kafka.connect.errors.ConnectException: ... Caused by: ElasticsearchException[connection refused] ..." }, { "id": 2, "state": "RUNNING", "worker_id": "kafka-connect:8083" } ] } The top-level "state": "RUNNING" is the first thing you see. During an incident with Slack pinging, it's easy to read that and move on. The failed task is buried in the tasks[] array. Expand it and there's the answer: task 1 is FAILED with ElasticsearchException: connection refused, timestamped 45 minutes ago. Exactly when the lag started. The task list for es-orders-sink. Task-0 and Task-2 are Running. Task-1 is Failed. The connector-level status still says "Running." With native tooling There is no dedicated CLI for Kafka Connect. You use curl: curl -s localhost:8083/connectors/es-orders-sink/status | jq . The JSON response nests tasks inside the connector object. A quick glance at the top-level "state": "RUNNING" is misleading. You need to parse the .tasks[] array and check each task's state individually. During an incident with Slack pinging, it's easy to see "RUNNING" and move on. With Conduktor Console Connector view shows each task as a separate row with its state, worker ID, and error trace. A failed task is immediately visible: red status, expandable error message, failure timestamp. No JSON parsing. What happens to task 1's partitions depends on the Connect version and configuration. In some deployments, the remaining tasks pick up the orphaned partitions through Connect's internal rebalancing. In others, the partitions go unconsumed until the task is manually restarted. Either way, the connector-level status stays RUNNING, the failed task stays FAILED, and if the remaining tasks don't pick up the slack (or if throughput drops because fewer tasks are sharing the load), lag grows. The core problem isn't the partition assignment. It's the visibility. The connector says it's healthy. One of its tasks is dead. No alerts fire on the connector itself. The failure is silent until either the consumer lag threshold is crossed or someone manually expands the task list. Depending on the threshold, that could be hours. **Skip Trace** and you see lag on three partitions and just restart the entire connector. It works. All tasks restart, task 1 reconnects, lag clears. But you've also interrupted tasks 0 and 2, which were healthy. On a high-throughput connector, that means brief data duplication (sink connectors commit offsets periodically, so a restart replays some messages) and a gap in the Elasticsearch index. Restarting only task 1 avoids both. You can only do that if you traced the problem to a specific task. Phase 5: Resolve Restart the failed task. Not the whole connector. Restart task 1 specifically. It reconnects to Elasticsearch and begins consuming from its last committed offset on partitions 3, 4, and 5. The 45-minute backlog of unindexed orders starts flowing into Elasticsearch. Lag clears as the task catches up. With native tooling Task-level restart requires a POST to a URL path nobody has memorized: curl -X POST localhost:8083/connectors/es-orders-sink/tasks/1/restart Get the task ID wrong and you restart a healthy task for nothing. There's no confirmation of what will happen before you execute. With Conduktor Console Click the restart button next to the failed task. The other tasks are untouched. Before closing, verify the Elasticsearch cluster is healthy. The network partition that killed task 1 was transient. The node replacement completed and connectivity is restored. If Elasticsearch were still down, restarting the task would fail again immediately. Phase 6: Prevent The connector-level status is not a health check The fix is straightforward: stop alerting on connector-level status and start alerting on task-level status. Most monitoring setups check the connector's status endpoint and alert if the connector state changes. But the connector state only changes if the connector *instance* fails, which is rare. Task failures are far more common, and they're completely invisible to connector-level monitoring. The alert should fire when any task transitions to FAILED. The second problem is that vanilla Kafka Connect has no built-in automatic task restart. When a task fails, it stays FAILED until someone manually calls the restart REST endpoint. For transient failures like network blips, this means the task sits dead for hours until a human notices. [Conduktor Console's auto-restart feature](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect#connector-auto-restart) can restart failed tasks automatically on a configurable interval (minimum once per minute). The service checks every few seconds, detects which individual tasks have failed, and restarts them independently without touching healthy tasks. The alternative is external scripts that poll the status API, which is exactly the kind of ad-hoc tooling that accumulates and nobody maintains. Connect's errors.tolerance and errors.retry.timeout configs handle a different problem: per-record errors within a running task (a single message that fails to serialize, for example). They don't help when the task itself fails due to an infrastructure error like a lost connection. Task-level failure and record-level failure are independent failure modes. What happened Root cause: an Elasticsearch network partition killed one of three sink connector tasks. The connector reported RUNNING because the connector instance was fine. The failed task's partitions went unconsumed for 45 minutes until the consumer lag threshold was crossed. The two-level status trap (connector state vs. task state) is one of the most common Kafka Connect misdiagnoses, and it's structurally invisible unless you expand the task list every time you check a connector. Incident 3 The breaking change: a schema update that passed the registry and broke every consumer The setup A different pipeline: the user-events topic captures user activity from the web application. Producers serialize with Avro using the Confluent Schema Registry. Three consumer groups read from this topic: analytics, personalization, and fraud detection. The schema subject user-events-value is set to BACKWARD compatibility and has been stable at version 2 for months. The pipeline. The producer deployed schema v3 with a new enum value in event_type. All three consumer groups are compiled with v2. The registry accepted v3 because BACKWARD compatibility passed. Thursday morning, the product team adds refund tracking to the web application. The producer's schema for user-events-value adds a new value REFUND to the event_type enum (previously: PAGE_VIEW, CLICK, SEARCH, ADD_TO_CART, CHECKOUT). The registry checks BACKWARD compatibility: can the new consumer (v3) read old data (v2)? Yes, because v3's enum is a superset of v2's. Check passes. Schema version 3 is registered. The producer starts emitting REFUND events. Phase 1: Detect Three consumer groups alert at once Within minutes, consumer lag alerts fire for all three consumer groups that read from user-events. Three alerts for the same topic is itself a signal. When multiple independent consumers fail simultaneously on the same topic, the problem is almost certainly in the data or the schema, not in the consumers themselves. Phase 2: Scope All partitions, all consumers, same inflection point Pick any of the three consumer groups. All six partitions show lag climbing at roughly the same rate. Consumer group state: Stable. No rebalances, no crashed instances. The consumers are running. They're just not making progress. The lag graph shows a clean inflection point: everything was fine until 90 minutes ago, then lag started growing linearly across all partitions. The uniformity is the signal. In Incident 1, one partition was stuck (pointing to a single message). In Incident 2, three partitions were stuck (pointing to one failed task). Here, *all* partitions on *all* consumer groups are affected equally. Something changed in the data itself. With native tooling Run kafka-consumer-groups.sh --describe three times, once for each consumer group. Manually compare the lag patterns. No lag history. You'd see current lag numbers but couldn't tell that all three groups started lagging at the same moment without pre-existing metrics infrastructure. With Conduktor Console Check each consumer group's lag graph. All three show the same inflection point at the same timestamp. The correlation is obvious. Phase 3: Inspect The messages look fine. The consumers can't read them. Open the user-events topic. Browse recent messages. They look normal. Properly encoded Avro, reasonable payloads, correct timestamps. The messages aren't malformed. They aren't poison pills. The message browser can decode them because it fetches the writer schema from the registry and uses it directly. But the consumers tell a different story. Check the consumer's application logs: INFO Processing batch from user-events... INFO offset=48201 p=2 OK (CLICK) INFO offset=48202 p=2 OK (PAGE_VIEW) ERROR offset=48203 p=2 FAILED AvroRuntimeException: Unknown symbol in enum EventType: REFUND ERROR offset=48206 p=2 FAILED Unknown symbol in enum EventType: REFUND WARN Last successful offset: 48202 WARN First failing offset: 48203 Not every new message fails. Only the ones where event_type is REFUND. Messages with CLICK or PAGE_VIEW still deserialize fine. But refund events are arriving steadily, so the consumer keeps hitting them and stalling. The error is specific: Unknown symbol in enum EventType: REFUND. The consumer's compiled reader schema has an enum with five values. The writer is sending a sixth value the reader doesn't know about. To understand why the registry allowed this, you need to know how Avro enum resolution works. When the consumer's deserializer reads a message, it extracts the schema ID from the first 5 bytes (the Confluent wire format: a magic byte followed by a 4-byte schema ID), fetches the writer schema from the registry, and tries to resolve it against the consumer's compiled reader schema. For enum fields, Avro checks whether every symbol the writer sends exists in the reader's enum. If the reader doesn't have the symbol and the enum has no default, resolution fails. Writer schema (v3) enum EventType: PAGE_VIEW, CLICK, SEARCH, ADD_TO_CART, CHECKOUT, REFUND Reader schema (v2) enum EventType: PAGE_VIEW, CLICK, SEARCH, ADD_TO_CART, CHECKOUT Reader encounters REFUND: ╳ unknown symbol, no default defined Worth noting how different this is from Incident 1. The poison pill was a valid message that the consumer's *business logic* couldn't handle. The consumer deserialized it fine, then crashed during processing. Here, the failure happens in the Kafka client library itself. The consumer can't read certain messages into a record at all. And unlike a poison pill, where one message blocks the partition, this failure is intermittent: only messages with the new enum value fail. The consumer processes some, fails on others, and lag grows unevenly. **Without Inspect,** you see all consumers lagging and assume it's a throughput problem. Maybe the partition count is too low, maybe the consumers need more instances. You scale up. Lag keeps growing. Or you restart the consumers, which changes nothing because the same deserialization failure happens on the first poll() after restart. You can't distinguish "consumers are slow" from "consumers can't read the messages" without checking the data and the error logs. Phase 4: Trace The schema changed. The registry didn't stop it. The deserialization errors point to a schema incompatibility. Open the user-events-value subject in the Schema Registry. Three versions. Diff version 2 against version 3: a new value REFUND was added to the event_type enum. That's it. One new enum symbol. Schema version 3 of user-events-value. Compatibility is BACKWARD (top right). The registry accepted the new enum value because the BACKWARD check passed. With native tooling Fetch both versions with curl: curl -s schema-registry:8081/subjects/user-events-value/versions/2 | jq .schema | jq -r . > v2.avsc curl -s schema-registry:8081/subjects/user-events-value/versions/3 | jq .schema | jq -r . > v3.avsc Pipe to files, run diff. The schema is returned as an escaped JSON string inside a JSON object, so you need the double jq to extract it. Spotting a single new enum value in a diff of escaped JSON during an incident is not fun. With Conduktor Console Open the subject. Select versions 2 and 3. Side-by-side diff highlights the new enum value immediately. Why did the registry accept this? The subject is set to BACKWARD compatibility. BACKWARD checks whether the *new* consumer (v3) can read *old* data (v2). It can: v3's enum is a superset of v2's, so old messages deserialize fine with the new reader. Check passes. But BACKWARD doesn't check the other direction. It doesn't ask whether *old* consumers (still running v2) can read *new* data (v3). That's FORWARD compatibility. And that check would have failed, because v2's enum doesn't include REFUND. BACKWARD: can the new reader handle old data? FORWARD: can the old reader handle new data? FULL: both. This incident passed BACKWARD but would have failed FORWARD or FULL. In any org where producers and consumers deploy on different schedules (which is most orgs), BACKWARD alone isn't enough. You need FULL, or you need to add a default to your enums. **Skip Trace** and you see deserialization errors, tell the producer team to roll back. They do. New messages stop including REFUND. But the messages already produced with REFUND are still sitting in the topic. The consumer hits them on the way through the backlog and fails again. The rollback helps going forward but doesn't fix the window of bad data already in the partitions. Phase 5: Resolve Two paths, depending on how fast you can deploy If the refund tracking feature is intentional and the consumer teams can deploy quickly, update the consumers' reader schema to include REFUND in the enum. Deploy, and the consumers catch up on the backlog. If consumer updates aren't ready, add a default to the enum in a new schema version (v4). In Avro, an enum with a default handles unknown symbols gracefully instead of throwing. Old consumers using v2 would still fail on REFUND messages, but consumers updated to v4 would map unknown symbols to the default. This buys time for the full rollout. For the messages already in the topic with REFUND, the consumers need to handle them somehow. If the consumer has error handling that skips failed records (dead letter topic, error tolerance), it works through the backlog automatically. If not, you may need to reset offsets past the REFUND messages. For the analytics group, losing some events is acceptable. For fraud detection, probably not. Phase 6: Prevent BACKWARD compatibility isn't enough The registry did its job. It checked BACKWARD compatibility and the change passed. The problem is that BACKWARD only protects one direction: new consumers reading old data. It doesn't protect old consumers reading new data. If your producers and consumers deploy on different schedules (and they almost certainly do), BACKWARD alone leaves a gap. Switch to FULL compatibility on subjects where multiple teams consume independently. FULL checks both directions: the new schema can read old data AND old schemas can read new data. The REFUND enum addition would have failed the FULL check because old readers can't handle the unknown symbol. Alternatively, always define a default on Avro enums. With a default, old consumers map unknown symbols to the default value instead of throwing. The data might not be perfect (a REFUND event shows up as UNKNOWN), but the consumer doesn't crash. This is a defensive coding practice that makes schema evolution safer regardless of the compatibility mode. The producer's build pipeline should also check FORWARD compatibility before deploying, even if the registry is set to BACKWARD. The Schema Registry's /compatibility endpoint can test against specific versions: check the new schema against v2 with FORWARD mode in CI. The build fails before the schema reaches production. What happened Root cause: a new enum value (REFUND) added to a schema subject with BACKWARD compatibility. The registry accepted it because the new reader can handle old data. But old consumers can't handle the new value, and the enum had no default. Three consumer groups failed on every message containing REFUND. The symptom (lag growth across all partitions on all consumers, correlated with the producer deployment) was the scoping signal that pointed toward a schema change rather than a consumer or infrastructure problem. The Pattern Three incidents, one framework, and where the chain breaks Every one of these incidents started with a consumer lag alert. The root causes were completely different. The framework handled all three because each phase asks a question rather than checks a fixed item. Phase Incident 1: Poison pill Incident 2: Dead sink task Incident 3: New enum value **Detect** Lag on order-processing-v2 Lag on connect-es-orders-sink Lag on 3 groups reading user-events **Scope** 1 partition stuck, group rebalancing 3 partitions growing, group stable All partitions, all groups, same inflection **Inspect** Bad message at stuck offset Fresh messages arriving, consumer not reading Deserialization errors on new messages **Trace** Schema and connector clean Sink connector task 1 FAILED New enum value REFUND, BACKWARD passed, FORWARD failed **Resolve** Skip offset by 1 Restart task 1 only Deploy updated consumers or register transitional schema **Prevent** DLT handling + drift monitoring Task-level alerting + auto-restart Switch to FULL compat + enum defaults + CI check The diagnostic chain broke at different points in each incident, and would have broken earlier with native tooling. In Incident 1, CLI tools make the Inspect phase expensive (Avro binary garbage, separate tool, manual offset navigation), so teams skip it and restart. In Incident 2, the Connect REST API hides the failed task behind a misleading connector-level RUNNING status. In Incident 3, the registry's BACKWARD check passed, so the team assumed the schema was safe. Spotting a new enum value in an escaped-JSON diff during an incident requires knowing exactly what to look for. What you can check this week You don't need new tooling to start closing these gaps. Four things you can audit right now: Check Why it matters How Consumer group assignment strategy If you're using the default eager protocol, one stuck consumer takes down the group (Incident 1) Check partition.assignment.strategy in your consumer configs Schema Registry compatibility settings If any subjects are on BACKWARD instead of FULL, a new enum value can break old consumers (Incident 3) curl your-registry:8081/config and check each subject override Connect monitoring granularity If you only alert on connector-level status, failed tasks are invisible (Incident 2) Check whether your alerts fire on task state, not just connector state Last investigation timeline Ask your team: how long did the last Kafka incident take from alert to root cause? If the answer is hours, you're breaking the chain somewhere. The answer tells you which phase is the bottleneck Teams that default to "restart and hope" aren't lazy or unskilled. They're making a rational economic decision: the cost of investigating with native tools exceeds the cost of restarting. The framework doesn't work by asking teams to try harder. It works when the cost of each phase drops low enough that following the chain is faster than skipping it. That's what [Conduktor Console](https://conduktor.io/console) changes. Not the framework itself. The framework is just a sequence of questions. What changes is the cost of answering each one. When the cost is low enough, teams stop breaking the chain. Keep reading the full playbook Three worked incidents with Kafka mechanics, terminal output, and CLI vs Console comparisons at every phase. Source: https://www.conduktor.io/resources/ebooks/the-kafka-debugging-playbook --- # When Kafka Becomes the Ledger: A Control-Plane Operating Model for Transaction and Payment Data What's in this paper[01Kafka as the system of recordWhy transaction and payment data lives on the log, and the mess that follows](#when-kafka-becomes-the-system-of-record)[02Per-app and per-cluster gapsWhere Kafka's native ACLs, schema, and admin controls fall short](#per-app-and-per-cluster-enforcement)[03Three principles to enforceThree principles: no invented data, no lost data, no trust](#the-kafka-control-plane-operating-model)[04A card deposit, fully tracedWhat the proxy owns and what the ledger owns, in one flow](#a-card-deposit-traced-through-the-proxy)[05The reconciliation payoffWhy each enforced field is one fewer manual exception](#the-reconciliation-payoff)[06The reference architectureOne proxy and one console across a multi-provider estate](#reference-architecture-the-kafka-control-plane)[07Putting a proxy in the pathLatency, availability, and failure modes to settle first](#putting-a-proxy-in-the-data-path)[08A control-plane audit checklistWhat an auditor can ask you to demonstrate](#can-you-prove-it-a-kafka-control-plane-self-assessment)[09What the CFO and CISO answer forThe cost of doing nothing, in fineable numbers](#what-the-cfo-and-ciso-answer-for)[10How Conduktor fits the modelHow Gateway and Console map to the operating model](#how-conduktor-supports-this-operating-model) Conduktor works with [banks](https://www.conduktor.io/customer-stories/cdc-informatique-governed-kafka-self-service), payment processors, card networks, and digital banks that run Apache Kafka. The data that moves money has moved onto Kafka, and the controls that keep it correct and access-governed are often spread across many application teams and multiple clusters. Many reimplement their own field encryption, deduplication logic, and schema checks. When the platform team faces an audit, it has no unified visibility, because that logic and enforcement sit in the wrong place. This paper is about an operating model, one we call the *Kafka control plane*, that puts that enforcement in one central place, so a platform team can prove correctness, retention, and access to an auditor. It is written for the person who owns the Kafka estate inside a regulated fintech: the Platform or Streaming Lead. It is meant to be co-signed by the people who share the risk: - **The CISO**, who wants an independent enforcement layer over managed Kafka. - **The Head of Data or DPO**, who owns the tension between GDPR erasure and AML retention. - **The CTO**, who owns the build-versus-buy decision. > **Scope.** By transaction and payment data we mean the streams that move money and prove it moved: payments, ledger entries, settlement, reconciliation, KYC, and audit streams. Low-latency trading and market data are out of scope, because they play by different rules (deliberate duplication to win a race, disposable intermediate state because only the latest price matters). Note that Kafka cannot be the ledger: funds reservation, overdraft handling, and balance invariants need linearizability and belong in a transactional store. The control plane owns only the delivery-layer correctness. For a listed fintech, or a subsidiary of a US-listed parent, there is a further layer: the moment these streams feed financial reporting, the streaming platform falls inside SOX IT general controls (ITGC), namely change control over what shapes the data, logical access over who can touch it, and IT operations over retention and recovery. SOX is mostly an organizational program of controls and ownership that no tool provides; what a control plane can provide is the ITGC evidence an auditor asks for. ## When Kafka becomes the system of record This data is on Kafka because of what it provides: - **Outbox and change-data-capture.** Get a committed state change reliably onto a topic. - **Event sourcing.** Make the log the source of truth that state is derived from. - **Audit trails.** Append-only records of what happened, who did it, and why. - **Webhook ingestion.** Land external transaction events on a topic before anything internal touches them. In each of these the streaming layer is the system of record, not a buffer. Kafka has become the ledger of record for delivery, even when the authoritative balance lives elsewhere. > *"Accountants don't use erasers."* > — an old accounting adage Finance has always worked this way. You never alter a posted entry, you add a correcting one, so the mistake and its fix both stay on the record for an auditor to follow. An append-only log is that same discipline expressed as a data structure, which is why correcting a reconciliation break with an `UPDATE` in a database is a fast track to a compliance finding. Kafka hands you the append-only structure for free; the guarantee that no one can quietly rewrite it is the part you still have to enforce. > *"Of course we need that, because our Kafka usage is for the payment and transaction of data."* > — Platform engineering lead, digital retail bank When Kafka carries transaction and payment data, every application team ends up re-implementing that correctness work on its own. One team writes its own field-encryption wrapper, another its own dedup barrier and schema checks in consumer code, and others mask PII before logging. The platform team handles the ACLs and the audit burden centrally, and cannot push all the controls it needs down to the application teams: that is too scattered and creates too much friction. So the work ends up half in the apps and half on the platform, inconsistent and hard to prove. To run Kafka inside its PCI-DSS zone, [Afterpay's payments platform](https://code.cash.app/implementing-kafka-in-the-payments-pci-dss-world) built three controls into client code, because standard Kafka clients do none of that: - **Card-number detection and obfuscation** into a custom Avro serializer. - **HMAC message-integrity verification** against a KMS key, into custom producer and consumer libraries. - **A chain of PrivateLink hops** to keep cardholder data away from teams that should not see it. It's the per-team model this paper is about: every one of those controls lives in client code that each team has to build, ship, and maintain, and the next team that touches it starts over. A fintech often runs a mix of self-managed Apache Kafka (on premises), AWS MSK, Confluent Cloud, or another flavor, each with its own access model, audit output, and encryption practice. There is no single place to apply one policy or prove one control. Governance is partial, and full of unknown unknowns. > *"And then we got derailed by all the audit stuff that we have to comply with and the findings. Right now March is all about compliance and getting our audit stuff in place for the entire team."* > — Platform / security manager, payments processor Compliance should not be a one-off task; it should be continuous. Today, when compliance work arrives, it is scattered across clusters and apps, and the platform team generally loses a big chunk of its roadmap to assembling the evidence by hand. Money software reduces to three principles, the ones the *Fintech Engineering Handbook* by Voytek Pituła names: no invented data, no lost data, no trust. The first two are about the data. The third, no trust, is about everyone who touches it: verify everything, and trust no source by default, not a provider, not another service, not your own operators. On Kafka, each one becomes something to enforce: - **No invented data** means idempotency. - **No lost data** means durability and erasure on an immutable log. - **No trust** means controlling who and what can touch the streams. ## Per-app and per-cluster enforcement The way Kafka is governed today is generally not enough, because enforcement is either incomplete or in the wrong place: **Kafka ACLs are coarse and insufficient.** Broker ACLs and authorizer logs record authorization grants and denials, not per-record consumption. Consumer-group offsets show progress, not identity-attributed reads. PCI Requirements 8 and 10 expect an audit trail that ties every access to cardholder data back to a named identity, so the question they force, who actually read this PAN (the cardholder's card number), is unanswerable at the data layer. You need identity-bound access to answer who is authorized and who is connected. DORA imposes similar access-control and audit-trail obligations. Shared service-account credentials, which are often the norm in Kafka clients (and in tooling that reuses existing ones), break the unique-ID attribution that PCI requires: several humans and tools hide behind one principal. This is also the classic SOX finding on a streaming estate: a service account nobody can identify still has write access to the general-ledger integration, and no one can say who granted it or who uses it. It is a segregation-of-duties gap in ITGC terms, a producer identity with an uncontrolled write path into financial reporting, and it is unanswerable at the broker, where authorization logs record grants and denials but not the human behind the principal. **No schema enforcement.** Any producer with write permissions can put a malformed or unbalanced transaction record on a topic, or serialize an amount as a bare JSON number (an IEEE-754 double, so money crossing the wire that way silently loses precision). Even when you have a Schema Registry, it does not enforce anything on the wire. **Operators are Gods.** Anyone with produce or admin rights can delete records, write tombstones, or change retention. An audit trail they can edit proves nothing to a regulator. **At-least-once means duplicates are guaranteed.** A retry, a consumer-group rebalance, or a consumer rewinding its offset all redeliver events you have already seen. Exactly-once delivery across system boundaries (an external PSP, a webhook) is impossible, and Kafka's own exactly-once semantics cover only Kafka-to-Kafka flows, which is precisely why an application-level idempotency pattern is still needed end to end. If the dedup barrier does not sit somewhere shared, every consumer has to reinvent its own idempotency check, or risk applying a money-moving effect twice. **[Encryption at rest and TLS in transit](https://www.conduktor.io/blog/kafka-credentials-leak-plaintext-or-ciphertext) are necessary and not sufficient.** PCI Requirement 3 is explicit that disk-level encryption alone does not protect stored PAN, and Requirement 4 expects strong transport encryption even on internal segments. At-rest encryption is transparent to the broker, so managed Kafka serves the data in whatever form the producer wrote it, plaintext the moment it is served: to an authenticated client, to a leaked credential, and to the provider's own break-glass operator inside their proxy. At-rest encryption protects a stolen or decommissioned disk, not the running broker, and pushing encryption client-side instead just moves the key-management burden onto every application team. **Encryption keeps the PAN in scope; only tokenization takes it out.** A field-level-encrypted PAN is still a reversible representation of the PAN, so the PCI SSC treats the ciphertext as cardholder data in scope: encrypting the topics protects the data but does not remove the brokers and consumers from the CDE. Tokenization is different. It replaces the PAN with a token that has no mathematical relationship to it, the mapping held in a segmented vault, so a topic carrying only tokens holds no cardholder data. Do it before the produce, and the log and every consumer downstream fall outside the CDE, subject to your QSA's scope determination. That shrinks a SAQ D assessment; it does not make you SAQ A eligible, since you still ingest the raw PAN at the edge. Format-preserving tokens keep schema validation and downstream joins working on a value shaped like a PAN, and what stays in scope is the ingestion edge and the vault itself. > *"From my understanding, you're encrypting the data for your client using Conduktor before it hits Confluent Cloud?"* > — Integration architect, retail & commercial bank That is the move: protect the data before it reaches the managed cluster, without making every app own an encryption library. **Erasure.** A GDPR request asks you to erase from a system that cannot be edited, where the data is replicated, sitting in long-lived compacted topics or topics with infinite retention (especially with Kafka Streams or ksqlDB), in tiered storage, copied across clusters, and duplicated by every downstream application into its own topics and databases. A common misconception: **Kafka is not just a temporary buffer.** Whether you like it or not, how it is used naturally leads to data being stored there indefinitely, whether through infinite retention (the full history stays) or compaction (the latest value per key stays), so anyone can sweep a topic and extract the current state of every entity still on it. Meanwhile AML requires record-keeping, and for crypto-asset service providers, MiCA requires that the same personal data be kept. A per-app deletion script cannot resolve a retain-versus-delete conflict on an append-only log. ## The Kafka control-plane operating model The operating model is this: move correctness, durability, and trust enforcement off every application and every cluster's native tooling, into one layer that applies to all clients and all clusters. That layer is an inline proxy every producer and consumer connects through, sitting in front of Kafka, paired with a governance console. The proxy enforces policy on the wire; the console is where that policy and the audit trail live. Trust no client, including your own engineers, operators, and partners. Enforce at the wire level, where no one can bypass it. The three principles below organize the model. ### No invented data: never move the same money twice Kafka can deliver the same transaction event more than once. A retry, a rebalance, or a replay sends it again, and the day one consumer mishandles it, a payment goes out twice or a balance is credited twice, the kind of error a customer and a regulator will complain about. Often each team writes their own protection against this, but one miss or weak implementation is enough to let a duplicate through. The deduplication itself still happens in the consumer, since it needs shared state the proxy does not hold. What the proxy enforces is the one precondition every consumer depends on: a single rule, for every producer, that every transaction event must carry a stable idempotency id. The estate then deduplicates on one guaranteed key instead of each team hoping every producer set one. The proxy's job is to guarantee this id is always present, so a redelivered event is recognizably the same event. Ordering is the same story: Kafka preserves order only within a partition, so an account's events stay ordered only when they share a partition key. Process a debit before the credit that funds it and the customer drops into an overdraft they were never actually in; this is why [Monzo treats per-account order as a correctness requirement](https://monzo.com/blog/vertically-scaling-ordered-consumption-using-kafka). A producer that omits the key lets the default partitioner scatter an account's events across partitions, and the wrong-order effect follows. The proxy can require a stable partition key on every transaction event, exactly as it requires the idempotency id, so same-key events land on one partition and Kafka holds their order. That is the precondition, not the whole job: keeping one in-flight event per key on the consumer side stays application logic, the same split as deduplication. The proxy guarantees the key is present; it does not itself reorder. ### No lost data: durability at both ends of the log A record only counts as durable if it reaches the log in the first place, and stays readable for as long as the regulator requires. - **On ingestion.** The proxy enforces that records arriving from the outbox or CDC path carry the schema and the stable id the rest of the system relies on. - **Over time.** The proxy enforces that a record actually uses a schema and that its payload matches the schema the record declares, so every transaction event lands well-formed and self-describing, which is what event sourcing, audit trails, and AML and MiCA multi-year retrieval rely on. With the outbox pattern, the atomicity between the ledger and the event lives inside the transactional store, out of the proxy's sight. The proxy does not reproduce that guarantee, and it cannot detect a change that was committed to the database but never published, because it only ever reasons about what passes through it. What it does guarantee is the shape of every event that does cross the delivery layer: the idempotency id is present, the partition key is present, the schema is valid. The store keeps the atomicity, the proxy keeps the delivery contract. A transaction event also carries multiple business dates: - Value date - Booking date - Settlement date These are separate fields. None of them is the default Kafka record timestamp the client adds when sending a record (wall-clock time). The proxy can require those fields to be present, so a producer cannot omit them or quietly substitute the record timestamp, which would confuse the rest of the ecosystem. Erasure on an immutable log is the hard part, and it is exactly where the DPO's two regulators pull opposite ways: GDPR Article 17 says erase the subject, while AML and MiCA say retain the record. The data is append-only, replicated, sitting in compacted or infinite-retention topics, copied into tiered storage and into every downstream sink, so a delete script can neither reach all of it nor satisfy both rules if it did. The outcome worth engineering for is narrow and specific: **erase the identity, keep the transaction.** Two moves get you there. - **Keep PII off the log where you can.** Reference each subject by an opaque internal identifier, and hold the PII in a separate, mutable store that can be redacted on request without touching the log. - **Crypto-shred the rest.** For the identifying fields that have to travel in the event, encrypt each subject's fields under a key unique to that subject, which is [crypto-shredding](https://www.conduktor.io/blog/crypto-shredding-in-kafka-a-cost-effective-way-to-ensure-compliance). Erasure becomes deleting that one key. The ciphertext stays on disk but is unreadable everywhere it was copied, including backups and tiered storage, as long as no copy of that key survives in a cache or a KMS backup and no consumer re-persisted the decrypted value downstream. Crypto-shredding on the log: encrypt each subject's identifying fields under a per-subject key, and erase by deleting that key. The identifying fields become permanently unreadable, including in backups and tiered storage, while the pseudonymous transaction survives for AML and MiCA retention. Every other subject is untouched under its own key. Deleting one subject's key leaves the pseudonymous record standing: the amount, currency, dates, and opaque identifier are still readable, so AML and MiCA keep the transaction they require, while the person behind it can no longer be identified. Two regulators satisfied by one action, and every other subject's records untouched under their own keys. The erasure has to survive a replay. Backfills are routine, you rerun history to fix a consumer bug or recompute a model. A replay reads the same log the live traffic reads, so once a shredded subject's key is gone, the identifying fields come back as ciphertext to the replay consumer exactly as they do to a live one, and stay unreadable. Because the key is destroyed at the source, not filtered downstream, no replay path reconstitutes an erased identity. That is the business outcome. "We cannot delete from Kafka" is precisely the storage-limitation gap an auditor writes up, and over-retention is a finding in its own right. Crypto-shredding converts that gap into a control you can demonstrate, and brings erasure-request turnaround inside the one-month GDPR window instead of an open-ended manual sweep across clusters and sinks. The honest parts stay honest. One key per subject means millions of keys with their own custody, availability, and recovery story, plus a safeguard against shredding a record still under retention or legal hold. The master key sits in a customer-controlled KMS, so the erasure decision stays with you rather than a vendor. And **ciphertext can still count as personal data while the key exists**, so validate the approach with your DPO and supervisory authority. > *"That is also something we are looking at, as we need to segregate PII data versus non-PII data."* > — Platform architect, digital-first bank / lending fintech ### No trust: enforcement on the way in, on the log, and on the way out This is the crux of the enforcement argument. Kafka's native trust surface is too weak for transaction and payment data, so the proxy adds what's required: - **On the way in.** The proxy asserts the shape money must take, rather than only rejecting known-bad values. An amount must be a string or an integer in its smallest unit, never a bare JSON number that is silently an IEEE-754 double, and a currency code must come from a controlled set, so a typo'd or attacker-supplied code is rejected at the wire before it enters the log. External, webhook, and CDC data is held to the same rule and rejected when it does not conform. This is also the point to tokenize a PAN: replace it at the wire, before the produce, with a token that has no mathematical relationship to the card number, so the log itself never carries cardholder data, instead of merely encrypting a value that stays in PCI scope. - **On the log itself.** Whatever the raw Kafka ACLs allow, the proxy denies deletes, tombstone writes, and destructive admin on ledger and audit topics, so append-only is enforced rather than assumed. - **On access.** Replace flat native ACLs with audited RBAC and least-privilege roles over topics and consumer groups. Add the controls regulators expect: scheduled access recertification, and four-eyes on sensitive admin such as topic deletion, permission changes, replay, and retention changes. These are the same controls SOX ITGC names under logical access and change control when the topics feed financial reporting: least-privilege on who can write, recurring access reviews, and segregation of duties so the person who ships a producer does not also hold an uncontrolled write path to ledger topics. A replay is itself an identity-attributed action in that trail, so you can show an auditor who reprocessed which topic and when. - **On the way out.** Mask and decrypt per reader at the wire, so a downstream service receives masked PII while an authorized service sees cleartext, with no client code change on either side. An ops or support engineer browsing a topic in a console gets the same protection through console-level masking. > *"That's where we're putting in Conduktor, to confirm who's coming in, and control and ensure that they have direct access only to their data. So in case an audit does come, we can guarantee that we are showing that this is what it is."* > — Platform / security manager, payments processor Your operating model becomes: - **One plane across vendors.** The same model applies across Confluent, MSK, Aiven, Redpanda, and self-managed Kafka from one plane, so there is one place to prove compliance instead of one silo per vendor. - **Governed self-service.** Application teams request topics or connectors and own their data within platform-defined guardrails, which lets the platform team delegate permissions and governance to the teams without losing the control that answers an audit. ## A card deposit, traced through the proxy Let's see the flow: take a card top-up, where money is coming in. Duplicates still have to be caught, since the provider can redeliver the same "captured", but the distinctive hard part here is trust: you cannot take the outside world's word for what happened. A card top-up split across two lanes: the webhook ingester verifies the signal and queries the provider for truth, the proxy guarantees every event that lands is keyed, schema-valid, and append-only, and the ledger does the posting arithmetic neither of them touches. The deposit process: - The user submits the deposit, and the request carries an idempotency key (because the submit can be retried) - The payment provider authorizes the card and places a hold (a reservation). The money is not yours yet, so nothing is credited. - The provider's webhook calls to say the payment was captured. The ingester verifies the signature of the payload and queries the provider's API to validate (the webhook only triggers the signal, it's not a source of truth). When the ingester produces the events, the proxy is what guarantees there is a stable idempotency id (so a re-delivered "captured" deduplicates to a single credit downstream) and that it matches its declared schema, before any downstream app sees it. Once the capture is confirmed, the money is in flight, captured by the provider but not yet settled to your bank, so it posts through a clearing account for the time being. > **Proxy vs ledger.** The proxy never does the accounting itself; that happens in the ledger. It does require the provider's transaction id on each event, so the later reconciliation is a key-join rather than a guess from amount and time. What happens next plays out over days and weeks. - **Settlement.** Days later the provider settles one transfer covering many deposits, and reconciliation matches that batch against the clearing account, one settlement to many transactions. - **Chargeback.** If the cardholder disputes the payment weeks later, the chargeback is a new compensating event that references the original posting, not an edit to it. - **Access.** Throughout, the cardholder's data is encrypted on produce and masked per reader, so an ops engineer reconciling the batch never sees a raw card number. This flow is about ledger logic and protecting the system: a duplicate delivery, or an operator's eyes, could invent, lose, or leak that data, which is why a proxy is required to enforce the necessary controls: idempotency id, valid schema, append-only, per-reader masking, and so on. ## The reconciliation payoff The card flow ends at reconciliation, the moment the processor's report, the bank's settlement file, and your own ledger have to agree. The breaks are usually discrepancies across systems: - Timestamps that disagree (wall-clock vs value date) - An amount that lost a cent to a float - A description truncated differently on each side - The same event counted twice (missing idempotency id) Each one surfaces as an exception a team has to clear by hand. Most of those issues are data problems. You can only enforce data on the side you control, the events on your Kafka log. This is why enforcing data at the wire keeps your side of the reconciliation from ever being the cause of a break: - **A required, well-formed provider transaction id and idempotency id** on every event, so your log always carries the key the reconciliation joins on, instead of falling back to matching on other fields like amount and time. - **Value, booking, and settlement dates as distinct, required fields**, so the log never collapses three dates into one ambiguous timestamp. It is still on the producer to put the right value in each. - **Amount as a minor-unit integer, currency from a controlled set, schema-validated types**, so no float sneaks into an amount field and no mis-typed or missing field reaches the log. - **Append-only audit on the Kafka side**, the who, what, and when to explain the origin of a discrepancy. - **A required, stable deduplication key on every event**, so your consumers have one key to deduplicate on. The proxy enforces that the key is present, the consumer does the actual dedup. This disciplines the Kafka side, and it makes investigations simpler: if the numbers disagree, you know the break came from an external source, not from your own log. ## Reference architecture: the Kafka control plane Every producer and consumer of that data passes through a Kafka proxy (data plane), and a Console (control plane) to govern every cluster behind it. The per-app encryption and per-cluster controls are enforced, so the same policy applies across every provider without any client code change. **Producers.** Applications, webhooks, CDC connectors reading a database write-ahead log, and external API calls. All of them carry transactional data or PII. **Consumers.** Downstream services and humans (ops or support). An authorized service receives cleartext; an ops user or a generic downstream service receives masked PII. **The proxy.** Every byte passes through it, to enforce the principles above: - Validates and rejects on schema, checks message integrity, enforces the idempotency id, and rate-limits per producer so a runaway job cannot starve a payment topic's throughput. - Marks ledger and audit topics append-only and denies destructive operations, then applies field-level encryption keyed per subject, feeding a crypto-shredding keystore. A single master key sits in a customer-controlled external KMS off to the side. - Masks and decrypts per reader. **A unified governance model** spanning every cluster. Visible in Console, it holds cross-provider RBAC and ownership, the self-service request flow with guardrails, the access-grant audit trail, and consumer-lag and delivery observability. Config-as-code feeds both the console policies and the proxy enforcements. The audit stream is exported to the SIEM you're using. > *"I've been chasing audit logs, trying and trying to get them into Splunk for all of our products."* > — Platform engineer, payments processor This user had a complex ecosystem with multiple Kafka clusters and vendors, and no proxy in front of it. ## Putting a proxy in the data path Architects may be skeptical at first (isn't a Kafka proxy a single point of failure?) until they remember they have dealt with proxies forever in the HTTP world: API Gateways, reverse proxies, nginx, traefik. A proxy is still a real decision, because every byte passes through it, which makes it an availability and latency question. **A Kafka proxy is not a dumb load balancer.** Kafka embeds broker addresses inside its own metadata responses: a client bootstraps, gets the full broker list, then connects directly to each partition leader. A Kafka proxy [rewrites those advertised addresses](https://www.conduktor.io/blog/what-a-kafka-proxy-can-do) to ensure clients always go through it. As with an HTTP reverse proxy, clients must talk only through it for the controls to apply; a Kafka proxy is the same, so you expose one address and hide the many brokers behind it. Reaching a managed cluster across a network otherwise means [a load balancer and a firewall opening for every broker](https://www.conduktor.io/blog/kafka-msk-cross-network-connectivity), redone each time you add one; behind the proxy, a new team points at a single endpoint and the cluster's own network is never touched. As with any proxy, a few questions need to be settled first: **What happens when a proxy fails.** You always run multiple proxies for resiliency, so even if one fails, or you roll-upgrade them, traffic keeps flowing. It's like Kafka itself: you can restart a broker and the system auto-adapts. Kafka proxies follow the same pattern: because they rely on Kafka to hold their state, Kafka stays your central point of failure, not the stateless proxies in front of it. **A Kafka proxy also simplifies failover.** Because clients only ever know the proxy's endpoint, failing a cluster over to its secondary is [a single API call](https://www.conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes) that reroutes all traffic with no client change or redeploy, which makes the failover reproducible and testable, useful for regulations like DORA that require continuity testing. It also removes the cross-team coordination that is usually one of the most painful parts of a DR or failover. **How it scales.** Proxies are stateless: instances run side by side in front of the Kafka clusters, sized to the throughput going through them and the use cases they enable (routing, verifying, altering payloads, and so on). **How it protects the payment path from a noisy neighbor.** On a cluster shared across teams, a runaway producer, typically a batch or analytics job, can saturate broker network, CPU, or IO and push the authorization path past its latency budget. Kafka's native client and broker quotas are the first line here: byte-rate and request-rate ceilings set per (user, client-id). But they are static, set by hand and revisited rarely, and they know nothing about which topic is on the critical path. The proxy adds a second, finer control: per-producer and per-consumer rate limits enforced on the wire and scoped to a tenant or a virtual cluster, so the batch job's throughput is capped before it starves the payment topic's SLA. As with a reverse proxy, you start small in a simple passthrough mode, then add controls: - Start in Kafka passthrough: it is just routing, changing the bootstrap servers the apps point at. - Introduce policies to observe and understand your Kafka traffic: see how apps talk to Kafka, and spot bad practices or even application bugs (over-committing, over-using transactions, and so on). - Turn enforcement on: now that you understand the traffic, talk to the teams doing odd things with Kafka (they may not even be aware), then start preventing those bad practices (the proxy will deny some behaviors from now on). **Proxy access has to be tightly secured.** A proxy terminates authentication, holds the masking and decryption logic, and reaches your KMS, which makes it a high-value target. It has to be hardened, isolated, and monitored like the brokers it fronts. ## Can you prove it? A Kafka control-plane self-assessment Gather your security and compliance teams and work through the questions below, against your estate as it stands today. Each is something you should be able to demonstrate to an auditor, and each maps to one of the three principles, and to a regulation where one applies. Nothing here assumes you have already adopted the control plane; the point is to find the gaps. Score each as enforced at a single control point, enforced in some apps or by convention, or not enforced. Note that "in some apps" is a gap: a control you cannot prove in one place is something an auditor will not accept. | # | Can you prove it? | Principle / regulation | |---|---|---| | 1 | A redelivered or replayed transaction event produces a single effect (shown with a replay test), because every event carries a stable idempotency id that consumers deduplicate on. | No invented data / idempotency | | 2 | A change committed to your source database reaches Kafka with no silent loss, reconciled database-to-topic with alerting on any gap, and the outbox or CDC path carries the schema and stable id the rest of the system relies on. | No lost data / outbox, CDC | | 3 | Today's code can read a transaction event written years ago, because backward and forward schema compatibility is enforced before a breaking schema is registered. | No lost data / AMLD 5y retention (MiCA scope: crypto-asset services) | | 4* | A GDPR Article 17 erasure request can be honored on an immutable topic, reaching backups and tiered storage, while only the AML-mandated field is retained and the rest is erased or anonymized. | Retention vs erasure / GDPR Art.17(3)(b) vs AMLD (MiCA scope: crypto-assets) | | 5 | An amount arrives as a string or a smallest-unit integer (never a bare JSON number) and a currency code comes from a controlled set, even from a producer you don't own. | No trust / precision, schema | | 6 | The integrity and source of an incoming webhook or external event is verified before it enters the log. | No trust / webhooks, APIs | | 7 | No one with produce or admin rights can delete, tombstone, or shorten retention on a ledger or audit topic. | No trust / immutability, PCI Req 10.5, SOX ITGC (change control) | | 8 | You can produce a who-did-what trail of every grant and revoke on in-scope topics, and run scheduled access recertification and four-eyes on sensitive admin. | No trust / PCI Req 7-8, DORA, SOX ITGC (logical access) | | 9 | You can attribute a read of a PAN or PII record to a unique authenticated identity, not a shared service account hiding several humans and jobs. | No trust / PCI Req 8, Req 10, SOX ITGC (logical access) | | 10 | An ops or support user can browse an in-scope topic without seeing raw PAN or IBAN, and a downstream service gets masked PII while an authorized service sees cleartext. | No trust / PCI Req 3, per-reader masking | | 11 | The same policy and access model are provably enforced across every provider in your estate (Confluent, MSK, Aiven, self-managed), not redefined per silo. | Estate-wide / DORA third-party oversight | | 12 | Application teams can request in-scope topics only within platform guardrails, and a request that falls outside policy is refused. | Operating model / self-service | | 13 | Raw CDC output is schema-validated and its PII fields are masked per reader, so a downstream consumer receives masked values for the fields it is not meant to read. | No trust / outbox, CDC | | 14 | A transaction event carries value, booking, and settlement dates as distinct fields, and none of them is silently the Kafka record timestamp (by default the producer's wall-clock at send time). | No lost data / value vs booking vs settlement date | | 15 | Sensitive fields are encrypted before they reach the broker, so the managed cluster, a leaked credential, and a break-glass operator never see cleartext, and the keys live in a KMS you control, so you can show who can decrypt a field and revoke that access by destroying the key. | No trust / PCI Req 3 (field encryption + key custody) | | 16 | Every Kafka connection is encrypted in transit (TLS), client-to-broker and broker-to-broker, including traffic that stays inside your own network, not only the connections coming from outside. | No trust / PCI Req 4 | | 17 | Every event for one account carries a stable partition key, so its events stay on one partition and are applied in order, and a debit never lands before the credit that funds it and puts the customer into a false overdraft. The proxy enforces the key is present; per-key in-flight serialization stays in the consumer. | No invented data / ordering, partition key | | 18 | A runaway producer, such as a batch or analytics job, cannot degrade an in-scope payment topic's latency SLA, because per-producer rate limits are enforced at a single control point and scoped to the tenant, on top of the native broker quotas. | Availability / SLA, DORA operational resilience | | 19 | A replay or backfill of historical topics does not re-expose PII that has since been erased or would be masked for the reader running it: a shredded subject's fields come back as ciphertext, and per-reader masking applies to the reprocessing consumer exactly as to a live one. | No trust / GDPR Art.17, per-reader masking | | 20* | A PAN is tokenized at the wire before the produce, with a token that has no mathematical relationship to the card number, so the topics and consumers behind the proxy carry no cardholder data and fall outside the PCI CDE (the ingestion edge and the vault stay in scope). | No trust / PCI scope reduction, smaller SAQ D | | 21 | No orphaned service account holds write access to a ledger or audit topic: every producer identity with a write path into financial-reporting streams is owned, inventoried, and traceable to who granted it and who uses it. | No trust / SOX ITGC (segregation of duties) | On item 4, the control plane executes the retain-versus-erase decision, it does not make it: whether a field is covered by the AML retention exemption (GDPR Article 17(3)(b)), and for how long, is a determination for your DPO and, where needed, your supervisory authority. On item 20, whether tokenized topics fall outside the CDE, and how much of your estate that removes from a SAQ D assessment, is a scope determination for your QSA: the control plane enforces the tokenization at the wire, the QSA validates the resulting scope. ## What the CFO and CISO answer for The cost of doing nothing is not only poor audit reviews; more importantly, it leaves you exposed to data breaches, loss of customer trust, and fines. - **Over-retention.** In January 2026, France's data regulator [fined Free Mobile and Free a combined 42 million euros](https://www.cnil.fr/en/sanction-free-2026), citing storage limitation for keeping subscriber data, including IBANs, with no purge process, alongside the breach that exposed it. - **Cardholder data.** The UK ICO fined British Airways 20 million pounds after card numbers, expiry, and CVV were skimmed for hundreds of thousands of customers. - **Misconfiguration at scale.** Capital One's misconfiguration reached about 106 million people and drew an 80 million dollar regulator penalty plus a 190 million dollar settlement. [IBM's Cost of a Data Breach](https://www.ibm.com/reports/data-breach) put the average financial-sector breach at 5.56 million dollars in 2025, second only to healthcare, while the US average across all sectors reached an all-time high of 10.22 million dollars. DORA, in force since January 2025, adds remediation orders and supervisory penalties for financial entities. For the critical ICT third-party providers they depend on, it adds periodic penalties of up to 1 percent of average daily worldwide turnover, charged for each day of non-compliance, for up to six months. For a listed fintech, or a subsidiary of a US-listed parent, SOX adds a different kind of exposure. A material weakness in the IT general controls over the systems that feed financial reporting is disclosed to the market, and management and the audit committee answer for it personally. An ungoverned write path into ledger topics, or an access trail you cannot produce on demand, is exactly the ITGC deficiency that turns into that disclosure. Encryption and identity and access management (IAM) are among the largest cost-reducing factors in the report, and they are the two controls this model puts at the proxy. The table below helps you understand your baseline, what you can/cannot measure, your gaps, and your target. | Metric | What it counts | Baseline (today) | Target | |---|---|---|---| | **Scope** | | | | | In-scope topics | Topics carrying CHD/PAN, PII, ledger, audit, KYC, or trade data; defines your PCI audit scope | # today | no target; this is your scope | | PAN-carrying topics not tokenized | In-scope topics holding a reversible PAN, in clear or encrypted, that could be tokenized to fall outside the CDE and shrink the SAQ; encryption alone does not remove them (#20) | # today | 0 | | Estate under one governance model | Clusters and providers governed by one RBAC model rather than siloed tooling (#11) | # of total | all | | **Coverage: protect the data** | | | | | Clear-text sensitive topics | In-scope topics where PAN or PII fields are unencrypted (PCI Req 3–4; #15) | # today | 0 | | Field-level encryption coverage | % of in-scope topics with sensitive fields encrypted before the broker (#15) | % today | 100% | | PII reaching unentitled readers | % of in-scope topics where a reader can get PII in clear that its role should not see (#10, #13) | % today | 0 | | Replays re-exposing erased or masked PII | Backfills or replays that re-emitted PII already erased, or served it in clear to a reader that should have been masked, verified with a replay test against a shredded subject (#19) | # per reporting period | 0 | | In-transit encryption coverage | % of connections on TLS, including broker-to-broker inside your network (PCI Req 4; #16) | % today | 100% | | Schema-validation coverage | % of in-scope topics that reject off-contract records (#5) | % today | 100% | | **Correctness: no invented or lost data** | | | | | Duplicate-effect incidents | Money-moving operations applied twice from a redelivered or replayed event (#1) | # per reporting period | 0 | | Out-of-order effect incidents | Money-moving operations applied in the wrong per-account order, such as a debit before the credit that funds it, from events scattered across partitions by a missing key (#17) | # per reporting period | 0 | | Lost writes (DB to topic) | Committed database changes that never reached Kafka, caught by database-to-topic reconciliation (#2) | # per reporting period | 0 | | Schema-incompatibility incidents | Consumers breaking on an incompatible contract change (#3) | # per reporting period | 0 | | Immutability incidents | Deletes, tombstones, or retention cuts on a ledger or audit topic (#7) | # per reporting period | 0 | | Topics with no purge policy | In-scope topics kept indefinitely with no scoped deletion; the over-retention an auditor fines (#4) | # today | 0 | | **Evidence and identity: prove access** | | | | | Non-unique identities on in-scope streams | Shared service accounts and any principal that hides several humans or jobs, so a PAN or PII read cannot be tied to one person (PCI Req 8, 10; #9) | # today | 0 | | Access-recertification gaps | Accounts on in-scope topics overdue for recertification (#8) | # today | 0 | | Erasure-request turnaround | Time to honor a GDPR Art.17 request across log, backups, tiered storage, and sinks (#4) | days today | within one month (Art.12(3)) | | DR recoverability | Days since the last successful restore test, and the gap to your RTO/RPO (DORA Art.11–12) | today | within your tested target | | Audit-prep effort | Person-days, and cost at your loaded rate, to assemble streaming-layer evidence per reporting period | days / $ today | lower each period | | **Operational burden: the operating model** | | | | | Apps carrying their own encryption, idempotency, or validation | Services with bespoke security or contract logic to consolidate | # today | lower | | ACL and governance ticket volume | Access-management tickets the central team handles per reporting period (#12) | # per reporting period | lower via guarded self-service | | Time to provision a governed topic | Request-to-ready with guardrails enforced (#12) | days today | lower | | Payment-path SLA breaches from contention | Times an in-scope payment topic missed its latency budget because another tenant or a batch job saturated shared broker capacity (#18) | # per reporting period | 0 | | **Exposure (in $)** | | | | | Estimated exposure | The IBM per-record breach cost ($160 above) applied to the PII records a single realistic incident would expose, not the entire in-scope base | $ per credible incident | falls as the coverage and correctness rows reach target | What we see in the field is often due to a lack of understanding or wrong assumptions: - **Infinite retention.** Kafka is not just a temporary buffer but a permanent store of regulated data (compacted topics, stream-processing state, and so on) that no one scoped for deletion. - **Answering 'who owns what' takes time.** Teams discover that working out who owns which topic is a painful process, and with thousands of topics it can take months. - **False comfort from scanners.** Schema-based PII scanners give false confidence, because producers put card numbers and emails into free-text fields the scanner never flags. IBM's 2025 report also highlights that **malicious insider attacks** topped the cost table for the second year running at 4.92 million dollars. Breaches spanning multiple environments cost 5.05 million dollars against 4.01 million on premises. And customer PII was both the most-stolen record type and among the priciest at 160 dollars each. Those are the over-privileged-access, multi-estate, and clear-text-PII gaps. Each of them can lead to an incident. ## How Conduktor supports this operating model To reduce exposure and introduce the necessary controls, Conduktor provides a governance plane and a Kafka proxy for the data plane. Every application and user reaches Kafka through Conduktor, so one control plane fronts any provider on any cloud, with no change to the clients or the clusters. ### Gateway is the proxy It is a Kafka-protocol proxy that applies policy on the wire with no client code change and no Kafka config change. That property is the reason the model is enforceable for producers and consumers you do not own. What it unlocks: - **Encryption.** Field-level and full-payload encryption with KMS-backed envelope encryption (DEK, KEK, EDEK), so PII and PAN are protected before data reaches the broker without each app owning encryption code. - **Tokenization.** Vault-backed, format-preserving tokenization through HashiCorp Vault's Transform engine, so a PAN is replaced at the wire with a token that has no mathematical relationship to the card number. Unlike encryption, which keeps the value reversible and in PCI scope, tokenization is what takes the topics and consumers behind the proxy out of the cardholder-data environment, the lever that shrinks a SAQ D assessment. The ingestion edge and the vault stay in scope, so it does not on its own make you SAQ A eligible, and the final scope determination is your QSA's call, so validate it with them. - **Crypto-shredding.** Deleting a per-subject key makes that subject's records permanently undecryptable through Gateway, reaching backups and tiered storage where tombstones cannot. Because the field is unreadable at the wire and not merely filtered for live traffic, a replay or backfill that rereads the log sees the same ciphertext, so reprocessing history cannot resurrect an erased identity. - **Per-reader masking on the wire.** Scopeable by service account, group, or virtual cluster, so different readers receive differently-masked bytes. The rule keys on the reader, not on whether the read is live or a replay, so a backfill consumer is masked identically to a live one and reprocessing cannot leak PII a live read would have hidden. - **Data Quality.** Using CEL, JSON Schema, or EnforceAvro rules, the proxy rejects malformed or wrong-schema records, and enforces contract rules on every transaction event, such as a required idempotency id and a stable partition key (the no-invented-data rules) or a currency code from a controlled set. The deduplication and the per-key ordering stay the consumer's job; the proxy guarantees the id is there to deduplicate on and the key is there so an account's events stay on one partition and are applied in order. - **Message-integrity signing and verification.** On records as they pass. - **Producer guardrails.** Enforce the safe client configuration no matter how a team set it: acks=all with min.insync.replicas ≥ 2, so a write is acknowledged by a quorum of in-sync replicas and survives a broker loss (acks=all on its own can still ack a single copy once the ISR shrinks), and enable.idempotence, so a producer's own retries do not turn one write into two within a session. That guarantee is per-partition and per-session, which is why the durable dedup key is still the business idempotency id above, not this flag. - **Throughput guardrails.** Producer and consumer rate limiting enforced at the proxy and scoped to a service account, group, or virtual cluster, finer than the broker's static, topic-blind quotas, so a runaway batch or analytics job cannot starve a payment topic's latency. It complements the native broker quotas rather than replacing them. That is how you build confidence that the data you depend on stays correct and under control. ### Console is the governance plane Then you need one cross-provider plane. Console governs Confluent, MSK, Aiven, Redpanda, and self-managed Kafka from a single pane, with global cross-cluster RBAC, data policies, and GitOps, while Gateway fronts the physical clusters underneath. On top of that sit ownership and self-service with platform guardrails, the access grant-and-revoke trail, consumer-lag and delivery observability, and audit-log export in a vendor-neutral format to an external SIEM. For a SOX audit, this is the ITGC evidence layer for the streaming platform: an inventory of who and what can write to ledger topics, service accounts that are owned and attributable rather than orphaned, and a grant-and-revoke trail that answers the logical-access and segregation-of-duties questions on demand. Console produces the evidence; the SOX control program, the ownership, and the sign-off stay yours. ### Not just a tool, a Platform Conduktor is a platform, not isolated tooling: - **Schema Registry Proxy** adds authentication, authorization, and observability on top of Confluent Schema Registry, wired into Console self-service, which supports the schema-evolution and schema-access angles of the model. - **Virtual Clusters** give strong tenant and partner namespace isolation, with their own authn/authz, topics, and policy scope, on a single physical cluster. Use them for subsidiary, line-of-business, and partner-zone isolation. That boundary is also a scope for throughput limits: a rate limit set per virtual cluster keeps one tenant's spike from eating another's share of the shared brokers, so the isolation covers performance and not only access. For per-internal-team multi-tenancy, Console self-service is the right primitive, not Virtual Clusters. - **Automation.** The CLI and config-as-code put topic, schema, and permission changes for in-scope streams through the same reviewed, version-controlled change trail as application code, so governance runs as pipelines rather than tickets, and the audit evidence assembles itself instead of costing person-days each cycle. - **Insights.** Consumer-lag and delivery observability, data-quality reporting, and usage visibility across the whole estate surface the metrics the self-assessment and risk matrix track, so a control gap shows up as a dashboard line before it shows up as an audit finding. - **Governed AI access.** [An MCP server and agent skills](https://www.conduktor.io/blog/conduktor-mcp-and-skills-for-ai-agents) let AI agents observe and operate the estate through the same governed path as any other client: an agent reads through the proxy, so it sees masked PII rather than raw PAN, and every action it takes is a least-privilege, audited grant. That is what makes an agent on money data safe: the control plane holds the context, so the agent is useful, and holds the permissions, so it stays in bounds. ### Why not build this A CTO will reasonably ask: why not build this ourselves? In isolation it can look simple: cover a few apps and producers, or set a convention you ask every team to honor, which enforces nothing. To hold a rule for producers and consumers you do not own, it has to run on the wire, in front of every client. That means a proxy. **Forking an open-source proxy is hard.** The knowledge it demands is real: the protocol's constraints and behavior, the compatibility rules. The Kafka protocol is not HTTP; it is stateful and more complex, and it often takes years to get comfortable with. Kafka is a distributed system, after all. The moment you mask a field or encrypt a payload, you are decoding and recompressing record batches, and you have to keep idempotence and transactions intact through the rewrite. Then you need a serious test harness. We've published ours, [have a look and see what it takes](https://www.conduktor.io/blog/testing-a-kafka-proxy). **Compatibility is a killer.** A team bumps its client library, your proxy downgrades the protocol because it does not support the new version, and the consumer group slides into a rebalance loop: no error, just lag climbing on a payments topic. You cannot reproduce it, because it depends on that client version, that fetch size, that session state, that rebalance timing. Every Kafka release and every client library is a new version of the compatibility surface you have to stay correct across. Then you have to manage a heterogeneous estate: - One access model and one audit story across Confluent, MSK, Aiven, and self-managed Kafka, each with its own auth. - Per-reader masking and field encryption fast enough to stay inside a payment latency budget, even when your KMS hiccups. - A key lifecycle for the millions of per-subject keys crypto-shredding needs. - Four-eyes and recertification on sensitive admin. - Audit evidence you can produce on demand. Keeping all of it correct, forever, is not a project that ends. It takes your rarest engineers, the ones comfortable with byte-level protocol, distributed-systems timing, and crypto on the hot path. **The question is not whether your team can build it.** Your good engineers can. It is whether you want to: - put your best engineers on infrastructure that vendors already master and sell for a fraction of the headcount it would cost you to build and maintain; - answer to the regulator for an uncertified component sitting in your data path; - keep it alive and evolving after the engineers who built it have moved on. Build-versus-buy here was never about the first version. Your edge was never rewriting the Kafka wire protocol. ## Where to start? Run back through the self-assessment above. What matters is how much of it you can demonstrate enforced at a single control point, rather than scattered across apps or missing entirely. | Where you stand | What it looks like | First move | |---|---|---| | Per-app and per-cluster | Correctness, retention, and access live in application code and native ACLs, and most of it cannot be shown to an auditor on demand. | Put a single control point in front of one in-scope topic and prove one rule end to end. | | Partial | Some controls are centralized, others are still per-app, and governance reaches some clusters but not the whole estate. | Extend coverage to the clusters and providers still ungoverned. | | Boundary-enforced | Most controls sit at a single control point across the estate, and the gaps that remain are known and named. | Close the named gaps and make each one provable to an auditor. | | One proxy, one plane | Every principle is enforced once, and provable across providers. | Hold the line: recertify access and keep the audit evidence current. | How to produce a gap report: - **Map** your in-scope topics against the self-assessment. - **Mark** which principles are enforced at a single control point today, in some apps or by convention, or nowhere. - **Turn** that into a gap report: the qualitative answers, quantified by the Baseline and Target metrics above. The *Fintech Engineering Handbook* by Voytek Pituła is recommended further reading: it catalogs the money-handling patterns in full, while this paper covered the operating model for enforcing the ones that land on Kafka. [Talk to us](https://www.conduktor.io/contact/demo?src=resources-ebooks-when-kafka-becomes-the-ledger) and we will help you build the gap report against your Kafka estate. ## Related resources - [Achieving Data Security for Kafka](https://www.conduktor.io/resources/ebooks/achieving-data-security-for-kafka): the security primer this paper builds on, covering encryption, masking, crypto-shredding, and key management. - [Governing Kafka Data for the AI Era](https://www.conduktor.io/resources/ebooks/governing-kafka-data-for-the-ai-era): the same control plane extended to AI and agent access. - [Federating Data Governance: Scaling Kafka Without Losing Control](https://www.conduktor.io/resources/ebooks/federating-data-governance-scaling-kafka-without-losing-control): the governed self-service model at estate scale. Keep reading the full guide Source: https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger --- # Where Kafka Costs Hide: A Field Guide What's in this paper [01 The shapes Kafka waste takes Field observations from real cost analyses](#wp-problem) [02 The four layers of Kafka cost What makes up Kafka TCO, and where this guide focuses](#wp-layers) [03 Partition overprovisioning Are partition counts justified by the topic's actual throughput?](#wp-pattern-1) [04 Retention misalignment Does retention match how consumers actually read?](#wp-pattern-2) [05 Cluster sprawl Is each cluster doing real work?](#wp-pattern-3) [06 Orphan and duplicate topics Are these topics still in active use?](#wp-pattern-4) [07 Inefficient client patterns Is the cumulative cost of client choices visible to the teams making them?](#wp-pattern-5) [08 Static capacity per resource Could pooled capacity replace dedicated allocation?](#wp-pattern-6) [09 Putting the patterns to work Compounding effects, response approaches, data to gather](#wp-conclusion) Executive summary Most teams running Kafka in production suspect something in the bill is wasted, but don't have an easy way to scope it. This guide is for platform leads, Kafka architects, and streaming engineers who want a structured way to find, quantify, and act on the waste in their own estate. How we measure recoverable The 25 to 40 percent figure covers what comes off through config changes, topic retirement, cluster consolidation, and replatforming where warranted. The flex capacity a healthy estate carries for growth and burst sits outside it. **The headline finding from the cost analyses we've run: about 25 to 40 percent of a typical Kafka infrastructure bill is recoverable, mostly from non-production environments where the business risk is low.** The waste isn't coming from usage. It's coming from how the estate is structured and provisioned, and from decisions made at topic creation that nobody ever revisits. The rest of the guide is the methodology behind that finding. Part 1 shows what the waste actually looks like when we run the analysis. The six patterns that follow each get their own section, with the diagnostic, what each costs at scale, how each surfaces on hosted versus self-managed infrastructure, and where the patterns cascade into each other. A closing section maps each pattern to the response approaches that fit, plus the data points worth gathering before any platform review or contract conversation. Part 1 The shapes Kafka waste takes Before getting into the framework, here is what we keep running into when we look at real Kafka estates. Each of these is a specific finding from a real cost analysis, anonymized but otherwise unedited. **The 300-partition topic with no traffic.** Sitting on a managed cluster that gets billed by capacity unit. The partition count was set at creation against a peak load that never came. Nobody has questioned it since, and nobody can lower it without recreating the topic and coordinating with every producer and consumer. **200 clusters for 200 projects.** One cluster per project. The strategy made sense at adoption: per-cluster isolation gave each team clean ownership and avoided the operational complexity of multi-tenancy. The economics didn't scale. Each cluster carried the same baseline cost regardless of how much it actually did, and there was no mechanism for any team to share infrastructure with another. **5,000 empty topics on a single cluster.** None producing, none consuming, none safe to delete. The platform team didn't own them and couldn't get the project teams to confirm which were genuinely abandoned and which were waiting on a consumer that ran monthly or quarterly. Topics accumulated indefinitely because deletion was a coordination problem nobody had time to solve. **A topic running 30 times more egress than ingress.** The application consuming it had grown internally to spawn hundreds of independent consumer instances, each polling at the default rate. Nothing in the platform's monitoring flagged this. It was a valid configuration that just happened to be expensive. **A monthly Kafka bill that doubled in a year.** $60K to $120K, while business throughput grew at a fraction of that pace. The doubling came from accumulation: new topics inheriting peak-load partition defaults, retention bumps that became permanent, clusters spun up for projects that were never decommissioned. None of these came from teams making mistakes. They came from teams running under the constraints Kafka imposes on every estate at scale: no built-in cost visibility, partition counts that can't be reduced once set, retention that became default policy nobody questions, cluster decisions that locked in years ago. **A well-run estate accumulates waste over time. The patterns just become invisible from inside day-to-day operations.** The pattern under the patterns **The waste is structural, not behavioral.** Every observation above came from a team that was making reasonable individual decisions. The patterns are what those decisions add up to over years. **Most of the recoverable spend is invisible from inside day-to-day operations.** Kafka doesn't expose cost-by-team, partition-vs-throughput, or aggregate utilization in a way that surfaces drift. **The methodology is in the rest of this guide.** Each pattern below has its own section, with the diagnostic to spot it, what it costs, and how it surfaces on hosted versus self-managed platforms. Part 2 The four layers of Kafka cost Before getting into the patterns, a quick frame on what makes up the total cost of ownership of a Kafka estate. The conversation often collapses into "the bill" when the real picture is broader. Four layers contribute. Layer What it covers How it surfaces **Infrastructure** Compute for brokers and controllers, storage multiplied by replication factor, networking Hosted: capacity units sized by throughput, partition count, and storage. Self-managed: servers, storage arrays, datacenter capacity, network gear **Ecosystem tooling** Schema Registry, Kafka Connect, stream processing engines, cross-cluster replication, monitoring Hosted: paid add-ons. Self-managed: internal compute and operational time **Vendor and licensing** Platform licenses, tier surcharges, feature add-ons, support contracts, professional services Negotiated annually or multi-year, often opaque to the platform team **Operational** Engineering time spent running clusters, responding to incidents, supporting internal consumers Self-managed: full broker operations burden. Hosted: cost analysis, configuration management, vendor coordination Each of these is a real line item in any Kafka environment, and any serious cost conversation has to acknowledge that Kafka is rarely just "the brokers." Each layer carries some mix of value-producing spend and structural drift. **Infrastructure is where the bill itself can be reduced most directly.** Ecosystem tooling costs follow architectural decisions the business has already committed to, and pulling them back means rethinking the use cases those tools support, not adjusting a config. Vendor and licensing costs are typically locked in for the contract term, so reductions there land at renewal rather than month-to-month. The operational layer is real and worth its own treatment, but the mechanisms there look different (governance, automation, federated ownership) and produce a different kind of return: not a smaller bill, but a small platform team able to support a growing number of internal Kafka users without growing in proportion. The four layers that contribute to total Kafka cost, sized by typical share of recoverable spend. Infrastructure is where the bill itself can be reduced most directly, and where this guide focuses. Within infrastructure, six patterns account for most of the addressable waste in the estates we've analyzed. Each gets its own section below. The patterns rarely sit in isolation: decisions made early in an estate's life cascade through several patterns at once, and the largest savings come from recognizing where they connect, which we call out as we go. Part 3 Partition overprovisioning Pattern 01 Are partition counts justified by the topic's actual throughput? What the pattern looks like Topics whose partition counts far exceed what their actual throughput requires. The count was usually set at creation against a peak load that never materialized, inherited from another topic, or pulled from a default template tuned for a different use case. Why it happens Partition counts are set at topic creation and cannot be reduced afterward without recreating the topic end to end and coordinating the switch with every producer and consumer. At creation time, nobody knows what throughput will eventually look like, so teams fall back on heuristics: Copy the count from another topic that seems similar. Apply a rule of thumb like "three times the broker count." Pick a number that feels safe. There is no feedback loop that says "you overprovisioned this by 10x," and it compounds silently. Internal topics created by stream processing engines inherit the source partition count, so one early decision cascades across every downstream operation. How to identify it Two queries get you most of the way there: List topics by partition count, sorted descending. For the top of that list, pull average throughput per partition over the last 30 days. Topics where average throughput per partition is far below the meaningful threshold for partition parallelism are candidates for concentration. In practice, a topic with 50 partitions averaging less than a megabyte per second per partition is almost certainly overprovisioned. The exception is workloads with bursty traffic that needs the partition count to absorb peaks, so filter for those carefully. What it costs Leaders vs. partition-replicas The partition counts shown on a topic config are leaders. What actually counts against broker capacity is partition-replicas (leaders × replication factor). With the typical RF of 3, a topic with 50 leader partitions takes 150 slots of broker capacity. When you sort the worst offenders, the relevant number is partition-replicas, not the leader count. The cost impact scales steeply with estate size. On small estates (a single cluster, a few hundred topics), partition overprovisioning is often invisible: the cluster runs fine because partition pressure is below the threshold where it triggers operational events. On medium estates (multi-cluster, thousands of topics), capacity tier expansions start getting triggered earlier than usage justifies, and the cumulative count begins to move the bill. On large estates (tens of thousands of topics across multiple clusters), the absolute waste grows substantially; concentration analyses routinely identify 50 to 70 percent of partitions as eligible for retirement. On hosted platforms On self-managed estates Partition-replica count drives capacity unit sizing. Crossing the per-broker threshold triggers an additional capacity unit: a step-function cost increase at the threshold. A topic overprovisioned by 50 partitions does not move the bill on its own, but can be the one that pushes a cluster past the ceiling. The same per-broker ceiling exists, but as a soft performance limit rather than a billing trigger. Cost surfaces as disk for segment files, memory for partition metadata, and network traffic for replication. Less visible per topic, but cumulative. Teams often only notice when capacity pressure produces performance issues or hardware procurement lead times become a constraint. The mechanism behind both is the same: brokers max out at roughly 4,000 partition-replicas each, including leaders and followers. **The savings on partition concentration tend to be larger than the per-topic math suggests because each consolidated topic delays the next capacity expansion on hosted, or the next hardware procurement cycle on self-managed.** Brokers max out at roughly 4,000 partition-replicas each. Once cumulative partition decisions push the cluster one over that ceiling, the platform's only answer is a full new capacity unit. Tiny change in demand, doubled bill. How this cascades Partition overprovisioning is rarely contained at the source topic. Stream processing engines (Kafka Streams, ksqlDB) create internal topics that inherit the source partition count, so an overprovisioned source cascades through every derived stream downstream. The fix is the same (right-size the source), but the leverage compounds because every dependent operation inherits the gain. Stories from the field **Two-thirds of topics eligible for concentration on a "well-tuned" estate.** **The setup.** A platform team running on a managed cloud platform assumed their environment was reasonably well tuned. The estate consisted of a handful of clusters split between non-prod and prod, carrying thousands of topics, tens of thousands of partitions, and several terabytes of storage. **The finding.** The concentration analysis surfaced patterns invisible from inside day-to-day operations. Roughly two-thirds of topics were eligible for partition concentration, with counts that could be collapsed toward 1 without loss of functionality. A similar share of total partition-replicas could be retired. The managed-platform capacity units the clusters had been sized for could drop by more than half. Projected annual savings ran well into the six figures, before any architectural change. **Where the waste concentrated.** The single largest share sat in the non-prod footprint: hundreds of per-developer test topics created by branching an existing topic, dead letter topics provisioned at high partition counts with no consumer activity, and an entire non-prod cluster carrying hundreds of topics on effectively zero throughput. The prod clusters told a subtler version of the same story, with partition counts set against peak load profiles from an earlier phase of the architecture and never revisited as traffic patterns evolved. Roughly half of the partitions on one prod cluster were collapsible without operational impact. Part 4 Retention misalignment Pattern 02 Does retention match how consumers actually read? What the pattern looks like Long retention applied uniformly across topics with very different consumption patterns. Topics retaining seven days of data when consumers read the most recent hour. Topics retaining indefinitely because someone once worried they might need to replay, even though no replay has ever happened. Why it happens Retention defaults get set early and rarely revisited. The cost of "more retention" is invisible until storage hits a threshold; the cost of "less retention" is a potential incident if a consumer falls behind. The asymmetry makes generous retention the safe choice from any individual decision, even when the cumulative effect is significant. The mechanism also has more moving parts than most teams realize. retention.ms doesn't operate alone. Three related configurations interact with it: **segment.ms** (default 7 days). Segments only become eligible for deletion after they close. A topic with 1-hour retention but the default segment.ms can hold a week of data on a low-throughput partition because the segment never rolls. **segment.bytes** (default 1 GB). A second trigger for segment closure. Whichever boundary is hit first (time or size) closes the segment. **retention.bytes** (per partition). A topic with retention.bytes=10 GB and 50 partitions can consume 500 GB before retention kicks in. Most teams set retention.ms once and never look at the related configurations. How to identify it Compare each topic's configured retention to the actual consumer-side read pattern. Look at each consumer group's committed offset position relative to the log-end-offset over time. Groups that consistently sit within the most recent few hours of data and never trail back further indicate consumers reading recent data only. Where this is true and there is no replay history, retention can usually be reduced toward the actual read window. The harder cases are topics where retention is doing real work and a configuration change is not the answer: Topics used for replay (historical reprocessing after a bug fix or schema change) Topics under a compliance window Topics with downstream batch consumption Audit three more things alongside retention.ms: **segment.ms**: whether segments are rolling fast enough for the configured retention to take effect. **retention.bytes**: per-partition, so total retention = retention.bytes × partition count × replication factor. **The gap between configured retention and actual disk usage**: large gaps usually trace back to segment-roll issues or producer timestamp skew rather than the retention setting itself. What it costs Storage cost scales linearly with retention, but the multipliers are easy to underestimate. **Total storage per topic** = throughput × retention × partition count × replication factor With the typical RF of 3, every gigabyte of useful data takes three gigabytes of disk. On managed platforms, generous retention either drives up direct storage charges or pushes the cluster into a larger capacity tier. On self-managed, it consumes disk that could otherwise host other workloads. Across a large estate, halving retention on the topics where it is safe to do so can reduce total storage by 20 to 40 percent. **Configured retention is a minimum, not a maximum.** Actual retention is shaped by segment timing: **Actual retention** = configured retention + segment roll time (up to 7 days) + check interval (5 min) + deletion delay (1 min) A cluster sized for 500 GB based on throughput times retention can run at 800 GB or more once segment timing is accounted for. For long retention windows driven by compliance or replay needs, tiered storage is worth evaluating as an architectural option rather than a configuration change. Offloading cold segments to object storage removes the replication multiplier on the cold tier (object storage handles durability internally), which can reduce per-gigabyte cost on historical data by roughly 3 to 9x. The tradeoff is read latency on cold data, which is acceptable for compliance archives but not hot consumer paths. On hosted platforms On self-managed estates Storage and replication are billed directly per gigabyte. Generous retention drives up the per-GB charge and can push the cluster into a larger capacity tier. The bill scales linearly with retention until you cross a tier threshold, then takes a step. Retention consumes broker disk that could otherwise host other workloads. The trigger to act is disk pressure rather than a billing line item. On-prem estates often only revisit retention when storage starts filling up, or when capacity expansion lead times become a problem. Stories from the field **1-hour retention. 7 days of disk used.** A platform team set 1-hour retention on a low-throughput logging topic and ran out of disk after a week. The cause was segment.ms still at the 7-day default. Segments were not rolling, so retention had nothing to delete. The gap between "what should be 1 hour of data" and "what was actually 7 days" was completely invisible from the topic config alone, since retention.ms read 1 hour as expected. The fix was a one-line config change to roll segments every 30 minutes. Part 5 Cluster sprawl Pattern 03 Is each cluster doing real work? What the pattern looks like Many clusters, most of them underutilized. Often the result of a "one project, one cluster" strategy adopted to provide isolation between teams or workloads. The problem is not multi-cluster as such. Plenty of estates run multiple clusters for legitimate reasons: Environment separation (dev, staging, prod) Geographic distribution to keep clusters near users Data residency requirements (GDPR, HIPAA) Isolating workload classes that genuinely do not mix (high-throughput batch versus real-time, large multi-tenant customers) Sprawl is what happens when clusters get added beyond those reasons, or stood up at adoption and never revisited as the estate matures. The patterns we see most often: **A cluster per project** created for organizational reasons rather than technical ones. **Environment stratification that went a level too deep**: separate dev and QA clusters where one would have done. **On-prem patterns carried into cloud unchanged**: clusters shaped by on-premises constraints that got migrated as-is, where the cost is now much more visible. Why it happens Multi-tenancy in Kafka is genuinely hard, and per-cluster isolation solved real problems: Blast radius containment Clean ownership Security boundaries Avoiding the operational complexity of quotas and ACLs On hosted platforms, spinning up a new cluster is frictionless and the cost is absorbed into a platform budget that individual project teams never see. On self-managed estates, the strategy is more often driven by compliance isolation requirements. Both paths lead to the same outcome: cluster count growing faster than the workload, with each cluster carrying the same baseline cost regardless of how much traffic it handles. Spawning a new cluster instead of building well-tuned multi-tenancy is often a rational substitute for operational expertise that the platform team has not been resourced for. Running a single large multi-tenant cluster well requires deep tuning experience and active governance, while standing up a new cluster per project does not. The substitution works in the moment, but it compounds: each new cluster reproduces the same trade rather than building the expertise that would let one cluster serve more workloads. How to identify it Pull three utilization ratios per cluster: peak ingress as a share of provisioned ingress, total partitions as a share of the platform's partition cap, and storage used as a share of provisioned storage. Clusters running below a meaningful threshold (typically 25 to 30 percent at peak load) on all three dimensions are consolidation candidates. A second pass on the list: ask which environment each cluster belongs to and whether the separations between them are doing real work. Four questions tend to surface consolidation candidates: Are dev and prod on different clusters? *Usually justified.* Are dev and QA on different clusters? *Often not justified.* Do regional clusters actually serve regional traffic? *If not, candidates for consolidation.* Does the same team operate multiple clusters for different applications? *Usually not justified once the team has operational maturity for multi-tenancy.* The harder analysis is which clusters can be consolidated without disrupting the teams that depend on them, since the original isolation strategy created organizational dependencies. Configuration drift between clusters that started identical also makes consolidation harder over time, because "merge cluster A into B" surfaces every retention setting, ACL, and quota inconsistency that has accumulated since they diverged. What it costs The baseline cost of a Kafka cluster does not shrink with usage. Every cluster requires a minimum broker count, controller infrastructure, baseline storage, networking, and operational overhead. On hosted platforms, this surfaces as a minimum capacity unit allocation. A cluster running at 5 percent of capacity costs the same as one running at 50 percent if both sit at the same tier. With 70 to 200 underutilized clusters in a sprawled estate, the aggregate baseline cost dwarfs the active workload. **The aggregate waste is often larger than partition overprovisioning, but it is less visible because each individual cluster looks fine in isolation.** Operating ten clusters consistently is not ten times the work of operating one, but it is meaningfully more, and the gap grows as cluster count grows. Configuration drift, version skew, certificate expiry tracking, and per-cluster monitoring overhead all scale with the number of clusters even when traffic does not. That overhead becomes part of what the platform team spends its time on, and ties back to the operational overhead discussion later in this guide. On hosted platforms On self-managed estates Spinning up a new cluster is frictionless, and the cost is absorbed into a platform budget that individual project teams never see. Each cluster carries a minimum capacity-unit allocation that does not shrink with usage. Consolidation requires merging clusters, which surfaces every accumulated configuration difference between them. Each cluster is a hardware footprint: minimum broker count, controller infrastructure, baseline storage, networking, and operational overhead. Compliance isolation requirements often drive sprawl here. The trigger to act is usually disk pressure, capacity expansion lead times, or operational pain rather than a billing line item. Stories from the field **200 clusters for 200 projects.** A platform team had grown to roughly 200 Kafka clusters supporting 200 projects, one cluster per project. The strategy made sense at adoption: per-cluster isolation avoided the operational complexity of multi-tenancy, gave each project clean ownership, and contained blast radius. Each cluster carried the same baseline cost regardless of traffic, so the majority of the estate was paying full price for fractional utilization. There was also no mechanism for any team to share infrastructure with another, which meant the only growth path was more clusters. How this cascades Cluster sprawl rarely produces only a cluster cost. When data lives in separate clusters, cross-project sharing becomes meaningfully harder: consumers in one cluster cannot subscribe to topics in another without a replication setup. The path of least resistance is to replicate the data each consumer needs into their own cluster, which is one of the primary drivers of Pattern 4 (topic proliferation and duplication). A decision to optimize for operational simplicity by spawning per-project clusters ends up generating both the direct cluster cost and the duplicate-topic cost downstream. Cleanup that addresses only the cluster count without the duplicate topics is incomplete; cleanup that addresses only the topics without consolidating clusters tends to recreate the same duplicates over time. Part 6 Topic proliferation and duplication Pattern 04 Are these topics still in active use? What the pattern looks like Two related dynamics. First, topics created for experiments, migrations, or short-lived projects that were never decommissioned. Second, duplicate topics created in parallel when teams cannot find an existing topic that already serves the same purpose, so they create their own. Three sub-cases show up underneath this: **Orphans.** Topics with no recent traffic and no active consumers, abandoned in place because nobody owns the cleanup. **Near-duplicates.** Topics with similar schemas, similar names, or the same source data with slightly different transformations, created because a team did not find the existing version. **Stream-processing topics.** Internal topics created by Kafka Streams, ksqlDB, or Flink. They inherit the source topic's partition count, so an overprovisioned source cascades into every derived stream downstream. They also accumulate when stream processing is used as a workaround for filtering: because Kafka cannot scope what a consumer sees on the original topic, teams write a filtered subset to a new topic, which duplicates the data once per unique subset that needs to be exposed (one per geo region, tenant, or status). Each sub-case has a different remediation path, and the diagnostic has to distinguish them. Why it happens Topics are easy to create and hard to delete. Most platform teams have no central catalog of who owns what or what is in active use. Without a discoverability mechanism, duplication is the path of least resistance: when a team cannot find what they need, recreating it is faster than tracking down the existing version. Once a topic exists, deleting it requires confirming nobody is reading from it, a coordination exercise the platform team rarely has time for. The result is accumulation in both directions: more topics created than needed, and few of the unneeded ones ever retired. Naming conventions either help or actively hurt. Names that encode business meaning (order-events-v3, customer-profile-snapshots) let a developer recognize a candidate for reuse at a glance. Names that encode implementation detail (svc-xyz-internal-v2, kstream-repartition-7d4f) tell that developer almost nothing, so the path of least resistance is to create their own. **Estates without enforced naming standards or a searchable catalog accumulate duplicates as a structural feature rather than a discipline failure.** How to identify it Two queries plus two more for stream-processing topics: Topics with zero traffic over the last 30 to 60 days and no active consumer groups: candidates for retirement. Topics with similar schemas or naming patterns: candidates for consolidation. Internal topics produced by stream processing jobs whose partition count exceeds the actual throughput of the derived stream: candidates for source-side right-sizing that will cascade through the pipeline. Internal topics that are filtered subsets of an upstream topic (similar schema, smaller volume, single downstream consumer): candidates for review. If the filter is for convenience, the logic can usually move to the consumer. If the filter is enforcing a security or compliance boundary, the duplication may be the right cost, but it is worth confirming the boundary still holds. The orphan case is the hard one. Many idle-looking topics are legitimately idle (DR topics, batch processing that runs monthly, schemas of record). Distinguishing legitimately idle topics from abandoned ones requires owner verification, which is itself a coordination problem. What it costs Each topic carries replication and metadata cost, even with zero traffic. At scale, thousands of orphaned topics are real money. The duplicate case is more expensive: each duplicate stream multiplies producer fanout and consumer cost in addition to the topic itself. Filter-workaround topics multiply differently: once per unique subset that needs to be exposed, so a source feeding 20 downstream regional teams becomes 20 derived topics plus the source, each carrying its own replication, storage, and metadata cost. Across a typical estate, retirable orphans plus de-duplicable streams plus filter-driven derived topics contribute meaningfully to total cost, and the visibility gap that produced them is itself a governance problem. The leverage on this work is larger than the per-topic line-item math suggests. Each orphan holds partition-replicas against the per-broker ceiling from Pattern 1. Removing them releases broker headroom and delays the next capacity expansion: Each retired topic propagates through the chain: partitions released, partition-replicas freed (multiplied by replication factor), broker headroom restored, and ultimately the next step-function capacity expansion delayed. The leverage compounds because the math compounds. The direct cost of carrying orphans, near-duplicates, and stream-processing topics adds up. The avoided capacity expansion the cleanup buys can be larger than the direct cost on its own. On hosted platforms On self-managed estates Each topic carries direct metadata cost and contributes to the partition-replica count that drives capacity tier upgrades. Topics with zero traffic still cost something. Orphans accumulate against the per-broker ceiling, accelerating the next expansion. The per-topic cost surfaces as broker memory for partition metadata and disk allocation for active segments. Less visible per topic but cumulatively significant once an estate hits the thousands-of-orphan-topics range. Can quietly degrade broker performance even before disk pressure surfaces. Stories from the field **5,000 empty topics. None deletable.** One platform team had more than 5,000 empty topics on a single cluster. None were producing or consuming. None could be safely deleted, because the platform team did not own them and could not confirm which were genuinely abandoned and which were waiting on a consumer that ran monthly or quarterly. Retirement required individual outreach to dozens of project teams, with no incentive on the other side to respond. The estate accumulated topics indefinitely because deletion was a coordination problem nobody had time to solve. How this cascades Topic proliferation is often a downstream effect of other patterns or of Kafka design constraints: **Cluster sprawl (Pattern 3)** makes cross-project data sharing harder, so teams replicate data into their own cluster as duplicates rather than subscribing across boundaries. **Partition overprovisioning (Pattern 1)** cascades through Streams and ksqlDB, producing chains of overprovisioned derived topics for every transformation. **Kafka's inability to scope what a consumer sees on a topic** drives a continuous trickle of filtered-subset topics whenever a downstream service needs only part of an upstream stream. Treating topic proliferation as a standalone cleanup ignores all three sources; the durable fix usually requires touching the upstream cause, not just the symptom. Part 7 Inefficient client patterns Pattern 05 Is the cumulative cost of client choices visible to the teams making them? What the pattern looks like Egress traffic running many times higher than ingress because of redundant consumer groups, unnecessary fan-out, or misconfigured clients. Producers configured with no compression. Consumers polling at high frequency without backoff. Several sub-patterns show up underneath: **Compression off.** Producer defaults are commonly off for backward-compatibility reasons, and the cost of "off" is invisible to the producer-owning team. **Consumer fan-out.** A topic that started with one consumer ends up with dozens as new services subscribe; each one is a multiplier on egress and broker work. **Aggressive polling.** Consumers configured at default poll rates with no backoff that hammer the broker on idle topics. **Cross-AZ traffic.** On hosted platforms, traffic between availability zones is a direct line item, and consumer-to-leader assignments that do not account for AZ topology pay this tax continuously. **Large messages without claim-check.** Topics carrying multi-megabyte messages that could be replaced with a reference to object storage, which is dramatically cheaper. Why it happens Clients are configured by application teams, who optimize for their own service rather than for the cluster they share. From any single application's perspective, "use the default" is the right call: compression off, default poll rates, default fan-out as new services subscribe. **From the cluster's perspective, hundreds of applications using defaults is what produces the inefficiency.** The cost penalty is invisible to the producer-owning team, and without a central audit or a guardrail at topic creation, no team has the incentive or the visibility to investigate. How to identify it Compare ingress to egress on a per-topic basis. Topics where egress exceeds ingress by more than a small multiplier indicate fan-out that may be addressable through architectural changes (consumer consolidation, claim-check patterns for large messages, caching at the consumer side). Compression auditing is straightforward: pull producer configurations and identify topics where a meaningful share of producers are not using compression. Three additional checks worth running: **Consumer count per topic.** Topics with high consumer counts and high replay rates are candidates for claim-check or consumer-side caching, since each replay multiplies fan-out. **Average message size.** Topics with messages above a few hundred KB are candidates for claim-check, where the payload goes to object storage and only the reference flows through Kafka. **Cross-AZ ratio.** On hosted platforms with cross-AZ billing, audit which topics have most of their consumers in different AZs from the partition leader; partition assignment can sometimes be reshaped to keep traffic in-AZ. What it costs The cost compounds across produce, replicate, and consume. A 1 MB batch with replication factor 3 generates: 1 MB of ingress 2 MB of internal replication networking 3 MB of storage (one copy per replica) Egress that scales with the consumer count Compression typically reduces all of these proportionally. Well-configured compression often achieves 5x to 10x reduction on text-heavy payloads, which translates into the same multiplier on the storage and network side of the bill. On hosted platforms On self-managed estates Egress is a direct line item, often per-GB. The cumulative effect of inefficient client patterns can rival the partition pattern in absolute terms. Cross-AZ traffic is typically billed at roughly double the in-AZ rate, so fan-out architectures that span AZs pay the multiplier twice. Compression directly reduces the bill. Cost surfaces as network capacity, replication overhead, and potentially saturated network links. Less visible per topic but cumulatively significant once an estate hits scale. Compression reduces network and disk pressure rather than a bill, which makes it easier to defer until something breaks. Stories from the field **Egress 30 times higher than ingress on one topic.** A platform team analyzing their estate discovered that one production topic had egress running 30 times higher than ingress. The topic was being consumed by a single application that had grown internally to spawn hundreds of independent consumer instances, each polling at the default rate. The configuration was technically valid; nothing in the platform's monitoring would have flagged it. The fix was an application-side architectural change to consolidate the consumers. The cost impact was significant in the months between when the duplication started and when the analysis surfaced it. Part 8 Static capacity per resource Pattern 06 Could pooled capacity replace dedicated allocation? What the pattern looks like Each topic, cluster, and connection holding dedicated broker capacity at all times, whether at peak load or sitting idle. Across an estate with hundreds of resources on different load profiles, the sum of those dedicated allocations runs well ahead of true aggregate demand, with no built-in way to pool underlying capacity across them. Concrete examples of where peaks misalign in real estates: **Batch jobs.** Nightly or monthly loads that run heavily for an hour and sit idle the rest of the time. **Dev and staging environments.** Low-traffic during nights and weekends by nature. **Time-zone-shifted workloads.** Different geographic regions peak on different clocks. **Seasonal traffic.** Retail holidays, financial month-end close, quarterly reporting cycles. In each case, the resource holds capacity sized for its peak around the clock, even though the peak only happens for a fraction of the time. Why it happens Kafka's design allocates capacity per resource. There is no built-in mechanism to share underlying broker capacity across topics or clusters that are not under load simultaneously. A topic provisioned for peak traffic continues to hold that capacity at low traffic, and an underutilized cluster cannot lend its headroom to a busier neighbor. How to identify it Plot the load profiles of the top resources by provisioned capacity on the same time axis, then compare the sum of their individual peaks to the aggregate peak across all of them. The gap between those two numbers is the pooled-capacity opportunity. The diagnostic is more involved than the others in this section because it requires correlating load profiles across resources, but the payoff is large in mature estates because peaks rarely align. Each resource holds capacity sized for its individual peak around the clock. Because peaks rarely align across resources, the aggregate peak across all of them is much lower than the sum of individual peaks. The gap is what pooled capacity recovers. Non-production environments are the easiest place to start the analysis. By nature they carry low and bursty traffic, so the over-allocation is most visible there and the risk to the business of pooling capacity is low. The same pattern exists in production, but the consolidation work is harder to justify because production teams reasonably optimize for risk over efficiency. Starting with non-prod produces evidence and savings that make the production conversation easier later. What it costs **This is the pattern that produces the largest absolute savings in big estates, because the aggregate over-allocation grows with resource count.** It is also the pattern that requires the most architectural change to address, since pooling capacity across resources is not a configuration setting; it is an infrastructure layer that sits between clients and brokers and has to be introduced deliberately. On hosted platforms On self-managed estates The pattern surfaces differently by pricing model. On step-function pricing (capacity units, broker counts, node counts), waste concentrates at threshold boundaries: nothing extra until you cross a tier, then a meaningful jump. On linear pricing (per-unit consumption, autoscaled tiers), waste accumulates continuously as headroom nobody is using. Hardware over-provisioning compounds. Infrastructure sized for peak load sits idle outside peaks, holding capacity it does not use. The trigger to act is when the next workload cannot fit on existing infrastructure and someone has to procure more, often months in advance. How this cascades This pattern compounds with the others above: **Pattern 1** is static capacity at the partition level: an overprovisioned partition holds broker capacity it does not use. **Pattern 3** is the same problem at the cluster level: an underutilized cluster holds infrastructure capacity it does not use. **Pattern 4**'s derived stream-processing topics inherit the static capacity of their source topics, multiplying the over-allocation downstream. Treating the patterns separately is useful for diagnostics; the underlying constraint is shared, which is why pooled capacity tends to produce the largest savings when an estate is mature enough to take it on. Conclusion Putting the patterns to work How waste compounds across the other layers The four cost layers introduced earlier are not independent. **Infrastructure waste pulls the other three layers up with it, which means the leverage on the diagnostics above is larger than the direct infrastructure bill alone suggests.** Ecosystem tooling: paying for the same work multiple times Stream processing engines and other ecosystem tools sit on top of the infrastructure and inherit its inefficiencies. The clearest example is filter-only stream processing: when a downstream service needs only part of an upstream topic, the common workaround is a Streams, ksqlDB, or Flink job that reads the source and writes a filtered subset to a new topic. In the estates we have looked at, this often accounts for the majority of stream processing usage; in some cases, 80 percent of Flink jobs are doing nothing more than pulling a subset of messages from one topic into another. The team ends up paying three times for the same work: **The stream processing engine.** Compute and licensing for Flink, ksqlDB, or Streams instances running the filter. **Developer time.** Building and maintaining the filter job, plus the operational overhead of treating it as a production service. **The resulting infrastructure.** The derived topic and its replicas, plus the partition-replicas it consumes against the per-broker ceiling from Pattern 1. Where the filter is not part of a larger transformation pipeline, application-side filtering removes all three costs at once. Pattern 4 is where the topic-level mechanics live; the point here is that the same waste also pulls the ecosystem-tooling line item up. Vendor and licensing: the percentage trap Vendor and licensing costs ripple from infrastructure in ways that are easy to underestimate. Several common line items are priced as a percentage of, or scoped against, total platform spend rather than as flat fees: **Support contracts.** On most hosted Kafka platforms, billed as a percentage of total platform spend, so an inflated infrastructure bill produces a proportionally inflated support line item without anyone deciding to pay more for support. **Tier upgrades and feature add-ons.** RBAC, audit logs, private networking, and multi-region replication, often gated behind tier upgrades that get triggered by cluster size or capacity-unit thresholds. **Professional services.** Engagements scoped against cluster footprint or capacity units, which scale up alongside the infrastructure they are meant to support. The corollary is that **cleaning up infrastructure automatically reduces the support line item by the same proportion.** The percentage stays the same; the base it applies to shrinks. No renegotiation required, no separate conversation, just the natural consequence of the infrastructure cleanup. For estates carrying premium feature add-ons or multi-region replication on top, the multipliers stack: every dollar of infrastructure waste pulls support, premium features, and any percentage-priced services up with it. The leverage on the cleanup is correspondingly larger. Operational overhead: the cost that does not show up on the bill Operational overhead is the layer that does not appear on the vendor bill at all. It surfaces as the time the platform team spends on manual operations that would not be necessary if the estate were better governed. The specific costs include: **Manual provisioning.** Ticket-based topic creation, partition changes that require platform team involvement for every request. **Firefighting.** Capacity emergencies when disks fill or capacity thresholds hit unexpectedly, requiring urgent intervention that pulls engineers off planned work. **Cross-team coordination.** Reaching out to dozens of project teams to confirm ownership before deleting topics, negotiating retention changes, managing exceptions. **Knowledge loss.** When the engineer who understood why certain decisions were made leaves, the remaining team spends time rediscovering context that should have been captured in tooling or policy. These compound alongside the infrastructure waste. A platform team spending 40 to 60 percent of its time on reactive operations and manual provisioning is common. That share of senior engineering time is the constraint on how many internal teams the platform can support, how well it supports them, and whether the platform team can do anything beyond keeping the lights on. Stories from the field **Nine-month lead time on new hardware.** A platform team running Kafka on-premises had a nine-month lead time on new machine provisioning. That number set the tempo for everything the team did. When storage on a cluster started to fill, they scrambled to retire topics, tighten retention, or reshuffle workloads, because waiting for hardware was not a viable answer. There was no proactive cleanup mechanism; cleanup happened when disks were about to fill. The same dynamic plays out in cloud estates, just with a different trigger: cleanup happens when the bill escalates to a level that draws leadership attention. Operational overhead is the constraint that limits how many internal teams a small platform team can support effectively. A meaningful share of the platform team's time goes to overhead that doesn't grow the platform's reach. Reducing that share is how a small team scales: more Kafka users, more use cases, and a better developer experience, without growing in proportion. What to do about each pattern Capturing the savings requires choosing a response that fits the pattern, the estate, and the team's capacity for coordination. Three response categories cover the work: **Update defaults.** Stop new topics, clusters, and clients from inheriting waste at creation. Low coordination, no immediate savings, slows future growth. **Optimize existing loads.** The hygiene and rightsizing work on what is already running. Weeks to months of effort; typically moves the bill 10 to 20 percent. **Rethink workloads.** Architectural changes for cases where the current shape no longer fits. Largest absolute savings in big estates, longest timelines. Most patterns can be addressed through more than one approach. The mapping is many-to-many, and the right starting point depends on the estate. Pattern Updating defaults Optimizing existing loads Rethinking workloads **Partition overprovisioning** Sensible partition defaults for new topics Right-size existing topics via recreation Pool partition capacity across topics **Retention not matched to consumer needs** Default retention by topic class Audit and tune existing topics Not typically required **Cluster sprawl** Default cluster tier for new use cases Consolidate underutilized clusters Move to pooled cluster capacity **Topic proliferation and duplication** Ownership and cataloguing at creation Retire orphans, deduplicate Discoverability layer **Inefficient client patterns** Compression enforcement, default fan-out limits Audit consumer groups, restructure fan-out Architectural changes to consumer model **Static capacity per resource** Not addressable through defaults Not addressable through optimization Capacity pooling layer Resist the urge to tag each pattern with a single primary response. The right choice depends on where the waste actually concentrates in your estate and how much coordination capacity the team has. Collect the data Acting on any of these patterns starts with collecting data the platform does not surface by default. Two kinds of data, in two different shapes. The first is technical. Where the cost is actually coming from in your estate: partition counts versus actual throughput, egress-to-ingress ratios per topic, cluster utilization at peak load, cost-by-team. None of these surface by default; pulling them together is deliberate work. The second is organizational. What your team and leadership are actually set up to change: drivers, cleanup history, where guardrails get bypassed, how engineers adapt to constraints today. This data lives in interviews, past-cleanup retrospectives, and conversations with leadership, not in cluster metrics. Both kinds matter. **The technical data tells you which patterns are present. The organizational data tells you which response approaches will actually land.** A serious analysis depends on both. Example questions for the organizational side These are the kinds of questions worth working through with the team and with leadership. Not a checklist, just examples of where the conversation tends to go when we run cost analyses with platform teams. 1. **What's driving the conversation right now?** A renewal, a leadership directive after a budget review, on-prem capacity pressure, or a sustained gap between business throughput and Kafka spend. The forcing function shapes which patterns are the priority and how much organizational support the work will have. 2. **Is there any mechanism that ties Kafka costs back to the app teams generating them?** Chargeback, showback, or even basic visibility into per-team consumption. Without a feedback loop, app teams have no incentive to right-size their own usage, and every cleanup becomes a centralized push by the platform team against teams that bear none of the cost themselves. 3. **What do engineers do when they need more partitions on a topic that already exists?** Recreate with full producer-consumer coordination (rare in practice), live with the constraint, or spawn a parallel topic. The actual workaround predicts how much partition overprovisioning is being baked in by individual decisions. 4. **What do teams do when they need data from a topic in another cluster?** Replicate, build a custom bridge, or duplicate the data into their own cluster. The default path predicts whether topic proliferation will reverse with cleanup or keep regenerating. 5. **How are guardrail exceptions managed?** Whether partition, retention, or cluster guardrails exist matters less than how exceptions are approved and tracked when strategic projects bypass them. Exceptions that accumulate without review become the largest pockets of waste in most estates. 6. **Where did the previous cleanup stall?** Cross-team coordination to confirm topic ownership, reluctance to touch live workloads, lack of leadership air cover, or political exceptions for strategic projects. The same blocker will usually hit the next cleanup. The cost analysis we offer covers both halves equally. We help you figure out exactly what data to pull and how to interpret it, and we work through which response approaches will actually move costs within your specific organizational context. Want to see how much you could save on your Kafka bill? Get a free Kafka cost analysis with our field engineering team. We will walk through your estate together, identify the waste patterns that apply, and give you a concrete estimate of where the savings are. [Get your Kafka Cost Analysis](https://www.conduktor.io/contact/kafka-cost-analysis?src=resources-ebooks-where-kafka-costs-hide-a-field-guide) Keep reading the full guide Source: https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide --- # Ebooks & Whitepapers Free guides from Kafka practitioners. - [When Kafka Becomes the Ledger](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger) — Whitepaper: When transaction and payment data lives on Kafka, much of fintech's correctness work moves from apps to the proxy. A control-plane operating model. - [Where Kafka Costs Hide: A Field Guide](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) — Whitepaper: For platform leads, Kafka architects, and streaming engineers who know Kafka is expensive but can't fully explain where or why. A structured way to find, quantify, and act on the hidden costs in your estate. - [Kafka Disaster Recovery Beyond Replication](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-beyond-replication) — Whitepaper: A complete Kafka DR strategy covering six technical areas beyond data replication, organized across three operational phases. Includes chaos testing methodology, compliance mapping, and a failover runbook. - [The Kafka Debugging Playbook: Six Phases from Alert to Root Cause](https://www.conduktor.io/resources/ebooks/the-kafka-debugging-playbook) — Whitepaper: A Kafka troubleshooting and debugging framework for developers, SREs, and DevOps teams. Three production incidents walked through from alert to root cause, covering consumer lag, Kafka Connect, Schema Registry, offset resets, and reducing MTTR. - [AWS MSK: Govern, Secure & Scale Kafka](https://www.conduktor.io/resources/ebooks/aws-msk-solution-brief) — Solution Brief: How Conduktor and AWS MSK work better together: encryption, governance, resilience, and multi-tenancy at the protocol level, with no code changes. - [Kafka Disaster Recovery Readiness Checklist](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-readiness-checklist) — Checklist: Six technical areas that determine whether you recover in minutes or hours. A practitioner-built checklist for Kafka disaster recovery beyond replication. - [The 2025 Data Streaming & AI Report](https://www.conduktor.io/resources/ebooks/the-2025-data-streaming-ai-survey-report) — Report: How 200 senior IT and data leaders use data streaming for AI, and why the governance gap across platforms is holding back their return on investment. - [The Journey to Self-Service](https://www.conduktor.io/resources/ebooks/the-journey-to-self-service) — Checklist: A six-phase checklist for moving Kafka from platform-team bottleneck to safe developer self-service, from assessment through governance to optimization. - [Federating Kafka Data Governance at Scale](https://www.conduktor.io/resources/ebooks/federating-data-governance-scaling-kafka-without-losing-control) — eBook: Escape the control-vs-autonomy dilemma in Kafka with federated data governance: shared guardrails, local autonomy, and how Conduktor delivers it. - [AI Can't Wait: Real-Time Data Quality](https://www.conduktor.io/resources/ebooks/ai-can-t-wait-why-real-time-systems-demand-upstream-data-quality) — eBook: Real-time AI acts on data the instant it arrives. See why data quality must be enforced upstream, in-stream, and how to build an AI-ready data strategy. - [Governing Kafka Data for the AI Era](https://www.conduktor.io/resources/ebooks/governing-kafka-data-for-the-ai-era) — Whitepaper: Kafka is AI's transport layer, but not secure out of the box. A field guide to governing streaming data quality, security, and compliance for GenAI. - [Achieving Data Security for Kafka](https://www.conduktor.io/resources/ebooks/achieving-data-security-for-kafka) — eBook: A practical guide to securing Apache Kafka: encryption in transit and at rest, RBAC, audit logging, plus a readiness checklist for platform teams. - [Kafka at Scale: 5 Challenges & Solutions](https://www.conduktor.io/resources/ebooks/kafka-at-scale-five-critical-challenges-and-how-to-solve-them) — eBook: Five Kafka challenges at scale: poor data quality, manual workflows, team silos, zombie costs, and legacy sprawl, plus how to solve each one. Source: https://www.conduktor.io/resources/hub --- # Kafka Resources Docs, 170+ glossary terms, tutorials, customer stories, and webinars. - [Documentation](https://docs.conduktor.io/) — Console & Conduktor Gateway setup, API reference, integrations - **Architecture** — Your VPC, your SSO, your KMS, fully customer-managed — architecture - **Product Releases** — August 2026: Live lineage, Flink SQL, and Chargeback — product-releases — New - **Product Demos** — Product walkthroughs — product-demos - **Customer Stories** — ROI, use cases, testimonials — customer-stories - **Ebooks & Guides** — Free downloads — resources/hub - **Events & Webinars** — Live and on-demand — events - [Kafkademy](https://www.conduktor.io/kafka) — Hands-on Kafka courses, beginner to expert - **Blog** — We Deleted a Kafka Topic. Which App Broke? (Sep 2026) — blog - **search** — Kafka Glossary — 170+ terms. Try: Semantic Layer for Streaming: Business Meaning for Real-Time Data — glossary Connect with us Questions? Ideas? Just want to chat Kafka? --- assets/icons/zendesk-icon.svg | Support | Get answers from our team | https://support.conduktor.io/ assets/icons/github-icon.svg | GitHub | Explore our open source projects | https://github.com/conduktor assets/icons/slack-icon.svg | Slack | Join 5,000+ Kafka practitioners | /slack Source: https://www.conduktor.io/resources --- # Schema Registry Proxy *Authorize. Audit. Govern.* A schema-registry-aware proxy that adds access control, auditing, and policy enforcement to the schema registry you already run. Point your clients at it. No code changes. [Talk to Us](https://www.conduktor.io/contact/demo?src=schema-registry-proxy) [Download Solution Brief](https://www.conduktor.io/assets/pdfs/use-cases/schema-registry-proxy.pdf) ## Let's protect the schema registry like we protect Kafka. We built the Schema Registry Proxy so every schema operation is authorized and on the record, on the registry you already run, with the identities you already manage. Drop in, and point to any provider. 01 · **aware** Speaks schema, not just HTTP The usual fix is nginx or an API gateway in front of the registry. Sensible, but it guards routes and methods, not subjects. It can't tell a new version from a destructive overwrite, because it has no idea what a subject or a schema is. With the proxy The proxy speaks the schema registry's own API. It reasons about **subjects, versions, and compatibility**, so every rule is expressed in the registry's own terms. 02 · **access** Authorization out of the box Open-source registries give you coarse roles or a password file, and no per-subject control. Confluent's answer is Confluent Platform RBAC, often $100k+ a year. Either way, anyone with access can change any schema, and the burden is on you. With the proxy **Per-subject read and write permissions**, checked on every request, on the registry you already run. And they come from where they already live: your Kafka ACLs or [federated ownership](https://www.conduktor.io/console#federated-ownership) in Console. No platform license. 03 · **audit** Put every operation on the record Out of the box, schema registry auditing doesn't exist. From vendors, it's another paid tier. A schema changes, consumers break, and there is no record of who did it, when, or why. Compliance asks for evidence and there is none to give. With the proxy Every operation is **traced end to end**, every denial is logged with who tried what, and every permission change is recorded. ## Grant · Enforce · Observe. Give each application its subjects, let the proxy check every call, and keep a record of everything. Permissions come from Console or the Kafka ACLs you already manage. 01 · grantGive each app its subjectsThrough federated ownership in Console, where applications own their subjects like they own their topics. Or standalone, on the Kafka ACLs you already operate. 02 · enforceEvery request checkedThe JWT is verified against your identity provider, the permission is checked against the subject, and only then is the call forwarded. Denied calls never reach the registry. 03 · observeKnow who did whatA trace for every operation and a log line for every denial, with the caller's identity. OpenTelemetry and Prometheus out of the box. With self-service With Kafka ACLs apiVersion: self-serve/v1 kind: ApplicationInstance metadata: application: "payments" name: "payments-prod" spec: cluster: "prod-kafka" serviceAccount: "payments-service" resources: - type: TOPIC patternType: PREFIXED name: "payments" - type: SUBJECT patternType: PREFIXED name: "payments" $ conduktor apply -f payments-prod.yaml ApplicationInstance/payments-prod: Created $ kafka-acls --bootstrap-server kafka:9092 \ --command-config admin.properties \ --add --allow-principal User:payments-service \ --operation WRITE --operation READ \ --topic payments --resource-pattern-type PREFIXED Adding ACLs for resource ResourcePattern(resourceType=TOPIC, name=payments, patternType=PREFIXED): (principal=User:payments-service, host=*, operation=WRITE, permissionType=ALLOW) (principal=User:payments-service, host=*, operation=READ, permissionType=ALLOW) # subjects inherit their topic's ACLs: # subject payments-value → topic "payments" # payments-service registers a new version $ curl -X POST http://srp:8080/subjects/payments-value/versions \ -H "Authorization: Bearer $PAYMENTS_TOKEN" \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d @new-schema.json {"id": 7} # analytics-service, never granted payments-*, tries the same HTTP/1.1 403 Forbidden { "error_code": 403, "message": "Forbidden: Service account 'analytics-service' is not authorized to Write resource 'payments-value'" } Denial log Trace POST /subjects/{subject}/versions 38ms │ http.method: POST │ http.target: /subjects/payments-value/versions │ http.status_code: 200 │ └─ HTTP POST schema-registry:8081 31ms # one trace per operation: proxy overhead, # backend latency, and status on every span { "@timestamp": "2026-06-05T01:12:09.482Z", "level": "WARN", "service": "schema-registry-proxy", "message": "Authorization failed: Service account 'analytics-service' is not authorized to Write resource 'payments-value'" } { "@timestamp": "2026-06-05T01:14:51.077Z", "level": "WARN", "service": "schema-registry-proxy", "message": "Authentication failed: JWT validation failed: Expired JWT" } ## The same registry, locked down. Put the proxy in front of the schema registry you already run. Nothing about your applications changes. Everything about who can touch schemas does. Before Schema Registry Proxy Some teams run the registry wide open. Most front it with a generic HTTP proxy that checks routes and tokens. Both guard the door, not the schemas behind it. **Neither setup knows what a subject is**, or who owns it. After Schema Registry Proxy Every request is authenticated against your identity provider and checked against per-subject permissions. Add [Conduktor Console](https://www.conduktor.io/console) for federated ownership and human schema management. **The proxy knows both**, on every request. ## A schema is a deliverable, not a runtime accident Most schema incidents start the same way: the schema is born as a side effect of the first message a service sends. Ship it on purpose instead, and the failures move left into code review, where they are cheap. - **Register it on purpose** — The Confluent serializer registers a schema on first produce, `auto.register.schemas` on by default. It lands whenever that code path first runs, often hours after deploy, so a break is hard to trace back to its cause. Provision schemas ahead of the deploy and the registry validates them before anything ships. - **One compatibility history per environment** — Dev accumulates more versions than QA, QA more than prod. Each has its own compatibility lineage, so a change that passes in dev can still be rejected against prod. Check it against the environment you are promoting into, and dev warns you before you get there. - **Schemas as code** — Keep subjects in version control, review them in a pull request, apply them from CI. The manifest the proxy reads is a file in your repo, so evolving a schema is a diff someone approves, not a call an app makes at runtime. Capabilities Built to drop into your stack. Works with any registry that implements the Confluent Schema Registry API, including Confluent Cloud, Apicurio, and Karapace. Per-subject permissionsRead and write per subject, with exact, prefix, and wildcard matching. Checked on every request, before anything reaches your registry. Your identity providerKeycloak, Auth0, Okta, Microsoft Entra ID, or any OAuth2/OIDC provider with a JWKS endpoint. Automatic key rotation included. Kafka-native permissionsDelivered through a Kafka topic from Console and applied in seconds, or read from your existing ACLs. No restarts, no new datastore. Fails closedIf a permission can't be verified, the request is denied. Errors and timeouts never fail open. Traces, logs, and metricsOne OpenTelemetry trace per operation, every denial logged with the caller's identity, and Prometheus metrics out of the box. Runs where Kafka runsA single container, on-premises, in the cloud, or air-gapped. Alongside Gateway or on its own. Resource policies · nextSet schema standards once: compatibility floors, format requirements, naming and namespace conventions, and required documentation, enforced at registration for every team and tool. Rate limiting · nextPer-principal request limits, so one bad deploy in a restart loop can't flood the registry every other team depends on. ## Four Ways to Run a Schema Registry Most teams are in one of the first three columns today. | | No protection | Generic HTTP proxy | Confluent Platform RBAC | Schema Registry Proxy | |---|---|---|---|---| | Authorization | None | Route allowlists | Per-subject RBAC | ✓ Per-subject, prefix and wildcard | | Understands | Nothing | URLs and methods | Subjects | ✓ Subjects and your Kafka identities | | Identity | None | Separate auth config | Confluent's identity plane | ✓ OIDC / OAuth2, the same accounts as your Kafka | | Works with | Any registry | Any registry | Confluent Platform only | ✓ Any registry with the Confluent SR API | | Licensing | Free | Engineering time | Full platform license | ✓ Standalone | | Maintenance | None | Your team | Confluent | ✓ Conduktor | Note for Confluent Platform Customers The Topic ACL Authorizer is deprecated. Your ACLs don't have to be. For years, the **Topic ACL Authorizer** let topic owners govern their own subjects. Confluent Platform 8.0 deprecates it, which puts every team that relies on it on a migration clock toward Confluent RBAC, an enterprise-license feature. Moving to RBAC is not a config flag. The topic-based grants your teams built up have to be re-modeled as role bindings and rewritten across every cluster and environment, on top of the platform license, a change program in its own right. There's a third option. The proxy's standalone mode is modeled after the authorizer: the same topic ACLs keep governing the same subjects. Point `schema.registry.url` at the proxy and upgrade on your own schedule. No RBAC migration, no platform license. ## Frequently Asked Questions **Do I need to change my application code?** No. Update `schema.registry.url` in your Kafka client configuration to point at the proxy. Producers and consumers continue using their existing serializers and deserializers. No code changes, no library swaps. **How are permissions managed?** Two ways. Through Conduktor Console, where subjects join your federated ownership model and changes apply in seconds via Kafka. Or standalone, where the proxy reads the Kafka topic ACLs you already manage. Console is not required. **Can it replace the Confluent Topic ACL Authorizer?** Yes. Standalone mode is modeled after it: the same topic ACLs govern the same subjects, with the same semantics. It works on Confluent Platform 8.0 and beyond, where the authorizer is deprecated. The [deployment guide](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-schema-registry-proxy) covers the exact mapping. **Which schema registries does the proxy support?** Confluent Schema Registry and any registry that implements the same REST API, including Confluent Cloud. **What can applications do through the proxy?** Only what they need: produce and consume against schemas that already exist. Evolution, meaning new versions, compatibility changes, and deletes, goes through a governed workflow like Console, so destructive and configuration endpoints are never on the path an application uses. And with runtime registration off, a client can no longer register a schema the first time it produces, hours after deploy, which is where auto-registration usually bites. **Why keep deletes and compatibility changes away from applications?** Because both are close to irreversible. Delete a subject or a version and every message still serialized against it becomes unreadable, so you lose the ability to deserialize your own history, the same way losing a key makes encrypted data unrecoverable. Loosening a subject's compatibility mode is quieter but just as damaging: the next incompatible version passes with no error at the time, and the guarantee every consumer relied on is gone. These are deliberate, authorized acts, not something an application or a stray UI should be able to trigger. **Why not just use the schema registry's own authorization?** Two native options exist and both disappoint. One infers permissions from Kafka, where whoever can write a topic can write its subject, which is what the Topic ACL Authorizer gave teams, now deprecated in Confluent Platform 8.0. The other is a native permissions API, but almost no ecosystem tooling grew around it, no Terraform, no Ansible, no wrappers, so few teams ever adopted it. That leaves Confluent RBAC and the full platform license. The proxy gives you per-subject authorization on any registry, with the Kafka identities you already have, without that trade. **What's the relationship between the Schema Registry Proxy and Conduktor Gateway?** The same philosophy at a different layer. Gateway sits in front of your Kafka clusters. The Schema Registry Proxy sits in front of your registry. Both enforce who can do what without touching clients, and they share the same identities. Deploy them together or independently. ## Ready to govern your schema registry? Our team can walk you through a deployment that fits your identity provider, your Kafka setup, and your compliance requirements. [Talk to Us](https://www.conduktor.io/contact/demo?src=schema-registry-proxy) [Download Solution Brief](https://www.conduktor.io/assets/pdfs/use-cases/schema-registry-proxy.pdf) ## Read more customer stories - [Bitvavo: DORA & MiCA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [Retail: 99% Faster Kafka Onboarding](https://www.conduktor.io/customer-stories/one-hour-kafka-onboarding-retail-streaming-operations) - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) Source: https://www.conduktor.io/schema-registry-proxy --- # Kafka Data Security & Encryption for Financial Services Protect sensitive financial data in motion. Unified encryption, masking, and key management within Kafka pipelines, delivering end-to-end compliance and centralized control for GRC and InfoSec teams. [See it in action](https://www.conduktor.io/contact/demo?src=solutions-industry-financial-services-data-security) Trusted by security teams at ## Why fragmented encryption doesn't work for finance. ### Inconsistent Coverage Encryption practices vary across runtimes and frameworks: Python, Kotlin, .NET, Flink, Connect. Each team implements their own approach, creating gaps. ### Operational Overhead Fragmented Vault and KMS integration across clusters complicates key rotation and audit. GRC teams lack visibility. ### Developer Friction Encryption slows delivery. Unclear ownership, inconsistent tooling, and schema mismatches between producers and connectors cause errors. ## Why Conduktor for Data Security - **Unified Encryption Layer** — Apply encryption and masking consistently across all Kafka flows: Flink, Connect, REST, any client - **Field-Level Protection** — Encrypt specific fields based on schema tags. Salary, SSN, card numbers protected without full-payload overhead - **Key Management Integration** — Connect to Vault (AppRole), KMS, Voltage, or Fortanix. Centralized rotation and lifecycle control - **Cross-Language Consistency** — Same encryption policies for .NET, Kotlin, Python, Flink, and REST clients. No per-team implementations - **Real-Time Audit Logs** — Every encrypt/decrypt operation logged. Immutable evidence for GRC, InfoSec, and regulators - **GRC Dashboards** — Track encryption coverage, exceptions, and compliance status across all clusters in one view - **Schema-Tag Enforcement** — Mark sensitive fields in your schema. Conduktor encrypts them automatically at the wire. No code changes required. - **Crypto-Shredding** — Delete encryption keys to render data unreadable. Meet data retention and right-to-erasure requirements. - **Full-Payload Encryption** — Start with full-payload encryption for immediate compliance, then evolve to field-level as policies mature. - **Exception Tracking** — Monitor and alert on unencrypted data flows. Identify gaps before auditors do. - **Automated Key Rotation** — Schedule key rotation through existing KMS systems. No manual intervention, no downtime. - **Zero Client Changes** — Conduktor Gateway, our Kafka proxy, handles encryption transparently. Existing producers and consumers continue working unchanged. ## How Data Security Works A pragmatic path from compliance baseline to field-level protection. - **Connect Key Management** — Integrate with Vault, KMS, or your existing key provider. Centralized control from day one - **Define Encryption Policies** — Tag sensitive fields in schemas or apply full-payload encryption. Policies enforce automatically - **Deploy Conduktor Gateway** — Our Kafka proxy applies encryption to all traffic. No client rewrites. Producers and consumers unchanged - **Monitor & Audit** — GRC dashboards show coverage, exceptions, and compliance status. Evidence ready for regulators ## Key Use Cases - **Loan and Credit Systems** — Mask salary, SSN, and account data while enabling risk models to operate on encrypted fields - **Fraud and AML Pipelines** — Encrypt device fingerprints and transaction payloads while preserving correlation for anomaly detection - **Healthcare and Insurance** — Apply field-level masking for PHI (diagnosis codes, policy numbers) before analytics or downstream exports - **Payments and Card Processing** — Tokenize card numbers and personal identifiers at the producer level before events reach Kafka - **KYC and Regulatory Auditing** — Enforce schema-tag encryption on customer identity streams with crypto-shredding for data retention control - **Data Governance Automation** — Integrate Kafka encryption with Vault or KMS to standardize key management and automate audit generation ## Read more customer stories - [Bitvavo: DORA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [European Payment Processor: PCI DSS](https://www.conduktor.io/customer-stories/securing-kafka-banking-financial-services) - [Swiss Post: Governed Kafka](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) ## Frequently Asked Questions **Do I need to modify my producers and consumers?** No. Conduktor Gateway encrypts at the wire layer. Your existing applications work unchanged. No code modifications, no library updates. **What key management systems are supported?** Conduktor integrates with HashiCorp Vault (AppRole), AWS KMS, Azure Key Vault, Google Cloud KMS, Voltage, and Fortanix. Custom integrations available. **Can I start with full-payload and move to field-level later?** Yes. Most organizations start with full-payload encryption to meet immediate compliance deadlines, then add field-level policies as their data classification matures. **How does field-level encryption work?** Two options: tag fields in your Avro or Protobuf schema, or use our flexible API to define encryption rules without touching schemas. Both approaches encrypt fields automatically at the wire. **How do consumers decrypt the data?** You define who can decrypt based on identity, group, or context. Supports contextual decryption rules for cross-continental restrictions, data residency, and regulatory boundaries. ## Ready to secure your Kafka data? See how Conduktor delivers end-to-end encryption without client changes. Our team can help you design an encryption strategy that meets your compliance requirements. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-financial-services-data-security) For the operating model behind these controls, read [When Kafka Becomes the Ledger](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger), our control-plane guide for transaction and payment data on Kafka. Source: https://www.conduktor.io/solutions/industry/financial-services/data-security --- # Compliant Real-Time Data Sharing for Financial Services Real-time data sharing across partners, regulators, and clouds without losing control. Masking, encryption, and auditability at every boundary, enabling compliant, trackable data exchange. [See it in action](https://www.conduktor.io/contact/demo?src=solutions-industry-financial-services-external-data-sharing) Trusted by data teams at ## Why traditional data sharing creates risk for finance. ### Partner Onboarding Delays Firewall rules, certificates, and network reviews delay every new partner. Weeks pass before first data exchange. ### Inconsistent Controls Multiple identity models (OIDC, tokens, BasicAuth) across partner ecosystems. No single control point for schema validation, ACLs, and encryption. ### Compliance Exposure No immutable audit evidence linking Kafka topics to regulator or SaaS endpoints. No FinOps visibility for chargeback. ## Why Conduktor for Data Sharing - **Unified Sharing Gateway** — Single control point for all external data flows. mTLS, ACLs, and schema validation enforced at the boundary - **Field-Level Masking** — Mask sensitive fields before exposure. Share transaction data while protecting PII - **BYOK Encryption** — Integrate with Voltage, Fortanix, or KMS. Your keys, your control, your compliance - **Flexible Authentication** — OIDC, API keys, or tokens. Onboard partners without rebuilding your identity model - **Immutable Audit Logs** — Every shared event logged with lineage. Evidence ready for PCI DSS, GLBA, and DORA - **FinOps Chargeback** — Track consumption per partner or business unit. Cost attribution and transparency across all data flows - **Zero-Trust Architecture** — Partners access only authorized topics. Topology and naming hidden. No cluster exposure - **Dynamic Masking** — Apply masking rules based on consumer identity. Same topic, different views - **Live Data Sharing** — Share Kafka data in real-time. No replication, no file exports, no data copies - **Lineage Tracking** — Trace every event from source topic to external consumer. Complete audit trail - **Throttling Controls** — Rate-limit external consumers. Protect cluster performance during partner surges - **Fast Onboarding** — Partners connect in days, not weeks of setup. Standard protocols, automated provisioning ## How Data Sharing Works From internal cluster to external partner, governed end-to-end. - **Define Sharing Policies** — Set which topics, fields, and events can be shared. Masking and encryption rules apply automatically - **Configure Partner Access** — Onboard partners with OIDC, tokens, or API keys. Access scoped to specific topics and fields - **Enable Live Streaming** — Partners consume Kafka data in real-time through the gateway. No replication or exports - **Monitor & Audit** — Track every access, generate compliance evidence, allocate costs per partner ## Key Use Cases - **Regulatory Reporting** — Deliver near-real-time reports to financial authorities with end-to-end lineage and immutable audit evidence - **Partner Connectivity** — Share Kafka topics externally via OIDC or token authentication with schema validation and BYOK encryption - **Credit and Risk Exchange** — Provide loan, payment, and scoring events to credit bureaus with field-level encryption and masking - **Vendor and SaaS Integrations** — Expose selected event streams to external providers under zero-trust controls - **B2B and Payment APIs** — Offer real-time data access to partner banks or payment networks with schema enforcement - **Data Marketplace** — Power monetized or cross-BU data exchanges with FinOps chargeback and consumption tracking ## Read more customer stories - [Bitvavo: Secure Data Flows](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [European Airline: Partner Communication](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) - [Swiss Post: Cross-Team Sharing](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) ## Frequently Asked Questions **Can I share data without replicating it?** Yes. Conduktor acts as a gateway. Partners consume directly from your Kafka cluster through controlled, masked, and audited channels. No data copies. **How do partners authenticate?** Conduktor supports OIDC, API keys, tokens, and mTLS. Use your existing identity provider or issue dedicated credentials per partner. **What masking options are available?** Field-level masking with multiple strategies: redaction, hashing, tokenization, or partial masking. Rules apply based on consumer identity. **How does this meet DORA requirements?** Immutable audit logs, complete lineage from source to consumer, and evidence of encryption coverage. Audit reports generated automatically. **Can I track what each partner is consuming?** Yes. FinOps dashboards track consumption per partner, topic, and time period. Use for usage attribution, internal chargeback, or commercial agreement reporting. ## Ready to share Kafka data securely? See how Conduktor enables compliant data exchange with partners and regulators. Our team can help you design a sharing strategy that meets your security requirements. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-financial-services-external-data-sharing) [See how real-time data sharing works](https://www.conduktor.io/solutions/use-case/data-sharing), or explore the [data streaming platform](https://www.conduktor.io/product/data-streaming-platform) it runs on. For the control-plane operating model behind regulated financial data on Kafka, read [When Kafka Becomes the Ledger](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger). Source: https://www.conduktor.io/solutions/industry/financial-services/external-data-sharing --- # Kafka Self-Service & Data Products for Financial Services Turn Kafka into a governed data product platform. Expose, secure, and manage Kafka topics as reusable data products, accelerating developer autonomy while maintaining full auditability and control. [See it in action](https://www.conduktor.io/contact/demo?src=solutions-industry-financial-services-governed-self-service) Trusted by platform teams at ## Why ticket-driven Kafka governance doesn't scale for finance. ### Data Locked in Silos Different business units manage isolated clusters and topics without shared visibility. Data exists but isn't discoverable or reusable. ### Manual Provisioning Bottlenecks ACLs, schemas, and connectors are manually managed. Platform teams own every change while developers wait in ticket queues. ### Compliance & Audit Gaps Shadow pipelines bypass GRC oversight. Missing lineage, consumer lag insights, and visibility into who consumes what. ## Why Conduktor for Self-Service - **Automated Provisioning** — Policy-driven templates for topics, ACLs, and connectors. Developers self-serve within guardrails - **Data Product Catalog** — Register topics as reusable data products with ownership, lineage, and access contracts - **Identity Integration** — Azure AD, Okta, LDAP, or IAM. Group-based access and least-privilege roles enforced automatically - **Consistent RBAC** — Same governance across Confluent, MSK, and self-managed. One policy, every cluster - **Terraform & GitOps** — Version-controlled, auditable provisioning through your existing CI/CD pipeline - **GRC Visibility** — Policy violations, encryption coverage, and user activity. All in one dashboard for compliance teams - **Policy Controls** — Enforce naming conventions, partition limits, retention settings, and schema standards automatically - **Approval Workflows** — Embed approvals into CI/CD or ServiceNow. Manual review where needed, automation everywhere else - **Ownership Model** — Assign data product owners. Clear accountability for schema versions, access approvals, and data quality - **Lineage Tracking** — See producers, consumers, and compliance status per data product. Know who uses what and why - **Operational Efficiency** — Eliminate tickets through automated provisioning and lifecycle cleanup. From weeks to minutes - **Developer Portal** — Self-service UI for topic discovery, access requests, and resource management ## How Data Products Work From request to production in minutes, not weeks. - **Define Guardrails** — Platform team sets policies: naming, schemas, retention, access. Guardrails apply automatically - **Register Data Products** — Topics become discoverable products with owners, schemas, and access metadata - **Enable Self-Service** — Developers request access, provision resources, and manage schemas within policy bounds - **Monitor & Audit** — Every change logged. Lineage tracked. Compliance evidence generated automatically ## Key Use Cases - **Data Product Enablement** — Publish Kafka topics as certified products with schema contracts, ownership, and access controls - **Developer Self-Service** — Engineers request and manage topics, ACLs, and connectors with built-in approvals and audit - **Hybrid Cloud Governance** — Synchronize identity and schema validation across Confluent, MSK, and self-managed clusters - **Compliance Automation** — Generate auditable logs of every access, schema update, and data product change for regulators - **Data Science Access** — ML and analytics teams subscribe to curated, masked Kafka data products safely - **Regulatory Sandboxes** — Reuse governed data products in test environments without violating masking or retention policies ## Read more customer stories - [FlixBus: Self-Service for 50+ Teams](https://www.conduktor.io/customer-stories/flix) - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Virgin Australia: 300 Hours/Month Saved](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) ## Frequently Asked Questions **What is a Kafka data product?** A data product is a Kafka topic packaged with ownership, schema, access metadata, and quality contracts. Teams discover and consume data products without understanding the underlying infrastructure. **How do you prevent teams from creating inconsistent resources?** Policy controls enforce naming conventions, partition limits, retention settings, and schema standards automatically. Teams can only create resources that comply with defined policies. **Can I still require approvals for production changes?** Yes. Conduktor supports configurable approval workflows. Require manual approval for production while allowing self-service for development environments. **How does this integrate with existing identity providers?** Conduktor integrates with Azure AD, Okta, LDAP, and IAM providers. Group memberships map to Kafka access. No separate credential management. **Does this work with multiple Kafka distributions?** Yes. Same governance layer across Confluent Cloud, AWS MSK, and self-managed Kafka. One policy applied everywhere. ## Ready for self-service Kafka? See how Conduktor enables developer autonomy without sacrificing governance. Our team can help you design the right self-service strategy for your organization. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-financial-services-governed-self-service) Source: https://www.conduktor.io/solutions/industry/financial-services/governed-self-service --- # Kafka for Financial Services | Data Security & Compliance Conduktor for Financial Services **Secure, compliant data streaming.** Protect sensitive financial data with field-level encryption, meet DORA and PCI-DSS requirements, and accelerate innovation with self-service Kafka. **Field-level encryption** - Encrypt PII and financial data at the wire. No code changes. 6 KMS backends including AWS, Azure, and HashiCorp Vault. **DORA & PCI-DSS compliance** - Audit trails, data masking, and access controls built for financial regulations. Proven in production at banks and payment processors. **Real-time fraud & AML** - Stream transactions across accounts and channels for pattern detection. Sub-second alerting for suspicious activity feeds your fraud detection and AML systems. **Self-service with guardrails** - Developers provision topics in minutes. Policies enforce naming conventions, schemas, and access automatically. ### Get in touch Why Financial Services Teams Choose Conduktor Self-Service That Satisfies Compliance Developers provision topics and access in minutes—while policies enforce data classification, audit logging, and access controls automatically. Every change is logged for regulators. Protect Data at the Wire Field-level encryption for PII and financial data—no code changes required. Complete audit trails and granular RBAC. Meet DORA, PCI-DSS, and GDPR with infrastructure proven at Tier 1 banks. Schema Enforcement for Regulated Data Enforce Avro or Protobuf schemas at the gateway. Block malformed trade events, payment messages, and customer records before they reach Kafka—keeping downstream risk systems clean. Deep Dive by Use Case - **Data Security & Encryption** — Field-level encryption, key management, and compliance for PCI DSS and GDPR. [Learn more](https://www.conduktor.io/solutions/industry/financial-services/data-security) - **Governed Self-Service** — Developer autonomy with automated provisioning, data products, and full auditability. [Learn more](https://www.conduktor.io/solutions/industry/financial-services/governed-self-service) - **External Data Sharing** — Share Kafka data with partners and regulators. Masking, encryption, and audit trails. [Learn more](https://www.conduktor.io/solutions/industry/financial-services/external-data-sharing) Where Conduktor Helps in Financial Services Conduktor connects to your existing Kafka infrastructure, whether Confluent, AWS MSK, or self-managed, and adds the governance, security, and self-service capabilities these use cases demand. For the operating model behind these use cases, read [When Kafka Becomes the Ledger](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger), our control-plane guide for transaction and payment data on Kafka. Real-time fraud detection Payment processing Anti-money laundering Trade execution Regulatory reporting MiFID II / CAT reporting Credit risk scoring Market data feeds Cross-border payments ISO 20022 migration Audit logging Account notifications Source: https://www.conduktor.io/solutions/industry/financial-services --- # Self-Service Kafka for Retail Teams Platform teams at retailers manage hundreds of developers. Self-service workspaces let teams provision topics, schemas, and access without tickets. Policies enforce standards automatically. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-retail-and-ecommerce-developer-self-service) Trusted by platform teams at ## Why ticket-driven Kafka doesn't scale for retail. ### Platform Team Bottleneck A small platform team handles every topic, ACL, and schema change for hundreds of developers. Tickets pile up. Projects wait. ### Shadow Tooling Spreads Teams bypass the platform with direct admin access and custom scripts. Naming conventions drift. Topic catalogs lose accuracy. ### Institutional Knowledge Lost The developer who set up Kafka left. New teams inherit topics they don't understand. No documentation, no owners, no samples. ## Why Conduktor for Developer Self-Service - **Team Workspaces** — Each team gets a workspace linked to IAM groups. Teams see and manage their own resources—not everyone else's - **Application Policies** — Policies encode naming conventions, retention limits, partition counts, and replication factors. Consistency without tickets - **Topic Catalog** — Search topics by owner, label, or schema. Find existing data before creating duplicates. Documentation and samples attached - **Guided Workflows** — Developers create topics, schemas, and connectors through self-service forms. Approvals route to platform team only when needed - **Service Account Lifecycle** — Teams create and manage their own service accounts. Full CRUD with ACL management. No shared credentials - **RBAC by Design** — Operators, developers, and data users get appropriate access. No over-provisioning. Audit trails for compliance - **Role-Based Access** — 50+ granular permissions. Topics, schemas, connectors, consumer groups—each with read, write, create, delete controls - **Guardrails Not Gates** — Policies prevent mistakes without blocking work. Teams move fast within safe boundaries - **Multi-Cluster View** — Manage Confluent Cloud, AWS MSK, and self-managed clusters from one UI. Teams see topics, not infrastructure - **Workspace Health** — Local teams monitor their own streams. Platform team keeps a global view of the entire estate - **Cost Visibility** — Chargeback reporting shows which teams and applications drive Kafka costs. FinOps without spreadsheets - **Faster Onboarding** — New developers create their first topic in under an hour. Workspaces, templates, and docs ready from day one ## How Self-Service Kafka Works for Retail Four steps from bottleneck to enabler. - **Define Workspaces** — Map workspaces to teams, brands, or domains. Link to IAM groups. Teams see their own slice of the Kafka estate - **Create Policies** — Encode patterns for topics, schemas, and connectors. Teams provision within policy guardrails. Minutes instead of weeks - **Build the Catalog** — Topics tagged with owners, documentation, and sample messages. Teams discover existing data before creating duplicates - **Measure Impact** — Track time saved, tickets avoided, and adoption rates. Platform team shifts from ticket processing to capability building ## Key Use Cases - **E-Commerce Teams** — Self-serve topics for checkout, cart, and order events. No waiting for platform team during feature sprints - **Merchandising** — Provision access to pricing and catalog event streams. Build integrations without platform dependencies - **Marketing & Personalization** — Request read access to customer event topics. Build recommendation engines with governed service accounts - **Supply Chain Operations** — Warehouse, inventory, and fulfillment teams get their own workspace for logistics events - **Analytics & Data Teams** — Consume operational topics for reporting. Governed pipelines feed the data warehouse - **Store Systems** — POS and store operations teams provision topics for in-store events. Regional workspaces for store networks For sharing Kafka data with external partners, see [supply chain sharing](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/supply-chain-sharing). For peak season preparation, see [resilience testing](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/peak-season-resilience). ## Read more customer stories - [Swiss Post: Self-Service Kafka](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Bitvavo: Developer Productivity](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) ## Frequently Asked Questions **How do workspaces map to our retail organization?** Workspaces can map to teams, brands, stores, regions, or any organizational unit. They're linked to IAM groups, so membership syncs automatically. **Can developers create any Kafka topic they want?** Teams create topics within policy guardrails. Policies enforce naming conventions, retention limits, partition counts, and replication factors. Non-standard requests go through approval workflows. **How does self-service work with our existing CI/CD?** Configuration lives in Git as declarative YAML. Conduktor syncs with Terraform or your CI/CD pipeline. Changes go through code review before applying. **Do we still need a central Kafka platform team?** Yes, but their role shifts from ticket processing to platform engineering. They define templates, policies, and patterns instead of handling every request. **How do we track platform team ROI?** Measure tickets avoided, time-to-provision, and developer adoption. Most teams see 100+ hours saved monthly once self-service is fully adopted. ## Ready to scale your retail Kafka platform? See how Conduktor enables self-service Kafka with guardrails. Our team can help you design a workspace and policy strategy for your organization. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-retail-and-ecommerce-developer-self-service) Source: https://www.conduktor.io/solutions/industry/retail-and-ecommerce/developer-self-service --- # Peak Season Kafka Resilience Black Friday, holiday rushes, flash sales. Test failover before customers arrive. Chaos engineering validates your DR plan. Automated cluster switching keeps orders flowing when incidents hit. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-retail-and-ecommerce-peak-season-resilience) Trusted by platform teams at ## Why peak season exposes Kafka weaknesses. ### Manual Failover Playbooks When Kafka goes down, teams scramble. Each application has its own failover procedure. Coordination takes hours while orders queue up. ### Untested DR Plans The disaster recovery plan exists on paper. Nobody knows if it works because testing it risks production. ### Peak Traffic Surprises Black Friday traffic is 5-10x normal. Systems that work fine in dev break under load. Connector backlogs, consumer lag, and broker overload hit simultaneously. ## Why Conduktor for Peak Season Resilience - **Chaos Testing** — Inject broker failures, network latency, and leader elections in pre-production. Validate application behavior before peak traffic hits - **Automated Failover** — Conduktor Gateway, a Kafka proxy, handles cluster switching. One API call moves all applications to the backup cluster—no individual reconfigurations - **Multi-Cluster Monitoring** — Track replication lag, consumer offsets, and broker health across primary and DR clusters from one console - **Latency Injection** — Test how applications handle slow brokers. Simulate network degradation between regions before it happens in production - **Message Corruption** — Inject corrupt messages to validate dead-letter queue handling. Ensure bad data doesn't break order processing - **Failover Validation** — Test the full failover sequence: cluster switch, offset translation, consumer restart. Know your RTO before you need it - **Chaos Interceptors** — Eight chaos testing interceptors: broken brokers, latency, leader elections, slow producers, message corruption, invalid schemas, duplicate messages - **Cluster Routing** — Conduktor Gateway handles connection routing during failover. Applications automatically connect to the backup cluster through Conduktor Gateway - **Real-Time Dashboards** — Monitor lag, throughput, and error rates during chaos tests. See exactly when and how systems degrade - **Production Safety** — Chaos testing runs inside Conduktor Gateway—test against production-like traffic without risking production data - **API-Driven Tests** — Trigger chaos tests via API. Integrate with your CI/CD pipeline for continuous validation before peak season - **One-Click Failover** — Switch clusters through API or Console. All applications follow automatically through Conduktor Gateway routing ## How Peak Season Preparation Works From untested DR plan to validated resilience in four steps. - **Baseline Performance** — Run chaos tests against your checkout and order pipelines. Document how applications behave under broker failures and network degradation - **Validate Failover** — Execute full failover to DR cluster. Measure actual RTO. Identify applications that need manual intervention - **Fix Gaps** — Address issues found during testing. Update runbooks. Automate manual steps through Conduktor Gateway policies - **Continuous Testing** — Schedule regular chaos tests. Catch regressions from deployments. Enter peak season with confidence ## Key Use Cases - **Order Management DR** — Test checkout pipeline failover before Black Friday. Validate that orders continue flowing when the primary cluster fails - **E-Commerce Checkout** — Inject latency into payment event streams. Ensure checkout completes even when downstream services slow down - **Inventory Sync** — Test what happens when inventory update consumers fall behind. Validate catch-up behavior and alerting - **Partner Integrations** — Simulate carrier API failures. Ensure order routing continues when shipping partners have outages - **Returns Processing** — Test refund event processing under load. Validate that holiday returns don't overwhelm the system - **Real-Time Analytics** — Inject broker failures during reporting windows. Ensure dashboards recover gracefully For Kafka cost allocation and compliance reporting, see [data protection](https://www.conduktor.io/solutions/use-case/data-protection). For self-service Kafka provisioning, see [developer self-service](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/developer-self-service). ## Read more customer stories - [Bitvavo: Chaos Testing](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [Swiss Post: Platform Resilience](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) ## Frequently Asked Questions **What chaos tests can Conduktor run on Kafka?** Eight interceptor types: simulate broken brokers, inject latency, trigger leader elections, slow producers/consumers, corrupt messages, inject invalid schema IDs, and duplicate messages. Tests run at the Conduktor Gateway layer. **Can we test failover without impacting production?** Yes. Run chaos tests against a production-like environment through Conduktor Gateway. Test traffic patterns and application behavior without risking real orders. **How does automated Kafka failover work?** Conduktor Gateway routes all application traffic. When you switch clusters, all applications automatically connect to the new cluster. No individual app reconfigurations required. **What if our applications don't support automatic failover?** Conduktor Gateway handles connection routing. Most applications need no changes. For stateful applications, we help you design offset translation and restart procedures. **How long does Kafka cluster failover take?** Conduktor Gateway switching is near-instant. Actual RTO depends on consumer restart behavior and offset translation. Chaos testing helps you measure and optimize your specific RTO. ## Validate your DR plan before peak season See how Conduktor helps retail teams test failover, inject chaos, and enter peak season with confidence. Get a demo tailored to your architecture. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-retail-and-ecommerce-peak-season-resilience) Source: https://www.conduktor.io/solutions/industry/retail-and-ecommerce/peak-season-resilience --- # Kafka Platform Modernization for Retail Migrate from VMs to Kubernetes, on-prem to Confluent Cloud, legacy connectors to modern integrations. One console manages hybrid environments during transition. No big-bang cutovers. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-retail-and-ecommerce-platform-modernization) Trusted by platform teams at ## Why retail Kafka modernization stalls. ### Infrastructure Sprawl Hundreds of VMs across data centers. Old Confluent Platform versions that can't upgrade without breaking applications. Technical debt compounds. ### Hybrid Complexity Cloud and on-prem clusters running in parallel. No unified view. Different tools, different credentials, different operational procedures. ### Migration Risk Moving applications to new clusters means changing connection strings, validating schemas, and testing failover. Downtime risk blocks progress. ## Why Conduktor for Kafka Modernization - **Multi-Cluster Visibility** — Manage Confluent Cloud, AWS MSK, Azure Event Hubs, and self-managed clusters from one console. See topics, consumers, and lag across all environments - **Hybrid Monitoring** — Track broker health, consumer groups, and connector status across on-prem and cloud. One dashboard during migration - **Schema Registry Integration** — Integrates with Confluent Schema Registry. Validate schema compatibility before migrating producers - **Connector Management** — Deploy, configure, and monitor Kafka Connect connectors across all clusters. Bridge legacy systems during transition - **Port-Based Routing** — Conduktor Gateway, a Kafka proxy, routes traffic by port. Migrate applications incrementally without connection string changes - **Multi-Environment RBAC** — Unified permissions across clusters. Teams get consistent access whether targeting on-prem or cloud - **Bridge Legacy Systems** — Manage connectors for IBM MQ, databases, and file systems. Bring legacy data into Kafka during modernization - **Cloud Migration Path** — Start with hybrid visibility. Migrate topics incrementally. Validate with chaos testing before cutover - **Topic Catalog** — Searchable inventory with owners, schemas, and labels across all clusters. Find topics regardless of where they live - **Consumer Lag Tracking** — Monitor consumer groups across old and new clusters. Identify applications falling behind during migration - **Audit Trails** — Centralized logs for all admin actions across environments. Compliance visibility during transition - **Zero-Downtime Migration** — Conduktor Gateway handles routing during migration. Applications follow automatically ## How Kafka Modernization Works From fragmented estate to unified platform. - **Inventory & Catalog** — Connect Conduktor to all existing clusters. Build a complete topic catalog with owners, schemas, and consumers - **Unified Operations** — Operate old and new clusters from one console. Consistent tooling reduces context switching - **Incremental Migration** — Move applications one at a time through Conduktor Gateway routing. Validate each migration before proceeding - **Decommission Legacy** — Monitor old cluster activity. Alert when topics are no longer consumed. Retire safely ## Key Use Cases - **VM to Kubernetes** — Migrate from PCF/Tanzu to AKS, EKS, or GKE. Conduktor manages Kafka regardless of deployment model - **On-Prem to Cloud** — Move from self-managed Confluent Platform to Confluent Cloud. Hybrid visibility during transition - **Connector Modernization** — Replace deprecated connectors with supported alternatives. Monitor both during migration - **Version Upgrades** — Validate schema compatibility and consumer behavior before upgrading Kafka versions - **Multi-Region Expansion** — Add clusters in new regions. Unified view across all environments - **Cost Optimization** — Identify unused topics and over-provisioned resources. Optimize before and after migration For peak season preparation, see [resilience testing](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/peak-season-resilience). For team self-service, see [developer self-service](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/developer-self-service). For sharing data with partners, see [supply chain sharing](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/supply-chain-sharing). ## Read more customer stories - [Swiss Post: Platform Modernization](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Bitvavo: Cloud Migration](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) ## Frequently Asked Questions **Can we migrate to cloud incrementally?** Yes. Connect Conduktor to both on-prem and cloud clusters. Migrate topics and applications one at a time. No big-bang cutover required. **How does Conduktor handle different Kafka versions?** Conduktor supports Apache Kafka, Confluent Platform, Confluent Cloud, AWS MSK, Azure Event Hubs, and other distributions. Unified interface regardless of version. **Does migration require application changes?** Conduktor Gateway handles routing. Most applications need no code changes—just updated connection configuration. Some stateful applications may need offset translation. **How do we track migration progress?** Topic catalog shows where topics live. Consumer group monitoring shows active usage. Audit logs track configuration changes across environments. **What about deprecated connectors?** Deploy new connectors alongside old ones. Monitor both through Conduktor. Validate data flow before decommissioning legacy connectors. ## Ready to modernize your Kafka platform? See how retail platform teams use Conduktor to manage hybrid Kafka environments during modernization. Get a demo tailored to your architecture. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-retail-and-ecommerce-platform-modernization) Source: https://www.conduktor.io/solutions/industry/retail-and-ecommerce/platform-modernization --- # Real-Time Supply Chain Data Sharing and Visibility Real-time supply chain data sharing with suppliers and carriers without exposing your Kafka infrastructure. Virtual clusters isolate partners. Field-level encryption protects sensitive data. Usage metering tracks consumption. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-retail-and-ecommerce-supply-chain-sharing) Trusted by platform teams at ## Why per-partner clusters don't scale for retail. ### Infrastructure Sprawl New Kafka clusters for each supplier, carrier, and marketplace inflate cost and operational load. Topics duplicated across environments. ### Slow Partner Onboarding Weeks to provision a new supplier connection. Network changes, security reviews, and custom integration work block time-sensitive partnerships. ### Data Exposure Risk Suppliers see more data than they need. No field-level controls. Audit trails scattered across systems. ## Why Conduktor for Supply Chain Sharing - **Virtual Clusters** — Create isolated environments for each partner on existing Kafka infrastructure. No new clusters, no VPC peering, no infrastructure sprawl - **Topic Filtering** — Share specific topics with each partner. They see only the data relevant to their integration—not your entire event stream - **Field-Level Encryption** — Encrypt or mask sensitive fields per partner contract. One topic serves multiple views with different data protection levels - **Usage Metering** — Track bytes produced and consumed per partner. Export via API for billing, chargeback, or capacity planning - **Rate Limiting** — Set throughput limits per partner. Premium partners get higher quotas. Prevent any single partner from overwhelming the cluster - **Audit Trail** — Centralized logs for all partner activity across clusters. Export to SIEM or compliance systems - **Conduktor Gateway Isolation** — Partners connect through Conduktor Gateway, a Kafka proxy. Kafka brokers stay inside your network, never exposed to external parties - **Crypto-Shredding** — Delete encryption keys to revoke partner access instantly. Meet GDPR right-to-erasure without data deletion - **Multi-Region Support** — Partner zones can bind to regional clusters. Route EU partners to EU clusters for data residency compliance - **Real-Time Visibility** — Monitor partner consumption, lag, and errors from Console. Identify integration issues before partners report them - **Self-Service Onboarding** — Partners request access via API or Console. Approval workflows route to platform team. Policies auto-enforce on approval - **No Replication** — Share topics without duplicating data. One source of truth, multiple controlled views ## How Partner Data Sharing Works From signed deal to live data feed without new infrastructure. - **Define Partner Zone** — Create a virtual cluster with topics, fields, and rate limits. Attach to IdP group or external identity provider - **Set Protection Policies** — Configure field-level encryption per partner contract. Mask pricing, PII, or competitive data as needed - **Enable Metering** — Track bytes and messages per partner. Use for cost attribution, chargeback, or commercial agreement reporting - **Onboard Partner** — Partner connects through Conduktor Gateway with their credentials. Policies enforce automatically. No cluster exposure ## Key Use Cases - **Supplier Integration** — Share demand forecasts and purchase orders with suppliers. Filter to show only their products. Encrypt pricing data - **Carrier Data Feeds** — Expose shipment and tracking events to logistics partners. Different tiers of access for different carrier classes - **Marketplace Integration** — Connect order and inventory data with Amazon, Shopify, and other marketplaces. Per-marketplace topic access - **3PL Warehousing** — Share inventory and order events with third-party logistics providers. Real-time visibility into stock movements - **Returns Processing** — Let return partners consume return authorization events. Track refund status back to your systems - **Loyalty Partners** — Share customer event streams with loyalty program partners. Encrypt PII, expose only program-relevant data For Kafka cost allocation and compliance, see [data protection](https://www.conduktor.io/solutions/use-case/data-protection). For internal team self-service, see [developer self-service](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/developer-self-service). [See how real-time data sharing works](https://www.conduktor.io/solutions/use-case/data-sharing), or explore the [data streaming platform](https://www.conduktor.io/product/data-streaming-platform) it runs on. ## Read more customer stories - [Swiss Post: Partner Data Sharing](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Bitvavo: Secure External Access](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) ## Frequently Asked Questions **Do partners need Kafka expertise to consume data?** No. Partners consume through standard Kafka clients. They see a simple, governed interface. Your infrastructure complexity stays hidden. **Can different partners see different fields from the same topic?** Yes. Field-level policies let you expose, mask, or encrypt specific fields per partner. One topic, multiple contractual views. **How does partner usage tracking work?** Usage metering tracks bytes, messages, and consumers per partner zone. Use for internal cost attribution, chargeback, or commercial agreement reporting. **How fast can we onboard a new supply chain partner?** A new partner zone can be configured in minutes via API or Console. No cluster provisioning, no network changes, no weeks of integration work. **How do we handle partners in different regions?** Partner zones can bind to regional clusters. Configure zones to route EU, US, or APAC partners to appropriate clusters for data residency. ## Launch your first partner zone See how retail platform teams use Conduktor to share Kafka data with suppliers and carriers without infrastructure sprawl. Get a personalized demo. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-retail-and-ecommerce-supply-chain-sharing) Source: https://www.conduktor.io/solutions/industry/retail-and-ecommerce/supply-chain-sharing --- # Kafka for Retail & E-Commerce Conduktor for Retail & E-Commerce **Real-time operations across every channel.** Manage order flows, sync inventory, and share data with partners—with governance, security, and peak-season resilience built in. **Order management & checkout** - Stream orders, payments, and fulfillment events across your commerce stack. Monitor critical pipelines in real-time. Automated failover keeps checkout flowing during incidents. **Inventory & supply chain** - Track stock levels across stores, warehouses, and DCs in real-time. Share demand signals with suppliers through governed Kafka topics. **Partner integrations** - Connect marketplaces, carriers, and suppliers to your event streams without exposing Kafka infrastructure. Virtual clusters isolate each partner with their own policies. **Self-service for teams** - Merchandising, marketing, and analytics teams provision Kafka access in minutes. No waiting for platform tickets. ### Get in touch Why Retail Teams Choose Conduktor Peak Season Resilience Black Friday, holiday rushes, flash sales. Chaos testing lets you validate failover before traffic spikes. Automated cluster switching keeps orders flowing when incidents hit. Platform Team Productivity Retailers often have hundreds of developers on Kafka. Self-service access and governed workspaces let teams move fast. Platform engineers stop processing tickets and start building capabilities. PCI-DSS & PII Protection Encrypt cardholder and customer data at the Conduktor Gateway layer—no code changes. Conduktor Gateway is a Kafka proxy that enforces encryption inline. Field-level encryption with AWS KMS, Azure Key Vault, or HashiCorp Vault. Full audit trails for compliance. Deep Dive by Use Case - **Peak Season Resilience** — Chaos testing, automated failover, and DR planning for revenue-critical Kafka. [Learn more](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/peak-season-resilience) - **Developer Self-Service** — Workspaces, policies, and topic catalogs for hundreds of development teams. [Learn more](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/developer-self-service) - **Supply Chain Sharing** — Share inventory and order data with suppliers and carriers securely. [Learn more](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/supply-chain-sharing) - **Platform Modernization** — Migrate from VMs to Kubernetes, on-prem to cloud, legacy to modern. [Learn more](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/platform-modernization) Where Conduktor Helps in Retail Conduktor connects to your existing Kafka infrastructure—Confluent Cloud, AWS MSK, Azure Event Hubs, or self-managed—and adds the governance, security, and resilience capabilities these use cases demand. Order management Inventory visibility Peak season readiness Partner integrations Customer 360 Returns processing PCI-DSS compliance Cost allocation Personalization feeds Data quality gates Hybrid infrastructure Team productivity ## Frequently Asked Questions **How does Conduktor help with Black Friday and peak season?** Conduktor Gateway enables chaos testing to validate your failover procedures before peak traffic hits. Test broker failures, latency injection, and cluster switching in pre-production. When incidents occur, automated cluster switching routes all applications to your backup cluster through a single API call. **Can we share Kafka data with suppliers without exposing our infrastructure?** Yes. Conduktor Gateway creates virtual clusters that give each partner isolated access to specific topics. Partners connect through Gateway—your Kafka brokers are never exposed. Field-level encryption protects sensitive data per partner contract. **How does self-service work for retail development teams?** Teams get application workspaces linked to your identity provider. Within their workspace, developers can create topics, schemas, and service accounts without tickets. Policies enforce naming conventions, retention limits, and partition standards automatically. **Does Conduktor work with our existing Kafka infrastructure?** Conduktor connects to Confluent Cloud, AWS MSK, Azure Event Hubs, and self-managed Apache Kafka clusters. You can manage multiple clusters from one console, including hybrid on-prem and cloud environments. **How do we handle PCI-DSS compliance for payment data in Kafka?** Conduktor Gateway encrypts cardholder data at the wire level—before it reaches Kafka. Field-level encryption integrates with AWS KMS, Azure Key Vault, HashiCorp Vault, and other key management systems. Full audit trails track all access for compliance reporting. Source: https://www.conduktor.io/solutions/industry/retail-and-ecommerce --- # Event Backbone Modernization: ESB & MQ to Kafka Replace legacy ESB, MQ, and middleware with a Kafka-based event backbone. Connect old systems into Kafka, manage hybrid on-prem and cloud clusters, and reduce custom integration code. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-transport-and-logistics-event-backbone-modernization) Trusted by platform teams at ## Why legacy integration patterns hold you back. ### Point-to-Point Sprawl Complex mesh of flows on ESB and MQ. Batch jobs and file drops where teams need real-time feeds. ### Parallel Stacks Kafka, legacy bus, and cloud tools growing in parallel with no clear backbone. No shared inventory of events, schemas, and consumers. ### Fragmented Operations Fuzzy ownership between app, integration, and platform teams. Fragmented monitoring and tracing across platforms and regions. ## Why Conduktor for Event Modernization - **Connector Lifecycle** — Deploy, configure, and monitor Kafka Connect connectors from one console. Manage MQ, EMS, CDC, and database connectors - **Hybrid Visibility** — One control plane for on-prem, Confluent Cloud, AWS MSK, Aiven, and self-managed clusters. See all environments, not just one - **Event Catalog** — Searchable inventory with owners, schemas, and labels. Teams find existing topics before creating duplicates - **Connector Monitoring** — Track connector status, lag, and errors across all Kafka Connect clusters. Alerts on failures - **Connector Auto-Restart** — Failed connectors automatically restart with configurable policies. Audit logs track every restart event - **Enterprise IAM** — RBAC via OIDC or LDAP. Map Kafka ACLs to Console groups. Encryption in transit and at rest - **Bridge Legacy Systems** — Manage Kafka Connect connectors for IBM MQ, Tibco EMS, RabbitMQ, and CDC sources. Deploy and monitor from one console - **Multi-Cloud Ready** — Confluent Cloud, AWS MSK, Aiven, and self-managed. Manage from one console - **Schema Registry Integration** — Integrates with Confluent Schema Registry and AWS Glue. Enforce Avro, Protobuf, or JSON schemas. Block breaking changes - **Consumer Monitoring** — Track consumer groups, lag, and offsets across all clusters. Identify bottlenecks - **Dead-Letter Queues** — Failed messages route to DLQ topics for inspection and replay. No data loss during schema validation - **Incremental Migration** — Move flows one at a time. Legacy and Kafka coexist during transition ## How Conduktor Enables Event Modernization From discovery to decommission, one platform guides the migration. - **Discover & Map** — Connect to existing Kafka Connect clusters. Document schemas and topic ownership before migration starts - **Bridge & Validate** — Deploy connectors from Conduktor Console. Monitor throughput and data quality during dual-write phase - **Catalog & Govern** — Register migrated topics with schemas and owners. Enforce access policies from day one - **Cutover & Retire** — Monitor consumer groups on new topics. Alert on lag before decommissioning old paths ## Key Use Cases - **Airline PSS Integration** — Connect booking, ticketing, and DCS systems to a central event hub. Replace batch extracts with real-time streams - **Retail POS Modernization** — Stream store POS and TLOG data to Kafka. Unify with e-commerce and supply chain events - **Manufacturing ERP-to-MES** — Bridge ERP orders and schedules to MES systems through Kafka. Real-time production visibility - **Grid Operations** — Connect SCADA, forecasting, and market systems. Replace point-to-point with event backbone - **Logistics TMS/WMS** — Align transportation and warehouse management with real-time shipment events - **Mainframe Offload** — Stream mainframe events to Kafka through CDC. Enable cloud-native consumers Once migrated, enable [self-service Kafka provisioning](https://www.conduktor.io/solutions/industry/transport-and-logistics/self-service-platform) for your teams. See how [Conduktor Gateway](https://www.conduktor.io/gateway) enforces data quality and schema validation. ## Read more customer stories - [Swiss Post: Event Platform](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Financial Services: ESB Migration](https://www.conduktor.io/customer-stories/securing-kafka-banking-financial-services) ## Frequently Asked Questions **Can we migrate from MQ to Kafka incrementally?** Yes. Bridge legacy flows into Kafka one at a time. Consumers can read from both during transition. No big-bang cutover required. **What legacy systems can Conduktor connect to Kafka?** Conduktor manages Kafka Connect connectors. Connectors available for IBM MQ, Tibco EMS, RabbitMQ, ActiveMQ, mainframe CDC, databases, and file systems. **How do we handle schema differences during migration?** Schema Registry enforces contracts. Transform legacy formats to canonical schemas at the connector level. **Does Kafka support ordering and exactly-once delivery?** Kafka preserves ordering per partition. Exactly-once semantics available with transactional producers and consumers. **Do we need to rewrite consumers when migrating to Kafka?** Not immediately. Bridge consumers can read from Kafka while legacy apps still use MQ. Migrate consumers at your own pace. ## See Event Backbone Modernization in Action Get a personalized demo with your architecture. We'll map your current state and show how Conduktor fits your migration path. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-transport-and-logistics-event-backbone-modernization) Source: https://www.conduktor.io/solutions/industry/transport-and-logistics/event-backbone-modernization --- # Real-Time Data Sharing Across Partner and Dealer Networks Real-time data sharing with partners and dealers without spinning up clusters. Secure zones with policy-based access, field-level masking, and usage tracking on your existing Kafka estate. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-transport-and-logistics-partner-data-sharing) Trusted by platform teams at ## Why per-partner clusters don't scale. ### Infrastructure Sprawl New Kafka clusters per tenant or partner inflate cost and operational load. Topics and schemas get duplicated across clusters. ### Security Fragmentation Inconsistent security and compliance rules between environments. Manual setup and rotation of certificates, keys, and identities for each external party. ### Slow Time-to-Revenue Long onboarding timelines involving many teams before a partner can consume or publish events. No clear way to package feeds as commercial data products. ## Why Conduktor for Kafka Partner Data Sharing - **Logical Partner Zones** — Create tenant zones on existing clusters. Each partner gets controlled access without dedicated infrastructure or VPC peering - **Conduktor Gateway** — Partners connect through a protocol-compatible proxy. Kafka brokers stay inside your network, never exposed - **Enterprise IAM** — Map partners to your IdP via OIDC or LDAP. Use groups and attributes for policy binding - **Field-Level Masking** — Encrypt, redact, or filter fields per partner contract. One topic serves multiple contractual views - **Usage Tracking** — Measure bytes produced and consumed per zone. Export via API for billing integration - **Audit Trail** — Centralized audit logs for all partner activity across clusters. Export to SIEM or compliance systems - **Policy-Based Access** — Control topics, fields, filters, and rate limits per partner zone. Policies enforce automatically - **Crypto-Shredding** — Delete encryption keys to revoke partner access instantly without data deletion. Meet GDPR right-to-erasure and retention requirements - **Multi-Cluster Zones** — Partner zones span Confluent Cloud, AWS MSK, Azure Event Hubs, and self-managed clusters. One consistent experience - **Rate Limiting** — Set throughput limits per partner zone. Premium partners get higher quotas - **Self-Service Onboarding** — Partners request access via API or Conduktor Console. Approval workflows route to platform team. Policies auto-enforce on approval - **No Replication** — Share topics without duplicating data. One source of truth, multiple controlled views ## How Partner Data Sharing Works From signed deal to live data feed without new infrastructure. - **Define Partner Zone** — Create a logical zone with topics, fields, filters, and rate limits. Attach to IdP group or external identity - **Set Protection Policies** — Configure field-level encryption and masking per partner contract. Bind zones to regional clusters for data residency - **Enable Metering** — Track bytes produced and consumed per zone. Export to billing systems for chargeback - **Onboard Partner** — Partner connects through Conduktor Gateway. Policies enforce automatically. Kafka brokers never exposed ## Key Use Cases - **OEM Dealer Networks** — Share vehicle telemetry and maintenance events with dealers. Different tiers of access and SLAs per dealer class - **Supplier Integration** — Expose production streams to design partners and suppliers. Keep sensitive fields encrypted per contract - **Logistics Partner Feeds** — Open warehouse, order, and shipment events to carriers and 3PLs for planning and capacity optimization - **Commercial Data Products** — Package Kafka feeds as products for partners with clear tiers, usage limits, and revenue sharing - **Aftermarket Services** — Share equipment telemetry with service partners for predictive maintenance and parts optimization - **Regional Rollouts** — Onboard local partners into standard tenant zones while central platform keeps control on policies and audit For Kafka cost allocation and compliance across your estate, see [Kafka platform governance and FinOps](https://www.conduktor.io/solutions/industry/transport-and-logistics/platform-governance). [See how real-time data sharing works](https://www.conduktor.io/solutions/use-case/data-sharing), or explore the [data streaming platform](https://www.conduktor.io/product/data-streaming-platform) it runs on. ## Read more customer stories - [Swiss Post: Governed Kafka](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Bitvavo: Secure Data Sharing](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) ## Frequently Asked Questions **Do external partners need Kafka knowledge to consume data?** No. Partners consume through standard Kafka clients or REST APIs. They see a simple, governed interface. Your complexity stays hidden. **Can different partners see different fields from the same topic?** Yes. Field-level policies let you expose, mask, or encrypt specific fields per partner. One topic, multiple views. **How does Kafka usage tracking and chargeback work?** Usage metering tracks bytes, messages, and consumers per partner zone. Use for internal cost attribution, chargeback, or commercial agreement reporting. **How do I share Kafka data with partners in different regions?** Partner zones can be bound to specific regional clusters. Configure zones to route to EU, US, or APAC clusters based on your data residency requirements. **How fast can we onboard a new Kafka data partner?** A new partner zone can be configured in minutes via API or Console. No cluster provisioning, no network changes. ## Launch your first partner zone See how platform teams use Conduktor to share Kafka data with partners without the infrastructure sprawl. Get a personalized demo with your architecture. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-transport-and-logistics-partner-data-sharing) Source: https://www.conduktor.io/solutions/industry/transport-and-logistics/partner-data-sharing --- # Platform Governance & FinOps for Critical Kafka Cost allocation, chargeback, and compliance without spreadsheets. FinOps visibility, encryption coverage, and data quality enforcement built for platform and finance teams. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-transport-and-logistics-platform-governance) Trusted by platform teams at ## Why Kafka governance becomes urgent at scale. ### Cost Blindness Rising Kafka and vendor costs with no FinOps view per team, product, or tenant. Nobody can say which team drives which share of the cost. ### Security Gaps PII and sensitive data spread across topics. Encryption and masking policies applied inconsistently. ### Data Quality Drift Data quality checks happen in the DWH, far from original streams. Bad data discovered too late to fix cheaply. ## Why Conduktor for Platform Governance - **Usage Metering** — Per-topic, per-tenant, per-service-account tracking. Attribute costs to cost centers, products, or business units - **FinOps Integration** — Export usage data via API. Build chargeback models and cost allocation reports - **PII Detection** — Tag sensitive fields in schemas with PII and GDPR markers. Apply encryption policies based on tags - **Field-Level Encryption** — Encrypt, mask, or tokenize at write time via Conduktor Gateway. Integrates with Vault, AWS KMS, Azure Key Vault, GCP KMS - **Stream Quality Gates** — Enforce schema compliance and CEL-based validation rules at produce time. Block bad data before it propagates - **Audit-Ready Logs** — Immutable records of access, configuration, and policy changes. Export to SIEM or archive for compliance - **Ownership Model** — Topics and service accounts tied to owners. Clear accountability per business unit - **Quota Management** — Quotas and limits per tenant, team, or environment. Prevent runaway workloads - **Topic Lifecycle** — Track topic activity and identify inactive topics. See last activity timestamps per topic - **Encryption Coverage** — Track which fields are encrypted via schema metadata and Conduktor Gateway policy enforcement - **Schema Validation** — Contracts validated at deployment and at write. Breaking changes blocked - **Stream Quality** — Schema validation and custom CEL rules enforced per topic. Non-compliant messages blocked or routed to DLQ ## How Platform Governance Works From cost mystery to transparent utility with measurable results. - **Cost Transparency** — Know which team, product, or cost center drives each dollar of Kafka spend. Answer finance questions in minutes, not weeks - **Resource Control** — Prevent runaway workloads before they impact production. Quotas and alerts stop cost surprises before they hit the bill - **Data Protection** — PII encrypted or masked before it lands in Kafka via Conduktor Gateway policy enforcement. No reliance on application-level compliance - **Quality Assurance** — Validate schema compliance and custom rules at produce time, before bad data reaches analytics or ML pipelines ## Key Use Cases - **FinOps & Chargeback** — Build cost views per plant, team, and product. Allocate Confluent Cloud and infrastructure costs to business units - **Sensitive Data Protection** — Encrypt PII in HR, customer, and order events. Mask trade and price data for non-privileged consumers - **Topic Catalog Cleanup** — Identify unused topics and connectors. Fix naming conventions and assign ownership - **Compliance Reporting** — Pull ready-made reports on access, retention, and encryption for audits. No more spreadsheet scrambles - **Stream Data Quality** — Validate schemas and custom rules on telemetry streams. Block malformed messages on order and supply chain feeds - **Cost Optimization** — Identify over-sized topics, unnecessary replication, and idle connectors. Reclaim wasted spend Need to [share Kafka data with external partners](https://www.conduktor.io/solutions/industry/transport-and-logistics/partner-data-sharing)? Governance policies extend to partner zones. See how [Conduktor Gateway](https://www.conduktor.io/gateway) enforces encryption and data quality at the wire level. ## Read more customer stories - [Bitvavo: Compliance & Governance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [Swiss Post: Platform Governance](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) ## Frequently Asked Questions **How do I implement Kafka cost allocation?** Metering tracks usage per topic, partition, and service account. You label these with cost centers, teams, or programs. Export to your FinOps tools or billing system. **How does Conduktor handle PII in Kafka?** Tag sensitive fields in schemas with PII and GDPR markers. Conduktor applies encryption and masking policies based on these tags automatically. **What KMS systems does Conduktor support for Kafka encryption?** HashiCorp Vault, AWS KMS, Azure Key Vault, Google Cloud KMS, and custom integrations. Keys managed centrally with automated rotation. **How do Kafka data quality checks work?** Define validation rules per topic using CEL expressions or JSON Schema. Checks run in the Conduktor Gateway layer at produce time. Non-compliant messages can be blocked or routed to a dead-letter topic. **Does Kafka governance work across multiple vendors?** Yes. Conduktor provides one governance layer across Confluent Cloud, AWS MSK, Azure Event Hubs, and self-managed clusters. ## Ready to govern your Kafka platform? See how Conduktor delivers FinOps, security, and data quality for critical Kafka infrastructure. Our team can help you build a governance strategy. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-transport-and-logistics-platform-governance) Source: https://www.conduktor.io/solutions/industry/transport-and-logistics/platform-governance --- # Self-Service Kafka for Domain Teams Ship data products without tickets. Workspaces give teams their own slice of Kafka. Templates enforce standards. Policies prevent drift. Platform teams scale without becoming a bottleneck. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-transport-and-logistics-self-service-platform) Trusted by platform teams at ## Why ticket-driven Kafka doesn't scale. ### Central Bottleneck A small central team handles every topic, ACL, connector, and schema change. Every new data flow goes through tickets. ### Shadow IT & Drift Teams bypass the platform with direct admin access and custom tools. Standards drift, topic catalogs lose structure. ### No Domain Ownership Kafka objects don't map to sites, assets, products, or programs. Nobody owns the data products on Kafka. ## Why Conduktor for Self-Service - **Team Workspaces** — Each domain gets a workspace linked to IAM groups. Teams see and manage their own resources - **Application Policies** — Policies encode defaults for naming, retention, partitions, and replication. Consistency without tickets - **Guided Workflows** — Teams create topics, schemas, and connectors through workflows. Approvals where needed, automation everywhere else - **Topic Catalog** — Search by product, asset, site, or program. See ownership, labels, and documentation - **Policy Engine** — Naming conventions, retention, and partitions enforced automatically via CEL rules. No drift - **GitOps Integration** — Works with Terraform or your existing CI/CD. Declarative config with code reviews and rollbacks - **Role-Based Access** — Operators, developers, and data users get appropriate access. No over-provisioning - **Guardrails Not Gates** — Policies prevent mistakes without blocking work. Teams move fast within safe boundaries - **Multi-Cluster View** — Manage Confluent Cloud, AWS MSK, Azure Event Hubs, and self-managed clusters from one UI. Teams see topics, not infrastructure - **Domain Health** — Local teams see the health of their own streams. Central team keeps a global view - **Service Account Lifecycle** — Teams manage their own service accounts with full CRUD and ACL management - **Faster Onboarding** — New team members create their first topic in under an hour. Workspaces, templates, and docs ready from day one ## How Self-Service Kafka Provisioning Works Four steps from gatekeeper to enabler. - **Define Workspaces** — Map workspaces to IAM groups and business domains. Teams see their own slice of the Kafka estate - **Create Policies** — Encode patterns for topics, schemas, and connectors. Teams provision within policy guardrails. Minutes instead of weeks - **Enable Workflows** — Teams request resources through guided forms. Approvals route to the right people - **Build the Catalog** — Topics and schemas tagged with owners, docs, and samples. Teams discover existing data before creating duplicates ## Key Use Cases - **Product Teams** — Self-serve topics, schemas, and access for new services. No waiting for platform team - **Factory & Plant Teams** — Connect MES, SCADA, and production line systems to Kafka using approved patterns - **Data Teams** — Request read access to operational topics. Consume through governed service accounts - **Supply Chain Operations** — Warehouse, order, and fulfillment teams get their own workspace for logistics events - **Platform Teams** — Expose standard paths to feed Kafka data into lakehouse or warehouse. One pattern, many teams - **Security & Compliance** — Review access and policy compliance per workspace. Audit trails ready Need to [migrate from ESB and MQ to Kafka](https://www.conduktor.io/solutions/industry/transport-and-logistics/event-backbone-modernization) first? Or [share Kafka data with external partners](https://www.conduktor.io/solutions/industry/transport-and-logistics/partner-data-sharing)? See how [Conduktor Console](https://www.conduktor.io/console) enables self-service. ## Read more customer stories - [Swiss Post: Governed Kafka](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Bitvavo: Developer Self-Service](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) ## Frequently Asked Questions **How do Kafka workspaces map to our org structure?** Workspaces can map to teams, products, plants, regions, or any organizational unit. They're linked to IAM groups, so membership syncs automatically. **Can teams create any Kafka topic they want?** Teams create topics within policy guardrails. Policies enforce naming, retention, partitions, and other standards. Custom requests go through approval workflows. **What if a team needs a Kafka resource outside the templates?** Exception workflows route requests to platform or security teams. Decisions are tracked and can inform future template updates. **How does Kafka self-service work with GitOps?** Kafka configuration lives in Git as declarative YAML. Conduktor syncs with Terraform or your CI/CD pipeline. Changes go through code review before applying. **Do we still need a central Kafka platform team?** Yes, but their role shifts from ticket processing to platform engineering. They define templates, policies, and patterns instead of handling every request. ## Ready to scale your Kafka platform? See how Conduktor enables self-service Kafka with guardrails. Our team can help you design a workspace and template strategy. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-industry-transport-and-logistics-self-service-platform) Source: https://www.conduktor.io/solutions/industry/transport-and-logistics/self-service-platform --- # Kafka for Transport & Logistics | Real-Time Fleet Data Conduktor for Transport & Logistics **Real-time visibility across your network.** Optimize fleet operations, track shipments, and share data with partners—all with enterprise security and governance. **Fleet & cold chain data** - Stream GPS coordinates, temperature sensors, and dispatch events from thousands of vehicles. Conduktor Gateway, a Kafka proxy, validates, routes, and encrypts telemetry at wire speed. **Supply chain visibility** - Track every shipment, pallet, and vehicle across carriers in a unified event stream. End-to-end visibility from warehouse to last-mile delivery. **Cross-entity data sharing** - Share parcel and tracking data between countries, divisions, and external partners—without replicating infrastructure. Filter events so each entity sees only the shipments relevant to them. **Self-service for operations** - New team members get Kafka access in minutes instead of weeks. Logistics teams provision topics and connectors without waiting on a central platform team. ### Get in touch Why Logistics Teams Choose Conduktor Integrate Partners in Days, Not Quarters New carrier? New country entity? Provision secure access to shipment events through self-service. No custom integrations or Flink jobs to filter data—Conduktor Gateway applies server-side filtering automatically. ELD & Compliance Ready Stream driver hours, rest periods, and vehicle inspections for ELD/HOS compliance. Automated audit trails and retention policies keep you regulator-ready. IoT Validation at Ingestion Validate GPS coordinates, temperature readings, and telemetry data through Conduktor Gateway. Block malformed sensor events before they corrupt ETA predictions and tracking systems. Deep Dive by Use Case - **Partner Data Sharing** — Expose Kafka data to partners with secure zones, metering, and revenue sharing. [Learn more](https://www.conduktor.io/solutions/industry/transport-and-logistics/partner-data-sharing) - **Self-Service Platform** — Domain workspaces, templates, and policies for engineering and operations teams. [Learn more](https://www.conduktor.io/solutions/industry/transport-and-logistics/self-service-platform) - **Event Modernization** — Replace ESB and MQ with a Kafka-based event backbone. [Learn more](https://www.conduktor.io/solutions/industry/transport-and-logistics/event-backbone-modernization) - **Platform Governance** — FinOps, encryption coverage, and data quality for critical Kafka infrastructure. [Learn more](https://www.conduktor.io/solutions/industry/transport-and-logistics/platform-governance) Where Conduktor Helps in Logistics Conduktor connects to your existing Kafka infrastructure, whether Confluent, AWS MSK, or self-managed, and adds the governance, security, and self-service capabilities these use cases demand. Fleet tracking Cold chain monitoring ELD/HOS compliance Shipment visibility ETA prediction Proof of delivery Last-mile tracking Carrier integration Cross-country sharing Exception alerts Customs clearance Warehouse events Carbon tracking Source: https://www.conduktor.io/solutions/industry/transport-and-logistics --- # Self-Service Kafka with Governance Controls Empower teams to manage their own data and applications with secure self-service controls, while ensuring compliance with centralized data policies and governance. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-accelerate-innovation-with-self-service-kafka-controls) Trusted by ## Why centralized Kafka management doesn't scale. ### Bottlenecks & Slow Approvals Development teams wait days or weeks for simple changes. Topic creation, ACL updates, and schema modifications all require tickets and manual approval. ### Compliance Risks Manual processes introduce security and governance gaps. Without automation, policies are inconsistently applied across teams and clusters. ### Increased Operational Costs Heavy reliance on platform teams strains resources. Engineers spend more time on tickets than innovation. ## Why Conduktor for Self-Service - **Enable Developer Autonomy** — Teams provision and manage Kafka resources independently within defined guardrails - **Scale Confidently** — Proactive monitoring and enforcement of best practices across all clusters - **Enforce Security & Governance** — Automated policies ensure compliance at every step without manual intervention - **Align Tech, Teams, & Processes** — Balance security with agility through configurable approval workflows - **Integrate Seamlessly** — GitOps, APIs, and RBAC support for smooth integration with existing workflows - **Reduce Onboarding Time** — New teams go from zero to productive in hours instead of weeks - **Policy Controls** — Set guardrails to keep topics consistent and cost-efficient. Enforce naming conventions, quotas, and standards automatically. - **Access Request Workflow** — Enable secure self-service with automated approvals. Teams request access, policies apply instantly, platform leaders stay in control. - **Application Catalog** — Platform team declares applications and rights, designates ownership. Application teams manage resources without involving platform. - **Schema Governance** — Enforce schema standards at the source. Prevent breaking changes before they reach production. - **Full Visibility** — Track usage, monitor compliance, and audit all changes from a single dashboard. - **GitOps Integration** — Manage Kafka resources as code. Version control, review, and deploy through your existing CI/CD pipeline. ## How Self-Service Works A simple four-step process to enable governed autonomy. - **Define Guardrails** — Central platform team sets security, governance, and resource policies that apply automatically - **Assign Application Owners** — Platform team declares applications and their rights, designates ownership to responsible teams - **Enable Self-Service** — Application teams provision and manage Kafka resources, grant permissions to collaborators—without involving platform team - **Monitor & Optimize** — Track usage patterns and adjust workflows dynamically based on real-time insights ## Read more customer stories - [FlixBus: Self-Service for 50+ Teams](https://www.conduktor.io/customer-stories/flix) - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Virgin Australia: 300 Hours/Month Saved](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) ## Frequently Asked Questions **What is self-service Kafka?** Self-service Kafka allows development teams to provision topics, manage ACLs, and configure resources without waiting for platform team tickets. Conduktor provides guardrails so teams move fast while staying compliant. **How do I prevent teams from creating inconsistent resources?** Conduktor's policy controls enforce naming conventions, partition limits, retention settings, and other standards automatically. Teams can only create resources that comply with your defined policies. **Can I still require approvals for sensitive operations?** Yes. Conduktor supports configurable approval workflows. You can require manual approval for production changes while allowing self-service for development environments. **Does this replace our existing Kafka management?** Conduktor adds a governance layer on top of your existing Kafka infrastructure. It works with any Kafka distribution—Confluent, Amazon MSK, or self-managed. **How does this integrate with GitOps?** Conduktor supports managing Kafka resources as code through APIs. Define topics, ACLs, and schemas in Git, and deploy through your CI/CD pipeline with policy validation at every step. ## Ready for self-service Kafka? See how Conduktor enables developer autonomy without sacrificing governance. Our team can help you design the right self-service strategy for your organization. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-accelerate-innovation-with-self-service-kafka-controls) Source: https://www.conduktor.io/solutions/use-case/accelerate-innovation-with-self-service-kafka-controls --- # Data Protection *Unblock Revenue. Protect Cash Flow.* *Prove Compliance.* Conduktor protects data at the wire level and governs access control centrally, so security is a feature, not a blocker. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-data-protection) Trusted by ## Kafka Scales. Security Doesn't Keep Up. Data flows through Kafka in plaintext. Access controls stop at the topic level. Teams build their own encryption. And when an auditor asks who accessed what, nobody has a clear answer. - **Revenue stuck on security approvals** — There's no centralized way to prove data is encrypted or access is governed. InfoSec ends up manually reviewing every new application, delaying data initiatives by months. - **Margins shrink from duplicated work** — Each team builds its own encryption and manages its own access controls. Over time, implementations drift, fields get missed, and a single gap can escalate into an organization-wide incident. - **Cash flow at risk from fines** — GDPR fines reach 4% of global revenue and HIPAA violations run up to $2M per category. Meanwhile, Kafka gives you topic-level ACLs and no audit trail of who accessed what data. ## Why Conduktor for data protection - **Encryption at every level** — Full-payload, field-level, and header-level encryption with eight algorithms including AES-GCM and ChaCha20-Poly1305. One policy, every application, zero code changes. Plug in your own KMS. [Learn more](https://www.conduktor.io/gateway#encrypt-consistently-across-apps) - **Crypto-shredding for GDPR** — Encrypt each record with a per-subject key. When a user invokes the right to erasure, destroy the key and every record becomes permanently unreadable. No topic deletion, no reprocessing. [Learn more](https://docs.conduktor.io/guide/reference/data-security#encryption) - **Dynamic data masking** — Mask fields per consumer at the wire level. A support team sees masked credit card numbers. A fraud team sees full data. [Learn more](https://docs.conduktor.io/guide/reference/data-security#data-masking-interceptor) - **Tokenization** — Format-preserving tokens via HashiCorp Vault Transform. Run analytics on protected fields without exposing the real values. [Learn more](https://docs.conduktor.io/guide/reference/data-security#tokenization) - **Cryptographic signing** — HMAC-SHA256 message signing at the gateway. Consumers verify integrity on fetch. Tampered payloads get flagged, meeting chain-of-custody requirements. [Learn more](https://www.conduktor.io/gateway#verify-message-integrity) - **Schema-aware protection** — Mark sensitive fields in Avro, JSON, or Protobuf schemas. Conduktor Gateway, a Kafka proxy, encrypts them automatically on produce and decrypts on authorized fetch, no client changes. [Learn more](https://docs.conduktor.io/guide/reference/data-security#encryption) - **Data quality enforcement** — Validate payloads against CEL or SQL rules at the proxy. Block or redirect non-compliant messages before they reach consumers. [Learn more](https://www.conduktor.io/gateway#stop-bad-data-at-the-source) - **Application audit trail** — Every application action logged: which service produced where, which consumer read what. IP, timestamp, and payload. [Learn more](https://docs.conduktor.io/guide/conduktor-in-production/admin/audit-logs#gateway) ## Why Conduktor for access control - **RBAC & SSO** — SSO via OIDC and LDAP. User and group-based roles. Assign permissions at the resource level. [Learn more](https://www.conduktor.io/console#secure-without-slowing-down) - **Data masking** — Mask PII and sensitive fields per user group in the Console UI. Different teams see different data. [Learn more](https://docs.conduktor.io/guide/conduktor-in-production/admin/data-mask) - **User audit trail** — Every user action logged with IP, timestamp, and payload. 70+ event types across Kafka, IAM, and governance actions. [Learn more](https://docs.conduktor.io/guide/conduktor-in-production/admin/audit-logs) - **ACL & service account management** — Create and manage Kafka ACLs and service accounts directly. View, edit, and delete access rules across clusters. [Learn more](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/service-accounts-acls) ## How they work together Console governs who and what can access your resources. Conduktor Gateway governs what happens to the data as it flows. - **Control access, protect data** — Console manages access for people (RBAC, SSO, masking) and applications (ACLs, service accounts). Conduktor Gateway protects data in motion with encryption, tokenization, crypto-shredding, and cryptographic signing. - **Prove it to auditors** — Console's audit trail covers user and admin actions. Conduktor Gateway covers application-level activity. Together you can show who accessed what and that data is encrypted and validated. ## Encryption comparison | Feature | Cluster-side encryption | Client-side field level | Conduktor encryption | |---------|------------------------|-------------------------|---------------------| | **Encryption type** | In-transit and at-rest (not end-to-end) | In-transit and at-rest | In-transit and at-rest | | **Granularity** | Entire payload | Field-level | Full-payload, field-level, or header-level | | **Implementation** | Requires configuration changes | Requires changes on each client | Configure once, applied everywhere | | **Regulatory compliance** | Limited (in-transit only) | Better for PII and sensitive data | PCI-DSS, GDPR, HIPAA | | **Multi-cluster** | Depends on cluster setup | Limited to clusters on MSP | Works across any Kafka vendor | | **Audit readiness** | Minimal | Moderate | Centralized policy visibility | | **Latency impact** | Low | Moderate, depends on client processing | Low, optimized at the proxy | ## Supported KMS integrations - **HashiCorp Vault** — Secret management, key rotation, and tokenization via Transform Engine. - **AWS KMS** — Native integration with AWS Key Management Service. - **Azure Key Vault** — Microsoft Azure's managed key solution. - **Google Cloud KMS** — Google Cloud Key Management for encryption keys. - **Fortanix DSM** — HSM-backed key management with confidential computing for regulated workloads. ## Measurable impact - **85% faster security implementation** — Per project vs. custom build. InfoSec signs off once, not once per team. - **$200K+ annual savings** — No more custom encryption builds. Fewer incidents from bad data. Less time on audit prep. - **500+ hours saved per year** — Across encryption implementation, access management, and audit preparation. - **75% faster team onboarding** — SSO, RBAC, and self-service ACLs mean new teams get access in minutes, not weeks of ticket queues. ## vs. the alternatives - **vs. Custom encryption per team** — Implementations drift. Fields get missed. Conduktor encrypts once at the wire level across all applications. - **vs. Native Kafka ACLs** — Topic-level only. No field masking, no encryption, no audit trail of who accessed what. Conduktor adds all three. - **vs. Managing access manually** — SSH access, shared credentials, and ticket queues for every permission change. Conduktor gives you SSO, RBAC, and self-service ACL management. ## Frequently Asked Questions **What's the difference between Conduktor encryption and Kafka's built-in TLS?** Kafka's TLS only encrypts data in transit between brokers and clients. Once data reaches the broker it's decrypted, and it sits in plaintext on disk. Anyone with broker or disk access can read it. Conduktor encrypts the payload (or specific fields) before it reaches Kafka, using your KMS keys. Data at rest stays as ciphertext. TLS protects the wire; Conduktor protects the data itself. **Does Conduktor encryption require code changes?** No. Conduktor Gateway encrypts at the wire level by intercepting messages before they reach Kafka. Producers and consumers keep using standard Kafka clients. Nothing changes on their end. **Can different consumers see different data from the same topic?** Yes. Conduktor Gateway masks fields dynamically based on consumer identity. A support team sees masked credit card numbers. A fraud team sees the full data. Same topic. **How do I integrate with my existing KMS?** Conduktor Gateway supports HashiCorp Vault, AWS KMS, Azure Key Vault, Google Cloud KMS, and Fortanix DSM. Configure the connection once. Conduktor Gateway handles key retrieval from there. Vault's Transform Engine is also supported for tokenization. **How do you handle GDPR right-to-erasure when Kafka is append-only?** Crypto-shredding. Conduktor Gateway encrypts records with a per-subject key (say, one key per customer ID). When a user invokes their right to be forgotten, you destroy that key. Every record encrypted with it becomes permanently unreadable. No topic deletion, no reprocessing, no reading every log to scrub PII. **What compliance frameworks does this cover?** Console's audit logs and RBAC address SOC2, ISO 27001, and GDPR access control requirements. Conduktor Gateway's encryption and masking address PCI-DSS, HIPAA, and GDPR encryption mandates. One covers who did what. The other covers how data is protected. **Does this work with Confluent Cloud, AWS MSK, and self-managed Kafka?** Yes. Works with any Kafka distribution. The security layer sits in front of your cluster regardless of the vendor. **How does access control work?** Console manages access for people (SSO, RBAC by user or group, UI masking) and applications (ACLs, service accounts). Conduktor Gateway adds virtual cluster isolation, centralized authentication, and per-tenant policies. Both produce their own audit trails. **I have other questions?** [Drop us a line](https://www.conduktor.io/contact?src=solutions-use-case-data-protection) and we'll get back to you. ## Ready to protect your Kafka data? See how Conduktor handles encryption, access controls, and audit trails across your Kafka infrastructure. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-data-protection) [Read the fintech deep-dive →](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger) ## Read more customer stories - [Bitvavo: DORA & MiCA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) - [Virgin Australia: 300 Hours/Month Saved](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) Source: https://www.conduktor.io/solutions/use-case/data-protection --- # Real-Time Data Sharing *Turn your streams into a partner-facing product.* Partners read live data from the Kafka you already run. Each one gets an isolated zone with its own credentials, and sees only the fields you choose to share. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-data-sharing) [Download Solution Brief](https://www.conduktor.io/assets/pdfs/use-cases/data-sharing.pdf) ## Sharing Kafka data externally usually means a second cluster A DMZ cluster, a MirrorMaker pipeline per partner, and engineering time for every new integration. It holds up while the partner list is short. - **Every partner adds infrastructure to run** — A replica cluster and a replication pipeline per partner. You patch and monitor each one whether or not the partner is consuming from it. - **One topic, different views per partner** — Partners need different fields from the same topic. Without per-reader filtering, the options are duplicating topics or writing a custom filter for each consumer. - **No single record of who accessed what** — Access lives in each partner's cluster config. When compliance asks for evidence of what a partner could read last quarter, someone reconstructs it by hand. ## Expose Kafka Events to Partners Without a Second Cluster BeforeAfter ## Why Conduktor for Real-Time Data Sharing - **Partner Zones** — Partners consume directly from the Kafka cluster you already run, through an isolated zone. The proxy handles the separation, so there is no second cluster to stand up and no replication to operate. - **Field-Level Masking & Encryption** — Share what partners need, mask what they don't. Masking and encryption run as Gateway interceptors scoped to each zone, before data leaves your network. Two partners can read the same topic and see different fields. - **Audit Trails** — Connections, authentications and denied requests are logged by default. Add the audit interceptor to record produce and fetch operations per topic, and export the trail to your SIEM for GDPR, HIPAA, SOC 2 and PCI-DSS reviews. - **Scoped Access Control** — Define which topics a partner can reach, and apply produce and consume rate limits per zone. Ending a partnership means deleting the zone: access goes with it, and nothing was replicated, so there is nothing to clean up. - **Native Kafka Protocol** — Partners connect with the Kafka client they already use. If they can consume from Kafka, they can consume from a Partner Zone. - **Topic Aliasing** — Expose a topic under a name that means something to the partner, without renaming anything internally. Inject or strip message headers per zone. How Data Sharing Works Every step happens on your side of the network. The partner receives credentials and a bootstrap endpoint. Create a Partner ZoneProvision the zone in Console or through the API. It carries its own credentials, its own topic list, and the rate limits you set for that partner. Select and Shape the DataChoose which topics to expose, and under which names. Apply field-level masking and encryption so each partner reads only what they are authorized to read. Connect with Standard Kafka ClientsShare the zone credentials with the partner. They point a standard Kafka client at Conduktor Gateway. No integration work on either side. Monitor Usage per PartnerConsole tracks ingress and egress bytes per zone. That is the number behind usage attribution and renewal conversations. Create a Partner ZoneProvision the zone in Console or through the API. It carries its own credentials, its own topic list, and the rate limits you set for that partner. Select and Shape the DataChoose which topics to expose, and under which names. Apply field-level masking and encryption so each partner reads only what they are authorized to read. Connect with Standard Kafka ClientsShare the zone credentials with the partner. They point a standard Kafka client at Conduktor Gateway. No integration work on either side. Monitor Usage per PartnerConsole tracks ingress and egress bytes per zone. That is the number behind usage attribution and renewal conversations. Measurable Impact Real results from teams sharing real-time data with partners on Conduktor. $150K+ Saved Per PartnershipPartners consume from your existing cluster via Partner Zones. No replica clusters, no Cluster Linking, no custom APIs to build or maintain — whether you're replacing existing infrastructure or avoiding the cost entirely. 2-4 Week Partner OnboardingWhat previously required months of per-partner engineering takes weeks. Reusable templates mean each new partner builds on the last, not from scratch. 2-3 Week Deal ApprovalAutomated audit logs and field-level encryption give compliance teams the evidence they need. Deals that stalled in review — or were blocked entirely — now close in weeks. New Revenue From Data You Already HaveReal-time data sharing opens data monetization opportunities from inventory feeds, telemetry, transaction patterns, and other data that was previously locked inside your platform. ## Approaches to Real-Time Data Sharing When shared data feeds a report, batch exchange works fine. When it drives an operational decision (rerouting a shipment, blocking a transaction, updating a price), it has to arrive as a stream. Conduktor's [data streaming platform](https://www.conduktor.io/product/data-streaming-platform) governs that layer, and Partner Zones extend the same controls across organizational boundaries. Replica Cluster + MirrorMakerConfluent Cluster LinkingConduktor Partner Zones Infrastructure requiredA cluster per partnerA Confluent destination cluster per partnerNone, uses the cluster you already run Data replicationYes, full topic replicationYes, full topic replicationNo, proxied access to the source topics Partner onboardingWeeks of engineeringCluster provisioning, then link setupMinutes to provision via API or Console Field-level access controlNot supportedNo field-level control in the link itselfPer-partner masking and encryption at the proxy Audit trailsManual, fragmentedCluster-level audit logsCentralized per zone in Console Infrastructure per partnerAn additional cluster to run and operateAn additional destination clusterNo additional Kafka cluster Works with any Kafka providerYesConfluent required on the destination sideYes Warehouse-native technologies like Databricks Delta Sharing solve a different problem: sharing tables and datasets between lakehouses, at table granularity. They do that well, and many organizations run both layers. If you're evaluating where each fits, see [Delta Sharing alternatives for real-time data](https://www.conduktor.io/compare/delta-sharing-alternatives), and [real-time data clean rooms](https://www.conduktor.io/glossary/real-time-data-clean-room) for privacy-preserving collaboration on live streams. ## Frequently Asked Questions **What is real-time data sharing?** Real-time data sharing gives external partners access to live event streams as they are produced, rather than periodic batch exports or table snapshots. With Conduktor, partners consume directly from your existing streams through isolated Partner Zones, with masking and encryption applied per reader. **How do we securely expose Kafka events to external partners?** Put Conduktor Gateway in front of the cluster and give each partner an isolated Partner Zone. The partner authenticates against the Gateway with credentials scoped to that zone, reads only the topics you exposed, and sees only the fields your masking rules allow. Nothing is replicated, and your internal topic names stay internal. **Do partners need their own Kafka cluster, or a special client?** No to both. Partners connect to your existing cluster through Conduktor Gateway using any standard Kafka client, librdkafka or Java included. No partner-side infrastructure is required beyond a consumer application. **How is shared data secured, and what evidence do compliance teams get?** Every Partner Zone enforces authentication and topic-level access control. Field-level masking and encryption run as Gateway interceptors scoped to the zone, applied before data leaves your network. Connections, authentications and denied requests are logged by default, and the audit interceptor extends that to produce and fetch operations per topic, which is the trail GDPR, SOC 2 and PCI-DSS reviews ask for. **Can different partners see different data from the same topic?** Yes. Masking and encryption interceptors are scoped per zone, so one partner reads full records while another reads masked fields, from the same source topic. For record-level filtering by field value, our team can walk through the right architecture for your data model. [Book a demo](https://www.conduktor.io/contact/demo?src=solutions-use-case-data-sharing) to discuss. **How is this different from Databricks Delta Sharing?** Delta Sharing shares tables and datasets from a lakehouse, at table granularity, and it does that well. Conduktor shares live streams: partners see events within seconds of production. Many organizations run both, see [Delta Sharing alternatives](https://www.conduktor.io/compare/delta-sharing-alternatives) for a detailed comparison. **We already have a DMZ cluster and MirrorMaker setup. Does this replace it?** For new partnerships, Partner Zones remove the replica cluster and the replication pipeline from the diagram. For existing setups, our team can walk you through how other organizations have moved across. **How do I track what each partner is consuming?** Console tracks ingress and egress bytes per Partner Zone, refreshed every few minutes. That is the number behind usage attribution and renewal conversations. **What happens when a partnership ends?** Delete the Partner Zone and the partner loses access. Credentials issued to a zone expire on their time to live rather than being revoked one by one, so deleting the zone is the clean cut. No data cleanup is required because nothing was replicated. **Can partners produce data back into my Kafka cluster?** Yes. A Partner Zone can grant write access as well as read, with the same authentication and topic-level controls. Encryption applies on the write path, masking applies when data is read back out. ## Go Deeper by Industry - **Transport & Logistics** — OEM dealer networks, supplier integration, logistics partner feeds, and commercial data products. [Explore](https://www.conduktor.io/solutions/industry/transport-and-logistics/partner-data-sharing) - **Financial Services** — Regulatory reporting, credit and risk exchange, and FinOps chargeback across partners, regulators, and clouds. [Explore](https://www.conduktor.io/solutions/industry/financial-services/external-data-sharing) - **Retail & E-commerce** — Supplier integration, carrier data feeds, marketplace connectivity, and loyalty partner data. [Explore](https://www.conduktor.io/solutions/industry/retail-and-ecommerce/supply-chain-sharing) Read more customer stories [Swiss Post scaled Kafka to 800+ users and 5x more applications with governance, RBAC, and self-service.Read Swiss Post's story →](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage)[Bitvavo meets DORA and MiCA obligations with automated audit trails and governed access across its Kafka platform.Read Bitvavo's story →](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica)[A leading European airline consolidated 25 Kafka clusters onto Confluent Cloud with Conduktor.Read the story →](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) # Talk through your partner architecture If you're running a DMZ cluster and a pipeline per partner today, we can walk through what replaces it, and what stays. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-data-sharing) [Download Solution Brief](https://www.conduktor.io/assets/pdfs/use-cases/data-sharing.pdf) Source: https://www.conduktor.io/solutions/use-case/data-sharing --- # Disaster Recovery & Failover *Single-Command Failover. Zero Application Changes.* *Recovery in Minutes, Not Hours.* When Kafka goes down, every minute counts. Conduktor Gateway, a Kafka proxy, handles cluster failover with a single API call. No application changes, no cross-team coordination. Recover in minutes instead of hours. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-disaster-recovery-and-failover) [Download Solution Brief](https://www.conduktor.io/assets/pdfs/use-cases/disaster-recovery-and-failover.pdf) ## Why Manual Kafka Failover Breaks Down at Scale The scripts, the runbooks, the war-room calls at 3 AM. They worked when you had a handful of applications on Kafka. But manual disaster recovery doesn't scale, and every new application makes Kafka failover harder. - **Coordination That Can't Keep Up** — When a primary cluster fails, recovery means updating bootstrap server configs across hundreds of applications and coordinating dozens of teams in parallel. **The complexity grows with every application you add to Kafka and every minute of coordination is a minute of downtime.** - **No Way to Prove Recovery Works** — Most organizations invest heavily in DR infrastructure but rarely test it. Production-like drills are too risky, so runbooks go stale and assumptions go unvalidated. **When a real incident hits, teams discover gaps in real time.** - **Security Gaps During Recovery** — Encryption policies, RBAC configurations, and audit logging are often inconsistent between primary and backup clusters, or skipped entirely during emergency cutover. **A failover that drops security controls is a compliance incident waiting to happen.** ## Why Conduktor for Disaster Recovery - **Transparent Cluster Switching** — Applications connect to Conduktor Gateway, not directly to Kafka. When failover triggers, Conduktor Gateway routes traffic to the backup cluster with no config changes, no application restarts, and no cross-team coordination needed. - **Unified State Switching** — Conduktor Gateway switches topic mappings, interceptor configs, consumer groups, and virtual cluster definitions to the backup cluster in a single coordinated operation. No manual reconfiguration, no drift between clusters. - **Security Continuity** — Encryption, access controls, and audit logging are stored in Conduktor Gateway's internal topics and carry over to the backup cluster when replicated with your cross-cluster replication tool. The same security policies you enforce on your primary cluster apply during and after failover. - **Application Resilience Testing** — Simulate broker failures, network partitions, and latency using Conduktor Gateway's chaos interceptors to validate how your applications handle infrastructure disruptions. Pair with the cluster-switching API to run end-to-end DR drills. - **Full Audit Trail** — Every cluster switch operation is captured in the audit log. Provide regulators and auditors with evidence of DR execution and compliance continuity. - **Failover-Ready Configuration** — Conduktor Gateway maintains your failover cluster configuration alongside the primary, so switching requires a single API call. Pair with your replication tool of choice to keep Gateway state and consumer offsets replicated to the backup cluster. Learn more: [Kafka DR Beyond Replication (whitepaper)](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-beyond-replication) · [Kafka failover configuration](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/failover) · [Chaos testing and DR drills](https://docs.conduktor.io/guide/use-cases/chaos-testing#perform-chaos-testing) · [Gateway audit logging](https://docs.conduktor.io/guide/conduktor-in-production/admin/audit-logs#gateway) · [Gateway overview](https://www.conduktor.io/gateway) · [Kafka data security](https://www.conduktor.io/solutions/use-case/data-protection) ## How Failover Works Four steps from setup to full recovery. - **Applications Connect to Gateway** — Your producers and consumers point to Conduktor Gateway instead of directly to Kafka brokers. This [architectural abstraction](https://www.conduktor.io/architecture) is the foundation of transparent failover. - **Replicate State Across Clusters** — Your cross-cluster replication tool (MirrorMaker 2, Confluent Replicator, or Cluster Linking) continuously replicates Gateway's internal topics, including consumer offsets, topic mappings, and security policies, to the backup cluster. Your failover target stays current. - **Trigger Cluster Switch** — When failover is needed, a single API call switches all traffic to the backup cluster. No application restarts, no config changes, no cross-team coordination. - **Resume Operations** — Applications reconnect through Gateway automatically. Consumers resume from their replicated offsets on the new cluster. Note that consumers may reprocess a small number of messages due to group rebalancing during the switch. Deep Dive ## How Conduktor Gateway Reduces Kafka DR from Hours to Minutes Traditional Kafka DR relies on manual runbooks and cross-team coordination during an outage. Conduktor Gateway eliminates that by sitting between your applications and Kafka, handling cluster switching and security policy enforcement while MirrorMaker keeps state replicated across clusters. [Read the full article ->](https://www.conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes) ![Conduktor Gateway failover architecture](https://www.conduktor.io/assets/images/blogs/blog2-hero.png) Deep Dive ## Kafka Disaster Recovery Beyond Replication The complete DR strategy covering the six technical areas beyond data replication, organized across three operational phases. Includes a chaos testing methodology, compliance mapping for DORA, SOC 2, PCI-DSS, and GDPR, and a step-by-step failover runbook. [Read the full whitepaper ->](https://www.conduktor.io/resources/ebooks/kafka-disaster-recovery-beyond-replication) ![Kafka Disaster Recovery Beyond Replication whitepaper](https://www.conduktor.io/assets/images/ebooks/kafka-disaster-recovery-beyond-replication.webp) ## Measurable Impact - **95% Faster Recovery** — Single-command failover reduces mean time to recovery from hours of manual coordination to minutes of transparent switching. - **$250k+ in Prevented Losses** — Kafka outages cost $300-500k+ per hour in financial services and e-commerce. Faster recovery directly reduces exposure. - **60+ Hours Saved Annually** — Eliminate manual runbook execution, cross-team coordination, and post-incident remediation time. - **Validated DR Posture** — Regular DR drills using chaos interceptors and the cluster-switching API replace annual exercises. Prove recovery works before you need it. ## Frequently Asked Questions **How does Conduktor Gateway handle failover without application changes?** Applications connect to Conduktor Gateway using standard Kafka clients. Conduktor Gateway manages the connection to the underlying Kafka cluster. When a failover is triggered, Conduktor Gateway re-routes all traffic to the backup cluster transparently. Your applications don't need new configs, restarts, or code changes. **What happens to consumer offsets during failover?** Consumer offsets are continuously replicated to the backup cluster using your replication tool of choice, whether that's MirrorMaker 2, Confluent Replicator, or Cluster Linking. When consumers reconnect through Gateway after a switch, they resume from their last replicated position on the new cluster. Consumers may reprocess a small number of messages due to replication lag and group rebalancing during the switch. **Can I test failover without affecting production?** Yes. Conduktor Gateway's chaos interceptors let you simulate broker failures, latency, and other disruptions to test application resilience. You can also run end-to-end DR drills by triggering the cluster-switching API in a staging environment to validate your recovery path without risking production traffic. **Does this work with Confluent Cloud, AWS MSK, and self-managed Kafka?** Yes. Conduktor Gateway works with any Kafka distribution. Your primary and backup clusters can even run on different providers, giving you true infrastructure independence for your DR strategy. **How fast is the actual cluster switch?** The cluster switch typically executes in seconds for most deployments. Conduktor Gateway re-establishes connections to the new cluster and resumes traffic routing automatically. End-to-end recovery, including application reconnection and consumer group rebalancing, typically completes in minutes. Actual time depends on deployment size and network conditions. **What about security and compliance during failover?** Encryption policies, RBAC configurations, and audit logging are enforced identically across both clusters. Conduktor Gateway maintains continuous audit trails throughout the failover process, so you can demonstrate uninterrupted compliance to regulators. ## Ready to simplify Kafka failover? See how Conduktor reduces recovery time from hours to minutes. Our team can help you design a DR strategy that eliminates cross-team coordination and validates recovery before you need it. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-disaster-recovery-and-failover) [Download Solution Brief](https://www.conduktor.io/assets/pdfs/use-cases/disaster-recovery-and-failover.pdf) ## Read more customer stories - [Bitvavo: DORA & MiCA Compliance](https://www.conduktor.io/customer-stories/bitvavo-ensures-compliance-dora-mica) - [Retail: 99% Faster Kafka Onboarding](https://www.conduktor.io/customer-stories/one-hour-kafka-onboarding-retail-streaming-operations) - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) Source: https://www.conduktor.io/solutions/use-case/disaster-recovery-and-failover --- # Kafka Cost Allocation & Chargeback *See the bill. Place it where it belongs.* Kafka spend keeps growing because no one outside the platform team can act on it. Conduktor attributes cost to the applications and teams that drive it, then gives those owners the visibility and the controls to act at the source. [Get your Kafka Cost Analysis](https://www.conduktor.io/contact/kafka-cost-analysis?src=solutions-use-case-kafka-chargeback) [Read the Field Guide](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) ## Visibility on one side, authority on the other The platform team can see the waste but cannot act on resources other teams created. The teams that can act cannot see what their choices cost. Kafka gives you neither a per-team view of spend nor a way to hand owners the controls to fix it, so cost grows by accretion and central cleanup never sticks. - ****Service accounts show usage, not ownership**** — Bytes in and out tell you how much data moved, never which team, project, or environment it belongs to. Finance cannot budget against raw technical identifiers. - ****Cleanup is a coordination problem**** — A platform team finds thousands of empty topics and cannot delete one of them, because they do not own them and cannot confirm which are abandoned and which are waiting on a monthly consumer. - ****Cost is shaped like the cluster, not the org**** — The bill rolls up by broker and partition, not by the team or product line that generated it, so no one recognizes their own share. - ****No lever for the people with context**** — The teams that know whether a topic is still needed have no view of its cost and no safe way to retire or right-size it. ## Why cost cleanup stalls: three stages of ownership Most organizations sit somewhere on the same progression. What changes across the stages is not who pays the bill, but whether the people positioned to act have the visibility and the controls to do it effectively. | | Stage 1 | Stage 2 | Stage 3 | |---|---|---|---| | **Who pays the bill** | Platform team carries it alone | Allocated as a flat share across teams | Teams pay in proportion to actual usage | | **Who can act on cost** | Platform team alone, without levers on resources they don't own | Teams now pay, but coarse data and no expertise to tell value from waste | Teams act on their share, with platform-team expertise on what matters | | **What enables the stage** | Default starting point | A model for dividing the bill, plus coarse usage views | Proportional attribution, self-service tooling, guardrails, and a working partnership | | **Where it breaks** | Accountability without the levers | Responsibility shifted, capability didn't follow | Requires the prior structures to be working | Stage 3 is where behavior actually changes, and it depends on two things landing together: spend attributed to the teams that own it, and those teams handed the controls to act. [Read the full breakdown →](https://www.conduktor.io/blog/your-platform-team-cant-fix-kafka-costs-alone) See the spend Attribution and waste detection are the foundation. Before any cleanup or accountability conversation, the spend has to map to how the business is structured, and the waste has to be visible to the people who own it. Chargeback ## Map Kafka spend to teams with Chargeback Chargeback turns raw consumption into an accounting view of spend. Costs roll up by application, service account, or any label, so reports line up with how the business is structured rather than how the infrastructure is shaped. - **Four cost axes**: storage, partitions, ingress, and egress, each with configurable unit costs tied to your actual contract terms - **Group by application, service account, or label** (team, department, environment, business unit), with drill-down into the topics and accounts behind every line - **Confluent Cloud direct**: pulls ingress and egress from the Confluent Metrics API and maps anonymous service accounts back to the teams that own them, with no Conduktor Gateway required - **Export to CSV** so finance can budget and forecast against numbers that match the org chart [Learn how Chargeback works →](https://docs.conduktor.io/guide/use-cases/chargeback) By application By business unit Insights ## Surface the waste behind the bill Attribution tells you who owns the spend. Insights tells you which of it is waste. The Cost Control view finds the topics you are paying for that you should not be, and ranks them so cleanup runs by impact, not by cluster. - **Empty, stale, and tiny topics** surfaced automatically, with idle time and partition count on each - **Ranked by storage, partitions, and throughput**, with the underlying pattern flagged so the biggest offenders rise to the top - **Filter by application, topic prefix, or cluster** so cleanup scopes to the topics that matter [Explore Kafka Health & Risks →](https://www.conduktor.io/solutions/use-case/kafka-health-and-risks) Empty Stale Tiny Hand owners the controls Visibility on its own does not change behavior. The shift to Stage 3 happens when the teams that can see their own bill also get the controls to act on it, without routing every change through a platform admin. Federated ownership ## Give every team its own bill, and the means to act Cost pushed centrally never sticks. Conduktor scopes both the visibility and the controls to the owning team, so each one sees its own consumption and can act at the source rather than waiting on the platform team. - **RBAC-aware Insights and Chargeback**: developers, SREs, and team leads see the health and cost of the topics they own, not just admins - **Delegated permissions through ApplicationGroups**: hand specific responsibilities, like requesting access, approving requests, and managing service accounts or API keys, to the people who should hold them - **Self-service with guardrails**: owners retire, right-size, and provision their own resources within bounds the platform team sets at creation time - **Ownership required on every new resource**, so orphan topics are blocked by construction rather than chased down later [Learn more about federated ownership →](https://www.conduktor.io/federated-ownership) Label coverage ## What changes when owners can see their own bill Attribution and delegated controls move an estate from central firefighting to owners acting at the source. These are the typical shifts we measure once teams can see and act on their own consumption. - **~90 percent less cleanup coordination** — Cleanup outreach drops from roughly 1.5 hours per project to under 15 minutes, because teams arrive already aware of their own consumption from the chargeback view. - **Stale topic rate from ~10% to ~3%** — When teams see their own bill, steady-state staleness drops 60 to 80 percent across the estates we measure. - **Attribution across every cluster and provider** — Confluent Cloud, Aiven, MSK, and self-managed, mapped to teams in one view rather than one cluster at a time. - **Support contract proportionally smaller** — Support priced as a percentage of platform spend shrinks automatically as the infrastructure it bills against shrinks, with no renegotiation. ## Frequently asked questions **How does cost attribution work across multiple clusters and providers?** Conduktor connects to every cluster regardless of provider and tracks consumption at the application level. Conduktor Gateway meters bytes in and out per service account, topic, and virtual cluster, while for Confluent Cloud the figures come straight from the Confluent Metrics API. Console aggregates all of it so cost breakdowns surface by team, application, or business label across the entire estate, not one cluster at a time. **Do I need Conduktor Gateway for Chargeback?** No. For Confluent Cloud, Chargeback pulls ingress and egress from the Confluent Metrics API and joins them with Console's storage and partition data. Conduktor Gateway adds the same ingress and egress metering for other cluster types. Storage and partition costs are available for every cluster either way. **Why does attribution change team behavior when central cleanup does not?** Central cleanup stalls because the platform team has visibility without authority and the project teams have authority without visibility. Putting each team's own bill in front of it closes that gap: the team that can retire a topic is now the team that can see what it costs. Pairing that with delegated controls means they can act without filing a ticket, which is what turns awareness into change. **How is this different from the Kafka Cost Optimization page?** Cost allocation and chargeback is about seeing and owning the spend. [Kafka cost optimization](https://www.conduktor.io/solutions/use-case/kafka-cost-optimization) is about cutting it, with levers like capacity pooling, topic views, and creation-time guardrails. Most teams start here, because attribution and ownership make every other lever land. **Does this work with managed Kafka?** Yes. Insights and Chargeback connect to every cluster regardless of provider, and managed-Kafka customers often see proportionally larger gains because per-unit infrastructure cost is higher and the attribution gap is wider. ## Ready to see where your Kafka bill comes from? Book a cost analysis with our field engineering team. We will map your current ownership model, attribute the spend to the teams behind it, and show where the gaps are before any cleanup begins. [Get your Kafka Cost Analysis](https://www.conduktor.io/contact/kafka-cost-analysis?src=solutions-use-case-kafka-chargeback) [Read the Field Guide →](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) ## Read more customer stories - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) - [Smart Farming: 10x Kafka Utilization](https://www.conduktor.io/customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk) - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) Source: https://www.conduktor.io/solutions/use-case/kafka-chargeback --- # Kafka Cost Optimization *Find waste. Attribute spend. Run leaner.* Kafka usage grows by accretion: more topics, clusters, integrations, and the operational overhead behind them. Conduktor gives platform teams the visibility, attribution, and architectural levers to find what's recoverable, run more efficiently, and keep the gains from eroding. [Get your Kafka Cost Analysis](https://www.conduktor.io/contact/kafka-cost-analysis?src=solutions-use-case-kafka-cost-optimization) [Read the Field Guide](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) ## Where Kafka Costs Actually Hide The Kafka bill is rarely one line item. It surfaces across four layers, and waste in one pulls the others up with it. Operational overhead is the layer most teams underestimate. Most platform teams find 25 to 40 percent recoverable spend across these layers without a replatform or a renegotiation. - ****Infrastructure**** — Brokers, partitions, storage, replication. The biggest layer and the easiest to act on. Six patterns drive most of the waste, from partition overprovisioning to topic proliferation. - ****Ecosystem tooling**** — Streams, ksqlDB, Flink, Connect. Sits on top of infrastructure and inherits its inefficiencies. Filter-only jobs alone can dominate stream-processing usage. - ****Vendor and licensing**** — Support contracts priced as a percentage of platform spend. Tier upgrades and add-ons scale with cluster footprint. Shrink the infrastructure, these shrink with it. - ****Operational overhead**** — The cost that never appears on the bill, and the largest one most teams miss. Platform-team time on manual provisioning, firefighting, and cleanup coordination that better tooling would absorb. [Read the Field Guide: Where Kafka Costs Hide →](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) ## From patterns to solutions Most Kafka estates carry the same six waste patterns. Conduktor helps in two ways: in identifying waste and addressing inefficiencies. | Pattern | What it looks like | How Conduktor helps | |---------|-------------------|---------------------| | **Partition overprovisioning** | Topics with 30+ partitions for use cases that need 3, brokers approaching the per-broker partition-replica ceiling | [Insights](#find-the-waste-hiding-in-your-estate) finds the over-partitioned topics. [Cost Guardrails](#stop-waste-at-creation-time) bound new ones at creation. | | **Retention misalignment** | Default retention on every topic regardless of consumer lag, retention longer than any consumer reads back | [Insights](#find-the-waste-hiding-in-your-estate) flags retention overrun. [Cost Guardrails](#stop-waste-at-creation-time) cap retention windows on new topics. | | **Cluster sprawl** | A cluster per project, environment, or business unit, each carrying its own broker fixed cost | [Capacity Pooling](#pool-capacity-instead-of-allocating-it) consolidates dedicated isolation clusters onto shared infrastructure. | | **Topic proliferation and duplication** | Orphan topics with no traffic, near-duplicates, derived topics from filter-only stream processing | [Insights](#find-the-waste-hiding-in-your-estate) surfaces orphans and duplicates. [Topic Views](#replace-filter-only-stream-processing) replace filter-only derivations. [Chargeback](#make-app-teams-own-their-costs) gets app teams retiring what they don't need. | | **Inefficient client patterns** | Producers without compression, idempotence misconfigured, consumers without partition awareness | [Cost Guardrails](#stop-waste-at-creation-time) enforce compression, idempotence, and ack policies on every producer. | | **Static capacity per resource** | Every topic gets dedicated partitions and replicas regardless of actual throughput | [Capacity Pooling](#pool-capacity-instead-of-allocating-it) replaces dedicated partitions with shared backing topics via concentration. | Identify Waste You can't fix what you can't see and so central cleanup never sticks. Insights finds the waste while Chargeback puts the bill in terms the business can act on. Insights ## Find the waste hiding in your estate Insights surfaces the topics you are paying for that you should not be: orphans, oversized, over-retained. The Cost Control view ranks them so cleanup runs by impact, not by cluster. - **Cost Control** ranks expensive topics by storage, partitions, and throughput, with the underlying pattern (over-partitioned, retention overrun, orphan) flagged on each - **Filter by application, topic prefix, or cluster** so cleanup scopes to the topics that matter - **RBAC-aware** so app owners see the topics they own and act on them without escalating to a platform admin [Learn more about Insights →](https://www.conduktor.io/solutions/use-case/kafka-health-and-risks) Insights Chargeback ## Make app teams own their costs Cleanup pushed centrally never sticks. Chargeback turns consumption into an accounting view of spend, so teams generating the cost see their own bill before anyone has to ask. - **Spend rolls up by application, service account, or any label** (team, department, business unit) so reports match the org chart, not the cluster topology - **Configurable unit costs** for storage, partitions, and ingress/egress tied to your actual contract terms - **Confluent Cloud direct**: pulls ingress and egress from the Confluent Metrics API, no Conduktor Gateway required [Learn how chargeback works →](https://docs.conduktor.io/guide/use-cases/chargeback) Chargeback Address Inefficiency Visibility surfaces the waste, but cutting it takes a different mix of levers: capacity pooling, topic views in place of filter-only stream processing, and creation-time guardrails on every new resource. Capacity Pooling ## Pool capacity instead of allocating it Most cost waste is structural: each team gets its own cluster, each topic gets its own dedicated partitions. Virtual clusters and topic concentration share the underlying infrastructure without breaking isolation. - **Virtual clusters** consolidate dedicated isolation clusters onto shared physical infrastructure, eliminating per-team broker overhead - **Topic concentration** multiplexes low-volume topics onto a single physical backing topic, dropping partition counts on sparse topics by 50 to 90 percent - **Hard isolation** between environments, brands, or business units with no new brokers required [Learn more about virtual clusters →](https://docs.conduktor.io/guide/conduktor-concepts/virtual-clusters) · [Topic concentration →](https://docs.conduktor.io/guide/conduktor-concepts/logical-topics#concentrated-topics) Virtual Clusters Topic Concentration # Two isolation boundaries on one physical cluster. # No new brokers, no new licenses, no naming conventions. apiVersion: gateway/v2 kind: VirtualCluster metadata: name: payments-team spec: type: Standard aclEnabled: true superUsers: - payments-admin apiVersion: gateway/v2 kind: VirtualCluster metadata: name: orders-team spec: type: Standard aclEnabled: true superUsers: - orders-admin # Sparse regional topics: 24 logical partitions each, # backed by 6 physical partitions on a shared topic. apiVersion: gateway/v2 kind: ConcentratedTopic metadata: name: customer-events-eu vCluster: payments-team spec: advertisedPartitions: 24 backingTopic: _concentrated_customer_events apiVersion: gateway/v2 kind: ConcentratedTopic metadata: name: customer-events-us vCluster: payments-team spec: advertisedPartitions: 24 backingTopic: _concentrated_customer_events # Two topics × 24 partitions = 48 advertised. # Physically backed by 6 partitions: 87.5% reduction. Topic Views ## Replace filter-only stream processing Most stream-processing jobs do one thing: read a topic, drop rows a consumer does not need, and write the rest somewhere new. The team pays three times: engine, dev time, derived infrastructure. Topic Views handle this at the proxy. - **SQL-based topic views** serve a filtered or projected subset at the proxy layer, with no new physical topic or partitions to pay for - **Caching** for high-frequency repetitive reads, reducing broker fetch load on the source topic [Learn more about topic views →](https://docs.conduktor.io/guide/use-cases/cache-data) SQL Topic View Caching # Replaces a Flink job that filtered "customers" to EU adults. # No derived topic, no new partitions, no Flink instance. apiVersion: gateway/v2 kind: Interceptor metadata: name: customers-eu-adults spec: pluginClass: io.conduktor.gateway.interceptor.VirtualSqlTopicPlugin priority: 100 config: virtualTopic: customers-eu-adults statement: | SELECT firstName, lastName, email, country FROM customers WHERE age >= 18 AND country IN ('FR', 'DE', 'ES') schemaRegistryConfig: host: http://schema-registry:8081 # High-frequency read patterns served from cache, # reducing broker fetch load and outbound bandwidth. apiVersion: gateway/v2 kind: Interceptor metadata: name: cache-reference-data spec: pluginClass: io.conduktor.gateway.interceptor.CacheInterceptorPlugin priority: 100 config: topic: "reference.*" cacheConfig: type: IN_MEMORY inMemConfig: cacheSize: 1000 expireTimeMs: 60000 Cost Guardrails ## Stop waste at creation time The cheapest cleanup is the one you never have to run. Bound partitions, retention, and replication at topic creation, require an owner on every new resource, and future loads inherit the discipline. - **Partition and retention bounds** enforced at topic creation, with override-to-fixed or block actions - **Replication factor enforcement** to prevent quietly-doubled replication on non-critical topics - **Producer policies** for compression and idempotence to standardize client efficiency - **Federated ownership** required on every new resource so orphan topics are blocked by construction [Learn more about safeguards →](https://docs.conduktor.io/guide/use-cases/apply-traffic-control-policies) · [Federated ownership →](https://www.conduktor.io/federated-ownership) Topic Creation Policy Producer Policy # New topics are bounded on partitions, retention, # and replication factor at creation time. apiVersion: gateway/v2 kind: Interceptor metadata: name: topic-cost-policy spec: pluginClass: io.conduktor.gateway.interceptor.safeguard.CreateTopicPolicyPlugin priority: 100 config: numPartition: min: 3 max: 12 action: OVERRIDE overrideValue: 6 replicationFactor: min: 3 max: 3 action: BLOCK retentionMs: min: 86400000 max: 604800000 action: OVERRIDE overrideValue: 259200000 # Producers without compression or idempotence are blocked. # Consistent client efficiency across every team. apiVersion: gateway/v2 kind: Interceptor metadata: name: producer-efficiency-policy spec: pluginClass: io.conduktor.gateway.interceptor.safeguard.ProducerPolicyPlugin priority: 100 config: compressionType: allowed: ["zstd", "lz4", "snappy"] action: BLOCK acks: required: "all" action: BLOCK enableIdempotence: required: true action: BLOCK ## Three approaches to sequencing the work The capabilities above are levers. How they get applied depends on which approach the team is taking. Most platform teams run all three in parallel: defaults catch new loads, optimization works through the existing estate, and architectural changes land over a longer horizon. - **Update defaults for new loads** — Set policies so new topics, clusters, and clients do not inherit waste. Partition defaults, retention policies, replication enforcement, and ownership requirements at creation time. Low coordination, fast to implement. Slows future cost growth without producing immediate savings. - **Optimize existing loads** — Hygiene and right-sizing on what is already running: tuning retention, retiring orphans, right-sizing partition counts, consolidating duplicate topics. The gating factor is coordinating with producers and consumers, not the technical work itself. Typically moves the infrastructure bill by 10 to 20 percent in weeks to months, with reductions of 50 percent or more in estates with significant accumulated waste. - **Rethink workloads** — Reshape data flows: pooled capacity at the cluster and topic layer, replacing filter-only stream processing with topic views, consolidating per-team clusters. The most variable approach in timeline and outcome, with the largest impact in big estates carrying years of accumulated structural decisions. ## What to expect Typical ranges from the estates we have analyzed. Your number depends on where you are starting from and which patterns dominate. - **25 to 40 percent recoverable** — The typical share of the Kafka infrastructure bill that's recoverable without a replatform. Configuration tuning and topic retirement do most of the work; consolidation and architectural changes close the rest. - **10 to 20 percent typical infra reduction** — What optimization on existing workloads usually moves the bill by, in weeks to months. Up to 50 percent in estates with significant accumulated orphans and over-partitioning. - **50 to 90 percent partition reduction** — Topic concentration on sparse low-throughput topics. Regional topics that would need hundreds of partitions back onto a fraction of the physical footprint. - **~90 percent less cleanup coordination** — Cleanup outreach drops from roughly 1.5 hours per project to under 15 minutes. Teams arrive already aware of their own consumption from the chargeback dashboard. - **Stale topic rate from ~10% to ~3%** — What happens when teams see their own bill. Steady-state staleness drops 60 to 80 percent across the estates we measure. - **Support contract proportionally smaller** — Support contracts on most hosted Kafka platforms are billed as a percentage of total platform spend. Cleaning up infrastructure shrinks that line item automatically, no renegotiation needed. ## Frequently Asked Questions **How much can I actually save?** A meaningful share of the typical Kafka infrastructure bill is recoverable through configuration changes, retirement, consolidation, and architectural levers. The 25 to 40 percent range is what we see across estates we analyze closely. Optimization on existing workloads typically moves the bill 10 to 20 percent in weeks to months, with reductions of 50 percent or more in estates carrying significant accumulated waste. Architectural changes like pooled capacity yield more variable returns with the largest impact in big estates. **Where do the savings tend to come from?** A small number of patterns account for most recoverable cost: partition overprovisioning, retention not matched to consumer needs, cluster sprawl, topic proliferation and duplication, inefficient client patterns, and static capacity per resource. The [field guide](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) covers each in detail. Estates vary in which dominate, but the data usually points to the biggest one or two quickly. **Where should I start?** Visibility first, almost always. Without a per-team breakdown of consumption, every other lever is operating blind. Insights and Chargeback are the typical starting point because they make the patterns and the accountability story concrete before any cleanup or architectural work begins. **What is topic concentration and how does it reduce costs?** Topic concentration multiplexes multiple low-volume logical topics onto a single physical backing topic. Applications still see independent topics with their own names and advertised partition counts, but the underlying storage is shared. Partition counts on sparse topics can drop 50 to 90 percent, directly lowering broker CPU, storage, and replication costs. Best fit is non-production environments and low-throughput production topics. **How does cost attribution work across multiple clusters and providers?** Conduktor connects to every cluster regardless of provider and tracks consumption at the application level. Conduktor Gateway meters bytes in and out per service account, topic, and virtual cluster. Console aggregates this into chargeback dashboards so cost breakdowns surface by team, application, or business label across the entire Kafka estate, not one cluster at a time. **Can I reduce costs without changing platforms?** Yes. Most cost work happens within the existing platform: tuning configurations, retiring waste, consolidating clusters, and pooling capacity through virtual clusters and topic concentration. Replatforming is a smaller set of cases and the longest commitment, and the first two categories of work usually close most of the gap. When a [Kafka migration](https://www.conduktor.io/solutions/use-case/kafka-migration) is planned, cost cleanup typically runs alongside it. **How do I keep the savings from eroding?** Defaults and policies are necessary but not sufficient. Holding the gains involves three things: visibility into what the estate contains and what it costs, clear ownership of every topic and cluster, and a regular cadence of reviewing what is actually being used. Federated ownership and self-service with guardrails are the structural pieces that hold up over years. **Does Conduktor work with managed Kafka (Confluent Cloud, AWS MSK)?** Yes. Console Insights and Chargeback connect to every cluster regardless of provider. Conduktor Gateway features like virtual clusters, topic concentration, and topic views work the same way, since clients connect to the gateway exactly like a broker. Managed-Kafka customers often see proportionally larger savings from concentration and view-based filtering because per-unit infrastructure cost is higher. ## Ready to find what is recoverable in your estate? Book a cost analysis with our field engineering team. We will walk through your estate together, identify the waste patterns that apply, and give a concrete estimate of where the savings sit. [Get your Kafka Cost Analysis](https://www.conduktor.io/contact/kafka-cost-analysis?src=solutions-use-case-kafka-cost-optimization) [Read the Field Guide →](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) ## Read more customer stories - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) - [Smart Farming: 10x Kafka Utilization](https://www.conduktor.io/customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk) - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) Source: https://www.conduktor.io/solutions/use-case/kafka-cost-optimization --- # Kafka Health & Risks *Stop Firefighting. Start Predicting.* Your Kafka monitoring shows what's happening now. Conduktor Insights shows what's about to break. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-kafka-health-and-risks) [Try Now](https://www.conduktor.io/get-started?src=solutions-use-case-kafka-health-and-risks) Trusted by ## You can't fix what you can't see Most teams discover Kafka risks during an incident. By then the data loss or the outage has already happened. - **Silent Data Loss Risks** — RF=1 with active consumers means zero redundancy on data people depend on. One broker failure, and it's gone. - **Under-Replication** — Topics running below their replication factor right now. Lose one more broker and writes block or data is lost. - **Partition Problems** — Too few partitions and you can't scale. Skewed partitions overload some brokers while others sit idle. - **Unknown VIPs** — Which topics are business-critical? Without visibility, the topic feeding 47 services gets the same treatment as a test topic. ## Topic intelligence for health and risk Conduktor Insights analyzes every topic across your clusters to surface exactly where reliability, performance, and governance need attention. Risk Analysis ## Identify risks before they become incidents Find topics that could cause data loss, performance degradation, or service disruptions. Insights correlates replication, volume, and consumer activity to surface risks that simple threshold alerts miss. - **Low replication factor detection**: Find single-replica topics instantly - **Partition skew analysis**: Identify uneven data distribution - **Single partition bottlenecks**: Spot topics that can't scale - **Actionable remediation recommendations** Data loss Cluster efficiency Load imbalance VIP Topics & Governance ## Govern your most critical topics Automatically identify your business-critical topics by consumer activity, throughput, and fanout, then check whether those exact topics are governed. The topics feeding the most services are the ones a schema gap or ad-hoc provisioning hurts most. - **Automatic VIP detection**: surface topics with high consumer counts, heavy throughput, and wide fanout - **Schema coverage on critical topics**: which VIP topics have a registered contract, and which leave consumers exposed - **Serialization consistency**: Avro, JSON, and Protobuf usage across your highest-impact streams - **Self-service coverage**: which critical topics were provisioned through governed channels versus created ad-hoc Topic Health ## Actionable health recommendations Get a comprehensive health score for your entire cluster and detailed recommendations for each topic that needs attention. - **Cluster health score**: Instant 0-100 assessment - **Per-topic configuration recommendations** - **Retention policy optimization suggestions** - **Prioritized action items based on impact** ## Insights for every team, not just admins - **RBAC-aware by default** — Insights respects Console permissions, so developers, SREs, and team leads see the health of the topics they own without a platform admin in the loop. - **Filter to what you own** — Scope every view by application or topic prefix pattern, so each team works from a view shaped by the topics they touch. - **Export for review** — Pull any view to CSV for offline analysis, audit evidence, or a regular governance cadence with stakeholders. Impact Measurable impact from teams using Conduktor Insights. Risks surfaced before incidentsFind RF=1 topics, under-replication, and partition skew before a broker failure turns them into an outage. 5 min to first insightInstant visibility without agents, code changes, or complex setup. 100% topic visibilityEvery topic analyzed across risk, VIP, governance, and health. Visibility for every teamRBAC-aware views put topic health in the hands of the people who own each topic. [Kafka Cost Allocation & ChargebackAttribute Kafka spend to the teams, applications, and topics driving it, then charge it back. Break down cluster costs across your organization and turn shared infrastructure into accountable budgets.See how →](https://www.conduktor.io/solutions/use-case/kafka-chargeback) ## Frequently asked questions **How is this different from Prometheus/Grafana?** Monitoring tells you what's happening now. Insights tells you what's about to become a problem. We analyze patterns across topics to surface risks you'd never think to alert on, like partition skew ratios or topics with RF=1 that haven't failed yet. **Do I need a large Kafka deployment to benefit?** No. Even small deployments accumulate hidden risk: single-replica topics, partition skew, schema gaps, and configuration drift. Insights surfaces these regardless of cluster size. **What about cost and chargeback?** Cost control sits alongside health and risk inside Insights. For the full cost story, including attribution and chargeback, see [Kafka Cost Allocation & Chargeback](https://www.conduktor.io/solutions/use-case/kafka-chargeback). **How does Insights integrate with my existing setup?** Insights connects read-only to your Kafka clusters. No agents to install, no code changes required. Works with Confluent, AWS MSK, Redpanda, and self-managed Kafka. [See the integration guide](https://docs.conduktor.io/guide/insights). # Get started with Kafka Health & Risks Move from reactive to predictive Kafka management. Get a personalized walkthrough of Conduktor Insights. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-kafka-health-and-risks) [Read Documentation](https://docs.conduktor.io/guide/insights) Source: https://www.conduktor.io/solutions/use-case/kafka-health-and-risks --- # Kafka Migration *Switch providers without rewriting applications.* *Cut migration timelines in half.* Whether you're migrating to the cloud, switching providers, or consolidating clusters as part of the move, Conduktor gives you the visibility and control to act without disruption. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-kafka-migration) [Download Solution Brief](https://www.conduktor.io/assets/pdfs/use-cases/migration-and-infrastructure-efficiency.pdf) ## Most Kafka Migrations Hit the Same Problems Migration projects stall when hidden dependencies surface mid-cutover, when application teams have to be coordinated for every cluster change, or when years of estate sprawl make any kind of move feel impossible. The technical work is rarely the hard part. Coordination, dependency discovery, and rollback safety are. - **Migrations That Stall or Blow Budget** — Migration budgets balloon 30-50% as hidden dependencies surface mid-project. Engineering capacity gets consumed by manual topology mapping and cross-team coordination. Critical blockers discovered during cutover force rollbacks and restarts. - **Application Changes Cascade Across Teams** — Every cluster move requires updating bootstrap servers, redeploying clients, and coordinating with every consuming team. What should be a platform task turns into a multi-quarter cross-team project. - **Estate Sprawl Slows Every Move** — Years of one-cluster-per-team accumulation make provider switches and cloud migrations feel impossible. Each cluster has its own owners, dependencies, and informal coordination that has to be unwound before anything can change. ## Why Conduktor for Kafka Migration - **Transparent Traffic Routing** — Conduktor Gateway, a Kafka proxy, sits between your applications and Kafka, enabling cluster switches without application code changes. Migrate cluster-by-cluster while applications keep running. - **Cross-Provider Visibility** — Unified view across OSS Kafka, MSK, Confluent, and Aiven. Automatically discover topics, consumer groups, connectors, and schemas across providers. Plan migration with complete resource data. - **Automatic Resource Discovery** — Index every topic, consumer group, connector, and schema across all connected clusters. Replace manual spreadsheets and team interviews with a complete resource inventory and dependency graph. - **Topic Aliasing** — Remap topic names transparently at the Conduktor Gateway layer. Applications continue using their existing topic names while the underlying infrastructure changes beneath them. - **Phased Cutover** — Migrate workloads incrementally rather than in a single risky cutover. Route specific topics to the new cluster while others stay on the old one, then cut over the rest when ready. - **Instant Rollback** — Conduktor Gateway routes traffic at the proxy layer, so cutting back to the old cluster takes seconds. No application redeploys, no DNS changes, no risk of permanent state. Learn more: [Conduktor Gateway overview](https://www.conduktor.io/gateway) · [Virtual clusters](https://docs.conduktor.io/guide/conduktor-concepts/virtual-clusters#virtual-clusters) · [Insights](https://www.conduktor.io/solutions/use-case/kafka-health-and-risks) ## How Migration Works Four steps from fragmented estate to migrated and stable. - **Map Your Kafka Estate** — Conduktor connects to all your clusters and automatically discovers topics, consumer groups, connectors, and schemas. You get a complete picture of what you're running, who's using it, and what depends on what. No more spreadsheets. - **Reduce What You Need to Migrate** — [Insights](https://www.conduktor.io/solutions/use-case/kafka-health-and-risks) flags empty, stale, and tiny topics that are better retired than migrated, plus high-impact topics with many consumers that warrant extra care during cutover. The result is a smaller, prioritized scope instead of lifting everything as-is. - **Cut Over Through Conduktor Gateway** — [Conduktor Gateway](https://www.conduktor.io/gateway) routes traffic transparently. Applications point to Conduktor Gateway, and the underlying cluster can change without code modifications, config updates, or cross-team coordination. The same architecture powers [disaster recovery and failover](https://www.conduktor.io/solutions/use-case/disaster-recovery-and-failover). - **Verify and Decommission** — With routing controlled at the Conduktor Gateway layer, you can move topics one at a time, verify each cutover in production, and cut over the rest once stability is confirmed. If anything goes wrong, rollback takes seconds rather than days. ## Measurable Impact - **Zero Downtime Cutover** — Transparent traffic routing enables cluster migrations without service interruption or application changes. - **50% Faster Migration Timeline** — Automatic resource relationship discovery and Conduktor Gateway-based routing replace months of manual coordination and cross-team planning. - **Zero Application Changes** — Applications continue using existing client configurations through cutover, eliminating coordination across product teams. - **Seconds to Roll Back** — Conduktor Gateway-level routing means rollback is reversing a config, not redeploying applications or restoring backups. ## Frequently Asked Questions **How does Conduktor enable migration without application changes?** Applications connect to Conduktor Gateway using standard Kafka clients. Conduktor Gateway manages the connection to the underlying cluster. When you're ready to migrate, Conduktor Gateway routes traffic to the new cluster transparently. Your applications don't need new configs, restarts, or code changes. You can migrate gradually and roll back instantly if needed. **Can I migrate without risking production?** Yes. Conduktor Gateway handles traffic routing transparently, so you can move workloads incrementally rather than in a single risky cutover. If you're consolidating clusters as part of the migration, virtual clusters provide logical isolation on shared physical infrastructure so teams maintain their own namespaces and access controls. **Does this work across different Kafka providers?** Yes. Conduktor provides a unified control plane across OSS Kafka, Amazon MSK, Confluent Cloud, Aiven, and self-managed deployments. You can migrate between providers without application awareness. **How do I prioritize what to migrate first?** Conduktor Insights surfaces empty, stale, and tiny topics across your clusters that are better retired than migrated, plus high-impact topics with many consumers that warrant extra care during cutover. Combined with the resource inventory in Console, you get the data needed to scope the work and prioritize by impact rather than guesswork. This cleanup work overlaps with broader [Kafka cost optimization](https://www.conduktor.io/solutions/use-case/kafka-cost-optimization). **What about topic naming conflicts during migration?** Conduktor Gateway supports topic aliasing, which lets you remap topic names at the proxy layer. Applications continue using their existing topic names while the physical topics are reorganized underneath. No naming collisions, no code changes. **How do I map resources before starting a migration?** Conduktor automatically indexes topics, consumer groups, connectors, and schemas across all connected clusters. Instead of manually building spreadsheets or interviewing application teams, you get a complete resource inventory that shows which consumer groups subscribe to which topics, connector-to-topic relationships, and resource ownership. This is what makes it possible to sequence migrations by complexity and avoid surprises during cutover. **What happens if something goes wrong during cutover?** Conduktor Gateway routes traffic at the proxy layer, so rolling back to the original cluster is a config change, not an application redeploy. If a problem surfaces during cutover, you can revert in seconds and address the issue without losing time or data. ## Ready to plan your Kafka migration? See how Conduktor accelerates Kafka migration without disrupting running applications. Our team can help you map your current estate and plan a phased cutover that delivers value in weeks, not quarters. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-kafka-migration) [Download Solution Brief](https://www.conduktor.io/assets/pdfs/use-cases/migration-and-infrastructure-efficiency.pdf) ## Read more customer stories - [European Airline: 25 Clusters to Cloud](https://www.conduktor.io/customer-stories/leading-european-airline-migrates-kafka-to-confluent-cloud-with-conduktor) - [Smart Farming: 10x Kafka Utilization](https://www.conduktor.io/customer-stories/accelerating-smart-farming-innovation-with-conduktor-and-amazon-msk) - [Swiss Post: 5x Kafka Growth](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage) Source: https://www.conduktor.io/solutions/use-case/kafka-migration --- # Operational Efficiency *Cut Costs. Reduce Risk. Ship Faster.* Conduktor gives organizations the confidence to operate Kafka without the operational tax, and gives teams the speed and autonomy to deliver. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-operational-efficiency) [Download Solution Brief](https://www.conduktor.io/assets/Conduktor-Operational-Efficiency-Solution-Brief.pdf) Trusted by ## Kafka Adoption Has Outpaced Your Ability to Operate It The gap between what Kafka can do and how teams actually operate it compounds fast. Every question, change, and incident routes to the same central team, and the symptoms show up everywhere. - **No visibility stalls delivery and buries teams in escalations** — Without cluster visibility, developers fall back on CLI tools and tickets. A 10-minute consumer lag investigation becomes a half-day blocker. - **Broad access creates risk no one can trace** — Broad ACLs, no RBAC enforcement, and no audit trail across topics mean that when a security incident hits, no one can reconstruct what happened or who had access. - **Manual processes throttle adoption and overload the platform team** — Topic creation, schema changes, and ACL requests all route through one team. Without standardized GitOps or CI/CD, every change waits in the queue. ## Why Conduktor for Operational Efficiency - **Core Kafka Operations** — One interface for clusters, topics, schemas, and connectors across any Kafka distribution. No CLI required, no context switching between tools. [Learn more](https://www.conduktor.io/console#simplify-kafka-operations) - **Data Exploration** — Browse, filter, and tail messages in production. Debug issues in minutes instead of hours, without writing custom consumers or asking for access. [Learn more](https://www.conduktor.io/console#accelerate-troubleshooting) - **Observability** — Real-time monitoring, proactive alerts to Slack and PagerDuty, and risk scoring that catches problems before they reach production. [Learn more](https://www.conduktor.io/console#prevent-issues-before-production) - **Security & Access Control** — SSO, granular RBAC, field-level masking, and complete audit trails. Meet compliance requirements without blocking developer productivity. [Learn more](https://www.conduktor.io/console#secure-without-slowing-down) - **Insights** — Health optimization, governance visibility, cost attribution by team, and data quality monitoring across your entire Kafka estate. [Learn more](https://www.conduktor.io/console#drive-data-driven-decisions) - **Federated Ownership** — Developers provision resources within automated guardrails. Application catalog defines ownership. Approval workflows replace ticket queues. [Learn more](https://www.conduktor.io/console#scale-ownership-across-teams) - **Automation** — REST API, CLI, Terraform provider, and self-healing connectors. Manage Kafka as code through your existing CI/CD pipelines. [Learn more](https://www.conduktor.io/console#reduce-operational-burden) What Changes With Conduktor Five outcomes that compound across your organization. Ship FasterFaster debugging means faster releases. Every hour reclaimed from Kafka troubleshooting is an hour building what customers need. Boost ProductivityOne interface replaces fragmented CLI tools to understand your entire Kafka estate at a glance. Cut CostsEliminate internal tooling maintenance, reduce incident frequency, and stop spending on manual coordination that self-service removes. Reduce RiskGranular RBAC replaces blanket admin access, automated validation catches misconfigs before production, and audit trails generate continuously. Scale With the Same TeamSelf-service provisioning frees the platform team from being overloaded and an inadvertent bottleneck. The same team that supported five teams now supports fifty. vs. the Alternatives Why teams choose Conduktor over the common approaches. - **vs. CLI & Siloed Expertise** — SSH access and scattered scripts don't scale. Console gives every developer cluster visibility instantly. - **vs. Building Internal Tooling** — Homegrown dashboards become their own project. Console is purpose-built so your team ships product, not tooling. - **vs. Confluent Control Center** — Monitoring without the operational model. No self-service, no guardrails, and no way to move faster with control. Measurable Impact Real results from platform teams using Conduktor. 80% Faster MTTRDevelopers investigate issues themselves instead of escalating, cutting troubleshooting from hours to minutes. $500K Annual SavingsFrom productivity gains, fewer incidents, and less manual coordination. 5,000+ Hours Returned YearlyAcross debugging, provisioning, access requests, and incident response. 75% Faster Team OnboardingNew teams self-serve within guardrails instead of waiting weeks in ticket queues. Read more customer stories [Swiss Post scaled Kafka to 800+ users and 5x more applications with governance, RBAC, and self-service.Read Swiss Post's story →](https://www.conduktor.io/customer-stories/how-swiss-post-governs-democratizes-kafka-usage)[FlixBus scaled Kafka to 50+ teams and 2,300+ topics with a data mesh architecture and federated governance.Read FlixBus's story →](https://www.conduktor.io/customer-stories/flix)[Virgin Australia saves 300 hours a month — faster onboarding, simplified workflows, and broader Kafka adoption.Read Virgin Australia's story →](https://www.conduktor.io/customer-stories/virgin-australia-increases-operational-efficiency-and-kafka-adoption-with-conduktor) ## Frequently Asked Questions **Do I need Conduktor Gateway too, or is Console enough?** Console is the modern Kafka UI and API for visibility, ownership, and autonomy across your clusters. It delivers operational efficiency on its own. Conduktor Gateway, a Kafka proxy, is an abstraction layer that adds infrastructure-level capabilities like automated failover, migration, and third-party data sharing, without requiring application code changes. Many teams start with Console and add Conduktor Gateway when they need it. **What Kafka distributions does this work with?** Console works with any Kafka-compatible platform: Apache Kafka, Confluent Platform, Confluent Cloud, AWS MSK, Redpanda, Aiven, and self-managed deployments. You can manage multiple distributions from a single interface. **How long does it take to see value?** Most teams are operational within a day. Connect your clusters, configure SSO, and developers start using Console immediately. Self-service policies and ownership models typically roll out over the first few weeks. **How does this integrate with our existing observability stack?** Console integrates with Prometheus for metrics export and supports alerting to Slack, Microsoft Teams, PagerDuty, and webhooks. It complements your existing monitoring rather than replacing it, adding Kafka-specific context your general tools lack. **Can we start with one team and expand?** Yes. Most organizations start with a single cluster or team, prove the value, then expand. Console's RBAC and multi-tenancy mean you can onboard teams incrementally without disrupting existing users. **I have other questions?** [Drop us a line](https://www.conduktor.io/contact?src=solutions-use-case-operational-efficiency) and we'll get back to you as soon as possible. # Ready to take control of Kafka operations? See how Conduktor cuts costs, reduces risk, and helps you ship faster across your Kafka operations. [Talk to Us](https://www.conduktor.io/contact/demo?src=solutions-use-case-operational-efficiency) [Download Solution Brief](https://www.conduktor.io/assets/Conduktor-Operational-Efficiency-Solution-Brief.pdf) Source: https://www.conduktor.io/solutions/use-case/operational-efficiency --- # Data Lineage and Exactly-Once at Trayport | Abubakar Zahid | Voices of the Kafka Community [Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Easier Said Than Done in Kafka AZ Guest Abubakar Zahid Backend Engineer, Trayport Host [Ron Kapoor](https://www.conduktor.io/authors/ron-kapoor) Technical Product Marketing, Conduktor Abubakar Zahid is a backend engineer at Trayport, an energy-trading platform, where he has worked on the data streaming and analytics platform since 2023. Running Kafka at scale in a regulated industry means the hard problems find him, whether he goes looking for them or not. In this conversation →Running Kafka at scale on an energy-trading platform →Why data lineage is easier said than done in a regulated environment →Auditing for missing data, and why it's a hard question to solve →The day-one lesson: exactly-once holds inside Kafka, and gets very hard once data leaves "It is easier to guarantee exactly-once within the Kafka ecosystem. But once your data leaves that, it is very challenging. I wish I knew that before I started." Abubakar Zahid, Backend Engineer at Trayport Interview highlight Easier said than done The two problems that follow Abubakar around: proving where your data came from in a regulated environment, and holding exactly-once once data leaves Kafka. [← More Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Source: https://www.conduktor.io/voices-of-the-kafka-community/abubakar-zahid --- # Optimizing Kafka Throughput at Human Security | Adrian Woodhead | Voices of the Kafka Community [Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Optimizing Kafka for Throughput Guest Adrian Woodhead Distinguished Engineer, Human Security Host [Ron Kapoor](https://www.conduktor.io/authors/ron-kapoor) Technical Product Marketing, Conduktor Adrian Woodhead is a Distinguished Engineer at Human Security, where bot-detection signal data from across the web flows through Kafka to decouple their front-end and back-end systems. He wrote up the throughput experiment in this conversation on the [Human Security engineering blog](https://www.humansecurity.com/tech-engineering-blog/optimizing-kafka-for-throughput/). In this conversation →How Human Security uses Kafka to decouple bot-detection signal data →Building a load-test harness to emulate production traffic →Why gzip pinned CPU through thread contention under load →How switching to the less efficient snappy raised throughput →The benchmarking discipline that made the result trustworthy "Snappy is less efficient than gzip, but it uses the CPU less. And we got much higher throughput." Adrian Woodhead, Distinguished Engineer at Human Security [← More Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Source: https://www.conduktor.io/voices-of-the-kafka-community/adrian-woodhead --- # Kafka's Role in AI Acceleration, and Open Formats at BNY | Ash Goel | Voices of the Kafka Community [Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Cutting Through the AI Hype AG Guest Ash Goel Data Engineering Lead, BNY Host [Ron Kapoor](https://www.conduktor.io/authors/ron-kapoor) Technical Product Marketing, Conduktor Ash Goel heads up a data engineering team at BNY. He spends a lot of time across the data ecosystem and its events, which gives him a grounded read on where the hype is regarding AI. In this conversation →Why he's a self-described "doomer" on the IBM acquisition of Confluent →Whether Kafka is really the backbone of AI yet →Why "we want to do AI" is a starting point, not a strategy →Starting from the problem instead of the buzzword →Convergence on open formats like Iceberg as the real enabler "They say, we want to do AI. And I say, tell me what you want to do. Then we can see how to fix it." Ash Goel, Data Engineering Lead at BNY Interview highlight "We want to do AI" is not a strategy Every platform claims to be the place to do AI. Ash's response is to ask what you actually want to do first, and work back from there. [← More Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Source: https://www.conduktor.io/voices-of-the-kafka-community/ash-goel --- # Kafka, AI Agents, and Data Quality at LinkedIn | Harshada Yesane | Voices of the Kafka Community [Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Kafka, AI Agents, and Data Quality at LinkedIn Guest Harshada Yesane Sr. Software Engineer, LinkedIn Host [Ron Kapoor](https://www.conduktor.io/authors/ron-kapoor) Technical Product Marketing, Conduktor Harshada Yesane is a Senior Software Engineer at LinkedIn, where she works on the open-source Brooklin project and Kafka at scale, and is exploring AI agents to support on-call rotations. She recently gave a talk at Current London on [building reliable CDC at trillion-message scale](https://current.confluent.io/london/sessions#session-SESS-237). In this conversation →Working on LinkedIn's open-source Brooklin project and Kafka at scale →Building AI agents to take some of the weight off 3am on-call →Why the agent is the easy part, and the data underneath is the hard part →Data freshness versus data correctness when hundreds of systems depend on you →Why she still calls Kafka paramount for any streaming solution "Building an AI agent is not a difficult thing. The data we have, we have to make sure it's very clean and appropriate, so our agents are more intelligent on top of it." Harshada Yesane, Sr. Software Engineer at LinkedIn Interview highlight Data correctness is the hard part Building the agent, or the data pipeline that feeds it, was never the hard part. Ensuring that data is correct, in real time, is. [← More Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Source: https://www.conduktor.io/voices-of-the-kafka-community/harshada-yesane --- # Kafka's Tricky Parts: Auth, Hot Topics, and Partitions | Hugh Evans | Voices of the Kafka Community [Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Kafka's Tricky Parts Guest Hugh Evans Senior Developer Advocate, Aiven Host [Ron Kapoor](https://www.conduktor.io/authors/ron-kapoor) Technical Product Marketing, Conduktor Hugh Evans is a Senior Developer Advocate at Aiven, where he looks after the Kafka community and runs the Get Kafka-Nated podcast and newsletter. Before advocacy he was a platform engineer building on Kafka himself. Safe to say he's shipped it, and now he's helping the community make sense of it. In this conversation →Why there is no magic number of partitions →Sizing partitions to your consumers and producers, not a target number →The overhead and confusion that over-partitioning creates →The first wall most people hit: authentication and unintuitive errors →Understanding Kafka properly before you design a system on top of it "Troubleshooting Kafka performance issues on hot topics can be tough. You've got to know what you're looking for, but get advice from other people in the community, look at stuff online, and you can pretty quickly start building that toolset for finding the sticking points in your deployment." Hugh Evans, Senior Developer Advocate at Aiven Interview highlight There is no magic number You need roughly as many partitions as you have consumers and producers. Chasing a magic number just adds overhead and confusion. [← More Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Source: https://www.conduktor.io/voices-of-the-kafka-community/hugh-evans --- # Kafka Architecture and Its Limits at Fortum | Jagadeesh Rajah | Voices of the Kafka Community [Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Kafka Architecture and Its Limits Guest Jagadeesh Rajah Kafka Architect, Fortum Host [Ron Kapoor](https://www.conduktor.io/authors/ron-kapoor) Technical Product Marketing, Conduktor Jagadeesh Rajah architects Kafka at Fortum, the Finnish energy company spanning B2B, B2C, and recycling. He came to it from an integration background, after Azure integration proved too slow for what his users needed. That experience shapes how he thinks about where one platform ends and another begins. In this conversation →Why Fortum moved to Confluent Kafka →Where Kafka's future sits with AI, MCP, and context →What good Kafka architecture looks like →His hot take: no single platform, Confluent included, can solve everything →Where API management, throttling, and a listening layer take over "Confluent cannot solve everything. For example, throttling. If you want to avoid certain users bombarding a topic, you should have throttling with API management." Jagadeesh Rajah, Fortum Interview highlight One platform can't do everything Throttling, traffic shaping, knowing your consumers. Those belong to an API-management layer, not the streaming platform. [← More Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Source: https://www.conduktor.io/voices-of-the-kafka-community/jagadeesh-rajah --- # Why Kafka Platforms Stall on People, Not Tech | Stephan Schiffner | Voices of the Kafka Community [Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) The Bottleneck Is Not the Technology SS Guest Stephan Schiffner CTO, Steadforce Host [Ron Kapoor](https://www.conduktor.io/authors/ron-kapoor) Technical Product Marketing, Conduktor Stephan Schiffner is the CTO at Steadforce, a Munich-based tech consultancy working in platform engineering, data streaming, and AI. He has watched Kafka platforms get built across a lot of client environments, and the blocker that shows up after the first version ships is rarely the technology. In this conversation →Why Kafka is the basis of many platforms, but never a solution alone →The real challenge after the first version ships: onboarding the business units →How internal IT becomes the bottleneck for the platform's own customers →The remedy: clear prioritization, good onboarding materials, and a genuinely self-service platform "The internal IT is kind of a bottleneck, because they don't manage to build all the requirements that are coming from the business units." Stephan Schiffner, CTO at Steadforce [← More Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Source: https://www.conduktor.io/voices-of-the-kafka-community/stephan-schiffner --- # How Fresh Does Your Data Really Need To Be | Vinzenz Halhammer | Voices of the Kafka Community [Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Not Everything Belongs in Kafka VH Guest Vinzenz Halhammer Data Engineer, Steadforce Host [Ron Kapoor](https://www.conduktor.io/authors/ron-kapoor) Technical Product Marketing, Conduktor Vinzenz Halhammer is a data engineer at Steadforce, a Munich-based IT consultancy, where he builds streaming applications for energy providers and grid operators. His definition of good Kafka architecture starts before any design work: evaluate what you actually want to achieve. In this conversation →Building streaming applications for energy providers and grid operators →Why good architecture starts with evaluating what you want to achieve →The freshness question: millisecond latency vs hourly or daily workloads →How matching data freshness to the use case improves efficiency and cost "Maybe I have some application where I need sub-second or millisecond latency. But maybe I also have a lot of applications, often in BI, where daily is enough, or hourly. So that I don't waste my precious Kafka resources." Vinzenz Halhammer, Data Engineer at Steadforce [← More Voices of the Kafka Community](https://www.conduktor.io/voices-of-the-kafka-community) Source: https://www.conduktor.io/voices-of-the-kafka-community/vinzenz-halhammer --- # Voices of the Kafka Community Kafka war stories, hot takes, and hard-won lessons from real practitioners. [Kafka, AI Agents, and Data Quality at LinkedInSee what Harshada Yesane, a Senior Software Engineer on LinkedIn's Brooklin project, has to say about her experience building AI agents.](https://www.conduktor.io/voices-of-the-kafka-community/harshada-yesane) [Kafka's Tricky Parts: Auth, Hot Topics, and PartitionsSee what Hugh Evans, a Senior Developer Advocate at Aiven, has to say about the parts of Kafka that are easier said than done.](https://www.conduktor.io/voices-of-the-kafka-community/hugh-evans) [Kafka's Role in AI Acceleration, and Open Formats at BNYSee what Ash Goel, who heads a data engineering team at BNY, has to say about cutting through the AI hype.](https://www.conduktor.io/voices-of-the-kafka-community/ash-goel) [Kafka Architecture and Its Limits at FortumSee what Jagadeesh Rajah, who architects Kafka at Fortum, has to say about why one platform can't solve everything.](https://www.conduktor.io/voices-of-the-kafka-community/jagadeesh-rajah) [Data Lineage and Exactly-Once at TrayportSee what Abubakar Zahid, a backend engineer at Trayport, has to say about data lineage and exactly-once semantics.](https://www.conduktor.io/voices-of-the-kafka-community/abubakar-zahid) [Optimizing Kafka Throughput at Human SecuritySee what Adrian Woodhead, a Distinguished Engineer at Human Security, has to say about why the less efficient compression won on throughput.](https://www.conduktor.io/voices-of-the-kafka-community/adrian-woodhead) [Why Kafka Platforms Stall on People, Not TechSee what Stephan Schiffner, the CTO at Steadforce, has to say about why internal IT becomes the bottleneck.](https://www.conduktor.io/voices-of-the-kafka-community/stephan-schiffner) [How Fresh Does Your Data Really Need To BeSee what Vinzenz Halhammer, a data engineer at Steadforce, has to say about what deserves your Kafka resources.](https://www.conduktor.io/voices-of-the-kafka-community/vinzenz-halhammer) Source: https://www.conduktor.io/voices-of-the-kafka-community --- # Access Control for Streaming: Securing Kafka Topics and Consumer Groups In traditional database systems, access control is straightforward: you grant users permissions to read, write, or modify specific tables. But streaming platforms work differently. Data flows continuously through topics, multiple consumers may read the same stream simultaneously, and producers can inject events at any moment. This creates unique security challenges. Consider a financial services company streaming payment transactions. The fraud detection team needs read access, the analytics team needs different data, and the payment processing service needs write access. Without proper access control, a misconfigured consumer could read sensitive customer data, or a rogue producer could inject fraudulent transactions. Unlike databases where access patterns are relatively static, streaming platforms serve multiple teams with different security clearances, compliance requirements, and data access needs. A single topic might contain data that some teams can read entirely, others can read partially, and some shouldn't access at all. This is where streaming-specific access control becomes critical. ## Why Access Control Matters in Streaming ![Access Control Architecture for Streaming Platforms](https://www.conduktor.io/assets/images/glossary/access-control-for-streaming-0.webp) ## Kafka ACLs: The Foundation Apache Kafka implements access control through Access Control Lists (ACLs). An ACL defines **who** (principal) can perform **which operations** (Read, Write, Create, etc.) on **which resources** (topics, consumer groups, cluster). Key concepts: - **Default-deny model**: Without explicit ACLs, authenticated users have no permissions - **Combining permissions**: Consumers need Read on both topic AND consumer group; producers need Write on topics plus IdempotentWrite for exactly-once semantics - **Prefixed patterns**: Use `team-alpha.*` to grant access to all topics with that prefix, dramatically reducing ACL management overhead For comprehensive coverage of ACL components, authorization flows, and production management strategies, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). ## Authentication: Proving Identity Before Kafka can enforce ACLs, it must verify identity through authentication. Kafka supports several mechanisms including SASL/PLAIN, SASL/SCRAM, mutual TLS (mTLS), and SASL/OAUTHBEARER. For detailed coverage of authentication mechanisms, configuration examples, and security tradeoffs, see [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). Most enterprises combine authentication methods: mTLS for internal services, OAuth for user-facing applications, and SASL/SCRAM for legacy systems. ## Getting Started: Enabling ACLs and First Steps Before using ACLs, you must enable authorization in your Kafka cluster by setting the `authorizer.class.name` property in `server.properties`: ```properties # For KRaft mode (Kafka 3.0+, default in Kafka 4.0+) authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer # For ZooKeeper mode (legacy, replaced by KRaft in Kafka 4.0+) authorizer.class.name=kafka.security.authorizer.AclAuthorizer # Configure super users who bypass ACL checks (use carefully) super.users=User:admin;User:kafka ``` After enabling ACLs, all access is denied by default. Use the `kafka-acls` command-line tool to grant permissions. For complete CLI examples covering consumer access, producer permissions, prefixed patterns, and ACL management, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). ## Authorization Patterns and Granularity Kafka supports permissions at multiple levels: cluster-wide administrative operations, topic-level read/write access, consumer group membership controls, and transactional ID permissions for exactly-once semantics. Common patterns: - **Producers**: Write on topics + IdempotentWrite for exactly-once - **Consumers**: Read on topics + Read on consumer groups - **Admins**: Create/Delete/Alter on cluster resources For team-based access patterns, environment separation strategies, and service-based permission models, see the authorization patterns section in [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). ## Multi-Tenancy and Isolation Multi-tenant streaming platforms serve multiple teams or business units on shared infrastructure. Access control becomes critical for isolation, preventing one tenant from accessing another's data. The simplest approach uses topic prefixes: `team-alpha.*`, `team-beta.*`. Each team gets Read and Write permissions only on their prefix. However, this doesn't prevent teams from seeing topic names or metadata from other teams. More sophisticated multi-tenancy requires: - **Namespace isolation**: Logical separation where teams cannot even discover other teams' resources. This requires custom authorizers or platform layers above Kafka. - **Quota management**: Per-tenant throughput quotas prevent noisy neighbors from monopolizing cluster resources. - **Separate security principals**: Each team uses different SASL credentials or client certificates, making audit trails clearer. - **Metadata filtering**: Custom authorizers that filter Describe operations, so teams only see their own topics. Multi-tenancy also impacts operational tools. Monitoring dashboards, schema registries, and stream processing frameworks all need tenant-aware access control to prevent information leakage. ### Virtual Clusters for Simplified Multi-Tenant Access Control Managing access control across multiple teams sharing a Kafka cluster requires careful ACL coordination that becomes unwieldy at scale. Conduktor Virtual Clusters simplify this by creating complete logical isolation within a single physical cluster, each tenant operates in what appears to be a dedicated Kafka environment with their own topics, consumer groups, and access policies, eliminating cross-tenant visibility entirely. Virtual Clusters enable per-tenant security postures where each virtual cluster enforces its own authentication requirements and ACL policies. Platform teams can delegate access control management to business unit administrators who manage permissions within their virtual cluster without requiring central team intervention for every change. For implementation guidance, see the [Virtual Clusters documentation](https://docs.conduktor.io/guide/conduktor-concepts/virtual-clusters). ## Enterprise Integration and Modern Approaches Enterprise environments demand integration with existing identity systems. Rather than managing Kafka-specific credentials, organizations want to use LDAP, Active Directory, or cloud identity providers. **LDAP/AD integration** typically works through SASL/PLAIN or SASL/GSSAPI (Kerberos). Kafka validates credentials against the directory service, and group memberships can inform authorization decisions through custom authorizers. ### From ACLs to RBAC While Kafka ACLs provide fine-grained control, managing thousands of individual ACL entries becomes unwieldy at enterprise scale. **Role-Based Access Control (RBAC)** solves this by introducing an abstraction layer above ACLs. Instead of granting permissions directly to users: ``` User:alice -> Read on topic:orders User:bob -> Read on topic:orders User:charlie -> Read on topic:orders ``` RBAC uses roles: ``` Role:data-analyst -> Read on topic:orders User:alice -> Role:data-analyst User:bob -> Role:data-analyst User:charlie -> Role:data-analyst ``` Benefits of RBAC: - **Simplified management**: Grant role to user instead of maintaining dozens of ACLs per user - **Consistency**: All analysts get identical permissions through the same role - **Auditability**: "Who has access to orders?" becomes "Who has the data-analyst role?" - **Onboarding/offboarding**: Add/remove role assignment, not individual ACLs **Conduktor** provides [enterprise RBAC for Kafka](https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac) that extends beyond Confluent's implementation. It enables organizations to: - Define custom roles aligned with organizational structure (data-engineer, ML-researcher, compliance-auditor) - Integrate with SSO providers (Okta, Azure AD) for centralized identity management - Apply RBAC across multiple Kafka clusters from a single interface - Enforce policies at the topic, consumer group, and schema level - Enable **self-service access requests**: Developers request roles/permissions through UI, managers approve via workflow, permissions auto-provision - Maintain audit trails showing role assignments and permission changes over time This becomes essential for regulated industries where proving "who had access to what data, when" is a compliance requirement. Self-service capabilities reduce bottlenecks on platform teams while maintaining governance controls, developers get access in minutes instead of waiting days for manual ACL configuration. - **Policy-as-code** treats access control definitions as version-controlled configuration. Tools like Terraform or GitOps workflows define ACLs and roles declaratively, ensuring changes are reviewed, auditable, and reproducible. This prevents permission sprawl where access accumulates without oversight. - **Attribute-based access control (ABAC)** makes authorization decisions based on attributes like time of day, client IP, data classification, or user department. While Kafka doesn't natively support ABAC, custom authorizers can implement these policies. ## Securing Your Streaming Architecture Access control is foundational to streaming platform security. Start by enabling authentication, implement least-privilege ACLs, and integrate with enterprise identity systems. As your platform grows, invest in RBAC abstractions, policy-as-code workflows, and automated access provisioning. Remember that access control is just one layer of defense. Combine it with encryption in transit (TLS), encryption at rest, network segmentation, and data masking for sensitive fields. Monitor authorization failures as potential security incidents. The streaming paradigm's continuous data flow and multiple consumers make access control more complex than traditional databases, but Kafka's ACL model provides the flexibility to implement enterprise-grade security. With proper planning and tooling, you can build a streaming platform that serves multiple teams securely. For production deployments, consider platforms that provide unified governance across your streaming infrastructure, making it easier to manage access control, monitor compliance, and maintain security as your streaming architecture grows. ## Related Concepts - [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) - Detailed implementation patterns for Kafka's native ACL system and modern authorization approaches. - [Data Access Control: RBAC and ABAC](https://www.conduktor.io/glossary/data-access-control-rbac-and-abac) - Role-based and attribute-based access control models for streaming platforms. - [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments) - Access control strategies for shared Kafka infrastructure serving multiple teams. ## Sources and References - [Apache Kafka Security Documentation](https://kafka.apache.org/documentation/#security) - Official documentation on Kafka's security features including authentication, authorization, and encryption - [Confluent RBAC for Apache Kafka](https://docs.confluent.io/platform/current/security/rbac/index.html) - Guide to Role-Based Access Control implementation on Confluent Platform - [Kafka Authorization and ACLs](https://docs.confluent.io/platform/current/kafka/authorization.html) - Comprehensive guide to Kafka Access Control Lists and authorization models - [SASL/OAUTHBEARER in Apache Kafka](https://cwiki.apache.org/confluence/display/KAFKA/KIP-255+-+OAuth+Authentication+via+SASL%2FOAUTHBEARER) - Kafka Improvement Proposal for OAuth 2.0 authentication integration - [AWS MSK Security Best Practices](https://docs.aws.amazon.com/msk/latest/developerguide/security.html) - Security best practices for Amazon Managed Streaming for Apache Kafka Source: https://www.conduktor.io/glossary/access-control-for-streaming --- # Agentic AI Pipelines: Streaming Data for Autonomous Agents The evolution from traditional machine learning models to autonomous AI agents represents a fundamental shift in how we build intelligent systems. While traditional ML models wait passively for inputs and return predictions, AI agents actively pursue goals, make multi-step decisions, and take actions in their environment. This autonomy creates new demands on data infrastructure, particularly the need for continuous, high-quality streaming data that provides agents with fresh context for decision-making. An agentic AI pipeline connects real-time operational data streams to autonomous agents, enabling them to react to events, retrieve relevant context, reason through problems, and execute actions, all while maintaining the data quality and governance controls essential for reliable AI operations. ## Key Concepts: Understanding the Components Key concepts to understand: - **RAG (Retrieval Augmented Generation)**: A technique where language models retrieve relevant information from external knowledge sources before generating responses. Instead of relying solely on training data, RAG systems query databases or document stores to ground their outputs in factual, up-to-date information. - **Vector Database**: A specialized database that stores data as high-dimensional vectors (embeddings) and enables similarity searches. When you ask "what are our return policies?", the vector database finds documents with semantically similar content, even if they don't contain the exact words you used. - **Embeddings**: Numerical representations of text, images, or other data that capture semantic meaning. Similar concepts have similar embeddings, enabling similarity searches. For example, "refund" and "money back" have embeddings close to each other in vector space. - **Context Window**: The maximum amount of text a language model can process at once, measured in tokens. Models like GPT-4 have context windows of 128,000 tokens (~100,000 words). When relevant information exceeds this limit, agents must select which context to include. - **Hallucination**: When language models generate plausible-sounding but factually incorrect information. Agents using stale or missing data are more prone to hallucination since they lack grounding facts. ## Understanding AI Agents: Beyond Traditional ML AI agents differ from traditional machine learning models in three critical ways: - **Autonomy**: Rather than simply mapping inputs to outputs, agents operate independently to achieve specified goals. They can break down complex objectives into subtasks, plan sequences of actions, and adapt their approach based on results. - **Tool Use**: Agents interact with external systems through tools, APIs, databases, search engines, or other services. This allows them to gather information, perform calculations, and take actions in the real world, extending their capabilities far beyond the model's training data. - **Multi-Step Reasoning**: Instead of single-shot inference, agents engage in iterative reasoning loops. They observe their environment, plan actions, execute them, and reflect on outcomes, often across multiple cycles before completing a task. Consider a customer service agent: it doesn't just classify support tickets. It reads the ticket, searches the knowledge base, retrieves the customer's order history, drafts a response, checks company policies, and sends the reply, all autonomously. Each step requires fresh, accurate data. ## Why Streaming Data Powers Agentic AI Agents depend on real-time operational data for three essential functions: - **Fresh Context**: An agent's decisions are only as good as its context. Stale data leads to incorrect assumptions and poor actions. When a trading agent analyzes market conditions, five-minute-old prices might as well be ancient history. When a logistics agent routes deliveries, it needs current traffic data, not yesterday's patterns. - **Event-Driven Triggers**: Agents don't operate on fixed schedules, they respond to events. A fraud detection agent springs into action when suspicious transactions occur. An incident response agent activates when system metrics cross thresholds. These triggers flow through streaming data pipelines, often from Kafka topics or event streams. - **Continuous Learning Loop**: Agents improve through feedback. Each action produces outcomes that inform future decisions. This requires streaming the results of agent actions back into the pipeline, creating a closed loop where agents learn from their operational performance in near real-time. Without streaming data, agents operate on stale assumptions, miss critical events, and fail to adapt to changing conditions. ## Agentic AI Pipeline Architecture A production agentic AI pipeline follows a consistent pattern: ![A production agentic AI pipeline follows a consistent pattern](https://www.conduktor.io/assets/images/glossary/agentic-ai-pipelines-0.webp) **1. Event Ingestion** Streaming platforms like Apache Kafka capture operational events: user actions, system metrics, transactions, sensor readings. These events flow continuously into topics organized by domain (orders, inventory, interactions). **2. Context Enrichment** Raw events trigger context retrieval. The agent needs more than just "order cancelled", it needs customer history, cancellation reasons, inventory status. This often involves: - Querying vector databases for semantic similarity searches - Retrieving from key-value stores for fast lookups - Joining event streams for correlated data - Accessing RAG (Retrieval Augmented Generation) systems for relevant documents **3. Agent Execution** The enriched context flows to the agent, which: - Analyzes the situation using its language model - Reasons through potential actions - Consults tools (APIs, databases, calculators) - Makes decisions based on its goal - Generates actions (API calls, messages, database updates) **4. Action Execution** Agent decisions become concrete actions streamed back into the operational environment: send an email, update a database, trigger a workflow, create a ticket. **5. Feedback Loop** Action outcomes stream back as new events, providing the agent with feedback and enabling monitoring systems to track agent performance, data quality, and business impact. ```python # Production agent loop with error handling and structured data from kafka import KafkaConsumer, KafkaProducer from typing import Dict, List, Any import json from datetime import datetime kafka_consumer = KafkaConsumer( 'support-tickets', bootstrap_servers='localhost:9092', group_id='customer-service-agent', value_deserializer=lambda m: json.loads(m.decode('utf-8')) ) kafka_producer = KafkaProducer( bootstrap_servers='localhost:9092', value_serializer=lambda m: json.dumps(m).encode('utf-8') ) def retrieve_context(event: Dict[str, Any], vector_db, redis) -> Dict[str, Any]: """ Retrieve enriched context from multiple sources. Returns: { 'customer_history': [...], 'similar_tickets': [...], 'knowledge_base': [...], 'product_info': {...} } """ ticket_id = event['ticket_id'] customer_id = event['customer_id'] # Fast lookup: customer data from Redis cache customer_history = redis.get(f"customer:{customer_id}") # Semantic search: find similar resolved tickets query_embedding = embed_text(event['description']) similar_tickets = vector_db.query( vector=query_embedding, top_k=5, filter={"status": "resolved"} ) # Semantic search: relevant knowledge base articles kb_articles = vector_db.query( vector=query_embedding, top_k=3, filter={"type": "knowledge_base"} ) return { 'customer_history': customer_history, 'similar_tickets': [t['metadata'] for t in similar_tickets['matches']], 'knowledge_base': [kb['metadata']['text'] for kb in kb_articles['matches']], 'product_info': event.get('product_id') } # Main agent processing loop for message in kafka_consumer: event = message.value try: # Step 1: Retrieve enriched context context = retrieve_context(event, vector_db, redis) # Step 2: Agent reasoning with structured input agent_input = { "ticket": event, "customer_history": context['customer_history'], "similar_cases": context['similar_tickets'], "knowledge_base": context['knowledge_base'], "goal": "resolve customer issue with empathy and accuracy" } # Agent returns structured output result = agent.invoke(agent_input) # result = { # "action": { # "type": "send_email", # "to": "customer@example.com", # "subject": "...", # "body": "..." # }, # "reasoning": "Customer reported missing item. Similar cases show...", # "confidence": 0.87, # "tools_used": ["email_api", "refund_system"] # } # Step 3: Stream actions to appropriate topics kafka_producer.send('agent-actions', { 'ticket_id': event['ticket_id'], 'action': result['action'], 'timestamp': datetime.utcnow().isoformat() }) # Step 4: Stream audit logs for governance kafka_producer.send('agent-audit-logs', { 'ticket_id': event['ticket_id'], 'agent_id': 'customer-service-v2', 'reasoning': result['reasoning'], 'confidence': result['confidence'], 'context_sources': list(context.keys()), 'timestamp': datetime.utcnow().isoformat() }) # Step 5: Stream feedback for learning kafka_producer.send('agent-feedback', { 'ticket_id': event['ticket_id'], 'action_type': result['action']['type'], 'confidence': result['confidence'] }) except Exception as e: # Dead letter queue for failed events kafka_producer.send('agent-dlq', { 'original_event': event, 'error': str(e), 'timestamp': datetime.utcnow().isoformat() }) ``` This architecture requires **low latency** (agents can't wait minutes for context), **high throughput** (supporting many concurrent agents), and **exactly-once semantics** (duplicate actions can cause serious problems). ## Data Requirements for Agent Reliability Agent quality correlates directly with data quality. Several requirements stand out: - **Latency**: Context retrieval must complete in milliseconds to seconds. An agent that waits 30 seconds for customer data provides a poor experience. Streaming platforms with low-latency connectors and fast vector databases (Pinecone, Weaviate, Milvus) are essential. - **Freshness**: Embeddings in vector stores must reflect current documents. If your knowledge base updates but embeddings lag by hours, agents retrieve outdated information. Streaming pipelines should trigger re-embedding when source documents change. - **Completeness**: Missing data forces agents to make assumptions or fail entirely. If inventory data is incomplete, an order fulfillment agent might promise products that aren't available. - **Consistency**: Agents often pull from multiple data sources. Inconsistencies between systems lead to contradictory context and confused reasoning. Change Data Capture (CDC) streams help maintain consistency by propagating updates across systems in real-time. ## Real-World Use Cases - **Customer Service Automation**: Agents handle support tickets by streaming in ticket events, retrieving customer history and product documentation, generating responses, and updating CRM systems, all autonomously for routine issues. - **Operational Monitoring**: DevOps agents monitor metric streams, detect anomalies, correlate across services, investigate root causes by querying logs and traces, and execute remediation workflows. - **Financial Trading**: Trading agents consume market data streams, analyze patterns, evaluate risk against portfolio state, and execute trades, with latency measured in milliseconds. - **Supply Chain Optimization**: Logistics agents process shipment events, weather data, and traffic conditions to dynamically reroute deliveries, reorder inventory, and optimize warehouse operations. Each use case demonstrates the same pattern: streaming events trigger agents that retrieve fresh context, reason through problems, and take actions that feed back into operational systems. ## Getting Started: Building Your First Agent For teams new to agentic AI, start with these recommended frameworks and a phased approach: ### Recommended Frameworks - **LangChain**: The most popular framework for building AI agents. Provides abstractions for agents, tools, memory, and chains. Excellent for prototyping and has extensive community support. - **LangGraph**: Extension of LangChain that adds explicit graph-based agent workflows. Better for complex multi-step agents where you want control over the reasoning flow. - **AutoGPT / AutoGen**: Frameworks for fully autonomous agents that plan and execute complex tasks with minimal human intervention. More experimental but powerful for research. - **LlamaIndex**: Specialized for RAG and data retrieval. Excellent when your agents primarily need to query and reason over documents. ### Phased Implementation Approach **Phase 1: Read-Only Agent (Low Risk)** Start with agents that only read data and provide recommendations without taking actions: - Monitor event streams and flag anomalies for human review - Answer questions from knowledge bases without modifying data - Provide draft responses that humans approve before sending **Phase 2: Constrained Actions (Medium Risk)** Add autonomous actions with strict guardrails: - Auto-respond to FAQs with approved templates - Update low-risk database fields (tags, categories) - Send notifications with bounded content **Phase 3: Autonomous Agents (High Risk)** Deploy fully autonomous agents for high-stakes operations: - Execute financial transactions - Make customer commitments - Modify production systems Each phase builds confidence in data quality, agent reliability, and governance processes before increasing autonomy. ### Essential Skills for Agent Teams Building production agentic AI pipelines requires: - **Streaming expertise**: Kafka, Flink, or similar platforms - **ML/AI knowledge**: Understanding LLM capabilities, limitations, prompt engineering - **Data engineering**: ETL, CDC, real-time data quality - **Software engineering**: Robust error handling, monitoring, distributed systems - **Domain expertise**: Understanding the business context where agents operate ## Challenges in Production Agentic Pipelines **Hallucination Risk**: Language models can generate plausible but incorrect information. When agents act on hallucinations, the consequences are real, wrong answers to customers, incorrect data updates, or misguided actions. *Mitigation strategies*: - Ground responses in retrieved facts from streaming data (RAG) - Add validation steps that verify agent outputs against known constraints - Implement confidence scoring and route low-confidence decisions to human review - Use structured output formats that constrain generation **Stale Data Impact**: Vector databases with outdated embeddings cause agents to retrieve irrelevant context. Even short delays (minutes to hours) degrade agent performance. *Mitigation strategies*: - Implement CDC streams from source systems to vector databases - Add timestamps to embeddings and filter by recency - Monitor embedding freshness metrics and alert on staleness - Use hybrid search (semantic + keyword) to catch recent exact matches **Context Window Limits**: Agents have finite context windows. When relevant information exceeds this limit, agents must summarize or select, potentially losing critical details. *Mitigation strategies*: - Rank retrieved context by relevance and include only top results - Use map-reduce patterns: summarize chunks first, then reason over summaries - Implement progressive context loading: start with essentials, add details if needed - Consider agents with larger context windows (Claude 3.5 Sonnet: 200K tokens) **Reliability and Error Handling**: Agents interact with many external systems, each of which can fail. Pipelines need graceful degradation. *Mitigation strategies*: - Implement circuit breakers for external API calls - Use exponential backoff for retries with maximum attempt limits - Route failed events to dead letter queues for manual review - Provide fallback responses when systems are unavailable - Set aggressive timeouts to prevent hanging on slow dependencies **Cost and Resource Management**: Running large language models for every event is expensive. A production agent handling 1M events/day can cost $10K+/month in LLM API fees. *Mitigation strategies*: - Batch similar requests when latency allows - Use smaller/cheaper models for simple classification tasks (routing, filtering) - Implement model routing: Claude Haiku for simple tasks, Opus for complex reasoning - Cache common patterns (FAQ responses, standard procedures) - Consider self-hosted models for high-volume, low-complexity scenarios ## Governance for Autonomous Agents Autonomous systems that take real actions demand rigorous governance: - **Action Auditing**: Every agent decision and action must be logged with full context: what data was retrieved, what reasoning was applied, what action was taken, and what outcome occurred. Streaming these audit logs to governance platforms enables compliance, debugging, and quality improvement. - **Guardrails and Constraints**: Agents need boundaries. A customer service agent shouldn't offer unlimited refunds. A trading agent needs position limits. These guardrails can be implemented through: - Pre-execution validation (checking actions against policies) - Streaming policy updates (real-time constraint propagation) - Post-execution review (flagging policy violations) - **Human-in-the-Loop Workflows**: Critical decisions should route to humans for approval. Streaming architectures support this through event-driven workflows: agent proposes action → streams to review queue → human approves/rejects → action executes or cancels. - **Data Quality Monitoring**: Agent performance correlates with data quality. Tracking metrics like embedding freshness, context retrieval latency, and data completeness helps identify when degraded data quality impacts agent decisions. Tools like **Conduktor** provide centralized governance for streaming data platforms, enabling teams to monitor data quality, enforce policies, and audit data access across the agentic AI pipeline, ensuring that agents operate on trusted, well-governed data streams. For data quality validation and testing, see [Conduktor's Data Quality capabilities](https://docs.conduktor.io/guide/use-cases/observe-data-quality). ## Integration Patterns: RAG and Vector Stores Most production agentic AI pipelines integrate with Retrieval Augmented Generation (RAG) systems: - **Streaming to Vector Stores**: As documents change (products updated, policies revised, knowledge articles published), CDC streams capture these changes and trigger re-embedding. The fresh embeddings stream into vector databases, ensuring agents retrieve current information. - **Hybrid Context Retrieval**: Agents often combine: - Semantic search (vector similarity for conceptual matches) - Keyword search (traditional indexes for exact matches) - Structured queries (databases for transactional data) - Time-series data (metrics streams for temporal context) - **Context Caching**: Frequently accessed context (product catalogs, common FAQs) can be cached, but caches must invalidate when source data streams updates. Event-driven cache invalidation keeps context fresh without constant re-retrieval. - **Feedback for Retrieval Improvement**: When agents mark retrieved context as helpful or unhelpful, these signals stream back to improve retrieval rankings, creating a continuous learning loop for the retrieval system itself. ## Monitoring and Observability Operating agentic AI pipelines requires monitoring across multiple dimensions: **Agent Performance Metrics**: - Task success rate - Action accuracy - Reasoning quality (human evaluation samples) - Goal completion time **Data Quality Metrics**: - Context retrieval latency - Embedding freshness - Data completeness scores - Stream lag and throughput - **Correlation Analysis**: The key insight comes from correlating these metrics. When does data staleness impact agent success rates? How does retrieval latency affect user satisfaction? These correlations guide infrastructure investment and quality improvements. - **Anomaly Detection**: Sudden changes in agent behavior often indicate data quality issues. If success rates drop 20%, check for stale embeddings, incomplete data, or stream processing delays. ## Summary Agentic AI pipelines combine streaming data platforms, vector databases, and large language models into autonomous systems that perceive, reason, and act. Agent quality depends directly on data quality: the freshness, completeness, and reliability of the streaming context that informs their decisions. Building production agentic AI pipelines requires: - Low-latency streaming infrastructure (Apache Kafka, Apache Flink) - Fast context retrieval (vector databases, caching layers) - Robust agent orchestration (error handling, retries, guardrails) - Governance (auditing, policies, human oversight) - Continuous monitoring (data quality, agent performance, correlation) The streaming data pipelines powering these agents are critical infrastructure and require the same engineering rigor and governance discipline as any mission-critical system. ## Related Concepts - [Streaming Data Pipeline](https://www.conduktor.io/streaming-data-pipeline) - Build data pipelines for autonomous agents - [Real-Time Analytics with Streaming Data](https://www.conduktor.io/real-time-analytics-with-streaming-data) - Analytics on agent performance and decisions - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/data-governance-framework-roles-and-responsibilities) - Govern agentic AI systems and data access ## Sources and References - [LangChain Documentation - Agents](https://python.langchain.com/docs/modules/agents/) - Comprehensive guide to building AI agents with LangChain framework - [Apache Kafka Streams Documentation](https://kafka.apache.org/documentation/streams/) - Official documentation for real-time stream processing with Kafka - [Pinecone Vector Database Documentation](https://docs.pinecone.io/) - Guide to implementing semantic search and RAG systems with vector databases - [OpenAI Retrieval Augmented Generation](https://platform.openai.com/docs/guides/retrieval-augmented-generation) - Best practices for implementing RAG with large language models - [Apache Flink Stream Processing](https://flink.apache.org/what-is-flink/flink-applications/) - Low-latency stream processing architecture for AI pipelines Source: https://www.conduktor.io/glossary/agentic-ai-pipelines --- # AI Discovery and Monitoring: Tracking AI Assets Across the Enterprise In the rush to implement AI solutions, many organizations have created a sprawling landscape of models, pipelines, and data flows that operate in silos across departments and teams. A data science team might deploy a fraud detection model, marketing launches a recommendation engine, and operations builds a predictive maintenance system, all without centralized visibility or coordination. This AI sprawl creates significant risks. Models trained on outdated data continue making predictions. Redundant systems waste compute resources. Compliance teams struggle to audit what AI is being used and how. Security vulnerabilities lurk in forgotten endpoints. The very innovations meant to drive business value become sources of operational debt and regulatory exposure. **AI discovery and monitoring** address this by building systematic visibility into every AI asset across the enterprise. Discovery answers "what AI do we have?", while monitoring answers "how is it performing?" Together, they form the foundation of AI governance. ## What is AI Discovery? ![AI Discovery and Monitoring Lifecycle](https://www.conduktor.io/assets/images/glossary/ai-discovery-and-monitoring-0.webp) AI discovery is the process of identifying, cataloging, and maintaining an inventory of all AI-related assets within an organization. These assets span a diverse landscape: - **Models**: Machine learning models in development, staging, and production environments (including traditional ML, deep learning, and foundation models) - **LLM Assets**: Foundation models, fine-tuned variants, prompt templates, embeddings, vector databases, and retrieval systems for RAG (Retrieval-Augmented Generation) applications - **Training pipelines**: Data processing workflows that prepare training datasets - **Feature pipelines**: Systems that compute and serve features for real-time inference - **Endpoints**: APIs and services that expose model predictions - **Training data**: Datasets used to build and validate models - **Inference data**: Real-time and batch data flowing through prediction systems Discovery extends across the entire AI lifecycle, from experimental notebooks in data science workstations to production-grade services handling millions of requests. It's not a one-time activity but a continuous process that keeps pace with rapid development cycles and evolving infrastructure. While discovery focuses on *what exists*, monitoring focuses on *how it behaves*. These practices are complementary: you can't effectively monitor what you haven't discovered, and discovery without monitoring leaves blind spots in operational performance. ## Why AI Discovery Matters The business case for AI discovery spans four critical dimensions: - **Compliance and Regulatory Requirements**: Regulations like GDPR, CCPA, the EU AI Act (enforced 2025), and US AI Executive Order requirements mandate organizations document what personal data their AI systems process, maintain AI Bill of Materials (AIBOM) for model transparency, provide Model Cards detailing model characteristics and limitations, and demonstrate measures protecting against bias and discrimination. Without comprehensive discovery, compliance teams can't even identify which systems fall under regulatory scope, let alone audit them effectively. - **Risk Management and Security**: Undocumented AI systems are security vulnerabilities waiting to be exploited. **Shadow AI**, models deployed by teams without IT oversight or central governance, may lack proper authentication, expose sensitive data, or make critical decisions without adequate testing. Discovery enables security teams to implement consistent policies, patch vulnerabilities, and ensure models meet organizational standards before reaching production. - **Cost Optimization**: AI workloads consume significant compute resources. Discovery reveals redundant models solving the same problem, underutilized systems that could be decommissioned, and opportunities to consolidate infrastructure. Organizations routinely find 20-30% cost savings by identifying and eliminating AI waste after implementing comprehensive discovery. - **Technical Debt Reduction**: Every organization has AI systems that outlived their usefulness but continue running because no one knows if they're still needed. Discovery provides the visibility to safely decommission obsolete models, reducing operational complexity and freeing teams to focus on high-value initiatives rather than maintaining legacy systems. ## Building an AI Asset Inventory A complete AI asset inventory is the single source of truth for your organization's AI landscape. The core components include: - **Models and Versions**: Each model entry should capture the algorithm type, version history, training date, accuracy metrics, owner, and deployment status. Version control is critical, production systems may depend on specific model versions, and rollbacks require knowing exactly what was deployed when. - **Training Data Lineage**: Document the datasets used to train each model, including data sources, transformation logic, and temporal snapshots. This enables reproducibility, helps diagnose performance issues, and supports compliance requirements around data usage and retention. For comprehensive lineage tracking practices, see [Data Lineage Tracking: Data from Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption). - **Features and Engineering**: Feature stores, centralized repositories where ML features are stored, managed, and served to models, are becoming central to modern ML architectures. Your inventory should track feature definitions, computation logic, dependencies, and which models consume which features. This prevents duplicate feature engineering and enables feature reuse across teams. For detailed coverage of feature store patterns, see [Feature Stores for Machine Learning](https://www.conduktor.io/glossary/feature-stores-for-machine-learning). - **Endpoints and APIs**: Production models are typically accessed through APIs. Catalog each endpoint's URL, authentication method, rate limits, SLA commitments, and consuming applications. This mapping is essential for impact analysis when changes are planned. - **Model registries** play a crucial role in maintaining this inventory. Tools like MLflow, Weights & Biases, and Neptune provide structured repositories where data scientists register models with standardized metadata. However, model registries alone aren't sufficient, they typically don't capture the broader context of data pipelines, feature engineering, and downstream consumers. Integration with **data catalogs** (like DataHub, Collibra, or Atlan) provides end-to-end visibility by connecting model metadata with the data assets they depend on and produce. Here's an example of registering a model with MLflow during training, automatically capturing parameters, metrics, and custom metadata: ```python import mlflow from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import accuracy_score # Start MLflow run with mlflow.start_run(run_name="fraud_detection_v1"): # Train model model = RandomForestClassifier(n_estimators=100, random_state=42) model.fit(X_train, y_train) # Log parameters mlflow.log_params({ "n_estimators": 100, "max_depth": 10, "training_data_version": "2025-12-01" }) # Log metrics predictions = model.predict(X_test) accuracy = accuracy_score(y_test, predictions) mlflow.log_metric("accuracy", accuracy) # Log model with metadata mlflow.sklearn.log_model( model, "model", registered_model_name="fraud_detection", metadata={ "owner": "data-science-team", "business_unit": "payments", "compliance_reviewed": True, "deployment_tier": "production" } ) ``` ## Discovery Methods and Techniques Organizations employ multiple strategies to discover and catalog AI assets: - **Metadata Scanning**: Automated tools scan infrastructure to identify AI workloads based on signatures like TensorFlow or PyTorch libraries, GPU usage patterns, or specific API frameworks. This passive approach catches systems that weren't formally registered but has limited insight into business context. - **API Monitoring**: Network analysis tools observe API traffic to identify machine learning inference endpoints based on request patterns, response structures, and performance characteristics. This reveals shadow AI deployed without proper documentation but requires sophisticated pattern recognition to distinguish ML APIs from other services. - **Lineage Tracing**: Following data flows backward from business applications reveals the models and pipelines that support them. Lineage tools track how data moves from sources through transformations to final consumption, mapping the complete chain from raw data to AI-driven decisions. This approach provides rich context but requires instrumentation of data pipelines. - **Declarative Registration**: The most reliable approach is requiring teams to explicitly register AI assets in centralized systems, ideally integrated into CI/CD pipelines so registration happens automatically during deployment. This provides high-quality metadata but only works with organizational discipline and enforcement. Most mature organizations combine these methods: automated discovery catches undocumented systems, while declarative registration ensures new systems are properly cataloged from the start. Here's an example of automated discovery using metadata scanning. This script scans AWS SageMaker to find all deployed models and endpoints, automatically building an inventory: ```python import boto3 import json def discover_sagemaker_models(region='us-east-1'): """ Scan AWS SageMaker to discover deployed models and endpoints """ sagemaker = boto3.client('sagemaker', region_name=region) # Discover all endpoints endpoints = sagemaker.list_endpoints()['Endpoints'] inventory = [] for endpoint in endpoints: endpoint_name = endpoint['EndpointName'] # Get endpoint details endpoint_config = sagemaker.describe_endpoint( EndpointName=endpoint_name ) # Get model details model_name = endpoint_config['EndpointConfigName'] # Collect metadata asset = { 'asset_id': endpoint_name, 'asset_type': 'ml_endpoint', 'status': endpoint_config['EndpointStatus'], 'creation_time': str(endpoint_config['CreationTime']), 'last_modified': str(endpoint_config['LastModifiedTime']), 'instance_type': endpoint_config['ProductionVariants'][0]['InstanceType'], 'model_name': model_name, 'region': region } # Extract tags for ownership and metadata tags = sagemaker.list_tags(ResourceArn=endpoint_config['EndpointArn']) asset['tags'] = {tag['Key']: tag['Value'] for tag in tags['Tags']} inventory.append(asset) return inventory # Run discovery discovered_assets = discover_sagemaker_models() print(f"Discovered {len(discovered_assets)} ML endpoints") ``` ## Monitoring Dimensions Once AI assets are discovered, continuous monitoring tracks their health and performance across multiple dimensions: - **Performance Metrics**: Track prediction latency, throughput, error rates, and resource utilization. Compare actual performance against SLA commitments. Set alerts for degradation that impacts user experience or breaches service agreements. - **Model Drift**: Monitor statistical properties of input data and model predictions to detect drift, when the data distribution shifts from what the model was trained on, degrading accuracy. Drift detection is crucial because models don't explicitly fail; they just become gradually less effective, often invisibly to users. For in-depth coverage of drift patterns and mitigation strategies, see [Model Drift in Streaming](https://www.conduktor.io/glossary/model-drift-in-streaming). - **Data Quality**: Track completeness, validity, and freshness of features fed to models. Missing values, schema changes, or stale data can silently corrupt predictions. Quality monitoring catches these issues before they cascade into business impact. For understanding quality dimensions, see [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). - **Usage Patterns**: Understand who uses each model, how often, and for what purposes. Usage tracking identifies models ready for decommissioning (no users) or requiring scaling (growing demand). It also supports chargeback models where consumers pay for the AI services they use. - **Cost Tracking**: Attribute infrastructure costs to specific models and teams. This enables ROI analysis (is this model worth what it costs?), budget accountability, and optimization efforts focused on the most expensive systems. Modern observability platforms like Arize, Fiddler, and WhyLabs specialize in AI-specific monitoring, providing purpose-built capabilities for drift detection, explainability, and fairness metrics that general-purpose monitoring tools lack. **LLM-Specific Monitoring** (2025+): For Large Language Model systems, additional monitoring dimensions include prompt effectiveness, token usage and costs, embedding quality, retrieval accuracy in RAG systems, context relevance, and hallucination detection. Tools like LangSmith, Weights & Biases Weave, and the open-source Evidently AI provide specialized observability for GenAI applications, tracking prompt-response patterns, semantic drift in embeddings, and retrieval performance in vector databases. Here's an example of implementing drift detection. For production use, consider specialized libraries like Evidently AI or NannyML that provide comprehensive drift detection, but this example demonstrates the core concept: ```python import numpy as np from scipy import stats from datetime import datetime, timedelta class DriftDetector: """ Monitor model input distributions for drift using statistical tests """ def __init__(self, reference_data, threshold=0.05): self.reference_data = reference_data self.threshold = threshold self.drift_history = [] def detect_drift(self, current_data, feature_name): """ Use Kolmogorov-Smirnov test to detect distribution drift """ # Perform KS test statistic, p_value = stats.ks_2samp( self.reference_data[feature_name], current_data[feature_name] ) drift_detected = p_value < self.threshold result = { 'timestamp': datetime.now(), 'feature': feature_name, 'statistic': statistic, 'p_value': p_value, 'drift_detected': drift_detected, 'severity': 'high' if p_value < 0.01 else 'medium' if p_value < self.threshold else 'none' } self.drift_history.append(result) if drift_detected: print(f"⚠️ Drift detected in {feature_name}: p-value={p_value:.4f}") return result def monitor_all_features(self, current_data): """ Monitor all numeric features for drift """ results = {} for feature in self.reference_data.select_dtypes(include=[np.number]).columns: results[feature] = self.detect_drift(current_data, feature) return results # Usage example detector = DriftDetector(reference_data=training_data) drift_results = detector.monitor_all_features(current_production_data) ``` ## Streaming-Specific Challenges AI systems built on streaming architectures present unique discovery and monitoring challenges: - **Real-time Model Serving**: Models that process event streams (fraud detection on payment events, personalization on clickstreams) operate in a fundamentally different paradigm than batch systems. Discovery must track event schemas, topic subscriptions, and the temporal dependencies between events and predictions. For building end-to-end streaming ML systems, see [Real-Time ML Pipelines](https://www.conduktor.io/glossary/real-time-ml-pipelines). - **Feature Pipelines**: Real-time feature engineering often involves complex streaming aggregations, windowed calculations, joins across multiple event streams, and stateful transformations. These pipelines are difficult to discover because the logic is distributed across stream processors, and the lineage is implicit in event flows rather than explicit in code. - **Event-Driven Architectures**: In platforms like Kafka, models consume events from topics and produce predictions to other topics, creating intricate graphs of dependencies. Discovery requires understanding these topic-level relationships and tracing data lineage through asynchronous event flows. For Kafka infrastructure monitoring, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). Governance platforms provide streaming-native capabilities to address these challenges, enabling teams to discover data products flowing through Kafka, enforce quality policies on event streams, and maintain visibility into the complex topologies that connect producers, stream processors, and consumers, including AI models and feature pipelines. This streaming-focused approach complements traditional model registries by capturing the real-time data context that batch-oriented tools miss. Here's an example of monitoring a real-time ML model consuming from Kafka. This example works with Kafka 4.0+ running in KRaft mode (the modern architecture that replaced ZooKeeper in 2025): ```python from kafka import KafkaConsumer, KafkaProducer import json import time from datetime import datetime class StreamingModelMonitor: """ Monitor real-time ML model performance in streaming environment """ def __init__(self, model, input_topic, output_topic): self.model = model self.consumer = KafkaConsumer( input_topic, bootstrap_servers=['localhost:9092'], value_deserializer=lambda m: json.loads(m.decode('utf-8')) ) self.producer = KafkaProducer( bootstrap_servers=['localhost:9092'], value_serializer=lambda v: json.dumps(v).encode('utf-8') ) self.output_topic = output_topic self.metrics = { 'predictions_count': 0, 'total_latency': 0, 'error_count': 0 } def process_stream(self): """ Consume events, make predictions, monitor performance """ for message in self.consumer: start_time = time.time() try: # Extract features from event event = message.value features = self.extract_features(event) # Make prediction prediction = self.model.predict([features])[0] # Calculate latency latency_ms = (time.time() - start_time) * 1000 # Update metrics self.metrics['predictions_count'] += 1 self.metrics['total_latency'] += latency_ms # Publish prediction with metadata result = { 'event_id': event.get('id'), 'prediction': float(prediction), 'timestamp': datetime.now().isoformat(), 'latency_ms': latency_ms, 'model_version': '1.2.0' } self.producer.send(self.output_topic, result) # Alert on high latency if latency_ms > 100: print(f"High latency warning: {latency_ms:.2f}ms") except Exception as e: self.metrics['error_count'] += 1 print(f"Prediction error: {e}") # Periodic metrics reporting if self.metrics['predictions_count'] % 1000 == 0: self.report_metrics() def extract_features(self, event): """Extract feature vector from Kafka event""" return [event.get('feature1'), event.get('feature2'), event.get('feature3')] def report_metrics(self): """Report aggregated metrics""" avg_latency = self.metrics['total_latency'] / self.metrics['predictions_count'] error_rate = self.metrics['error_count'] / self.metrics['predictions_count'] print(f"Metrics - Count: {self.metrics['predictions_count']}, " f"Avg Latency: {avg_latency:.2f}ms, Error Rate: {error_rate:.2%}") ``` ## Governance Workflows Discovery and monitoring enable proactive governance workflows throughout the AI lifecycle: - **Approval and Certification**: Before deployment, new models pass through review gates where architecture boards assess risk, security teams verify data protection, and compliance teams confirm regulatory alignment. Discovery systems integrate with these workflows, preventing uncertified models from reaching production. - **Lifecycle Management**: Formal processes govern model transitions between development, staging, and production environments. Each transition triggers validation checks, documentation requirements, and stakeholder notifications tracked in the inventory system. - **Retirement and Deprecation**: When models become obsolete, governance workflows ensure safe decommissioning. The inventory reveals all downstream consumers, enabling impact analysis and migration planning. Formal sunset processes notify stakeholders, archive artifacts for compliance, and prevent accidental re-deployment. - **Audit Trails**: Every change to model configuration, training data, or deployment status is logged with timestamps and responsible parties. These audit trails support compliance reporting, incident investigations, and continuous improvement of AI operations. Mature organizations encode these workflows in their discovery and monitoring platforms, automating routine checks and providing clear handoffs between teams. For broader governance practices, see [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities). ## Building an AI Operations Center Leading organizations are establishing centralized **AI Operations Centers** that consolidate visibility and coordination: - **Centralized Dashboards**: Executive dashboards provide at-a-glance views of the entire AI estate, how many models in production, performance trends, cost trajectories, and compliance posture. These visualizations make AI operations tangible to leadership and enable data-driven investment decisions. - **Integration with MLOps and DataOps**: AI operations don't exist in isolation. The operations center integrates model lifecycle management (MLOps) with data pipeline orchestration (DataOps), providing unified visibility into the dependencies between data and models. This integration enables end-to-end impact analysis: "if we change this dataset, which models are affected?" - **Team Collaboration**: The operations center is a collaboration hub where data scientists, ML engineers, data engineers, and platform teams coordinate. Shared visibility into the AI landscape reduces duplicate work, enables knowledge sharing, and clarifies ownership boundaries. - **Incident Response**: When models degrade or fail, the operations center provides the context for rapid troubleshooting, recent changes, dependency mapping, historical performance baselines, and contact information for responsible teams. This dramatically reduces mean time to resolution. Building an operations center is as much organizational as technical. It requires executive sponsorship, cross-functional collaboration, and cultural acceptance that AI governance enables rather than inhibits innovation. ## Summary AI discovery and monitoring are not compliance burdens — they're operational necessities. Organizations with visibility into their AI assets can move faster and scale more reliably than those operating without it. The path to mature AI operations begins with discovery: cataloging what exists, understanding dependencies, and establishing baseline monitoring. From there, teams build governance workflows, optimize resource allocation, and create the transparency that regulators and business stakeholders need. As AI becomes standard infrastructure, discovery and monitoring will integrate with broader observability, security, and governance platforms rather than existing as separate disciplines. ## Related Concepts - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/data-governance-framework-roles-and-responsibilities) - Establish governance for AI assets - [Streaming Data Pipeline](https://www.conduktor.io/streaming-data-pipeline) - Monitor pipelines feeding AI models - [Schema Registry and Schema Management](https://www.conduktor.io/schema-registry-and-schema-management) - Track schemas for AI data flows ## Sources and References 1. **MLflow Documentation** - MLflow Model Registry and Tracking: [https://mlflow.org/docs/latest/model-registry.html](https://mlflow.org/docs/latest/model-registry.html) - Comprehensive guide to model versioning, lifecycle management, and metadata tracking. 2. **Google Cloud - Best Practices for ML Engineering**: [https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning](https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning) - Industry best practices for MLOps, including model monitoring and governance frameworks. 3. **Arize AI - ML Observability Guide**: [https://arize.com/blog/ml-observability/](https://arize.com/blog/ml-observability/) - Detailed overview of model monitoring, drift detection, and observability patterns for production ML systems. 4. **AWS - Machine Learning Governance**: [https://docs.aws.amazon.com/sagemaker/latest/dg/governance.html](https://docs.aws.amazon.com/sagemaker/latest/dg/governance.html) - Practices for governing ML workflows, including model discovery, lineage tracking, and compliance. 5. **DataHub Project - Metadata Architecture**: [https://datahubproject.io/docs/metadata-model/](https://datahubproject.io/docs/metadata-model/) - Open-source framework for building comprehensive data and ML asset catalogs with lineage tracking. Source: https://www.conduktor.io/glossary/ai-discovery-and-monitoring --- # Amazon MSK: Managed Kafka on AWS Amazon Managed Streaming for Apache Kafka (MSK) is AWS's fully managed service for running Apache Kafka clusters in the cloud. Launched in 2019, MSK removes much of the operational burden of running Kafka while maintaining compatibility with the Apache Kafka ecosystem. For organizations already running on AWS, MSK provides a straightforward way to adopt event-driven architectures and real-time data pipelines without managing Kafka infrastructure themselves. ## What is Amazon MSK? Amazon MSK is a fully managed service that handles the deployment, configuration, and maintenance of Apache Kafka clusters on AWS infrastructure. It automates provisioning, patching, and scaling while giving you control over Kafka configurations. MSK runs open-source Apache Kafka, which means applications, tooling, and skills transfer directly. You get standard Kafka APIs and protocols without vendor lock-in at the application layer. AWS handles the underlying infrastructure: compute instances, storage volumes, networking, and availability zone distribution. The service offers multiple deployment options: MSK Provisioned (traditional instance-based clusters), MSK Provisioned with Express brokers (2024+, optimized for high throughput and rapid scaling), and MSK Serverless (automatic scaling based on demand). ## Architecture and Core Components ![Amazon MSK Architecture](https://www.conduktor.io/assets/images/glossary/amazon-msk-managed-kafka-on-aws-0.webp) An MSK cluster consists of Apache Kafka brokers distributed across multiple availability zones within an AWS region. AWS manages the underlying EC2 instances, EBS volumes for storage, and KRaft controllers for cluster coordination. As of 2024, MSK supports KRaft mode (Kafka's native consensus protocol), which eliminates the need for ZooKeeper and provides improved scalability and faster metadata operations. When you create an MSK cluster, you specify: - The number of broker nodes and their distribution across availability zones - Instance types (from kafka.m5.large to kafka.m7g.24xlarge for provisioned clusters) - Storage capacity per broker - Network configuration (VPC, subnets, security groups) MSK automatically handles broker replacement if hardware fails, patches software with minimal downtime, and enables automatic minor version upgrades. Multi-AZ deployment ensures high availability: if one availability zone becomes unavailable, the cluster continues operating with replicas in other zones. For example, a typical production MSK cluster might have six brokers spread across three availability zones (two brokers per zone), using kafka.m5.xlarge instances with 1TB of storage each. This configuration provides both fault tolerance and adequate capacity for moderate-throughput workloads. ## MSK Provisioned: Standard vs Express Brokers As of 2024, AWS offers two types of provisioned brokers with significantly different performance and cost characteristics: ### Standard Brokers (Traditional MSK Provisioned) Standard brokers use EC2 instances with attached EBS volumes. They offer: - Predictable, instance-based pricing - Wide range of instance types (kafka.m5.large to kafka.m7g.24xlarge) - Fixed storage capacity that can be expanded (but not reduced) - Traditional scaling: add brokers and rebalance partitions manually ### Express Brokers (Recommended for 2025+) - **Express brokers** represent AWS's optimized Kafka broker architecture, launched in 2024. They provide dramatic improvements over standard brokers: - **Performance Advantages**: - **3x higher throughput** per broker compared to standard brokers - **20x faster scaling**: Add or remove capacity in ~5 minutes vs 20-40 minutes for standard brokers - **Unlimited storage**: Pay-as-you-go storage model with no pre-provisioning required - **Lower latency**: Optimized I/O path reduces p99 latency by 30-40% **Cost Model**: - Per-GB throughput pricing instead of instance hours - Separate charges for compute, storage, and data transfer - Generally 20-30% more expensive than standard brokers for steady-state workloads - More economical for bursty workloads due to elastic capacity **When to Use Express Brokers**: - High-throughput streaming applications (>100 MB/s per broker) - Variable workloads requiring rapid scaling - Applications sensitive to rebalancing downtime - Long retention requirements (unlimited storage reduces storage management overhead) **When to Use Standard Brokers**: - Predictable workloads with steady throughput - Cost optimization for low-throughput use cases - Existing clusters without urgent performance needs - **Migration Path**: AWS recommends Express brokers for new production clusters. Existing standard broker clusters can migrate using MSK Replicator (see Cross-Region Replication section below) to avoid downtime. - **Example Comparison**: ``` Standard Broker Cluster: - 6x kafka.m5.xlarge brokers - 1TB storage per broker - ~$600/month + data transfer - Manual scaling: 20-40 minutes - Throughput: ~50 MB/s per broker Express Broker Cluster: - Pay-per-GB throughput - Unlimited storage (pay for usage) - ~$750/month for similar throughput - Elastic scaling: ~5 minutes - Throughput: ~150 MB/s per broker ``` ## Key Features and Capabilities - **Integrated Security**: MSK integrates deeply with AWS security services. You can authenticate clients using IAM, TLS certificates, or SASL/SCRAM credentials stored in AWS Secrets Manager. Encryption in transit uses TLS, while encryption at rest uses AWS KMS. Fine-grained access control through IAM policies lets you define who can produce or consume from specific topics. - **Monitoring and Observability**: MSK automatically publishes metrics to Amazon CloudWatch, including broker CPU, memory, disk usage, and Kafka-specific metrics like request rates and under-replicated partitions. You can set alarms and create dashboards without additional configuration. - **Storage Tiering**: MSK Tiered Storage automatically moves older data from broker storage to Amazon S3, reducing costs while keeping data accessible. This is particularly valuable for workloads requiring long retention periods without paying for expensive broker storage. - **Apache Kafka Compatibility**: MSK supports multiple Kafka versions and stays current with Apache Kafka releases. You control when to upgrade, and MSK performs rolling updates that maintain cluster availability. - **Serverless Option**: MSK Serverless eliminates capacity planning entirely. It automatically provisions and scales compute and storage resources based on your workload, charging only for the throughput and storage you use. ## MSK in the Data Streaming Ecosystem Amazon MSK is the backbone for real-time data streaming architectures on AWS. It connects naturally with other AWS services: Kinesis Data Analytics for stream processing, Lambda functions for event-driven triggers, S3 for data lake integration, and RDS for change data capture patterns. - **Kafka Connect Integration**: MSK works with Kafka Connect for integrating external systems. You can run self-managed Connect clusters or use MSK Connect, a managed service that deploys and scales Kafka Connect workers. Common patterns include streaming database changes using Debezium connectors or moving data to S3 using sink connectors. - **Stream Processing with Apache Flink**: MSK pairs well with Amazon Managed Service for Apache Flink (formerly Kinesis Data Analytics) for stateful stream processing. Flink applications can consume from MSK topics, perform complex transformations and aggregations, then write results back to MSK or other sinks. - **Schema Management**: While MSK doesn't include a schema registry, you can deploy AWS Glue Schema Registry or third-party registries. Governance platforms provide schema registry capabilities along with data governance and observability features, helping teams manage schema evolution and compatibility across producers and consumers. - **Developer Experience**: Managing MSK clusters at scale requires good tooling. While AWS Console provides basic management, governance platforms offer richer interfaces for topic management, consumer group monitoring, message inspection, and data quality enforcement. This becomes especially valuable as organizations scale from a few topics to hundreds or thousands. ## Operational Considerations - **Cost Structure**: MSK pricing includes broker instance hours, storage costs, and data transfer. A three-broker cluster using kafka.m5.large instances (2 vCPUs, 8 GB memory each) with 500 GB storage per broker costs approximately $500-600/month before data transfer. MSK Serverless can be more economical for variable or unpredictable workloads. For implementing chargeback and cost attribution across MSK clusters to understand per-team and per-topic costs, see [Conduktor's Cost Control Insights](https://docs.conduktor.io/guide/insights/cost-control). - **Scaling**: You can scale MSK clusters vertically by changing broker instance types or horizontally by adding brokers. Storage expansion happens online without downtime. However, rebalancing partitions across new brokers requires manual partition reassignment or tools that automate the process. - **Monitoring**: Beyond CloudWatch metrics, production deployments benefit from comprehensive monitoring. This includes tracking consumer lag, partition distribution, broker disk usage trends, and data quality issues. Governance platforms can enforce policies on data quality, schema compliance, and access patterns before problems reach production. - **Disaster Recovery and Cross-Region Replication**: MSK clusters are regional. AWS provides **MSK Replicator**, a fully managed cross-region replication service that simplifies multi-region deployments without managing MirrorMaker 2 infrastructure. MSK Replicator features: - **Automatic replication**: Configure source and target MSK clusters; AWS handles replication logic - **Topic filtering**: Replicate specific topics or patterns, not entire clusters - **Consumer group offset sync**: Enables seamless failover by replicating consumer offsets - **Exactly-once delivery**: Prevents duplicate messages during replication - **Monitoring integration**: CloudWatch metrics for replication lag and throughput Use cases: - **Active-passive DR**: Replicate production data to standby region for disaster recovery - **Active-active**: Bidirectional replication for multi-region writes (requires careful conflict resolution) - **Data aggregation**: Replicate from multiple regional clusters to central analytics cluster - **Migration**: Zero-downtime migration from standard brokers to Express brokers or between regions Cost: MSK Replicator charges per GB replicated plus inter-region data transfer fees. Typically adds 20-40% to total MSK costs for DR scenarios. ## Use Cases and When to Choose MSK **Best Fit Scenarios**: - Organizations already using AWS services extensively - Teams that want Kafka's ecosystem without managing infrastructure - Applications requiring event-driven architectures with high throughput - Microservices needing reliable inter-service communication - Real-time analytics and data pipeline workloads - **Real-World Example**: A financial services company might use MSK to stream transaction events from payment systems to fraud detection services, analytics platforms, and audit logging systems. MSK handles millions of events per day while the team focuses on business logic rather than cluster operations. - **Consider Alternatives When**: - You need maximum control over Kafka internals and hardware tuning - Your workload is primarily on other cloud providers (consider their managed Kafka offerings) - You have specialized compliance requirements that preclude managed services - Your team has deep Kafka operations expertise and wants full control MSK Serverless works well for development environments, unpredictable traffic patterns, or teams wanting to start small without capacity planning. MSK Provisioned suits production workloads with predictable throughput where you want control over instance types and storage. ## Summary Amazon MSK simplifies running Apache Kafka on AWS by handling infrastructure management, patching, and availability while maintaining compatibility with the Kafka ecosystem. It offers two deployment models: provisioned clusters for predictable workloads and serverless for variable demand. MSK integrates well with AWS services and the broader streaming ecosystem, supporting Kafka Connect, Apache Flink, and various schema management solutions. Operational aspects like monitoring, scaling, and disaster recovery require planning, though AWS handles much of the infrastructure complexity. For AWS-focused organizations wanting Kafka without the operational overhead, MSK is a solid foundation for real-time data streaming. Success depends on proper cluster sizing, monitoring, security configuration, and tooling for governance and developer experience. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - The open-source streaming platform that MSK is built on - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Essential for right-sizing MSK clusters and controlling costs - [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership) - Understanding the full economic impact of managed Kafka services ## Sources and References 1. **AWS Documentation**: [Amazon MSK Developer Guide](https://docs.aws.amazon.com/msk/) - Official AWS documentation covering architecture, configuration, and best practices. 2. **Apache Kafka Documentation**: [Apache Kafka Documentation](https://kafka.apache.org/documentation/) - Core Kafka concepts and features that MSK implements. 3. **AWS Blog**: [Best Practices for Running Apache Kafka on AWS](https://aws.amazon.com/blogs/big-data/) - AWS Big Data Blog with MSK use cases, performance optimization, and architecture patterns. 4. **AWS Case Studies**: [Netflix and Amazon MSK](https://aws.amazon.com/solutions/case-studies/netflix-msk/) - Real-world implementation examples from major MSK users. 5. **Confluent Platform**: [Kafka on AWS Best Practices](https://www.confluent.io/blog/) - Kafka ecosystem insights applicable to MSK deployments, including monitoring and operations patterns. Source: https://www.conduktor.io/glossary/amazon-msk-managed-kafka-on-aws --- # Apache Iceberg Apache Iceberg is an open table format that brings **database-like reliability and transactional guarantees** to massive datasets stored in cloud object stores. Created at Netflix to address the limits of traditional data lakes, Iceberg provides the missing structure and control that teams need to manage analytical data at scale. It has become a foundational layer for **data lakehouse architectures**, bridging real-time streaming ingestion from systems like Kafka and analytical queries on Spark, Trino, or Flink. ## Why Table Formats Exist Raw data lakes built on Parquet or ORC files are flexible but fragile. They lack atomic operations, consistent schema management, and efficient file handling. This makes data engineering complex and error-prone as datasets grow and evolve. Common challenges include: * **No atomicity:** Multiple concurrent writers can corrupt datasets. * **Rigid schemas:** Evolving or renaming columns often requires rewriting entire tables. * **Performance overhead:** Query engines must scan thousands of files to infer structure. * **Small files problem:** Frequent micro-batch writes from streaming systems create excessive metadata and degrade query performance. Table formats like **Apache Iceberg, Delta Lake, and Apache Hudi** were created to solve these problems by adding a **metadata layer** that tracks versions, schemas, and file locations. Iceberg stands out for its open specification, flexible partitioning, and support for concurrent streaming and batch workloads. ## How Apache Iceberg Works Iceberg organizes data through multiple metadata layers that manage both data and schema evolution efficiently. ### Key Components 1. **Data files:** Contain actual records, typically in Parquet or ORC format. 2. **Manifests:** List data files and include statistics like record counts and partition values. 3. **Snapshots:** Immutable views representing the state of a table at a specific time. 4. **Metadata file:** The top-level JSON file that points to the active snapshot and stores schema history. Each write operation in Iceberg creates a **new snapshot** without altering existing data. This *copy-on-write* model allows atomic commits and concurrent readers. For comprehensive coverage of Iceberg's internal architecture, metadata layers, and snapshot mechanics, see [Iceberg Table Architecture: Metadata and Snapshots](https://www.conduktor.io/glossary/iceberg-table-architecture-metadata-and-snapshots). A simplified example: ![A simplified example](https://www.conduktor.io/assets/images/glossary/apache-iceberg-0.webp) ### Hidden Partitioning and Partition Evolution Traditional data lakes expose partition columns directly, often leading to brittle query logic. Iceberg's **hidden partitioning** abstracts this detail away, users can query logical fields without worrying about how data is stored. If access patterns change, **partition evolution** allows new partition strategies (for example, switching from `day()` to `hour()`) without rewriting existing data. ### Metadata-Only Operations Iceberg supports operations that work purely on metadata, like listing partitions or updating schemas, without scanning data files. This design keeps both maintenance and queries efficient at scale. ## ACID Transactions and Time Travel Apache Iceberg brings **ACID transactions** to data lakes through **optimistic concurrency control**. Multiple writers can attempt to modify the same table, but only one commit succeeds, ensuring atomic updates without locking. Readers always see consistent snapshots, and failed writers automatically retry using the latest table state. ### Time Travel Queries Iceberg records a history of all table snapshots, allowing you to query data as it existed at any point in time: ```sql SELECT * FROM orders VERSION AS OF 1082348923412; SELECT * FROM orders TIMESTAMP AS OF '2025-11-01 10:00:00'; ``` This feature supports **debugging**, **reproducible analytics**, and **regulatory compliance**. To manage costs, Iceberg allows administrators to define **snapshot retention and orphan file cleanup** policies, balancing governance and storage efficiency. ## Apache Iceberg in the Lakehouse Ecosystem In a data lakehouse, Iceberg is the **storage and transaction layer** that unifies batch and interactive analytics over cloud storage. Major query engines, **Spark, Trino, Snowflake, Flink, and Dremio**, natively support Iceberg, enabling teams to query the same datasets with different compute engines. This interoperability makes Iceberg ideal for multi-engine and multi-cloud environments. Iceberg integrates with metadata catalogs such as **AWS Glue**, **Hive Metastore**, **Project Nessie**, and **Apache Polaris**, allowing unified table discovery, access control, and schema management. For detailed catalog comparisons, configuration, and migration strategies, see [Iceberg Catalog Management: Hive, Glue, and Nessie](https://www.conduktor.io/glossary/iceberg-catalog-management-hive-glue-and-nessie). Combined with features like **Z-order clustering**, **configurable file compaction**, and **metadata pruning**, Iceberg delivers warehouse-grade performance on top of open storage. ## Apache Iceberg and Data Streaming Streaming data systems like Kafka and Flink continuously generate small batches of events, a pattern that doesn't fit traditional batch-oriented storage. Iceberg bridges this gap by supporting **incremental ingestion and CDC (Change Data Capture)** at scale. For understanding how Kafka streaming integrates with data lakes, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). For CDC implementation details, see [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium). ### Streaming Patterns 1. **Append mode:** Ideal for high-volume event streams that continually add new data. 2. **Upsert mode:** Designed for CDC pipelines using tools like **Debezium**, where updates and deletes must be reflected in analytical tables. For example, Flink's Iceberg connector enables exactly-once ingestion with upsert support: ```sql CREATE TABLE orders_iceberg ( order_id BIGINT, customer_id BIGINT, amount DECIMAL(10,2), updated_at TIMESTAMP(3), PRIMARY KEY (order_id) NOT ENFORCED ) WITH ( 'connector' = 'iceberg', 'catalog-type' = 'rest', -- Modern REST catalog (Polaris, Nessie) recommended over Hive 'catalog-uri' = 'http://catalog-server:8181', 'warehouse' = 's3://my-warehouse/iceberg', 'write.distribution-mode' = 'hash' -- Distribute by primary key for upserts ); ``` The `PRIMARY KEY` declaration (available in Flink 1.15+) enables upsert semantics automatically, while `write.distribution-mode = 'hash'` ensures updates to the same key are processed by the same writer for consistency. Iceberg supports **file compaction** to consolidate the small files generated by streaming writes, preventing performance degradation and maintaining healthy table structures over time. Note that compaction must be explicitly configured and scheduled (using Flink actions, Spark procedures, or dedicated compaction jobs), it doesn't happen automatically without configuration. This combination of streaming ingestion and governed storage makes Iceberg central to **real-time analytics**, **fraud detection**, and **AI feature store** architectures. ## Summary Apache Iceberg transforms the data lake into a **reliable, versioned, and queryable** foundation for modern analytics. It fixes the gaps of traditional data lakes through strong transactional guarantees, schema evolution, and metadata-driven performance. In streaming environments, Iceberg enables **real-time ingestion** from Kafka or Flink into consistent analytical tables, while managing file compaction and schema consistency. Governance platforms extend this reliability to the operational layer, ensuring every Kafka-to-Iceberg pipeline remains secure, observable, and governed. ## Related Concepts - [Real-Time Analytics with Streaming Data](https://www.conduktor.io/real-time-analytics-with-streaming-data) - [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/what-is-change-data-capture-cdc-fundamentals) - [Data Lineage: Tracking Data from Source to Consumption](https://www.conduktor.io/data-lineage-tracking-data-from-source-to-consumption) ## Sources and References * [Apache Iceberg Official Documentation](https://iceberg.apache.org/) * [Netflix Tech Blog: Introducing Iceberg](https://netflixtechblog.com/) * [Apache Flink: Iceberg Connector Guide](https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/table/iceberg/) * [Confluent: Kafka Connect Iceberg Sink Connector](https://docs.confluent.io/kafka-connectors/iceberg/current/overview.html) * [Databricks Blog: Understanding Table Formats in Lakehouses](https://www.databricks.com/blog/) Source: https://www.conduktor.io/glossary/apache-iceberg --- # Apache Kafka Apache Kafka has become the backbone of real-time data systems. It powers everything from payment tracking to fraud detection to application logs. This article explains what Kafka is, how event streaming works, and why it matters for modern data architectures. It integrates both foundational concepts and practical operational details used in production systems today. ## From Queues to Streams Before Kafka, most systems shared data through message queues or scheduled ETL jobs. These methods moved data in batches and required tight coupling between producers and consumers. Traditional queues worked well for point-to-point communication, but modern systems needed more: multiple consumers reading the same data, the ability to replay events after errors, and throughput in the millions of messages per second. Streaming systems changed that model. Instead of moving data at fixed intervals, they process it continuously as events happen. Every click, transaction, or sensor reading can be captured and processed in real time. This shift from discrete messages to immutable event streams represents a deeper change in how we think about data. Streams act as durable logs of facts that can be replayed to rebuild application state or debug failures. They bridge the gap between real-time processing and historical replay, connecting transactional systems to analytical platforms in one continuous flow. For example, a bank can stream transactions through Kafka to detect fraud instantly while still retaining the full event history for auditing and model retraining. ## What Apache Kafka Is Apache Kafka is a distributed platform for event streaming. It is both a **messaging system** and a **durable event store**. Kafka was originally built at LinkedIn to handle high data volumes with low latency and open-sourced in 2011. Today it handles trillions of events per day in large enterprises. Kafka plays three main roles in modern data infrastructure: 1. **Message Broker**, Kafka decouples producers from consumers, allowing multiple applications to publish and subscribe independently. 2. **Storage System**, Events are written to disk and replicated across brokers, enabling fault tolerance and replay from any point in time. 3. **Stream Processing Platform**, Using Kafka Streams or external frameworks like Flink, data can be aggregated or transformed in motion. Kafka's durability sets it apart from traditional messaging systems. It doesn't delete events once consumed. Instead, data is retained for a configurable period, allowing consumers to reprocess or recover from failures without data loss. ## Core Concepts and Architecture Kafka's architecture is designed for horizontal scalability, strong ordering, and high throughput. ### Topics and Partitions A **topic** is a named stream of events, like a table in a database. Topics are split into **partitions**, which are ordered, immutable logs of records. Each record in a partition has an **offset**, a unique sequential ID. Partitioning allows Kafka to scale by distributing data across servers. For deep dives into partition assignment strategies, rebalancing, and performance implications, see [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). ### Brokers and Clusters A **broker** is a Kafka server that stores event data and serves read/write requests. Multiple brokers form a **cluster**, which collectively manages replication and fault tolerance. A partition typically has one **leader** (handling reads/writes) and several **followers** that replicate data for durability. If a broker fails, a follower is promoted to leader automatically. ### Cluster Coordination: From ZooKeeper to KRaft - **Critical 2025 Update**: Kafka's cluster coordination mechanism has undergone a fundamental architectural change. - **ZooKeeper (Legacy, Deprecated in Kafka 4.0)**: Historically, Kafka relied on Apache ZooKeeper for cluster coordination, tracking broker membership, electing partition leaders, and storing cluster metadata. ZooKeeper added operational complexity: a separate system to deploy, monitor, and scale. Organizations needed ZooKeeper expertise alongside Kafka knowledge. **KRaft (Kafka Raft Metadata Mode) - Production Ready in Kafka 3.3+**: KRaft removes the ZooKeeper dependency entirely, implementing Kafka's own consensus protocol based on the Raft algorithm. This represents the biggest architectural change in Kafka's history. **Why KRaft Matters**: - **Simpler operations**: No ZooKeeper ensemble to manage separately - **Faster metadata operations**: Leader elections complete in milliseconds instead of seconds - **Higher partition scalability**: Supports millions of partitions per cluster (ZooKeeper topped out at ~200K) - **Improved recovery**: Cluster recovery after failures is 10x faster - **Unified architecture**: One system instead of two reduces operational surface area **KRaft Architecture**: ![**KRaft Architecture**](https://www.conduktor.io/assets/images/glossary/apache-kafka-0.webp) **Deployment Modes**: - **Combined mode**: Controllers and brokers run on the same nodes (simpler for smaller clusters) - **Separated mode**: Dedicated controller nodes (recommended for large production clusters) **Migration from ZooKeeper to KRaft**: Kafka 3.6+ supports live migration from ZooKeeper to KRaft without downtime: 1. Add KRaft controllers to existing ZooKeeper cluster 2. Migrate metadata from ZooKeeper to KRaft 3. Remove ZooKeeper dependency 4. Continue operating in KRaft mode **When to Use KRaft**: - **New deployments**: All new Kafka clusters should use KRaft (ZooKeeper mode is deprecated) - **Existing clusters**: Plan migration roadmap; ZooKeeper support ends in Kafka 4.0 (expected 2024-2025) - **High partition count**: KRaft is essential for clusters with 100K+ partitions - **Cloud deployments**: Managed services (Confluent Cloud, Amazon MSK) are transitioning to KRaft by default ### Producers and Consumers - **Producers** send records to topics. They can control where records go, for example, hashing by customer ID to preserve order for that key. - **Consumers** read records and track their progress via offsets. They can operate independently or as part of **consumer groups**, which distribute partitions among members for parallelism. For detailed consumer group mechanics, offset management, and rebalancing strategies, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). A simple producer configuration might look like this: ```properties bootstrap.servers=broker1:9092,broker2:9092 acks=all key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer retries=3 ``` This ensures messages are acknowledged only after full replication, providing strong durability guarantees. ## How Data Flows Through Kafka Each event in Kafka follows a clear path: 1. **Production**, The producer serializes and sends the record to the appropriate partition. 2. **Replication**, The leader broker writes it to disk and replicates it to follower brokers. 3. **Acknowledgment**, The producer receives confirmation once enough replicas have written the data (based on `acks` setting). 4. **Consumption**, Consumers poll Kafka for new records and process them at their own pace. Because consumers pull data, Kafka supports both near real-time and delayed processing. Offsets allow flexible replay and exactly-once semantics in combination with Kafka Streams or Flink. ### Example: Stream Aggregation A Kafka Streams application can aggregate events over time windows to track session activity: ```java stream.groupByKey() .windowedBy(SessionWindows.with(Duration.ofMinutes(5))) .aggregate( SessionData::new, (key, value, aggregate) -> aggregate.update(value), Materialized.as("session-store") ); ``` This type of processing lets teams build stateful real-time applications, for instance, aggregating clickstream data or monitoring user sessions without external databases. ## Kafka in Modern Data Streaming Kafka is the backbone of a broader streaming ecosystem. It connects real-time applications, analytical platforms, and machine learning pipelines. * **Change Data Capture (CDC):** Tools like **Debezium** capture database changes and stream them into Kafka topics, eliminating polling and keeping data synchronized across systems. * **Stream Processing:** Frameworks like **Apache Flink** consume Kafka streams to perform complex transformations and stateful analytics with exactly-once guarantees. * **Lakehouse Integration:** Platforms such as **Databricks** or **Snowflake** can ingest Kafka data directly into Delta Lake or Iceberg tables, combining streaming and batch processing under a unified model. This integration moves data from operational systems into analytical or AI pipelines. For instance, a retail company might stream purchase events into Kafka, use Flink to enrich them with customer data, and store the results in a Delta table for real-time dashboards. ## End-to-End Streaming Architecture Below is how Kafka fits into a real-world streaming ecosystem: ![Below is how Kafka fits into a real-world streaming ecosystem](https://www.conduktor.io/assets/images/glossary/apache-kafka-1.webp) This structure shows how raw operational data flows through Kafka, gets processed and enriched in motion, and ends up in analytical and governance systems, all in near real time. ## Managing Visibility, Security, and Control Operating Kafka at enterprise scale introduces challenges that go beyond streaming performance. ### Access Control and RBAC Kafka's built-in **ACLs** define who can read, write, or manage topics. In large organizations with hundreds of users, this model often scales poorly. Many teams adopt **Role-Based Access Control (RBAC)** to simplify management. Instead of granting ACLs per user, RBAC assigns roles such as `data-engineer`, `analyst`, or `auditor`, each with predefined permissions. For example: * `data-engineer`: Can produce and consume data on development topics. * `analyst`: Can consume from certified topics only. * `auditor`: Read-only access to all logs and schema versions. Governance platforms enforce and visualize these RBAC policies across clusters, integrating with LDAP or SSO providers to align identity and access. ### Schema Evolution and Audit Scenario Kafka's **Schema Registry** ensures producers and consumers share compatible message formats. Every schema change is versioned and validated to prevent breaking downstream systems. In regulated environments (like finance or healthcare), teams often require **data audit trails**: * When a schema changes, a record of who changed it and when must be logged. * When sensitive data moves between topics, lineage must show where it originated and who accessed it. * Audit teams must verify that data masking and encryption policies are consistently enforced. Governance platforms can export audit logs, showing which topics contain sensitive data and which users or apps interacted with them. This closes the gap between data engineering and compliance without slowing down development velocity. ### Observability and Governance Monitoring Kafka involves tracking broker health, consumer lag, and throughput. Lag indicates how far a consumer is behind the producer, a key signal for diagnosing slow processing or overload. Visibility across topics, consumer groups, and schemas is critical for debugging and compliance. Governance platforms centralize these aspects, offering: * Dashboards for lag, throughput, and partition health * Topic ownership tracking * Fine-grained role-based access and field-level data masking * Policy automation and audit logs for governance This level of visibility turns Kafka from a raw cluster into a governed data platform, reducing operational risk while improving developer autonomy. ## Common Use Cases and Next Steps Kafka supports a wide range of real-time and data integration use cases: * **Financial Services:** Fraud detection and risk scoring from live transaction streams * **E-commerce:** Order events and inventory updates processed in real time * **Telecommunications:** Customer behavior tracking and network event monitoring * **Data Synchronization:** Database change capture pipelines feeding warehouses or lakehouses * **AI and Machine Learning:** Real-time feature ingestion for model scoring Teams often start with simple pub-sub pipelines, then evolve into event-driven architectures and stream processing. Success requires investing in schema management, governance, and developer training to handle streaming's continuous nature compared to batch data. ## Summary Apache Kafka has evolved from a simple messaging system into the foundation of real-time data infrastructure. Its durability, scalability, and ability to integrate with processing and storage systems make it central to modern streaming architectures. Kafka enables continuous data flow between operational and analytical systems, bridging the gap between events and decisions. Yet operating it securely and transparently remains complex, a space where governance platforms provide value through access control, lineage, and observability. Understanding Kafka's fundamentals, topics, partitions, offsets, replication, and consumers, is essential for anyone designing reliable, event-driven systems in today's data landscape. ## Related Concepts - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Essential for managing message formats and ensuring compatibility across producers and consumers in Kafka. - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines) - Complements Kafka's streaming capabilities by enabling seamless integration with external data systems. - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Critical for building reliable streaming applications with strong delivery guarantees. ## Sources and References * [Apache Kafka Documentation](https://kafka.apache.org/documentation/) * [Confluent Schema Registry Documentation](https://docs.confluent.io/platform/current/schema-registry/index.html) * [Apache Flink Kafka Connector Docs](https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/) * [Debezium Documentation](https://debezium.io/documentation/reference/stable/connectors/index.html) * [Databricks Structured Streaming Kafka Guide](https://docs.databricks.com/en/structured-streaming/kafka.html) Source: https://www.conduktor.io/glossary/apache-kafka --- # API Gateway Patterns for Data Platforms Modern data platforms handle diverse data sources, multiple protocols, and complex access patterns. API gateways provide a critical abstraction layer that simplifies client access while enforcing security, governance, and performance controls. Understanding gateway patterns helps teams build scalable, secure data architectures. ## What is an API Gateway in Data Platforms? An API gateway is a server that provides a single entry point for multiple backend services. In data platforms, gateways sit between data consumers and data sources, whether those sources are REST APIs, databases, or streaming platforms like Apache Kafka. The gateway handles cross-cutting concerns that would otherwise need to be implemented in every service: authentication, rate limiting, protocol translation, logging, and request routing. This consolidation reduces complexity and creates consistent behavior across the platform. For data platforms specifically, API gateways solve several problems. They provide a unified interface to heterogeneous data sources, translate between protocols (like REST to Kafka), enforce data governance policies, and protect backend systems from overload. ## Core API Gateway Patterns ![## Core API Gateway Patterns](https://www.conduktor.io/assets/images/glossary/api-gateway-patterns-for-data-platforms-0.webp) ### Routing and Service Discovery The routing pattern directs requests to appropriate backend services based on URL paths, headers, or request content. A gateway might route `/customers/*` to a customer database API while `/orders/*` goes to an order processing service. Dynamic routing extends this by discovering services at runtime. When new data sources come online, the gateway automatically includes them without manual configuration changes. This pattern is essential in cloud-native data platforms where services scale dynamically. ### Protocol Translation Data platforms often need to bridge different communication protocols. A common scenario involves HTTP clients accessing Kafka topics. The gateway translates REST requests into Kafka producer calls and streams Kafka messages back as HTTP responses. Confluent's REST Proxy exemplifies this pattern, allowing HTTP clients to produce and consume Kafka messages without native Kafka client libraries. This enables broader ecosystem integration, though with performance tradeoffs: HTTP serialization overhead typically reduces throughput to ~20K events/second per deployment compared to 100K+ events/second for native Kafka clients. The tradeoff favors accessibility over raw performance, ideal for low-throughput integrations, web clients, and services where native Kafka libraries aren't feasible. ### Request and Response Aggregation The aggregation pattern combines multiple backend calls into a single gateway response. For example, a dashboard might need customer data, recent orders, and account balance. Instead of three separate client requests, the gateway fetches all three and returns a unified response. This pattern reduces network round trips and simplifies client code. However, it requires careful design to avoid creating overly complex gateway logic that duplicates business logic better placed in dedicated services. ### Rate Limiting and Throttling Protecting backend systems from overload is critical in data platforms. Rate limiting restricts how many requests a client can make in a time window. This prevents individual consumers from monopolizing resources or accidentally creating cascading failures. Throttling extends rate limiting by actively slowing down requests rather than rejecting them. A gateway might queue excess requests and process them at a sustainable rate. This smooths traffic spikes while maintaining system stability. ## API Gateways for Data Streaming Systems Streaming platforms like Apache Kafka present unique gateway requirements. Traditional HTTP gateways assume request-response patterns, but streaming systems are inherently asynchronous and continuous. ### Stream-Specific Gateway Patterns Kafka gateways must handle long-lived connections for consuming message streams. A WebSocket gateway can maintain persistent connections that stream Kafka messages to browsers or mobile apps. This enables real-time data delivery without constant polling. For comprehensive understanding of Kafka's core architecture and how topics, partitions, and consumer groups work, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ![api-gateway-patterns-for-data-platforms diagram 2](https://www.conduktor.io/assets/images/glossary/api-gateway-patterns-for-data-platforms-1.webp) Topic-based routing becomes the streaming equivalent of path-based routing. The gateway maps logical topics to physical Kafka topics, allowing topic reorganization without breaking client code. It can also merge multiple topics into a single consumer stream or split a single topic into multiple logical views. ### Real-World Example: Event-Driven API Gateway Consider an e-commerce platform using Kafka for event streaming. Mobile apps need real-time order updates, but implementing Kafka clients in mobile apps creates maintenance challenges and security risks. An API gateway solves this by exposing a WebSocket endpoint at `/orders/stream`. When a mobile app connects, the gateway: 1. Authenticates the user via JWT token 2. Determines which Kafka topics the user can access 3. Creates a Kafka consumer scoped to that user's orders 4. Streams order events back through the WebSocket connection 5. Manages consumer lifecycle, including offset commits and reconnection handling **Important**: The gateway must carefully manage consumer group membership and offset commits. Options include: - **Unique consumer group per WebSocket session**: Simplest but resource-intensive - **Shared consumer group with external offset storage**: Store delivery tracking in Redis/database - **Stateful gateway instances**: Use sticky sessions so reconnections hit the same gateway instance The choice depends on delivery semantics (at-least-once vs at-most-once) and reconnection requirements. This pattern keeps Kafka internals hidden while providing real-time capabilities to clients. For detailed consumer group mechanics and offset management strategies, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). ### Specialized Kafka Gateways Specialized gateways extend traditional API gateway patterns specifically for Kafka environments. They sit between applications and Kafka clusters, enforcing policies that would be difficult to implement at the application level. These gateways support encryption and decryption of sensitive fields, ensuring that applications never see raw PII data. They also provide topic virtualization, where teams can access logical topic names that map to physical topics according to governance rules. For multi-tenant Kafka clusters, specialized gateways enforce tenant isolation by validating that applications only access their authorized topics and consumer groups. This prevents accidental cross-tenant data leakage while maintaining a unified Kafka infrastructure. ## Security and Governance Patterns ### Authentication and Authorization Gateways centralize authentication by validating credentials once rather than in every backend service. Common patterns include JWT validation, OAuth flows, and API key verification. Authorization determines what authenticated users can access. In data platforms, this often means enforcing row-level or column-level security. A gateway might filter datasets based on user roles or mask sensitive fields before returning data. ### Audit Logging and Compliance Comprehensive audit trails are essential for data governance. Gateways log every request with details like user identity, requested resources, timestamp, and response status. This creates a single source of truth for data access patterns. For compliance regimes like GDPR or HIPAA, gateways can enforce retention policies, log data lineage, and provide evidence of proper data handling. The centralized position makes gateways ideal for compliance enforcement. For comprehensive audit logging patterns in streaming environments, see [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). ## Performance and Scalability Considerations ### Caching Strategies Gateways can cache frequently accessed data to reduce backend load and improve response times. Time-based expiration works for relatively static data, while event-based invalidation suits data that changes in response to specific events. For streaming platforms, caching becomes more complex. A gateway might cache the latest message from each Kafka topic, serving recent data instantly while streaming updates as they arrive. This balances latency and freshness. ### Circuit Breaker Pattern When backend services fail, circuit breakers prevent cascading failures. The gateway monitors error rates and automatically stops sending requests to failing services. After a timeout, it allows test requests through to check if the service recovered. This pattern is critical in data platforms where a single slow database query could back up the entire system. Circuit breakers fail fast and preserve capacity for healthy services. ### Gateway Scalability The gateway itself must not become a bottleneck. Horizontal scaling runs multiple gateway instances behind a load balancer. Stateless gateway design ensures that any instance can handle any request. For high-throughput streaming scenarios, gateways may need specialized infrastructure. Kafka gateways benefit from co-location with Kafka brokers to minimize latency and network overhead. ## Real-World Implementation Example A financial services company processes millions of transactions daily through Kafka. Different teams need access to transaction data, but security requirements vary by team. Their gateway implementation includes: - API key authentication for internal services and OAuth for external partners - Role-based topic access where retail banking teams see only retail transactions - PII masking that redacts customer names and account numbers for analytics teams - Rate limiting of 1000 requests per minute per API key - Circuit breakers that fail over to cached data when Kafka is unavailable This architecture allows diverse consumers to access the same underlying data platform while maintaining strict security and performance guarantees. The gateway handles complexity that would otherwise require custom code in dozens of applications. ## Backend for Frontend (BFF) Pattern The BFF pattern addresses a common problem with monolithic API gateways: they accumulate client-specific logic that bloats the gateway and couples it to specific frontends. Instead of one gateway serving all clients, each client type gets its own optimized backend service. ### Why BFF Matters Different clients have different needs: - **Mobile apps** need minimal payloads (battery/bandwidth constraints) - **Web dashboards** need rich aggregated data - **IoT devices** need efficient binary protocols - **Partner APIs** need stable, versioned contracts A single gateway serving all these clients either: 1. Returns everything (over-fetching for most clients) 2. Returns minimal data (under-fetching, requiring multiple requests) 3. Grows complex conditional logic based on client type ### BFF Architecture ![### BFF Architecture](https://www.conduktor.io/assets/images/glossary/api-gateway-patterns-for-data-platforms-2.webp) ### BFF Benefits for Data Platforms - **Domain ownership**: Each frontend team owns their BFF, enabling independent deployment and iteration without coordinating with other teams. - **Optimized queries**: Mobile BFF fetches only fields needed for mobile UI, reducing Kafka message deserialization and database queries. - **Protocol flexibility**: Web BFF can offer GraphQL while IoT BFF uses efficient binary protocols (Protobuf, CBOR). - **Simplified testing**: Each BFF has a focused scope, making integration testing easier than testing a monolithic gateway with conditional paths. ### BFF Considerations - **Code duplication**: Common logic (authentication, rate limiting) may be duplicated across BFFs. Solve with shared libraries or a thin common gateway layer. - **Operational overhead**: Multiple BFFs mean multiple deployments. Container orchestration (Kubernetes) and infrastructure-as-code reduce this burden. - **When not to use BFF**: Small teams with simple requirements benefit from a unified gateway. BFF pattern pays off as client diversity and team count grow. ## Summary API gateways are essential components of modern data platforms, providing unified access to diverse data sources while enforcing security, governance, and performance policies. Core patterns like routing, protocol translation, aggregation, and rate limiting apply across all gateway implementations. For data streaming systems like Kafka, gateways enable new use cases by bridging HTTP and streaming protocols, managing long-lived connections, and enforcing fine-grained access controls. Patterns like circuit breakers and caching ensure gateways enhance rather than degrade system performance. Successful gateway implementations balance flexibility with governance. They simplify client access without creating a monolithic bottleneck. As data platforms grow in complexity, well-designed API gateways become increasingly critical to maintaining security, performance, and developer productivity. ## Related Concepts - [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming) - Access control mechanisms that API gateways enforce to protect streaming data platforms - [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) - Fine-grained authorization patterns that gateways can implement for Kafka-based data platforms - [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments) - Multi-tenant patterns that API gateways enable by providing isolation and resource controls ## Sources and References 1. Richardson, C. (2018). *Microservices Patterns: With Examples in Java*. Manning Publications. Chapter 8: External API Patterns. 2. Confluent Documentation. (2024). "REST Proxy API Reference." https://docs.confluent.io/platform/current/kafka-rest/api.html 3. AWS Architecture Center. (2024). "API Gateway Pattern." https://aws.amazon.com/architecture/ 4. Stopford, B. (2018). *Designing Event-Driven Systems*. O'Reilly Media. Chapter 11: Event Streams and APIs. Source: https://www.conduktor.io/glossary/api-gateway-patterns-for-data-platforms --- # Audit Logging for Streaming Platforms Audit logging is the practice of recording who did what, when, and where in a system. For streaming platforms like Apache Kafka, Apache Flink, or Pulsar, audit logging becomes both critical and challenging due to the distributed nature of these systems and the high volume of operations they handle. As organizations adopt streaming platforms for mission-critical workloads, they face increasing pressure to demonstrate compliance with regulations like GDPR, HIPAA, SOC2, and PCI-DSS. Audit logs provide the paper trail needed to prove that data is handled correctly, access is controlled, and changes are tracked. ## Why Audit Logging Matters for Streaming Platforms Streaming platforms sit at the heart of modern data architectures, moving sensitive data between systems in real-time. A single misconfigured topic ACL or unauthorized schema change can expose private information or break downstream applications. Audit logging serves several key purposes: - **Compliance and Regulatory Requirements**: Many regulations require organizations to maintain detailed records of who accessed what data and when. For example, GDPR requires demonstrating that personal data is protected, while SOC2 mandates tracking of administrative changes. - **Security Incident Investigation**: When a security breach occurs, audit logs help teams reconstruct what happened. Did someone delete a critical topic? Who modified the consumer group offsets? Which user updated the schema registry? - **Operational Troubleshooting**: Beyond security, audit logs help debug production issues. If a consumer suddenly stops processing messages, audit logs can reveal whether someone changed the ACLs, modified the schema, or altered the topic configuration. - **Change Management**: In large organizations, multiple teams may interact with the same streaming platform. Audit logs provide accountability and help track changes across team boundaries. ## Unique Challenges in Streaming Environments Unlike traditional databases with centralized audit logs, streaming platforms present unique challenges: - **Distributed Architecture**: Kafka clusters consist of multiple brokers, ZooKeeper/KRaft nodes, Schema Registries, and Connect workers. Each component has its own logs and security model, making centralized auditing difficult. - **High Volume of Events**: Streaming platforms process millions of messages per second. While you typically don't audit every message (that would be prohibitive), you do need to audit administrative operations, which can still generate substantial log volume. - **Multiple Access Patterns**: Users interact with streaming platforms through various interfaces: CLI tools, REST APIs, web UIs, and programmatic clients. Each access path needs consistent audit logging. - **Immutability Concerns**: Audit logs themselves must be tamper-proof. If someone can modify or delete audit logs, they lose their value for compliance and security. ## What Should Be Audited ![Audit Logging Flow for Streaming Platforms](https://www.conduktor.io/assets/images/glossary/audit-logging-for-streaming-platforms-0.webp) A comprehensive audit logging strategy for streaming platforms should capture several categories of events: ### Administrative Operations - **Topic Management**: Creating, deleting, or modifying topics (partition count, retention, compaction settings) - **ACL Changes**: Granting or revoking permissions for users or service accounts - **Configuration Updates**: Changes to broker configurations, quotas, or cluster settings - **User Management**: Creating, modifying, or deleting service accounts and API keys ### Schema Registry Events - **Schema Evolution**: Registering new schemas, updating compatibility settings - **Schema Deletion**: Removing schemas (soft or hard deletion) - **Access Control**: Changes to schema-level permissions ### Consumer Group Operations - **Offset Resets**: Manual offset resets can cause duplicate processing or data loss - **Group Membership**: Tracking which consumers join or leave groups - **Group Deletion**: Removing consumer groups and their committed offsets ### Connect and Stream Processing - **Connector Lifecycle**: Creating, updating, pausing, or deleting connectors - **Task Failures**: Failed tasks that might indicate configuration or permission issues ## Implementation Patterns ### Native Kafka Audit Logging Apache Kafka itself provides limited built-in audit logging through authorizer plugins. The `AclAuthorizer` logs authorization decisions, but these logs are scattered across broker log files and can be difficult to aggregate and analyze. For production environments, organizations typically implement custom authorizers that write audit events to a dedicated Kafka topic. This approach has several advantages: ``` Topic: audit-logs Message: { "timestamp": "2025-12-07T10:30:45Z", "user": "data-eng-team", "operation": "DELETE_TOPIC", "resource": "customer-events", "outcome": "SUCCESS", "source_ip": "10.0.1.45" } ``` By storing audit logs in Kafka itself, you get the platform's durability and retention capabilities. The audit topic can be consumed by SIEM systems, stored in long-term archival storage, or analyzed in real-time for suspicious activity. ### Centralized Audit Collection For environments with multiple Kafka clusters, Schema Registries, and Connect clusters, centralized audit collection becomes essential. This typically involves: 1. **Standardized Audit Format**: Define a common schema for all audit events across components 2. **Central Audit Topic or Database**: Aggregate logs from all sources into a queryable store 3. **Retention Policies**: Implement long-term retention (often 7+ years for compliance) 4. **Access Controls**: Protect audit logs with strict permissions to prevent tampering Governance platforms provide built-in audit logging that captures operations across the entire Kafka ecosystem through a unified interface. When users create topics, modify schemas, or change ACLs, these actions are automatically logged with full context (who, what, when, why). This centralized approach simplifies compliance reporting and reduces the burden of implementing custom audit solutions. ### Real-World Example: Schema Registry Audit Trail Consider a financial services company using Kafka to stream transaction data. They use Avro schemas in Schema Registry to ensure data quality. Here's why audit logging matters: A developer accidentally updates a schema with a breaking change, removing a required field. Within minutes, downstream applications start failing. Without audit logs, the team would need to manually review recent schema versions and guess who made the change. With proper audit logging, the team immediately sees: ``` 2025-12-07 14:23:18 | user: john.doe@company.com operation: REGISTER_SCHEMA subject: transactions-value version: 12 compatibility: BACKWARD result: SUCCESS ``` The team can contact John directly, understand the reasoning, and quickly roll back to version 11 while they fix the issue. ## Best Practices - **Immutable Storage**: Write audit logs to append-only storage or use Kafka topics with `retention.ms=-1` (infinite retention) and strict ACLs preventing deletion. Note: `delete.retention.ms` is a different setting used for tombstone retention in compacted topics, not for general retention. - **Include Context**: Capture not just the action, but the context: source IP, authentication method, API client version, and any relevant metadata. - **Separate Audit Traffic**: Use dedicated topics or partitions for audit logs to prevent them from being mixed with application data. - **Monitor Audit Logs**: Set up alerts for suspicious patterns like bulk deletions, privilege escalations, or access from unusual IP addresses. - **Regular Review**: Compliance frameworks require periodic review of audit logs. Automate reports for security and compliance teams. - **Test Restoration**: Verify that audit logs can be restored from backups. An audit log you can't retrieve during an investigation is worthless. ## Integration with Compliance Frameworks Different compliance frameworks have specific audit logging requirements: - **SOC2**: Requires tracking all administrative changes, user access, and data modifications with timestamps and user attribution. - **GDPR**: Mandates logging access to personal data, data deletions (right to be forgotten), and data exports (data portability). For comprehensive GDPR implementation guidance, see [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams). - **HIPAA**: Requires tracking all access to protected health information (PHI), including who viewed what data and when. - **PCI-DSS**: Demands detailed logs of access to cardholder data environments, with secure storage and regular review. Streaming platforms need to map their audit events to these framework requirements. Tracking what data flows where becomes essential, for comprehensive guidance on building this visibility, see [Data Lineage Tracking: Data from Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption). For a worked example of mapping these audit requirements to a regulated financial system on Kafka, see [When Kafka Becomes the Ledger](https://www.conduktor.io/resources/ebooks/when-kafka-becomes-the-ledger). For example, a GDPR-compliant audit log might track: - Which topics contain personal data (through metadata tagging) - Who accessed those topics (consumer groups, connect workers) - Data deletion operations (topic cleanup, compaction) - Export operations (connector pulls for user data requests) ## Summary Audit logging is a fundamental requirement for operating streaming platforms in regulated environments. While Apache Kafka and related technologies provide some basic audit capabilities, production deployments typically require custom solutions or dedicated platforms to achieve comprehensive, centralized audit logging. Key takeaways: - Audit logs must capture administrative operations, schema changes, ACL modifications, and consumer group operations - The distributed nature of streaming platforms makes centralized audit collection challenging but essential - Audit logs should be immutable, context-rich, and stored with long retention periods - Different compliance frameworks have specific requirements that must be mapped to streaming platform operations - Governance platforms can simplify audit logging by providing unified audit trails across the entire Kafka ecosystem By implementing proper audit logging, organizations gain the visibility needed for compliance, security incident response, and operational troubleshooting. ## Related Concepts - [Streaming Audit Logs](https://www.conduktor.io/glossary/streaming-audit-logs) - Implementing audit trails for real-time systems - [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming) - Automated governance with audit trails - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) - Organizational structure for audit compliance ## Sources and References - [Apache Kafka Documentation - Security](https://kafka.apache.org/documentation/#security) - Official documentation on Kafka authorization and security features - [Confluent - Audit Logs](https://docs.confluent.io/platform/current/security/audit-logs/audit-logs-concepts.html) - Confluent's approach to audit logging in Confluent Platform and Confluent Cloud - [NIST Special Publication 800-92 - Guide to Computer Security Log Management](https://csrc.nist.gov/publications/detail/sp/800-92/final) - Comprehensive guide on security log management best practices - [SOC 2 Compliance Requirements](https://www.aicpa.org/interestareas/frc/assuranceadvisoryservices/aicpasoc2report.html) - AICPA's framework for security, availability, and confidentiality controls - [GDPR Article 30 - Records of Processing Activities](https://gdpr-info.eu/art-30-gdpr/) - Requirements for maintaining records of data processing Source: https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms --- # Automated Data Quality Testing: A Practical Guide for Modern Data Pipelines Data quality issues cost organizations millions annually through incorrect analytics, failed ML models, and broken downstream systems. As data pipelines grow more complex, especially with real-time streaming architectures, manual testing becomes impractical. Automated data quality testing is essential for maintaining trust in your data infrastructure. ## Why Automate Data Quality Testing? Traditional manual data validation doesn't scale. When dealing with hundreds of data sources, schema evolution, and continuous data flows, you need systematic, automated approaches to catch issues before they propagate downstream. Automated testing provides: - **Early detection**: Catch schema changes, null values, and data anomalies immediately - **Continuous validation**: Test data quality in real-time as it flows through pipelines - **Regression prevention**: Ensure transformations don't break existing data contracts - **Documentation**: Tests serve as executable specifications of data expectations ## Core Testing Dimensions ![Data Quality Testing Dimensions](https://www.conduktor.io/assets/images/glossary/automated-data-quality-testing-0.webp) Effective data quality testing covers multiple dimensions. For broader context on quality dimensions and how they relate to organizational data strategy, see [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). ### 1. Schema Validation Ensure data structures match expected schemas, particularly critical in streaming environments where schema evolution can break consumers. ```python from pydantic import BaseModel, Field, field_validator from typing import Optional from datetime import datetime class UserEvent(BaseModel): user_id: str = Field(..., min_length=1) event_type: str timestamp: datetime properties: dict revenue: Optional[float] = None @field_validator('revenue') @classmethod def validate_revenue(cls, v): if v is not None and v < 0: raise ValueError('Revenue cannot be negative') return v @field_validator('event_type') @classmethod def validate_event_type(cls, v): allowed_types = {'page_view', 'click', 'purchase', 'signup'} if v not in allowed_types: raise ValueError(f'Event type must be one of {allowed_types}') return v # Validation in streaming pipeline def validate_event(event_json: dict) -> bool: try: UserEvent(**event_json) return True except Exception as e: log_validation_error(e, event_json) return False ``` ### 2. Data Completeness Check for missing values, null rates, and required field presence. ```python import pandas as pd from typing import List, Dict class CompletenessValidator: def __init__(self, required_fields: List[str], max_null_rate: float = 0.05): self.required_fields = required_fields self.max_null_rate = max_null_rate def validate_batch(self, df: pd.DataFrame) -> Dict[str, any]: results = { 'passed': True, 'issues': [] } # Check required fields exist missing_fields = set(self.required_fields) - set(df.columns) if missing_fields: results['passed'] = False results['issues'].append(f'Missing required fields: {missing_fields}') # Check null rates for field in self.required_fields: if field in df.columns: null_rate = df[field].isna().sum() / len(df) if null_rate > self.max_null_rate: results['passed'] = False results['issues'].append( f'{field} null rate {null_rate:.2%} exceeds threshold {self.max_null_rate:.2%}' ) return results # Usage validator = CompletenessValidator(required_fields=['user_id', 'timestamp', 'event_type']) validation_result = validator.validate_batch(events_df) assert validation_result['passed'], f"Validation failed: {validation_result['issues']}" ``` ### 3. Statistical Validation Detect anomalies using statistical boundaries and historical patterns. ```python import numpy as np from scipy import stats class StatisticalValidator: def __init__(self, historical_mean: float, historical_std: float, z_threshold: float = 3.0): self.mean = historical_mean self.std = historical_std self.z_threshold = z_threshold def validate_metric(self, current_value: float) -> Dict[str, any]: z_score = abs((current_value - self.mean) / self.std) is_anomaly = z_score > self.z_threshold return { 'value': current_value, 'z_score': z_score, 'is_anomaly': is_anomaly, 'expected_range': ( self.mean - self.z_threshold * self.std, self.mean + self.z_threshold * self.std ) } # Example: Validate daily record count daily_count_validator = StatisticalValidator( historical_mean=1_000_000, historical_std=50_000, z_threshold=3.0 ) current_count = 1_200_000 result = daily_count_validator.validate_metric(current_count) if result['is_anomaly']: alert(f"Anomalous record count: {current_count} (z-score: {result['z_score']:.2f})") ``` ## Streaming Data Quality with Kafka For streaming pipelines, data quality testing must happen in real-time. When using Apache Kafka, streaming management tools provide valuable capabilities for monitoring and validating data quality in real-time contexts. For foundational understanding of Kafka architecture and streaming patterns, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ### Real-Time Validation Pattern ```python from kafka import KafkaConsumer, KafkaProducer import json class StreamingQualityTester: def __init__(self, bootstrap_servers: str, consumer_group: str = 'quality-tester'): self.consumer = KafkaConsumer( 'raw-events', bootstrap_servers=bootstrap_servers, group_id=consumer_group, auto_offset_reset='earliest', enable_auto_commit=False, # Manual commit for exactly-once value_deserializer=lambda m: json.loads(m.decode('utf-8')) ) self.valid_producer = KafkaProducer( bootstrap_servers=bootstrap_servers, acks='all', # Wait for all replicas value_serializer=lambda m: json.dumps(m).encode('utf-8') ) self.dlq_producer = KafkaProducer( bootstrap_servers=bootstrap_servers, acks='all', value_serializer=lambda m: json.dumps(m).encode('utf-8') ) self.metrics = { 'processed': 0, 'valid': 0, 'invalid': 0 } def run(self): try: for message in self.consumer: self.metrics['processed'] += 1 event = message.value # Run validation suite validation_results = self.validate_event(event) if validation_results['passed']: self.valid_producer.send('validated-events', event) self.metrics['valid'] += 1 else: # Send to dead letter queue with error details error_event = { **event, '_validation_errors': validation_results['errors'], '_original_topic': 'raw-events', '_timestamp': message.timestamp } self.dlq_producer.send('validation-dlq', error_event) self.metrics['invalid'] += 1 # Commit offset after successful processing self.consumer.commit() # Report metrics periodically if self.metrics['processed'] % 1000 == 0: self.report_metrics() finally: # Ensure clean shutdown self.consumer.close() self.valid_producer.close() self.dlq_producer.close() def validate_event(self, event: dict) -> Dict[str, any]: errors = [] # Schema validation try: UserEvent(**event) except Exception as e: errors.append(f'Schema validation failed: {str(e)}') # Business rule validation if event.get('event_type') == 'purchase' and not event.get('revenue'): errors.append('Purchase events must include revenue') return { 'passed': len(errors) == 0, 'errors': errors } def report_metrics(self): success_rate = self.metrics['valid'] / self.metrics['processed'] * 100 print(f"Processed: {self.metrics['processed']}, " f"Success Rate: {success_rate:.2f}%") ``` ### Quality Monitoring for Kafka Streams Streaming management platforms provide visual monitoring and testing capabilities for Kafka streams: - **Monitor schema registry**: Track schema evolution and catch breaking changes. For details on schema management patterns, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). - **Validate message format**: Configure validation rules for incoming data. For implementing validation with Conduktor, see [Enforcing Data Quality](https://docs.conduktor.io/guide/use-cases/enforce-data-quality). - **Dead letter queue management**: Easily inspect and replay failed messages. For error handling patterns, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). Set up quality gates to automatically route messages through validation topics, making it easy to visualize quality metrics and troubleshoot issues. For observing data quality metrics, see [Observing Data Quality with Conduktor](https://docs.conduktor.io/guide/use-cases/observe-data-quality). ### Continuous Validation with Data Quality Policies Conduktor Data Quality Policies complement automated testing by providing infrastructure-level continuous validation. Data Quality Policies create Rules defining expected message formats and content that attach to specific topics, creating a centralized quality enforcement layer. In observe-only mode, Policies record violations without impacting message flow; when integrated with Conduktor Gateway (a Kafka proxy), they validate records before production, blocking or marking non-compliant messages. This layered approach combines development-time testing with production-time enforcement, catching edge cases that testing environments might miss. For implementation patterns, see [Data Quality Policies](https://docs.conduktor.io/guide/conduktor-concepts/data-quality-policies). ## Implementing a Testing Framework Build a comprehensive testing framework that runs continuously: ```python class DataQualityTestSuite: def __init__(self, name: str): self.name = name self.tests = [] self.results = [] def add_test(self, test_name: str, test_func: callable): self.tests.append((test_name, test_func)) def run(self, data: any) -> bool: self.results = [] all_passed = True for test_name, test_func in self.tests: try: result = test_func(data) self.results.append({ 'test': test_name, 'passed': result['passed'], 'details': result.get('details', '') }) if not result['passed']: all_passed = False except Exception as e: self.results.append({ 'test': test_name, 'passed': False, 'details': f'Test failed with exception: {str(e)}' }) all_passed = False return all_passed def generate_report(self) -> str: report = [f"\nTest Suite: {self.name}"] report.append("=" * 50) for result in self.results: status = "PASS" if result['passed'] else "FAIL" report.append(f"[{status}] {result['test']}") if result['details']: report.append(f" Details: {result['details']}") passed = sum(1 for r in self.results if r['passed']) total = len(self.results) report.append(f"\nResults: {passed}/{total} tests passed") return "\n".join(report) ``` ## Best Practices 1. **Test early and often**: Validate data at ingestion, transformation, and output stages 2. **Separate validation logic**: Keep quality tests decoupled from business logic 3. **Monitor quality metrics**: Track validation success rates, common failure patterns 4. **Design for failure**: Use dead letter queues and graceful degradation 5. **Version your tests**: Treat quality tests as code with proper version control 6. **Balance strictness**: Too strict validation creates false positives; too lenient misses real issues ## Summary Automated data quality testing shifts data reliability from a reactive problem to a proactive one. By validating across schema, completeness, and statistical dimensions — particularly in streaming architectures — you build data systems that teams can trust. For production-grade implementation using established frameworks, see [Great Expectations: Data Testing Framework](https://www.conduktor.io/glossary/great-expectations-data-testing-framework). For formal agreements between data producers and consumers, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). Automated testing reduces debugging time, improves confidence in data-driven decisions, and speeds up incident resolution. ## Related Concepts - [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) - Comprehensive approach to quality management - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Schema validation infrastructure - [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) - Contract-based validation strategies ## Sources and References - [Great Expectations Documentation](https://docs.greatexpectations.io/) - Leading open-source framework for data validation and quality testing - [Apache Kafka Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html) - Schema validation and evolution management for streaming data - [Pydantic Data Validation](https://docs.pydantic.dev/) - Python library for data validation using type annotations - [dbt Data Testing](https://docs.getdbt.com/docs/build/tests) - Testing framework for analytics engineering and data transformations - [AWS Glue Data Quality](https://docs.aws.amazon.com/glue/latest/dg/glue-data-quality.html) - Automated data quality monitoring for data pipelines Source: https://www.conduktor.io/glossary/automated-data-quality-testing --- # Avro vs Protobuf vs JSON Schema: Kafka Serialization Compared (2026) **Avro, Protobuf, and JSON Schema** are the three schema formats most used with Kafka. Avro is built for schema evolution and pairs with a Schema Registry so each message carries only a compact schema ID; Protobuf compiles to fast, strongly typed code across languages; JSON Schema validates human-readable JSON with no code generation step. The choice sets serialization speed, payload size, and how safely a schema can change once producers and consumers are live. This article compares these three popular schema formats, examining their design philosophies, serialization approaches, and practical tradeoffs in real-world applications. ## Introduction to Schema Formats ![Schema Format Comparison Overview](https://www.conduktor.io/assets/images/glossary/avro-vs-protobuf-vs-json-schema-0.webp) Schema formats serve two primary purposes: defining the structure of data and enabling validation. In distributed systems, schemas provide a contract between producers and consumers, ensuring data compatibility across services and over time. Without schemas, systems rely on implicit assumptions about data structure. When a service changes its data format, downstream consumers may break unexpectedly. Schemas make these contracts explicit and machine-readable. The three formats covered here take different approaches. Avro and Protobuf focus on compact binary serialization with strong schema evolution support. JSON Schema focuses on validating human-readable JSON documents. Understanding these differences helps you choose the right tool for your use case. ## Understanding Binary Serialization First, a note on what "binary serialization" means. When you serialize data to binary format, you convert it into a sequence of bytes that machines can process efficiently. Unlike human-readable text formats (like JSON), binary formats optimize for: - **Size**: Compressed representation using techniques like variable-length integers - **Speed**: Direct memory mapping without parsing text - **Schema enforcement**: Structure embedded in the format itself The tradeoff is readability, you can't open a binary file in a text editor and understand it. You need the schema and appropriate deserialization tools. ## Avro: Compact and Schema Evolution Apache Avro is a binary serialization format created within the Hadoop ecosystem. It stores data in a compact binary format while keeping the schema separate from the data itself. Avro's defining characteristic is its approach to schema evolution. When reading Avro data, you provide both the writer's schema (used when data was written) and the reader's schema (expected by the current application). Avro resolves differences between these schemas at read time, enabling backward and forward compatibility. **Key features**: - Binary encoding with small message sizes - Schema stored separately from data (often in a Schema Registry) - Dynamic typing - no code generation required - Rich support for schema evolution with resolution rules **Example Avro schema**: ```json { "type": "record", "name": "User", "fields": [ {"name": "id", "type": "long"}, {"name": "username", "type": "string"}, {"name": "email", "type": ["null", "string"], "default": null} ] } ``` Avro excels when schema evolution is critical. Adding a field with a default value allows old readers to process new data and new readers to process old data without changes. This makes Avro popular in data lakes and streaming platforms where data schemas change frequently. ## Protocol Buffers: Performance and Cross-Language Support Protocol Buffers (Protobuf) is Google's language-neutral serialization format. Unlike Avro, Protobuf uses code generation to create strongly-typed classes in your target programming language. Protobuf achieves excellent performance through its compact binary format and the use of generated code. The schema, defined in `.proto` files, specifies field numbers that are used in the binary encoding. This makes Protobuf messages extremely efficient. **Key features**: - Very compact binary encoding - Strong typing through code generation - Field numbers enable schema evolution - Extensive language support with official compilers **Example Protobuf schema**: ```protobuf syntax = "proto3"; message User { int64 id = 1; string username = 2; optional string email = 3; } ``` Schema evolution in Protobuf relies on field numbers. You can add new fields, mark fields as deprecated, or change field names without breaking compatibility, as long as field numbers remain stable. Removing fields requires careful consideration of backward compatibility. Protobuf's code generation approach provides compile-time type safety but requires a build step. This makes it particularly popular in microservices architectures where strong typing and performance are priorities. ## JSON Schema: Human-Readable Validation JSON Schema takes a fundamentally different approach. Rather than defining a serialization format, JSON Schema validates JSON documents. The data remains human-readable JSON text. JSON Schema defines rules for what constitutes valid JSON. It specifies types, required fields, format constraints, and complex validation logic. This makes it powerful for API validation and configuration files where human readability matters. **Key features**: - Validates standard JSON (no custom serialization) - Human-readable data and schema - Rich validation capabilities (regex, ranges, custom formats) - No code generation or build step required **Example JSON Schema**: ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "id": {"type": "integer"}, "username": {"type": "string"}, "email": {"type": "string", "format": "email"} }, "required": ["id", "username"] } ``` The tradeoff with JSON Schema is efficiency. JSON text is significantly larger than binary formats, and parsing JSON is slower than deserializing binary data. However, for use cases where debugging, human inspection, or web APIs are important, this tradeoff may be acceptable. JSON Schema's evolution model is less formalized than Avro or Protobuf. You can add optional fields or loosen validation rules without breaking existing documents, but there's no built-in schema resolution mechanism. ## Schema Formats in Data Streaming All three formats are commonly used with Apache Kafka and other streaming platforms, but with different patterns and tradeoffs. - **Avro with Kafka** is perhaps the most common combination. Kafka messages contain only the binary data and a schema ID. The Schema Registry stores schemas centrally and ensures compatibility between producers and consumers. Confluent's implementation provides automatic schema validation and evolution checks. - **Protobuf with Kafka** follows a similar pattern but leverages code generation. Producers and consumers use generated classes, providing type safety at compile time. Schema Registry support for Protobuf enables the same centralized schema management as Avro. - **JSON Schema with Kafka** is used when human readability is valuable, such as in development environments or when non-technical users need to inspect messages. The larger message size impacts throughput and storage costs in high-volume scenarios. - **Real-world example**: A financial services company might use Protobuf for high-frequency trading data (millions of messages per second), Avro for analytical data lakes (frequent schema changes), and JSON Schema for configuration events (human readability for operations teams). Tools like Conduktor provide unified interfaces for working with all three formats. Developers can inspect messages, validate schemas, and test compatibility rules without writing custom deserialization code. This is particularly valuable in environments where different teams choose different formats for different use cases. For managing schemas across these formats, see [Conduktor's Schema Registry Management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry). ## Schema Evolution and Compatibility All three formats support schema evolution, but managing compatibility becomes critical in production systems. When using Schema Registry (Confluent Schema Registry, AWS Glue Schema Registry), you configure compatibility modes that determine which schema changes are allowed: - **Backward compatibility**: New schemas can read old data (most common, consumers upgrade before producers) - **Forward compatibility**: Old schemas can read new data (producers upgrade before consumers) - **Full compatibility**: Both backward and forward compatible (most restrictive) For a comprehensive guide to schema compatibility modes, validation rules, and practical evolution patterns, see the [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) article. Schema Registry works with all three formats covered here (Avro, Protobuf, JSON Schema) and provides the safety guardrails needed to evolve schemas without breaking production systems. ## Choosing the Right Format The choice between these formats depends on several factors: **Choose Avro when**: - Schema evolution is a primary concern - You need dynamic typing and runtime flexibility - Working with data lakes or analytical workloads - Integration with Hadoop ecosystem tools **Choose Protobuf when**: - Maximum performance is critical - You want strong compile-time type safety - Building microservices with multiple languages - Code generation fits your development workflow **Choose JSON Schema when**: - Human readability is essential - Working with web APIs or browser clients - Schema validation without custom serialization - Developer tooling and debugging are priorities - **Performance comparison**: In typical benchmarks, Protobuf produces the smallest messages (often 20-30% smaller than Avro), followed by Avro, with JSON being 3-5x larger than binary formats. Serialization and deserialization speed follows similar patterns. - **Ecosystem and tooling**: All three formats have mature ecosystems. Avro benefits from Hadoop integration. Protobuf has official Google support and extensive language bindings. JSON Schema has ubiquitous support in web technologies. Many organizations don't choose just one format. They use Protobuf for high-performance services, Avro for data pipelines, and JSON Schema for APIs. The key is establishing clear guidelines for when to use each format and ensuring your infrastructure supports all chosen formats. ## Summary Apache Avro, Protocol Buffers, and JSON Schema each excel in different scenarios. Avro provides excellent schema evolution with dynamic typing, making it ideal for data pipelines where schemas change frequently. Protocol Buffers delivers maximum performance and type safety through code generation, fitting high-throughput microservices. JSON Schema prioritizes human readability and validation of standard JSON, serving web APIs and configuration use cases. In data streaming environments like Apache Kafka, all three formats integrate well with Schema Registry for centralized schema management. The choice depends on your specific requirements around performance, type safety, schema evolution, and human readability. Understanding these tradeoffs lets you pick the right format for each component. Many production systems use multiple formats, applying each where its strengths fit the requirements. ## Related Concepts - [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka) - Comprehensive guide to how Kafka uses Avro, Protobuf, and JSON Schema for message serialization and deserialization. - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Learn how Schema Registry provides centralized management for all three schema formats with compatibility enforcement. - [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) - Best practices for evolving schemas safely across different serialization formats while maintaining compatibility. ## Sources and References 1. [Apache Avro Documentation](https://avro.apache.org/docs/) - Official specification and schema evolution rules 2. [Protocol Buffers Language Guide](https://protobuf.dev/programming-guides/proto3/) - Google's official Protobuf documentation 3. [JSON Schema Specification](https://json-schema.org/specification) - Complete JSON Schema standards 4. [Confluent Schema Registry Documentation](https://docs.confluent.io/platform/current/schema-registry/index.html) - Multi-format schema management for Kafka 5. [Schema Evolution in Avro, Protocol Buffers and Thrift](https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html) - Martin Kleppmann's technical analysis of schema evolution approaches Source: https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema --- # Azure Event Hubs and Kafka Compatibility Azure Event Hubs is Microsoft's fully managed, cloud-native event streaming platform designed to handle millions of events per second. One of its most significant features is built-in compatibility with the Apache Kafka protocol, allowing organizations to use existing Kafka client applications with minimal changes. This compatibility bridges the gap between on-premises Kafka deployments and Azure's managed services, offering a path for cloud migration and hybrid architectures. Understanding how Event Hubs implements Kafka compatibility is important for architects and engineers evaluating cloud streaming options or planning migrations. ## Understanding Kafka Protocol Compatibility ![Azure Event Hubs Kafka Protocol Compatibility](https://www.conduktor.io/assets/images/glossary/azure-event-hubs-and-kafka-compatibility-0.webp) Azure Event Hubs implements the Apache Kafka producer and consumer APIs, supporting Kafka clients version 1.0 and later. This means applications written using standard Kafka client libraries can connect to Event Hubs by simply changing the connection configuration. The compatibility layer works by translating Kafka protocol requests into Event Hubs operations. When a Kafka producer sends messages, Event Hubs receives them as events. Similarly, Kafka consumers can read from Event Hubs partitions using familiar consumer group semantics. ### How It Works Event Hubs exposes a Kafka-compatible endpoint that listens on port 9093 (standard Kafka SSL port). To connect a Kafka client, you configure the bootstrap server to point to your Event Hubs namespace and provide authentication credentials using SASL/PLAIN or SASL/OAUTHBEARER. Example connection configuration: ``` bootstrap.servers=your-namespace.servicebus.windows.net:9093 security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="[connection-string]"; ``` In this configuration, an Event Hub maps to a Kafka topic, and partitions in Event Hubs map directly to Kafka partitions. This one-to-one mapping preserves ordering guarantees and partition assignment logic. ## Migration Strategies and Use Cases Organizations adopt Event Hubs with Kafka compatibility for several strategic reasons: - **Cloud Migration**: Companies moving to Azure can migrate Kafka workloads without rewriting applications. By pointing existing Kafka producers and consumers to Event Hubs endpoints, teams can validate cloud performance before committing to deeper architectural changes. - **Hybrid Architectures**: Some organizations maintain on-premises Kafka clusters while using Event Hubs for specific workloads. For example, a financial services company might keep sensitive trading data on-premises while streaming analytics events to Azure for processing with Azure Stream Analytics or Azure Databricks. - **Managed Service Benefits**: Event Hubs eliminates operational overhead associated with Kafka cluster management. Teams avoid tasks like broker configuration, partition rebalancing, and version upgrades, allowing them to focus on application development. - **Integration with Azure Ecosystem**: Event Hubs natively integrates with Azure services like Azure Functions, Azure Data Explorer, and Azure Synapse Analytics. Applications can stream data through Event Hubs using Kafka protocols while downstream processing leverages Azure-native tools. ## Feature Comparison: Event Hubs vs Native Kafka While Event Hubs supports Kafka protocols, it's not a drop-in replacement for all Kafka features. Understanding these differences is critical for migration planning. ### Supported Features - Producer and consumer APIs (Kafka 1.0+) - Consumer groups with offset management - Partition-level ordering - Idempotent producers - Transactional writes (in premium and dedicated tiers) ### Limitations and Differences - **Kafka Streams API (Preview)**: Event Hubs now supports Kafka Streams API in preview for Premium and Dedicated tiers, enabling stateful stream processing. However, production use should consider Azure Stream Analytics or Flink as mature alternatives. - **No Native Kafka Connect**: Event Hubs doesn't provide a native Connect framework. Connectors must run separately and connect to Event Hubs as a Kafka endpoint. - **Topic Administration**: Creating and configuring Event Hubs requires using Azure Portal, CLI, or ARM templates rather than Kafka admin APIs. - **Log Compaction (GA)**: Event Hubs now supports log compaction in General Availability across all tiers, enabling key-value store patterns and stateful applications. - **Message Size**: Event Hubs supports up to 1 MB messages in Standard tier and up to 20 MB in Dedicated tier (versus Kafka's configurable limits). These limitations mean Event Hubs works best for producer-consumer patterns rather than complex stream processing or connector-heavy architectures. ## Working with Event Hubs Kafka Endpoint Migrating a Kafka application to Event Hubs typically involves four steps: 1. **Create Event Hubs Namespace**: Provision an Event Hubs namespace in Azure, selecting the appropriate tier (Standard, Premium, or Dedicated) based on throughput and feature requirements. 2. **Configure Connection Strings**: Retrieve the connection string from Azure Portal and configure Kafka clients to authenticate using SASL. 3. **Map Topics to Event Hubs**: Each Kafka topic requires a corresponding Event Hub. Create these through Azure Portal or Infrastructure as Code templates. 4. **Test and Validate**: Deploy applications to a staging environment and verify message production, consumption, and consumer group behavior. A real-world example: An e-commerce platform migrated its order processing pipeline from self-managed Kafka to Event Hubs. The team changed only connection configuration in their Spring Boot microservices, deployed to Azure Kubernetes Service, and achieved 99.9% uptime with reduced operational burden. ## Monitoring and Management in Multi-Protocol Environments Operating Event Hubs with Kafka clients introduces monitoring challenges. Azure provides native metrics through Azure Monitor, but teams accustomed to Kafka tooling may find the transition difficult. Key monitoring considerations: - **Partition Metrics**: Track throughput, lag, and consumer group offsets across partitions - **Error Rates**: Monitor authentication failures, throttling events, and protocol errors - **Schema Registry**: Event Hubs now includes integrated Schema Registry (free with every namespace), providing schema validation and evolution management compatible with Confluent Schema Registry APIs - **Express Tier**: New cost-effective tier for dev/test workloads with auto-inflate capabilities Management platforms help teams operate multi-protocol streaming environments by providing unified monitoring, schema management, and data governance across both native Kafka clusters and Event Hubs. This is particularly valuable for organizations running hybrid architectures or gradually migrating workloads to Azure. Teams should also establish alerting for consumer lag and throughput saturation to maintain service level objectives during migration. ## Summary Azure Event Hubs provides Kafka protocol compatibility that lets organizations reuse existing Kafka clients while adopting Azure's managed streaming platform. The compatibility layer supports core producer and consumer patterns, making it suitable for lift-and-shift migrations and hybrid cloud architectures. Event Hubs has evolved significantly: Kafka Streams API is now in preview, log compaction is GA, integrated Schema Registry is available for free, and new tiers like Express provide cost-effective options. However, teams should carefully evaluate their specific use cases, particularly around stream processing maturity and connector dependencies, before migrating. For organizations committed to Azure, Event Hubs offers significant operational benefits: reduced management overhead, native integration with Azure services, and enterprise-grade reliability. When combined with proper monitoring and governance tools, Event Hubs can serve as an effective foundation for cloud-native event streaming architectures. The decision to adopt Event Hubs involves balancing protocol compatibility benefits against feature limitations, cost, and long-term architectural goals. For many teams, the combination of Kafka familiarity and Azure ecosystem integration makes Event Hubs a practical choice for cloud streaming. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - The streaming platform that Event Hubs provides compatibility with - [Introduction to Confluent Cloud](https://www.conduktor.io/glossary/introduction-to-confluent-cloud) - Alternative managed Kafka service for multi-cloud deployments - [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership) - Comparing the economics of managed streaming platforms ## Sources and References 1. [Azure Event Hubs for Apache Kafka](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview) - Microsoft Learn 2. [Apache Kafka Protocol Documentation](https://kafka.apache.org/protocol) - Apache Software Foundation 3. [Azure Event Hubs quotas and limits](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) - Microsoft Learn 4. [Migrating to Azure Event Hubs for Apache Kafka](https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-migration-guide) - Microsoft Learn 5. [Azure Architecture Center - Event-driven architecture](https://learn.microsoft.com/en-us/azure/architecture/guide/architecture-styles/event-driven) - Microsoft Learn Source: https://www.conduktor.io/glossary/azure-event-hubs-and-kafka-compatibility --- # Backpressure Handling in Streaming Systems In distributed streaming systems, data flows continuously from producers through processing pipelines to consumers. When downstream components can't keep up with the rate of incoming data, backpressure occurs. Understanding and properly handling backpressure is essential for building reliable, scalable streaming applications. ## What is Backpressure? Backpressure is the resistance or feedback signal that occurs when a downstream system component cannot process data as quickly as it's being produced upstream. Think of it like water flowing through connected pipes of different diameters, when a narrow pipe can't handle the flow from a wider one, pressure builds up. ![backpressure-handling-in-streaming-systems diagram 1](https://www.conduktor.io/assets/images/glossary/backpressure-handling-in-streaming-systems-0.webp) In streaming systems, backpressure manifests when: - Producers generate events faster than consumers can process them - A processing stage takes longer than expected (due to complex transformations, external API calls, or database writes) - Network congestion slows data transfer between components - Downstream systems experience temporary performance degradation Without proper handling, backpressure leads to unbounded memory growth, system crashes, data loss, or cascading failures across your infrastructure. ## Why Backpressure Matters in Streaming Unlike batch processing where you control when jobs run, streaming systems operate continuously. This creates unique challenges: - **Memory Exhaustion**: If a consumer keeps accepting data it can't process, buffers fill up until the system runs out of memory and crashes. - **Data Loss**: When buffers overflow, systems may silently drop events or fail to acknowledge messages, leading to data loss. - **Latency Spikes**: As queues grow, end-to-end latency increases. An event that normally processes in milliseconds might take minutes during backpressure situations. - **Cascading Failures**: Backpressure in one component can propagate upstream, affecting the entire pipeline and connected systems. - **Resource Waste**: Over-provisioned systems that don't handle backpressure intelligently waste compute and storage resources buffering data unnecessarily. Real-world streaming applications must handle variable data rates. A retail system might see 10x traffic during flash sales. An IoT platform might receive bursts when sensors reconnect after network outages. Proper backpressure handling ensures systems remain stable during these variations. ## Common Backpressure Strategies Different strategies address backpressure depending on your requirements for data completeness, latency, and cost: ### Buffering Store incoming data temporarily until the downstream system catches up. Buffers smooth out short-term spikes but have limits. Kafka topics act as distributed buffers, storing messages until consumers process them. Configure buffer sizes based on expected spike duration and available memory. ### Throttling and Blocking Slow down producers when consumers can't keep up. In Kafka, this happens when producers receive a `buffer.memory` exception and must wait before sending more messages. Reactive Streams implementations use this approach, where subscribers signal demand and publishers respect it. ### Load Shedding and Sampling Drop some data to protect system stability. For metrics or monitoring data where approximate values suffice, sampling every 10th event during high load maintains system health. Implement this with filters that check system load and probabilistically drop events. ### Elastic Scaling Automatically add consumer instances when lag increases. Kubernetes can scale consumer deployments based on metrics. Kafka consumer groups automatically rebalance partitions across new instances. ### Batching and Windowing Process multiple events together to improve throughput. Instead of writing each event individually to a database, batch 100 events and write once. This reduces per-event overhead and helps consumers catch up. ## Backpressure in Popular Streaming Platforms ### Apache Kafka Kafka handles backpressure through several mechanisms: - **Partitions and Consumer Groups**: Scale consumption by adding partitions and consumer instances. Each partition is consumed by one consumer in a group. For detailed consumer group mechanics, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). - **Consumer Lag Monitoring**: Track how far behind consumers are. High lag indicates backpressure. See [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) for monitoring strategies. - **Producer Controls**: Configure `buffer.memory`, `max.block.ms`, and `linger.ms` to control producer behavior when brokers can't keep up. - **Quotas**: Enforce rate limits on producers and consumers to prevent any single client from overwhelming the system. - **KRaft Mode (Kafka 4.0+)**: The modern consensus protocol improves metadata operations, reducing coordination overhead during scaling events. KRaft's faster leader elections help systems recover from backpressure-induced failures more quickly than legacy ZooKeeper mode. ### Apache Flink Flink implements automatic backpressure propagation: When a downstream operator (like a sink writing to a database) becomes slow, Flink automatically slows upstream operators. This happens through network buffers, when buffers fill, the sender blocks until space becomes available. **Flink 1.13+ Visual Backpressure Monitoring**: The web UI provides enhanced backpressure visualization, showing per-operator backpressure status with color-coded indicators (green, yellow, red). Task metrics now include busy/idle time ratios, making it easier to identify bottlenecks. The Task Manager metrics page displays buffer pool usage and network buffer details for granular troubleshooting. ### Kafka Streams Kafka Streams applications are both consumers and producers. Backpressure happens naturally through Kafka's consumer behavior. If processing slows, the application commits offsets less frequently and processes fewer records per poll. Configure `max.poll.records` and `max.poll.interval.ms` to control this behavior. ### Apache Pulsar Pulsar provides more granular backpressure control than Kafka: - **Flow Control**: Consumers send permits indicating how many messages they can receive - **Delayed Message Delivery**: Producers can specify delivery delays when backpressure occurs - **Multi-layer Architecture**: Separation of serving and storage layers helps isolate backpressure issues ## Real-World Scenarios and Solutions ### Scenario 1: IoT Sensor Data Ingestion A factory operates 10,000 sensors, each sending temperature readings every second (10,000 events/sec). The analytics database can only handle 5,000 writes per second. - **Problem**: Without backpressure handling, the processing application's memory fills up with pending writes, eventually causing OutOfMemory errors. - **Solution**: Implement a combination of strategies: 1. Batch database writes (insert 100 readings per transaction instead of individual inserts) 2. Use Kafka with sufficient partitions (20 partitions allowing parallel consumption) 3. Add consumer instances to parallelize processing 4. Implement sampling for non-critical sensors during peak periods 5. Monitor consumer lag and alert when it exceeds thresholds ### Scenario 2: Event Processing Pipeline An e-commerce platform processes order events through multiple stages: validation, inventory check, payment processing, and fulfillment. The payment service has variable response times (50ms to 2000ms depending on payment provider). - **Problem**: When payment processing slows, events back up, increasing end-to-end latency from seconds to minutes. - **Solution**: 1. Use Kafka between stages to decouple components 2. Configure the payment processing stage with appropriate timeouts and retries 3. Implement circuit breakers to fail fast when payment services are down 4. Scale payment processors based on queue depth 5. Monitor lag across each processing stage to identify bottlenecks quickly When consumer lag in the payment processing topic exceeds 10,000 messages, the operations team receives alerts and can investigate whether to scale consumers or if the payment provider is experiencing issues. ## Best Practices and Monitoring ### Design for Backpressure - **Set Explicit Limits**: Configure memory limits, buffer sizes, and timeouts explicitly rather than relying on defaults - **Plan for Peak Load**: Design capacity for 2-3x normal load to handle spikes - **Implement Graceful Degradation**: Define which data can be sampled or delayed when backpressure occurs - **Use Dead Letter Queues**: Route problematic messages to separate topics for later investigation instead of blocking processing. See [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling) for implementation patterns. ### Monitor Key Metrics Essential metrics for detecting backpressure: - **Consumer Lag**: The number of messages consumers are behind producers - **Processing Time**: How long each event takes to process (p50, p95, p99 percentiles) - **Queue Depth**: Number of messages waiting in buffers - **Memory Utilization**: Heap usage in consumers and producers - **Error Rates**: Timeouts, failed writes, rejected messages ### Tools for Visibility Kafka's built-in metrics expose lag and throughput, but interpreting raw metrics requires effort. Streaming management platforms provide [dashboards that visualize consumer lag](https://docs.conduktor.io/guide/monitor-brokers-apps/index), identify slow consumers, and show partition-level throughput. This visibility helps teams quickly identify whether backpressure stems from under-provisioned consumers, slow downstream systems, or data skew across partitions. During incident response, seeing which specific consumer groups are lagging and which partitions are affected saves valuable troubleshooting time. ### Testing Backpressure Scenarios Don't wait for production to test backpressure handling: - **Chaos Engineering**: Deliberately slow down consumers or inject latency in downstream systems - **Load Testing**: Gradually increase producer rate to find breaking points - **Partition Testing**: Verify behavior when adding/removing partitions - **Consumer Failure**: Test what happens when consumers crash during high load ## Summary Backpressure is an inevitable challenge in streaming systems where components operate at different speeds. Understanding how backpressure manifests and implementing appropriate handling strategies, buffering, throttling, load shedding, and elastic scaling, ensures your streaming applications remain stable and performant. Modern streaming platforms like Kafka and Flink provide built-in mechanisms for handling backpressure, but you must configure them appropriately for your use case. Monitor consumer lag, processing times, and queue depths to detect backpressure early. Design systems with explicit limits and graceful degradation paths. Whether you're building IoT data pipelines, event-driven microservices, or real-time analytics, proper backpressure handling is essential for reliability. Start by understanding your throughput requirements, identify potential bottlenecks, implement monitoring, and test under load. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Platform with built-in backpressure mechanisms through consumer groups - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Tracking processing delays to detect backpressure - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Sizing infrastructure to handle throughput requirements ## Sources and References - [Apache Kafka Documentation - Producer and Consumer Configurations](https://kafka.apache.org/documentation/) - Official documentation covering producer buffer management, consumer lag, and flow control mechanisms - [Apache Flink Documentation - Back Pressure Monitoring](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/monitoring/back_pressure/) - Detailed explanation of Flink's automatic backpressure propagation and monitoring tools - [Reactive Streams Specification](https://www.reactive-streams.org/) - Foundation specification for asynchronous stream processing with non-blocking backpressure - Kleppmann, Martin. "Designing Data-Intensive Applications" (O'Reilly Media, 2017) - Chapter 11 covers stream processing architectures and handling variable data rates - [Confluent Blog - Kafka Producer and Consumer Internals](https://www.confluent.io/blog/) - Technical deep-dives on Kafka's internal mechanisms for flow control and performance tuning Source: https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems --- # Building a Business Glossary for Data Governance One of the most persistent challenges in data management is establishing a common language across business and technical teams. A business glossary provides the semantic layer that connects business terminology with technical data assets, and forms the foundation of effective data governance. For a deeper exploration of semantic layers in streaming contexts, see [Semantic Layer for Streaming](https://www.conduktor.io/glossary/semantic-layer-for-streaming). ## Understanding the Business Glossary ![Business Glossary Architecture](https://www.conduktor.io/assets/images/glossary/building-a-business-glossary-for-data-governance-0.webp) A business glossary is more than a simple dictionary of terms. It's a living, governed repository that defines business concepts, their relationships, and their mappings to physical data assets. While a data dictionary focuses on technical metadata, column names, data types, and constraints, a business glossary operates at a higher level of abstraction, capturing the meaning and context that data holds within your organization. Business glossaries are typically managed within data catalogs, which provide the technical infrastructure for discovery, search, and governance. For an overview of modern data catalog capabilities, see [What is a Data Catalog? Modern Data Discovery](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery). For instance, "customer" might mean different things across departments. Marketing might define it as anyone who has engaged with promotional content, while finance considers only those who have completed a purchase. Without a business glossary, these inconsistencies propagate through dashboards, reports, and analytics, leading to conflicting insights and eroded trust in data. ## Core Components of an Effective Business Glossary ### Business Terms and Definitions Each term in your glossary should include a clear, agreed-upon definition that reflects how your organization uses the concept. Avoid technical jargon in these definitions, they should be understandable to business stakeholders. Include context about when and how the term applies, along with any specific business rules or calculations. ### Relationships and Hierarchies Terms rarely exist in isolation. Document how concepts relate to one another through hierarchies, associations, and dependencies. A "revenue" term might have child terms like "gross revenue," "net revenue," and "recurring revenue," each with distinct definitions and calculation methods. ### Data Asset Mappings The real power of a business glossary emerges when you link business terms to their technical implementations. Map terms to database tables, columns, API endpoints, and even streaming topics. This bidirectional linkage allows business analysts to find the data they need and data engineers to understand the business impact of technical changes. When combined with data lineage tracking, glossary terms flow through transformation pipelines, enabling end-to-end business context from source to consumption. For detailed coverage of lineage tracking, see [Data Lineage: Tracking Data from Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption). ### Ownership and Stewardship Assign clear ownership to each term. Data stewards should be subject matter experts who can validate definitions, approve changes, and resolve ambiguities. This governance layer ensures the glossary remains authoritative and current. For detailed guidance on establishing roles and responsibilities, see [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities). ## Building Your Glossary: A Practical Approach ### Start with High-Impact Terms Don't attempt to catalog your entire organizational vocabulary at once. Begin with the terms that appear most frequently in critical business processes, reports, and decisions. Focus on areas where misalignment has caused concrete problems, missed targets, compliance issues, or conflicting analytics. Conduct workshops with cross-functional teams to document these terms. These sessions often reveal surprising discrepancies in understanding and provide opportunities to build consensus early. ### Establish Governance Workflows Create formal processes for proposing, reviewing, and approving glossary entries. This might include: - Submission templates that ensure consistent information capture - Review committees with representatives from business and technical teams - Approval workflows with appropriate stakeholders - Change management processes for updating existing terms - Versioning to track how definitions evolve over time ### Integrate with Technical Metadata The value of your business glossary multiplies when integrated with your technical metadata infrastructure. Modern data catalogs (2025) provide sophisticated glossary capabilities with automated linking: **Enterprise Data Catalogs:** - **Atlan**: AI-powered glossary with automated term propagation and column-level lineage - **Collibra**: Workflow-driven governance with business glossary as the central hub - **Microsoft Purview**: Cloud-native catalog integrating with Azure ecosystem and Microsoft 365 - **DataHub (LinkedIn)**: Open-source metadata platform with real-time glossary updates - **Alation**: Behavioral AI that learns from data usage patterns to suggest glossary mappings **Cloud-Native Options:** - **AWS Glue Data Catalog**: Native integration with AWS services and Lake Formation - **Google Cloud Data Catalog**: Automatic discovery and tagging with Data Loss Prevention (DLP) These platforms automatically link business terms to database schemas, data warehouse models, BI dashboards, and streaming topics. This integration enables impact analysis, when a business term changes, you can immediately see which reports, dashboards, pipelines, and systems are affected. For detailed coverage of technical versus business metadata and their relationship, see [Metadata Management: Technical vs Business Metadata](https://www.conduktor.io/glossary/metadata-management-technical-vs-business-metadata). ## Streaming Integration and Real-Time Glossaries As organizations adopt streaming architectures for real-time data processing, the business glossary must extend to cover event streams and topics. Streaming platforms like Apache Kafka have become critical infrastructure, yet the business meaning of event payloads often remains opaque. Document your streaming terms with the same rigor as batch data. Define what business events your topics represent, the business entities they describe, and how event attributes map to business concepts. For example, a "payment.processed" event topic should have glossary entries explaining the business meaning of each field in the event payload. ### Practical Streaming Glossary Integration **Schema Registry Integration:** Link business terms directly to schema definitions in Confluent Schema Registry or AWS Glue Schema Registry: ```json { "namespace": "com.example.payments", "type": "record", "name": "PaymentProcessed", "doc": "Business Glossary: Payment Transaction Event - represents completed payment processing", "fields": [ { "name": "transactionId", "type": "string", "doc": "Glossary Term: Transaction ID - unique identifier for payment transaction" }, { "name": "amount", "type": "double", "doc": "Glossary Term: Transaction Amount - gross payment value before fees" }, { "name": "currency", "type": "string", "doc": "Glossary Term: Currency Code - ISO 4217 three-letter currency code" } ] } ``` **Stream Governance Tools (2025):** Modern platforms enable glossary integration: - **Conduktor**: Topic and schema management with access approval workflows and naming convention enforcement - **DataHub**: Real-time metadata ingestion from Kafka with glossary term propagation - **StreamSets**: Data pipeline observability with business context from glossary terms These tools enforce naming conventions, validate topic alignment with approved business terms, and maintain semantic consistency across batch and streaming ecosystems. Business analysts can discover relevant topics through business terminology rather than navigating technical names and schemas. For implementation details on schema management and evolution, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). For broader streaming governance patterns, refer to [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ## Maintaining Glossary Quality ### Regular Audits and Updates Schedule periodic reviews of glossary content. Business terminology evolves as your organization grows, enters new markets, or undergoes digital transformation. Stale definitions undermine trust and adoption. ### Usage Analytics Monitor which terms are searched, viewed, and referenced most frequently. Low-engagement terms might need better definitions or may not be as important as initially thought. High-engagement terms deserve extra attention to ensure their accuracy and completeness. ### Feedback Mechanisms Enable users to suggest corrections, request new terms, or ask questions about definitions. This crowdsourced input keeps your glossary aligned with actual business usage while building a culture of shared data ownership. ## AI-Powered Glossary Management (2025) Modern glossary platforms use AI and machine learning to automate parts of glossary management: ### Automated Term Extraction AI models scan existing documentation, code comments, data dictionaries, and SQL queries to identify candidate business terms automatically. Natural language processing (NLP) extracts frequently used concepts and suggests them for glossary inclusion, dramatically reducing manual cataloging effort. ### Intelligent Definition Generation Large language models (LLMs) can draft initial term definitions based on context from: - Database column names and comments - Business documentation and wiki pages - Historical usage patterns in queries and reports - Similar terms in the existing glossary These AI-generated definitions serve as starting points for subject matter experts, accelerating glossary population while maintaining human oversight for accuracy. ### Smart Term Matching and Classification Machine learning algorithms automatically: - Match technical column names to business terms (e.g., "cust_id" → "Customer Identifier") - Classify terms into categories (financial, operational, customer-facing) - Detect synonyms and suggest term consolidation - Identify potential duplicates or conflicting definitions - Automatically tag sensitive terms (PII, financial data) for compliance For comprehensive strategies on data classification and tagging, see [Data Classification and Tagging Strategies](https://www.conduktor.io/glossary/data-classification-and-tagging-strategies). ### Usage Analytics and Recommendations AI-powered systems track how data assets are queried and consumed, then recommend: - Which terms need better documentation based on search patterns - Which datasets should be linked to specific business terms - High-value terms that deserve prioritization based on usage frequency Tools like Atlan's AI-powered glossary assistant, Alation's behavioral AI, and Collibra's ML-driven recommendations demonstrate these capabilities in production environments. For related automation in data quality, see [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing) and [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework). ## Data Mesh and Domain-Specific Glossaries In data mesh architectures, business glossaries operate at both global and domain levels, reflecting the decentralized nature of data ownership. ### Domain Glossaries Each data domain (e.g., Customer, Product, Finance) maintains its own glossary of domain-specific terms, owned and curated by domain teams. Domain glossaries define: - **Domain-specific entities**: Terms unique to that domain's context - **Local definitions**: How the domain interprets shared concepts - **Domain data products**: Business meaning of published data products For example, the Customer domain defines "Customer Lifetime Value" while the Finance domain defines "Revenue Recognition Period", both valid within their contexts. ### Global Enterprise Glossary The global glossary establishes common terminology across domains: - **Cross-domain entities**: Terms used by multiple domains (e.g., "Customer", "Order") - **Standardized definitions**: Agreed-upon meanings that all domains respect - **Federated governance**: Coordination between domain glossaries ### Glossary Federation Pattern Modern implementations support glossary federation where: 1. Domain teams maintain autonomy over domain-specific terms 2. Global glossary defines shared vocabulary and standards 3. Data catalog platforms (DataHub, Purview, Atlan) automatically aggregate and link domain glossaries 4. Conflicts are surfaced through automated governance checks This approach balances consistency with domain autonomy, preventing the glossary from becoming a centralized bottleneck. For comprehensive coverage of data mesh principles, see [Data Mesh Principles and Implementation](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation). For data product governance patterns, refer to [Building and Managing Data Products](https://www.conduktor.io/glossary/building-and-managing-data-products) and [Data Product Governance](https://www.conduktor.io/glossary/data-product-governance). ## Common Challenges and Best Practices ### Challenge: Low Adoption - **Symptom**: Glossary exists but few people use it. - **Solutions**: - Embed glossary directly in data tools (BI platforms, SQL editors, data notebooks) - Make glossary search the default in your data catalog - Gamify contributions with leaderboards and recognition - Demonstrate ROI with concrete examples of time saved or errors prevented ### Challenge: Glossary Becomes Stale - **Symptom**: Definitions are outdated or terms are deprecated. - **Solutions**: - Automate freshness checks by comparing glossary terms to actual data usage - Set review cycles tied to business planning (quarterly, annually) - Alert term owners when usage patterns change significantly - Archive unused terms rather than deleting them (maintain history) ### Challenge: Conflicting Definitions Across Departments - **Symptom**: Multiple teams define the same term differently. - **Solutions**: - Document context-specific definitions with clear scope (e.g., "Customer - Marketing definition") - Establish a cross-functional data council to resolve conflicts - Use qualified names (namespaces) to distinguish domain-specific interpretations - Use data mesh patterns for federated glossary management ### Challenge: Glossary Scope Creep - **Symptom**: Glossary becomes unwieldy with thousands of rarely-used terms. - **Solutions**: - Start with top 50-100 high-impact terms - Require business justification for new term additions - Archive low-engagement terms after analysis - Focus on terms that drive decisions, not every database column ## Measuring Success A successful business glossary should demonstrate measurable impact: - **Time to Insight**: Reduced time for analysts to find and understand data (target: 30-50% reduction) - **Data Trust**: Decreased incidents of misinterpreted data leading to wrong decisions - **Onboarding Velocity**: Faster onboarding of new team members to data assets (target: 2-3x faster) - **Compliance**: Improved compliance through clear documentation of regulated terms - **Reuse Rate**: Higher reuse of existing data assets versus duplicate creation (target: 40%+ reuse rate) - **Engagement Metrics**: Search volume, term views, and contribution rates Track these metrics quarterly and report to leadership to demonstrate glossary value and justify continued investment. ## Summary A business glossary aligns teams around shared definitions and connects business concepts to the technical assets that implement them. Start with high-impact terms, establish clear governance workflows, and integrate with your technical metadata infrastructure. Modern platforms can automate term extraction and classification, but human oversight is still needed to keep definitions accurate and current. Whether your data flows through databases, warehouses, or real-time streaming platforms like Kafka, a well-maintained glossary ensures everyone works from the same definitions. ## Related Concepts - [Metadata Management: Technical vs Business Metadata](https://www.conduktor.io/glossary/metadata-management-technical-vs-business-metadata) - Understanding different types of metadata in governance - [Data Classification and Tagging Strategies](https://www.conduktor.io/glossary/data-classification-and-tagging-strategies) - Organizing and labeling data for governance - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing technical schemas alongside business definitions ## Sources 1. DAMA International - "DAMA-DMBOK: Data Management Body of Knowledge" (2023 Edition) - https://www.dama.org/cpages/body-of-knowledge 2. Atlan - "The Modern Data Catalog: AI-Powered Glossary Management" (2025) - https://atlan.com/glossary/ 3. Microsoft - "Microsoft Purview Data Catalog Documentation" (2025) - https://learn.microsoft.com/azure/purview/ 4. DataHub Project - "DataHub Glossary and Metadata Management" (2025) - https://datahubproject.io/docs/ 5. Collibra - "Business Glossary Best Practices and Governance Workflows" (2024) - https://www.collibra.com/glossary/ 6. Confluent - "Stream Governance and Schema Registry Integration" (2025) - https://docs.confluent.io/platform/current/schema-registry/ 7. Alation - "Building a Business Glossary: Implementation Guide" (2024) - https://www.alation.com/blog/business-glossary/ 8. Gartner - "Magic Quadrant for Metadata Management Solutions" (2024) - https://www.gartner.com/en/documents/metadata-management Source: https://www.conduktor.io/glossary/building-a-business-glossary-for-data-governance --- # Building a Data Quality Framework Data quality is the foundation of trustworthy analytics, reliable machine learning models, and sound business decisions. Establishing a data quality framework is both a technical exercise and an organizational one — it affects every data-driven initiative across the organization. ## Understanding the Data Quality Framework A data quality framework is a structured approach to measuring, monitoring, and improving data quality across your organization. It encompasses the processes, metrics, tools, and governance structures needed to ensure data meets the quality standards required for its intended use. The framework typically addresses six core dimensions of data quality: accuracy, completeness, consistency, timeliness, validity, and uniqueness. Each dimension requires specific validation rules, measurement approaches, and remediation strategies tailored to your organization's data landscape and business requirements. ## Designing Your Framework Architecture ![Data Quality Framework Architecture](https://www.conduktor.io/assets/images/glossary/building-a-data-quality-framework-0.webp) Begin by establishing clear ownership and accountability structures. Every dataset should have a designated data steward responsible for quality standards, while data quality analysts focus on implementing and monitoring quality checks. This separation of concerns ensures both business context and technical rigor are applied to quality management. Your framework architecture should include three primary layers: the **detection layer** that identifies quality issues, the **measurement layer** that quantifies quality levels, and the **reporting layer** that communicates quality status to stakeholders. These layers work together to create a continuous feedback loop that drives quality improvement. Consider both batch and streaming data patterns in your design. While batch processing allows for comprehensive quality checks on historical data, streaming data requires real-time validation to catch issues before they propagate through your data pipelines. This dual approach ensures quality controls operate at the speed of your business. ## Defining Quality Metrics and Validation Rules Quality metrics provide quantifiable measures of data health. Start with foundational metrics like completeness rates (percentage of non-null values), accuracy scores (comparison against trusted sources), and consistency checks (verification across systems). These metrics should be specific, measurable, and aligned with business impact. Validation rules translate quality requirements into executable checks. For example, a customer email field might require format validation (proper email structure), uniqueness validation (no duplicates), and domain validation (only approved domains). Document each rule with its business rationale, severity level, and expected threshold. Here's a practical example of implementing data quality validation in Python: ```python import re from dataclasses import dataclass from typing import List, Optional from enum import Enum class Severity(Enum): CRITICAL = "critical" MAJOR = "major" MINOR = "minor" @dataclass class ValidationResult: field: str passed: bool severity: Severity message: Optional[str] = None class DataQualityValidator: def __init__(self): self.approved_domains = ['company.com', 'partner.com'] def validate_email(self, email: str) -> List[ValidationResult]: """Validate email field against multiple quality rules.""" results = [] # Completeness check FIRST (avoid regex on null/empty values) if not email or email.strip() == "": results.append(ValidationResult( field="email", passed=False, severity=Severity.CRITICAL, message="Email field is empty" )) return results # Early return, don't check format/domain on empty values # Format validation email_pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$' if not re.match(email_pattern, email): results.append(ValidationResult( field="email", passed=False, severity=Severity.CRITICAL, message=f"Invalid email format: {email}" )) return results # Don't check domain if format is invalid # Domain validation (only if format is valid) if '@' in email: domain = email.split('@')[1] if domain not in self.approved_domains: results.append(ValidationResult( field="email", passed=False, severity=Severity.MAJOR, message=f"Unapproved domain: {domain}" )) # If no issues found, return success if not results: results.append(ValidationResult( field="email", passed=True, severity=Severity.MINOR )) return results def calculate_completeness_rate(self, records: List[dict], field: str) -> float: """Calculate completeness rate for a specific field.""" total = len(records) if total == 0: return 0.0 non_null = sum(1 for r in records if r.get(field) is not None and str(r.get(field)).strip() != "") return (non_null / total) * 100 # Example usage validator = DataQualityValidator() # Validate individual record results = validator.validate_email("user@company.com") for result in results: print(f"{result.severity.value}: {result.message if result.message else 'Passed'}") # Calculate completeness for a dataset records = [ {"email": "user1@company.com", "age": 25}, {"email": "", "age": 30}, {"email": "user3@company.com", "age": None} ] completeness = validator.calculate_completeness_rate(records, "age") print(f"Age field completeness: {completeness:.1f}%") ``` Implement a tiered severity model for quality issues: **critical** issues that block data usage, **major** issues that significantly impact reliability, and **minor** issues that should be monitored but don't prevent usage. This classification helps prioritize remediation efforts and set appropriate alerting thresholds. Create quality rules that are both preventive and detective. Preventive rules enforce quality at data entry or ingestion points, while detective rules identify issues in existing data. For streaming scenarios, preventive rules are particularly valuable as they stop bad data before it enters downstream systems. ## Building Quality Scorecards Quality scorecards aggregate metrics into actionable dashboards that communicate data health to various stakeholder groups. Executive scorecards might show overall quality trends and business impact, while operational scorecards provide detailed breakdowns by dataset, data domain, or quality dimension. Design scorecards with clear visual indicators, green, yellow, and red status indicators quickly communicate whether quality meets expectations. Include trend lines to show improvement or degradation over time, and contextualize metrics with business impact assessments that translate quality scores into business terms. Your scorecards should answer key questions: Which datasets have the highest quality issues? Are quality trends improving or declining? What is the business impact of current quality levels? Which data domains require immediate attention? These insights drive prioritization and resource allocation for quality improvement initiatives. ## Implementing Streaming Quality Checks Modern data architectures increasingly rely on streaming platforms like Apache Kafka for real-time data movement. Integrating quality checks into streaming pipelines ensures issues are caught immediately rather than discovered downstream when they've already impacted analytics or operations. Use schema validation as your first line of defense in streaming scenarios. Governance platforms provide real-time schema enforcement capabilities, ensuring every message conforms to expected structure before entering your data platform. This prevents schema evolution issues and malformed data from propagating through your systems. Implement stateful streaming quality checks for metrics that require aggregation or historical context. For example, detecting duplicate records in a stream requires maintaining state about previously seen keys. Similarly, checking for referential integrity or validating business rules often requires looking up reference data or maintaining running aggregations. Here's an example of implementing streaming quality checks using Kafka Streams: ```java import org.apache.kafka.streams.*; import org.apache.kafka.streams.kstream.*; import java.time.Duration; import java.util.*; // Data model class CustomerRecord { public String customerId; public String email; public Integer age; public List qualityErrors = new ArrayList<>(); } public class StreamingQualityValidator { public static void main(String[] args) { Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "data-quality-validator"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); StreamsBuilder builder = new StreamsBuilder(); // Read incoming data stream KStream sourceStream = builder.stream("customer-events"); // Branch stream into valid and invalid records Map> branches = sourceStream .split(Named.as("quality-")) .branch((key, record) -> isValid(record), Branched.as("valid")) .branch((key, record) -> !isValid(record), Branched.as("invalid")) .noDefaultBranch(); // Send valid records to main topic branches.get("quality-valid") .to("customer-validated"); // Send invalid records to quarantine for review branches.get("quality-invalid") .mapValues(record -> enrichWithQualityErrors(record)) .to("customer-quarantine"); // Detect duplicates using stateful processing KStream validStream = branches.get("quality-valid"); validStream .groupByKey() .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5))) .count() .toStream() .filter((windowedKey, count) -> count > 1) .map((windowedKey, count) -> KeyValue.pair(windowedKey.key(), "Duplicate detected: " + count + " occurrences")) .to("quality-alerts"); KafkaStreams streams = new KafkaStreams(builder.build(), props); streams.start(); } private static boolean isValid(CustomerRecord record) { // Completeness checks if (record.email == null || record.email.isEmpty()) { return false; } // Format validation if (!record.email.matches("^[A-Za-z0-9+_.-]+@(.+)$")) { return false; } // Business rule validation if (record.age != null && (record.age < 0 || record.age > 150)) { return false; } return true; } private static CustomerRecord enrichWithQualityErrors(CustomerRecord record) { List errors = new ArrayList<>(); if (record.email == null || record.email.isEmpty()) { errors.add("Missing required field: email"); } else if (!record.email.matches("^[A-Za-z0-9+_.-]+@(.+)$")) { errors.add("Invalid email format"); } if (record.age != null && (record.age < 0 || record.age > 150)) { errors.add("Age out of valid range"); } record.qualityErrors = errors; return record; } } ``` Consider quality check performance carefully in streaming contexts. Quality validations must process data at the rate it arrives without creating backpressure or latency. Design checks to be computationally efficient and scalable, using techniques like sampling for expensive validations that don't need to run on every record. For backpressure handling strategies, see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems). ## Modern Data Quality Tools (2025) The data quality tooling landscape has evolved significantly. Modern frameworks complement custom validation code with declarative quality definitions and automated monitoring: - **Soda Core**: Open-source data quality testing framework using YAML-based quality checks. Define data contracts and quality rules declaratively, integrating with orchestration tools like Airflow and Dagster. - **dbt Tests and Contracts**: dbt (data build tool) now includes native data quality testing and contracts. Define expectations in SQL and enforce them during transformation workflows. dbt contracts (introduced in dbt 1.5+) provide explicit interfaces between models with automatic validation. - **Elementary Data**: Observability platform for dbt that provides automatic data quality monitoring, anomaly detection, and lineage tracking. Complements dbt's built-in testing with ML-based anomaly detection. - **Great Expectations**: Comprehensive data validation framework with extensive built-in expectations. See [Great Expectations: Data Testing Framework](https://www.conduktor.io/glossary/great-expectations-data-testing-framework) for implementation patterns. - **Data Contracts (ODCS)**: The Open Data Contract Standard (ODCS v3.1.0) provides a specification for formal data contracts. See [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) for implementation guidance. These tools integrate with streaming platforms like Kafka, enabling real-time quality validation alongside custom code solutions. ## Monitoring and Platform Integration Governance platforms provide comprehensive monitoring capabilities for streaming-based data quality frameworks. Use data lineage features to track how quality issues propagate through your pipelines, identifying exactly where problems originate and which downstream consumers are affected. Use platform governance policies to enforce quality standards at the infrastructure level. Define policies that automatically reject messages failing validation criteria, route problematic data to quarantine topics for review, or trigger alerts when quality thresholds are breached. These automated governance controls ensure quality standards are consistently applied across all streaming data flows. For actionable implementation with Conduktor, see [Data Quality Policies](https://docs.conduktor.io/guide/conduktor-concepts/data-quality-policies) and [Governance Insights](https://docs.conduktor.io/guide/insights/governance). Testing capabilities in modern platforms allow you to validate quality rules before deploying them to production. Create test scenarios with both valid and invalid data to verify your validation logic catches expected issues without generating false positives. This testing discipline prevents quality frameworks from becoming obstacles to legitimate data flows. ## Establishing Continuous Improvement A data quality framework is never truly complete, it evolves with your data landscape and business needs. Establish regular reviews of quality metrics, validation rules, and scorecards to ensure they remain relevant and effective. Incorporate feedback from data consumers about quality issues they encounter in practice. Measure the business impact of quality improvements to demonstrate value and secure ongoing investment in quality initiatives. Track metrics like reduced incident response time, decreased data rework efforts, or improved decision confidence. These outcomes connect technical quality metrics to business value. Create feedback loops between quality measurement and data platform improvements. When quality issues reveal systemic problems, inadequate data validation, missing reference data, or architectural gaps, use these insights to drive platform enhancements that prevent entire classes of quality issues. ## Related Concepts - [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing) - Implementing continuous testing practices - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) - Core quality metrics to track - [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) - Formalizing quality expectations ## Summary Building a data quality framework requires balancing technical rigor with practical business needs. Clear metrics, both batch and streaming validation, actionable scorecards, and governance platform integration create a foundation for trustworthy data and confident decision-making. ## Sources and References 1. **Data Quality: The Accuracy Dimension** by Jack E. Olson (Morgan Kaufmann, 2003) - Comprehensive guide to data quality dimensions and measurement techniques. 2. **DAMA-DMBOK: Data Management Body of Knowledge** (2nd Edition, 2017) - Industry standard framework for data governance and quality management practices. Available at: https://www.dama.org/cpages/body-of-knowledge 3. **Designing Data-Intensive Applications** by Martin Kleppmann (O'Reilly, 2017) - Chapter 12 covers data quality, consistency, and validation patterns in distributed systems. 4. **Apache Kafka Documentation - Stream Processing** - Official documentation on implementing stateful stream processing for data validation. Available at: https://kafka.apache.org/documentation/streams/ 5. **The Data Quality Dimensions: A Comprehensive Framework** by Richard Y. Wang and Diane M. Strong, MIT Sloan School of Management (1996) - Foundational research paper defining the six dimensions of data quality used across the industry. Source: https://www.conduktor.io/glossary/building-a-data-quality-framework --- # Building a Modern Data Lake on Cloud Storage Data lakes have become the foundation of modern data architectures, storing large volumes of structured and unstructured data while keeping flexibility for diverse analytics workloads. Unlike traditional data warehouses with rigid schemas, data lakes follow a "store first, structure later" approach that fits well with cloud-native storage. ## Why Cloud Object Storage? Cloud object storage services, AWS S3, Azure Data Lake Storage Gen2, and Google Cloud Storage, have emerged as the de facto standard for data lake implementations. Their advantages are compelling: - **Infinite scalability**: No capacity planning required. Your storage grows elastically with your data, from gigabytes to exabytes. - **Cost-efficiency**: Pay only for what you use, with tiered storage classes (hot, cool, archive) that optimize costs based on access patterns. Storage costs have dropped dramatically, making it economical to retain years of historical data. - **Durability and availability**: Built-in replication across availability zones provides 99.999999999% (11 nines) durability, eliminating the need to manage complex backup strategies. - **Separation of storage and compute**: Unlike traditional systems, you can scale processing power independently from storage, running multiple analytics engines (Spark, Presto, Athena) against the same dataset. ## Architectural Patterns ### The Three-Zone Architecture A well-designed data lake typically follows a three-zone pattern that mirrors the data refinement process. For detailed best practices on implementing this architecture, see [Data Lake Zones: Bronze, Silver, Gold Architecture](https://www.conduktor.io/glossary/data-lake-zones-bronze-silver-gold-architecture). - **Raw Zone (Bronze Layer)**: Immutable source data stored exactly as ingested. This might include JSON logs from applications, CSV exports from databases, or binary files like images. Partition by ingestion date (e.g., `s3://datalake/raw/system_logs/year=2025/month=12/day=07/`) to enable efficient pruning. - **Curated Zone (Silver Layer)**: Cleansed, validated, and transformed data in standardized formats. Convert to columnar formats like Parquet or ORC for better compression and query performance. This layer often includes deduplication, schema enforcement, and data quality checks. - **Analytics Zone (Gold Layer)**: Business-ready datasets optimized for specific use cases. These are often aggregated, denormalized, and partitioned by business dimensions (region, product line, customer segment) rather than just time. ### Storage Format Selection Choosing the right file format significantly impacts performance and cost: - **Parquet**: Columnar format ideal for analytical queries. Instead of storing data row-by-row, Parquet stores each column separately, enabling queries to read only the columns they need (reducing I/O by 10-100x). Excellent compression ratios and predicate pushdown support (filtering data at storage level before reading into memory). Use for structured data that's queried by specific columns. - **Avro**: Row-oriented with strong schema evolution support (ability to add/remove fields without breaking existing data readers). Preferred for write-heavy workloads and streaming data where you need to preserve exact record order. - **Open Table Formats (Iceberg/Delta Lake/Hudi)**: Modern table formats that bring ACID transactions (atomicity, consistency, isolation, durability), time travel, and schema evolution to object storage. Apache Iceberg has emerged as the industry standard in 2024-2025, with adoption across Snowflake, Databricks, AWS, Google Cloud, and nearly all major data platforms. These formats are essential for scenarios requiring updates, deletes, or consistent reads during writes. For detailed coverage of Iceberg's architecture and capabilities, see [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg). ## Platform-Specific Implementations ### AWS S3 Data Lake ```yaml # S3 bucket structure s3://production-datalake/ raw/ clickstream/ year=2025/month=12/day=07/ events-{timestamp}.json.gz curated/ user_sessions/ region=us-east/ part-00000.parquet analytics/ daily_metrics/ report_date=2025-12-07/ metrics.parquet ``` Key S3 features to use: - **S3 Intelligent-Tiering**: Automatically moves objects between access tiers based on usage patterns - **S3 Select**: Query data in place using SQL without downloading entire files - **S3 Express One Zone** (2024+): High-performance storage class delivering single-digit millisecond latency for frequently accessed data, ideal for real-time analytics workloads - **S3 Tables** (2024+): AWS-native Apache Iceberg table management with automatic optimization, compaction, and query performance improvements. Simplifies lakehouse operations without separate table format maintenance - **AWS Glue Data Catalog**: Central metadata repository that integrates with Athena, EMR, and Redshift Spectrum. Now includes native Iceberg table support (2024+) with full schema evolution and time travel capabilities - **S3 Event Notifications**: Trigger Lambda functions or SQS messages when new data arrives ### Azure Data Lake Storage Gen2 ADLS Gen2 combines blob storage's scalability with hierarchical namespaces that enable directory-level operations and POSIX permissions (Unix-style read/write/execute permissions familiar to Linux users): ```bash # ADLS Gen2 structure with hierarchical namespace abfss://datalake@storageaccount.dfs.core.windows.net/ /raw/transactional_db/orders/year=2025/month=12/ /curated/customer_360/ /analytics/sales_dashboard/ ``` Key capabilities: - **Azure Synapse Analytics**: Integrated analytics workspace combining data engineering, warehousing, and BI - **Azure Databricks**: Optimized Spark with Delta Lake and Unity Catalog support (2024+) for unified batch and streaming with centralized governance - **Role-Based Access Control (RBAC)**: Fine-grained security at the directory and file level - **Microsoft Purview** (evolved from Azure Purview): Unified data governance providing automated data discovery, classification, and lineage tracking across your entire data estate ### Google Cloud Storage GCS offers a unified object storage API with automatic performance optimization: ```python # GCS bucket organization gs://company-datalake/ raw/pubsub_events/ curated/enriched_events/ analytics/ml_features/ ``` Notable features: - **BigQuery External Tables**: Query GCS data directly without loading into BigQuery - **Cloud Storage FUSE**: Mount buckets as file systems for legacy applications - **Autoclass**: Automatic storage class transitions based on access patterns ## Unified Table Catalogs and Governance (2024-2025) A critical evolution in modern data lakes is the emergence of unified table catalogs that provide centralized metadata management across table formats, engines, and clouds: **Open Source Catalogs**: - **Apache Polaris**: Open catalog for Apache Iceberg supporting multi-engine access with centralized governance - **Project Nessie**: Git-like semantics for data lakes with branching, tagging, and versioning of table metadata - **Unity Catalog** (Databricks, now open-sourced): Cross-platform catalog supporting Delta Lake, Iceberg, and Hudi with fine-grained access control **Platform-Integrated Catalogs**: - **AWS Glue with Iceberg**: Native Iceberg support enabling consistent table access across Athena, EMR, Redshift, and third-party engines - **Google BigLake**: Unified metadata layer providing fine-grained access control across BigQuery, Spark, and other engines - **Azure Unity Catalog**: Centralized governance for Databricks and expanding to Synapse integration These catalogs solve the historic "metadata sprawl" problem where each engine maintained separate metadata, leading to inconsistencies and governance challenges. In 2025, unified catalogs are considered essential for production data lakes. For an architectural overview of lakehouse patterns built on these catalogs, see [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture). ## Integrating Streaming Data Modern data lakes must handle both batch and streaming ingestion. Apache Kafka infrastructure often feeds real-time data into your lake. For comprehensive streaming ingestion patterns, see [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse). ### Kafka to Data Lake Pipeline ![### Kafka to Data Lake Pipeline](https://www.conduktor.io/assets/images/glossary/building-a-modern-data-lake-on-cloud-storage-0.webp) When streaming events into your data lake, modern patterns combine Kafka with table formats for reliable, exactly-once ingestion. For detailed Kafka fundamentals, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). **Modern Streaming Best Practices (2025)**: 1. **Use Kafka Connect with Iceberg**: The Iceberg Sink Connector (Apache Kafka 3.x+) provides native table format support with automatic schema evolution and compaction. For building robust data pipelines with Kafka Connect, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines). 2. **Implement exactly-once semantics**: Kafka transactions (Kafka 3.0+) combined with Iceberg's ACID guarantees ensure data isn't duplicated during failures. Exactly-once delivery is now the standard for production streaming data lakes. 3. **Use Apache Flink 1.18+ for complex transformations**: Flink's native Iceberg sink provides millisecond-latency writes with automatic compaction and sorting, ideal for real-time analytics on lakehouse tables. 4. **Handle schema evolution**: Use Avro or Protobuf with schema registry to manage changes without breaking downstream consumers. For schema management patterns, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). Both Iceberg and Delta Lake support automatic schema evolution when ingesting from evolving schemas. ## Best Practices - **Partition strategically**: Over-partitioning creates too many small files (inefficient for queries due to high metadata overhead), while under-partitioning forces full scans. Aim for files between 128MB-1GB, large enough to amortize open/close costs but small enough for parallel processing. Modern table formats like Iceberg handle this automatically through compaction. - **Implement data lifecycle policies**: Automatically archive or delete old data in the raw zone. Move infrequently accessed curated data to cheaper storage tiers. - **Secure by default**: Enable encryption at rest and in transit. Use IAM roles and policies for access control. Implement data catalog tagging for classification (PII, confidential, public). - **Monitor and optimize**: Track storage costs by zone and team. Use query performance metrics to identify partitioning improvements. Set up alerts for failed ingestion jobs. - **Document metadata**: Maintain a data catalog (AWS Glue, Microsoft Purview, Google Dataplex) describing datasets, owners, lineage, and SLAs. Poor metadata management is a common data lake failure mode. Modern catalogs integrate with unified table formats to provide automated lineage tracking and data discovery. For comprehensive coverage, see [What is a Data Catalog: Modern Data Discovery](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery). ## Summary Building a modern data lake on cloud storage requires balancing flexibility with governance. A structured zone-based approach combined with open table formats like Apache Iceberg and unified catalogs creates a scalable analytics foundation. The shift toward lakehouse architectures has made table formats and unified catalogs essential, not optional. They provide the ACID guarantees, time travel, and governance that production data lakes require. Start with clear patterns and adjust as requirements evolve. ## Related Concepts - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) - [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/what-is-change-data-capture-cdc-fundamentals) - [Data Lineage: Tracking Data from Source to Consumption](https://www.conduktor.io/data-lineage-tracking-data-from-source-to-consumption) ## Sources and References - [AWS S3 Data Lake Best Practices](https://docs.aws.amazon.com/prescriptive-guidance/latest/defining-bucket-names-data-lakes/welcome.html) - AWS guidance on building data lakes with S3 - [Azure Data Lake Storage Gen2 Documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) - Microsoft's hierarchical namespace and integration patterns - [Google Cloud Storage Best Practices](https://cloud.google.com/storage/docs/best-practices) - GCS optimization and architecture patterns - [Apache Parquet Documentation](https://parquet.apache.org/docs/) - Columnar storage format for efficient analytics - [Delta Lake Documentation](https://docs.delta.io/) - ACID transactions and time travel for data lakes on cloud storage Source: https://www.conduktor.io/glossary/building-a-modern-data-lake-on-cloud-storage --- # Building and Managing Data Products Data products change how organizations think about data. Rather than treating data as a byproduct of applications or raw material locked in databases, data products package it as consumable, reliable, and well-defined assets that serve specific user needs. This article explores what data products are, how to design and build them, their relationship with streaming architectures, and best practices for governance and operations. ## What Are Data Products? ![Data Product Architecture and Lifecycle](https://www.conduktor.io/assets/images/glossary/building-and-managing-data-products-0.webp) A data product is a self-contained, reusable data asset designed with clear ownership, well-defined interfaces, and quality guarantees. Unlike traditional datasets or reports, data products are built with product thinking: they have users, versioned APIs, documentation, and service-level objectives. Key characteristics include: - **Clear ownership**: A dedicated team responsible for the product's quality and evolution - **Well-defined interface**: Schemas, APIs, or contracts that consumers can depend on - **Discovery and documentation**: Metadata, lineage, and usage examples - **Quality guarantees**: Data quality checks, SLAs for freshness and completeness - **Versioning**: Backward compatibility and deprecation policies For example, a "Customer 360" data product might aggregate customer data from multiple sources, provide a standardized schema via an API, guarantee 99.9% uptime, and update within 5 minutes of source changes. ## Designing Data Products: The Product Mindset Building effective data products requires understanding your users and their needs. Data product consumers might include data analysts, machine learning engineers, business intelligence tools, or downstream applications. Start with user discovery. What questions do they need to answer? What latency is acceptable? What format works best? A real-time fraud detection system needs sub-second streaming data, while a monthly executive dashboard can work with batch updates. Define clear interfaces early. For batch data products, this might be a REST API or a well-documented table schema. For streaming products, this means Kafka topic schemas with clear semantics: is each event a full snapshot or a delta? What guarantees does ordering provide? Consider versioning from day one. Breaking changes are inevitable as requirements evolve. Semantic versioning (v1, v2) lets you evolve the product while giving consumers time to migrate. Document deprecation timelines clearly. ## Technical Architecture and Implementation Data products can be implemented using various architectural patterns. Modern approaches commonly use event-driven and domain-driven design principles. In a data mesh architecture, data products are owned by domain teams rather than a centralized data team. Each domain (e.g., payments, customer service) publishes data products relevant to their business area. This decentralization improves scalability and domain expertise but requires strong governance. For detailed coverage of data mesh principles and implementation patterns, see [Data Mesh Principles and Implementation](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation). Technical implementation considerations include: - **Data modeling**: Use schemas that balance flexibility and strictness. Avro, Protobuf, and JSON Schema are popular for streaming data products because they support schema evolution while maintaining compatibility. Avro and Protobuf offer stronger type safety and smaller message sizes, while JSON Schema provides better accessibility for JavaScript-based consumers. - **Storage and serving layers**: Batch data products might use object storage (S3) with query engines (Athena, Trino). Streaming products use message brokers (Kafka 4.0+ with KRaft mode and Tiered Storage for scalable retention) or databases optimized for real-time queries. Modern lakehouse formats (Apache Iceberg, Delta Lake) bridge batch and streaming, enabling data products that serve both access patterns efficiently. For lakehouse architecture patterns, see [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture). - **Transformation pipelines**: Tools like Apache Flink (1.18+), Spark Structured Streaming, or dbt (Core 1.7+) transform raw data into polished data products. Flink's unified batch and streaming processing makes it particularly well-suited for building consistent data products. These pipelines should be versioned, tested, and monitored like application code. - **Access patterns**: Provide multiple access methods when possible. A single data product might offer batch exports, REST APIs for point queries, and streaming subscriptions for real-time updates. ## Data Products in Streaming Architectures Streaming architectures are particularly well-suited for data products because they enable real-time, event-driven data sharing across an organization. In Apache Kafka (4.0+), a data product might be represented as one or more topics with strict schema contracts managed through Schema Registry. For instance, a "payment-events" data product could be a Kafka topic where each message represents a payment transaction with a well-defined Avro schema. Consumers subscribe to this topic knowing the schema won't break unexpectedly. Kafka's KRaft mode (replacing ZooKeeper) simplifies operational management, while Tiered Storage enables cost-effective long-term retention for historical data products. Apache Flink can transform raw event streams into refined data products. A raw clickstream might be processed by Flink to produce a "user-session-events" data product that groups clicks into sessions, enriches them with user metadata, and outputs to a new Kafka topic. Streaming data products enable powerful patterns: - **Event-driven microservices**: Services consume data products to react to business events in real time. For architectural patterns, see [Event-Driven Microservices Architecture](https://www.conduktor.io/glossary/event-driven-microservices-architecture). - **Real-time analytics**: Stream processing applications compute aggregations and alerts directly from data products - **Change data capture (CDC)**: Database changes become consumable data products for downstream systems. For CDC implementation details, see [CDC for Real-Time Data Warehousing](https://www.conduktor.io/glossary/cdc-for-real-time-data-warehousing). Governance platforms help manage streaming data products by providing centralized schema governance, ensuring consumers see consistent data contracts, and monitoring topic health and consumer lag to verify SLAs are met. ## Governance and Quality Management Data products require ongoing governance to maintain trust and usability. - **Schema management** is critical. As data products evolve, schemas must change in compatible ways. Enforce compatibility rules (backward, forward, or full compatibility) through automated checks. Schema registry integrations let teams enforce these rules and preview breaking changes before deployment. For robust schema management, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). - **Data quality** must be continuously validated. Define quality metrics relevant to your product: completeness (no missing required fields), accuracy (values within expected ranges), and freshness (data arrives within SLA windows). For comprehensive coverage of these dimensions, see [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). Implement automated quality checks in your pipelines using tools like Great Expectations (1.x), Soda Core, or dbt tests. These frameworks integrate with streaming pipelines to validate data products in real time and alert when violations occur. - **Ownership and accountability** are foundational. Each data product needs a clear owner responsible for SLAs, breaking changes, and support. In data mesh architectures, domain teams own their products, but a central governance team might define standards and provide platforms. - **Access control** ensures sensitive data products are only available to authorized users. Role-based access control (RBAC) and attribute-based access control (ABAC) can restrict who can read or write to specific topics or tables. For detailed implementation guidance, see [Data Access Control: RBAC and ABAC](https://www.conduktor.io/glossary/data-access-control-rbac-and-abac). - **Lineage and observability** help users understand data flow. Document where data comes from, how it's transformed, and who consumes it. This transparency builds trust and simplifies debugging. For detailed coverage of lineage tracking, see [Data Lineage: Tracking Data from Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption). ### Self-Service for Data Product Management Managing data products at scale requires enabling domain teams to operate independently without creating governance gaps. Self-Service frameworks address this by providing teams autonomy within guardrails, domain teams manage their data products through declarative configuration files in version control (GitOps), and the platform validates changes against centralized governance policies before automatically provisioning resources. This approach eliminates ticket-based bottlenecks while maintaining consistency. Version control creates audit trails, automated compatibility checks prevent breaking changes, and access control policies are enforced centrally. Platform teams shift from gatekeepers to enablers. For implementation guidance, see [Self-Service](https://docs.conduktor.io/guide/conduktor-concepts/self-service). ## Operationalizing Data Products Production data products require operational rigor similar to software services. - **Monitoring** should cover both technical and product metrics. Track infrastructure health (throughput, latency, error rates) and product-specific KPIs (freshness, consumer count, quality violations). Modern observability platforms (Datadog, Grafana with Prometheus, Confluent Control Center) and distributed tracing tools (OpenTelemetry, Jaeger) provide end-to-end visibility across data products. For data freshness tracking, see [Data Freshness Monitoring and SLA Management](https://www.conduktor.io/glossary/data-freshness-monitoring-sla-management). Set up alerts when SLAs are at risk. - **Consumer analytics** reveal how your data product is used. Which consumers are most active? Are any experiencing lag? Tools like Kafka Lag Exporter, Burrow, or Confluent Control Center track consumer group metrics across all data products. For understanding consumer group behavior, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). Conduktor provides comprehensive consumer insights through [VIP Topics](https://docs.conduktor.io/guide/insights/vip-topics) that help identify critical data products and their consumer patterns. This information guides capacity planning and helps identify when consumers might need support during schema changes. Consumer monitoring dashboards that show lag, consumption rates, and trends across all consumers of a data product provide valuable insights. - **Lifecycle management** includes versioning, deprecation, and retirement. When introducing breaking changes, announce them early, provide migration guides, and support old versions during a transition period. Retire unused data products to reduce maintenance burden. - **Documentation** is essential for adoption. Include schema definitions, sample queries, SLA commitments, known limitations, and contact information for the owning team. Treat documentation as part of the product, not an afterthought. - **Example**: A retail company's "inventory-updates" data product streams real-time stock changes. The team monitors message throughput, schema validation errors, and consumer lag. When they need to add a "warehouse-location" field, they publish v2 with the new field as optional, maintain v1 for six months, notify all consumers via email and Slack, and provide code examples for migration. ## Summary Building and managing data products transforms data from a passive asset into an active, consumable resource that drives business value. By applying product thinking to data, organizations create reliable, discoverable, and high-quality data assets that serve diverse users. Key principles include: - Treat data as a product with users, interfaces, and SLAs - Design with clear ownership and governance from the start - Use streaming architectures for real-time, event-driven data products - Enforce schema contracts and data quality standards - Monitor operations and consumer health continuously - Document thoroughly and communicate changes proactively Whether you're building batch analytics datasets or real-time event streams, the data product approach keeps your data infrastructure trustworthy and usable as it grows. ## Related Concepts - [Data Lineage: Tracking Data from Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption) - Understanding data flow and dependencies is essential for managing data product lifecycles and impact analysis - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) - Core quality metrics that every data product must define and maintain - [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) - Critical patterns for evolving data product interfaces while maintaining consumer compatibility ## Sources and References 1. Dehghani, Zhamak. *Data Mesh: Delivering Data-Driven Value at Scale*. O'Reilly Media, 2022. [O'Reilly](https://www.oreilly.com/library/view/data-mesh/9781492092384/) 2. Kleppmann, Martin. *Designing Data-Intensive Applications*. O'Reilly Media, 2017. [O'Reilly](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/) 3. Confluent. "Building Data Products with Apache Kafka." Confluent Blog, 2023. [Confluent](https://www.confluent.io/blog/) 4. Uber Engineering. "Building Uber's Data Platform: From Batch to Real-Time." Uber Engineering Blog. [Uber Engineering](https://www.uber.com/blog/engineering/) 5. Sadalage, Pramod and Martin Fowler. "Evolutionary Database Design." martinfowler.com, 2016. [martinfowler.com](https://martinfowler.com/articles/evodb.html) Source: https://www.conduktor.io/glossary/building-and-managing-data-products --- # Building Recommendation Systems with Streaming Data Recommendation systems drive some of the most engaging experiences on the internet, from Netflix's video suggestions to Amazon's product recommendations. Traditional systems operate on batch-processed data, but modern applications need real-time personalization that responds to user behavior as it happens. That requires different architecture patterns built on streaming data platforms. ## Understanding Recommendation Systems Recommendation systems analyze user behavior, preferences, and contextual signals to suggest relevant content, products, or services. Traditional approaches use collaborative filtering (finding similar users or items), content-based filtering (analyzing item attributes), or hybrid methods combining both. The classic batch approach processes historical data periodically, perhaps daily or hourly, to update recommendations. A user's morning browsing session might not influence their afternoon recommendations. This delay creates a disconnect between user intent and system response. Streaming recommendation systems process events as they occur. When a user clicks a product, adds items to a cart, or watches a video, the system immediately updates features and potentially refreshes recommendations. This responsiveness creates more engaging experiences and captures fleeting user interests before they fade. ## The Challenge of Real-Time Recommendations Moving from batch to streaming recommendations introduces significant technical challenges. Batch systems benefit from complete datasets and predictable processing windows. Streaming systems must handle incomplete information, out-of-order events, and strict latency requirements. Latency becomes critical. Users expect recommendations to update within seconds, not minutes. This requires fast feature computation, efficient model serving, and optimized data pipelines. A streaming architecture must balance freshness with computational cost, not every event justifies recomputing all recommendations. Data consistency poses another challenge. In distributed streaming systems, events may arrive out of order or be duplicated. A user might rate a movie before the system processes their viewing event. Handling these scenarios requires careful event timestamp management and idempotent processing logic. Model complexity creates trade-offs. Complex deep learning models that work well in batch processing may be too slow for real-time serving. Streaming systems often use simpler models that can execute in milliseconds, or employ techniques like model distillation to compress larger models into faster approximations. ## Streaming Architecture for Recommendations A streaming recommendation architecture typically involves several key components working together. Apache Kafka captures user events in real-time and routes them through the system. Every click, view, purchase, or rating becomes an event flowing through Kafka topics. For foundational Kafka concepts, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) and [Kafka Topics, Partitions, Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). Feature engineering happens continuously using stream processing frameworks like Apache Flink or Kafka Streams. These systems maintain running aggregations, items viewed in the last hour, categories explored this session, time spent on different content types. Features stay fresh, updating with each new event. A feature store bridges streaming and serving. Modern platforms like Tecton, Feast, Hopsworks, or cloud-native solutions (AWS SageMaker Feature Store, Vertex AI Feature Store) maintain low-latency access to both batch-computed features (user demographics, long-term preferences) and streaming features (recent activity, session context). These systems typically provide sub-10ms read latency and handle feature freshness guarantees, ensuring models query consistent, up-to-date features during inference. For detailed coverage of feature store architectures, see [Feature Stores for Machine Learning](https://www.conduktor.io/glossary/feature-stores-for-machine-learning). Model serving requires specialized infrastructure. Modern frameworks include TensorFlow Serving, Ray Serve (for Python-native models), Seldon Core, KServe, and MLflow Model Serving. These platforms provide APIs for model inference with single-digit millisecond latency and support A/B testing, canary deployments, and automatic scaling. Some architectures pre-compute candidate recommendations and use streaming features only for final ranking, reducing computational load. For real-time ML inference patterns, see [Real-Time ML Inference with Streaming Data](https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data). Vector databases have become essential for modern recommendation systems using embedding-based approaches. Platforms like Pinecone, Weaviate, Milvus, Qdrant, and cloud solutions (Azure AI Search, Amazon OpenSearch with vector support) enable real-time similarity search over millions of item embeddings. These systems integrate with streaming pipelines to update embeddings as new items arrive or user preferences change, enabling semantic recommendations that go beyond traditional collaborative filtering. For comprehensive coverage, see [Vector Databases and Streaming Architectures](https://www.conduktor.io/glossary/vector-databases-and-streaming-architectures) and [Vector Embeddings in Streaming](https://www.conduktor.io/glossary/vector-embeddings-in-streaming). ## Implementation Patterns Consider an e-commerce platform implementing streaming recommendations. When a user views a product, several processes activate simultaneously. The event flows into Kafka, triggering multiple consumers that update different aspects of the recommendation system. For comprehensive e-commerce streaming patterns, see [E-Commerce Streaming Architecture Patterns](https://www.conduktor.io/glossary/e-commerce-streaming-architecture-patterns). One consumer updates session features: products viewed, categories explored, price ranges considered. Another updates the user's real-time preference vector, adjusting weights based on recent behavior. A third consumer may trigger re-ranking of recommended products already loaded in the user's browser. The architecture might look like this: ![The architecture might look like this](https://www.conduktor.io/assets/images/glossary/building-recommendation-systems-with-streaming-data-0.webp) Here's a concrete example using Kafka Streams to compute real-time recommendation features: ```java // Compute user session features from click events StreamsBuilder builder = new StreamsBuilder(); KStream events = builder.stream("user-events"); // Calculate items viewed per category in the last hour KTable, Long> categoryViews = events .filter((key, event) -> event.getEventType().equals("PRODUCT_VIEW")) .groupBy((key, event) -> event.getUserId() + ":" + event.getCategory(), Grouped.with(Serdes.String(), userEventSerde)) .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1))) .count(); // Compute average session duration KTable avgSessionDuration = events .groupByKey() .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30))) .aggregate( () -> new SessionStats(), (key, event, stats) -> stats.update(event), Materialized.with(Serdes.String(), sessionStatsSerde) ) .mapValues(stats -> stats.averageDuration()); // Write features to feature store topic categoryViews.toStream().to("recommendation-features"); ``` For Apache Flink users, similar feature computation might look like: ```java // Flink DataStream API for computing recommendation features StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream events = env .addSource(new FlinkKafkaConsumer<>("user-events", new UserEventSchema(), properties)); // Compute sliding window aggregations with watermarks DataStream categoryFeatures = events .assignTimestampsAndWatermarks( WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10)) .withTimestampAssigner((event, timestamp) -> event.getTimestamp()) ) .keyBy(event -> event.getUserId()) .window(SlidingEventTimeWindows.of(Time.hours(1), Time.minutes(5))) .aggregate(new CategoryViewAggregator()); categoryFeatures.addSink(new FlinkKafkaProducer<>("recommendation-features", new CategoryFeatureSchema(), properties)); ``` These examples demonstrate how stream processing continuously maintains fresh features. For in-depth coverage of Kafka Streams patterns, see [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). For Flink implementation details, see [Flink DataStream API: Building Streaming Applications](https://www.conduktor.io/glossary/flink-datastream-api-building-streaming-applications). Feature computation uses windowing to balance recency and stability. A tumbling window of 5 minutes aggregates very recent activity. A sliding window of 1 hour captures session-level behavior. Daily windows track longer-term trends. The model considers all three time scales. Event-time processing ensures correct ordering. Even if events arrive late or out of sequence, the stream processor uses event timestamps to maintain accurate feature values. Watermarks determine when to finalize window computations, balancing lateness tolerance against result freshness. For a detailed explanation of watermarks and handling late-arriving data, see [Event Time and Watermarks in Flink](https://www.conduktor.io/glossary/event-time-and-watermarks-in-flink) and [Handling Late-Arriving Data in Streaming](https://www.conduktor.io/glossary/handling-late-arriving-data-in-streaming). ## Real-World Applications Netflix provides a compelling example of streaming recommendations at scale. When you pause on a title, that dwell-time event updates your preference signals in real-time. The next row of recommendations may already reflect this micro-signal of interest. Netflix processes billions of events daily through Kafka, using sophisticated stream processing to maintain fresh features for hundreds of millions of users. For broader context on real-time ML architectures, see [Real-Time ML Pipelines](https://www.conduktor.io/glossary/real-time-ml-pipelines) and [Clickstream Analytics with Kafka](https://www.conduktor.io/glossary/clickstream-analytics-with-kafka). Spotify uses streaming data to power its Discover Weekly and Daily Mix features. As users listen to songs, skip tracks, or add songs to playlists, these events flow through streaming pipelines. The system balances immediate feedback (you're in a workout mood right now) with long-term preferences (you generally prefer indie rock). Uber Eats adjusts restaurant recommendations based on real-time signals like current location, time of day, and recent searches. The system must process location updates, search queries, and ordering events while maintaining low latency for mobile users. Stream processing enables this responsiveness across millions of concurrent sessions. ## Monitoring and Operating Streaming Recommendations Operating streaming recommendation systems requires robust monitoring and debugging capabilities. Unlike batch systems where failures are obvious and contained, streaming systems can silently degrade through data quality issues, processing lag, or feature drift. Monitoring Kafka topics becomes essential. Modern tools like Kafka Lag Exporter (integrated with Prometheus and Grafana) provide real-time visibility into consumer lag metrics. Are user events flowing at expected rates? Are there sudden spikes or drops that indicate upstream issues? Is consumer lag growing, suggesting the feature computation pipeline cannot keep up with event volume? For comprehensive monitoring patterns, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) and [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). Data quality checks must run continuously. Are event schemas valid? Are feature values within expected ranges? Do session boundaries make sense? Modern data observability platforms (Monte Carlo, Datadog Data Streams Monitoring, Conduktor) help teams monitor Kafka clusters, detect anomalies, inspect message content, and debug [data quality issues](https://docs.conduktor.io/guide/use-cases/observe-data-quality) before they impact recommendation quality. For systematic approaches, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) and [Data Quality vs Data Observability: Key Differences](https://www.conduktor.io/glossary/data-quality-vs-data-observability-key-differences). Access control matters particularly for recommendation systems, which process sensitive user behavior data. Kafka topics containing viewing history, purchase data, or browsing patterns require careful governance under regulations like GDPR and CCPA. Apache Kafka 4.0 with KRaft mode provides enhanced security capabilities and simplified ACL management. Teams need to audit who accesses which topics and ensure data masking policies are enforced for sensitive fields. For security best practices, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) and [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming). Testing streaming recommendation pipelines requires special considerations. Teams need to replay historical events to validate feature computation logic, test new model versions against production traffic, and verify that updates don't degrade recommendation quality. This requires infrastructure to capture production events, replay them in test environments, and compare results. For testing strategies, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). Model drift and data drift pose ongoing challenges for streaming recommendations. As user behavior patterns shift over time, models trained on historical data may degrade. Continuous monitoring of recommendation quality metrics (click-through rate, conversion rate, engagement time) helps detect when retraining becomes necessary. For strategies to detect and handle drift, see [Model Drift in Streaming](https://www.conduktor.io/glossary/model-drift-in-streaming) and [Data Drift in Streaming](https://www.conduktor.io/glossary/data-drift-in-streaming). ## Summary Building recommendation systems with streaming data transforms how applications personalize user experiences. By processing events in real-time, systems can respond to user intent with millisecond latency, creating more engaging and relevant interactions. The architecture requires careful orchestration of event streaming (Apache Kafka 4.0 with KRaft), stream processing (Apache Flink, Kafka Streams), feature stores (Tecton, Feast, Hopsworks), vector databases (Pinecone, Weaviate, Milvus), and model serving infrastructure (Ray Serve, KServe, TensorFlow Serving). Each component must balance freshness, accuracy, and computational cost. Modern 2025 platforms provide enterprise-grade capabilities for real-time feature computation, embedding-based similarity search, and low-latency model inference. Real-world implementations at Netflix, Spotify, and Uber demonstrate the power of streaming recommendations. These systems process billions of events daily while maintaining low latency and high availability, leveraging the latest streaming technologies and ML infrastructure. Success requires the right architecture plus monitoring (Kafka Lag Exporter, Prometheus, Grafana), debugging, and governance (ACLs, data masking, compliance). As recommendation systems incorporate vector embeddings and LLM-powered personalization, operational complexity grows and platform tooling becomes more important for maintaining data quality and system reliability. ## Related Concepts - [Streaming Data Pipeline](https://www.conduktor.io/streaming-data-pipeline) - Build end-to-end recommendation pipelines - [Real-Time Analytics with Streaming Data](https://www.conduktor.io/real-time-analytics-with-streaming-data) - Analytics for recommendation performance - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/data-quality-dimensions-accuracy-completeness-and-consistency) - Ensure high-quality recommendation inputs ## Sources and References 1. **Netflix Technology Blog** - "Distributed Time-Travel for Feature Generation" ([https://netflixtechblog.com/distributed-time-travel-for-feature-generation-389cccdd3907](https://netflixtechblog.com/distributed-time-travel-for-feature-generation-389cccdd3907)) 2. **Apache Kafka Documentation** - "Kafka Streams Use Cases" ([https://kafka.apache.org/documentation/streams/](https://kafka.apache.org/documentation/streams/)) 3. **Uber Engineering Blog** - "Building Uber's Eats Personalization Engine" ([https://www.uber.com/blog/eats-personalization/](https://www.uber.com/blog/eats-personalization/)) 4. **Spotify Engineering** - "How Machine Learning Powers Spotify's Recommendations" ([https://engineering.atspotify.com/](https://engineering.atspotify.com/)) 5. **Apache Flink Documentation** - "Real-time Feature Engineering" ([https://flink.apache.org/use-cases.html](https://flink.apache.org/use-cases.html)) Source: https://www.conduktor.io/glossary/building-recommendation-systems-with-streaming-data --- # CDC for Microservices: Event-Driven Architectures Change Data Capture (CDC) has become a key enabler for event-driven microservices architectures. By capturing database changes as streams of events, CDC connects traditional data storage to real-time event processing, letting services react to data changes with minimal latency. This article covers how CDC powers event-driven architectures, supports patterns like event sourcing and CQRS, and integrates with streaming platforms to build resilient microservices. For a comprehensive guide on implementing CDC with specific tools and configurations, see [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium). For a comparison of different CDC approaches, refer to [Log-Based vs Query-Based CDC Comparison](https://www.conduktor.io/glossary/log-based-vs-query-based-cdc-comparison). ## Understanding CDC in Event-Driven Systems ### The Event-Driven Architecture Challenge Microservices architectures aim to decompose monolithic applications into independent, loosely coupled services. However, maintaining data consistency and enabling inter-service communication without tight coupling presents significant challenges: - **Service Isolation**: Each microservice owns its data, but services need to react to changes in other services - **Dual Writes Problem**: Writing to a database and publishing an event as separate operations creates consistency risks. For example, if you save an order to the database and then publish an OrderCreated event to Kafka, a crash between these operations leaves your system in an inconsistent state, the order exists in the database but downstream services never receive the event. - **Real-Time Data Propagation**: Services require near-real-time visibility into data changes across the system Traditional approaches like API polling or message queues require application-level coordination, increasing complexity and creating coupling between services. ### CDC as an Event Backbone CDC transforms database changes into a stream of events automatically, without requiring application code changes. This provides several architectural benefits: - **Automatic Event Generation**: Every INSERT, UPDATE, and DELETE operation becomes an event, ensuring complete capture of state changes without manual event publishing. - **Guaranteed Ordering**: CDC preserves the order of changes as they occurred in the database transaction log (the internal log that databases use to ensure durability and replication), maintaining causal consistency. - **Low Latency**: Modern CDC implementations can capture and stream changes within milliseconds, enabling near-real-time event propagation. ![cdc-for-microservices-event-driven-architectures diagram 1](https://www.conduktor.io/assets/images/glossary/cdc-for-microservices-event-driven-architectures-0.webp) ## Event Sourcing and CDC ### Event Sourcing Fundamentals Event sourcing is an architectural pattern where the state of an application is determined by a sequence of events rather than storing only the current state. Each state change is captured as an immutable event, creating a complete audit trail. CDC complements event sourcing by: 1. **Deriving Events from State Changes**: Traditional databases store current state; CDC extracts the change events from the transaction log 2. **Event Store Population**: CDC can populate event stores from existing databases without application refactoring 3. **Hybrid Approaches**: Combine application-generated domain events with CDC-derived data change events For detailed coverage of event sourcing patterns and implementation strategies with Kafka, see [Event Sourcing Patterns with Kafka](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka) and [CQRS and Event Sourcing with Kafka](https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka). ### Practical Event Sourcing with CDC Each database operation generates a CDC event that downstream services can consume to maintain their own views or trigger business logic. For example, as an order moves through states (PENDING → CONFIRMED → SHIPPED), CDC captures each transition as an event (OrderCreated, OrderConfirmed, OrderShipped). **Example CDC Event Structure:** When an order record is updated in PostgreSQL, Debezium generates a CDC event like this: ```json { "before": { "id": 12345, "customer_id": 789, "status": "PENDING", "total": 99.99, "updated_at": "2025-01-15T10:30:00Z" }, "after": { "id": 12345, "customer_id": 789, "status": "CONFIRMED", "total": 99.99, "updated_at": "2025-01-15T10:35:00Z" }, "source": { "version": "2.5.0.Final", "connector": "postgresql", "name": "order-db", "ts_ms": 1705318500000, "db": "order_service", "schema": "public", "table": "orders" }, "op": "u", "ts_ms": 1705318500123 } ``` This event captures both the `before` and `after` state, allowing consumers to understand exactly what changed, when, and in which database. ## CQRS: Command Query Responsibility Segregation ### CDC-Powered Read Models CQRS separates write operations (commands) from read operations (queries) by maintaining separate models for each. CDC is instrumental in keeping read models synchronized with the write model: **Write Model (Command Side)**: - Handles business logic and state transitions - Single source of truth - Optimized for transactional consistency **Read Models (Query Side)**: - CDC streams changes from write model - Denormalized, optimized for specific query patterns - Multiple read models for different use cases ### Building Read Models with CDC CDC events can be consumed to build specialized read models. For example, consuming product database changes to update an Elasticsearch search index in real-time. This approach enables: - **Specialized Views**: Create read models optimized for specific query patterns (search, reporting, caching) - **Polyglot Persistence**: Materialize data into different storage engines (Elasticsearch, Redis, MongoDB) - **Independent Scaling**: Scale read and write paths independently ## Streaming Ecosystem Integration ### Apache Kafka and CDC Apache Kafka has become the de facto standard for event streaming in microservices architectures. CDC integrates with Kafka through connectors like Debezium. **Modern Kafka Features for CDC (2025)**: - **KRaft Mode (Kafka 4.0+)**: Kafka now runs without ZooKeeper, using its built-in Raft consensus protocol for metadata management. This simplifies CDC deployments by eliminating ZooKeeper dependencies and improving startup times and failover. - **Exactly-Once Semantics (EOS)**: Kafka's idempotent producers and transactional capabilities ensure CDC events are delivered exactly once, preventing duplicate event processing even during failures. - **Tiered Storage**: Store older CDC events in cloud object storage (S3, Azure Blob) while keeping recent data on local disks, enabling cost-effective long-term event retention for audit and replay scenarios. **CDC Tools Ecosystem (2025)**: - **Debezium 2.x+**: The leading open-source CDC platform with improved performance, support for all major databases (PostgreSQL, MySQL, MongoDB, SQL Server, Oracle), and enhanced Kafka Connect integration. - **Apache Flink CDC**: Native CDC connectors for Flink providing low-latency streaming with exactly-once guarantees, ideal for real-time data pipelines and stream processing. - **Kafka Streams & ksqlDB**: Process CDC events in real-time using stream processing, enabling event enrichment, filtering, and aggregation directly within the Kafka ecosystem. See [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams) and [ksqlDB for Real-Time Data Processing](https://www.conduktor.io/glossary/ksqldb-for-real-time-data-processing) for details. For foundational knowledge about Apache Kafka, refer to [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). To understand Kafka Connect's role in CDC pipelines, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines). ### The Outbox Pattern The outbox pattern solves the dual writes problem by ensuring atomic writes to the database and event publication. It's a critical pattern for maintaining consistency in event-driven architectures: **How the Outbox Pattern Works**: 1. Application writes business data and events to an outbox table in a single transaction 2. CDC monitors the outbox table for changes 3. CDC streams outbox events to Kafka 4. Events are consumed by downstream services 5. Optionally, processed events are cleaned up from the outbox The outbox pattern provides: - **Atomicity**: Database writes and event publication succeed or fail together - **Reliability**: No lost events due to message broker failures - **Ordering Guarantees**: Events are published in the order they were written **Outbox Pattern Implementation Example:** ```sql -- Create outbox table CREATE TABLE outbox ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), aggregate_type VARCHAR(255) NOT NULL, aggregate_id VARCHAR(255) NOT NULL, event_type VARCHAR(255) NOT NULL, payload JSONB NOT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); -- Application code writes both business data and outbox event atomically BEGIN; INSERT INTO orders (id, customer_id, status, total) VALUES ('12345', '789', 'CONFIRMED', 99.99); INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload) VALUES ( 'Order', '12345', 'OrderConfirmed', '{"orderId": "12345", "customerId": "789", "total": 99.99}'::jsonb ); COMMIT; ``` ```yaml # Debezium connector configuration for outbox pattern name: outbox-connector config: connector.class: io.debezium.connector.postgresql.PostgresConnector database.hostname: postgres database.port: 5432 database.user: cdc_user database.dbname: order_service table.include.list: public.outbox transforms: outbox transforms.outbox.type: io.debezium.transforms.outbox.EventRouter transforms.outbox.table.field.event.type: event_type transforms.outbox.table.field.event.key: aggregate_id transforms.outbox.table.field.event.payload: payload transforms.outbox.route.topic.replacement: ${routedByValue}.events ``` The Debezium outbox transformation extracts events from the outbox table and routes them to appropriate Kafka topics, transforming database rows into clean business events. For a comprehensive guide to the outbox pattern and its implementation, see [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing). To understand exactly-once guarantees in this context, refer to [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka). ### Governance and Visibility As CDC-based event-driven architectures scale, managing the proliferation of event streams, schemas, and consumers becomes challenging. Governance platforms provide critical visibility and control: **Schema Management**: - CDC generates events with evolving schemas as database schemas change - Schema registries track schema versions and enforce compatibility rules - Prevents breaking changes from propagating through the event pipeline - See [Avro vs Protobuf vs JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema) for schema format comparisons and [Schema Registry for Kafka](https://www.conduktor.io/glossary/schema-registry-for-kafka) for registry implementation **Quality and Monitoring**: - Monitor CDC lag and throughput metrics with [Kafka Connect Management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) - Set alerts for anomalous event patterns or consumption delays - Validate data quality rules on CDC event streams using [Data Quality tools](https://docs.conduktor.io/guide/use-cases/observe-data-quality) **Example Governance Use Case**: When a database schema change is proposed (e.g., renaming a column in the products table): 1. Lineage views show all services consuming product CDC events 2. Schema compatibility checks identify which consumers would break 3. Teams coordinate migration: new column added alongside old one 4. CDC captures both columns during transition period 5. Consumers migrate to new schema at their own pace 6. Old column deprecated and removed after all consumers migrated This governance layer prevents the chaos that often accompanies event-driven architectures at scale. ## Service Integration Patterns ### Inter-Service Communication CDC enables several patterns for service integration: **1. Database-to-Database Synchronization**: Services maintain local caches or denormalized views of data owned by other services. ![cdc-for-microservices-event-driven-architectures diagram 2](https://www.conduktor.io/assets/images/glossary/cdc-for-microservices-event-driven-architectures-1.webp) **2. Event-Driven Workflows**: Services coordinate complex workflows through event choreography. For example, the Inventory Service listens for OrderCreated events, reserves inventory, and writes to its local database. CDC then publishes InventoryReserved or InventoryInsufficient events for downstream services. **3. Polyglot Persistence**: Different services use optimal storage technologies (e.g., PostgreSQL for transactional data, Elasticsearch for search, Redis for caching) while staying synchronized through CDC events. ### Error Handling and Resilience CDC-based integration requires robust error handling: - **Dead Letter Queues**: Failed events are routed to DLQs for analysis and replay. See [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling) for implementation patterns. - **Idempotency**: Consumers must handle duplicate events gracefully by tracking processed event IDs - **Circuit Breakers**: Prevent cascading failures when downstream services are unavailable - **Compensation Events**: Publish compensating events when operations need to be reversed ## Best Practices and Considerations ### Schema Evolution Database schema changes directly impact CDC events. Follow these practices: - **Backward Compatibility**: Add columns instead of renaming; deprecate gradually - **Schema Registry**: Use Confluent Schema Registry or similar to version and validate schemas - **Consumer Resilience**: Design consumers to handle missing or additional fields gracefully ### Performance Optimization CDC can generate high event volumes. Optimize with: - **Filtering**: Configure CDC to capture only relevant tables and columns - **Batching**: Consume events in batches for higher throughput - **Partitioning**: Use appropriate Kafka partition keys to distribute load ### Monitoring and Observability Implement comprehensive monitoring using modern observability tools: **Key CDC Metrics (2025)**: - **CDC Lag**: Time between database change and event availability in Kafka - **Replication Slot Growth**: Monitor PostgreSQL replication slots (Write-Ahead Log segments retained for CDC) to prevent disk exhaustion - **Consumer Lag**: Track how far behind consumers are from the latest events using tools like Kafka Lag Exporter or Burrow. Understanding consumer groups is essential, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) - **Schema Compatibility**: Alert on schema incompatibilities detected by Schema Registry - **Throughput & Errors**: Monitor CDC connector throughput, error rates, and restart counts **Modern Observability Stack**: - **OpenTelemetry**: Standardized telemetry collection for distributed tracing across CDC pipelines, Kafka brokers, and consuming services. See [Distributed Tracing for Kafka Applications](https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications) for implementation guidance. - **Prometheus & Grafana**: Metrics collection and visualization for CDC lag, Kafka throughput, and consumer health. For comprehensive Kafka metrics, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). - **Kafka Lag Exporter**: Dedicated tool for monitoring consumer group lag with Prometheus integration - **Conduktor Platform**: Unified governance and monitoring for Kafka-based CDC architectures Governance platforms provide centralized monitoring across the entire CDC pipeline, aggregating metrics from connectors, brokers, and consumers. ### Security and Compliance CDC streams contain sensitive data: - **Encryption**: Encrypt data in transit (TLS) and at rest. See [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka) for implementation details. - **Access Control**: Implement fine-grained ACLs on Kafka topics. Refer to [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) for access control strategies. - **Data Masking**: Use Kafka Connect transformations to mask PII before events reach consumers. For PII handling strategies, see [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming) and [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). - **Audit Logging**: Track who accesses CDC event streams. For comprehensive audit logging, see [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). ## Summary Change Data Capture has become a foundational technology for modern event-driven microservices architectures. By automatically transforming database changes into event streams, CDC enables: - **True Event-Driven Architecture**: Services react to data changes in real-time without tight coupling - **Event Sourcing**: Complete audit trails and state reconstruction from change events - **CQRS Implementation**: Synchronized read models optimized for different query patterns - **Reliable Integration**: The outbox pattern ensures atomic event publication and data consistency - **Streaming Ecosystem**: Seamless integration with Apache Kafka for scalable event distribution The combination of CDC with streaming platforms like Kafka, governance tools, and architectural patterns like outbox and CQRS creates resilient, scalable microservices ecosystems. As organizations continue to decompose monoliths and embrace event-driven architectures, CDC will remain a critical enabler of loosely coupled, reactive systems. Successful CDC implementation requires attention to schema evolution, performance, monitoring, and security. With the right tooling and practices, teams can build microservices architectures that are both flexible and maintainable. ## Related Concepts - [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) - Core CDC concepts and patterns that enable event-driven microservices architectures. - [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) - Practical implementation guide for setting up CDC connectors to power microservices communication. - [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling) - Essential error handling patterns for CDC-based event streams in microservices systems. ## Sources and References **Core Technologies:** - [Debezium Documentation](https://debezium.io/documentation/) - Open-source CDC platform for databases - [Apache Kafka Documentation](https://kafka.apache.org/documentation/) - Distributed event streaming platform - [Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/) - Schema versioning and compatibility **Architectural Patterns:** - Martin Fowler - [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html) - Martin Fowler - [CQRS](https://martinfowler.com/bliki/CQRS.html) - Chris Richardson - [Microservices Patterns: Transactional Outbox](https://microservices.io/patterns/data/transactional-outbox.html) **Tools and Platforms:** - [Conduktor](https://www.conduktor.io/) - Apache Kafka governance, monitoring, and development platform - [Kafka Connect](https://docs.confluent.io/platform/current/connect/) - Integration framework for Kafka **Best Practices:** - [Debezium Tutorial](https://debezium.io/documentation/reference/tutorial.html) - Getting started with CDC - [Designing Event-Driven Systems](https://www.confluent.io/designing-event-driven-systems/) - Comprehensive guide by Ben Stopford - [Building Microservices](https://www.oreilly.com/library/view/building-microservices-2nd/9781492034018/) - Sam Newman (2nd Edition) Source: https://www.conduktor.io/glossary/cdc-for-microservices-event-driven-architectures --- # CDC for Real-Time Data Warehousing Real-time data warehousing is now a standard requirement for data-driven organizations. Traditional batch ETL processes that run nightly or hourly can no longer meet the demand for immediate insights. Change Data Capture (CDC) provides the foundation for real-time data warehouses by continuously capturing and streaming database changes as they occur. ## Understanding CDC in the Data Warehouse Context Change Data Capture is a design pattern that identifies and captures changes made to data in a database, then delivers those changes to downstream systems in real-time or near-real-time. Unlike traditional full-table extracts that read entire datasets repeatedly, CDC tracks only the incremental changes, inserts, updates, and deletes, making it highly efficient for data warehousing scenarios. In a data warehouse context, CDC bridges operational databases (OLTP - Online Transaction Processing systems optimized for transactional workloads) and analytical databases (OLAP - Online Analytical Processing systems optimized for complex queries and aggregations). By capturing changes at the source and streaming them to the warehouse, organizations can maintain synchronized, up-to-date analytical datasets without impacting source system performance or requiring large batch processing windows. CDC operates through several mechanisms: - **Log-based CDC**: Reads transaction logs from the source database (e.g., MySQL binlog, PostgreSQL WAL - Write-Ahead Log) - **Trigger-based CDC**: Uses database triggers to capture changes into shadow tables - **Query-based CDC**: Polls tables for changes using timestamp or version columns Log-based CDC is generally preferred for data warehousing because it has minimal performance impact on source systems and captures all changes reliably without requiring schema modifications. For comprehensive coverage of CDC fundamentals and mechanisms, see [What is Change Data Capture (CDC): Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). ## Incremental Loading Patterns Incremental loading is the cornerstone of efficient real-time data warehousing. Rather than reprocessing entire datasets, incremental patterns apply only the changes captured by CDC to the target warehouse. ### Append-Only Pattern In this pattern, all CDC events are appended to the warehouse as immutable records. Each change creates a new row with metadata indicating the operation type and timestamp. This approach preserves complete history and simplifies stream processing, but requires additional logic for querying current state. ```sql CREATE TABLE customer_changes ( change_id BIGINT PRIMARY KEY, customer_id INT, name VARCHAR(255), email VARCHAR(255), operation_type VARCHAR(10), -- INSERT, UPDATE, DELETE change_timestamp TIMESTAMP, source_transaction_id VARCHAR(100) ); ``` ### Upsert Pattern The upsert (update-or-insert) pattern applies CDC changes directly to warehouse tables, maintaining current state. Updates and inserts modify existing rows or create new ones, while deletes remove records. This pattern provides simpler querying but requires merge capabilities in the warehouse. ```sql MERGE INTO customer_warehouse AS target USING customer_cdc_stream AS source ON target.customer_id = source.customer_id WHEN MATCHED AND source.operation = 'UPDATE' THEN UPDATE SET name = source.name, email = source.email WHEN MATCHED AND source.operation = 'DELETE' THEN DELETE WHEN NOT MATCHED AND source.operation = 'INSERT' THEN INSERT (customer_id, name, email) VALUES (source.customer_id, source.name, source.email); ``` ### Slowly Changing Dimensions (SCD) For maintaining historical context, SCD patterns track how dimension attributes change over time. Type 2 SCD creates a new row for each change with validity timestamps, enabling historical analysis while preserving current state. ## Streaming Ecosystem Integration Modern CDC pipelines use streaming platforms to decouple data capture from consumption, providing scalability, fault tolerance, and flexibility. ### Architecture Components A typical CDC streaming architecture includes: - **Debezium** captures changes from source databases by reading transaction logs and converting them to change events. It supports multiple databases (MySQL, PostgreSQL, MongoDB, Oracle, SQL Server) and produces standardized event formats. - **Apache Kafka** is the distributed event streaming platform, receiving CDC events from Debezium and buffering them for consumption. Kafka topics organize events by source table, and partitioning ensures ordering guarantees per key. For Kafka topic design considerations, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines). - **Apache Flink** or other stream processors consume CDC events from Kafka, transform them as needed, and write to the data warehouse. Flink provides exactly-once processing semantics (guaranteeing each change event is processed once and only once, even during failures), stateful operations, and windowing capabilities essential for complex transformations. For details on guaranteeing data consistency, see [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka). ### Reference Architecture ![### Reference Architecture](https://www.conduktor.io/assets/images/glossary/cdc-for-real-time-data-warehousing-0.webp) ### Debezium Configuration Example ```json { "name": "postgres-cdc-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres-db.example.com", "database.port": "5432", "database.user": "debezium_user", "database.password": "${secret:debezium-secrets:postgres-password}", "database.dbname": "production_db", "database.server.name": "prod", "table.include.list": "public.customers,public.orders", "plugin.name": "pgoutput", "slot.name": "debezium_slot", "publication.name": "debezium_publication", "topic.prefix": "prod", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "transforms": "unwrap", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "false" } } ``` This configuration captures changes from specific PostgreSQL tables using the native `pgoutput` logical decoding plugin (available in PostgreSQL 10+, recommended for Debezium 2.x+) and publishes them to Kafka topics with the prefix `prod` (e.g., `prod.public.customers`). The `ExtractNewRecordState` transformation simplifies the event structure by extracting the after-state of changes, making downstream processing easier. For detailed Debezium implementation guidance, see [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium). ### Modern CDC Platforms and 2025 Tooling While Debezium remains the leading open-source CDC solution (version 3.0+ as of 2024/2025), the CDC landscape has evolved with several modern alternatives: **Cloud-Native CDC Services**: - **AWS Database Migration Service (DMS)**: Fully managed CDC for AWS RDS and Aurora databases, with native integration to Redshift, S3, and Kinesis - **Google Cloud Datastream**: Serverless CDC service for MySQL, PostgreSQL, and Oracle, streaming to BigQuery, Cloud Storage, or Pub/Sub - **Azure Data Factory with Change Feed**: Native CDC for Azure Cosmos DB and SQL Database **Unified Data Integration Platforms**: - **Airbyte** (2025): Open-source data integration platform with 300+ connectors, including CDC connectors for major databases with configuration UI and monitoring - **Fivetran**: Managed CDC platform with automatic schema drift handling and built-in transformations - **Striim**: Real-time data integration with CDC, in-flight processing, and multi-cloud support **Kafka 4.0+ Enhancements** (2024/2025): - Native **KRaft mode** (ZooKeeper removed): Simplified Kafka deployment and improved metadata performance for CDC pipelines - **Kafka Connect improvements**: Enhanced connector management, better error handling, and improved exactly-once support - **Tiered storage**: Cost-effective long-term retention of CDC events in object storage (S3, GCS, Azure Blob) These modern platforms reduce operational overhead and provide managed CDC infrastructure, though open-source solutions like Debezium offer greater flexibility and cost control for sophisticated use cases. For Kafka fundamentals and architecture, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ### Governance and Visibility In production CDC pipelines, governance and operational visibility become critical. Streaming governance platforms provide centralized monitoring and management for Kafka-based CDC infrastructures: - **Schema validation**: Ensures CDC events conform to expected schemas before reaching the warehouse - **Data quality monitoring**: Tracks event volumes, delays, and anomalies in CDC streams with [Data Quality Monitoring](https://docs.conduktor.io/guide/use-cases/observe-data-quality) - **Access control**: Manages which teams can consume specific CDC topics - **Pipeline observability**: Visualizes data flows from source databases through Kafka to warehouses using [Kafka Connect Management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) This governance layer prevents data quality issues from propagating to analytical systems and provides the operational transparency needed to maintain SLAs for real-time reporting. ## Data Lake Ingestion with CDC CDC pipelines often feed both data warehouses and data lakes, creating a unified approach to real-time analytics across structured and semi-structured data. ### Initial Snapshots and Bootstrap Strategy Before streaming incremental changes, CDC pipelines must capture the current state of source tables through initial snapshots. This bootstrap process ensures the target warehouse starts with complete data before applying ongoing changes: 1. **Snapshot Capture**: Debezium or similar tools read the entire table(s) using consistent snapshots (leveraging database-specific mechanisms like PostgreSQL's `REPEATABLE READ` transactions) 2. **Streaming Transition**: After snapshot completion, the CDC process switches to reading transaction logs from the point captured during the snapshot 3. **Backfill Strategies**: For large tables, consider splitting snapshots into chunks or using parallel snapshot threads to reduce initial load time Modern CDC tools handle this transition automatically, preventing data loss during the snapshot-to-streaming switch. For large-scale migrations, see [CDC for Microservices Event-Driven Architectures](https://www.conduktor.io/glossary/cdc-for-microservices-event-driven-architectures). ### Lake House Architecture The lake house pattern combines data lake flexibility with warehouse performance. CDC events land initially in the data lake (S3, Azure Data Lake) in raw format, then incremental transformations promote data through bronze, silver, and gold layers. - **Bronze Layer**: Raw CDC events stored as Parquet or Avro files, partitioned by timestamp - **Silver Layer**: Cleaned and deduplicated events with basic transformations applied - **Gold Layer**: Business-aggregated datasets optimized for analytics Technologies like **Delta Lake**, **Apache Iceberg**, and **Apache Hudi** provide ACID transactions and time-travel capabilities on lake storage, enabling upserts and deletes that traditional data lakes cannot handle efficiently. **Modern Table Format Features (2025)**: - **Apache Iceberg 1.4+**: Native upsert support with merge-on-read optimization, partition evolution without rewriting data, and multi-table transactions - **Apache Hudi 0.14+**: Record-level indexes for faster upserts, clustering for improved query performance, and metadata table for change tracking - **Apache XTable (formerly OneTable)**: Universal table format translator enabling cross-format interoperability (read Iceberg as Delta, etc.) - **Delta Lake 3.0+**: Deletion vectors for efficient row-level deletes, liquid clustering (automatic data organization), and improved merge performance For details on lakehouse patterns, see [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture) and [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg). ### Incremental Processing Strategy ```python # Example: Flink streaming job for CDC to Delta Lake from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import StreamTableEnvironment, EnvironmentSettings # Configure Flink for exactly-once processing env = StreamExecutionEnvironment.get_execution_environment() env.enable_checkpointing(60000) # Checkpoint every 60 seconds env.get_checkpoint_config().set_checkpointing_mode(CheckpointingMode.EXACTLY_ONCE) # Create Table Environment settings = EnvironmentSettings.in_streaming_mode() t_env = StreamTableEnvironment.create(env, settings) # Define Kafka source with CDC events t_env.execute_sql(""" CREATE TABLE customer_cdc ( customer_id INT, name STRING, email STRING, operation STRING, ts TIMESTAMP(3), WATERMARK FOR ts AS ts - INTERVAL '5' SECOND ) WITH ( 'connector' = 'kafka', 'topic' = 'prod.public.customers', 'properties.bootstrap.servers' = 'kafka:9092', 'properties.group.id' = 'flink-cdc-consumer', 'format' = 'json', 'scan.startup.mode' = 'earliest-offset' ) """) # Define Delta Lake sink t_env.execute_sql(""" CREATE TABLE customer_delta ( customer_id INT, name STRING, email STRING, PRIMARY KEY (customer_id) NOT ENFORCED ) WITH ( 'connector' = 'delta', 'path' = 's3://data-lake/silver/customers', 'write.mode' = 'upsert' ) """) # Process CDC stream and upsert to Delta Lake t_env.execute_sql(""" INSERT INTO customer_delta SELECT customer_id, name, email FROM customer_cdc WHERE operation IN ('INSERT', 'UPDATE') """) ``` This Flink job demonstrates exactly-once processing semantics with checkpointing, ensuring no data loss or duplication when streaming CDC events from Kafka to Delta Lake. The watermark configuration handles late-arriving events, and the upsert mode ensures the Delta table reflects the current state of each customer. ## Operational Considerations Implementing CDC for real-time data warehousing requires careful attention to operational aspects that impact reliability and performance. ### Schema Evolution CDC pipelines must handle schema changes in source databases without breaking downstream consumers. Use schema registries (Confluent Schema Registry, Apicurio) to version and validate schemas. Configure warehouses to support schema evolution through automatic column additions or versioned tables. For detailed guidance on managing schema changes, see [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) and [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). ### Backpressure and Lag Management Monitor consumer lag between CDC event production and warehouse ingestion. High lag indicates **backpressure**, a condition where the warehouse or stream processor cannot keep up with the incoming change volume, causing events to accumulate in Kafka. Address this through: - **Horizontal scaling of stream processors**: Add more Flink task managers or Kafka Streams instances - **Warehouse optimization**: Improve clustering, partitioning, and indexing strategies - **Micro-batching**: Reduce write overhead by batching multiple CDC events into single warehouse transactions - **Filtering irrelevant changes**: Drop unnecessary columns or tables at the CDC source to reduce data volume - **Monitoring tools**: Use Kafka Lag Exporter, Burrow, or Confluent Control Center to track consumer lag metrics For comprehensive coverage of backpressure patterns, see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems). For monitoring consumer lag, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). ### Disaster Recovery CDC pipelines must support failure recovery without data loss. Kafka's retention policies should accommodate reasonable downtime, and checkpoint mechanisms in stream processors ensure exactly-once processing. Maintain runbooks for rebuilding state after catastrophic failures, including procedures for historical backfills from database snapshots. For disaster recovery strategies and high availability patterns, see [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters) and [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability). ### Performance Metrics and SLAs Establish clear performance targets for CDC pipelines to ensure real-time data warehousing meets business requirements: **Latency Metrics**: - **End-to-end latency**: Time from database commit to warehouse availability (typical target: < 60 seconds for real-time use cases) - **Replication lag**: Time between source database change and CDC event capture (target: < 5 seconds) - **Processing lag**: Time for stream processor to consume and transform events (target: < 10 seconds) **Throughput Metrics**: - **Events per second**: CDC event production rate per table/topic - **Warehouse write rate**: Insert/update operations per second in target warehouse - **Data volume**: GB/hour of change data captured and processed **Reliability Metrics**: - **Data loss**: Zero tolerance for production CDC pipelines (requires exactly-once semantics) - **Duplicate rate**: Percentage of duplicate events reaching the warehouse (target: 0% with idempotent writes) - **Pipeline availability**: Uptime percentage (typical target: 99.9%+) Monitor these metrics using tools like Prometheus, Grafana, Datadog, or cloud-native monitoring services. Set up alerts for SLA violations and establish escalation procedures. For streaming SLA management, see [SLA for Streaming](https://www.conduktor.io/glossary/sla-for-streaming). ### Data Quality and Validation CDC pipelines must maintain data quality as changes flow from operational systems to analytical warehouses: **Validation Strategies**: - **Schema validation**: Ensure CDC events match expected schema versions before processing - **Row count reconciliation**: Periodically compare source table counts with warehouse row counts - **Checksum validation**: Verify data integrity by comparing aggregated checksums between source and target - **Business rule validation**: Apply domain-specific validation rules (e.g., non-negative account balances, valid date ranges) **Quality Monitoring Tools (2025)**: - **Great Expectations**: Define data quality expectations on CDC streams and warehouse tables - **Soda Core**: Open-source data quality framework with SQL-based checks - **Monte Carlo**: Data observability platform with automated anomaly detection for CDC pipelines - **Elementary**: dbt-native data quality monitoring Implement data quality checks at multiple stages: at CDC capture (Debezium level), during stream processing (Flink/Kafka Streams), and post-warehouse ingestion (batch validation jobs). For comprehensive data quality strategies, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) and [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). ## Summary Change Data Capture transforms traditional batch-oriented data warehousing into responsive, real-time analytics infrastructure. By capturing incremental changes through log-based mechanisms and streaming them via platforms like Kafka, organizations achieve near-instantaneous data availability without overwhelming source systems or warehouse resources. Successful CDC implementation combines appropriate loading patterns (append-only, upsert, SCD) with robust streaming architectures built on Debezium, Kafka, and stream processors like Flink. Integration with data lakes through lake house architectures extends CDC benefits to both structured analytics and flexible exploration workloads. Operational maturity requires governance tools, careful schema evolution, and proactive monitoring. A well-implemented CDC pipeline delivers near-real-time data availability while maintaining the reliability and accuracy that analytical systems require. ## Related Concepts - [Streaming ETL vs Traditional ETL](https://www.conduktor.io/glossary/streaming-etl-vs-traditional-etl) - Compare traditional batch ETL approaches with CDC-based streaming ETL for real-time data warehouse loading. - [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) - Foundational CDC concepts that enable real-time data warehousing architectures. - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) - Learn how to maintain data quality in CDC pipelines feeding data warehouses. ## Sources and References - **Apache Kafka Documentation**: [https://kafka.apache.org/documentation/](https://kafka.apache.org/documentation/) - **Debezium Documentation**: [https://debezium.io/documentation/](https://debezium.io/documentation/) - **Apache Flink CDC Connectors**: [https://nightlies.apache.org/flink/flink-cdc-docs-stable/](https://nightlies.apache.org/flink/flink-cdc-docs-stable/) - **Delta Lake Documentation**: [https://docs.delta.io/](https://docs.delta.io/) - **Slowly Changing Dimensions**: Kimball, R. & Ross, M. (2013). *The Data Warehouse Toolkit*. Wiley. - **Change Data Capture Patterns**: Kleppmann, M. (2017). *Designing Data-Intensive Applications*. O'Reilly Media. Source: https://www.conduktor.io/glossary/cdc-for-real-time-data-warehousing --- # Chaos Engineering for Kafka **Chaos engineering for Kafka** is the practice of deliberately injecting failures (broker outages, latency spikes, message corruption) into streaming systems to verify resilience before production incidents occur. Proxy-based approaches enable this testing without infrastructure access or risk to actual data. Streaming platforms fail in ways that batch systems never experience. A Kafka broker goes down during peak traffic, consumers lag behind as partitions rebalance, or corrupted messages propagate through downstream services before anyone notices. **Chaos engineering for Kafka means proactively injecting these failures** to discover weaknesses before they cause outages. The challenge with traditional chaos engineering approaches: killing brokers or introducing network partitions requires infrastructure access and risks impacting production data. **Protocol-aware proxies solve this** by intercepting Kafka traffic and injecting failures at the client-visible layer, without touching actual brokers. ## Why Kafka Needs Chaos Testing ![Kafka Failure Points for Chaos Testing](https://www.conduktor.io/assets/images/glossary/chaos-engineering-for-kafka-0.webp) Kafka's distributed architecture creates failure scenarios that don't exist in simpler systems: - **Broker failures during rebalancing**: When a broker goes down, Kafka elects new partition leaders and consumers rebalance. Applications must handle temporary `NOT_LEADER_OR_FOLLOWER` errors and connection resets gracefully. - **Consumer lag cascades**: A slow consumer triggers rebalancing, which causes other consumers to pause, creating a lag spiral that can take minutes to recover. - **Exactly-once semantics under failure**: Idempotent producers and transactions behave correctly during normal operation, but what happens when brokers return stale `INVALID_PRODUCER_EPOCH` errors? - **Schema evolution failures**: A consumer receives a message with an unknown schema ID. Does it crash, skip the record, or enter an infinite retry loop? Testing these scenarios in production risks data loss or service degradation. Testing them in staging often misses configuration differences. Chaos engineering bridges this gap by injecting failures in a controlled way. ## Common Kafka Failure Scenarios to Test Effective chaos engineering targets specific failure modes rather than random disruption: - **Broker unavailability**: Simulate `UNKNOWN_SERVER_ERROR` or `CORRUPT_MESSAGE` responses to test producer retry logic and consumer error handling - **Leader election delays**: Inject `LEADER_NOT_AVAILABLE`, `NOT_LEADER_OR_FOLLOWER`, and `BROKER_NOT_AVAILABLE` errors to validate client behavior during partition leadership changes - **Latency spikes**: Add 500ms-2s delays to produce and fetch requests to test timeout configurations and backpressure handling - **Message corruption**: Append random bytes to message payloads to verify consumer deserialization error handling - **Duplicate messages**: Return the same message multiple times to test idempotency in downstream processing - **Invalid schema IDs**: Overwrite schema registry IDs with invalid values to test deserialization failure paths Each scenario maps to a real-world failure. Broker unavailability mirrors hardware failures or network partitions. Latency spikes replicate cross-AZ network degradation. Duplicate messages simulate what happens when consumers crash before committing offsets. ## Proxy-Based Chaos Testing with Conduktor Gateway Traditional chaos tools require infrastructure access to kill processes or inject network faults. Conduktor Gateway takes a different approach: it sits between Kafka clients and brokers, intercepting the Kafka protocol and injecting failures at the application layer. This architecture enables chaos testing without: - SSH access to broker nodes - Kubernetes privileges to kill pods - Network-level fault injection tools - Risk of corrupting actual topic data Gateway's chaos interceptors simulate failures that clients experience, not infrastructure-level outages. The brokers remain healthy while clients receive error responses that match real failure scenarios. For complete interceptor documentation, see [Chaos Testing in Conduktor Gateway](https://docs.conduktor.io/guide/use-cases/chaos-testing). ### Simulating Broken Brokers Test how producers and consumers handle broker errors: ```yaml pluginClass: io.conduktor.gateway.interceptor.chaos.SimulateBrokenBrokersPlugin config: rateInPercent: 25 errorMap: FETCH: UNKNOWN_SERVER_ERROR PRODUCE: CORRUPT_MESSAGE ``` 25% of requests return errors. Monitor your application's retry behavior and alert thresholds. ### Simulating Latency Test timeout configurations under network degradation: ```yaml pluginClass: io.conduktor.gateway.interceptor.chaos.SimulateSlowBrokerPlugin config: rateInPercent: 50 minLatencyMs: 200 maxLatencyMs: 2000 ``` Half of requests experience 200ms-2s latency. Validates `request.timeout.ms` and `session.timeout.ms` settings. ### Simulating Leader Elections Test client resilience to partition leadership changes: ```yaml pluginClass: io.conduktor.gateway.interceptor.chaos.SimulateLeaderElectionsErrorsPlugin config: rateInPercent: 30 ``` Clients receive `LEADER_NOT_AVAILABLE`, `NOT_LEADER_OR_FOLLOWER`, and `BROKER_NOT_AVAILABLE` errors. ### Simulating Message Corruption Verify consumer deserialization error handling: ```yaml pluginClass: io.conduktor.gateway.interceptor.chaos.FetchSimulateMessageCorruptionPlugin config: topic: "orders.*" rateInPercent: 10 sizeInBytes: 50 ``` 10% of messages have random bytes appended. Tests whether consumers log errors or crash. For all chaos interceptors, see [Chaos Testing in Conduktor Gateway](https://docs.conduktor.io/guide/use-cases/chaos-testing). ## Running Chaos Experiments Safely Chaos engineering requires discipline to avoid causing the very outages you're trying to prevent: - **Start with low failure rates**: Begin at 5-10% and increase gradually. A 100% failure rate in production isn't chaos engineering, it's an outage. - **Scope experiments narrowly**: Apply interceptors to specific virtual clusters, topics, or consumer groups rather than all traffic. Gateway's `scope` configuration limits blast radius. - **Define success metrics before starting**: "End-to-end latency stays below 2s" or "Consumer lag recovers within 60 seconds." Without measurable criteria, you can't determine if the experiment succeeded. - **Monitor during experiments**: Watch broker metrics, consumer lag, producer error rates, and application logs in real-time. Abort if metrics exceed acceptable thresholds. - **Document findings**: Each experiment should produce actionable results: configuration changes, code fixes, or increased confidence in existing resilience. A typical experiment workflow: 1. Define hypothesis: "If 20% of produce requests fail, producer retries will succeed and no messages will be lost" 2. Configure interceptor with 20% failure rate on a non-critical topic 3. Run load test while monitoring producer success rate and topic message count 4. Verify message counts match (no data loss) and producer metrics show retries 5. Remove interceptor and document results ## Summary Chaos engineering for Kafka validates that streaming applications **survive the failures they'll inevitably encounter** in production. Broker outages, latency spikes, leader elections, and message corruption all occur in real deployments. The choice is whether to discover resilience gaps during controlled experiments or during 3 AM incidents. **Proxy-based chaos testing** with tools like Conduktor Gateway enables failure injection without infrastructure access, making it practical to run experiments regularly as configurations and applications evolve. Start with the failure scenarios that match your operational experience: if broker restarts have caused issues before, test leader election handling first. The goal isn't to prove systems are perfectly resilient. It's to **find the configuration values, retry policies, and error handling paths** that need work before production traffic reveals them. ## Related Concepts - [Zero Trust Architecture for Kafka](https://www.conduktor.io/glossary/zero-trust-architecture-for-kafka): Security framework where chaos testing validates continuous verification mechanisms - [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems): Broader chaos engineering principles for Kafka, Flink, and other streaming platforms - [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters): Planning for and recovering from cluster-level failures - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka): Understanding delivery guarantees under failure conditions ## Sources and References - [Conduktor Gateway Chaos Testing Documentation](https://docs.conduktor.io/guide/use-cases/chaos-testing) - Complete reference for chaos interceptors including configuration options and error types - [Apache Kafka Documentation - Design and Reliability Guarantees](https://kafka.apache.org/documentation/#design_reliability) - Kafka's built-in failure handling and delivery semantics - [Principles of Chaos Engineering](https://principlesofchaos.org/) - Foundational principles from Netflix's chaos engineering practice - Rosenthal, C., & Hochstein, L. (2020). *Chaos Engineering: System Resiliency in Practice*. O'Reilly Media Source: https://www.conduktor.io/glossary/chaos-engineering-for-kafka --- # Chaos Engineering for Streaming Systems Chaos engineering is the discipline of experimenting on a distributed system to build confidence in its ability to withstand turbulent conditions in production. When applied to streaming systems like Apache Kafka, Apache Flink, or Apache Pulsar, chaos engineering becomes essential for ensuring that real-time data pipelines remain resilient under failure conditions. This article covers how chaos engineering principles apply to streaming architectures, the unique challenges these systems present, and practical approaches for testing resilience in production-like environments. ## What is Chaos Engineering? ![Chaos Engineering Process for Streaming Systems](https://www.conduktor.io/assets/images/glossary/chaos-engineering-for-streaming-systems-0.webp) Chaos engineering originated at Netflix with the creation of Chaos Monkey, a tool that randomly terminates instances in production to ensure that systems can survive unexpected failures. The core principle is simple: proactively inject failures into your system to discover weaknesses before they cause outages. The practice follows a scientific method: 1. Define a steady state that represents normal system behavior 2. Hypothesize that this steady state will continue in both control and experimental groups 3. Introduce variables that reflect real-world events (server failures, network latency, resource exhaustion) 4. Observe the differences between control and experimental groups 5. Disprove the hypothesis by finding weaknesses, or gain confidence in system resilience For streaming systems, this methodology is particularly valuable because failures can cascade quickly through distributed pipelines, affecting downstream consumers and business-critical applications. ## Why Streaming Systems Need Chaos Engineering Streaming platforms present unique challenges that make chaos engineering especially important: - **Real-time guarantees**: Unlike batch systems where failures can be retried during the next scheduled run, streaming systems must handle failures while maintaining near-real-time data delivery. A broker failure in Kafka, for example, should not cause data loss or significant processing delays. - **Stateful processing**: Stream processors like Apache Flink maintain state across millions of events. When failures occur, state must be recovered correctly, or the system may produce incorrect results. Testing state recovery under various failure scenarios is critical. - **Distributed coordination**: Streaming systems rely on consensus protocols for coordination. Kafka 4.0+ uses KRaft (the modern Raft-based consensus protocol that replaced ZooKeeper). Network partitions, leader elections, and split-brain scenarios can cause subtle bugs that only appear under specific failure conditions. KRaft's faster metadata operations (milliseconds vs seconds for leader elections) improve failure recovery, making chaos testing even more critical to validate these improvements. For Kafka architecture details, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). - **Consumer group dynamics**: Kafka consumer groups automatically rebalance when consumers join or leave. These rebalances can cause temporary processing pauses. Understanding how your system behaves during rebalances under load is essential for capacity planning. ## Common Failure Scenarios in Streaming Systems Effective chaos engineering requires understanding the failure modes specific to streaming architectures. Here are the most impactful scenarios to test: - **Broker failures**: Randomly terminate broker instances to verify that partition replicas take over seamlessly. Monitor metrics like under-replicated partitions and client reconnection times. - **Network partitions**: Introduce latency or completely partition network segments to test how producers and consumers handle timeouts. This reveals issues with configuration values like `request.timeout.ms` and `session.timeout.ms`. - **Consumer lag injection**: Artificially slow down consumers to create lag, then observe how the system recovers. This tests autoscaling policies and lag monitoring alerts. - **Resource exhaustion**: Limit CPU, memory, or disk I/O on brokers or stream processors to simulate resource contention. This helps validate performance under degraded conditions. - **State corruption**: For stateful stream processors, intentionally corrupt state stores to verify that recovery mechanisms work correctly from checkpoints or snapshots. ## Chaos Engineering Practices for Streaming Implementing chaos engineering for streaming systems requires a structured approach: ### Start Small and Build Confidence Begin experiments in non-production environments that closely mirror production configurations. Use realistic data volumes and traffic patterns. As confidence grows, carefully introduce experiments in production during low-traffic periods with strong monitoring in place. ### Define Clear Success Metrics Before running experiments, establish measurable criteria for success: - End-to-end latency remains below SLA thresholds - Zero data loss (verify message counts match) - Consumer lag recovers within acceptable time windows - No cascading failures to downstream systems ### Use Gradual Rollout Apply the "blast radius" principle by limiting experiments to specific partitions, topics, or consumer groups. For example, terminate one broker in a three-broker cluster rather than all brokers simultaneously. Gradually increase the scope as you validate resilience. ### Automate and Repeat Manual chaos experiments are valuable for learning, but automated, continuous chaos engineering provides ongoing validation. Tools like Chaos Mesh, Litmus, or custom scripts can inject failures on a schedule, ensuring that resilience is maintained as the system evolves. ## Real-World Example: Broker Failure During Peak Load Consider a streaming pipeline processing financial transactions. The pipeline runs on a Kafka cluster with three brokers, each hosting partition leaders for different topics. The team wants to verify that losing one broker won't cause transaction loss or unacceptable latency. **Experiment setup**: 1. Define steady state: 99th percentile end-to-end latency < 500ms, zero data loss 2. Inject failure: Terminate broker-2 using `kill -9` during peak trading hours 3. Observe behavior: Monitor partition leadership changes, producer retry behavior, consumer lag, and end-to-end latency **Results**: The experiment reveals that while Kafka successfully fails over to replica leaders, producer retries cause a 2-second spike in latency because the default `retries` and `retry.backoff.ms` settings are too conservative. The team adjusts these configurations and re-runs the experiment, confirming that latency now stays below 600ms even during broker failures. ## Observability and Tooling for Chaos Experiments Successful chaos engineering depends on comprehensive observability. Without visibility into system behavior, it's impossible to determine whether experiments succeed or reveal problems. Key observability requirements include: - **Metrics collection**: Track broker health, partition metrics, consumer lag, throughput, and latency at high resolution - **Distributed tracing**: Trace individual messages through the pipeline to identify bottlenecks during failures - **Log aggregation**: Centralize logs from all components to correlate failure injection with system responses Platforms like Conduktor provide centralized visibility into Kafka clusters, making it easier to monitor chaos experiments. Features like real-time consumer lag tracking, cluster health dashboards, and topic inspection help teams quickly identify issues during failure injection. [Monitor your chaos experiments](https://docs.conduktor.io/guide/monitor-brokers-apps/index) and use [Conduktor's Insights](https://docs.conduktor.io/guide/insights/index) to track the impact. This observability is essential for understanding the impact of chaos experiments and validating that recovery mechanisms work as expected. ## Best Practices and Considerations When implementing chaos engineering for streaming systems, keep these principles in mind: - **Document runbooks**: Create clear runbooks for each experiment, including rollback procedures. This ensures that anyone on the team can safely run or abort experiments. - **Coordinate with stakeholders**: Inform downstream teams about planned chaos experiments, especially in production. Unexpected behavior in your streaming pipeline may affect their services. - **Test recovery, not just failure**: It's not enough to verify that the system survives a failure. Test how quickly it recovers and whether it returns to the defined steady state. - **Combine with game days**: Run coordinated exercises where teams respond to chaos experiments as if they were real incidents. This builds muscle memory for incident response. - **Don't neglect configuration**: Many streaming failures result from misconfiguration rather than code bugs. Test configuration changes (like replication factors or retention policies) under failure conditions. ## Modern Chaos Engineering Tools (2025) The chaos engineering tooling landscape has matured significantly: - **Conduktor Gateway**: Kafka proxy that enables [controlled chaos testing](https://docs.conduktor.io/guide/use-cases/chaos-testing) without modifying applications. Inject failures, simulate latency, and test resilience by intercepting and manipulating Kafka traffic at the protocol level. Combines chaos engineering with observability, providing real-time visibility into experiment impact on cluster health and consumer behavior. - **Chaos Mesh**: Cloud-native chaos engineering platform for Kubernetes. Supports network chaos, pod failures, I/O chaos, and time chaos. Excellent for testing containerized Kafka and Flink deployments. - **Litmus**: CNCF chaos engineering framework with extensive fault injection scenarios. Provides reusable chaos experiments and integrates with CI/CD pipelines for continuous chaos testing. - **AWS Fault Injection Simulator (FIS)**: Managed service for running chaos experiments on AWS infrastructure. Supports EC2, ECS, EKS, and RDS with built-in experiment templates. - **Azure Chaos Studio**: Microsoft's chaos engineering service for Azure resources. Test resilience of AKS clusters, VMs, and managed services with controlled fault injection. - **OpenTelemetry Integration**: Modern chaos tools integrate with OpenTelemetry for distributed tracing during experiments, making it easier to understand failure propagation across microservices. These tools complement custom scripts and provide production-grade chaos experimentation capabilities with safety controls and rollback mechanisms. ## Summary Chaos engineering is essential for building resilient streaming systems. By proactively injecting failures into platforms like Kafka and Flink, teams can discover weaknesses before they cause production outages. Streaming systems are particularly sensitive to failures due to real-time requirements, stateful processing, and distributed coordination. Start with well-defined experiments in controlled environments, establish clear success metrics, and gradually expand the scope as confidence grows. Combine chaos engineering with strong observability and monitoring to understand system behavior under failure conditions. ## Related Concepts - [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications) - Comprehensive testing approaches including chaos testing - [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters) - Planning for and recovering from failures - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) - Observability during chaos experiments ## Sources and References 1. Rosenthal, C., & Hochstein, L. (2017). *Chaos Engineering: Building Confidence in System Behavior through Experiments*. O'Reilly Media. 2. Netflix Technology Blog. (2011). "The Netflix Simian Army." Available at: https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116 3. Apache Kafka Documentation. "Kafka Reliability Guarantees." Available at: https://kafka.apache.org/documentation/#design_reliability 4. Kreps, J., Narkhede, N., & Rao, J. (2011). "Kafka: A Distributed Messaging System for Log Processing." Proceedings of the NetDB Workshop. 5. Garg, N. (2022). "Chaos Engineering for Distributed Systems." Uber Engineering Blog. Available at: https://eng.uber.com/chaos-engineering/ Source: https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems --- # CI/CD Best Practices for Streaming Applications Continuous Integration and Continuous Deployment (CI/CD) practices have become standard in modern software development, but streaming applications present unique challenges that traditional CI/CD approaches don't fully address. Unlike stateless request-response services, streaming applications process data continuously, maintain state, and must evolve schemas without breaking downstream consumers. This article covers best practices for CI/CD pipelines designed for streaming applications built on platforms like Apache Kafka, Apache Flink, Kafka Streams, and similar event-driven architectures. For foundational knowledge about these platforms, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka), [What is Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing), and [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). ![CI/CD Pipeline for Streaming Applications](https://www.conduktor.io/assets/images/glossary/cicd-best-practices-for-streaming-applications-0.webp) ## Understanding the Unique Challenges Streaming applications differ from traditional services in several critical ways that impact CI/CD strategy. First, they are **stateful**. A Kafka Streams application or Flink job might maintain gigabytes of state that must be preserved across deployments. Losing this state could mean reprocessing millions of events or losing aggregated data entirely. Second, they operate **continuously**. Traditional web services can experience brief downtime during deployments, but streaming applications often require zero-downtime deployments to avoid data loss or processing gaps. Third, they involve **schema evolution**. As data models change, producers and consumers must remain compatible. A breaking schema change can cascade through an entire streaming ecosystem, affecting dozens of downstream applications. These characteristics require CI/CD practices that go beyond standard deployment automation. ## Testing Strategies for Streaming Applications Effective testing is foundational to any CI/CD pipeline, but streaming applications require a multi-layered testing approach. For comprehensive coverage of testing patterns, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). - **Unit tests** should cover business logic in isolation. For Kafka Streams applications, frameworks like Kafka Streams Test Utils provide a TopologyTestDriver that allows testing stream processing logic without running a full Kafka cluster. Similarly, Flink provides testing harnesses for validating operator behavior. - **Integration tests** verify that components work together correctly. These tests should use embedded Kafka clusters (like the one provided by Testcontainers) to validate actual message production, consumption, and processing. With Kafka 4.0+ running in KRaft mode (ZooKeeper-free), test environments are faster to start and simpler to configure, making integration testing more efficient. For details on KRaft, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). For example: ```java @Test public void testOrderProcessing() { // Start embedded Kafka with KRaft (Kafka 4.0+) kafka.start(); // Produce test events producer.send(new ProducerRecord<>("orders", order)); // Consume and verify output ConsumerRecord result = consumer.poll(); assertEquals("COMPLETED", result.value().getStatus()); } ``` Modern testing frameworks for streaming applications include: - **Testcontainers** (Java, Python, Go) for spinning up real Kafka clusters in Docker - **kafka-python** with pytest fixtures for Python applications - **kafka-streams-test-utils** for testing Kafka Streams topologies - **Flink Testing Harness** for validating Flink operators - **Contract testing** ensures compatibility between producers and consumers. Schema registries play a crucial role here. By validating schemas against a registry during CI builds, teams can catch breaking changes before they reach production. Modern tools like Confluent Schema Registry's Maven/Gradle plugins, or AWS Glue Schema Registry's validation APIs, can automate schema validation and compatibility checks as part of the CI pipeline. Additionally, tools like Schemathesis can generate contract tests automatically from OpenAPI/AsyncAPI specifications, preventing incompatible schema changes from being deployed. - **End-to-end tests** validate entire data pipelines in staging environments. These tests should include realistic data volumes and processing patterns to catch performance issues and edge cases. For validating resilience under failure conditions, see [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems). ## Managing State and Schema Evolution State management and schema evolution are perhaps the most critical aspects of CI/CD for streaming applications. For **stateful applications**, deployments must preserve processing state. Apache Flink addresses this through savepoints, consistent snapshots of application state taken before deployment. For detailed information, see [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing). A proper CI/CD pipeline should: 1. Trigger a savepoint before stopping the application 2. Deploy the new version 3. Restore from the savepoint when starting the new version Kafka Streams handles this through state stores and changelog topics. For more details, see [State Stores in Kafka Streams](https://www.conduktor.io/glossary/state-stores-in-kafka-streams). When upgrading a Kafka Streams application, ensure that state store formats remain compatible or plan for state rebuilding. **Schema evolution** requires careful governance. For comprehensive guidance, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). Adopt these practices: - Use a schema registry (Confluent Schema Registry, AWS Glue Schema Registry) as the source of truth - Enforce compatibility rules (backward, forward, or full compatibility) - Include schema validation in CI pipelines, reject builds that introduce incompatible schemas - Version schemas alongside application code - Test both old and new schema versions in integration tests Modern data governance platforms like Atlan, Collibra, or open-source tools like DataHub (2025) provide schema management capabilities that integrate with CI/CD workflows. These platforms automatically validate schema changes, track schema lineage, and ensure compatibility across environments. Additionally, data quality tools like Soda Core, Great Expectations, or Monte Carlo can validate data contracts as part of CI pipelines. For detailed guidance on implementing data quality testing, see [Great Expectations Data Testing Framework](https://www.conduktor.io/glossary/great-expectations-data-testing-framework) and [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework). ## Deployment Patterns for Zero-Downtime Achieving zero-downtime deployments for streaming applications requires thoughtful deployment strategies. - **Blue-green deployments** work well for stateless streaming applications. Deploy the new version (green) alongside the existing version (blue), validate its behavior, then switch traffic. For Kafka consumers, this means starting a new consumer group with the new application version, monitoring its lag and error rates, then stopping the old consumer group once confidence is established. For details on consumer groups, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). - **Canary deployments** are ideal for gradually rolling out changes. Deploy the new version to a small subset of instances or partitions first. For example, if your Kafka Streams application processes 100 partitions, initially deploy the new version to instances handling just 10 partitions. Monitor metrics closely, and if no issues arise, gradually expand to more instances. - **Rolling updates with savepoints** (for Flink) provide zero-downtime deployments for stateful applications: 1. Trigger a savepoint on the running job 2. Cancel the job gracefully 3. Deploy new version to Kubernetes/YARN 4. Submit new job, restoring from savepoint This approach preserves exactly-once processing guarantees while updating application logic. For Kafka-based applications running in Kubernetes, use rolling updates with proper health checks and readiness probes to ensure each instance is fully initialized before routing traffic. **Progressive delivery** tools like Flagger or Argo Rollouts can automate canary deployments with automatic rollback based on metrics. These tools integrate with Prometheus to monitor success rates, latency, and custom streaming metrics (like consumer lag), automatically rolling back deployments if metrics degrade. ## Infrastructure as Code and Environment Parity Streaming applications often involve complex infrastructure, Kafka clusters, schema registries, stream processing frameworks, and supporting databases. Managing this infrastructure manually across development, staging, and production environments leads to configuration drift and deployment failures. **Infrastructure as Code** (IaC) ensures consistency. For comprehensive guidance on Kafka deployments, see [Infrastructure as Code for Kafka Deployments](https://www.conduktor.io/glossary/infrastructure-as-code-for-kafka-deployments) and [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes). Use tools like: - **Terraform** for provisioning Kafka clusters, schema registries, and cloud resources - **Kubernetes Helm charts** for deploying Flink jobs or Kafka Streams applications - **GitOps workflows** (ArgoCD, FluxCD) for declarative deployments Define Kafka topics, partitions, replication factors, and configurations in code. This allows CI/CD pipelines to automatically create or update topics as part of deployment processes. Maintain **environment parity** between development and production. While production may run larger Kafka clusters, the topic configurations, schema registry settings, and application configurations should be identical. Use environment-specific variable files while keeping core infrastructure definitions consistent. Store all infrastructure code in version control alongside application code. When an application requires a new Kafka topic or schema, the PR should include both the application changes and the infrastructure updates. ## CI/CD Pipeline Implementation Modern CI/CD platforms provide excellent support for streaming application workflows. Here are patterns for common platforms: **GitHub Actions** example for Kafka Streams: ```yaml name: Kafka Streams CI/CD on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up JDK 21 uses: actions/setup-java@v4 with: java-version: '21' - name: Run tests with Testcontainers run: ./gradlew test - name: Validate schemas run: ./gradlew schemaRegistryCompatibilityCheck deploy: needs: test if: github.ref == 'refs/heads/main' steps: - name: Build Docker image run: docker build -t streaming-app:${{ github.sha }} . - name: Deploy to Kubernetes run: kubectl apply -f k8s/ ``` - **GitLab CI** offers similar capabilities with built-in container registry and Kubernetes integration. **Jenkins** remains popular for complex streaming workflows with its extensive plugin ecosystem. - **Containerization best practices** for streaming applications: Use **multi-stage Docker builds** to minimize image size: ```dockerfile FROM gradle:8-jdk21 AS build COPY . /app WORKDIR /app RUN gradle build FROM eclipse-temurin:21-jre-alpine COPY --from=build /app/build/libs/app.jar /app.jar ENTRYPOINT ["java", "-jar", "/app.jar"] ``` For production deployments, consider **distroless images** or **Alpine-based images** to reduce attack surface and image size. Streaming applications often run 24/7, so security and efficiency are paramount. ## Monitoring and Validation in Production Even with thorough testing, production monitoring is essential for validating deployments and catching issues early. **Key metrics to monitor** include: - Consumer lag, indicates processing performance and potential bottlenecks (see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring)) - Error rates and dead letter queue volumes (see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling)) - Processing throughput (messages/second) - State store sizes (for stateful applications) - Schema validation failures - End-to-end latency Set up **automated validation** as part of deployments. After deploying a new version: 1. Monitor consumer lag, it should decrease or remain stable 2. Check error rates, spikes indicate bugs or incompatibilities 3. Validate data quality, sample output messages to ensure correct processing 4. Compare metrics against baseline, significant deviations warrant investigation **Alerting** should be deployment-aware. Configure alerts to be more sensitive immediately after deployments, when regressions are most likely. Modern observability stacks for streaming applications typically include: - **Prometheus + Grafana** for metrics collection and visualization - **Kafka Lag Exporter** for dedicated consumer lag monitoring - **OpenTelemetry** for distributed tracing across streaming pipelines - **Datadog, New Relic, or Confluent Cloud** for managed observability - **Burrow** (LinkedIn's open-source tool) for consumer lag monitoring These tools provide centralized observability for Kafka, letting teams track consumer lag, throughput, and schema usage across all environments from a single interface. That visibility matters most during and after deployments. Implement **rollback procedures**. Despite careful testing, production issues can occur. Have automated rollback mechanisms that can quickly revert to the previous version if critical metrics exceed thresholds. ## Summary CI/CD for streaming applications requires adapting traditional DevOps practices to accommodate continuous processing, statefulness, and schema evolution. Key best practices include: - **Implement multi-layered testing**, unit, integration, contract, and end-to-end tests specific to streaming patterns - **Manage state carefully**, use savepoints (Flink) or state stores (Kafka Streams) to preserve processing state across deployments - **Enforce schema compatibility**, integrate schema validation into CI pipelines and use schema registries - **Choose appropriate deployment patterns**, blue-green, canary, or rolling updates with savepoints depending on application requirements - **Use Infrastructure as Code**, maintain environment parity and version infrastructure alongside application code - **Monitor actively**, track streaming-specific metrics like consumer lag and validate deployments in production Following these practices helps teams reach the same deployment velocity and reliability for streaming applications that they expect from traditional services, while respecting the unique requirements of event-driven architectures. ## Related Concepts - [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications) - Comprehensive testing approaches for streaming pipelines - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Essential metrics for CI/CD validation - [Infrastructure as Code for Kafka Deployments](https://www.conduktor.io/glossary/infrastructure-as-code-for-kafka-deployments) - Terraform and automation for streaming platforms ## Sources and References 1. Fowler, Martin. "Continuous Integration." martinfowler.com, May 2006. https://martinfowler.com/articles/continuousIntegration.html 2. Confluent. "Kafka Streams Testing." Confluent Documentation, 2024. https://docs.confluent.io/platform/current/streams/developer-guide/test-streams.html 3. Apache Flink. "Savepoints." Apache Flink Documentation, 2024. https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/state/savepoints/ 4. Stopford, Ben. "Testing Event-Driven Systems." Confluent Blog, 2019. https://www.confluent.io/blog/testing-event-driven-systems/ 5. Kleppmann, Martin. "Designing Data-Intensive Applications." O'Reilly Media, 2017. Chapter 11: Stream Processing. Source: https://www.conduktor.io/glossary/cicd-best-practices-for-streaming-applications --- # Clickstream Analytics with Kafka Every click, scroll, and interaction on a website or mobile app generates useful data. Understanding this user behavior in real-time can drive personalization, detect fraud, and improve customer experiences. Clickstream analytics with Apache Kafka lets organizations capture, process, and act on this data at scale. ![Clickstream Analytics Architecture with Kafka](https://www.conduktor.io/assets/images/glossary/clickstream-analytics-with-kafka-0.webp) ## What is Clickstream Analytics? Clickstream analytics involves tracking and analyzing the sequence of user interactions with digital platforms. Each event, page views, button clicks, video plays, cart additions, creates a data point that reveals user intent and behavior patterns. Traditional analytics systems process this data in batches, often with delays of hours or days. But modern applications need immediate insights. A user browsing products expects personalized recommendations now, not tomorrow. Fraud detection systems must identify suspicious patterns in seconds, not after the damage is done. This shift from batch to real-time processing is where streaming platforms become essential. ## The Challenge of Real-Time Clickstream Processing Clickstream data presents several challenges that make traditional databases and batch systems inadequate. - **Volume and velocity** are the first obstacles. Popular websites generate millions of events per minute. A single user session might produce dozens of events within seconds. Any system handling clickstream data must ingest and process this continuous flood without dropping events or creating bottlenecks. - **Event ordering and timing** matter significantly. Understanding user journeys requires preserving the sequence of actions. Did the user add items to cart before or after viewing the pricing page? These temporal relationships are critical for funnel analysis and session reconstruction. - **Multiple consumers** need access to the same data. The marketing team wants to track campaign effectiveness. Product teams analyze feature usage. The data science team builds recommendation models. Each group needs the raw clickstream, often with different processing requirements and latencies. Traditional message queues struggle with these requirements. They weren't designed for persistent, replayable event streams that multiple systems can consume independently. ## Kafka as the Backbone for Clickstream Data Apache Kafka's architecture solves these clickstream challenges through several key capabilities. Kafka treats events as an immutable, append-only log (a data structure where new entries are only added to the end, never modified or deleted) distributed across multiple servers. When a clickstream event arrives, it's written to a topic partition based on a key, typically the user ID or session ID. This ensures all events for a given user flow to the same partition, maintaining order. The distributed log architecture provides durability and scalability. Events are replicated across brokers, protecting against hardware failures. Adding more partitions and brokers scales the system horizontally as traffic grows. Kafka's consumer model (using consumer groups where each application can independently read events) allows multiple applications to read the same clickstream data independently. Each consumer tracks its own position in the log. The analytics team can process events in real-time while the data lake loader batches them for long-term storage, all from the same topics. For detailed coverage of consumer coordination and offset management, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). - **Retention policies** let organizations balance storage costs with replay requirements. Clickstream topics might retain data for 7 days, allowing systems to recover from failures or reprocess events when bugs are fixed. - **Modern Kafka architecture** (Kafka 3.3+ and Kafka 4.0+) has moved to KRaft mode, eliminating the ZooKeeper dependency that was previously required for cluster coordination. This simplifies operations, reduces latency, and improves scalability for high-throughput clickstream workloads. For production clickstream systems, KRaft mode is now the recommended deployment model. For deeper understanding of Kafka's core architecture, see [Kafka Topics, Partitions, Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). ## Building a Clickstream Pipeline with Kafka A typical clickstream architecture with Kafka includes several layers, each handling specific concerns. - **Data capture** happens through lightweight JavaScript libraries or mobile SDKs that track user interactions. These collectors batch events and send them via HTTP to ingestion services. The ingestion layer validates events and writes them to Kafka topics. - **Topic design** impacts performance and maintainability. Common patterns include: - Single `clickstream-events` topic with event type as a field - Separate topics per event type: `page-views`, `button-clicks`, `purchases` - Partitioning by user ID to maintain ordering per user session For comprehensive guidance on topic design decisions, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) and [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). **Stream processing** transforms raw events into analytics-ready data. This might include: - Sessionization: grouping events into user sessions based on time windows - Enrichment: joining clickstream events with user profile data - Aggregation: computing metrics like pages per session or conversion rates **Storage and serving** typically involves multiple systems. Hot path analytics go to databases like Elasticsearch or ClickHouse for real-time dashboards. Cold path data flows to data lakes (S3, HDFS) for historical analysis and ML training. Modern lakehouse formats like Apache Iceberg or Delta Lake provide ACID transactions and time travel capabilities for clickstream data lakes. Kafka Connect simplifies this data movement with pre-built connectors for sinks like Elasticsearch, S3, and databases. For details on building data integration pipelines, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines). For data lake architecture patterns, see [Building a Modern Data Lake on Cloud Storage](https://www.conduktor.io/glossary/building-a-modern-data-lake-on-cloud-storage). Here's a production-ready topic configuration example (Kafka 3.x/4.x): ``` Topic: clickstream-events Partitions: 12 Partition Key: user_id Replication Factor: 3 Retention: 7 days (168 hours) Min In-Sync Replicas: 2 Compression Type: lz4 Max Message Bytes: 1MB Cleanup Policy: delete ``` This configuration ensures: - Events for each user maintain order (via partition key) - High availability and fault tolerance (replication factor 3, min ISR 2) - Efficient storage and network usage (lz4 compression) - Balanced load across consumer instances - Protection against data loss with at least 2 in-sync replicas ## Stream Processing for Clickstream Analytics Raw clickstream events require processing to become useful insights. Stream processing frameworks like Kafka Streams (3.x+), Apache Flink (1.18+), and ksqlDB excel at this transformation. The choice depends on complexity: Kafka Streams for moderate stateful processing, Flink for complex event-time operations, and ksqlDB for SQL-based transformations. - **Sessionization** is a fundamental operation. It groups events into sessions based on inactivity gaps. If a user stops interacting for 30 minutes, the current session closes and a new one begins with their next action. This allows calculation of session duration, pages per session, and bounce rates. - **Funnel analysis** tracks user progression through defined steps, for example, product view → add to cart → checkout → purchase. Stream processing can compute conversion rates at each step in real-time, alerting teams when drop-off rates spike. - **Personalization engines** consume clickstream data to update user profiles and recommendation models. When a user browses winter coats, the system immediately adjusts product suggestions for subsequent page loads. For comprehensive coverage of streaming-based personalization, see [Building Recommendation Systems with Streaming Data](https://www.conduktor.io/glossary/building-recommendation-systems-with-streaming-data). Here's a Kafka Streams example for sessionization (Kafka Streams 3.x+): ```java import org.apache.kafka.streams.StreamsBuilder; import org.apache.kafka.streams.kstream.*; import java.time.Duration; // Define the stream from clickstream-events topic KStream clicks = builder.stream("clickstream-events"); // Create session windows with 30-minute inactivity gap KTable, Long> sessionCounts = clicks .groupByKey() .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(30))) .count(Materialized.as("session-counts-store")); // Write session metrics to output topic sessionCounts .toStream() .map((windowedKey, count) -> { String userId = windowedKey.key(); long sessionStart = windowedKey.window().start(); long sessionEnd = windowedKey.window().end(); return KeyValue.pair(userId, new SessionMetrics(userId, sessionStart, sessionEnd, count)); }) .to("session-metrics"); ``` This implementation: - Groups events by user ID (the key) - Creates session windows with 30-minute inactivity gaps - Counts events per session and materializes state for queries - Emits session metrics with timing information to a downstream topic For exactly-once processing guarantees in clickstream pipelines (preventing duplicate event counting), configure Kafka Streams with `processing.guarantee=exactly_once_v2`. For details, see [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka). For deeper coverage of Kafka Streams patterns, see [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). **Alternative: ksqlDB for SQL-based processing** For teams preferring SQL over Java/Scala code, ksqlDB provides a SQL interface for stream processing: ```sql -- Create stream from clickstream topic CREATE STREAM clickstream_events ( user_id VARCHAR KEY, event_type VARCHAR, page_url VARCHAR, event_timestamp BIGINT ) WITH ( KAFKA_TOPIC='clickstream-events', VALUE_FORMAT='JSON' ); -- Compute session counts with 30-minute windows CREATE TABLE session_counts AS SELECT user_id, COUNT(*) as event_count, WINDOWSTART as session_start, WINDOWEND as session_end FROM clickstream_events WINDOW SESSION (30 MINUTES) GROUP BY user_id EMIT CHANGES; ``` For comprehensive ksqlDB coverage, see [ksqlDB for Real-Time Data Processing](https://www.conduktor.io/glossary/ksqldb-for-real-time-data-processing). ## Data Quality and Governance in Clickstream Pipelines Production clickstream systems must handle schema evolution, data quality issues, and operational visibility. - **Schema management** becomes critical as event formats evolve. Adding new fields to track additional user interactions shouldn't break downstream consumers. Schema registries enforce compatibility rules and version schemas as they change. - **Data quality** issues are inevitable. Client-side code might send malformed events. Network issues can cause duplicates. Processing bugs might corrupt data. Validation at ingestion time, dead letter queues for invalid events, and monitoring for anomalies help maintain quality. For comprehensive coverage of handling invalid events, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). For data quality dimensions and testing approaches, see [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). - **Operational visibility** requires monitoring throughout the pipeline. Are events flowing at expected rates? Are consumers keeping up with producers? Are there spikes in error rates or latency? Modern tooling for clickstream pipeline observability (2025) includes: - **Conduktor Platform**: Comprehensive Kafka management ([Topics](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), [Monitoring](https://docs.conduktor.io/guide/monitor-brokers-apps/index)), schema registry UI, data quality monitoring, and access control - **Kafka Lag Exporter**: Prometheus-based consumer lag monitoring with alerting capabilities - **Kafka UI**: Open-source web interface for cluster management and topic inspection - **Distributed tracing**: OpenTelemetry integration for end-to-end event flow visibility For comprehensive monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) and [Distributed Tracing for Kafka Applications](https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications). This observability layer is particularly valuable for large organizations where multiple teams produce and consume clickstream data. ## Real-World Use Cases and Best Practices Major technology companies have proven clickstream analytics with Kafka at massive scale. - **Netflix** uses Kafka to process hundreds of billions of events daily, tracking viewing behavior to power recommendations and detect streaming issues in real-time. - **LinkedIn** built its real-time analytics infrastructure on Kafka, processing member interactions to personalize feeds and suggest connections. - **E-commerce platforms** use clickstream data for dynamic pricing, inventory optimization, and fraud detection. When patterns suggest a bot attack or payment fraud, systems can block transactions within milliseconds. - **Best practices** from these implementations include: - Start with a simple topic design and evolve based on actual access patterns - Partition by user ID to maintain event ordering within user sessions - Monitor consumer lag metrics to ensure consumers keep pace with producers (use Kafka Lag Exporter or Conduktor) - Implement sampling for ultra-high-volume scenarios where 100% processing isn't necessary - Use separate topics for sensitive events requiring stricter access controls (PII, payment data) - Enable compression (lz4 or zstd) to reduce storage and network costs - Configure exactly-once semantics for critical metrics (session counts, conversion rates) - Implement dead letter queues for malformed events to prevent processing failures - Test failure scenarios: What happens when consumers fall behind? When brokers restart? - Consider tiered storage (Kafka 3.6+) for long-term retention without excessive disk costs - Use KRaft mode (Kafka 3.3+/4.0+) for simplified operations and better scalability For event-driven architecture patterns applicable to clickstream systems, see [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture). For e-commerce specific patterns, see [E-commerce Streaming Architecture Patterns](https://www.conduktor.io/glossary/e-commerce-streaming-architecture-patterns). ## Summary Clickstream analytics with Kafka enables real-time understanding of user behavior at scale. Kafka's distributed log architecture handles the volume, velocity, and multiple-consumer requirements that clickstream data demands. By combining Kafka (3.x/4.x with KRaft mode) with stream processing frameworks like Kafka Streams, Apache Flink, or ksqlDB, organizations can transform raw click events into actionable insights, powering personalization, fraud detection, and analytics. Building production-grade clickstream pipelines requires attention to data quality, schema evolution, and operational monitoring. Modern tooling (2025) like Kafka Lag Exporter, Conduktor Platform, and distributed tracing with OpenTelemetry provides comprehensive observability. Exactly-once semantics prevent duplicate processing, while dead letter queues handle invalid events gracefully. The architecture patterns and best practices developed by companies processing billions of events daily provide a proven foundation for implementing clickstream analytics. As digital experiences grow more interactive and users expect immediate personalization, real-time clickstream processing becomes more important. Kafka's modern architecture with KRaft, tiered storage, and enhanced monitoring provides the infrastructure to make this practical at any scale. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/apache-kafka) - Distributed streaming platform for clickstream data ingestion - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) - Integrate clickstream data with downstream systems - [Exactly-Once Semantics](https://www.conduktor.io/exactly-once-semantics) - Guarantee accurate event counting in clickstream pipelines ## Sources and References 1. [Confluent - Clickstream Analysis](https://www.confluent.io/blog/) - Architecture patterns and best practices for Kafka-based clickstream analytics 2. [Netflix Tech Blog - Keystone Real-time Stream Processing Platform](https://netflixtechblog.com/keystone-real-time-stream-processing-platform-a3ee651812a) - Netflix's experience processing billions of events with Kafka 3. [LinkedIn Engineering - Brooklin: Near Real-Time Data Streaming at Scale](https://engineering.linkedin.com/blog/) - LinkedIn's approach to real-time member behavior analytics 4. [Apache Kafka Documentation](https://kafka.apache.org/documentation/) - Official documentation covering core concepts and stream processing 5. [Designing Data-Intensive Applications by Martin Kleppmann](https://dataintensive.net/) - Foundational concepts for stream processing and event-driven architectures Source: https://www.conduktor.io/glossary/clickstream-analytics-with-kafka --- # Kafka Consumer Lag: Definition & Monitoring **Kafka consumer lag** is the number of messages a consumer group still has to read on a partition. It is calculated as the partition's log-end offset minus the group's last committed offset, reported per partition; the group's lag is the sum across the partitions it owns. Lag is the single most useful signal for whether a Kafka consumer is keeping up with its topic, and the first metric most teams alert on. ![Consumer Lag Concept Visualization](https://www.conduktor.io/assets/images/glossary/consumer-lag-monitoring-0.webp) For example, if a partition's latest message has an offset of 10,000 and your consumer has committed offset 9,500, the consumer lag is 500 messages. This number tells you how far behind your consumer is from real-time processing. Lag exists for each partition independently. When monitoring consumer groups that subscribe to topics with multiple partitions, you need to track lag across all assigned partitions to get a complete picture of consumer health. The aggregate lag across all partitions often serves as the primary health indicator for the entire consumer group. ## Why Consumer Lag Matters Consumer lag directly impacts several critical aspects of your streaming architecture: - **Data Freshness**: High lag means your downstream systems are processing stale data. If you're building real-time dashboards or fraud detection systems, excessive lag can render your insights useless or allow fraudulent transactions to slip through. - **SLA Compliance**: Many organizations have strict SLAs around data processing latency. A consumer that falls behind may violate these agreements, potentially triggering contractual penalties or damaging customer trust. - **System Health Indication**: Sudden increases in lag often signal underlying problems, overloaded consumers, network issues, database bottlenecks, or resource constraints. Lag is frequently the first visible symptom of degraded system performance. - **Capacity Planning**: Trending lag metrics over time help you understand when to scale your consumer infrastructure. Consistently growing lag indicates that your current capacity cannot keep pace with the incoming message rate. ## Root Causes of Consumer Lag Understanding what causes lag is essential for effective remediation: - **Processing Bottlenecks**: The most common cause is simply that message processing takes too long. This might be due to expensive computations, slow database queries, external API calls, or inefficient code paths. - **Insufficient Consumer Resources**: If consumers don't have enough CPU, memory, or network bandwidth, they cannot process messages fast enough. This often manifests as high CPU utilization or memory pressure on consumer instances. - **Rebalancing Events**: When consumers join or leave a group, Kafka triggers a rebalance that temporarily halts processing. Frequent rebalances, caused by unhealthy consumers, network issues, or misconfigured timeouts, create periodic lag spikes. - **Backpressure from Downstream Systems**: Consumers often write to databases, send HTTP requests, or publish to other systems. If these downstream dependencies slow down or become unavailable, consumers get blocked waiting for responses. - **Message Skew**: Uneven distribution of messages across partitions or "hot" partitions receiving disproportionate traffic can cause some consumers to fall behind while others remain idle. ## Monitoring Consumer Lag Effective lag monitoring requires multiple approaches and tools: - **Kafka Metrics**: Kafka exposes lag metrics through JMX (Java Management Extensions). The `kafka.consumer:type=consumer-fetch-manager-metrics,client-id=` MBean provides `records-lag-max` and `records-lag` metrics for each partition. - **Consumer Group Commands**: The Kafka CLI includes the `kafka-consumer-groups.sh` tool for inspecting consumer group status: ```bash kafka-consumer-groups --bootstrap-server localhost:9092 \ --describe --group my-consumer-group ``` This command shows current offset, log-end offset, and lag for each partition, providing a real-time snapshot of consumer health. **Dedicated Lag Monitoring Tools**: - **Kafka Lag Exporter**: Modern Prometheus-based exporter that provides granular lag metrics with minimal overhead. Supports multiple clusters and integrates seamlessly with Grafana dashboards. - **Burrow**: LinkedIn's HTTP API-based lag monitor with configurable evaluation rules and multi-cluster support. - **Kafdrop / Kowl (Redpanda Console)**: Web UIs that provide real-time lag visibility and consumer group management. **Observability Platforms**: Modern monitoring platforms like Prometheus, Grafana, Datadog, and New Relic offer Kafka integrations that collect lag metrics automatically. These platforms enable visualization, alerting, and correlation with other system metrics. When implementing lag monitoring, track both absolute lag (number of messages behind) and lag growth rate. A static lag of 1,000 messages might be acceptable if it remains stable, but lag growing by 100 messages per second indicates a serious problem requiring immediate attention. ## Setting Up Effective Alerts Alert configuration requires balancing sensitivity with noise reduction: - **Threshold-Based Alerts**: Set absolute lag thresholds based on your SLAs and typical message rates. For example, alert when lag exceeds 10,000 messages or when any partition lags by more than 5 minutes of wall-clock time. - **Trend-Based Alerts**: Monitor lag growth over time windows. Alert when lag increases by more than 20% over a 5-minute period, indicating consumers are falling further behind rather than catching up. - **Anomaly Detection**: Use statistical approaches to detect unusual lag patterns. If your typical lag is 500 messages ±200, an anomaly detector can alert when lag suddenly jumps to 5,000, even if that's still below your absolute threshold. - **Partition-Level vs. Group-Level**: Decide whether to alert on individual partition lag or aggregate consumer group lag. Partition-level alerts catch localized issues, while group-level alerts focus on overall health. Avoid alert fatigue by tuning thresholds to your specific workloads and using alert suppression during planned maintenance windows or deployments. ## Remediation Strategies When lag occurs, systematic remediation is essential: - **Scale Consumer Instances**: The most direct solution is horizontal scaling. Add more consumer instances to your consumer group, and Kafka will automatically rebalance partitions across the expanded pool. This works when lag is caused by insufficient processing capacity. - **Optimize Processing Logic**: Profile your consumer code to identify bottlenecks. Optimize database queries, cache frequently accessed data, batch external API calls, or move expensive operations to asynchronous processing. - **Adjust Consumer Configuration**: Tune parameters like `fetch.min.bytes`, `fetch.max.wait.ms`, and `max.poll.records` to optimize throughput. Increasing the batch size can improve efficiency if you're processing messages in small batches. - **Partition Rebalancing**: If message distribution is skewed, consider adding partitions to the topic and implementing a better partitioning strategy. Ensure your partition key distributes messages evenly across partitions. - **Temporary Parallel Processing**: For one-time lag remediation, temporarily spin up additional consumer groups reading from the same topic to catch up faster. This requires ensuring your processing is idempotent and can handle duplicate operations. ## Consumer Group Dynamics and Lag Consumer lag is intrinsically tied to consumer group behavior: Each consumer in a group is assigned specific partitions through Kafka's partition assignment strategy. Lag exists per partition, so understanding which consumer handles which partition helps localize performance issues. When consumers are unevenly loaded, perhaps one handles more partitions or more expensive messages, lag patterns become uneven across the group. Monitoring per-partition lag reveals these imbalances. Rebalancing temporarily increases lag because all consumers must stop processing, coordinate the new partition assignment, and resume. Minimizing rebalance frequency through stable consumer instances and appropriate timeout configurations helps maintain consistent lag levels. ## Modern Consumer Group Protocol (Kafka 4.0 / KIP-848) Kafka 4.0 introduces a next-generation consumer group protocol (KIP-848) that significantly improves rebalancing and lag management: **Key Improvements**: - **Incremental Rebalancing**: Instead of stopping all consumers during rebalance, only affected partitions are reassigned, reducing lag spikes - **Static Membership Enhancements**: Better support for stable consumer group membership, minimizing unnecessary rebalances - **Faster Coordination**: Consumer group state management is more efficient, reducing rebalance duration from seconds to milliseconds - **Improved Lag Visibility**: Enhanced metrics provide better insight into rebalancing impact on lag **Migration Considerations**: The new protocol is backward compatible but requires consumer upgrades to leverage improvements. Monitor rebalance frequency and duration before and after migration to quantify benefits. ## Advanced Patterns Sophisticated streaming architectures employ lag-aware patterns: - **Lag-Based Autoscaling**: Use lag metrics to trigger automatic scaling of consumer infrastructure. When lag exceeds a threshold for a sustained period, automatically provision additional consumer instances. When lag normalizes, scale back down to optimize costs. Kubernetes-based deployments can use KEDA (Kubernetes Event-Driven Autoscaling) with Kafka lag metrics as triggers. - **Priority Processing**: Implement separate consumer groups with different priorities. A high-priority group might process a subset of critical messages first, ensuring low lag for important data even if the comprehensive consumer group falls behind. - **Time-Based Lag Monitoring**: In addition to message count lag, monitor time-based lag (how old the messages being processed are). For time-sensitive applications, a 5-minute time lag might be more meaningful than 10,000 message lag. - **Lag-Aware Load Shedding**: When lag becomes critical, implement strategies to shed load temporarily, processing only high-priority messages, sampling data, or bypassing expensive enrichment steps to catch up faster. See [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) for related patterns. - **Circuit Breaker Integration**: Connect lag monitoring to circuit breaker patterns. If a downstream dependency is causing backpressure and increasing lag, open the circuit breaker to prevent further degradation while alerting operators. ## Tools and Ecosystem The Kafka ecosystem provides rich tooling for lag management: - **Kafka CLI Tools**: Built-in commands like `kafka-consumer-groups.sh` provide essential lag inspection capabilities for troubleshooting and manual monitoring. - **Burrow**: LinkedIn's Burrow offers comprehensive lag evaluation with configurable rules, HTTP API for integration, and multi-cluster support. It's particularly valuable for large-scale deployments. - **Streaming Management Platforms**: Tools like [Conduktor Console](https://www.conduktor.io/kafka-consumer-lag) provide multi-cluster lag dashboards with per-team ownership, threshold-based alerts on offset or time lag, and ties to access control and audit. For the vendor-aware tutorial on monitoring lag with Conduktor, see [Monitor Kafka Consumer Lag](https://www.conduktor.io/kafka-consumer-lag). - **Kafka Connect**: For Connect-based streaming pipelines, lag monitoring extends to source and sink connectors. Tracking connector lag ensures end-to-end pipeline health. - **Stream Processing Frameworks**: Frameworks like Kafka Streams and Flink expose lag metrics specific to their processing models, including state store lag and processing time lag. ## Related Concepts - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) - Comprehensive monitoring strategies covering broker health, partition replication, and performance metrics beyond consumer lag. - [Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide) - Optimization techniques to reduce consumer lag through improved throughput and processing efficiency. - [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) - Patterns for managing flow control when consumer lag indicates backpressure from downstream systems. ## Sources and References - [Apache Kafka Monitoring Documentation](https://kafka.apache.org/documentation/#monitoring) - Official guide to Kafka metrics and monitoring approaches - [Confluent Monitoring Consumer Lag](https://docs.confluent.io/platform/current/kafka/monitoring.html#consumer-lag-monitoring) - Best practices for tracking and analyzing consumer lag - [LinkedIn Burrow](https://github.com/linkedin/Burrow) - Open-source consumer lag monitoring tool with evaluation rules - [Kafka Consumer Groups CLI](https://kafka.apache.org/documentation/#basic_ops_consumer_group) - Command-line tools for inspecting consumer group status - [Prometheus JMX Exporter for Kafka](https://github.com/prometheus/jmx_exporter) - Exposing Kafka JMX metrics for monitoring platforms Source: https://www.conduktor.io/glossary/consumer-lag-monitoring --- # CQRS and Event Sourcing with Kafka Command Query Responsibility Segregation (CQRS) and Event Sourcing address scalability and auditability by rethinking how data is modeled and stored. While each pattern offers distinct benefits, they work well together, especially when built on Apache Kafka. This article covers how CQRS and Event Sourcing work individually and together, why Kafka is a natural fit, and what you need to implement these patterns successfully. For broader context on event-driven systems, see [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture) and [Event-Driven Microservices Architecture](https://www.conduktor.io/glossary/event-driven-microservices-architecture). ## Understanding CQRS: Separating Reads from Writes Command Query Responsibility Segregation (CQRS) is a pattern that separates the write model (commands) from the read model (queries) in an application. Instead of using a single data model for both operations, CQRS maintains distinct models optimized for their specific purposes. The write side handles commands that change application state. These commands go through validation, business logic, and eventually modify the system. The read side handles queries that retrieve data, often from denormalized views optimized for specific query patterns. This separation provides several advantages. Read and write workloads can scale independently, which is valuable when query traffic far exceeds write traffic. Each model can use different data stores optimized for its access patterns, for example, a normalized database for writes and a document store or search index for reads. CQRS also improves security and performance. You can apply different security policies to commands versus queries, and optimize each side without compromise. ## Event Sourcing: State as a Sequence of Events Event Sourcing takes a fundamentally different approach to data persistence. Instead of storing the current state of an entity, Event Sourcing stores the complete sequence of events that led to that state. The current state is derived by replaying these events. Consider a bank account. Traditional systems store the current balance. Event Sourcing stores every transaction: AccountOpened, MoneyDeposited, MoneyWithdrawn. To determine the current balance, you replay these events from the beginning. This approach offers powerful capabilities. You get a complete audit trail automatically, every change is recorded as an event with full context. You can reconstruct past states by replaying events up to any point in time, enabling temporal queries and debugging. If you discover bugs in your business logic, you can fix the code and replay events to correct the current state. Events are immutable facts about what happened. Once written, they never change. This immutability simplifies concurrency, debugging, and distributed system reasoning. ### Optimizing Event Replay with Snapshots For long-lived aggregates with thousands of events, replaying the entire history on every read becomes impractical. **Snapshots** solve this performance challenge by periodically saving the current state as a checkpoint. Instead of replaying 10,000 account transactions, you load the most recent snapshot (taken at event 9,500) and replay only the 500 events since then. Snapshots are stored alongside events but treated differently, they're performance optimizations, not the source of truth. If a snapshot is corrupted or missing, you can always rebuild it from events. Kafka's **log compaction** feature supports snapshot strategies. By using a separate compacted topic keyed by aggregate ID, you can store the latest snapshot for each entity. When consumers need to rebuild state, they first read the snapshot, then continue with the event stream. For more details on this Kafka feature, see [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained). ```java // Snapshot strategy example public AccountState loadAccount(String accountId) { // Try loading latest snapshot AccountSnapshot snapshot = snapshotStore.getLatest(accountId); long startPosition = snapshot != null ? snapshot.getLastEventPosition() : 0; // Replay events since snapshot List events = eventStore.getEventsSince(accountId, startPosition); AccountState state = snapshot != null ? snapshot.getState() : new AccountState(); return events.stream().reduce(state, (acc, event) -> acc.apply(event), (a, b) -> b); } ``` ## Why Kafka is the Ideal Foundation Apache Kafka's architecture aligns perfectly with CQRS and Event Sourcing requirements. Kafka is a distributed commit log that stores streams of events durably and provides both pub-sub and queue semantics. For a comprehensive introduction to Kafka's core concepts, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). Kafka topics serve as the event store. Events are written to topics and retained based on configurable policies, for days, weeks, or indefinitely. This durable log becomes the source of truth for Event Sourcing. Unlike traditional message queues that delete messages after consumption, Kafka retains events, allowing new consumers to process the entire history. Kafka's partitioning enables horizontal scaling. Events with the same key (like a customer ID) go to the same partition, maintaining order for that entity. Multiple partitions allow parallel processing across many consumers. This is crucial when rebuilding read models from potentially millions of events. Kafka's exactly-once semantics ensure events aren't lost or duplicated, which is essential for financial or transactional systems. As of Kafka 4.0+ (2024-2025), exactly-once processing is more robust with improved transactional guarantees. To achieve exactly-once delivery, configure producers with `enable.idempotence=true` and use transactional IDs for coordinated writes across multiple partitions. For detailed coverage of this critical feature, see [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka). The platform's performance, handling millions of events per second, supports even the most demanding real-time applications. Modern Kafka deployments (4.0+) use KRaft mode, which removes the dependency on Apache ZooKeeper for metadata management. This simplifies operations, reduces infrastructure complexity, and improves cluster startup times and scalability. KRaft mode is the default and recommended configuration for all new Kafka clusters as of 2025. Kafka Connect and Kafka Streams provide integrations with external systems and stream processing. For an introduction to building streaming applications, see [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). Apache Flink also integrates with Kafka for more complex stateful computations and event-time processing. ## Implementation Patterns and Architecture Implementing CQRS and Event Sourcing with Kafka follows a typical pattern. The command side receives commands (HTTP requests, API calls), validates them, and publishes events to Kafka topics representing the business domain. ![cqrs-and-event-sourcing-with-kafka diagram 1](https://www.conduktor.io/assets/images/glossary/cqrs-and-event-sourcing-with-kafka-0.webp) For example, in an e-commerce system, a "PlaceOrder" command might produce an "OrderPlaced" event containing the order details. This event is written to an "orders" topic in Kafka. The read side consists of one or more consumers that subscribe to these event topics and build materialized views. Each view is optimized for specific query patterns. One consumer might build a denormalized view in PostgreSQL for transactional queries. Another might index data in Elasticsearch for full-text search. A third might populate Redis for low-latency lookups. These materialized views are called **projections**, read-optimized representations of data derived from the event stream. Projections transform and aggregate events into formats suitable for queries. Since they're derived from events, you can rebuild or modify projections without affecting the write side or losing data. Using Kafka Streams or ksqlDB, you can build sophisticated projections with joins, aggregations, and windowing. These tools operate directly on Kafka topics, treating them as tables and streams. When debugging or developing these systems, tools like Conduktor help visualize event flows through [topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), inspect message contents, and test consumer behavior. This visibility is valuable when troubleshooting why a read model isn't updating correctly or when understanding the impact of schema changes. ### Practical Example: Order Management Consider an order management system. Commands include CreateOrder, ShipOrder, CancelOrder. Each produces corresponding events: OrderCreated, OrderShipped, OrderCanceled. Here's what an event looks like in practice: ```json { "eventId": "550e8400-e29b-41d4-a716-446655440000", "eventType": "OrderCreated", "timestamp": "2025-01-15T10:30:00Z", "aggregateId": "order-12345", "version": 1, "data": { "orderId": "order-12345", "customerId": "customer-789", "items": [ {"productId": "prod-001", "quantity": 2, "price": 29.99} ], "totalAmount": 59.98, "status": "CREATED" } } ``` The command handler validates commands and publishes events to Kafka: ```java // Command Handler (Write Side) public class OrderCommandHandler { private final KafkaProducer producer; public void handle(CreateOrderCommand command) { // Validate command validateOrder(command); // Create event OrderCreatedEvent event = new OrderCreatedEvent( UUID.randomUUID(), command.getOrderId(), command.getCustomerId(), command.getItems(), Instant.now() ); // Publish to Kafka with order ID as key for partitioning ProducerRecord record = new ProducerRecord<>("order-events", event.getOrderId(), event); producer.send(record, (metadata, exception) -> { if (exception == null) { log.info("Event published to partition {} at offset {}", metadata.partition(), metadata.offset()); } }); } } ``` The read side consumes events and builds materialized views (projections): ```java // Consumer building read model (Read Side) public class OrderReadModelConsumer { private final OrderRepository repository; @KafkaListener(topics = "order-events", groupId = "order-view-builder") public void consume(ConsumerRecord record) { OrderEvent event = record.value(); switch (event.getType()) { case ORDER_CREATED: OrderCreatedEvent created = (OrderCreatedEvent) event; OrderView view = new OrderView( created.getOrderId(), created.getCustomerId(), created.getTotalAmount(), "CREATED" ); repository.save(view); break; case ORDER_SHIPPED: repository.updateStatus(event.getOrderId(), "SHIPPED"); break; case ORDER_CANCELED: repository.updateStatus(event.getOrderId(), "CANCELED"); break; } } } ``` Multiple read models consume the "order-events" topic, partitioned by order ID: - An operational dashboard queries a PostgreSQL view showing current order statuses - A reporting system aggregates data in a data warehouse for analytics - A notification service triggers emails when orders ship Each consumer maintains its own projection, a denormalized view built from events. If you add a new analytics requirement, you create a new consumer, replay the event history from the beginning, and build the new view without touching the write side. ## Challenges and Trade-offs While powerful, these patterns introduce complexity. Eventual consistency is inherent, there's a delay between writing an event and updating read models. Applications must handle scenarios where a command succeeds but the read model hasn't updated yet. Event schema evolution requires careful planning. Events are immutable, but your understanding of the domain evolves. You need strategies for handling multiple event versions and migrating between schemas. Modern schema management tools help manage this complexity: - **Confluent Schema Registry** or **Apicurio Registry** provide centralized schema storage with version control and compatibility checking - **Apache Avro**, **Protocol Buffers**, or **JSON Schema** define structured event formats with evolution rules (see [Avro vs Protobuf vs JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema) for a comparison) - **Compatibility modes** enforce constraints: - **Backward compatibility**: New consumers can read old events (safe to add optional fields) - **Forward compatibility**: Old consumers can read new events (safe to remove optional fields) - **Full compatibility**: Both forward and backward (safest for most use cases) In practice, version your events explicitly and handle multiple versions in consumers: ```java public void consume(ConsumerRecord record) { OrderEvent event = record.value(); // Handle different event versions switch (event.getVersion()) { case 1: processOrderEventV1(event); break; case 2: processOrderEventV2(event); break; default: log.warn("Unknown event version: {}", event.getVersion()); } } ``` Schema registries integrate with Kafka producers and consumers to automatically validate schemas before publishing and deserialize events correctly based on their schema version. Operational overhead increases. You're managing event stores, multiple read models, and the infrastructure to keep them synchronized. Storage grows since events accumulate over time, though Kafka's retention policies help manage this. Kafka offers two retention strategies: - **Time-based or size-based retention**: Delete old events after a specified time (e.g., 30 days) or when storage reaches a limit - **Log compaction**: For keyed topics, retain only the latest value for each key, automatically removing older versions while preserving the most recent state Log compaction is particularly useful for snapshot topics in Event Sourcing. Instead of storing every snapshot ever created for an entity, compaction keeps only the most recent one. This provides efficient state recovery without unbounded storage growth. Not every system needs these patterns. Simple CRUD applications add unnecessary complexity with CQRS and Event Sourcing. These patterns shine in complex domains with high scalability requirements, critical audit needs, or sophisticated read patterns that differ significantly from write patterns. ## Summary CQRS and Event Sourcing are complementary patterns that excel when built on Apache Kafka's event streaming platform. CQRS separates read and write concerns, enabling independent scaling and optimization. Event Sourcing provides complete audit trails and the ability to reconstruct past states. Kafka's durable, partitioned log is a good fit for implementing these patterns in production. They introduce real complexity and operational overhead, but the benefits for scalable, auditable, event-driven systems are worth it when the use case calls for them. Success requires understanding the trade-offs, investing in proper tooling and monitoring, and applying these patterns where they provide clear value over simpler alternatives. For additional Event Sourcing implementation patterns and best practices, see [Event Sourcing Patterns with Kafka](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka). ## Related Concepts - [Event Sourcing Patterns with Kafka](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka) - Detailed implementation patterns for event sourcing - [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions) - Managing distributed transactions in event-driven systems - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Ensuring data consistency with transactional guarantees ## Sources and References 1. Fowler, Martin. "CQRS." martinfowler.com, 2011. https://martinfowler.com/bliki/CQRS.html 2. Fowler, Martin. "Event Sourcing." martinfowler.com, 2005. https://martinfowler.com/eaaDev/EventSourcing.html 3. Stopford, Ben. "Designing Event-Driven Systems." O'Reilly Media, 2018. https://www.confluent.io/designing-event-driven-systems/ 4. Narkhede, Neha, Gwen Shapira, and Todd Palino. "Kafka: The Definitive Guide." O'Reilly Media, 2017. 5. Boner, Jonas, et al. "Reactive Microsystems: The Evolution of Microservices at Scale." O'Reilly Media, 2017. https://www.oreilly.com/library/view/reactive-microsystems/9781491994368/ Source: https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka --- # Cross-AZ Traffic in Streaming: Managing Costs and Latency Cross-availability-zone (cross-AZ) traffic refers to data transfer between different availability zones within the same cloud region. Availability zones are isolated data centers with independent power, cooling, and networking, designed to provide fault tolerance and high availability. In streaming architectures, cross-AZ traffic occurs whenever data moves between components deployed in different zones. This includes broker-to-broker replication, consumer fetches from brokers in other zones, and producer writes to leaders in different zones. For foundational Kafka concepts including brokers, topics, and partitions, see [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). Unlike traffic within a single availability zone (typically free) or between regions (significantly more expensive), cross-AZ traffic occupies a middle ground that can become a substantial cost factor at scale. Understanding and optimizing this traffic is essential for cost-effective streaming deployments. ![Cross-AZ Traffic Architecture](https://www.conduktor.io/assets/images/glossary/cross-az-traffic-streaming-0.webp) ## Cost and Performance Implications Cloud providers charge for cross-AZ data transfer, with pricing typically ranging from $0.01 to $0.02 per GB transferred. While this may seem modest, streaming platforms processing terabytes daily can accumulate significant costs: - **AWS**: $0.01/GB in each direction ($0.02/GB total for bidirectional) - **Google Cloud Platform**: $0.01/GB - **Azure**: $0.01/GB between zones For a Kafka cluster processing 10 TB/day with a replication factor of 3 and distributed consumers, cross-AZ traffic can easily reach 15-20 TB/day, translating to $150-400/day or $4,500-12,000/month in data transfer costs alone. Beyond costs, cross-AZ traffic introduces latency. While typically modest (1-2ms within a region), this latency compounds in streaming architectures where data passes through multiple stages. For latency-sensitive applications, minimizing cross-AZ hops becomes a performance requirement, not just a cost optimization. ## Cross-AZ Traffic in Kafka Apache Kafka generates cross-AZ traffic through several mechanisms. With Kafka 4.0 (released 2024) and the transition to KRaft consensus replacing ZooKeeper, cross-AZ traffic patterns have evolved with improved metadata distribution and reduced coordination overhead: ### Replication Traffic With a replication factor greater than 1, Kafka replicates partition data across multiple brokers. If these brokers reside in different availability zones, replication generates cross-AZ traffic. For a replication factor of 3, this can triple your ingress data volume. For detailed coverage of replication mechanisms and high availability, see [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability). ### Consumer Fetch Patterns Consumers fetch data from partition leaders. If a consumer in zone A fetches from a leader in zone B, all consumed data traverses zones. With multiple consumer groups, this traffic multiplies rapidly. For in-depth understanding of consumer group mechanics, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). ### Producer Acknowledgments Producers receive acknowledgments from partition leaders. When `acks=all`, producers wait for all in-sync replicas to acknowledge, potentially generating cross-AZ traffic for both writes and acknowledgments. ### Partition Leadership Distribution Kafka distributes partition leadership across brokers. Without rack awareness, leaders may be evenly distributed across zones, maximizing cross-AZ traffic potential. For detailed coverage of partition distribution strategies, see [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). ## Optimization Strategies ### Rack Awareness Configuration Kafka's rack awareness feature allows you to assign brokers to logical racks (availability zones). When enabled, Kafka distributes partition replicas across racks, ensuring high availability while enabling zone-aware optimizations. Configure rack awareness in broker properties: ```properties broker.rack=us-east-1a ``` With rack awareness enabled, Kafka ensures replicas for each partition span multiple racks, preventing data loss if an entire zone fails while providing the foundation for follower fetching optimizations. ### Follower Fetching for Consumers Introduced in Kafka 2.4 and now standard in Kafka 3.x and 4.x, follower fetching allows consumers to fetch data from the nearest replica rather than always fetching from the partition leader. This dramatically reduces cross-AZ traffic when consumers and follower replicas are co-located. Enable follower fetching by setting: ```properties # Consumer configuration client.rack=us-east-1a ``` Combined with broker rack awareness, Kafka routes consumer fetches to replicas in the same availability zone, potentially reducing consumer-related cross-AZ traffic by 60-80%. In Kafka 4.0, follower fetching has been further optimized with improved replica lag detection and faster failover to leader fetches when necessary. ### Zone-Aware Partition Assignment For partition assignment strategies, consider implementing custom assignors that account for zone locality. While Kafka's default assignors focus on balance, zone-aware assignors can co-locate consumers with partition replicas, minimizing cross-AZ fetches. ### Strategic Topic Placement For topics with predictable consumption patterns, consider strategic placement: - **Internal topics**: If consumed only by brokers, ensure leaders are distributed to minimize cross-AZ replication - **Single-zone topics**: For less critical data, place all replicas in a single zone to eliminate cross-AZ traffic entirely (sacrificing multi-zone availability) - **Read-heavy topics**: Ensure replicas exist in all zones where consumers run ### Tiered Storage for Historical Data Kafka 4.0 introduced production-ready Tiered Storage, which offloads older segments to object storage (S3, GCS, Azure Blob) while keeping recent data on local broker disks. This significantly impacts cross-AZ traffic patterns: **Benefits for Cross-AZ Traffic:** - **Reduced replication costs**: Historical data in object storage doesn't require multi-AZ replication across Kafka brokers - **Lower ongoing costs**: Object storage replication is handled by cloud providers, typically at lower rates - **Selective local storage**: Keep only hot data (recent messages) on expensive multi-AZ broker storage **Configuration example:** ```properties # Broker configuration for Tiered Storage (Kafka 4.0+) remote.log.storage.system.enable=true remote.log.manager.task.interval.ms=60000 # Move segments to tiered storage after 1 hour local.retention.ms=3600000 local.retention.bytes=10737418240 ``` Tiered Storage is particularly effective for topics with long retention requirements but infrequent historical data access, reducing cross-AZ traffic by 30-50% for such workloads. ## Monitoring and Measurement Effective optimization requires visibility into cross-AZ traffic patterns: ### Key Metrics - **Network bytes in/out per broker**: Track at the availability zone level - **Replication bytes**: Monitor broker replication traffic - **Consumer fetch bytes**: Track by consumer group and zone - **Partition distribution**: Monitor leader and replica distribution across zones ### Monitoring Tools Cloud providers offer native monitoring for cross-AZ traffic: - **AWS CloudWatch**: NetworkIn/NetworkOut metrics with availability zone dimensions - **GCP Cloud Monitoring**: Network egress metrics by zone - **Azure Monitor**: Network traffic metrics with zone filtering Kafka metrics provide additional insight: - `kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec` - `kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec` - `kafka.server:type=FetcherStats,name=BytesPerSec` **Modern Monitoring Stack (2025):** For comprehensive cross-AZ traffic monitoring, modern deployments typically use: - **Prometheus + Grafana**: Industry-standard observability stack with JMX Exporter for Kafka metrics - **Kafka Lag Exporter**: Tracks consumer lag by partition, enabling zone-specific lag monitoring - **Cloud provider cost dashboards**: AWS Cost Explorer, GCP Cost Management, Azure Cost Analysis with resource tagging - **Kafka UI tools**: Conduktor for comprehensive visual monitoring and management For comprehensive coverage of Kafka monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ### Cost Attribution Implement tagging strategies to attribute cross-AZ costs to specific teams, applications, or topics. This enables data-driven optimization decisions and cost awareness across your organization. For detailed guidance on implementing cost attribution and chargeback models that include network transfer costs, see [Conduktor's Cost Control Insights](https://docs.conduktor.io/guide/insights/cost-control). ## Trade-offs and Best Practices ### High Availability vs Cost Optimization The fundamental trade-off in cross-AZ optimization is between high availability and cost. Multi-zone deployments provide fault tolerance against zone failures but generate cross-AZ traffic. Consider: - **Mission-critical topics**: Maintain replication factor 3 across zones - **Development environments**: Use single-zone deployment or replication factor 2 - **Archival topics**: Lower replication requirements may be acceptable For disaster recovery strategies beyond availability zone failures, see [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters). ### Performance vs Redundancy Follower fetching improves performance and reduces costs but creates dependencies on local replicas. If a zone's replicas fall behind, consumers may experience increased latency or fall back to cross-AZ fetches. For comprehensive performance optimization techniques, see [Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide). ### Zone-Aware Consumer Configuration Deploy consumers with explicit zone awareness: 1. Set `client.rack` to match consumer deployment zone 2. Ensure sufficient partition replicas in consumer zones 3. Monitor consumer lag by zone to detect replica lag issues ### Managed Service Considerations Managed Kafka services handle some optimizations automatically: - **Amazon MSK**: Automatically configures rack awareness across availability zones. As of 2025, MSK supports Kafka 3.x with KRaft mode, reducing metadata-related cross-AZ traffic However, you still control consumer deployment and topic configuration, making zone-awareness essential even with managed services. Tools like Conduktor help monitor and optimize these deployments across any Kafka infrastructure. ## Cost Modeling and Planning Understanding cross-AZ costs is essential for capacity planning and infrastructure budgeting. For comprehensive capacity planning guidance, see [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning). ### Estimating Monthly Costs Calculate potential cross-AZ costs using this formula: ``` Monthly Cross-AZ Cost = ( (Daily Ingress × Replication Factor × Cross-AZ Percentage) + (Daily Ingress × Consumer Groups × Cross-AZ Fetch Percentage) ) × 30 × Cost per GB ``` ### Real-World Scenario Consider a deployment with: - 10 TB/day ingress - Replication factor of 3 - 4 consumer groups - 50% of replicas in different zones - 70% of consumer fetches cross-zone (before optimization) - $0.01/GB cross-AZ cost **Before optimization:** - Replication: 10 TB × 2 (additional replicas) × 0.5 = 10 TB/day cross-AZ - Consumer fetches: 10 TB × 4 × 0.7 = 28 TB/day cross-AZ - Total: 38 TB/day × 30 × $0.01 = $11,400/month **After follower fetching:** - Replication: 10 TB/day (unchanged) - Consumer fetches: 10 TB × 4 × 0.1 = 4 TB/day cross-AZ (reduced to 10%) - Total: 14 TB/day × 30 × $0.01 = $4,200/month - **Monthly savings: $7,200** ### Optimization ROI Even modest optimizations deliver substantial returns. Implementing follower fetching typically requires minimal engineering effort (configuration changes) while delivering 30-60% reductions in cross-AZ traffic costs. ## Governance and Best Practices Managing cross-AZ traffic at scale requires governance policies and enforcement mechanisms. Establishing topic creation policies, deployment standards, and cost accountability ensures sustainable optimization. Governance platforms let you manage Kafka deployments through topic configuration policies, cluster monitoring, and cost visibility. Enforcing zone-aware configurations and tracking traffic patterns helps maintain optimization gains as your streaming infrastructure evolves. ## Key Takeaways - Cross-AZ traffic costs $0.01-0.02/GB and accumulates rapidly in streaming deployments - Kafka generates cross-AZ traffic through replication, consumer fetches, and partition leadership - Rack awareness and follower fetching are essential optimizations, reducing cross-AZ traffic by 60-80% - Kafka 4.0's Tiered Storage reduces cross-AZ costs by 30-50% for topics with long retention requirements - KRaft consensus (default in Kafka 4.0) reduces metadata-related cross-AZ traffic compared to ZooKeeper - Monitor cross-AZ traffic at both cloud and application layers for comprehensive visibility using Prometheus, Kafka Lag Exporter, and cloud provider dashboards - Balance high availability requirements against cost optimization opportunities - Managed services provide some automatic optimizations but still require zone-aware configuration - Cost modeling and ROI analysis justify optimization investments - Governance policies sustain long-term optimization gains By understanding cross-AZ traffic patterns, implementing targeted optimizations, and maintaining visibility through monitoring, you can build cost-effective streaming architectures that deliver high availability without excessive cloud data transfer costs. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Platform architecture impacting cross-AZ traffic patterns - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Planning infrastructure to optimize cross-AZ costs - [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters) - Multi-AZ strategies for high availability ## Sources and References - [Apache Kafka 4.0 Documentation](https://kafka.apache.org/documentation/) - Official documentation including Tiered Storage and KRaft consensus - [Apache Kafka Rack Awareness](https://kafka.apache.org/documentation/#rack) - Official documentation on configuring rack awareness for multi-AZ deployments - [Kafka Follower Fetching (KIP-392)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica) - Kafka Improvement Proposal enabling zone-aware consumer fetches - [Kafka Tiered Storage (KIP-405)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage) - Tiered Storage architecture and configuration - [AWS Data Transfer Pricing](https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer) - Understanding cross-AZ and inter-region transfer costs - [Amazon MSK Best Practices](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html) - Multi-AZ deployment and cost optimization strategies - [Conduktor Platform](https://www.conduktor.io/) - Kafka management and monitoring for multi-AZ deployments Source: https://www.conduktor.io/glossary/cross-az-traffic-streaming --- # Cross-Organization Data Sharing Patterns Organizations increasingly need to share data beyond their boundaries. Supply chain partners exchange inventory updates, financial institutions collaborate on fraud detection, and healthcare networks share patient records across hospital systems. Each scenario requires careful consideration of technical architecture, security, and governance. This article explores common patterns for cross-organization data sharing, the role of data streaming platforms, and practical considerations for implementation. ## Common Data Sharing Patterns Organizations use several architectural patterns to share data, each with distinct characteristics and trade-offs. ![cross-organization-data-sharing-patterns diagram 1](https://www.conduktor.io/assets/images/glossary/cross-organization-data-sharing-patterns-0.webp) ### Point-to-Point Integration The simplest approach connects two organizations through direct integration. One system sends data via APIs, file transfers, or database connections to another system. This pattern works well for limited partnerships but becomes difficult to manage as the number of connections grows. With N organizations, you potentially need N(N-1)/2 connections. Point-to-point integration often uses REST APIs, SFTP, or direct database access. While straightforward to implement initially, this pattern creates tight coupling and makes it hard to add new partners or change data formats. ### Hub-and-Spoke A hub-and-spoke pattern centralizes data sharing through an intermediary platform. Organizations connect to the hub, which handles routing, transformation, and delivery. This reduces the number of connections from N² to N and provides a single place to enforce policies and monitor data flows. Cloud data platforms, API gateways, and data marketplaces often implement hub-and-spoke patterns. The hub manages authentication, rate limiting, and data format conversions, simplifying integration for participating organizations. ### Mesh and Event-Driven Patterns Modern architectures increasingly use event-driven patterns where organizations publish events to shared event streams. Other organizations subscribe to relevant events and process them independently. This creates a loosely coupled mesh where producers and consumers don't need direct knowledge of each other. For foundational event-driven concepts, see [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture) and [Event Stream Fundamentals](https://www.conduktor.io/glossary/event-stream-fundamentals). Event-driven patterns work particularly well for real-time data sharing and scenarios where multiple organizations need the same data. A manufacturer might publish inventory events that flow to distributors, logistics providers, and retailers simultaneously. ## Security and Access Control Models Cross-organization data sharing requires robust security at multiple levels. ### Authentication and Authorization Organizations must verify the identity of data consumers (authentication) and control what data they can access (authorization). Common approaches include: - **OAuth 2.0 and OIDC** for API-based sharing, allowing organizations to grant limited access without sharing credentials. In 2025, Workload Identity Federation enables cross-cloud authentication without managing service account keys. - **Mutual TLS (mTLS)** for service-to-service communication, ensuring both parties authenticate each other with certificate-based verification - **API keys and tokens** for simpler scenarios, though these require careful rotation and management - **SASL/SCRAM or SASL/OAUTHBEARER** for Kafka authentication, with OAuth providing better support for token-based, time-limited access For Kafka-specific authentication, see [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth) and [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka). Example OAuth 2.0 client credentials flow for cross-organization API access: ```python import requests from requests.auth import HTTPBasicAuth # Partner organization obtains access token token_response = requests.post( 'https://auth.primary-org.com/oauth/token', auth=HTTPBasicAuth(client_id, client_secret), data={ 'grant_type': 'client_credentials', 'scope': 'data:read:inventory' } ) access_token = token_response.json()['access_token'] # Use token to access shared data API data_response = requests.get( 'https://api.primary-org.com/v1/inventory', headers={'Authorization': f'Bearer {access_token}'} ) inventory_data = data_response.json() ``` Authorization typically uses role-based access control (RBAC) or attribute-based access control (ABAC). RBAC grants permissions based on predefined roles (e.g., "partner-org-reader"), while ABAC makes access decisions based on dynamic attributes like user department, data classification, time of day, or request context. For example, a healthcare network might grant a partner hospital read access to specific patient records based on attributes like patient consent, treating physician, and active care relationship. For comprehensive access control patterns, see [Data Access Control: RBAC and ABAC](https://www.conduktor.io/glossary/data-access-control-rbac-and-abac). Example ABAC policy for cross-organization data access: ```json { "effect": "allow", "principal": "organization:partner-hospital-123", "action": "kafka:consume", "resource": "topic:patient-records", "conditions": { "patientConsent": "granted", "careRelationship": "active", "dataClassification": "phi-shared", "accessTime": "business-hours" } } ``` ### Data Masking and Encryption Sensitive data often requires masking or encryption before sharing. Organizations might: - Encrypt data in transit using TLS 1.3+ and at rest using encryption keys managed by each organization - Apply field-level masking to hide sensitive attributes like social security numbers or account details - Use tokenization to replace sensitive values with non-sensitive tokens that can be mapped back only by authorized systems - Apply dynamic data masking based on the consumer's identity and permissions For streaming data encryption, see [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka) and [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming). Example data masking before cross-organization sharing: ```python import hashlib import json def mask_sensitive_fields(transaction, consumer_org): """Mask data based on consuming organization's permissions.""" masked = transaction.copy() # Always mask full account numbers for external orgs if consumer_org != 'internal': masked['account_number'] = f"****{transaction['account_number'][-4:]}" masked['customer_name'] = "REDACTED" # Replace SSN with one-way hash for fraud detection if 'ssn' in transaction: masked['ssn_hash'] = hashlib.sha256( transaction['ssn'].encode() ).hexdigest() del masked['ssn'] # Preserve transaction patterns and amounts for analysis # (amount, timestamp, merchant_category remain) return masked # Original transaction transaction = { "transaction_id": "tx_12345", "account_number": "1234567890", "customer_name": "John Doe", "ssn": "123-45-6789", "amount": 245.50, "merchant_category": "retail", "timestamp": "2025-01-15T10:30:00Z" } # Masked version for partner fraud detection system shared_transaction = mask_sensitive_fields(transaction, consumer_org='partner-bank') ``` A financial institution sharing transaction data for fraud analysis might mask customer names and account numbers while preserving transaction patterns and amounts, allowing partners to detect fraud without exposing PII. For PII handling strategies, see [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). ## Data Streaming in Cross-Organization Scenarios Event streaming platforms like Apache Kafka have become popular for cross-organization data sharing because they support real-time data flows, decouple producers from consumers, and provide built-in durability and replay capabilities. Modern streaming platforms (Kafka 4.0+, Apache Pulsar) have evolved significantly in 2025, offering native multi-tenancy, improved security models, and built-in data sharing capabilities that simplify cross-organization integration. ### Kafka for Multi-Organization Data Sharing Organizations can use Kafka in several ways for cross-organization sharing: **Multi-Cluster Replication**: Each organization runs its own Kafka cluster. MirrorMaker 2 or Confluent Cluster Linking replicates selected topics between clusters, maintaining data sovereignty while enabling sharing. With Kafka 4.0+, KRaft mode (replacing ZooKeeper) simplifies cluster management and improves replication performance. This pattern works well when organizations want complete control over their infrastructure. For detailed cluster replication setup, see [Kafka MirrorMaker 2 for Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication). **Shared Cluster with Multi-Tenancy**: Organizations share a Kafka cluster but use ACLs and quotas to isolate data and enforce access policies. Kafka 4.0+ provides enhanced multi-tenancy features including improved quota management and namespace isolation. This reduces operational overhead but requires careful security configuration. For comprehensive ACL configuration, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) and [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments). **Event Streaming as Integration Layer**: Kafka acts as the hub in a hub-and-spoke pattern. Organizations publish events to central topics, and consumers subscribe based on their needs. Schema Registry (Confluent Schema Registry, AWS Glue Schema Registry, or Apicurio Registry) ensures data compatibility across organizations. For schema management best practices, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). ### Real-World Example: Supply Chain Data Sharing Consider a manufacturer sharing real-time inventory data with distributors. The manufacturer publishes inventory events to a Kafka topic. Each distributor subscribes to events for products they carry. When inventory drops below a threshold, distributors automatically adjust orders. This event-driven approach means the manufacturer doesn't need to know which distributors exist or how they use the data. New distributors can join by subscribing to the topic without requiring changes to the manufacturer's systems. Stream governance platforms help manage this complexity by providing topic-level access control, schema validation, and data masking capabilities to configure organizational access, apply transformations, and monitor data flows across boundaries. ### Modern Data Sharing Platforms (2025) Several specialized platforms have emerged to simplify cross-organization data sharing: - **Conduktor Partner Zones**: Enables organizations to share Kafka topics across cloud accounts and regions with fine-grained access control (see [Conduktor Partner Zones](https://docs.conduktor.io/guide/conduktor-concepts/partner-zones) for secure cross-organization data sharing). Producers maintain ownership while granting read access to external consumers through governance policies. This approach eliminates data duplication and reduces operational complexity. For a vendor-aware walkthrough, see [real-time data sharing](https://www.conduktor.io/solutions/use-case/data-sharing). - **Delta Sharing**: An open protocol for secure data sharing that works with data lake tables (Delta Lake, Apache Iceberg, Apache Hudi). Organizations share live data without copying it, using a simple REST API for access control. Recipients can query shared data using their preferred tools (Spark, pandas, Power BI) without requiring direct cloud storage access. For how table sharing and stream sharing differ, see [Delta Sharing alternatives for real-time data sharing](https://www.conduktor.io/compare/delta-sharing-alternatives). For implementation details, see [Zero-Copy Data Sharing](https://www.conduktor.io/glossary/zero-copy-data-sharing). - **Apache Iceberg REST Catalog**: Provides unified metadata access across organizations. Teams can share table metadata and enforce fine-grained access controls at the table and column level while maintaining separate storage accounts. - **Managed API Gateways**: Cloud platforms (AWS API Gateway, Google Cloud Apigee, Azure API Management) offer built-in authentication, rate limiting, and monitoring for API-based data sharing. GraphQL Federation enables organizations to expose unified APIs while maintaining separate backend services. For API gateway patterns, see [API Gateway Patterns for Data Platforms](https://www.conduktor.io/glossary/api-gateway-patterns-for-data-platforms). ## Governance and Compliance Considerations Cross-organization data sharing requires clear governance frameworks to manage responsibilities, quality, and compliance. For broader governance frameworks, see [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) and [Data Mesh Principles and Implementation](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation). ### Data Contracts and SLAs Organizations should establish formal agreements defining: - **Data contracts** that specify schemas, update frequencies, and data quality expectations - **Service level agreements (SLAs)** covering availability, latency, and support - **Change management processes** for schema evolution and breaking changes For implementing data contracts, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). For SLA monitoring and management, see [Data Freshness Monitoring & SLA Management](https://www.conduktor.io/glossary/data-freshness-monitoring-sla-management) and [SLA for Streaming](https://www.conduktor.io/glossary/sla-for-streaming). These contracts prevent misunderstandings and provide a foundation for resolving issues when they arise. ### Regulatory Compliance Data sharing across organizations often involves regulatory requirements: - **GDPR** requires explicit consent for personal data sharing and mandates data minimization - **HIPAA** governs healthcare data sharing and requires business associate agreements - **Financial regulations** like PCI DSS control how payment card data can be shared For detailed GDPR compliance guidance, see [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams). Organizations must map data flows to understand which regulations apply and implement appropriate controls. Audit logs tracking who accessed what data when are essential for compliance and incident response. For audit logging implementation, see [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) and [Streaming Audit Logs](https://www.conduktor.io/glossary/streaming-audit-logs). ## Implementation Challenges and Best Practices Several practical challenges arise when implementing cross-organization data sharing. ### Network Connectivity Organizations often operate behind firewalls with restricted network access. Options include: - **API gateways** that provide controlled access through public endpoints - **VPN tunnels** for private connectivity between networks - **Cloud-based integration platforms** that both organizations can reach without direct network connections Event streaming platforms can simplify network challenges. Organizations connect outbound to a shared Kafka cluster, avoiding the need for inbound firewall rules. ### Schema Evolution Data formats evolve over time, creating compatibility challenges. Best practices include: - Use schema registries to version and validate schemas - Apply backward and forward compatibility rules so changes don't break consumers - Provide advance notice of breaking changes and support multiple schema versions during transitions For comprehensive schema evolution strategies, see [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) and [Schema Evolution in Apache Iceberg](https://www.conduktor.io/glossary/schema-evolution-in-apache-iceberg). Schema management tools help teams validate schema changes before deployment and understand which consumers might be affected by changes. ### Monitoring and Observability Understanding data flows across organizations requires monitoring at multiple levels: - **Infrastructure metrics** tracking throughput, latency, and errors (using Prometheus, Grafana, DataDog, or cloud-native monitoring) - **Business metrics** measuring data freshness, completeness, and quality - **Security metrics** detecting unusual access patterns or potential breaches using anomaly detection - **Distributed tracing** with OpenTelemetry to track data flows across organizational boundaries Modern observability platforms in 2025 provide: - **OpenTelemetry integration**: Standardized instrumentation for Kafka producers, consumers, and stream processors, enabling end-to-end trace correlation across organizations - **Real-time alerting**: Automated notifications when SLAs are violated or anomalies detected - **Cross-organization dashboards**: Unified views showing data flow health across all participating organizations For Kafka-specific monitoring, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) and [Distributed Tracing for Kafka Applications](https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications). Example OpenTelemetry instrumentation for cross-organization Kafka producer: ```python from opentelemetry import trace from opentelemetry.instrumentation.kafka import KafkaInstrumentor from kafka import KafkaProducer # Initialize OpenTelemetry tracing tracer = trace.get_tracer(__name__) KafkaInstrumentor().instrument() # Create instrumented Kafka producer producer = KafkaProducer( bootstrap_servers='kafka.primary-org.com:9092', security_protocol='SASL_SSL', sasl_mechanism='OAUTHBEARER' ) # Traces automatically include organization context with tracer.start_as_current_span("share_inventory_data") as span: span.set_attribute("organization", "manufacturer-123") span.set_attribute("partner_count", 5) producer.send( 'inventory-updates', key=b'product-sku-456', value=inventory_event_json ) producer.flush() ``` Centralized monitoring platforms that aggregate metrics from all participants help teams quickly identify and resolve issues. In 2025, organizations increasingly use federated observability where each org maintains control over their telemetry data while sharing aggregated health metrics. ## Summary Cross-organization data sharing enables collaboration while introducing complexity in architecture, security, and governance. The most appropriate pattern depends on the number of partners, real-time requirements, and control needs. Point-to-point integration works for simple scenarios but doesn't scale. Hub-and-spoke patterns centralize management at the cost of creating a single point of control. Event-driven architectures using platforms like Kafka 4.0+ provide loose coupling and real-time capabilities while requiring careful attention to security and schema management. In 2025, modern data sharing platforms have significantly simplified cross-organization integration: - **Stream sharing technologies** (Conduktor) eliminate data replication with governance policies - **Zero-copy protocols** (Delta Sharing, Iceberg REST catalog) enable secure data access without duplication - **Enhanced multi-tenancy** in Kafka 4.0+ with KRaft improves isolation and performance - **OpenTelemetry integration** provides standardized observability across organizational boundaries - **Workload Identity Federation** simplifies cross-cloud authentication Successful implementations establish clear data contracts, implement layered security (authentication, authorization, and encryption), and maintain monitoring with distributed tracing. Modern platforms with built-in schema management, access control, and data masking reduce the operational burden of managing these data flows. For quality assurance in shared data, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) and [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). As organizations participate in more data ecosystems, these patterns and practices become necessary for reliable, secure, and scalable data sharing. ## Related Concepts - [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) - Fine-grained access control essential for securing cross-organization data streams - [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth) - Authentication mechanisms for verifying identity across organizational boundaries - [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka) - Protecting sensitive data as it moves between organizations - [Real-Time Data Clean Room](https://www.conduktor.io/glossary/real-time-data-clean-room) - Governed environment where organizations share live streams under enforced privacy controls ## Sources and References 1. Apache Kafka Documentation: "Multi-Tenancy and Data Isolation" - https://kafka.apache.org/documentation/#multitenancy 2. Conduktor Platform Documentation - https://www.conduktor.io/ 3. Apache Kafka 4.0 Documentation: "KRaft Mode and Security" - https://kafka.apache.org/documentation/#kraft 4. Delta Sharing Protocol Specification (2024) - https://github.com/delta-io/delta-sharing 5. OpenTelemetry Kafka Instrumentation (2025) - https://opentelemetry.io/docs/instrumentation/kafka/ 6. Martin Kleppmann: "Designing Data-Intensive Applications" - Chapter 11 on Stream Processing and Data Integration 7. AWS Whitepaper: "Securely Share Data Across AWS Accounts" - https://docs.aws.amazon.com/whitepapers/latest/architecting-hipaa-security-and-compliance-on-aws/data-sharing.html 8. GDPR Official Text: Articles on Data Processing and Third-Party Sharing - https://gdpr-info.eu/ Source: https://www.conduktor.io/glossary/cross-organization-data-sharing-patterns --- # Crypto Shredding for Kafka: GDPR-Compliant Data Deletion **Crypto shredding** is a data deletion technique that makes encrypted data permanently unreadable by destroying the encryption keys. Unlike physical deletion, the ciphertext remains in Kafka but becomes cryptographically meaningless without its key, satisfying GDPR's "right to be forgotten" without modifying immutable logs. Kafka's append-only architecture creates a fundamental tension with data privacy regulations. When a user exercises their "right to be forgotten" under GDPR, you cannot simply delete their records from an immutable log. The data persists in topic partitions, replicas, and consumer state stores. Crypto shredding resolves this by making data **unreadable** rather than physically deleting it. Encrypt each user's data with their own key, then destroy the key when deletion is required. The ciphertext remains in Kafka, but without the key, it becomes meaningless bytes, cryptographically indistinguishable from random noise. This approach satisfies GDPR Article 17 (Right to Erasure) requirements while preserving Kafka's architectural integrity. The data is effectively "deleted" from a privacy perspective: no one can access it, even with full access to the underlying storage. ## How Crypto Shredding Works Crypto shredding combines field-level encryption with per-user key management. The architecture uses envelope encryption with three key types: ![Crypto Shredding Architecture with Envelope Encryption](https://www.conduktor.io/assets/images/glossary/crypto-shredding-for-kafka-0.webp) **Key terminology:** - **KEK (Key Encryption Key)**: Master key stored in your external KMS (AWS KMS, HashiCorp Vault, Azure Key Vault). Never leaves the KMS. - **DEK (Data Encryption Key)**: Per-user key generated by Conduktor Gateway that encrypts actual record fields. - **EDEK (Encrypted DEK)**: The DEK encrypted by the KEK. Stored safely because it cannot be decrypted without the KEK. The encryption flow: 1. Conduktor Gateway intercepts records on produce 2. Extracts a unique identifier (e.g., `userId`) to determine the key ID 3. Generates or retrieves a DEK for that user 4. Encrypts specified fields (e.g., `email`, `visa`) with the DEK 5. Stores the EDEK in a dedicated Kafka topic (Encryption Keys Store) 6. Produces the encrypted record to the target topic When crypto shredding is triggered, you tombstone the EDEK record in the keys store. Without the DEK, Conduktor Gateway cannot decrypt the data, and neither can anyone else. ## Conduktor Gateway Configuration for Crypto Shredding Conduktor Gateway, a Kafka proxy, implements crypto shredding through the `gateway-kms://` key scheme. This delegates key storage to a dedicated Kafka topic while using your external KMS for the master key. ### Encryption Interceptor ```yaml # Per-user encryption keys for crypto shredding pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin config: topic: customers kmsConfig: vault: uri: http://vault:8200 token: ${VAULT_TOKEN} gateway: masterKeyId: vault-kms://vault:8200/transit/keys/master-key fields: - fieldName: email keySecretId: gateway-kms://user-{{record.value.userId}} - fieldName: visa keySecretId: gateway-kms://user-{{record.value.userId}} ``` Key points: - `gateway-kms://user-{{record.value.userId}}`: Creates a unique DEK per user via mustache templating - `masterKeyId`: References the KEK in your external KMS (Vault, AWS, Azure, or GCP) - Fields not listed remain unencrypted for querying ### Decryption Interceptor ```yaml pluginClass: io.conduktor.gateway.interceptor.DecryptPlugin config: topic: customers kmsConfig: vault: uri: http://vault:8200 token: ${VAULT_TOKEN} gateway: masterKeyId: vault-kms://vault:8200/transit/keys/master-key ``` If a key has been shredded, Conduktor Gateway returns encrypted data per the configured `errorPolicy`. For complete configuration, see [Conduktor Gateway encryption documentation](https://docs.conduktor.io/guide/use-cases/encrypt-kafka). ## The Crypto Shredding Process When a user requests data deletion, you perform crypto shredding by tombstoning their EDEK in the Encryption Keys Store topic. ![Crypto Shredding Deletion Process](https://www.conduktor.io/assets/images/glossary/crypto-shredding-for-kafka-1.webp) ### Step-by-Step Deletion 1. **Identify the key ID**: For a user with `userId: 101`, the key ID is `gateway-kms://user-101` 2. **Find all EDEKs**: Scan `_conduktor_gateway_encryption_keys` for records matching this key ID. Due to distributed processing, multiple EDEKs may exist with different UUIDs: ```json {"algorithm":"AES128_GCM","keyId":"gateway-kms://user-101","uuid":"abc-123"} {"algorithm":"AES128_GCM","keyId":"gateway-kms://user-101","uuid":"def-456"} ``` 3. **Tombstone each EDEK**: Produce a null-value record for each key: ```bash echo '{"algorithm":"AES128_GCM","keyId":"gateway-kms://user-101","uuid":"abc-123"}|NULL' | \ kafka-console-producer \ --bootstrap-server gateway:6969 \ --topic _conduktor_gateway_encryption_keys \ --property "parse.key=true" \ --property "key.separator=|" \ --property "null.marker=NULL" ``` 4. **Verify**: Consume from the original topic. The shredded user's data now returns encrypted (unreadable), while other users' data decrypts normally. ### Important Considerations - **Tombstone all UUIDs**: Multiple Conduktor Gateway nodes may create duplicate EDEKs for the same user. Ensure all are tombstoned. - **New records still encrypt**: Shredding only affects historical data. New records for the same user would create a new DEK. - **Topic compaction timing**: The tombstoned records eventually compact away, but timing depends on Kafka's compaction settings. For a hands-on tutorial, see [Configure crypto shredding](https://docs.conduktor.io/guide/tutorials/crypto-shredding). ## Why Crypto Shredding vs Alternatives ### Compared to Tombstone Records Standard Kafka tombstones (`key=userId, value=null`) only mark logical deletion. Consumers must implement filtering logic, and downstream systems may already have copies of the data. With crypto shredding, the data itself becomes unreadable everywhere. ### Compared to Log Compaction Log compaction eventually removes older records with the same key, but: - No guarantees on timing - Replicas and backups may retain data - Downstream consumers may have persistent copies Crypto shredding provides immediate, cryptographic certainty. ### Compared to Pseudonymization Pseudonymization (storing PII separately with tokens) requires managing a separate PII store and complex join logic. Crypto shredding keeps data in Kafka's natural structure while enabling compliant deletion. ## Production Considerations ### Cost and Performance Conduktor Gateway's KMS significantly reduces KMS costs for high-volume scenarios: - **Single KEK**: Only one master key in your external KMS - **Many DEKs**: Per-user keys stored as encrypted records in Kafka - **Local caching**: DEKs are cached in Conduktor Gateway memory with configurable TTL Without Conduktor Gateway KMS, storing millions of per-user keys directly in AWS KMS or Vault would be cost-prohibitive and create a performance bottleneck. ### Key Security The master key (KEK) is the crown jewel: - Stored only in your external KMS (Vault, AWS, Azure, GCP) - Never exposed to Conduktor Gateway - Rotate per your organization's policy - If compromised, all data protected by that KEK is at risk DEKs are protected by: - Encryption with the KEK before storage - Compacted Kafka topic with access controls - Conduktor Gateway authentication requirements ### Compliance Audit Trail Maintain records of: - When crypto shredding was requested (deletion request timestamp) - When tombstones were produced (execution timestamp) - Which key IDs were shredded - Verification that encrypted data is returned post-shredding This audit trail demonstrates GDPR compliance during regulatory review. ### Error Handling Configure the decryption interceptor's `errorPolicy`: - `return_encrypted`: Returns unreadable ciphertext (supports crypto shredding) - `fail_fetch`: Fails the consumer request - `crypto_shred_safe_fail_fetch`: Fails unless key not found (crypto shredding case) For most GDPR scenarios, `return_encrypted` is appropriate, it allows consumers to continue processing while making shredded data permanently inaccessible. ## Related Concepts - [Zero Trust Architecture for Kafka](https://www.conduktor.io/glossary/zero-trust-architecture-for-kafka): Crypto shredding fits within a zero trust security model where data protection extends beyond access control - [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams): Broader GDPR implementation strategies including consent management and data subject rights - [Encryption at Rest and In Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka): Infrastructure-level encryption that complements field-level crypto shredding - [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming): Alternative approaches to protecting sensitive data in streams ## Sources and References - [Conduktor Crypto Shredding Tutorial](https://docs.conduktor.io/guide/tutorials/crypto-shredding) - Step-by-step implementation guide with Docker Compose environment. - [Conduktor Gateway Encryption Reference](https://docs.conduktor.io/guide/reference/data-security#encryption-interceptors) - Complete configuration options for encryption and decryption interceptors. - [GDPR Article 17 - Right to Erasure](https://gdpr-info.eu/art-17-gdpr/) - Legal basis for the right to be forgotten. - [NIST Key Management Guidelines (SP 800-57)](https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final) - Best practices for cryptographic key management. Source: https://www.conduktor.io/glossary/crypto-shredding-for-kafka --- # Dark Data Tax: The Hidden Costs of Unused Data Organizations collect vast amounts of data that is never analyzed, queried, or used for decision-making. This unused information, known as "dark data," creates a hidden tax that extends far beyond storage costs. For a related but distinct challenge, see [Data Obesity](https://www.conduktor.io/glossary/data-obesity), which explores the broader problem of excessive data volume. The dark data tax represents the cumulative financial, technical, and organizational burden of maintaining data that provides no business value. While cloud storage may be inexpensive, the true costs manifest in data pipeline maintenance, computational overhead, engineering time, cognitive load on data teams, and increased security and compliance risk. ![Dark Data Tax Cost Accumulation](https://www.conduktor.io/assets/images/glossary/dark-data-tax-0.webp) ## The Three Pillars of the Dark Data Tax The dark data tax can be understood through three interconnected dimensions: - **Financial costs** include not just storage, but the compute resources required to ingest, serialize, transform, and move data through pipelines. Cloud providers charge for data transfer, query execution, and processing, costs that accumulate whether or not anyone uses the resulting datasets. - **Technical costs** emerge when data engineers spend valuable time maintaining pipelines, debugging failures, and optimizing schemas for data that serves no purpose. Every broken pipeline requires investigation, every schema change needs coordination, and every data quality issue demands attention. - **Organizational costs** appear as cognitive overhead. When data platforms contain thousands of unused tables, topics, or datasets, data consumers struggle to find reliable information. Trust in the platform erodes, productivity decreases, and teams waste time navigating "data swamps" instead of extracting insights. ## The Operational Tax: Beyond Storage Costs A common misconception is that the primary cost of dark data is storage. Cloud object storage services like Amazon S3 or Google Cloud Storage charge only a few cents per gigabyte per month. At this price point, storage itself rarely becomes the limiting factor. The real operational tax emerges elsewhere in the data lifecycle: - **Data ingestion and movement** consume significant resources. Every record ingested into a data warehouse or lakehouse incurs compute costs for parsing, validation, and transformation. Data transferred between regions or across cloud providers generates egress fees that can far exceed storage costs. - **Pipeline maintenance** represents an ongoing engineering investment. Data engineers build ETL or ELT processes to move data from source systems to analytical stores. These pipelines require monitoring, error handling, schema evolution management, and periodic optimization. When a pipeline breaks at 3 AM to load data that hasn't been queried in six months, the cost is paid in engineer time, burnout, and opportunity cost. - **Query and compute overhead** affects active users. In data warehouses, every table consumes metadata storage and appears in query planning. Unused tables slow down data discovery, clutter autocomplete suggestions, and occasionally get accidentally joined in queries, increasing execution time and costs. Consider a typical scenario: A product team requests a new data stream to analyze user behavior. Data engineers build the pipeline, create transformations, and load data into a warehouse table. Three months later, the product initiative is cancelled, but the pipeline continues running. Over the next year, the organization pays for continuous ingestion, transformation compute, storage, and maintenance, all for data that will never be analyzed. ## The Dark Data Problem in Streaming Systems Streaming data platforms like Apache Kafka, Apache Flink, and Apache Pulsar amplify the dark data tax in several ways: - **High-throughput continuous costs**: Unlike batch systems that process data periodically, streaming platforms operate continuously. Every message sent to a Kafka topic incurs network bandwidth, serialization overhead (converting data to efficient formats like Avro, Protocol Buffers, or JSON), and broker storage costs. If a topic receives millions of events per day that are never consumed or analyzed, you're paying constant compute and network costs with zero return. - **Real-time operational urgency**: Streaming systems are typically monitored with high priority because they often support real-time applications. When a stream fails, alerts fire, on-call engineers investigate, and teams treat the issue as urgent. If that stream feeds a dashboard that nobody looks at or a downstream system that was deprecated months ago, the operational overhead becomes pure waste. - **Consumer group proliferation**: In Kafka, multiple consumer groups (independent sets of consumers that each process all messages from a topic) can read from the same topic. Over time, consumer groups may be created for experiments, prototypes, or temporary analyses. Abandoned consumer groups continue tracking offsets and consuming cluster metadata. While the cost per consumer group is small, at scale this creates clutter and makes it difficult to understand actual data usage patterns. For detailed coverage of consumer group mechanics, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). - **Retention and compaction overhead**: Streaming platforms must decide how long to retain data. Longer retention periods provide flexibility for new use cases and recovery scenarios, but also increase storage costs and compaction overhead (the process of consolidating and removing duplicate or obsolete records). When topics contain dark data, organizations pay for retention without benefit. For more on compaction mechanics, see [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained). A data streaming team at a financial services company discovered that 40% of their Kafka topics hadn't been consumed in over 90 days. These topics continued generating data, consuming broker resources, and triggering alerts when brokers experienced performance degradation. By identifying and deprecating unused topics, they reduced cluster load by 35% and eliminated dozens of false-alert incidents per month. **Modern operational improvements (2025)**: Kafka 4.0 and KRaft mode (Kafka's ZooKeeper-less architecture) significantly reduce the operational overhead of managing streaming platforms. KRaft eliminates the need to maintain a separate ZooKeeper ensemble, reducing infrastructure costs and operational complexity. This makes it easier to identify and deprecate dark data topics without worrying about the coordination overhead that ZooKeeper previously imposed. For comprehensive coverage of KRaft benefits, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). Tools like Kafka Lag Exporter and Burrow provide real-time visibility into consumer lag and inactive consumer groups, helping teams quickly identify unused streams. For monitoring strategies, refer to [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). ## The Liability Tax: Security and Compliance Risks Dark data doesn't just cost money and engineering time, it creates legal and security risks that can result in substantial penalties and reputational damage. - **Expanded attack surface**: Every database, data lake, or streaming topic represents a potential target for attackers. Unused datasets are often less monitored than active production systems, making them attractive targets. If a forgotten database contains customer information and gets breached, the organization faces the same regulatory consequences as if it were actively used data. - **Compliance violations**: Data privacy regulations like GDPR, CCPA, and HIPAA require organizations to minimize data collection, implement retention limits, and honor deletion requests. Dark data often violates these principles. You may be storing personally identifiable information (PII) in a deprecated table, unaware that it should have been deleted per a customer's request. The fine for GDPR violations can reach 4% of annual global revenue. For detailed guidance on compliance requirements, see [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams) and [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). - **Data residency and sovereignty**: Different jurisdictions have varying requirements about where data can be stored and processed. Dark data that has been replicated across regions or cloud providers may violate data residency requirements without anyone realizing it. Streaming platforms that replicate data across data centers for reliability can inadvertently create compliance issues when data is never purged. - **Audit and discovery burden**: When regulators or legal teams request information about data handling practices, dark data creates significant overhead. Teams must investigate abandoned datasets, determine what they contain, and assess compliance status, all for information that provides no business value. A healthcare organization discovered during a compliance audit that they were retaining patient event streams in Kafka topics for over two years, far exceeding their stated retention policy of 90 days. While the data was encrypted, the extended retention violated their own privacy commitments and created unnecessary risk. Implementing automated retention policies and governance checks prevented similar issues in the future. ## Identifying and Reducing Dark Data Reducing the dark data tax requires a combination of technical tooling, organizational processes, and cultural change. Modern platforms provide sophisticated capabilities for tracking and managing data assets: - **Implement data observability**: Deploy tools that track dataset usage, query frequency, and consumer activity. Modern data observability platforms (2025) like Monte Carlo, Datadog Data Streams Monitoring, and Confluent Observability provide comprehensive visibility into data pipelines, automatically detecting unused datasets and anomalies. For data warehouses, monitor table access patterns using query logs and metadata APIs. For streaming platforms, track consumer lag, active consumer groups, and topic throughput using tools like Kafka Lag Exporter or Burrow. Datasets that show no consumption activity for 60-90 days are strong candidates for deprecation. To understand the foundational principles, see [What is Data Observability: The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars). - **Establish lifecycle policies**: Define clear retention policies based on data type and business value. Operational logs might be retained for 30 days, analytical data for one year, and compliance data for seven years. Automate enforcement of these policies using platform features like Kafka's retention settings, S3 lifecycle rules, data warehouse table expiration, or lakehouse table format features. Modern lakehouse formats like Apache Iceberg, Delta Lake, and Apache Hudi (2025) provide sophisticated retention and time-travel capabilities that make it easier to manage data lifecycles while maintaining compliance and enabling auditing. For detailed coverage of lakehouse architectures, see [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture) and [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg). - **Create data catalogs**: Maintain metadata about dataset purpose, ownership, consumers, and business value. Modern data catalog and governance platforms (2025) like OpenMetadata, Atlan, Collibra, Apache Atlas, and DataHub provide automated discovery, lineage tracking, and usage analytics. These tools help teams understand what data exists, who uses it, and why it was created. When a dataset's purpose is unclear or its owner has left the organization, it's likely dark data. Many platforms now integrate data quality monitoring through tools like Soda Core, automatically flagging datasets with quality issues or no recent consumption. For more on catalog capabilities, see [What is a Data Catalog: Modern Data Discovery](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery). - **Implement approval workflows**: Require justification before creating new data pipelines or topics. A lightweight review process, where teams document the use case, expected consumers, and retention requirements, prevents the creation of speculative datasets that may never be used. Consider implementing data contracts that formalize agreements between data producers and consumers, ensuring that new data assets have committed downstream users before resources are allocated. For detailed guidance on data contracts, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). - **Enable self-service governance**: For streaming platforms, governance tools provide visibility into Kafka clusters, topic usage, and consumer groups. Data teams can identify unused topics, review schema evolution, and enforce access control policies without requiring deep Kafka expertise. This democratizes governance and reduces the burden on central platform teams. - **Conduct regular audits**: Schedule quarterly reviews of data assets. For each dataset, ask: Is this being actively used? Does it serve a documented business purpose? Is it compliant with our retention policies? Datasets that fail these tests should be deprecated or archived. - **Implement soft deletion**: Rather than immediately deleting potentially dark data, mark it as deprecated and prevent new consumers. After a grace period (e.g., 30 days), if no one objects, proceed with deletion. This reduces the risk of accidentally removing valuable datasets while still eliminating waste. ## Summary The dark data tax is a hidden but significant cost that affects organizations at every scale. While cloud storage is inexpensive, the true burden emerges in compute overhead, pipeline maintenance, engineering time, cognitive load, and security risk. For streaming data platforms, the tax is particularly acute. Continuous high-throughput processing, real-time operational urgency, and complex consumer ecosystems multiply costs when data goes unused. A Kafka topic that receives millions of unused events per day generates constant network, serialization, and storage costs while consuming engineering attention for monitoring and maintenance. For a comprehensive analysis of streaming costs, see [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership). Reducing the dark data tax requires technical solutions, data observability, automated retention policies, and governance tooling, combined with organizational discipline. By implementing usage tracking, establishing lifecycle policies, and conducting regular audits, data teams can identify and eliminate dark data before it accumulates. For broader governance context, see [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities). The goal is not to minimize all data collection, but to ensure that every dataset serves a documented purpose and delivers value proportional to its cost. As data volumes grow, the organizations that stay ahead will be those that distinguish between valuable data assets and expensive dark data liabilities. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Streaming platforms where dark data tax often manifests through unused topics and consumer groups - [Data Obesity](https://www.conduktor.io/glossary/data-obesity) - The related problem of excessive data volume impacting system performance - [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership) - Understanding the full economic burden of streaming infrastructure including dark data ## Sources and References - Gartner, "How to Deal with Dark Data" - Research on identifying and managing unused data assets - Apache Kafka Documentation - Topic configuration, retention policies, and consumer group management - General Data Protection Regulation (GDPR) Article 5 - Data minimization and storage limitation principles - California Consumer Privacy Act (CCPA) - Requirements for data collection and retention - AWS Documentation - Data transfer pricing and S3 lifecycle management - "The Data Warehouse Toolkit" by Ralph Kimball - Best practices for dimensional modeling and data governance - Apache Flink Documentation - Stream processing concepts and operational best practices Source: https://www.conduktor.io/glossary/dark-data-tax --- # Data Access Control: RBAC and ABAC Access control determines who can interact with your data and what actions they can perform. As organizations scale and data architectures become more complex, particularly with real-time streaming platforms, choosing the right access control model matters. This article covers two fundamental approaches: Role-Based Access Control (RBAC) and Attribute-Based Access Control (ABAC), with practical examples for streaming data infrastructure. Modern implementations leverage tools like Open Policy Agent (OPA), Keycloak, AWS Cedar, and cloud-native IAM systems to enforce these models at scale. For comprehensive Kafka-specific authorization patterns, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). ![RBAC vs ABAC Access Control Models](https://www.conduktor.io/assets/images/glossary/data-access-control-rbac-and-abac-0.webp) ## Understanding RBAC: The Foundation Role-Based Access Control assigns permissions based on organizational roles. Users inherit permissions through their assigned roles, creating a layer of abstraction between individuals and access rights. ### Core RBAC Concepts In RBAC, you define three primary components: 1. **Roles**: Collections of permissions (e.g., "Data Engineer," "Analytics Reader") 2. **Permissions**: Specific actions on resources (e.g., "read topic," "write to consumer group") 3. **Users**: Individuals assigned to one or more roles This model excels in organizations with clearly defined job functions and relatively stable access patterns. ### RBAC in Streaming Platforms Consider a Kafka deployment. With Kafka 4.0+ and KRaft mode, authorization has evolved beyond simple ACLs to support pluggable authorizers and integration with enterprise identity providers. You might structure your RBAC model as follows: ```yaml # Example RBAC configuration roles: - name: streaming-producer permissions: - resource: topic pattern: "sales.*" operations: [WRITE, DESCRIBE] - name: streaming-consumer permissions: - resource: topic pattern: "sales.*" operations: [READ, DESCRIBE] - resource: consumer-group pattern: "analytics-*" operations: [READ] - name: platform-admin permissions: - resource: "*" pattern: "*" operations: [ALL] users: - email: engineer@company.com roles: [streaming-producer] - email: analyst@company.com roles: [streaming-consumer] ``` In this configuration, a data engineer can write to sales topics but cannot read from them directly, while analysts can consume from these topics through designated consumer groups. This separation of concerns is fundamental to RBAC's security model. **Modern Authorization Implementations (2025):** - **Keycloak + OAuth2**: Integrate Kafka with Keycloak for centralized RBAC using OAuth2/OIDC tokens. Kafka clients authenticate with bearer tokens, and roles are extracted from token claims. - **Cloud-native IAM**: AWS MSK supports IAM-based authentication where AWS IAM roles and policies define Kafka permissions. Azure Event Hubs similarly integrates with Azure AD RBAC. - **OPA (Open Policy Agent)**: While traditionally used for ABAC, OPA can enforce RBAC policies through its Kafka authorizer plugin, centralizing policy management across your stack. For authentication mechanisms that enable these authorization models, see [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). ### RBAC Advantages - **Simplicity**: Easy to understand and implement. New employees can be quickly onboarded by assigning them to appropriate roles. - **Auditability**: Clear role assignments make compliance audits straightforward. You can easily answer "Who has access to customer PII?" by identifying users in relevant roles. - **Scalability**: Adding new users is simple, just assign existing roles. No need to configure individual permissions for each person. ### RBAC Limitations RBAC struggles with fine-grained, context-sensitive requirements. What if a data engineer should only access production topics during business hours? What if analysts from the EMEA region shouldn't access US customer data? These scenarios require workarounds or additional layers, which is where ABAC excels. ## ABAC: Fine-Grained Control Attribute-Based Access Control evaluates access decisions based on attributes of the user, resource, action, and environment. Instead of "Does this user have the 'data-engineer' role?" ABAC asks "Does this user's department, clearance level, location, and current time satisfy the policy for this resource?" ### ABAC Components ABAC policies consider multiple attribute categories: - **Subject attributes**: User department, security clearance, location, employment status - **Resource attributes**: Data classification, owner, creation date, geographic origin - **Action attributes**: Read, write, delete, modify schema - **Environmental attributes**: Time of day, IP address, authentication strength, security context ### ABAC in Streaming Platforms Modern data governance platforms enable ABAC policies for Kafka clusters. ABAC policies are typically written in specialized policy languages that evaluate conditions across attributes. The most common pattern uses a structure like: - **subject.attribute**: Properties of the requesting user/service (e.g., `subject.region`, `subject.department`) - **resource.attribute**: Properties of the data being accessed (e.g., `resource.data_region`, `resource.classification_level`) - **environment.attribute**: Contextual factors (e.g., `environment.time`, `environment.ip_address`) - **action**: The operation being requested (READ, WRITE, DELETE, etc.) Here's a realistic example using a policy language similar to OPA or Cedar: ```yaml # Example ABAC policy policies: - name: regional-data-access description: Users can only access data from their region rule: | subject.region == resource.data_region && subject.clearance >= resource.classification_level - name: business-hours-production description: Production writes restricted to business hours except for on-call rule: | (environment.time >= "09:00" && environment.time <= "17:00") || subject.on_call_status == true effect: allow actions: [WRITE] resources: topics: ["production.*"] - name: pii-data-protection description: PII data requires additional authentication rule: | resource.contains_pii == true && environment.mfa_authenticated == true && subject.pii_training_completed == true effect: allow actions: [READ, WRITE] resources: topics: ["*.customer-data"] ``` In this ABAC model, access decisions are dynamic and contextual. An engineer might have access to a topic at 10 AM but not at 10 PM. A consumer application in the EU region can access EU customer topics but not US ones, regardless of the service account's other privileges. **Real-World ABAC with Open Policy Agent (OPA):** OPA has become the de facto standard for policy-based authorization in cloud-native environments. Here's how you'd implement the regional data access policy in OPA's Rego language for Kafka: ```rego # OPA Rego policy for Kafka authorization package kafka.authz import future.keywords.if import future.keywords.in # Allow access if user region matches data region allow if { input.operation.name == "Read" topic_region := topic_metadata[input.resource.name].region topic_region == input.principal.region } # Allow access if user has sufficient clearance allow if { topic_classification := topic_metadata[input.resource.name].classification input.principal.clearance_level >= topic_classification } # Topic metadata (could be fetched from a data catalog) topic_metadata := { "eu.customer-data": { "region": "eu", "classification": 3 }, "us.customer-data": { "region": "us", "classification": 3 } } ``` This Rego policy integrates with Kafka through an OPA authorizer plugin and evaluates in real-time as clients make requests. For data classification strategies that feed these policies, see [Data Classification and Tagging Strategies](https://www.conduktor.io/glossary/data-classification-and-tagging-strategies). ### ABAC Advantages - **Granularity**: Extremely fine-grained control based on any measurable attribute. - **Flexibility**: Policies adapt to changing contexts without modifying role assignments. - **Regulatory compliance**: Naturally aligns with data residency, privacy regulations (GDPR, CCPA), and classification-based security requirements. - **Dynamic adaptation**: Automatically responds to attribute changes. When an employee changes departments, their access automatically adjusts based on the new department attribute. ### ABAC Challenges - **Complexity**: Policy creation requires careful planning and testing. Unintended access denials can disrupt operations. - **Performance**: Evaluating complex policies in real-time can introduce latency, particularly in high-throughput streaming scenarios. OPA policy evaluation typically adds 1-10ms per request in well-optimized deployments, but complex policies with external data fetches can add 50-100ms+. For Kafka clusters handling 100,000+ messages/second, this overhead requires careful caching and policy optimization strategies. - **Debugging**: Troubleshooting "Why was I denied access?" becomes more complex when multiple attributes interact. Modern tools like OPA's decision logs and policy tracing help by showing which rules failed, but require additional infrastructure. ## Choosing Between RBAC and ABAC The choice isn't always binary. Many organizations implement hybrid models. Use **RBAC** when: - Organizational roles clearly map to data access needs - Access patterns are relatively static - Simplicity and operational ease are priorities - You're starting your access control journey Use **ABAC** when: - Regulatory requirements demand fine-grained, contextual controls - Data classification levels require dynamic enforcement - Multi-tenancy or regional data segregation is critical - Environmental factors (time, location, authentication method) should influence access Use **Both** when: - RBAC provides the baseline (e.g., "data-engineer" role) - ABAC adds contextual restrictions (e.g., "but only during business hours for production") ## Hybrid Implementation Approach Modern streaming governance platforms provide both RBAC and ABAC capabilities. Conduktor Platform offers UI-driven access management that translates to underlying ACLs or policy engines. You can define roles for common access patterns while layering ABAC policies for specialized requirements: ```yaml # Hybrid approach example rbac: roles: - name: data-engineer base_permissions: - topics: [READ, WRITE, DESCRIBE] abac: policies: - name: production-safeguard apply_to_roles: [data-engineer] conditions: - topic.environment == "production" - user.peer_review_approved == true OR user.on_call == true ``` This hybrid model gives engineers broad access to development and staging environments through RBAC, while ABAC gates production access with additional requirements. **Related Access Control Concerns:** - For audit trails of who accessed what, see [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) and [Conduktor's audit logs](https://docs.conduktor.io/guide/conduktor-in-production/admin/audit-logs) - For protecting sensitive data within allowed topics, see [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming) - For detecting and preventing PII exposure, see [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams) - For encryption alongside access control, see [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka) ## Conclusion RBAC and ABAC both have a place in data access control. RBAC provides operational simplicity and clear administrative boundaries, while ABAC delivers the precision needed for complex, dynamic environments. As streaming data platforms become central to enterprise architecture, knowing when to apply each model is important for Security Engineers and Data Governance Officers. In 2025, the access control landscape continues to evolve. Relationship-Based Access Control (ReBAC), modeling access based on relationships between entities (e.g., "can access if user.manager == resource.owner"), is gaining traction for complex organizational hierarchies. Tools like Google Zanzibar, SpiceDB, and AWS Cedar are making these advanced models more accessible. **Implementation Strategy:** 1. Start with RBAC for foundational access control using native Kafka ACLs or cloud IAM 2. Layer ABAC policies for compliance requirements (data residency, classification-based access) 3. Integrate with centralized identity providers (Keycloak, Azure AD, Okta) for authentication 4. Implement audit logging to track access decisions 5. Use policy-as-code approaches (OPA, Cedar) for version-controlled, testable access policies For broader data governance frameworks that incorporate access control, see [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities). ## Related Concepts - [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming) - Practical implementation of RBAC and ABAC in Kafka and streaming platforms. - [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) - How Kafka's native ACLs map to RBAC and ABAC models. - [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming) - Automated enforcement of RBAC and ABAC policies using tools like OPA. ## Sources and References **Standards and Research:** - **NIST RBAC Standard**: NIST. (2004). "Role Based Access Control (RBAC) and Role Based Security" - [NIST RBAC Model](https://csrc.nist.gov/projects/role-based-access-control) - **XACML Specification**: OASIS. (2013). "eXtensible Access Control Markup Language (XACML) Version 3.0" - [OASIS XACML Standard](http://docs.oasis-open.org/xacml/3.0/xacml-3.0-core-spec-os-en.html) - **ABAC Guide**: NIST Special Publication 800-162. (2014). "Guide to Attribute Based Access Control (ABAC) Definition and Considerations" - [NIST SP 800-162](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-162.pdf) - **Access Control Models**: Ferraiolo, D. F., et al. (2001). "Proposed NIST Standard for Role-Based Access Control" - ACM Transactions on Information and System Security - **Modern Access Control**: Hu, V. C., et al. (2015). "Attribute-Based Access Control" - IEEE Computer Magazine, Vol. 48, No. 2 **Implementation Tools (2025):** - **Apache Kafka Authorization**: [Kafka Security Documentation](https://kafka.apache.org/documentation/#security_authz) - ACLs and pluggable authorizers - **Open Policy Agent (OPA)**: [OPA Kafka Authorization](https://www.openpolicyagent.org/docs/latest/kafka-authorization/) - Policy-based access control - **AWS Cedar**: [Cedar Policy Language](https://www.cedarpolicy.com/) - Amazon's open-source authorization policy language - **Keycloak**: [Keycloak Authorization Services](https://www.keycloak.org/docs/latest/authorization_services/) - OAuth2/OIDC with fine-grained authorization - **Zanzibar**: Pang, R., et al. (2019). "Zanzibar: Google's Consistent, Global Authorization System" - USENIX ATC 2019 Source: https://www.conduktor.io/glossary/data-access-control-rbac-and-abac --- # Data Classification and Tagging Strategies In data architectures built on streaming platforms like Apache Kafka, data classification and tagging are core parts of a data governance framework. As organizations process millions of events per second, understanding what data flows through your systems and how sensitive it is matters for compliance, security, and operational efficiency. ![Data Classification Levels and Tagging](https://www.conduktor.io/assets/images/glossary/data-classification-and-tagging-strategies-0.webp) ## Understanding Data Classification Data classification is the systematic organization of data into categories based on sensitivity, regulatory requirements, and business criticality. This process enables organizations to apply appropriate security controls, access policies, and retention strategies to different data types. For Data Governance Officers and Security Engineers, classification serves multiple purposes: it reduces risk exposure, ensures compliance with regulations like GDPR and CCPA (see [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams)), and optimizes resource allocation by focusing protection efforts where they matter most. ## Classification Levels and Frameworks A well-designed classification framework typically includes four to five levels: - **Public Data**: Information that can be freely shared without risk, such as marketing materials or publicly available product information. - **Internal Data**: Information meant for internal use only, like employee directories or internal communications, which could cause minor inconvenience if exposed. - **Confidential Data**: Sensitive business information such as financial records, strategic plans, or customer data that could cause significant harm if disclosed. - **Restricted Data**: Highly sensitive information including personally identifiable information (PII), protected health information (PHI), payment card data, or trade secrets requiring the highest level of protection. For detailed PII handling strategies, see [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). Some organizations add a fifth tier for regulated data requiring specific compliance controls under frameworks like HIPAA, PCI-DSS, or SOX. ## Tagging Strategies for Streaming Data In streaming architectures, classification metadata must travel with the data itself. Traditional database-centric approaches don't translate directly to event streams, requiring new strategies. ### Message Header Tagging Apache Kafka supports message headers (introduced in Kafka 0.11+, enhanced in Kafka 4.0 with KRaft mode), making them ideal for carrying classification metadata. For comprehensive Kafka header usage patterns, see [Using Kafka Headers Effectively](https://www.conduktor.io/glossary/using-kafka-headers-effectively). Each event can include headers like: ``` classification: CONFIDENTIAL pii-fields: email,phone,ssn retention-policy: 90-days compliance-tags: GDPR,CCPA ``` This approach keeps classification data separate from business payloads while ensuring it remains available to downstream consumers and governance tools. Here's a complete example of how to set classification headers when producing messages in Java: ```java import org.apache.kafka.clients.producer.*; import org.apache.kafka.common.header.Headers; import org.apache.kafka.common.header.internals.RecordHeader; import java.nio.charset.StandardCharsets; import java.util.Properties; public class ClassifiedProducer { public static void main(String[] args) { Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); try (KafkaProducer producer = new KafkaProducer<>(props)) { String topic = "confidential.customer.events"; String key = "customer-123"; String value = "{\"email\":\"user@example.com\",\"phone\":\"+1234567890\"}"; ProducerRecord record = new ProducerRecord<>(topic, key, value); // Add classification headers Headers headers = record.headers(); headers.add(new RecordHeader("classification", "CONFIDENTIAL".getBytes(StandardCharsets.UTF_8))); headers.add(new RecordHeader("pii-fields", "email,phone".getBytes(StandardCharsets.UTF_8))); headers.add(new RecordHeader("retention-policy", "90-days".getBytes(StandardCharsets.UTF_8))); headers.add(new RecordHeader("compliance-tags", "GDPR,CCPA".getBytes(StandardCharsets.UTF_8))); producer.send(record, (metadata, exception) -> { if (exception != null) { exception.printStackTrace(); } else { System.out.printf("Sent classified record to topic %s partition %d offset %d%n", metadata.topic(), metadata.partition(), metadata.offset()); } }); } } } ``` In Python using the confluent-kafka library: ```python from confluent_kafka import Producer def delivery_report(err, msg): if err is not None: print(f'Message delivery failed: {err}') else: print(f'Message delivered to {msg.topic()} [{msg.partition()}] at offset {msg.offset()}') conf = {'bootstrap.servers': 'localhost:9092'} producer = Producer(conf) topic = 'confidential.customer.events' key = 'customer-123' value = '{"email":"user@example.com","phone":"+1234567890"}' # Add classification headers headers = [ ('classification', b'CONFIDENTIAL'), ('pii-fields', b'email,phone'), ('retention-policy', b'90-days'), ('compliance-tags', b'GDPR,CCPA') ] producer.produce( topic=topic, key=key, value=value, headers=headers, callback=delivery_report ) producer.flush() ``` ### Reading and Enforcing Classification Headers Downstream consumers should read classification headers and enforce appropriate handling policies. Here's how to read classification metadata: ```java import org.apache.kafka.clients.consumer.*; import org.apache.kafka.common.header.Header; import java.nio.charset.StandardCharsets; import java.time.Duration; import java.util.Arrays; import java.util.Properties; public class ClassificationAwareConsumer { public static void main(String[] args) { Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("group.id", "classification-aware-consumer"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); try (KafkaConsumer consumer = new KafkaConsumer<>(props)) { consumer.subscribe(Arrays.asList("confidential.customer.events")); while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord record : records) { // Extract classification metadata String classification = getHeaderValue(record, "classification"); String piiFields = getHeaderValue(record, "pii-fields"); String retentionPolicy = getHeaderValue(record, "retention-policy"); System.out.printf("Processing record with classification: %s%n", classification); // Apply appropriate handling based on classification if ("RESTRICTED".equals(classification) || "CONFIDENTIAL".equals(classification)) { // Enforce encryption for sensitive data // Log access for audit trail System.out.println("Applying enhanced security controls for sensitive data"); auditAccess(record.key(), classification, piiFields); // Mask PII fields if needed String processedValue = maskPiiFields(record.value(), piiFields); processMessage(processedValue); } else { // Standard processing for non-sensitive data processMessage(record.value()); } } } } } private static String getHeaderValue(ConsumerRecord record, String headerKey) { Header header = record.headers().lastHeader(headerKey); return header != null ? new String(header.value(), StandardCharsets.UTF_8) : null; } private static void auditAccess(String key, String classification, String piiFields) { // Log to audit system System.out.printf("AUDIT: Accessed %s data [key=%s, pii-fields=%s]%n", classification, key, piiFields); } private static String maskPiiFields(String value, String piiFields) { // Implement field masking logic based on pii-fields return value; // Simplified for example } private static void processMessage(String value) { // Business logic here } } ``` In Python: ```python from confluent_kafka import Consumer, KafkaException import json def get_header_value(headers, key): """Extract header value by key""" if headers: for header_key, header_value in headers: if header_key == key: return header_value.decode('utf-8') if header_value else None return None def audit_access(key, classification, pii_fields): """Log access to sensitive data""" print(f"AUDIT: Accessed {classification} data [key={key}, pii-fields={pii_fields}]") def mask_pii_fields(value, pii_fields): """Mask PII fields based on classification""" # Implement masking logic return value def process_message(value): """Business logic for processing messages""" pass conf = { 'bootstrap.servers': 'localhost:9092', 'group.id': 'classification-aware-consumer', 'auto.offset.reset': 'earliest' } consumer = Consumer(conf) consumer.subscribe(['confidential.customer.events']) try: while True: msg = consumer.poll(timeout=1.0) if msg is None: continue if msg.error(): raise KafkaException(msg.error()) # Extract classification metadata from headers classification = get_header_value(msg.headers(), 'classification') pii_fields = get_header_value(msg.headers(), 'pii-fields') retention_policy = get_header_value(msg.headers(), 'retention-policy') print(f"Processing record with classification: {classification}") # Apply appropriate handling based on classification if classification in ['RESTRICTED', 'CONFIDENTIAL']: # Enforce enhanced security controls audit_access(msg.key().decode('utf-8') if msg.key() else None, classification, pii_fields) # Mask PII if needed processed_value = mask_pii_fields(msg.value().decode('utf-8'), pii_fields) process_message(processed_value) else: # Standard processing process_message(msg.value().decode('utf-8')) except KeyboardInterrupt: pass finally: consumer.close() ``` ### Schema Registry Integration Schema Registry is Confluent's centralized service for managing and validating schemas in Kafka environments. By embedding classification metadata directly in schemas, you ensure that every message using that schema inherits the classification automatically, preventing unclassified data from entering your system. For detailed Schema Registry implementation, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). Confluent Schema Registry supports custom properties that can include: ```json { "name": "CustomerEvent", "type": "record", "metadata": { "classification": "CONFIDENTIAL", "contains_pii": true, "pii_fields": ["email", "phone", "address"] } } ``` Here's a complete example of registering an Avro schema with classification metadata: ```java import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient; import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient; import org.apache.avro.Schema; import org.apache.avro.SchemaBuilder; import java.util.HashMap; import java.util.Map; public class SchemaRegistrationExample { public static void main(String[] args) throws Exception { String schemaRegistryUrl = "http://localhost:8081"; SchemaRegistryClient schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryUrl, 100); // Define Avro schema with classification Schema customerSchema = SchemaBuilder .record("CustomerEvent").namespace("com.example") .fields() .requiredString("customerId") .requiredString("email") .requiredString("phone") .optionalString("address") .endRecord(); // Add classification metadata as schema properties Map properties = new HashMap<>(); properties.put("classification", "CONFIDENTIAL"); properties.put("contains_pii", "true"); properties.put("pii_fields", "email,phone,address"); properties.put("retention_policy", "90-days"); properties.put("compliance_tags", "GDPR,CCPA"); // Register schema with metadata String subject = "customer-events-value"; io.confluent.kafka.schemaregistry.avro.AvroSchema avroSchema = new io.confluent.kafka.schemaregistry.avro.AvroSchema(customerSchema); int schemaId = schemaRegistry.register(subject, avroSchema); System.out.println("Registered schema with ID: " + schemaId); // Optionally, update subject-level metadata schemaRegistry.updateCompatibility(subject, "BACKWARD"); } } ``` Using the Schema Registry REST API to register a schema with metadata: ```bash curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \ --data '{ "schemaType": "AVRO", "schema": "{\"type\":\"record\",\"name\":\"CustomerEvent\",\"namespace\":\"com.example\",\"fields\":[{\"name\":\"customerId\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"},{\"name\":\"phone\",\"type\":\"string\"},{\"name\":\"address\",\"type\":[\"null\",\"string\"],\"default\":null}]}", "metadata": { "properties": { "classification": "CONFIDENTIAL", "contains_pii": "true", "pii_fields": "email,phone,address", "retention_policy": "90-days", "compliance_tags": "GDPR,CCPA" } } }' \ http://localhost:8081/subjects/customer-events-value/versions ``` ### Topic-Level Classification Implementing a topic naming convention that includes classification information provides immediate visibility. For example: - `public.web.clickstream` - `confidential.customer.profile-updates` - `restricted.payment.transactions` This strategy enables quick identification and allows security tools to apply policies based on topic patterns. ### Comparison of Classification Approaches Different classification strategies serve different use cases. Here's when to use each: | Approach | Best For | Pros | Cons | |----------|----------|------|------| | **Message Headers** | Dynamic, event-level classification | Flexible, no schema changes needed, per-message granularity | Adds network overhead, requires consumer awareness | | **Schema Registry Metadata** | Schema-level enforcement | Centralized governance, prevents unclassified data, integrates with tooling | Less flexible, requires Schema Registry, all messages share classification | | **Topic Naming** | High-level organization | Immediate visibility, simple to implement, works with all tools | Coarse-grained, requires topic proliferation, hard to change | | **Combination** | Enterprise environments | Comprehensive coverage, defense in depth | More complexity to manage | Most production environments use a combination: topic naming for broad organization, Schema Registry for enforcement, and headers for event-specific nuances. ## Modern Governance Tools and Technologies (2025) The data governance landscape has evolved significantly, with new tools and platforms specifically designed for streaming data classification and governance. ### Conduktor Stream Governance Conduktor provides comprehensive stream governance features that integrate classification directly into the data streaming workflow: - **Stream Catalog**: Browse and manage your data streams with topic-level metadata - **Data Quality Rules**: Define and enforce classification-based quality rules - **Business Metadata**: Attach business context to technical classifications ```bash # Example: Setting classification using Confluent CLI (Kafka 4.0+) confluent schema-registry schema create \ --subject customer-events-value \ --schema-file customer.avsc \ --metadata '{"classification":"CONFIDENTIAL","pii_fields":["email","phone"]}' ``` ### AI-Powered Classification and PII Detection Machine learning-based tools now automate classification decisions by analyzing data patterns: **Automated PII Detection Tools (2025)**: - **AWS Macie for Kafka**: Integrates with Amazon MSK to automatically detect PII in streaming data - **Azure Purview with Event Hubs**: Provides ML-based classification for Azure event streams - **Google Cloud DLP API**: Real-time scanning of Kafka messages for sensitive data patterns - **Open-source alternatives**: Presidio (Microsoft), Pilar (AWS Labs) Example using AWS Macie patterns in a Kafka consumer: ```python import boto3 from confluent_kafka import Consumer # Initialize AWS Macie client for PII detection macie_client = boto3.client('macie2', region_name='us-east-1') def detect_pii_with_ml(message_content): """Use ML to detect PII in message content""" # This is a conceptual example - actual implementation would use # AWS Macie custom data identifiers or similar ML models response = macie_client.test_custom_data_identifier( regex='[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}', sampleText=message_content ) return response['matchCount'] > 0 def auto_classify_message(message): """Automatically classify message based on content analysis""" contains_pii = detect_pii_with_ml(message) if contains_pii: return { 'classification': 'CONFIDENTIAL', 'auto_detected': 'true', 'pii_detected': 'true', 'detection_method': 'ML' } return {'classification': 'INTERNAL'} ``` ### Data Catalogs and Metadata Management Modern data catalogs provide centralized classification management across streaming and batch systems: **Open-Source Solutions**: - **OpenMetadata** (2025): Native Kafka integration, automated lineage tracking, ML-based classification suggestions - **DataHub** (LinkedIn): Kafka Connect integration, real-time metadata updates, classification propagation - **Amundsen** (Lyft): Search-first catalog with Kafka stream discovery **Commercial Solutions**: - **Atlan**: Real-time classification sync with Kafka, automated compliance workflows - **Collibra**: Enterprise-grade governance with Kafka stream cataloging - **Alation**: Active metadata intelligence for streaming platforms These tools sync classification metadata bidirectionally with Kafka, ensuring consistency between your streaming platform and enterprise governance systems. ### Field-Level Encryption for Classified Data For highly sensitive data, modern Kafka platforms support field-level encryption that integrates with classification tags: ```java import io.confluent.kafka.schemaregistry.encryption.FieldEncryptionExecutor; import io.confluent.kafka.serializers.KafkaAvroSerializer; // Configure field-level encryption based on classification Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("schema.registry.url", "http://localhost:8081"); // Enable field-level encryption for PII fields props.put("rule.executors", "encrypt"); props.put("rule.executors.encrypt.class", "io.confluent.kafka.schemaregistry.encryption.FieldEncryptionExecutor"); props.put("rule.executors.encrypt.param.kek.id", "my-kek"); props.put("rule.executors.encrypt.param.kms.type", "aws-kms"); props.put("rule.executors.encrypt.param.kms.key.id", "arn:aws:kms:..."); // Messages with classification=RESTRICTED will automatically encrypt PII fields KafkaProducer producer = new KafkaProducer<>(props); ``` This approach ensures that even if data is intercepted or improperly accessed, sensitive fields remain encrypted based on their classification level. ### Data Contracts and Classification Data contracts (an emerging pattern in 2025) formalize the agreement between data producers and consumers, including classification requirements: ```yaml # data-contract.yaml for customer-events topic domain: customer name: customer-events version: 2.1.0 classification: level: CONFIDENTIAL sensitivity_fields: - email: PII - phone: PII - customer_id: PII retention: 90_days compliance: [GDPR, CCPA] schema: type: avro specification: customer-event-v2.avsc quality: - field: email rule: valid_email_format - field: phone rule: valid_phone_format owners: - team: customer-data-platform contact: customer-data@company.com ``` Tools like **Confluent Schema Registry with Data Contracts** or **Soda Core** can validate that producers honor these classification commitments. For detailed contract implementation, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). ### Classification in Stream Processing Pipelines When data flows through stream processing frameworks like Kafka Streams or Apache Flink, classification tags should propagate automatically. Modern frameworks (2025) support classification-aware processing: **Kafka Streams with Classification Propagation** (Kafka 4.0+): ```java StreamsBuilder builder = new StreamsBuilder(); KStream stream = builder.stream("customer-events"); // Classification metadata propagates through transformations KStream enriched = stream .mapValues((key, value) -> { // Transform data return enrichEvent(value); }) // Headers (including classification) automatically propagate .filter((key, value) -> value.isValid()); // Write to output topic with original classification preserved enriched.to("enriched-customer-events"); ``` **Apache Flink with Classification Handling** (Flink 1.18+): ```java DataStream stream = env .addSource(new FlinkKafkaConsumer<>("customer-events", schema, props)); // Preserve classification metadata through transformations stream .map(new ClassificationAwareMapFunction()) .addSink(new FlinkKafkaProducer<>( "processed-customer-events", new ClassificationPreservingSerializer(), props )); ``` For more on stream processing patterns that preserve metadata, see [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams) and [What is Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). ## Practical Implementation Strategies ### Start with Data Discovery Before classifying data, you need to know what you have. Implement automated scanning tools that inspect message payloads and schemas to identify potential PII, financial data, or other sensitive information. Many organizations discover sensitive data flowing through systems they believed contained only operational metrics. For comprehensive data discovery approaches, see [Building a Business Glossary for Data Governance](https://www.conduktor.io/glossary/building-a-business-glossary-for-data-governance). ### Establish Clear Ownership Every topic and data stream should have a designated data owner responsible for classification decisions. This accountability ensures classifications remain accurate as data evolves and prevents the "classify everything as confidential" problem that reduces the framework's effectiveness. ### Automate Classification Manual classification doesn't scale in streaming environments. Leverage schema validation, producer interceptors, and governance platforms to automatically tag data based on content patterns, source systems, and field names. ### Implement Progressive Controls Apply security controls proportional to classification levels. Public data might require only basic access logging, while restricted data demands encryption at rest and in transit, strict access controls, audit trails, and limited retention periods. For comprehensive security patterns, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) and [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming). ### Regular Classification Reviews Data sensitivity changes over time. Customer emails might become public after opt-in to marketing lists. Financial projections become less sensitive after quarterly earnings releases. Schedule regular reviews to ensure classifications remain appropriate. ## Integrating with Enterprise Security Classification and tagging strategies must integrate with broader security infrastructure to provide defense-in-depth. Connect Kafka classification metadata to: - **Identity and Access Management (IAM)**: Use classification tags to drive Role-Based Access Control (RBAC) policies in Kafka ACLs (Access Control Lists) and authorization systems. For example, only users with "restricted-data-access" role can consume from topics tagged as RESTRICTED. See [Data Access Control RBAC and ABAC](https://www.conduktor.io/glossary/data-access-control-rbac-and-abac) for implementation patterns. - **Data Loss Prevention (DLP)**: Feed classification metadata to DLP systems for monitoring egress points and preventing unauthorized data exfiltration. DLP tools can automatically block or quarantine messages with CONFIDENTIAL or RESTRICTED classifications from being sent to unauthorized destinations. - **Security Information and Event Management (SIEM) platforms**: Include classification context in security event logs for better threat detection and forensics. When a security incident occurs, knowing the classification level of accessed data helps prioritize response efforts. For audit logging implementation, see [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). - **Data catalogs**: Synchronize classification information with enterprise data catalogs for unified governance across streaming and batch systems. This ensures analysts and data scientists see consistent classification whether accessing Kafka streams or data warehouse tables. See [What is a Data Catalog](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery) for more details. - **Encryption systems**: Automatically apply encryption policies based on classification levels. RESTRICTED data might require field-level encryption, while CONFIDENTIAL data needs encryption in transit. See [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka) for comprehensive encryption strategies. ## Conclusion Data classification and tagging are the foundation of governance in streaming architectures. By implementing structured classification frameworks and using Kafka's native capabilities like message headers and Schema Registry, organizations can maintain security and compliance without sacrificing streaming performance. The key is automation, clear ownership, and integration with existing security infrastructure. Start with critical data flows and expand systematically. With proper classification and tagging, your organization gains visibility, reduces risk, and builds trust in your data platform. ## Related Concepts - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing schemas with classification metadata - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) - Organizational structures for classification governance - [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) - Tracking access to classified data ## Sources and References 1. **Apache Kafka Documentation - Message Headers**: [Kafka Record Headers](https://kafka.apache.org/documentation/) - Official documentation on implementing message headers in Apache Kafka for metadata propagation. 2. **Confluent Schema Registry Documentation**: [Schema Registry Overview](https://docs.confluent.io/platform/current/schema-registry/index.html) - Comprehensive guide on using Schema Registry for schema management and metadata. 3. **NIST Special Publication 800-122**: [Guide to Protecting the Confidentiality of Personally Identifiable Information (PII)](https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-122.pdf) - Federal guidelines for PII classification and protection strategies. 4. **GDPR Article 32 - Security of Processing**: [EU General Data Protection Regulation](https://gdpr-info.eu/art-32-gdpr/) - European Union requirements for implementing appropriate technical measures for data classification and security. 5. **OWASP Data Classification Guide**: [OWASP Application Security Verification Standard](https://owasp.org/www-project-application-security-verification-standard/) - Industry best practices for data classification in application security contexts. Source: https://www.conduktor.io/glossary/data-classification-and-tagging-strategies --- # Data Contracts for Reliable Pipelines Data pipelines break when producers change data formats without warning consumers. A missing field, a type change from integer to string, or an unexpected null value can cascade through downstream systems, causing processing failures, incorrect analytics, and broken dashboards. Data contracts formalize the agreement between data producers and consumers, defining not just the structure of data, but also its quality, semantics, and ownership. This article explores how data contracts work, why they matter for streaming pipelines, and how to implement them effectively. ![Data Contract Components and Flow](https://www.conduktor.io/assets/images/glossary/data-contracts-for-reliable-pipelines-0.webp) ## What Are Data Contracts? A data contract is a formal specification that defines: - **Schema**: The structure, field names, and data types - **Semantics**: What each field means and how it should be used - **Quality expectations**: Constraints like nullability, ranges, and uniqueness - **SLAs**: Timeliness, completeness, and availability guarantees - **Ownership**: Who produces the data and who to contact for changes Unlike informal documentation that quickly becomes outdated, data contracts are enforced programmatically. They shift data quality left, catching breaking changes before they reach production pipelines. Consider an e-commerce platform where the order service publishes events to Kafka. Without a contract, the service might change `order_total` from an integer (cents) to a float (dollars), breaking every downstream consumer that expects integer arithmetic. A data contract would flag this incompatible change during development. ## Core Components of a Data Contract ### Schema Definition The foundation is a machine-readable schema. For streaming systems, this typically means Avro, Protobuf, or JSON Schema. For a detailed comparison of these formats, see [Avro vs Protobuf vs JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema). These formats support: - Primitive types (string, integer, boolean, etc.) - Complex types (nested objects, arrays, maps) - Optional vs. required fields - Default values Example Avro schema for an order event: ```json { "type": "record", "name": "OrderCreated", "namespace": "com.example.orders", "fields": [ {"name": "order_id", "type": "string"}, {"name": "customer_id", "type": "string"}, {"name": "order_total_cents", "type": "long"}, {"name": "created_at", "type": "long", "logicalType": "timestamp-millis"}, {"name": "currency", "type": "string", "default": "USD"} ] } ``` ### Quality Rules Beyond structure, contracts specify constraints: - `order_total_cents` must be positive - `customer_id` must match UUID format - `created_at` must be within the last 24 hours (detecting replay issues) - Required fields cannot be null These rules can be validated at write time (producer validation) or read time (consumer validation). For comprehensive coverage of quality dimensions, see [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). ### Compatibility Modes Contracts define how schemas can evolve. Confluent Schema Registry supports multiple modes: - **Backward compatibility**: New schema can read old data (safe to add optional fields with defaults) - Example: Adding `optional string payment_method = "credit_card"` works because old records will use the default - Most common mode for consumer-driven evolution - **Forward compatibility**: Old schema can read new data (safe to remove optional fields) - Example: Removing an optional field that consumers ignore - Useful when producers evolve faster than consumers - **Full compatibility**: Both backward and forward (most restrictive but safest) - Only optional field additions with defaults allowed - Recommended for critical business data with many consumers - **None**: No compatibility checks (dangerous for production) - Use only for development or isolated topics with single consumer ### Ownership and Documentation Contracts should specify: - Team or service responsible for the data - Contact information for questions - Business meaning of fields - Known limitations or edge cases For enterprise-scale governance of data contracts, see [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities). ## Data Contracts in Streaming Pipelines Streaming systems like Apache Kafka and Apache Flink are prime candidates for data contracts. Unlike batch systems where you can inspect data after the fact, streaming pipelines process data continuously. A breaking change can corrupt state stores, trigger incorrect alerts, or cause cascading failures across multiple consumers. ### Schema Registry Integration Schema Registry acts as the contract enforcement layer. For detailed implementation guidance, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). When a Kafka producer serializes a message: 1. Producer checks if the schema is registered 2. If new, Schema Registry validates compatibility with previous versions 3. If compatible, schema is registered with a new version ID 4. Message includes schema ID in the payload 5. Consumer uses schema ID to deserialize correctly This prevents incompatible changes from reaching Kafka topics. A producer attempting to publish an incompatible schema receives an immediate error. ### Stream Processing Validation Flink and Kafka Streams applications can validate data against contracts at runtime. Modern implementations use side outputs to route invalid records to dead letter queues: ```java // Flink 1.18+ example: validate order events with dead letter queue OutputTag invalidOrders = new OutputTag("invalid-orders"){}; SingleOutputStreamOperator validOrders = env .addSource(new FlinkKafkaConsumer<>("orders", ConfluentRegistryAvroDeserializationSchema.forSpecific(Order.class, schemaRegistryUrl), properties)) .process(new ProcessFunction() { @Override public void processElement(Order order, Context ctx, Collector out) { // Validate quality rules if (order.getOrderTotal() <= 0) { ctx.output(invalidOrders, order); // Route to DLQ return; } if (order.getCustomerId() == null || order.getCustomerId().isEmpty()) { ctx.output(invalidOrders, order); return; } out.collect(order); // Valid record } }); // Route invalid records to dead letter topic validOrders.getSideOutput(invalidOrders) .addSink(new FlinkKafkaProducer<>("orders-dlq", ...)); ``` This pattern prevents pipeline crashes while preserving invalid data for investigation. For detailed error handling patterns, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). ### Multi-Consumer Coordination Streaming topics often have many consumers: analytics databases, machine learning pipelines, microservices, monitoring systems. Each consumer depends on the contract. When a producer needs to make a breaking change, the contract provides a coordination mechanism: 1. Propose the new schema version 2. Identify all consumers (via contract metadata) 3. Coordinate migration timeline 4. Deploy backward-compatible intermediate version if possible 5. Migrate consumers before removing old fields Without contracts, this coordination happens ad-hoc through incidents and debugging sessions. ## Implementing Data Contracts ### Choose a Schema Format For streaming systems, Avro offers the best balance of performance, tooling, and schema evolution support. Protobuf is popular in polyglot environments. JSON Schema works for less performance-critical applications. Beyond schema formats, modern data contract platforms provide comprehensive contract management: **Schema-First Approach (2025):** - **Confluent Schema Registry + Schema Linking**: Manages schemas across multiple Kafka clusters with bi-directional replication - **Open Data Contract Standard (ODCS)**: YAML-based specification defining schemas, quality rules, SLAs, and ownership in one document - **Bitol**: Data contract specification language with built-in validation and code generation **Data Quality Platforms (2025):** - **Soda Core**: Open-source framework for defining data quality checks as code, integrates with streaming pipelines - **Great Expectations**: Provides expectations (assertions) on data, with Kafka integration via custom data sources - **Monte Carlo**: Data observability platform with ML-driven anomaly detection for contract violations - **Atlan**: Data catalog with contract enforcement and lineage tracking across streaming and batch systems These tools shift from pure schema validation to comprehensive data quality contracts covering completeness, freshness, distribution, and custom business rules. For practical implementation patterns, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework). ### Establish Governance Process Define who can change schemas, how changes are reviewed, and what compatibility modes are required for different data tiers: - Critical business data: Full compatibility - Internal service communication: Backward compatibility - Experimental features: No compatibility guarantees ### Automate Validation Integrate schema validation into CI/CD pipelines: ```bash # Confluent Schema Registry: validate compatibility before deployment curl -X POST http://localhost:8081/compatibility/subjects/orders-value/versions/latest \ -H "Content-Type: application/vnd.schemaregistry.v1+json" \ -d @order_created.avsc # Or using Confluent CLI (requires Confluent Platform or Cloud) confluent schema-registry schema create --subject orders-value \ --schema order_created.avsc \ --compatibility BACKWARD ``` ### Monitor Contract Violations Track metrics on: - Schema validation failures - Deserialization errors - Quality rule violations - Consumer lag spikes (often caused by malformed data) For comprehensive monitoring approaches, see [What is Data Observability: The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars). Modern schema management platforms provide visibility into schema usage across Kafka clusters: - **Confluent Schema Registry**: The industry standard, provides REST API for schema storage, compatibility checking, and version management - **Karapace**: Open-source alternative compatible with Confluent's API, popular for self-hosted deployments - **AWS Glue Schema Registry**: Native AWS integration with automatic schema discovery and evolution tracking - **Apicurio Registry**: Open-source registry supporting multiple formats (Avro, Protobuf, JSON Schema) with role-based access control These platforms show which consumers use which schema versions, enabling coordinated migrations before breaking changes. They support data quality rules that validate message contents against business logic, catching issues like negative order totals or invalid email formats. For implementing data quality enforcement with Conduktor, see [Data Quality Policies](https://docs.conduktor.io/guide/conduktor-concepts/data-quality-policies) and [Enforcing Data Quality](https://docs.conduktor.io/guide/use-cases/enforce-data-quality). ## Benefits and Challenges ### Benefits - **Prevents production incidents**: Breaking changes are caught before deployment - **Enables self-service**: Consumers can discover and trust data without direct producer coordination - **Documents data semantics**: Schema serves as living documentation - **Supports compliance**: Contracts can enforce PII handling, retention policies, and data classification ### Challenges - **Organizational change**: Requires buy-in from producers who may see contracts as overhead - **Initial setup cost**: Defining schemas for existing data can be time-consuming - **Evolution complexity**: Managing compatibility across dozens of consumers requires coordination - **Performance overhead**: Schema validation adds latency (typically <1ms, but matters for high-throughput systems) ## Best Practices - **Start with critical data**: Implement contracts for your most important topics first - **Make schemas discoverable**: Use a schema registry UI or catalog to browse available data - **Version everything**: Even if you think a field will never change, version it - **Test compatibility**: Include schema compatibility tests in your CI pipeline - **Document semantics, not just structure**: Explain what `order_total_cents` means, don't just specify it's a long - **Plan for evolution**: Design schemas with optional fields and defaults to enable backward-compatible additions. See [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) for detailed guidance - **Establish clear ownership**: Every schema should have a responsible team ## Summary Data contracts transform data pipelines from fragile, undocumented systems into reliable, self-service platforms. Formalizing agreements between producers and consumers prevents breaking changes, enables safe schema evolution, and provides living documentation. For streaming pipelines built on Kafka, Flink, and similar technologies, contracts are particularly valuable. Errors cascade quickly in streaming, and the multi-consumer pattern means a single change can affect many systems simultaneously. Implementing data contracts requires organizational commitment and tooling investment, but the payoff is real: fewer production incidents, faster development cycles, and greater trust in data across the organization. ## Related Concepts - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Core infrastructure for enforcing data contracts - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) - Quality expectations defined in contracts - [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) - Systematic approach to contract validation ## Sources and References 1. Chad Sanderson - "Data Contracts: The Foundation of Data Mesh" - https://dataproducts.substack.com/ 2. Confluent Documentation - "Schema Registry Overview" - https://docs.confluent.io/platform/current/schema-registry/ 3. Andrew Jones - "Designing Data Contracts" - https://www.datamesh-architecture.com/ 4. Gwen Shapira - "Kafka: The Definitive Guide" (O'Reilly) - Schema management and compatibility 5. Martin Kleppmann - "Designing Data-Intensive Applications" (O'Reilly) - Schema evolution and data modeling Source: https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines --- # Data Drift in Streaming: Detecting and Managing Unexpected Changes In streaming data systems, the assumption that data maintains consistent patterns and structures over time is rarely true. Data drift — where data characteristics change unexpectedly — creates real challenges for real-time pipelines and machine learning models. Understanding and managing drift is essential for reliable streaming applications. ![Types of Data Drift](https://www.conduktor.io/assets/images/glossary/data-drift-in-streaming-0.webp) ## Understanding Types of Drift Data drift is an umbrella term that encompasses several distinct types of changes, each with different implications for streaming systems. - **Schema drift** occurs when the structure of your data changes. A new field appears in your event schema, an existing field's type changes from integer to string, or a required field becomes optional. In streaming systems, schema drift can break downstream consumers that expect a specific structure. For example, if a payment event suddenly includes a new `currency_code` field, consumers that parse these events may fail if they're not designed to handle unexpected fields. - **Data drift** (sometimes called statistical drift) refers to changes in the statistical properties of your data, the distribution, range, or patterns within the data itself. The same schema might persist, but the values flowing through your pipeline shift. Average transaction amounts might increase, user activity patterns might change, or the ratio between different event types might evolve. This type of drift is particularly insidious because it often goes undetected by schema validation alone. - **Concept drift** affects machine learning models specifically. It occurs when the relationship between input features and target predictions changes over time. A fraud detection model trained on historical patterns may degrade as fraudsters adapt their tactics. The model's inputs (transaction features) might follow the same schema and distributions, but their predictive relationship to fraud has fundamentally changed. For comprehensive coverage of ML-specific drift challenges, see [Model Drift in Streaming](https://www.conduktor.io/glossary/model-drift-in-streaming). ## Root Causes of Drift in Streaming Drift emerges from various sources, many of which are inherent to evolving business systems. - **Upstream system changes** are the most common culprit. When a microservice updates its event schema without coordination, or when a new version of a mobile app starts sending additional fields, downstream streaming pipelines must adapt. These changes often happen without warning, especially in distributed organizations where teams operate independently. - **New data sources** introduce variation when integrated into existing pipelines. Merging events from a newly acquired company, adding a third-party data feed, or incorporating IoT sensor data can shift overall data distributions. Each source brings its own quirks, formats, and value ranges. - **Business evolution** naturally causes drift. Seasonal patterns change market behavior, new product features alter user engagement, and regulatory changes force schema modifications. A subscription service expanding to new countries might see payment methods, currencies, and purchase patterns shift dramatically. - **Bugs and data quality issues** create unexpected drift. A faulty sensor might start reporting extreme values, a code bug might corrupt certain fields, or a misconfigured producer might send malformed events. Unlike intentional changes, these issues require immediate detection and remediation. ## Detecting Drift in Real-Time Systems Early detection is critical because drift's impact compounds over time. Streaming systems require continuous monitoring to catch drift before it cascades into downstream failures. - **Schema validation** provides the first line of defense. Tools like Schema Registry (Confluent, AWS Glue) enforce contracts between producers and consumers. When a producer attempts to publish an event that violates the registered schema, the system rejects it immediately. This prevents incompatible data from entering your pipeline but only catches structural drift, not statistical or conceptual changes. - **Statistical monitoring** tracks data distributions over time. Calculating metrics like mean, median, standard deviation, and percentile distributions for numerical fields helps identify when values shift from historical norms. For example, if your transaction amount field historically averages $47 with a standard deviation of $22, but recent data shows an average of $89 with standard deviation $45, this signals significant drift requiring investigation. For categorical fields, monitoring the frequency distribution of different values reveals when rare categories become common or vice versa. - **Automated drift detection tests** compare current data windows against baseline distributions using statistical hypothesis testing: - **Kolmogorov-Smirnov (K-S) test**: Compares continuous distributions (e.g., transaction amounts, response times). Use when you need to detect any change in the overall shape of the distribution. Threshold: p-value < 0.05 typically indicates significant drift. - **Chi-squared test**: Detects shifts in categorical distributions (e.g., status codes, product categories). Ideal for comparing frequency distributions across categories. Works best with sufficient sample sizes in each category. - **Population Stability Index (PSI)**: Quantifies the shift between two distributions using a single score. PSI < 0.1 indicates minimal drift, 0.1-0.25 indicates moderate drift requiring investigation, PSI > 0.25 signals significant drift demanding immediate action. When statistical tests exceed predefined thresholds, alerts trigger for investigation. Here's a practical example of implementing drift detection in Python: ```python import numpy as np from scipy import stats from typing import Dict, List, Tuple from dataclasses import dataclass from enum import Enum class DriftSeverity(Enum): NONE = "none" MODERATE = "moderate" SIGNIFICANT = "significant" @dataclass class DriftResult: field: str severity: DriftSeverity metric: str value: float threshold: float message: str class StreamingDriftDetector: def __init__(self, baseline_data: List[float]): """Initialize with baseline distribution from training/historical data""" self.baseline_data = np.array(baseline_data) self.baseline_mean = np.mean(baseline_data) self.baseline_std = np.std(baseline_data) def detect_ks_drift(self, current_data: List[float], threshold: float = 0.05) -> DriftResult: """Kolmogorov-Smirnov test for continuous distributions""" statistic, p_value = stats.ks_2samp(self.baseline_data, current_data) if p_value < threshold: severity = DriftSeverity.SIGNIFICANT message = f"Significant distribution shift detected (p={p_value:.4f})" else: severity = DriftSeverity.NONE message = f"No significant drift (p={p_value:.4f})" return DriftResult( field="continuous_field", severity=severity, metric="ks_test_p_value", value=p_value, threshold=threshold, message=message ) def detect_psi_drift(self, current_data: List[float], n_bins: int = 10) -> DriftResult: """Population Stability Index for distribution shift""" # Create bins based on baseline data _, bin_edges = np.histogram(self.baseline_data, bins=n_bins) # Calculate distributions baseline_counts, _ = np.histogram(self.baseline_data, bins=bin_edges) current_counts, _ = np.histogram(current_data, bins=bin_edges) # Convert to proportions baseline_props = baseline_counts / len(self.baseline_data) current_props = current_counts / len(current_data) # Calculate PSI (avoid division by zero) psi = 0 for baseline_prop, current_prop in zip(baseline_props, current_props): if baseline_prop > 0 and current_prop > 0: psi += (current_prop - baseline_prop) * np.log(current_prop / baseline_prop) # Interpret PSI if psi > 0.25: severity = DriftSeverity.SIGNIFICANT message = f"Significant drift (PSI={psi:.4f}) - immediate action required" elif psi > 0.1: severity = DriftSeverity.MODERATE message = f"Moderate drift (PSI={psi:.4f}) - investigation recommended" else: severity = DriftSeverity.NONE message = f"Minimal drift (PSI={psi:.4f})" return DriftResult( field="continuous_field", severity=severity, metric="psi", value=psi, threshold=0.1, message=message ) def detect_summary_stats_drift(self, current_data: List[float], std_threshold: float = 2.0) -> List[DriftResult]: """Detect drift using summary statistics (mean, std deviation)""" current_mean = np.mean(current_data) current_std = np.std(current_data) results = [] # Check if mean has drifted significantly (more than N standard deviations) mean_shift = abs(current_mean - self.baseline_mean) / self.baseline_std if mean_shift > std_threshold: results.append(DriftResult( field="continuous_field", severity=DriftSeverity.SIGNIFICANT, metric="mean_shift", value=mean_shift, threshold=std_threshold, message=f"Mean shifted from {self.baseline_mean:.2f} to {current_mean:.2f} " f"({mean_shift:.2f} standard deviations)" )) # Check if variance has changed significantly variance_ratio = current_std / self.baseline_std if self.baseline_std > 0 else 0 if variance_ratio > 1.5 or variance_ratio < 0.67: severity = DriftSeverity.MODERATE if variance_ratio < 2.0 else DriftSeverity.SIGNIFICANT results.append(DriftResult( field="continuous_field", severity=severity, metric="std_ratio", value=variance_ratio, threshold=1.5, message=f"Standard deviation changed from {self.baseline_std:.2f} " f"to {current_std:.2f} (ratio={variance_ratio:.2f})" )) if not results: results.append(DriftResult( field="continuous_field", severity=DriftSeverity.NONE, metric="summary_stats", value=mean_shift, threshold=std_threshold, message="Summary statistics within expected ranges" )) return results # Example usage: Monitoring transaction amounts baseline_transactions = [45.20, 52.10, 38.90, 61.50, 44.30, 55.80, 49.70, 42.60, 58.90, 47.20] * 100 # 1000 baseline samples current_transactions = [87.40, 95.30, 102.50, 78.90, 91.20, 88.60, 93.80, 85.70, 97.10, 90.40] * 100 # 1000 current samples detector = StreamingDriftDetector(baseline_transactions) # Run drift detection ks_result = detector.detect_ks_drift(current_transactions) psi_result = detector.detect_psi_drift(current_transactions) stats_results = detector.detect_summary_stats_drift(current_transactions) print(f"K-S Test: {ks_result.message}") print(f"PSI: {psi_result.message}") for result in stats_results: print(f"Summary Stats: {result.message}") ``` **Model performance monitoring** specifically targets concept drift. Track prediction accuracy, precision, recall, and other relevant metrics on a continuous basis. When a fraud detection model's false positive rate suddenly increases or a recommendation engine's click-through rate drops, concept drift is likely occurring. ## Impact on Streaming Pipelines Drift manifests in streaming systems through various failure modes, each with different severity and visibility. - **Broken consumers** are the most visible impact. When schema drift introduces incompatible changes, consuming applications crash, throw exceptions, or stall. A consumer expecting an integer `user_id` field will fail when it encounters a string UUID. These failures are immediate and obvious but disruptive to operations. - **Silent data corruption** is more dangerous because it goes unnoticed. When a consumer can technically parse drifted data but misinterprets it, corrupt data flows downstream. A percentage field changing from 0-100 scale to 0-1 scale might be read successfully but produce incorrect calculations. These errors accumulate in data warehouses, dashboards, and reports before anyone notices. - **Failed processing jobs** occur when assumptions about data quality break down. Aggregations produce unexpected results, joins fail to find matches due to key format changes, or filtering logic becomes irrelevant. A streaming job that filters events where `status = "completed"` will miss events if the upstream system changes the value to `status = "COMPLETED"`. - **Cascading failures** happen when drift in one component triggers failures in dependent systems. A recommendation service experiencing concept drift might generate poor suggestions, leading to lower user engagement, which in turn affects analytics dashboards, A/B testing frameworks, and business metrics, all depending on the same degraded data source. ## Impact on Machine Learning Models For ML models consuming streaming data, drift directly affects prediction quality and business outcomes. - **Model degradation** occurs gradually as concept drift widens the gap between training data and production reality. A model trained on pre-pandemic e-commerce behavior would struggle with post-pandemic patterns. Without retraining, prediction accuracy erodes, and business value diminishes. - **Feature distribution shifts** cause data drift to affect model inputs. If a model expects average transaction values around $50 but suddenly receives values around $500 due to market changes, predictions become unreliable even if the underlying concept remains stable. Many models are sensitive to input ranges that deviate from training distributions. - **Retraining triggers** must balance responsiveness against stability. Retrain too frequently and you chase noise; retrain too infrequently and drift degrades performance. Automated retraining pipelines use drift detection metrics to trigger updates when distributions shift beyond acceptable thresholds, ensuring models stay current without constant churn. ## Prevention Through Governance Preventing drift requires organizational processes and technical controls working in concert. - **Schema registries** enforce contracts between data producers and consumers. By centralizing schema definitions and validating compatibility rules (backward, forward, or full compatibility), registries prevent breaking changes from reaching production. Producers must evolve schemas following compatibility guidelines, ensuring consumers continue functioning as schemas change. - **Producer-consumer contracts** establish explicit agreements about data formats, value ranges, required fields, and evolution policies. These contracts, whether formal (like Protobuf definitions) or documented (like API specifications), create shared expectations. When changes are necessary, contract owners coordinate migrations rather than surprising downstream teams. For implementation guidance, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). - **Governance policies** define how data can evolve. Data governance platforms enable organizations to enforce validation rules, approval workflows for schema changes, and quality checks before data reaches production streams. Policies might require backward compatibility for all schema changes, mandate documentation for new fields, or restrict who can modify critical event types. - **Testing in non-production environments** validates changes before they reach production streams. Staging environments that mirror production topologies allow teams to test schema migrations, verify consumer compatibility, and assess performance impacts. Canary deployments gradually roll out changes while monitoring for drift-related issues. ## Remediation Strategies When drift occurs despite prevention efforts, effective remediation minimizes impact. - **Backward compatibility** allows old consumers to continue functioning while new consumers leverage enhanced schemas. Adding optional fields with default values, widening field types (integer to long), or maintaining deprecated fields during migration periods keeps systems operational during transitions. - **Versioning strategies** enable multiple schema versions to coexist. Topic naming conventions (like `payment-events-v2`) or version fields within messages allow producers to migrate gradually while consumers update on their own schedules. This decoupling prevents forcing coordinated deployments across all services. - **Graceful degradation** designs consumers to handle unexpected data. Rather than failing on unknown fields, consumers ignore them. When expected fields are missing, they use sensible defaults or skip processing. When types mismatch, they log warnings and continue processing valid records. This resilience prevents complete failures from drift. - **Dead letter queues** capture problematic events for later analysis and reprocessing. When drift causes processing failures, routing failed events to a separate queue preserves them for investigation. Teams can fix issues, update consumers, and replay failed events once systems are drift-compatible. For detailed coverage of error handling patterns, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). Here's an example of implementing drift-aware error handling with dead letter queues: ```python from kafka import KafkaConsumer, KafkaProducer import json import logging from typing import Dict, Any, Optional logger = logging.getLogger(__name__) class DriftAwareConsumer: def __init__(self): self.consumer = KafkaConsumer( 'payment-events', bootstrap_servers=['localhost:9092'], value_deserializer=lambda m: json.loads(m.decode('utf-8')), auto_offset_reset='latest', enable_auto_commit=False ) self.dlq_producer = KafkaProducer( bootstrap_servers=['localhost:9092'], value_serializer=lambda m: json.dumps(m).encode('utf-8') ) # Expected schema baseline self.expected_fields = {'payment_id', 'amount', 'currency', 'status'} self.expected_types = { 'payment_id': str, 'amount': (int, float), 'currency': str, 'status': str } def validate_schema(self, event: Dict[str, Any]) -> Optional[str]: """Validate event schema and return error message if invalid""" # Check for missing required fields missing_fields = self.expected_fields - set(event.keys()) if missing_fields: return f"Schema drift: Missing required fields {missing_fields}" # Check field types for field, expected_type in self.expected_types.items(): if field in event: if not isinstance(event[field], expected_type): actual_type = type(event[field]).__name__ expected = expected_type.__name__ if not isinstance(expected_type, tuple) \ else f"({', '.join(t.__name__ for t in expected_type)})" return f"Schema drift: Field '{field}' has type {actual_type}, expected {expected}" return None def send_to_dlq(self, event: Dict[str, Any], error: str, topic: str): """Send failed event to dead letter queue with enriched metadata""" dlq_message = { 'original_event': event, 'error': error, 'source_topic': topic, 'timestamp': event.get('timestamp', 'unknown'), 'drift_type': 'schema_drift' if 'Schema drift' in error else 'data_quality' } self.dlq_producer.send('payment-events-dlq', value=dlq_message) logger.warning(f"Event sent to DLQ: {error}") def process_with_graceful_degradation(self, event: Dict[str, Any]) -> bool: """Process event with graceful degradation for non-critical drift""" try: # Validate schema schema_error = self.validate_schema(event) if schema_error: self.send_to_dlq(event, schema_error, 'payment-events') return False # Validate business rules with defaults for optional fields amount = event['amount'] currency = event.get('currency', 'USD') # Default currency if missing status = event['status'] # Validate data ranges (detect data drift) if amount < 0 or amount > 1000000: error = f"Data drift: Amount {amount} outside expected range [0, 1000000]" self.send_to_dlq(event, error, 'payment-events') return False # Process valid event logger.info(f"Processed payment {event['payment_id']}: " f"{currency} {amount} - {status}") return True except Exception as e: error = f"Processing error: {str(e)}" self.send_to_dlq(event, error, 'payment-events') return False def run(self): """Main consumer loop with drift handling""" try: for message in self.consumer: event = message.value success = self.process_with_graceful_degradation(event) if success: self.consumer.commit() # Note: We commit even on failure since event is in DLQ finally: self.consumer.close() self.dlq_producer.close() # Usage if __name__ == "__main__": logging.basicConfig(level=logging.INFO) consumer = DriftAwareConsumer() consumer.run() ``` ## Tools and Platforms Modern tooling helps detect, prevent, and manage drift across streaming pipelines. - **Schema registries** like Confluent Schema Registry (v7.x+), AWS Glue Schema Registry, and Karapace (open-source alternative) centralize schema management and enforce compatibility rules. They integrate with streaming platforms to validate events at production time. For detailed coverage of schema management strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). - **Data quality frameworks** provide comprehensive validation capabilities: - **Great Expectations (GX 1.0+)**: Production-grade data validation framework with 300+ built-in expectations. The August 2024 release introduced breaking API changes with a fluent interface. See [Great Expectations: Data Testing Framework](https://www.conduktor.io/glossary/great-expectations-data-testing-framework) for streaming integration patterns. - **Soda Core**: Open-source framework using YAML-based quality checks, designed for modern data stacks with native dbt and orchestration tool integration. - **Elementary Data**: Observability platform for dbt with ML-based anomaly detection, automatic data quality monitoring, and lineage tracking. - **dbt Tests and Contracts**: dbt 1.5+ includes native data contracts with automatic validation during transformation workflows, providing explicit interfaces between models. **Drift detection platforms** specifically target statistical and concept drift: - **Evidently AI**: Specialized ML monitoring for data drift, concept drift, and prediction drift with visual reports and real-time detection capabilities. - **Amazon SageMaker Model Monitor**: Automated drift detection for deployed ML models with integration into AWS streaming architectures. - **Observability platforms** including Datadog, Prometheus with Grafana, and OpenTelemetry-based solutions track drift metrics alongside system metrics. Dashboards visualize distribution changes, anomaly detection algorithms identify unusual patterns, and alerting systems notify teams when thresholds breach. - **Governance platforms** provide centralized control over streaming data ecosystems, combining schema management, data quality validation, access control, and compliance enforcement in unified platforms to make drift prevention an organizational capability rather than individual team responsibility. ## Building Drift-Resilient Systems Managing data drift requires treating it as a normal condition rather than an exceptional failure. Streaming systems that embrace change through robust schemas, continuous monitoring, and graceful handling remain resilient as data evolves. Start by establishing schema contracts with backward compatibility requirements. Implement statistical monitoring to detect distribution shifts early. Design consumers to degrade gracefully when encountering unexpected data. Build feedback loops that trigger model retraining when concept drift degrades predictions. Treat drift detection as seriously as error monitoring, both indicate your system's health. For comprehensive approaches to quality management, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) and [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing). When drift causes incidents, refer to [Data Quality Incidents](https://www.conduktor.io/glossary/data-quality-incidents) for response strategies. The organizations that succeed with streaming data don't prevent all drift — they build systems that detect, adapt to, and recover from it automatically. Resilience comes from treating drift as a normal condition, not an exceptional one. ## Related Concepts - [Streaming Data Pipeline](https://www.conduktor.io/streaming-data-pipeline) - Build drift-resilient data pipelines - [Schema Registry and Schema Management](https://www.conduktor.io/schema-registry-and-schema-management) - Manage schema evolution to prevent drift - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/data-quality-dimensions-accuracy-completeness-and-consistency) - Quality checks to detect drift ## Sources 1. Confluent - "Detecting and Managing Schema Drift in Apache Kafka" - https://www.confluent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you-really-need-one/ 2. Google Cloud - "Detecting and Managing Data Drift in ML Systems" - https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning 3. Amazon SageMaker - "Model Monitor for Data Quality and Model Drift" - https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html 4. Martin Kleppmann - "Designing Data-Intensive Applications" (O'Reilly) - Schema evolution and compatibility 5. Databricks - "Monitoring Data Drift in Production ML Models" - https://www.databricks.com/blog/2019/09/18/productionizing-machine-learning-from-deployment-to-drift-detection.html Source: https://www.conduktor.io/glossary/data-drift-in-streaming --- # Data Freshness Monitoring: SLA Management Ensuring that data arrives on time is just as important as ensuring it arrives correctly. Data freshness monitoring and SLA management are core practices for maintaining reliable data pipelines and meeting business expectations. This article covers how to implement freshness monitoring, define meaningful SLAs, and respond when data misses timeliness requirements. Freshness is one of the five pillars of data observability, alongside data quality, volume, schema, and lineage. For a comprehensive overview, see [What is Data Observability: The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars). To understand how freshness relates to other quality dimensions, refer to [Data Quality vs Data Observability: Key Differences](https://www.conduktor.io/glossary/data-quality-vs-data-observability-key-differences). ![Data Freshness Pipeline with Timestamps](https://www.conduktor.io/assets/images/glossary/data-freshness-monitoring-sla-management-0.webp) ## Understanding Data Freshness Data freshness refers to how current or up-to-date your data is relative to when it was generated or should be available for consumption. While data quality focuses on correctness, freshness focuses on timeliness. A dataset might be perfectly accurate but completely useless if it arrives hours late for a time-sensitive business decision. Freshness manifests differently across pipeline architectures: - **Batch pipelines** typically measure freshness as the time between scheduled runs and actual completion. If your daily ETL job should complete by 6 AM but finishes at 9 AM, you have a three-hour freshness violation. - **Streaming pipelines** measure freshness as the end-to-end latency from event generation to availability in the target system. For a Kafka-based pipeline, this includes time spent in topics, processing delays, and sink write latency. - **Hybrid systems** combine both paradigms, often ingesting streaming data but aggregating it in batch windows, creating complex freshness requirements at multiple stages. ## Defining Meaningful SLAs Service Level Agreements for data freshness should be derived from business requirements, not arbitrary technical targets. The process begins with understanding how data is consumed: - **Business-driven SLAs** emerge from asking the right questions. Does the marketing team need customer behavior data within 15 minutes to trigger personalized campaigns? Does the finance dashboard require yesterday's transactions by 8 AM? These business needs translate directly into technical SLAs. - **Tiered SLAs** recognize that not all data has the same urgency. Critical real-time fraud detection might require sub-second freshness with 99.9% reliability, while monthly reporting aggregates might tolerate day-old data with 95% reliability. This tiering allows you to allocate engineering resources where they matter most. - **Composite SLAs** acknowledge that modern pipelines have multiple stages. For a streaming pipeline ingesting from Kafka, processing with Apache Flink 1.19+, and landing in a data warehouse, you might define: - Kafka ingestion lag: < 5 seconds (p95 - 95th percentile, meaning 95% of events meet this target) - Processing latency: < 30 seconds (p95) - End-to-end freshness: < 60 seconds (p95) The percentile notation (p95, p99) indicates that the specified percentage of requests must meet the target. p95 means 95 out of 100 events arrive within the threshold, while allowing 5% outliers. This is more realistic than requiring 100% compliance, which would force you to design for worst-case scenarios. Using p99 (99th percentile) is stricter, only 1% of events can exceed the threshold, appropriate for critical use cases like fraud detection. Each component contributes to the overall SLA, and monitoring each stage helps isolate issues quickly. ## Implementing Freshness Monitoring Effective freshness monitoring requires instrumentation at multiple levels: ### Event Timestamp Tracking The foundation is capturing accurate timestamps throughout the pipeline. Every event should carry: - **Event time**: when the event actually occurred (source system timestamp) - **Ingestion time**: when it entered your system (Kafka producer timestamp) - **Processing time**: when it was transformed (Flink/Streams processing timestamp) - **Availability time**: when it became queryable (sink write timestamp) With these timestamps, you can calculate critical metrics like **event-time lag** (processing time minus event time) and **end-to-end latency** (availability time minus event time). **Example: Enriching Events with Pipeline Timestamps** Here's a practical example using Kafka Streams to track freshness through the pipeline: ```java // Kafka Streams topology that enriches events with pipeline timestamps StreamsBuilder builder = new StreamsBuilder(); KStream orders = builder.stream("orders", Consumed.with(Serdes.String(), orderEventSerde)); KStream enrichedOrders = orders .mapValues((readOnlyKey, order) -> { EnrichedOrder enriched = new EnrichedOrder(order); // Event time: from source system (already in order object) enriched.setEventTime(order.getCreatedAt()); // Ingestion time: Kafka record timestamp enriched.setIngestionTime(context().timestamp()); // Processing time: current system time enriched.setProcessingTime(System.currentTimeMillis()); // Calculate freshness metrics long ingestionLag = enriched.getIngestionTime() - enriched.getEventTime(); long processingLag = enriched.getProcessingTime() - enriched.getEventTime(); enriched.setIngestionLagMs(ingestionLag); enriched.setProcessingLagMs(processingLag); return enriched; }); // Write to sink with availability time added by the connector enrichedOrders.to("enriched-orders", Produced.with(Serdes.String(), enrichedOrderSerde)); ``` In the downstream sink (e.g., Snowflake, BigQuery), add a final timestamp when data becomes queryable: ```sql -- Snowflake table with freshness tracking CREATE TABLE enriched_orders ( order_id STRING, event_time TIMESTAMP_NTZ, -- From source system ingestion_time TIMESTAMP_NTZ, -- When entered Kafka processing_time TIMESTAMP_NTZ, -- When processed by Streams availability_time TIMESTAMP_NTZ, -- When loaded to warehouse event_to_available_latency_ms NUMBER AS ( DATEDIFF('millisecond', event_time, availability_time) ) ); -- Query to monitor current freshness SLA compliance SELECT DATE_TRUNC('hour', availability_time) as hour, PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY event_to_available_latency_ms) as p95_latency_ms, PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY event_to_available_latency_ms) as p99_latency_ms, COUNT(*) as record_count FROM enriched_orders WHERE availability_time >= DATEADD('day', -1, CURRENT_TIMESTAMP()) GROUP BY 1 ORDER BY 1 DESC; ``` This approach provides granular visibility into where latency accumulates, enabling targeted optimization. ### Streaming Integration Monitoring For Kafka-based architectures, consumer lag is your primary freshness indicator, it measures how far behind consumers are from the latest messages in a topic. Modern monitoring tools provide real-time visibility into consumer group lag across all topics and partitions. When lag increases, it signals that consumers can't keep pace with producers, directly impacting data freshness. **Modern Kafka Monitoring Tools (2025)** Several specialized tools have emerged for Kafka lag monitoring: **Kafka Lag Exporter** (open-source) exports consumer group lag metrics to Prometheus, enabling sophisticated alerting and visualization in Grafana. It works with Kafka 2.0+ and supports the modern KRaft mode introduced in Kafka 4.0: ```bash # Deploy Kafka Lag Exporter with Helm helm repo add kafka-lag-exporter https://seglo.github.io/kafka-lag-exporter/ helm install kafka-lag-exporter kafka-lag-exporter/kafka-lag-exporter \ --set clusters[0].name=production \ --set clusters[0].bootstrapBrokers=kafka:9092 ``` - **Burrow** (LinkedIn's open-source tool) provides HTTP endpoints for lag monitoring and automatically evaluates consumer health status, distinguishing between "stuck" consumers and those simply processing a backlog. - **Conduktor** offers built-in lag monitoring dashboards with automatic alerting, ideal for teams managing Kafka clusters. The Consumer Lag metric tracks both record count and time lag (how old the oldest unconsumed message is). [Learn more about monitoring brokers and apps in Conduktor](https://docs.conduktor.io/guide/monitor-brokers-apps/index). - **Enterprise Observability Platforms** like Datadog Data Streams Monitoring and New Relic provide comprehensive pipeline visibility, combining Kafka lag with infrastructure metrics to correlate performance issues. Streaming monitoring capabilities should extend beyond basic lag metrics to provide: - Historical lag trends to identify degradation patterns - Per-partition lag breakdown to detect imbalanced consumption - Consumer group health metrics to catch failed instances - Alert thresholds that trigger before SLA violations occur - Time-based lag (not just record count) to measure actual freshness impact This level of visibility is crucial for streaming pipelines where even brief lag spikes can cascade into SLA failures. For detailed coverage of consumer groups and lag mechanics, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). ### Heartbeat and Canary Metrics Passive monitoring detects problems after they occur. Active monitoring catches issues proactively: **Heartbeat records** are synthetic events injected at regular intervals with known timestamps. By tracking when these records appear downstream, you measure actual pipeline latency without waiting for real data. If heartbeats stop appearing or show increasing delays, you know freshness is degrading. Here's a simple heartbeat implementation using a scheduled producer: ```python from kafka import KafkaProducer import json import time from datetime import datetime from apscheduler.schedulers.blocking import BlockingScheduler producer = KafkaProducer( bootstrap_servers=['localhost:9092'], value_serializer=lambda v: json.dumps(v).encode('utf-8') ) def send_heartbeat(): """Send a heartbeat event every minute""" heartbeat = { 'type': 'heartbeat', 'source': 'freshness-monitor', 'timestamp': datetime.utcnow().isoformat(), 'timestamp_ms': int(time.time() * 1000) } producer.send('pipeline-heartbeats', value=heartbeat) producer.flush() print(f"Heartbeat sent at {heartbeat['timestamp']}") # Schedule heartbeat every minute scheduler = BlockingScheduler() scheduler.add_job(send_heartbeat, 'interval', minutes=1) scheduler.start() ``` Downstream, monitor when heartbeats arrive to detect pipeline degradation before it affects real data. **Canary pipelines** run simplified versions of production workloads continuously, providing constant freshness signals even during low-traffic periods when real events are sparse. ### Data Quality Framework Integration Modern data quality frameworks include freshness testing as a first-class feature. These tools integrate naturally into your CI/CD and data pipelines: **dbt Freshness Tests** (dbt 1.0+) define acceptable data age directly in your source configurations: ```yaml # models/sources.yml version: 2 sources: - name: production_db database: postgres freshness: warn_after: {count: 12, period: hour} error_after: {count: 24, period: hour} loaded_at_field: updated_at tables: - name: orders freshness: warn_after: {count: 1, period: hour} error_after: {count: 2, period: hour} - name: customers # Uses source-level freshness defaults ``` Run freshness checks as part of your dbt pipeline: ```bash # Check freshness for all sources dbt source freshness # Fail the pipeline if any source fails error threshold dbt source freshness --select source:production_db ``` For more on dbt testing practices, see [dbt Tests and Data Quality Checks](https://www.conduktor.io/glossary/dbt-tests-and-data-quality-checks). **Great Expectations 1.0+** provides sophisticated freshness expectations with custom thresholds: ```python import great_expectations as gx from datetime import datetime, timedelta context = gx.get_context() # Define freshness expectation validator = context.sources.pandas_default.read_dataframe(df) validator.expect_column_max_to_be_between( column="event_timestamp", min_value=datetime.utcnow() - timedelta(hours=2), max_value=datetime.utcnow(), meta={ "notes": "Data must be no older than 2 hours for real-time dashboard SLA" } ) # Run validation in your pipeline checkpoint = context.add_or_update_checkpoint( name="freshness_check", validations=[ { "batch_request": batch_request, "expectation_suite_name": "orders_freshness_suite", } ], ) result = checkpoint.run() if not result.success: # Trigger alert or fail pipeline raise ValueError(f"Freshness SLA violated: {result}") ``` **Soda Core 3.0+** (open-source alternative to Soda Cloud) uses simple YAML checks: ```yaml # checks/orders_freshness.yml checks for orders: - freshness(event_timestamp) < 1h: name: Orders must be fresh within 1 hour warn: when required - freshness(event_timestamp) < 2h: name: Orders must be fresh within 2 hours fail: when required # Check with custom SQL - freshness using max(event_timestamp): warn: when > 1h fail: when > 2h ``` Run Soda checks in your pipeline: ```bash soda scan -d production -c configuration.yml checks/orders_freshness.yml ``` **Monte Carlo** and other enterprise data observability platforms provide automated freshness anomaly detection using machine learning. They learn your typical data arrival patterns and alert when deviations occur, reducing manual threshold tuning. For more on data quality frameworks, see [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). ## Responding to Freshness Violations Monitoring without action is just observation. When freshness SLAs are breached, your response should be systematic. For detailed incident response procedures, see [Data Incident Management and Root Cause Analysis](https://www.conduktor.io/glossary/data-incident-management-and-root-cause-analysis). ### Automated Alerting Configure multi-level alerts: - **Warning thresholds** at 70-80% of SLA targets give teams time to investigate before violations - **Critical alerts** at SLA boundaries trigger immediate response - **Severity escalation** routes persistent violations to on-call engineers Context-rich alerts include not just the metric value but also recent trends, affected data domains, and potential impact on downstream consumers. ### Root Cause Analysis Freshness violations typically stem from: - **Volume spikes**: sudden increases in data volume overwhelming processing capacity - **Resource constraints**: CPU, memory, or I/O saturation in compute or storage layers - **Dependency failures**: upstream system delays or outages - **Configuration issues**: inefficient processing logic or suboptimal parallelism Effective monitoring isolates which component is responsible. If Kafka consumer lag is stable but warehouse availability is delayed, the issue is in the sink, not ingestion or processing. ### Mitigation Strategies Short-term mitigations include: - Scaling compute resources to process backlog faster - Temporarily increasing parallelism or partition count - Pausing non-critical workloads to free resources - Implementing backpressure to prevent cascade failures (see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems)) Long-term improvements address systemic issues: - Optimizing processing logic to reduce per-event latency - Right-sizing infrastructure based on realistic capacity planning - Implementing data tiering to separate critical from non-critical flows - Adding circuit breakers and graceful degradation patterns ## Building a Freshness-Aware Culture Technical solutions alone don't ensure fresh data. Organizational practices matter: - **SLA documentation** should be centralized, version-controlled, and accessible to both engineering and business stakeholders. When everyone understands the commitments, they can make informed decisions about dependencies. Consider incorporating freshness SLAs into your data contracts, explicit agreements between data producers and consumers. For detailed guidance, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). - **Regular SLA reviews** adapt targets as business needs evolve. Quarterly reviews assess whether SLAs remain relevant and achievable. - **Freshness dashboards** provide self-service visibility. Data consumers should see current freshness metrics for datasets they depend on, reducing surprise and enabling proactive mitigation. Integrating freshness metrics with data lineage tools provides end-to-end visibility into how delays propagate through dependent datasets. For more on tracking data flow, see [Data Lineage Tracking: Data From Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption). ## Conclusion Data freshness monitoring and SLA management shift data engineering from reactive fire-fighting to proactive reliability. By defining business-driven SLAs, implementing monitoring across batch and streaming pipelines, and building systematic response processes, you ensure that data arrives when it's needed. The investment pays off beyond meeting SLAs. It provides early warning of capacity issues, validates architectural decisions, and builds trust with data consumers. Fresh data isn't optional when real-time decisions drive business outcomes. Start with your most critical pipelines, establish baseline SLAs, and expand coverage gradually. Tools like Kafka Lag Exporter, dbt freshness checks, Great Expectations, Soda Core, and platforms like Monte Carlo cover most of what you need. Good tooling and a team that understands freshness metrics will get you most of the way there. ## Related Concepts - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Detailed streaming lag tracking strategies - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) - Understanding timeliness in context - [What is Data Observability: The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars) - Freshness as part of observability ## Sources and References - [Google SRE Book - Service Level Objectives](https://sre.google/sre-book/service-level-objectives/) - Foundational guidance on defining and managing SLAs and SLOs - [Apache Flink Metrics and Monitoring](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/) - Tracking latency and processing time in stream processing (Flink 1.19+) - [Kafka Lag Exporter](https://github.com/seglo/kafka-lag-exporter) - Open-source Prometheus exporter for Kafka consumer lag metrics, supports KRaft mode - [Datadog Data Streams Monitoring](https://www.datadoghq.com/blog/data-streams-monitoring/) - End-to-end latency tracking for Kafka pipelines (2025) - [dbt Source Freshness](https://docs.getdbt.com/reference/resource-properties/freshness) - Implementing freshness checks in analytics pipelines (dbt 1.0+) - [Great Expectations Freshness Expectations](https://docs.greatexpectations.io/docs/reference/expectations/) - Data validation framework with time-based expectations (GX 1.0+) - [Soda Core Data Quality Checks](https://docs.soda.io/soda-core/soda-core.html) - Open-source data quality framework with freshness monitoring (Soda 3.0+) - [Burrow - Kafka Consumer Lag Monitoring](https://github.com/linkedin/Burrow) - LinkedIn's consumer lag monitoring tool with health evaluation - [Conduktor Platform](https://www.conduktor.io/) - Kafka management and monitoring platform with comprehensive lag and latency metrics Source: https://www.conduktor.io/glossary/data-freshness-monitoring-sla-management --- # Data Governance Framework: Roles and Responsibilities Implementing a data governance framework requires more than policies and procedures — it requires clear roles and responsibilities across your organization. Without well-defined accountability, even well-designed governance strategies fail to deliver value. This guide covers the essential roles within a data governance framework and how they work together to ensure data quality, compliance, and strategic value. ![Data Governance Organizational Structure](https://www.conduktor.io/assets/images/glossary/data-governance-framework-roles-and-responsibilities-0.webp) ## Why Roles Matter in Data Governance Data governance is inherently cross-functional. Data flows through multiple departments, systems, and stakeholders, each with different perspectives and priorities. A clear role structure ensures that everyone understands their responsibilities, decision-making authority, and accountability for data-related outcomes. When roles are ambiguous, organizations face several challenges: - Data quality issues persist without clear ownership - Compliance gaps emerge from unclear accountability - Strategic initiatives stall due to decision-making bottlenecks - Teams duplicate efforts or work at cross-purposes A well-structured framework assigns responsibility while enabling collaboration, creating a culture where data is treated as a strategic asset. ## Role Overview at a Glance Before diving into details, here's a quick comparison of key governance roles: | Role | Primary Focus | Decision Authority | Time Commitment | Technical Depth | |------|--------------|-------------------|----------------|----------------| | **Data Governance Council** | Strategic direction | High - approves policies | Quarterly meetings | Low - business focused | | **Data Governance Officer** | Program leadership | Medium - coordinates execution | Full-time | Medium - bridging role | | **Data Owner** | Domain accountability | High - approves access | Part-time oversight | Low - business domain expert | | **Data Steward** | Operational execution | Medium - implements policies | Full-time or significant part-time | Medium-High - domain + technical | | **Data Custodian** | Technical implementation | Low - follows policies | Full-time | High - infrastructure specialist | | **Compliance Officer** | Regulatory alignment | Medium - defines requirements | Full-time | Medium - regulatory + technical | | **Data Product Owner** | Product management | High - within domain | Full-time | Medium-High - product + technical | | **AI/ML Governance Officer** | Model governance | Medium - ML-specific policies | Full-time | High - ML/AI specialist | This structure creates a balance between strategic oversight, operational execution, and technical implementation. ## Core Governance Roles ### Data Governance Council The Data Governance Council sits at the top of the governance hierarchy, providing strategic direction and executive sponsorship. Typically composed of senior leaders including the CTO, Chief Data Officer, and heads of key business units, this council sets policies, approves standards, and allocates resources for governance initiatives. **Key Responsibilities:** - Define governance strategy aligned with business objectives - Approve data policies, standards, and procedures - Resolve escalated issues and conflicts - Monitor governance program effectiveness - Ensure adequate funding and resources The council meets quarterly or bi-annually, focusing on strategic decisions rather than day-to-day operations. Their visible commitment signals to the organization that data governance is a business priority, not just an IT initiative. ### Data Governance Officer The Data Governance Officer (DGO) serves as the operational leader of the governance program, translating executive vision into actionable initiatives. Reporting to the Chief Data Officer or CTO, the DGO coordinates between technical teams, business stakeholders, and compliance functions. **Key Responsibilities:** - Design and implement governance frameworks - Facilitate governance council meetings and decisions - Monitor compliance with data policies - Coordinate training and awareness programs - Track governance metrics and KPIs - Manage relationships with data stewards and owners For organizations managing streaming data platforms, the DGO must understand real-time data challenges, including event schema management, data lineage in stream processing, and access control for streaming topics. Modern governance platforms help DGOs visualize and manage streaming environments with tools like: - **OpenMetadata** and **DataHub** for open-source data cataloging and lineage tracking - **Conduktor** for Kafka cluster governance, monitoring, and stream governance - **Atlan** and **Collibra** for enterprise-wide data governance with streaming integration - **Apache Atlas** for metadata management in distributed data ecosystems These tools provide centralized governance capabilities without becoming bottlenecks to real-time data flows. For detailed guidance on implementing governance for streaming data, see [Building and Managing Data Products](https://www.conduktor.io/glossary/building-and-managing-data-products) and [Data Mesh Principles and Implementation](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation). ### Data Owners Data Owners are senior business leaders accountable for specific data domains. A Chief Marketing Officer might own customer data, while a Chief Financial Officer owns financial data. This business-led ownership ensures that data governance serves business objectives rather than existing solely as a technical exercise. **Key Responsibilities:** - Define business rules and quality standards for their domain - Approve access requests to sensitive data - Set data retention and archival policies - Accountable for compliance within their domain - Allocate resources for data quality initiatives Data Owners have decision-making authority but delegate day-to-day management to Data Stewards. Their involvement ensures governance decisions reflect business priorities and risk tolerance. ### Data Stewards Data Stewards are the operational champions of data governance, working closely with technical teams and business users. While Data Owners set policy, Data Stewards implement and enforce it. They serve as subject matter experts for their assigned data domains. **Key Responsibilities:** - Document data definitions, lineage, and business context - Monitor data quality and investigate anomalies - Coordinate data quality remediation efforts - Maintain data catalogs and metadata - Review and process access requests - Provide training to data consumers In streaming environments, Data Stewards manage schema registries (see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management)), define topic naming conventions, and ensure proper data classification tags flow through Kafka topics. They work with platform teams to implement governance controls without impeding real-time data flows. For comprehensive approaches to data quality monitoring, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) and [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing). ### Data Custodians Data Custodians are technical professionals responsible for the physical management and security of data. Database administrators, platform engineers, and security specialists fill this role, implementing the technical controls that enforce governance policies. **Key Responsibilities:** - Implement access controls and encryption - Manage backup and recovery procedures - Monitor system performance and availability - Apply security patches and updates - Execute data retention and deletion procedures For streaming platforms, Data Custodians configure authentication and authorization for Kafka clusters (running on Kafka 4.0+ with KRaft mode for simplified operations), implement encryption in transit and at rest, and manage disaster recovery procedures. Modern governance platforms like Conduktor enable Data Custodians to implement fine-grained access controls through intuitive interfaces (see [Conduktor RBAC setup](https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac)) and comprehensive audit logging (see [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms)) without extensive custom development. For security implementation details, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). ### Compliance Officers Compliance Officers ensure that data practices align with regulatory requirements such as GDPR, CCPA, HIPAA, and industry-specific regulations. They bridge governance and legal/regulatory functions, translating complex requirements into practical controls. **Key Responsibilities:** - Monitor regulatory landscape and assess impact - Define compliance requirements for data handling - Conduct privacy impact assessments - Manage data breach response procedures - Coordinate regulatory audits and reporting - Review and approve data sharing agreements In real-time data environments, Compliance Officers must address unique challenges such as ensuring the right to deletion in event streams, maintaining audit trails for streaming data access, and managing consent across distributed systems. ## Emerging Roles in 2025 As data governance evolves to meet new technological challenges, several specialized roles have emerged or gained prominence: ### Data Product Owner In Data Mesh architectures, the Data Product Owner combines business domain expertise with technical understanding to manage data as a product. This role is central to federated data governance models where domains own and govern their own data products. **Key Responsibilities:** - Define data product strategy and roadmap - Ensure data product meets quality, discoverability, and usability standards - Own the data product lifecycle from creation to retirement - Manage data product SLAs and consumer relationships - Collaborate with platform teams on infrastructure needs The Data Product Owner bridges traditional Data Owner and Data Steward responsibilities but with product management discipline. For detailed guidance, see [Building and Managing Data Products](https://www.conduktor.io/glossary/building-and-managing-data-products) and [Data Product Governance](https://www.conduktor.io/glossary/data-product-governance). ### AI/ML Governance Officer With the explosion of machine learning and AI systems in 2024-2025, the AI/ML Governance Officer ensures responsible development and deployment of AI models. This role focuses on model governance, fairness, explainability, and ethical AI practices. **Key Responsibilities:** - Define and enforce ML model governance policies - Oversee model risk management and validation - Ensure model documentation and lineage tracking - Monitor for model bias, drift, and fairness issues - Coordinate responsible AI practices across teams - Manage AI/ML compliance with emerging regulations This role is particularly critical for organizations implementing LLM-powered applications, where data governance intersects with prompt engineering, fine-tuning datasets, and retrieval-augmented generation (RAG) pipelines. ### LLM Safety and Ethics Officer As organizations adopt Large Language Models (LLMs) and generative AI, a specialized role has emerged to govern these powerful technologies. The LLM Safety Officer focuses on preventing misuse, ensuring ethical deployment, and managing risks unique to generative AI. **Key Responsibilities:** - Define guardrails for LLM inputs and outputs - Prevent training data contamination and leakage - Monitor for jailbreak attempts and prompt injection attacks - Ensure responsible use of synthetic data - Manage consent for data used in model training - Coordinate red-teaming exercises for AI safety This role works closely with Compliance Officers on regulations like the EU AI Act and emerging AI governance frameworks. ### Cloud FinOps Data Governance Specialist As data infrastructure moves to cloud platforms, the intersection of financial operations and data governance requires specialized expertise. This role ensures cost-effective governance without sacrificing control. **Key Responsibilities:** - Optimize data storage costs while maintaining retention policies - Monitor and control compute costs for data processing - Implement data lifecycle management for cost optimization - Balance data accessibility with storage tier economics - Track governance-related infrastructure costs This role is essential in organizations running large-scale streaming platforms where data retention and processing costs can escalate quickly. ## Role Interactions in Practice These roles don't operate in isolation. Consider a scenario where a marketing team wants to integrate real-time customer behavior data from a Kafka stream into their analytics platform: 1. **Data Steward** receives the request and validates it aligns with customer data policies 2. **Compliance Officer** reviews for privacy implications and consent requirements 3. **Data Owner** approves based on business value and risk assessment 4. **Data Custodian** implements technical access controls and monitoring 5. **Data Governance Officer** tracks the request for metrics and identifies process improvements This collaborative approach balances innovation with control, enabling teams to move quickly while maintaining governance standards. ## Implementing Your Framework Start by identifying executives who can serve on your Data Governance Council and champion the initiative. Select a Data Governance Officer with both technical understanding and business acumen. Identify natural Data Owners based on your organizational structure, and recruit Data Stewards from teams with strong domain knowledge. Document roles clearly, but start with a lightweight framework that grows with your program maturity. Perfect is the enemy of good in governance, it's better to establish clear ownership for critical data domains and expand coverage over time. ### Modern Implementation Approaches **For Traditional Centralized Governance:** - Implement a data catalog like OpenMetadata, Atlan, or Collibra - Establish RACI matrices for each data domain - Use tools like Apache Atlas for metadata management - Focus on critical data assets first, then expand **For Data Mesh / Federated Governance:** - Start with defining data product standards - Implement federated computational governance (policy as code) - Use self-serve platforms for domain teams - Balance autonomy with global standards - See [Data Mesh Principles and Implementation](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation) for detailed guidance **For Streaming Data Platforms:** - Invest in Kafka-native governance tools (Conduktor for comprehensive governance and monitoring) - Implement schema governance early with Schema Registry - Establish topic naming and tagging conventions - Enable self-serve access with guardrails - Track data lineage through stream processing pipelines - See [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) **For AI/ML Environments:** - Add ML model governance tools (MLflow, Weights & Biases with governance extensions) - Implement feature store governance - Establish model cards and documentation standards - Create guardrails for LLM deployments - Monitor for bias and drift continuously Modern governance platforms provide role-based access control, automated lineage tracking, and policy enforcement without becoming bottlenecks. The key is choosing tools that fit your architecture, cloud-native for cloud workloads, streaming-native for real-time platforms, and federated for Data Mesh implementations. ## Balancing Autonomy and Governance with Self-Service Traditional governance frameworks create tension between control and agility, centralized approval processes slow development teams, leading to shadow IT or governance circumvention. Self-Service frameworks resolve this by enabling teams to manage resources independently through version-controlled configuration files (GitOps), with the platform automatically validating requests against governance policies before provisioning. This approach eliminates bottlenecks while ensuring adherence to organizational standards. Version control creates audit trails for Compliance Officers, Data Owners retain approval authority through pull request workflows, and consistent policy enforcement eliminates manual configuration errors. For implementation guidance, see [Self-Service](https://docs.conduktor.io/guide/conduktor-concepts/self-service). ## Conclusion A successful data governance framework depends on clearly defined roles working together. From strategic leadership in the Governance Council to day-to-day execution by Data Stewards and Custodians, each role contributes to treating data as a strategic asset rather than a liability. As data environments evolve, new roles emerge: Data Product Owners for Data Mesh architectures, AI/ML Governance Officers for responsible AI deployment, and LLM Safety Officers for generative AI governance. These specialized roles complement traditional governance structures. Whether you're implementing centralized governance, adopting Data Mesh principles, managing streaming platforms, or deploying AI systems, the key is establishing clear accountability while enabling teams to move. With the right roles, responsibilities, and tooling, data governance becomes an enabler rather than a bottleneck. For related guidance on specific governance domains, see: - [Data Product Governance](https://www.conduktor.io/glossary/data-product-governance) for product-centric governance - [Metadata Management: Technical vs Business Metadata](https://www.conduktor.io/glossary/metadata-management-technical-vs-business-metadata) for metadata governance - [What is a Data Catalog: Modern Data Discovery](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery) for discovery and cataloging - [Building a Business Glossary for Data Governance](https://www.conduktor.io/glossary/building-a-business-glossary-for-data-governance) for semantic governance ## Related Concepts - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Centralized schema governance for streaming data - [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) - Tracking data access and changes for compliance - [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) - Formal agreements between data producers and consumers ## Sources and References - [DAMA-DMBOK Data Governance Framework](https://www.dama.org/cpages/body-of-knowledge) - Industry-standard framework for data management and governance roles - [Data Governance Institute Best Practices](https://datagovernance.com/the-dgi-data-governance-framework/) - Comprehensive guide to data governance implementation - [Data Mesh Principles (Zhamak Dehghani)](https://martinfowler.com/articles/data-mesh-principles.html) - Foundational article on federated data governance - [OpenMetadata Documentation](https://docs.open-metadata.org/) - Open-source metadata management and governance platform - [Confluent Stream Governance](https://docs.confluent.io/platform/current/streams/governance.html) - Kafka-native governance capabilities and best practices - [Microsoft Azure Purview Roles](https://learn.microsoft.com/en-us/azure/purview/catalog-permissions) - Role-based access control for data governance platforms - [Collibra Data Governance Operating Model](https://www.collibra.com/us/en/data-governance-operating-model) - Framework for defining governance roles and responsibilities - [GDPR Data Protection Roles](https://gdpr.eu/data-protection-officer/) - Regulatory requirements for data governance and compliance roles - [EU AI Act](https://artificialintelligenceact.eu/) - Emerging AI governance regulation requiring specialized roles - [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) - Framework for AI/ML governance and risk management Source: https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities --- # Data Incident Management and Root Cause Analysis Data incidents are inevitable. Whether you're managing batch pipelines, real-time streaming systems, or hybrid architectures, knowing how to respond to incidents and perform root cause analysis is critical for maintaining data reliability and stakeholder trust. ![Incident Response Lifecycle](https://www.conduktor.io/assets/images/glossary/data-incident-management-and-root-cause-analysis-0.webp) ## Understanding Data Incidents A data incident occurs when data quality, availability, or timeliness deviates from expected standards, impacting downstream consumers. Unlike application incidents that might affect user experience immediately, data incidents can be insidious, cascading through pipelines, corrupting analytics, and undermining business decisions before anyone notices. Common incident types include: - **Data Quality Issues**: Schema drift, null values, duplicates, or invalid data types - **Pipeline Failures**: Job crashes, resource exhaustion, or dependency failures - **Latency Problems**: Service Level Agreement (SLA) breaches, backpressure in streaming systems, or batch delays - **Data Loss**: Missing partitions, incomplete ingestion, or failed writes - **Schema Conflicts**: Breaking changes in upstream systems or incompatible transformations ## The Incident Response Lifecycle Effective incident management follows a structured lifecycle that minimizes Mean Time to Resolution (MTTR, the average time it takes to resolve an incident) while preventing recurrence. ### 1. Detection and Alerting The first step is knowing an incident occurred. Modern data platforms should implement multi-layered monitoring: - **Data Quality Monitoring**: Automated checks on row counts, schema validation, null rates, and business logic constraints. Modern tools like Great Expectations 1.0+, Soda Core 3.x, and dbt tests validate both technical correctness and business expectations. For detailed coverage of data quality frameworks, see [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). - **Freshness Monitoring**: Track when data was last updated. For streaming systems like Apache Kafka 4.0+ (now running KRaft mode without ZooKeeper), monitor consumer lag across topics using tools like Kafka Lag Exporter, Burrow, or Confluent Control Center to identify when processing falls behind production. For more on monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). - **Pipeline Health**: Monitor job execution status, resource utilization, and dependency chains. Modern incident management platforms like Datadog Incident Management, PagerDuty Incident Workflows with AIOps, and Grafana IRM (Incident Response Management) provide unified visibility across data pipelines. Failed jobs should trigger immediate alerts with context about the failure point. - **Anomaly Detection**: Statistical methods and machine learning models to identify unusual patterns that might not trigger rule-based alerts but indicate emerging problems. Cloud-native observability platforms (AWS CloudWatch Anomaly Detection, Azure Monitor, GCP Operations Suite) and data observability tools (Monte Carlo, Soda Core) provide automated anomaly detection for data metrics. ### 2. Triage and Assessment When an alert fires, rapid triage determines incident severity and guides response: - **Impact Analysis**: Identify affected datasets, downstream dependencies, and business processes. Understanding the blast radius (the scope of systems and users affected by the incident) helps prioritize response efforts. - **Severity Classification**: Not all incidents are equal. Classify based on: - Business impact (revenue, compliance, critical reports) - Affected user count - Data recoverability - Time sensitivity **Initial Investigation**: Quickly gather context, recent deployments, configuration changes, upstream system status, and error logs. For streaming platforms, check broker health, partition assignments, and consumer group states. ### 3. Mitigation and Resolution The goal is restoring service while minimizing additional damage: - **Immediate Mitigation**: Stop the bleeding. This might mean pausing pipelines, rolling back deployments, or switching to backup data sources. In Kafka environments, you might need to reset consumer offsets or pause specific consumers to prevent poison messages (corrupted or malformed messages that cause consumer crashes) from blocking processing. For error handling patterns, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). - **Root Cause Investigation**: While mitigating, begin deeper investigation. Streaming systems add complexity, issues might stem from broker configuration, partition rebalancing (redistribution of topic partitions across consumers), message serialization, or consumer processing logic. Modern observability approaches using OpenTelemetry and distributed tracing help examine topic configurations, consumer group assignments, and message contents, accelerating diagnosis. For advanced debugging techniques, see [Distributed Tracing for Kafka Applications](https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications). - **Fix Implementation**: Apply the fix with appropriate testing. For streaming systems, consider: - Will the fix handle backlog replay correctly? - Are there ordering guarantees to maintain? - Will processing the backlog cause downstream issues? **Data Reconciliation**: Determine if reprocessing is needed. Streaming architectures often provide natural reprocessing mechanisms through offset management, but care must be taken with exactly-once semantics (guaranteeing each message is processed exactly one time, never duplicated or lost) and idempotent consumers (consumers that produce the same result even if they process the same message multiple times). ## Root Cause Analysis: Going Deeper Resolving an incident is only half the battle. Root cause analysis prevents recurrence and drives systematic improvement. ### The Five Whys Technique Start with the observed symptom and ask "why" repeatedly until reaching the fundamental cause: **Example**: 1. Why did the dashboard show incorrect metrics? → Pipeline produced wrong aggregations 2. Why were aggregations wrong? → Duplicate events in the source topic 3. Why were there duplicates? → Producer retry logic without idempotent settings 4. Why wasn't idempotence configured? → Default producer configuration was used 5. Why were defaults used? → No standardized producer configuration template The root cause isn't the duplicates, it's the lack of standardized configuration management. **Code Example - Configuring Idempotent Kafka Producer (Kafka 3.0+)**: ```java Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); // Enable idempotence to prevent duplicates props.put("enable.idempotence", "true"); props.put("acks", "all"); // Required for idempotence props.put("retries", Integer.MAX_VALUE); // Safe with idempotence props.put("max.in.flight.requests.per.connection", "5"); // Kafka 3.0+ allows up to 5 KafkaProducer producer = new KafkaProducer<>(props); ``` ### Contributing Factor Analysis Complex systems rarely fail from a single cause. Identify contributing factors: - **Technical Factors**: Schema evolution without compatibility checks, insufficient resources, missing backpressure handling - **Process Factors**: Lack of deployment validation, insufficient testing, unclear ownership - **Organizational Factors**: Siloed teams, unclear escalation paths, inadequate documentation For streaming systems, consider: - Producer configurations (acks, retries, idempotence) - Broker settings (retention, replication, partitioning) - Consumer patterns (offset management, rebalancing strategies) - Network reliability and partition leadership - KRaft mode configurations (Kafka 4.0+ replaces ZooKeeper with internal consensus) **Code Example - Monitoring Consumer Lag (Kafka 4.0+)**: ```bash # Check consumer lag using kafka-consumer-groups command kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --describe --group my-consumer-group # Output shows lag per partition: # GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG # my-consumer-group events 0 1500 2000 500 # my-consumer-group events 1 1800 1800 0 # For programmatic monitoring, use Kafka Lag Exporter or AdminClient API ``` ### Preventive Measures Root cause analysis should produce actionable improvements: - **Automated Safeguards**: Implement schema validation, data quality gates, and compatibility checks. Kafka Schema Registry with compatibility enforcement prevents many schema-related incidents. Modern data quality tools like Great Expectations 1.0+ can run as pipeline gates, blocking bad data before it propagates. - **Improved Monitoring**: Add specific checks targeting the incident class. If consumer lag caused the issue, implement granular lag monitoring with appropriate thresholds using Kafka Lag Exporter or Burrow. Shift from threshold-based alerting to SLO-based alerting (Service Level Objectives) for more context-aware incident detection. - **Documentation and Runbooks**: Capture incident response procedures, architecture decisions, and configuration standards. This organizational memory prevents repeated mistakes. Modern incident management platforms like PagerDuty and Grafana IRM integrate runbooks directly into alert workflows. - **Testing Enhancement**: Add regression tests, chaos engineering experiments, or failure injection to validate resilience. Test how your streaming system handles broker failures, network partitions, or poison messages. Tools like Chaos Mesh and Litmus help inject failures into Kafka clusters to validate recovery procedures. For more on proactive testing, see [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems). ## Modern Observability and Incident Management (2025) The shift toward observability-driven incident management emphasizes understanding system behavior through metrics, logs, and traces rather than just reacting to alerts. - **OpenTelemetry Adoption**: The industry standard for instrumentation now enables unified observability across data pipelines. OpenTelemetry provides automatic instrumentation for Kafka clients, stream processors, and data applications, creating end-to-end visibility from producers to consumers. - **SLO-Based Alerting**: Instead of arbitrary thresholds (e.g., "alert if lag > 1000"), define Service Level Objectives based on business impact (e.g., "99.9% of messages processed within 5 minutes"). This reduces alert fatigue and focuses on what matters. - **AI-Powered Incident Management**: Modern platforms like PagerDuty AIOps, Datadog Watchdog, and Grafana Machine Learning analyze historical incident patterns to predict issues, suggest root causes, and automate routine responses. - **Data Observability Platforms**: Purpose-built tools like Monte Carlo, Soda Core 3.x, and Elementary Data provide data-specific observability, tracking data quality, freshness, volume, and schema changes automatically. These complement traditional infrastructure monitoring. ## Best Practices for Data Platform Teams - **Establish Clear Ownership**: Every dataset and pipeline should have defined owners responsible for monitoring, incident response, and maintenance. Data mesh principles emphasize domain ownership with clear accountability. - **Implement Incident Review Processes**: Post-incident reviews should be blameless, focused on systemic improvement rather than individual mistakes. Document findings in a searchable incident database accessible to all teams. - **Build Observability In**: Design systems with debugging in mind. Comprehensive logging, distributed tracing (using OpenTelemetry), and metadata tracking enable faster diagnosis. Instrument custom metrics for business-specific KPIs, not just infrastructure metrics. - **Automate Recovery**: Where possible, implement self-healing mechanisms, automatic retries with exponential backoff, dead letter queues for poison messages, and circuit breakers for failing dependencies. For streaming error patterns, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). - **Maintain Communication Channels**: During incidents, clear communication with stakeholders prevents panic and sets appropriate expectations. Modern incident management platforms integrate with Slack, Microsoft Teams, and status page tools for automated stakeholder updates. - **Practice Chaos Engineering**: Regularly test your incident response capabilities through controlled failure injection. Validate that your monitoring detects issues, alerts fire appropriately, and recovery procedures work as documented. ## Conclusion Effective incident management and root cause analysis are core competencies for data and platform engineering teams. Structured detection, response, and learning processes minimize MTTR, prevent recurrence, and build more resilient data platforms. The modern approach combines traditional incident response with observability-driven practices and proactive chaos engineering. Tools like OpenTelemetry, Kafka Lag Exporter, Great Expectations 1.0+, and purpose-built data observability platforms provide deep visibility into data system behavior. The goal isn't eliminating all incidents — that's unrealistic in complex distributed systems. Focus on rapid detection, efficient response, thorough root cause analysis, and continuous improvement through automated testing and chaos engineering. Over time, this builds both technical reliability and organizational capability. ## Related Concepts - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Detecting streaming pipeline incidents through lag metrics - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Platform requiring specialized incident response strategies - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Processing guarantees that prevent duplicate-related incidents ## Related Topics - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) - Understanding data quality metrics for incident detection - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) - Comprehensive Kafka monitoring strategies - [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling) - Patterns for handling failed messages - [Distributed Tracing for Kafka Applications](https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications) - End-to-end visibility in streaming systems - [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems) - Proactive resilience testing - [Data Quality Incidents](https://www.conduktor.io/glossary/data-quality-incidents) - Specific incident patterns in data quality ## Sources 1. Google SRE Book - "Managing Incidents" - https://sre.google/sre-book/managing-incidents/ 2. Atlassian - "Incident Management for High-Velocity Teams" - https://www.atlassian.com/incident-management 3. PagerDuty - "Incident Response Documentation and Postmortems" - https://response.pagerduty.com/ 4. Apache Kafka Documentation - "Operations and Monitoring" (Kafka 4.0+) - https://kafka.apache.org/documentation/#operations 5. AWS - "Building a Data Quality Framework" - https://aws.amazon.com/blogs/big-data/build-a-data-quality-framework/ 6. OpenTelemetry Documentation - "Getting Started" - https://opentelemetry.io/docs/ 7. Kafka Lag Exporter - GitHub Repository - https://github.com/seglo/kafka-lag-exporter 8. Monte Carlo Data Observability - Documentation - https://docs.getmontecarlo.com/ 9. Soda Core 3.x - "Data Quality Monitoring" - https://docs.soda.io/ 10. Grafana IRM - "Incident Response Management" - https://grafana.com/products/cloud/irm/ Source: https://www.conduktor.io/glossary/data-incident-management-and-root-cause-analysis --- # Data Lake Zones: Bronze, Silver, Gold Architecture The Medallion Architecture, also known as the Bronze-Silver-Gold pattern, has become a standard way to organize analytical data. This layered approach provides a clear framework for progressive data refinement, letting data teams balance raw data preservation with the need for high-quality, analytics-ready datasets. One clarification up front: Bronze-Silver-Gold and the storage architecture underneath it are two separate concerns. Medallion describes *how* data is refined through quality tiers. A data lake, a data warehouse, and a lakehouse are choices about *where and how* that data is physically stored. The pattern actually predates the lakehouse. It mirrors the layering data warehouses have long used (staging, then cleansed, then data marts), and it runs just as well on a warehouse or plain object storage as it does on a lakehouse. **You can have a lakehouse with no medallion layers, and a medallion pipeline with no lakehouse.** The two get conflated mainly because Databricks popularized "medallion architecture" and the "lakehouse" together, so people absorbed them as one package rather than two independent decisions. ## Understanding the Medallion Architecture The Medallion Architecture divides your data lake into three distinct zones, each serving a specific purpose in the data pipeline: ![The Medallion Architecture divides your data lake into three distinct zones, each serving a specific purpose in the data pipeline](https://www.conduktor.io/assets/images/glossary/data-lake-zones-bronze-silver-gold-architecture-0.webp) This layered approach ensures data lineage traceability while providing different consumption patterns for various use cases. ## Bronze Layer: The Foundation of Truth The Bronze layer serves as your raw data landing zone. Here, data arrives in its most pristine form, exactly as it exists at the source. This layer operates on a fundamental principle: **preserve everything, transform nothing**. ### Key Characteristics - **Append-only architecture**: Data is never deleted or modified, ensuring complete historical records - **Schema-on-read**: Data is stored in flexible formats (JSON, Avro, Parquet) without strict schema validation at write time. Schema is applied when reading/querying, allowing you to adapt to source changes without breaking ingestion - **Full fidelity**: Captures all source data, including metadata like ingestion timestamps, source system identifiers, and Kafka offsets - **Partitioning**: Typically organized by ingestion date (e.g., `date=2025-01-15`) for efficient time-based querying and data retention ### Streaming Integration For real-time data pipelines, the Bronze layer excels at consuming streaming data. Apache Kafka topics feed directly into Bronze tables, often using formats like JSON or Avro that preserve the source structure. Modern Kafka platforms like **Conduktor** provide essential visibility into streaming data flows, helping teams manage topics, monitor consumer lag, validate schemas, and track data quality metrics as events flow into the Bronze layer. See [Conduktor's topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) and [Schema Registry integration](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry) for comprehensive streaming governance. This operational insight ensures that your Bronze ingestion pipeline remains healthy and performant. ![data-lake-zones-bronze-silver-gold-architecture diagram 2](https://www.conduktor.io/assets/images/glossary/data-lake-zones-bronze-silver-gold-architecture-1.webp) ### Implementation Pattern ```sql -- Bronze layer typically stores data with minimal transformation -- Using Delta Lake (2025) CREATE TABLE bronze.raw_transactions ( raw_data STRING, source_system STRING, ingestion_timestamp TIMESTAMP, date_partition DATE ) USING DELTA PARTITIONED BY (date_partition) TBLPROPERTIES ( 'delta.enableChangeDataFeed' = 'true', -- Enable CDC for downstream processing 'delta.autoOptimize.optimizeWrite' = 'true' -- Automatic file sizing ); ``` For more details on modern table formats, see [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg) and [Delta Lake Transaction Log](https://www.conduktor.io/glossary/delta-lake-transaction-log-how-it-works). ## Silver Layer: Cleaned and Conformed The Silver layer represents your cleaned, validated, and standardized data. This is where data engineering rigor comes into play, transforming raw data into a reliable foundation for analytics. ### Key Characteristics - **Data quality enforcement**: Invalid records are filtered or corrected using validation rules - **Schema standardization**: Consistent data types and column names across sources - **Deduplication**: Removes duplicate records based on business keys - **Change data capture (CDC)**: Efficiently tracks and processes only changed/new records from Bronze, reducing processing costs. For detailed CDC patterns, see [What is Change Data Capture](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) - **PII handling**: Sensitive data is masked, encrypted, or tokenized for compliance ### Transformation Logic The Bronze-to-Silver transformation applies business rules while maintaining data integrity: ```sql -- Silver layer transformation example (Spark SQL / Delta Lake) CREATE TABLE silver.transactions AS WITH parsed_data AS ( SELECT CAST(get_json_object(raw_data, '$.transaction_id') AS BIGINT) AS transaction_id, CAST(get_json_object(raw_data, '$.amount') AS DECIMAL(10,2)) AS amount, CAST(get_json_object(raw_data, '$.currency') AS STRING) AS currency, to_timestamp(get_json_object(raw_data, '$.timestamp')) AS transaction_timestamp, current_timestamp() AS processed_at, ingestion_timestamp, -- Add row number for deduplication ROW_NUMBER() OVER (PARTITION BY get_json_object(raw_data, '$.transaction_id') ORDER BY ingestion_timestamp DESC) AS row_num FROM bronze.raw_transactions WHERE get_json_object(raw_data, '$.transaction_id') IS NOT NULL -- Data quality filter AND CAST(get_json_object(raw_data, '$.amount') AS DECIMAL(10,2)) > 0 -- Business rule ) SELECT transaction_id, amount, currency, transaction_timestamp, processed_at FROM parsed_data WHERE row_num = 1; -- Keep only the most recent record per transaction_id ``` ### Streaming in Silver For streaming architectures, the Silver layer often implements **incremental processing**. Modern streaming engines consume from Bronze and write to Silver in near real-time: - **Apache Spark Structured Streaming** (3.5+): Processes micro-batches with Delta Lake's merge operations - **Apache Flink** (1.18+): True streaming with exactly-once semantics and state management - **dbt** (1.7+): Incremental models for batch transformations with SQL Schema registry integration (Confluent Schema Registry or AWS Glue Schema Registry) ensures that as schemas evolve in Kafka topics, your Silver layer transformations can adapt accordingly, preventing breaking changes from propagating downstream. For more on schema management, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). ## Gold Layer: Business Value The Gold layer contains curated, business-level datasets optimized for specific consumption patterns. This is where data becomes actionable, powering dashboards, reports, and machine learning models. ### Key Characteristics - **Aggregated metrics**: Pre-calculated KPIs and business metrics for fast dashboard queries - **Denormalized structures**: Optimized for query performance with pre-joined data - **Slowly Changing Dimensions (SCD Type 2)**: Tracks historical changes by maintaining multiple versions of dimension records with effective dates - **Feature stores**: ML-ready feature sets for machine learning models - **Dimensional modeling**: Star or snowflake schemas optimized for BI tools like Tableau and Power BI ### Business-Driven Design Unlike Bronze and Silver, which are technical layers, Gold is organized around business domains: ![Unlike Bronze and Silver, which are technical layers, Gold is organized around business domains](https://www.conduktor.io/assets/images/glossary/data-lake-zones-bronze-silver-gold-architecture-2.webp) ### Example Gold Table ```sql -- Gold layer: Business-ready aggregate CREATE TABLE gold.daily_revenue_summary AS SELECT DATE(transaction_timestamp) AS revenue_date, currency, country_code, COUNT(DISTINCT customer_id) AS unique_customers, COUNT(transaction_id) AS transaction_count, SUM(amount) AS total_revenue, AVG(amount) AS avg_transaction_value, PERCENTILE(amount, 0.5) AS median_transaction_value FROM silver.transactions WHERE transaction_status = 'completed' GROUP BY DATE(transaction_timestamp), currency, country_code; ``` ## Modern Table Formats (2025) The Medallion Architecture is typically implemented using modern **lakehouse table formats** that provide ACID transactions, schema evolution, and time travel capabilities: ### Delta Lake Delta Lake is the most widely adopted format, offering: - **ACID transactions** for reliable concurrent writes - **Time travel** to query historical versions of data - **Liquid Clustering** (2025): Replaces traditional partitioning with adaptive, multi-dimensional clustering for better query performance. See [Delta Lake Liquid Clustering](https://www.conduktor.io/glossary/delta-lake-liquid-clustering-modern-partitioning) - **Deletion Vectors** (2025): Efficient row-level deletes without rewriting entire files. See [Delta Lake Deletion Vectors](https://www.conduktor.io/glossary/delta-lake-deletion-vectors-efficient-row-level-deletes) - **Change Data Feed**: Native CDC support for incremental processing ### Apache Iceberg Apache Iceberg provides enterprise-grade features: - **Hidden partitioning**: No partition columns needed in queries - **Partition evolution**: Change partitioning schemes without rewriting data - **Multi-catalog support**: Works with Hive, Glue, Nessie, and Unity Catalog - **Metadata management**: Efficient snapshot management and metadata pruning For comprehensive coverage, see [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture) and [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse). ### Implementation Example ```sql -- Modern Bronze table with Delta Lake (2025) CREATE TABLE bronze.raw_events USING DELTA TBLPROPERTIES ( 'delta.enableChangeDataFeed' = 'true', 'delta.autoOptimize.optimizeWrite' = 'true', 'delta.autoOptimize.autoCompact' = 'true' ) AS SELECT * FROM stream_source; -- Silver table with Liquid Clustering CREATE TABLE silver.events USING DELTA CLUSTER BY (event_type, date); -- Adaptive clustering replaces static partitions ``` ## Architecture Benefits ### 1. Separation of Concerns Each layer has a distinct responsibility, making the data pipeline easier to understand, maintain, and troubleshoot. ### 2. Incremental Development Teams can iterate on Silver and Gold transformations without re-ingesting raw data, as Bronze preserves the complete history. ### 3. Data Quality Gates Issues caught at Silver prevent bad data from reaching production analytics in Gold, protecting decision-making processes. ### 4. Performance Optimization By progressively refining data, queries against Gold tables execute faster than querying raw Bronze data directly. ### 5. Regulatory Compliance The immutable Bronze layer provides an audit trail, while Silver and Gold can implement data retention policies and privacy controls. ## Best Practices - **Idempotency**: Ensure all transformations can be safely re-run without creating duplicates or inconsistencies. Use merge operations (MERGE INTO in SQL) rather than INSERT to handle updates gracefully. - **Incremental Processing**: Process only new or changed data between layers to minimize compute costs. Delta Lake's Change Data Feed and Iceberg's incremental scan make this efficient. See [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables). - **Data Lineage**: Maintain metadata about data transformations to track how Gold metrics are derived from raw sources. Modern catalogs like Unity Catalog and tools like dbt automatically capture lineage. - **Schema Evolution**: Plan for schema changes by versioning your data and using backward-compatible transformations. See [Schema Evolution in Apache Iceberg](https://www.conduktor.io/glossary/schema-evolution-in-apache-iceberg) for best practices. - **Data Quality**: Implement validation checks at each layer boundary. Use frameworks like Great Expectations for comprehensive testing. See [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) and [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing). - **Monitoring**: Use **Conduktor** for Kafka/streaming monitoring (consumer lag, schema validation, throughput) and data observability platforms for data quality metrics. Implement alerting for pipeline failures and SLA violations. ## Conclusion The Bronze-Silver-Gold architecture gives data teams a clear structure for building scalable, maintainable data lakes. Separating raw data preservation (Bronze), data cleansing (Silver), and business aggregation (Gold) makes it easier to deliver high-quality analytics while adapting to changing requirements. Modern lakehouse formats (Delta Lake, Apache Iceberg) bring ACID transactions, schema evolution, and time travel to data lakes. Combined with streaming platforms like Kafka 4.0+ (with KRaft) and monitoring tools like **Conduktor**, teams can build real-time medallion architectures that process data continuously while maintaining quality and governance. Whether you're implementing batch processing or real-time streaming ingestion, the medallion pattern provides the organizational structure needed for data engineering at any scale. ## Related Concepts - [Real-Time Analytics with Streaming Data](https://www.conduktor.io/real-time-analytics-with-streaming-data) - [Data Pipeline Orchestration with Streaming](https://www.conduktor.io/data-pipeline-orchestration-with-streaming) - [Streaming ETL vs Traditional ETL](https://www.conduktor.io/streaming-etl-vs-traditional-etl) ## Related Topics ### Table Formats & Architecture - [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg) - Modern table format with schema evolution and hidden partitioning - [Delta Lake Liquid Clustering](https://www.conduktor.io/glossary/delta-lake-liquid-clustering-modern-partitioning) - 2025 approach to optimizing table layout - [Delta Lake Transaction Log](https://www.conduktor.io/glossary/delta-lake-transaction-log-how-it-works) - How ACID transactions work in Delta Lake - [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture) - Combining data lake and warehouse capabilities ### Data Quality & Processing - [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) - Implementing validation across layers - [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing) - Testing strategies for data pipelines - [What is Change Data Capture](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) - Efficient incremental processing patterns ### Streaming Integration - [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse) - Real-time data ingestion patterns - [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables) - Writing streaming data to Delta/Iceberg tables - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Streaming platform for Bronze layer ingestion ### Schema Management - [Schema Evolution in Apache Iceberg](https://www.conduktor.io/glossary/schema-evolution-in-apache-iceberg) - Handling schema changes gracefully - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing schemas in streaming pipelines ## Sources and References - [Databricks Medallion Architecture](https://www.databricks.com/glossary/medallion-architecture) - Official guide to implementing Bronze-Silver-Gold layers with Delta Lake - [Delta Lake Best Practices](https://docs.delta.io/latest/best-practices.html) - Optimizing data lake performance with ACID transactions - [Apache Iceberg Table Format](https://iceberg.apache.org/docs/latest/) - Modern table format for large-scale analytics with schema evolution - [AWS Lake Formation Best Practices](https://docs.aws.amazon.com/lake-formation/latest/dg/best-practices.html) - Data lake organization and governance on AWS - [dbt Semantic Layer](https://docs.getdbt.com/docs/use-dbt-semantic-layer/dbt-semantic-layer) - Building business-ready metrics layers on top of data lakes Source: https://www.conduktor.io/glossary/data-lake-zones-bronze-silver-gold-architecture --- # Data Lineage: Tracking Data From Source to Consumption Understanding where data comes from, how it transforms, and where it ends up is no longer optional. Data lineage provides this visibility, creating a map of your data's journey through your entire ecosystem. For data engineers managing complex pipelines and governance officers ensuring compliance, data lineage is essential. ![Data Lineage Flow](https://www.conduktor.io/assets/images/glossary/data-lineage-tracking-data-from-source-to-consumption-0.webp) ## What Is Data Lineage? Data lineage is the documentation and visualization of data's complete lifecycle. It captures the path data takes from its origin through every transformation, aggregation, and movement until it reaches its final destination, whether that's a dashboard, machine learning model, or downstream application. Think of data lineage as a genealogy tree for your data. Just as a family tree shows relationships across generations, data lineage reveals the ancestry and descendants of every data element in your system. This includes not just the "what" and "where," but also the "how" and "when" of data transformations. ## Why Data Lineage Matters ### Regulatory Compliance With regulations like GDPR, CCPA, and industry-specific mandates, organizations must demonstrate where personal or sensitive data originates, how it's processed, and who accesses it. Data lineage provides the audit trail needed to prove compliance and respond to data subject requests. For comprehensive guidance on governance frameworks, see [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities). ### Impact Analysis Before making changes to a data source or transformation logic, you need to understand the ripple effects. Data lineage enables impact analysis: if you modify a source table schema or alter a transformation rule, you can immediately identify every downstream consumer affected by that change. This prevents breaking production dashboards, reports, or ML models. ### Root Cause Analysis When data quality issues arise, incorrect metrics in a report, anomalous ML predictions, data lineage helps you trace the problem back to its source. You can navigate upstream through transformations to identify where corruption, logic errors, or unexpected data entered the pipeline. For strategies on managing quality issues, see [Data Quality Incidents](https://www.conduktor.io/glossary/data-quality-incidents) and [Data Quality vs Data Observability: Key Differences](https://www.conduktor.io/glossary/data-quality-vs-data-observability-key-differences). ### Data Discovery and Understanding New team members or data consumers need to understand what data exists and where it comes from. Data lineage serves as living documentation, helping users discover datasets and understand their trustworthiness based on source quality and transformation complexity. For more on enabling discovery, see [What is a Data Catalog? Modern Data Discovery](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery). ## Components of Effective Data Lineage ### Source-Level Lineage This captures where data originates: databases, APIs, file systems, message queues, or streaming platforms like Apache Kafka. For streaming architectures, source-level lineage must track not just Kafka topics but also schema registry versions, producer applications, and partition strategies. ### Column-Level Lineage Beyond tracking tables or datasets, column-level lineage maps individual fields through transformations. This granularity is crucial for compliance scenarios where you need to track specific personally identifiable information (PII) fields or for understanding how a specific metric is calculated. ### Transformation Lineage Every SQL query, ETL job, streaming processor, or script that modifies data should be captured in your lineage graph. This includes the logic applied, the execution schedule or trigger, and the technology used (dbt models, Spark jobs, Flink applications, etc.). ### Operational Metadata Lineage isn't complete without context: when did transformations run? How long did they take? Did they succeed or fail? Were there data quality validations applied? This operational layer enriches lineage with the temporal and health dimensions. ## Data Lineage in Streaming Architectures Traditional batch-oriented lineage tools often struggle with streaming pipelines where data flows continuously through distributed systems. Streaming introduces unique challenges: ### Real-Time Tracking In streaming architectures built on Apache Kafka, data flows through topics, is processed by stream processors (Kafka Streams, Flink, Spark Streaming), and may be enriched by joining with other streams or reference data. Lineage tools must capture these relationships in real-time, not just after batch completion. ### Schema Evolution Streaming systems frequently handle schema evolution, where message structures change over time. Lineage systems must track schema versions from registries and map how downstream consumers adapt to these changes. For detailed coverage of managing schema changes, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). ### Complex Topology A single event might trigger transformations across multiple microservices, each publishing to different topics. Lineage must represent these complex, event-driven topologies, showing how an event in one topic cascades through the system. ## Implementing Data Lineage ### Automated Extraction Manual lineage documentation fails quickly in dynamic environments. Modern approaches use automated extraction: - **Query parsing**: Analyzing SQL queries, dbt models, or transformation code to extract source-to-target relationships - **Metadata APIs**: Integrating with data platforms (Snowflake, Databricks, BigQuery) that expose lineage through APIs - **Log analysis**: Parsing execution logs to infer data movement patterns - **Instrumentation**: Embedding lineage capture directly into ETL frameworks or streaming applications ### OpenLineage: The Standard for Lineage Collection OpenLineage has emerged as the open standard for data lineage collection, providing a vendor-neutral specification for capturing lineage metadata. It defines a common event model that pipelines emit at runtime, enabling consistent lineage tracking across heterogeneous systems. Key features of OpenLineage (2025): - **Universal integration**: Supported by Apache Airflow, Spark, Flink, dbt, Dagster, and major orchestration platforms - **Real-time emission**: Lineage events are emitted as pipelines run, not reconstructed later - **Facets system**: Extensible metadata model capturing data quality metrics, schema versions, and operational context - **Backend flexibility**: Events can be sent to Marquez, DataHub, Apache Amundsen, or custom backends Example OpenLineage event for a Kafka streaming job: ```python from openlineage.client import OpenLineageClient from openlineage.client.run import RunEvent, RunState, Job from openlineage.client.facet import DataQualityMetricsInputDatasetFacet client = OpenLineageClient(url="http://marquez:5000") # Emit start event for Flink job reading from Kafka client.emit( RunEvent( eventType=RunState.START, job=Job(namespace="kafka-streaming", name="fraud-detection-processor"), inputs=[{ "namespace": "kafka://prod-cluster", "name": "transactions-topic", "facets": { "schema": { "fields": [ {"name": "transaction_id", "type": "string"}, {"name": "amount", "type": "decimal"}, {"name": "timestamp", "type": "timestamp"} ] } } }], outputs=[{ "namespace": "kafka://prod-cluster", "name": "flagged-transactions-topic" }] ) ) ``` This approach provides automatic lineage tracking without manual documentation. ### Modern Lineage Platforms Several open-source and commercial platforms now provide comprehensive lineage tracking: **Open-Source Solutions:** - **DataHub** (LinkedIn): Graph-based metadata platform with rich lineage visualization, support for Kafka, Flink, Spark, and batch systems. Includes impact analysis and data quality integration. - **Apache Amundsen** (Lyft): Focuses on data discovery with lineage as a core feature. Strong integration with Airflow and batch pipelines. - **Marquez**: OpenLineage reference implementation, purpose-built for collecting and visualizing lineage from diverse sources. **For Kafka Streaming Platforms:** For organizations running Apache Kafka, **Conduktor Gateway** acts as a transparent proxy between Kafka clients and brokers, capturing all produce/consume activity in audit logs. This connection-level visibility gives teams insight into which applications interact with which topics, providing a foundation for understanding data flows across the Kafka environment without modifying application code. Key capabilities: - Audit logs capturing all producer and consumer connections across topics - Visibility into which applications write to and read from which topics - Schema Registry integration for schema validation and management - Policy enforcement and access controls for Kafka resources - Audit trails for compliance and governance ### Integration with Data Catalogs Data lineage works best when integrated with a comprehensive data catalog. Modern governance platforms provide topic catalogs and lineage visualization showing how data flows through topics, connectors, and stream processors. This integration gives data engineers a single pane of glass to understand both what data exists and how it moves through streaming platforms. For batch and lakehouse architectures, tools like dbt automatically generate lineage from SQL transformations, while Spark lineage can be captured through OpenLineage integrations. The key is choosing platforms that support your specific technology stack and can aggregate lineage from multiple sources into a unified view. ### Visualization and Querying Lineage data must be queryable and visual. Graph databases often underpin lineage systems, enabling queries like "show me all tables dependent on this source" or "trace this field back to its origin." Visualization tools render these graphs as interactive diagrams where users can explore upstream and downstream dependencies. ### Lineage in Modern Transformation Tools **dbt (Data Build Tool)** has revolutionized lineage tracking for SQL-based transformations. Every dbt model automatically generates lineage by parsing SQL dependencies: ```yaml # dbt generates lineage from model dependencies # models/marts/customer_lifetime_value.sql select c.customer_id, c.customer_name, sum(o.order_amount) as lifetime_value from {{ ref('stg_customers') }} c join {{ ref('stg_orders') }} o on c.customer_id = o.customer_id group by c.customer_id, c.customer_name ``` dbt's `ref()` function creates explicit lineage relationships, and running `dbt docs generate` produces interactive lineage graphs showing all model dependencies. This lineage is also exportable to platforms like DataHub or Amundsen through dbt's metadata artifacts. ### Integrating Data Quality with Lineage Modern lineage platforms integrate data quality metrics directly into lineage graphs. When a data quality test fails, lineage helps identify: - **Upstream impact**: Which source or transformation introduced the issue? - **Downstream impact**: Which reports, dashboards, or ML models are affected? Tools like **Soda Core** and **Great Expectations** can emit OpenLineage events containing data quality results, allowing platforms to display quality scores alongside lineage paths. This creates "data trust graphs" where users see not just where data comes from, but how reliable it is. Example integration: ```python # Great Expectations with OpenLineage from great_expectations.data_context import DataContext from openlineage.client import OpenLineageClient context = DataContext() results = context.run_checkpoint("daily_validation") # Emit data quality facet to lineage platform client = OpenLineageClient(url="http://marquez:5000") client.emit_data_quality_facet( dataset="orders_table", metrics={ "row_count": 50000, "null_percentage": 0.01, "validation_success": results.success } ) ``` ## Best Practices for Data Lineage ### Start with Critical Paths Don't attempt to capture lineage for every data flow immediately. Begin with business-critical pipelines: those feeding executive dashboards, regulatory reports, or revenue-impacting ML models. Prove value, then expand coverage. ### Automate Everything Manual lineage documentation is dead on arrival. Invest in automation and integration with your existing tools. If you use dbt, leverage its built-in lineage. If you run Spark jobs, instrument them to emit lineage metadata. ### Include Business Context Technical lineage alone isn't sufficient. Enrich your lineage with business glossary terms, data ownership information, and quality metrics. When a business analyst looks at lineage, they should understand not just the technical path but also what the data means. For guidance on building business context, see [Building a Business Glossary for Data Governance](https://www.conduktor.io/glossary/building-a-business-glossary-for-data-governance) and [Metadata Management: Technical vs Business Metadata](https://www.conduktor.io/glossary/metadata-management-technical-vs-business-metadata). ### Keep It Fresh Stale lineage is worse than no lineage, it creates false confidence. Ensure your lineage system captures changes automatically and reflects the current state of your pipelines, not last quarter's architecture. ### Enable Self-Service Data lineage should be accessible to everyone who needs it: data engineers debugging pipelines, analysts understanding report sources, and governance officers auditing compliance. Provide intuitive interfaces and search capabilities so users can answer their own lineage questions. ## Conclusion Data lineage transforms data infrastructure from a black box into a transparent, understandable system. For data engineers, it's a debugging tool and change management safeguard. For governance officers, it's the foundation of compliance and trust. In streaming architectures where data flows continuously through distributed systems, lineage tracking is especially important. As data ecosystems grow more complex — spanning batch and streaming pipelines, cloud and on-premise systems, structured and unstructured data — investing in data lineage is a requirement for maintaining control and ensuring quality. Implementing automated lineage capture, integrating with catalogs and governance tools, and making lineage accessible across your organization turns data from an opaque resource into a well-understood, trustworthy asset. ## Related Concepts - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing schema evolution and versioning in data pipelines - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) - Organizational structures for data governance - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Tracking streaming data freshness and pipeline health ## Sources - [OpenLineage: Open Standard for Data Lineage](https://openlineage.io/) - [DataHub: Metadata Platform with Lineage](https://datahubproject.io/) - [Apache Amundsen: Data Discovery and Lineage](https://www.amundsen.io/) - [Marquez: OpenLineage Reference Implementation](https://marquezproject.ai/) - [dbt Documentation and Lineage](https://docs.getdbt.com/docs/collaborate/documentation) - [Great Expectations Data Quality](https://greatexpectations.io/) - [Soda Core: Data Quality Testing](https://www.soda.io/core) - [Conduktor: Kafka Governance Platform](https://www.conduktor.io/) Source: https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption --- # Data Masking and Anonymization for Streaming As organizations increasingly rely on real-time data pipelines to power applications, analytics, and machine learning, protecting sensitive information in streaming systems has become critical. Unlike batch processing where data can be transformed before storage, streaming data moves continuously through pipelines, requiring protection mechanisms that work at scale and in real time. Data masking and anonymization are two complementary approaches to protecting sensitive information. They share the goal of preventing unauthorized access to personal or confidential data, but differ in purpose and technique. Knowing how to apply these methods in streaming architectures is important for compliance, security, and trust. ## Understanding Masking and Anonymization - **Data masking** replaces sensitive data with realistic but fictitious values, preserving the format and type of the original data. The goal is to create a version of the data that can be used for development, testing, or analytics without exposing actual sensitive information. Masked data is reversible in some implementations (like tokenization), allowing authorized systems to retrieve the original values when necessary. - **Anonymization** goes further by removing or transforming data in a way that makes it impossible or extremely difficult to identify individuals. Anonymized data is typically irreversible and aims to meet strict privacy regulations like GDPR, which requires that personal data be processed in a way that prevents identification. In streaming systems, both techniques must be applied while data is in motion, adding complexity around performance, consistency, and maintaining referential integrity across multiple data streams. ## The Business Case for Data Protection in Streaming Several factors drive the need for data protection in streaming architectures: - **Compliance requirements** like GDPR, CCPA, and HIPAA mandate specific handling of personal data. GDPR Article 32 requires "pseudonymization and encryption of personal data," and violations can result in fines up to 4% of global revenue. Streaming systems that handle customer data, financial transactions, or health records must implement protection mechanisms to avoid legal and financial consequences. For comprehensive GDPR implementation guidance, see [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams). - **Security and access control** become more complex in streaming environments where data flows through multiple systems and teams. Development teams need realistic data for testing, analytics teams require access to aggregated insights, and production systems need full fidelity. Each use case requires different levels of data protection. For strategies on implementing role-based and attribute-based access controls, see [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming) and [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). - **Data sharing and partnerships** often involve sending data to third parties or across organizational boundaries. Masking and anonymization enable safer data sharing by reducing the risk of exposing sensitive information while maintaining the utility of the data for its intended purpose. The real-time nature of streaming systems means that protection mechanisms must be applied with minimal latency, making the choice of technique and implementation approach critical. ## Data Masking Techniques for Streaming Several masking techniques can be applied to streaming data, each with different characteristics: - **Static masking** involves creating a masked copy of data before it enters the stream. This approach is simple and performant but doesn't handle new or updated records dynamically. It's best suited for datasets that change infrequently. - **Dynamic masking** applies transformations in real-time as data flows through the pipeline. This ensures all data is protected regardless of when it enters the system. Common dynamic masking methods include: - **Substitution**: Replacing real values with fake but realistic data (e.g., replacing "John Doe" with "Jane Smith") - **Shuffling**: Randomly redistributing values within a column to break associations with other attributes - **Nulling or redaction**: Removing sensitive fields entirely or replacing them with asterisks - **Encryption**: Transforming data using cryptographic algorithms, making it unreadable without the decryption key. For end-to-end encryption strategies, see [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka) - **Format-preserving encryption (FPE)** is particularly useful in streaming contexts because it maintains the data type and format. For example, a 16-digit credit card number remains a 16-digit number after encryption, ensuring downstream systems continue to function without modification. - **Tokenization** replaces sensitive data with randomly generated tokens stored in a secure vault. The streaming system works with tokens while the actual values remain protected. This is reversible and maintains referential integrity across streams. Here's an example of how customer data might be masked in a Kafka stream: ```json // Original message { "customerId": "CUST12345", "email": "john.doe@example.com", "ssn": "123-45-6789", "purchaseAmount": 299.99, "timestamp": "2025-03-15T10:30:00Z" } // Masked version for analytics team { "customerId": "CUST12345", "email": "j***@example.com", "ssn": "***-**-6789", "purchaseAmount": 299.99, "timestamp": "2025-03-15T10:30:00Z" } ``` The customerId is preserved for analytics, while email and SSN are partially masked, and the purchase amount remains unchanged for business analysis. ## Anonymization Strategies for Streaming Anonymization techniques aim to prevent re-identification while preserving data utility: - **Pseudonymization** replaces identifying fields with pseudonyms. While technically reversible (unlike true anonymization), it provides strong protection when the mapping between pseudonyms and real identities is securely stored separately. - **Aggregation and generalization** reduce precision to prevent identification. For streaming location data, instead of transmitting precise GPS coordinates (37.7749° N, 122.4194° W), the system might generalize to city level (San Francisco, CA) or zip code (94102). - **K-anonymity** ensures that any individual's data cannot be distinguished from at least k-1 other individuals. In a stream processing context, this might involve buffering records and only releasing them once the anonymity threshold is met. For example, a Flink job could aggregate user events by generalizing age to ranges (20-30, 30-40) and locations to regions until each group contains at least 5 individuals. - **Differential privacy** adds carefully calibrated noise to data to prevent identification while preserving statistical properties. This is particularly useful for real-time analytics streams where aggregate metrics (counts, averages) need to be computed without exposing individual records. The challenge in streaming systems is applying these techniques while maintaining low latency and handling the continuous nature of the data. ## Implementation in Streaming Platforms Implementing data protection in streaming architectures requires deciding where to apply transformations in the pipeline: - **At the producer level**: Applications can mask or anonymize data before publishing to Kafka topics. This ensures sensitive data never enters the streaming platform. However, it requires modifying every producer application and doesn't provide flexibility for different consumer needs. - **At the broker or platform level**: Kafka brokers can be configured with stream processors that apply transformations. This centralizes the logic and doesn't require changing applications. Platforms like Confluent Schema Registry can enforce data contracts, and tools like Conduktor provide [data masking capabilities](https://docs.conduktor.io/guide/conduktor-in-production/admin/data-mask) at the platform level, allowing teams to define and enforce masking policies centrally without modifying individual applications. - **Within stream processors**: Apache Flink, Kafka Streams, or ksqlDB can apply transformations as data flows through processing topologies. This approach offers flexibility to create multiple views of the same data for different purposes. - **At the consumer level**: Consumers can apply transformations when reading data. This provides maximum flexibility but requires implementing protection logic in every consumer and doesn't prevent unauthorized consumers from accessing unprotected data. A common pattern is to use topic-based access control combined with stream processors that publish different versions of data: ![A common pattern is to use topic-based access control combined with stream processors that publish different versions of data](https://www.conduktor.io/assets/images/glossary/data-masking-and-anonymization-for-streaming-0.webp) Production systems read from raw topics, analytics teams use masked versions, and public dashboards consume aggregated metrics. ## Modern Tools and 2025 Capabilities Several tools and platform capabilities have changed how teams approach data masking in 2025: - **Kafka 4.0+ with KRaft** brings improved security foundations that enhance masking implementations. With ZooKeeper removed, the simplified architecture makes it easier to implement consistent security policies across the cluster. KRaft's metadata management provides better audit trails for tracking which transformations are applied to which topics. For comprehensive audit logging strategies, see [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). - **Conduktor Gateway** offers proxy-level data masking that sits between producers/consumers and Kafka brokers. This approach allows teams to define masking policies centrally without modifying application code. Gateway can apply different masking rules based on the consumer's identity, enabling a single topic to serve multiple teams with different data access levels. For example: ```yaml # Conduktor Gateway masking policy interceptors: - type: dataMasking config: rules: - field: $.email maskingType: EMAIL_DOMAIN consumerGroup: analytics-team - field: $.ssn maskingType: HASH_SHA256 consumerGroup: ml-training - field: $.creditCard maskingType: REDACT consumerGroup: support-team ``` **Apache Flink 1.18+** introduced enhanced Python APIs and improved state backends that make implementing custom masking functions more accessible. The new hybrid state backend optimizes performance for stateful masking operations like k-anonymity buffering: ```python # Flink 1.18+ Python API for masking from pyflink.datastream import StreamExecutionEnvironment from pyflink.datastream.functions import MapFunction class MaskSensitiveData(MapFunction): def map(self, event): event['email'] = self.mask_email(event['email']) event['ssn'] = self.hash_ssn(event['ssn']) return event def mask_email(self, email): username, domain = email.split('@') return f"{username[0]}***@{domain}" def hash_ssn(self, ssn): import hashlib return hashlib.sha256(ssn.encode()).hexdigest()[:16] env = StreamExecutionEnvironment.get_execution_environment() stream = env.from_source(kafka_source, watermark_strategy, "customer-events") masked_stream = stream.map(MaskSensitiveData()) ``` **Kafka Connect Single Message Transforms (SMTs)** provide a lightweight approach for masking data in transit. The `MaskField` and `ReplaceField` transformations can be chained to protect sensitive data without writing custom code. For more details on SMT patterns, see [Kafka Connect Single Message Transforms](https://www.conduktor.io/glossary/kafka-connect-single-message-transforms): ```json { "name": "customer-events-connector", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "transforms": "maskEmail,hashSSN", "transforms.maskEmail.type": "org.apache.kafka.connect.transforms.MaskField$Value", "transforms.maskEmail.fields": "email", "transforms.maskEmail.replacement": "***MASKED***", "transforms.hashSSN.type": "org.apache.kafka.connect.transforms.HoistField$Value", "transforms.hashSSN.field": "ssn" } } ``` **Kafka Streams with stateful masking** enables more sophisticated anonymization patterns. The Processor API allows implementing k-anonymity by buffering records until the anonymity threshold is met: ```java // Kafka Streams k-anonymity processor (Kafka 3.6+) public class KAnonymityProcessor implements Processor { private KeyValueStore> buffer; private final int k = 5; @Override public void process(Record record) { String generalizedKey = generalize(record.value()); List group = buffer.get(generalizedKey); if (group == null) group = new ArrayList<>(); group.add(record.value()); if (group.size() >= k) { // Release anonymized group group.forEach(event -> context().forward( new Record<>(generalizedKey, anonymize(event), record.timestamp()) )); buffer.delete(generalizedKey); } else { buffer.put(generalizedKey, group); } } } ``` **Schema Registry integration** has improved with better support for tracking masked fields through schema metadata. Using Avro or Protobuf with Schema Registry allows teams to mark fields as sensitive and enforce masking at the serialization layer. For comprehensive schema management patterns, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). For PII-specific handling patterns and detection techniques, see [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). ## Challenges and Trade-offs Implementing data protection in streaming systems involves several challenges: - **Performance impact**: Encryption, tokenization, and anonymization operations add processing overhead. Format-preserving encryption can be computationally expensive, and techniques like k-anonymity may require buffering data, increasing latency. - **Referential integrity**: When the same entity appears in multiple streams (e.g., customer data in order events and support tickets), applying consistent masking is critical. A tokenization service must return the same token for the same input to maintain relationships across topics. - **Join operations**: Anonymization that generalizes or adds noise can make joining streams difficult or impossible. If one stream generalizes age to ranges and another keeps exact ages, correlating records becomes problematic. - **Key handling**: For techniques that use encryption or tokenization, securely managing and rotating keys in a distributed streaming environment requires careful architecture. Keys must be accessible to authorized processors while remaining protected from unauthorized access. - **Compliance complexity**: Different regulations have different requirements. GDPR's "right to be forgotten" requires the ability to delete or anonymize specific individuals' data, which is challenging in append-only systems like Kafka where historical data is immutable. For strategies on enforcing data policies, see [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming). - **State management**: Stream processors applying anonymization techniques like k-anonymity need to maintain state about which records have been processed and released, adding operational complexity. Teams must balance protection requirements with performance, functionality, and operational overhead based on their specific compliance needs and use cases. ## Summary Data masking and anonymization protect sensitive information in streaming systems. Masking preserves data format while hiding actual values, making it suitable for non-production environments and analytics. Anonymization removes identifying information to prevent re-identification and meets stricter compliance requirements. Streaming systems add complexity because data is in constant motion, performance is critical, and different consumers often need different levels of access. Implementation approaches range from producer-side transformation to platform-level enforcement to stream processor-based filtering, each with different trade-offs. Key considerations: choosing the right technique (masking for flexibility vs. anonymization for compliance), deciding where in the pipeline to apply transformations (closer to producers for security, closer to consumers for flexibility), managing performance impact, and maintaining referential integrity across streams. ## Related Concepts - [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams) - Detecting PII in streams before applying masking and anonymization techniques. - [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams) - How masking and anonymization support GDPR requirements like pseudonymization and data minimization. - [Data Classification and Tagging Strategies](https://www.conduktor.io/glossary/data-classification-and-tagging-strategies) - Classifying data sensitivity to determine appropriate masking and anonymization approaches. ## Sources and References 1. **European Union General Data Protection Regulation (GDPR)** - Official regulation text covering requirements for personal data protection, pseudonymization, and the right to erasure. Available at: https://gdpr-info.eu/ 2. **Apache Kafka Security Documentation** - Comprehensive guide to authentication, authorization, and encryption in Kafka. Available at: https://kafka.apache.org/documentation/#security 3. **NIST Special Publication 800-188: De-Identifying Government Datasets** - Technical guidance on de-identification techniques, risk assessment, and best practices from the National Institute of Standards and Technology. Available at: https://csrc.nist.gov/publications/detail/sp/800-188/final 4. **Differential Privacy: A Survey of Results** (Cynthia Dwork, 2008) - Foundational academic paper on differential privacy theory and applications in data release and analysis. 5. **Apache Flink Documentation** - Official documentation for Apache Flink stream processing including state management and masking patterns. Available at: https://flink.apache.org/ 6. **Conduktor Documentation** - Platform guides for data masking, governance, and security in Kafka environments. Available at: https://docs.conduktor.io/ Source: https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming --- # Data Mesh Principles and Implementation As organizations scale their data operations, traditional centralized data architectures often become bottlenecks. Data lakes and warehouses promise unified access to data, but they concentrate ownership and expertise in a single team, creating dependencies and slowing down the entire organization. Data Mesh emerged as an alternative that decentralizes data ownership while maintaining discoverability and governance. ## Introduction to Data Mesh Data Mesh is a socio-technical approach to data architecture introduced by Zhamak Dehghani in 2019. Rather than treating data as a byproduct collected in a central repository, Data Mesh treats data as a product owned and maintained by the teams closest to its source. **Key Terms:** - **SLA (Service Level Agreement)**: A commitment about performance metrics like availability (99.9% uptime) or latency (< 5 minute delay) - **ETL (Extract, Transform, Load)**: The traditional process of moving data from sources to a data warehouse - **Domain**: A distinct area of business logic (e.g., orders, customers, inventory in an e-commerce system) - **Avro**: A binary data serialization format commonly used with Kafka for schemas The core insight is that the team building an e-commerce checkout system understands checkout data better than any central data team ever could. They know its semantics, quality issues, and how it should be modeled. Data Mesh puts these teams in charge of publishing high-quality data products that other teams can consume. This is a shift from "data as a side effect" to "data as a first-class concern" within each domain. ## The Four Core Principles Data Mesh is built on four foundational principles that work together to enable decentralized data architecture. ### Domain-Oriented Decentralized Data Ownership Instead of a central data team owning all data pipelines and transformations, each business domain owns its data products. In an e-commerce company, the customer domain would own customer data products, the inventory domain would own inventory data products, and so on. This ownership includes the responsibility for data quality, schema evolution, documentation, and SLAs. Domain teams understand their data's context and can make informed decisions about its structure and evolution. ### Data as a Product Each domain treats its data as a product with consumers, quality standards, and lifecycle management. A data product is not just raw data, it's discoverable, addressable, trustworthy, self-describing, and secure. For example, a "customer-events-v1" data product might include: - A well-defined schema with semantic descriptions - Quality metrics and SLAs (e.g., 99.9% availability, < 5 minute latency) - Clear versioning and deprecation policies - Access controls and compliance documentation - Usage examples and documentation For detailed guidance on building and managing data products, see [Building and Managing Data Products](https://www.conduktor.io/glossary/building-and-managing-data-products). For governance aspects, refer to [Data Product Governance](https://www.conduktor.io/glossary/data-product-governance). ### Self-Serve Data Infrastructure as a Platform To enable domain teams to own their data products without becoming infrastructure experts, organizations need a self-serve data platform. This platform provides standardized tools for creating, publishing, discovering, and consuming data products. The platform handles cross-cutting concerns like deployment, monitoring, access control, and compliance, while domain teams focus on their specific data products. Think of it as "infrastructure as code" but for data products. ### Federated Computational Governance Rather than centralized control, governance in Data Mesh is federated across domains but standardized through computational policies. In simpler terms: each domain team governs their own data products, but they all follow the same automated rules that are enforced by code rather than manual reviews. For instance, policies might automatically enforce: - Schema compatibility rules (e.g., no breaking changes without version bumps) - Data privacy regulations (e.g., PII must be encrypted) - Naming conventions and metadata requirements - Quality thresholds before publishing This allows domains to move fast while maintaining organizational standards. For comprehensive coverage of governance frameworks, see [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities). ## Data Mesh vs Traditional Architectures Traditional data architectures centralize data in warehouses or lakes managed by specialized teams. While this provides a single source of truth, it creates several problems: ![Traditional data architectures centralize data in warehouses or lakes managed by specialized teams. While this provides a single source of truth, it creates several problems](https://www.conduktor.io/assets/images/glossary/data-mesh-principles-and-implementation-0.webp) - **Scaling bottlenecks**: All data requests flow through the central team, creating dependencies and delays. - **Context loss**: The central team lacks deep domain knowledge, leading to mismodeled data and misunderstood semantics. - **Coupling**: Changes in source systems require coordination with the central team, slowing down both teams. Data Mesh inverts this model. Instead of data flowing into a central repository, domain teams publish data products that consumers access directly. The central platform team provides tooling, not data. This is similar to how microservices architecture moved from monolithic applications to decentralized services owned by different teams. ## Implementation Challenges and Considerations Implementing Data Mesh is not purely technical, it requires organizational change. - **Cultural shift**: Teams must take on new responsibilities for data quality and consumer support. This requires training, tooling, and incentive alignment. - **Duplication concerns**: Multiple domains might model similar concepts differently (e.g., "customer" defined differently by marketing and billing). This requires good discovery and documentation to help consumers find the right data products. - **Platform investment**: Building a self-serve data platform requires significant upfront investment before domain teams can become productive. - **Governance complexity**: Federated governance requires sophisticated automation to enforce policies without creating bottlenecks. Organizations should start small, typically with one or two domains, prove the value, and then expand incrementally. ## Partner Zones for Federated Data Sharing A practical challenge in Data Mesh implementations is enabling secure data sharing between domains or with external partners while maintaining single source of truth. Conduktor Partner Zones address this by providing dedicated zones with customized access to Kafka topics without data duplication, domain teams share data products with specific partners while maintaining the original topic as the authoritative source. Partner Zones also provide traffic analytics showing consumption patterns within each zone, enabling domain teams to understand usage and make informed decisions about their data products' evolution. Learn more in the [Partner Zones documentation](https://docs.conduktor.io/guide/conduktor-concepts/partner-zones). ## Data Mesh and Data Streaming Data Mesh and streaming technologies like Apache Kafka are natural complements. Kafka's architecture aligns closely with Data Mesh principles. ![data-mesh-principles-and-implementation diagram 2](https://www.conduktor.io/assets/images/glossary/data-mesh-principles-and-implementation-1.webp) - **Decentralized ownership**: Kafka topics can be owned and managed by individual domain teams, who publish events representing their domain's data products. - **Real-time data products**: Instead of batch-only data products, streaming enables real-time data products. A "customer-events-v1" Kafka topic becomes a continuously updated data product. - **Schema governance**: Kafka's Schema Registry provides the foundation for data contracts, version management, and compatibility checks, key requirements for the "data as a product" principle. For details on schema management strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). - **Self-serve infrastructure**: Platforms like Conduktor provide a self-serve layer on top of Kafka, allowing domain teams to create topics, manage schemas, and configure access controls through intuitive interfaces without requiring deep Kafka expertise. This addresses the "self-serve infrastructure" principle by democratizing access to streaming infrastructure. See the [Conduktor Self-Service documentation](https://docs.conduktor.io/guide/conduktor-concepts/self-service) for details on enabling teams to manage their own data products. Consumers can discover this data product through a catalog, understand its schema and SLAs, and request access, all without contacting the order team directly. Conduktor Gateway helps with federated governance by acting as a proxy layer that enforces naming conventions, schema compatibility rules, data quality checks, and access policies automatically across all domains. This lets teams operate independently while organizational standards are still enforced. For comprehensive guidance on building data products with proper governance, see [Conduktor's Data Products guide](https://docs.conduktor.io/guide/conduktor-concepts/data-products). ## Modern Data Mesh Patterns (2024-2025) As Data Mesh has matured, several patterns and practices have emerged as industry standards: ### Data Contracts Data contracts have become essential for Data Mesh implementations. A data contract is a formal agreement between data producers and consumers that specifies: ```yaml # Example data contract for a data product contract: name: order-events-v1 owner: orders-domain-team schema: format: avro version: 1.2.0 compatibility: backward sla: availability: 99.9% latency_p99: 1000ms freshness: 60s quality: completeness: 99% validity_rules: - order_total > 0 - customer_id is not null ``` Data contracts provide clear expectations and enable automated validation. When a producer attempts to publish data that violates the contract, it's rejected immediately, preventing data quality issues from propagating downstream. ### Data Product Observability Modern Data Mesh implementations include comprehensive observability for each data product: - **Performance metrics**: Throughput, latency, consumer lag - **Quality metrics**: Completeness, validity, consistency scores - **Usage analytics**: Active consumers, query patterns, access frequency - **Cost attribution**: Infrastructure costs allocated per data product This observability enables domain teams to treat data products like software products, with clear SLIs (Service Level Indicators) and SLOs (Service Level Objectives). For understanding quality metrics, see [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). ### Kafka 4.0 and KRaft Apache Kafka 4.0 (released 2024) eliminates ZooKeeper dependency through KRaft consensus. This simplifies Data Mesh implementations by: - Reducing infrastructure complexity (fewer components to manage) - Improving metadata operations (faster topic creation, partition rebalancing) - Enabling better multi-tenancy through improved controller scalability For Data Mesh, this means domain teams can create and manage data products with less operational overhead and faster provisioning times. ### Cost Allocation and Chargeback As Data Mesh scales, organizations need mechanisms to allocate infrastructure costs to domain teams. Modern implementations track: - Storage costs per topic (data product) - Compute costs for processing (Kafka Streams, Flink applications) - Network transfer costs - Schema Registry and catalog usage This enables internal chargeback models where domain teams have "skin in the game" for their data products' efficiency. Conduktor provides built-in [Chargeback capabilities](https://docs.conduktor.io/guide/conduktor-concepts/chargeback) to track and allocate streaming costs to teams and data products. ## Practical Implementation Steps Organizations looking to adopt Data Mesh should follow an incremental approach: 1. **Identify pilot domains**: Choose one or two domains with clear ownership and well-understood data to start. 2. **Define data product standards**: Establish what constitutes a good data product, schema requirements, documentation, SLAs, etc. 3. **Build platform capabilities**: Start with basic self-serve capabilities like topic creation, schema management, and access control. Expand based on domain team feedback. For topic design guidance, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines). 4. **Implement computational governance**: Encode key policies as automated checks (schema validation, naming conventions, quality metrics). 5. **Create discovery and cataloging**: Make data products discoverable through a catalog that shows available products, their schemas, owners, and SLAs. 6. **Measure and iterate**: Track metrics like time-to-publish for new data products, consumer satisfaction, and data quality incidents. Use these to improve the platform. The goal is not to achieve perfect Data Mesh from day one, but to build capabilities incrementally while proving value. ## Summary Data Mesh moves from centralized control to decentralized ownership. Its four principles — domain-oriented ownership, data as a product, self-serve infrastructure, and federated governance — work together to enable scalable data operations. Implementation requires significant organizational and technical investment, but the payoff is faster data product development, better data quality through domain expertise, and fewer bottlenecks. Streaming technologies like Apache Kafka are a natural fit for Data Mesh, enabling real-time data products with strong governance through schemas and automated policies. Start small, prove value with pilot domains, and expand incrementally. ## Related Concepts - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) - Essential governance structures for implementing federated computational governance in data mesh architectures - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Critical infrastructure for maintaining data contracts and schema governance across domain-owned data products - [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) - Formalized agreements between data producers and consumers that enable data-as-a-product principles ## Sources and References 1. Dehghani, Zhamak. "Data Mesh: Delivering Data-Driven Value at Scale." O'Reilly Media, 2022. 2. Dehghani, Zhamak. "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh." Martin Fowler's website, 2019. [https://martinfowler.com/articles/data-monolith-to-mesh.html](https://martinfowler.com/articles/data-monolith-to-mesh.html) 3. Dehghani, Zhamak. "Data Mesh Principles and Logical Architecture." Martin Fowler's website, 2020. [https://martinfowler.com/articles/data-mesh-principles.html](https://martinfowler.com/articles/data-mesh-principles.html) 4. Apache Kafka Documentation. "Kafka Architecture and Design." Official documentation covering topics, schemas, and governance. [https://kafka.apache.org/documentation/](https://kafka.apache.org/documentation/) 5. Thoughtworks Technology Radar. "Data Mesh." Multiple editions covering Data Mesh adoption and patterns. [https://www.thoughtworks.com/radar](https://www.thoughtworks.com/radar) Source: https://www.conduktor.io/glossary/data-mesh-principles-and-implementation --- # Data Obesity: When Data Infrastructure Becomes Bloated Organizations face a counterintuitive problem: too much data can be as harmful as too little. Data obesity occurs when an organization's data infrastructure becomes bloated because the volume of ingested data significantly outpaces the organization's ability to extract value from it. Unlike healthy data growth, where increasing volumes correlate with improved insights and measurable business outcomes, data obesity is accumulation without purpose. The system becomes burdened by petabytes of information that slow operations, inflate costs, and reduce agility. This phenomenon is particularly acute in streaming architectures, where the velocity of data ingestion can mask underlying inefficiencies until performance degradation becomes severe. ![Data Obesity Symptoms](https://www.conduktor.io/assets/images/glossary/data-obesity-0.webp) ## The Three Symptoms of Data Obesity ### System Lethargy: Performance Impact The most immediate symptom of data obesity is degraded performance. Massive, uncurated datasets slow down every operation that touches them. Query latency increases. Batch jobs take hours instead of minutes. Real-time systems start to lag. In streaming environments, this manifests as "fat" payloads that cause backpressure (downstream processing slowdowns due to overwhelming data volume) throughout the pipeline. When event producers serialize entire database rows, including large BLOB fields, audit columns, and metadata that downstream consumers never touch, they force every component in the chain to process, transmit, and store unnecessary bytes. For detailed coverage of backpressure patterns, see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems). A Kafka consumer that needs only a customer ID and purchase amount shouldn't receive a 50KB payload containing product images, full address history, and marketing preferences. Yet this pattern is common in Change Data Capture (CDC), the process of capturing database changes as events, implementations where convenience takes precedence over efficiency. For more on CDC patterns, refer to [What is Change Data Capture (CDC)](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). ### Operational Rigidity: Loss of Agility Obese data systems are difficult to maneuver. What should be routine operations, upgrading a database, migrating to a new cloud region, implementing a new storage tier, become high-risk, multi-month projects. The cost isn't just time. When your streaming platform contains 500TB of data with 90-day retention, even small architectural changes require careful choreography. Reprocessing historical data becomes prohibitively expensive. Testing new features against realistic data volumes becomes impractical. Teams become locked into legacy architectures simply because the migration path is too daunting. For capacity planning strategies to avoid this rigidity, see [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning). This rigidity creates a dangerous cycle: the system becomes harder to change, so technical debt accumulates, making future changes even more difficult. ### Network Congestion: The "Junk Food" Effect Not all data provides equal nutritional value. Ingesting low-value data acts like consuming empty calories, it fills up your infrastructure without providing business benefit. Consider a microservices architecture where services emit detailed debug logs to a central event stream "just in case." These logs consume network bandwidth, increase serialization overhead, and inflate storage costs. When a genuine performance issue arises, finding signal in this noise becomes harder, not easier. In streaming systems, network congestion from unnecessary data manifests as: - Increased cross-availability-zone transfer costs - Higher serialization/deserialization CPU usage - Longer replication lag between Kafka brokers - Slower producer acknowledgments due to I/O saturation ## Data Obesity in Streaming Architectures Streaming platforms like Apache Kafka, Apache Flink, and Pulsar are particularly vulnerable to obesity because they're designed for high throughput. This strength becomes a weakness when data volume lacks corresponding value. ### Fat Payloads in Change Data Capture CDC tools capture every change to a database and stream it to downstream consumers. The default configuration often includes the entire row, before and after states, regardless of what consumers actually need. A typical example: an e-commerce company streams order updates from PostgreSQL to Kafka. Each order record contains customer data (name, email, shipping address), product details (descriptions, images, specifications), payment information, and audit fields. Most consumers need only the order ID and status, yet they receive 100KB per event. When order volume reaches 10,000 per minute, this translates to nearly 1GB/minute of unnecessary network traffic and storage consumption. ### Infinite Retention Without Purpose Treating Kafka as "forever storage" creates obesity at the infrastructure level. Some organizations configure topics with infinite retention or multi-year retention periods without clear use cases. The rationale is usually defensive: "We might need this data someday." But indefinite retention has real costs: - Storage expenses that scale linearly with data volume - Slower consumer group rebalancing as partition sizes grow - Increased recovery time when brokers fail - Higher operational complexity for upgrades and migrations Effective streaming architectures distinguish between hot data (active processing), warm data (recent historical queries), and cold data (long-term archival). Conflating these tiers in a single Kafka cluster creates unnecessary burden. For cold data archival, modern lakehouse architectures provide better economics and query capabilities than indefinite Kafka retention. Streaming data to Delta Lake or Apache Iceberg tables enables cost-effective long-term storage with ACID guarantees and SQL analytics. For details on lakehouse patterns, see [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture) and [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables). **Modern Solution: Tiered Storage (Kafka 3.6+)** Kafka's tiered storage feature (production-ready as of Kafka 3.6) addresses the retention obesity problem by automatically moving older log segments to object storage (S3, Azure Blob, GCS) while keeping recent data on local disks. This enables: - **Unlimited retention at reduced cost**: Store years of data at object storage prices instead of high-performance disk - **Faster broker operations**: Local disk contains only hot data, speeding up recovery and rebalancing - **Cost optimization**: Pay ~$0.02/GB/month for cold data vs. ~$0.10/GB/month for provisioned disk - **Simplified operations**: No manual archival pipelines or separate long-term storage systems For organizations on Kafka 4.0+ with KRaft mode, tiered storage becomes even more efficient due to improved metadata handling and elimination of ZooKeeper coordination overhead. This combination represents the modern approach to retention management without infrastructure obesity. For more on Kafka's evolution, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) and [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). ### Backpressure Cascades When producers overwhelm consumers with "obese" streams, backpressure propagates upstream. Consumers fall behind, lag metrics increase, and the system enters a degraded state where catch-up becomes impossible without intervention. For monitoring techniques to detect and respond to consumer lag, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). This often creates a vicious cycle: 1. Fat payloads slow consumer processing 2. Consumers fall behind, increasing memory pressure 3. Out-of-memory errors force restarts 4. Restarts cause rebalancing (redistribution of partitions across consumer instances), further slowing the consumer group 5. Meanwhile, producers continue adding data, widening the gap ## Real-World Impact: Case Studies ### Case Study 1: Financial Services CDC Implementation A financial services firm implemented CDC to stream transaction data from their core banking system to a fraud detection service. The initial implementation streamed complete customer records (including scanned documents stored as BLOBs) with each transaction event. Within three months, they faced: - Kafka cluster storage costs exceeding $50,000/month - Average event processing latency of 8 seconds (vs. 200ms target) - Regular consumer group failures from out-of-memory errors - Inability to add new consumers without cluster capacity expansion The solution involved schema redesign: transaction events referenced customer IDs rather than embedding full records. Document BLOBs were stored separately with URL references. This reduced average payload size from 2.1MB to 4KB, a 99.8% reduction. Results: latency dropped to 150ms, storage costs decreased by 85%, and the system could scale to 5x transaction volume without additional infrastructure. ### Case Study 2: E-Commerce Event Stream Obesity An e-commerce platform emitted product view events including the entire product catalog entry (description, images, reviews, inventory across all warehouses). Analytics consumers needed only product ID and category. The bloated events caused: - 40TB monthly network transfer costs between regions - Flink jobs consuming 3x necessary CPU for deserialization - 7-day retention limit (vs. desired 30 days) due to storage constraints By implementing projection at the source, emitting only required fields, they reduced payload size by 94%, enabling 90-day retention at lower total cost than the original 7-day configuration. ## The Diet: Mitigation Strategies ### Filter at the Source The most effective obesity prevention happens before data enters your streaming platform. Source-side filtering ensures only valuable data consumes downstream resources. **For Kafka Connect CDC:** Configure Single Message Transforms (SMTs) to: - Drop unnecessary fields from change events - Filter out rows that don't meet relevance criteria - Extract nested structures into separate topics For detailed SMT patterns, see [Kafka Connect: Single Message Transforms](https://www.conduktor.io/glossary/kafka-connect-single-message-transforms). Example connector configuration to reduce payload obesity: ```json { "name": "postgres-cdc-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "tasks.max": "1", "database.hostname": "postgres.example.com", "database.port": "5432", "database.user": "debezium", "database.dbname": "orders", "topic.prefix": "cdc", "transforms": "dropFields,filterLargeImages,route", "transforms.dropFields.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.dropFields.exclude": "internal_notes,audit_log,created_by_ip", "transforms.filterLargeImages.type": "org.apache.kafka.connect.transforms.Filter", "transforms.filterLargeImages.predicate": "hasLargeBlob", "predicates": "hasLargeBlob", "predicates.hasLargeBlob.type": "org.apache.kafka.connect.transforms.predicates.RecordIsTombstone" } } ``` This configuration drops audit fields, filters records with large BLOBs, and routes lean events to appropriate topics, reducing payload sizes by 70-90%. **For application producers:** - Implement schema-driven serialization (Avro, Protobuf) - Design events with consumer needs in mind, not just producer convenience - Use event types with specific purposes rather than generic "data dump" events - Establish data contracts that define payload expectations and SLAs For establishing clear expectations between producers and consumers, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). For overall topic design principles, refer to [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines). ### Aggressive Compaction For topics maintaining current state (e.g., customer profiles, product catalogs), log compaction automatically removes historical values for each key, retaining only the latest. For deep dive into compaction mechanics, see [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained). Compaction strategies and configuration example: ```properties # Topic configuration for state management cleanup.policy=compact # Trigger compaction when 50% of log contains old values (default: 50%) min.cleanable.dirty.ratio=0.5 # More aggressive: trigger at 30% for faster space recovery # min.cleanable.dirty.ratio=0.3 # Minimum time before a message can be compacted (default: 0) min.compaction.lag.ms=0 # Maximum time before forcing compaction (default: Long.MAX_VALUE) max.compaction.lag.ms=86400000 # 24 hours # Tombstone retention (how long delete markers remain) delete.retention.ms=86400000 # 24 hours # Segment settings that affect compaction efficiency segment.ms=3600000 # 1 hour - smaller segments compact faster segment.bytes=104857600 # 100MB ``` **Recommended configuration for obesity prevention:** - State topics: `cleanup.policy=compact` with aggressive `min.cleanable.dirty.ratio=0.3` - Event topics: `cleanup.policy=delete` with retention based on actual query patterns (typically 7-30 days) - Hybrid: `cleanup.policy=compact,delete` for state topics that also need time-based expiration For topics with time-series data, implement retention policies aligned with actual query patterns. If 95% of queries touch data from the last 7 days, 90-day retention may be excessive. ### Schema Evolution and Projection As consumer needs evolve, schemas should too. Removing deprecated fields prevents perpetuating obesity through inertia. For comprehensive coverage of schema management, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). Strategies: - Use schema registries to version and govern event structures - Implement backward-compatible changes that deprecate then remove unused fields - Create topic-specific projections for different consumer groups rather than one-size-fits-all events **Example:** Instead of a single "OrderEvent" with 50 fields, create: - `order.created.minimal` (10 fields for real-time dashboards) - `order.created.full` (50 fields for data warehouse) - `order.created.audit` (20 fields for compliance systems) For serialization format comparisons that impact payload size, refer to [Avro vs Protobuf vs JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema). ## Governance as Prevention Technology alone cannot prevent data obesity, organizational governance is essential. ### Visibility and Monitoring You can't manage what you don't measure. Effective governance requires visibility into: - Per-topic data volume and growth rates - Payload size distributions (average, p95, p99) - Consumer utilization (which fields are actually accessed) - Cost attribution (which teams/services drive storage and network costs) Modern tooling for data obesity monitoring: - **Conduktor Platform**: Comprehensive Kafka management with payload inspection, schema governance, and cost tracking dashboards - **Conduktor Gateway**: Kafka proxy for data governance that can enforce payload size limits, schema validation, and consumption quotas in real-time - **Kafka Lag Exporter**: Open-source tool for monitoring consumer lag patterns that indicate processing bottlenecks - **Cruise Control**: LinkedIn's open-source tool for automated cluster rebalancing and capacity planning For comprehensive governance frameworks, see [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) and [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming). ### Policy Enforcement Governance platforms enable organizations to define and enforce lean data policies: - **Schema validation rules:** Prevent registration of schemas exceeding size thresholds or containing prohibited field types - **Topic lifecycle management:** Automatically archive or delete topics that haven't been consumed in X days - **Quota management:** Limit producer throughput or storage per team/application to prevent runaway growth ### Cost Attribution and Accountability When storage and network costs are treated as shared overhead, individual teams lack incentive to optimize. Chargeback models that attribute costs to producing teams create accountability. Governance platforms can: - Track which services produce which topics - Calculate storage and throughput costs per topic - Generate reports showing each team's infrastructure footprint - Trigger alerts when teams exceed allocated budgets This economic feedback loop encourages teams to question whether they truly need to stream every field, keep every event, or maintain indefinite retention. For implementing chargeback mechanisms that make data obesity costs visible to producing teams, see [Conduktor's chargeback documentation](https://docs.conduktor.io/guide/conduktor-concepts/chargeback). **FinOps for Streaming (2025 Best Practices):** Modern organizations apply FinOps principles to streaming infrastructure: - **Cost visibility dashboards**: Real-time tracking of per-topic, per-team cloud costs (compute, storage, network egress) - **Budget alerts**: Automated notifications when topics exceed allocated spend - **Right-sizing recommendations**: ML-driven analysis suggesting optimal partition counts, retention periods, and compression settings - **Cross-AZ transfer optimization**: Identifying and minimizing expensive cross-availability-zone traffic Data obesity directly correlates with "dark data tax", the hidden costs of maintaining unused or low-value data. To identify topics consuming disproportionate resources, use [VIP Topics insights](https://docs.conduktor.io/guide/insights/vip-topics) which highlight high-cost, high-volume topics requiring optimization attention. For more on this economic challenge, see [Dark Data Tax](https://www.conduktor.io/glossary/dark-data-tax) and [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership). ## Summary Data obesity isn't inevitable. It results from prioritizing short-term convenience ("let's capture everything") over long-term sustainability. Healthy data infrastructure requires: - **Intentionality:** Every byte ingested should have a defined purpose and expected consumer. Default to lean payloads and expand only when justified by specific use cases. - **Right-sizing:** Match retention policies to actual query patterns. Distinguish between hot, warm, and cold data with appropriate storage tiers. - **Continuous optimization:** Schema evolution isn't a one-time activity. As business needs change, data structures should adapt — removing deprecated fields and avoiding perpetual accumulation. - **Economic accountability:** Make data costs visible and attributable. When teams understand the infrastructure impact of their decisions, they make better tradeoffs. - **Governance as culture:** Technology platforms can enforce policies, but sustainable change requires organizational commitment to treating data infrastructure as a finite resource, not an infinite dumping ground. The organizations that handle data-intensive workloads well aren't the ones that collect the most data — they're the ones that collect the right data and stay agile enough to evolve as needs change. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Streaming platform where data obesity often manifests through bloated payloads and excessive retention - [Dark Data Tax](https://www.conduktor.io/glossary/dark-data-tax) - The hidden costs of unused data that often accompanies data obesity - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Right-sizing infrastructure to prevent obesity while meeting performance requirements ## Sources and References 1. **Kleppmann, Martin.** "Designing Data-Intensive Applications." O'Reilly Media, 2017. Chapters on data storage patterns and system scalability. 2. **Apache Kafka Documentation.** "Tiered Storage" (Kafka 3.6+). Official documentation on configuring and managing tiered storage for cost-effective retention. 3. **Apache Kafka Documentation.** "KRaft Mode" (Kafka 4.0+). Guide to ZooKeeper-free Kafka operations and improved metadata management. 4. **Confluent Documentation.** "Kafka Configuration Reference - Log Compaction." Best practices for managing topic retention and compaction strategies. 5. **Narkhede, Neha, et al.** "Kafka: The Definitive Guide, 2nd Edition." O'Reilly Media, 2021. Updated sections on performance tuning and capacity planning. 6. **Beyer, Betsy, et al.** "Site Reliability Engineering: How Google Runs Production Systems." O'Reilly Media, 2016. Chapters on capacity planning and performance management. 7. **FinOps Foundation.** "Cloud FinOps for Streaming Data Platforms" (2025). Best practices for cost optimization and accountability in streaming infrastructure. 8. **DataCouncil Conference Proceedings.** Various talks on data platform economics and ROI measurement (2020-2025). 9. **Gartner Research.** "Managing Data Obesity: Storage Optimization Strategies for Modern Data Platforms." Reports on data management costs and organizational impact (2024-2025). 10. **Apache Flink Documentation.** "State Backends and Checkpointing." Guidelines for managing state size in streaming applications. 11. **Conduktor Documentation.** "Data Governance with Conduktor Gateway" (2025). Real-time policy enforcement and payload management patterns. Source: https://www.conduktor.io/glossary/data-obesity --- # Data Pipeline Orchestration with Streaming Modern data platforms increasingly rely on streaming architectures to deliver real-time insights and power event-driven applications. While traditional batch orchestration tools like Apache Airflow have matured over the past decade, streaming introduces fundamentally different orchestration challenges. ## Understanding Pipeline Orchestration Pipeline orchestration refers to the automated management of data workflows, including scheduling tasks, managing dependencies between operations, handling failures, and monitoring execution. In traditional batch processing, orchestration tools coordinate discrete jobs that run on a schedule, hourly ETL jobs, nightly data warehouse loads, or weekly report generation. These batch orchestrators excel at directed acyclic graphs (DAGs) where task B waits for task A to complete. They provide retry logic, alerting, and visibility into pipeline execution. Tools like Apache Airflow, Prefect, and Dagster have become industry standards for this paradigm. ## The Streaming Orchestration Challenge Streaming data processing fundamentally differs from batch in one critical way: it's continuous rather than scheduled. A Kafka consumer reading from a topic doesn't "finish" in the traditional sense, it runs indefinitely, processing events as they arrive. This creates several orchestration challenges: - **Deployment vs Runtime Management**: In batch systems, orchestration means scheduling when jobs run. In streaming, it often means deploying long-running applications and ensuring they stay healthy. A Flink job processing clickstream data might run for months without restarting. - **Stateful Processing**: Streaming applications maintain state across millions of events. Orchestrating these systems means managing checkpoints, state backends, and ensuring exactly-once processing semantics survive failures and redeployments. For detailed coverage of state management in Flink, see [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing). For exactly-once semantics implementation, refer to [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka). - **Dynamic Scaling**: Streaming workloads vary throughout the day. Orchestration must handle scaling consumer groups, rebalancing partitions, and adjusting resources without data loss. To understand how consumer groups coordinate and rebalance, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). Consider an e-commerce fraud detection pipeline: ![Consider an e-commerce fraud detection pipeline](https://www.conduktor.io/assets/images/glossary/data-pipeline-orchestration-with-streaming-0.webp) Traditional batch orchestration doesn't fit, this pipeline processes events continuously, scales dynamically based on traffic, and maintains complex state about user behavior patterns. ## Orchestration Approaches for Streaming Pipelines Organizations typically adopt one of three orchestration strategies for streaming systems, often using a combination: ### Infrastructure Orchestration Traditional orchestration tools manage the streaming infrastructure itself rather than the data flow. Airflow can deploy Kafka connectors, manage Flink job submissions, handle schema registry updates, and coordinate infrastructure changes. ![data-pipeline-orchestration-with-streaming diagram 2](https://www.conduktor.io/assets/images/glossary/data-pipeline-orchestration-with-streaming-1.webp) This approach treats streaming components as long-running infrastructure that needs periodic updates rather than repeatedly scheduled tasks. ### Stream-Native Orchestration Stream processing frameworks have built-in orchestration capabilities. Kafka Connect manages connector lifecycle, load balancing, and failure recovery. ksqlDB orchestrates stream processing through SQL statements. Flink's job manager handles checkpointing, savepoints, and recovery. With Flink 1.18+ and its unified batch-streaming architecture, orchestration becomes more flexible. The same Flink job can process both bounded (batch) and unbounded (streaming) data, allowing orchestration systems to treat backfilling historical data and processing real-time streams as variations of the same pipeline rather than separate workflows. These tools orchestrate at the stream processing level, managing how data flows through topics, how consumers coordinate, and how processing state is maintained. The "orchestration" is implicit in the streaming platform's design. ### Kubernetes-Based Orchestration Container orchestration platforms like Kubernetes increasingly manage streaming applications. Modern operators like Strimzi 0.40+ manage Kafka 4.0 clusters with KRaft mode (eliminating ZooKeeper dependencies), the Flink Kubernetes Operator 1.7+ handles Flink 1.18+ jobs with improved autoscaling, and standard Kubernetes primitives (deployments, services, config maps) orchestrate the entire streaming infrastructure. This approach provides consistent orchestration across streaming and non-streaming components, unified monitoring, and declarative infrastructure management. With Kafka 4.0's KRaft mode, orchestration becomes simpler as there's no need to coordinate ZooKeeper clusters alongside Kafka, reducing operational complexity and improving deployment reliability. For detailed guidance on Kubernetes deployments, see [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) and [Infrastructure as Code for Kafka Deployments](https://www.conduktor.io/glossary/infrastructure-as-code-for-kafka-deployments). ## Streaming and Data Pipeline Orchestration in Practice Apache Kafka ecosystems demonstrate the interplay between different orchestration layers. A complete streaming pipeline involves: - **Topic Management**: Creating topics with appropriate partitioning, replication, and retention policies. Changes must be coordinated across environments. - **Schema Evolution**: As data structures evolve, schemas in the schema registry must be updated compatibly. Producers and consumers must handle multiple schema versions during transitions. For comprehensive guidance on managing schemas, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). - **Connector Lifecycle**: Kafka Connect source and sink connectors need deployment, configuration updates, and monitoring. A connector failure can create data gaps that require orchestrated recovery procedures. For detailed coverage of building and managing connectors, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines). - **Consumer Group Coordination**: Multiple applications consuming from the same topics need coordination. If a new consumer version deploys, orchestration ensures graceful handoff without duplicate processing or data loss. - **State Management**: Applications using Kafka Streams or Flink maintain local state stores. Orchestration must handle state migration during scaling events or version upgrades. Platforms like Conduktor help orchestrate these aspects by providing centralized management of topics, schemas, and connectors with [Kafka Connect](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect), and monitoring the health of consumer groups and connectors in real time. ## Best Practices for Streaming Orchestration ### Embrace Idempotency and Exactly-Once Semantics Unlike batch jobs that can simply rerun, streaming pipelines must handle duplicate events and out-of-order processing. Design orchestration around Kafka's transactional producers, Flink's exactly-once processing, and idempotent consumers. ### Schema-First Development Enforce schema validation at ingestion. Use the schema registry as a source of truth. Orchestrate schema changes through a review process that includes compatibility checks and consumer impact analysis. Breaking schema changes can cascade through an entire streaming pipeline. ### Monitor Lag, Not Just Errors Traditional batch orchestration focuses on task success or failure. Streaming orchestration must monitor consumer lag, the gap between produced and consumed messages. Growing lag indicates problems even when no errors appear in logs. Modern tools like Kafka Lag Exporter (Prometheus-based) provide real-time lag metrics that integrate with orchestration workflows. Platforms like Conduktor offer comprehensive monitoring dashboards that track consumer lag, throughput, and pipeline health across your entire streaming infrastructure. Monitor connector health and status with [Kafka Connect Management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) and validate data quality using [built-in monitoring tools](https://docs.conduktor.io/guide/use-cases/observe-data-quality). Orchestration systems should trigger alerts when lag exceeds thresholds and potentially scale resources automatically. For comprehensive monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ### Design for Reprocessing Build streaming pipelines that can rewind to earlier offsets and reprocess data. This enables fixing bugs in processing logic, recovering from data quality issues, and handling schema evolution problems. Orchestration should include tools for controlled reprocessing. ### Dead Letter Queues for Poison Pills Individual malformed events shouldn't crash entire streaming pipelines. Orchestrate error handling through dead letter topics that capture problematic events for later analysis and reprocessing. For detailed implementation guidance, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). ### Testing in Streaming Environments Traditional orchestration includes testing through integration tests that run complete DAGs. Streaming requires testing continuous processing, state evolution, and failure scenarios. Tools that can replay production-like event streams into test environments help validate orchestration logic. Chaos engineering for streaming pipelines has become essential in 2025. Conduktor Gateway, a Kafka proxy, enables testing failure scenarios by injecting faults, simulating network issues, and validating exactly-once guarantees under adverse conditions. Orchestration systems should include automated testing stages that verify pipeline resilience before production deployment. ## Summary Pipeline orchestration for streaming systems requires rethinking traditional batch patterns. While batch orchestration focuses on scheduling discrete jobs and managing task dependencies, streaming orchestration manages continuously running infrastructure, coordinates state across distributed systems, and ensures data flows reliably through complex topologies. Successful streaming orchestration combines multiple approaches: traditional tools like Airflow for infrastructure deployment and updates, stream-native capabilities within Kafka and Flink for runtime coordination, and Kubernetes for container orchestration. Streaming pipelines are living systems that require ongoing management, not periodic execution. ## Related Concepts - [Streaming Data Pipeline](https://www.conduktor.io/streaming-data-pipeline) - Understanding the components that orchestration manages - [Streaming ETL vs Traditional ETL](https://www.conduktor.io/streaming-etl-vs-traditional-etl) - How orchestration differs between batch and streaming - [Data Contracts for Reliable Pipelines](https://www.conduktor.io/data-contracts-for-reliable-pipelines) - Enforcing contracts in orchestrated streaming pipelines ## Sources and References 1. Kleppmann, M. (2017). *Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems*. O'Reilly Media. Chapter 11: Stream Processing. 2. Apache Kafka Documentation. (2025). "Kafka Connect" and "Kafka Streams Architecture." https://kafka.apache.org/documentation/ 3. Apache Flink Documentation. (2025). "Application Development" and "Operations & Deployment." https://flink.apache.org/ 4. Narkhede, N., Shapira, G., & Palino, T. (2017). *Kafka: The Definitive Guide*. O'Reilly Media. 5. Conduktor Documentation. (2025). "Kafka Orchestration and Management." https://docs.conduktor.io/ Source: https://www.conduktor.io/glossary/data-pipeline-orchestration-with-streaming --- # Data Product Governance: Building Trustworthy Data Assets As organizations shift from centralized data platforms to distributed data architectures, the concept of data products has become a fundamental building block. Treating data as a product isn't just about technology — it requires a governance framework that ensures data products are discoverable, trustworthy, and valuable to their consumers. This article covers how to establish governance practices that turn raw data streams into reliable, well-managed data products. For an overview of data products in practice, see [Building and Managing Data Products](https://www.conduktor.io/glossary/building-and-managing-data-products). For organizational structures and responsibilities, refer to [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities). ![Data Product Governance Framework](https://www.conduktor.io/assets/images/glossary/data-product-governance-0.webp) ## Understanding Data Products A **data product** is more than just a dataset or a collection of raw data. It's a self-contained, intentionally designed data asset that serves a specific analytical or operational purpose. While raw data might be the unprocessed events flowing through your Kafka topics, and a dataset might be a static snapshot extracted for analysis, a data product is: - **Purposefully designed** with specific consumer needs in mind - **Self-describing** with comprehensive metadata and documentation - **Quality-assured** with explicit guarantees about freshness, completeness, and accuracy - **Actively maintained** with clear ownership and SLAs - **Discoverable** through catalogs and standardized interfaces For example, a "Customer 360 Profile" data product aggregates customer interactions, purchases, and preferences into a unified view with guaranteed freshness within 5 minutes, far more valuable than raw clickstream events or isolated transactional datasets. ## Core Data Product Principles Effective data product governance rests on several foundational principles: - **Ownership**: Every data product must have a designated owner, typically a domain team that understands the business context and can make decisions about the product's evolution. This isn't just a name in a spreadsheet; owners are accountable for the product's quality, availability, and support. - **Documentation**: Data products require comprehensive documentation covering schema definitions, business logic, known limitations, usage examples, and support contacts. Documentation should be living and version-controlled alongside the product itself. - **Service Level Agreements (SLAs)**: Data products must declare explicit guarantees about latency, freshness, availability, and quality. A real-time fraud detection stream might guarantee 99.9% availability with sub-second latency, while a daily marketing report might guarantee delivery by 6 AM with 99.5% completeness. - **Discoverability**: Consumers should be able to find relevant data products through searchable catalogs with business-friendly terminology, not just technical identifiers. ## Governance Dimensions Comprehensive data product governance spans multiple dimensions: ### Quality Governance Quality governance establishes standards and monitors adherence to data quality requirements. This includes: - **Freshness SLAs**: Maximum acceptable data age (e.g., "customer events processed within 2 minutes") - **Completeness guarantees**: Expected volume ranges and missing data thresholds - **Accuracy contracts**: Validation rules, acceptable error rates, and reconciliation processes - **Schema compliance**: Enforcement of data contracts and compatibility rules For deep dives into quality dimensions, see [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). For contract implementation, refer to [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). ### Access Governance Access governance balances self-service enablement with necessary controls: - **Authentication and authorization**: Who can discover, view, and consume each data product - **Privacy and compliance**: PII handling, data residency, retention policies - **Self-service with guardrails**: Enabling consumption without requiring central approvals while preventing misuse - **Audit trails**: Tracking who accesses what data for compliance and security ### Lineage Governance Understanding data flow and transformations is critical for trust and debugging: - **Source tracking**: Where does the data originate? - **Transformation documentation**: What business logic has been applied? - **Dependency mapping**: What downstream products or systems rely on this data? - **Impact analysis**: What breaks if this product changes or becomes unavailable? For detailed lineage tracking approaches, see [Data Lineage Tracking: Data from Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption). ### Lifecycle Governance Data products evolve over time, requiring structured lifecycle management: - **Versioning strategies**: Semantic versioning for schemas and contracts - **Deprecation policies**: How long old versions are supported, migration paths - **Change notification**: Alerting consumers to upcoming changes - **Sunset procedures**: Safe decommissioning of retired products ## Ownership Models Successful data product governance distributes responsibility appropriately: - **Domain Teams** own the data products generated within their business domain. The marketing team owns customer engagement metrics; the logistics team owns shipment tracking data. This domain ownership ensures those closest to the business context make decisions about data quality and evolution. - **Data Product Owners** within each domain team serve as the primary point of contact. They define SLAs, approve schema changes, prioritize enhancements, and coordinate with consumers. - **Data Stewards** provide governance oversight, ensuring consistency across products, maintaining the data catalog, and enforcing organizational policies without dictating domain-specific decisions. - **Platform Teams** provide the infrastructure and tooling that makes governance scalable, data catalogs, quality monitoring, access controls, and lineage tracking. ## Quality Contracts and SLAs The most critical governance element is the explicit quality contract each data product commits to. For streaming data products, this typically includes: - **Freshness SLAs**: "Order events appear in the stream within 30 seconds of transaction completion" creates a measurable commitment that can be monitored and alerted on. - **Completeness Guarantees**: "During business hours, order volume should be 800-1200 events/minute; alerts trigger if outside this range" helps consumers trust the data or know when something is wrong. - **Accuracy Standards**: "Product prices reconcile with the source system within $0.01" provides confidence in downstream analytics. - **Availability Commitments**: "99.9% uptime during 6 AM - 11 PM EST" sets expectations for when the product can be reliably consumed. These contracts should be codified, monitored automatically, and violations should trigger alerts to both owners and consumers. Modern governance platforms like Conduktor provide comprehensive policy enforcement, including data quality validation, schema governance, access controls, and SLA monitoring across Kafka topics (see [Conduktor Governance Insights](https://docs.conduktor.io/guide/insights/governance) for real-time visibility into policy compliance). Conduktor Gateway (2025+), a Kafka proxy, enables advanced governance patterns like chaos testing, data masking, and topic-level policy enforcement without modifying application code. ## Discoverability and Self-Service Access A data product that can't be found can't be used. Effective discoverability requires: - **Business Glossaries**: Mapping technical data product names to business terms ("order_events_v2" is "Real-time Order Stream") - **Rich Metadata**: Schema documentation, sample queries, known use cases, performance characteristics, and contact information, all searchable and centralized - **Data Catalogs**: Centralized registries where teams can browse available products, understand their purpose, and request access - **Usage Analytics**: Showing which products are popular, who's using them, and for what purposes helps teams discover battle-tested products Self-service access means consumers can start using a data product without lengthy approval processes, provided they have appropriate authorization and understand the quality contracts. Access requests should be automated, audit trails maintained, and onboarding documentation readily available. ## Implementing Data Contracts with Modern Tools - **Data contracts** formalize the agreement between data product producers and consumers. In 2025, several frameworks enable codified, testable contracts: - **Soda Core (2025)** enables declarative data quality checks: ```yaml # Product: customer_orders_stream # Contract: data-contract.yml checks for orders: - freshness(event_time) < 5m - row_count between 100 and 5000 - missing_count(customer_id) = 0 - invalid_percent(order_total) < 0.1% - schema: fail: when required column missing: [order_id, customer_id, order_total] when wrong column type ``` **Great Expectations 1.x** provides programmatic expectations: ```python # Data product contract validator import great_expectations as gx context = gx.get_context() # Define expectations for customer_orders product expectation_suite = context.add_expectation_suite("customer_orders_v2") validator = context.get_validator( batch_request=customer_orders_batch, expectation_suite_name="customer_orders_v2" ) # Freshness SLA validator.expect_column_max_to_be_between( column="event_timestamp", min_value=datetime.now() - timedelta(minutes=5), max_value=datetime.now() ) # Completeness guarantee validator.expect_column_values_to_not_be_null(column="customer_id") # Schema enforcement validator.expect_table_columns_to_match_ordered_list( column_list=["order_id", "customer_id", "order_total", "event_timestamp"] ) # Run validation results = validator.validate() ``` For comprehensive data quality frameworks, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework). **Policy-as-Code with OpenPolicyAgent (OPA)** enables governance automation: ```rego # Data product access policy package data_product.access default allow = false # Allow access if user has required role and data classification matches allow { input.user.roles[_] == "data_analyst" input.data_product.classification == "internal" input.purpose in ["analytics", "reporting"] } # Sensitive data requires additional approval allow { input.user.roles[_] == "data_scientist" input.data_product.classification == "sensitive" input.approval.status == "approved" time.now_ns() < input.approval.expires_at } ``` Modern data catalogs like **OpenMetadata** and **DataHub** (2025) provide centralized governance: - Automated metadata extraction from streaming sources - AI-powered business glossary mapping - Lineage tracking with column-level granularity - Embedded data quality monitoring - Self-service access request workflows For metadata strategies, see [Metadata Management: Technical vs Business Metadata](https://www.conduktor.io/glossary/metadata-management-technical-vs-business-metadata) and [What is a Data Catalog: Modern Data Discovery](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery). ## Lifecycle Management in Streaming Environments Streaming data products present unique lifecycle challenges: **Schema Evolution**: As business requirements change, schemas must evolve. Governance requires compatibility rules and version management: - **Forward compatibility**: New schema can read old data (safe for consumers) - **Backward compatibility**: Old schema can read new data (safe for producers) - **Full compatibility**: Both forward and backward compatible (safest for all parties) Breaking changes require coordinated migrations with sufficient notice to consumers. For detailed schema evolution strategies, see [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). - **Continuous Deployment**: Unlike batch datasets with discrete versions, streaming products continuously produce data. Changes must be deployed without disruption, often requiring dual-write periods or feature flags. - **Deprecation Windows**: When retiring a data product or version, provide sufficient migration time. A typical policy might be: 30 days notice for minor changes, 90 days for major changes, 6 months for deprecation. - **Migration Support**: Provide migration guides, backward compatibility periods, and direct support to high-value consumers during transitions. - **Version Transparency**: Consumers should always know which version they're consuming and when it will be deprecated. This metadata should be discoverable in the data catalog and potentially embedded in the data stream itself. ## Measuring Data Product Health Effective governance requires measuring both product health and governance adherence: **Product Health Metrics**: - SLA compliance rates (freshness, completeness, accuracy) - Availability percentages - Consumer count and growth trends - Query/consumption volumes - Support ticket volumes and resolution times **Governance Adherence Metrics**: - Percentage of products with complete documentation - Percentage with active owners and up-to-date SLAs - Time-to-discovery for new products - Access request fulfillment time - Schema change policy violations **Adoption Indicators**: - Number of unique consumers per product - Diversity of use cases - Reduction in duplicate data efforts - Self-service access rate vs. manual requests Teams should regularly review these metrics in data product reviews, celebrating successes and addressing problem areas. Products consistently failing SLAs or lacking consumers may need investment or retirement. ## Connection to Data Mesh Principles Data product governance operationalizes several data mesh principles: - **Domain Ownership** aligns with domain-oriented decentralization, where teams own their data products rather than centralizing all data into a monolith. - **Data as a Product** thinking requires governance frameworks that ensure products meet consumer needs with quality guarantees. - **Self-Service Data Infrastructure** depends on governance providing the guardrails that make self-service safe and compliant. - **Federated Computational Governance** balances centralized standards with domain autonomy, global policies enforced locally. Even if you're not implementing a full data mesh architecture, these governance practices create more trustworthy, discoverable, and valuable data products in any distributed data environment. For comprehensive coverage of data mesh implementation, see [Data Mesh Principles and Implementation](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation). ## Conclusion Data product governance turns raw data streams into reliable, trustworthy assets. By establishing clear ownership, documenting quality contracts, enabling discoverability, and managing lifecycles thoughtfully, organizations can build data products that consumers trust and adopt widely. Effective governance isn't about control — it's about enablement. It gives domain teams the structure to move quickly while keeping data products reliable, compliant, and valuable. ## Related Concepts - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) - Organizational structures and roles that support effective data product governance - [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) - Formalized agreements that define quality guarantees and SLAs for data products - [Metadata Management: Technical vs Business Metadata](https://www.conduktor.io/glossary/metadata-management-technical-vs-business-metadata) - Understanding the different types of metadata critical for data product discoverability and governance ## Sources and References - [Data Mesh Principles and Logical Architecture](https://martinfowler.com/articles/data-mesh-principles.html) - [The Data Product: Building Blocks and Best Practices](https://www.thoughtworks.com/insights/blog/data-strategy/what-are-data-products) - [Data Governance in Practice: A Framework for Modern Organizations](https://www.oreilly.com/library/view/data-governance-the/9781492063483/) - [OpenMetadata: Open Source Data Catalog](https://open-metadata.org/) - [Conduktor Documentation](https://docs.conduktor.io/) Source: https://www.conduktor.io/glossary/data-product-governance --- # Data Quality Incidents: Detection, Response, and Prevention ## What is a Data Quality Incident? A **data quality incident** occurs when data flowing through streaming systems fails to meet established quality standards, resulting in impact to downstream consumers, analytics, or business operations. Unlike normal statistical variance or expected fluctuations in data patterns, incidents represent significant deviations that require human intervention and remediation. In streaming architectures, data quality incidents are particularly critical because they propagate in real-time to multiple downstream systems. A schema violation in a Kafka topic might cascade to break consumers, corrupt data lakes, and trigger false alerts across monitoring systems. The velocity and volume of streaming data mean that incidents can affect millions of records within minutes if not detected and contained quickly. The key distinction between an incident and normal variance lies in **impact and deviation from acceptable bounds**. A 5% increase in null values might be within normal operating parameters, while a sudden 50% spike in nulls represents an incident requiring immediate attention. Organizations define these thresholds through **Service Level Objectives (SLOs)**, measurable targets for system reliability and data quality, and data quality contracts that specify acceptable ranges for metrics like completeness, accuracy, and timeliness. For detailed coverage of these quality dimensions, see [Data Quality Dimensions](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency). ![Data Quality Incident Response](https://www.conduktor.io/assets/images/glossary/data-quality-incidents-0.webp) ## Types of Data Quality Incidents ### Schema Violations and Format Errors Schema violations occur when incoming data doesn't match the expected structure. In streaming systems using Apache Kafka with Schema Registry, this might manifest as: - Producers sending data with missing required fields - Type mismatches (sending strings where integers are expected) - Addition of unexpected fields that break strict schema enforcement - Incompatible schema evolution (non-backward compatible changes) Modern Schema Registry (2025) offers advanced compatibility modes like `FULL_TRANSITIVE` that check compatibility across all schema versions, preventing incidents from incompatible evolution. These incidents often cause immediate consumer failures, as applications cannot deserialize or process malformed records. For comprehensive schema management practices, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). ### Null Explosions and Missing Data A **null explosion** happens when a normally populated field suddenly contains null or missing values at abnormally high rates. Common causes include: - Upstream service failures that result in partial data - Configuration errors in data producers - Database replication lag causing incomplete record retrieval - API timeouts leading to default null values Missing critical business data (customer IDs, timestamps, transaction amounts) can invalidate entire analytical pipelines and business reports. ### Duplicate Records Duplicates in streaming systems arise from: - Producer retries due to transient network failures - At-least-once delivery semantics without proper deduplication - Replay scenarios where data is reprocessed - Multiple producers writing the same logical events While some duplication is acceptable in idempotent systems, excessive duplicates skew aggregations, inflate metrics, and waste processing resources. ### Data Delays and Lag Spikes Timeliness incidents occur when: - Event-time timestamps show growing lag from processing-time - Producer throughput drops below expected rates - Network partitions delay message delivery - Consumer lag grows beyond acceptable thresholds (e.g., > 1 hour for near-real-time systems) Delays can render time-sensitive applications ineffective, such as fraud detection or real-time recommendations. **Modern monitoring (2025)**: Tools like [Kafka Lag Exporter](https://github.com/seglo/kafka-lag-exporter) provide Prometheus metrics for consumer lag with configurable alerting thresholds. Kafka 4.0's KRaft mode eliminates ZooKeeper-related latency issues, reducing incident surface area. For detailed consumer lag strategies, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). ## Detection Methods ### Automated Quality Checks and Validation Rules Proactive monitoring involves continuous validation of streaming data against defined rules: - **Completeness checks**: Monitor null rates, record counts, and required field presence - **Accuracy checks**: Validate data ranges, enum values, and business rule compliance - **Consistency checks**: Verify referential integrity and cross-field relationships - **Timeliness checks**: Measure event-time vs. processing-time lag - **Example validation with Soda Core (2025)**: ```yaml # checks.yml checks for orders_stream: - missing_count(order_id) = 0 - invalid_percent(status) < 1%: valid values: ['pending', 'completed', 'cancelled'] - freshness(event_timestamp) < 5m - duplicate_count(order_id) < 0.1% ``` **Governance platforms** like Conduktor enable policy enforcement at the Kafka protocol level, validating data quality before it enters topics and preventing bad data from polluting streams. Conduktor Gateway, a Kafka proxy, can intercept messages, apply validation rules, and route violations to dead letter queues automatically. For practical implementation, see [Enforcing Data Quality with Conduktor](https://docs.conduktor.io/guide/use-cases/enforce-data-quality) and [Observing Data Quality](https://docs.conduktor.io/guide/use-cases/observe-data-quality). For comprehensive testing strategies, see [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing) and [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework). ### Anomaly Detection and Statistical Monitoring Beyond rule-based validation, statistical anomaly detection identifies unusual patterns: - Standard deviation analysis for numeric fields - Time-series forecasting to detect unexpected volume changes - Distribution drift detection comparing current data to historical baselines - Outlier detection for individual record values **2025 ML-based detection**: Modern observability platforms integrate machine learning models trained on historical data patterns to automatically flag deviations. OpenTelemetry provides standardized instrumentation for streaming applications, enabling correlation of data quality metrics with infrastructure signals. These systems can detect subtle incidents like gradual data drift that rule-based checks might miss. For related drift patterns, see [Data Drift in Streaming](https://www.conduktor.io/glossary/data-drift-in-streaming) and [Model Drift in Streaming](https://www.conduktor.io/glossary/model-drift-in-streaming). ### Preventing Incidents with Data Quality Policies Conduktor Data Quality Policies provide systematic prevention of data quality incidents by creating Rules that define expected message formats and content. These attach to specific topics, enabling centralized quality enforcement. In observe-only mode, Policies record violations without impacting flow; when integrated with Conduktor Gateway, they validate records before production, blocking non-compliant messages or marking them with violation metadata. This centralized approach catches schema violations, null explosions, and format errors at the infrastructure level rather than requiring each producer to implement validation independently, ensuring consistent quality standards. For implementation guidance, see [Data Quality Policies](https://docs.conduktor.io/guide/conduktor-concepts/data-quality-policies). ### User Reports and Feedback Loops Despite automated monitoring, downstream consumers often detect quality issues first: - Dashboard users noticing missing or incorrect data - Business stakeholders questioning unexpected metric changes - Consumers reporting processing errors or unexpected behavior Establishing clear channels for users to report suspected data quality issues creates a critical feedback loop for incident detection. ## Severity Classification and Incident Response ### Severity Levels Organizations typically classify incidents across four severity levels: - **Critical (P0)**: Complete data loss, major schema breaks, or incidents affecting critical business operations. Requires immediate response and escalation. - **High (P1)**: Significant data quality degradation affecting multiple systems or important analytical workloads. Response within 1-4 hours. - **Medium (P2)**: Moderate quality issues with workarounds available or limited impact. Response within business day. - **Low (P3)**: Minor issues with minimal impact. Tracked for resolution in normal workflow. ### Incident Response Process A structured response process ensures consistent handling: - **1. Detect**: Automated monitoring or user report identifies potential incident - **2. Assess**: On-call engineer evaluates severity, scope, and impact - **3. Contain**: Implement immediate mitigation to prevent further damage (pause producers, reroute consumers to dead letter queues, isolate affected data) - **4. Resolve**: Identify root cause and implement fix (repair data, deploy corrected code, adjust configurations) - **5. Review**: Conduct post-mortem to prevent recurrence For error isolation strategies, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). ### Root Cause Analysis for Streaming Incidents Effective RCA in streaming systems requires examining multiple layers: - **Producer layer**: Code changes, configuration updates, dependency failures - **Infrastructure layer**: Network issues, broker failures, resource exhaustion - **Schema layer**: Evolution mistakes, registry failures - **Data source layer**: Upstream system changes, database issues **Modern tooling (2025)**: OpenTelemetry distributed tracing connects data quality incidents across the entire pipeline, from producer instrumentation through Kafka brokers to consumer processing. Audit logging platforms capture all configuration changes and schema modifications with timestamps, enabling precise correlation with incident onset. Kafka 4.0's improved observability APIs provide richer metadata for incident investigation. For implementation guidance, see [Distributed Tracing for Kafka Applications](https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications) and [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). ## Communication and Post-Mortems ### Stakeholder Notification Strategies Timely communication prevents confusion and enables affected teams to take protective action: - **Immediate notification**: Alert directly impacted consumers and data owners - **Status page updates**: Provide public incident status for broader organization - **Regular updates**: Share progress every 30-60 minutes during active incidents - **Resolution notification**: Confirm when normal operations resume and any required actions Use targeted communication channels (Slack, PagerDuty, email) based on severity and audience. ### Blameless Post-Mortem Practices Post-incident reviews focus on **system improvement rather than individual blame**: 1. **Document timeline**: Reconstruct event sequence with precise timestamps 2. **Identify root cause**: Use "five whys" to uncover underlying systemic issues 3. **Analyze contributing factors**: Environmental, organizational, or technical factors that enabled the incident 4. **Define action items**: Specific, assignable improvements with owners and deadlines 5. **Share learnings**: Distribute post-mortem widely to improve organizational knowledge Blameless culture encourages transparency and prevents future incidents by addressing systemic weaknesses rather than individual errors. ## Prevention and Metrics ### Data Contracts and Validation **Data contracts** define explicit agreements between producers and consumers about data structure, quality, and SLAs. Contracts specify: - Required and optional fields - Data types and formats - Acceptable value ranges - Quality thresholds (max null rate, duplicate rate) - Timeliness guarantees **Modern contract enforcement (2025)**: Governance platforms like Conduktor enable enforcement of data policies and contracts at the infrastructure level, validating data before it reaches consumers and providing early detection of violations. Conduktor Gateway acts as an intelligent proxy layer, inspecting messages in real-time against contract definitions and preventing violating messages from entering topics. Schema-based approaches using Protobuf with [buf](https://buf.build/) or Avro with Schema Registry provide compile-time and runtime validation, catching contract violations before deployment. For contract implementation patterns, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines). ### Testing Strategies Preventing incidents requires comprehensive testing: - **Schema compatibility tests**: Verify evolution doesn't break consumers - **Data quality unit tests**: Validate transformation logic with edge cases using dbt tests or Great Expectations 1.0+ - **Integration tests**: Test producer-consumer interactions with realistic data - **Chaos engineering**: Intentionally inject failures to verify detection and recovery **2025 testing practices**: Modern CI/CD pipelines integrate data quality gates that prevent deployment of code or schema changes that violate contracts. Conduktor Gateway enables controlled chaos testing by injecting schema violations, delays, or duplicates in non-production environments to validate incident response procedures. For testing methodologies, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications), [Great Expectations Data Testing Framework](https://www.conduktor.io/glossary/great-expectations-data-testing-framework), and [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems). ### Key Metrics Three critical metrics measure incident management effectiveness: - **MTTD (Mean Time to Detect)**: Average time from incident occurrence to detection. Target: < 5 minutes for critical systems. - **MTTR (Mean Time to Resolve)**: Average time from detection to full resolution. Target varies by severity but < 1 hour for P0 incidents. - **Incident Frequency**: Number of quality incidents per week/month, categorized by type and severity. Track trends to measure prevention effectiveness. ## Building Incident Response Playbooks ### Creating Runbooks for Common Scenarios Playbooks provide step-by-step procedures for frequent incident types: **Schema violation playbook**: 1. Identify affected topic and producer 2. Check Schema Registry for recent changes 3. Pause affected producer 4. Validate consumer compatibility 5. Roll back schema or update producer code 6. Resume processing and verify recovery **Null explosion playbook**: 1. Query recent data to quantify impact 2. Identify affected fields and producers 3. Check upstream data sources 4. Implement filtering or default values in consumers 5. Fix root cause in producer 6. Backfill missing data if required **Duplicate detection playbook**: 1. Measure duplicate rate using unique keys 2. Identify duplicate source (replay, retry, multiple producers) 3. Implement deduplication in affected consumers 4. Correct producer configuration 5. Clean duplicate records if necessary ### Tools and Automation **Effective incident response (2025) relies on modern tooling**: - **Automated alerting**: PagerDuty, Opsgenie for on-call escalation with intelligent routing - **Monitoring and observability**: - Grafana, Datadog for real-time visibility into Kafka metrics - Kafka Lag Exporter for Prometheus-based consumer lag monitoring - OpenTelemetry for distributed tracing across streaming pipelines - **Data quality platforms**: - Soda Core for declarative data quality checks - Great Expectations 1.0+ for Python-based validation - dbt tests for transformation quality validation - **Governance and policy enforcement**: - Conduktor for Kafka cluster management, governance, and monitoring - Conduktor Gateway for protocol-level policy enforcement and chaos testing - **Runbook automation**: Scripts and Kubernetes operators for common mitigation actions - **Collaboration tools**: Slack, Microsoft Teams for incident coordination **Kafka 4.0 benefits**: KRaft mode simplifies incident response by eliminating ZooKeeper dependencies, reducing operational complexity and failure modes. Improved metrics APIs provide richer incident context. Automation reduces MTTD and MTTR by enabling instant response to common scenarios. For operational best practices, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) and [DataOps for Streaming](https://www.conduktor.io/glossary/dataops-for-streaming). ## Conclusion Data quality incidents in streaming systems require proactive detection, rapid response, and systematic prevention. Comprehensive monitoring, clear severity classification, structured response processes, and blameless post-mortems all reduce the impact of quality issues. Automated validation combined with statistical anomaly detection and well-defined playbooks lets teams detect incidents quickly and resolve them efficiently. Measuring MTTD, MTTR, and incident frequency shows where to focus prevention efforts. Current best practices emphasize prevention through data contracts, governance platforms like Conduktor, and testing with modern frameworks (Soda Core, Great Expectations 1.0+, dbt tests). Kafka 4.0's KRaft mode and improved observability reduce incident complexity, while OpenTelemetry provides end-to-end visibility across distributed streaming architectures. For understanding the relationship between data quality and observability, see [Data Quality vs Data Observability: Key Differences](https://www.conduktor.io/glossary/data-quality-vs-data-observability-key-differences). For incident management patterns, refer to [Data Incident Management and Root Cause Analysis](https://www.conduktor.io/glossary/data-incident-management-and-root-cause-analysis). ## Related Concepts - [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) - Systematic prevention strategies - [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing) - Early detection through testing - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Detecting freshness-related incidents ## Sources and References - [Site Reliability Engineering: How Google Runs Production Systems](https://sre.google/books/) - [Incident Response and Management for Data Engineering Teams](https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/) - [Great Expectations Documentation: Data Quality Testing](https://docs.greatexpectations.io/) - [Apache Kafka Operations Guide](https://kafka.apache.org/documentation/#operations) - [Blameless Post-Mortems and Just Culture](https://www.atlassian.com/incident-management/postmortem/blameless) Source: https://www.conduktor.io/glossary/data-quality-incidents --- # Data Quality vs Data Observability: Key Differences As data pipelines grow more complex and real-time requirements become the norm, two distinct yet complementary approaches have emerged: data quality and data observability. These terms are often used interchangeably, but they represent different philosophies and practices. ![Data Quality vs Data Observability](https://www.conduktor.io/assets/images/glossary/data-quality-vs-data-observability-key-differences-0.webp) ## What is Data Quality? Data quality is a reactive, testing-based approach that validates whether data meets predefined expectations and business rules. Think of it as a checkpoint system where data must pass specific tests before being considered valid. At its core, data quality answers the question: "Does this data meet our standards?" It involves defining explicit rules and constraints, then verifying that incoming data conforms to these requirements. Common data quality checks include: - **Completeness**: Are all required fields populated? - **Accuracy**: Do values fall within expected ranges? - **Consistency**: Do related data points align across systems? - **Uniqueness**: Are there unwanted duplicates? - **Timeliness**: Is data arriving within acceptable time windows? For example, a data quality test might verify that all customer email addresses follow a valid format, that order amounts are positive numbers, or that timestamps fall within a reasonable range. These tests are typically implemented as assertions in your data pipeline, often using frameworks like [Great Expectations](https://www.conduktor.io/glossary/great-expectations-data-testing-framework) (GX 1.0+), [dbt tests](https://www.conduktor.io/glossary/dbt-tests-and-data-quality-checks), or Soda Core. The strength of data quality lies in its precision. You define exactly what "good" looks like, and your system alerts you when data deviates from these expectations. However, this precision is also a limitation: you can only catch issues you've explicitly tested for. Unknown problems or emerging patterns will slip through undetected. ## What is Data Observability? Data observability takes a proactive, monitoring-based approach inspired by software observability practices. Rather than testing for specific conditions, observability continuously monitors the behavior and health of your data systems, looking for anomalies and unexpected patterns. Data observability answers a broader question: "What is happening in our data systems, and why?" For detailed coverage of observability fundamentals, see [What is Data Observability? The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars). It focuses on five key pillars: - **Freshness**: Is data arriving on schedule? - **Volume**: Are record counts within normal ranges? - **Schema**: Has the structure of data changed unexpectedly? - **Distribution**: Are value distributions behaving normally? - **Lineage**: How does data flow through our systems? Instead of predefined tests, observability systems use baseline metrics, statistical analysis, and machine learning to detect anomalies. Modern observability platforms (as of 2025) increasingly leverage AI-powered anomaly detection that adapts to seasonal patterns, business cycles, and evolving data characteristics. For instance, if your user events table typically receives 10,000 records per hour but suddenly drops to 100, an observability platform would flag this volume anomaly even without an explicit test. The power of data observability is its ability to surface unknown issues. By monitoring the overall health of your data ecosystem, including integration with OpenTelemetry for distributed tracing and correlation across systems, it can catch problems you didn't anticipate. However, this breadth comes with trade-offs: observability systems may generate false positives and require tuning to understand what constitutes normal behavior. ## Key Differences ### Philosophy Data quality is prescriptive: you define the rules. Data observability is descriptive: the system learns patterns and detects deviations. ### Coverage Data quality provides deep validation for known requirements. Data observability provides broad monitoring for unexpected issues. ### Implementation Data quality tests are explicit assertions in your code. Observability requires instrumentation, metrics collection, and analysis infrastructure. ### Response Time Data quality catches issues at test execution time, often blocking bad data from propagating. Observability detects trends and anomalies over time, enabling faster root cause analysis. ### Mental Model Think of data quality as unit tests for your data, while data observability is like application performance monitoring (APM) for your data pipelines. ## Streaming Integration: A Critical Context The distinction between data quality and observability becomes even more important in streaming environments. Traditional batch processing allows for comprehensive testing before data moves downstream, but streaming systems demand different approaches. In streaming architectures built on platforms like Apache Kafka, data flows continuously with minimal latency. Modern governance platforms like Conduktor provide comprehensive visibility into Kafka clusters, schema registries, and data flows, bridging the gap between quality testing and observability monitoring. To implement data quality policies in your streaming pipelines, see [Conduktor's Data Quality Policies guide](https://docs.conduktor.io/guide/conduktor-concepts/data-quality-policies) and [Enforcing Data Quality](https://docs.conduktor.io/guide/use-cases/enforce-data-quality). For streaming use cases: - **Data quality** can be implemented as stream processors that validate records in real-time, potentially routing invalid data to [dead letter queues](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling) for later analysis and reprocessing - **Data observability** monitors stream health metrics like [consumer lag](https://www.conduktor.io/glossary/consumer-lag-monitoring), partition distribution, throughput rates, and schema evolution You might set up data quality rules to validate message schemas against your registry, while simultaneously using observability to monitor whether consumers are keeping up with producers or if topic partition counts are balanced. The real-time nature of streaming makes observability particularly valuable. Rather than waiting for batch tests to run, observability can alert you within seconds when message volumes drop, lag increases, or data distributions shift unexpectedly. ## Using Both Together The most effective data reliability strategy combines both approaches. For a comprehensive framework that integrates quality and observability, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework). 1. **Use data quality** for critical business rules and compliance requirements. If customer IDs must be unique or transaction amounts must be positive, enforce these with explicit tests. Implement [automated data quality testing](https://www.conduktor.io/glossary/automated-data-quality-testing) to catch issues early in your pipelines. 2. **Use data observability** for holistic system health and anomaly detection. Monitor freshness, volume, and distribution to catch emerging issues. Track [data quality dimensions](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) across your entire data ecosystem. 3. **Let observability inform quality rules**. When observability surfaces a recurring anomaly, codify it as a quality test. This creates a feedback loop that continuously strengthens your data contracts. 4. **Use quality test failures to enrich observability context**. When tests fail, capture metadata that helps explain the failure in your observability platform. Modern tools support integration between validation frameworks and observability dashboards. ## Data Contracts: Bridging Quality and Observability A growing best practice in 2025 is the use of data contracts that formalize expectations between data producers and consumers. These contracts combine elements of both quality testing and observability monitoring: - **Contract definitions** specify schema, freshness SLAs, volume expectations, and quality thresholds - **Quality tests** validate that data meets contract specifications - **Observability monitoring** tracks contract compliance over time and alerts on violations - **Lineage tracking** identifies impact when contracts are breached For organizations implementing data contracts, see [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) for implementation patterns. ## Choosing the Right Approach For data engineers and data quality analysts, consider: - Start with **data quality** for well-understood, critical requirements - Add **data observability** as your systems scale and complexity increases - Implement **observability first** in streaming environments where comprehensive testing isn't feasible - Use both when reliability is paramount - Consider **data contracts** to formalize expectations and combine quality + observability practices ## Conclusion Data quality and data observability aren't competing approaches — they're complementary. Quality provides precise, rule-based validation for known requirements. Observability offers broad monitoring for system health and emerging issues. The goal isn't to choose one over the other. The precision of quality testing and the breadth of observability monitoring work together as a data reliability strategy. ## Related Concepts - [What is Data Observability: The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars) - Deep dive into observability fundamentals - [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) - Implementing systematic quality management - [Data Freshness Monitoring: SLA Management](https://www.conduktor.io/glossary/data-freshness-monitoring-sla-management) - Timeliness as a quality dimension ## Sources and References - [The Five Pillars of Data Observability](https://www.montecarlodata.com/blog-what-is-data-observability/) - [Great Expectations 1.0+ Documentation](https://docs.greatexpectations.io/) - [Soda Core: Open-Source Data Quality Testing](https://docs.soda.io/soda-core/) - [Observability Engineering: Achieving Production Excellence](https://www.oreilly.com/library/view/observability-engineering/9781492076438/) - [Data Quality Fundamentals (Google Cloud)](https://cloud.google.com/architecture/dq-fundamentals) - [OpenTelemetry for Data Observability](https://opentelemetry.io/) - [Apache Kafka Monitoring and Observability](https://kafka.apache.org/documentation/#monitoring) - [Data Contracts: From Ground Zero to Automated Governance](https://www.datamesh-architecture.com/data-contracts) Source: https://www.conduktor.io/glossary/data-quality-vs-data-observability-key-differences --- # Data Versioning in Streaming: Managing Event History In traditional databases, versioning is straightforward: you have snapshots, backups, and transaction logs. But in streaming systems where millions of events flow continuously, how do you preserve history, enable reproducibility, and maintain governance? Data versioning in streaming contexts is the practice of tracking and managing different versions of your event data, schemas, and derived datasets over time. This capability is critical for machine learning reproducibility, regulatory compliance, debugging production issues, and maintaining trust in your data pipelines. Let's explore how to implement effective versioning strategies in streaming architectures. ![Data Versioning Layers](https://www.conduktor.io/assets/images/glossary/data-versioning-in-streaming-0.webp) ## Streaming fundamentals Before getting into versioning strategies, here are the key concepts: - **Event**: A record representing something that happened at a specific time, a user action, sensor reading, database change, or business transaction. Events are immutable facts about the past. - **Stream**: A continuous, ordered sequence of events flowing over time. Unlike batch datasets that have a defined beginning and end, streams are unbounded and always evolving. - **Producer**: An application or service that writes events to a stream. For example, a web server producing user clickstream events, or a database producing change data capture events. - **Consumer**: An application that reads and processes events from a stream. Consumers might aggregate data, trigger actions, update dashboards, or feed machine learning models. - **Offset**: A sequential position number assigned to each event within a partition. Think of it like a line number in a log file, it identifies exactly where an event appears in the stream. Offsets enable precise replay and checkpointing. - **Partition**: A subdivision of a stream that enables parallel processing. Events with the same key (like customer ID) go to the same partition, maintaining order within that partition while allowing the overall stream to scale horizontally. - **Schema**: The structure and data types of an event, its fields, their types, and validation rules. Schemas act as contracts between producers and consumers, ensuring data compatibility. Understanding these building blocks is essential because versioning in streaming systems operates at multiple levels: versioning the schemas that define event structure, versioning the events themselves through offsets and timestamps, and versioning the derived datasets created from streams. ## Why Data Versioning Matters in Streaming Unlike batch systems that process static datasets, streaming systems handle continuously evolving data. This creates unique challenges and requirements: - **Reproducibility**: Machine learning models need to be retrained on the exact same data to validate improvements. Without versioning, you cannot reproduce the training dataset that was used three months ago, making model debugging impossible. - **Auditing and Compliance**: Regulatory requirements often demand the ability to reconstruct exactly what happened at any point in time. Financial services, healthcare, and other regulated industries must prove what data was processed and when. - **Debugging Production Issues**: When a pipeline produces incorrect results, you need to replay events exactly as they appeared at the time of failure. Versioning enables time-travel debugging to isolate the root cause. - **Safe Evolution**: As business requirements change, event schemas evolve. Versioning ensures that consumers can handle both old and new event formats gracefully, preventing breaking changes from cascading through your system. ## Schema Versioning: The Foundation The most fundamental form of versioning in streaming systems is schema versioning. Every event schema should carry a version identifier, typically managed through a Schema Registry. A Schema Registry is a centralized repository that stores and validates event schemas. When a producer writes an event, it registers the schema (if new) and includes a schema ID in the message. Consumers fetch the schema by ID to deserialize and validate incoming events. This ensures producers and consumers agree on data structure without hardcoding schemas in application code. **Modern Schema Registry Options (2025):** - **Open Source**: Karapace (Aiven's Kafka Schema Registry), Apicurio Registry (Red Hat), Confluent Schema Registry Community Edition - **Cloud-Native**: AWS Glue Schema Registry, Azure Schema Registry, Upstash Schema Registry - **Commercial Platforms**: Conduktor Platform provides enterprise-grade schema governance with validation, testing, and policy enforcement across multiple registries Schema registries support multiple serialization formats: **Avro** (compact binary, strong typing), **Protobuf** (efficient, backward/forward compatible), and **JSON Schema** (human-readable, flexible). Avro remains popular for its schema evolution capabilities, while Protobuf is preferred for polyglot environments. Consider an e-commerce order event that evolves over time: ```json // Version 1 { "orderId": "12345", "customerId": "c-789", "amount": 99.99 } // Version 2 - Added currency field with default { "orderId": "12345", "customerId": "c-789", "amount": 99.99, "currency": "USD" // New field with default value } // Version 3 - Added payment method { "orderId": "12345", "customerId": "c-789", "amount": 99.99, "currency": "USD", "paymentMethod": "credit_card" // Another new field } ``` With schema versioning, downstream consumers can detect which version they're processing and apply appropriate transformation logic. The Schema Registry enforces **compatibility modes** to prevent breaking changes: - **BACKWARD**: New schema can read data written with previous schema (most common). You can add optional fields or remove fields with defaults. - **FORWARD**: Old schema can read data written with new schema. You can add fields with defaults or remove optional fields. - **FULL**: Both backward and forward compatible. Most restrictive but safest for long-running systems. - **NONE**: No compatibility checks. Dangerous, use only during prototyping. A consumer handling multiple versions might look like: ```python def process_order(event): schema_version = event.schema_id if schema_version == 1: # Handle v1: assume USD if no currency return Order( order_id=event['orderId'], customer_id=event['customerId'], amount=event['amount'], currency='USD', payment_method='unknown' ) elif schema_version == 2: # Handle v2: default payment method return Order( order_id=event['orderId'], customer_id=event['customerId'], amount=event['amount'], currency=event['currency'], payment_method='unknown' ) else: # v3 and beyond # Handle latest schema return Order( order_id=event['orderId'], customer_id=event['customerId'], amount=event['amount'], currency=event['currency'], payment_method=event['paymentMethod'] ) ``` This explicit version handling ensures consumers gracefully adapt to schema evolution without breaking when encountering older events during replay or reprocessing. For deeper coverage of schema management patterns, see [Schema Registry and Schema Management](https://www.conduktor.io/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/schema-evolution-best-practices). ## Event-Level Versioning Strategies Beyond schemas, individual events themselves can carry version metadata. This enables fine-grained control over how different versions are processed: - **Semantic Versioning**: Events include version numbers that signal breaking changes (major), new features (minor), or bug fixes (patch). Consumers can route events based on version compatibility. - **Event Type Versioning**: Instead of evolving a single event type, create new types for major changes. For example, `OrderCreated` might evolve into `OrderCreatedV2`, allowing both versions to coexist during migration periods. - **Timestamp-Based Versioning**: Every event carries production and ingestion timestamps. These enable time-travel queries to reconstruct state as it existed at any historical moment. ## Time-Based Versioning: Offsets and Watermarks Apache Kafka 4.0+ (stable in 2025 with KRaft mode, removing ZooKeeper dependency) provides built-in versioning through offsets and timestamps. Each event receives a sequential offset number within its partition, creating an immutable, ordered log. Offsets enable precise replay: you can restart a consumer from any historical point and reprocess events exactly as they occurred. Combined with timestamp indexing, you can query specific time ranges. **Example: Offset-Based Replay** ```python from kafka import KafkaConsumer, TopicPartition # Replay events from a specific offset consumer = KafkaConsumer(bootstrap_servers=['localhost:9092']) partition = TopicPartition('orders', 0) consumer.assign([partition]) # Seek to offset 10000 to replay from that point consumer.seek(partition, 10000) for message in consumer: print(f"Offset {message.offset}: {message.value}") if message.offset >= 15000: break # Stop at offset 15000 ``` **Watermarks** extend this concept to handle late-arriving data, a common challenge in distributed systems where network delays, clock skew, or batch processing cause events to arrive out of order. For example, an event timestamped 2:00 PM might arrive after an event timestamped 2:05 PM. Watermarks represent a threshold timestamp where the system considers all earlier events to have arrived. Once a watermark passes time T, the system assumes it has seen all events with timestamps ≤ T. This enables: - **Window Closing**: Aggregation windows can be finalized and results emitted - **State Snapshots**: Processing state can be checkpointed at specific watermark positions - **Reproducible Processing**: Replaying from a watermark recreates identical results Stream processing frameworks like Apache Flink use watermarks extensively. Versioning watermarks, tracking which watermark position was used for specific computations, lets you snapshot processing state at specific points, enabling reproducible stream processing and time-travel debugging. For more on watermark mechanics and late data handling, see [Event Time and Watermarks in Flink](https://www.conduktor.io/event-time-and-watermarks-in-flink) and [Handling Late-Arriving Data in Streaming](https://www.conduktor.io/handling-late-arriving-data-in-streaming). ## Versioning for Machine Learning Workflows ML pipelines have unique versioning requirements. Training data must be frozen at specific points in time to ensure reproducibility: - **Training Data Snapshots**: Periodically materialize streaming features into versioned datasets. "Point-in-time correct" means capturing feature values exactly as they existed at a specific timestamp, avoiding data leakage where future information accidentally influences historical training data. - **Modern Feature Store Options (2025):** - **Open Source**: Feast (CNCF project), Feathr (LinkedIn) - **Cloud-Native**: AWS SageMaker Feature Store, Azure ML Feature Store, Google Vertex AI Feature Store - **Commercial**: Tecton, Hopsworks Feature stores create temporal snapshots, when you request features for timestamp T, you get the latest known values as of time T, never data from the future. This temporal consistency is critical for ML reproducibility. **Feature Versioning**: As feature engineering logic evolves, version the transformations themselves. Feature stores track both the data and the code that generated it, enabling model retraining with identical inputs. ```python # Example: Feast feature retrieval with point-in-time correctness from feast import FeatureStore store = FeatureStore(repo_path=".") # Get features as they existed on 2024-12-01 training_data = store.get_historical_features( entity_df=orders, # Orders from December 2024 features=[ "customer_profile:total_purchases", "customer_profile:avg_order_value", "customer_profile:days_since_last_purchase" ], event_timestamp_col="order_timestamp" ).to_df() # This ensures no data leakage—features reflect state at order time ``` **Model Lineage**: Connect model versions back to the specific data versions used for training. This creates an audit trail from predictions back through the model to the source events, essential for debugging model drift and regulatory compliance. For comprehensive coverage of feature store patterns, see [Feature Stores for Machine Learning](https://www.conduktor.io/feature-stores-for-machine-learning). ## Storage-Backed Versioning: Modern Table Formats Modern data lakes support versioning at the storage layer, treating data like Git treats code. These **table formats** sit between compute engines (Spark, Flink, Trino) and object storage (S3, ADLS, GCS), providing ACID transactions, schema evolution, and time travel over raw files. **Delta Lake Time Travel** (Delta 3.x in 2025) maintains transaction logs that record every change to a table. Queries can specify `VERSION AS OF` or `TIMESTAMP AS OF` to read historical snapshots. ```sql -- Query orders table as it existed on December 1st, 2024 SELECT * FROM orders TIMESTAMP AS OF '2024-12-01 00:00:00' WHERE customer_id = 'c-789'; -- Query a specific version number (from transaction log) SELECT * FROM orders VERSION AS OF 42 WHERE order_date = '2024-12-01'; -- Compare current vs historical data SELECT current.total_amount - historical.total_amount AS amount_diff FROM orders current LEFT JOIN orders TIMESTAMP AS OF '2024-11-01' historical ON current.order_id = historical.order_id; ``` - **Apache Iceberg** provides similar capabilities with hidden partitioning, snapshot isolation, and better handling of schema evolution. Multiple readers can query different versions simultaneously without interference. Iceberg is particularly strong for large-scale analytics workloads. - **Apache Hudi** (Hadoop Upserts Deletes and Incrementals) specializes in incremental processing and CDC (Change Data Capture) use cases. It offers "incremental queries" to fetch only changed records between two timestamps. - **Apache Paimon** (incubating, 2025) is a newer streaming lake storage format designed specifically for real-time data lakes. It provides streaming reads/writes with low latency while maintaining batch query performance. - **LakeFS** creates Git-like branches, commits, and tags for object storage. Data engineers can create experimental branches for testing new pipelines, merge validated changes, and roll back problematic deployments. This brings software engineering practices to data management. These systems solve the storage cost challenge through snapshot isolation and incremental changes. Only the delta between versions needs storage, not complete copies of entire datasets. Combined with streaming ingestion from Kafka (using Kafka's tiered storage feature in 4.0+ for cost-effective long-term retention), this creates end-to-end versioned data platforms. For deeper dives into table format internals, see [Iceberg Table Architecture: Metadata and Snapshots](https://www.conduktor.io/iceberg-table-architecture-metadata-and-snapshots) and [Streaming to Lakehouse Tables](https://www.conduktor.io/streaming-to-lakehouse-tables). ## Implementing Versioned Streaming Architectures A complete versioning strategy combines multiple layers: 1. **Schema versioning** in Schema Registry for event structure 2. **Offset-based versioning** in Kafka for event replay 3. **Snapshot versioning** in feature stores for ML reproducibility 4. **Storage versioning** in data lakes for long-term time travel Best practices include: - **Immutable Events**: Never update events in place. Instead, publish new events that supersede previous ones. This preserves complete history for auditing. For patterns on building systems around immutable event logs, see [Event Sourcing Patterns with Kafka](https://www.conduktor.io/event-sourcing-patterns-with-kafka). - **Version Metadata**: Include schema version, event version, and timestamps in every message. Make versioning explicit rather than implicit. - **Retention Policies**: Balance versioning needs against storage costs. Kafka can retain weeks or months of events; data lakes can retain years with tiered storage. For maintaining current state while preserving history, see [Kafka Log Compaction Explained](https://www.conduktor.io/kafka-log-compaction-explained) and [Tiered Storage in Kafka](https://www.conduktor.io/tiered-storage-in-kafka). - **Automated Testing**: Validate that consumers handle multiple schema versions correctly. Test backward and forward compatibility before deploying changes. ## Data Contracts and Governance Versioning is inseparable from data governance. Governance platforms enable teams to enforce versioning policies, require schema registration, and validate data contracts across streaming infrastructure. **Data contracts** specify not just schema versions but also SLAs, ownership, and evolution rules. A contract might declare: - Schema compatibility mode (BACKWARD, FORWARD, FULL) - Required vs. optional fields - Valid value ranges and constraints - Deprecation timelines for old versions - Owner teams and contact information - Expected throughput and latency SLAs When a producer wants to introduce a breaking change, governance systems can block the deployment and suggest backward-compatible alternatives. **Conduktor Platform** provides enterprise-grade data governance for streaming systems, including: - **Schema Governance**: Enforce compatibility rules, validate schemas against contracts with [Schema Registry management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry), and track schema evolution across environments - **Data Quality Rules**: Validate event content beyond schema structure (value ranges, business logic, data freshness) - **Self-Service Discovery**: Enable data consumers to find datasets, understand versions, and explore topic metadata using [topic discovery](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) - **Policy Enforcement**: Prevent breaking changes from reaching production, require approvals for schema evolution This prevents the chaos that emerges when teams evolve events independently without considering downstream impacts. Versioning combined with governance creates a controlled evolution path for streaming architectures, ensuring changes are deliberate, tested, and communicated to affected teams. For broader governance patterns, see [Policy Enforcement in Streaming](https://www.conduktor.io/policy-enforcement-in-streaming) and [Streaming Data Product Asset](https://www.conduktor.io/streaming-data-product-asset). ## Challenges and Tradeoffs Data versioning in streaming systems comes with costs: - **Storage Overhead**: Retaining historical versions consumes storage. Tiered storage and compaction strategies help, but there's always a tradeoff between retention and cost. - **Complexity**: Supporting multiple versions increases code complexity. Transformation logic must handle various schemas, and testing becomes more demanding. - **Performance**: Reading historical versions may be slower than querying current state. Indexing and caching strategies are necessary for acceptable query performance. - **Consistency**: In distributed streaming systems, determining "what version was active at time T" requires careful timestamp coordination and clock synchronization. ## Conclusion Data versioning transforms streaming systems from ephemeral pipelines into reliable, auditable platforms. Combining schema versioning, offset-based replay, storage time travel, and governance policies gives you reproducibility, compliance, and safe evolution. The investment pays off in faster debugging, more reliable ML models, and stronger regulatory confidence. Versioning is no longer optional in modern streaming architectures — it's a baseline requirement for production systems. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Platform providing offset-based versioning and replay capabilities - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Processing guarantees essential for reproducible versioning - [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters) - Protecting versioned event history ## Sources - [Delta Lake Time Travel Documentation](https://docs.delta.io/latest/delta-batch.html#-deltatimetravel) - [Apache Iceberg Table Versioning](https://iceberg.apache.org/docs/latest/) - [LakeFS Documentation: Git for Data Lakes](https://docs.lakefs.io/) - [Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/) - [Feast Feature Store Versioning](https://docs.feast.dev/) Source: https://www.conduktor.io/glossary/data-versioning-in-streaming --- # DataOps for Streaming: Operational Excellence in Real-Time Systems As organizations increasingly rely on real-time data streaming for critical business operations, the need for strong operational practices has grown. DataOps — the application of DevOps principles to data systems — provides the methodology and tooling to deliver streaming systems with speed, quality, and reliability. ![DataOps Principles](https://www.conduktor.io/assets/images/glossary/dataops-for-streaming-0.webp) ## What is DataOps? DataOps is a collaborative, process-oriented methodology that brings together data engineers, platform teams, and stakeholders to improve the quality, speed, and reliability of data analytics and pipelines. Drawing from DevOps, Agile, and lean manufacturing principles, DataOps emphasizes: - **Automation**: Reducing manual intervention through CI/CD, infrastructure as code, and automated testing - **Collaboration**: Breaking down silos between data engineering, platform operations, and business teams - **Continuous improvement**: Iterative development with rapid feedback loops - **Quality focus**: Built-in testing, monitoring, and validation at every stage - **Observability**: Comprehensive monitoring and alerting to detect and resolve issues quickly For batch-oriented data systems, DataOps has proven valuable. For streaming systems operating 24/7 with millisecond latencies, it becomes essential. ## Why Streaming Demands DataOps Streaming architectures present unique operational challenges that make DataOps practices critical: - **Continuous operation**: Unlike batch jobs that run on schedules, streaming pipelines operate continuously. A deployment error or configuration drift can immediately impact downstream consumers, making automated testing and safe deployment strategies non-negotiable. - **Complex dependencies**: Streaming systems involve multiple components, message brokers, stream processors, state stores, connectors, each with their own configurations, versions, and operational characteristics. Managing this complexity manually is error-prone and doesn't scale. - **Real-time requirements**: When data must flow within seconds or milliseconds, there's no room for lengthy manual reviews or slow deployment processes. Automation enables teams to move fast while maintaining reliability. - **Stateful processing**: Many streaming applications maintain state, making deployments and scaling more complex than stateless services. DataOps practices help manage state migrations, backups, and recovery procedures systematically. ## Core DataOps Practices for Streaming ### Infrastructure as Code Treating infrastructure configuration as versioned code is foundational to DataOps. For streaming systems, this means: **Declarative definitions**: Define Kafka topics, schemas, connectors, stream processing applications, and their configurations in code using tools like Terraform, Pulumi, or Kubernetes operators. ```hcl # Example using Terraform Kafka provider terraform { required_providers { kafka = { source = "Mongey/kafka" version = "~> 0.7" } } } resource "kafka_topic" "user_events" { name = "user.events.v1" partitions = 12 replication_factor = 3 config = { "retention.ms" = "604800000" "cleanup.policy" = "delete" "compression.type" = "lz4" "min.insync.replicas" = "2" } } ``` - **Version control**: Store all infrastructure definitions in Git, enabling change tracking, code review, and rollback capabilities. - **Reproducibility**: Deploy identical environments for development, staging, and production from the same codebase, reducing environment-specific bugs. - **KRaft-aware configuration**: With Apache Kafka's move to KRaft mode (production-ready since 3.3, mandatory in 4.0+), infrastructure code should target KRaft-based clusters rather than ZooKeeper-dependent configurations. This simplifies operations by removing ZooKeeper management from your DataOps workflows and enables faster metadata operations. ### Automated Testing Strategies Comprehensive testing ensures streaming applications behave correctly before reaching production: **Unit testing**: Test individual stream processing functions in isolation. For Kafka Streams applications, the TopologyTestDriver (a Kafka Streams testing utility) enables fast, local testing without external dependencies. ```java // Example: Testing a Kafka Streams topology TopologyTestDriver testDriver = new TopologyTestDriver(topology, config); TestInputTopic inputTopic = testDriver.createInputTopic("input-topic", stringSerde, stringSerde); TestOutputTopic outputTopic = testDriver.createOutputTopic("output-topic", stringSerde, longSerde); inputTopic.pipeInput("key1", "value1"); assertEquals(1L, outputTopic.readValue()); ``` - **Integration testing**: Validate interactions between components using embedded brokers or containerized environments (Testcontainers with Kafka). Test schema compatibility, serialization, error handling, and state management. Verify [data quality](https://www.conduktor.io/glossary/building-a-data-quality-framework) rules are enforced at pipeline boundaries. - **End-to-end testing**: Deploy complete pipelines in staging environments and validate end-to-end behavior with realistic data volumes and patterns. Test failover scenarios, exactly-once semantics, and recovery from failures. - **Property-based testing**: Generate test cases automatically to verify streaming application properties hold across diverse inputs and edge cases. ### CI/CD for Streaming Pipelines Continuous integration and deployment pipelines automate the path from code commit to production: **Automated builds**: Compile applications, run tests, and build container images on every commit. Fail fast when tests don't pass. ```yaml # Example: GitHub Actions workflow for Kafka Streams application name: CI/CD Pipeline on: [push] jobs: test-and-build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up JDK 21 uses: actions/setup-java@v4 with: java-version: '21' - name: Run unit tests run: ./gradlew test - name: Build Docker image run: docker build -t myapp:${{ github.sha }} . - name: Push to registry run: docker push myapp:${{ github.sha }} ``` - **Deployment automation**: Use GitOps tools like ArgoCD or FluxCD to automate deployments to Kubernetes clusters, ensuring production matches your Git repository state. Traditional CI/CD platforms like Jenkins or GitHub Actions can also orchestrate deployments. - **Configuration validation**: Automatically validate topic configurations, schema compatibility, and application settings before deployment. Tools like Conduktor can validate configurations against governance policies and operational best practices. - **Gradual rollouts**: Implement deployment patterns that minimize risk and enable quick rollback. ## Deployment Patterns for Streaming Safe deployment strategies are crucial when applications process continuous data streams: **Blue-Green Deployments**: Run old and new versions of an application simultaneously, routing traffic to the new version only after validation. For streaming, this means both versions consume from the same topics with different consumer groups, allowing instant rollback by switching active consumers. ```java // Example: Blue-green deployment with consumer groups // Blue (current): consumer-group-v1 // Green (new): consumer-group-v2 // Both consume same topics, switch traffic via routing logic Properties config = new Properties(); config.put(ConsumerConfig.GROUP_ID_CONFIG, "consumer-group-v2"); // Green version ``` - **Canary Deployments**: Deploy new versions to a small subset of instances first, monitoring metrics and error rates before rolling out to all instances. For Kafka Streams applications, this might mean directing a single partition to the new version initially. Monitor [consumer lag](https://www.conduktor.io/glossary/consumer-lag-monitoring) and error rates closely during canary phases. - **Rolling Deployments**: Gradually replace old instances with new ones, maintaining availability throughout. Critical for stateful applications where state must be migrated or rebuilt during deployment. Kafka Streams handles state store migrations automatically, but monitor state restoration progress and [backpressure](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) during transitions. - **Shadow Mode**: Run new versions alongside production, processing the same data but without affecting downstream systems, enabling validation with production traffic before full deployment. Use separate output topics for shadow deployments to compare results without impacting downstream consumers. ## Monitoring and Observability Comprehensive observability enables teams to understand system behavior and quickly identify issues: - **Metrics collection**: Track application-level metrics (processing rate, latency, error rate) and infrastructure metrics (CPU, memory, network, disk) using tools like Prometheus, Datadog, or New Relic. Kafka exposes JMX metrics that should be scraped and monitored continuously. - **Key performance indicators**: Monitor [consumer lag](https://www.conduktor.io/glossary/consumer-lag-monitoring) (how far consumers are behind producers), throughput, end-to-end latency, and error rates. Set SLOs (Service Level Objectives, quantifiable targets like "99.9% of messages processed within 500ms") and alert when they're breached. ```yaml # Example: Prometheus alerting rule for consumer lag - alert: HighConsumerLag expr: kafka_consumer_lag > 10000 for: 5m labels: severity: warning annotations: summary: "Consumer group {{ $labels.group }} has high lag" ``` - **Distributed tracing**: Implement tracing across streaming pipelines to understand request flows and identify bottlenecks. Modern observability uses OpenTelemetry (the industry-standard successor to OpenTracing/OpenCensus) with backends like Jaeger, Tempo, or cloud-native solutions. Trace context propagation through Kafka message headers enables end-to-end visibility. - **Log aggregation**: Centralize logs from all components using ELK stack (Elasticsearch, Logstash, Kibana), Splunk, or cloud-native solutions. Structured logging enables efficient querying during incident investigation. - **Dashboards**: Create role-specific dashboards for operators, developers, and business stakeholders. Visualize pipeline health, data flow, and business metrics in real-time. Tools like Conduktor provide purpose-built dashboards for Kafka operations, complementing general-purpose monitoring platforms. ## Incident Response and On-Call Practices When issues occur in 24/7 streaming systems, rapid response is essential: **Runbooks**: Document common failure scenarios and remediation steps. Include commands for scaling, restarting, and rollback procedures. ```markdown # Example runbook entry ## High Consumer Lag Alert - **Symptoms**: Consumer lag > 10,000 messages for > 5 minutes - **Impact**: Delayed data processing, potential SLA breach - **Investigation Steps**: 1. Check consumer health: `kafka-consumer-groups --describe --group ` 2. Verify partition distribution and rebalancing 3. Check for slow processing (CPU, GC, external dependencies) 4. Review error logs for exceptions **Remediation**: - Scale consumers horizontally if capacity-limited - Restart stuck consumers - Increase processing parallelism if I/O-bound - Rollback recent deployment if regression introduced ``` - **Alerting strategy**: Configure alerts that are actionable and minimize false positives. Alert on symptoms (high latency, errors) rather than just causes (high CPU). Follow the [consumer lag monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) best practices for streaming-specific alerting. - **On-call rotation**: Establish clear on-call responsibilities with escalation paths. Ensure on-call engineers have access to necessary tools and documentation. - **Post-incident reviews**: Conduct blameless postmortems after incidents to identify root causes and prevent recurrence. Document learnings and update runbooks. - **Chaos engineering**: Proactively test system resilience by injecting failures in non-production environments, validating failover mechanisms and recovery procedures. Tools like Conduktor Gateway, a Kafka proxy, enable controlled failure injection (network delays, message corruption, broker failures) without modifying application code. See [chaos engineering for streaming systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems) for comprehensive testing strategies. ## Cross-Team Collaboration DataOps success depends on effective collaboration across organizational boundaries: - **Shared ownership**: Data engineers and platform teams jointly own streaming systems' reliability and performance. - **Self-service platforms**: Build internal platforms that let data teams provision infrastructure, deploy applications, and monitor systems without constant platform team involvement. Platform teams provide guardrails and observability while data teams move independently. - **Clear interfaces**: Define clear contracts between teams through schemas, APIs, and SLAs. [Governance frameworks](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) help establish practices that clarify data ownership, [access policies](https://www.conduktor.io/glossary/access-control-for-streaming), and quality standards across streaming platforms. Schema registries enforce compatibility rules, preventing breaking changes from propagating to consumers. - **Knowledge sharing**: Regular demos, documentation, and training sessions ensure teams stay aligned on best practices and platform capabilities. ## DataOps Tooling Ecosystem Modern DataOps relies on a rich ecosystem of tools: - **Infrastructure as Code**: Terraform, Pulumi, CloudFormation for provisioning cloud resources and Kafka infrastructure; Kubernetes operators for declarative Kafka cluster management - **CI/CD**: GitHub Actions, GitLab CI, Jenkins for automated testing and builds; ArgoCD, FluxCD for GitOps-based deployment orchestration - **Containerization**: Docker for packaging streaming applications; Kubernetes for orchestration with horizontal pod autoscaling and rolling updates - **Monitoring**: Prometheus + Grafana for metrics collection and visualization; Datadog, New Relic for comprehensive observability platforms; JMX exporters for Kafka-specific metrics - **Observability**: OpenTelemetry for instrumentation and trace context propagation; Jaeger, Tempo, or Zipkin for distributed tracing backends; ELK stack (Elasticsearch, Logstash, Kibana) or Splunk for log aggregation - **Schema management**: Schema Registry (open-source or managed) for Avro/Protobuf/JSON Schema versioning; AWS Glue Schema Registry for AWS-native workflows; schema compatibility validation in CI pipelines - **Kafka operations and governance**: Conduktor for comprehensive Kafka management, monitoring, data exploration, and governance; Conduktor Gateway for proxy-based security, chaos testing, and policy enforcement; UI tools like Kafka UI, AKHQ for operational visibility - **Testing**: Testcontainers for integration testing with real Kafka brokers; TopologyTestDriver for Kafka Streams unit testing; Conduktor Gateway for chaos engineering and failure injection - **Security**: Vault, AWS Secrets Manager for credential management; cert-manager for TLS certificate automation; OAuth/OIDC providers for authentication ## Measuring DataOps Success Quantifiable metrics help teams assess DataOps maturity and identify improvement areas: - **Deployment frequency**: How often can you safely deploy changes to production? High-performing teams deploy multiple times per day. Track this metric per application and team. - **Lead time**: Time from code commit to running in production. Shorter lead times enable faster iteration. Target: < 1 hour for non-stateful applications, < 4 hours for stateful applications requiring state migration. - **Mean time to recovery (MTTR)**: How quickly can you restore service after an incident? Automation and observability reduce MTTR. Target: < 15 minutes for automated rollbacks, < 1 hour for complex incidents. - **Change failure rate**: Percentage of deployments causing incidents. Effective testing and gradual rollouts minimize failures. Target: < 5% for production deployments. - **Data quality metrics**: Track schema validation failures, data completeness, and accuracy to ensure pipeline quality. Implement [automated data quality testing](https://www.conduktor.io/glossary/automated-data-quality-testing) and monitor quality SLOs. - **Pipeline availability**: Percentage of time streaming pipelines are processing data within SLA. Target: 99.9% (43 minutes downtime/month) or better for critical pipelines. - **Developer satisfaction**: Survey teams regularly to identify friction points in development and deployment workflows. High satisfaction correlates with productivity and retention. ## Security and Compliance in DataOps Integrating security into DataOps workflows ensures streaming systems meet compliance requirements without slowing development: - **Secrets management**: Never commit credentials to version control. Use secret management tools (Vault, AWS Secrets Manager) integrated into CI/CD pipelines to inject secrets at runtime. - **Access control**: Implement [role-based access control](https://www.conduktor.io/glossary/access-control-for-streaming) and audit all administrative actions. Define who can deploy to production, modify topic configurations, or access sensitive data through policy-as-code. - **Encryption**: Enforce TLS for data in transit and encryption at rest for state stores and logs. Automate certificate rotation to prevent expiration incidents. - **Compliance automation**: Embed compliance checks into CI/CD pipelines. Validate data retention policies, PII handling, and geographic restrictions before deployment. [Audit logging](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) provides the trail necessary for compliance verification. - **Shift-left security**: Catch security issues during development through automated scanning of container images, dependency vulnerability checks, and configuration validation. ## Conclusion DataOps practices provide the operational foundation for streaming architectures at scale. Automation, rigorous testing, safe deployment strategies, and comprehensive observability let teams deliver streaming systems that are both agile and reliable. The investment pays off: faster delivery of new features, fewer production incidents, and teams that can change things without fear. With KRaft-based Kafka, OpenTelemetry, GitOps deployment patterns, and platforms like Conduktor, the tooling is mature. Start small — automate one pain point, add one test suite, improve one deployment process — and build from there. ## Related Concepts - [CI/CD Best Practices for Streaming Applications](https://www.conduktor.io/glossary/cicd-best-practices-for-streaming-applications) - Deployment automation for streaming systems - [Infrastructure as Code for Kafka Deployments](https://www.conduktor.io/glossary/infrastructure-as-code-for-kafka-deployments) - Managing streaming infrastructure as code - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) - Quality frameworks for DataOps ## Sources and References - [The DataOps Manifesto](https://dataopsmanifesto.org/) - [Accelerate: Building and Scaling High Performing Technology Organizations](https://itrevolution.com/product/accelerate/) - [Apache Kafka Operations Guide](https://kafka.apache.org/documentation/#operations) - [Site Reliability Engineering: How Google Runs Production Systems](https://sre.google/books/) - [OpenTelemetry Documentation](https://opentelemetry.io/docs/) - [DORA Metrics and DevOps Research](https://dora.dev/) Source: https://www.conduktor.io/glossary/dataops-for-streaming --- # dbt Tests and Data Quality Checks: Building Reliable Data Pipelines Data quality is the foundation of trustworthy analytics. As data pipelines grow in complexity, ensuring data integrity becomes critical. dbt (data build tool) provides a testing framework that lets Analytics Engineers and Data Quality Analysts define, execute, and monitor data quality checks throughout the transformation pipeline. ![dbt testing framework architecture](https://www.conduktor.io/assets/images/glossary/dbt-tests-and-data-quality-checks-0.webp) ## Understanding dbt's Testing Framework dbt's testing approach treats data quality as code, enabling version control, peer review, and automated validation. Tests in dbt are essentially SELECT queries that return failing rows. If a test returns zero rows, it passes; any rows returned indicate failures that need attention. ### Generic Tests vs. Singular Tests dbt offers two primary testing approaches: **Generic tests** are reusable, parameterized tests that can be applied to any column or model. The four built-in generic tests are: - `unique`: Ensures all values in a column are unique - `not_null`: Validates that a column contains no null values - `accepted_values`: Confirms values match a predefined list - `relationships`: Enforces referential integrity between tables (ensures foreign key values exist in the referenced table) **Singular tests** are custom SQL queries stored in the `tests/` directory, providing flexibility for complex business logic validation. ## Implementing Basic Data Quality Checks Let's start with a practical example. Consider a customer orders model where we need to ensure data quality: ```yaml # models/schema.yml version: 2 models: - name: fct_orders description: "Fact table containing order transactions" columns: - name: order_id description: "Unique identifier for each order" tests: - unique - not_null - name: customer_id description: "Foreign key to customers dimension" tests: - not_null - relationships: to: ref('dim_customers') field: customer_id - name: order_status description: "Current status of the order" tests: - accepted_values: values: ['pending', 'processing', 'shipped', 'delivered', 'cancelled'] - name: order_amount description: "Total order amount in USD" tests: - not_null ``` This example uses the standard built-in tests. For more advanced validations, you can leverage community packages like **dbt-utils** and **dbt-expectations**, which provide additional test types. Install them via `packages.yml`: ```yaml # packages.yml packages: - package: dbt-labs/dbt_utils version: 1.3.0 - package: calogica/dbt_expectations version: 0.10.4 ``` Then run `dbt deps` to install. Here's an extended example using these packages: ```yaml # models/schema.yml (extended) models: - name: fct_orders columns: - name: order_amount tests: - not_null - dbt_utils.expression_is_true: expression: ">= 0" - name: created_at tests: - dbt_expectations.expect_column_values_to_be_between: min_value: "'2020-01-01'" max_value: "CURRENT_DATE + INTERVAL '1 day'" ``` Running `dbt test` executes all defined tests and reports failures, enabling quick identification of data quality issues. A successful test run looks like: ```bash $ dbt test Running with dbt=1.8.0 Found 5 models, 12 tests, 0 snapshots, 0 analyses, 0 macros, 0 operations, 0 seed files, 0 sources 12:34:56 Running with dbt=1.8.0 12:34:57 1 of 12 START test not_null_fct_orders_order_id ...................... [RUN] 12:34:57 1 of 12 PASS not_null_fct_orders_order_id ............................ [PASS in 0.12s] 12:34:57 2 of 12 START test unique_fct_orders_order_id ........................ [RUN] 12:34:57 2 of 12 FAIL 3 unique_fct_orders_order_id ............................ [FAIL 3 in 0.15s] Completed with 1 error and 0 warnings: Failure in test unique_fct_orders_order_id (models/schema.yml) Got 3 results, configured to fail if != 0 compiled Code at target/compiled/my_project/models/schema.yml/unique_fct_orders_order_id.sql ``` Failed tests indicate data quality issues requiring investigation. Use `dbt test --store-failures` to save failing rows for analysis. ## Advanced Testing with Custom Assertions Beyond generic tests, singular tests enable complex validations. Create a file `tests/assert_order_totals_match.sql`: ```sql -- Validate that order totals equal sum of line items WITH order_totals AS ( SELECT order_id, order_amount FROM {{ ref('fct_orders') }} ), line_item_totals AS ( SELECT order_id, SUM(quantity * unit_price) AS calculated_total FROM {{ ref('fct_order_lines') }} GROUP BY order_id ) SELECT o.order_id, o.order_amount, l.calculated_total, ABS(o.order_amount - l.calculated_total) AS difference FROM order_totals o INNER JOIN line_item_totals l ON o.order_id = l.order_id WHERE ABS(o.order_amount - l.calculated_total) > 0.01 ``` This test ensures financial accuracy by validating that order totals match the sum of their line items, with a small tolerance for rounding differences. ## Unit Testing SQL Models (dbt v1.8+) A major advancement in dbt testing arrived with **unit tests** in dbt v1.8 (2024). Unlike data tests that run against your actual data warehouse, unit tests validate transformation logic using mock data, similar to unit tests in software engineering. Unit tests are defined in YAML and test specific models with predefined inputs and expected outputs: ```yaml # models/schema.yml unit_tests: - name: test_order_total_calculation model: fct_orders given: - input: ref('stg_orders') rows: - {order_id: 1, subtotal: 100.00, tax_rate: 0.10, shipping: 5.00} - {order_id: 2, subtotal: 50.00, tax_rate: 0.10, shipping: 3.00} expect: rows: - {order_id: 1, order_amount: 115.00} - {order_id: 2, order_amount: 58.00} - name: test_handles_null_shipping model: fct_orders given: - input: ref('stg_orders') rows: - {order_id: 3, subtotal: 100.00, tax_rate: 0.10, shipping: null} expect: rows: - {order_id: 3, order_amount: 110.00} ``` Run unit tests with `dbt test --select test_type:unit`. This provides fast feedback during development without needing actual data, making it ideal for: - Testing edge cases (nulls, zeros, negative values) - Validating complex calculation logic - Regression testing when refactoring models - Development environments where production data isn't available **Best practice**: Combine unit tests for logic validation with data tests for data quality validation. Unit tests ensure your code works correctly; data tests ensure your data meets quality standards. ## Test Coverage and Quality Metrics Measuring test coverage helps identify gaps in your data quality strategy. Use dbt packages like `dbt-coverage` to analyze which models and columns lack tests: ```bash dbt-coverage compute documentation --cov-report coverage-doc.json dbt-coverage compute test --cov-report coverage-test.json ``` Aim for comprehensive coverage on critical business metrics and primary keys. Not every column requires testing, but understanding your coverage helps prioritize testing efforts. ## Streaming Integration and Real-Time Data Quality Modern data architectures increasingly incorporate streaming data. While dbt traditionally operates on batch transformations, integrating with streaming platforms enables near-real-time quality validation. ### Streaming Data Quality Integration Kafka management platforms can complement dbt's testing framework for streaming scenarios. Here's how to architect an integrated approach: **Architecture Pattern:** 1. Stream events flow through Kafka topics 2. Governance platforms validate schema compliance and basic data quality rules 3. Data lands in your data warehouse (incremental materialization) 4. dbt tests run on micro-batches to validate transformations 5. Failed tests trigger alerts through monitoring systems Example incremental model with streaming considerations: ```sql -- models/fct_streaming_events.sql {{ config( materialized='incremental', unique_key='event_id', on_schema_change='fail' -- Fail if upstream schema changes unexpectedly ) }} SELECT event_id, user_id, event_type, event_timestamp, properties, _kafka_partition, -- Kafka metadata for debugging and partitioning _kafka_offset -- Kafka metadata for exactly-once processing FROM {{ source('kafka_raw', 'user_events') }} {% if is_incremental() %} -- Only process new events since last run WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM {{ this }}) {% endif %} ``` **Note**: When streaming data from Kafka to your warehouse, many connectors (Kafka Connect, Fivetran, Airbyte) automatically add metadata columns like `_kafka_partition` and `_kafka_offset`. These are valuable for debugging data issues and ensuring exactly-once processing semantics. Corresponding tests for streaming data: ```yaml # models/schema.yml models: - name: fct_streaming_events tests: # Ensure no duplicate events (composite uniqueness) # This catches cases where the same event_id appears with different timestamps - dbt_utils.unique_combination_of_columns: combination_of_columns: - event_id - event_timestamp columns: - name: event_id tests: - not_null - name: event_timestamp tests: - not_null # Validate events aren't too far in the future (clock skew tolerance) - dbt_utils.expression_is_true: expression: "<= CURRENT_TIMESTAMP + INTERVAL '5 minutes'" # Check for reasonable recency (catch stuck pipelines) - dbt_utils.expression_is_true: expression: ">= CURRENT_TIMESTAMP - INTERVAL '7 days'" config: severity: warn # Warn instead of failing for old data ``` ### Orchestrating Quality Checks For streaming workflows, consider running dbt tests on a schedule (e.g., every 15 minutes) to catch issues quickly: ```yaml # .github/workflows/dbt-streaming-tests.yml name: Streaming Data Quality Checks on: schedule: - cron: '*/15 * * * *' # Every 15 minutes jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: '3.11' - name: Install dbt run: | pip install dbt-core dbt-snowflake # or your warehouse adapter - name: Run dbt tests on streaming models run: | dbt test --select tag:streaming --fail-fast ``` **Modern Alternative**: For teams using dbt Cloud (2024-2025), leverage built-in CI/CD and scheduled runs instead of managing GitHub Actions: ```yaml # dbt_cloud.yml (dbt Cloud configuration) jobs: - name: Streaming Quality Checks schedule: "*/15 * * * *" # Every 15 minutes commands: - dbt test --select tag:streaming on_failure: - notify_slack: "#data-alerts" - create_incident: true ``` dbt Cloud provides integrated monitoring, automatic retries, and observability features that simplify production data quality operations. ## Best Practices for Data Quality at Scale 1. **Start with critical paths**: Focus testing efforts on models that directly impact business decisions 2. **Test early and often**: Run tests in development, CI/CD, and production environments 3. **Document test intent**: Add clear descriptions to help team members understand validation logic 4. **Configure test severity appropriately**: Use `severity: warn` for non-critical issues and `severity: error` for critical failures: ```yaml # dbt_project.yml tests: +store_failures: true # Save failing rows for investigation +store_failures_as: table # or 'view' +schema: dbt_test_failures # Individual test configuration models: - name: fct_orders columns: - name: order_amount tests: - not_null # Critical - fails the build - dbt_utils.expression_is_true: expression: ">= 0" config: severity: warn # Non-critical - warns but doesn't fail ``` 5. **Store test failures for analysis**: Enable `store_failures: true` to save failing rows in your warehouse, making debugging faster 6. **Monitor test performance**: Track test execution times to prevent bottlenecks. Use `dbt test --select state:modified+` to test only changed models in CI 7. **Integrate with alerting**: Connect test failures to Slack, PagerDuty, or other notification systems 8. **Use Elementary Data for observability**: Consider data observability tools like Elementary (open-source) for automatic anomaly detection and test result dashboards ## Conclusion dbt's testing framework makes data quality a first-class concern. Modern dbt (v1.8+) provides a complete testing toolkit: - **Data tests** validate that actual data meets quality standards - **Unit tests** verify transformation logic works correctly with mock data - **Severity configurations** allow graceful degradation for non-critical issues - **Store failures** enables deep investigation of quality issues Combining generic tests for common patterns, singular tests for complex business logic, unit tests for transformation validation, and integration with streaming platforms lets teams build pipelines that maintain quality from source to consumption. Treat tests as living documentation that evolves with your data models. As your understanding of data quality requirements deepens, refine your testing strategy to catch issues before they reach stakeholders. ## Related Concepts - [Great Expectations: Data Testing Framework](https://www.conduktor.io/great-expectations-data-testing-framework) - Complementary testing framework for data validation beyond dbt - [Automated Data Quality Testing](https://www.conduktor.io/automated-data-quality-testing) - Broader patterns for automated testing across data pipelines - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/data-quality-dimensions-accuracy-completeness-and-consistency) - Understanding what to test for ## Sources and References - [dbt Documentation: Data Tests](https://docs.getdbt.com/docs/build/data-tests) - [dbt Documentation: Unit Tests](https://docs.getdbt.com/docs/build/unit-tests) - [dbt Utils Package: Advanced Testing](https://github.com/dbt-labs/dbt-utils) - [dbt Expectations Package](https://github.com/calogica/dbt-expectations) - [dbt Best Practices: Testing Guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview) - [Elementary Data: Open-Source Data Observability](https://docs.elementary-data.com/) - [dbt Cloud: Continuous Integration](https://docs.getdbt.com/docs/deploy/continuous-integration) - [Great Expectations Integration with dbt](https://docs.greatexpectations.io/docs/deployment_patterns/how_to_use_great_expectations_with_dbt) - [Analytics Engineering: Testing and Data Quality](https://www.getdbt.com/analytics-engineering/transformation/data-quality/) Source: https://www.conduktor.io/glossary/dbt-tests-and-data-quality-checks --- # Dead Letter Queues for Error Handling When building event-driven systems, not every message can be processed successfully. Network failures, schema mismatches, invalid data, and application bugs all contribute to processing failures. Dead Letter Queues (DLQs) provide a systematic approach to handle these failures without losing data or blocking the processing pipeline. ![Dead Letter Queue flow diagram](https://www.conduktor.io/assets/images/glossary/dead-letter-queues-for-error-handling-0.webp) ## What is a Dead Letter Queue? A Dead Letter Queue is a designated location where messages that cannot be processed successfully are sent for later inspection and handling. Rather than discarding failed messages or retrying them indefinitely, the system routes them to a separate queue where they can be examined, debugged, and potentially reprocessed. The concept originates from traditional message queue systems but has become essential in modern streaming architectures. The core idea is simple: when a message fails processing after a defined number of retry attempts, move it to the DLQ and continue processing other messages. This pattern prevents problematic messages from blocking the entire processing pipeline. Without a DLQ, a single corrupted message could cause a consumer to crash repeatedly, stalling all downstream processing. ## How Dead Letter Queues Work The typical DLQ flow follows these steps: 1. A consumer attempts to process a message 2. Processing fails due to an error (deserialization failure, validation error, downstream service unavailable) 3. The system retries processing according to configured retry policies 4. After exhausting retries, the message is routed to the DLQ 5. The original consumer continues processing subsequent messages 6. Operations teams investigate DLQ messages and determine remediation steps Most implementations preserve metadata alongside failed messages, including error details, timestamps, retry counts, and the original topic or queue. This context is crucial for debugging and determining whether messages should be reprocessed. ## Dead Letter Queues in Data Streaming Dead Letter Queues are particularly important in data streaming platforms like Apache Kafka, where continuous processing of high-volume event streams is critical. ### DLQs in Modern Kafka (KRaft Mode) With Kafka 4.0's removal of ZooKeeper in favor of KRaft (Kafka Raft metadata mode), DLQ topic management becomes more streamlined through the KRaft metadata layer. DLQ topics benefit from faster metadata operations and improved cluster recovery times, making error handling more responsive in large-scale deployments. The simplified architecture also reduces operational complexity when managing DLQ topics across multiple environments. ### Kafka Connect Error Handling Kafka Connect provides built-in DLQ support for sink connectors (connectors that write data from Kafka to external systems like databases or cloud storage). When a sink connector cannot write a record to the destination system, it can route the failed record to a designated Kafka topic. This prevents the connector from halting on errors while preserving failed records for analysis. Configuration example: ```properties errors.tolerance=all errors.deadletterqueue.topic.name=dlq-orders errors.deadletterqueue.topic.replication.factor=3 errors.deadletterqueue.context.headers.enable=true errors.deadletterqueue.key.serializer=org.apache.kafka.common.serialization.StringSerializer errors.deadletterqueue.value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer errors.log.include.messages=true ``` The `errors.deadletterqueue.context.headers.enable` setting adds headers containing error information (including the exception class name, message, and stack trace), making debugging significantly easier. The `errors.log.include.messages` option provides detailed error logging for faster troubleshooting. Using `ByteArraySerializer` for the value serializer ensures failed messages are preserved exactly as received, regardless of deserialization issues. ### Stream Processing Applications Stream processing frameworks like Kafka Streams and Apache Flink often implement custom DLQ patterns. A common approach involves catching exceptions during processing and producing failed messages to a separate topic. #### Kafka Streams 3.x+ Pattern Modern Kafka Streams applications use the `FixedKeyProcessor` API for DLQ handling: ```java // Kafka Streams 3.x+ DLQ pattern stream.processValues(() -> new FixedKeyProcessor() { private FixedKeyProcessorContext context; private KafkaProducer dlqProducer; @Override public void init(FixedKeyProcessorContext context) { this.context = context; this.dlqProducer = new KafkaProducer<>(dlqProducerConfig()); } @Override public void process(FixedKeyRecord record) { try { ProcessedValue result = processMessage(record.value()); context.forward(record.withValue(result)); } catch (Exception e) { // Send to DLQ with error context DLQRecord dlqRecord = DLQRecord.builder() .originalKey(record.key()) .originalValue(record.value()) .errorMessage(e.getMessage()) .stackTrace(getStackTrace(e)) .timestamp(System.currentTimeMillis()) .originalHeaders(record.headers()) .build(); dlqProducer.send(new ProducerRecord<>("dlq-topic", record.key(), dlqRecord)); } } @Override public void close() { if (dlqProducer != null) { dlqProducer.close(); } } }); ``` This pattern maintains streaming throughput while isolating problematic messages. The `FixedKeyProcessor` API ensures type safety and proper key handling when routing to DLQs. #### Apache Flink Side Outputs Pattern Apache Flink 1.18+ uses side outputs for elegant DLQ routing, allowing failed messages to be processed in separate streams: ```java // Flink 1.18+ DLQ pattern using side outputs OutputTag dlqTag = new OutputTag("dlq-output"){}; SingleOutputStreamOperator mainStream = stream .process(new ProcessFunction() { @Override public void processElement(Event event, Context ctx, Collector out) { try { out.collect(processEvent(event)); } catch (Exception e) { // Route to DLQ side output with error context DLQMessage dlqMessage = DLQMessage.builder() .originalEvent(event) .errorMessage(e.getMessage()) .stackTrace(ExceptionUtils.getStackTrace(e)) .processingTime(ctx.timestamp()) .build(); ctx.output(dlqTag, dlqMessage); } } }); // Route DLQ messages to a Kafka sink DataStream dlqStream = mainStream.getSideOutput(dlqTag); dlqStream.sinkTo(KafkaSink.builder() .setBootstrapServers(bootstrapServers) .setRecordSerializer(dlqSerializer) .build()); ``` Side outputs provide type-safe error handling without mixing failed and successful processing paths, maintaining clean stream topology. ### Consumer Group Error Handling Application consumers can implement DLQ patterns by catching processing exceptions and producing failed messages to a DLQ topic within the same transaction scope (when using Kafka transactions). This ensures atomic DLQ routing and offset commits, preventing message loss. Without transactional guarantees, DLQ routing typically provides at-least-once semantics, where a message might appear in the DLQ multiple times if the consumer crashes after producing to the DLQ but before committing the offset. For more details on transactional processing, see [exactly-once semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) and [Kafka transactions deep dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). ## Common Use Cases and Patterns Dead Letter Queues address several recurring scenarios in distributed systems: - **Deserialization Failures**: When message schemas evolve incompatibly or producers send malformed data, consumers may fail to deserialize messages. Rather than crashing, the consumer routes the unparseable message to a DLQ. Proper [schema registry and schema management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) helps prevent many deserialization issues before they occur. - **Validation Errors**: Business rule violations or constraint failures can render messages unprocessable. A DLQ allows these messages to be examined and potentially corrected or discarded based on business requirements. - **Transient vs. Persistent Failures**: Distinguishing between temporary failures and permanent failures helps determine appropriate retry strategies: - **Transient failures**: Network timeouts, downstream service temporarily unavailable, rate limit exceeded. These should be retried 3-5 times with exponential backoff before moving to DLQ. - **Persistent failures**: Invalid JSON schema, malformed data, constraint violations, unsupported message version. These should be routed to DLQ immediately without retries. Understanding this distinction prevents wasting resources retrying messages that will never succeed, while giving legitimately failed operations a chance to recover. A circuit breaker pattern (which automatically stops retry attempts after detecting repeated failures) can help prevent retry storms when transient failures become prolonged outages. **Schema Evolution Issues**: When consumers cannot handle messages produced with newer schemas, DLQs provide a buffer while systems are updated to support the new format. Following [schema evolution best practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) helps minimize these scenarios. ### Real-World Example: E-Commerce Order Processing Consider an e-commerce system processing order events from a `orders` topic. The order fulfillment service encounters orders with invalid product IDs that fail inventory lookups. Rather than blocking all order processing, these problematic orders move to a `dlq-orders` topic. The DLQ message includes rich context: ```json { "headers": { "original.topic": "orders", "error.class": "ProductNotFoundException", "error.message": "Product ID 'XYZ-999' not found in inventory database", "retry.count": "3", "failed.timestamp": "2025-12-16T14:30:00Z" }, "key": "order-12345", "value": { "orderId": "order-12345", "customerId": "customer-789", "items": [{"productId": "XYZ-999", "quantity": 2}], "total": 89.99 } } ``` The data quality team investigates the DLQ and discovers the issue: the product catalog synchronization job failed overnight, leaving inventory data stale. After fixing the sync issue and updating the product database, they replay the DLQ messages back to the original `orders` topic using their platform's DLQ management tools. The orders process successfully on the second attempt, and customers receive their fulfillment notifications. This scenario demonstrates how DLQs integrate with broader [data quality dimensions](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) and [data quality incident](https://www.conduktor.io/glossary/data-quality-incidents) management practices. ## Best Practices and Considerations Implementing effective DLQ strategies requires careful planning: **Monitor DLQ Growth**: Rising DLQ message counts often indicate systemic issues requiring immediate attention. Set up alerts when DLQ volumes exceed thresholds. Modern observability practices using OpenTelemetry can track key DLQ metrics: - `dlq.messages.total`: Total count of messages routed to DLQ - `dlq.messages.rate`: Messages per second entering DLQ - `dlq.processing.latency`: Time from initial processing attempt to DLQ routing - `dlq.error.type`: Categorization of error types (deserialization, validation, timeout) These metrics help identify patterns and prioritize remediation efforts. Implementing comprehensive [Kafka cluster monitoring](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) and [data observability](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars) practices ensures DLQ issues are detected and resolved quickly. - **Retention Policies**: Configure appropriate retention for DLQ topics. Messages may need to persist longer than standard topics to allow thorough investigation. - **Include Rich Context**: Capture error messages, stack traces, timestamps, and original source information. This metadata accelerates debugging and root cause analysis. - **Reprocessing Strategy**: Establish clear procedures for reprocessing DLQ messages after fixing underlying issues. This might involve manual replay, automated retry with backoff, or permanent archival. - **Avoid Infinite Loops**: Ensure reprocessed messages don't end up back in the DLQ. Implement circuit breakers (mechanisms that automatically stop retry attempts after detecting repeated failures) or tracking to detect and prevent retry loops. - **Separate DLQs by Error Type**: Consider using different DLQ topics for different error categories (serialization errors, validation failures, downstream service errors). This enables targeted monitoring and remediation. ## Tooling and Platform Support Managing Dead Letter Queues effectively requires visibility into failed messages and streamlined workflows for analysis and reprocessing. Platforms like Conduktor provide dedicated features for DLQ management in Kafka environments. Teams can inspect DLQ messages, view error context in a unified interface, and replay messages back to source topics after resolving issues. This reduces the operational burden of manually managing DLQ topics through command-line tools or custom scripts. ### Testing DLQ Behavior with Conduktor Gateway Before deploying DLQ patterns to production, teams need confidence that error routing logic works correctly under various failure scenarios. [Conduktor Gateway, a Kafka proxy, enables systematic testing](https://docs.conduktor.io/guide/use-cases/chaos-testing) of DLQ behavior through policy-based failure injection: - **Deserialization error simulation**: Inject malformed data to verify DLQ routing for schema violations - **Downstream service failure testing**: Simulate timeout and connection errors to validate retry logic - **Transient failure patterns**: Test exponential backoff and circuit breaker behavior - **Message corruption scenarios**: Verify handling of messages with missing required fields This [chaos engineering](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems) approach helps teams discover edge cases and validate DLQ configurations before encountering real production failures. Combined with comprehensive [testing strategies for streaming applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications), teams can build confidence in their error handling logic. When evaluating DLQ tooling, look for capabilities including: - Message browsing and search across DLQ topics - Error context visualization (stack traces, headers, metadata) - Selective or bulk message replay with filtering - Integration with monitoring and alerting systems - Audit trails for DLQ operations - Failure injection for testing (via proxy like Conduktor Gateway) These features transform DLQs from passive error repositories into active components of operational workflows. ## Summary Dead Letter Queues are essential for resilient event-driven systems. Routing failed messages to a designated queue lets the system maintain processing throughput while preserving data for analysis and remediation. In streaming platforms like Kafka, DLQs prevent individual message failures from blocking entire pipelines. Kafka Connect provides native DLQ support, while stream processing applications implement custom DLQ patterns. Effective DLQ implementation requires careful attention to monitoring, metadata capture, retention policies, and reprocessing strategies. With proper tooling and operational procedures, DLQs become a proactive component of system reliability, not just a fallback. ## Related Concepts - [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) - Managing system load and preventing message failures - [Exactly-Once Semantics](https://www.conduktor.io/glossary/exactly-once-semantics) - Ensuring reliable message processing and DLQ routing - [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture) - Architectural patterns requiring robust error handling ## Sources and References 1. Confluent Documentation: [Kafka Connect Error Handling and Dead Letter Queues](https://docs.confluent.io/platform/current/connect/concepts.html#error-handling) 2. Apache Kafka Documentation: [Kafka Streams Exception Handling](https://kafka.apache.org/documentation/streams/developer-guide/error-handling) 3. Hohpe, G. and Woolf, B. (2003): *Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions*, Addison-Wesley Professional 4. AWS Documentation: [Amazon SQS Dead-Letter Queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html) 5. Kleppmann, M. (2017): *Designing Data-Intensive Applications*, O'Reilly Media Source: https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling --- # Debezium vs Airbyte: CDC Approaches **Debezium** is an open-source log-based Change Data Capture (CDC) platform most commonly deployed as Kafka Connect source connectors — it captures every row-level change from database transaction logs and streams them as events to Kafka topics in real time. **Airbyte** is an open-source data integration platform that supports polling-based and (for selected database sources) log-based CDC capture, targeting batch or micro-batch data movement primarily to data warehouses and lakes. The core difference: Debezium is a real-time streaming CDC engine designed around Kafka; Airbyte is a broader ELT orchestration platform where CDC is a capture mechanism for incremental database syncs. ## TL;DR | Dimension | Debezium | Airbyte | |---|---|---| | Primary use case | Real-time streaming CDC to Kafka | ELT data integration (batch, micro-batch, CDC) | | CDC mechanism | Log-based (transaction log tailing) | Log-based CDC + polling (full refresh, incremental) | | Delivery latency | Near real-time (sub-second to seconds) | Micro-batch (minutes) or batch (hours) | | Output target | Kafka topics (via Kafka Connect) | Data warehouses, lakes, databases, SaaS tools | | Kafka dependency | Mandatory for Kafka Connect deployment (also: Debezium Server for non-Kafka sinks) | None (standalone platform) | | Deployment | Kafka Connect workers | Airbyte server + workers (Docker / K8s) | | Connector count | ~30 database sources | Hundreds of sources and destinations | | License | Apache 2.0 | Elastic License 2.0 (core open source) | | Managed offering | Debezium Cloud (Red Hat) | Airbyte Cloud | | State management | Kafka Connect offsets + database-specific (e.g., MySQL binlog position, PostgreSQL LSN) | Airbyte internal state store | ## What is Debezium? Debezium is an open-source CDC platform built by Red Hat, designed to tail database transaction logs and emit row-level change events. The most common deployment is as a **Kafka Connect source connector**: each connector monitors one database instance and translates insert, update, and delete operations into structured events on Kafka topics. Debezium also supports standalone deployment via Debezium Server (which can route to non-Kafka sinks like Kinesis, Pub/Sub, or HTTP) and the embedded engine (library mode for custom applications). Supported databases include PostgreSQL (logical replication / pgoutput), MySQL (binlog), MongoDB (change streams), SQL Server (SQL Server Agent CDC), Oracle (LogMiner), and others. Because Debezium reads the transaction log rather than polling tables, it captures every change — including deletes — with low latency and minimal database load. Change events are routed to per-table Kafka topics (e.g., `dbserver1.public.orders`), where downstream consumers (Flink, Kafka Streams, Sink connectors) pick them up. See [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) and [What is Change Data Capture (CDC)?](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) for deeper coverage. ## What is Airbyte? Airbyte is an open-source data integration platform that orchestrates **ELT** (Extract, Load, Transform) pipelines from hundreds of sources to dozens of destinations. Sources include databases, SaaS APIs (Stripe, Salesforce, GitHub), files (S3, GCS), and event streams. Destinations include data warehouses (Snowflake, BigQuery, Redshift, Databricks), databases, and files. Airbyte supports two primary sync modes per stream: - **Full refresh**: extract all records every sync cycle - **Incremental**: extract records modified since the last sync For supported database sources (PostgreSQL, MySQL, SQL Server, MongoDB, Oracle), incremental sync can be powered by **CDC** (log-based capture using an embedded Debezium instance) rather than cursor-based polling. CDC is a capture mechanism, not a separate sync mode — it enables higher-fidelity incremental syncs including deletes. For database sources that support it, Airbyte's incremental CDC implementation uses an embedded Debezium instance within its connector runtimes — Airbyte manages the CDC lifecycle (snapshot, streaming, offset management) as part of the platform. Not all sources use Debezium; SaaS and API connectors use polling or webhook-based patterns. ## Architecture compared ### CDC mechanism **Debezium (log-based, always)**: Debezium exclusively uses database transaction logs. For PostgreSQL, it uses logical replication (pgoutput or decoderbufs plugin). For MySQL, it reads the binary log (binlog). For MongoDB, it uses change streams. This means: - Changes are captured as they are committed — sub-second latency - Deletes are captured (they appear as events in the log) - No polling queries hit the source database - The database must be configured to enable logical replication / binlog (requires database-level permissions) **Airbyte (log-based CDC + polling)**: Airbyte connectors can use multiple strategies depending on the source: - **CDC mode** (available for select connectors): embeds Debezium to read transaction logs, similar latency to native Debezium - **Incremental cursor**: queries `WHERE updated_at > last_sync_cursor` periodically — typically minutes to hours between syncs; deletes are NOT captured - **Full refresh**: reads entire table on each sync For most Airbyte use cases, syncs run on a schedule (hourly, daily) rather than continuously. This is intentional — Airbyte is optimized for data warehouse loading where freshness in minutes is acceptable. ### Kafka dependency **Debezium**: Kafka Connect is not optional. Debezium connectors run inside a Kafka Connect cluster, which requires a running Kafka cluster for offset storage, schema registry (if using Avro), and event delivery. The output is Kafka topics. Downstream consumers must read from Kafka. This makes Debezium the right choice when your architecture already centers on Kafka. **Airbyte**: Has no Kafka dependency. Sources connect directly to destinations. Airbyte can read from Kafka (Kafka source connector exists), but Kafka is not required for its operation. If your destination is a data warehouse and you don't have Kafka infrastructure, Airbyte is simpler to adopt. ### Output targets Debezium's output is Kafka topics. To load data into a database or data warehouse, you need a Kafka Connect sink connector (JDBC sink, Snowflake connector, BigQuery connector, etc.) or a stream processor (Flink, Kafka Streams) to transform and route the events. This is a multi-step pipeline: DB → Debezium → Kafka → Sink Connector → Destination. Airbyte's output is a direct connection from source to destination. DB → Airbyte → Data Warehouse. Fewer moving parts for the warehouse-loading use case. ### Connector ecosystem Debezium: ~30 database sources, focused on relational and NoSQL databases that expose transaction logs. No SaaS connectors. Airbyte: hundreds of sources and destinations, covering databases, SaaS APIs, files, and messaging systems (see the Airbyte connector catalog for current counts). Much broader coverage for data warehouse loading use cases. ### State and ordering Debezium tracks state as Kafka Connect offsets — database-specific positions (MySQL binlog file + position, PostgreSQL LSN, MongoDB resume token). Reconnecting after a gap resumes from the last committed offset. Airbyte tracks its own sync state (cursor values, CDC offsets) in its internal metadata store. If Airbyte is restarted, it resumes from the last recorded state. Both preserve event ordering within a partition/stream for their CDC modes. Debezium guarantees ordering within a Kafka partition (per table, per key). Airbyte's ordering guarantees depend on the destination's ingestion behavior. ## Operational trade-offs **Debezium advantages:** - True real-time streaming — sub-second latency for change events - Captures every change including deletes, schema changes (DDL events in some connectors) - Deep integration with Kafka ecosystem: Kafka Connect SMTs, Schema Registry, downstream Flink/Kafka Streams processing - No polling load on the source database - Apache 2.0 license, fully open source **Debezium disadvantages:** - Requires Kafka infrastructure (Kafka Connect cluster, Kafka brokers) - Database configuration required (logical replication slots, binlog enabled, LogMiner access) - Connector configuration is complex: snapshot mode, replication slot management, schema history topics, heartbeat configuration - No built-in transformations or destination-aware routing — requires additional connector or processor **Airbyte advantages:** - Hundreds of connectors covering databases, SaaS APIs, files (see Airbyte catalog for current count) - Simple UI for pipeline configuration — no Kafka expertise required - Direct source-to-destination without intermediate message bus - Dbt integration for in-warehouse transformations - Lower barrier to entry for teams without Kafka infrastructure **Airbyte disadvantages:** - Batch/micro-batch oriented — minutes of latency minimum, often hours - CDC mode (when available) is Debezium-embedded but managed for batch delivery, not true streaming - Non-CDC modes miss deletes - Elastic License 2.0 (EL2) restricts offering Airbyte as a managed service ## When to choose Debezium - You need **real-time streaming** of database changes (sub-second latency) - Your architecture already uses Kafka — you want changes flowing into Kafka topics for downstream processors - You need to **capture deletes** and schema changes reliably - You are building [CDC for microservices](https://www.conduktor.io/glossary/cdc-for-microservices-event-driven-architectures) or [CDC for real-time data warehousing](https://www.conduktor.io/glossary/cdc-for-real-time-data-warehousing) - You need the [outbox pattern](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing) for reliable event publishing from transactional databases - Your team has Kafka operations expertise ## When to choose Airbyte - You need to **sync data to a data warehouse** (Snowflake, BigQuery, Redshift) on a scheduled basis - You need **SaaS source connectors** (Salesforce, Stripe, GitHub, etc.) alongside database sources - You don't have Kafka infrastructure and don't want to build it - Minutes of latency is acceptable for your analytics use case - Your team wants a **UI-driven pipeline configuration** with minimal code ## Can Debezium and Airbyte coexist? Yes — they occupy different layers of a data architecture: - Use **Debezium** for real-time operational use cases: streaming CDC into Kafka, event-driven microservices, real-time analytics pipelines - Use **Airbyte** for batch ELT to data warehouses: historical loads, SaaS API ingestion, daily/hourly refreshes for BI A common pattern: Debezium feeds Kafka (operational streaming tier) while Airbyte feeds the data warehouse (analytical batch tier). Both read the same source database but serve different consumers with different latency requirements. See also: [Log-Based vs Query-Based CDC Comparison](https://www.conduktor.io/glossary/log-based-vs-query-based-cdc-comparison). **Does Airbyte use Debezium?** For supported database sources (such as PostgreSQL and MySQL) configured with CDC-based incremental sync, Airbyte embeds Debezium within its connector runtime to read the transaction log. Airbyte manages the Debezium lifecycle; you configure Airbyte, not Debezium directly. SaaS connectors and non-database sources use polling or webhook patterns — not Debezium. **Can Debezium load data directly into Snowflake or BigQuery?** Not directly. Debezium outputs to Kafka topics. Loading into Snowflake or BigQuery requires a Kafka Connect sink connector for that destination (e.g., Snowflake Kafka Connector, BigQuery Kafka Connector). This multi-hop pipeline adds latency and operational components but enables real-time streaming into the warehouse, which batch tools like Airbyte cannot match. **Is Debezium reliable? It seems complex.** Debezium is production-proven at scale (Netflix, Shopify, Airbnb). The complexity is real: replication slots must be managed to prevent WAL bloat in PostgreSQL, schema evolution requires careful handling, and the initial snapshot of large tables must be managed. Tools like Conduktor can help manage Kafka Connect connectors including Debezium connector lifecycle. **What is the license difference between Debezium and Airbyte?** Debezium is Apache 2.0 — fully permissive, including for managed service use. Airbyte uses the Elastic License 2.0 (EL2) for most components, which prohibits third parties from offering Airbyte as a managed service. For internal use, both licenses are effectively permissive. Source: https://www.conduktor.io/glossary/debezium-vs-airbyte --- # Delta Lake Deletion Vectors: Efficient Row-Level Deletes Row-level modifications in large-scale data lakes have traditionally been expensive. When you need to delete or update a few rows in a multi-gigabyte Parquet file, the standard approach requires rewriting the entire file. This write amplification problem becomes particularly acute with frequent small updates — GDPR compliance deletions, real-time corrections, or slowly changing dimensions. Delta Lake's deletion vectors feature, introduced in Delta Lake 2.3.0 (2023), addresses this by enabling efficient row-level deletes without rewriting data files. As of 2025, this feature is mature and available in Delta Lake 3.x+ with improved integration with Liquid Clustering and UniForm. This article covers how deletion vectors work, their performance characteristics, and best practices for production use. ![Deletion vectors vs traditional delete](https://www.conduktor.io/assets/images/glossary/delta-lake-deletion-vectors-efficient-row-level-deletes-0.webp) ## What Are Deletion Vectors? Deletion vectors are a metadata-based approach to marking rows as deleted without physically removing them from Parquet files. Instead of rewriting entire files when deleting rows, Delta Lake maintains a bitmap structure (a space-efficient data structure where each bit represents whether a row is deleted, essentially a list of true/false flags, one per row) that tracks which rows should be considered deleted during query execution. Each deletion vector is a compact binary structure that encodes the positions of deleted rows within a specific data file. These vectors are stored separately from the data files themselves, typically as part of the Delta transaction log or as small sidecar files (separate metadata files stored alongside the data) referenced by the log. Here's a simplified example of how a deletion vector works: ``` Data file: [row0, row1, row2, row3, row4] Deletion vector: [0, 0, 1, 0, 1] // Bitmap marks rows 2 and 4 as deleted Query result: [row0, row1, row3] // Deleted rows filtered at read time ``` The key innovation is that deletion vectors decouple logical deletion from physical file rewriting. When a query reads a data file with an associated deletion vector, the query engine filters out the marked rows at read time, presenting a consistent view of the data without the write overhead. This means queries do slightly more work at read time, but avoid massive write overhead (typically adding 5-15% read overhead while reducing write volume by 100-1000x). ## How Deletion Vectors Work When you execute a DELETE operation on a Delta table with deletion vectors enabled, the following process occurs: 1. **Row Identification**: Delta Lake identifies which data files contain rows matching the delete predicate 2. **Vector Creation**: For each affected file, a deletion vector is created or updated to mark the deleted row positions 3. **Metadata Update**: The transaction log is updated with references to the deletion vectors, but the original data files remain unchanged 4. **Read-Time Filtering**: Subsequent queries consult both the data files and their deletion vectors to return only non-deleted rows **Concurrent Delete Handling**: Starting with Delta Lake 3.0 (2024), deletion vectors support improved optimistic concurrency control. Multiple writers can delete different rows in the same file simultaneously, with automatic conflict resolution at commit time. This enables higher throughput for multi-writer DELETE workloads without the coordination overhead of traditional file-level locking. Consider this example: ```sql -- Enable deletion vectors on a new table CREATE TABLE user_events ( user_id BIGINT, event_type STRING, timestamp TIMESTAMP, data STRING ) USING DELTA TBLPROPERTIES ('delta.enableDeletionVectors' = 'true'); -- Insert sample data INSERT INTO user_events VALUES (1001, 'login', '2025-01-15 10:00:00', '{"ip": "192.168.1.1"}'), (1002, 'purchase', '2025-01-15 10:15:00', '{"amount": 99.99}'), (1003, 'logout', '2025-01-15 10:30:00', '{}'); -- Delete specific rows (uses deletion vectors) DELETE FROM user_events WHERE user_id = 1002; ``` In this example, instead of rewriting the entire Parquet file, Delta Lake creates a deletion vector marking the row with `user_id = 1002` as deleted. The physical file remains unchanged, but queries will exclude this row. ### UPDATE Operations with Deletion Vectors While DELETE operations only mark rows as deleted, UPDATE operations in Delta Lake 3.0+ can leverage deletion vectors for the "delete" phase of the read-modify-write cycle: ```sql -- Update uses deletion vectors internally UPDATE user_events SET data = '{"ip": "10.0.0.1"}' WHERE user_id = 1001; -- Internally: marks old row deleted via deletion vector, -- writes new row to new data file ``` This hybrid approach reduces write amplification for updates by avoiding full file rewrites while still creating new data for the updated rows. The old version is marked as deleted using a deletion vector, and the new version is written to a separate data file, minimizing the I/O overhead. ## Performance Benefits and Trade-offs The primary advantage of deletion vectors is **reduced write amplification**. Traditional delete operations on columnar formats require a copy-on-write approach (a pattern where any modification creates a new copy of the entire file) where entire files are rewritten. For large files, this means writing gigabytes of data to delete a handful of rows. ### Performance Comparison Consider a table with 1GB data files where you need to delete 1% of rows: **Traditional Approach:** - Files rewritten: 1GB per file - I/O overhead: Full file scan + full file write - Latency: Proportional to file size (~10-30 seconds per file) **Deletion Vectors Approach:** - Files rewritten: 0 bytes - Metadata written: ~10-100KB deletion vector - Latency: Near-constant time (~100-500ms) This represents a **100-1000x reduction** in write volume for small delete operations. ### Trade-offs to Consider While deletion vectors offer significant write performance improvements, they introduce read-time overhead: 1. **Read Performance**: Queries must load and apply deletion vectors, adding small overhead (typically 5-15%) 2. **File Fragmentation**: Deleted rows still occupy storage until compaction runs 3. **Compaction Requirements**: Periodic OPTIMIZE operations are needed to physically remove deleted rows ```sql -- Check deletion vector statistics DESCRIBE DETAIL user_events; -- Compact files to remove deleted rows physically OPTIMIZE user_events; -- For tables with frequent deletes, schedule regular compaction OPTIMIZE user_events WHERE timestamp < current_date() - INTERVAL 7 DAYS; ``` ## Enabling and Configuring Deletion Vectors Deletion vectors can be enabled at table creation or on existing tables: ```sql -- Enable on existing table ALTER TABLE user_events SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true'); -- Configure thresholds for automatic file rewriting ALTER TABLE user_events SET TBLPROPERTIES ( 'delta.deletionVectors.threshold' = '0.3', -- Rewrite file if >30% deleted 'delta.targetFileSize' = '128MB' ); ``` The `delta.deletionVectors.threshold` property controls when Delta Lake switches from using deletion vectors to rewriting files. If more than 30% of a file's rows are deleted, subsequent operations will rewrite the file to reclaim space (meaning files with heavy deletes eventually get rewritten automatically). ### Python API Example Deletion vectors can also be configured programmatically using PySpark: ```python # Enable deletion vectors programmatically (PySpark) from delta.tables import DeltaTable dt = DeltaTable.forPath(spark, "/path/to/user_events") dt.alter().setProperties({ 'delta.enableDeletionVectors': 'true', 'delta.deletionVectors.threshold': '0.3' }).execute() # Delete with deletion vectors dt.delete("user_id = 1002") # Update with deletion vectors dt.update( condition="user_id = 1001", set={"data": "'\"ip\": \"10.0.0.1\"}'"} ) ``` ### When to Use Deletion Vectors Deletion vectors are most effective when: - **Delete operations affect <20% of rows** in a file - **Deletes occur frequently** (daily or more) - **Write latency is critical** (real-time pipelines) - **Compliance requires timely deletion** (GDPR, CCPA) They are less beneficial when: - Deletes affect entire partitions (use DROP PARTITION instead) - Batch deletes remove >50% of data (traditional rewrite is more efficient) - Tables are rarely queried (read overhead not amortized) ## Streaming Ecosystem Integration Because deletion vectors avoid file rewrites, they're particularly valuable in streaming scenarios where data is continuously arriving and deletes must be processed without blocking writers. Deletion vectors integrate seamlessly with streaming pipelines, enabling real-time delete propagation without disrupting data ingestion workflows. ### Streaming Deletes with Structured Streaming Delta Lake's deletion vectors work with Spark Structured Streaming to handle Change Data Capture (CDC) scenarios: ```sql -- Streaming CDC pipeline with deletes CREATE OR REPLACE TEMP VIEW user_changes AS SELECT * FROM cloud_files( '/data/cdc-stream/', 'json', map('cloudFiles.inferColumnTypes', 'true') ); -- Apply changes using MERGE (leverages deletion vectors for deletes) MERGE INTO user_events target USING user_changes source ON target.user_id = source.user_id WHEN MATCHED AND source.operation = 'DELETE' THEN DELETE WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *; ``` This pattern allows streaming applications to process deletes efficiently without blocking writers or causing write amplification. ### Cross-Format Compatibility with UniForm Delta UniForm (introduced in 2024) enables Iceberg-compatible readers to access Delta tables with deletion vectors. When UniForm is enabled, deletion vectors are automatically translated to Iceberg's positional delete files, allowing heterogeneous query engines to read the same data: ```sql -- Enable UniForm on deletion vector table ALTER TABLE user_events SET TBLPROPERTIES ( 'delta.enableDeletionVectors' = 'true', 'delta.universalFormat.enabledFormats' = 'iceberg' ); -- Now both Delta and Iceberg readers can query with deletes applied -- Delta: SELECT * FROM delta.`/path/to/user_events` -- Iceberg: SELECT * FROM iceberg.user_events -- via Iceberg catalog ``` This enables query engines like Trino, Flink, Snowflake, and other Iceberg-compatible tools to read Delta tables with row-level deletes without requiring data rewrites. The deletion semantics are preserved across formats, ensuring consistent results regardless of the query engine used. ### Governance and Observability In production environments, tracking deletion vector usage is critical for capacity planning and performance monitoring. Data governance tools can provide visibility into Delta Lake operations by analyzing the transaction log, including: - **Deletion vector growth rates**: Monitor metadata overhead to predict storage requirements - **File-level fragmentation**: Identify tables needing compaction based on deletion ratios - **Query performance impact**: Track read-time overhead from deletion vectors across workloads - **Compliance tracking**: Verify deletion operations complete within SLA windows for regulatory requirements By integrating with Delta Lake's transaction log, governance platforms can audit which records were deleted, when, and by whom, critical for GDPR Article 17 (right to erasure) compliance and other data privacy regulations. ## Limitations and Considerations While deletion vectors solve write amplification, they introduce several operational considerations: ### Storage Overhead Deletion vectors consume metadata storage proportional to the number of deleted rows. For tables with sustained high delete rates, this metadata can accumulate: ```sql -- Monitor deletion vector overhead (Databricks) DESCRIBE DETAIL user_events; -- For detailed deletion vector statistics from transaction log: SELECT add.path AS file_path, add.deletionVector.pathOrInlineDv AS dv_location, add.deletionVector.cardinality AS deleted_rows, add.size_in_bytes AS file_size FROM ( SELECT explode(add) AS add FROM delta.`/path/to/table/_delta_log/*.json` ) WHERE add.deletionVector IS NOT NULL; ``` In PySpark, you can access these metrics programmatically: ```python from delta.tables import DeltaTable dt = DeltaTable.forPath(spark, "/path/to/user_events") details = dt.detail().collect()[0] print(f"Files: {details.numFiles}") print(f"Size: {details.sizeInBytes / 1024**3:.2f} GB") ``` ### Compatibility Deletion vectors require: - Delta Lake 2.3.0+ readers - Databricks Runtime 12.0+ or Apache Spark 3.3+ with Delta 2.3.0+ - Reader protocol version 3, writer protocol version 7 Protocol versions ensure compatibility between readers and writers. Version 3/7 means only newer clients (since 2023) can use deletion vectors. Tables using deletion vectors cannot be read by older clients. Consider this when sharing data across environments: ```sql -- Check protocol versions DESCRIBE HISTORY user_events; -- Downgrade if needed (removes deletion vectors) ALTER TABLE user_events SET TBLPROPERTIES ( 'delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '2' ); -- Databricks-specific command to purge deletion vectors: REORG TABLE user_events APPLY (PURGE); -- For OSS Delta Lake, run OPTIMIZE to consolidate files: OPTIMIZE user_events; -- Then vacuum to clean up old files VACUUM user_events; ``` ### VACUUM and Deletion Vector Cleanup VACUUM operations remove data files older than the retention period, but deletion vector metadata is cleaned up through different mechanisms: ```sql -- Standard vacuum removes old data files (default: 7 days retention) VACUUM user_events RETAIN 168 HOURS; -- Deletion vectors are cleaned automatically when: -- 1. Their referenced data files are vacuumed -- 2. OPTIMIZE rewrites files (consolidates deletion vectors into rewritten files) -- 3. REORG TABLE APPLY (PURGE) explicitly removes them (Databricks only) ``` Monitor deletion vector cleanup to ensure metadata doesn't accumulate excessively: ```sql -- Check for deletion vector metadata overhead DESCRIBE DETAIL user_events; -- Look for high deletion ratios indicating need for OPTIMIZE: -- If deletionVectorSizeInBytes is growing or many files have deletion vectors, -- schedule regular OPTIMIZE operations to consolidate and reclaim space ``` ### Z-Ordering and Data Skipping Deletion vectors work with Delta Lake's data skipping optimizations, but heavily deleted files may still be scanned unnecessarily. You can combine deletion vectors with Z-ORDERING for optimal performance: ```sql OPTIMIZE user_events ZORDER BY (user_id, timestamp); ``` Note: Z-ORDERING is a technique that arranges data by multiple columns to improve query performance through better data skipping (the query engine can skip entire files based on column statistics). ### Deletion Vectors with Liquid Clustering As of Delta Lake 3.1+ (2024), deletion vectors integrate seamlessly with Liquid Clustering, the successor to Z-ORDERING. Liquid Clustering automatically adapts partitioning strategies based on query patterns while maintaining deletion vector efficiency: ```sql -- Create table with liquid clustering and deletion vectors CREATE TABLE user_events_clustered ( user_id BIGINT, event_type STRING, timestamp TIMESTAMP, data STRING ) USING DELTA CLUSTER BY (user_id, DATE(timestamp)) TBLPROPERTIES ('delta.enableDeletionVectors' = 'true'); -- Clustering and deletion vectors work together automatically DELETE FROM user_events_clustered WHERE user_id = 1002; -- No manual OPTIMIZE ZORDER needed; liquid clustering adapts dynamically -- OPTIMIZE still runs to consolidate small files and clean deletion vectors OPTIMIZE user_events_clustered; ``` Liquid Clustering eliminates the need to manually choose clustering columns and reorder data. The clustering strategy evolves based on actual query patterns, while deletion vectors continue to provide efficient row-level deletes without file rewrites. This combination is the recommended approach for new tables in 2025, as it provides: - Automatic clustering adaptation (no manual Z-ORDERING tuning) - Efficient row-level deletes (deletion vectors) - Reduced maintenance overhead (fewer OPTIMIZE operations needed) For more details on Liquid Clustering, see the [Delta Lake Liquid Clustering](https://www.conduktor.io/glossary/delta-lake-liquid-clustering-adaptive-data-layout) article. ## Summary Delta Lake deletion vectors solve the write amplification problem inherent in columnar storage formats. By maintaining metadata-based deletion markers instead of rewriting entire files, deletion vectors enable: - **100-1000x reduction** in write volume for small deletes - **Sub-second delete latency** for compliance-driven row removal - **Seamless streaming integration** for CDC pipelines - **Backward-compatible compaction** through OPTIMIZE operations - **Cross-format compatibility** via UniForm for Iceberg readers (2024+) - **Concurrent delete handling** with improved optimistic concurrency (Delta 3.0+) The trade-offs are manageable: slight read overhead (5-15%), increased metadata storage, and the need for periodic compaction. For workloads with frequent small deletes, GDPR compliance, real-time corrections, or CDC pipelines, deletion vectors significantly improve operational efficiency. Best practices for 2025: 1. Enable deletion vectors for tables with frequent (<20% rows) deletes 2. **Use Liquid Clustering instead of Z-ORDERING** for new tables (Delta 3.1+) 3. Schedule regular OPTIMIZE operations to reclaim storage and consolidate deletion vectors 4. Monitor deletion vector overhead using DESCRIBE DETAIL and transaction log queries 5. Enable UniForm for cross-format compatibility if using multiple query engines 6. Leverage concurrent delete capabilities for multi-writer scenarios (Delta 3.0+) 7. Configure VACUUM retention policies to balance time-travel needs with storage costs As data lakes increasingly support transactional workloads and real-time updates, deletion vectors bring database-like mutation capabilities to open lakehouse architectures without sacrificing the scalability of columnar formats. The 2024-2025 improvements in Liquid Clustering integration, UniForm support, and concurrent delete handling make deletion vectors an essential feature for modern data lakehouse deployments. ## Related Concepts - [Delta Lake Transaction Log: How It Works](https://www.conduktor.io/delta-lake-transaction-log-how-it-works) - Understanding the transaction log that tracks deletion vectors - [Optimizing Delta Tables: OPTIMIZE and Z-ORDER](https://www.conduktor.io/optimizing-delta-tables-optimize-and-z-order) - Compaction strategies to consolidate deletion vectors - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/data-quality-dimensions-accuracy-completeness-and-consistency) - Ensuring data quality when performing row-level deletes ## Sources and References 1. **Delta Lake Documentation - Deletion Vectors**: [https://docs.delta.io/latest/delta-deletion-vectors.html](https://docs.delta.io/latest/delta-deletion-vectors.html) 2. **Databricks Blog - "Deletion Vectors: Fast and Efficient Deletes on Delta Lake"**: [https://www.databricks.com/blog/2023/03/03/deletion-vectors-fast-and-efficient-deletes-delta-lake.html](https://www.databricks.com/blog/2023/03/03/deletion-vectors-fast-and-efficient-deletes-delta-lake.html) 3. **Delta Lake Protocol - Reader/Writer Versions**: [https://github.com/delta-io/delta/blob/master/PROTOCOL.md](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) 4. **Apache Parquet Documentation**: [https://parquet.apache.org/docs/](https://parquet.apache.org/docs/) 5. **GDPR Article 17 - Right to Erasure**: [https://gdpr-info.eu/art-17-gdpr/](https://gdpr-info.eu/art-17-gdpr/) 6. **Databricks Documentation - OPTIMIZE Command**: [https://docs.databricks.com/sql/language-manual/delta-optimize.html](https://docs.databricks.com/sql/language-manual/delta-optimize.html) Source: https://www.conduktor.io/glossary/delta-lake-deletion-vectors-efficient-row-level-deletes --- # Delta Lake Liquid Clustering: Modern Partitioning Traditional partitioning strategies in data lakes present a difficult tradeoff: optimize for current query patterns and risk poor performance as workloads evolve, or maintain multiple copies of data organized differently. Delta Lake's Liquid Clustering takes a different approach — adaptive, automatic clustering that continuously optimizes data layout without the brittleness of static partitioning schemes. ![Traditional partitioning vs liquid clustering](https://www.conduktor.io/assets/images/glossary/delta-lake-liquid-clustering-modern-partitioning-0.webp) ## Understanding the Partitioning Problem Traditional Hive-style partitioning creates physical directory structures based on column values. While this approach provides predictable query performance for filters on partition columns, it introduces several challenges: - **Physical Directory Explosion**: High-cardinality columns create thousands of small files, degrading metadata operations and query planning performance. A table partitioned by `user_id` with millions of users becomes unmanageable. - **Query Pattern Lock-in**: Once you partition by `date` and `region`, queries filtering by `product_category` cannot benefit from data skipping. Changing partition schemes requires full table rewrites. - **Maintenance Overhead**: Small file problems accumulate rapidly. A streaming ingestion writing to hundreds of partitions every minute creates millions of tiny files, requiring constant OPTIMIZE operations. - **Multi-dimensional Access**: Modern analytics often require slicing data across multiple dimensions simultaneously. Traditional partitioning forces you to choose which dimension to optimize, leaving other access patterns suboptimal. ## Data Skipping: The Performance Foundation Before diving into Liquid Clustering, it's essential to understand **data skipping**, the core mechanism that makes both partitioning and clustering performant. Data skipping allows query engines to avoid reading files that definitely don't contain relevant data. Delta Lake maintains statistics (min/max values, null counts) for each column in each file. When you query `WHERE event_timestamp > '2025-12-01'`, the engine checks these statistics and skips any files whose maximum timestamp is before December 2025, without opening those files at all. Traditional partitioning achieves data skipping through directory structure (skip entire partition directories). Liquid Clustering achieves it through fine-grained statistics on clustered columns within files, enabling multi-dimensional skipping without directory proliferation. ## What is Liquid Clustering? Liquid Clustering replaces physical partitioning with flexible, multi-dimensional clustering that adapts to actual query patterns. Instead of creating directory hierarchies, Delta Lake colocates related data within files based on specified clustering columns, then continuously refines this layout as data arrives and query patterns emerge. The key innovation is **decoupling logical organization from physical storage**. Your data remains in a flat file structure while Delta Lake maintains internal metadata about how data is clustered within files. This enables: - **Multi-dimensional clustering**: Cluster by `date`, `region`, and `product_category` simultaneously - **Adaptive optimization**: OPTIMIZE operations incrementally improve layout based on data distribution - **No small file proliferation**: Write operations don't create per-value directories - **Flexible query patterns**: All clustering columns benefit from data skipping ## Implementing Liquid Clustering ### Creating a Liquid Clustered Table Liquid Clustering is enabled during table creation with the `CLUSTER BY` clause: ```sql CREATE TABLE events ( event_id STRING, user_id STRING, event_timestamp TIMESTAMP, event_type STRING, region STRING, properties MAP ) USING delta CLUSTER BY (event_timestamp, region, event_type); ``` This creates a table clustered across three dimensions. Unlike partitioning, this doesn't create separate directories, instead, Delta Lake organizes data within files to colocate rows with similar values across these columns. ### Choosing Clustering Columns Select clustering columns based on common filter predicates and join keys in your queries: - **High-cardinality columns**: Unlike partitioning, Liquid Clustering handles high-cardinality columns efficiently. Clustering by `user_id` works well without creating millions of directories. - **Multiple filter dimensions**: Include all columns frequently used in WHERE clauses. A table accessed by time ranges, geographic filters, and event types should cluster on all three. - **Ordering matters**: List columns in order of filter selectivity, how much data typical filters eliminate. More selective columns (like timestamp ranges that filter out 90%+ of data) should typically come first, followed by medium-cardinality columns (region might filter 70-80%), then categorical columns (event_type might filter 50%). This ordering maximizes data skipping efficiency by pruning files more aggressively on the first clustering column. ### Write and Optimize Operations Data written to liquid clustered tables is automatically organized, but incremental optimization improves layout quality: ```sql -- Standard write operations work unchanged INSERT INTO events SELECT * FROM raw_events WHERE event_date = current_date(); -- Incremental optimization refines clustering OPTIMIZE events; -- Full optimization for major layout improvements OPTIMIZE events FULL; ``` The OPTIMIZE command incrementally reorders data within files to improve clustering quality. Unlike traditional partition optimization, this operation is incremental, it processes only files that would benefit from reorganization. ## Migration from Partitioned Tables Migrating from traditional partitioning to Liquid Clustering requires careful planning to avoid query downtime: ### Strategy 1: In-Place Conversion For smaller tables or during maintenance windows: ```sql -- Remove existing partitioning ALTER TABLE events SET TBLPROPERTIES ( 'delta.enableDeletionVectors' = 'true' ); -- Enable liquid clustering ALTER TABLE events CLUSTER BY (event_timestamp, region, event_type); -- Optimize to apply clustering OPTIMIZE events FULL; ``` This approach rewrites the entire table, which may be expensive for large datasets. ### Strategy 2: Parallel Table Migration For production systems requiring zero downtime: ```sql -- Create new clustered table CREATE TABLE events_clustered USING delta CLUSTER BY (event_timestamp, region, event_type) AS SELECT * FROM events; -- Switch application queries via view CREATE OR REPLACE VIEW events AS SELECT * FROM events_clustered; -- Backfill historical data incrementally INSERT INTO events_clustered SELECT * FROM events_partitioned WHERE event_date BETWEEN '2024-01-01' AND '2024-01-31'; ``` This pattern enables gradual migration with fallback capability. ### Validation and Performance Testing Before full migration, validate query performance improvements: ```sql -- Compare query performance EXPLAIN COST SELECT COUNT(*) FROM events WHERE event_timestamp > '2024-12-01' AND region = 'us-west-2' AND event_type = 'purchase'; -- Check clustering statistics DESCRIBE DETAIL events; ``` Monitor data skipping metrics to confirm that clustering improves file pruning for common query patterns. ## Streaming Ecosystem Integration Liquid Clustering integrates seamlessly with streaming architectures, addressing common challenges in real-time data processing: ### Structured Streaming Integration Spark Structured Streaming writes to liquid clustered tables work identically to standard Delta tables: ```python # Streaming write to clustered table (PySpark) from pyspark.sql.functions import from_json, col from pyspark.sql.types import StructType, StructField, StringType, TimestampType event_schema = StructType([ StructField("event_id", StringType()), StructField("user_id", StringType()), StructField("event_timestamp", TimestampType()), StructField("event_type", StringType()), StructField("region", StringType()) ]) (spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "localhost:9092") .option("subscribe", "events") .load() .select(from_json(col("value").cast("string"), event_schema).alias("data")) .select("data.*") .writeStream .format("delta") .option("checkpointLocation", "/checkpoints/events") .trigger(processingTime="5 minutes") .toTable("events")) # Already clustered by (event_timestamp, region, event_type) ``` The streaming job writes data continuously while Delta Lake maintains clustering quality through background optimization. ### Continuous Optimization Pattern Combine streaming writes with scheduled optimization: ```python # Streaming job continuously writes # Separate optimization job runs periodically from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() # Run every hour to maintain clustering quality # Use SQL for reliability and compatibility spark.sql("OPTIMIZE events") # For more control over optimization scope: # spark.sql("OPTIMIZE events WHERE event_timestamp >= current_date() - INTERVAL 7 DAYS") ``` This pattern ensures that streaming micro-batches don't degrade clustering quality over time. ### Governance and Visibility with Conduktor When managing streaming data pipelines that write to Delta Lake, governance tooling becomes critical. Conduktor provides visibility into the Kafka layer of your pipeline, helping teams govern the data flowing into Delta Lake tables. For liquid clustered tables receiving streaming data, Conduktor enables: - **Schema evolution tracking**: Monitor schema changes in upstream Kafka topics that affect Delta table structure and clustering columns. Use [Schema Registry management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry) to track schema versions - **Data quality validation**: Ensure clustering column values meet expected distributions before writes, catching data quality issues that would degrade clustering performance - **Audit logging**: Conduktor Gateway, a Kafka proxy, has audit logs that capture all producer/consumer connections to Kafka topics, providing visibility into which applications write to which topics upstream of your Delta tables - **Performance monitoring**: Identify streaming jobs that write poorly distributed data affecting clustering quality, with alerts for anomalous data distributions Conduktor Gateway can also test resilience scenarios, simulating Kafka broker failures or network partitions, to verify that your streaming-to-Delta pipeline maintains data integrity and clustering quality under adverse conditions. ## Advanced Patterns and Best Practices ### Time-Based Clustering with Retention Combine Liquid Clustering with time-based retention policies: ```sql CREATE TABLE metrics ( metric_id STRING, timestamp TIMESTAMP, value DOUBLE, environment STRING, tags MAP ) USING delta CLUSTER BY (timestamp, environment) TBLPROPERTIES ( 'delta.deletedFileRetentionDuration' = 'interval 7 days' ); -- Efficient time-based deletion with clustering -- Clustering by timestamp ensures files are organized by time, -- making this DELETE scan far fewer files DELETE FROM metrics WHERE timestamp < current_timestamp() - INTERVAL 90 DAYS; ``` Clustering by timestamp ensures deletion operations efficiently target relevant files. ### Multi-Tenant Table Design Liquid Clustering excels in multi-tenant scenarios: ```sql CREATE TABLE tenant_data ( tenant_id STRING, created_at TIMESTAMP, data_type STRING, payload BINARY ) USING delta CLUSTER BY (tenant_id, created_at, data_type); ``` Queries filtering by `tenant_id` benefit from aggressive file pruning, isolating tenant data without physical partitioning overhead. ### Monitoring Clustering Quality Track clustering effectiveness through Delta table statistics: ```sql -- View table-level clustering information DESCRIBE DETAIL events; -- Check clustering effectiveness via file-level statistics -- Note: This requires accessing Delta's internal metadata SELECT file_path, num_records, size_bytes / 1024 / 1024 as size_mb FROM delta.`/path/to/events/`.files ORDER BY file_path; -- Alternative: Use Databricks-specific commands if available ANALYZE TABLE events COMPUTE STATISTICS FOR COLUMNS event_timestamp, region, event_type; -- Review statistics DESCRIBE EXTENDED events event_timestamp; ``` Well-clustered tables show: - **Tight min/max ranges per file**: Each file contains a narrow range of values for clustering columns (e.g., one file contains events from 2025-12-15 10:00 to 10:15, not scattered across months) - **Consistent file sizes**: Files are similarly sized (target 128MB-1GB) rather than having many small files - **High data skipping rates**: Query metrics show most files are skipped for typical queries **Success Criteria**: A well-clustered table typically shows 80-95% of files skipped for queries filtering on clustering columns, compared to 0-30% for unclustered tables. ## 2025 Features and Enhancements Delta Lake's liquid clustering capabilities have matured significantly, with several key enhancements in Delta Lake 3.0+ and ecosystem improvements: ### Unity Catalog Integration Unity Catalog (the unified governance layer for lakehouse platforms) now provides enhanced clustering management: ```sql -- Create clustered table with Unity Catalog governance CREATE TABLE main.analytics.events ( event_id STRING, event_timestamp TIMESTAMP, region STRING, event_type STRING ) USING delta CLUSTER BY (event_timestamp, region, event_type) TBLPROPERTIES ( 'delta.enableChangeDataFeed' = 'true', 'delta.autoOptimize.optimizeWrite' = 'true', 'delta.autoOptimize.autoCompact' = 'true' ); -- Unity Catalog tracks clustering metadata and optimization history DESCRIBE HISTORY main.analytics.events; ``` Unity Catalog maintains lineage information showing how clustering configuration changes impact downstream queries and dashboards. ### Auto-Optimize and Auto-Compaction Modern Delta Lake configurations include automatic optimization to reduce operational overhead: ```sql -- Enable auto-compaction for streaming writes ALTER TABLE events SET TBLPROPERTIES ( 'delta.autoOptimize.optimizeWrite' = 'true', -- Optimizes write file sizes 'delta.autoOptimize.autoCompact' = 'true' -- Automatically runs compaction ); ``` These properties ensure that streaming micro-batches are automatically compacted without manual OPTIMIZE commands, maintaining clustering quality continuously. ### Photon Engine Optimizations The Photon vectorized query engine includes specific optimizations for liquid clustered tables: - **Vectorized statistics evaluation**: Faster file pruning during query planning for clustered columns - **Adaptive scan batching**: Dynamically adjusts file read batching based on clustering quality - **Cluster-aware data caching**: Preferentially caches data from well-clustered files with high reuse potential These optimizations provide 2-5x query performance improvements on clustered tables compared to earlier engines. ### Cloud Storage Optimizations Liquid clustering works seamlessly with modern cloud storage tiers: - **AWS S3 Express One Zone**: Low-latency storage for frequently accessed clustered data with single-digit millisecond latencies - **Azure Premium Blob Storage**: Optimized for clustered table operations with consistent low-latency access - **GCS Turbo Replication**: Fast cross-region replication for globally distributed clustered tables Clustering combined with cloud tiering enables cost optimization: frequently queried recent data in fast storage tiers, historical data in standard storage, all with consistent clustering performance. ## Performance Considerations ### When Liquid Clustering Excels - **Multi-dimensional query patterns**: Analytics requiring filters across multiple columns simultaneously - **Evolving workloads**: Query patterns that change over time as business needs shift - **High-cardinality clustering**: Tables needing organization by columns with thousands of distinct values - **Streaming ingestion**: Continuous writes where partition explosion creates operational burden ### When Traditional Partitioning Remains Viable - **Stable, single-dimension access**: Tables always queried by a single low-cardinality column (e.g., daily batch processing by date) - **Physical data isolation requirements**: Compliance scenarios requiring separate storage paths per partition - **Legacy tooling compatibility**: Systems expecting physical partition directory structures ### Cost and Performance Tradeoffs Liquid Clustering trades write-time organization for read-time performance: - **Write performance**: Slightly slower writes due to clustering metadata maintenance (typically 5-10% overhead) - **Optimization cost**: Regular OPTIMIZE operations consume compute resources (plan for hourly or daily optimization jobs) - **Query performance**: Substantially improved for multi-dimensional queries through better data skipping (typical improvements: 60-90% reduction in data scanned, 3-8x faster query execution for analytical workloads) - **Storage efficiency**: Reduced storage from fewer small files and better compression within clustered files (typically 15-30% storage savings compared to unoptimized tables) **Real-world example**: A production events table with 500 million rows showed: - Before clustering: 18,000 files, average query scanning 45 GB, 32-second typical query time - After clustering: 850 files, average query scanning 8 GB, 6-second typical query time - 82% reduction in data scanned, 81% faster queries, 25% smaller storage footprint ## Summary Delta Lake Liquid Clustering replaces rigid partitioning schemes with adaptive, multi-dimensional clustering. By decoupling logical organization from physical storage layout, it eliminates the brittleness and operational overhead of traditional partitioning while improving query performance across diverse access patterns. Key points for data engineers: - **Flexible optimization**: Cluster across multiple high-cardinality columns without directory explosion - **Adaptive performance**: Data layout improves continuously through incremental optimization (enabled by auto-compaction in 2025) - **Streaming-friendly**: Integrates with real-time ingestion patterns, with Conduktor providing governance for Kafka-to-Delta pipelines - **Simplified operations**: Eliminates partition maintenance, small file management, and layout redesign projects - **Unity Catalog integration**: Enterprise governance with lineage tracking and clustering metadata management - **Proven performance**: Typical improvements include 60-90% reduction in data scanned and 3-8x faster queries The migration path from existing partitioned tables is straightforward, and the 2025 auto-optimization capabilities make liquid clustering a production-ready, self-managing data organization strategy. ## Related Concepts - [Introduction to Lakehouse Architecture](https://www.conduktor.io/introduction-to-lakehouse-architecture) - The foundational architecture that Liquid Clustering optimizes - [Optimizing Delta Tables: OPTIMIZE and Z-ORDER](https://www.conduktor.io/optimizing-delta-tables-optimize-and-z-order) - Traditional optimization approaches that Liquid Clustering supersedes - [Streaming to Lakehouse Tables](https://www.conduktor.io/streaming-to-lakehouse-tables) - How streaming writes interact with Liquid Clustering ## Sources and References - [Delta Lake Liquid Clustering Documentation](https://docs.delta.io/latest/delta-clustering.html) - Official Delta Lake documentation on Liquid Clustering features and implementation - [Databricks Liquid Clustering Guide](https://www.databricks.com/blog/2023/04/24/liquid-clustering-delta-lake.html) - Comprehensive guide on Liquid Clustering architecture and best practices - [Delta Lake GitHub Repository](https://github.com/delta-io/delta) - Open-source Delta Lake implementation and protocol specifications - [Structured Streaming with Delta Lake](https://docs.delta.io/latest/delta-streaming.html) - Integration patterns for streaming workloads - [Delta Lake Performance Tuning](https://docs.delta.io/latest/optimizations-oss.html) - Optimization strategies for Delta Lake tables including clustering Source: https://www.conduktor.io/glossary/delta-lake-liquid-clustering-modern-partitioning --- # Delta Lake Transaction Log: How It Works Delta Lake transforms cloud object storage into a reliable, **ACID-compliant** data platform through a single critical component: the transaction log. ACID stands for Atomicity (operations fully succeed or fail), Consistency (data remains valid), Isolation (concurrent operations don't interfere), and Durability (committed changes persist). Understanding how this transaction log works is essential for data engineers building modern data lakehouses — it underpins every feature that differentiates Delta Lake from raw Parquet files on S3 or ADLS. This article covers the architecture, mechanics, and operational characteristics of Delta Lake's transaction log. ## What Is the Delta Lake Transaction Log? The Delta Lake transaction log, also called the DeltaLog, is an ordered record of every transaction committed to a Delta table since its creation. Stored as a series of JSON files in the `_delta_log` subdirectory of your table, this log serves as the single source of truth for the table's current state. Each transaction appends a new JSON file numbered sequentially (e.g., `00000000000000000000.json`, `00000000000000000001.json`). These files contain metadata about: - **Add operations**: New data files added to the table - **Remove operations**: Files marked as logically deleted - **Metadata changes**: Schema evolution, table properties - **Protocol changes**: Delta Lake version requirements - **Commit info**: Timestamp, operation type, isolation level When a reader queries a Delta table, it reconstructs the current state by reading the transaction log from the beginning (or the last checkpoint) and applying each transaction sequentially. This append-only structure makes the log naturally immutable and provides built-in version control. ![delta-lake-transaction-log-how-it-works diagram 1](https://www.conduktor.io/assets/images/glossary/delta-lake-transaction-log-how-it-works-0.webp) ## The Optimistic Concurrency Commit Protocol Delta Lake uses an **optimistic concurrency control** protocol to enable multiple writers to modify a table simultaneously while maintaining ACID guarantees. "Optimistic" means Delta Lake assumes conflicts between writers are rare and doesn't lock resources upfront. Instead, each writer proceeds independently, and conflicts are detected only at commit time. This contrasts with "pessimistic" locking approaches (like traditional databases) that acquire locks before modifications, potentially blocking other writers even when conflicts wouldn't occur. The optimistic approach provides better performance in data lake scenarios where: - Most writes add new data (appends) rather than modifying existing rows - Writers often work on different partitions or time ranges - The cost of occasional retries is lower than the overhead of distributed locking This protocol relies on atomic file operations supported by cloud object stores (S3, Azure Blob Storage, Google Cloud Storage). ### How Concurrent Writes Work When a writer wants to commit changes: 1. **Read Phase**: The writer reads the latest version of the transaction log to understand the current state 2. **Execution Phase**: The writer performs the operation (write new Parquet files, compute statistics, etc.) 3. **Validation Phase**: Before committing, the writer re-reads the log to check if any new commits occurred 4. **Commit Phase**: If no conflicts exist, the writer attempts to atomically write the next sequential log entry ![delta-lake-transaction-log-how-it-works diagram 2](https://www.conduktor.io/assets/images/glossary/delta-lake-transaction-log-how-it-works-1.webp) The atomicity guarantee comes from cloud storage's conditional PUT operations (e.g., S3's PUT-if-absent, ADLS's create-if-not-exists). These operations either fully succeed or fully fail, there's no in-between state where a file is "partially written." When S3 receives two simultaneous PUT-if-absent requests for the same file, exactly one succeeds and the other fails immediately. The failed writer detects the conflict, re-validates against the new state, and retries if the operation is still valid. ### Multi-Statement Transactions (Delta 3.0+) Starting with Delta Lake 3.0 (2024), multi-statement transactions enable grouping multiple operations into a single atomic commit. This ensures that multiple related changes appear as one version in the transaction log, providing true ACID semantics across operations. ```sql -- Begin a multi-statement transaction BEGIN TRANSACTION; -- Multiple operations that commit together DELETE FROM orders WHERE status = 'cancelled'; UPDATE customers SET orders_count = orders_count - 1 WHERE customer_id IN (SELECT customer_id FROM orders WHERE status = 'cancelled'); INSERT INTO audit_log VALUES (current_timestamp(), 'cleanup', user()); -- Commit all changes as a single transaction log entry COMMIT; ``` Without multi-statement transactions, each operation would create a separate transaction log entry, potentially leaving the table in an inconsistent intermediate state if a failure occurs between operations. With multi-statement transactions, either all operations succeed together or all fail together, maintaining referential integrity. In the transaction log, a multi-statement transaction appears as a single version containing all `add`, `remove`, and `cdc` actions from all statements: ```json { "commitInfo": { "timestamp": 1735689600000, "operation": "TRANSACTION", "operationMetrics": { "numStatements": "3", "numTargetRowsDeleted": "150", "numTargetRowsUpdated": "45", "numTargetRowsInserted": "1" } } } ``` This feature is particularly valuable for maintaining data consistency in complex ETL workflows, enforcing business rules across multiple tables, and ensuring atomic schema migrations. ### Conflict Detection and Resolution Delta Lake categorizes conflicts into two types: **1. Blind Append Conflicts** - Two writers adding new data with no dependency on existing rows. These typically proceed without conflict because writers are adding different data. Example of compatible blind appends: ![Example of compatible blind appends](https://www.conduktor.io/assets/images/glossary/delta-lake-transaction-log-how-it-works-2.webp) **2. Read-Modify-Write Conflicts** - Operations that depend on reading and filtering existing data. These must detect conflicts because concurrent changes might affect which rows match the predicate. Example of read-modify-write conflict: ``` Initial state (v4): Table contains 100 inactive customers Writer A: UPDATE customers SET status = 'archived' WHERE status = 'inactive' Writer B: DELETE FROM customers WHERE status = 'inactive' Both operations: 1. Read v4 and identify the same 100 rows 2. Write new data files 3. Attempt to commit Outcome: - Writer A commits v5 successfully - Writer B's commit fails: "The files you read (inactive customers) were modified by v5" - Writer B must retry: re-read v5, discover A already archived the rows, adjust DELETE predicate accordingly ``` For read-modify-write operations, Delta Lake checks if the files read during the operation were modified by concurrent transactions. If so, the transaction fails and must retry with the updated state. The retry logic typically succeeds after re-reading and re-evaluating against the new table version. ```json // Example transaction log entry (simplified) { "commitInfo": { "timestamp": 1735689600000, "operation": "MERGE", "operationMetrics": { "numTargetRowsUpdated": "1250", "numTargetRowsInserted": "340" } }, "add": { "path": "part-00005-abc123.snappy.parquet", "partitionValues": {"date": "2025-01-01"}, "size": 524288, "modificationTime": 1735689600000, "dataChange": true, "stats": "{\"numRecords\": 1590, \"minValues\": {...}, \"maxValues\": {...}}" }, "remove": { "path": "part-00001-xyz789.snappy.parquet", "deletionTimestamp": 1735689600000 } } ``` ## Checkpointing: Managing Transaction Log Growth As tables evolve through hundreds or thousands of commits, reading the entire transaction log becomes inefficient. Delta Lake addresses this through checkpointing. ### How Checkpoints Work Every 10 commits (by default), Delta Lake generates a checkpoint file that represents the complete table state at that version. This checkpoint is a Parquet file containing the same information as the aggregated JSON log entries up to that point. ![delta-lake-transaction-log-how-it-works diagram 4](https://www.conduktor.io/assets/images/glossary/delta-lake-transaction-log-how-it-works-3.webp) Checkpoint files use the naming pattern `00000000000000000010.checkpoint.parquet` and allow readers to: 1. Start from the checkpoint (version 10) 2. Apply only subsequent transactions (11, 12, 13...) 3. Reconstruct the current state efficiently ### What's Inside a Checkpoint? A checkpoint is essentially a Parquet-encoded snapshot of the transaction log state at a specific version. It contains: - **All active data files**: Every `add` action that hasn't been logically deleted by a `remove` action - **Current table metadata**: Schema definition, partition columns, table properties - **Protocol version**: Current reader/writer version requirements and enabled features - **Statistics**: Aggregated file-level statistics for data skipping What checkpoints **don't** include: - Removed files (already filtered out) - Historical `commitInfo` entries (optimization metadata not needed for state reconstruction) - Individual transaction details (only the net result matters) Example: If transactions 0-10 added 50 files and removed 12 files, the checkpoint at version 10 contains only the 38 currently active files, not the full history of all 62 operations. This compression dramatically reduces the amount of data readers must process to understand the current table state, especially for long-lived tables with thousands of commits. For very large tables, Delta Lake can create multi-part checkpoints split across multiple Parquet files, with a JSON manifest coordinating the parts: ![For very large tables, Delta Lake can create multi-part checkpoints split across multiple Parquet files, with a JSON manifest coordinating the parts](https://www.conduktor.io/assets/images/glossary/delta-lake-transaction-log-how-it-works-4.webp) Checkpointing runs automatically during write operations, ensuring readers have recent snapshots even on frequently updated tables. ## Time Travel and Versioning The transaction log's immutable, append-only nature enables time travel queries, allowing you to query historical versions of your data. Each transaction log entry represents a discrete version of the table. You can query historical data using: ```sql -- Query table as of version 42 SELECT * FROM my_table VERSION AS OF 42 -- Query table as of timestamp SELECT * FROM my_table TIMESTAMP AS OF '2025-01-15 10:00:00' ``` ### How Time Travel Queries Execute When you run a time travel query, Delta Lake performs these steps: ![When you run a time travel query, Delta Lake performs these steps](https://www.conduktor.io/assets/images/glossary/delta-lake-transaction-log-how-it-works-5.webp) **Important**: Time travel requires that the Parquet data files still exist on storage. If `VACUUM` has deleted old files, time travel to older versions will fail with "file not found" errors, even though the transaction log entries remain. Delta Lake maps timestamps to versions by reading the `commitInfo.timestamp` fields in the transaction log. For timestamp-based queries, Delta Lake scans the log to find the most recent version committed before the specified timestamp, then executes the version-based query. This feature supports: - **Auditing**: Track data lineage and changes over time - **Reproducibility**: Recreate historical analyses with exact data states - **Rollback**: Revert tables to previous versions after errors - **A/B testing**: Compare results across different table states ### VACUUM and Data Retention Tradeoffs The retention of old data files and log entries is controlled by the `VACUUM` command and retention policies. Understanding the tradeoff between storage costs and time travel capabilities is critical for production deployments. **What VACUUM Does:** ```sql -- Remove files older than 7 days (must be >= 7 days for safety) VACUUM my_table RETAIN 168 HOURS; ``` - **Keeps**: Transaction log JSON entries (history metadata is preserved) - **Deletes**: Actual Parquet data files marked as removed and older than retention period - **Impact**: You can see WHAT changed in the transaction log, but cannot query old data if files are deleted **The Tradeoff:** | Retention Period | Time Travel | Storage Cost | Use Case | |-----------------|-------------|--------------|----------| | 7 days | Recent versions only | Lowest | Cost-sensitive, no compliance requirements | | 30 days (default) | Monthly history | Moderate | Balanced production workloads | | 90+ days | Extended history | Higher | Compliance, long-running experiments | **Example Scenario:** ``` Day 0: Table created, version 0 Day 10: Current version is 100, VACUUM with 7-day retention runs Day 15: Analyst runs: SELECT * FROM table VERSION AS OF 50 Result: ERROR - "File not found: part-00042.parquet" Reason: Version 50 data files were deleted by VACUUM on Day 10 However: Transaction log still shows version 50 metadata You can see: WHAT changed at v50 (commitInfo, operation type) You cannot: Query the actual data from v50 ``` **Best Practices:** 1. **Set retention based on requirements**: Align with your SLAs for data recovery and historical analysis 2. **Run VACUUM regularly**: Prevent unbounded storage growth from accumulating old files 3. **Monitor retention**: Alert when VACUUM falls behind or retention period is too aggressive 4. **Document policies**: Ensure teams understand time travel limitations after VACUUM By default, Delta Lake retains 30 days of history, though this is configurable via `delta.deletedFileRetentionDuration` table property. ## Streaming Integration and Real-Time Processing The transaction log architecture makes Delta Lake particularly well-suited for streaming data pipelines. Streaming frameworks can efficiently tail the transaction log to process new data incrementally. ### Streaming Reads Apache Spark Structured Streaming and other engines can use Delta tables as streaming sources by monitoring the transaction log for new commits. The streaming engine tracks the last processed version and reads only newly added files referenced in subsequent transactions, enabling efficient incremental processing. ### Streaming Writes Streaming writes to Delta tables benefit from **exactly-once semantics** (ensuring each record is processed exactly one time, never duplicated or lost) through **idempotent retries** (operations that can safely be repeated without changing the result). If a streaming batch fails mid-commit, the retry attempts to create the same transaction log entry (with the same version number), which either succeeds or discovers the previous attempt succeeded. This streaming integration extends Delta Lake's value beyond batch processing into real-time architectures, supporting use cases like: - **CDC (Change Data Capture)** ingestion with MERGE operations: capturing row-level changes from operational databases and syncing them to the lakehouse - Real-time analytics dashboards - Event-driven architectures with guaranteed ordering - **Lambda architecture** implementations with unified batch/streaming code: architectures that process data through both batch and real-time paths, converging results in a unified view For organizations managing complex streaming topologies, platforms like **Conduktor** provide governance and visibility across streaming pipelines, complementing Delta Lake's transaction guarantees with operational monitoring, schema validation, and data quality checks. ## Transaction Log Schema and Internals Understanding the transaction log's internal structure helps when debugging, optimizing, or building custom tooling around Delta Lake. ### Delta Kernel: Language-Agnostic Transaction Log Access Introduced in 2023 and maturing in 2025, Delta Kernel is a set of libraries that provide low-level, language-agnostic APIs for reading and writing the Delta transaction log. Unlike the full Delta Lake libraries (which are tightly coupled to Spark), Delta Kernel enables building native Delta Lake integrations in any language. Delta Kernel provides: - **Transaction log parsing**: Read and interpret JSON log entries without Spark dependencies - **Checkpoint reading**: Efficiently process Parquet checkpoint files - **Protocol compliance**: Ensure correct handling of features like deletion vectors, column mapping, and UniForm - **State reconstruction**: Build the current table state from the transaction log This enables native Delta Lake support in: - **Query engines**: Rust-based engines (DataFusion), Go applications, C++ systems - **Data tools**: ETL frameworks, data quality tools, catalog systems - **Streaming platforms**: Real-time processors that need direct Delta access Delta Kernel has become the foundation for expanding Delta Lake's ecosystem beyond the JVM, making it a truly universal lakehouse format in 2025. ### Core Action Types Each transaction log entry contains one or more actions: | Action | Purpose | |--------|---------| | `add` | Register a new data file in the table | | `remove` | Mark a data file as logically deleted | | `metaData` | Define/update schema, partition columns, configuration | | `protocol` | Specify minimum Delta Lake version required | | `commitInfo` | Record operation metadata, metrics, provenance | | `txn` | Support idempotent transactions with application IDs | | `cdc` | Capture change data for Change Data Feed (CDF) | ### Change Data Feed (CDF) Change Data Feed is a Delta Lake feature that tracks row-level changes (inserts, updates, deletes) in the transaction log. When enabled, CDF generates additional files containing change data with metadata indicating the type of operation. ```sql -- Enable Change Data Feed on a table ALTER TABLE user_profiles SET TBLPROPERTIES (delta.enableChangeDataFeed = true); -- Query changes between versions SELECT * FROM table_changes('user_profiles', 10, 20); ``` The transaction log records CDF files using the `cdc` action: ```json { "cdc": { "path": "_change_data/cdc-00001-a3f9b2c8.parquet", "partitionValues": {}, "size": 4096, "dataChange": false } } ``` Each CDF record includes: - The changed row data - `_change_type`: "insert", "update_preimage", "update_postimage", or "delete" - `_commit_version`: Transaction log version where the change occurred - `_commit_timestamp`: Timestamp of the change CDF is essential for: - **CDC pipelines**: Propagating changes downstream to data warehouses or operational systems - **Incremental processing**: Processing only changed data rather than full table scans - **Audit trails**: Tracking exactly what changed, when, and in which version - **Real-time analytics**: Streaming change events to dashboards and alerting systems Change Data Feed has become a mainstream feature in 2025 for building reactive data architectures, complementing streaming platforms like Apache Kafka with guaranteed change capture at the storage layer. ### Statistics and Data Skipping The `add` actions include per-file statistics in the `stats` field: ```json { "add": { "path": "date=2025-01-15/part-00000.parquet", "stats": "{\"numRecords\":10000,\"minValues\":{\"id\":1,\"timestamp\":\"2025-01-15T00:00:00\"},\"maxValues\":{\"id\":10000,\"timestamp\":\"2025-01-15T23:59:59\"}}" } } ``` These statistics enable data skipping: when a query filters on `id = 5000`, Delta Lake reads the transaction log, checks the min/max statistics, and skips files where `5000 < minValues.id` or `5000 > maxValues.id`. This dramatically reduces the amount of data scanned, improving query performance without requiring manual partition management. For example, if you query `WHERE date = '2025-01-15'` on a table with a year of data, Delta Lake checks the min/max statistics and only reads files containing January 15 data, potentially skipping 99% of files in the dataset. In practice, data skipping can reduce query times from minutes to seconds, and I/O costs by 10-100x depending on query selectivity. ### Deletion Vectors: Efficient Row-Level Operations Starting with Delta Lake 2.4 (2023) and increasingly adopted in 2025, deletion vectors provide a performance optimization for UPDATE, DELETE, and MERGE operations that affect only a small percentage of rows in large files. Traditional Delta Lake uses copy-on-write: updating a single row in a 1GB Parquet file requires rewriting the entire file. Deletion vectors change this by storing a bitmap of deleted row positions separately, allowing Delta Lake to mark rows as deleted without file rewrites. ```sql -- Enable deletion vectors for a table ALTER TABLE my_table SET TBLPROPERTIES('delta.enableDeletionVectors' = 'true'); -- Now DELETE and UPDATE operations are much faster DELETE FROM my_table WHERE user_id = 12345; -- Only writes a small deletion vector, not the entire file ``` In the transaction log, deletion vectors appear as a new field in the `add` action: ```json { "add": { "path": "part-00005-abc123.snappy.parquet", "size": 524288, "deletionVector": { "storageType": "u", "pathOrInlineDv": "ab3e7c9f8d2a1b5c", "offset": 1, "sizeInBytes": 40, "cardinality": 327 } } } ``` Deletion vectors dramatically improve write performance for selective updates (10-100x faster) and reduce write amplification, making Delta Lake more efficient for use cases like CDC ingestion, real-time updates, and GDPR deletion requests. For a comprehensive exploration of deletion vectors and their performance characteristics, see [Delta Lake Deletion Vectors: Efficient Row-Level Deletes](https://www.conduktor.io/glossary/delta-lake-deletion-vectors-efficient-row-level-deletes). ### Protocol Evolution The `protocol` action specifies: - `minReaderVersion`: Minimum Delta Lake reader version needed - `minWriterVersion`: Minimum Delta Lake writer version needed This allows Delta Lake to evolve with new features (column mapping, deletion vectors, [liquid clustering](https://www.conduktor.io/glossary/delta-lake-liquid-clustering-modern-partitioning), identity columns) while maintaining compatibility. Older clients that don't support new features will fail fast with clear error messages rather than producing incorrect results. ### UniForm: Universal Format Compatibility Introduced in Delta Lake 3.0 (2024) and widely adopted in 2025, UniForm (Universal Format) extends the transaction log to simultaneously maintain compatibility with Apache Iceberg and Apache Hudi formats. This allows readers that understand Iceberg or Hudi to query Delta tables without conversion. ```sql -- Enable UniForm for multi-format reads CREATE TABLE products ( product_id BIGINT, name STRING, price DECIMAL(10,2) ) USING DELTA TBLPROPERTIES( 'delta.universalFormat.enabledFormats' = 'iceberg' ); ``` When UniForm is enabled, Delta Lake automatically maintains additional metadata alongside the transaction log: - **For Iceberg**: Generates Iceberg metadata files (`metadata/*.json`) and manifest files - **For Hudi**: Maintains Hudi timeline and metadata structures The transaction log remains the authoritative source of truth, but Delta Lake's write operations automatically synchronize changes to the enabled format metadata. This enables: - **Tool interoperability**: Use Iceberg-native query engines (Trino, Dremio) on Delta tables - **Gradual migration**: Transition between table formats without full data rewrites - **Multi-engine analytics**: Different teams using different engines can share the same physical tables The transaction log's `protocol` action includes UniForm configuration, ensuring readers understand which formats are available: ```json { "protocol": { "minReaderVersion": 3, "minWriterVersion": 7, "readerFeatures": ["deletionVectors"], "writerFeatures": ["deletionVectors", "universalFormat"] } } ``` ## Performance Considerations and Best Practices To maximize the transaction log's effectiveness: 1. **Checkpoint frequency**: Adjust `delta.checkpointInterval` based on commit frequency (default: 10) 2. **Statistics collection**: Ensure statistics are computed for filter columns to enable data skipping 3. **Log retention**: Balance history requirements with storage costs using `delta.logRetentionDuration` 4. **Concurrent writes**: Design ETL patterns to minimize read-modify-write conflicts 5. **File size**: Target 128MB-1GB data files to keep transaction log entries manageable For tables with extremely high commit rates (thousands per hour), consider: - Batching smaller writes into larger transactions - Using OPTIMIZE to consolidate small files periodically - Monitoring transaction log size and checkpoint generation latency ## Summary The Delta Lake transaction log is the foundational mechanism that transforms object storage into a reliable lakehouse platform. By maintaining an ordered, immutable log of all table operations, Delta Lake delivers: - **ACID guarantees** through optimistic concurrency control - **Time travel** via version history and timestamp mapping - **Efficient reads** through checkpointing and data skipping statistics - **Streaming integration** with exactly-once semantics and incremental processing - **Schema evolution** with protocol versioning and backward compatibility Understanding the transaction log helps data engineers design more efficient pipelines, debug production issues, and leverage features like time travel and concurrent writes effectively. As Delta Lake evolves with deletion vectors and liquid clustering, the transaction log remains the core abstraction that makes these capabilities possible. ## Related Concepts - [Introduction to Lakehouse Architecture](https://www.conduktor.io/introduction-to-lakehouse-architecture) - Understanding the lakehouse paradigm that Delta Lake enables - [Apache Iceberg](https://www.conduktor.io/apache-iceberg) - Alternative table format with similar ACID transaction capabilities - [Streaming to Lakehouse Tables](https://www.conduktor.io/streaming-to-lakehouse-tables) - Patterns for writing streaming data to Delta tables with exactly-once semantics ## Sources and References - [Delta Lake Transaction Log Protocol Specification](https://github.com/delta-io/delta/blob/master/PROTOCOL.md) - Official protocol documentation - [Diving Into Delta Lake: Unpacking The Transaction Log](https://www.databricks.com/blog/2019/08/21/diving-into-delta-lake-unpacking-the-transaction-log.html) - Databricks engineering deep dive - [Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores](https://www.vldb.org/pvldb/vol13/p3411-armbrust.pdf) - VLDB 2020 research paper - [Delta Lake GitHub Repository](https://github.com/delta-io/delta) - Open-source implementation and examples - [Apache Spark Structured Streaming + Delta Lake](https://docs.delta.io/latest/delta-streaming.html) - Streaming integration guide Source: https://www.conduktor.io/glossary/delta-lake-transaction-log-how-it-works --- # Disaster Recovery Strategies for Kafka Clusters Disaster recovery (DR) planning is critical for any production Kafka deployment. As organizations increasingly rely on real-time data streaming for mission-critical applications, the ability to recover from failures — hardware malfunctions, datacenter outages, or catastrophic events — is essential for business continuity. A solid disaster recovery strategy for Kafka goes beyond basic replication. It requires understanding recovery time objectives (RTO), recovery point objectives (RPO), and the trade-offs between consistency, availability, and cost. ## Understanding Kafka's Built-in Replication Before implementing complex DR strategies, it's important to understand Kafka's native replication capabilities, which form the foundation of any resilience plan. Kafka replicates data across multiple brokers within a cluster. Each partition has one leader and multiple follower replicas that maintain in-sync replicas (ISR), the set of replicas that have caught up with the leader's log. The replication factor determines how many copies of the data exist. For production workloads, a replication factor of at least 3 is recommended. Two critical configurations impact data durability: - `min.insync.replicas`: The minimum number of replicas that must acknowledge a write for it to be considered successful. Setting this to 2 (with a replication factor of 3) provides strong durability guarantees while tolerating one broker failure. - `acks=all`: Producer configuration ensuring messages are acknowledged only after being written to all in-sync replicas. | Configuration | Durability | Availability | Use Case | |--------------|------------|--------------|----------| | `replication.factor=3`, `min.insync.replicas=1` | Low | High | Non-critical logs, metrics | | `replication.factor=3`, `min.insync.replicas=2` | High | Medium | Most production workloads | | `replication.factor=3`, `min.insync.replicas=3` | Very High | Low | Critical financial transactions | For example, a financial services company processing payment transactions might configure topics with `replication.factor=3` and `min.insync.replicas=2`, ensuring that even if one broker fails, data remains available and no transactions are lost. Producer applications should also be configured for durability: ```java Properties props = new Properties(); props.put("bootstrap.servers", "kafka-cluster:9092"); props.put("acks", "all"); props.put("retries", Integer.MAX_VALUE); props.put("max.in.flight.requests.per.connection", 5); props.put("enable.idempotence", true); props.put("compression.type", "zstd"); ``` This configuration ensures messages aren't lost during network failures or broker restarts, with idempotence preventing duplicates during retries. However, single-cluster replication only protects against individual broker failures, not datacenter-wide disasters. ## Multi-Datacenter Replication Patterns To protect against datacenter failures, organizations implement multi-datacenter replication using several architectural patterns. The choice depends on recovery objectives, budget, and operational complexity tolerance. ![disaster-recovery-strategies-for-kafka-clusters diagram 1](https://www.conduktor.io/assets/images/glossary/disaster-recovery-strategies-for-kafka-clusters-0.webp) | Pattern | RTO | RPO | Complexity | Cost Efficiency | Best For | |---------|-----|-----|------------|-----------------|----------| | Active-Passive | Minutes-Hours | Seconds-Minutes | Low | Medium (50% idle) | Most enterprises | | Active-Active | Near-Zero | Seconds | High | High (100% utilized) | Global services | | Stretch Cluster | Zero (automatic) | Zero | Medium | High | Single-region HA | ### Active-Passive (Warm Standby) In this pattern, one cluster (active) handles all production traffic while a secondary cluster (passive) receives replicated data but doesn't serve client requests. During a disaster, applications failover to the passive cluster. MirrorMaker 2 (MM2) is the recommended tool for cross-cluster replication. It replicates topics, consumer groups, ACLs, and configurations between clusters while preserving message ordering and offsets through checkpoint synchronization. ```properties # mm2.properties - Active-Passive Configuration clusters = primary, secondary primary.bootstrap.servers = primary-kafka:9092 secondary.bootstrap.servers = secondary-kafka:9092 # Enable one-way replication primary->secondary.enabled = true primary->secondary.topics = orders.*, inventory.*, customers.* # Preserve consumer group offsets for seamless failover sync.group.offsets.enabled = true sync.group.offsets.interval.seconds = 60 emit.checkpoints.enabled = true emit.checkpoints.interval.seconds = 60 # Replicate topic configurations sync.topic.configs.enabled = true sync.topic.acls.enabled = true # Ensure durability replication.factor = 3 checkpoints.topic.replication.factor = 3 offset-syncs.topic.replication.factor = 3 ``` This approach provides simpler operational management and clear failover procedures, but results in unused infrastructure capacity during normal operations. The secondary cluster typically runs with 50-70% capacity reserved for failover scenarios. ### Active-Active (Multi-Active) In active-active deployments, multiple clusters in different regions simultaneously serve production traffic. Applications write to their nearest cluster, and data is replicated bidirectionally. This pattern minimizes latency for globally distributed users and eliminates unused capacity. However, it introduces complexity around conflict resolution and requires careful coordination of schema changes and topic configurations. Applications must implement idempotent processing to handle potential duplicate messages from bidirectional replication. An e-commerce platform with users across North America and Europe might deploy active-active Kafka clusters in both regions, allowing each region to process orders locally while keeping product catalogs synchronized across regions. Order events flow locally with sub-100ms latency, while cross-region catalog updates can tolerate 1-2 second replication lag. ### Stretch Clusters Some organizations deploy single Kafka clusters across multiple availability zones or nearby datacenters. Brokers are distributed across locations, and rack awareness (configured via `broker.rack` property) ensures replicas are placed in different zones to maximize resilience. This provides automatic failover without application changes but requires low-latency (<10ms), high-bandwidth (10Gbps+) connectivity between locations and doesn't protect against region-wide failures. ### KRaft Mode Considerations Modern Kafka deployments use KRaft (Kafka Raft metadata mode) instead of ZooKeeper, which simplifies DR operations. In KRaft mode, metadata is stored in Kafka itself, eliminating the need to replicate and backup a separate ZooKeeper ensemble. For multi-datacenter KRaft deployments: - Controller quorum nodes should be distributed across datacenters - Use an odd number of controllers (typically 3 or 5) for quorum decisions - Plan controller placement to survive datacenter failures while maintaining quorum KRaft's unified architecture reduces DR complexity and provides faster controller failover (typically under 1 second) compared to ZooKeeper-based deployments. ## Backup and Recovery Mechanisms While replication provides high availability, backups offer protection against logical failures like accidental topic deletion, application bugs corrupting data, or security incidents. Modern Kafka deployments leverage tiered storage for efficient backup strategies. ### Tiered Storage for Long-Term Backup Kafka 3.6+ includes production-ready tiered storage (KIP-405), which automatically offloads older log segments to object storage while keeping recent data on local disks. This provides cost-effective, long-term data retention for DR scenarios. ```properties # server.properties - Enable Tiered Storage remote.log.storage.system.enable=true remote.log.manager.task.interval.ms=30000 # S3 configuration remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.RemoteLogManager rsm.config.remote.storage.type=s3 rsm.config.s3.bucket.name=kafka-dr-backup rsm.config.s3.region=us-east-1 # Define retention for local vs remote storage log.retention.hours=168 # 7 days local log.remote.retention.hours=8760 # 1 year remote ``` With tiered storage, organizations can maintain years of historical data for compliance and disaster recovery without expensive local storage, while still supporting topic recovery from any point in the retention window. ### Topic Snapshots and Export For critical topics requiring independent backups beyond tiered storage, organizations export data to object storage using tools like: - **Kafka Connect S3 Sink Connector**: Continuously exports topic data to S3/GCS/Azure Blob Storage - **Custom consumers**: Purpose-built applications that snapshot specific topics periodically ### Consumer Offset Preservation DR plans must account for consumer offsets to resume processing at the correct position after failover. MirrorMaker 2 automatically replicates consumer group offsets between clusters using offset translation, but organizations should also: - Periodically backup the `__consumer_offsets` topic using tiered storage or snapshots - Maintain offset mappings in external databases for critical consumer groups - Document offset reset procedures for scenarios where offset synchronization fails ### State Store Backups for Kafka Streams Applications using Kafka Streams maintain local state stores (RocksDB-backed key-value stores). DR strategies should include: - **Changelog topics**: Ensure changelog topics have adequate replication factor (3+) and are included in cross-cluster replication - **State store backup**: Configure Kafka Streams to periodically backup state stores to shared storage - **Recovery procedures**: Document how to rebuild state stores from changelog topics after datacenter failover ```java Properties streamsConfig = new Properties(); streamsConfig.put(StreamsConfig.STATE_DIR_CONFIG, "/var/kafka-streams/state"); streamsConfig.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 3); // Ensure changelog topics are replicated for DR streamsConfig.put(StreamsConfig.producerPrefix(ProducerConfig.ACKS_CONFIG), "all"); ``` ## Monitoring and Testing DR Plans A disaster recovery plan is only valuable if it works when needed. Regular testing and continuous monitoring are essential. ### Monitoring Requirements Key metrics to monitor for DR readiness include: | Metric | Warning Threshold | Critical Threshold | Impact | |--------|------------------|-------------------|---------| | MirrorMaker 2 replication lag | >30 seconds | >300 seconds | Increased RPO during failover | | Under-replicated partitions | >0 for 5 minutes | >5% of partitions | Data durability at risk | | Consumer lag | >10,000 messages | >100,000 messages | Extended recovery time | | Broker availability | 80% | >90% | May prevent replication | | Network throughput to DR site | <50% capacity | <20% capacity | Replication lag increases | **Conduktor Platform** provides centralized monitoring across multiple Kafka clusters with: - Real-time replication lag tracking between primary and DR clusters via [monitoring dashboards](https://docs.conduktor.io/guide/monitor-brokers-apps/index) - [Alerting](https://docs.conduktor.io/guide/monitor-brokers-apps/alerts) on under-replicated partitions and unhealthy brokers - Consumer lag monitoring across all consumer groups - Cluster health dashboards for primary and DR environments through [Insights](https://docs.conduktor.io/guide/insights/index) - Configuration drift detection between environments with [risk analysis](https://docs.conduktor.io/guide/insights/risk-analysis) Modern observability platforms integrate with Kafka's JMX metrics and support OpenTelemetry for distributed tracing across the entire streaming pipeline. ### DR Testing Procedures Organizations should regularly conduct failover drills, simulating various failure scenarios: **Quarterly Full Failover Tests** - Controlled failover to secondary datacenter with production-like load - Measure actual RTO (time to restore service) and RPO (data loss) - Validate MirrorMaker 2 offset synchronization for seamless consumer resumption - Test DNS/load balancer reconfiguration for client redirection - Verify application reconnection logic handles cluster endpoint changes **Monthly Partial Tests** - Simulate single broker failures to validate in-sync replica behavior - Test producer and consumer retry mechanisms - Verify monitoring alerts trigger correctly - Practice runbook procedures without full failover **Chaos Engineering with Conduktor Gateway** - **Conduktor Gateway** acts as a proxy layer for Kafka, enabling [controlled fault injection for chaos testing](https://docs.conduktor.io/guide/use-cases/chaos-testing) - Inject network latency between datacenters to test replication lag handling - Simulate broker failures without actual production impact - Test partition leader election scenarios - Validate application behavior under degraded conditions Documenting each test, measuring actual RTO/RPO achieved, and updating runbooks based on lessons learned ensures DR plans remain effective as systems evolve. ## Disaster Recovery in the Data Streaming Ecosystem Kafka rarely operates in isolation. Modern data streaming architectures include producers, consumers, stream processing frameworks (Kafka Streams, Apache Flink), and downstream systems. Effective DR strategies must consider the entire streaming pipeline. If Kafka fails over to a secondary cluster, producers must redirect traffic, stream processing applications must reconfigure cluster endpoints, and consumers must switch to the new cluster without data loss or duplication. ### Coordinating Pipeline Failover Service discovery mechanisms and orchestration tools help coordinate these transitions: **DNS-Based Failover** - Use DNS with low TTL (30-60 seconds) for Kafka bootstrap servers - Update DNS records during failover to point to DR cluster - Applications automatically reconnect to new cluster after TTL expires - Simple but introduces TTL delay and requires proper client reconnection logic **Application-Level Configuration** - Store cluster endpoints in centralized configuration (Consul, etcd, Kubernetes ConfigMaps) - Applications watch for configuration changes and reconnect dynamically - Provides fine-grained control but requires custom reconnection logic **Load Balancer Abstraction** - Place load balancers in front of Kafka clusters - Redirect traffic at network layer during failover - Transparent to applications but adds latency and potential bottleneck ### Stream Processing Considerations **Apache Flink Failover** Flink applications require careful checkpoint and savepoint management for DR: - Take savepoints before planned failovers to preserve exactly-once processing state - Restart Flink jobs with updated Kafka cluster endpoints after failover - Ensure checkpoint storage (S3, HDFS) is accessible from DR environment - Configure Flink to tolerate temporary Kafka unavailability during transitions **Kafka Streams Failover** Kafka Streams applications benefit from MirrorMaker 2's offset synchronization: - Changelog topics must be replicated to DR cluster via MirrorMaker 2 - Applications can resume processing from synchronized offsets after failover - State stores rebuild automatically from changelog topics if local state is lost ### Schema Registry DR Schema registries store message schemas and require their own DR strategy. If using Schema Registry (Confluent Schema Registry, Karapace, or compatible alternatives): - **Karapace** is the recommended open-source Schema Registry implementation - Replicate schema data between primary and DR environments using registry-specific replication - Some organizations use MirrorMaker 2 to replicate the `_schemas` topic between clusters - Maintain schema compatibility rules consistently across environments **Conduktor Platform** provides schema registry governance features including: - Centralized schema management across multiple clusters - Schema compatibility validation before deployment - Configuration drift detection between primary and DR schema registries ## Recovery Objectives and Trade-offs Different business requirements demand different DR strategies. Understanding RTO and RPO helps determine the appropriate approach. ### Recovery Time Objective (RTO) RTO defines how quickly systems must be restored after a failure. Different strategies offer different RTO characteristics: | Strategy | Typical RTO | Automation Level | Factors | |----------|------------|------------------|---------| | Active-Active | <1 minute | Full (automatic routing) | DNS/LB switching, client reconnection | | Active-Passive (Automated) | 5-15 minutes | High (scripted failover) | MM2 lag, DNS propagation, application startup | | Active-Passive (Manual) | 30-120 minutes | Low (runbook-based) | Human coordination, validation steps | | Backup Recovery | 2-24+ hours | Low | Data restoration time, cluster rebuild | ### Recovery Point Objective (RPO) RPO defines the maximum acceptable data loss during a failure: **Within-Cluster RPO (near-zero)** - Synchronous replication using `acks=all` and `min.insync.replicas=2+` - Messages acknowledged only after replication to multiple brokers - Protects against broker failures but not datacenter failures **Cross-Cluster RPO (seconds to minutes)** - MirrorMaker 2 replication is asynchronous by design - Typical replication lag: 1-30 seconds under normal conditions - Degraded network or high throughput can increase lag to minutes - Monitor replication lag continuously to understand actual RPO ### Cost-Benefit Analysis Organizations must balance RTO/RPO requirements against infrastructure costs: **Stock Trading Platform (High-Frequency Trading)** - Requirement: RPO <1 second, RTO <1 minute - Solution: Active-active across regions with synchronous writes - Cost: 2x infrastructure + cross-region bandwidth - Justification: Regulatory compliance, revenue impact of downtime **E-Commerce Order Processing** - Requirement: RPO <30 seconds, RTO <15 minutes - Solution: Active-passive with automated failover - Cost: 1.7x infrastructure (70% DR capacity) - Justification: Balance between availability and cost **Log Aggregation System** - Requirement: RPO <10 minutes, RTO <2 hours - Solution: Periodic backups to object storage, manual recovery - Cost: 1.1x infrastructure (backup storage only) - Justification: Non-critical data, acceptable loss window ## Summary Disaster recovery for Kafka clusters requires a layered approach combining native replication, multi-datacenter architectures, backup mechanisms, and comprehensive testing. - **Foundation**: Single-cluster replication with `replication.factor=3`, `min.insync.replicas=2`, and `acks=all` protects against individual broker failures and provides the baseline for any DR strategy. - **Multi-Datacenter Protection**: Organizations choose between active-passive (simple, cost-effective), active-active (zero-RTO, globally distributed), or stretch clusters (automatic failover within regions) based on RTO/RPO requirements and budget constraints. - **Modern Tooling**: MirrorMaker 2 handles cross-cluster replication with offset synchronization. Tiered storage (Kafka 3.6+) provides cost-effective long-term backup. KRaft mode simplifies DR by eliminating ZooKeeper dependencies. - **Ecosystem Coordination**: Successful DR strategies consider the entire streaming pipeline, producers, consumers, stream processors (Flink, Kafka Streams), schema registries (Karapace), and downstream systems must all be included in failover procedures. - **Monitoring and Testing**: Platforms like Conduktor provide centralized visibility across clusters, while Conduktor Gateway enables chaos engineering for realistic DR testing. Regular failover drills ensure procedures work when disasters actually occur. Matching DR approaches to actual business requirements — RTO, RPO, budget — determines what level of protection makes sense. The patterns here scale from simple backup-and-restore to zero-downtime active-active setups. ## Related Concepts - [Kafka MirrorMaker 2 for Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication) - Deep dive into cross-cluster replication architecture and configuration for multi-region DR strategies. - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Plan DR cluster sizing to handle failover traffic while balancing cost and performance requirements. - [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems) - Validate DR procedures through controlled failure injection and resilience testing. ## Sources and References 1. Apache Kafka Documentation - "Replication" - https://kafka.apache.org/documentation/#replication 2. Apache Kafka Documentation - "MirrorMaker 2 (Geo-Replication)" - https://kafka.apache.org/documentation/#georeplication 3. Apache Kafka Documentation - "KRaft Mode" - https://kafka.apache.org/documentation/#kraft 4. KIP-405: Kafka Tiered Storage - https://cwiki.apache.org/confluence/display/KAFKA/KIP-405 5. Confluent Documentation - "Multi-Datacenter Architectures" - https://docs.confluent.io/platform/current/multi-dc-deployments/index.html 6. Conduktor Platform Documentation - https://www.conduktor.io/ 7. Karapace Schema Registry - https://github.com/aiven/karapace 8. AWS Best Practices - "Disaster Recovery for Apache Kafka" - https://aws.amazon.com/blogs/big-data/ Source: https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters --- # Diskless Kafka: Direct-to-S3 Topics **Diskless Kafka** is a topic type where brokers write incoming records directly to object storage such as Amazon S3 instead of to replicated local disk. Writes are leaderless (any broker can accept any partition), and a coordinator sequences the uploaded data and assigns offsets, moving durability off the brokers entirely. ![diskless-kafka diagram 1](https://www.conduktor.io/assets/images/glossary/diskless-kafka-0.webp) ## What KIP-1150 diskless topics are [KIP-1150](https://kafka-options-explorer.conduktor.io/kip/1150/) introduces **diskless topics** as an opt-in topic type that lives alongside classic topics in the same cluster. The design's own framing is "Diskless is to No-Disks as Serverless is to No-Servers": the disks do not disappear, but they stop being the durability layer for active data. Because it is a per-topic choice, operators can keep latency-sensitive workloads on classic disk-backed topics and move cost-sensitive, high-volume ones to diskless. The mechanics live in follow-up KIPs that were still under discussion as of mid-2026: - **[KIP-1163](https://kafka-options-explorer.conduktor.io/kip/1163/) (Diskless Core)**: the write and read path - **[KIP-1164](https://kafka-options-explorer.conduktor.io/kip/1164/) (Diskless Coordinator)**: the batch coordinator - **KIP-1165 (Object Compaction)**: merging small objects ## How the write path works: leaderless writes and the batch coordinator Classic Kafka assigns each partition a single leader broker; every produce request for that partition routes to the leader, which appends to its local log and replicates to follower brokers. Diskless removes the leader. Any broker can accept a produce request for any partition. That's a big change. **How does it work now?** A broker buffers incoming batches from many partitions, then uploads them together as a single **shared log segment object** (SLSO). Unlike a classic log segment (one partition per file) or a tiered-storage segment (one sealed partition segment), one SLSO holds data from many partitions written in the same time window. This eliminates cross-broker replication for active data; durability is provided by the object store's own redundancy (S3 has 99.999999999% durability and stores every object across at least three AZs, can't beat it). Ordering and offsets move to a "Diskless Coordinator", a new replicated-state-machine component that sequences uploaded batches, assigns the official per-partition offsets, and stores the batch-to-object metadata (for the read path to locate records). A produce acknowledgement returns only after the batch is durable in object storage *and* the coordinator has assigned its offset. That two-step commit, with no local-disk fast path, is where the latency comes from with diskless topics. Because small SLSOs create read amplification (a consumer of one partition would touch many objects), **object compaction agents** run in the background to reorder batches by offset, regroup them by topic-partition, and rewrite them into fewer, larger objects optimized for reads. ![diskless-kafka diagram 2](https://www.conduktor.io/assets/images/glossary/diskless-kafka-1.webp) ## Diskless vs tiered storage (KIP-405) The two features are easy to confuse because both push data to S3. The big difference is that [Tiered storage](https://www.conduktor.io/glossary/tiered-storage-in-kafka) keeps the active write path exactly as it is (records land on a leader's replicated local disk) and only offloads *sealed, inactive* segments to object storage after the fact. Diskless removes local-disk replication for *active* data by writing through to object storage on the produce path itself. | Dimension | Tiered storage ([KIP-405](https://kafka-options-explorer.conduktor.io/kip/405/)) | Diskless (KIP-1150) | |---|---|---| | Status | GA since Kafka 3.9 | Accepted March 2026; not in mainline | | Write path | Leader broker → replicated local disk | Any broker → object storage directly | | Partition leadership | Per-partition leader | Leaderless | | What goes to S3 | Sealed/inactive segments only | Active data, immediately | | Cross-AZ replication | Still on the write path | Eliminated for diskless topics | | Object layout | One partition per remote segment | Multi-partition Shared Log Segment | | Ordering/offsets | Broker log | Batch Coordinator | | Write latency | Unchanged (local disk) | Higher (object-store round trip) | A cluster can run classic, tiered, and diskless topics side by side, each chosen for its retention and latency profile. ## Cost and latency trade-offs The origin of diskless is to reduce cross-AZ replication traffic. Classic Kafka replicates each write across brokers in different AZs for durability, and vendors report that inter-AZ networking can reach 80–88% of cloud Kafka infrastructure cost (as of 2026, AWS and GCP both charge roughly $0.01/GiB per direction, ≈$0.02/GiB round-trip, for inter-zone transfer). Diskless routes durability through the object store, whose replication is internal and not separately billed, which cuts *infrastructure* cost. Diskless adds a different cost: object stores bill per API request, so uploading many small SLSOs generates many PUTs, plus per-GB-month storage. Net savings therefore depend on throughput and object/compaction sizing: a low-volume topic writing tiny objects can see request charges eat much of the replication saving. > On Azure, inter-zone networking has historically been free, so diskless does not reduce the bill there. The savings come almost entirely from removing inter-AZ replication charges. See [cross-AZ traffic in streaming](https://www.conduktor.io/glossary/cross-az-traffic-streaming). The trade-off is latency. A produce that must land in object storage and be sequenced by the coordinator takes anywhere from ~100ms to 1–2s, versus a few milliseconds for classic disk-based topics. Both topic types can coexist in one cluster, this is a per-topic tuning decision, not a cluster-wide commitment. ## The diskless ecosystem KIP-1150 is an effort to standardize patterns vendors are already shipping: - **Aiven Inkless**: the open (AGPLv3) MVP fork implementing KIP-1150. Runs on Kafka 4.x, uses PostgreSQL as the coordinator (or an in-memory mode for testing), and supports S3, GCS, and Azure Blob (MinIO for local testing). - **WarpStream**: stateless agents, no local WAL, direct-to-S3, roughly 400–600ms P99; now part of Confluent/IBM. - **AutoMQ**: an Apache Kafka fork with a pluggable WAL: an S3 WAL (~500ms) or an EBS WAL (sub-10ms) for lower latency. - **Confluent Freight**: Using Kora's "direct write" leaderless mode; Confluent reports up to ~90% cost saving, latency up to 1–2s, no transactions. - **Redpanda**: markets Cloud Topics as a direct-to-object-storage write mode: payloads go to object storage on the produce path and keeps metadata and Raft consensus on local disk. Note that when there is a coordinator in the data path, its availability *is* the data-plane SLA. Inkless depends on a database; WarpStream on a managed metadata plane. Tools like Conduktor add operational visibility across these varied backends, tracking produce/consume latency, offset lag, and topic-level configuration so teams can see how each backend behaves. **Is diskless Kafka available in Apache Kafka today?** No. KIP-1150 was accepted in March 2026 as a directional KIP, but it is not in mainline Apache Kafka. Stock releases have no diskless topic configuration. The implementation KIPs (1163/1164/1165) are still under discussion, and the only running implementation is the Aiven Inkless fork. **How is diskless Kafka different from tiered storage?** Tiered storage (KIP-405) keeps the active write path on replicated local disk and only offloads sealed segments to S3. Diskless removes disk replication for active data by writing directly to object storage on the produce path. They are complementary and can coexist per topic. **What is the batch coordinator?** A new replicated-state-machine component that sequences the batches brokers upload to object storage, assigns official per-partition offsets, and stores batch-to-object metadata for reads. It sits in the data path, so its availability directly affects the topic's SLA. **Why is diskless Kafka cheaper?** It eliminates cross-AZ replication traffic, which vendors report can be 80–88% of cloud Kafka infrastructure cost; durability moves to the object store's internal redundancy instead of inter-broker network transfer. Object-store API request and per-GB storage charges add some cost back, so net savings depend on throughput and object sizing. The saving is largest on AWS and GCP; on Azure, where inter-zone traffic is free, there is little benefit. **What is the latency cost of diskless topics?** Produce-to-consume latency rises to hundreds of milliseconds up to 1–2 seconds, versus sub-100ms for classic disk-based topics, because each write must reach object storage and be sequenced by the coordinator. KIP-1150 lets classic and diskless topics coexist, so latency-sensitive workloads stay on classic topics. ## Related Pages - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka): the KIP-405 cold-tier offload that diskless complements rather than replaces. - [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka): the metadata quorum model that coordinator-based designs build on. - [Cross-AZ Traffic in Streaming](https://www.conduktor.io/glossary/cross-az-traffic-streaming): the inter-zone replication cost that diskless is designed to eliminate. - [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership): how object-storage write paths reshape overall streaming cost. - [Kafka Brokers Explained](https://www.conduktor.io/glossary/kafka-brokers-explained): the leader-based write path that leaderless diskless writes replace. - [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability): the cross-broker replication diskless offloads to the object store. ## Sources - [KIP-1150: Diskless Topics, Apache Kafka wiki](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150:+Diskless+Topics) - [KIP-405: Kafka Tiered Storage, Apache Kafka wiki](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405:+Kafka+Tiered+Storage) - [The Hitchhiker's Guide to Diskless Kafka, Aiven](https://aiven.io/blog/guide-diskless-apache-kafka-kip-1150) - [KIP-1150 Accepted, and the Road Ahead, Aiven](https://aiven.io/blog/kip-1150-accepted-and-the-road-ahead) - [aiven/inkless: Fork of Apache Kafka implementing KIP-1150 (GitHub)](https://github.com/aiven/inkless) - [How KIP-1150 Diskless Topics makes Kafka stateless, 2 Minute Streaming](https://blog.2minutestreaming.com/p/diskless-kafka-topics-kip-1150) - [A Fork in the Road: Deciding Kafka's Diskless Future, Jack Vanlightly](https://jack-vanlightly.com/blog/2025/10/22/a-fork-in-the-road-deciding-kafkas-diskless-future) - [Confluent Cloud Freight Clusters are Generally Available, Confluent](https://www.confluent.io/blog/freight-clusters-are-generally-available/) - [KIP-1150 Diskless Topics Explained, AutoMQ Blog](https://www.automq.com/blog/kip-1150-explained-diskless-topics-kafka-future) Source: https://www.conduktor.io/glossary/diskless-kafka --- # Distributed Tracing for Kafka Applications Monitoring distributed systems is challenging. When a single request spans multiple services, understanding what happened and where things went wrong requires visibility into the entire journey. This is where distributed tracing becomes essential, especially for event-driven architectures built on Apache Kafka. ## Introduction to Distributed Tracing Distributed tracing is an observability technique that tracks requests as they flow through multiple services in a distributed system. Each operation creates a "span" that records timing data, metadata, and relationships to other spans. These spans are collected into a "trace" that represents the complete journey of a request. In traditional synchronous systems, like HTTP-based microservices, tracing is relatively straightforward. A request enters the system, flows through various services via direct API calls, and returns a response. The call chain is explicit and easy to trace. For Kafka-based applications, however, the story is different. Kafka decouples producers from consumers using asynchronous message passing. There are no direct transactions or synchronous call chains to follow. Messages are written to topics, buffered, and consumed independently. This architectural pattern creates implicit dependencies that are difficult to observe without proper instrumentation. ## The Unique Challenges of Tracing Kafka Applications Kafka's architecture introduces several challenges for distributed tracing: - **Asynchronous Processing:** Unlike HTTP requests that have clear request-response patterns, Kafka messages are fire-and-forget. A producer sends a message without waiting for consumers to process it. This breaks the traditional parent-child relationship that tracing systems rely on. - **Temporal Decoupling:** Messages may sit in Kafka topics for seconds, minutes, or even longer before being consumed. The temporal gap between production and consumption makes it difficult to correlate related operations. - **One-to-Many Relationships:** A single message may be consumed by multiple consumer groups, each processing it differently. Tracing must account for these branching paths. - **Message Loss Visibility:** When messages disappear, you need to identify exactly where in the pipeline the loss occurred, during production, storage in Kafka, or consumption. Despite these challenges, distributed tracing for Kafka is feasible and genuinely useful for understanding the health and performance of event-driven systems. ## How Distributed Tracing Works with Kafka The key to tracing Kafka applications is context propagation, passing trace metadata along with each message so that spans can be properly linked across service boundaries. ![distributed-tracing-for-kafka-applications diagram 1](https://www.conduktor.io/assets/images/glossary/distributed-tracing-for-kafka-applications-0.webp) ### Context Propagation via Message Headers Kafka messages support headers, key-value pairs similar to HTTP headers, which have been a core feature since Kafka 0.11 (and are standard in modern Kafka 3.x/4.x deployments). OpenTelemetry and other tracing frameworks use these headers to propagate trace context following the W3C Trace Context specification. When a producer sends a message, the tracing instrumentation automatically injects trace metadata into the message headers. This typically includes: - Trace ID: A unique identifier for the entire trace - Span ID: An identifier for the current span - Parent Span ID: The ID of the span that initiated this operation - Sampling decision: Whether this trace should be recorded When a consumer receives the message, it extracts this context from the headers and creates a new span that's linked to the producer's span. This maintains the trace continuity across the asynchronous boundary. Here's what the trace context headers look like in a real Kafka message: ``` traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01 tracestate: congo=t61rcWkgMzE ``` The `traceparent` header follows the W3C standard and includes the version, trace ID, parent span ID, and sampling decision. This compact representation adds approximately 150-200 bytes to message headers. ### Interceptors and Instrumentation Points Kafka clients support interceptors (introduced via Kafka Improvement Proposal KIP-42), which allow custom code to be executed during key lifecycle events. Tracing frameworks leverage interceptors to: - Create spans when messages are produced - Inject trace context into message headers - Extract trace context when messages are consumed - Create spans for consumption operations - Record timing data, topic names, partition information, and offsets The resulting spans typically have operation names like `kafka.produce` and `kafka.consume`, with attributes defined by OpenTelemetry semantic conventions (v1.24+) that include: - `messaging.system`: Always "kafka" - `messaging.destination.name`: The Kafka topic name - `messaging.kafka.message.key`: The message key - `messaging.kafka.message.offset`: The message offset - `messaging.kafka.destination.partition`: The partition number - `messaging.kafka.consumer.group`: The consumer group (for consumers) These standardized attributes ensure consistent trace visualization across different tools and platforms. This metadata allows you to trace individual messages through your entire system, from the moment they're produced until they're consumed and processed. ## Instrumentation Approaches There are multiple approaches to instrumenting Kafka applications for distributed tracing, ranging from zero-code automatic instrumentation to fine-grained manual control. ### Automatic (Zero-Code) Instrumentation Automatic instrumentation uses agents that attach to your application at runtime and inject tracing code without requiring source code changes. This is particularly effective for Java applications. The agent uses bytecode instrumentation to automatically wrap Kafka client calls and create spans. This approach requires no code changes and works with existing applications, making it ideal for quickly adding observability to legacy systems. **Advantages:** - No code changes required - Works with third-party libraries - Quick to implement **Disadvantages:** - Less control over what's traced - May not capture business-specific context - Potential performance overhead ### Manual Instrumentation Manual instrumentation involves explicitly adding tracing code to your application. This gives you fine-grained control over what's traced and allows you to add custom attributes. **Advantages:** - Full control over spans and attributes - Can add business context - Better performance tuning **Disadvantages:** - Requires code changes - More maintenance overhead - Developer training needed Most organizations start with automatic instrumentation for quick wins, then add manual instrumentation for critical paths where custom context is valuable. ### Example: Manual Instrumentation with OpenTelemetry Here's a practical example of instrumenting a Kafka producer and consumer with OpenTelemetry in Java: ```java // Producer with OpenTelemetry tracing import io.opentelemetry.api.trace.Span; import io.opentelemetry.api.trace.Tracer; import io.opentelemetry.context.Context; import io.opentelemetry.context.propagation.TextMapSetter; import org.apache.kafka.clients.producer.*; import org.apache.kafka.common.header.Headers; public class TracedKafkaProducer { private final KafkaProducer producer; private final Tracer tracer; public TracedKafkaProducer(KafkaProducer producer, Tracer tracer) { this.producer = producer; this.tracer = tracer; } public void send(String topic, String key, String value) { // Create a span for the produce operation Span span = tracer.spanBuilder("kafka.produce") .setSpanKind(SpanKind.PRODUCER) .setAttribute("messaging.system", "kafka") .setAttribute("messaging.destination.name", topic) .startSpan(); try (var scope = span.makeCurrent()) { ProducerRecord record = new ProducerRecord<>(topic, key, value); // Inject trace context into message headers Context current = Context.current(); TextMapSetter setter = (headers, k, v) -> headers.add(k, v.getBytes()); GlobalOpenTelemetry.getPropagators() .getTextMapPropagator() .inject(current, record.headers(), setter); // Send the message producer.send(record, (metadata, exception) -> { if (exception != null) { span.recordException(exception); span.setStatus(StatusCode.ERROR); } else { span.setAttribute("messaging.kafka.message.offset", metadata.offset()); span.setAttribute("messaging.kafka.destination.partition", metadata.partition()); } span.end(); }); } } } // Consumer with OpenTelemetry tracing public class TracedKafkaConsumer { private final KafkaConsumer consumer; private final Tracer tracer; public void poll() { ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord record : records) { // Extract parent context from headers Context parentContext = GlobalOpenTelemetry.getPropagators() .getTextMapPropagator() .extract(Context.current(), record.headers(), (headers, key) -> { Header header = headers.lastHeader(key); return header != null ? new String(header.value()) : null; }); // Create child span linked to producer span Span span = tracer.spanBuilder("kafka.consume") .setParent(parentContext) .setSpanKind(SpanKind.CONSUMER) .setAttribute("messaging.system", "kafka") .setAttribute("messaging.destination.name", record.topic()) .setAttribute("messaging.kafka.message.offset", record.offset()) .setAttribute("messaging.kafka.destination.partition", record.partition()) .startSpan(); try (var scope = span.makeCurrent()) { // Process the message processMessage(record); span.setStatus(StatusCode.OK); } catch (Exception e) { span.recordException(e); span.setStatus(StatusCode.ERROR); } finally { span.end(); } } } private void processMessage(ConsumerRecord record) { // Your business logic here } } ``` This example demonstrates key concepts: - Creating producer and consumer spans with proper span kinds - Injecting trace context into Kafka message headers - Extracting parent context to link consumer spans to producer spans - Adding OpenTelemetry semantic convention attributes - Recording errors and exceptions in spans For Python, the approach is similar using the `opentelemetry-instrumentation-kafka-python` library, which provides automatic instrumentation: ```python from opentelemetry import trace from opentelemetry.instrumentation.kafka import KafkaInstrumentor from kafka import KafkaProducer, KafkaConsumer # Automatically instrument Kafka clients KafkaInstrumentor().instrument() # Now all Kafka operations are automatically traced producer = KafkaProducer(bootstrap_servers='localhost:9092') producer.send('my-topic', b'message') consumer = KafkaConsumer('my-topic', bootstrap_servers='localhost:9092') for message in consumer: # Consumer operations are automatically traced process_message(message.value) ``` ### eBPF-Based Tracing (2025 Approach) A newer approach gaining adoption in 2025 is eBPF (Extended Berkeley Packet Filter) based tracing, which provides observability with near-zero overhead. Tools like Pixie and Odigos use eBPF to automatically capture Kafka operations at the kernel level without requiring application changes or agents. **Advantages:** - Truly zero overhead (< 1% CPU impact) - No application restarts required - Works with compiled languages without instrumentation - Captures network-level metrics alongside traces **Disadvantages:** - Requires Linux kernel 4.14+ with eBPF support - Limited to operation-level tracing (no custom business context) - More complex infrastructure setup eBPF-based tracing is particularly valuable for high-throughput Kafka deployments where traditional instrumentation overhead becomes noticeable. ## Visualization and Analysis Tools Once your Kafka applications are instrumented, you need a backend system to collect, store, and visualize traces. ### Jaeger Jaeger is an open-source distributed tracing platform originally developed by Uber and now part of the Cloud Native Computing Foundation (CNCF). It provides: - A collector that receives spans from applications - Storage backends (Elasticsearch, Cassandra, Kafka) - A UI for searching and visualizing traces - Service dependency graphs Jaeger can use Kafka as a buffer between collectors and storage, which is useful for high-volume environments. It also supports Zipkin-compatible instrumentation, making migration easier. ### Zipkin Zipkin, open-sourced by Twitter in 2012, is another popular tracing system. It bundles the collector, storage, API, and UI into a single process, making it simpler to deploy for smaller systems. Zipkin also supports Kafka as a transport mechanism for receiving spans. ### OpenTelemetry Backends OpenTelemetry is vendor-neutral and can export traces using OTLP (OpenTelemetry Protocol) over gRPC or HTTP to multiple backends, including: - **Open-source solutions**: Jaeger, Zipkin, Grafana Tempo, SigNoz, Uptrace - **Commercial observability platforms**: New Relic, Datadog, Honeycomb, Splunk - **Cloud provider services**: AWS X-Ray, Google Cloud Trace, Azure Monitor In 2025, Grafana Tempo has emerged as a popular choice for Kafka tracing due to its cost-effective object storage backend and seamless integration with Grafana dashboards. SigNoz offers an open-source alternative to commercial APM platforms with built-in support for Kafka-specific visualizations. When analyzing Kafka traces, look for: - **End-to-End Latency:** The time from message production to consumption completion. High latency might indicate consumer lag or processing bottlenecks. - **Orphan Spans:** Spans with no parent could indicate message loss or incomplete instrumentation. - **Error Rates:** Traces with error status help identify failing operations. - **Partition Skew:** If certain partitions show consistently higher latency, you may have a hot partition problem. ## Best Practices and Considerations Implementing distributed tracing for Kafka applications requires careful consideration of several factors. ### Payload Size Impact Tracing metadata adds 150-200 bytes to each message's headers. For small messages, this can be significant overhead. If your average message is 500 bytes, tracing adds 30-40% to the payload size. Before enabling tracing in production: - Test in a development environment - Verify your Kafka message size limits can accommodate the additional headers - Monitor storage costs, as larger messages consume more disk space - Consider sampling strategies for high-volume topics ### Sampling Strategies Tracing every message in a high-throughput Kafka system can generate enormous amounts of data and impact performance. Sampling reduces the volume while maintaining visibility. Common strategies: - **Probabilistic sampling:** Trace a fixed percentage of messages (e.g., 1% of all messages) - **Rate limiting:** Trace a maximum number of messages per second regardless of volume - **Priority sampling:** Always trace errors or slow operations, sample normal operations at lower rates - **Head-based sampling:** Decide at the start of a trace (when the producer creates a span) whether to record it. This decision propagates through the trace. Simple but may miss interesting traces that only become problematic later. - **Tail-based sampling:** Decide after seeing the complete trace by buffering spans and analyzing the entire trace before deciding to keep or discard it. For example, keep all traces with errors or latency > 1 second, but sample only 1% of successful fast traces. More accurate and intelligent, but requires more infrastructure (buffering, coordination between collectors) and adds processing latency. ### Environment Variables and Configuration Key OpenTelemetry configuration settings for Kafka applications: - `OTEL_SERVICE_NAME`: Logical name of your service (critical for visualization, e.g., "payment-processor") - `OTEL_TRACES_EXPORTER`: Where to send traces (`jaeger`, `zipkin`, `otlp`) - `OTEL_EXPORTER_OTLP_ENDPOINT`: The collector endpoint (e.g., `http://localhost:4318`) - `OTEL_EXPORTER_OTLP_PROTOCOL`: Protocol to use (`grpc`, `http/protobuf`, `http/json`) - `OTEL_TRACES_SAMPLER`: Sampling strategy (`always_on`, `always_off`, `traceidratio`, `parentbased_traceidratio`) - `OTEL_TRACES_SAMPLER_ARG`: Sampling rate for probabilistic sampling (e.g., `0.01` for 1%) - `OTEL_RESOURCE_ATTRIBUTES`: Additional resource attributes (e.g., `deployment.environment=production,service.version=1.2.3`) Example configuration for a production Kafka consumer: ```bash export OTEL_SERVICE_NAME="order-consumer" export OTEL_TRACES_EXPORTER="otlp" export OTEL_EXPORTER_OTLP_ENDPOINT="https://tempo.example.com:4318" export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" export OTEL_TRACES_SAMPLER="parentbased_traceidratio" export OTEL_TRACES_SAMPLER_ARG="0.05" # 5% sampling export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production,service.version=2.1.0,kafka.consumer.group=order-processors" ``` ### Testing Before Production Always test tracing in non-production environments: 1. Verify trace context propagates correctly between producers and consumers 2. Confirm spans appear in your visualization tool with correct parent-child relationships 3. Measure the performance impact on throughput and latency 4. Check that message size limits aren't exceeded 5. Ensure consumer groups handle traced messages correctly ### Integration with Existing Observability Distributed tracing is most powerful when combined with other observability signals: - **Metrics:** Track message rates, consumer lag, and throughput alongside traces - **Logs:** Correlate log entries with trace IDs for detailed debugging - **Infrastructure Monitoring:** Connect application traces with Kafka broker metrics Platforms like Conduktor provide complementary observability features for Kafka infrastructure, including [real-time monitoring of topics, consumer groups, and cluster health](https://docs.conduktor.io/guide/monitor-brokers-apps/index). While distributed tracing shows you the application-level flow of messages, tools like Conduktor give you visibility into the Kafka layer itself, partition assignments, replication status, and configuration. Conduktor Gateway, a Kafka proxy, is particularly valuable for testing your distributed tracing implementation. It acts as a proxy layer where you can [inject chaos scenarios](https://docs.conduktor.io/guide/use-cases/chaos-testing) (network latency, message loss, broker failures) to verify that your traces accurately capture failures and help you diagnose issues. This allows you to validate that: - Traces correctly identify message loss and where it occurred - Error spans are created when messages fail to produce or consume - Retry logic is properly traced with parent-child relationships - Latency spikes are visible in trace timelines Together, application-level tracing and infrastructure monitoring form a complete observability picture for Kafka systems. ## Modern Considerations for 2025 Several developments in 2025 have enhanced distributed tracing for Kafka applications: ### Kafka 4.x and KRaft Mode With Kafka's move to KRaft (removing ZooKeeper dependency), tracing has become more streamlined. The simplified architecture reduces the number of moving parts to monitor, and modern Kafka clients include better built-in instrumentation hooks. ### Cost Optimization in Cloud Environments Trace storage costs can become significant in cloud environments with high-volume Kafka deployments. Modern strategies include: - **Adaptive sampling**: Dynamically adjust sampling rates based on message volume and error rates - **Trace retention policies**: Keep detailed traces for 7 days, aggregated metrics for 30+ days - **Object storage backends**: Use Grafana Tempo with S3/GCS for cost-effective long-term storage - **Selective attribute collection**: Only capture expensive attributes (message payloads) for sampled traces ### AI/ML Observability Machine learning pipelines increasingly use Kafka for feature pipelines and model training data. Tracing these workflows presents unique challenges: - Training jobs may process millions of messages in batch - Feature transformations create complex trace trees - Model inference traces need correlation with training data lineage OpenTelemetry's semantic conventions are evolving to support ML-specific attributes alongside messaging attributes, enabling end-to-end visibility from raw events through feature engineering to model predictions. ## Summary Distributed tracing solves the observability challenges inherent in Kafka-based applications. While Kafka's asynchronous, decoupled architecture makes traditional monitoring difficult, modern tracing frameworks like OpenTelemetry provide the tools needed to maintain visibility across the entire message lifecycle. Key takeaways: - Distributed tracing tracks messages from producers through Kafka to consumers using trace context propagated in message headers following the W3C Trace Context specification - Multiple instrumentation approaches are available: automatic agents, manual instrumentation, and eBPF-based tracing, each with different trade-offs - Modern backends like Grafana Tempo, Jaeger, and SigNoz provide cost-effective visualization and analysis capabilities - Proper implementation requires attention to OpenTelemetry semantic conventions, sampling strategies, and thorough testing - Combining application-level tracing with infrastructure monitoring (like Conduktor and Conduktor Gateway) creates comprehensive observability - 2025 advances include eBPF-based tracing, improved cloud cost management, and better support for ML/AI workloads As event-driven architectures grow more complex, distributed tracing becomes less optional. With thoughtful implementation, you get the visibility to troubleshoot quickly, optimize performance, and understand how your system actually behaves in production. ## Related Concepts - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Tracking consumer performance with distributed traces - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) - Infrastructure observability alongside application tracing - [Data Freshness Monitoring and SLA Management](https://www.conduktor.io/glossary/data-freshness-monitoring-sla-management) - Using traces to validate latency SLAs ## Related Articles - [Consumer Lag](https://www.conduktor.io/glossary/consumer-lag) - Understanding and monitoring consumer lag, which traces help diagnose - [Backpressure](https://www.conduktor.io/glossary/backpressure) - Managing flow control in streaming systems, visible through trace latencies - [Chaos Engineering](https://www.conduktor.io/glossary/chaos-engineering) - Testing system resilience with tools like Conduktor Gateway - [Message Ordering](https://www.conduktor.io/glossary/message-ordering) - Guaranteeing order in Kafka, which affects trace interpretation ## Sources and References - [OpenTelemetry Semantic Conventions for Messaging](https://opentelemetry.io/docs/specs/semconv/messaging/) - v1.24+ standards - [W3C Trace Context Specification](https://www.w3.org/TR/trace-context/) - Standard for trace context propagation - [Instrumenting Apache Kafka clients with OpenTelemetry](https://opentelemetry.io/blog/2022/instrument-kafka-clients/) - [The Importance of Distributed Tracing for Apache-Kafka-Based Applications](https://www.confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications/) - [Kafka with OpenTelemetry: Distributed Tracing Guide](https://last9.io/blog/kafka-with-opentelemetry/) - [Optimizing Kafka Tracing with OpenTelemetry: Boost Visibility & Performance](https://newrelic.com/blog/how-to-relic/optimizing-kafka-tracing-with-opentelemetry-boost-visibility-performance) - [Tracing Apache Kafka With OpenTelemetry](https://www.instaclustr.com/blog/tracing-apache-kafka-with-opentelemetry/) - [Grafana Tempo Documentation](https://grafana.com/docs/tempo/latest/) - Cost-effective trace storage - [SigNoz Kafka Monitoring](https://signoz.io/docs/instrumentation/kafka/) - Open-source APM for Kafka Source: https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications --- # E-Commerce Streaming Architecture Patterns Modern e-commerce platforms run on real-time data. Customers expect instant inventory updates, personalized recommendations, and consistent experiences across web, mobile, and physical stores. Batch-oriented architectures struggle to keep up, which is why many teams are moving to streaming-first approaches built on [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture) principles. This article covers the core patterns that enable real-time e-commerce operations, the technologies involved, and the practical challenges teams run into. ## Introduction to E-Commerce Streaming Challenges E-commerce platforms generate thousands to millions of events per second: product views, cart additions, purchases, inventory adjustments, pricing changes, and shipping updates. Processing these events in real-time is not merely a technical preference but a business necessity. Consider the classic inventory problem: a popular item is shown as available on the website, but by the time a customer completes checkout, it's out of stock. This creates customer frustration and operational overhead. Batch processing with 15-minute or hourly updates exacerbates this issue. Beyond inventory, e-commerce systems must handle: - Real-time fraud detection during checkout - Dynamic pricing based on demand and competition - Personalized product recommendations that reflect recent browsing behavior - Coordinated fulfillment across warehouses and stores - Near-instant customer notification across email, SMS, and push channels These requirements demand architecture patterns that treat data as continuous streams of events rather than periodic snapshots. ## Core Architecture Patterns for E-Commerce Streaming Several proven patterns form the foundation of streaming e-commerce architectures. ### Event Sourcing Event sourcing stores every state change as an immutable event. Rather than updating a database record for an order status, the system appends events like "OrderPlaced," "PaymentConfirmed," "ItemsShipped," and "OrderDelivered." This pattern provides a complete audit trail and enables temporal queries: "What was the inventory state at 2 PM yesterday?" It also allows rebuilding derived views by replaying events, which proves valuable when introducing new features or fixing bugs. For detailed coverage of implementing event sourcing with Kafka, see [Event Sourcing Patterns with Kafka](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka). ### Command Query Responsibility Segregation (CQRS) CQRS separates write operations (commands) from read operations (queries). Commands modify state and generate events, while queries read from optimized materialized views built from those events. **Example: Product Search in E-Commerce** When a merchant updates product details, the system: 1. **Command side:** Writes the change to a transactional database and publishes a "ProductUpdated" event to Kafka 2. **Event processing:** A stream processor consumes the event and updates multiple read-optimized views: - An Elasticsearch index for full-text search - A Redis cache for fast product detail lookups - A recommendation engine's graph database When customers search for "blue running shoes," the query reads from Elasticsearch, a denormalized view optimized for text search with product attributes, inventory status, and pricing all pre-joined. This query never touches the transactional write database, allowing each side to scale independently. Writes prioritize consistency and durability, while reads optimize for speed and specific access patterns. ### Change Data Capture (CDC) Many e-commerce platforms integrate with existing relational databases. CDC tools capture database changes (inserts, updates, deletes) and publish them as events to a streaming platform like Kafka. This allows legacy order management or ERP systems to participate in real-time architectures without requiring application rewrites. CDC bridges traditional and modern components, enabling gradual migration strategies. For implementation details, see [Log-Based vs Query-Based CDC Comparison](https://www.conduktor.io/glossary/log-based-vs-query-based-cdc-comparison). ## Real-Time Inventory and Order Processing Inventory accuracy directly impacts revenue and customer satisfaction. Streaming architectures enable real-time inventory visibility across all channels. When a customer purchases an item online, the system publishes an "InventoryReserved" event. Warehouse management systems, store point-of-sale systems, and the website all consume this event and update their local views accordingly. This approach, called eventual consistency with event-driven synchronization, balances speed with accuracy. **Example: Multi-Channel Inventory Synchronization** A customer buys the last unit of a product online. Within milliseconds: 1. The checkout service publishes an "OrderCreated" event to Kafka 2. An inventory service consumes this event and publishes "InventoryReserved" 3. The product catalog service updates availability status 4. Store systems receive the update and remove the item from in-store pickup options 5. A notification service alerts warehouse staff to fulfill the order ![e-commerce-streaming-architecture-patterns diagram 1](https://www.conduktor.io/assets/images/glossary/e-commerce-streaming-architecture-patterns-0.webp) This entire flow completes in under a second, preventing overselling and improving operational efficiency. Order orchestration also benefits from streaming patterns. Rather than a monolithic order service, microservices coordinate through events. Payment verification, fraud checks, inventory allocation, and shipping preparation happen in parallel where possible, reducing total order processing time. ### Partitioning Strategy for E-Commerce Events Kafka topics are divided into partitions for parallelism and scalability. E-commerce systems typically partition events by customer ID or order ID to ensure related events maintain ordering. For example, all events for order #12345 must be processed in sequence: OrderCreated → PaymentProcessed → InventoryAllocated → OrderShipped. By partitioning on order_id, Kafka guarantees these events land in the same partition, preserving order. However, different orders process independently across partitions, enabling high throughput. Inventory events often use product_id as the partition key, ensuring all updates for a specific product are processed sequentially while different products update in parallel. For comprehensive guidance on choosing partition keys and strategies, see [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). ### Reliable Event Publishing with Transactional Outbox A critical challenge is ensuring database changes and Kafka events stay synchronized. If an order service writes to its database but fails before publishing the OrderCreated event, the system becomes inconsistent. The **transactional outbox pattern** solves this by writing events to an outbox table within the same database transaction. A separate process (often using CDC) reads the outbox table and publishes events to Kafka. This guarantees that database state changes and event publishing succeed or fail together, maintaining system-wide consistency. For a comprehensive guide to this pattern, see [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing). ### Example: Publishing Order Events with Kafka Here's a simplified example of publishing an order creation event using Kafka's Java producer with exactly-once semantics: ```java Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("transactional.id", "order-producer-1"); props.put("enable.idempotence", "true"); KafkaProducer producer = new KafkaProducer<>(props); producer.initTransactions(); try { producer.beginTransaction(); // Publish order created event ProducerRecord record = new ProducerRecord<>( "orders", orderId, // partition key ensures ordering per order orderEventJson ); producer.send(record); // Publish inventory reserved event ProducerRecord inventoryRecord = new ProducerRecord<>( "inventory", productId, inventoryEventJson ); producer.send(inventoryRecord); producer.commitTransaction(); } catch (Exception e) { producer.abortTransaction(); throw e; } ``` This ensures both events are published atomically, either both succeed or both fail, maintaining consistency across topics. ## Customer Experience and Personalization Streams Real-time personalization requires processing customer behavior as it happens. When a customer views products, these events feed recommendation engines that update suggestions within the same session. Streaming architectures enable: - **Session-based recommendations:** Analyzing current browsing behavior to suggest complementary products - **Real-time customer segmentation:** Updating customer profiles based on purchase patterns and assigning to dynamic segments for targeted promotions - **Dynamic pricing:** Adjusting prices based on demand signals, competitor pricing, and inventory levels - **Abandoned cart recovery:** Triggering immediate follow-up when customers leave items in their cart **Example: Real-Time Recommendation Pipeline** A fashion retailer uses Kafka to stream clickstream data to Apache Flink. Flink maintains stateful aggregations of customer preferences and product affinities. When a customer views a dress, Flink: 1. Updates the customer's style profile based on attributes (color, pattern, price range) 2. Queries similar products from a pre-computed graph 3. Publishes personalized recommendations to a cache 4. The web application retrieves and displays these recommendations in milliseconds This real-time processing increases conversion rates by showing relevant products while customer intent is high. For related use cases, see [Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming) which discusses pattern detection techniques applicable to personalization engines. ## Data Streaming Technologies in E-Commerce Apache Kafka has become the de facto standard for e-commerce event streaming. Its distributed, fault-tolerant design handles high throughput while maintaining event ordering guarantees within partitions. Modern deployments (Kafka 4.0+) use KRaft mode, eliminating ZooKeeper dependencies and simplifying operational complexity. Key Kafka capabilities for e-commerce: - **Durability:** Events persist to disk, enabling replay for recovery or new consumer applications - **Scalability:** Horizontal scaling supports growing transaction volumes - **Exactly-once semantics:** Critical for financial transactions and inventory operations. This guarantees that even with failures or retries, an order payment is processed exactly once, preventing duplicate charges or lost transactions. Kafka achieves this through idempotent producers and transactional writes. For a deep dive, see [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka). - **Consumer groups:** Multiple applications process the same events independently. For details on consumer group behavior, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). Stream processing frameworks complement Kafka: - **Apache Flink** (version 1.18+) excels at stateful stream processing with low latency. E-commerce use cases include real-time analytics, complex event processing for fraud detection, and maintaining materialized views. Flink's SQL API enables business analysts to query streaming data using familiar SQL syntax. For implementation guidance, see [Flink DataStream API: Building Streaming Applications](https://www.conduktor.io/glossary/flink-datastream-api-building-streaming-applications) and [Flink SQL and Table API for Stream Processing](https://www.conduktor.io/glossary/flink-sql-and-table-api-for-stream-processing). - **Kafka Streams** provides a lightweight library for building streaming applications. Its simplicity makes it attractive for teams already invested in the Kafka ecosystem. Use it for straightforward transformations and aggregations where low latency matters more than complex stateful logic. For an introduction, see [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). - **Apache Spark Structured Streaming** bridges batch and streaming paradigms, useful for organizations with existing Spark expertise. Its micro-batching approach trades some latency for throughput efficiency. The choice depends on latency requirements (Flink for sub-second, Spark for few seconds), team skills, stateful processing complexity, and integration with existing infrastructure. ## Monitoring and Governance Challenges Operating streaming architectures at e-commerce scale introduces significant operational challenges. ### Schema Evolution Product catalogs change frequently, new attributes, seasonal categories, promotional fields. Each change potentially impacts dozens of microservices consuming product events. Managing schema evolution without breaking consumers requires discipline and tooling. Schema registries enforce compatibility rules, preventing producers from publishing incompatible changes. Teams must decide between forward compatibility (new consumers can read old data), backward compatibility (old consumers can read new data), or full compatibility (both directions work) based on their deployment practices. For detailed coverage of compatibility modes and best practices, see [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) and [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). Platforms like **Conduktor** provide comprehensive governance capabilities including schema visualization and impact analysis. These tools help teams understand how schema changes ripple through the ecosystem before deployment. This governance becomes critical when dozens of teams independently develop services consuming shared event streams. ### Data Quality and Monitoring Real-time systems fail in real-time. Monitoring must detect issues like: - Unexpected event volume spikes or drops - Consumer lag indicating processing bottlenecks - Schema validation failures - Duplicate or out-of-order events E-commerce platforms require monitoring at multiple levels: infrastructure metrics (broker health, disk usage), application metrics (processing latency, error rates), and business metrics (order completion rates, inventory accuracy). **Conduktor** provides unified monitoring across Kafka clusters, showing consumer lag, throughput metrics, and schema registry health in a single interface. Get started with [Conduktor Monitoring](https://docs.conduktor.io/guide/monitor-brokers-apps/index) to track your e-commerce streaming infrastructure. For chaos engineering and testing data quality policies, **Conduktor Gateway** acts as a proxy that can inject faults, enforce data contracts, and simulate production issues in test environments. Distributed tracing (using OpenTelemetry) helps diagnose issues across microservices. When an order fails, teams need to trace the flow from checkout through payment, inventory, and fulfillment services to identify the failure point. ### Compliance and Data Privacy E-commerce systems process sensitive customer data subject to regulations like GDPR and CCPA. Streaming architectures must implement: - **Data masking:** Removing or encrypting PII in events. See [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams) for practical techniques. - **Retention policies:** Automatically purging events after specified periods - **Access controls:** Ensuring only authorized services consume sensitive topics. For detailed coverage, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). - **Audit trails:** Tracking who accessed what data and when These requirements demand robust governance frameworks with data masking and access control features to enforce these policies consistently across streaming infrastructure. For broader compliance considerations, see [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams). ## Summary E-commerce streaming architecture patterns enable the real-time responsiveness modern customers expect. Event sourcing, CQRS, and CDC provide the foundational patterns, while technologies like Apache Kafka and Flink deliver the required performance and scalability. Successful implementations start with high-value use cases — inventory synchronization, real-time recommendations — and expand from there. Build expertise gradually, and invest in monitoring, governance, and developer tooling as the platform grows. Moving from batch to streaming is a real architectural shift. Development practices, operational procedures, and even team structure need to evolve. The tradeoff: reduced overselling, better personalization, and faster fulfillment for teams willing to make that investment. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/apache-kafka) - Distributed streaming platform for event-driven e-commerce - [Exactly-Once Semantics](https://www.conduktor.io/exactly-once-semantics) - Critical for payment processing and inventory operations - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/data-governance-framework-roles-and-responsibilities) - Govern streaming data across e-commerce teams ## Sources and References - [Confluent - E-Commerce Reference Architecture](https://www.confluent.io/blog/event-driven-architecture-for-ecommerce-with-kafka/) - Detailed exploration of event-driven patterns for retail, including schema design and use cases - Kleppmann, Martin. "Designing Data-Intensive Applications" (O'Reilly Media, 2017) - Chapter 11 covers stream processing fundamentals and architectural patterns applicable to e-commerce - [Shopify Engineering Blog - Scaling to Millions of Transactions](https://shopify.engineering/) - Real-world case studies on handling Black Friday traffic and real-time inventory management - [AWS - Real-Time Analytics and Insights for Retail](https://aws.amazon.com/retail/real-time-analytics/) - Cloud architecture patterns for streaming retail data with managed services - Narkhede, Neha et al. "Kafka: The Definitive Guide" (O'Reilly Media, 2021) - Comprehensive coverage of Kafka architecture, operations, and e-commerce use cases including exactly-once semantics Source: https://www.conduktor.io/glossary/e-commerce-streaming-architecture-patterns --- # Encryption at Rest and In Transit for Kafka Kafka often carries sensitive data — financial transactions, personal records, proprietary business events. Protecting it requires two complementary controls: encryption in transit and encryption at rest. This article covers both mechanisms, how to implement them in modern Kafka deployments, and how they fit into broader security architectures covering authentication, authorization, and compliance. ## Understanding Encryption in Transit Encryption in transit protects data as it moves between different components of your Kafka cluster. In a typical Kafka deployment, data flows between producers, brokers, consumers, and administrative tools. Without encryption, this data travels as plaintext across the network, making it vulnerable to interception through packet sniffing or man-in-the-middle attacks. ![encryption-at-rest-and-in-transit-for-kafka diagram 1](https://www.conduktor.io/assets/images/glossary/encryption-at-rest-and-in-transit-for-kafka-0.webp) Kafka implements encryption in transit using Transport Layer Security (TLS), formerly known as SSL. When TLS is enabled, all communication between clients and brokers becomes encrypted, ensuring that even if network traffic is intercepted, the data remains unreadable to unauthorized parties. TLS provides three key security benefits: 1. **Confidentiality**: Encrypts data to prevent eavesdropping during transmission 2. **Integrity**: Detects any tampering or modification of data in transit 3. **Authentication**: When combined with certificate verification, confirms the identity of communicating parties Modern Kafka deployments should use TLS 1.3, which offers significant advantages over TLS 1.2. TLS 1.3 reduces handshake latency by 33%, eliminates deprecated cipher suites (including vulnerable algorithms like RC4 and 3DES), provides improved forward secrecy, and delivers better performance on modern CPUs with hardware acceleration. For example, a financial services company streaming credit card transactions through Kafka would use TLS 1.3 encryption to ensure that transaction data cannot be captured and read by attackers monitoring network traffic between their payment processing microservices. The reduced handshake latency of TLS 1.3 also minimizes the performance impact on high-frequency trading systems where every millisecond matters. Organizations planning for long-term security should also be aware of post-quantum cryptography developments. While not yet standard in Kafka, NIST's post-quantum algorithms (CRYSTALS-Kyber for key establishment and CRYSTALS-Dilithium for signatures) are being integrated into TLS 1.3 implementations to protect against future quantum computing threats. Forward-thinking security teams are monitoring these developments and preparing migration strategies. ## Understanding Encryption at Rest Encryption at rest protects data stored on disk. In Kafka, this primarily means protecting log segments stored on broker filesystems. Even with encrypted network communication, data written to disk remains vulnerable if storage devices are physically stolen, improperly decommissioned, or accessed by unauthorized users with system-level privileges. Unlike encryption in transit, Kafka does not provide built-in encryption at rest. Instead, organizations typically rely on filesystem-level or disk-level encryption mechanisms provided by the operating system or storage infrastructure. Common approaches include: - **Linux Unified Key Setup (LUKS)**: Provides full-disk encryption for Linux systems, transparently encrypting all data written to the partition - **dm-crypt**: Offers block device encryption at the kernel level, working underneath the filesystem layer - **eCryptfs**: Provides filesystem-level encryption with per-file granularity - **Cloud provider encryption**: AWS EBS encryption, Azure Disk Encryption, and Google Cloud persistent disk encryption offer managed encryption with integrated key management - **Hardware-based encryption**: Self-encrypting drives (SEDs) implement encryption in the drive controller, offering high performance with minimal CPU overhead A healthcare organization handling patient records through Kafka event streams would implement encryption at rest to meet HIPAA (Health Insurance Portability and Accountability Act) compliance requirements, ensuring that even if backup tapes or decommissioned drives fall into the wrong hands, the data remains protected. HIPAA specifically requires encryption of electronic protected health information (ePHI) both in transit and at rest, with documented key management procedures. Modern cloud deployments often use envelope encryption, where data is encrypted with a data encryption key (DEK), and the DEK itself is encrypted with a key encryption key (KEK) managed by a cloud KMS service like AWS KMS, Azure Key Vault, or Google Cloud KMS. This architecture separates data encryption from key management, enabling centralized key rotation and access policies without re-encrypting all stored data. ## Implementing Encryption in Transit for Kafka Configuring TLS encryption in Kafka involves several steps. First, you must generate or obtain SSL certificates for your brokers. Each broker needs a keystore containing its private key and certificate, along with a truststore containing certificates of trusted certificate authorities (CAs). ### Certificate Generation and Management In 2025, manual certificate management is largely obsolete for production Kafka deployments. Modern approaches use: - **cert-manager** (Kubernetes): Automates certificate issuance, renewal, and rotation for Kafka running on Kubernetes - **HashiCorp Vault PKI**: Provides automated certificate generation with short TTLs (time-to-live), reducing blast radius from compromised certificates - **AWS Certificate Manager (ACM) Private CA**: Integrates with AWS infrastructure for managed certificate lifecycle - **Let's Encrypt with ACME protocol**: Offers free, automated certificates for internet-facing Kafka endpoints These tools eliminate the operational burden of tracking certificate expiration dates and performing manual rotations, a common source of production outages. ### Broker Configuration The broker configuration requires enabling SSL on the appropriate listeners. Here's a modern broker configuration: ```properties # Enable SSL listener (TLS 1.3 with KRaft) listeners=SSL://kafka1.example.com:9093 security.inter.broker.protocol=SSL # Keystore configuration (broker's certificate and private key) ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks ssl.keystore.password=keystore-password ssl.key.password=key-password # Truststore configuration (CA certificates for client validation) ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks ssl.truststore.password=truststore-password # Require client authentication for mutual TLS (mTLS) ssl.client.auth=required # TLS 1.3 specific settings (2025 best practice) ssl.enabled.protocols=TLSv1.3,TLSv1.2 ssl.cipher.suites=TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256 # Certificate validation ssl.endpoint.identification.algorithm=https ``` ### Client Configuration Producer and consumer clients must also be configured with appropriate SSL properties: ```properties security.protocol=SSL ssl.truststore.location=/var/private/ssl/client.truststore.jks ssl.truststore.password=truststore-password # For mutual TLS authentication ssl.keystore.location=/var/private/ssl/client.keystore.jks ssl.keystore.password=client-keystore-password ssl.key.password=client-key-password # Enable TLS 1.3 ssl.enabled.protocols=TLSv1.3 ``` ### Simplified Management with Conduktor Platforms like Conduktor simplify the process of connecting to SSL-enabled clusters by handling certificate configuration and validation through their interfaces. Learn how to [encrypt Kafka with Conduktor](https://docs.conduktor.io/guide/use-cases/encrypt-kafka). Conduktor provides: - Visual certificate upload and validation - Automatic hostname verification - Team-based credential sharing without exposing private keys - Certificate expiration monitoring and alerts - Integrated testing to verify SSL connectivity before deploying applications This dramatically reduces the learning curve for teams new to Kafka security and prevents common misconfigurations that lead to connection failures. ## Implementing Encryption at Rest for Kafka The most common approach to encryption at rest is operating system-level disk encryption. On Linux systems, LUKS provides a standard for full-disk encryption. Once configured, this encryption is transparent to Kafka, the broker reads and writes data normally while the OS handles encryption and decryption automatically. ### Cloud Provider Encryption For cloud deployments, enabling provider-managed encryption is often the simplest approach: - **AWS EBS encryption**: Encrypts volumes using AES-256 encryption with keys managed through AWS Key Management Service (KMS). Supports both AWS-managed keys and customer-managed keys (CMK) for enhanced control - **Azure Disk Encryption**: Uses BitLocker (Windows) or dm-crypt (Linux) with keys stored in Azure Key Vault. Integrates with Azure RBAC for access control - **Google Cloud persistent disk encryption**: Automatic encryption of all data at rest with Google-managed keys, or customer-managed encryption keys (CMEK) via Cloud KMS These managed services handle key rotation, logging, and compliance reporting automatically, significantly reducing operational overhead. ### Key Management Best Practices The key consideration with encryption at rest is key management. Encryption keys must be protected and rotated regularly according to security policies: - **Hardware Security Modules (HSMs)**: Provide FIPS 140-2 Level 3 certified protection for encryption keys, required for many compliance frameworks - **Cloud KMS services**: AWS KMS, Azure Key Vault, and Google Cloud KMS offer managed HSM-backed key storage with automated rotation and audit logging - **Key rotation schedules**: Industry best practice recommends rotating encryption keys every 90 days for sensitive data, with automated rotation preventing human error - **Separation of duties**: Key administrators should be separate from data administrators, preventing any single person from accessing both encrypted data and decryption keys Modern zero-trust architectures treat key management as a critical control plane, with comprehensive audit trails showing every key access and usage. ### Application-Level Encryption Some organizations implement application-level encryption, where data is encrypted by producers before being sent to Kafka and decrypted by consumers after retrieval. This approach provides end-to-end protection but requires careful key management and increases application complexity. Benefits of application-level encryption: - Data remains encrypted in Kafka brokers, protecting against broker compromise - Enables selective encryption of specific fields (PII protection) - Supports different encryption keys for different data sensitivity levels Drawbacks include: - Prevents Kafka from compressing encrypted data effectively - Requires key distribution to all producers and authorized consumers - Adds latency for encryption/decryption operations - Complicates troubleshooting (cannot inspect message contents in brokers) Governance platforms like Conduktor can implement encryption and decryption at an intermediate layer, providing a middle ground between application-level and infrastructure-level encryption. This approach centralizes key management and encryption policies while maintaining data utility within Kafka for operations like monitoring and debugging. ## Field-Level Encryption with Conduktor Gateway While infrastructure-level encryption (TLS and disk encryption) protects data in transit and at rest, field-level encryption provides granular protection for specific sensitive fields within messages. Conduktor Gateway, a Kafka proxy, implements field-level encryption through [interceptors](https://docs.conduktor.io/guide/conduktor-concepts/interceptors), enabling organizations to selectively encrypt sensitive data in structured messages (Avro, Protobuf, JSON) without the overhead of encrypting entire payloads. Gateway supports schema-based encryption (where fields tagged in the schema registry are automatically encrypted) and list-based encryption (where fields are specified in interceptor configuration). It integrates with enterprise Key Management Systems including HashiCorp Vault, Azure Key Vault, AWS KMS, and Google Cloud KMS using envelope encryption (DEK/KEK pattern) for secure key management and rotation. Encryption can occur on-produce (data encrypted before entering Kafka) or on-consume (data encrypted before delivery to specific consumers), enabling role-based access where different consumer groups receive different levels of data visibility. For complete encryption implementation guidance, see [Encrypting Kafka with Conduktor](https://docs.conduktor.io/guide/use-cases/encrypt-kafka). ## Performance and Operational Considerations Encryption introduces computational overhead that impacts throughput and latency, though 2025 hardware and software optimizations have significantly reduced these impacts. ### TLS Performance in 2025 Modern TLS 1.3 implementations with current hardware show: - **CPU overhead**: 3-8% on brokers and clients with AVX-512 and AES-NI instruction sets (down from 5-15% in older implementations) - **Handshake latency**: TLS 1.3's 1-RTT handshake adds 1-5ms for new connections, with 0-RTT resumption eliminating this for subsequent connections - **Throughput impact**: Minimal at typical message sizes (>1KB); more noticeable for high-volume scenarios with tiny messages (<100 bytes) - **Connection pooling**: Long-lived Kafka connections amortize handshake costs; proper client configuration reuses connections across thousands of messages Intel and AMD processors with dedicated AES instructions (AES-NI) can encrypt/decrypt at line speed, while ARM processors with cryptography extensions (ARMv8-A) offer similar capabilities. AWS Graviton3 and Graviton4 instances provide exceptional price-performance for encrypted Kafka workloads. ### Encryption at Rest Performance Hardware-accelerated encryption at rest using AES-256 typically adds less than 3% overhead. Self-encrypting drives (SEDs) implement encryption in the drive controller, delivering performance indistinguishable from unencrypted disks while providing transparent protection. Cloud provider managed encryption (EBS, Azure Disk) uses hardware acceleration and adds negligible overhead, typically unmeasurable in Kafka benchmarks. ### Capacity Planning When planning capacity for encrypted Kafka clusters: - Allocate 5-10% additional CPU headroom for encryption overhead - Monitor CPU utilization metrics: `kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec` alongside CPU percentage - Test under realistic load patterns, encryption impact varies with message size and throughput characteristics - Consider hardware with dedicated encryption acceleration for cost-effective scaling ### Certificate and Key Rotation Key rotation procedures should be established and tested regularly: - **Automated certificate rotation**: Use cert-manager, Vault, or cloud-native tools to rotate certificates every 30-90 days - **Zero-downtime rotation**: Kafka supports hot certificate reloading, update keystores without broker restart (requires Kafka 2.5+) - **Pre-expiration monitoring**: Alert 30, 14, and 7 days before certificate expiration to prevent outages - **Encryption key rotation**: Rotate KMS keys annually or per compliance requirements, using envelope encryption to avoid re-encrypting all data Automated rotation eliminates human error, the leading cause of certificate-related outages. Tools like Conduktor monitor certificate expiration across all clusters and alert operations teams well in advance. ### Compliance Requirements Understanding industry-specific compliance requirements helps determine appropriate encryption scope: - **GDPR (General Data Protection Regulation)**: Requires "appropriate technical measures" for personal data protection, strongly recommending encryption for sensitive data - **HIPAA (Health Insurance Portability and Accountability Act)**: Mandates encryption of electronic protected health information (ePHI) in transit and at rest - **PCI DSS (Payment Card Industry Data Security Standard)**: Requires encryption of cardholder data in transit over public networks and encryption of stored data - **SOC 2 Type II**: Expects encryption controls as part of security and confidentiality trust service criteria - **FedRAMP (Federal Risk and Authorization Management Program)**: Mandates FIPS 140-2 validated cryptographic modules for government data These frameworks increasingly expect encryption by default, not as an optional control. Modern security architectures treat encryption as foundational, not exceptional. ## Encryption and the Broader Security Architecture Encryption works best as part of a comprehensive security strategy that includes authentication, authorization, and monitoring: - **Authentication**: Encryption in transit often provides authentication through mutual TLS (mTLS), where both client and server certificates verify identity. Learn more in [Kafka Authentication: SASL, SSL, and OAuth](https://www.conduktor.io/kafka-authentication-sasl-ssl-oauth) and [mTLS for Kafka](https://www.conduktor.io/mtls-for-kafka) - **Authorization**: After authentication, Kafka ACLs determine what encrypted connections can access. See [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/kafka-acls-and-authorization-patterns) - **Comprehensive security**: Encryption is one pillar of Kafka security. Review [Kafka Security Best Practices](https://www.conduktor.io/kafka-security-best-practices) for the complete security picture For data governance and compliance: - **GDPR compliance**: Encryption helps satisfy GDPR's data protection requirements. See [GDPR Compliance for Data Teams](https://www.conduktor.io/gdpr-compliance-for-data-teams) - **PII protection**: Combine encryption with field-level masking and tokenization. Review [PII Detection and Handling in Event Streams](https://www.conduktor.io/pii-detection-and-handling-in-event-streams) ## Summary Encryption at rest and in transit are complementary security measures essential for protecting sensitive data in Apache Kafka deployments. Encryption in transit uses TLS 1.3 to protect data moving between producers, brokers, and consumers, while encryption at rest protects data stored on broker disks using filesystem or disk-level encryption. In 2025, encryption implementation has become significantly more manageable through automation tools like cert-manager, HashiCorp Vault, and cloud-native KMS services. These tools handle certificate lifecycle management, key rotation, and compliance reporting, eliminating the operational burden that historically made encryption challenging. Modern hardware acceleration (AES-NI, AVX-512, ARM cryptographic extensions) reduces encryption overhead to 3-8% for TLS and under 3% for encryption at rest. TLS 1.3 delivers improved performance while eliminating deprecated cipher suites, making it the standard for new deployments. Organizations handling regulated or sensitive data should treat encryption as a baseline, not optional. TLS 1.3, managed disk encryption, automated key rotation, and monitoring together cover the full data lifecycle in a Kafka deployment. Platforms like Conduktor simplify encryption implementation by providing visual certificate management, expiration monitoring, and policy enforcement, reducing complexity for teams implementing security controls across distributed Kafka environments. ## Related Concepts - [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth) - Authentication mechanisms that work with TLS encryption to secure Kafka connections. - [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka) - Mutual TLS authentication using certificates for both encryption and identity verification. - [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) - Comprehensive security strategy that includes encryption as one pillar of defense-in-depth. ## Sources and References 1. **Apache Kafka Documentation - Security**: Official guide covering TLS configuration, encryption, and authentication mechanisms for Kafka: https://kafka.apache.org/documentation/#security 2. **RFC 8446 - The Transport Layer Security (TLS) Protocol Version 1.3**: The official specification for TLS 1.3, detailing protocol improvements and security enhancements: https://datatracker.ietf.org/doc/html/rfc8446 3. **NIST Special Publication 800-111 - Guide to Storage Encryption Technologies**: Comprehensive guidance on encryption at rest, including full-disk encryption and key management: https://csrc.nist.gov/publications/detail/sp/800-111/final 4. **NIST Special Publication 800-52 Rev. 2 - Guidelines for TLS Implementations**: Federal guidelines for secure TLS configuration, cipher suite selection, and certificate validation: https://csrc.nist.gov/publications/detail/sp/800-52/rev-2/final 5. **NIST Post-Quantum Cryptography Project**: Information on quantum-resistant algorithms and migration planning for long-term security: https://csrc.nist.gov/projects/post-quantum-cryptography 6. **OWASP Transport Layer Protection Cheat Sheet**: Best practices for implementing TLS encryption in applications and infrastructure: https://cheatsheetseries.owasp.org/cheatsheets/Transport_Layer_Protection_Cheat_Sheet.html 7. **Cloud Security Alliance - Encryption in Transit Best Practices**: Industry guidance on securing data in motion across cloud and hybrid environments: https://cloudsecurityalliance.org/ Source: https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka --- # Event-Driven Architecture Event-driven architecture (EDA) is a design pattern built around one idea: instead of services calling each other directly, they emit events and let other services react. This gets you loose coupling, independent scalability, and real-time responsiveness — at the cost of some additional operational complexity. This article covers the core principles, common patterns, implementation tradeoffs, and what it takes to operate EDA in production. ## What is Event-Driven Architecture? Event-driven architecture is a design pattern where systems communicate through events, immutable records of state changes or significant occurrences, rather than through direct, synchronous requests. An event is simply a notification that something happened: a change in the state of a system or a significant occurrence that other parts of the system might care about. For example, a user clicking "Place Order" generates an event. Unlike traditional request-response architectures where Service A calls Service B directly and waits for a response, EDA allows Service A to publish the event and immediately continue its work, relying on the event infrastructure to handle delivery. The core components include: - **Event Producers**: Services or applications that detect state changes and publish events. Producers are often unaware of which consumers will react to the event. - **Event Brokers**: The intermediary system responsible for receiving events from producers, persisting them, and routing them to interested consumers. This transport layer is crucial for achieving decoupling. - **Event Consumers**: Services that subscribe to events and react accordingly. Consumers act asynchronously and independently of one another. ![event-driven-architecture diagram 1](https://www.conduktor.io/assets/images/glossary/event-driven-architecture-0.webp) The general data flow: Producer publishes event → Broker ingests and persists event → Consumer(s) retrieve event and react. ## Core Principles: Decoupling, Asynchrony, and Reactivity ### Loose Coupling In EDA, producers and consumers are decoupled, they don't need to know about each other's existence, location, or implementation details. The checkout service publishes "OrderPlaced" events without knowing whether one consumer, ten consumers, or zero consumers are listening. Adding a new consumer requires no code changes to the producer service. This allows services to evolve independently, be deployed separately, and scale based on their own requirements rather than being limited by upstream or downstream dependencies. ### Asynchronous Communication Unlike synchronous API calls that block until receiving a response, event-driven communication is inherently asynchronous. Producers publish events and continue processing without waiting for consumers to react. This prevents cascading failures, if the inventory service is temporarily unavailable, the checkout service continues operating, and events accumulate for processing once the inventory service recovers. ### Reactive Processing and Real-Time Responsiveness Systems react to events in real time rather than polling for changes or operating on fixed schedules. When a payment completes, the fulfillment process begins immediately through event processing rather than waiting for a batch job to detect the status change. This reactivity enables lower latency, more responsive user experiences, and more efficient resource utilization. ## Event-Driven Patterns: Choreography vs Orchestration ### Event Choreography In choreography, services react independently to events without centralized control. When an "OrderPlaced" event occurs, the inventory service reserves stock, the payment service processes payment, the fulfillment service begins shipping, and the notification service sends updates, all independently. No single service orchestrates this workflow. This pattern maximizes decoupling and service autonomy but can make workflows harder to understand and debug. ### Event Orchestration In orchestration, a central workflow engine coordinates the process by sending commands to services and waiting for response events. A saga orchestrator sends commands sequentially: reserve stock, process payment, ship order, waiting for confirmation events between steps. This pattern provides centralized visibility and control, making workflows easier to understand and monitor, but introduces a coordination point that reduces some decoupling benefits. **When to Use Each**: Choose choreography for simple workflows where services have clear, independent responsibilities. Use orchestration for complex, long-running workflows requiring compensation logic, human intervention, or centralized monitoring, such as financial transactions or order fulfillment with multiple failure scenarios. ## Streaming Platforms as the Event Backbone ### Apache Kafka: Beyond Message Queues Apache Kafka has emerged as the dominant platform for event-driven systems due to its unique architecture. Unlike traditional message queues that delete messages after consumption, Kafka is a distributed, durable, and ordered commit log (a persistent, append-only data structure where events are stored sequentially): - **Durability and Replayability**: Kafka persists events for a configurable period (days, weeks, or indefinitely). If a consumer fails, it can resume from where it left off. New services can replay the entire history of events to build their initial state, enabling event sourcing (an architectural pattern where all state changes are stored as a sequence of events), where the event log becomes the system of record. For detailed coverage of this pattern, see [Event Sourcing Patterns with Kafka](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka). - **Ordering Guarantees**: Events with the same partition key (a routing attribute that determines which partition an event goes to, such as order_id or customer_id) are written to the same partition, ensuring they're processed in the order they occurred. This maintains business logic integrity, all events for order #12345 process sequentially, while events for order #67890 process in parallel on different partitions. For partition strategy details, see [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). - **Scalable Distribution**: Multiple independent consumers read the same events at their own pace without interfering. A real-time analytics service and a data warehouse loader both consume "OrderPlaced" events simultaneously, each processing at different speeds. For consumer group mechanics, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). - **Elastic Buffering**: Kafka accommodates high event traffic bursts by acting as a buffer between producers and consumers, preventing overload. Modern Kafka 4.0+ clusters with optimized configurations can handle millions of events per second per broker, with throughput scaling linearly as brokers are added. ### Modern Kafka Architecture: KRaft Mode Since Kafka 4.0 (2024), Apache Kafka has eliminated its dependency on ZooKeeper through KRaft (Kafka Raft), a built-in consensus protocol that simplifies operations and improves performance: - **Simplified Operations**: KRaft removes the need to deploy and maintain a separate ZooKeeper ensemble, reducing operational complexity and infrastructure requirements. Cluster metadata is now managed internally by Kafka controllers. - **Faster Recovery**: Metadata changes propagate more quickly through the Raft consensus protocol, enabling faster leader elections and cluster recovery after failures, reducing downtime from minutes to seconds in some scenarios. - **Better Scalability**: KRaft supports larger clusters (10,000+ partitions) with improved metadata performance. Metadata operations that previously scaled with ZooKeeper's limitations now scale with Kafka's distributed log architecture. - **Production Ready**: KRaft became production-ready in Kafka 3.3 and is now the standard deployment mode in Kafka 4.0+. All new Kafka deployments should use KRaft mode. For migration details, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## Building Event-Driven Systems: Design Considerations ### Event Schema Design Events need well-defined schemas (structured definitions of event format and fields) to ensure producers and consumers agree on structure. Use schema registries to enforce contracts between services and manage schema evolution. A schema registry validates event schemas before allowing producers to publish, preventing malformed events from breaking downstream consumers. Common schema formats include **Avro** (compact binary format with strong typing), **Protobuf** (Google's efficient binary protocol), and **JSON Schema** (human-readable but larger). Avro and Protobuf offer better performance and built-in versioning support. Design events as immutable facts about what happened, including event type and version, timestamp, entity identifiers, and relevant state changes. For detailed schema management patterns, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). ### Handling Eventual Consistency Event-driven systems embrace eventual consistency, state across services converges over time rather than being immediately consistent. Design systems to handle this by using correlation IDs (unique identifiers like order-123-correlation-id) that propagate through all events in a workflow, allowing you to trace the entire chain from OrderPlaced through StockReserved to PaymentCompleted. Implement the Saga Pattern to manage multi-step business transactions through a sequence of local transactions. If one step fails, compensating transactions (rollback actions that undo previous steps) undo previous steps, for example, if payment fails after inventory reservation, a compensation transaction releases the reserved stock. For comprehensive Saga implementation details, see [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions). ### Event Processing Styles - **Event Notification (Simple Reaction)**: A consumer performs an immediate, isolated action based on the received event, for example, sending a confirmation email when OrderPlaced arrives. - **Event Stream Processing (Stateful Computation)**: Consumers use dedicated stream processing engines like Apache Flink (1.19+) or Kafka Streams for complex, continuous, and stateful computations. For example, fraud detection reads PaymentAttempt events, maintains state per user, and blocks attempts after detecting five failures within 60 seconds. For framework comparisons, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink) and [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). ### Ensuring Idempotency Network failures and retries mean events may be delivered multiple times. Design consumers to be idempotent, processing the same event twice produces the same result as processing it once. Common idempotency techniques include: 1. **Event ID Tracking**: Store processed event IDs in a database. Before processing, check if the event ID exists; if so, skip processing. ```python def process_event(event): if redis.exists(f"processed:{event.id}"): return # Already processed # Process the event update_inventory(event.product_id, event.quantity) # Mark as processed (with expiration) redis.setex(f"processed:{event.id}", 86400, "1") ``` 2. **Natural Idempotency**: Design operations that are inherently idempotent. For example, setting `order_status = "SHIPPED"` multiple times has the same effect. 3. **Exactly-Once Semantics**: Modern Kafka (3.0+) provides exactly-once processing guarantees through transactions. When enabled, Kafka ensures each event is processed exactly once, even with retries. For implementation details, see [Exactly-Once Semantics](https://www.conduktor.io/glossary/exactly-once-semantics) and [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). ### Error Handling and Dead Letter Queues Implement retry logic with exponential backoff for transient failures. For events that fail repeatedly, route them to dead letter queues, separate Kafka topics where failed events are stored for manual inspection. This prevents poison messages (malformed or problematic events that cause repeated processing failures) from blocking subsequent events while preserving failed events for investigation. Dead letter queues enable operations teams to inspect failures, fix issues, and replay events once resolved. Modern error handling strategies also include circuit breakers (automatically stopping processing after repeated failures to prevent cascade effects) and alerting on DLQ accumulation. For reliable event publishing from databases, the **Outbox Pattern** ensures events are published atomically with database transactions, preventing data inconsistencies when services crash between database commits and event publishing. See [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing) for implementation details. ## Observability and Governance in Event-Driven Systems As event-driven architectures scale, operational visibility becomes critical. Traditional monitoring focused on individual services misses the cross-service event flows that define business processes. ### Key Governance Challenges - **Tracking Event Flows**: Understanding how events flow through distributed systems, which services produce and consume each event type, processing times, and failure locations, requires sophisticated observability using correlation IDs to reconstruct complete business transaction paths. - **Event Lineage and Dependencies**: Before modifying an event schema, teams need to identify all downstream consumers and assess impact. Without centralized visibility, this requires manually surveying teams across repositories. - **Schema Management**: Schema validation prevents producers from publishing malformed events. When producers attempt new schema versions, registries validate backward compatibility before allowing changes. - **Monitoring Processing Latency**: End-to-end latency spans multiple services and event hops. Teams track metrics like time-in-broker and processing-time-per-event to identify optimization opportunities. - **Maintaining Event Catalogs**: As organizations adopt EDA, event types grow rapidly. Teams need catalogs documenting available events, schemas, business meaning, producing services, and consuming services. ### Modern Governance Tooling Governance platforms like **Conduktor** address these challenges by providing centralized visibility across event-driven systems: - **Topic Discovery**: Track which services produce and consume each event type, with visibility into topic metadata, schemas, and consumer groups across the architecture - **Consumer Lag Monitoring**: Identify bottlenecks and processing delays by tracking how far behind consumers are from the latest events - **Event Catalog Management**: Maintain searchable catalogs with schema documentation, ownership information, and business context. Teams can explore and manage [Kafka topics](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) through a unified interface - **Security Policy Enforcement**: Apply access controls, data masking, and audit logging centrally across all event streams - **Distributed Tracing**: Trace event flows through multiple services using correlation IDs to debug complex business processes - **Testing and Chaos Engineering**: [Conduktor Gateway](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/index), a Kafka proxy, enables intercepting, modifying, or failing event flows to test resilience without disrupting production systems These capabilities transform event-driven architectures from difficult-to-observe distributed systems into manageable platforms with full operational visibility. ## Summary Event-driven architecture enables building reactive, loosely coupled systems that scale independently and respond to changes in real time. Modern streaming platforms like Apache Kafka 4.0+ with KRaft have transformed EDA from theoretical pattern into practical infrastructure, providing the durability, ordering guarantees, and scalability needed for production systems. Getting EDA right requires attention across several dimensions: choosing the right coordination pattern (choreography vs orchestration), designing schemas with versioning in mind, building idempotent consumers, handling distributed transactions with Sagas, propagating correlation IDs, monitoring consumer lag, and establishing governance practices for schema enforcement and access control. As systems grow, governance tends to become as demanding as the technical implementation. Platforms like Conduktor provide centralized visibility, testing capabilities, and operational controls for running event-driven architectures at scale. For related patterns, see [Event-Driven Microservices Architecture](https://www.conduktor.io/glossary/event-driven-microservices-architecture) and [Event Stream Fundamentals](https://www.conduktor.io/glossary/event-stream-fundamentals). ## Related Concepts - [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram) - Understanding Kafka's foundational components for event-driven systems - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing event schema evolution and validation - [Event-Driven Microservices Architecture](https://www.conduktor.io/glossary/event-driven-microservices-architecture) - Applying event-driven patterns to microservices architectures ## Sources and References - Fowler, Martin. [What do you mean by 'Event-Driven'?](https://martinfowler.com/articles/201701-event-driven.html) martinfowler.com, 2017. - Apache Software Foundation. [Apache Kafka Documentation: Introduction and Use Cases](https://kafka.apache.org/intro) - Fowler, Martin. [The Many Meanings of Event-Driven Architecture](https://www.youtube.com/watch?v=STKCRSUsyP0). Goto Conference 2017. Source: https://www.conduktor.io/glossary/event-driven-architecture --- # Event-Driven Microservices Architecture Modern applications require flexibility, scalability, and resilience. As systems grow in complexity, traditional request-response architectures often struggle to meet these demands. Event-driven microservices architecture addresses these challenges by combining the modularity of microservices with the loose coupling and asynchronous communication of event-driven design. ## What is Event-Driven Microservices Architecture? Event-driven microservices architecture is a design pattern where independent services communicate by publishing and consuming events rather than making direct synchronous calls. An event represents a state change or significant occurrence in the system, such as "OrderPlaced" or "PaymentProcessed." In this architecture, services don't need to know about each other directly. Instead, they publish events to an event broker when something noteworthy happens. Other services that care about those events subscribe to them and react accordingly. This decoupling enables services to evolve independently and reduces cascading failures when one service becomes unavailable. The key distinction from traditional microservices is the communication pattern. Instead of Service A calling Service B's API directly, Service A publishes an event that Service B listens for. This shift from "tell me" to "I'll tell you when something happens" fundamentally changes how systems are designed. ## Core Components and Patterns ### Event Broker The event broker is the central component of an event-driven architecture. It receives events from producers and delivers them to consumers. Popular event brokers include Apache Kafka, RabbitMQ, and Amazon EventBridge. The broker provides persistence, ordering guarantees, and the ability to replay events. ### Producers and Consumers Producers are services that publish events when their internal state changes. Consumers subscribe to specific event types and execute business logic in response. A single event may have multiple consumers, each handling it differently based on their responsibilities. ### Common Patterns - **Event Sourcing** stores every state change as an event, creating an immutable audit log. Instead of storing current state, you store the sequence of events that led to that state. This enables time travel debugging and complete system reconstruction. - **Command Query Responsibility Segregation (CQRS)** separates write operations (commands) from read operations (queries). Events update specialized read models optimized for specific queries, improving performance and scalability. For detailed implementation patterns combining these approaches, see [CQRS and Event Sourcing with Kafka](https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka). - **Saga Pattern** manages distributed transactions across multiple services. Instead of a two-phase commit, each service performs its local transaction and publishes an event. If something fails, compensating events undo previous operations. For a comprehensive guide to implementing sagas, see [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions). - **Outbox Pattern** ensures reliable event publishing by writing events to a database table in the same transaction as business data, then asynchronously publishing them to the event broker. This prevents lost events when services crash after updating data but before publishing. For implementation details, see [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing). ## Benefits and Trade-offs ### Benefits - **Loose Coupling**: Services don't need to know about downstream consumers. You can add new services that react to existing events without modifying producers. - **Scalability**: Services can be scaled independently based on their event processing needs. If one consumer can't keep up, you can add more instances without affecting other parts of the system. - **Resilience**: When a service goes down, events are retained by the broker. Once the service recovers, it can process missed events. This prevents data loss and enables self-healing systems. - **Flexibility**: New features can be built by subscribing to existing events. This accelerates development and reduces the risk of breaking existing functionality. ### Trade-offs - **Increased Complexity**: Debugging distributed systems is harder than monoliths. Understanding how data flows through multiple asynchronous services requires sophisticated monitoring and tracing tools. - **Eventual Consistency**: Unlike synchronous systems where you get immediate confirmation, event-driven systems embrace eventual consistency. Different services may temporarily have different views of the data. - **Message Ordering**: Ensuring events are processed in the correct order across partitions and services requires careful design. Out-of-order processing can lead to incorrect state. ## Event-Driven Architecture and Data Streaming Data streaming platforms like Apache Kafka and Apache Flink are the foundation of modern event-driven microservices. While a simple message queue might work for basic use cases, streaming platforms provide critical capabilities for production systems. Modern Kafka deployments (Kafka 4.0+) use **KRaft mode**, which eliminates the ZooKeeper dependency and simplifies operations. KRaft provides faster metadata propagation, improved scalability, and easier cluster management, making it the recommended architecture for new deployments in 2025. For more details on this architectural change, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ### Event Persistence and Replay Unlike traditional message queues that delete messages after consumption, Kafka retains events for a configured period. This enables new services to replay historical events, recover from failures, and reprocess data with updated logic. You can rebuild entire databases from the event stream if needed. ### Stream Processing Apache Flink and Kafka Streams enable real-time event processing. You can filter, aggregate, join, and transform events as they flow through the system. For example, you might compute running totals, detect patterns, or enrich events with data from other sources. For comprehensive coverage of stream processing frameworks, see [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) and [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). ### Scalability Through Partitioning Kafka partitions events by key, enabling parallel processing across multiple consumer instances. If you partition orders by customer ID, different consumers can independently process different customers' orders, dramatically increasing throughput. To understand how consumer groups enable parallel processing, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). ### Schema Evolution As your system evolves, event structures change. Schema registries enforce compatibility rules and enable consumers to handle multiple event versions gracefully. This prevents breaking changes from cascading through your system. The Schema Registry manages event schemas with compatibility modes (backward, forward, full) that ensure consumers can handle both old and new versions. Avro, Protobuf, and JSON Schema are common serialization formats that enable type safety and schema evolution. For detailed guidance on managing schemas, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). For comparing serialization formats, see [Avro vs Protobuf vs JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema). Tools like Conduktor provide visual monitoring of event flows, schema validation, and data governance capabilities across your streaming infrastructure. Teams can use [self-service capabilities](https://docs.conduktor.io/guide/conduktor-concepts/self-service) to request and manage [Kafka topics](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) with proper governance guardrails, making it easier to track schema evolution and enforce compatibility rules. ## Implementation Challenges ### Exactly-Once Semantics Ensuring each event is processed exactly once is difficult in distributed systems. Network failures can cause duplicate deliveries or lost messages. Kafka provides exactly-once semantics, but implementing it correctly requires understanding idempotent producers and transactional consumers. Example of a Kafka producer with exactly-once semantics (Java): ```java Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer"); props.put("schema.registry.url", "http://localhost:8081"); // Enable idempotence for exactly-once semantics props.put("enable.idempotence", "true"); props.put("acks", "all"); props.put("retries", Integer.MAX_VALUE); // Transactional ID for exactly-once across multiple partitions props.put("transactional.id", "order-service-producer-1"); KafkaProducer producer = new KafkaProducer<>(props); // Initialize transactions producer.initTransactions(); try { producer.beginTransaction(); // Send multiple events atomically producer.send(new ProducerRecord<>("order-events", orderId, orderEvent)); producer.send(new ProducerRecord<>("audit-events", orderId, auditEvent)); producer.commitTransaction(); } catch (Exception e) { producer.abortTransaction(); throw e; } ``` This configuration ensures events are never duplicated and transactions are atomic across topics. ### Event Schema Design Poorly designed event schemas create technical debt. Events should be immutable and self-contained with all necessary context. Avoid referencing external state that might change. Use descriptive names and include metadata like timestamps and correlation IDs for debugging. Here's an example of a well-designed event schema using Avro: ```json { "type": "record", "name": "OrderPlaced", "namespace": "com.example.orders.events", "fields": [ {"name": "eventId", "type": "string"}, {"name": "eventTime", "type": "long", "logicalType": "timestamp-millis"}, {"name": "orderId", "type": "string"}, {"name": "customerId", "type": "string"}, {"name": "items", "type": { "type": "array", "items": { "type": "record", "name": "OrderItem", "fields": [ {"name": "productId", "type": "string"}, {"name": "quantity", "type": "int"}, {"name": "pricePerUnit", "type": "double"} ] } }}, {"name": "totalAmount", "type": "double"}, {"name": "shippingAddress", "type": { "type": "record", "name": "Address", "fields": [ {"name": "street", "type": "string"}, {"name": "city", "type": "string"}, {"name": "postalCode", "type": "string"}, {"name": "country", "type": "string"} ] }}, {"name": "correlationId", "type": "string"} ] } ``` This schema is self-contained with all order details, includes metadata (eventId, eventTime, correlationId), and uses strong typing. Consumers can process this event without additional database lookups. ### Monitoring and Observability Traditional request-response monitoring doesn't work for asynchronous systems. You need distributed tracing to follow events across service boundaries, metrics for lag and throughput, and alerting for processing delays. Understanding end-to-end latency requires correlation IDs that flow through the entire event chain. Modern observability in 2025 uses **OpenTelemetry** for standardized distributed tracing across microservices. OpenTelemetry provides automatic instrumentation for Kafka producers and consumers, enabling you to trace events from producer to consumer with minimal code changes. Key monitoring tools include: - **Kafka Lag Exporter**: Prometheus-based consumer lag monitoring - **Burrow**: LinkedIn's consumer lag checking tool - **Kafdrop**: Web UI for viewing topics, browsing messages, and monitoring consumer groups - **Conduktor**: Enterprise platform for comprehensive monitoring, governance, and testing For production systems, monitor these critical metrics: - **Consumer lag**: Time delay between event production and consumption - **Throughput**: Messages per second produced and consumed - **Error rates**: Failed message processing and dead letter queue size - **End-to-end latency**: Total time from event creation to final processing ### Testing Testing event-driven systems is complex. Unit tests work for individual services, but integration tests must verify event flows across multiple services. Consider contract testing to ensure producers and consumers agree on event formats, and use test event streams to validate behavior without affecting production. Modern testing strategies for 2025 include: - **Chaos Engineering**: Test system resilience by simulating failures. [Conduktor Gateway](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/index), a Kafka proxy, enables chaos testing scenarios like network latency, broker failures, message corruption, and throttling without affecting production infrastructure. - **Contract Testing**: Use tools like Pact to verify producer-consumer agreements. This ensures schema compatibility before deployment. - **Integration Testing**: Use embedded Kafka (Testcontainers) for realistic testing environments. This allows you to test complete event flows without requiring a full cluster. - **Performance Testing**: Validate throughput and latency under load to ensure your architecture meets SLAs before production deployment. ## Real-World Examples ### E-commerce Order Processing ![### E-commerce Order Processing](https://www.conduktor.io/assets/images/glossary/event-driven-microservices-architecture-0.webp) When a customer places an order, the Order Service publishes an "OrderPlaced" event. The Inventory Service consumes this event and reserves stock, then publishes "InventoryReserved." The Payment Service listens for inventory confirmation and processes payment, publishing "PaymentProcessed." Finally, the Shipping Service reacts to successful payment and schedules delivery. If payment fails, a "PaymentFailed" event triggers compensating actions in other services to unreserve inventory and cancel the order. Each service maintains its own database and operates independently, but they coordinate through events to complete the business process. ### Financial Transaction Processing Banks use event sourcing to maintain complete audit trails of every transaction. Each account operation (deposit, withdrawal, transfer) is stored as an event. The current balance is computed by replaying all events for that account. This provides regulatory compliance, enables fraud detection through pattern analysis, and supports disaster recovery by rebuilding state from events. ## Summary Event-driven microservices architecture combines the independence of microservices with the loose coupling of event-driven design. Services communicate through events published to a broker, enabling scalability, resilience, and flexibility. However, this approach introduces complexity around eventual consistency, debugging, and message ordering. Data streaming platforms like Apache Kafka provide the foundation for production event-driven systems, offering persistence, replay, stream processing, and scalability through partitioning. Success requires careful attention to schema design, monitoring, testing, and operational tooling. This architecture excels when you need independent service scaling, complex workflows across multiple systems, or complete audit trails. It's particularly valuable in domains like e-commerce, financial services, and real-time analytics where business processes span multiple bounded contexts. Start small with a few services and core events, then expand as you gain experience with the patterns and operational practices. ## Related Concepts - [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions) - Managing distributed transactions across microservices with compensating actions - [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing) - Ensuring atomic database updates and event publishing - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing event schema evolution across microservices ## Sources and References 1. Fowler, Martin. "What do you mean by 'Event-Driven'?" martinfowler.com, 2017. https://martinfowler.com/articles/201701-event-driven.html 2. Newman, Sam. "Building Microservices: Designing Fine-Grained Systems." O'Reilly Media, 2021. 3. Richardson, Chris. "Pattern: Event-driven architecture." Microservices.io. https://microservices.io/patterns/data/event-driven-architecture.html 4. Apache Kafka Documentation. "KRaft: Apache Kafka Without ZooKeeper." Apache Software Foundation, 2025. https://kafka.apache.org/documentation/#kraft 5. OpenTelemetry. "OpenTelemetry for Apache Kafka." Cloud Native Computing Foundation, 2025. https://opentelemetry.io/ 6. Stopford, Ben. "Designing Event-Driven Systems." 2018. Source: https://www.conduktor.io/glossary/event-driven-microservices-architecture --- # Event Sourcing Patterns with Kafka Event sourcing stores the state of a system as a sequence of events rather than as a current snapshot. When combined with Apache Kafka's distributed log capabilities, it enables scalable, auditable, and resilient systems. This article covers the core patterns, implementation strategies, and practical considerations for building event-sourced systems with Kafka. ## Understanding Event Sourcing Event sourcing treats events as the source of truth for application state. Instead of storing only the current state of an entity in a database, event sourcing persists every state change as an immutable event. The current state can be reconstructed by replaying these events from the beginning. For example, instead of storing a bank account balance as a single value that gets updated, an event-sourced system stores individual `AccountOpened`, `MoneyDeposited`, and `MoneyWithdrawn` events. The current balance is derived by processing all events in sequence. This approach provides several benefits: complete audit trails, temporal queries (viewing state at any point in time), and the ability to rebuild state from scratch. However, it also introduces complexity in managing event schemas, handling event replay, and ensuring performance. ## Why Kafka Excels at Event Sourcing Apache Kafka's architecture aligns naturally with event sourcing principles. Kafka topics function as distributed, durable, append-only logs, exactly what event sourcing requires. Modern Kafka deployments (Kafka 4.0+) use KRaft consensus for metadata management, eliminating ZooKeeper dependencies and simplifying operational complexity for event-sourced systems. Key Kafka features that support event sourcing include: - **Immutability and Durability**: Events written to Kafka are immutable and persisted durably across multiple brokers, ensuring no data loss. - **Replay Capability**: Consumers can replay events from any point in the log by adjusting their offset, enabling state reconstruction and processing corrections. For details on managing consumer offsets and coordination, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). - **Ordering Guarantees**: Kafka maintains event order within a partition, critical for correctly reconstructing state when events must be processed sequentially. - **Scalability**: Kafka's partitioning allows horizontal scaling of event storage and processing across multiple consumers. Unlike traditional databases that optimize for current state queries, Kafka is built for sequential event storage and streaming, making it well-suited for event sourcing. ## Core Event Sourcing Patterns with Kafka ### Event Store Pattern The most fundamental pattern uses Kafka topics as event stores. Each aggregate or entity type has a dedicated topic (e.g., `account-events`, `order-events`). Events are keyed by entity ID to ensure all events for a specific entity land in the same partition, maintaining order. ![event-sourcing-patterns-with-kafka diagram 1](https://www.conduktor.io/assets/images/glossary/event-sourcing-patterns-with-kafka-0.webp) ### Snapshot Pattern Replaying thousands or millions of events to reconstruct state can be slow. The snapshot pattern periodically saves the current state as a snapshot, then only replays events after the snapshot. Kafka's compacted topics are useful here. By periodically publishing a snapshot event with the same key as the aggregate, Kafka's log compaction retains the latest snapshot while eventually removing older events, balancing storage with replay speed. For details on log compaction mechanics, see [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained). ### CQRS Integration Event sourcing often pairs with Command Query Responsibility Segregation (CQRS). Commands write events to Kafka, while read models are built by consuming those events into optimized query stores (databases, search indexes, caches). ![event-sourcing-patterns-with-kafka diagram 2](https://www.conduktor.io/assets/images/glossary/event-sourcing-patterns-with-kafka-1.webp) A typical flow: a command handler validates a command, writes events to Kafka, then separate consumers project those events into read models tailored for specific query patterns. ### Event Replay for Corrections When business logic changes or bugs are fixed, event replay allows reprocessing historical events to correct derived state. Kafka Streams applications can be reset to offset zero to rebuild state tables from scratch. ## Implementation Considerations ### Schema Evolution Events are long-lived, so schema evolution is critical. Using a schema registry with formats like Avro or Protobuf enables backward and forward compatibility. Events must be designed to evolve without breaking existing consumers. For detailed coverage of schema management strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). For example, adding optional fields is safe, but removing required fields or changing types requires careful versioning. Platforms like Conduktor provide schema management capabilities that help teams visualize schema evolution, validate compatibility rules, and prevent breaking changes before they reach production. ### Partitioning Strategy Choosing the right partition key ensures event ordering. Typically, the aggregate ID (customer ID, order ID) serves as the key. However, if multiple aggregates interact, consider whether strict ordering across aggregates is needed, which might require single-partition topics or other coordination mechanisms. For comprehensive guidance on partitioning decisions, see [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). ### Event Versioning As events evolve, versioning strategies become necessary. Common approaches include: - **Upcasting**: Transform old event versions to new formats during consumption - **Versioned event types**: Use distinct event types for different versions (e.g., `OrderCreatedV1`, `OrderCreatedV2`) - **Schema registry validation**: Enforce compatibility checks automatically ### Handling Failures Event sourcing with Kafka requires idempotent consumers. Since events may be replayed, consumers must handle duplicate processing gracefully. Kafka 3.0+ provides mature exactly-once semantics through transactions, enabling atomic writes across multiple partitions and topics. Alternatively, implementing idempotency keys in downstream systems prevents duplicate effects. For comprehensive coverage of idempotency and transactional guarantees, see [Exactly-Once Semantics](https://www.conduktor.io/glossary/exactly-once-semantics). ## Common Challenges and Solutions - **Debugging Complex Event Chains**: Tracing how a sequence of events led to a particular state can be difficult. Platforms like Conduktor provide event flow visualization, filtering, and inspection capabilities that make it easier to understand event sequences and identify issues in event-sourced systems. - **Performance at Scale**: Replaying millions of events is expensive. Combine snapshots, appropriate retention policies, and event compaction. For frequently accessed aggregates, maintain in-memory state caches. - **Eventual Consistency**: Event-sourced systems are eventually consistent. The write side (events) and read side (projections) may be temporarily out of sync. Design UIs and APIs to handle this gracefully, perhaps showing "processing" states or using optimistic updates. - **Schema Governance**: Without proper governance, schema proliferation and breaking changes create chaos. Establishing schema review processes, compatibility policies, and centralized schema management prevents these issues. ## Real-World Applications - **Financial Services**: Banks use event sourcing for transaction logs. Every debit, credit, and fee becomes an immutable event in Kafka. Regulatory compliance benefits from complete audit trails, and account balances can be reconstructed at any historical point for dispute resolution. - **E-commerce Order Management**: An order progresses through states like `OrderPlaced`, `PaymentProcessed`, `OrderShipped`, `OrderDelivered`. Each state transition is an event. Kafka stores this complete order history, enabling analytics, customer service queries, and integration with warehouse systems that consume shipping events. - **IoT and Sensor Data**: IoT systems generate continuous streams of sensor events. Event sourcing captures every measurement, enabling historical analysis, anomaly detection, and model training on complete datasets. ## Related Patterns Event sourcing often works in combination with other architectural patterns: - **[Outbox Pattern](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing)**: Ensures reliable event publishing by storing events alongside application state in a transactional outbox before publishing to Kafka - **[Saga Pattern](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions)**: Manages distributed transactions across event-sourced services using compensating events - **CQRS**: Separates command (write) and query (read) responsibilities, naturally complementing event sourcing's append-only write model ## Summary Event sourcing with Kafka provides a solid foundation for building auditable, scalable, and flexible systems. Kafka's log-based architecture naturally supports the immutability, ordering, and replay requirements of event sourcing. By applying patterns like event stores, snapshots, and CQRS, and by carefully managing schemas, partitioning, and failure handling, teams can build systems that maintain complete history while delivering high performance. Key takeaways: - Use Kafka topics as durable, ordered event stores - Apply snapshots to optimize replay performance - Leverage schema registries for evolution and compatibility - Design for idempotency and eventual consistency - Consider governance and debugging tools for production operations - Use testing tools like Conduktor Gateway, a Kafka proxy, to simulate failures and validate event replay behavior Event sourcing is not right for every system, but when audit trails, temporal queries, or flexible event-driven processing are requirements, Kafka provides the infrastructure to implement it effectively. ## Related Concepts - [CQRS and Event Sourcing with Kafka](https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka) - Combining CQRS with event sourcing for scalable architectures - [Exactly-Once Semantics](https://www.conduktor.io/glossary/exactly-once-semantics) - Ensuring data consistency in event-sourced systems - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing event schema evolution for long-lived event stores ## Sources and References 1. Fowler, Martin. "Event Sourcing." martinfowler.com, 2005. https://martinfowler.com/eaaDev/EventSourcing.html 2. Narkhede, Neha, Gwen Shapira, and Todd Palino. *Kafka: The Definitive Guide*. O'Reilly Media, 2017. 3. Boner, Jonas, et al. "Reactive Microsystems: The Evolution of Microservices at Scale." O'Reilly Media, 2017. 4. Stopford, Ben. "Designing Event-Driven Systems." O'Reilly, 2018. 5. Apache Kafka Documentation. "Event Sourcing and Stream Processing." Apache Software Foundation, 2025. https://kafka.apache.org/documentation/ Source: https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka --- # Event Streams: The Foundation of Real-Time Architectures Event streams are at the core of modern real-time data architectures. Every click on a website, every transaction in a payment system, every sensor reading from an IoT device — these are events flowing through distributed systems. Understanding event streams is essential for building scalable, real-time applications that react to changes as they happen. ![Event stream architecture](https://www.conduktor.io/assets/images/glossary/event-stream-fundamentals-0.webp) ## What is an Event Stream? An event stream is a continuous, ordered sequence of events produced by operational systems. Unlike traditional databases that store the current state, event streams capture *what happened* over time as an immutable log of facts. To understand event streams, it helps to contrast them with other data systems: - **Event Stream vs Database**: A database stores the current state of your system. If a user updates their email address, the old value is typically overwritten. An event stream, however, records *both* events: "EmailChanged from old@example.com" and "EmailChanged to new@example.com". You can reconstruct state at any point in time by replaying events. - **Event Stream vs Message Queue**: Message queues are designed for task distribution and typically delete messages once consumed. Event streams are designed for data distribution and retain events for a configurable period (hours, days, or indefinitely). Multiple consumers can independently read the same events without affecting each other. The key insight is that event streams treat data as a first-class, replayable asset rather than ephemeral messages or mutable records. ## Anatomy of an Event Each event in a stream consists of several components that provide structure and context: - **Key**: An optional identifier that determines how events are partitioned and ordered. Events with the same key are guaranteed to be ordered and stored in the same partition. For example, all events for "user-123" would share the same key. - **Value**: The actual event payload, typically containing business data. This might be a JSON object describing a purchase, an Avro record of a sensor reading, or any serialized data format. - **Timestamp**: When the event occurred. Event streams typically support two timestamp types: the time when the event was created (event time) and when it was written to the stream (ingestion time). - **Headers**: Key-value metadata about the event, such as correlation IDs for tracing, content type information, or application-specific metadata that doesn't belong in the value. For practical guidance on leveraging headers, see [Using Kafka Headers Effectively](https://www.conduktor.io/glossary/using-kafka-headers-effectively). - **Metadata**: System-generated information like partition number, offset (position in the stream), and checksum for integrity verification. This structure allows consumers to process events intelligently, filtering by headers, ordering by timestamp, or grouping by key, without parsing the entire value payload. ## Core Characteristics of Event Streams Event streams have several defining characteristics that distinguish them from other data systems: - **Immutable**: Once an event is written to the stream, it cannot be modified or deleted (except by retention policies). This immutability provides a reliable audit trail and enables time-travel debugging. - **Append-only**: New events are always added to the end of the stream. This simple write pattern enables extremely high throughput, streaming platforms like Kafka can handle millions of events per second on commodity hardware. - **Ordered**: Events within a partition are strictly ordered. This ordering guarantee is critical for maintaining consistency. If a user creates an account and then updates their profile, consumers will always see these events in the correct sequence. - **Partitioned**: Event streams are divided into partitions for parallelism and scalability. Each partition is an independent, ordered sequence. Partitioning allows horizontal scaling: more partitions mean more parallel consumers and higher throughput. For guidance on choosing partition keys and strategies, see [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). These characteristics make event streams ideal for distributed systems where multiple services need access to the same events without coordination or locking. ## Producers, Consumers, and Decoupled Communication Event streams enable a powerful pattern: decoupled, asynchronous communication between services. - **Producers** are applications or services that write events to streams. A producer doesn't know or care who consumes its events. An e-commerce checkout service might produce "OrderPlaced" events without knowing whether they'll be consumed by an inventory service, an email service, or a fraud detection system. - **Consumers** read events from streams independently. Each consumer maintains its own offset (position in the stream), allowing it to: - Process events at its own pace - Replay events from any point in time - Fail and recover without affecting other consumers - Join or leave without coordination For detailed coverage of consumer mechanics and consumer groups, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) and [Kafka Producers and Consumers](https://www.conduktor.io/glossary/kafka-producers-and-consumers). This decoupling provides enormous flexibility. You can add new consumers to react to existing events without modifying producers. If your fraud detection service goes down, it can resume processing from where it left off without losing data. Consider a real-world example: ride-sharing applications. When a ride is completed, the mobile app produces a single "RideCompleted" event containing ride details. This event flows to multiple independent consumers: the billing service calculates charges, the rating service prompts for feedback, the analytics service updates driver performance metrics, and the data warehouse ingests it for reporting. Each service operates independently, and adding new functionality requires only adding a new consumer. ## Event Stream Platforms and Technologies Several platforms implement event streaming with different trade-offs: - **Apache Kafka** is the most widely adopted event streaming platform. Kafka organizes events into topics (logical streams) divided into partitions. It provides strong ordering guarantees, configurable retention, and a rich ecosystem of connectors and stream processing tools. Kafka excels at high throughput and is commonly used for log aggregation, real-time analytics, and event-driven microservices. As of Kafka 4.0 (2024), Kafka runs in KRaft mode by default, removing the ZooKeeper dependency for simpler operations and improved scalability. For detailed information on Kafka's core architecture, see [Kafka Topics, Partitions, and Brokers](https://www.conduktor.io/glossary/kafka-architecture-diagram). - **Apache Pulsar** is a newer alternative that separates compute and storage layers, enabling independent scaling. Pulsar topics support both streaming and queuing semantics, making it versatile for different use cases. Its multi-tenancy features make it attractive for service provider scenarios. - **AWS Kinesis** is a fully managed streaming service in AWS. Kinesis streams provide similar functionality to Kafka topics but with tight integration into the AWS ecosystem. Organizations already on AWS often choose Kinesis for its operational simplicity, though it has lower throughput limits per shard compared to Kafka partitions. - **Azure Event Hubs** and **Google Cloud Pub/Sub** provide similar managed streaming capabilities in their respective cloud ecosystems. While these platforms differ in implementation details, they all provide the core event streaming primitives: durable, ordered, partitioned logs of events with independent consumer offsets. For understanding Kafka's modern architecture without ZooKeeper, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## Architectural Patterns and Use Cases Event streams enable several powerful architectural patterns: - **Event Sourcing** stores all changes to application state as a sequence of events. Instead of storing current account balances, you store "MoneyDeposited" and "MoneyWithdrawn" events. The current balance is computed by replaying events. This provides perfect audit trails and enables time-travel queries: "What was this account's balance on March 15th?" For reliable event publishing patterns, see [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing). - **CQRS (Command Query Responsibility Segregation)** separates write and read models. Commands produce events to the stream, and multiple read models (materialized views) are built by consuming those events. This allows optimizing each read model for specific query patterns without compromising the write model. - **Event-Driven Architecture** builds systems as loosely coupled services that communicate through events. Services react to events rather than making direct API calls, reducing dependencies and improving resilience. For comprehensive coverage of this pattern, see [Event-Driven Microservices Architecture](https://www.conduktor.io/glossary/event-driven-microservices-architecture). Event streams are foundational to modern data architectures: - **Lakehouse architectures** consume event streams from operational systems, landing events in object storage (like S3) for analytics while also serving real-time queries. For detailed coverage, see [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture) and [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse). - **Data Mesh** uses event streams as the interoperability layer between domain-specific data products, enabling decentralized data ownership with centralized discoverability. - **Real-time analytics** processes event streams as they arrive, powering dashboards, alerting, and automated responses with sub-second latency. See [Real-Time Analytics with Streaming Data](https://www.conduktor.io/glossary/real-time-analytics-with-streaming-data) for implementation patterns. Consider a financial services example: a trading platform produces trade execution events to a stream. One consumer updates user portfolios in real-time (CQRS read model), another feeds a risk management system, another lands events in a data lake for regulatory reporting, and a stream processing application detects suspicious trading patterns. All from a single event stream, with each consumer operating independently. ## Schema Evolution, Retention, and Governance As event streams become central to data architectures, managing their evolution and governance becomes critical. - **Schema Management**: Event schemas evolve as businesses change. You might add new fields to capture additional information or deprecate old fields. Schema registries enforce compatibility rules: consumers can read events produced with newer or older schemas without breaking. Common strategies include forward compatibility (new consumers read old events), backward compatibility (old consumers read new events), and full compatibility (both directions). For detailed guidance on schema evolution strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). - **Retention Policies** determine how long events remain in the stream. Short retention (hours or days) suits transient events like application logs. Long retention (months or years) enables reprocessing historical data. Infinite retention is common for event sourcing where events are the source of truth. - **Log Compaction** provides an alternative to time-based retention. In a compacted topic, the stream retains only the latest event for each key. This is perfect for change data capture: you keep the current state of each database row without storing every historical change. For an in-depth explanation of log compaction mechanics, see [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained). - **Governance Challenges** emerge as event streams scale: - **Discovery**: Which streams exist? What events do they contain? Who produces and consumes them? - **Access Control**: Who should be allowed to read or write specific topics? - **Data Quality**: Are events conforming to expected schemas? Are critical fields populated? - **Compliance**: Which streams contain sensitive data requiring encryption or access logging? Modern governance platforms like Conduktor provide centralized visibility across Kafka clusters, showing consumer lag and data flows. These platforms enforce schema validation, manage access controls through policies rather than low-level ACLs, and provide data quality monitoring to detect anomalies in event streams. [Conduktor Gateway](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/index), a Kafka proxy, adds an additional layer for testing scenarios including chaos engineering and protocol validation. For organizations running multiple Kafka clusters across teams, [self-service capabilities](https://docs.conduktor.io/guide/conduktor-concepts/self-service) ensure consistent policies and visibility without requiring every team to become Kafka experts. For more on monitoring, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ## Summary Event streams represent a fundamental shift in how we build data systems — from storing state to recording facts, from synchronous coupling to asynchronous flow, from batch processing to continuous processing. Key concepts: - Event streams are immutable, ordered, partitioned logs of events, distinct from both databases and message queues - Each event carries structured data (key, value, timestamp, headers) with system metadata - Producers and consumers are decoupled, enabling independent scaling and flexible architectures - Platforms like Kafka (KRaft mode in 4.0+), Pulsar, and Kinesis provide different implementations with similar core primitives - Architectural patterns like event sourcing and CQRS leverage event streams to build auditable, scalable systems - Modern data architectures — lakehouses, data mesh, real-time analytics — rely on event streams as their foundation - Schema evolution, retention policies, and governance tools are essential as event streaming scales ## Related Concepts - [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram) - Deep dive into Kafka's foundational components for event streaming - [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture) - Architectural patterns built on event streams - [Streaming Data Pipeline](https://www.conduktor.io/glossary/streaming-data-pipeline) - Building end-to-end pipelines with event streams ## Sources and References 1. Kleppmann, Martin. *Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems*. O'Reilly Media, 2017. (Chapters on Replication, Partitioning, and Stream Processing) 2. Kreps, Jay. "The Log: What every software engineer should know about real-time data's unifying abstraction." *LinkedIn Engineering Blog*, December 2013. [https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying) 3. Apache Kafka Documentation. "Introduction to Kafka Streams." *Apache Software Foundation*. [https://kafka.apache.org/documentation/streams/](https://kafka.apache.org/documentation/streams/) 4. Stopford, Ben. *Designing Event-Driven Systems: Concepts and Patterns for Streaming Services with Apache Kafka*. O'Reilly Media, 2018. 5. AWS Documentation. "What Is Amazon Kinesis Data Streams?" *Amazon Web Services*. [https://docs.aws.amazon.com/streams/latest/dev/introduction.html](https://docs.aws.amazon.com/streams/latest/dev/introduction.html) 6. Narkhede, Neha, Gwen Shapira, and Todd Palino. *Kafka: The Definitive Guide*. O'Reilly Media, 2017. Source: https://www.conduktor.io/glossary/event-stream-fundamentals --- # Event Time and Watermarks in Flink In stream processing, understanding when events occurred versus when they are processed is fundamental to building accurate real-time applications. Apache Flink provides mechanisms for handling event time semantics through watermarks, which allow temporal processing pipelines to handle out-of-order data and late arrivals. ![Event time and watermarks flow](https://www.conduktor.io/assets/images/glossary/event-time-and-watermarks-in-flink-0.webp) ## Understanding Time Semantics in Stream Processing Flink supports three distinct notions of time: - **Processing Time** refers to the wall-clock time of the machine executing the streaming operation. While simple to implement, processing time offers no guarantees about determinism or correctness when events arrive out of order. - **Event Time** represents the timestamp embedded in the event itself, indicating when the event actually occurred at the source. This semantic provides deterministic results regardless of when events arrive at the processing system. - **Ingestion Time** captures the timestamp when an event enters the Flink system. It sits between processing time and event time in terms of determinism and complexity. For applications requiring accurate temporal reasoning, such as sessionization (see [Session Windows in Stream Processing](https://www.conduktor.io/glossary/session-windows-in-stream-processing)), fraud detection, or time-based aggregations, event time is essential. However, event time processing introduces complexity: how does the system know when all events for a given time window have arrived? ## Watermarks: The Event Time Progress Indicator Watermarks are Flink's mechanism for measuring progress in event time. A watermark with timestamp `t` asserts that all events with timestamps less than or equal to `t` have arrived. This allows Flink to trigger time-based operations like window computations. Watermarks flow through the streaming topology as special records. When an operator receives a watermark, it can: 1. Trigger computations for windows whose end time is before the watermark 2. Emit results for completed windows 3. Forward the watermark downstream The key challenge is balancing two competing concerns: - **Too aggressive watermarks** (advancing time too quickly) may cause late data to be dropped or misclassified - **Too conservative watermarks** (advancing time too slowly) increase latency as the system waits longer before triggering computations ## Watermark Strategies and Generators Flink provides several built-in watermark strategies and allows custom implementations. ### Bounded Out-of-Orderness The most common strategy assumes events arrive within a maximum delay: ```java DataStream stream = env .fromSource(kafkaSource, WatermarkStrategy .forBoundedOutOfOrderness(Duration.ofSeconds(5)) .withTimestampAssigner((event, timestamp) -> event.getTimestamp()), "Kafka Source"); ``` This strategy generates watermarks that lag behind the maximum observed timestamp by the specified duration (5 seconds here). Events arriving more than 5 seconds late will be considered late data. ### Monotonous Timestamps For sources that emit events in strictly increasing timestamp order: ```java WatermarkStrategy strategy = WatermarkStrategy .forMonotonousTimestamps() .withTimestampAssigner((event, timestamp) -> event.getTimestamp()); ``` This strategy generates watermarks equal to the latest observed timestamp, suitable for sources like database change streams or ordered log files. ### Custom Watermark Generators For complex scenarios, implement custom watermark logic: ```java public class CustomWatermarkGenerator implements WatermarkGenerator { private long maxTimestamp; private long outOfOrdernessMillis = 5000; @Override public void onEvent(Event event, long eventTimestamp, WatermarkOutput output) { maxTimestamp = Math.max(maxTimestamp, eventTimestamp); } @Override public void onPeriodicEmit(WatermarkOutput output) { output.emitWatermark(new Watermark(maxTimestamp - outOfOrdernessMillis)); } } ``` Flink calls `onPeriodicEmit()` at regular intervals (configured via `pipeline.auto-watermark-interval`, default 200ms), allowing the generator to emit watermarks based on accumulated state. In production, tune this interval based on your latency requirements: lower values (50-100ms) for low-latency applications, higher values (500ms-1s) for high-throughput batch-oriented workloads. ## Handling Late Data Despite watermark strategies, some events will inevitably arrive after their corresponding windows have been triggered. Flink provides several mechanisms for late data handling: ### Allowed Lateness Windows can be configured to accept late data for a specified duration after the watermark passes: ```java stream .keyBy(Event::getUserId) .window(TumblingEventTimeWindows.of(Time.minutes(5))) .allowedLateness(Time.minutes(1)) .aggregate(new AggregateFunction()); ``` When late data arrives within the allowed lateness period, Flink re-triggers the window computation and emits updated results. This enables correction of previously emitted results at the cost of downstream complexity. ### Side Outputs For auditing or alternative processing, late data can be redirected to a side output: ```java OutputTag lateDataTag = new OutputTag("late-data"){}; SingleOutputStreamOperator result = stream .keyBy(Event::getUserId) .window(TumblingEventTimeWindows.of(Time.minutes(5))) .sideOutputLateData(lateDataTag) .aggregate(new AggregateFunction()); DataStream lateData = result.getSideOutput(lateDataTag); ``` This approach allows separate handling of late events, perhaps logging them for analysis or applying compensating logic. For comprehensive coverage of late data strategies beyond watermarks, see [Handling Late Arriving Data in Streaming](https://www.conduktor.io/glossary/handling-late-arriving-data-in-streaming). ## Integration with Kafka and the Streaming Ecosystem When consuming from Apache Kafka, watermark configuration becomes critical for end-to-end correctness. For foundational understanding of Kafka's architecture and how it relates to stream processing, see [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). Kafka topics often contain out-of-order events due to: - Multiple producers writing concurrently - Network delays between producers and brokers - Partition-level ordering but not global ordering Flink's Kafka connector supports per-partition watermark generation. Each partition maintains its own watermark, and Flink merges them by taking the minimum across all partitions. For example, if partition 0 has watermark 12:00:00, partition 1 has 12:00:05, and partition 2 has 11:59:50, Flink uses 11:59:50 as the overall watermark, ensuring no events from any partition are incorrectly processed as late: ```java KafkaSource source = KafkaSource.builder() .setBootstrapServers("localhost:9092") .setTopics("events") .setValueOnlyDeserializer(new EventDeserializationSchema()) .build(); DataStream stream = env.fromSource( source, WatermarkStrategy .forBoundedOutOfOrderness(Duration.ofSeconds(10)) .withTimestampAssigner((event, ts) -> event.getTimestamp()), "Kafka Source" ); ``` For visibility into watermark propagation and late data patterns, platforms like Conduktor provide [monitoring capabilities](https://docs.conduktor.io/guide/monitor-brokers-apps/index) that surface metrics about event time lag, watermark advancement, and late event rates across your streaming topology. These tools offer real-time dashboards for tracking watermark progression, identifying lagging partitions, and alerting on excessive late data. ## Watermark Alignment and Parallel Streams In complex topologies with multiple input streams, watermark alignment becomes important. Flink takes the minimum watermark across all inputs to an operator, ensuring no operator processes data beyond any input's event time progress. For instance, when joining two streams (see [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns)), if stream A has watermark 12:05:00 and stream B has watermark 12:03:00, Flink uses 12:03:00, preventing the join from advancing past timestamps that stream B hasn't yet reached. This conservative approach prevents incorrect results but can cause pipeline stalls if one partition or source falls behind. Flink 1.15 introduced watermark alignment features (enhanced in 1.18+) that can pause faster sources to prevent excessive skew: ```java KafkaSource source = KafkaSource.builder() .setBootstrapServers("localhost:9092") .setTopics("events") .setValueOnlyDeserializer(new EventDeserializationSchema()) .build(); env.fromSource( source, WatermarkStrategy .forBoundedOutOfOrderness(Duration.ofSeconds(5)) .withWatermarkAlignment("alignment-group", Duration.ofSeconds(20)), "Kafka Source" ); ``` This configuration groups sources into alignment groups and limits the maximum drift between aligned sources, improving throughput while maintaining correctness. ## Modern Watermark Features (Flink 1.18+) Recent Flink versions have introduced significant improvements to watermark handling: ### Enhanced Watermark Metrics (Flink 1.18+) Flink 1.18 introduced more granular watermark metrics, exposing per-source and per-operator watermark values through the metrics system. This enables better observability: ```java // Watermark metrics are now automatically exposed: // - watermarkAlignment.drift: Maximum drift in alignment group // - currentInputWatermark: Watermark for each input // - currentOutputWatermark: Operator's emitted watermark ``` These metrics integrate with monitoring systems like Prometheus, allowing teams to track watermark progression and identify bottlenecks in real-time. ### Improved Watermark Propagation (Flink 1.19+) Flink 1.19 optimized watermark propagation in complex topologies, reducing overhead in pipelines with many operators. The runtime now batches watermark updates more efficiently, improving throughput in watermark-heavy applications by 10-15%. ### Watermark Strategies for Unified Source API Modern connectors built on the unified Source API (including the Kafka connector used throughout this article) provide better integration with watermark strategies: ```java // Flink 1.18+ provides better control over per-partition watermark generation WatermarkStrategy strategy = WatermarkStrategy .forBoundedOutOfOrderness(Duration.ofSeconds(5)) .withTimestampAssigner((event, ts) -> event.getTimestamp()) .withIdleness(Duration.ofMinutes(1)) .withWatermarkAlignment("alignment-group", Duration.ofSeconds(20)); ``` The unified approach ensures consistent watermark behavior across all source connectors, simplifying configuration and debugging. For deeper exploration of building streaming applications with these APIs, see [Flink DataStream API: Building Streaming Applications](https://www.conduktor.io/glossary/flink-datastream-api-building-streaming-applications). ## Best Practices and Edge Cases ### Idle Sources When a partition or source stops emitting events, its watermark stops advancing, potentially stalling downstream operators. Flink's `withIdleness()` configuration allows sources to be marked idle: ```java WatermarkStrategy .forBoundedOutOfOrderness(Duration.ofSeconds(5)) .withIdleness(Duration.ofMinutes(1)) .withTimestampAssigner((event, ts) -> event.getTimestamp()) ``` After one minute without events, the source is considered idle and its watermark is ignored when computing the minimum across inputs. ### Debugging Watermark Issues Common symptoms of watermark problems include: - **Windows never triggering**: Watermarks not advancing (check timestamp extraction and source idleness) - **Excessive late data**: Watermark advancing too aggressively (increase out-of-orderness bound) - **High latency**: Watermark advancing too conservatively (reduce out-of-orderness bound or check for straggling partitions) Enable watermark logging and metrics to diagnose these issues in production. ## Watermark Strategies in Flink SQL For teams using Flink SQL (increasingly common in 2025), watermark strategies are declared directly in table DDL: ```sql CREATE TABLE events ( user_id STRING, event_type STRING, event_timestamp BIGINT, event_time AS TO_TIMESTAMP_LTZ(event_timestamp, 3), WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND ) WITH ( 'connector' = 'kafka', 'topic' = 'events', 'properties.bootstrap.servers' = 'localhost:9092', 'format' = 'json' ); ``` The `WATERMARK FOR` clause defines the watermark strategy declaratively. The expression `event_time - INTERVAL '5' SECOND` creates a bounded out-of-orderness strategy with a 5-second delay, equivalent to the DataStream API's `forBoundedOutOfOrderness(Duration.ofSeconds(5))`. For strictly ordered data, use the current timestamp: ```sql WATERMARK FOR event_time AS event_time ``` Flink SQL automatically propagates watermarks through queries, making event time processing accessible to SQL users without requiring Java/Scala knowledge. This has made Flink SQL a popular choice for stream processing in 2025, particularly in data teams with strong SQL expertise. For more advanced stream processing patterns in SQL, see [Flink SQL and Table API for Stream Processing](https://www.conduktor.io/glossary/flink-sql-and-table-api-for-stream-processing). ## Summary Event time processing with watermarks enables Flink applications to handle out-of-order data and produce accurate temporal results. Key takeaways: - **Event time semantics** provide deterministic results based on when events actually occurred, essential for temporal reasoning - **Watermarks** signal progress in event time, allowing time-based operations to trigger while balancing latency and completeness - **Watermark strategies** must be tuned based on source characteristics: use bounded out-of-orderness for typical Kafka scenarios, monotonous timestamps for ordered sources, and custom generators for complex patterns - **Late data handling** through allowed lateness and side outputs provides flexibility for correcting results or auditing late arrivals - **Kafka integration** requires careful consideration of per-partition watermarks and alignment strategies - **Observability** through monitoring tools helps track watermark propagation and tune strategies in production Understanding event time and watermarks is essential for building production-grade streaming applications that handle real-world data characteristics while maintaining correctness and performance. ## Related Concepts - [Watermarks and Triggers in Stream Processing](https://www.conduktor.io/glossary/watermarks-and-triggers-in-stream-processing) - Comprehensive guide to watermark strategies and trigger mechanisms across stream processing frameworks - [Handling Late-Arriving Data in Streaming](https://www.conduktor.io/glossary/handling-late-arriving-data-in-streaming) - Advanced techniques for managing late events beyond basic watermark configuration - [Windowing in Apache Flink](https://www.conduktor.io/glossary/windowing-in-apache-flink-tumbling-sliding-and-session-windows) - How watermarks interact with different window types in Flink ## Sources and References - [Apache Flink Documentation: Event Time and Watermarks](https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/) - [Apache Flink Documentation: Generating Watermarks](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/event-time/generating_watermarks/) - [Flink Kafka Connector Documentation](https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/) - [Watermark Alignment in Flink 1.15](https://flink.apache.org/2022/06/09/watermark-alignment.html) Source: https://www.conduktor.io/glossary/event-time-and-watermarks-in-flink --- # Kafka Exactly-Once: Producers + Transactions **Exactly-once semantics (EOS)** in Kafka guarantees that every record is produced and consumed exactly once, even across broker failures and retries. Kafka implements EOS via idempotent producers (sequence-number dedup) and transactions (atomic multi-partition writes). It is the strongest delivery guarantee, at higher latency and coordination cost than at-least-once. ## Understanding Delivery Guarantees: At-Most-Once, At-Least-Once, and Exactly-Once Data processing guarantees define how a system handles records when failures occur. Distributed systems offer three fundamental delivery semantic models, each representing different trade-offs between reliability, performance, and implementation complexity. ### At-Most-Once Delivery The system processes a record at most once. If a failure occurs, the system will not retry, potentially leading to data loss. This occurs when producers don't wait for acknowledgment (`acks=0`) or consumers commit offsets before processing messages. **Use cases:** High-throughput telemetry, metrics collection, or log aggregation where occasional data loss is acceptable and performance is critical. Offers highest throughput and lowest latency because no retry logic or coordination is needed. ### At-Least-Once Delivery The system guarantees a record will be processed, but failure recovery may cause the record to be processed more than once, leading to data duplication. This is the default setting for most message systems and the most common semantic in production. - **Use cases:** Most data pipelines where duplicate processing can be handled through idempotent operations or deduplication logic. For example, updating a customer's current address is naturally idempotent — writing the same address twice produces the same result. - **Configuration:** Producer with `acks=all` and retries enabled; consumer commits offsets only after successful processing. ### Exactly-Once Delivery (EOS) The system ensures that every record is processed exactly one time, avoiding both data loss and data duplication. Messages are processed precisely once even in the presence of producer retries, broker failures, or consumer crashes. - **Use cases:** Financial transactions, inventory updates, billing systems, or any operation where duplicates cause incorrect results. Processing a payment twice charges the customer incorrectly; missing a payment fails to record revenue. - **Trade-offs:** Slightly higher latency (typically 2-5ms) and lower throughput (10-20% reduction) compared to at-least-once, plus increased configuration and operational complexity. ## What is Exactly-Once Semantics? Exactly-once semantics means that the effect of processing a message happens exactly once, even if the system encounters failures, retries, or network issues. In Kafka, this involves two key guarantees: 1. **Idempotent production**: A producer can safely retry sending messages without creating duplicates 2. **Transactional processing**: Messages are produced, consumed, and processed as atomic operations Kafka introduced exactly-once semantics in version 0.11.0 and has continued to refine the implementation in subsequent releases. Kafka 2.5 introduced the improved `exactly_once_v2` protocol for Kafka Streams, and Kafka 4.0+ brings significant performance improvements to transactional processing in KRaft mode, with faster transaction commits and reduced coordinator overhead. ## The Challenge of Exactly-Once: Why It's Considered Hard Achieving EOS is complex because failures in distributed systems are asynchronous and unpredictable. ### Producer Retries and Duplicate Writes When a producer sends a message to the broker and doesn't receive an acknowledgment due to a network timeout, it faces ambiguity: did the broker receive the message but fail before acknowledging? Or did the message never arrive? Without additional mechanisms, retrying risks writing the same message twice (violating exactly-once); not retrying risks data loss (violating at-least-once). ### Consumer Recovery and Duplicate Processing When a consumer fails after reading a record but before committing its offset, it must restart and will re-read and re-process the last record. The challenge intensifies with stateful operations (aggregations, joins, windows) because the consumer must coordinate reading from source, updating internal state, writing results to sink, and committing source offsets, all atomically. ### Coordinating Distributed State Exactly-once semantics requires atomic operations across multiple components: producers writing to topics, consumers processing messages, stream processors updating state, and external systems persisting results, all as a single logical transaction. This coordination problem compounds when operations span multiple partitions, topics, or separate systems. ## How Kafka Achieves Exactly-Once Semantics Kafka's exactly-once implementation relies on three core mechanisms: idempotent producers, transactions, and transactional consumers. ![exactly-once-semantics-in-kafka diagram 1](https://www.conduktor.io/assets/images/glossary/exactly-once-semantics-in-kafka-0.webp) ### Idempotent Producers: Preventing Duplicate Writes The idempotent producer guarantees that retrying the same send request will only ever write the record once to the Kafka log. Kafka assigns each producer a unique Producer ID (PID) during initialization and tracks a sequence number for each message sent to each partition. This sequence number increments with each message. When a producer retries, Kafka detects duplicate messages by comparing sequence numbers and simply acknowledges success without writing the message again. **Configuration (Kafka 3.0+ defaults):** ```properties enable.idempotence=true # Default in Kafka 3.0+ acks=all retries=Integer.MAX_VALUE max.in.flight.requests.per.connection=5 ``` Idempotence is enabled by default in Kafka 3.0+, making exactly-once semantics more accessible. Prior versions required explicit configuration. For comprehensive coverage of producer configuration and behavior, see [Kafka Producers](https://www.conduktor.io/glossary/kafka-producers). ### Transactional Producers: Atomic Multi-Partition Writes Transactional producers extend idempotence to allow atomic writes across multiple partitions and topics. A Transaction Coordinator in the broker manages a two-phase commit process (a distributed algorithm that ensures all participants agree to commit or abort), guaranteeing either all messages in the batch are committed or none are. To use transactions, assign a `transactional.id` to your producer. This ID is persistent across producer restarts, allowing Kafka to fence out zombie producers (old instances that haven't fully shut down) and prevent split-brain scenarios. **Configuration:** ```properties enable.idempotence=true transactional.id=unique-transactional-id-per-producer-instance ``` **Producer code pattern:** ```java producer.initTransactions(); try { producer.beginTransaction(); producer.send(new ProducerRecord<>("output-topic", key, value)); producer.sendOffsetsToTransaction(offsets, consumerGroupId); producer.commitTransaction(); } catch (Exception e) { producer.abortTransaction(); } ``` This enables the read-process-write cycle to be atomic: read from input topic, process, write to output topic, and commit input offsets all succeed together or all fail together. For the full mechanics of Kafka's transactional implementation, see [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). ### Transactional Consumers: Reading Only Committed Data Consumers must configure `isolation.level=read_committed` to participate in exactly-once semantics. With this setting, they only see messages from committed transactions; in-flight and aborted writes are filtered out. **Configuration:** ```properties isolation.level=read_committed enable.auto.commit=false ``` When processing messages transactionally, the consumer reads messages, processes them, produces results to output topics, and commits its offsets within a single transaction. If the transaction fails, none of the effects are visible. For more information on consumer offset management and group coordination, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). Tools like Conduktor provide visual monitoring of transaction markers in Kafka topics through [topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) and real-time tracking of transactional producer states, making it easier to debug exactly-once configurations, verify transaction completion, and troubleshoot issues like stalled transactions or coordinator problems. ### KRaft and Transaction Performance (Kafka 4.0+) With Kafka 4.0 (released 2024), KRaft (Kafka Raft) replaced ZooKeeper as the metadata management system and became the default deployment mode. This architectural shift significantly improves exactly-once semantics performance and reliability: **Transaction Coordinator Improvements:** - **Faster Failover:** Transaction coordinator recovery time reduced from seconds to milliseconds, minimizing transaction timeout risk during broker failures - **Reduced Metadata Overhead:** Transaction state replication is more efficient without ZooKeeper coordination, reducing broker CPU and network usage - **Improved Commit Latency:** Transaction commits are 5-10% faster thanks to simpler metadata operations through the Raft protocol **Operational Benefits:** - **Simplified Architecture:** Eliminates ZooKeeper dependencies, reducing operational complexity for exactly-once deployments - **Better Scalability:** Supports more concurrent transactions per broker without metadata bottlenecks - **Enhanced Monitoring:** Transaction coordinator metrics are now integrated directly into Kafka's native monitoring KRaft is the default and recommended mode for Kafka 4.0+. ZooKeeper mode is deprecated and will be removed in future versions. For exactly-once workloads, migrating to KRaft provides measurable performance improvements and operational simplification. ## Exactly-Once in Stream Processing Stream processing frameworks like Kafka Streams and Apache Flink build on Kafka's exactly-once semantics to provide end-to-end processing guarantees. While Kafka provides the transactional messaging layer, stream processing engines manage state correctly during failures through checkpointing. ### Kafka Streams Kafka Streams provides exactly-once semantics out of the box when you set `processing.guarantee=exactly_once_v2` (introduced in Kafka 2.5 and the recommended setting for Kafka 2.5+). It coordinates consumer offsets, operator state snapshots, and output commits within a single Kafka transaction. When a failure occurs, Kafka Streams rolls back to the last successful transaction and resumes processing without loss or duplication. Kafka 4.0+ further improves Kafka Streams EOS performance with optimized transaction handling in KRaft mode. For a comprehensive introduction to Kafka Streams architecture, see [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). ### Apache Flink Flink achieves exactly-once semantics through distributed checkpointing coordinated with Kafka transactions using two-phase commit: first, all operators prepare the transaction by persisting state; second, all commit together. Flink periodically snapshots operator state, writes output records to Kafka without committing, then commits all transactions together only after all operators successfully complete their checkpoint. If a failure occurs before a checkpoint, Flink rolls back to the last successful checkpoint and Kafka aborts the incomplete transaction. **Flink 1.19+ Improvements (2024):** - **Enhanced Checkpoint Coordinator:** Reduces checkpoint completion time by 10-15%, accelerating recovery from failures - **Better Timeout Handling:** Improved detection and recovery from checkpoint timeouts during transaction coordination - **Reduced State Overhead:** Optimized checkpoint metadata storage reduces the cost of maintaining exactly-once guarantees These improvements make Flink's exactly-once mode more performant and reliable for production workloads processing millions of events per second. For detailed coverage of Flink's checkpointing and state management, see [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing). To compare the two approaches, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink). ## Achieving End-to-End EOS: Source, Processor, and Sink True end-to-end exactly-once semantics requires coordination across all three pipeline stages. - **Source Requirements:** The source must be re-readable, allowing consumers to reset positions and replay data after failures. Kafka's durable log architecture excels here. - **Processor Requirements:** The processor must be transactional, coordinating state updates with input consumption and output production. Flink and Kafka Streams provide this via distributed checkpointing and two-phase commits. - **Sink Requirements:** The sink must be either idempotent (safely handling duplicate writes like Cassandra or key-value stores) or transactional (participating in two-phase commit like PostgreSQL or MySQL). Non-compliant sinks require application-level deduplication. ## Trade-offs and Performance Considerations Exactly-once semantics comes with measurable costs. The additional coordination and bookkeeping required to guarantee exactly-once delivery impacts throughput and latency. - **Latency:** Transactional producers add 2-5 milliseconds due to transaction coordination (waiting for the transaction coordinator to acknowledge commits). - **Throughput:** The two-phase commit protocol and additional metadata management reduce maximum throughput by 10-20% compared to at-least-once semantics. - **Broker Load:** Transaction coordinators and additional state tracking consume more broker resources. Each transactional producer maintains state, and transaction logs require disk space. - **Operational Complexity:** Requires monitoring transaction status and handling zombie transactions. ### When Exactly-Once Is Worth It **Choose exactly-once semantics when:** - Duplicates cause incorrect results (financial transactions, inventory updates, billing) - Applications maintain monetary values or non-idempotent state - Compliance requires guaranteed processing (audit trails, regulatory reporting) - The cost of incorrect data exceeds the performance overhead **Consider at-least-once when:** - Downstream operations are naturally idempotent - Performance requirements are extreme (sub-millisecond latency) - The application can deduplicate through business logic Platforms like Conduktor help you understand the performance impact of exactly-once semantics by providing comprehensive visibility into producer and consumer metrics through [topic monitoring](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), transaction success rates, coordinator performance, and end-to-end latency. ## Monitoring and Validating Exactly-Once Behavior Operating exactly-once pipelines requires specific monitoring. **Key Metrics:** - **Transaction Coordinator Lag:** Indicates broker overload or configuration issues - **Aborted Transactions:** Spikes suggest application errors or infrastructure issues - **Producer Transaction Timeouts:** Detects slow processing or oversized transactions - **Fence Occurrences:** When a new producer instance with the same `transactional.id` starts, Kafka fences out (blocks) the old instance to prevent zombie producers from corrupting data. Frequent fencing suggests transaction timeouts being too short, producers restarting too often, or duplicate transactional IDs. - **Zombie Transaction Detection:** Transactions remaining open can block downstream consumers **Validation Techniques:** - End-to-end counting with unique identifiers - Application-level duplicate detection for validation - Transaction state inspection using Kafka tools - State consistency checks comparing checkpoint state with topic offsets For comprehensive approaches to validating exactly-once behavior in production, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). ## Governance and Compliance In enterprise environments, governance is essential to ensure EOS is correctly configured. Platforms like Conduktor provide real-time visibility into transaction status across all Kafka clusters through [topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), enabling operators to identify and abort zombie transactions, enforce policies requiring EOS for sensitive data streams, track which applications have exactly-once enabled, and monitor transaction health metrics in centralized dashboards. **Testing Exactly-Once with Conduktor Gateway:** Validating exactly-once behavior under failure conditions requires chaos testing. Conduktor Gateway acts as a proxy layer between producers/consumers and Kafka brokers, enabling controlled failure injection: - **Transaction Coordinator Failures:** Simulate coordinator crashes during transaction commits to verify proper recovery - **Network Partitions:** Introduce network delays or disconnections to test timeout handling and retry logic - **Broker Failures:** Trigger broker unavailability during two-phase commits to validate transaction rollback - **Consumer Rebalancing:** Force consumer group rebalances mid-transaction to test fence handling This testing validates that applications correctly maintain exactly-once guarantees even when infrastructure fails, providing confidence before production deployment. ## Real-World Use Cases - **Financial Services:** A payments platform uses exactly-once semantics to ensure that each payment instruction is processed exactly once. When a customer initiates a wire transfer, the system publishes a transaction event to Kafka. Downstream services consume this event to debit the sender's account, credit the receiver's account, and record the transaction for compliance. Without exactly-once guarantees, a retry could cause a double charge or duplicate accounting entry. - **E-commerce Order Processing:** An online retailer processes orders through Kafka. When a customer places an order, an order service publishes the event transactionally along with inventory updates. The inventory service consumes these events with exactly-once semantics to ensure that each order decrements inventory counts exactly once. This prevents overselling products due to duplicate inventory decrements or underselling due to lost messages. For more e-commerce streaming patterns, see [E-commerce Streaming Architecture Patterns](https://www.conduktor.io/glossary/e-commerce-streaming-architecture-patterns). ## Summary Exactly-once semantics is the highest data integrity guarantee in data streaming, essential for maintaining correct state in financial, billing, inventory, and compliance-driven applications. - **Why it matters:** EOS eliminates both data loss and duplication by ensuring every record is processed exactly once. For financial transactions and regulatory compliance, that guarantee prevents incorrect results from duplicate processing or missing data. - **How it works:** Kafka achieves EOS through idempotent producers (sequence-number deduplication), transactional producers (atomic multi-partition writes via two-phase commit), and transactional consumers (read_committed isolation). Stream processors like Kafka Streams and Apache Flink build on these primitives via checkpointing. - **What it costs:** 2-5ms additional latency, 10-20% throughput reduction. Acceptable for critical workflows; use at-least-once elsewhere. - **Where platforms fit:** Governance and observability platforms help manage exactly-once configurations at scale: tracking settings, monitoring transaction states, enforcing policies, and meeting regulatory requirements for data integrity. ## Related Concepts - [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive) - Full mechanics of Kafka's transactional implementation - [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions) - Managing distributed transactions with exactly-once guarantees - [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink) - Comparing the two stream processing approaches to exactly-once - [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing) - How Flink's checkpointing integrates with Kafka transactions - [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications) - Validating exactly-once behavior in production - [Kafka UI: The Complete Guide](https://www.conduktor.io/kafka-ui) - Inspect producer/consumer state, decode payloads, and debug transactional workflows from a UI - [Kafka Streams exactly-once (hands-on)](https://www.conduktor.io/kafka-streams/exactly-once) - What exactly_once_v2 actually guarantees in a Streams app, and its scope ## Sources and References 1. Apache Kafka Documentation: [Exactly Once Semantics](https://kafka.apache.org/documentation/#semantics) - Official documentation on Kafka's delivery semantics and transactional API 2. Confluent Blog: [Exactly-Once Semantics Are Possible: Here's How Kafka Does It](https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/) - In-depth technical explanation of Kafka's exactly-once implementation 3. KIP-98: [Exactly Once Delivery and Transactional Messaging](https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging) - Original Kafka Improvement Proposal introducing exactly-once semantics 4. Apache Flink Documentation: [Fault Tolerance Guarantees of Data Sources and Sinks](https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/guarantees/) - How Flink integrates with Kafka for exactly-once processing 5. Jay Kreps: [Transactions in Apache Kafka](https://www.confluent.io/blog/transactions-apache-kafka/) - Architectural rationale for Kafka's transactional model 6. Apache Kafka: [KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum) - KRaft architecture enabling improved transaction performance Source: https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka --- # Fine-Grained Access Control for Kafka Kafka's native ACLs operate at the topic level: a principal can read or write an entire topic, or they cannot. But real-world security requirements rarely align with topic boundaries. A single topic might contain customer records where the analytics team should see purchase history but not payment card numbers, where EU analysts should access only EU customer data, and where PII fields must be masked for third-party integrations. This is the gap between coarse-grained and fine-grained access control. Organizations discover that topic-level permissions are just the starting point. True fine-grained control means enforcing access at the record or field level — without duplicating data across dozens of filtered topics. ## The Granularity Spectrum Access control in streaming platforms operates across multiple levels, each with different implementation complexity and operational overhead. ![Fine-Grained Access Control Granularity Levels](https://www.conduktor.io/assets/images/glossary/fine-grained-access-control-for-kafka-0.webp) - **Cluster level**: Superusers with full administrative access. Binary: either you manage everything or nothing. - **Topic level**: Standard Kafka ACLs. A principal can Read from `orders` or Write to `payments`. No visibility into what's inside the topic. - **Partition level**: Rarely used for security. Consumer groups can be assigned specific partitions, but this isn't access control: any consumer in the group can poll any partition. - **Record level**: Filter records based on key, headers, or payload attributes. The analytics-eu consumer only receives records where `region=EU`. Requires a filtering layer above Kafka. - **Field level**: The finest granularity. Mask credit card numbers, redact PII fields, or transform values based on the consumer's clearance. A single record might look different to different consumers. Most organizations implement cluster and topic-level controls through native ACLs, then discover they need record or field-level filtering for compliance, multi-tenancy, or data sharing scenarios. ## Why Topic-Level ACLs Aren't Enough Three common scenarios expose the limitations of coarse-grained access: - **Mixed-sensitivity topics**: A `customer-events` topic contains registration, login, purchase, and support events. The analytics team needs purchase patterns but shouldn't access support tickets containing complaint details. Creating separate topics fragments the customer timeline and complicates event ordering. - **Regulatory compliance**: GDPR requires data minimization: consumers should receive only the data they need. A downstream system processing shipping labels doesn't need customer email addresses or phone numbers, even if it needs their shipping address from the same record. - **Multi-tenant analytics**: A SaaS platform streams usage events from all tenants through a single topic. Each tenant's analytics dashboard should see only their own data, but replicating to per-tenant topics creates operational overhead proportional to tenant count. Native Kafka ACLs cannot solve these problems. You either grant Read on the topic (exposing everything) or deny it (exposing nothing). The workarounds — creating filtered topic copies, client-side filtering, or trusting consumers to ignore unauthorized data — all have significant drawbacks. ## Achieving Field-Level Access Control Field-level access control requires intercepting data between Kafka and consumers, then transforming or filtering based on policy. Several approaches exist, each with different trade-offs. ### Schema-Aware Filtering When topics use Schema Registry (Avro, Protobuf, or JSON Schema), the schema defines which fields exist. A policy engine can reference schema field names to determine what to mask or remove: ```yaml # Policy: mask PII for analytics-consumer policies: - principal: "analytics-consumer" topic: "customer-events" fields: - name: "email" action: mask # alice@company.com → a***@c*****.com - name: "phone" action: redact # removed entirely - name: "ssn" action: deny # consumer cannot access records with this field populated ``` The filtering layer deserializes records using the schema, applies field transformations, and re-serializes before delivering to consumers. This requires CPU overhead but preserves schema evolution:when new fields are added, policies can specify default actions. ### Virtual Topics and Views Rather than modifying data in-flight, virtual topics present a filtered view of an underlying physical topic. Consumers connect to `customer-events-analytics` which internally reads from `customer-events` and applies transformations. This pattern mirrors database views. The underlying data remains unchanged, but different consumers see different projections. Virtual topics can also: - Filter records by key pattern or header values - Route subsets of data (EU records to one virtual topic, US to another) - Apply rate limiting per virtual topic Conduktor Gateway implements virtual topics through its [Virtual Clusters and aliasing features](https://docs.conduktor.io/guide/conduktor-concepts/virtual-clusters/), enabling field-level transformations without modifying producer or consumer code. ### Proxy-Based Enforcement A Kafka proxy sits between clients and brokers, inspecting every request and response. For Produce requests, it can reject messages containing unauthorized fields. For Fetch responses, it can filter or transform records before returning them to consumers. Proxy architecture enables: - **Transparent enforcement**: Consumers use standard Kafka clients. They don't know filtering is happening. - **Centralized policy**: One policy engine governs all access, simplifying audit and compliance. - **Protocol-level control**: Block specific API calls (like DescribeConfigs) for certain principals. The trade-off is latency. Every message passes through an additional network hop and deserialization cycle. For high-throughput clusters (100K+ messages/second), proxy overhead must be carefully measured. ## Context-Aware and Dynamic Policies Static policies ("analytics-consumer can read customer-events with email masked") cover many use cases but miss context-dependent requirements. - **Time-based access**: Production data access allowed only during business hours. On-call engineers can access sensitive topics outside hours. - **Purpose-based access**: The ML training pipeline can access full customer records. The real-time dashboard sees only aggregates. - **Location-based access**: EU-based consumers can access EU customer data. Cross-region access requires additional authorization. These scenarios require evaluating attributes beyond principal identity. Open Policy Agent (OPA) and AWS Cedar provide policy languages for attribute-based decisions: ```rego # OPA policy: time-based production access package kafka.authz import future.keywords.if allow if { input.topic == "production.orders" input.operation == "Read" is_business_hours } allow if { input.topic == "production.orders" input.operation == "Read" input.principal.on_call == true } is_business_hours if { hour := time.clock(time.now_ns())[0] hour >= 9 hour < 18 } ``` Dynamic policies require a policy engine evaluating each request. For Kafka, this means either a custom authorizer (for topic-level decisions) or a proxy layer (for record/field-level decisions). ## Implementation Patterns for Kafka ![Fine-Grained Access Control Implementation Architecture](https://www.conduktor.io/assets/images/glossary/fine-grained-access-control-for-kafka-1.webp) ### Pattern 1: Proxy with Field-Level Encryption and Masking Conduktor Gateway provides multiple mechanisms for fine-grained data protection: - **Field-level encryption**: Encrypt sensitive fields at produce time. Data is encrypted before reaching Kafka, so even brokers and administrators cannot read protected fields. - **Contextual decryption**: Only virtual clusters with the DecryptPlugin configured receive decrypted values. Others see encrypted ciphertext. The same physical topic serves multiple consumers with different views based on which virtual cluster they connect through. - **Data masking**: Transform field values (partial masking, redaction) based on policy. Unlike encryption, masking is lossy: the original value cannot be recovered. ```yaml # Encrypt PII fields on produce (scoped to passthrough vCluster) pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin config: recordValue: fields: - fieldName: email keySecretId: vault-kms://vault:8200/transit/keys/pii-key - fieldName: payment.cardNumber keySecretId: vault-kms://vault:8200/transit/keys/payment-key --- # Decrypt for ml-training vCluster only pluginClass: io.conduktor.gateway.interceptor.DecryptPlugin scope: vCluster: ml-training --- # Mask for analytics vCluster pluginClass: io.conduktor.gateway.interceptor.FieldLevelDataMaskingPlugin scope: vCluster: analytics config: policies: - name: mask-email rule: { type: MASK_ALL } fields: [email] - name: mask-card rule: { type: MASK_LAST_N, numberOfChars: 12 } fields: [payment.cardNumber] ``` Full configuration details: [Conduktor Gateway Data Security Reference](https://docs.conduktor.io/guide/reference/data-security/) This layered approach means: - **ML training pipelines** (via `ml-training` vCluster) receive fully decrypted data - **Analytics consumers** (via `analytics` vCluster) receive masked data - **Other consumers** see encrypted ciphertext (no decryption interceptor configured) ### Pattern 2: Stream Processing Transformation Use Kafka Streams or Flink to create filtered derivative topics: ```java // Kafka Streams: create analytics-safe topic StreamsBuilder builder = new StreamsBuilder(); builder.stream("customer-events", Consumed.with(stringSerde, customerEventSerde)) .mapValues(event -> { // Mask PII fields event.setEmail(mask(event.getEmail())); event.setPhone(null); return event; }) .to("customer-events-analytics", Produced.with(stringSerde, customerEventSerde)); ``` This creates a separate topic with transformed data. Advantages: no proxy overhead for reads. Disadvantages: data duplication, additional infrastructure, potential consistency issues between topics. ### Pattern 3: Consumer-Side Filtering with Policy Enforcement Trust consumers to enforce policies, but verify through audit logging and periodic compliance checks. This works for internal teams with strong governance but fails for external data sharing where you cannot trust the consumer. Implement via Schema Registry validation: schemas include field-level annotations marking sensitivity, and consumers must demonstrate compliance through code review or automated scanning. ## Operational Considerations - **Performance overhead**: Field-level filtering requires deserializing every record. For JSON or Avro, this adds 0.1-1ms per record depending on schema complexity. At 100K records/second, a proxy-based approach needs significant CPU resources. Benchmark with production-representative loads before deploying. - **Audit requirements**: Fine-grained access control is only valuable if you can prove it's working. Log every access decision, every transformation applied, and every policy change. Conduktor provides [audit logging](https://docs.conduktor.io/platform/navigation/settings/audit-log/) for tracking who accessed what data with which transformations. - **Policy drift**: Policies written today may not match tomorrow's requirements. Schema evolution adds fields that policies don't cover. New consumer applications may not be assigned appropriate policies. Implement policy-as-code with version control and review processes. - **Testing access policies**: How do you verify that analytics-consumer actually receives masked data? Build integration tests that consume as each principal and validate field values. Conduktor Gateway supports testing interceptors in isolation before deploying to production. ## Summary Fine-grained access control extends Kafka security beyond topic-level ACLs to record and field-level enforcement. While native Kafka cannot filter by field, proxy-based architectures enable schema-aware masking, virtual topics, and dynamic policy evaluation without modifying producers or consumers. Organizations typically start with topic-level ACLs, then add fine-grained controls when compliance requirements, multi-tenancy, or data sharing scenarios demand more precision. The implementation choice — proxy, stream processing, or trusted consumers — depends on latency tolerance, operational complexity, and trust boundaries. Conduktor Gateway provides a production-ready approach to fine-grained control through field-level encryption, contextual decryption, and data masking — serving multiple consumers with different data views from the same topics without data duplication. ## Related Concepts - [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns): Foundation for topic-level access control that fine-grained policies extend - [Data Access Control: RBAC and ABAC](https://www.conduktor.io/glossary/data-access-control-rbac-and-abac): Role and attribute-based models that inform fine-grained policy design - [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming): Techniques for protecting sensitive fields within records ## Sources and References - [Apache Kafka Documentation - Authorization and ACLs](https://kafka.apache.org/documentation/#security_authz) - Native Kafka authorization model and its granularity limitations. - [NIST SP 800-162: Guide to Attribute Based Access Control](https://csrc.nist.gov/publications/detail/sp/800-162/final) - Framework for fine-grained, attribute-based authorization policies. - [Open Policy Agent - Kafka Authorization](https://www.openpolicyagent.org/docs/latest/kafka-authorization/) - Policy engine integration for dynamic access decisions. - [Conduktor Gateway Documentation](https://docs.conduktor.io/gateway/) - Proxy-based implementation of field-level access control and virtual clusters. Source: https://www.conduktor.io/glossary/fine-grained-access-control-for-kafka --- # Flink DataStream API: Building Streaming Applications The DataStream API is Apache Flink's primary interface for building streaming applications. It provides a declarative programming model that handles the distributed processing complexity while giving you control over state management, time semantics, and fault tolerance. Unlike batch processing frameworks that operate on bounded datasets, the DataStream API is designed for continuous, unbounded streams of data. This makes it suitable for use cases such as real-time analytics, fraud detection, event-driven applications, and IoT data processing. The API supports Java, Scala, and Python (PyFlink), allowing developers to write type-safe transformations that Flink optimizes and executes across distributed clusters. As of Flink 2.0+ (released in 2025), the DataStream API has matured with significant improvements to state backends, async state operations, and unified source/sink APIs that simplify integration with systems like Apache Kafka. For a broader introduction to Flink's capabilities, see [What is Apache Flink? Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). ![Flink DataStream API transformations](https://www.conduktor.io/assets/images/glossary/flink-datastream-api-building-streaming-applications-0.webp) ## Core Concepts and Architecture ### DataStream Abstraction A `DataStream` represents an immutable, distributed collection of elements flowing through a Flink application. DataStreams are created from sources, transformed through various operations, and ultimately written to sinks. Each transformation produces a new DataStream, forming a directed acyclic graph (DAG) that Flink compiles into an optimized execution plan. ### Execution Environment Every Flink application begins by obtaining an execution environment, which is the entry point for defining sources and configuring runtime parameters: ```java StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.enableCheckpointing(60000); // Enable checkpointing every 60 seconds env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); ``` Note: Since Flink 1.12+, event time is the default time characteristic, so `setStreamTimeCharacteristic()` is no longer needed and was removed in Flink 2.0. The execution environment determines whether the application runs locally for development or is submitted to a distributed cluster for production workloads. ### Sources and Sinks Sources ingest data into the streaming pipeline from external systems such as Apache Kafka, file systems, or custom connectors. Sinks write processed data to downstream storage or messaging systems. Since Flink 1.18+, the unified Source API (FLIP-27) has become the standard for implementing sources, replacing the older `SourceFunction` interface. The new API provides better support for bounded/unbounded sources, checkpointing, and event time. Similarly, the Sink API (FLIP-143) replaces `SinkFunction` with a more robust interface supporting exactly-once semantics and flexible writer implementations. ## Essential Transformations The DataStream API offers a rich set of transformations for manipulating streaming data. Understanding these operations is fundamental to building effective stream processing applications. ### Basic Operations - **map**: Applies a function to each element, producing one output per input - **filter**: Selects elements matching a predicate - **flatMap**: Produces zero or more outputs per input, useful for parsing and tokenization ```java DataStream events = env.fromElements("user:1,action:click", "user:2,action:view"); DataStream parsedEvents = events .flatMap((String value, Collector out) -> { String[] parts = value.split(","); if (parts.length == 2) { String userId = parts[0].split(":")[1]; String action = parts[1].split(":")[1]; out.collect(new UserAction(userId, action, System.currentTimeMillis())); } }) .returns(UserAction.class); ``` ### Keyed Streams and Stateful Operations Partitioning data by key enables stateful processing, where Flink maintains state for each key independently. The `keyBy` operation creates a `KeyedStream`, which supports aggregations, reductions, and custom stateful functions: ```java DataStream actions = ...; DataStream actionCounts = actions .keyBy(action -> action.getUserId()) .window(TumblingEventTimeWindows.of(Time.minutes(5))) .aggregate(new CountAggregateFunction()); ``` Keyed streams ensure that all events with the same key are processed by the same parallel instance, maintaining consistency for stateful operations. ### Rich Functions Rich functions extend basic transformations with lifecycle methods and access to runtime context. The `RichMapFunction`, `RichFlatMapFunction`, and similar classes provide `open()` and `close()` methods for resource initialization and cleanup, as well as access to state and metrics: ```java public class EnrichmentMapper extends RichMapFunction { private transient ValueState profileState; @Override public void open(Configuration parameters) { ValueStateDescriptor descriptor = new ValueStateDescriptor<>("user-profile", UserProfile.class); profileState = getRuntimeContext().getState(descriptor); } @Override public EnrichedEvent map(Event event) throws Exception { UserProfile profile = profileState.value(); return new EnrichedEvent(event, profile); } } ``` For high-performance state access patterns, Flink 1.19+ introduces the Async State API, which allows non-blocking state operations that can significantly improve throughput in I/O-bound applications. This is particularly beneficial when using remote state backends. For comprehensive coverage of state management patterns, see [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing). ## Windowing and Time Semantics Time is a critical dimension in stream processing. Flink distinguishes between event time (when events actually occurred) and processing time (when events are processed by the system). ### Event Time Processing Event time processing enables correct results even when events arrive out of order or delayed. Applications assign timestamps to events and define watermarks, which signal progress in event time: ```java DataStream events = env .fromSource(kafkaSource, WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10)) .withTimestampAssigner((event, timestamp) -> event.getTimestamp()), "Kafka Source" ); ``` Watermarks allow Flink to determine when all events up to a certain timestamp have arrived, triggering window computations and handling late data appropriately. For a deep dive into watermark mechanics, see [Watermarks and Triggers in Stream Processing](https://www.conduktor.io/glossary/watermarks-and-triggers-in-stream-processing). For strategies to handle events that arrive after watermarks have passed, see [Handling Late-Arriving Data in Streaming](https://www.conduktor.io/glossary/handling-late-arriving-data-in-streaming). ### Window Types Windows group events into finite sets for aggregation. Flink supports several window types: - **Tumbling Windows**: Fixed-size, non-overlapping intervals (e.g., every 5 minutes) - **Sliding Windows**: Overlapping intervals that slide by a specified amount - **Session Windows**: Dynamic windows based on inactivity gaps ```java DataStream readings = ...; DataStream averages = readings .keyBy(reading -> reading.getSensorId()) .window(TumblingEventTimeWindows.of(Time.minutes(5))) .reduce((r1, r2) -> new SensorReading( r1.getSensorId(), Math.max(r1.getTimestamp(), r2.getTimestamp()), (r1.getValue() + r2.getValue()) / 2 )); ``` ## Kafka Integration and Ecosystem Connectivity Apache Kafka is the most common data source and sink for Flink applications, enabling scalable, fault-tolerant streaming pipelines. ### Flink Kafka Connector The Flink Kafka connector provides exactly-once semantics through Flink's checkpointing mechanism and Kafka's transactional producers: ```java KafkaSource kafkaSource = KafkaSource.builder() .setBootstrapServers("localhost:9092") .setTopics("events") .setGroupId("flink-consumer-group") .setValueOnlyDeserializer(new EventDeserializationSchema()) .setStartingOffsets(OffsetsInitializer.earliest()) .build(); DataStream stream = env.fromSource( kafkaSource, WatermarkStrategy.noWatermarks(), "Kafka Source" ); ``` The connector handles offset management, consumer group coordination, and automatic failover, integrating seamlessly with Flink's state management. ### Monitoring and Governance with Conduktor Production streaming applications require monitoring and governance. Conduktor provides visibility into the Kafka topics that Flink reads from and writes to, including consumer group lag and topic throughput. By [monitoring Kafka topics](https://docs.conduktor.io/guide/monitor-brokers-apps/index), teams can identify bottlenecks in the Kafka layer, track data quality, and ensure SLA compliance. Conduktor's capabilities include real-time monitoring of consumer group lag (critical for Flink applications consuming from Kafka), [schema registry integration](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry), and data quality validation on Kafka messages entering or leaving Flink pipelines. ## Practical Implementation Example The following example demonstrates a complete streaming application that reads user events from Kafka, aggregates click counts per user over 10-minute windows, and writes results to another Kafka topic: ```java public class ClickCountApplication { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.enableCheckpointing(30000); // Configure Kafka source KafkaSource source = KafkaSource.builder() .setBootstrapServers("localhost:9092") .setTopics("user-events") .setGroupId("click-count-app") .setValueOnlyDeserializer(new JsonDeserializationSchema<>(UserEvent.class)) .build(); // Define streaming pipeline DataStream events = env.fromSource( source, WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(5)) .withTimestampAssigner((event, ts) -> event.getTimestamp()), "Kafka Source" ); DataStream clickCounts = events .filter(event -> "click".equals(event.getAction())) .keyBy(UserEvent::getUserId) .window(TumblingEventTimeWindows.of(Time.minutes(10))) .aggregate(new ClickAggregateFunction()); // Write to Kafka sink KafkaSink sink = KafkaSink.builder() .setBootstrapServers("localhost:9092") .setRecordSerializer(new ClickCountSerializer("click-counts")) .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE) .build(); clickCounts.sinkTo(sink); env.execute("Click Count Application"); } } ``` This pattern is common in production environments: reading from Kafka, applying stateful transformations with windowing, and writing results back to Kafka for downstream consumption. ## 2025 Features and Modern Patterns Flink 1.18+ through 2.0+ has introduced several important enhancements to the DataStream API: - **Unified Source/Sink APIs**: The FLIP-27 Source API and FLIP-143 Sink API provide standardized interfaces with better support for exactly-once semantics, backpressure handling, and hybrid bounded/unbounded sources. - **Async State API (Flink 1.19+)**: Non-blocking state access improves throughput for stateful operations, particularly when using RocksDB or remote state backends. - **Changelog State Backend**: Materializes state changes immediately to distributed storage, reducing recovery time while maintaining performance for normal operations. - **Python DataStream API (PyFlink)**: Full-featured Python API enabling data engineers to build Flink applications in Python while leveraging the same runtime optimizations as Java/Scala applications. - **Improved Kafka Integration**: Enhanced partition discovery, better offset management, and optimized serialization for high-throughput scenarios. These features make the DataStream API more performant, easier to use, and better suited for modern cloud-native deployments. ## Summary The Flink DataStream API provides a powerful, expressive framework for building streaming applications. Key concepts include: - **DataStream abstraction** for representing continuous data flows - **Rich transformations** supporting both stateless and stateful processing - **Event time processing** with watermarks for handling out-of-order data - **Flexible windowing** for aggregating events over time intervals - **Kafka integration** for scalable, fault-tolerant data pipelines - **Modern APIs** with unified sources/sinks and async state (Flink 1.18+/2.0+) - **Kafka topic monitoring and governance** through platforms like Conduktor (consumer lag, schema compliance, data quality on Kafka topics) The combination of Flink's processing capabilities and Kafka's messaging infrastructure is the basis for most modern real-time data platforms. For alternative declarative approaches to stream processing in Flink, explore [Flink SQL and Table API for Stream Processing](https://www.conduktor.io/glossary/flink-sql-and-table-api-for-stream-processing). ## Related Concepts - [Event Time and Watermarks in Flink](https://www.conduktor.io/glossary/event-time-and-watermarks-in-flink) - Essential concepts for accurate time-based processing in the DataStream API - [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns) - Advanced join strategies and data enrichment techniques - [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) - Managing throughput and flow control in DataStream applications ## References and Further Reading - Apache Flink Documentation: DataStream API - https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/overview/ - Flink Kafka Connector Documentation - https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/ - Stream Processing with Apache Flink (O'Reilly) by Fabian Hueske and Vasiliki Kalavri - Event Time and Watermarks in Flink - https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/ Source: https://www.conduktor.io/glossary/flink-datastream-api-building-streaming-applications --- # Flink SQL and Table API for Stream Processing Flink SQL and the Table API let data engineers and analysts process unbounded streams using familiar SQL syntax or programmatic APIs in Java, Scala, and Python. This makes stream processing accessible without sacrificing the performance needed for production workloads. For a comprehensive introduction to Apache Flink's architecture and capabilities, see [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). ![Dynamic tables and continuous queries](https://www.conduktor.io/assets/images/glossary/flink-sql-and-table-api-for-stream-processing-0.webp) ## Understanding Dynamic Tables and Continuous Queries Traditional SQL operates on static tables in batch mode, queries execute once and return a result set. Flink SQL introduces a fundamentally different concept: **dynamic tables** that continuously evolve as new data arrives. A dynamic table represents a stream of data as a table abstraction. When you write a SQL query against a dynamic table, Flink creates a **continuous query** that never terminates. Instead, it continuously consumes new rows and updates its results in real time. This paradigm shift allows you to use standard SQL for stream processing without learning entirely new syntax. For example, consider an append-only log of events flowing into a table. As new events arrive, they appear as new rows. A continuous query processes these rows incrementally, producing updates to an output dynamic table. This output can be materialized to external systems like Kafka topics, Elasticsearch indices, or key-value stores. Flink supports two primary stream types: - **Append streams**: New rows are only inserted (e.g., log events, sensor readings) - **Changelog streams**: Rows can be inserted, updated, or deleted (e.g., database CDC streams). These streams emit change events with operation types: INSERT (+I), UPDATE_BEFORE (-U), UPDATE_AFTER (+U), and DELETE (-D). Understanding these stream semantics is crucial because they determine what operations are valid and how state is managed. ## Flink SQL: Writing Streaming Queries Flink SQL allows you to express stream processing logic using standard SQL syntax. The same query can run on both bounded (batch) and unbounded (streaming) datasets without modification. ### Basic Aggregation Query Here's a simple continuous query that counts employees per department from an incoming stream: ```sql SELECT dept_id, COUNT(*) as emp_count FROM employee_information GROUP BY dept_id; ``` This query maintains state for each department, updating the count as new employee records arrive. The output is a changelog stream that emits updates whenever counts change. ### Window Aggregations Windowing allows you to bound computations over time intervals. Flink SQL supports tumbling, sliding, and session windows. **Tumbling Window Example** (non-overlapping fixed intervals): ```sql SELECT TUMBLE_START(event_time, INTERVAL '1' MINUTE) as window_start, product_id, SUM(quantity) as total_quantity FROM orders GROUP BY TUMBLE(event_time, INTERVAL '1' MINUTE), product_id; ``` This query aggregates orders into one-minute non-overlapping windows, computing total quantities per product. **Sliding Window Example** (overlapping intervals): ```sql SELECT HOP_END(event_time, INTERVAL '30' SECOND, INTERVAL '1' MINUTE) as window_end, sensor_id, AVG(temperature) as avg_temp FROM sensor_readings GROUP BY HOP(event_time, INTERVAL '30' SECOND, INTERVAL '1' MINUTE), sensor_id; ``` This creates one-minute windows that slide forward every 30 seconds, providing overlapping time-based aggregations useful for smoothing time-series data. ## Table API: Programmatic Stream Processing While Flink SQL provides declarative query capabilities, the Table API offers a programmatic alternative for Java, Scala, and Python developers. The Table API sits at the same abstraction level as SQL but allows you to compose queries using code rather than SQL strings. **Java Table API Example**: ```java import org.apache.flink.table.api.*; TableEnvironment tableEnv = TableEnvironment.create(EnvironmentSettings.inStreamingMode()); // Define a source table tableEnv.executeSql( "CREATE TABLE transactions (" + " account_id STRING," + " amount DECIMAL(10, 2)," + " event_time TIMESTAMP(3)," + " WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND" + ") WITH (" + " 'connector' = 'kafka'," + " 'topic' = 'transactions'," + " 'properties.bootstrap.servers' = 'localhost:9092'" + ")" ); // Use Table API to query Table result = tableEnv.from("transactions") .groupBy($("account_id")) .select( $("account_id"), $("amount").sum().as("total_amount") ); // Convert to changelog stream (retraction/update events) for DataStream API tableEnv.toChangelogStream(result).print(); // Or materialize directly to a sink table tableEnv.executeSql("INSERT INTO output_table SELECT * FROM " + result); ``` The Table API is particularly useful when you need to: - Mix SQL with custom business logic - Build reusable query components - Work within a strongly-typed programming environment - Dynamically construct queries based on runtime conditions ### Python Table API (PyFlink) As of Flink 2.x in 2025, **PyFlink** has matured into a production-ready option for Python developers. The Python Table API mirrors the Java/Scala API while leveraging Python's ecosystem: ```python from pyflink.table import EnvironmentSettings, TableEnvironment # Create streaming table environment env_settings = EnvironmentSettings.in_streaming_mode() table_env = TableEnvironment.create(env_settings) # Define source table table_env.execute_sql(""" CREATE TABLE transactions ( account_id STRING, amount DECIMAL(10, 2), event_time TIMESTAMP(3), WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND ) WITH ( 'connector' = 'kafka', 'topic' = 'transactions', 'properties.bootstrap.servers' = 'localhost:9092', 'format' = 'json' ) """) # Use Table API result = table_env.from_path("transactions") \ .group_by(col("account_id")) \ .select(col("account_id"), col("amount").sum.alias("total")) # Execute query result.execute().print() ``` PyFlink now supports user-defined functions (UDFs), pandas integration for vectorized operations, and seamless interoperability with Python ML libraries, making it ideal for data science workflows. ## Catalogs and Metadata Management Starting with Flink 1.18+ and continuing through the 2.x releases in 2025, **Catalogs** have become central to managing table metadata across environments. Catalogs provide a unified way to register, discover, and version table definitions. Flink supports multiple catalog implementations: - **GenericInMemoryCatalog**: Default in-memory catalog for development - **HiveCatalog**: Integrates with Hive Metastore for table metadata persistence - **JdbcCatalog**: Stores metadata in PostgreSQL or other JDBC databases - **Iceberg Catalog**: Native integration with Apache Iceberg tables **Example: Using Hive Catalog for persistent table definitions**: ```java TableEnvironment tableEnv = TableEnvironment.create(EnvironmentSettings.inStreamingMode()); // Register Hive Catalog tableEnv.executeSql( "CREATE CATALOG hive_catalog WITH (" + " 'type' = 'hive'," + " 'hive-conf-dir' = '/opt/hive/conf'" + ")" ); // Use the catalog tableEnv.useCatalog("hive_catalog"); // Tables created now persist in Hive Metastore tableEnv.executeSql("CREATE TABLE my_table (id INT, name STRING) ..."); ``` Catalogs enable teams to share table definitions across Flink jobs, environments, and even with other query engines like Spark and Trino, creating a unified data layer. ## Kafka Integration and the Streaming Ecosystem Apache Kafka serves as the backbone of most streaming architectures, and Flink provides first-class integration through its Kafka SQL connector. ### Defining Kafka Tables You can create a table backed by a Kafka topic directly in SQL: ```sql CREATE TABLE user_events ( user_id STRING, event_type STRING, timestamp BIGINT, properties MAP, -- Computed column: converts Unix timestamp (ms) to TIMESTAMP type event_time AS TO_TIMESTAMP(FROM_UNIXTIME(timestamp / 1000)), -- Watermark: allows 10 seconds of out-of-order events WATERMARK FOR event_time AS event_time - INTERVAL '10' SECOND ) WITH ( 'connector' = 'kafka', 'topic' = 'user-events', 'properties.bootstrap.servers' = 'localhost:9092', 'properties.group.id' = 'flink-consumer', 'format' = 'json', 'scan.startup.mode' = 'earliest-offset' ); ``` This definition includes a computed column for event time and a watermark strategy for handling out-of-order events. ### Change Data Capture (CDC) Support Flink SQL natively supports CDC formats like Debezium, allowing you to process database changelog streams as they happen. The **Upsert Kafka** connector interprets changelog streams and maintains the latest state per key: For foundational understanding of CDC concepts and patterns, see [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). ```sql CREATE TABLE user_profiles ( user_id STRING, name STRING, email STRING, updated_at TIMESTAMP(3), PRIMARY KEY (user_id) NOT ENFORCED ) WITH ( 'connector' = 'upsert-kafka', 'topic' = 'user-profiles', 'properties.bootstrap.servers' = 'localhost:9092', 'key.format' = 'json', 'value.format' = 'json' ); ``` This setup automatically handles inserts, updates, and deletes, maintaining only the current state for each user_id. ### Governance and Visibility As streaming pipelines grow in complexity, governance becomes critical. **Conduktor** provides comprehensive visibility into Kafka topics, schemas, and data flows for production environments. When building Flink SQL applications that consume from or produce to Kafka, Conduktor allows teams to [monitor Kafka topics](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) for throughput and consumer lag, [validate schema compatibility with Schema Registry](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry), and enforce data quality policies to complement Flink's processing capabilities. ## 2025 Features: Unified Execution and Modern SQL Flink 2.1 and 2.2 (released in 2025) introduce several game-changing features that further blur the line between batch and streaming processing: ### Materialized Tables **Materialized tables** allow you to define continuously updated tables that Flink automatically refreshes: ```sql -- Define a materialized table that continuously aggregates CREATE MATERIALIZED TABLE sales_summary REFRESH = CONTINUOUS AS SELECT product_id, COUNT(*) as order_count, SUM(amount) as total_revenue FROM orders GROUP BY product_id; -- Query the materialized view like a regular table SELECT * FROM sales_summary WHERE total_revenue > 10000; ``` Flink manages the materialization automatically, handling checkpointing and state management behind the scenes. ### Delta Joins for Reduced State Traditional stream-stream joins require maintaining full state for both inputs. **Delta joins** (introduced in Flink 2.x) significantly reduce memory footprint by only storing changes: ```sql -- Delta join maintains only recent changes instead of full state SELECT o.order_id, o.amount, c.customer_name FROM orders o JOIN customers FOR SYSTEM_TIME AS OF o.order_time AS c ON o.customer_id = c.customer_id; ``` This optimization is particularly valuable for joins with large dimension tables. ### Unified Batch and Streaming Mode Flink 2.x provides true unified execution where the same SQL query seamlessly handles both bounded (historical) and unbounded (real-time) data without code changes: ```java // Same code works for both batch and streaming TableEnvironment tableEnv = TableEnvironment.create( EnvironmentSettings.newInstance() .inStreamingMode() // or .inBatchMode() .build() ); // Query works identically in both modes tableEnv.executeSql("SELECT COUNT(*) FROM orders WHERE amount > 100"); ``` The optimizer automatically chooses the most efficient execution plan based on the mode and data characteristics. ## Practical Considerations ### State Management Stateful operations like aggregations and joins require Flink to maintain state. Flink manages this state efficiently using RocksDB (an embedded key-value store) for large state backends, with automatic checkpointing for fault tolerance. Be mindful of state growth, unbounded aggregations without proper windowing can lead to ever-growing state. For detailed coverage of state backends, checkpointing strategies, and savepoint management, see [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing). ### Watermarks and Event Time For time-based operations, Flink uses watermarks to track event time progress. Watermarks handle out-of-order events by allowing some lateness before closing windows. The Kafka connector can emit per-partition watermarks, which Flink merges to determine overall stream progress. For a deep dive into watermark strategies, late data handling, and trigger mechanisms, see [Watermarks and Triggers in Stream Processing](https://www.conduktor.io/glossary/watermarks-and-triggers-in-stream-processing). ### Delivery Guarantees With checkpointing enabled, Flink's Kafka connector provides exactly-once delivery semantics. Configure the `sink.semantic` option to choose between at-least-once, exactly-once, or none, depending on your consistency requirements and performance trade-offs. ## Summary Flink SQL and the Table API democratize stream processing by providing familiar SQL interfaces and programmatic APIs for real-time data pipelines. Dynamic tables and continuous queries transform traditional SQL into a powerful stream processing paradigm, while Kafka integration connects Flink to the broader streaming ecosystem. Key takeaways: - **Flink SQL** enables declarative stream processing using standard SQL syntax with continuous queries on dynamic tables - **Dynamic tables** evolve continuously, supporting append and changelog semantics with clear operation types - **Table API** provides programmatic query composition for Java, Scala, and Python (PyFlink) - **Catalogs** (Flink 1.18+) enable persistent metadata management across Hive, JDBC, and Iceberg - **Kafka connectors** integrate seamlessly with CDC streams, upsert patterns, and Schema Registry - **2025 features** include materialized tables, delta joins for reduced state, and unified batch/streaming execution - **State management** and **watermarks** are essential for correctness and performance For data engineers building real-time pipelines, Flink SQL offers a balance between ease of use and sophisticated stream processing capabilities. For data analysts familiar with SQL, it provides a path to work directly with streaming data without learning low-level APIs. PyFlink extends these capabilities to the Python ecosystem with pandas integration and ML library support. The latest Flink releases (2.1 and 2.2 in 2025) add features like materialized tables, delta joins for reduced state, and AI model integration. ## Related Concepts - [Real-Time Analytics with Streaming Data](https://www.conduktor.io/glossary/real-time-analytics-with-streaming-data) - Building analytics dashboards and metrics with Flink SQL - [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns) - Implementing complex join patterns in Flink SQL and Table API - [Handling Late-Arriving Data in Streaming](https://www.conduktor.io/glossary/handling-late-arriving-data-in-streaming) - Configuring watermarks and late data handling in Flink SQL DDL ## Sources and References - [Apache Flink Table API & SQL Documentation](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/overview/) - [Continuous Queries on Dynamic Tables - Apache Flink Blog](https://flink.apache.org/2017/03/30/continuous-queries-on-dynamic-tables/) - [Flink SQL Demo: Building an End-to-End Streaming Application](https://flink.apache.org/2020/07/28/flink-sql-demo-building-an-end-to-end-streaming-application/) - [Apache Flink Kafka SQL Connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/) - [Apache Flink 2.2.0 Release Announcement](https://flink.apache.org/2025/12/04/apache-flink-2.2.0-advancing-real-time-data--ai-and-empowering-stream-processing-for-the-ai-era/) Source: https://www.conduktor.io/glossary/flink-sql-and-table-api-for-stream-processing --- # Flink State Management and Checkpointing Apache Flink maintains stateful computations with strong fault tolerance guarantees. Understanding state management and checkpointing is essential for building reliable streaming applications that recover from failures without data loss or duplication. For a broader overview of Flink's architecture and capabilities, see [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). ## Understanding State in Flink State in Flink represents information about past events that influences the processing of future events. Flink supports two primary types of state: **Keyed State** is associated with a specific key and is accessible only by operations on the same key. This is the most common form of state in streaming applications. Flink provides several keyed state primitives: - `ValueState`: Stores a single value per key - `ListState`: Stores a list of elements per key - `MapState`: Stores a map of key-value pairs per key - `ReducingState`: Stores a single value representing aggregation results - `AggregatingState`: Similar to ReducingState but with different input/output types **Operator State** (or non-keyed state) is associated with parallel instances of operators rather than with specific keys. This is commonly used for source operators that need to track partition offsets or maintain buffered data. State serves critical functions in streaming applications: maintaining session windows, tracking user interactions over time, computing running aggregations, and implementing complex event processing patterns. Without proper state management, streaming applications would be limited to stateless transformations. For practical implementation details, see [Flink DataStream API: Building Streaming Applications](https://www.conduktor.io/glossary/flink-datastream-api-building-streaming-applications). ## State Backends: Storage and Retrieval Flink abstracts state storage through configurable state backends, which determine how state is stored locally during computation, and checkpoint storage, which determines how state is persisted during checkpoints. Starting with Flink 1.13, this separation provides clearer configuration and better flexibility. ![flink-state-management-and-checkpointing diagram 1](https://www.conduktor.io/assets/images/glossary/flink-state-management-and-checkpointing-0.webp) - **HashMapStateBackend** (introduced in Flink 1.13, replacing MemoryStateBackend) stores state as Java objects in the TaskManager's JVM heap. It provides fast state access but is limited by available heap memory. Suitable for state sizes up to hundreds of megabytes and development/testing environments. - **EmbeddedRocksDBStateBackend** (updated API in Flink 1.13+, replacing RocksDBStateBackend) uses an embedded RocksDB instance to store state on local disk. This backend supports state sizes larger than available memory and provides incremental checkpointing, making it the preferred choice for production applications with large state requirements (multi-gigabyte state). RocksDB serializes state objects, which adds CPU overhead but enables practically unlimited state size. The checkpoint storage is configured separately and determines where completed checkpoints are persisted: JobManager memory (for testing only), distributed file systems (HDFS, S3, Azure Blob Storage, Google Cloud Storage), or other durable storage systems. Configuration example for modern Flink 1.13+ state backend (Java): ```java import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend; import org.apache.flink.runtime.state.storage.FileSystemCheckpointStorage; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // Configure RocksDB state backend with incremental checkpoints EmbeddedRocksDBStateBackend rocksDBBackend = new EmbeddedRocksDBStateBackend(); rocksDBBackend.setIncrementalCheckpointsEnabled(true); env.setStateBackend(rocksDBBackend); // Configure checkpoint storage separately env.getCheckpointConfig().setCheckpointStorage( new FileSystemCheckpointStorage("s3://my-bucket/checkpoints") ); // Enable checkpointing with 60-second interval env.enableCheckpointing(60000); ``` Configuration example using YAML (flink-conf.yaml): ```yaml # State backend configuration (Flink 1.13+) state.backend: rocksdb state.backend.incremental: true # Checkpoint storage configuration state.checkpoints.dir: s3://my-bucket/checkpoints state.savepoints.dir: s3://my-bucket/savepoints # RocksDB-specific tuning state.backend.rocksdb.localdir: /tmp/rocksdb ``` State backend selection depends on state size requirements, recovery time objectives, and available infrastructure. Small state (< 100 MB) fits in HashMapStateBackend, while multi-gigabyte state requires EmbeddedRocksDBStateBackend with incremental checkpointing enabled. ## Checkpointing: The Fault Tolerance Mechanism Checkpointing is Flink's primary mechanism for fault tolerance. A checkpoint represents a consistent snapshot of the entire application state and stream positions across all operators. If a failure occurs, Flink restarts the application from the most recent successful checkpoint. Flink implements the Chandy-Lamport algorithm for distributed snapshots, modified to work efficiently with streaming data. This algorithm ensures all operators capture their state at logically consistent points in the data stream, preventing partial processing or duplication. The process works as follows: ![Flink implements the Chandy-Lamport algorithm for distributed snapshots, modified to work efficiently with streaming data. This algorithm ensures all operators capture their state at logically consistent points in the data stream, preventing partial processing or duplication. The process works as follows](https://www.conduktor.io/assets/images/glossary/flink-state-management-and-checkpointing-1.webp) 1. The JobManager triggers a checkpoint by injecting special barrier markers into the data streams 2. Barriers flow through the topology with the data stream 3. When an operator receives barriers from all input channels, it snapshots its state to the configured checkpoint storage 4. The operator then forwards the barrier to downstream operators 5. Once all operators acknowledge the checkpoint, it's marked as complete Checkpointing configuration requires careful tuning: ```yaml # flink-conf.yaml execution.checkpointing.interval: 60000 # Checkpoint every 60 seconds execution.checkpointing.mode: EXACTLY_ONCE execution.checkpointing.timeout: 600000 # 10 minute timeout execution.checkpointing.max-concurrent-checkpoints: 1 execution.checkpointing.min-pause: 30000 # Minimum 30s between checkpoints state.backend.incremental: true # Enable incremental checkpoints for RocksDB ``` Checkpoint intervals represent a trade-off: shorter intervals reduce potential data reprocessing after failure but increase overhead and storage costs. Production applications typically checkpoint every 30-120 seconds, depending on data volume and recovery time requirements. The **exactly-once** processing semantics guarantee that each record affects the final results exactly once, even in the presence of failures. This requires coordinating checkpoints with external systems (sources and sinks) that support transactional or idempotent writes. For more on exactly-once guarantees in Kafka, see [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). ## Savepoints: Versioned Application State While checkpoints are automatic and lightweight, savepoints are manually triggered, versioned snapshots of application state. Savepoints enable operational workflows that checkpoints cannot: - **Application upgrades**: Deploy new application logic while maintaining existing state - **Cluster migration**: Move applications between Flink clusters - **Flink version upgrades**: Update the Flink runtime version - **A/B testing**: Run different application versions with identical initial state - **Time travel**: Revert to a known good state after detecting data quality issues Creating and using savepoints: ```bash # Trigger a savepoint ./bin/flink savepoint [targetDirectory] # Start job from savepoint ./bin/flink run -s myApplication.jar # Dispose of old savepoint ./bin/flink savepoint -d ``` Savepoints are larger and slower than checkpoints because they use a canonical, version-stable format designed for long-term storage. Applications should trigger savepoints before planned maintenance rather than relying on automatic checkpoints. ## Integration with Kafka and the Streaming Ecosystem Flink's state management integrates deeply with Apache Kafka through coordinated checkpointing. When Flink checkpoints successfully, it commits Kafka consumer offsets transactionally, ensuring exactly-once processing end-to-end. The Kafka source connector participates in Flink's checkpointing by: 1. Storing current partition offsets as operator state 2. Committing offsets to Kafka only after checkpoint completion 3. Restoring offsets from state upon recovery This coordination prevents message loss and duplication: ```java KafkaSource source = KafkaSource.builder() .setBootstrapServers("broker:9092") .setTopics("events") .setGroupId("flink-consumer") .setStartingOffsets(OffsetsInitializer.committedOffsets()) .setValueOnlyDeserializer(new EventDeserializer()) .build(); DataStream events = env.fromSource( source, WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10)), "Kafka Source" ); ``` Similarly, the Kafka sink provides exactly-once guarantees through two-phase commit protocol, coordinating with Flink's checkpointing. Transactions are committed only when checkpoints complete successfully. For organizations managing complex streaming infrastructures, governance platforms like Conduktor provide visibility into the Kafka topics used in Flink pipelines. [Monitor Kafka topics](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) to track consumption lag, offset commits, and topic throughput to identify data flow issues on the Kafka side before they impact production systems. [Schema Registry integration](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry) ensures event format compatibility across your Flink pipelines. For comprehensive lag monitoring strategies, see [Consumer Lag Monitoring and Alerting](https://www.conduktor.io/glossary/consumer-lag-monitoring-and-alerting) and [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ## Production Considerations and Best Practices Successful production deployments require attention to several operational aspects: - **State Size Monitoring**: Track state growth over time using Flink's metrics system. Unbounded state growth often indicates logic errors (improper cleanup of old keys, window state not being evicted). Set up alerts when state size grows unexpectedly. Monitor metrics such as `state.backend.rocksdb.size` and `checkpoint.duration` to identify issues early. - **Checkpoint Alignment**: Unaligned checkpoints (introduced in Flink 1.11, stabilized in 1.13+) reduce backpressure impact during checkpointing by allowing barriers to overtake buffered data. Enable for high-throughput applications where checkpoint alignment causes slowdowns: ```yaml execution.checkpointing.unaligned: true execution.checkpointing.aligned-checkpoint-timeout: 60s # Fallback to unaligned after 60s ``` **State TTL (Time-To-Live)**: Configure state time-to-live to automatically clean up expired entries, preventing unbounded state growth: ```java StateTtlConfig ttlConfig = StateTtlConfig .newBuilder(Time.days(7)) .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite) .setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired) .build(); ValueStateDescriptor descriptor = new ValueStateDescriptor<>("state", String.class); descriptor.enableTimeToLive(ttlConfig); ``` **Incremental Checkpointing**: For RocksDB backend, enable incremental checkpoints to avoid uploading full state snapshots. This dramatically reduces checkpoint duration and network I/O for large state: ```yaml state.backend.incremental: true ``` Incremental checkpoints only upload changes since the last checkpoint, reducing checkpoint time from minutes to seconds for multi-gigabyte state. **Generalized Log-Based Incremental Checkpoints**: Flink 1.18+ introduced generalized incremental checkpoints (FIP-306) that work with any state backend, not just RocksDB. This enables incremental checkpointing for HashMapStateBackend, combining fast heap access with efficient checkpointing: ```yaml execution.checkpointing.incremental: true dstl.dstl.enabled: true # Distributed State Transaction Log ``` **Changelog State Backend**: Flink 1.15+ introduced and Flink 1.18+ stabilized the changelog state backend, which materializes state changes immediately to distributed storage. This reduces recovery time from minutes to seconds at the cost of higher I/O during normal operation. As of 2025, this is production-ready for latency-sensitive applications: ```yaml state.backend.changelog.enabled: true state.backend.changelog.storage: filesystem # or memory ``` **Retention Policies**: Configure checkpoint retention to keep completed checkpoints even after job cancellation, enabling state recovery for debugging or reprocessing scenarios: ```yaml execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION execution.checkpointing.max-retained-checkpoints: 3 ``` ## Summary Flink's state management and checkpointing system provides the foundation for fault-tolerant stateful stream processing. Keyed and operator state enable complex computations across time, while configurable state backends balance performance and scalability requirements. Checkpointing implements the Chandy-Lamport algorithm to create consistent distributed snapshots with exactly-once semantics, and savepoints enable operational flexibility through versioned state snapshots. Production deployments require careful configuration of checkpoint intervals, state backends (typically RocksDB with incremental checkpointing), and integration with external systems like Kafka. Monitoring state size, checkpoint duration, and alignment metrics helps identify issues before they cause outages. For broader Flink architectural decisions, see [Flink vs Spark Streaming: When to Choose Each](https://www.conduktor.io/glossary/flink-vs-spark-streaming-when-to-choose-each) and [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink). Understanding these mechanisms is what makes Flink reliable for mission-critical streaming applications that must maintain correctness even when failures occur. For testing strategies to validate your state management implementation, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). ## Related Concepts - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - How Flink coordinates with Kafka for end-to-end exactly-once processing - [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive) - Understanding the transactional guarantees that enable exactly-once Kafka sinks - [Handling Late-Arriving Data in Streaming](https://www.conduktor.io/glossary/handling-late-arriving-data-in-streaming) - Strategies for managing late data with checkpointing and state ## Sources and References - Apache Flink Documentation: [State Backends](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/state/state_backends/) - Apache Flink Documentation: [Checkpointing](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/fault-tolerance/checkpointing/) - Apache Flink Documentation: [Savepoints](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/state/savepoints/) - Carbone, P., et al. (2017): "State Management in Apache Flink" - VLDB Conference - Apache Flink Documentation: [Kafka Connector](https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/) Source: https://www.conduktor.io/glossary/flink-state-management-and-checkpointing --- # Flink vs Kafka Streams: When to Choose **Apache Flink** is a distributed stream processing framework with its own cluster runtime — you submit jobs to a Flink cluster (JobManager + TaskManagers). **Kafka Streams** is a Java library you embed directly in your application — no separate cluster, no job submission, just a JAR. The core difference: Flink is infrastructure you operate; Kafka Streams is code you ship. ## TL;DR | Dimension | Apache Flink | Kafka Streams | |---|---|---| | Deployment model | Cluster runtime (JobManager + TaskManagers) | Library embedded in your application | | Language support | Java, Scala, Python, SQL | Java (Scala wrapper deprecated since 4.3) | | Kafka dependency | Optional (any source/sink) | Mandatory (reads/writes Kafka only) | | State backend | RocksDB, HashMap, custom | RocksDB (local), changelog topics for durability | | Fault tolerance | Checkpoints to durable storage (S3, HDFS) | Changelog topics in Kafka | | Event-time semantics | Full: watermarks, late data handling, timers | Full: event-time windowing, grace periods | | Windowing | Tumbling, sliding, session, global, custom | Tumbling, hopping, session, sliding (KS 3.x) | | SQL / Table API | Yes (Flink SQL, Table API) | No native SQL (ksqlDB is separate) | | Batch processing | Unified (Flink handles batch and stream) | No — streaming only | | Scaling model | Task-level parallelism set at submission | Partition-based, scales with app instances | | Operational complexity | High (cluster management, checkpoints) | Low (standard JVM service operations) | | Ecosystem | Broad (non-Kafka sources/sinks) | Kafka-centric | ## What is Apache Flink? Apache Flink is a distributed stateful stream processing framework. You write a processing job (Java, Scala, Python, or SQL), package it, and submit it to a running Flink cluster. The cluster consists of a **JobManager** (coordinator: scheduling, checkpointing, failure recovery) and one or more **TaskManagers** (workers: execute processing tasks). State is stored in a configurable backend — typically RocksDB for large state, HashMap for small in-memory state. Fault tolerance uses periodic checkpoints: Flink snapshots state to durable external storage (S3, HDFS, GCS) so jobs can recover from exactly where they left off without reprocessing. See [What is Apache Flink? Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) for a deeper treatment. ## What is Kafka Streams? Kafka Streams is a Java library (part of the Apache Kafka project since 0.10) for building stateful stream processing applications that read from and write to [Kafka topics](https://www.conduktor.io/glossary/kafka-topics-explained). A Scala wrapper exists but was deprecated in Kafka 4.3 and is targeted for removal in Kafka 5.0. There is no separate cluster: you add the `kafka-streams` dependency to your application and deploy instances like any other microservice. Kafka handles partition assignment and rebalancing across running instances. Local state is stored in RocksDB; durability comes from **changelog topics** in Kafka — if an instance crashes and restarts (possibly on a different host), it rebuilds local state by replaying the changelog. See [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams) and [State Stores in Kafka Streams](https://www.conduktor.io/glossary/state-stores-in-kafka-streams) for detailed coverage. ## Architecture compared ### Cluster-managed vs embedded library **Flink**: Requires a running Flink cluster before any job can execute. In Kubernetes environments, this typically means a Flink Operator (e.g., Apache Flink Kubernetes Operator) managing JobManager and TaskManager pods. Jobs are submitted via REST API or CLI. Multiple jobs can share a cluster (session mode) or each job can have a dedicated cluster (application mode). You get centralized job management, a web UI, and cluster-level metrics. **Kafka Streams**: No cluster to manage. Your application IS the processing cluster. Deploy multiple instances of your app; Kafka Streams uses Kafka's [consumer group](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) protocol to distribute partitions across instances automatically. Scale out: run more instances. Scale in: stop some instances; Kafka rebalances within minutes. No separate web UI — observability is through your application's metrics and Kafka's consumer group metrics. ### State management **Flink state backends:** - `RocksDBStateBackend`: state stored on TaskManager local disk, checkpointed to remote storage. Handles very large state (terabytes) that doesn't fit in memory. Suited for joins with large lookup tables, long-window aggregations. - `HashMapStateBackend`: state stored in JVM heap, checkpointed to remote storage. Fast for small-to-medium state but limited by TaskManager memory. - Checkpoints are full or incremental snapshots to S3/HDFS/GCS. On failure, Flink restores from the last successful checkpoint and reprocesses events since that checkpoint. **Kafka Streams state stores:** - RocksDB by default, stored on the application instance's local disk. - Each state store has a corresponding **changelog topic** in Kafka. Every write to local state is mirrored to the changelog topic. - On failure/restart, Kafka Streams rebuilds state by replaying the changelog. For large state, this standby replica concept (configurable via `num.standby.replicas`) pre-warms state on standby instances to minimize recovery time. - State is co-partitioned: state store partitions align with input topic partitions, which simplifies joins but constrains topology design. ### Event-time semantics Both support full event-time processing with watermarks: **Flink**: WatermarkStrategy assigns watermarks to events; the JobManager propagates watermarks across operators. Flink's event-time model handles out-of-order events, configurable late data strategies (drop, side output, or allow), and triggers. See [Event Time and Watermarks in Flink](https://www.conduktor.io/glossary/event-time-and-watermarks-in-flink) and [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing). **Kafka Streams**: Timestamps come from Kafka record metadata (`CreateTime` or `LogAppendTime`). Stream time advances as records arrive; late records are handled with configurable grace periods per window. Kafka Streams does not use Flink-style explicit watermarks — windowing advances based on stream time derived from record timestamps, with punctuation callbacks (wall-clock or stream-time) for processing-time triggers. ### SQL and batch Flink provides a unified Table API and Flink SQL that works on both streaming and batch data. You can write a single SQL query that runs on a bounded dataset (batch) or an unbounded stream. This is powerful for teams with SQL expertise and mixed batch/stream workloads. See [Flink SQL and Table API](https://www.conduktor.io/glossary/flink-sql-and-table-api-for-stream-processing). Kafka Streams has no built-in SQL. ksqlDB (a separate product from Confluent) adds SQL over Kafka Streams, but it is its own deployment and operational surface. ### Sources and sinks **Flink**: Source/sink agnostic. Flink connectors exist for Kafka, Kinesis, Pulsar, Cassandra, JDBC, Elasticsearch, S3, Hudi, Iceberg, Delta Lake, and many others. Flink is commonly used in non-Kafka architectures. **Kafka Streams**: Kafka-only. Input must come from Kafka topics; output must go to Kafka topics. External lookups are possible via `GlobalKTable` (backed by a topic) or custom state store implementations, but the primary data flow is Kafka ↔ Kafka Streams ↔ Kafka. ## Operational trade-offs **Flink advantages:** - Handles very large state (terabytes) via RocksDB + checkpoints to object storage - Cluster-level resource management and multi-job orchestration - SQL support for analyst-friendly pipeline development - Source/sink agnostic — works with any data system, not just Kafka - Unified batch and stream processing in one framework **Flink disadvantages:** - Requires cluster infrastructure (JobManager, TaskManagers, checkpoint storage, cluster monitoring) - Higher operational burden: job versioning, checkpoint management, TaskManager memory tuning - Savepoints (for schema evolution or job upgrades) add operational ceremony - Harder to debug locally — requires a local Flink cluster or MiniCluster test harness **Kafka Streams advantages:** - Zero additional infrastructure — deploy like any microservice - Fault tolerance via Kafka changelog — no external storage dependency - Simple horizontal scaling: add instances, Kafka rebalances - Easier local development and testing (EmbeddedKafka or TestTopologyDriver) - Natural fit for microservices architectures where each service owns its processing **Kafka Streams disadvantages:** - Kafka-only: cannot consume from or produce to non-Kafka systems natively - No SQL interface — topology must be defined in Java/Scala code - State store rebuild from changelog can be slow for large state on cold start (mitigated by standbys) - No batch processing support ## When to choose Flink - Your pipeline consumes from or writes to **non-Kafka sources/sinks** (JDBC, S3, Kinesis, Iceberg, etc.) - You need **very large state** that exceeds what comfortably fits on a Kafka Streams instance's disk - You require **SQL-based pipeline authoring** for analyst-driven transformations - You have **batch and stream workloads** that benefit from a unified execution engine - Your organization already runs Flink for other workloads (shared cluster amortizes cost) - You need **complex event patterns** (CEP) via Flink's CEP library ## When to choose Kafka Streams - Your entire pipeline is **Kafka ↔ Kafka** — no external sources or sinks needed - You want **zero infrastructure overhead** — embed processing in your microservice - Your team is **Java-first** and prefers library APIs over framework deployments - Your state is **moderate in size** (up to tens of GB per partition, manageable on instance disk) - You are building a **microservice** that owns both its business logic and its stream processing - **Local development simplicity** is important — TestTopologyDriver makes unit testing easy ## Migration considerations - Porting a Kafka Streams topology to Flink (or vice versa) requires rewriting the processing code — the APIs are different (Flink DataStream API vs Kafka Streams DSL). - State migration is the hard part: Kafka Streams state lives in changelog topics (accessible via Kafka); Flink state lives in checkpoints on object storage (opaque binary format). There is no tooling for direct state migration between the two. - If you need SQL, migrating from Kafka Streams to Flink SQL is the more common direction — Kafka Streams topology is translated into Flink SQL Table API definitions. See also: the existing [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink) page for additional architectural depth. For a hands-on Kafka Streams walkthrough, see the [Learn Kafka Streams course](https://www.conduktor.io/kafka-streams). **Do I need a Kafka cluster to use Flink?** No. Flink is source/sink agnostic. You can use Flink with Kafka as one of many connectors, but Flink also works with Kinesis, Pulsar, JDBC, S3, and more. Kafka Streams, by contrast, requires Kafka — it reads from and writes to Kafka topics exclusively. **Can Kafka Streams handle large state?** Yes, but with caveats. Kafka Streams uses RocksDB for local state, so state size is bounded by instance disk. For very large state (hundreds of GB to terabytes), recovery from changelog topics can be slow unless standby replicas are configured. Flink with RocksDB + checkpoints to object storage scales more gracefully for massive state. **Is Flink harder to operate than Kafka Streams?** Yes, meaningfully so. Flink requires a cluster (JobManager + TaskManagers), checkpoint storage configuration, job lifecycle management (submit, cancel, upgrade, savepoint), and cluster-level monitoring. Kafka Streams is a library — you operate it like any other JVM application, with no additional infrastructure. **Which has better exactly-once guarantees?** Both support exactly-once semantics. Kafka Streams achieves EOS via Kafka transactions (producer + consumer in a single transaction per commit interval). Flink achieves exactly-once internal state via checkpoints; end-to-end exactly-once with Kafka sinks uses a two-phase commit protocol coordinated with Flink checkpoints — both the source and sink must support it. See [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) for the Kafka side of this story. Source: https://www.conduktor.io/glossary/flink-vs-kafka-streams --- # GDPR Compliance for Data Teams: Navigating Privacy in Modern Data Architectures The General Data Protection Regulation (GDPR) has changed how organizations handle personal data. For data teams working with streaming architectures and distributed systems, compliance presents technical challenges that go beyond traditional database management. - **Who Does GDPR Apply To?** GDPR applies to any organization that processes personal data of EU residents, regardless of where the organization is located. This includes both "data controllers" (who determine purposes and means of processing) and "data processors" (who process data on behalf of controllers). - **What Are the Penalties for Non-Compliance?** GDPR violations can result in fines of up to €20 million or 4% of annual global turnover, whichever is higher. Beyond financial penalties, non-compliance can lead to reputational damage, loss of customer trust, and operational disruptions. As of 2025, enforcement has intensified, with regulators actively monitoring AI/ML data usage. This article explores practical strategies and technical implementations for achieving GDPR compliance in data-intensive streaming environments. ![GDPR compliance workflow for streaming data systems](https://www.conduktor.io/assets/images/glossary/gdpr-compliance-for-data-teams-0.webp) **Key GDPR Timelines for Data Teams:** - **30 days**: Maximum time to fulfill data subject access requests (DSARs) - **72 hours**: Required notification window for data breaches to supervisory authorities - **1 month**: Default timeframe to respond to data subject requests (erasure, rectification, etc.) - **Without undue delay**: Obligation to inform data subjects of breaches when they're at high risk ## Understanding GDPR's Core Principles for Data Teams GDPR establishes seven foundational principles that data teams must embed into their technical architecture: lawfulness, fairness, and transparency; purpose limitation; data minimization; accuracy; storage limitation; integrity and confidentiality; and accountability. These principles translate into specific technical requirements that affect every layer of your data infrastructure. The principle of **data minimization** requires teams to collect only what is necessary for specific purposes. In practice, this means implementing schema validation and filtering mechanisms at ingestion points. For streaming platforms like Apache Kafka, this might involve deploying data governance tools that enforce field-level policies before messages reach downstream consumers. For guidance on detecting and handling sensitive data in streams, see [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams) and [PII Leakage Prevention](https://www.conduktor.io/glossary/pii-leakage-prevention). **Storage limitation** demands that personal data be retained only as long as necessary. Data teams must implement automated retention policies with configurable time-to-live (TTL) settings across all storage layers, from streaming platforms to data warehouses and analytics databases. ## Technical Implementation of Data Subject Rights GDPR grants individuals eight fundamental rights regarding their personal data. For data teams, the most technically challenging are the right to access, right to rectification, right to erasure (the "right to be forgotten"), and right to data portability. ### Right to Access and Portability When a data subject requests access to their personal data, your systems must be capable of identifying and retrieving all records across distributed systems within the GDPR-mandated 30-day window. This requires: - **Unified identity management**: Implement a consistent user identifier schema across all systems to enable efficient data retrieval - **Data catalog and lineage tracking**: Maintain comprehensive metadata about where personal data resides and how it flows through your pipeline. For detailed coverage of data cataloging, see [What is a Data Catalog: Modern Data Discovery](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery) - **Export mechanisms**: Build automated processes to extract, format, and deliver data in machine-readable formats (typically JSON or CSV) ### Right to Erasure: The Streaming Challenge The right to be forgotten presents particular challenges in streaming architectures where data is immutable by design. Apache Kafka, for instance, uses append-only logs that cannot be modified retroactively without compromising the integrity of the event stream. Data teams have several architectural options to address this: **Tombstone Records**: A tombstone is a special message with a null value that signals a deletion. When published to Kafka topics with a specific key (e.g., user ID), it marks that key's data for removal. Downstream consumers must implement logic to honor these tombstones and filter out deleted records during processing. Here's a practical implementation for handling the right to erasure in streaming systems: ```python from kafka import KafkaProducer, KafkaConsumer import json from datetime import datetime class DataDeletionManager: def __init__(self, bootstrap_servers): self.producer = KafkaProducer( bootstrap_servers=bootstrap_servers, value_serializer=lambda v: json.dumps(v).encode('utf-8') if v else None ) self.deletion_topic = 'user-deletion-requests' def process_deletion_request(self, user_id): """Process right to erasure request for a user.""" # Step 1: Publish tombstone to user data topics self.producer.send( 'user-events', key=user_id.encode('utf-8'), value=None # Tombstone record ) # Step 2: Record deletion event for audit trail deletion_event = { 'user_id': user_id, 'timestamp': datetime.utcnow().isoformat(), 'requested_by': 'data_subject', 'status': 'initiated' } self.producer.send(self.deletion_topic, value=deletion_event) # Step 3: Delete from databases and data warehouses self._delete_from_databases(user_id) # Step 4: Remove PII from separate PII store self._delete_pii_mapping(user_id) self.producer.flush() return {'user_id': user_id, 'status': 'deletion_initiated'} def _delete_from_databases(self, user_id): """Delete user data from all databases.""" # Example: PostgreSQL deletion # NOTE: Use environment variables or secure credential management (e.g., AWS Secrets Manager, HashiCorp Vault) import psycopg2 import os conn = psycopg2.connect( dbname=os.getenv('DB_NAME'), user=os.getenv('DB_USER'), password=os.getenv('DB_PASSWORD'), host=os.getenv('DB_HOST') ) cursor = conn.cursor() cursor.execute("DELETE FROM user_profiles WHERE user_id = %s", (user_id,)) cursor.execute("DELETE FROM user_events WHERE user_id = %s", (user_id,)) conn.commit() conn.close() def _delete_pii_mapping(self, user_id): """Remove PII mapping for pseudonymization.""" # Delete from PII vault/store pass class TombstoneAwareConsumer: def __init__(self, bootstrap_servers, topic): self.consumer = KafkaConsumer( topic, bootstrap_servers=bootstrap_servers, value_deserializer=lambda m: json.loads(m.decode('utf-8')) if m else None ) self.deleted_users = set() def process_messages(self): """Process messages while respecting tombstones.""" for message in self.consumer: if message.value is None: # Tombstone detected - record deleted user user_id = message.key.decode('utf-8') self.deleted_users.add(user_id) print(f"User {user_id} has been deleted - skipping all future events") else: # Regular event - check if user is deleted user_id = message.value.get('user_id') if user_id not in self.deleted_users: self._process_event(message.value) else: print(f"Skipping event for deleted user {user_id}") def _process_event(self, event): """Process valid event.""" pass # Usage deletion_mgr = DataDeletionManager(['localhost:9092']) deletion_mgr.process_deletion_request('user-123') ``` - **Data Pseudonymization**: Store personal identifiable information (PII) separately from event data, using tokenized references. When deletion is requested, remove the PII mapping while preserving anonymized event history for analytics. - **Log Compaction with Key-Based Deletion**: Configure Kafka topics with log compaction enabled. When a deletion request arrives, publish a tombstone record with the user's identifier as the key, allowing Kafka to eventually remove all records for that key. For detailed coverage of log compaction mechanics, see [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained). - **Policy Enforcement Layer**: Data governance platforms like Conduktor provide capabilities that enable teams to implement deletion policies, [data masking](https://docs.conduktor.io/guide/conduktor-in-production/admin/data-mask), and field-level encryption across Kafka clusters. Conduktor's policy enforcement features can intercept, filter, and transform messages based on compliance rules without requiring changes to producer or consumer applications, making GDPR compliance enforcement more manageable at scale. ## Consent Management in Streaming Systems GDPR requires explicit, informed consent before processing personal data for specific purposes. Data teams must implement consent as a first-class attribute in their data models. In streaming architectures, consent should be treated as event data itself. When a user grants or revokes consent, publish a consent event to a dedicated topic. Downstream processors can then join stream data with the latest consent state to determine whether processing is lawful. Consider this approach: 1. **Consent Events Topic**: Maintain a compacted topic containing the latest consent preferences for each user 2. **Stream Enrichment**: Join data streams with consent state before processing 3. **Conditional Processing**: Implement stream processors that route data based on consent status, processing consented data normally while quarantining or dropping non-consented data Here's a practical example of consent event publishing and validation: ```python from kafka import KafkaProducer import json from datetime import datetime class ConsentManager: def __init__(self, bootstrap_servers): self.producer = KafkaProducer( bootstrap_servers=bootstrap_servers, value_serializer=lambda v: json.dumps(v).encode('utf-8') ) self.consent_topic = 'user-consent-events' def record_consent(self, user_id, purposes, granted=True): """Record user consent for specific data processing purposes.""" consent_event = { 'user_id': user_id, 'timestamp': datetime.utcnow().isoformat(), 'purposes': purposes, # e.g., ['marketing', 'analytics'] 'granted': granted, 'consent_version': '1.0' } # Use user_id as key for log compaction self.producer.send( self.consent_topic, key=user_id.encode('utf-8'), value=consent_event ) self.producer.flush() def revoke_consent(self, user_id, purposes): """Revoke user consent for specific purposes.""" self.record_consent(user_id, purposes, granted=False) # Usage consent_mgr = ConsentManager(['localhost:9092']) consent_mgr.record_consent('user-123', ['marketing', 'analytics'], granted=True) ``` Stream processors can then validate consent before processing personal data: ```python from kafka import KafkaConsumer import json class ConsentAwareProcessor: def __init__(self, bootstrap_servers): # Consumer for consent state (compacted topic) self.consent_consumer = KafkaConsumer( 'user-consent-events', bootstrap_servers=bootstrap_servers, value_deserializer=lambda m: json.loads(m.decode('utf-8')), auto_offset_reset='earliest' ) # Build consent state cache self.consent_cache = {} self._load_consent_state() def _load_consent_state(self): """Load latest consent state for all users.""" for message in self.consent_consumer: user_id = message.key.decode('utf-8') consent_data = message.value self.consent_cache[user_id] = consent_data def can_process(self, user_id, purpose): """Check if we have consent to process data for given purpose.""" consent = self.consent_cache.get(user_id) if not consent: return False return consent.get('granted', False) and purpose in consent.get('purposes', []) def process_event(self, event): """Process event only if consent is granted.""" user_id = event.get('user_id') if self.can_process(user_id, 'analytics'): # Process the event return self._perform_analysis(event) else: # Drop or quarantine the event return None ``` ## Data Protection by Design and Default Article 25 of GDPR requires "data protection by design and default," meaning privacy considerations must be integrated into systems from the earliest development stages. For data teams, this translates into: - **Encryption at Rest and in Transit**: All personal data must be encrypted using industry-standard algorithms (AES-256 for storage, TLS 1.3 for transmission) - **Field-Level Encryption**: Encrypt specific PII fields while leaving non-sensitive data plaintext for analytics. This enables you to maintain data utility while protecting privacy. - **Key Management**: Enterprise GDPR compliance requires robust key management using services like AWS Key Management Service (KMS), Azure Key Vault, Google Cloud KMS, or HashiCorp Vault. These services provide key rotation, access auditing, and centralized control essential for regulatory compliance. Never hardcode encryption keys in application code. Here's an implementation example for field-level encryption in streaming data: ```python from cryptography.fernet import Fernet from kafka import KafkaProducer, KafkaConsumer import json import base64 class FieldEncryptor: def __init__(self, encryption_key=None): """Initialize with encryption key or generate new one.""" if encryption_key: self.key = encryption_key else: self.key = Fernet.generate_key() self.cipher = Fernet(self.key) def encrypt_fields(self, data, sensitive_fields): """Encrypt specified fields in the data dictionary.""" encrypted_data = data.copy() for field in sensitive_fields: if field in encrypted_data: # Encrypt the field value plaintext = str(encrypted_data[field]).encode('utf-8') encrypted_value = self.cipher.encrypt(plaintext) encrypted_data[field] = base64.b64encode(encrypted_value).decode('utf-8') # Mark field as encrypted for downstream processing encrypted_data[f'{field}_encrypted'] = True return encrypted_data def decrypt_fields(self, data, sensitive_fields): """Decrypt specified fields in the data dictionary.""" decrypted_data = data.copy() for field in sensitive_fields: if field in decrypted_data and decrypted_data.get(f'{field}_encrypted'): # Decrypt the field value encrypted_value = base64.b64decode(decrypted_data[field]) plaintext = self.cipher.decrypt(encrypted_value) decrypted_data[field] = plaintext.decode('utf-8') del decrypted_data[f'{field}_encrypted'] return decrypted_data # Usage in Kafka producer encryptor = FieldEncryptor() producer = KafkaProducer( bootstrap_servers=['localhost:9092'], value_serializer=lambda v: json.dumps(v).encode('utf-8') ) # User event with PII user_event = { 'user_id': 'user-123', 'email': 'user@example.com', 'name': 'John Doe', 'activity': 'page_view', 'timestamp': datetime.utcnow().isoformat() + 'Z' } # Encrypt sensitive fields before publishing sensitive_fields = ['email', 'name'] encrypted_event = encryptor.encrypt_fields(user_event, sensitive_fields) producer.send('user-events', value=encrypted_event) ``` - **Role-Based Access Control (RBAC)**: Implement granular permissions that restrict access to personal data based on job function and necessity. Modern data governance platforms provide fine-grained access controls at the topic, consumer group, and even field level. For comprehensive guidance on implementing access controls in Kafka, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) and [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). - **Data Masking and Redaction**: Automatically mask or redact sensitive fields when data moves to non-production environments or when accessed by roles without appropriate clearance. ## Audit Trails and Accountability GDPR's accountability principle requires organizations to demonstrate compliance through comprehensive documentation. Data teams must implement: - **Access Logging**: Record every access to personal data, including who accessed it, when, what data was accessed, and for what purpose. For implementation patterns, see [Streaming Audit Logs](https://www.conduktor.io/glossary/streaming-audit-logs) - **Processing Records**: Maintain detailed records of all data processing activities, including data sources, purposes, categories of recipients, and retention periods - **Schema Evolution Tracking**: Document all changes to data structures that contain personal data, ensuring you can trace how data models evolved over time - **Automated Compliance Reports**: Build dashboards and reporting mechanisms that provide real-time visibility into compliance posture, including retention policy adherence, consent rates, and data subject request fulfillment metrics ## 2025 GDPR Enforcement: AI/ML and Automation As of 2025, GDPR enforcement has evolved to address modern data processing challenges, particularly around artificial intelligence and machine learning systems. ### AI/ML Data Processing Requirements When training machine learning models on personal data, GDPR requires: - **Explicit Consent for AI Training**: Data subjects must explicitly consent to their data being used for AI/ML model training, separate from other processing purposes - **Model Explainability**: Organizations must be able to explain automated decision-making processes to data subjects (Article 22 - Right to Explanation) - **Training Data Lineage**: Maintain comprehensive records of which personal data was used to train which models and when - **Model Retraining After Deletion**: When users exercise the right to be forgotten, organizations may need to retrain models that were trained on their data Here's a practical approach to tracking consent for AI/ML processing: ```python class MLConsentManager: def __init__(self, bootstrap_servers): self.producer = KafkaProducer( bootstrap_servers=bootstrap_servers, value_serializer=lambda v: json.dumps(v).encode('utf-8') ) self.ml_consent_topic = 'ml-training-consent' def record_ml_consent(self, user_id, model_purpose, granted=True): """Record consent specifically for ML model training.""" consent_event = { 'user_id': user_id, 'timestamp': datetime.utcnow().isoformat(), 'model_purpose': model_purpose, # e.g., 'fraud_detection', 'recommendation_engine' 'granted': granted, 'consent_version': '2.0', 'ai_training': True # Flag for AI/ML specific consent } self.producer.send( self.ml_consent_topic, key=user_id.encode('utf-8'), value=consent_event ) self.producer.flush() def log_training_data_usage(self, model_id, user_ids): """Track which user data was used to train which models.""" training_event = { 'model_id': model_id, 'timestamp': datetime.utcnow().isoformat(), 'user_count': len(user_ids), 'user_ids': user_ids[:100], # Sample for audit purposes 'event_type': 'model_training' } self.producer.send('ml-audit-trail', value=training_event) self.producer.flush() ``` ### Data Subject Access Request (DSAR) Automation The 30-day window for fulfilling data subject access requests (DSARs) has driven automation requirements in 2025: - **Automated Data Discovery**: Tools that automatically scan data stores to locate personal data across distributed systems - **Self-Service DSAR Portals**: User-facing interfaces where data subjects can submit requests and track progress - **Orchestrated Deletion Workflows**: Automated workflows that ensure deletion propagates across all systems (streaming platforms, databases, backups, and ML models) - **Compliance Monitoring**: Real-time dashboards tracking DSAR fulfillment rates and identifying bottlenecks Modern data governance platforms can integrate with streaming platforms to enforce deletion policies and data masking at the infrastructure level, supporting the technical components of DSAR fulfillment workflows. ## Building a Compliant Streaming Architecture Modern data teams working with streaming platforms must architect for compliance from the ground up. A compliant streaming architecture typically includes: 1. **Ingestion Layer**: Data validation, schema enforcement, and initial consent verification 2. **Governance Layer**: Policy enforcement, data classification, and transformation rules 3. **Processing Layer**: Stream processors that honor consent, implement retention policies, and respect data subject rights 4. **Storage Layer**: Encrypted, access-controlled data stores with automated retention and deletion 5. **Audit Layer**: Comprehensive logging and monitoring of all data access and processing activities ## Conclusion GDPR compliance for data teams is not just a legal checkbox. Treating privacy as a core architectural requirement — rather than an afterthought — produces better systems that respect user rights while delivering business value. The technical challenges are real, particularly around implementing the right to erasure in immutable event streams and managing consent across distributed systems. With careful architectural planning and the right tooling for governance enforcement, data teams can build systems that are both GDPR-compliant and technically sound. ## Related Concepts - [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams) - Automated detection and protection of personal data in streaming systems to support GDPR compliance. - [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming) - Pseudonymization and anonymization techniques required for GDPR Article 32 compliance. - [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) - Comprehensive audit trails for demonstrating GDPR accountability and tracking data subject requests. ## Sources and References 1. **European Union GDPR Official Text** - [EUR-Lex - General Data Protection Regulation](https://eur-lex.europa.eu/eli/reg/2016/679/oj) - The official legal text of the GDPR, providing authoritative guidance on all articles and requirements. 2. **Information Commissioner's Office (ICO) - Guide to GDPR** - [ICO GDPR Guide](https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/) - Comprehensive guidance from the UK's data protection authority on implementing GDPR requirements. 3. **Apache Kafka Documentation - Security and GDPR** - [Kafka Security Documentation](https://kafka.apache.org/documentation/#security) - Technical documentation on implementing security, encryption, and compliance features in Apache Kafka. 4. **NIST Special Publication 800-53** - [Security and Privacy Controls](https://csrc.nist.gov/publications/detail/sp/800-53/rev-5/final) - US government framework for information security controls applicable to GDPR compliance. 5. **European Data Protection Board - Guidelines on Data Protection by Design and by Default** - [EDPB Guidelines](https://edpb.europa.eu/our-work-tools/general-guidance/guidelines-recommendations-best-practices_en) - Official guidance on implementing Article 25 requirements for data protection by design. Source: https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams --- # Great Expectations: Data Testing Framework > **Important**: This article covers Great Expectations 1.0+ (released August 2024), which introduced breaking API changes from previous versions. If you're using GX 0.x, see the [official migration guide](https://docs.greatexpectations.io/docs/0.18/reference/learn/migration_guide/). Data quality problems tend to surface only after they've already hit downstream systems or business decisions. Great Expectations (GX) addresses this by providing a Python-based framework for testing, documenting, and profiling your data pipelines. ![Great Expectations validation workflow architecture](https://www.conduktor.io/assets/images/glossary/great-expectations-data-testing-framework-0.webp) ## What is Great Expectations? Great Expectations is an open-source data validation framework that enables data teams to express what they "expect" from their data through assertions called Expectations. Think of it as unit testing for your data, instead of testing code behavior, you're testing data quality, schema compliance, and business logic. The framework goes beyond simple validation by generating data documentation, maintaining data quality metrics over time, and integrating seamlessly into modern data workflows. ## Getting Started Install Great Expectations and initialize a project: ```bash pip install great_expectations great_expectations init # Creates project structure ``` ## Core Concepts ### Expectations Expectations are declarative assertions about your data. They're the building blocks of data quality tests. Great Expectations provides over 300 built-in Expectations covering common validation scenarios: ```python import great_expectations as gx # Initialize a Data Context context = gx.get_context() # Set up a pandas datasource with fluent API data_source = context.data_sources.add_pandas("pandas_datasource") data_asset = data_source.add_dataframe_asset(name="customer_data") batch_definition = data_asset.add_batch_definition_whole_dataframe("customer_batch") # Create an Expectation Suite suite = context.suites.add( gx.core.expectation_suite.ExpectationSuite(name="customer_suite") ) # Add expectations to the suite suite.add_expectation( gx.expectations.ExpectTableRowCountToBeBetween( min_value=1000, max_value=1000000 ) ) suite.add_expectation( gx.expectations.ExpectColumnValuesToNotBeNull(column="customer_id") ) suite.add_expectation( gx.expectations.ExpectColumnValuesToBeUnique(column="customer_id") ) suite.add_expectation( gx.expectations.ExpectColumnValuesToBeInSet( column="status", value_set=["active", "inactive", "pending"] ) ) ``` ### Batch Definitions **Batch Definitions** specify how to access your data. They replaced the batch_request pattern from GX 0.x: ```python # For a CSV file csv_asset = data_source.add_csv_asset( name="customer_csv", filepath_or_buffer="customer_data.csv" ) batch_definition = csv_asset.add_batch_definition_whole_file("customer_batch") # For dataframes (in-memory) df_asset = data_source.add_dataframe_asset(name="customer_df") batch_definition = df_asset.add_batch_definition_whole_dataframe("df_batch") ``` ### Data Assistants (Auto-Profiling) Data Assistants automatically generate Expectation Suites by analyzing your data: ```python from great_expectations.rule_based_profiler.data_assistant import OnboardingDataAssistant # Get a batch for profiling batch = batch_definition.get_batch() # Use OnboardingDataAssistant to auto-generate expectations data_assistant = OnboardingDataAssistant( name="onboarding_assistant", batch_request=batch.batch_request ) # Generate suite result = data_assistant.run() suite = result.get_expectation_suite() ``` ### Validation Definitions and Checkpoints **Validation Definitions** link a Batch Definition to an Expectation Suite. **Checkpoints** then execute one or more validations: ```python # Create a Validation Definition validation_definition = context.validation_definitions.add( gx.core.validation_definition.ValidationDefinition( name="customer_validation", data=batch_definition, suite=suite, ) ) # Create a Checkpoint checkpoint = context.checkpoints.add( gx.checkpoint.checkpoint.Checkpoint( name="customer_checkpoint", validation_definitions=[validation_definition] ) ) # Run validation checkpoint_result = checkpoint.run() # Check results if checkpoint_result.success: print("✅ All validations passed!") else: print("❌ Validation failed:") for result in checkpoint_result.run_results.values(): for validation_result in result["validation_result"].results: if not validation_result.success: print(f" - {validation_result.expectation_config.type}") print(f" {validation_result.result}") ``` **Data Docs**: Validation results automatically generate HTML documentation showing data quality metrics, trends, and validation history. These docs serve as living documentation of your data contracts. ## Batch Data Validation For traditional batch processing pipelines, Great Expectations integrates with data warehouses, lakes, and processing frameworks using the fluent SQL datasource API: ```python # PostgreSQL database validation datasource = context.data_sources.add_postgres( name="postgres_datasource", connection_string="postgresql://user:password@localhost:5432/database" ) # Add a table asset table_asset = datasource.add_table_asset( name="orders", table_name="orders" ) # Create batch definition for the entire table batch_definition = table_asset.add_batch_definition_whole_table("orders_batch") # Create expectations for the orders table orders_suite = context.suites.add( gx.core.expectation_suite.ExpectationSuite(name="orders_suite") ) orders_suite.add_expectation( gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id") ) orders_suite.add_expectation( gx.expectations.ExpectColumnValuesToBeUnique(column="order_id") ) # Create validation definition and checkpoint validation_def = context.validation_definitions.add( gx.core.validation_definition.ValidationDefinition( name="orders_validation", data=batch_definition, suite=orders_suite, ) ) orders_checkpoint = context.checkpoints.add( gx.checkpoint.checkpoint.Checkpoint( name="orders_checkpoint", validation_definitions=[validation_def] ) ) # Run validation result = orders_checkpoint.run() ``` ## Streaming Data Validation Modern data architectures increasingly rely on streaming data. Great Expectations can validate streaming data by integrating with Apache Kafka, Kinesis, or other streaming platforms. ### Kafka Integration Here's how to validate streaming data from Kafka using GX 1.0+: ```python from kafka import KafkaConsumer, KafkaProducer import json import pandas as pd import great_expectations as gx import logging logger = logging.getLogger(__name__) # Initialize GX context context = gx.get_context() # Set up dataframe datasource for streaming validation data_source = context.data_sources.add_pandas("streaming_pandas") df_asset = data_source.add_dataframe_asset(name="streaming_events") batch_definition = df_asset.add_batch_definition_whole_dataframe("streaming_batch") # Create expectation suite for streaming events streaming_suite = context.suites.add( gx.core.expectation_suite.ExpectationSuite(name="streaming_events_suite") ) streaming_suite.add_expectation( gx.expectations.ExpectColumnValuesToNotBeNull(column="customer_id") ) streaming_suite.add_expectation( gx.expectations.ExpectColumnValuesToBeInSet( column="event_type", value_set=["page_view", "purchase", "signup"] ) ) # Kafka consumer setup consumer = KafkaConsumer( 'customer-events', bootstrap_servers=['localhost:9092'], value_deserializer=lambda m: json.loads(m.decode('utf-8')), auto_offset_reset='latest', enable_auto_commit=False, # Manual commit after successful validation ) # Dead letter queue producer for failed validations dlq_producer = KafkaProducer( bootstrap_servers=['localhost:9092'], value_serializer=lambda m: json.dumps(m).encode('utf-8') ) def handle_validation_failure(messages, checkpoint_result): """Send failed messages to dead letter queue with error details""" for result in checkpoint_result.run_results.values(): validation_result = result["validation_result"] if not validation_result.success: failure_details = { "messages": messages, "validation_errors": [ { "expectation": r.expectation_config.type, "result": str(r.result) } for r in validation_result.results if not r.success ], "statistics": validation_result.statistics } dlq_producer.send('data-quality-dlq', value=failure_details) logger.error(f"Validation failed: {validation_result.statistics}") # Streaming validation function def validate_streaming_batch(messages): """Validate a micro-batch of streaming messages""" df = pd.DataFrame(messages) # Get batch with dataframe batch = batch_definition.get_batch(batch_parameters={"dataframe": df}) # Create validation definition validation_def = context.validation_definitions.add( gx.core.validation_definition.ValidationDefinition( name=f"streaming_validation_{pd.Timestamp.now().timestamp()}", data=batch_definition, suite=streaming_suite, ) ) # Create and run checkpoint checkpoint = context.checkpoints.add( gx.checkpoint.checkpoint.Checkpoint( name=f"streaming_checkpoint_{pd.Timestamp.now().timestamp()}", validation_definitions=[validation_def] ) ) checkpoint_result = checkpoint.run() if not checkpoint_result.success: handle_validation_failure(messages, checkpoint_result) return checkpoint_result.success # Process streaming data in micro-batches batch_size = 100 message_buffer = [] try: for message in consumer: message_buffer.append(message.value) if len(message_buffer) >= batch_size: success = validate_streaming_batch(message_buffer) if success: # Commit offset after successful validation consumer.commit() # Clear buffer regardless of validation result message_buffer = [] finally: consumer.close() dlq_producer.close() ``` ## Custom Expectations For domain-specific validation logic, you can create custom Expectations. The `ColumnMapExpectation` base class pattern remains available in GX 1.0+: ```python from great_expectations.expectations.expectation import ColumnMapExpectation class ExpectColumnValuesToBeValidEmail(ColumnMapExpectation): """Expect column values to be valid email addresses""" map_metric = "column_values.match_regex" success_keys = ("mostly", "regex") default_kwarg_values = { "regex": r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$", "mostly": 1.0, } # Usage suite.add_expectation( ExpectColumnValuesToBeValidEmail(column="email_address") ) ``` > **Note**: Custom expectation development patterns may have evolved in GX 1.0+. Consult the [official custom expectations documentation](https://docs.greatexpectations.io/docs/core/customize_expectations/overview/) for the latest implementation guidance. ## Best Practices 1. **Start Simple**: Begin with basic Expectations like null checks and uniqueness constraints before adding complex validations. 2. **Version Control**: Store Expectation Suites in version control alongside your data pipeline code. 3. **Incremental Adoption**: Implement GX incrementally, starting with critical datasets. 4. **Monitor Trends**: Use Data Docs to track data quality metrics over time and identify degradation patterns. 5. **Fail Fast**: Configure validation workflows to halt pipelines on critical validation failures. Use conditional logic based on `checkpoint_result.success`. 6. **Performance in Streaming**: For high-throughput streams, consider: - **Sampling**: Validate every Nth message instead of every message - **Time windows**: Batch messages over fixed time intervals (e.g., validate every 10 seconds) - **Adaptive validation**: Increase validation frequency when anomalies are detected 7. **Integration with Orchestration**: Integrate GX with orchestration tools like Airflow, Prefect, or Dagster for production deployments. Use the `airflow-provider-great-expectations` package for Airflow integration. ## Conclusion Great Expectations turns data quality from a reactive debugging exercise into a proactive testing discipline. By defining Expectations, creating validation suites, and integrating checks into batch and streaming pipelines, data teams can catch issues early and reduce time spent on data quality incidents. Whether you're validating nightly batch loads in Snowflake or real-time events streaming through Kafka, Great Expectations gives you the framework to ensure your data meets the standards your business depends on. ## Related Concepts - [dbt Tests and Data Quality Checks](https://www.conduktor.io/dbt-tests-and-data-quality-checks) - Complementary testing approach within transformation pipelines - [Automated Data Quality Testing](https://www.conduktor.io/automated-data-quality-testing) - Broader automation strategies for data quality - [Data Contracts for Reliable Pipelines](https://www.conduktor.io/data-contracts-for-reliable-pipelines) - Defining and enforcing data quality expectations ## Sources - [Great Expectations Official Documentation](https://docs.greatexpectations.io/) - [GX Core 1.0+ Quick Start](https://docs.greatexpectations.io/docs/core/introduction/try_gx/) - [GX 0.x to 1.0 Migration Guide](https://docs.greatexpectations.io/docs/0.18/reference/learn/migration_guide/) - [Great Expectations GitHub Repository](https://github.com/great-expectations/great_expectations) - [Great Expectations Changelog](https://docs.greatexpectations.io/docs/core/changelog/) - [Custom Expectations Documentation](https://docs.greatexpectations.io/docs/core/customize_expectations/overview/) Source: https://www.conduktor.io/glossary/great-expectations-data-testing-framework --- # Handling Late-Arriving Data in Streaming Dealing with data that arrives later than expected is one of the harder aspects of stream processing. Unlike batch processing where you have a complete dataset before computation begins, streaming systems must decide when to produce results even as new data continues to arrive. Understanding late-arriving data is essential for building accurate and reliable streaming applications. ## Understanding Event Time vs Processing Time The root of the late data problem lies in the difference between two fundamental time concepts in streaming systems. ![handling-late-arriving-data-in-streaming diagram 1](https://www.conduktor.io/assets/images/glossary/handling-late-arriving-data-in-streaming-0.webp) - **Event time** is when an event actually occurred in the real world. For example, when a sensor reading was taken, when a user clicked a button, or when a financial transaction was initiated. This timestamp is typically embedded in the event payload itself. - **Processing time** is when the streaming system processes the event. This is when your Kafka consumer reads the message or when your Flink job processes the record. In an ideal world, events would arrive in perfect event time order, and processing time would closely match event time. In reality, network delays, system outages, mobile devices going offline, clock skew across distributed systems, and various other factors cause events to arrive out of order or significantly delayed. Consider an IoT deployment where thousands of sensors monitor industrial equipment. Sensors in areas with poor connectivity might buffer readings locally and transmit them in batches when connectivity improves. A temperature reading from 10 minutes ago might arrive after readings from 5 minutes ago, creating challenges for time-based aggregations and alerts. Late-arriving data is particularly challenging for session windows, which dynamically adjust their boundaries based on event timing. For detailed coverage of session-based processing, see [Session Windows in Stream Processing](https://www.conduktor.io/glossary/session-windows-in-stream-processing). ## Common Causes of Late-Arriving Data Late data occurs for many reasons in distributed systems: - **Network issues** are the most common cause. Temporary connectivity problems, packet loss, or routing issues can delay message delivery. Mobile applications are particularly susceptible, as devices move between networks or lose connectivity entirely. - **System failures and retries** also contribute to late data. When a producer crashes and restarts, it might resend events that were already processed by downstream consumers. Kafka's at-least-once delivery guarantees mean duplicate or delayed messages are possible. - **Distributed clock skew** can cause events to appear late even when they arrive promptly. If different systems have clocks that are out of sync, an event timestamped on one server might appear to be from the past or future when processed on another server. - **Data corrections and updates** represent intentional late data. Financial systems often receive transaction adjustments or corrections hours or days after the initial transaction. These late corrections must be incorporated into historical calculations. ## Strategies for Handling Late Data Modern stream processing frameworks provide several mechanisms for handling late-arriving data while maintaining result accuracy. ### Watermarks Watermarks are the primary mechanism for tracking event time progress in streaming systems. A watermark is a special signal that indicates "all events with timestamps less than T have been seen." More precisely, a watermark represents a heuristic estimate of event time progress. ![handling-late-arriving-data-in-streaming diagram 2](https://www.conduktor.io/assets/images/glossary/handling-late-arriving-data-in-streaming-1.webp) For example, a watermark of 10:15:00 suggests that the system has likely received all events with timestamps before 10:15:00. When a window computation spans 10:00:00 to 10:15:00, the system can trigger the computation and produce results once the watermark advances past 10:15:00. Watermarks are necessarily heuristic because stream processing systems cannot know with certainty whether more late data will arrive. The watermark strategy must balance completeness (waiting longer to include more late data) against latency (producing results quickly). Apache Flink provides flexible watermark strategies including periodic watermarks generated at regular intervals and punctuated watermarks based on special markers in the stream. Since Flink 1.18, the WatermarkStrategy API has been further simplified with built-in strategies like `forBoundedOutOfOrderness()` and `forMonotonousTimestamps()`, making it easier to handle common late data patterns. Kafka Streams automatically tracks stream time as the maximum timestamp seen so far, with Kafka Streams 3.7+ offering improved grace period semantics for more predictable late data handling. ### Allowed Lateness Even with watermarks, some data will inevitably arrive after the watermark has passed. The allowed lateness parameter defines a grace period during which late data is still incorporated into computations. When a window closes based on the watermark, the system doesn't immediately discard the window state. Instead, it keeps the window open for the allowed lateness period. Late events arriving during this grace period trigger recomputation and emission of updated results. For instance, with an allowed lateness of 5 minutes, an event with timestamp 10:12:00 arriving at 10:21:00 (after the 10:15:00 watermark but within the grace period) would still update the 10:00-10:15 window and emit a corrected result. ### Side Outputs and Late Data Streams Some data arrives so late that it falls outside even the allowed lateness window. Rather than silently dropping this data, modern frameworks allow routing it to side outputs or separate late data streams. This pattern enables multiple strategies for handling extremely late data: - Log late data for auditing and debugging - Store late data in a separate system for batch reprocessing - Trigger alerts when late data volumes exceed thresholds - Merge late data results in downstream systems that can handle corrections ## Implementation in Streaming Frameworks Different streaming platforms provide various approaches to handling late data. **Apache Flink** offers comprehensive event time support with customizable watermark generators, allowed lateness configuration, and side outputs for late data. Flink's DataStream API makes event time processing explicit, requiring developers to specify timestamp extraction and watermark generation strategies. For foundational understanding of Flink's architecture and state management, see [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). For detailed coverage of windowing strategies in Flink, see [Windowing in Apache Flink: Tumbling, Sliding, and Session Windows](https://www.conduktor.io/glossary/windowing-in-apache-flink-tumbling-sliding-and-session-windows). Here's a practical example of handling late data in Flink 1.18+: ```java DataStream events = // ... source .assignTimestampsAndWatermarks( WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofMinutes(5)) .withTimestampAssigner((event, timestamp) -> event.getEventTime()) ); SingleOutputStreamOperator results = events .keyBy(Event::getUserId) .window(TumblingEventTimeWindows.of(Time.minutes(15))) .allowedLateness(Time.minutes(5)) .sideOutputLateData(lateDataTag) .aggregate(new EventAggregator()); // Access late data that arrived beyond allowed lateness DataStream lateEvents = results.getSideOutput(lateDataTag); ``` **Kafka Streams** provides built-in support for windowed operations with configurable grace periods. The framework automatically manages watermarks based on stream time and allows specifying how long windows remain open for late data. Kafka Streams 3.7+ improved grace period behavior to be more consistent and predictable. The framework also supports suppression operators that can wait for late data before emitting final results. For an introduction to Kafka Streams fundamentals, see [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). Example with Kafka Streams 3.7+: ```java KStream events = builder.stream("events"); events .groupByKey() .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(15)) .grace(Duration.ofMinutes(5))) // Grace period for late data .aggregate( () -> new EventAggregate(), (key, event, aggregate) -> aggregate.add(event), Materialized.as("events-store") ) .suppress(Suppressed.untilWindowCloses(unbounded())) // Wait for late data .toStream() .to("results"); ``` **Apache Beam** (and cloud services like Google Dataflow) pioneered many late data handling concepts, including the separation of event time and processing time, watermarks, and triggers. Beam's model allows precise control over when results are materialized and how late data updates are handled. For deeper understanding of watermark mechanics and triggering strategies, refer to [Watermarks and Triggers in Stream Processing](https://www.conduktor.io/glossary/watermarks-and-triggers-in-stream-processing). ## Monitoring and Observability Effective late data handling requires good observability. Key metrics to track include: - **Watermark lag** measures the difference between current event time (watermark position) and current processing time. Growing watermark lag indicates the system is falling behind in processing events. - **Late event count and latency** tracks how many events arrive after watermarks and how late they are. Sudden increases might indicate upstream problems or changing data patterns. - **Window trigger frequency** for recomputation shows how often late data causes windows to re-emit results. Excessive recomputation might indicate watermark strategies need tuning. Monitoring event timestamp distributions across topics can reveal producers with clock skew or connectivity problems. **Modern monitoring tools** like Conduktor provide [visibility into Kafka topics](https://docs.conduktor.io/guide/monitor-brokers-apps/index), including consumer lag monitoring and throughput metrics. Tracking consumer lag on the Kafka topics that Flink reads from helps identify when processing falls behind and late data patterns are emerging. For production deployments, tools like Prometheus and Grafana integrate well with Flink and Kafka Streams to expose late data metrics through standard metric endpoints. ## 2025 Advances in Late Data Handling Modern stream processing frameworks have significantly improved late data handling capabilities: - **Flink 1.18+ improvements** include more efficient watermark alignment across parallel operators, reducing memory overhead for windows with long allowed lateness periods. The new `WatermarkStrategy.forGenerator()` API provides fine-grained control over watermark generation logic, including custom strategies that adapt to changing data patterns. - **Kafka Streams 3.7+ enhancements** provide clearer semantics around grace periods, with better documentation of when windows close and state is purged. The `ofSizeAndGrace()` method makes it explicit when windows will no longer accept late data, improving predictability for downstream consumers. - **Adaptive watermarking** is an emerging pattern where systems dynamically adjust watermark delays based on observed late data patterns. Instead of static watermark strategies, machine learning models can predict optimal watermark delays based on historical latency patterns, time of day, and data source characteristics. - **Exactly-once semantics** have become more robust in handling late data. Both Flink and Kafka Streams now provide stronger guarantees that late data updates are processed exactly once, even in the presence of failures and reprocessing. ## Best Practices When designing systems to handle late-arriving data, consider these guidelines: - **Choose appropriate watermark strategies** based on your data characteristics. If events arrive mostly in order with occasional stragglers, aggressive watermarks with moderate allowed lateness work well. For highly out-of-order data, more conservative watermarks or longer grace periods may be necessary. - **Set allowed lateness based on business requirements**, not arbitrary values. Analyze your actual late data patterns and balance accuracy needs against state management costs. Longer allowed lateness means more state to maintain. - **Design downstream systems to handle updates**. Since late data can trigger result recomputation, consumers of your streaming results must gracefully handle updated values for previously computed time windows. - **Monitor and alert on late data patterns**. Sudden changes in late data frequency or severity often indicate infrastructure problems, upstream failures, or data quality issues that require investigation. - **Test with realistic late data scenarios**. Many streaming application bugs only appear when processing out-of-order or late-arriving data. Include late data scenarios in your testing strategy. Tools like Conduktor Gateway (a Kafka proxy) can simulate network delays and out-of-order message delivery to test late data handling in controlled environments. For comprehensive testing approaches, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). ## Summary Handling late-arriving data is a fundamental challenge in stream processing that distinguishes it from batch processing. The difference between event time and processing time, combined with the realities of distributed systems, means data will inevitably arrive out of order or delayed. Modern streaming frameworks provide robust mechanisms for handling late data through watermarks, allowed lateness, and side outputs. Watermarks enable event time-based computations by providing heuristic estimates of stream progress. Allowed lateness offers a grace period for incorporating late events into computations. Side outputs ensure extremely late data isn't lost. As of 2025, frameworks like Flink 1.18+ and Kafka Streams 3.7+ have significantly improved late data handling with better APIs, clearer semantics, and more efficient implementations. Tools like Conduktor provide comprehensive monitoring to understand late data patterns in production, while Conduktor Gateway enables realistic testing of late data scenarios. Successful late data handling requires understanding your data patterns, choosing appropriate watermark strategies, configuring reasonable allowed lateness based on business requirements, and maintaining good observability. Designing systems that explicitly account for late-arriving data — rather than ignoring it — produces streaming applications that are both timely and accurate. ## Related Concepts - [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) - Framework with sophisticated watermark and late data handling - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Streaming platform providing offset-based replay capabilities - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Processing guarantees when handling late data ## Sources and References - Akidau, Tyler, et al. "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing." VLDB Endowment, 2015. [https://research.google/pubs/pub43864/](https://research.google/pubs/pub43864/) - Apache Flink Documentation. "Event Time and Watermarks." [https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/](https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/) - Akidau, Tyler. "Streaming 101: The world beyond batch." O'Reilly Radar, 2015. [https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/](https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/) - Apache Kafka Documentation. "Kafka Streams - Time Concepts." [https://kafka.apache.org/documentation/streams/](https://kafka.apache.org/documentation/streams/) - Kleppmann, Martin. "Designing Data-Intensive Applications." O'Reilly Media, 2017. Chapter 11: Stream Processing. Source: https://www.conduktor.io/glossary/handling-late-arriving-data-in-streaming --- # Healthcare Data Streaming Use Cases Healthcare organizations are adopting real-time data streaming architectures to improve patient outcomes and operational efficiency. Unlike traditional batch processing systems that analyze data hours or days after collection, streaming platforms like Apache Kafka 4.0+ and Apache Flink process healthcare data as it's generated, enabling immediate insights and faster responses to critical events. Modern streaming platforms have matured significantly for healthcare workloads. Apache Kafka 4.0+ operates in KRaft mode (removing ZooKeeper dependencies for simplified operations), offers tiered storage for long-term retention of patient data, and provides enhanced security features critical for HIPAA compliance. For detailed coverage of KRaft architecture, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). This shift addresses real challenges: managing data from thousands of connected medical devices, coordinating care across multiple systems, and detecting life-threatening conditions before they become emergencies. ![Healthcare streaming architecture with device integration and analytics](https://www.conduktor.io/assets/images/glossary/healthcare-data-streaming-use-cases-0.webp) ## Real-Time Patient Monitoring and Critical Care Intensive care units (ICUs) generate continuous streams of patient data from ventilators, cardiac monitors, infusion pumps, and other medical devices. Each device may produce hundreds of data points per second, including vital signs, medication dosages, and equipment status. Data streaming platforms consume these device feeds in real time, applying stream processing logic to detect anomalies, trigger alerts, and update clinical dashboards. For example, a streaming pipeline might correlate heart rate variability with blood pressure trends to identify early signs of septic shock, alerting clinicians minutes or hours before traditional monitoring systems would flag the issue. Apache Kafka topics can organize patient data by ward, device type, or clinical priority, while stream processors like Kafka Streams or Apache Flink apply complex event processing rules. These systems must handle high throughput with low latency, delays of even a few seconds can impact critical care decisions. ## Medical Device Integration and IoT The proliferation of connected medical devices, from bedside monitors to implantable sensors and consumer wearables, creates massive volumes of streaming health data. Remote patient monitoring programs track chronic conditions like diabetes and heart disease by continuously ingesting data from glucose meters, blood pressure cuffs, and smartwatches. These IoT ecosystems rely on event-driven architectures where devices publish measurements to streaming platforms. The data flows through validation and enrichment stages before being routed to clinical systems, analytics platforms, or long-term storage. For comprehensive coverage of IoT streaming patterns, protocols, and edge processing strategies, see [IoT Data Streaming Architectures](https://www.conduktor.io/glossary/iot-data-streaming-architectures). A typical architecture might use MQTT or HTTP for device ingestion, with an Apache Kafka cluster serving as the central event backbone. Stream processors normalize data formats, filter noise, and detect threshold violations. For instance, a streaming application might monitor a diabetic patient's continuous glucose monitor (CGM) and automatically alert care teams when blood sugar levels exceed safe ranges for more than 15 minutes. ## Clinical Decision Support and Predictive Analytics Clinical decision support systems (CDSS) leverage streaming data to provide real-time recommendations at the point of care. By analyzing current patient state alongside historical patterns and clinical guidelines, these systems help clinicians make more informed decisions. Stream processing enables predictive models that continuously score patient risk. A sepsis prediction model might consume lab results, vital signs, and medication data in real time, recalculating risk scores every time new information arrives. When risk crosses a threshold, the system can trigger care protocols, order additional tests, or notify rapid response teams. Machine learning models deployed in streaming pipelines must handle concept drift, the phenomenon where the statistical properties of the target variable change over time, meaning patient populations and disease patterns evolve, requiring models to adapt. For example, a sepsis prediction model trained on pre-pandemic data may need retraining to account for COVID-19 complications. Streaming architectures support online learning and model retraining by capturing prediction outcomes and feeding them back into training pipelines. For implementation patterns, see [Real-Time ML Pipelines](https://www.conduktor.io/glossary/real-time-ml-pipelines) and [Model Drift in Streaming](https://www.conduktor.io/glossary/model-drift-in-streaming). ## Healthcare Interoperability and Data Exchange Healthcare data exists in silos across electronic health records (EHRs), laboratory information systems (LIS), picture archiving systems (PACS), and countless other specialized applications. Achieving interoperability requires real-time data exchange using standards like HL7 v2, HL7 FHIR, and DICOM. Streaming platforms serve as integration backbones that mediate between these heterogeneous systems. When a patient visits the emergency department, their record might trigger a cascade of events: retrieving medical history from the EHR, checking medication lists from the pharmacy system, and pulling recent imaging from PACS, all happening in seconds rather than minutes. HL7 FHIR's event-driven design aligns naturally with streaming architectures. FHIR subscriptions allow systems to publish resource changes (new lab results, updated medications, revised diagnoses) to Kafka topics, where downstream consumers can react in real time. Schema registries become critical for managing the evolution of FHIR resource definitions and ensuring compatibility across system versions. For comprehensive coverage of schema management strategies, versioning, and compatibility rules, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). Here's an example of a Kafka producer publishing a FHIR patient resource update: ```python from kafka import KafkaProducer import json # Initialize producer with encryption producer = KafkaProducer( bootstrap_servers=['kafka-broker:9093'], security_protocol='SSL', ssl_cafile='/path/to/ca-cert', ssl_certfile='/path/to/client-cert', ssl_keyfile='/path/to/client-key', value_serializer=lambda v: json.dumps(v).encode('utf-8') ) # FHIR Patient resource update event fhir_patient_update = { "resourceType": "Patient", "id": "patient-12345", "meta": { "versionId": "2", "lastUpdated": "2025-12-08T10:30:00Z" }, "identifier": [{ "system": "http://hospital.example.org/patients", "value": "MRN-987654" }], "name": [{ "family": "Smith", "given": ["John"] }], "telecom": [{ "system": "phone", "value": "+1-555-0100", "use": "mobile" }], "birthDate": "1985-03-15", "address": [{ "line": ["123 Main St"], "city": "Boston", "state": "MA", "postalCode": "02101" }] } # Publish to Kafka topic producer.send( 'fhir.patient.updates', key=b'patient-12345', value=fhir_patient_update ) producer.flush() ``` ## Compliance, Security, and Data Governance Healthcare data streaming must comply with strict regulations including HIPAA in the United States, GDPR in Europe, and various regional privacy laws. These requirements affect every aspect of streaming architecture, from encryption and access controls to audit logging and data retention. For comprehensive guidance on GDPR requirements for data teams, see [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams). Data in motion requires end-to-end encryption using TLS for network transmission and encryption at rest for data stored in Kafka topics. Access control lists (ACLs) limit which applications can produce to or consume from specific topics containing protected health information (PHI). For detailed security implementation patterns, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) and [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). Audit trails must capture who accessed what data and when. Streaming platforms generate their own audit events, creating meta-streams (streams about streams), dedicated Kafka topics that track data lineage, schema changes, and consumer access patterns. These audit streams feed compliance dashboards and support forensic investigation when breaches occur. For detailed implementation patterns for audit logging in streaming platforms, see [Streaming Audit Logs](https://www.conduktor.io/glossary/streaming-audit-logs). Data governance tools become essential for managing consent, data retention policies, and right-to-deletion requirements. When a patient exercises their right to be forgotten under GDPR, systems must identify and purge all related events across topics and downstream systems, a complex operation that requires careful tracking of data lineage. Here's an example of processing patient consent events in a streaming pipeline: ```python from kafka import KafkaConsumer from datetime import datetime import json # Consumer for consent management stream consumer = KafkaConsumer( 'patient.consent.updates', bootstrap_servers=['kafka-broker:9093'], security_protocol='SSL', ssl_cafile='/path/to/ca-cert', ssl_certfile='/path/to/client-cert', ssl_keyfile='/path/to/client-key', value_deserializer=lambda m: json.loads(m.decode('utf-8')), group_id='consent-processor' ) # Process consent events for message in consumer: consent_event = message.value if consent_event['action'] == 'REVOKE': # Patient revoked consent - trigger data deletion patient_id = consent_event['patientId'] data_category = consent_event['category'] # e.g., 'research', 'marketing' print(f"Processing consent revocation for patient {patient_id}") print(f"Category: {data_category}") print(f"Timestamp: {consent_event['timestamp']}") # Trigger data deletion workflow initiate_data_purge(patient_id, data_category) elif consent_event['action'] == 'GRANT': # Patient granted new consent patient_id = consent_event['patientId'] consent_scope = consent_event['scope'] print(f"Recording consent grant for patient {patient_id}") update_consent_registry(patient_id, consent_scope) def initiate_data_purge(patient_id, category): """Initiate GDPR-compliant data deletion across all systems""" # Publish deletion event to downstream systems deletion_event = { 'patientId': patient_id, 'category': category, 'action': 'DELETE', 'timestamp': datetime.utcnow().isoformat() } # Send to deletion orchestration topic # downstream consumers will handle purging from their systems pass def update_consent_registry(patient_id, scope): """Update central consent registry""" # Store consent in persistent registry pass ``` Governance platforms like Conduktor provide capabilities to enforce data policies, monitor compliance, and manage the lifecycle of sensitive healthcare data flowing through streaming systems. Conduktor offers data masking ([documentation](https://docs.conduktor.io/guide/conduktor-in-production/admin/data-mask)), audit trails, and policy enforcement features specifically designed for regulated industries like healthcare, helping organizations maintain HIPAA and GDPR compliance while managing Kafka clusters at scale. For proactive alerts on compliance violations or system issues, configure [Conduktor Alerts](https://docs.conduktor.io/guide/monitor-brokers-apps/alerts). ## Streaming Architecture Considerations Building healthcare streaming systems requires careful architectural decisions. High availability is non-negotiable, downtime in critical monitoring systems can have life-threatening consequences. This demands multi-datacenter replication, automated failover, and robust disaster recovery plans. For cross-cluster replication strategies, see [Kafka MirrorMaker 2 for Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication). Latency requirements vary by use case. Critical alerts must propagate in seconds (typically under 5 seconds end-to-end), while population health analytics might tolerate minute-level delays. Event ordering matters for clinical accuracy, lab results must be processed in the sequence they were collected, and medication administration events must maintain precise timestamps. Data quality deserves special attention in healthcare contexts. Invalid readings from malfunctioning devices, duplicate events from system retries, and schema inconsistencies across sources can corrupt analytics and trigger false alarms. Stream processing pipelines must include validation, deduplication, and error handling at every stage. Modern Kafka deployments (4.0+) benefit from several features particularly valuable for healthcare: - **Tiered Storage**: Enables long-term retention of patient data (required for compliance) without expanding expensive broker storage. Historical data moves to cost-effective object storage while remaining queryable. See [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). - **KRaft Mode**: Simplifies operations by removing ZooKeeper dependencies, reducing failure points in critical healthcare infrastructure. - **Enhanced Security**: Native support for OAuth 2.0 and improved encryption options align with healthcare security requirements. See [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). Monitoring and observability are critical. Healthcare teams need real-time visibility into pipeline health, data quality metrics, consumer lag, and system performance. Conduktor provides unified monitoring dashboards, alerting, and troubleshooting capabilities specifically designed for complex Kafka deployments in regulated environments. ## Summary Healthcare data streaming lets organizations move from reactive, batch-oriented systems to real-time care delivery. The use cases span patient monitoring in critical care, medical device integration, clinical decision support, and interoperability frameworks that break down data silos. For broader streaming analytics patterns applicable to healthcare, see [Real-Time Analytics with Streaming Data](https://www.conduktor.io/glossary/real-time-analytics-with-streaming-data). Success requires addressing unique healthcare challenges: stringent compliance requirements, complex data formats like HL7 FHIR, high reliability demands, and the need to integrate legacy systems with modern streaming architectures. Organizations that master these challenges gain competitive advantages through improved patient outcomes, operational efficiency, and the ability to leverage data for continuous care improvement. Modern platforms like Apache Kafka 4.0+ with KRaft mode, combined with governance tools like Conduktor, provide the foundation for building compliant, scalable, and reliable healthcare streaming systems that meet both technical and regulatory requirements. As healthcare continues its digital transformation, streaming platforms will become foundational infrastructure, as essential as electronic health records are today. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/apache-kafka) - Distributed streaming platform for healthcare data pipelines - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/data-governance-framework-roles-and-responsibilities) - Govern sensitive healthcare data - [GDPR Compliance for Data Teams](https://www.conduktor.io/gdpr-compliance-for-data-teams) - Ensure regulatory compliance for patient data ## Sources and References 1. HL7 International. "FHIR R4 Specification - Subscriptions Framework." HL7.org, https://www.hl7.org/fhir/subscription.html 2. U.S. Department of Health & Human Services. "HIPAA Security Rule - Technical Safeguards." HHS.gov, https://www.hhs.gov/hipaa/for-professionals/security/index.html 3. Hathaliya, Jigna J., and Sudeep Tanwar. "An exhaustive survey on security and privacy issues in Healthcare 4.0." Computer Communications, vol. 153, 2020, pp. 311-335. doi:10.1016/j.comcom.2020.02.018 4. Apache Software Foundation. "Apache Kafka Use Cases - Healthcare Data Pipelines." Kafka.apache.org, https://kafka.apache.org/documentation/#uses 5. Roine, Roy, et al. "Real-time clinical decision support: The role of streaming analytics in healthcare." Journal of Medical Systems, vol. 45, no. 3, 2021. doi:10.1007/s10916-020-01704-3 Source: https://www.conduktor.io/glossary/healthcare-data-streaming-use-cases --- # High Value Assets: Protecting Critical Data in Streaming Not all data is created equal. In any organization, certain data assets represent disproportionate value, or risk. These **High Value Assets (HVAs)** are the crown jewels that, if compromised, could cause severe business disruption, regulatory penalties, or reputational damage. In streaming architectures, where data flows continuously through multiple systems and services, identifying and protecting HVAs becomes both more critical and more complex. A single Kafka topic carrying payment transactions, personally identifiable information (PII), or intellectual property may represent millions of dollars in potential liability. Yet the real-time nature of streaming means traditional perimeter-based "castle-and-moat" security approaches often fall short. Modern security architectures embrace **zero trust principles**: verify every access request, assume breach, minimize blast radius, and enforce least-privilege access, regardless of network location. For HVAs in streaming systems, zero trust means authenticating and authorizing every producer and consumer, encrypting all data in transit, and continuously monitoring for anomalous behavior. For comprehensive security patterns, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) and [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). ![High Value Asset protection framework for streaming data](https://www.conduktor.io/assets/images/glossary/high-value-assets-streaming-0.webp) The cost of HVA compromise extends beyond immediate financial loss. Organizations face regulatory fines, litigation expenses, customer churn, and lasting damage to brand trust. According to IBM's 2024 Cost of a Data Breach Report, the average cost of a data breach reached $4.88 million globally, representing a 10% increase from 2023. Breaches involving sensitive customer data and regulated information command the highest price tags, with healthcare breaches averaging $9.77 million. For streaming systems processing HVAs at scale, where a single compromised topic could expose millions of records, the stakes are exponentially higher. ## Classifying High Value Assets Effective HVA protection starts with proper classification. Organizations must establish clear, consistent criteria for determining which data assets qualify as high-value. Three primary dimensions drive HVA classification: - **Business Impact:** Assets that directly affect revenue, operations, or strategic advantage. In streaming systems, this includes topics carrying financial transactions, pricing data, supply chain events, or real-time analytics feeding business-critical decisions. The key question: "What would happen if this data became unavailable, corrupted, or exposed?" - **Sensitivity:** Data subject to privacy laws, competitive advantage concerns, or confidentiality agreements. Personal health information (PHI), payment card data (PCI), customer PII, trade secrets, and merger/acquisition details all qualify. Streaming platforms often aggregate multiple sensitive data types into derived topics, elevating their HVA status. - **Regulatory Requirements:** Certain regulations explicitly mandate HVA programs. NIST Cybersecurity Framework, CMMC (Cybersecurity Maturity Model Certification), and various financial services regulations require organizations to identify and protect high-value or high-risk data assets. Failure to properly classify and safeguard regulatory HVAs can trigger compliance violations. In streaming contexts, HVA classification becomes dynamic. A topic containing anonymized data might transition to HVA status when joined with re-identification vectors through stream processing. Time-series data that seems innocuous in isolation may reveal competitive intelligence when analyzed in aggregate. Your HVA inventory must reflect these transformations. ## Discovering and Cataloging HVAs Building an HVA inventory requires systematic discovery across your streaming landscape. This process presents unique challenges compared to traditional databases: - **Dynamic Discovery:** Streaming topics proliferate rapidly. Teams create new data products, join streams, and derive new topics without always following formal approval processes. Automated discovery tools that continuously scan your Kafka clusters, schema registries, and stream processing applications become essential. These tools should capture topic metadata, schema definitions, data lineage, and consumer patterns. - **Content Classification:** Understanding what's *in* a topic requires more than reading topic names. Schema analysis helps, but real HVAs often hide in nested fields or emerge from combinations of seemingly innocuous attributes. Data classification engines that sample streaming data, apply machine learning models, and flag sensitive patterns provide crucial visibility. For PII detection specifically, see [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). - **Lineage Tracking:** Where did this data come from? Where does it go? HVA classification must follow data lineage. If a source system contains HVA data, downstream topics inheriting that data likely qualify as HVAs too, unless effective **de-identification** (removing identifiers to prevent re-identification), **encryption** (encoding data to prevent unauthorized access), or **filtering** (removing sensitive fields entirely) occurs in transit. Each technique serves different purposes: filtering eliminates data permanently, masking obscures it reversibly for certain users, and encryption protects it while maintaining full fidelity for authorized consumers. Modern governance platforms provide tools specifically designed for streaming architectures, making it easier to manage and protect HVAs across complex topologies. Tools like Apache Atlas and OpenLineage provide open-source lineage tracking, while **Conduktor** offers commercial-grade policy enforcement, access controls, and audit logging for Kafka environments. Your HVA catalog should document: - Asset identifier (cluster, topic, schema) - Classification rationale (business impact, sensitivity, regulatory) - Data owner and steward - Protection requirements - Retention policies - Approved consumers - Review date **Example HVA Catalog Entry:** ```yaml Topic: payments-prod-us-east-1 Cluster: prod-financial-cluster Classification: High Value Asset Rationale: - Business Impact: HIGH - Processes $50M+ daily transactions - Sensitivity: HIGH - Contains PCI DSS cardholder data - Regulatory: PCI DSS 4.0, SOC 2 Type II Owner: Payments Platform Team (payments-team@company.com) Data Steward: Jane Chen (jane.chen@company.com) Protection Requirements: - TLS 1.3 encryption in transit - Volume encryption at rest (AWS KMS) - OAuth 2.0 authentication required - ACLs restricted to 3 authorized service accounts - Audit logging enabled with 7-year retention Retention Policy: 90 days (regulatory requirement) Approved Consumers: - payment-processing-service (Write) - fraud-detection-service (Read) - compliance-reporting-service (Read) Last Review: 2025-10-15 Next Review: 2026-01-15 ``` Treat this catalog as a living document. Schedule quarterly reviews, trigger reassessment when data usage changes, and automate alerts when new potential HVAs appear. For schema governance and evolution tracking, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). ## Risk-Based Protection Strategies Once identified, HVAs demand proportional protection. The principle of risk-based security means applying stronger controls to higher-value assets while avoiding over-engineering protections for lower-risk data. - **Encryption:** HVAs should be encrypted at rest and in transit. For streaming systems, this means TLS for all broker connections, encrypted storage for Kafka log segments, and potentially application-level encryption for sensitive fields. - **Envelope encryption** provides an additional security layer for maximum-sensitivity HVAs: data encryption keys (DEKs) encrypt your actual data, while master keys stored in hardware security modules (HSMs) or cloud key management services (AWS KMS, Azure Key Vault, GCP Cloud KMS) encrypt the DEKs themselves. This separation means compromising storage doesn't expose master keys, and key rotation becomes manageable without re-encrypting all data. Example encryption configuration for Kafka brokers (2025 standards): ```properties # Enable TLS 1.3 for client connections (Kafka 3.0+) listeners=SSL://kafka-broker:9093 ssl.protocol=TLSv1.3 ssl.enabled.protocols=TLSv1.3 ssl.cipher.suites=TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256 # SSL certificates and keystores # Keystore: contains broker's private key and certificate (broker identity) # Truststore: contains trusted CA certificates (for validating client certificates) ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks ssl.keystore.password=${KEYSTORE_PASSWORD} ssl.key.password=${KEY_PASSWORD} ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks ssl.truststore.password=${TRUSTSTORE_PASSWORD} ssl.client.auth=required # Enable encryption at rest log.dirs=/var/kafka-logs-encrypted # Use LUKS (Linux), dm-crypt, or cloud-native volume encryption # SASL/SCRAM for authentication (Kafka 2.0+) sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512 sasl.enabled.mechanisms=SCRAM-SHA-512 security.inter.broker.protocol=SASL_SSL ``` **Access Control:** Implement least-privilege access through fine-grained ACLs. HVA topics should restrict both producers and consumers to explicitly authorized service accounts. Role-based access control (RBAC) policies should clearly document who can read, write, or administer HVA assets. For highly sensitive topics, consider implementing dual-authorization requirements where multiple approvals are needed before granting access. Example ACL configuration for HVA topics (Kafka 3.x+ with KRaft mode): ```bash # Modern KRaft-mode ACL management (Kafka 3.0+, required for Kafka 4.0+) # Note: --bootstrap-server replaces deprecated --authorizer-properties with ZooKeeper # Allow only specific service account to produce to HVA topic kafka-acls --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add \ --allow-principal User:payment-service \ --operation Write \ --topic payments-hva # Allow only analytics service to consume kafka-acls --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add \ --allow-principal User:analytics-service \ --operation Read \ --topic payments-hva \ --group analytics-consumer-group # Deny all other access explicitly (evaluated after allow rules) kafka-acls --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add \ --deny-principal User:* \ --operation All \ --topic payments-hva # For enterprise environments, Conduktor provides GUI-based ACL management # with visual policy builders, change tracking, and audit trails ``` **OAuth 2.0 and Modern Authentication:** Modern HVA protection increasingly uses OAuth 2.0 and OpenID Connect (OIDC) for fine-grained, time-limited access control. Rather than managing long-lived service account credentials directly, OAuth enables centralized identity providers (Okta, Auth0, Azure AD, Keycloak) to issue short-lived access tokens with specific scopes and claims. Kafka 3.1+ supports OAuth 2.0 through the `org.apache.kafka.common.security.oauthbearer` SASL mechanism. This enables: - **Token-based authentication** with automatic expiration and renewal - **Fine-grained authorization** based on JWT claims (user roles, groups, scopes) - **Centralized credential management** eliminating distributed secret storage - **Audit integration** with identity provider logs for complete access tracking For detailed OAuth implementation patterns, see [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). Example OAuth configuration for HVA consumer: ```properties # Client OAuth configuration (Kafka 3.1+) security.protocol=SASL_SSL sasl.mechanism=OAUTHBEARER sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \ clientId="hva-analytics-service" \ clientSecret="${OAUTH_CLIENT_SECRET}" \ scope="kafka.hva.read" \ tokenEndpointUrl="https://idp.example.com/oauth/token"; sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler ``` - **Network Segmentation:** Isolate HVA processing in dedicated clusters or brokers with restricted network access. Use private network segments, strict firewall rules, and **jump hosts** (secure intermediary servers requiring multi-factor authentication) for administrative access. This limits the blast radius if other parts of your streaming infrastructure become compromised. - **Data Masking and Tokenization:** Where possible, mask or tokenize sensitive fields before they enter streaming pipelines. Stream processing applications can dynamically mask PII for non-HVA consumers while preserving full fidelity for authorized analytics. Format-preserving encryption maintains data utility while protecting HVA content. - **Monitoring and Anomaly Detection:** HVAs require enhanced monitoring beyond standard observability. Track access patterns, data volumes, schema changes, and authentication events. Alert on unusual consumer activity, unexpected topic configurations, or access attempts from unauthorized networks. SIEM (Security Information and Event Management) systems like Splunk, Elastic Security, or AWS Security Hub enable correlation with broader security events across your infrastructure. Modern HVA monitoring increasingly incorporates **AI/ML-based anomaly detection** to identify subtle attack patterns that rule-based systems miss, such as credential stuffing, gradual data exfiltration, or insider threat behaviors. Example monitoring configuration using JMX metrics (captured via Prometheus for Kafka): ```yaml # Prometheus JMX exporter config for HVA monitoring rules: - pattern: kafka.server<>Count name: kafka_topic_messages_in_total labels: topic: "$1" type: COUNTER - pattern: kafka.server<>Count name: kafka_topic_bytes_in_total labels: topic: "$1" type: COUNTER # Alert rules for HVA topics alerts: - alert: HVAUnauthorizedAccess expr: increase(kafka_topic_failed_authentication_total{topic=~".*-hva"}[5m]) > 5 annotations: summary: "Unauthorized access attempts on HVA topic {{ $labels.topic }}" - alert: HVAUnusualVolume expr: rate(kafka_topic_messages_in_total{topic=~".*-hva"}[5m]) > 2 * avg_over_time(rate(kafka_topic_messages_in_total{topic=~".*-hva"}[5m])[1h:5m]) annotations: summary: "Unusual message volume on HVA topic {{ $labels.topic }}" ``` ## Governance and Compliance HVA protection isn't just technical, it requires robust governance processes: - **Approval Workflows:** Creating new HVA topics or granting HVA access should trigger formal approval workflows. Data owners must review and approve requests. Security teams should validate that proposed protections meet policy requirements. Governance platforms like **Conduktor** can automate these workflows, routing requests through proper channels and maintaining immutable audit trails (see [Conduktor Self-Service](https://docs.conduktor.io/guide/conduktor-concepts/self-service) for approval workflow automation). For comprehensive audit logging patterns, see [Streaming Audit Logs](https://www.conduktor.io/glossary/streaming-audit-logs). - **Regulatory Mapping:** Document which regulations apply to each HVA. PCI DSS governs payment data, HIPAA covers healthcare information, GDPR protects EU personal data, and CCPA addresses California consumer data. Each regulation imposes specific protection requirements. Your HVA program must map these requirements to technical controls and demonstrate compliance through regular assessments. For GDPR-specific considerations, see [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams). - **Business Continuity:** HVAs receive priority in disaster recovery planning. **Recovery Time Objectives (RTOs)** define maximum acceptable downtime before business impact becomes severe, while **Recovery Point Objectives (RPOs)** specify maximum acceptable data loss measured in time. HVA topics typically demand RTOs under 15 minutes and RPOs under 1 minute. Implement cross-region replication using MirrorMaker 2 or cloud-native replication, maintain verified backups, and regularly test failover procedures. When HVA topics go down, critical business processes halt, your continuity plans must reflect this reality. For replication strategies, see [Kafka MirrorMaker 2 for Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication). **Change Management:** Changes affecting HVAs demand extra scrutiny. Schema evolution, retention policy updates, or ACL modifications should follow change control procedures with impact analysis and rollback plans. Automated governance tools can enforce these procedures, preventing unauthorized changes from reaching production. ## Measuring Protection Effectiveness You can't improve what you don't measure. HVA protection programs require metrics demonstrating effectiveness: **Coverage Metrics:** - Percentage of HVAs with encryption enabled - HVAs with documented ownership - HVAs with current access reviews - Time-to-classify new HVA assets **Control Effectiveness:** - Failed access attempts to HVA topics - Mean time to detect (MTTD) HVA incidents - Mean time to respond (MTTR) to HVA incidents - Policy violations detected and remediated **Compliance Metrics:** - HVAs meeting regulatory requirements - Audit findings related to HVAs - Days since last HVA inventory review - Training completion for HVA stewards **Business Metrics:** - HVA incidents per quarter - Financial impact of HVA-related incidents - HVA availability percentage - Cost per HVA (protection overhead) Establish baselines, set improvement targets, and report metrics to leadership regularly. When metrics reveal gaps, trigger remediation. When incidents occur involving HVAs, conduct root cause analysis and update controls accordingly. ## Conclusion: Building an HVA Program High Value Asset protection represents a maturity milestone for data governance programs. Organizations that successfully implement HVA programs demonstrate risk management sophistication, regulatory compliance, and business alignment. Start by identifying your most critical streaming data assets. Establish clear classification criteria and build your initial HVA inventory. Don't aim for perfection, start with your top 10-20 most critical topics and expand from there. Implement proportional protections based on risk. Not every control applies to every HVA, but every HVA should have documented protection requirements and verified implementations. Automate where possible. Manual HVA management doesn't scale in streaming architectures where topics number in the hundreds or thousands. Governance platforms that provide automated discovery, classification, policy enforcement, and compliance reporting make this tractable. Threats evolve, business priorities shift, and new regulations emerge. Regular reviews, continuous monitoring, and adaptive controls keep HVA protections current. ## Related Concepts - [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) - Fine-grained access control patterns essential for protecting high-value assets in streaming environments - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) - Governance structures that support HVA identification, classification, and protection workflows - [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka) - Encryption strategies critical for protecting high-value data as it flows through streaming systems ## Sources and References - **NIST Special Publication 800-53**: Security and Privacy Controls for Information Systems and Organizations - **NIST Cybersecurity Framework**: High Value Asset Management Guidelines - **ISO/IEC 27001:2022**: Information Security Management Systems - **PCI DSS v4.0**: Payment Card Industry Data Security Standard - **HIPAA Security Rule**: 45 CFR Part 164, Subpart C - Security Standards for the Protection of Electronic Protected Health Information - **GDPR Article 32**: Security of Processing Requirements - **Apache Kafka Security Documentation**: Encryption and Authentication Configuration - **CMMC Model**: Cybersecurity Maturity Model Certification - Asset Management Practices - **Cost of a Data Breach Report 2023**: IBM Security and Ponemon Institute Source: https://www.conduktor.io/glossary/high-value-assets-streaming --- # Iceberg Catalog Management: REST, Hive, Glue, and Nessie Apache Iceberg has become a widely-used table format for data lakes, offering ACID transactions, schema evolution, and time travel. At the heart of Iceberg's architecture is the catalog: a component responsible for tracking table metadata, managing table locations, and coordinating concurrent operations. Understanding the differences between catalog implementations matters for data architects and platform engineers building scalable data infrastructures. This article covers the major catalog options available in 2025: the vendor-neutral REST Catalog (recommended for new deployments), traditional Hive Metastore, AWS Glue for cloud-native architectures, and Project Nessie for Git-like versioning workflows. For foundational concepts about Iceberg's architecture, see [Apache Iceberg Table Architecture: Metadata and Snapshots](https://www.conduktor.io/glossary/iceberg-table-architecture-metadata-and-snapshots). ![Comparison of Iceberg catalog implementations](https://www.conduktor.io/assets/images/glossary/iceberg-catalog-management-hive-glue-and-nessie-0.webp) ## Understanding Iceberg Catalogs An Iceberg catalog serves as the authoritative source for table metadata, maintaining pointers to the current metadata file for each table. When a query engine or processing framework needs to read or write an Iceberg table, it first consults the catalog to locate the table's metadata. The catalog's responsibilities include: - **Table Discovery**: Maintaining a registry of available tables and their locations - **Metadata Pointer Management**: Tracking the current metadata file for each table - **Atomic Updates**: Ensuring that metadata updates occur atomically to prevent corruption - **Namespace Management**: Organizing tables into hierarchical namespaces or databases The choice of catalog implementation impacts operational complexity, scalability, multi-engine compatibility, and advanced features like versioning and branching. ## REST Catalog (Recommended for New Deployments) The REST Catalog, standardized in Iceberg 1.0+ (2022) and refined through 2025, has become the recommended approach for new implementations. It provides a vendor-neutral HTTP API that any query engine can use, eliminating the need for engine-specific catalog implementations. ### Architecture and Characteristics The REST Catalog defines a standard HTTP API specification for catalog operations. Implementations can be backed by any storage system, databases, object stores, or specialized metadata stores. This abstraction allows organizations to switch backend implementations without changing client configurations. **Advantages:** - **Vendor Neutrality**: Works consistently across Spark, Flink, Trino, Dremio, and other engines - **Simplified Client Configuration**: Single HTTP endpoint instead of engine-specific settings - **Modern API Design**: RESTful interface with JWT authentication support - **Cloud-Native**: Naturally fits serverless and containerized deployments - **Extensibility**: Custom implementations can add features like caching, auditing, or access control - **Active Development**: Apache Polaris (incubating since 2024) provides an open-source implementation with advanced access control **Limitations:** - Requires HTTP server infrastructure (though lightweight) - Network latency for remote catalog access - Relatively newer ecosystem compared to Hive Metastore ### Configuration Example ```python # Spark 3.5+ Configuration for REST Catalog from pyspark.sql import SparkSession spark = SparkSession.builder \ .appName("IcebergRESTCatalog") \ .config("spark.sql.catalog.rest_catalog", "org.apache.iceberg.spark.SparkCatalog") \ .config("spark.sql.catalog.rest_catalog.catalog-impl", "org.apache.iceberg.rest.RESTCatalog") \ .config("spark.sql.catalog.rest_catalog.uri", "https://catalog-server:8181") \ .config("spark.sql.catalog.rest_catalog.warehouse", "s3://bucket/warehouse") \ .config("spark.sql.catalog.rest_catalog.token", "eyJhbGc...") \ .getOrCreate() # Create table using REST catalog spark.sql(""" CREATE TABLE rest_catalog.analytics.events ( event_id STRING, timestamp TIMESTAMP, user_id STRING, event_type STRING, properties MAP ) USING iceberg PARTITIONED BY (days(timestamp)) TBLPROPERTIES ( 'format-version' = '2', -- Iceberg V2 format (1.0+) 'write.delete.mode' = 'merge-on-read', -- Optimized deletes 'write.update.mode' = 'merge-on-read' -- Optimized updates ) """) ``` ### Apache Polaris: Open-Source REST Catalog Apache Polaris (donated by Snowflake in 2024) is an open-source REST Catalog implementation offering: - Multi-catalog support within a single server - Fine-grained access control policies - Multi-tenant isolation - Integration with OAuth 2.0 and OpenID Connect ## Hive Metastore Catalog The Hive Metastore has long been the de facto metadata repository in the Hadoop ecosystem. Iceberg's Hive catalog implementation leverages this existing infrastructure, making it a natural choice for organizations with established Hive deployments. ### Architecture and Characteristics The Hive Metastore uses a relational database (typically MySQL, PostgreSQL, or MariaDB) to store metadata. For Iceberg tables, the metastore stores the table location and a pointer to the current metadata file. The actual table metadata, manifests, and data files reside in object storage or HDFS. **Advantages:** - Broad compatibility with existing tools and engines - Mature and well-understood operational model - Native support in most query engines (Spark, Trino, Flink) **Limitations:** - No built-in versioning or branching capabilities - Potential bottleneck for high-concurrency environments - Requires separate infrastructure management ### Configuration Example ```python # Spark 3.5+ Configuration for Hive Metastore Catalog from pyspark.sql import SparkSession spark = SparkSession.builder \ .appName("IcebergHiveCatalog") \ .config("spark.sql.catalog.hive_catalog", "org.apache.iceberg.spark.SparkCatalog") \ .config("spark.sql.catalog.hive_catalog.type", "hive") \ .config("spark.sql.catalog.hive_catalog.uri", "thrift://metastore-host:9083") \ # Hive Metastore Thrift endpoint .config("spark.sql.catalog.hive_catalog.warehouse", "s3://bucket/warehouse") \ # Table storage location .enableHiveSupport() \ # Required for Hive catalog integration .getOrCreate() # Create a table using Hive catalog spark.sql(""" CREATE TABLE hive_catalog.db.events ( event_id STRING, timestamp TIMESTAMP, user_id STRING, event_type STRING ) USING iceberg PARTITIONED BY (days(timestamp)) -- Time-based partitioning for efficient queries """) ``` ## AWS Glue Catalog AWS Glue provides a fully managed metadata catalog service that integrates seamlessly with the AWS ecosystem. The Glue catalog implementation for Iceberg offers serverless operation without the overhead of managing metastore infrastructure. ### Architecture and Characteristics The Glue catalog stores Iceberg table metadata in a managed service, eliminating the need for database administration. It integrates naturally with other AWS services like Athena, EMR, and Redshift Spectrum. **Advantages:** - Serverless and fully managed, no infrastructure to maintain - Fine-grained IAM-based access control - Native integration with AWS analytics services - Built-in data cataloging and discovery features **Limitations:** - AWS ecosystem lock-in - API rate limits may impact high-frequency operations - Limited versioning capabilities compared to specialized solutions ### Configuration Example ```python # Spark 3.5+ Configuration for AWS Glue Catalog spark = SparkSession.builder \ .appName("IcebergGlueCatalog") \ .config("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog") \ .config("spark.sql.catalog.glue_catalog.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \ # Glue-specific implementation .config("spark.sql.catalog.glue_catalog.warehouse", "s3://bucket/warehouse") \ # S3 warehouse location .config("spark.sql.catalog.glue_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \ # S3-optimized file operations .getOrCreate() # Create table using Glue catalog spark.sql(""" CREATE TABLE glue_catalog.analytics.user_sessions ( session_id STRING, start_time TIMESTAMP, end_time TIMESTAMP, page_views INT ) USING iceberg TBLPROPERTIES ( 'write.format.default' = 'parquet', -- File format 'write.parquet.compression-codec' = 'snappy' -- Compression for storage efficiency ) """) ``` ### Alternative: JDBC Catalog for Simpler Deployments For organizations not tied to AWS or Hive infrastructure, the JDBC Catalog offers a lightweight alternative backed by any relational database (PostgreSQL, MySQL, etc.): ```python # JDBC Catalog configuration spark = SparkSession.builder \ .config("spark.sql.catalog.jdbc_catalog", "org.apache.iceberg.spark.SparkCatalog") \ .config("spark.sql.catalog.jdbc_catalog.catalog-impl", "org.apache.iceberg.jdbc.JdbcCatalog") \ .config("spark.sql.catalog.jdbc_catalog.uri", "jdbc:postgresql://localhost:5432/iceberg") \ .config("spark.sql.catalog.jdbc_catalog.jdbc.user", "iceberg") \ .config("spark.sql.catalog.jdbc_catalog.jdbc.password", "password") \ .config("spark.sql.catalog.jdbc_catalog.warehouse", "s3://bucket/warehouse") \ .getOrCreate() ``` ## Project Nessie Catalog Project Nessie introduces Git-like versioning semantics to data lakes. For readers unfamiliar with Git: it's a system that tracks changes over time with branches (parallel versions), commits (snapshots), and merges (combining changes), Nessie applies these concepts to data tables. It provides branches, tags, and commit history for Iceberg tables, enabling advanced workflows like experimentation, rollback, and cross-table consistency. ### Architecture and Characteristics Nessie 2.x (released 2023, matured through 2025) stores catalog state in a versioned data structure, supporting multiple branches of table metadata. This allows data engineers to create isolated environments for development, testing, and production within the same catalog, similar to how software developers work on feature branches before merging to production. **Advantages:** - **Git-like Branching**: Create isolated environments for testing without affecting production - **Cross-table Transactions**: Atomically commit changes across multiple tables - **Full Audit Trail**: Complete commit history showing who changed what and when - **Safe Experimentation**: Test schema changes or data transformations, then rollback if needed - **Time Travel Across Tables**: Query consistent snapshots across multiple related tables - **Multi-table Atomic Commits**: Ensure consistency in multi-table operations **Limitations:** - Relatively newer technology with smaller adoption (though growing rapidly) - Additional operational complexity compared to simpler catalogs - Requires understanding of version control concepts - More moving parts to monitor and maintain ### Configuration Example ```python # Spark 3.5+ Configuration for Nessie 2.x Catalog spark = SparkSession.builder \ .appName("IcebergNessieCatalog") \ .config("spark.sql.catalog.nessie", "org.apache.iceberg.spark.SparkCatalog") \ .config("spark.sql.catalog.nessie.catalog-impl", "org.apache.iceberg.nessie.NessieCatalog") \ .config("spark.sql.catalog.nessie.uri", "http://nessie-server:19120/api/v2") \ # Nessie 2.x uses /api/v2 .config("spark.sql.catalog.nessie.ref", "main") \ # Default branch (like 'main' in Git) .config("spark.sql.catalog.nessie.warehouse", "s3://bucket/warehouse") \ .config("spark.sql.catalog.nessie.authentication.type", "BEARER") \ # Optional: JWT authentication .getOrCreate() # Create a branch for development (isolated environment) spark.sql("CREATE BRANCH IF NOT EXISTS dev IN nessie FROM main") # Switch to dev branch and create table spark.sql("USE REFERENCE dev IN nessie") spark.sql(""" CREATE TABLE nessie.analytics.experiments ( experiment_id STRING, variant STRING, metrics MAP, created_at TIMESTAMP ) USING iceberg TBLPROPERTIES ( 'format-version' = '2' -- Use Iceberg V2 format ) """) # After testing, view changes before merging spark.sql("SHOW LOG IN nessie") # Merge changes back to main after validation (atomic cross-table commit) spark.sql("MERGE BRANCH dev INTO main IN nessie") ``` **Nessie 2.x Improvements:** - Enhanced REST API with better performance - Improved conflict resolution for concurrent operations - Native support for Iceberg views (1.5+) - Better integration with access control systems - Reduced metadata overhead for large catalogs ## Streaming Ecosystem Integration Iceberg catalogs are central to streaming architectures, where real-time data pipelines continuously ingest and process events. The catalog choice impacts streaming framework integration, particularly with Apache Flink, Spark Structured Streaming, and Kafka-based pipelines. For comprehensive coverage of streaming to lakehouses, see [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables). ### Streaming Considerations - **Catalog Commit Frequency**: Streaming jobs commit snapshots at regular intervals (e.g., every checkpoint in Flink). The catalog must handle these frequent metadata updates efficiently without becoming a bottleneck. - **Multi-Writer Scenarios**: Stream processing often involves multiple parallel writers. The catalog must provide optimistic concurrency control to prevent conflicts, this means writers detect and retry when concurrent updates occur, rather than blocking each other. - **Schema Evolution**: Streaming schemas evolve over time as business requirements change. The catalog should support backward-compatible schema changes without disrupting active streams. For detailed guidance, see [Schema Evolution in Apache Iceberg](https://www.conduktor.io/glossary/schema-evolution-in-apache-iceberg). ### Apache Flink Example ```java // Flink 1.19+ streaming write to Iceberg with REST Catalog (recommended) StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.enableCheckpointing(60000); // Checkpoint every 60 seconds (controls commit frequency) StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env); // Configure REST catalog (modern approach) tableEnv.executeSql( "CREATE CATALOG rest_catalog WITH (" + " 'type' = 'iceberg'," + " 'catalog-impl' = 'org.apache.iceberg.rest.RESTCatalog'," + " 'uri' = 'https://catalog-server:8181'," + " 'warehouse' = 's3://bucket/warehouse'" + ")" ); // Define Kafka source (for context - see kafka-connect-building-data-integration-pipelines.md) tableEnv.executeSql( "CREATE TABLE kafka_source (" + " event_id STRING," + " event_time TIMESTAMP(3)," + " user_id STRING," + " event_type STRING," + " WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND" + ") WITH (" + " 'connector' = 'kafka'," + " 'topic' = 'events'," + " 'properties.bootstrap.servers' = 'localhost:9092'," + " 'format' = 'json'" + ")" ); // Streaming INSERT to Iceberg with automatic commits at checkpoints tableEnv.executeSql( "INSERT INTO rest_catalog.db.events " + "SELECT event_id, event_time, user_id, event_type " + "FROM kafka_source" ); ``` For detailed Flink streaming patterns, see [Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) and [Flink DataStream API](https://www.conduktor.io/glossary/flink-datastream-api-building-streaming-applications). ### Governance and Visibility For organizations managing complex streaming pipelines with Iceberg tables, data governance platforms provide visibility into catalog operations. **Conduktor** offers governance capabilities for Kafka-to-Iceberg pipelines, including: - **Schema Registry Integration**: Monitor schema evolution from Kafka topics to Iceberg tables - **Pipeline Visibility**: Monitor streaming data flows from Kafka topics through to table storage via dashboards and audit logs - **Access Control Auditing**: Track who accesses which catalogs and tables - **Policy Enforcement**: Set governance rules for catalog operations and schema changes - **Migration Visibility**: Monitor catalog migrations and ensure consistency during transitions This governance layer becomes particularly valuable when managing multiple catalogs or migrating between catalog implementations, ensuring that metadata changes and access patterns remain auditable and compliant. Conduktor integrates with all major catalog types (REST, Hive, Glue, Nessie) to provide unified visibility across your data platform. ## Catalog Migration Strategies Organizations may need to migrate between catalog implementations as requirements evolve. Common migration scenarios include moving from Hive Metastore to AWS Glue for operational simplification, or adopting Nessie for advanced versioning capabilities. ### Migration Approaches **1. Dual-Catalog Operation** Temporarily maintain both catalogs, registering tables in both systems during the transition period. This allows gradual migration of workloads. ```python # Register table in both catalogs spark.sql("CREATE TABLE hive_catalog.db.events USING iceberg ...") spark.sql("CREATE TABLE glue_catalog.db.events USING iceberg LOCATION 's3://...'") ``` **2. Metadata Export and Import** Export table metadata from the source catalog and re-register in the target catalog. The underlying data files remain unchanged. ```python # Export metadata location from Hive hive_location = spark.sql("DESCRIBE EXTENDED hive_catalog.db.events") \ .filter("col_name = 'Location'").collect()[0].data_type # Register in Glue catalog spark.sql(f"CREATE TABLE glue_catalog.db.events USING iceberg LOCATION '{hive_location}'") ``` **3. Catalog Proxy Pattern** Use a catalog proxy or abstraction layer that can route requests to different underlying catalogs based on configuration. ### Migration Considerations - **Downtime Requirements**: Determine if zero-downtime migration is necessary - **Consistency Guarantees**: Ensure atomic cutover to prevent split-brain scenarios - **Access Control**: Map permissions from source to target catalog - **Validation**: Verify table accessibility and metadata integrity post-migration ## Catalog Selection Criteria Choosing the right catalog implementation depends on several factors: **REST Catalog** is recommended when: - Starting a new Iceberg deployment in 2025 - You want vendor-neutral, engine-agnostic architecture - Cloud-native or containerized environments are preferred - You need flexibility to change backend storage without affecting clients - Modern authentication (JWT, OAuth) is required - Consider Apache Polaris for open-source implementation with advanced access control **Hive Metastore** is appropriate when: - You have existing Hive infrastructure and expertise - Multi-cloud or on-premises deployment is required with existing Hadoop ecosystem - Broad tool compatibility is the primary concern (though REST catalog now matches this) - You're migrating from traditional Hive tables to Iceberg - Advanced versioning is not a requirement **AWS Glue** is suitable when: - Your infrastructure is exclusively AWS-based - You prefer fully managed services over self-hosted solutions - Integration with AWS analytics services (Athena, EMR, Redshift Spectrum) is critical - You want zero infrastructure management for the catalog - IAM-based access control fits your security model **JDBC Catalog** works well for: - Small to medium deployments - When you already have a PostgreSQL/MySQL database available - Simple requirements without advanced features - Development and testing environments **Project Nessie** excels when: - You need Git-like versioning, branching, and tagging - Cross-table consistency and multi-table atomic commits are critical - You want to support isolated development/staging/production environments - Audit trails and safe experimentation workflows are important - Time travel across multiple related tables is required ## Related Concepts - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) - [Data Lineage: Tracking Data from Source to Consumption](https://www.conduktor.io/data-lineage-tracking-data-from-source-to-consumption) - [Data Pipeline Orchestration with Streaming](https://www.conduktor.io/data-pipeline-orchestration-with-streaming) ## Summary The choice between REST Catalog, Hive Metastore, AWS Glue, JDBC Catalog, and Project Nessie significantly impacts your platform's capabilities and operational characteristics. **As of 2025, the REST Catalog is recommended for new deployments** due to its vendor-neutral API, broad engine support, and modern architecture. Apache Polaris provides an open-source implementation with enterprise-grade access control. Hive Metastore offers broad compatibility and maturity but requires infrastructure management, best for existing Hadoop environments. AWS Glue provides serverless convenience within the AWS ecosystem, ideal for AWS-native architectures. JDBC Catalog offers a lightweight option for simpler deployments. Project Nessie introduces powerful Git-like versioning semantics, enabling advanced workflows like isolated environments and multi-table transactions, at the cost of increased operational complexity. For streaming workloads (covered in [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables)), catalog performance under frequent commits and multi-writer scenarios becomes critical. Integration with governance platforms like **Conduktor** helps maintain visibility and control as catalog complexity grows, particularly for Kafka-based data pipelines. When evaluating catalog options, consider your infrastructure constraints, operational capabilities, versioning requirements, and long-term platform evolution. Many organizations benefit from a multi-catalog strategy, using different implementations for different use cases or environments. For related topics, see [Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture), [Iceberg Table Maintenance](https://www.conduktor.io/glossary/maintaining-iceberg-tables-compaction-and-cleanup), and [Metadata Management](https://www.conduktor.io/glossary/metadata-management-technical-vs-business-metadata). ## Sources and References - [Apache Iceberg Documentation - Catalogs](https://iceberg.apache.org/docs/latest/configuration/) - [AWS Glue Catalog Integration with Iceberg](https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html) - [Project Nessie Documentation](https://projectnessie.org/docs/) - [Iceberg Catalog Specification](https://iceberg.apache.org/spec/#catalog) - [Apache Flink Iceberg Integration](https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/table/iceberg/) - [Iceberg Hive Metastore Catalog](https://iceberg.apache.org/docs/latest/hive/) - [Tabular: Iceberg Catalog Best Practices](https://tabular.io/blog/) Source: https://www.conduktor.io/glossary/iceberg-catalog-management-hive-glue-and-nessie --- # Iceberg Table Architecture: Metadata and Snapshots Apache Iceberg has become a widely-used table format for data lakehouses, addressing limitations in traditional data lake architectures. Its metadata layer enables ACID transactions, snapshot isolation, and schema evolution — capabilities previously available only in proprietary data warehouses. For a broader overview of Iceberg's features and ecosystem, see [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg). For practical catalog implementation guidance, refer to [Iceberg Catalog Management: Hive, Glue, and Nessie](https://www.conduktor.io/glossary/iceberg-catalog-management-hive-glue-and-nessie). ## The Three-Layer Metadata Architecture Iceberg's architecture is built on three distinct metadata layers that work together to track table state and data files: ![Iceberg's architecture is built on three distinct metadata layers that work together to track table state and data files](https://www.conduktor.io/assets/images/glossary/iceberg-table-architecture-metadata-and-snapshots-0.webp) ### Catalog Layer The catalog serves as the entry point to an Iceberg table, maintaining a pointer to the current metadata file. The catalog's atomic update mechanism ensures only one metadata file is considered "current," preventing split-brain scenarios during concurrent writes. **Modern Catalog Implementations (2025)**: Since Iceberg 1.5, the **REST Catalog** has emerged as the standard for cloud-native deployments. Unlike traditional catalogs that depend on Hive Metastore or cloud-specific services, REST Catalog provides a vendor-neutral HTTP API for catalog operations: ```properties # Modern REST Catalog configuration catalog-impl=org.apache.iceberg.rest.RESTCatalog uri=https://catalog.example.com credential= warehouse=s3://my-warehouse/ ``` REST Catalog advantages include: - **Vendor neutrality**: Works across any cloud or on-premises infrastructure - **Multi-tenancy**: Built-in namespace isolation for different teams - **Fine-grained security**: OAuth2/token-based authentication and authorization - **Simplified operations**: No external metastore dependencies to manage Other catalog options remain available for specific use cases: - **Hive Metastore**: Legacy compatibility for existing Hadoop ecosystems - **AWS Glue**: Native AWS integration with IAM-based access control - **Nessie**: Git-like versioning for data with multi-table transactions - **JDBC Catalog**: Lightweight option using relational databases ### Metadata Files Each metadata file is a JSON document containing: - **Current snapshot reference**: Points to the active table state - **Snapshot history**: List of all previous snapshots with timestamps - **Schema**: Current and historical table schemas - **Partition spec**: How the table is partitioned (supports evolution) - **Sort order**: Optional ordering specification for data optimization - **Table properties**: Configuration and operational metadata Metadata files are immutable and versioned. Each write operation creates a new metadata file, enabling time travel to any previous table state. ### Manifest Lists and Manifest Files The manifest list (Avro format) contains references to all manifest files for a given snapshot. Each manifest file tracks a subset of data files along with: - **File-level statistics**: Row counts, column bounds (min/max values), null counts, and distinct value counts - **Partition values**: For partitioned tables, the partition key-value pairs for each file - **File metadata**: File size, format (Parquet/ORC/Avro), file paths in object storage **Advanced Statistics with Puffin Files (Iceberg 1.4+)**: Since Iceberg 1.4, the **Puffin file format** enables storage of advanced statistics that go beyond simple min/max bounds: - **Theta sketches**: Probabilistic cardinality estimation for high-cardinality columns - **Bloom filters**: Membership testing to skip files that definitely don't contain a value - **NDV (Number of Distinct Values)**: Accurate distinct counts for query optimization - **Histogram data**: Value distribution information for cost-based optimization ```sql -- Query engines automatically use Puffin statistics when available SELECT DISTINCT user_id FROM events WHERE user_id = 'abc123'; -- Bloom filter eliminates non-matching files SELECT COUNT(DISTINCT user_id) FROM events; -- Uses Theta sketch for estimation ``` Puffin files are referenced in the manifest list and stored separately from manifest files, allowing statistics to be updated independently of the core metadata structure. This hierarchical structure enables efficient query planning. Engines can read manifest files and Puffin statistics to determine which data files are relevant for a query without scanning the entire dataset, a critical optimization for large tables with millions of files. ## Snapshot Isolation and ACID Transactions Snapshots are the cornerstone of Iceberg's transactional guarantees. Each snapshot represents an immutable, consistent view of the table at a specific point in time. ### How Snapshots Work When a write operation occurs: 1. New data files are written to object storage 2. New manifest files are created describing these data files 3. A new manifest list aggregates all relevant manifests 4. A new metadata file is created with updated snapshot information 5. The catalog pointer is atomically updated to the new metadata file If the atomic catalog update fails (due to concurrent modification), the write operation is retried with the latest table state, ensuring **serializable isolation**, the highest level of transaction isolation where concurrent operations appear to execute sequentially, preventing anomalies like dirty reads or lost updates. ### Snapshot Benefits **Time Travel**: Query historical table states by specifying a snapshot ID or timestamp: ```sql SELECT * FROM orders FOR SYSTEM_TIME AS OF '2025-01-15 10:00:00'; ``` For detailed time travel patterns and use cases, see [Time Travel with Apache Iceberg](https://www.conduktor.io/glossary/time-travel-with-apache-iceberg). **Rollback**: Revert to previous snapshots instantly without data movement: ```sql CALL catalog.system.rollback_to_snapshot('orders', 12345678901234); ``` **Incremental Processing**: Read only changes between snapshots for efficient ETL: ```sql SELECT * FROM orders FOR SYSTEM_VERSION AS OF 100 WHERE _iceberg_snapshot_id > 99; ``` ### Branching and Tagging (Iceberg 1.5+) Iceberg 1.5 introduced **branches** and **tags** for managing multiple table versions simultaneously, think of them as Git branches for your data: ```sql -- Create a branch for experimental changes ALTER TABLE orders CREATE BRANCH IF NOT EXISTS experimental; -- Write to a specific branch INSERT INTO orders.branch_experimental VALUES (12345, 'customer@example.com', 599.99, CURRENT_TIMESTAMP); -- Create a tag to mark an important snapshot (e.g., end of quarter) ALTER TABLE orders CREATE TAG IF NOT EXISTS q1_2025_snapshot; -- Query from a branch SELECT * FROM orders VERSION AS OF 'experimental'; -- Query from a tag SELECT * FROM orders VERSION AS OF 'q1_2025_snapshot'; -- Fast-forward main branch to experimental after testing CALL catalog.system.fast_forward('orders', 'main', 'experimental'); ``` **Use Cases for Branches**: - **Isolated testing**: Test schema changes, data migrations, or quality rules without affecting production queries - **Staging environments**: Create `dev`, `staging`, and `prod` branches within the same table - **A/B testing**: Compare different data transformation approaches on branches before merging - **Data science experimentation**: Allow data scientists to experiment with feature engineering on separate branches **Use Cases for Tags**: - **Compliance snapshots**: Tag end-of-period snapshots for regulatory retention - **Rollback points**: Mark known-good states before risky operations - **Reproducibility**: Tag data versions used for model training or reports - **Audit trails**: Immutable references to historical data states Each branch maintains its own snapshot lineage and can have independent retention policies, making branches ideal for data engineering workflows that need production isolation. ## Schema Evolution and Partition Evolution Unlike traditional formats, Iceberg supports safe schema and partition evolution without requiring full table rewrites. ### Schema Evolution Iceberg tracks schema changes with unique field IDs that remain constant even when columns are renamed or reordered. Supported operations include: - Add columns (anywhere in the schema) - Drop columns - Rename columns - Update column types (with compatible promotions) - Reorder columns Each schema change creates a new schema version in the metadata file, but existing data files remain valid. Query engines use the schema evolution history to correctly read older files. For comprehensive schema evolution patterns, best practices, and migration strategies, see [Schema Evolution in Apache Iceberg](https://www.conduktor.io/glossary/schema-evolution-in-apache-iceberg). ### Partition Evolution Partition specifications can evolve over time. For example, you might start with daily partitioning and later switch to hourly: ```sql -- Initial partition spec ALTER TABLE events SET PARTITION SPEC (day(event_time)); -- Evolve to hourly partitioning ALTER TABLE events SET PARTITION SPEC (hour(event_time)); ``` Iceberg maintains partition evolution history, ensuring queries correctly apply partition filters regardless of when data was written. For detailed partitioning strategies and performance optimization techniques, see [Iceberg Partitioning and Performance Optimization](https://www.conduktor.io/glossary/iceberg-partitioning-and-performance-optimization). ## Metadata Tables for Operational Monitoring Iceberg exposes internal metadata through queryable **metadata tables**, enabling operational insights without external tools: ```sql -- View snapshot history SELECT snapshot_id, parent_id, timestamp_ms, operation, summary FROM orders.snapshots ORDER BY timestamp_ms DESC LIMIT 10; -- Inspect all data files in current snapshot SELECT file_path, file_format, record_count, file_size_in_bytes, partition, value_counts, null_value_counts, lower_bounds, upper_bounds FROM orders.files WHERE file_size_in_bytes > 100000000; -- Files larger than 100MB -- Analyze manifest files SELECT path, length, partition_spec_id, added_snapshot_id, added_data_files_count FROM orders.manifests ORDER BY added_snapshot_id DESC; -- Review table history (all changes) SELECT made_current_at, snapshot_id, is_current_ancestor FROM orders.history WHERE made_current_at > CURRENT_TIMESTAMP - INTERVAL '7' DAY; -- Check partition statistics SELECT partition, record_count, file_count, spec_id FROM orders.partitions WHERE record_count > 1000000; -- High-volume partitions ``` **Common Operational Use Cases**: - **Performance monitoring**: Identify small files that need compaction - **Cost analysis**: Track data file growth and storage costs by partition - **Debugging**: Investigate snapshot lineage and failed operations - **Capacity planning**: Analyze partition distribution and file count trends - **Compliance**: Query snapshot history for audit trails Metadata tables integrate seamlessly with BI tools and monitoring dashboards, providing observability into Iceberg's internal state. ## Streaming Ecosystem Integration Iceberg's architecture is well-suited for streaming data ingestion, providing exactly-once semantics and low-latency visibility of new data. For architectural patterns and best practices for streaming ingestion, see [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse) and [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables). ### Kafka and Flink Integration Apache Flink (1.18+) provides native Iceberg sink connectors that leverage snapshots for exactly-once processing: ```java // Modern Flink 1.18+ with Iceberg connector StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // Enable checkpointing for exactly-once guarantees env.enableCheckpointing(60000, CheckpointingMode.EXACTLY_ONCE); env.getCheckpointConfig().setMinPauseBetweenCheckpoints(30000); // Configure Kafka source KafkaSource kafkaSource = KafkaSource.builder() .setBootstrapServers("localhost:9092") .setTopics("events-topic") .setGroupId("flink-iceberg-sink") .setDeserializer(new RowDataDeserializationSchema()) .setStartingOffsets(OffsetsInitializer.latest()) .build(); DataStream stream = env.fromSource( kafkaSource, WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(5)), "Kafka Source" ); // Configure Iceberg catalog and table Configuration config = new Configuration(); config.setString("type", "iceberg"); config.setString("catalog-type", "rest"); config.setString("uri", "http://catalog.example.com"); config.setString("warehouse", "s3://bucket/warehouse/"); TableLoader tableLoader = TableLoader.fromCatalog( CatalogLoader.rest("rest_catalog", config, new HashMap<>()), TableIdentifier.of("default", "events") ); // Write to Iceberg with modern API FlinkSink.forRowData(stream) .tableLoader(tableLoader) .equalityFieldColumns(Arrays.asList("event_id")) // Upsert mode .upsert(true) // Enable upserts for CDC patterns .distributionMode(DistributionMode.HASH) .writeParallelism(4) .build(); env.execute("Kafka to Iceberg Pipeline"); ``` Each Flink checkpoint triggers an Iceberg commit, creating a new snapshot. If the job fails and restarts from a checkpoint, Iceberg's atomic commits prevent duplicate data. ### Streaming Read Patterns Iceberg supports streaming reads through incremental snapshots: - **Micro-batch streaming**: Poll for new snapshots at regular intervals - **Continuous streaming**: Subscribe to catalog notifications for new commits - **CDC patterns**: Use row-level deletes and position deletes for change data capture (see [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) for CDC to Iceberg patterns) Tools like Apache Flink, Apache Spark Structured Streaming, and Kafka Connect can consume Iceberg tables incrementally, enabling real-time analytics pipelines. ### Governance and Visibility When building streaming pipelines with Iceberg, data governance becomes critical. **Conduktor** provides comprehensive visibility and governance for Kafka-to-Iceberg pipelines: - **Pipeline monitoring**: Monitor data flow from Kafka topics through transformations to Iceberg snapshots via dashboards and audit logs - **Schema compatibility monitoring**: Validate that Kafka producer schemas match Iceberg table schemas before commits - **Data quality enforcement**: Apply validation rules and quality checks before data reaches Iceberg tables - **Access auditing**: Track which consumers (streaming and batch) access Iceberg tables and at what snapshot versions - **Performance monitoring**: Monitor Flink checkpoint durations, Iceberg commit latency, and snapshot growth rates - **Chaos testing with Conduktor Gateway**: Use the Kafka proxy Conduktor Gateway to test pipeline resilience by simulating Kafka failures, slow brokers, or network partitions This governance layer ensures that the flexibility of Iceberg's architecture doesn't compromise data quality or compliance requirements in production streaming environments. ## Metadata Optimization and Maintenance While Iceberg's metadata architecture provides powerful capabilities, it requires periodic maintenance to prevent metadata bloat. For comprehensive maintenance procedures and strategies, see [Maintaining Iceberg Tables: Compaction and Cleanup](https://www.conduktor.io/glossary/maintaining-iceberg-tables-compaction-and-cleanup). ### Snapshot Expiration Old snapshots should be periodically expired to remove unreferenced data and manifest files: ```sql CALL catalog.system.expire_snapshots( table => 'orders', older_than => TIMESTAMP '2025-01-01 00:00:00', retain_last => 100 ); ``` This operation: - Removes metadata files for expired snapshots - Deletes unreferenced manifest files - Marks unreferenced data files for deletion (actual deletion happens during orphan file cleanup) ### Manifest File Compaction As tables accumulate many small manifest files, query planning can slow down. Manifest compaction consolidates small manifests: ```sql CALL catalog.system.rewrite_manifests('orders'); ``` ### Orphan File Cleanup Failed writes can leave orphaned data files. Periodic cleanup removes these files: ```sql CALL catalog.system.remove_orphan_files( table => 'orders', older_than => TIMESTAMP '2025-01-01 00:00:00' ); ``` ### Metadata Caching Query engines cache manifest files and statistics to avoid repeated reads. Understanding cache behavior is crucial for performance tuning in high-throughput environments. ## Performance Considerations Iceberg's metadata architecture directly impacts query performance: - **Partition Pruning**: Manifest files contain partition statistics, enabling engines to skip entire manifests during query planning. - **Column Statistics**: Min/max values and null counts allow predicate pushdown without reading data files. - **Manifest Filtering**: When only specific partitions are modified, Iceberg reuses unchanged manifest files across snapshots, reducing metadata overhead. - **Vectorized Reads**: Manifest files in Avro format support efficient vectorized reading for large-scale query planning. For large tables (billions of rows, millions of files), proper partition design and regular metadata maintenance are essential for maintaining sub-second query planning. ## Summary Apache Iceberg separates metadata into distinct layers — catalog, metadata files, manifest lists, and manifest files — which together provide: - **ACID transactions** through atomic catalog updates and immutable snapshots - **Snapshot isolation** enabling time travel and concurrent read/write workloads - **Schema and partition evolution** without data rewrites - **Efficient query planning** through hierarchical metadata and statistics - **Streaming integration** with exactly-once semantics for real-time pipelines Understanding these architectural components matters for data engineers building modern data platforms. The metadata layer is the foundation that enables Iceberg to deliver warehouse-like reliability on data lake infrastructure. As organizations adopt streaming-first architectures with tools like Kafka and Flink, Iceberg's snapshot model provides the transactional guarantees needed for production-grade real-time analytics. ## Related Concepts - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) - [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/what-is-change-data-capture-cdc-fundamentals) - [Data Lineage: Tracking Data from Source to Consumption](https://www.conduktor.io/data-lineage-tracking-data-from-source-to-consumption) **Related Articles**: - [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg) - Comprehensive overview of Iceberg features - [Time Travel with Apache Iceberg](https://www.conduktor.io/glossary/time-travel-with-apache-iceberg) - Advanced time travel patterns - [Schema Evolution in Apache Iceberg](https://www.conduktor.io/glossary/schema-evolution-in-apache-iceberg) - Schema evolution best practices - [Iceberg Partitioning and Performance Optimization](https://www.conduktor.io/glossary/iceberg-partitioning-and-performance-optimization) - Partitioning strategies - [Maintaining Iceberg Tables: Compaction and Cleanup](https://www.conduktor.io/glossary/maintaining-iceberg-tables-compaction-and-cleanup) - Maintenance procedures - [Iceberg Catalog Management: Hive, Glue, and Nessie](https://www.conduktor.io/glossary/iceberg-catalog-management-hive-glue-and-nessie) - Catalog implementations - [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse) - Streaming architecture patterns - [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) - CDC to Iceberg integration ## Sources and References - [Apache Iceberg Official Documentation - Table Spec](https://iceberg.apache.org/spec/) - [Apache Iceberg Official Documentation - Metadata](https://iceberg.apache.org/docs/latest/metadata/) - [Netflix Technology Blog - Iceberg: A Fast Table Format](https://netflixtechblog.com/iceberg-tables-turning-the-database-inside-out-11e8b91f5677) - [Apache Flink Documentation - Iceberg Connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/iceberg/) - [Tabular Blog - Understanding Iceberg Snapshots](https://tabular.io/blog/how-iceberg-snapshots-work/) - [Dremio Blog - Iceberg Metadata Optimization](https://www.dremio.com/blog/apache-iceberg-metadata-optimization/) Source: https://www.conduktor.io/glossary/iceberg-table-architecture-metadata-and-snapshots --- # Debezium CDC with Kafka: Setup & Examples **Debezium** is an open-source CDC platform built on Kafka Connect that captures row-level changes from database transaction logs and streams them as Kafka events. It reads from the WAL rather than polling tables, capturing every insert, update, and delete with minimal impact on the source. Supported databases include MySQL, PostgreSQL, MongoDB, SQL Server, and Oracle. ## Understanding Debezium's CDC Approach Debezium captures row-level changes from database transaction logs rather than querying tables directly. This log-based approach offers several advantages: minimal impact on source database performance, capture of all change types (inserts, updates, deletes), and preservation of the exact order of operations. When a database transaction commits, the changes are first written to the transaction log (also called write-ahead log or WAL - these terms are used interchangeably throughout database systems). Debezium connectors read these logs, parse the binary format, and convert change events into structured messages that flow through Kafka topics. Each table typically maps to a dedicated topic, with each message representing a single row change. **Key Terminology:** - **Transaction Log / WAL (Write-Ahead Log)**: The database's internal log recording all changes before they're committed to disk. Used for crash recovery and replication. - **Replication Slot**: A persistent position marker in the transaction log (PostgreSQL-specific) that prevents the database from discarding log data before Debezium processes it. - **Offset**: Debezium's position tracking mechanism stored in Kafka that records how far through the transaction log it has processed. Enables fault tolerance and exactly-once processing. The key benefit of this architecture is that Debezium acts as a passive observer. Unlike trigger-based CDC solutions that add overhead to database operations, log-based CDC reads from logs that databases already maintain for replication and recovery purposes. ## Connector Architecture and Components A Debezium deployment consists of several components working together. At the core is the Kafka Connect framework, which provides the runtime environment for Debezium connectors. Each connector is database-specific (MySQL, PostgreSQL, MongoDB, SQL Server, Oracle, etc.) and understands how to parse that database's transaction log format. For comprehensive coverage of Kafka Connect architecture and patterns, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines). ![implementing-cdc-with-debezium diagram 1](https://www.conduktor.io/assets/images/glossary/implementing-cdc-with-debezium-0.webp) The connector runs as a task within Kafka Connect and maintains its own state, tracking which portion of the transaction log has been processed. This state is stored in Kafka topics, enabling fault tolerance: if a connector crashes and restarts, it resumes from where it left off without losing or duplicating events. Debezium also includes a snapshot mechanism. When a connector first starts, it can optionally perform an initial snapshot of existing table data before switching to log-based streaming. This ensures downstream systems receive both historical data and ongoing changes. ### CDC Event Structure Understanding the message format is crucial for downstream processing. Here's an example of a Debezium event for a new order insertion: ```json { "before": null, "after": { "id": 1001, "customer_id": 5, "order_date": "2025-01-15", "status": "pending", "total": 99.99 }, "source": { "version": "2.5.0.Final", "connector": "postgresql", "name": "orders_server", "ts_ms": 1705305600000, "snapshot": "false", "db": "orders_db", "sequence": "[\"123456789\",\"123456800\"]", "schema": "public", "table": "orders", "txId": 789, "lsn": 24023128, "xmin": null }, "op": "c", "ts_ms": 1705305601000, "transaction": null } ``` Key fields: - **before**: Row state before the change (null for inserts) - **after**: Row state after the change (null for deletes) - **op**: Operation type - `c` (create/insert), `u` (update), `d` (delete), `r` (read/snapshot) - **source**: Metadata about the change origin, including timestamp, database coordinates, and transaction information - **ts_ms**: Timestamp when Debezium processed the event For updates, both `before` and `after` contain values, allowing consumers to see exactly what changed. This structure supports complex downstream logic like maintaining materialized views or triggering business workflows. ## Setting Up Your First Debezium Connector Let's walk through configuring a PostgreSQL Debezium connector. Before starting, ensure you have: - Kafka and Kafka Connect running - PostgreSQL configured with logical replication enabled (`wal_level = logical`) - A replication slot and publication created for the tables you want to capture Here's a complete, production-ready connector configuration: ```json { "name": "postgres-orders-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres.example.com", "database.port": "5432", "database.user": "debezium_user", "database.password": "${env:DB_PASSWORD}", "database.dbname": "orders_db", "database.server.name": "orders_server", "plugin.name": "pgoutput", "publication.name": "dbz_publication", "publication.autocreate.mode": "filtered", "table.include.list": "public.orders,public.order_items", "topic.prefix": "cdc.postgres.orders", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "heartbeat.interval.ms": "5000", "heartbeat.topics.prefix": "__debezium-heartbeat" } } ``` Key configuration elements: - `database.server.name`: A logical name identifying this database server, used in topic naming - `plugin.name`: The PostgreSQL logical decoding plugin. `pgoutput` is the default output plugin since PostgreSQL 10+, requiring no additional installation. It's the recommended choice for modern deployments. Alternative plugins like `wal2json` and `decoderbufs` are now deprecated and should be avoided. - `publication.name`: The PostgreSQL publication that defines which tables to replicate - `publication.autocreate.mode`: Debezium 2.x+ can automatically create publications - `filtered` mode creates publications for only the specified tables - `table.include.list`: Specifies which tables to capture (use `table.exclude.list` to exclude specific tables) - `topic.prefix`: Prefix for all Kafka topics created by this connector Deploy this connector by POSTing the JSON to your Kafka Connect REST API: ```bash curl -X POST http://localhost:8083/connectors \ -H "Content-Type: application/json" \ -d @postgres-connector.json ``` ## Configuration Patterns and Best Practices **Snapshot Modes**: Control how initial snapshots are handled with `snapshot.mode`. Common values include: - `initial`: Perform a snapshot on first run (default) - `always`: Always perform a snapshot on startup - `never`: Skip snapshots, only stream changes - `when_needed`: Perform snapshot if no offset exists - `initial_only`: Perform snapshot then stop the connector - `exported`: Use database's native export functionality (PostgreSQL 15+, with improvements in PostgreSQL 16 and 17 for better performance and parallel execution) **Incremental Snapshots**: A powerful 2025 feature that allows re-snapshotting tables without stopping the connector or locking tables: ```json { "incremental.snapshot.enabled": true, "signal.data.collection": "public.debezium_signal", "snapshot.max.threads": 4, "snapshot.fetch.size": 10000 } ``` Create a signal table in your source database: ```sql CREATE TABLE debezium_signal ( id VARCHAR(42) PRIMARY KEY, type VARCHAR(32) NOT NULL, data TEXT ); ``` Trigger an incremental snapshot by inserting into the signal table: ```sql INSERT INTO debezium_signal VALUES ( 'snapshot-orders-' || NOW()::TEXT, 'execute-snapshot', '{"data-collections": ["public.orders"], "type": "incremental"}' ); ``` **Heartbeat Configuration**: Enable heartbeats to ensure offset commits during low database activity: ```json { "heartbeat.interval.ms": "5000", "heartbeat.topics.prefix": "__debezium-heartbeat" } ``` This prevents consumer lag accumulation and ensures connectors maintain their position in the transaction log. **Message Transformations**: Debezium supports Single Message Transforms (SMTs) to modify events before they're written to Kafka. The `ExtractNewRecordState` SMT is particularly useful as it simplifies the event structure by extracting just the "after" state of a row change, making downstream consumption easier. For example, without SMT, the full Debezium event structure includes `before`, `after`, `source`, and `op` fields. With `ExtractNewRecordState`: ```json { "transforms": "unwrap", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "false", "transforms.unwrap.delete.handling.mode": "rewrite", "transforms.unwrap.add.fields": "op,db,table,ts_ms" } ``` This configuration: - Extracts only the `after` state for inserts/updates - Preserves delete events by rewriting them with the `before` state - Adds metadata fields (`op`, `db`, `table`, `ts_ms`) to the simplified record - Produces flattened records easier for downstream consumers to process The simplified output looks like: ```json { "id": 1001, "customer_id": 5, "order_date": "2025-01-15", "status": "pending", "total": 99.99, "__op": "c", "__db": "orders_db", "__table": "orders", "__ts_ms": 1705305601000 } ``` For more on transformation patterns, see [Kafka Connect Single Message Transforms](https://www.conduktor.io/glossary/kafka-connect-single-message-transforms). - **Handling Schema Changes**: Debezium tracks schema evolution through a schema history topic. This ensures connectors can correctly interpret older log entries even after table schema changes. Configure appropriate retention policies for this topic to prevent data loss. - **Performance Tuning**: For high-throughput scenarios based on 2025 production benchmarks: ```json { "max.batch.size": "2048", "max.queue.size": "8192", "poll.interval.ms": "100", "producer.override.batch.size": "1000000", "producer.override.linger.ms": "500", "producer.override.compression.type": "lz4" } ``` These Kafka producer overrides can reduce snapshot times by 25% or more. Monitor connector lag using JMX metrics or Kafka Connect's REST API at `/connectors/{name}/status`. ## Kafka 4.0 and KRaft Mode Integration Debezium 2.5+ fully supports Kafka 4.0's KRaft mode, which eliminates the ZooKeeper dependency for Kafka cluster management. This represents a significant architectural improvement for production deployments, offering better scalability, simplified operations, and faster metadata propagation. When deploying Debezium with KRaft-based Kafka clusters: - **No Configuration Changes**: From the connector's perspective, KRaft mode is transparent. Your connector configurations remain identical. - **Kafka Connect Setup**: Ensure your Kafka Connect cluster is configured for KRaft by removing any `zookeeper.connect` properties and using bootstrap servers. - **Internal Topics Management**: Offset, config, and status topics are now managed by Kafka controllers instead of ZooKeeper. ```json { "offset.storage.topic": "debezium-offsets", "config.storage.topic": "debezium-configs", "status.storage.topic": "debezium-status", "offset.storage.replication.factor": "3", "config.storage.replication.factor": "3", "status.storage.replication.factor": "3", "offset.storage.partitions": "25", "config.storage.partitions": "1", "status.storage.partitions": "5" } ``` For production KRaft deployments, ensure these internal topics have appropriate replication factors (typically 3) and configure partition counts based on your connector scale. Higher partition counts for offset topics improve parallelism when running many connectors. **Performance Benefits**: KRaft mode reduces metadata latency from ~50ms to ~5ms in typical deployments, resulting in faster connector task rebalancing and improved recovery times. For detailed KRaft architecture, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## Cloud-Native Deployment Patterns Modern Debezium deployments use cloud-native patterns for better reliability and operational efficiency. ### Kubernetes with Strimzi The Strimzi operator provides declarative Kafka Connect management on Kubernetes, with first-class support for Debezium connectors: ```yaml apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: debezium-connect-cluster annotations: strimzi.io/use-connector-resources: "true" spec: version: 3.6.0 replicas: 3 bootstrapServers: kafka-cluster-kafka-bootstrap:9093 tls: trustedCertificates: - secretName: kafka-cluster-cluster-ca-cert certificate: ca.crt config: group.id: debezium-connect-cluster offset.storage.topic: debezium-offsets config.storage.topic: debezium-configs status.storage.topic: debezium-status config.storage.replication.factor: 3 offset.storage.replication.factor: 3 status.storage.replication.factor: 3 build: output: type: docker image: my-registry.io/debezium-connect:2.5.0 plugins: - name: debezium-postgres-connector artifacts: - type: tgz url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/2.5.0.Final/debezium-connector-postgres-2.5.0.Final-plugin.tar.gz - name: debezium-mysql-connector artifacts: - type: tgz url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/2.5.0.Final/debezium-connector-mysql-2.5.0.Final-plugin.tar.gz resources: requests: memory: "2Gi" cpu: "1000m" limits: memory: "4Gi" cpu: "2000m" ``` Deploy individual connectors as Kubernetes resources: ```yaml apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnector metadata: name: postgres-orders-connector labels: strimzi.io/cluster: debezium-connect-cluster spec: class: io.debezium.connector.postgresql.PostgresConnector tasksMax: 1 config: database.hostname: postgres.database.svc.cluster.local database.port: "5432" database.user: debezium database.password: "${secret:postgres-credentials:password}" database.dbname: orders_db database.server.name: orders-server plugin.name: pgoutput publication.name: dbz_publication publication.autocreate.mode: filtered table.include.list: public.orders,public.order_items topic.prefix: cdc.postgres.orders heartbeat.interval.ms: "5000" heartbeat.topics.prefix: "__debezium-heartbeat" ``` **Key Kubernetes Benefits**: - **Declarative Management**: Define connectors as YAML resources managed by GitOps workflows - **Automatic Scaling**: Scale Kafka Connect pods based on CPU/memory metrics - **Secret Management**: Integrate with Kubernetes secrets for database credentials - **Health Monitoring**: Leverage Kubernetes liveness/readiness probes for automatic recovery For comprehensive Kubernetes deployment patterns, see [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) and [Strimzi Kafka Operator](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes). ### Debezium Server (Standalone Mode) Debezium Server 2.x provides a lightweight alternative to Kafka Connect for scenarios where you need CDC without full Kafka Connect infrastructure. It can stream changes to Kafka, Amazon Kinesis, Google Pub/Sub, Apache Pulsar, or custom sinks. **Configuration Example** (`application.properties`): ```properties # Source Configuration (PostgreSQL) debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector debezium.source.offset.storage.file.filename=/debezium/offsets/offsets.dat debezium.source.offset.flush.interval.ms=5000 debezium.source.database.hostname=postgres.example.com debezium.source.database.port=5432 debezium.source.database.user=debezium debezium.source.database.password=${DB_PASSWORD} debezium.source.database.dbname=inventory debezium.source.topic.prefix=dbserver1 debezium.source.plugin.name=pgoutput debezium.source.publication.autocreate.mode=filtered debezium.source.table.include.list=public.orders,public.customers # Sink Configuration (Kafka) debezium.sink.type=kafka debezium.sink.kafka.producer.bootstrap.servers=kafka:9092 debezium.sink.kafka.producer.key.serializer=org.apache.kafka.common.serialization.StringSerializer debezium.sink.kafka.producer.value.serializer=org.apache.kafka.common.serialization.StringSerializer debezium.sink.kafka.producer.acks=all debezium.sink.kafka.producer.compression.type=lz4 # For cloud storage sinks (alternative to Kafka) # debezium.sink.type=kinesis # debezium.sink.kinesis.region=us-east-1 ``` **When to Use Debezium Server**: - **Edge Deployments**: Running CDC at remote locations with limited infrastructure - **Serverless Environments**: Lambda, Cloud Run, or container instances where full Kafka Connect is overkill - **Non-Kafka Targets**: Streaming directly to Kinesis, Pub/Sub, or custom HTTP endpoints - **Simplified Operations**: Single-process deployment with minimal operational overhead Deploy as a container: ```dockerfile FROM quay.io/debezium/server:2.5 COPY application.properties /debezium/conf/ EXPOSE 8080 ``` Debezium Server includes a health endpoint at `/q/health` for monitoring and a Prometheus metrics endpoint at `/q/metrics` for observability. ## Integrating with the Kafka Ecosystem Debezium connectors produce events to Kafka topics, making them immediately available to the broader Kafka ecosystem. This integration enables several powerful patterns: - **Stream Processing**: Use Kafka Streams, ksqlDB, or Apache Flink to process CDC events in real-time. For example, joining order changes with customer data to build enriched materialized views. Apache Flink has become the industry standard for stateful stream processing in 2025, offering superior state management and exactly-once guarantees. For a detailed comparison, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink). - **Data Lake Ingestion**: Connect Debezium to sink connectors that write to object storage or data warehouses, creating an automated pipeline from operational databases to analytical systems. For lakehouse integration, see [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables) and [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse). - **Event-Driven Architecture**: CDC events can trigger downstream microservices. An order status change captured by Debezium might trigger fulfillment systems, notification services, or analytics pipelines. For related patterns, see [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing) and [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions). - **Schema Management**: Integrate Debezium with Schema Registry to enforce schema evolution rules and maintain a centralized schema catalog. Debezium 2.x+ supports multiple serialization formats: - **Avro with Schema Registry** (recommended for production): ```json { "key.converter": "io.confluent.connect.avro.AvroConverter", "key.converter.schema.registry.url": "http://schema-registry:8081", "key.converter.schemas.enable": true, "value.converter": "io.confluent.connect.avro.AvroConverter", "value.converter.schema.registry.url": "http://schema-registry:8081", "value.converter.schemas.enable": true } ``` **Protobuf** (Debezium 2.1+, ideal for high-performance scenarios): ```json { "key.converter": "io.confluent.connect.protobuf.ProtobufConverter", "key.converter.schema.registry.url": "http://schema-registry:8081", "value.converter": "io.confluent.connect.protobuf.ProtobufConverter", "value.converter.schema.registry.url": "http://schema-registry:8081" } ``` **JSON Schema** (balance between readability and schema enforcement): ```json { "key.converter": "io.confluent.connect.json.JsonSchemaConverter", "key.converter.schema.registry.url": "http://schema-registry:8081", "value.converter": "io.confluent.connect.json.JsonSchemaConverter", "value.converter.schema.registry.url": "http://schema-registry:8081" } ``` Schema Registry ensures backward and forward compatibility as database schemas evolve, preventing breaking changes from impacting downstream consumers. For comprehensive schema management strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). **Monitoring and Governance**: Governance platforms provide visibility into Debezium topics, helping teams monitor data quality, track schema evolution, and govern access to sensitive CDC streams. Conduktor offers comprehensive data governance capabilities including [data masking](https://docs.conduktor.io/guide/conduktor-concepts/interceptors) for CDC pipelines containing PII and quality monitoring through [topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) - all without modifying connector configurations. This allows teams to enforce privacy policies and compliance requirements centrally. ## Common Patterns and Use Cases - **Database Replication**: Replicate data from operational databases to read replicas, analytics databases, or search indexes without impacting source database performance. - **Event Sourcing**: Capture all database changes as immutable events, creating an audit trail and enabling temporal queries. - **Cache Invalidation**: Propagate database changes to distributed caches in real-time, ensuring cache consistency without complex invalidation logic. - **Microservices Data Synchronization**: Share data between microservices while maintaining service autonomy. Each service consumes relevant CDC topics to maintain its own local copy of data owned by other services. For architectural considerations, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) for foundational concepts and [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) for consumption patterns. ## Monitoring and Troubleshooting Successful Debezium deployments require comprehensive observability. Modern Debezium 2.5+ includes enhanced monitoring capabilities integrated with cloud-native observability stacks. ### Prometheus Metrics and Grafana Dashboards Debezium exposes detailed metrics via JMX and Prometheus endpoints. Key metrics to track: ```yaml # Kafka Connect with Prometheus JMX exporter apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: debezium-connect-cluster spec: metricsConfig: type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: connect-metrics key: metrics-config.yml ``` **Critical Metrics**: - `debezium_metrics_MilliSecondsSinceLastEvent`: Lag between database change and Debezium processing - `debezium_metrics_TotalNumberOfEventsSeen`: Total events processed (monitors throughput) - `debezium_metrics_NumberOfEventsFiltered`: Events excluded by filters (validates configuration) - `debezium_metrics_SnapshotRunning`: Snapshot status (0 = complete, 1 = in progress) - `debezium_metrics_Connected`: Database connection health - `kafka_connect_connector_status`: Connector state (1 = running, 0 = failed) Grafana dashboards are available from the Debezium community, providing pre-built visualizations for connector health, throughput, lag, and snapshot progress. ### OpenTelemetry Integration Debezium 2.5+ supports OpenTelemetry for distributed tracing, enabling end-to-end visibility from database transaction to Kafka consumer: ```json { "opentelemetry.enabled": "true", "opentelemetry.trace.enabled": "true", "opentelemetry.metrics.enabled": "true" } ``` This creates trace spans for each CDC event, allowing correlation of database changes with downstream processing in tools like Jaeger or Zipkin. ### Common Issues and Solutions **Replication Slot Growth**: If a connector is stopped for extended periods, PostgreSQL replication slots can accumulate WAL files, consuming disk space. Monitor with: ```sql SELECT slot_name, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS lag_size FROM pg_replication_slots; ``` Mitigation: Set `max_wal_size` and `wal_keep_size` in PostgreSQL configuration, and configure alerts when lag exceeds thresholds (e.g., 10GB). Consider using `slot.drop.on.stop=true` for non-production connectors. **Connector Task Failures**: Check logs via Kafka Connect API: ```bash curl http://localhost:8083/connectors/postgres-orders-connector/status | jq ``` Common causes: database connection issues, insufficient permissions, or schema changes that violate compatibility rules. **High Memory Usage During Snapshots**: Large initial snapshots can cause OOM errors. Solutions: - Reduce `snapshot.fetch.size` (default 10000) - Increase Kafka Connect heap size - Use `snapshot.mode=exported` for PostgreSQL 15+ (offloads snapshot to database) - Enable incremental snapshots to spread load over time **Network Partitions**: Debezium connectors must maintain persistent connections to both the database and Kafka. Network issues can cause connector failures. Implement: - Connection retry policies: `database.initial.statements` for connection validation - Health checks via Kafka Connect REST API - Alerting on connector state changes For comprehensive monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ## Security Configuration Production Debezium deployments require multiple security layers to protect sensitive data in transit and at rest. ### Database Connection Security **SSL/TLS for PostgreSQL**: ```json { "database.sslmode": "verify-full", "database.sslcert": "/path/to/client-cert.pem", "database.sslkey": "/path/to/client-key.pem", "database.sslrootcert": "/path/to/ca-cert.pem" } ``` **SSL/TLS for MySQL**: ```json { "database.ssl.mode": "required", "database.ssl.truststore": "/path/to/truststore.jks", "database.ssl.truststore.password": "${secret:db-ssl:truststore-password}", "database.ssl.keystore": "/path/to/keystore.jks", "database.ssl.keystore.password": "${secret:db-ssl:keystore-password}" } ``` ### Kafka Security Integration **SASL/SCRAM Authentication with TLS**: ```json { "producer.security.protocol": "SASL_SSL", "producer.sasl.mechanism": "SCRAM-SHA-512", "producer.sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"debezium-user\" password=\"${secret:kafka-creds:password}\";", "producer.ssl.truststore.location": "/path/to/kafka.truststore.jks", "producer.ssl.truststore.password": "${secret:kafka-ssl:truststore-password}" } ``` **mTLS (Mutual TLS)**: ```json { "producer.security.protocol": "SSL", "producer.ssl.truststore.location": "/path/to/kafka.truststore.jks", "producer.ssl.truststore.password": "${secret:kafka-ssl:truststore-password}", "producer.ssl.keystore.location": "/path/to/kafka.keystore.jks", "producer.ssl.keystore.password": "${secret:kafka-ssl:keystore-password}", "producer.ssl.key.password": "${secret:kafka-ssl:key-password}" } ``` ### Secrets Management **Kubernetes Secrets** (with Strimzi): ```yaml apiVersion: v1 kind: Secret metadata: name: postgres-credentials type: Opaque stringData: password: "secure-db-password" --- apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnector spec: config: database.password: "${secret:postgres-credentials:password}" ``` **External Secrets Providers** (HashiCorp Vault, AWS Secrets Manager): ```json { "config.providers": "vault", "config.providers.vault.class": "io.confluent.connect.secretregistry.ConnectSecretRegistryConfigProvider", "config.providers.vault.param.master.encryption.key": "${VAULT_TOKEN}", "database.password": "${vault:debezium/postgres:password}" } ``` ### Data Masking and PII Protection For sensitive data in CDC streams, implement field-level transformations: ```json { "transforms": "maskPII", "transforms.maskPII.type": "org.apache.kafka.connect.transforms.MaskField$Value", "transforms.maskPII.fields": "ssn,credit_card", "transforms.maskPII.replacement": "****MASKED****" } ``` For more sophisticated data governance including dynamic masking, tokenization, and role-based access, Conduktor provides governance capabilities that work with Debezium CDC pipelines. For comprehensive security patterns, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices), [Kafka Authentication (SASL, SSL, OAuth)](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth), and [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka). ## Summary Debezium provides a production-ready platform for implementing Change Data Capture across various database systems. By leveraging transaction logs, it captures database changes with minimal overhead while maintaining strict ordering guarantees and exactly-once semantics. Modern Debezium 2.5+ deployments in 2025 benefit from significant enhancements: Kafka 4.0 KRaft integration, cloud-native deployment patterns via Kubernetes and Strimzi, Debezium Server for lightweight scenarios, comprehensive observability with OpenTelemetry and Prometheus, and production-grade security with mTLS and secrets management. The key to successful implementation lies in understanding your specific requirements: choosing appropriate snapshot modes, configuring message transformations, and integrating with downstream consumers. When combined with Kafka's distributed architecture and ecosystem tools, Debezium enables real-time data pipelines that bridge operational and analytical systems. Start with a single table connector in a non-production environment to understand the event format and behavior. Gradually expand to more tables, implement monitoring, and integrate with downstream systems. This incremental approach reduces risk while building team expertise with CDC patterns. For broader streaming architecture context, see [Streaming Data Pipeline](https://www.conduktor.io/glossary/streaming-data-pipeline), [Real-Time Data Streaming](https://www.conduktor.io/glossary/what-is-real-time-data-streaming), and [Streaming ETL vs Traditional ETL](https://www.conduktor.io/glossary/streaming-etl-vs-traditional-etl). ## Related Concepts - [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) - Foundational CDC concepts and patterns that underpin Debezium's log-based approach to capturing database changes. - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines) - Understand the Kafka Connect framework that Debezium runs on for distributed, fault-tolerant CDC processing. - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Learn how Debezium leverages Kafka's exactly-once guarantees to ensure reliable CDC event delivery without duplicates. ## Sources and References - [Debezium Documentation](https://debezium.io/documentation/) - [Debezium Connector for PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) - [PostgreSQL Logical Replication](https://www.postgresql.org/docs/current/logical-replication.html) - [Kafka Connect Documentation](https://kafka.apache.org/documentation/#connect) - [Debezium Tutorial](https://debezium.io/documentation/reference/stable/tutorial.html) - [Single Message Transforms (SMTs)](https://kafka.apache.org/documentation/#connect_transforms) Source: https://www.conduktor.io/glossary/implementing-cdc-with-debezium --- # Infrastructure as Code for Kafka Deployments Managing Apache Kafka deployments at scale presents unique challenges. Unlike stateless applications, Kafka requires careful orchestration of brokers, persistent storage, network configurations, and numerous operational resources like topics, ACLs, and connectors. Infrastructure as Code (IaC) has emerged as a critical practice for teams seeking to deploy Kafka reliably across multiple environments while maintaining consistency and reducing human error. ## What is Infrastructure as Code? Infrastructure as Code is the practice of managing and provisioning infrastructure through machine-readable definition files rather than manual configuration or interactive tools. Instead of clicking through cloud consoles or running ad-hoc commands, teams define their desired infrastructure state in code that can be version-controlled, reviewed, and automatically applied. The benefits extend beyond automation. IaC provides documentation as code, your infrastructure definitions become the source of truth for how systems are configured. Changes go through the same review processes as application code, creating an audit trail and enabling rollbacks when issues arise. For distributed systems like Kafka, where configuration drift can lead to subtle but serious problems, this consistency is invaluable. ![Infrastructure as Code workflow for Kafka deployments](https://www.conduktor.io/assets/images/glossary/infrastructure-as-code-for-kafka-deployments-0.webp) ## The Kafka Infrastructure Challenge Kafka deployments involve multiple layers of infrastructure that must work in harmony. At the foundation, you need compute resources (virtual machines or containers) with appropriate CPU, memory, and network configurations. Storage requires careful planning, Kafka brokers need persistent volumes that can handle high throughput and provide the necessary durability guarantees. Beyond the infrastructure layer, Kafka has its own configuration complexity. Historically, clusters depended on ZooKeeper for coordination, adding another distributed system to manage. With Kafka 4.0+ (released in 2024), KRaft mode has become the default and recommended approach, eliminating ZooKeeper entirely. KRaft mode introduces new considerations around controller quorum setup, dedicated controller nodes that manage cluster metadata using Kafka's own Raft consensus protocol. Each broker requires specific configurations for log retention, replication factors, and inter-broker communication. For more details on KRaft architecture, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). On top of the cluster itself, operational resources multiply quickly. A production Kafka deployment typically includes dozens or hundreds of topics, each with specific partition counts, replication settings, and retention policies (see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) for best practices). Security configurations add access control lists, user credentials, and encryption settings, covered in detail in [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) and [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices). Kafka Connect deployments require connector configurations and associated infrastructure (see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines)). Schema registries need their own setup and version management (see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management)). Managing this complexity manually is error-prone and doesn't scale. When you need to replicate environments for development, staging, and production, or deploy across multiple regions, manual processes break down quickly. ## IaC Tools for Kafka Deployments Several tools have emerged to tackle Kafka infrastructure management, each with different strengths. - **Terraform** has become a popular choice for Kafka deployments. The Terraform Kafka provider allows you to define topics, ACLs, and quotas as code. Combined with cloud provider modules, you can manage the entire stack from virtual machines to Kafka resources in a single workflow. Terraform's state management and plan-preview capabilities help teams understand changes before applying them. - **Kubernetes Operators** like Strimzi provide a cloud-native approach to Kafka deployment. These operators use custom resources to define Kafka clusters declaratively, handling the complexity of pod scheduling, storage provisioning, and rolling updates. The operator pattern fits naturally with containerized environments and provides built-in automation for common operational tasks. - **Ansible** offers procedural automation through playbooks that can configure both infrastructure and Kafka resources. While less declarative than Terraform or operators, Ansible's flexibility makes it suitable for complex migration scenarios or organizations with existing Ansible investments. - **Cloud Provider Modules** enable integration with managed Kafka services like AWS MSK, Azure Event Hubs, and Google Cloud Pub/Sub. These modules handle cloud-specific configurations while maintaining the benefits of infrastructure as code across hybrid environments. ## GitOps and Continuous Deployment GitOps has emerged as a powerful pattern for managing Kafka infrastructure in 2025. GitOps extends IaC principles by using Git as the single source of truth and automatically synchronizing infrastructure state with Git repository contents. **ArgoCD** and **FluxCD** are the leading GitOps tools for Kubernetes-based Kafka deployments. These tools continuously monitor Git repositories and automatically apply changes to Kubernetes clusters, including Kafka resources managed by operators like Strimzi. When you update a Kafka cluster definition in Git, ArgoCD or FluxCD detects the change and orchestrates the rolling update automatically, with built-in rollback capabilities if issues arise. GitOps patterns provide several advantages for Kafka operations: - **Audit trail**: Every infrastructure change is a Git commit, providing complete history of who changed what and when - **Drift detection**: GitOps tools continuously reconcile actual cluster state with desired state in Git, alerting on manual changes - **Disaster recovery**: Entire Kafka deployments can be reconstructed from Git repository contents - **Multi-cluster management**: Single Git repository can define infrastructure across development, staging, production, and multiple regions For Kubernetes-based Kafka deployments, see [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) and [Strimzi Kafka Operator for Kubernetes](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes) for detailed implementation guidance. ## Cloud-Native and Managed Services Modern Kafka deployments increasingly use cloud provider services that work well with IaC workflows: - **AWS MSK (Managed Streaming for Apache Kafka)** now supports KRaft mode and can be provisioned entirely through Terraform or CloudFormation. MSK handles broker provisioning, patching, and scaling while allowing teams to manage topics, ACLs, and Connect configurations as code. - **Azure Event Hubs** provides a Kafka-compatible API and integrates with Azure Resource Manager templates and Terraform. Teams can define Event Hubs namespaces and event hubs (equivalent to topics) declaratively alongside other Azure resources. - **Google Cloud Pub/Sub** offers Kafka-compatible endpoints through its Pub/Sub Lite service, manageable via Terraform and Google Cloud Deployment Manager. These managed services reduce operational burden while maintaining IaC benefits. Teams define cluster configurations, security policies, and monitoring as code without managing underlying infrastructure. For production deployments, combining managed control planes with IaC for application-level resources (topics, schemas, connectors) provides an optimal balance of convenience and control. ## Infrastructure as Code in Data Streaming Environments Data streaming platforms present unique IaC requirements that distinguish them from traditional application infrastructure. Streaming systems are inherently stateful and often run continuously for months or years. Any infrastructure change must account for in-flight data and avoid disrupting processing pipelines. IaC enables streaming teams to maintain consistency across environments while adapting configurations to each environment's needs. Your development environment might use single-broker Kafka with minimal replication, while production requires a multi-zone cluster with replication factor three. With IaC, these differences are parameterized rather than ad-hoc, reducing the risk of configuration drift causing issues. Disaster recovery planning becomes more robust with IaC. When infrastructure definitions exist as code, rebuilding a failed cluster or creating a new cluster in a different region becomes a repeatable process rather than an emergency scramble to remember undocumented settings. Teams can even test their recovery procedures by regularly spinning up new clusters from code in test environments. For multi-region deployments, see [Kafka MirrorMaker 2 for Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication). Multi-tenant streaming platforms particularly benefit from IaC. When onboarding a new team or application, infrastructure teams can apply tested templates that provision topics, service accounts, ACLs, and monitoring in a standardized way. This self-service approach scales better than manual provisioning while maintaining security and operational standards. For multi-tenancy patterns, see [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments). ## Best Practices for Kafka IaC Successful Kafka IaC implementations follow several key practices. Version control is foundational, all infrastructure definitions should live in Git repositories with the same protections as application code. This enables code review, creates an audit trail, and allows teams to understand how infrastructure evolved over time. Separate environments should use the same IaC code with different parameters or variable files. This "write once, deploy many" approach ensures that your production infrastructure actually matches what was tested in lower environments. Consider using workspaces in Terraform or separate namespaces in Kubernetes to isolate environments while sharing code. Test infrastructure changes before applying them to production. Terraform's `plan` command and Kubernetes' dry-run capabilities let you preview changes. For critical changes like modifying partition counts or replication factors, test in a non-production environment first to verify the impact. Implement proper secret management. Never commit credentials or sensitive configuration to version control. Use secret management tools like HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets, and reference them in your IaC code. Document non-obvious decisions through code comments and README files. While IaC serves as documentation, context about why certain configurations were chosen helps future maintainers understand the system. ## Real-World Implementation Example Here's a practical example using Terraform to manage Kafka topics. This snippet demonstrates how teams can define topics with specific configurations and manage them as code: ```hcl terraform { required_providers { kafka = { source = "Mongey/kafka" version = "~> 0.8" } } } provider "kafka" { bootstrap_servers = var.kafka_bootstrap_servers tls_enabled = true # For KRaft mode clusters (Kafka 4.0+), no ZooKeeper configuration needed } resource "kafka_topic" "orders" { name = "customer.orders" replication_factor = 3 partitions = 12 config = { "retention.ms" = "604800000" # 7 days "compression.type" = "snappy" "min.insync.replicas" = "2" "cleanup.policy" = "delete" } } resource "kafka_acl" "orders_producer" { resource_type = "Topic" resource_name = kafka_topic.orders.name acl_principal = "User:order-service" acl_operation = "Write" acl_permission_type = "Allow" } ``` This code defines a topic with production-appropriate settings: three-way replication for durability, 12 partitions for parallelism, and a seven-day retention period. The ACL grants write permissions to the order service. Running `terraform apply` creates these resources, and changes to the code trigger updates to keep infrastructure aligned with the definition. ## Complementary Tooling and Testing While IaC tools handle infrastructure provisioning, additional platforms enhance the complete deployment lifecycle: **Conduktor** provides comprehensive Kafka management capabilities that complement IaC workflows. Conduktor's platform includes: - **Configuration management**: Define topic standards, naming conventions, and governance policies that validate IaC-created resources against organizational requirements - **Testing integration**: [Conduktor Gateway, a Kafka proxy, enables chaos engineering and integration testing](https://docs.conduktor.io/guide/use-cases/chaos-testing) for Kafka deployments. Teams can test how applications handle broker failures, network partitions, or replication lag, validating that IaC-defined resilience settings actually work - **Observability**: Real-time [monitoring of topics, consumer groups, and cluster health](https://docs.conduktor.io/guide/monitor-brokers-apps/index), helping validate that deployed infrastructure performs as expected (see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) for comprehensive monitoring strategies) - **Security governance**: Enforces data masking, encryption, and access policies across environments, ensuring IaC-defined security configurations align with compliance requirements - **Terraform automation**: [Automate Conduktor configuration with Terraform](https://docs.conduktor.io/guide/conduktor-in-production/automate/terraform-automation) for complete infrastructure-as-code workflows For example, after Terraform provisions a new topic, Conduktor can automatically verify it meets naming standards, has appropriate retention policies, and complies with data classification rules. Conduktor Gateway can then inject failures to test that the replication factor and min.insync.replicas settings actually provide the expected durability. This layered approach, IaC for provisioning, complementary platforms for governance and testing, creates robust, well-tested Kafka deployments. For comprehensive testing approaches, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). For sizing infrastructure correctly, see [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning). ## Summary Infrastructure as Code transforms Kafka deployments from manual, error-prone processes into repeatable, testable, and auditable workflows. By codifying infrastructure decisions, teams gain consistency across environments, simplified disaster recovery, and the ability to scale operations without proportionally scaling operational overhead. The specific tools matter less than the principles: version-controlled definitions, automated application of changes, separation of environments through parameterization, and thorough testing before production deployment. Whether using Terraform, Kubernetes operators, or other tools, these practices enable teams to manage Kafka's inherent complexity while maintaining reliability. For operational best practices beyond IaC, see [Kafka Admin Operations and Maintenance](https://www.conduktor.io/glossary/kafka-admin-operations-and-maintenance). As streaming platforms grow in scale, IaC becomes a necessity. Infrastructure automation tools combined with platforms for resource management make reliable streaming data systems achievable. ## Related Concepts - [Strimzi Kafka Operator for Kubernetes](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes) - Kubernetes-native IaC approach for Kafka using operators and custom resources with GitOps workflows. - [CI/CD Best Practices for Streaming Applications](https://www.conduktor.io/glossary/cicd-best-practices-for-streaming-applications) - Integrate IaC-managed Kafka infrastructure into continuous deployment pipelines. - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Use capacity planning insights to parameterize IaC templates for different environment sizes and workloads. ## Sources and References 1. HashiCorp Terraform Kafka Provider Documentation - https://registry.terraform.io/providers/Mongey/kafka/latest/docs 2. Strimzi: Kubernetes Operator for Apache Kafka - https://strimzi.io/documentation/ 3. Confluent: Automation and Infrastructure as Code - https://docs.confluent.io/platform/current/installation/overview.html 4. Martin Fowler, "Infrastructure as Code" - https://martinfowler.com/bliki/InfrastructureAsCode.html 5. Apache Kafka Operations Documentation - https://kafka.apache.org/documentation/#operations Source: https://www.conduktor.io/glossary/infrastructure-as-code-for-kafka-deployments --- # Integrating LLMs with Streaming Platforms Large Language Models (LLMs) have become powerful tools for understanding and generating human-like text, while streaming platforms like Apache Kafka and Apache Flink excel at processing high-volume, real-time data. Combining these technologies enables organizations to build intelligent, event-driven applications that can analyze, enrich, and respond to data as it flows through their systems. The 2025 LLM landscape offers diverse options: proprietary models like GPT-4, Claude 3.5 Sonnet, and Gemini 2.0 provide strong capabilities with streaming APIs, while open-source alternatives like Llama 3.3, Mistral, and Qwen enable on-premises deployment for sensitive data. Modern LLMs support structured outputs, function calling, and streaming responses, features that integrate naturally with event-driven architectures. This article covers the technical patterns, implementation approaches, and practical considerations for integrating LLMs with streaming platforms. ![LLM integration with streaming data architecture](https://www.conduktor.io/assets/images/glossary/integrating-llms-with-streaming-platforms-0.webp) ## Why Integrate LLMs with Streaming Platforms? The integration of LLMs with streaming platforms addresses several business and technical needs. Real-time AI applications require immediate responses to events, whether that's analyzing customer support tickets, detecting fraudulent transactions, or moderating user-generated content. Streaming platforms provide the infrastructure to handle high-throughput data ingestion and distribution, while LLMs add natural language understanding and generation capabilities. Together, they enable use cases that neither technology could handle effectively alone. Traditional batch processing approaches introduce latency that makes them unsuitable for time-sensitive applications. When a customer submits a support ticket, waiting hours for batch processing to analyze and route it creates poor user experiences. Streaming architectures allow events to trigger LLM inference immediately, reducing response times from hours to seconds. Streaming platforms also offer built-in reliability features like message persistence, replay capabilities, and exactly-once semantics. These features are essential when dealing with LLM operations that may be expensive, non-deterministic, or subject to rate limits. ## RAG and Real-Time Context Retrieval Augmented Generation (RAG) has become a foundational pattern for building LLM applications that need fresh, domain-specific knowledge. Traditional RAG systems query static knowledge bases, but streaming architectures enable real-time RAG pipelines where context updates continuously as events flow through the system. For detailed coverage of RAG with streaming data, see [RAG Pipelines with Real-Time Data](https://www.conduktor.io/glossary/rag-pipelines-with-real-time-data). In a streaming RAG architecture, events trigger vector embedding generation, which updates vector databases like Pinecone, Weaviate, or Qdrant in real-time. When an LLM inference request arrives, the system retrieves the most current relevant documents, ensuring responses reflect the latest information. This pattern is essential for customer support systems that need to reference recent interactions, fraud detection that learns from new attack patterns, or recommendation engines that adapt to trending content. For real-time embedding strategies, see [Vector Embeddings in Streaming](https://www.conduktor.io/glossary/vector-embeddings-in-streaming). For architectural patterns connecting vector databases with streaming platforms, see [Vector Databases and Streaming Architectures](https://www.conduktor.io/glossary/vector-databases-and-streaming-architectures). Streaming platforms handle the orchestration: Kafka topics carry raw events, stream processors generate embeddings (either via embedding APIs or local models), and change data capture streams database updates back into Kafka for downstream consumers. This creates feedback loops where LLM outputs become inputs for future context, continuously improving relevance. For CDC patterns that enable these feedback loops, see [What is Change Data Capture? CDC Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). ## Common Integration Patterns Several architectural patterns have emerged for integrating LLMs with streaming platforms. ### Consumer-Transform-Producer Pattern The most straightforward pattern involves consuming events from a topic, sending them to an LLM API, and producing the results to another topic. A microservice or stream processing application reads messages from an input topic, constructs prompts for the LLM, calls the API, and writes the responses to an output topic. For example, a support ticket system might consume tickets from a `support-requests` topic, use an LLM to classify urgency and extract key information, then produce enriched tickets to a `classified-tickets` topic for routing to the appropriate team. ### Event-Driven Inference In this pattern, specific events trigger LLM inference rather than processing every message. A stream processor filters events based on criteria, then invokes the LLM only for relevant cases. This reduces costs and latency by avoiding unnecessary API calls. A fraud detection system might only invoke an LLM when a transaction exceeds certain thresholds or matches suspicious patterns, rather than analyzing every transaction. ### Streaming ETL for LLM Training Organizations can use streaming platforms to collect and prepare training data for fine-tuning LLMs. Events flow through transformation pipelines that clean, anonymize, and structure data before storing it in formats suitable for model training. This pattern creates feedback loops where production data continuously improves model performance, though it requires careful attention to data quality and privacy. For ETL patterns, see [Streaming ETL vs Traditional ETL](https://www.conduktor.io/glossary/streaming-etl-vs-traditional-etl). ### Kafka Connect for LLM Integration Kafka Connect provides a declarative approach for integrating LLMs with streaming data through custom connectors or single message transforms (SMTs). Organizations can build source connectors that poll LLM APIs for batch inference results, or sink connectors that send batched events to LLM endpoints. For Kafka Connect fundamentals, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines). For transformation patterns, see [Kafka Connect Single Message Transforms](https://www.conduktor.io/glossary/kafka-connect-single-message-transforms). While less flexible than stream processors for complex workflows, Kafka Connect simplifies deployment and operations for straightforward LLM enrichment use cases. It's particularly effective for batch-oriented LLM operations where latency requirements are measured in seconds rather than milliseconds. ## Technical Implementation Approaches Implementing LLM integration with streaming platforms involves several technical considerations. ### Using Kafka Streams or Flink Stream processing frameworks like Kafka Streams and Apache Flink provide natural integration points for LLM calls. Modern implementations use structured outputs (JSON Schema or function calling) to ensure consistent parsing, and async patterns to handle LLM latency without blocking. A Kafka Streams application can implement custom processors that invoke LLM APIs with proper error handling: ```java StreamsBuilder builder = new StreamsBuilder(); KStream tickets = builder.stream("support-requests"); KStream enriched = tickets.mapValues(ticket -> { try { // Build prompt with structured output schema String prompt = buildPrompt(ticket); JsonSchema schema = JsonSchema.builder() .properties("urgency", "string") .properties("category", "string") .properties("summary", "string") .required("urgency", "category", "summary") .build(); // Use structured outputs for reliable parsing (GPT-4, Claude 3.5+) LLMResponse response = llmClient.completeWithSchema(prompt, schema); return enrichTicket(ticket, response); } catch (RateLimitException e) { // Send to retry topic with backoff throw new RetryableException(e); } catch (LLMException e) { // Send to dead letter queue for manual review sendToDeadLetterQueue(ticket, e); return ticket; // Pass through unchanged } }); enriched.to("enriched-tickets"); ``` For foundational Kafka Streams concepts, see [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). Flink offers similar capabilities with its AsyncFunction interface for non-blocking LLM calls, allowing stateful processing and access to timers for handling LLM timeouts. For Flink processing patterns, see [What is Apache Flink? Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). ### Managing Latency and Throughput LLM API calls introduce latency that can impact stream processing throughput. A single LLM inference might take 500ms to several seconds, which becomes a bottleneck when processing high-volume streams. Parallelization helps by running multiple stream processor instances, each handling a partition of the input topic. Batching requests where the LLM API supports it can improve throughput, though it trades off latency. Some teams implement caching layers to avoid redundant LLM calls for similar inputs. A Redis cache keyed by prompt content can serve repeated queries without API calls, though cache invalidation requires careful consideration. ### Streaming LLM Responses Modern LLM APIs support streaming responses that emit tokens incrementally rather than waiting for complete generation. This pattern reduces perceived latency in real-time applications, allowing downstream systems to process partial results while generation continues. In event-driven architectures, streaming LLM responses can be captured as multiple events in a Kafka topic. Each token or token chunk becomes an event, enabling real-time display in user interfaces, progressive processing in downstream services, or early termination when sufficient information is available. This is particularly valuable for chatbots, content generation systems, and interactive analytics where immediate feedback improves user experience. Kafka Streams or Flink can aggregate streaming tokens back into complete messages for services that require full context, while still enabling low-latency display layers to consume individual tokens as they arrive. ### Function Calling and Tool Use 2025 LLMs support function calling (also called tool use), allowing models to invoke external APIs and integrate results into their reasoning. In streaming architectures, this creates powerful patterns where LLMs can query databases, fetch real-time data from Kafka topics, or trigger actions in external systems, forming the foundation for agentic AI systems. A fraud detection LLM might use function calling to query recent transaction history from a state store, check velocity rules in a feature store, or look up customer risk profiles from a database, all within a single inference request. The streaming platform orchestrates these lookups, maintains state stores for fast access, and ensures consistent data across function calls. For managing state in stream processing, see [State Stores in Kafka Streams](https://www.conduktor.io/glossary/state-stores-in-kafka-streams). For autonomous agents built on streaming data, see [Agentic AI Pipelines: Streaming Data for Autonomous Agents](https://www.conduktor.io/glossary/agentic-ai-pipelines). ### Error Handling and Reliability LLM APIs can fail due to rate limits, timeouts, or service issues. Robust implementations include retry logic with exponential backoff, circuit breakers to prevent cascading failures, and dead letter queues for messages that repeatedly fail processing. For patterns on handling failed messages, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). Monitoring API rate limits and costs is essential. Stream processors should track metrics like API call volume, latency percentiles, token usage, and error rates. Platforms like Conduktor provide comprehensive monitoring for Kafka-based LLM pipelines, offering visibility into message flow, [topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), processing metrics, consumer lag, and cost tracking across the entire system. For Kafka-specific monitoring approaches, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) and [Consumer Lag: Monitoring and Managing Streaming Health](https://www.conduktor.io/glossary/consumer-lag-monitoring). ## Challenges and Best Practices Several challenges arise when integrating LLMs with streaming platforms. - **Cost Management**: LLM API calls can be expensive at scale. Track token usage carefully and consider strategies like prompt optimization to reduce tokens, caching common responses, and filtering events before LLM processing. Some organizations run smaller, self-hosted models (Llama 3.3, Mistral, Qwen) for simpler tasks while reserving powerful cloud APIs (GPT-4, Claude 3.5) for complex cases. For cost visibility across streaming infrastructure, see [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership). - **Data Privacy**: Streaming applications often process sensitive data. Ensure LLM providers meet compliance requirements for your industry. Consider techniques like data anonymization before sending to LLM APIs, or running models on-premises when data cannot leave your infrastructure. For PII handling in streams, see [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams) and [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming). - **Schema Evolution**: While 2025 LLMs support structured outputs via JSON Schema, managing schema evolution across streaming pipelines remains critical. Define clear schemas for structured data extraction from LLM responses. Schema registries help manage these data contracts as they evolve, ensuring data consistency across streaming pipelines. For schema management patterns, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). - **Testing**: LLM outputs are non-deterministic, making traditional testing approaches challenging. Focus on property-based testing that verifies outputs meet requirements rather than exact matching. Test error handling paths thoroughly since LLM failures are common. For comprehensive testing strategies, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). - **Prompt Engineering**: The quality of LLM responses depends heavily on prompt design. Maintain prompt templates in version control, test changes systematically, and monitor output quality in production. Include examples in prompts (few-shot learning) to guide the model toward desired formats. Modern LLMs support system prompts, user/assistant message structure, and function definitions, use these features to improve consistency. - **LLM Governance**: Organizations must track LLM usage, costs, and outputs for compliance and quality assurance. Implement logging for all LLM requests and responses, track which models and prompts are used in production, and establish approval processes for new LLM integrations. For AI governance patterns, see [Shadow AI: Governing Unauthorized AI in the Enterprise](https://www.conduktor.io/glossary/shadow-ai-governance) and [AI Discovery and Monitoring](https://www.conduktor.io/glossary/ai-discovery-and-monitoring). ## Real-World Use Cases - **Customer Support Automation**: A SaaS company streams incoming support tickets through Kafka. An LLM analyzes each ticket to extract the problem description, classify urgency (low/medium/high), identify the product area, and suggest potential solutions based on the knowledge base. The enriched tickets are routed to appropriate support queues, and low-priority issues with clear solutions trigger automated responses. This reduces first-response time from hours to minutes while ensuring complex issues reach specialized teams. - **Content Moderation**: A social media platform processes millions of user posts daily. Posts flow through Kafka topics to content moderation services. An LLM analyzes posts flagged by keyword filters or user reports, determining whether they violate community guidelines. The system produces moderation decisions to downstream topics that trigger actions like hiding posts, notifying moderators for review, or updating user reputation scores. This hybrid approach combines scalable keyword filtering with nuanced LLM analysis only where needed. - **Fraud Detection**: A payment processor streams transaction events through Flink. The system maintains state about user behavior patterns and flags anomalous transactions. For flagged transactions, an LLM analyzes the transaction context, user history, and merchant details to generate risk assessments using function calling to query feature stores and user profiles in real-time. These assessments combine with rule-based scoring to make real-time approval or decline decisions. The LLM's ability to understand complex patterns complements traditional fraud rules, catching sophisticated attacks that rules miss. For detailed patterns, see [Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming). - **Real-Time Recommendations**: E-commerce platforms use LLMs with streaming data to generate personalized product recommendations. As users browse, click events flow through Kafka to recommendation services that use RAG patterns to retrieve similar products from vector databases, then invoke LLMs to generate natural language explanations for why products match user interests. For recommendation architecture, see [Building Recommendation Systems with Streaming Data](https://www.conduktor.io/glossary/building-recommendation-systems-with-streaming-data). ## Summary Integrating LLMs with streaming platforms combines the real-time processing capabilities of technologies like Kafka and Flink with the natural language understanding of modern AI models. This integration enables applications that can analyze, enrich, and respond to events intelligently as they occur. The 2025 landscape offers significant advantages: structured outputs eliminate parsing errors, streaming responses reduce latency, function calling enables complex workflows, and diverse model options (from GPT-4 and Claude 3.5 to open-source Llama 3.3 and Mistral) support various deployment models and cost profiles. Success requires careful attention to architecture patterns, performance optimization, cost management, and operational concerns. The consumer-transform-producer pattern provides a straightforward starting point, while more sophisticated approaches like RAG with real-time context, selective event-driven inference, and vector database integration enable advanced use cases at scale. Key considerations include managing API latency through parallelization and async patterns, implementing robust error handling with dead letter queues and circuit breakers, and using structured outputs to ensure data consistency across streaming pipelines. Organizations should also address data privacy through masking and anonymization, monitor costs closely with platforms like Conduktor, and invest in prompt engineering to maximize output quality. For foundational Kafka concepts, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). For event-driven architecture principles, see [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture). For ML integration patterns, see [Real-Time ML Inference with Streaming Data](https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data) and [Real-Time ML Pipelines](https://www.conduktor.io/glossary/real-time-ml-pipelines). As LLMs become faster and more capable and streaming platforms add better AI support, this integration pattern will matter more for real-time applications. ## Related Concepts - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) - Integrate LLM APIs with streaming data - [Real-Time Analytics with Streaming Data](https://www.conduktor.io/real-time-analytics-with-streaming-data) - Analytics on LLM outputs and performance - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/data-governance-framework-roles-and-responsibilities) - Govern LLM usage and data access ## Sources and References 1. Apache Kafka Documentation: "Kafka Streams" - https://kafka.apache.org/documentation/streams/ 2. OpenAI API Documentation: "Structured Outputs" - https://platform.openai.com/docs/guides/structured-outputs 3. Anthropic Claude Documentation: "Function Calling and Tool Use" - https://docs.anthropic.com/en/docs/build-with-claude/tool-use 4. Apache Flink Documentation: "Event-driven Applications" - https://flink.apache.org/what-is-flink/use-cases/#event-driven-applications 5. Google Cloud: "Gemini API Streaming" - https://ai.google.dev/gemini-api/docs/text-generation 6. AWS: "Building Real-time AI Applications with Amazon Bedrock and Kafka" - https://aws.amazon.com/blogs/machine-learning/ 7. Martin Kleppmann: "Designing Data-Intensive Applications" (O'Reilly, 2017) - Chapters on stream processing fundamentals 8. Conduktor Documentation: "Monitoring Kafka Applications" - https://docs.conduktor.io/ Source: https://www.conduktor.io/glossary/integrating-llms-with-streaming-platforms --- # Introduction to Confluent Cloud As organizations adopt real-time data architectures, managing Apache Kafka infrastructure becomes increasingly complex. Confluent Cloud addresses this by providing a fully managed, cloud-native Kafka service that handles operational overhead while delivering enterprise-grade features. This article covers what Confluent Cloud is, how it works, and when organizations should consider using it versus self-managed Apache Kafka with platforms like Conduktor for governance and monitoring. For foundational Kafka concepts, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ![Confluent Cloud architecture and ecosystem integration](https://www.conduktor.io/assets/images/glossary/introduction-to-confluent-cloud-0.webp) ## What is Confluent Cloud? Confluent Cloud is a fully managed Apache Kafka service developed by Confluent, the company founded by the original creators of Apache Kafka. It provides a complete data streaming platform that runs across multiple cloud providers including AWS, Google Cloud, and Microsoft Azure. Unlike self-hosted Kafka deployments, Confluent Cloud removes the operational burden of managing brokers, maintaining clusters, scaling infrastructure, and handling upgrades. The platform handles these tasks automatically while providing guaranteed uptime SLAs (Service Level Agreements - commitments to availability percentages). As of 2025, Confluent Cloud runs on Apache Kafka 4.0+ using KRaft mode (Kafka Raft metadata mode), which eliminates the ZooKeeper dependency that existed in earlier Kafka versions. This architecture provides faster metadata operations, improved scalability, and simpler cluster management. For more on this transition, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). The service goes beyond basic Kafka functionality by including additional components like Schema Registry, managed Apache Flink and ksqlDB for stream processing, and a rich ecosystem of pre-built connectors for data integration. ## Core Capabilities and Features Confluent Cloud provides several key capabilities that distinguish it from self-managed Kafka deployments. - **Multi-Cloud and Multi-Region Support**: Organizations can deploy clusters across AWS, GCP, and Azure in different regions worldwide. This enables data locality compliance, disaster recovery strategies, and global data distribution patterns. - **Fully Managed Infrastructure**: Confluent handles all infrastructure operations including broker management, storage scaling, software patching, and security updates. Teams can focus on building streaming applications rather than maintaining infrastructure. - **Elastic Scaling**: Clusters can scale up or down based on demand without manual intervention. Storage automatically expands as data grows, and compute resources adjust to handle throughput changes. - **Integrated Ecosystem**: The platform includes Schema Registry for data governance, Kafka Connect for integrating with external systems, managed Apache Flink and ksqlDB for stream processing, and Cluster Linking for cross-cluster data replication. Cluster Linking enables real-time data mirroring between Kafka clusters for disaster recovery, multi-region architectures, and cloud migrations without custom replication code. - **Stream Governance (2025)**: Confluent Cloud includes advanced governance capabilities such as Stream Lineage for tracking data flows, data quality rules enforcement, and business metadata management. These features provide visibility into how data moves through streaming pipelines. - **Enterprise Security**: Built-in encryption at rest and in transit, role-based access control (RBAC - Role-Based Access Control), audit logging, and compliance with standards like SOC 2, ISO 27001, and HIPAA. Private networking options include AWS PrivateLink, Azure Private Link, and GCP Private Service Connect for secure connectivity without exposing traffic to the public internet. For detailed security practices, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices). ## Architecture and Deployment Models Confluent Cloud offers four main cluster types designed for different use cases and requirements (as of 2025). - **Basic Clusters** provide a single-zone deployment suitable for development, testing, and non-critical workloads. They offer a cost-effective entry point with limited availability guarantees. - **Standard Clusters** deliver multi-zone deployment within a single region, providing high availability with 99.95% uptime SLA. These clusters suit production workloads requiring reliability without extreme scale requirements. - **Dedicated Clusters** offer isolated infrastructure with customizable configurations, private networking options, and 99.99% uptime SLA. They support the highest throughput and storage requirements for mission-critical enterprise applications. - **Serverless Clusters (2025)** provide on-demand Kafka capacity that automatically scales based on actual usage without pre-provisioning resources. Organizations pay only for data transferred and stored, making serverless ideal for variable workloads, development environments, and proof-of-concepts where predictable costs matter less than operational simplicity. For example, a financial services company might use a Dedicated cluster for transaction processing in their primary region, with Cluster Linking replicating data to Standard clusters in other regions for analytics and disaster recovery. For infrastructure planning, see [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning). ## Confluent Cloud in the Data Streaming Ecosystem Confluent Cloud sits at the center of modern data streaming architectures, integrating with various technologies and platforms. - **Apache Kafka Core**: At its foundation, Confluent Cloud provides fully compatible Apache Kafka APIs. Applications using standard Kafka clients can connect without modification, ensuring portability and avoiding vendor lock-in. For core concepts, see [Kafka Topics, Partitions, Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). - **Stream Processing**: The platform includes both managed Apache Flink (as of 2025) and ksqlDB, enabling SQL-based stream processing directly within Confluent Cloud. Teams can also connect external stream processing engines or Kafka Streams applications running in their own infrastructure. For Flink details, see [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) and [ksqlDB for Real-Time Data Processing](https://www.conduktor.io/glossary/ksqldb-for-real-time-data-processing). - **Data Integration**: Kafka Connect provides pre-built connectors for databases, data warehouses, cloud storage, and SaaS applications. This enables building real-time data pipelines without custom integration code. For detailed connector patterns, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines). - **Schema Management**: Schema Registry provides centralized schema storage and evolution management. It ensures data compatibility across producers and consumers, preventing breaking changes in streaming pipelines. See [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) for implementation details. - **Observability Integration**: Confluent Cloud exports metrics to monitoring platforms like Datadog, Prometheus, and CloudWatch, enabling integration with existing observability stacks. For comprehensive monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). This ecosystem approach allows organizations to build complete data streaming platforms where Confluent Cloud handles the message broker while integrating with specialized tools for processing, storage, and analysis. ## Common Use Cases Organizations adopt Confluent Cloud for various real-time data scenarios. - **Event-Driven Microservices**: Services communicate through Kafka topics, enabling loose coupling and asynchronous processing. When a customer places an order in an e-commerce platform, the order service publishes an event that inventory, payment, and notification services consume independently. - **Real-Time Analytics**: Streaming data from applications, IoT devices, or clickstreams into data warehouses or analytics platforms. A ride-sharing company might stream location updates, ride events, and payment transactions for real-time dashboards and dynamic pricing algorithms. - **Data Integration and CDC**: Capturing database changes in real-time using change data capture (CDC) connectors and replicating them to analytics systems. This creates near-real-time data warehouses without batch ETL processes. For CDC fundamentals, see [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). - **Log Aggregation**: Centralizing application logs, metrics, and traces from distributed systems for analysis and troubleshooting. Multiple services across regions send logs to Kafka topics, which feed into log analysis platforms. - **Machine Learning Feature Stores**: Streaming feature data to ML models for real-time predictions. A fraud detection system processes transaction events in real-time, enriching them with historical patterns before feeding them to ML models. ## Operating Confluent Cloud at Scale While Confluent Cloud manages infrastructure, organizations still need to handle application-level operations and governance. - **Monitoring and Observability**: Confluent Cloud provides built-in metrics dashboards showing throughput, latency, and consumer lag. Teams should establish alerting on key metrics like under-replicated partitions, consumer lag exceeding thresholds, and connector failures. The 2025 Stream Designer visual interface provides real-time topology views of data flows. For consumer lag patterns, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). - **Security and Access Control**: Implementing proper RBAC policies ensures teams have appropriate access levels. Service accounts with API keys enable applications to authenticate securely without sharing credentials. For detailed access patterns, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). - **Topic and Schema Governance**: As Kafka deployments grow, managing hundreds or thousands of topics becomes challenging. Establishing naming conventions, retention policies, and schema evolution standards prevents sprawl and maintains data quality. See [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) for best practices. - **Comparison with Self-Managed Kafka**: Organizations running self-managed Apache Kafka can achieve similar governance and monitoring capabilities using platforms like Conduktor. Conduktor provides topic visualization, schema management, consumer group monitoring, data masking, and permission management across self-managed Kafka clusters. This approach offers more infrastructure control while maintaining operational efficiency through modern tooling. For self-managed operations, see [Kafka Admin Operations and Maintenance](https://www.conduktor.io/glossary/kafka-admin-operations-and-maintenance). - **Cost Optimization**: Monitoring usage patterns helps optimize costs. This includes adjusting retention periods, archiving cold data to object storage using Tiered Storage, rightsizing cluster types, and using committed use discounts for predictable workloads. The 2025 Kora engine provides improved storage efficiency and faster queries. See [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka) for archival strategies. ## Pricing and Total Cost of Ownership Confluent Cloud uses consumption-based pricing with several components (as of 2025). - **Cluster Costs**: Fixed hourly rates based on cluster type (Basic, Standard, Dedicated, Serverless). Dedicated clusters charge for provisioned capacity, while Serverless clusters charge only for actual usage, making them ideal for variable workloads. - **Data Transfer**: Charges for data written to and read from Kafka. The 2025 Freight tier optimizes inter-region data transfer costs for Cluster Linking and multi-region deployments, offering significant savings compared to standard cloud provider egress fees. - **Storage**: Costs for data retained in Kafka topics, with charges varying by retention period and cluster type. Tiered Storage (powered by the Kora engine) provides lower-cost archival storage for historical data while maintaining query performance. - **Additional Services**: Separate pricing for Schema Registry, managed Apache Flink compute resources, ksqlDB processing units, and Kafka Connect connectors. Stream Governance features are included in Enterprise tiers. - **2025 Pricing Improvements**: The Kora storage engine reduces storage costs by up to 50% through improved compression and encoding. Combined with Tiered Storage, organizations can retain longer histories economically. When comparing with self-managed Kafka, organizations should consider the total cost of ownership. Self-managed deployments require infrastructure costs (compute, storage, networking), operational staff time for maintenance and on-call support, expertise for upgrades and troubleshooting, and potential costs of downtime. However, organizations gain full control over infrastructure, security configurations, and can use platforms like Conduktor for governance and monitoring without vendor lock-in. Confluent Cloud typically provides better economics for small to medium deployments, teams without deep Kafka expertise, or when engineering time is expensive. Very large deployments with predictable workloads and experienced Kafka teams may find self-managed Apache Kafka with Conduktor more cost-effective while maintaining operational efficiency. For implementing cost visibility and chargeback across any Kafka deployment (cloud or self-managed), see [Conduktor's Cost Control Insights](https://docs.conduktor.io/guide/insights/cost-control). For detailed cost analysis, see [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership). ## Summary Confluent Cloud provides a fully managed Apache Kafka 4.0+ platform running on KRaft architecture that eliminates infrastructure management while delivering enterprise features across multiple clouds. As of 2025, the platform includes Serverless clusters, managed Apache Flink, the Kora storage engine, Stream Governance, and enhanced security through private networking options. The platform integrates naturally into data streaming ecosystems, providing not just Kafka but also stream processing (Flink and ksqlDB), schema management, and data integration capabilities. Common use cases include event-driven microservices, real-time analytics, CDC pipelines, and machine learning feature stores. While Confluent Cloud handles infrastructure operations, organizations must still manage application-level concerns like monitoring, security, governance, and cost optimization. The service trades infrastructure control for operational simplicity, making it attractive for teams that want to focus on building streaming applications rather than managing Kafka clusters. **Choosing Between Confluent Cloud and Self-Managed Kafka**: The decision depends on several factors: - **Choose Confluent Cloud** when you need rapid deployment, lack Kafka expertise, want predictable SLAs, or prefer consumption-based pricing - **Choose Self-Managed Kafka with Conduktor** when you need maximum infrastructure control, have experienced Kafka teams, require specific security configurations, or operate at massive scale where self-management becomes cost-effective Both approaches can deliver production-grade streaming platforms: Confluent Cloud through managed services, and self-managed Kafka through modern platforms like Conduktor that provide governance, monitoring, and operational tooling without vendor lock-in. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - The foundational open-source platform that Confluent Cloud is built on - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Critical for right-sizing both cloud and self-managed deployments - [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership) - Analyzing the full economic impact of managed vs. self-hosted Kafka ## Sources and References - [Confluent Cloud Documentation (2025)](https://docs.confluent.io/cloud/current/overview.html) - Official documentation covering architecture, features, KRaft mode, and best practices - [Apache Kafka 4.0+ Documentation](https://kafka.apache.org/documentation/) - Core Kafka concepts, KRaft mode, and APIs that Confluent Cloud implements - [Confluent Cloud Pricing (2025)](https://www.confluent.io/confluent-cloud/pricing/) - Detailed pricing for Serverless, Dedicated clusters, Kora engine, and Freight tier - [Confluent Kora Engine Technical Overview](https://www.confluent.io/blog/kora-engine-cloud-native-kafka-storage/) - Architecture and performance improvements in 2025 storage engine - [Apache Flink on Confluent Cloud](https://docs.confluent.io/cloud/current/flink/overview.html) - Managed Flink for stream processing integration - [Conduktor Platform Documentation](https://docs.conduktor.io/) - Governance and monitoring for self-managed Kafka environments - [Gartner Market Guide for Event Stream Processing (2025)](https://www.gartner.com/en/documents/event-stream-processing) - Industry analysis of streaming platforms Source: https://www.conduktor.io/glossary/introduction-to-confluent-cloud --- # Introduction to Kafka Streams Stream processing has become essential for modern data architectures. Organizations need to react to events in real-time, whether detecting fraud, personalizing user experiences, or monitoring system health. Kafka Streams provides a lightweight yet powerful solution for building these real-time applications. This article introduces Kafka Streams, explaining its core concepts, architecture, and practical applications. You'll learn how it fits into the broader stream processing landscape and how to get started building your own streaming applications. ## What is Kafka Streams? Kafka Streams is a client library for building stream processing applications on top of Apache Kafka. Unlike heavyweight frameworks that require separate clusters, Kafka Streams runs as part of your application. It's simply a Java library you add to your project. For foundational understanding of Apache Kafka's architecture, topics, and partitions, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). As of Kafka 3.6+ and Kafka 4.0, Kafka Streams has matured significantly with features like versioned state stores, improved exactly-once semantics, and full compatibility with KRaft mode (Kafka's ZooKeeper-free architecture). These advancements make it production-ready for the most demanding real-time applications. For details on KRaft mode, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). The key advantage is simplicity. You write standard Java applications that read from and write to Kafka topics. No separate processing cluster is needed. Your application instances can be deployed, scaled, and managed like any other microservice. Kafka Streams provides high-level abstractions for common stream processing patterns: - Filtering and transforming events - Joining multiple streams - Aggregating data over time windows - Maintaining stateful computations - Handling late-arriving data These capabilities enable you to build sophisticated real-time applications without dealing with low-level complexities of distributed stream processing. ## Core Architecture and Components Understanding Kafka Streams architecture helps you design better applications and troubleshoot issues effectively. ### Stream Processing Topology A Kafka Streams application defines a processing topology - a graph of stream processors connected by streams. The topology describes how data flows through your application: ![A Kafka Streams application defines a processing topology - a graph of stream processors connected by streams. The topology describes how data flows through your application](https://www.conduktor.io/assets/images/glossary/introduction-to-kafka-streams-0.webp) 1. **Source Processor**: Reads records from Kafka topics 2. **Stream Processors**: Transform, filter, or aggregate data 3. **Sink Processor**: Writes results back to Kafka topics Each processor performs a specific operation. Data flows from sources through processors to sinks, with intermediate results stored in Kafka topics. ### State Stores Many stream processing operations require state. For example, counting events or joining streams needs to remember previous data. Kafka Streams provides state stores - fault-tolerant, local databases embedded in your application. State stores are backed by Kafka topics called changelog topics. Every state change is written to Kafka, ensuring durability. If an application instance fails, another instance can rebuild the state from the changelog. **Modern State Store Features (Kafka 3.5+):** Kafka Streams 3.5+ introduced **versioned state stores**, which maintain historical versions of each key's value with timestamps. This enables: - Time-travel queries to retrieve values as they existed at specific points in time - More accurate stream-stream joins with better handling of out-of-order data - Improved consistency for temporal queries ```java // Example: Creating a versioned state store StoreBuilder> storeBuilder = Stores.versionedKeyValueStoreBuilder( Stores.persistentVersionedKeyValueStore("user-profiles-versioned", Duration.ofHours(24)), Serdes.String(), Serdes.Long() ); ``` ### Exactly-Once Processing Guarantees Kafka Streams supports **exactly-once semantics (EOS)**, ensuring each record is processed exactly once even during failures. This is critical for applications like financial transactions or inventory management where duplicate processing causes serious problems. Enable exactly-once processing with the `processing.guarantee` configuration: ```java Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "word-count-app"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2); ``` Kafka 3.0+ introduced **exactly-once semantics v2 (EXACTLY_ONCE_V2)**, which significantly improves performance by reducing coordination overhead between brokers. Always use v2 for new applications. For in-depth coverage of state store internals, RocksDB configuration, and optimization strategies, see [State Stores in Kafka Streams](https://www.conduktor.io/glossary/state-stores-in-kafka-streams). ### Parallel Processing Kafka Streams automatically parallelizes processing based on Kafka's partition model. Each partition can be processed independently. If your input topic has 10 partitions, you can run 10 application instances, each processing one partition. This design provides elastic scalability. Add more application instances to handle increased load. Remove instances when load decreases. Kafka's consumer group protocol handles partition assignment automatically. For detailed coverage of how Kafka Streams uses consumer groups for coordination, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). For partitioning strategies that maximize parallel processing, see [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). ## Stream Processing Fundamentals Kafka Streams introduces several abstractions representing different types of data streams. ### KStream: Event Streams A KStream represents an unbounded stream of events. Each record is a new fact. For example, user clicks, sensor readings, or transaction events are naturally modeled as KStreams. KStreams are append-only. A new record with the same key doesn't replace previous records - it's simply another event in the stream. ```java // Create a KStream from a Kafka topic KStream clicks = builder.stream("user-clicks"); // Filter clicks from mobile devices KStream mobileClicks = clicks.filter( (key, value) -> value.contains("mobile") ); // Transform the stream KStream processed = mobileClicks.mapValues( value -> value.toUpperCase() ); // Write results to an output topic processed.to("mobile-clicks-processed"); ``` ### KTable: Changelog Streams A KTable represents a changelog stream - a sequence of updates to a table. Each record with a given key updates the current value for that key. Think of it as a database table continuously updated by a stream of changes. KTables are useful for representing current state. For example, user profiles, product inventory, or account balances are naturally modeled as KTables. ```java // Create a KTable from a compacted Kafka topic KTable userProfiles = builder.table("user-profiles"); // A new record with key "user123" replaces the previous value // Materialize the table for interactive queries KTable materializedProfiles = userProfiles.mapValues( profile -> profile, Materialized.>as("user-profiles-store") ); // Convert KTable to KStream to send updates downstream KStream profileUpdates = userProfiles.toStream(); ``` **KStream vs KTable Decision Guide:** - Use **KStream** when each event matters independently (transactions, clicks, sensor readings) - Use **KTable** when you care about current state (user settings, inventory levels, account balances) - A KTable is essentially a KStream with log compaction - only the latest value per key is retained For details on how log compaction works and its role in KTables, see [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained). To understand the transactional guarantees underlying exactly-once processing, see [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). ### Stateless vs Stateful Operations Stateless operations process each record independently: - `map`: Transform each record - `filter`: Select records matching a condition - `flatMap`: Transform one record into zero or more records Stateful operations require remembering previous data: - `aggregate`: Combine records with the same key - `join`: Combine records from multiple streams - `windowing`: Group records into time-based windows Stateful operations use state stores and are more complex but enable powerful stream processing patterns. For detailed coverage of windowing strategies including tumbling, sliding, and session windows, see [Session Windows in Stream Processing](https://www.conduktor.io/glossary/session-windows-in-stream-processing). For join patterns and stream enrichment techniques, see [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns). To understand how Kafka Streams handles out-of-order data and event time processing, see [Watermarks and Triggers in Stream Processing](https://www.conduktor.io/glossary/watermarks-and-triggers-in-stream-processing). ## Building Your First Kafka Streams Application Here is a simple word count application - the "Hello World" of stream processing. This example demonstrates core Kafka Streams concepts. ### Project Setup First, add Kafka Streams to your project. For Maven: ```xml org.apache.kafka kafka-streams 3.6.0 ``` For Gradle: ```gradle implementation 'org.apache.kafka:kafka-streams:3.6.0' ``` ### The Word Count Application Our application reads text from an input topic, counts word occurrences, and writes results to an output topic. ![introduction-to-kafka-streams diagram 2](https://www.conduktor.io/assets/images/glossary/introduction-to-kafka-streams-1.webp) This simple application demonstrates several concepts: 1. **Stateless transformation**: `flatMapValues` splits lines into words 2. **Grouping**: `groupBy` organizes words with the same key 3. **Stateful aggregation**: `count` maintains running totals in a state store 4. **Output**: Results are written to another Kafka topic ### Complete Code Example Here's the complete, runnable word count application: ```java import org.apache.kafka.common.serialization.Serdes; import org.apache.kafka.streams.KafkaStreams; import org.apache.kafka.streams.StreamsBuilder; import org.apache.kafka.streams.StreamsConfig; import org.apache.kafka.streams.kstream.KStream; import org.apache.kafka.streams.kstream.KTable; import org.apache.kafka.streams.kstream.Materialized; import org.apache.kafka.streams.kstream.Produced; import java.util.Arrays; import java.util.Properties; public class WordCountApplication { public static void main(String[] args) { // Configure the Streams application Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "word-count-app"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass()); props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass()); // Build the topology StreamsBuilder builder = new StreamsBuilder(); // Read from input topic KStream textLines = builder.stream("text-input"); // Process the stream KTable wordCounts = textLines // Split each line into words .flatMapValues(line -> Arrays.asList(line.toLowerCase().split("\\W+"))) // Group by word .groupBy((key, word) -> word) // Count occurrences .count(Materialized.as("word-counts-store")); // Write results to output topic wordCounts.toStream() .to("word-counts", Produced.with(Serdes.String(), Serdes.Long())); // Start the application KafkaStreams streams = new KafkaStreams(builder.build(), props); // Add shutdown hook for graceful shutdown Runtime.getRuntime().addShutdownHook(new Thread(streams::close)); streams.start(); } } ``` **Key Configuration Parameters:** - `APPLICATION_ID_CONFIG`: Unique identifier for this application. Also serves as the consumer group ID and prefix for internal topics - `BOOTSTRAP_SERVERS_CONFIG`: Kafka cluster connection string - `DEFAULT_KEY_SERDE_CLASS_CONFIG` / `DEFAULT_VALUE_SERDE_CLASS_CONFIG`: Serialization/deserialization classes for keys and values ### Running the Application The application runs as a standard Java process. Deploy it like any microservice - in containers, on VMs, or in Kubernetes. To scale, simply launch more instances. Kafka Streams automatically distributes partitions across instances. **Before running:** 1. Create input and output topics: ```bash kafka-topics --create --topic text-input --bootstrap-server localhost:9092 --partitions 3 kafka-topics --create --topic word-counts --bootstrap-server localhost:9092 --partitions 3 ``` 2. Run the application: ```bash java -jar word-count-app.jar ``` 3. Test with sample data: ```bash echo "hello world" | kafka-console-producer --topic text-input --bootstrap-server localhost:9092 echo "hello kafka" | kafka-console-producer --topic text-input --bootstrap-server localhost:9092 ``` 4. View results: ```bash kafka-console-consumer --topic word-counts --bootstrap-server localhost:9092 \ --from-beginning --property print.key=true --property key.separator=" : " ``` ## Real-World Use Cases Kafka Streams powers diverse real-time applications across industries. ### Fraud Detection Financial services use Kafka Streams to detect fraudulent transactions in real-time. The application aggregates recent transactions per user, calculates risk scores, and flags suspicious patterns immediately. ![introduction-to-kafka-streams diagram 3](https://www.conduktor.io/assets/images/glossary/introduction-to-kafka-streams-2.webp) This pattern enables immediate action - blocking transactions, alerting users, or triggering manual review. ### Metrics and Monitoring Applications generate continuous metrics - request rates, error counts, latency percentiles. Kafka Streams aggregates these metrics in real-time, providing up-to-the-second dashboards without expensive batch processing. ### Event-Driven Microservices Modern architectures use events to communicate between services. Kafka Streams processes these events, maintaining materialized views, enforcing business rules, and orchestrating complex workflows. For example, an e-commerce system might use Kafka Streams to: - Update inventory after orders - Calculate customer lifetime value - Generate personalized recommendations - Detect and prevent overselling Each of these runs as an independent Kafka Streams application, reading from and writing to Kafka topics. ## Monitoring and Troubleshooting Production Kafka Streams applications require careful monitoring and operational tooling. ### Key Metrics to Monitor Kafka Streams exposes metrics through JMX: - **Lag**: How far behind the application is from the latest data - **Throughput**: Records processed per second - **State store size**: Memory and disk usage - **Rebalances**: Partition assignment changes High lag indicates the application can't keep up with incoming data. Frequent rebalances suggest instability. Monitoring these metrics is essential for maintaining healthy stream processing. For comprehensive monitoring strategies and alerting best practices, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ### Debugging Stream Processing Stream processing applications are inherently complex. Data flows through multiple stages, state is distributed, and timing matters. Debugging requires specialized tools. Platforms like Conduktor provide visibility into the Kafka topics your Streams application reads from and writes to through [topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), helping you understand consumer lag and throughput at each stage. When issues arise, these tools significantly reduce debugging time. ### Testing Strategies Kafka Streams includes the **TopologyTestDriver** for unit testing without running Kafka. This enables fast, reliable testing as part of your CI/CD pipeline. **Example unit test using TopologyTestDriver:** ```java import org.apache.kafka.common.serialization.Serdes; import org.apache.kafka.streams.StreamsBuilder; import org.apache.kafka.streams.StreamsConfig; import org.apache.kafka.streams.TestInputTopic; import org.apache.kafka.streams.TestOutputTopic; import org.apache.kafka.streams.TopologyTestDriver; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import java.util.Properties; import static org.junit.jupiter.api.Assertions.assertEquals; public class WordCountApplicationTest { private TopologyTestDriver testDriver; private TestInputTopic inputTopic; private TestOutputTopic outputTopic; @BeforeEach public void setup() { Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:1234"); StreamsBuilder builder = new StreamsBuilder(); // Build your topology here (same as production code) testDriver = new TopologyTestDriver(builder.build(), props); inputTopic = testDriver.createInputTopic("text-input", Serdes.String().serializer(), Serdes.String().serializer()); outputTopic = testDriver.createOutputTopic("word-counts", Serdes.String().deserializer(), Serdes.Long().deserializer()); } @AfterEach public void tearDown() { testDriver.close(); } @Test public void testWordCount() { // Send test data inputTopic.pipeInput("hello world"); inputTopic.pipeInput("hello kafka"); // Verify results assertEquals("hello", outputTopic.readKeyValue().key); assertEquals(1L, outputTopic.readKeyValue().value); assertEquals("world", outputTopic.readKeyValue().key); assertEquals(1L, outputTopic.readKeyValue().value); assertEquals("hello", outputTopic.readKeyValue().key); assertEquals(2L, outputTopic.readKeyValue().value); assertEquals("kafka", outputTopic.readKeyValue().key); assertEquals(1L, outputTopic.readKeyValue().value); } } ``` **Benefits of TopologyTestDriver:** - **No Kafka cluster required**: Tests run in milliseconds, not seconds - **Deterministic**: Control input timing and ordering precisely - **State inspection**: Access state stores directly for assertions - **CI/CD friendly**: Fast enough for every commit For comprehensive testing strategies including integration testing and chaos engineering approaches, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). ## Summary Kafka Streams provides a powerful yet accessible approach to stream processing. As a lightweight library rather than a separate framework, it integrates naturally into modern application architectures. Key takeaways: - **Simplicity**: No separate cluster required - just a Java library - **Modern features (Kafka 3.6+/4.0)**: Versioned state stores, improved exactly-once semantics v2, full KRaft compatibility - **Scalability**: Automatic parallelization based on Kafka partitions - **Fault tolerance**: State backed by Kafka changelog topics with automatic recovery - **Rich abstractions**: KStream and KTable model different stream semantics - **Production-ready**: Built-in metrics, TopologyTestDriver for testing, and operational tools - **Exactly-once processing**: EXACTLY_ONCE_V2 ensures no duplicate processing even during failures Whether you're building fraud detection, real-time analytics, or event-driven microservices, Kafka Streams provides the building blocks for reliable stream processing. Start with simple transformations and gradually adopt more advanced patterns as your needs grow. The stream processing landscape includes alternatives like Apache Flink and ksqlDB, each with different trade-offs. Kafka Streams excels when you want library-based processing, tight Kafka integration, and operational simplicity. For a detailed comparison, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink). For SQL-based stream processing, see [ksqlDB for Real-Time Data Processing](https://www.conduktor.io/glossary/ksqldb-for-real-time-data-processing). For real-world implementation examples, see [Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming) which demonstrates advanced Kafka Streams patterns for production systems. ## Related Concepts - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Deep dive into Kafka's exactly-once processing guarantees and EXACTLY_ONCE_V2 - [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive) - Understand the transactional guarantees underlying exactly-once processing in Kafka Streams - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Essential metrics and strategies for monitoring Kafka Streams application lag - [Learn Kafka Streams (hands-on course)](https://www.conduktor.io/kafka-streams) - Production-grounded walkthrough of the DSL, state, joins, and running it in production ## Sources and References - [Apache Kafka Documentation - Streams API](https://kafka.apache.org/documentation/streams/) - Official documentation covering architecture, API reference, and developer guide - [Confluent Kafka Streams Tutorial](https://docs.confluent.io/platform/current/streams/) - Comprehensive tutorials and best practices from Confluent, the company founded by Kafka's creators - "Kafka: The Definitive Guide" by Neha Narkhede, Gwen Shapira, and Todd Palino (O'Reilly Media, 2021) - In-depth book covering Kafka fundamentals and stream processing patterns - [Apache Kafka Streams GitHub Repository](https://github.com/apache/kafka/tree/trunk/streams) - Source code and examples demonstrating Kafka Streams capabilities - "Designing Event-Driven Systems" by Ben Stopford (O'Reilly Media, 2018) - Explores event-driven architectures and stream processing design patterns Source: https://www.conduktor.io/glossary/introduction-to-kafka-streams --- # Introduction to Lakehouse Architecture Modern organizations face a recurring challenge: how to efficiently store, process, and analyze massive volumes of diverse data types. For years, data teams have had to choose between the rigid structure of data warehouses and the flexible chaos of data lakes. Lakehouse architecture offers a unified platform that combines both. ## What is Lakehouse Architecture? Lakehouse architecture combines the flexibility and cost-effectiveness of data lakes with the performance and ACID (Atomicity, Consistency, Isolation, Durability) transaction guarantees of data warehouses. Rather than maintaining separate systems for different workloads, a lakehouse provides a single, unified platform for all your data needs. At its core, a lakehouse stores data in low-cost object storage (similar to data lakes) while implementing a metadata and governance layer on top (similar to data warehouses). This architecture enables both business intelligence queries and machine learning workloads to operate on the same data without requiring costly and error-prone data movement between systems. The key innovation is the **table format layer**, technologies like Apache Iceberg 1.7+, Delta Lake 3.x, and Apache Hudi 1.x that sit between storage and compute, providing transactional guarantees and metadata management while keeping data in open formats. ## The Evolution: From Warehouses to Lakes to Lakehouses ### Traditional Data Warehouses Data warehouses emerged in the 1980s as specialized databases optimized for analytical queries. They provide: - **Structured data storage** with predefined schemas - **ACID transactions** ensuring data consistency - **High-performance SQL queries** for business intelligence - **Data quality enforcement** through schema validation However, warehouses come with significant limitations. They're expensive to scale, struggle with unstructured data (images, videos, logs), and require costly ETL (Extract, Transform, Load) processes to load data. Their rigid schemas make them inflexible for rapidly changing business needs. ### The Data Lake Era Data lakes gained popularity in the 2010s as organizations needed to store massive volumes of diverse data types at lower costs. Built on technologies like Hadoop and cloud object storage, data lakes offer: - **Schema-on-read flexibility** allowing storage of any data format (structure is applied when reading, not writing) - **Cost-effective storage** using commodity hardware or cloud storage - **Support for unstructured data** including logs, images, and streaming data - **Native machine learning support** with direct access to raw data Despite these advantages, data lakes introduced new problems. Without proper governance, they often became "data swamps", disorganized repositories where data quality degraded over time. They lacked ACID transaction support, making it difficult to ensure data consistency. Performance for SQL queries was often poor compared to warehouses. ### The Lakehouse Solution Lakehouse architecture emerged in the early 2020s to address the limitations of both approaches. By adding a transaction and metadata layer on top of data lake storage, lakehouses provide: - **Unified storage** for all data types in open formats (primarily Parquet) - **ACID transactions** through modern table formats like [Delta Lake](https://www.conduktor.io/delta-lake-transaction-log-how-it-works) 3.x, [Apache Iceberg](https://www.conduktor.io/apache-iceberg) 1.7+, or Apache Hudi 1.x - **Schema evolution** supporting both schema-on-write (enforce structure during writes) and schema-on-read (apply structure during reads) - **Performance optimization** through Z-ordering, [liquid clustering](https://www.conduktor.io/delta-lake-liquid-clustering-modern-partitioning), bloom filters, and [deletion vectors](https://www.conduktor.io/delta-lake-deletion-vectors-efficient-row-level-deletes) - **Built-in governance** with fine-grained access controls, audit logging, and catalog integration - **Time travel** allowing queries against historical table versions for auditing and debugging ## Core Components of Lakehouse Architecture A typical lakehouse architecture consists of several key layers: ![A typical lakehouse architecture consists of several key layers](https://www.conduktor.io/assets/images/glossary/introduction-to-lakehouse-architecture-0.webp) ### Storage Layer The foundation uses cost-effective object storage (AWS S3, Azure Data Lake Storage Gen2, or Google Cloud Storage) to store data in open file formats. **Parquet has emerged as the de facto standard** for lakehouse implementations due to its columnar format, excellent compression, and widespread support across all query engines. While ORC and Avro are supported, Parquet's efficient columnar storage and predicate pushdown capabilities make it ideal for analytical workloads. This ensures vendor independence and cost efficiency. ### Metadata & Transaction Layer This critical layer provides lakehouse capabilities through modern table formats. As of 2025, the three major formats have converged on core features while maintaining distinct strengths: - **[Apache Iceberg 1.7+](https://www.conduktor.io/apache-iceberg)**: Created by Netflix, now the most widely adopted format with support for REST catalogs, puffin statistics files, and [hidden partitioning](https://www.conduktor.io/iceberg-partitioning-and-performance-optimization). Excels at metadata management and [schema evolution](https://www.conduktor.io/schema-evolution-in-apache-iceberg). Offers the strongest multi-engine support (Spark, Trino, Flink, Presto, Dremio). - **[Delta Lake 3.x](https://www.conduktor.io/delta-lake-transaction-log-how-it-works)**: Developed by Databricks, now features [Liquid Clustering](https://www.conduktor.io/delta-lake-liquid-clustering-modern-partitioning) for automatic data layout optimization, [Deletion Vectors](https://www.conduktor.io/delta-lake-deletion-vectors-efficient-row-level-deletes) for efficient row-level deletes, and UniForm for cross-format compatibility. Strong integration with Spark and [time travel](https://www.conduktor.io/time-travel-with-apache-iceberg) capabilities. - **Apache Hudi 1.x**: Optimized by Uber, excels at incremental data processing with copy-on-write and merge-on-read storage types. Best for CDC pipelines and frequent upsert workloads. These formats track which files belong to which table version, manage concurrent writes through optimistic concurrency control, enable [time travel queries](https://www.conduktor.io/time-travel-with-apache-iceberg), and maintain performance statistics for query optimization. ### Catalog Layer The catalog layer manages table metadata, permissions, and discovery across the lakehouse. Modern catalogs include: - **Polaris Catalog**: Snowflake's open-source REST catalog for Iceberg, offering centralized metadata management - **Project Nessie**: Git-like catalog with branching and versioning for data, enabling data-as-code workflows - **Unity Catalog**: Databricks' unified governance layer supporting Delta Lake, Iceberg, and Hudi - **AWS Glue Catalog**: AWS-native catalog with [strong Iceberg integration](https://www.conduktor.io/iceberg-catalog-management-hive-glue-and-nessie) - **Hive Metastore**: Legacy catalog still widely used but being phased out in favor of REST catalogs The catalog enables [data governance](https://www.conduktor.io/data-governance-framework-roles-and-responsibilities), lineage tracking, and cross-platform table discovery. ### Query Engine Layer As of 2025, compute engines have matured with robust lakehouse support: - **Apache Spark 3.5+**: Universal compute engine with excellent support for all three table formats. Handles both batch and [streaming workloads](https://www.conduktor.io/flink-vs-spark-streaming-when-to-choose-each). - **Trino**: Distributed SQL query engine optimized for interactive analytics, with strong Iceberg support - **Apache Flink 1.19+**: [Stream processing powerhouse](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) with growing batch capabilities and native Iceberg integration - **Presto**: Facebook's original distributed SQL engine, now largely superseded by Trino These engines use the metadata layer to optimize query plans, perform predicate pushdown, skip irrelevant files, and read only necessary columns (projection pushdown). ### Analytics & ML Layer Business intelligence tools, data science platforms, and machine learning frameworks connect directly to the lakehouse, eliminating the need for separate systems. ## Streaming Integration in Lakehouse Architecture One of the most powerful aspects of lakehouse architecture is its native support for both batch and streaming data processing. Modern data platforms must handle real-time data streams from applications, IoT devices, and event-driven systems, implementing what's known as **Kappa architecture**, a unified streaming approach that has largely replaced the older Lambda architecture pattern. ### Streaming Data Ingestion Lakehouse platforms integrate directly with streaming technologies: - **[Apache Kafka](https://www.conduktor.io/apache-kafka)** is the primary streaming layer for real-time event ingestion, providing durable, scalable message queuing - **Apache Flink 1.19+** and **Spark 3.5+ Structured Streaming** process streams before writing to the lakehouse, with both engines now supporting native Iceberg writes - **[Kafka Connect](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines)** with Iceberg/Delta Lake sinks enables direct streaming ingestion without custom code - **Change Data Capture (CDC)** tools like Debezium continuously sync database changes into the lakehouse via Kafka ### Example: Streaming to Iceberg with Flink Here's a practical example of writing streaming data from Kafka to an Iceberg table: ```java // Flink 1.19+ streaming to Iceberg StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); TableEnvironment tableEnv = TableEnvironment.create(env); // Configure Iceberg catalog tableEnv.executeSql( "CREATE CATALOG iceberg_catalog WITH (" + "'type'='iceberg'," + "'catalog-impl'='org.apache.iceberg.rest.RESTCatalog'," + "'uri'='https://your-catalog.com/api/v1')" ); // Read from Kafka and write to Iceberg with exactly-once semantics tableEnv.executeSql( "INSERT INTO iceberg_catalog.db.events " + "SELECT event_id, user_id, event_type, event_time " + "FROM kafka_source" ); ``` ### Stream Processing Management with Conduktor Managing streaming infrastructure requires specialized expertise and governance. **Conduktor** provides comprehensive capabilities for data teams building lakehouse ingestion pipelines: - **Kafka Cluster Management**: Monitor and manage Kafka clusters with intuitive dashboards, tracking lag, throughput, and consumer health. See [Conduktor's topic management guide](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) for monitoring streaming ingestion - **Data Governance**: Enforce schema validation, data contracts, and quality policies on streaming data before it reaches the lakehouse. Learn more about [Schema Registry integration](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry) - **Testing & Validation**: Use Conduktor Gateway, a Kafka proxy, to inject chaos scenarios, test failure handling, and validate exactly-once semantics - **Pipeline Debugging**: Visualize data flows, inspect message payloads, and troubleshoot connector configurations using [Kafka Connect management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) Integrating Conduktor with a lakehouse setup helps organizations build reliable end-to-end data pipelines with consistent data quality from ingestion through analytics. ### Unified Batch and Streaming (Kappa Architecture) Modern lakehouse table formats enable **Kappa architecture**, a unified streaming paradigm that processes all data as streams. This approach has superseded Lambda architecture by supporting both: - **Batch writes** for large-scale data loads and historical backfills - **Streaming writes** with ACID guarantees for real-time upserts and incremental updates - **Unified queries** that read both batch and streaming data from the same tables - **Exactly-once semantics** ensuring no duplicate or lost records This eliminates the complexity of maintaining separate hot and cold data paths while providing consistent data across all query patterns. ## Benefits of Lakehouse Architecture ### Cost Efficiency By consolidating data warehouses and data lakes into a single platform, organizations reduce: - Storage costs through object storage pricing - Data duplication across multiple systems - Operational overhead of managing separate platforms - ETL pipeline complexity and associated compute costs ### Simplified Data Management Data teams benefit from: - Single source of truth for all analytics and ML workloads - Reduced data movement and synchronization issues - Unified security and governance policies - Simplified data lineage and compliance ### Performance and Flexibility Lakehouses deliver: - Near-warehouse performance for SQL queries through optimizations - Direct access to raw data for machine learning - Support for diverse workloads on the same data - Real-time and batch processing in one platform ## Getting Started with Lakehouse Architecture For organizations considering lakehouse adoption in 2025: 1. **Assess current architecture**: Identify pain points with existing warehouse and lake separation. Calculate costs of data duplication, ETL complexity, and operational overhead. 2. **Choose a table format**: - **[Apache Iceberg](https://www.conduktor.io/apache-iceberg)**: Best choice for multi-engine environments (Spark, Trino, Flink). Strongest community momentum and vendor support in 2025. - **[Delta Lake](https://www.conduktor.io/delta-lake-transaction-log-how-it-works)**: Best if you're heavily invested in Databricks or Spark-centric workflows. UniForm enables cross-format compatibility. - **Apache Hudi**: Best for CDC-heavy workloads with frequent upserts. 3. **Select a catalog strategy**: Choose between REST catalogs (Polaris, Nessie) for cloud-native flexibility or Unity Catalog for comprehensive governance. Avoid Hive Metastore for new implementations. 4. **Start with a pilot**: Migrate a single analytical use case (e.g., event analytics, customer 360) to validate the approach. Measure query performance, cost savings, and operational complexity. 5. **Build streaming pipelines**: - Set up [Kafka infrastructure](https://www.conduktor.io/apache-kafka) for event streaming - Use [Kafka Connect](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) or [Flink](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) for lakehouse writes - Implement Conduktor for monitoring, governance, and testing 6. **Implement governance from day one**: - Establish [data governance framework](https://www.conduktor.io/data-governance-framework-roles-and-responsibilities) with clear ownership - Build [business glossary](https://www.conduktor.io/building-a-business-glossary-for-data-governance) for term standardization - Implement [data catalog](https://www.conduktor.io/what-is-a-data-catalog-modern-data-discovery) for discovery and lineage 7. **Train your team**: Invest in upskilling data engineers on table format internals, streaming patterns, and catalog management. Lakehouse architecture requires different thinking than traditional warehouses. ## Comparison: Data Warehouse vs Data Lake vs Lakehouse | Feature | Data Warehouse | Data Lake | Lakehouse (2025) | |---------|---------------|-----------|------------------| | **Storage Format** | Proprietary | Open (any format) | Open (primarily Parquet) | | **Schema** | Schema-on-write | Schema-on-read | Both supported | | **ACID Transactions** | Yes | No | Yes (Iceberg/Delta/Hudi) | | **Performance** | Excellent for BI | Poor for structured queries | Near-warehouse for BI, excellent for ML | | **Cost** | High (proprietary compute+storage) | Low (object storage) | Low (object storage + open compute) | | **Data Types** | Structured only | All types | All types | | **Time Travel** | Limited | No | Yes (snapshot-based) | | **Schema Evolution** | Difficult | Easy | Easy with compatibility checks | | **Governance** | Built-in | Requires external tools | Built-in (catalog layer) | | **Best For** | BI dashboards, reports | Raw data archival, ML | Unified analytics + ML + streaming | ## Conclusion Lakehouse architecture unifies the capabilities of data warehouses and data lakes, reducing complexity while improving performance, cost, and flexibility. As of 2025, the ecosystem has matured significantly: - **Table formats** (Iceberg 1.7+, Delta Lake 3.x, Hudi 1.x) provide production-grade ACID guarantees and performance optimizations - **Query engines** (Spark 3.5+, Trino, Flink 1.19+) offer solid multi-format support - **Catalogs** (Polaris, Nessie, Unity Catalog) enable governance and data discovery - **Streaming integration** via Kafka and Flink enables real-time data ingestion with exactly-once semantics Whether you're starting fresh or evolving existing infrastructure, the lakehouse approach is practical for organizations that need to combine analytics, ML, and streaming workloads on a single platform. ## Related Concepts - [Streaming ETL vs Traditional ETL](https://www.conduktor.io/streaming-etl-vs-traditional-etl) - [Real-Time Analytics with Streaming Data](https://www.conduktor.io/real-time-analytics-with-streaming-data) - [Data Pipeline Orchestration with Streaming](https://www.conduktor.io/data-pipeline-orchestration-with-streaming) ## Related Articles - [Apache Iceberg: Open Table Format for Huge Analytic Tables](https://www.conduktor.io/apache-iceberg) - [Delta Lake Transaction Log: How It Works](https://www.conduktor.io/delta-lake-transaction-log-how-it-works) - [Schema Evolution in Apache Iceberg](https://www.conduktor.io/schema-evolution-in-apache-iceberg) - [Time Travel with Apache Iceberg](https://www.conduktor.io/time-travel-with-apache-iceberg) - [Iceberg Partitioning and Performance Optimization](https://www.conduktor.io/iceberg-partitioning-and-performance-optimization) - [Delta Lake Liquid Clustering: Modern Partitioning](https://www.conduktor.io/delta-lake-liquid-clustering-modern-partitioning) - [Delta Lake Deletion Vectors: Efficient Row-Level Deletes](https://www.conduktor.io/delta-lake-deletion-vectors-efficient-row-level-deletes) - [Iceberg Catalog Management: Hive, Glue, and Nessie](https://www.conduktor.io/iceberg-catalog-management-hive-glue-and-nessie) - [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/data-governance-framework-roles-and-responsibilities) - [What is a Data Catalog: Modern Data Discovery](https://www.conduktor.io/what-is-a-data-catalog-modern-data-discovery) ## Sources - [Apache Iceberg 1.7.0 Documentation](https://iceberg.apache.org/docs/1.7.0/) - [Delta Lake 3.x Documentation](https://docs.delta.io/latest/index.html) - [Apache Hudi 1.x Documentation](https://hudi.apache.org/docs/overview/) - [Apache Spark 3.5+ Structured Streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) - [Apache Flink 1.19 Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/) - [Polaris Catalog Documentation](https://www.polaris.io/) - [Project Nessie Documentation](https://projectnessie.org/) - [The Data Lakehouse: Building the Next Generation of Data Platforms (2021)](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) - [Databricks: What is a Data Lakehouse?](https://www.databricks.com/glossary/data-lakehouse) Source: https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture --- # IoT Data Streaming Architectures The Internet of Things has transformed how we collect and process data. Billions of connected devices generate continuous streams of information, from industrial sensors monitoring equipment health to smart home devices tracking energy consumption. Traditional batch processing architectures cannot handle the volume, velocity, and variety of IoT data. This is where streaming architectures become essential. IoT data streaming architectures enable organizations to ingest, process, and act on sensor data in real-time. These systems must handle millions of events per second, support diverse data formats, and provide low-latency processing for time-sensitive applications. ## Core Components of IoT Streaming Architectures An effective IoT streaming architecture consists of several interconnected layers that work together to move data from devices to actionable insights. ![iot-data-streaming-architectures diagram 1](https://www.conduktor.io/assets/images/glossary/iot-data-streaming-architectures-0.webp) ### Data Ingestion Layer The ingestion layer collects data from IoT devices and routes it into the streaming platform. MQTT 5.0 (Message Queuing Telemetry Transport) is the standard protocol for device-to-cloud communication in 2025, offering enhanced features like shared subscriptions for load balancing, message expiry intervals, and improved authentication flows. Its lightweight nature and support for unreliable networks make it ideal for constrained IoT devices. Apache Kafka is the central message broker, providing durable storage and high-throughput ingestion. Modern IoT architectures use Kafka 4.0+ running in KRaft mode (without ZooKeeper), which simplifies deployment and improves metadata handling for millions of IoT device connections. IoT devices publish messages to MQTT topics, which are then bridged to Kafka topics through protocol bridge services (lightweight applications that translate MQTT messages into Kafka records). This architecture decouples device communication from downstream processing, allowing the system to scale independently. For detailed coverage of Kafka's architecture, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ### Stream Processing Layer Once data enters the platform, stream processing engines transform, enrich, and analyze it in real-time. Apache Flink 1.19+ and Kafka Streams are popular choices for IoT workloads. They support stateful processing, windowing operations (time-based grouping of events), and complex event processing (detecting patterns across multiple events). Flink's exactly-once semantics ensure critical IoT data like energy metering or industrial safety alerts are processed without loss or duplication. Its event time processing handles out-of-order events common when IoT devices buffer data during network interruptions. For comparing these technologies, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink). For example, a smart city traffic management system might use Flink to aggregate vehicle speed data over five-minute windows, detect congestion patterns, and trigger alerts when traffic slows below thresholds. For more on windowing strategies, see [Windowing in Apache Flink](https://www.conduktor.io/glossary/windowing-in-apache-flink-tumbling-sliding-and-session-windows). ### Storage Layer IoT architectures require multiple storage systems optimized for different access patterns. Hot data used for real-time dashboards goes into time-series databases like InfluxDB, TimescaleDB, QuestDB, or Apache IoTDB. These databases are optimized for time-stamped sensor readings and support efficient range queries. Historical data for analytics moves to data lakes built on object storage using formats like Apache Iceberg or Delta Lake, which provide ACID transactions and schema evolution. For details on lakehouse architectures, see [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture) and [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse). The storage strategy depends on retention requirements, query patterns, and cost constraints. IoT data volumes can grow rapidly, so implementing tiered storage and data lifecycle policies is essential. ## Architectural Patterns for IoT Streaming Different IoT use cases require different architectural approaches. Understanding these patterns helps teams design systems that match their specific requirements. ### Edge-to-Cloud Architecture Edge computing processes data close to IoT devices before sending it to the cloud. This pattern reduces bandwidth costs, improves latency for local decisions, and continues functioning during network outages. Modern edge deployments use frameworks like AWS IoT Greengrass v2, Azure IoT Edge 2.0, Google Cloud IoT Edge, or lightweight Kubernetes distributions like K3s. These platforms enable running containerized stream processing applications (Kafka Streams, Flink) directly on edge gateways with resource constraints as low as 1GB RAM. Edge nodes run lightweight stream processing to filter noise, aggregate readings, and detect anomalies. Only relevant data or summarized metrics flow to the cloud for long-term storage and advanced analytics. Edge devices can also run ML inference models (TinyML, TensorFlow Lite) for real-time decision-making without cloud connectivity. A manufacturing plant might deploy edge processing on factory floor gateways. These gateways analyze vibration sensor data from machinery in real-time, triggering immediate maintenance alerts while sending hourly summaries to central systems for trend analysis. For more on real-time ML, see [Real-Time ML Inference with Streaming Data](https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data). ### Lambda and Kappa Architectures Lambda architecture maintains separate batch and stream processing paths, eventually reconciling them for complete accuracy. The batch layer reprocesses historical data to correct any streaming errors, while the speed layer provides real-time results. Kappa architecture simplifies this by using only stream processing. All data flows through the streaming platform, and batch operations become replay operations over the same stream. For IoT systems with well-defined schemas and reliable processing logic, Kappa reduces operational complexity. ## Data Streaming Technologies in IoT Modern IoT architectures rely on specialized streaming technologies designed to handle the unique challenges of sensor data. ### Apache Kafka for IoT Kafka 4.0+ excels at IoT ingestion due to its high throughput, durability, and scalability. Topics can be partitioned by device ID or geographic region to parallelize processing. KRaft mode eliminates ZooKeeper, simplifying operations and improving metadata scalability for millions of IoT devices. Kafka's retention policies let organizations keep raw sensor data for days or weeks, enabling reprocessing if business logic changes. For large-scale deployments, Tiered Storage offloads older IoT data to cost-effective object storage while keeping recent data on fast local disks. This dramatically reduces storage costs for long retention requirements. For details, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). Schema Registry ensures IoT data maintains consistent structure as devices evolve. When a sensor manufacturer updates firmware and adds new fields, Schema Registry manages backward compatibility across the streaming pipeline using versioned Avro, Protobuf, or JSON schemas. Each IoT message references a schema ID, enabling consumers to deserialize data correctly even as device fleets evolve over months or years. For schema management strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). ### Apache Flink for Real-Time Analytics Flink 1.19+ processes IoT streams with exactly-once semantics, critical for applications like energy metering or financial transactions. Its event time processing handles out-of-order events common in IoT systems where devices may buffer data during network interruptions. Watermarks track event time progress, enabling accurate windowing even with delayed sensor readings. Flink's stateful stream processing enables complex operations like sessionization of user interactions with smart devices or correlation of events from multiple sensors to detect system-wide patterns. State backends (RocksDB for large state, heap for low latency) can be configured based on IoT workload characteristics. For comprehensive coverage, see [What is Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) and [Watermarks and Triggers in Stream Processing](https://www.conduktor.io/glossary/watermarks-and-triggers-in-stream-processing). ### Protocol Bridges and Integration IoT ecosystems use various protocols beyond MQTT 5.0, including: - **CoAP (Constrained Application Protocol)** with DTLS 1.3 for secure communication on resource-constrained devices - **HTTP/3 over QUIC** for reliable, low-latency communication with reduced head-of-line blocking - **AMQP 1.0** for enterprise messaging with transactional semantics - **Proprietary protocols** from industrial equipment manufacturers Streaming architectures employ protocol bridges, lightweight services (often written in Go or Rust for performance) that translate between device protocols and Kafka. These bridges handle authentication, protocol translation, and message batching before publishing to Kafka topics. Cloud IoT platforms like AWS IoT Core, Azure IoT Hub, and Google Cloud IoT Core provide managed protocol bridges with built-in device management, certificate provisioning, and fleet monitoring. Protocol bridges standardize data formats early in the pipeline, simplifying downstream processing and enabling a single Kafka cluster to ingest data from diverse IoT device fleets. For serialization strategies, see [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka). ## Challenges and Best Practices Building production IoT streaming architectures requires addressing several operational and technical challenges. ### Scalability and Resource Management IoT deployments can grow from thousands to millions of devices rapidly. Architectures must scale horizontally by adding brokers, processing nodes, and storage capacity without downtime. Auto-scaling based on ingestion rates and processing lag helps manage variable workloads. Kubernetes-based deployments using operators like Strimzi enable dynamic scaling of Kafka clusters in response to IoT traffic patterns. For Kubernetes deployment strategies, see [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) and [Strimzi Kafka Operator](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes). Kafka cluster management becomes critical at scale. Conduktor provides comprehensive visibility into partition distribution, consumer lag, cluster health, and topic configuration ([Topics Guide](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics)), enabling teams to identify bottlenecks before they impact data freshness. Real-time monitoring of consumer lag helps detect when downstream processors cannot keep pace with IoT device ingestion rates ([Monitoring](https://docs.conduktor.io/guide/monitor-brokers-apps/index)). For monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ### Data Quality and Schema Evolution IoT sensors operate in harsh environments and may send malformed data due to hardware failures or network corruption. Stream processing logic should validate data against schemas, filter invalid readings, and route anomalies to dead-letter queues for investigation. Data quality frameworks can monitor IoT streams for anomalies like missing sensor readings, out-of-range values, or timestamp drift. For error handling patterns, see [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling). As device fleets evolve, schemas change. Implementing schema versioning and compatibility rules (forward, backward, full) ensures new device versions don't break existing processing pipelines. Schema Registry enforces compatibility policies, preventing incompatible schema changes that would break downstream consumers. Schema governance features help teams manage schema evolution across distributed IoT deployments with thousands of device types. For detailed guidance, see [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). ### Security and Access Control IoT data often contains sensitive information requiring encryption in transit and at rest. Modern IoT architectures use TLS 1.3 for device-to-cloud communication, providing improved security and reduced handshake latency. Mutual TLS (mTLS) authenticates both devices and servers using X.509 certificates, preventing unauthorized devices from connecting to the platform. Kafka supports authentication via SSL certificates, SASL (SCRAM, PLAIN, GSSAPI), or OAuth 2.0 for machine-to-machine communication. Fine-grained access control using ACLs ensures only authorized applications can consume specific sensor data streams. For example, building management applications can read HVAC sensor data but not security camera streams. For comprehensive security guidance, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices), [Kafka Authentication](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth), [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns), and [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka). ### Monitoring and Observability Production IoT systems require comprehensive monitoring of device connectivity, message throughput, processing latency, and data quality metrics. Modern observability stacks use OpenTelemetry for instrumentation, exporting traces, metrics, and logs to centralized platforms. Key IoT metrics to monitor include: - **Device connectivity**: Online/offline status, connection duration, reconnection frequency - **Ingestion rates**: Messages per second per device type, payload sizes, topic throughput - **Processing lag**: Consumer lag across stream processing applications, end-to-end latency from device to storage - **Data quality**: Schema validation failures, null value rates, out-of-range sensor readings Distributed tracing (OpenTelemetry traces through MQTT bridges, Kafka, and Flink) helps diagnose issues spanning edge gateways, message brokers, and processing engines. Prometheus scrapes metrics from Kafka exporters, Flink job managers, and custom IoT metrics endpoints. Grafana dashboards visualize device fleet health, data pipelines, and alerting thresholds. For monitoring infrastructure, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ## IoT Streaming Architecture Example Here's a practical example of publishing IoT sensor data to Kafka via an MQTT bridge: ```python # IoT device publishing temperature readings via MQTT import paho.mqtt.client as mqtt import json import time from datetime import datetime def publish_sensor_data(): client = mqtt.Client() client.tls_set(ca_certs="ca.pem", certfile="device.crt", keyfile="device.key") client.connect("mqtt.iot-platform.com", 8883) while True: reading = { "device_id": "temp-sensor-001", "timestamp": datetime.utcnow().isoformat(), "temperature": 22.5, "humidity": 45.2, "location": {"lat": 37.7749, "lon": -122.4194} } # Publish to MQTT topic (bridged to Kafka) client.publish("sensors/temperature", json.dumps(reading)) time.sleep(10) # Kafka Streams processing IoT data from kafka import KafkaConsumer import json consumer = KafkaConsumer( 'iot.sensors.temperature', bootstrap_servers=['kafka-1:9092', 'kafka-2:9092'], security_protocol='SASL_SSL', sasl_mechanism='SCRAM-SHA-512', auto_offset_reset='latest', value_deserializer=lambda m: json.loads(m.decode('utf-8')) ) for message in consumer: sensor_data = message.value # Detect temperature anomalies if sensor_data['temperature'] > 30.0: print(f"Alert: High temperature detected at {sensor_data['device_id']}") # Publish alert to downstream topic ``` This architecture enables: - Secure device communication using TLS 1.3 and X.509 certificates - Protocol translation from MQTT to Kafka for unified stream processing - Real-time anomaly detection with sub-second latency - Horizontal scalability to millions of devices For building Kafka-based data pipelines, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines) and [Streaming Data Pipeline](https://www.conduktor.io/glossary/streaming-data-pipeline). ## Real-World IoT Streaming Use Cases ### Smart City Traffic Management A metropolitan area deploys 10,000 traffic sensors across intersections, measuring vehicle counts, speeds, and pedestrian activity. These sensors publish readings every second to MQTT topics, bridged to Kafka. Flink processes the streams to calculate real-time congestion scores, predict traffic patterns using historical data, and optimize traffic light timing. City operators view live dashboards built on the processed streams, while urban planners query historical data from the data lake to design infrastructure improvements. ### Industrial Predictive Maintenance Manufacturing equipment contains hundreds of sensors monitoring temperature, vibration, pressure, and electrical current. An edge gateway collects readings every 100 milliseconds, running local anomaly detection algorithms. When anomalies exceed thresholds, the gateway publishes detailed diagnostic data to Kafka topics. Cloud-based Flink jobs correlate patterns across multiple machines, predict component failures, and automatically schedule maintenance before breakdowns occur. This reduces unplanned downtime by 40% and extends equipment lifespan. ## Summary IoT data streaming architectures allow organizations to act on sensor data in real-time. These systems combine specialized ingestion protocols like MQTT 5.0 and HTTP/3 with streaming platforms like Apache Kafka 4.0+ and Apache Flink 1.19+ to handle massive scale and low-latency requirements. Modern IoT architectures in 2025 rely on: - **KRaft-mode Kafka** for simplified operations and improved scalability to millions of devices - **Edge computing frameworks** (AWS IoT Greengrass v2, Azure IoT Edge, K3s) for local processing and ML inference - **Cloud IoT platforms** (AWS IoT Core, Azure IoT Hub) for managed protocol bridges and device management - **Schema Registry** for managing data evolution across heterogeneous device fleets - **OpenTelemetry, Prometheus, and Grafana** for comprehensive observability - **Conduktor** for Kafka management, monitoring, and governance at scale Effective architectures balance edge and cloud processing, implement solid data quality controls, and scale as device fleets grow. Following patterns like edge-to-cloud designs and applying best practices for security (TLS 1.3, mTLS), schema management, and monitoring helps teams build reliable IoT streaming systems. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/apache-kafka) - Distributed streaming platform for IoT data ingestion - [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) - Real-time IoT analytics and pattern detection - [Consumer Lag Monitoring](https://www.conduktor.io/consumer-lag-monitoring) - Monitor IoT data pipeline health and performance ## Sources and References 1. Apache Kafka Documentation - "Use Cases: Internet of Things" (https://kafka.apache.org/uses) 2. MQTT Version 5.0 Specification - OASIS Standard (https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html) 3. Eclipse Mosquitto - "MQTT and Kafka Integration Patterns" (https://mosquitto.org/) 4. AWS IoT Core Documentation - "Architecture Best Practices" (https://docs.aws.amazon.com/iot/) 5. Azure IoT Hub Documentation - "Device-to-Cloud Communication Guidance" (https://docs.microsoft.com/azure/iot-hub/) 6. Apache Flink Documentation - "Event Time and Watermarks" (https://nightlies.apache.org/flink/flink-docs-release-1.19/) 7. Conduktor Platform - "Kafka Management for IoT Workloads" (https://www.conduktor.io/) 8. OpenTelemetry Documentation - "Distributed Tracing for IoT Systems" (https://opentelemetry.io/docs/) Source: https://www.conduktor.io/glossary/iot-data-streaming-architectures --- # Kafka ACLs and Authorization Patterns Security in distributed systems is not optional. As Apache Kafka deployments grow from proof-of-concept experiments to mission-critical production platforms, controlling who can read, write, and manage streaming data becomes essential. Access Control Lists (ACLs) provide the foundation for authorization in Kafka, enabling organizations to enforce fine-grained permissions across topics, consumer groups, and cluster operations. This article covers how Kafka ACLs work in modern KRaft-based clusters (Kafka 4.0+), authorization patterns including OAuth 2.0 and Open Policy Agent integration, and practical approaches to managing access control at scale. ## Prerequisites Before working with Kafka ACLs, you should understand: - Basic Kafka concepts (topics, partitions, producers, consumers) - Authentication mechanisms (see [Kafka Authentication: SASL, SSL, and OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth)) - The distinction between authentication (proving identity) and authorization (controlling access) - KRaft mode architecture (see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka)) ## Understanding Kafka ACLs An Access Control List in Kafka defines permissions that determine which principals (authenticated users or applications) can perform specific operations on particular resources. Unlike authentication, which verifies identity, authorization ensures that authenticated entities only access what they are permitted to. Kafka ACLs follow a structured format with four core components: - **Principal**: The identity attempting the operation - **Resource**: The Kafka resource being accessed - **Operation**: The action being performed - **Permission Type**: Allow or Deny ### Principal Types Principals represent authenticated identities in Kafka ACLs. The format depends on the authentication mechanism: - **User principals**: `User:username` or `User:CN=service-name` (for mTLS, see [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka)) - **Group principals**: `Group:team-analytics` (when using group-based authorization) - **Wildcard**: `User:*` grants permissions to all authenticated users (use with extreme caution) For example, an ACL might specify: "User:analytics-app is allowed to Read from Topic:clickstream-events." Without this explicit permission, the analytics application would be denied access even if properly authenticated. ### ACL Storage in KRaft Mode In modern Kafka deployments (Kafka 3.3+), ACLs are stored in the `__cluster_metadata` topic managed by KRaft controllers. This represents a significant improvement over the legacy approach where ACLs were stored in ZooKeeper (replaced by KRaft in Kafka 4.0+), providing: - **Faster propagation**: Metadata changes replicate through Kafka's native replication, reducing latency from seconds to milliseconds - **Simplified operations**: No external ZooKeeper cluster to manage and monitor - **Better scalability**: The metadata topic scales with Kafka's proven log architecture - **Consistent views**: All brokers read from the same replicated log, eliminating metadata inconsistencies Legacy note: Kafka versions prior to 3.3 stored ACLs in ZooKeeper (legacy, replaced by KRaft in Kafka 4.0+) under `/kafka-acl` znodes. If migrating from ZooKeeper to KRaft, see [ZooKeeper to KRaft Migration](https://www.conduktor.io/glossary/zookeeper-to-kraft-migration). ### Authorizer Implementation Kafka evaluates ACLs through the **StandardAuthorizer** class (introduced in Kafka 3.0, replacing the older AclAuthorizer). When a client attempts an operation, Kafka checks all applicable ACLs and denies access unless an explicit Allow rule exists. This deny-by-default model provides strong security guarantees. To enable authorization, configure your broker with: ```properties # Kafka 3.0+ configuration authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer # Deny access by default if no ACL found (CRITICAL for security) allow.everyone.if.no.acl.found=false # Superusers bypass all ACL checks (for admin operations) super.users=User:admin;User:kafka-operator ``` ## How Kafka Authorization Works Authorization in Kafka operates through the Authorizer interface. The default implementation, StandardAuthorizer, evaluates permissions whenever a client performs an operation. Consider a producer attempting to write to a topic. The authorization flow includes: ![Consider a producer attempting to write to a topic. The authorization flow includes](https://www.conduktor.io/assets/images/glossary/kafka-acls-and-authorization-patterns-0.webp) 1. The client authenticates (via SASL, mTLS, or another mechanism) 2. The principal identity is extracted (e.g., `User:order-service`) 3. The producer attempts a Write operation on a Topic resource 4. AclAuthorizer queries stored ACLs for matching rules 5. If an Allow rule exists, the operation proceeds; otherwise, it is denied A typical producer needs Write, Describe, and optionally Create permissions. Consumers require even more permissions because they interact with both topics and consumer groups (Read and Describe on both resources). Missing any of these ACLs results in authorization failures that can be difficult to diagnose in production (see the Troubleshooting section below). ## Resource Patterns and Operations Kafka supports multiple resource pattern types that control how ACLs match resources, enabling flexible permission management. ### Resource Pattern Types - **LITERAL**: Exact match only. An ACL for `--resource-pattern-type literal --topic orders` only applies to the topic named exactly "orders". - **PREFIXED**: Matches resources starting with the specified prefix. An ACL for `--resource-pattern-type prefixed --topic team-a.` applies to `team-a.orders`, `team-a.payments`, `team-a.inventory`, etc. - **MATCH**: (Not commonly used for ACL creation) Matches resources that would be matched by literal or prefixed patterns. Example using prefixed patterns: ```bash # Grant Team A write access to all topics starting with "team-a." kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:team-a-app \ --operation Write \ --operation Describe \ --resource-pattern-type prefixed \ --topic team-a. # This grants access to: # - team-a.orders # - team-a.payments # - team-a.analytics # But NOT to team-b.orders or shared.events ``` Prefixed patterns dramatically reduce ACL sprawl in environments with many topics. Instead of creating individual ACLs for 50 team-specific topics, one prefixed ACL covers them all. ### Common Operations by Resource Type Different Kafka resources support different operations: **Topic Operations**: - Read: Consume messages from the topic - Write: Produce messages to the topic - Create: Create the topic (auto-creation) - Delete: Delete the topic - Describe: View topic metadata and configuration - Alter: Modify topic configuration - DescribeConfigs: View topic configurations - AlterConfigs: Modify topic configurations **Group Operations** (for consumer groups): - Read: Join the consumer group and consume - Describe: View consumer group metadata and offsets - Delete: Delete the consumer group **Cluster Operations**: - Create: Create topics at the cluster level - ClusterAction: Execute cluster-level operations - Describe: View cluster metadata - Alter: Modify cluster-level settings - DescribeConfigs: View broker configurations - AlterConfigs: Modify broker configurations - IdempotentWrite: Enable idempotent producer writes **TransactionalId Operations** (for transactional producers): - Describe: View transactional state - Write: Use the transactional ID for transactions ### Practical ACL Examples **Basic Producer** (writing to a specific topic): ```bash # Producer needs Write and Describe on the topic kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:payment-service \ --operation Write \ --operation Describe \ --topic payments.completed # For idempotent producers, also grant ClusterAction kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:payment-service \ --operation IdempotentWrite \ --cluster ``` **Basic Consumer** (reading from a topic): ```bash # Consumer needs Read and Describe on the topic kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:analytics-service \ --operation Read \ --operation Describe \ --topic payments.completed # Consumer also needs Read on the consumer group kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:analytics-service \ --operation Read \ --group analytics-group ``` **Stream Processing Application** (reading from input, writing to output): ```bash # Input topic permissions kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:fraud-detector \ --operation Read \ --operation Describe \ --topic transactions.raw # Output topic permissions kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:fraud-detector \ --operation Write \ --operation Describe \ --operation Create \ --topic transactions.fraud-alerts # Consumer group for state management kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:fraud-detector \ --operation Read \ --group fraud-detector-app # For Kafka Streams internal topics (if using) kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:fraud-detector \ --operation All \ --resource-pattern-type prefixed \ --topic fraud-detector- ``` **Admin/Operator** (managing topics and configurations): ```bash # Grant broad permissions for cluster administration kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:kafka-admin \ --operation All \ --topic '*' \ --cluster \ --group '*' # Or use superuser configuration (recommended) # In server.properties: super.users=User:kafka-admin ``` ## Common Authorization Patterns Organizations typically adopt authorization patterns that align with their operational structure and security requirements. ### Team-Based Access Control Many companies organize ACLs around team boundaries. An analytics team might have read-only access to all topics, while application teams have full control over their own topics: - Analytics team: Read access to `clickstream.*`, `orders.*`, `inventory.*` - Orders team: Full access to `orders.*` topics and consumer groups - Inventory team: Full access to `inventory.*` topics and consumer groups This pattern mirrors role-based access control (RBAC) principles and simplifies permission management as teams scale. ### Environment Separation Production, staging, and development environments often require different authorization policies. A common pattern uses separate Kafka clusters with distinct ACL configurations: - Production: Strict ACLs requiring explicit approval for all operations - Staging: Relaxed ACLs allowing broader access for testing - Development: Minimal ACLs or even disabled authorization Some organizations use Kafka topic prefixes (e.g., `dev.`, `staging.`, `prod.`) to implement environment separation within a single cluster, though this approach requires careful ACL management. ### Service-Based Permissions Microservices architectures benefit from service-specific principals where each service has a unique identity and receives only the permissions it needs: - `User:payment-service`: Write to `payments.completed`, Read from `orders.validated` - `User:notification-service`: Read from `payments.completed`, `orders.shipped` - `User:analytics-service`: Read from all topics via wildcard ACL This principle of least privilege minimizes blast radius if a service is compromised. ## Modern Authorization Patterns for 2025 Beyond traditional ACLs, modern Kafka deployments increasingly adopt advanced authorization patterns that integrate with cloud-native security frameworks and organizational identity systems. ### OAuth 2.0 and OIDC Integration OAuth 2.0 with OpenID Connect (OIDC) has become the standard authentication mechanism for cloud-native applications. Kafka 2.0+ supports OAuth through the SASL/OAUTHBEARER mechanism, enabling integration with identity providers like Okta, Auth0, Keycloak, or cloud provider IAM systems. In OAuth-based authorization: 1. Clients authenticate with the identity provider and receive a JWT token 2. The JWT token is presented to Kafka brokers 3. Kafka validates the token and extracts principal identity from claims 4. Standard ACLs are evaluated using the extracted principal This pattern enables centralized identity management, single sign-on (SSO), and dynamic credential rotation without restarting Kafka clients. Example broker configuration for OAuth: ```properties # Enable OAuth authentication sasl.enabled.mechanisms=OAUTHBEARER sasl.mechanism.inter.broker.protocol=OAUTHBEARER # Configure OAuth validation listener.name.sasl_ssl.oauthbearer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required; listener.name.sasl_ssl.oauthbearer.sasl.login.callback.handler.class=io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler listener.name.sasl_ssl.oauthbearer.sasl.server.callback.handler.class=io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler # OAuth issuer URL oauth.token.endpoint.uri=https://auth.company.com/oauth/token oauth.jwks.endpoint.uri=https://auth.company.com/oauth/jwks oauth.valid.issuer.uri=https://auth.company.com ``` Example producer configuration: ```properties security.protocol=SASL_SSL sasl.mechanism=OAUTHBEARER sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \ oauth.client.id="payment-service" \ oauth.client.secret="***" \ oauth.token.endpoint.uri="https://auth.company.com/oauth/token"; sasl.login.callback.handler.class=io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler ``` Once OAuth authentication establishes identity, traditional ACLs control what the authenticated principal can access. For comprehensive OAuth setup, see [Kafka Authentication: SASL, SSL, and OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). ### Open Policy Agent (OPA) Integration Open Policy Agent provides policy-based authorization using the Rego policy language. OPA decouples authorization logic from application code, enabling centralized policy management and sophisticated authorization rules based on multiple attributes. Instead of static ACLs, OPA can make authorization decisions based on: - Principal identity and group membership - Resource attributes (topic names, partitions) - Operation types - Contextual information (time of day, client IP, request metadata) - External data sources (user databases, compliance systems) Example OPA policy for time-based access control: ```rego package kafka.authz import future.keywords.if # Allow analytics team to read only during business hours allow if { input.principal.name == "analytics-service" input.operation.name == "Read" is_business_hours } is_business_hours if { hour := time.clock(time.now_ns())[0] hour >= 9 hour < 18 } # Allow payment service full access anytime allow if { input.principal.name == "payment-service" startswith(input.resource.name, "payments.") } ``` Integrating OPA with Kafka requires a custom authorizer that delegates decisions to the OPA service. Several open-source implementations exist, including those from Bisnode and Confluent's commercial offerings. OPA excels in environments requiring complex authorization logic that goes beyond simple principal-resource-operation matching. For implementing broader policy enforcement, see [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming). ### Attribute-Based Access Control (ABAC) ABAC extends authorization beyond identity to include attributes of the subject, resource, action, and environment. This enables dynamic, context-aware access control. Example ABAC scenarios: - Grant access based on data classification: Only principals with "PII-access" attribute can read topics tagged as "contains-pii" - Environment-based rules: Production data can only be accessed from production networks - Project-based access: Users can access topics tagged with their project ID - Compliance-driven controls: GDPR-protected data requires additional authorization checks ABAC typically requires: 1. A metadata system tagging resources with attributes 2. An identity provider supplying subject attributes 3. A policy engine evaluating attribute-based rules (like OPA) 4. Integration points in the Kafka authorizer ### Role-Based Access Control (RBAC) RBAC assigns permissions to roles rather than individual principals, simplifying management in large organizations. Instead of granting "User:alice" access to 50 topics, you grant the "data-analyst" role access and assign users to roles. Traditional Kafka ACLs don't natively support roles, but RBAC can be implemented through: 1. **Group-based ACLs**: Use `Group:analysts` principals when your authentication system supports group extraction 2. **Prefix-based role simulation**: Use topic naming conventions like `role-analysts.*` with prefixed ACLs 3. **External RBAC systems**: Integrate with LDAP, Active Directory, or cloud IAM systems that map roles to Kafka principals 4. **Commercial platforms**: Conduktor provides native RBAC support with visual role management Example RBAC pattern using topic prefixes: ```bash # Define role through topic naming: role-analysts.* topics kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:alice \ --allow-principal User:bob \ --allow-principal User:charlie \ --operation Read \ --resource-pattern-type prefixed \ --topic role-analysts. # Any topic named role-analysts.* is readable by these users # New topics following the convention automatically inherit permissions ``` For multi-team environments, see [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments) for comprehensive RBAC strategies. ## ACLs in Data Streaming Ecosystems Kafka rarely operates in isolation. Modern data streaming platforms integrate Kafka with stream processors (Apache Flink, Kafka Streams), data warehouses (Snowflake, BigQuery), and operational databases. Authorization must extend across this entire ecosystem. Stream processing applications often require complex ACL configurations. A Flink job reading from multiple input topics, maintaining state, and writing to output topics needs permissions for: - Reading from input topics - Writing to output topics - Managing consumer groups for checkpointing - Accessing internal Kafka topics for coordination (if applicable) Schema registries add another authorization layer. While Kafka ACLs control access to data, schema registry permissions govern who can modify data contracts. Misaligned permissions between Kafka and schema registries can create operational headaches. Change Data Capture (CDC) connectors reading from databases and writing to Kafka require careful ACL planning. These connectors often use service accounts with broad permissions, making them high-value targets for attackers. Real-time analytics platforms that query Kafka topics (via tools like ksqlDB or Kafka Streams) need read access but should not have write permissions unless explicitly required. Separating read and write permissions prevents accidental data corruption. ## Managing ACLs at Scale As Kafka clusters grow, ACL management becomes a significant operational challenge. A cluster with hundreds of topics, dozens of consumer groups, and numerous applications can accumulate thousands of ACL entries. Manual ACL management using CLI commands quickly becomes error-prone. Common problems include: - Typos in principal names or topic patterns - Forgetting required permissions (e.g., Describe along with Read) - Stale ACLs for decommissioned applications - No audit trail for ACL changes - Difficulty understanding current permission state - Inconsistent ACL patterns across teams - Authorization failures with unclear root causes Organizations address these challenges through automation, infrastructure-as-code, and specialized tooling. ### Infrastructure-as-Code for ACLs Treating ACLs as declarative infrastructure enables version control, peer review, and automated deployment. Store ACL definitions in YAML or JSON files alongside application code: Example YAML ACL definition: ```yaml # acls/payment-service.yaml principals: - name: payment-service type: User acls: - resource: Topic name: payments.completed pattern: LITERAL operations: - Write - Describe permission: Allow - resource: Topic name: orders.validated pattern: LITERAL operations: - Read - Describe permission: Allow - resource: Group name: payment-service-group pattern: LITERAL operations: - Read permission: Allow - resource: Cluster operations: - IdempotentWrite permission: Allow ``` Apply ACLs via CI/CD pipeline: ```bash #!/bin/bash # Apply ACLs from YAML definitions for acl_file in acls/*.yaml; do echo "Applying ACLs from $acl_file" # Use a tool like yq to parse YAML and generate kafka-acls.sh commands # Or use a dedicated ACL management tool ./apply-acls.sh $acl_file done ``` Benefits of infrastructure-as-code for ACLs: - **Version control**: Every ACL change tracked in git history - **Peer review**: Pull requests require approval before ACL changes - **Rollback capability**: Revert to previous ACL state if issues occur - **Documentation**: ACL definitions serve as documentation - **Consistency**: Standardized format across all applications - **Testing**: Validate ACL definitions in CI before applying ### Conduktor Platform for ACL Management **Conduktor** provides enterprise-grade ACL management with [visual RBAC capabilities](https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac) and governance features that dramatically simplify operations at scale. Key Conduktor capabilities for ACLs: - **Visual ACL Editor**: Create and modify ACLs through an intuitive UI instead of complex CLI commands. See all permissions for a principal or resource in one view, eliminating the need to parse kafka-acls.sh output. - **ACL Validation**: Conduktor validates ACL configurations before applying them, catching common mistakes like missing Describe permissions or incorrect resource patterns. Dry-run mode shows what would change without actually modifying ACLs. - **Permission Explorer**: Understand "who can access what" through powerful search and filtering. Find all principals with access to a topic, or all resources a service can access. Identify over-permissioned accounts quickly. - **Audit Logging**: Every ACL change is logged with timestamp, user, and justification. Compliance teams can track permission changes over time and generate audit reports. - **Testing with Conduktor Gateway**: Test ACL configurations in a safe environment using Conduktor Gateway, a Kafka proxy, and its authorization testing features. Gateway can simulate different principals and operations to verify ACLs work as intended before deploying to production. Gateway also enables chaos engineering for authorization - intentionally inject authorization failures to test application error handling. - **Self-Service ACL Requests**: Enable developers to request ACL changes through Conduktor's approval workflow. Platform teams review and approve requests, maintaining governance while reducing operational toil. Example workflow with Conduktor: 1. Developer requests Read access to `customer-events` topic via Conduktor UI 2. Platform team reviews request, sees which permissions already exist 3. Conduktor generates the complete ACL configuration (including Describe) 4. Approval triggers automated deployment via Conduktor's API 5. Change logged in audit trail with requestor and approver For authorization testing and chaos engineering, Conduktor Gateway acts as a proxy that can: - Enforce additional authorization policies beyond Kafka ACLs - Test application behavior when authorization fails - Gradually roll out ACL changes to subset of clients - Monitor authorization attempts in real-time Learn more: https://conduktor.io ### Listing and Auditing ACLs Regular ACL audits prevent permission sprawl and identify security issues: ```bash # List all ACLs in the cluster kafka-acls.sh --bootstrap-server localhost:9092 --list # List ACLs for a specific principal kafka-acls.sh --bootstrap-server localhost:9092 --list \ --principal User:payment-service # List ACLs for a specific topic kafka-acls.sh --bootstrap-server localhost:9092 --list \ --topic payments.completed # Export all ACLs for backup/analysis kafka-acls.sh --bootstrap-server localhost:9092 --list > acls-backup-$(date +%Y%m%d).txt ``` Audit questions to ask regularly: - Which principals have wildcard access (`--topic '*'`)? - Are there ACLs for decommissioned services? - Do any principals have more permissions than required? - Are superusers properly restricted? - Do all production services use dedicated principals? ### Programmatic ACL Management For advanced automation, use the Kafka Admin API to manage ACLs programmatically: ```java // Java example using Kafka Admin Client Properties props = new Properties(); props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); AdminClient admin = AdminClient.create(props); // Create ACL for payment service AclBinding acl = new AclBinding( new ResourcePattern(ResourceType.TOPIC, "payments.completed", PatternType.LITERAL), new AccessControlEntry("User:payment-service", "*", AclOperation.WRITE, AclPermissionType.ALLOW) ); CreateAclsResult result = admin.createAcls(Collections.singleton(acl)); result.all().get(); // Wait for completion admin.close(); ``` Programmatic management enables: - Dynamic ACL creation based on service discovery - Automated ACL lifecycle tied to application deployment - Integration with external approval systems - Complex ACL patterns based on organizational policies ### Monitoring Authorization Failures Monitor authorization metrics to detect misconfigurations and potential security incidents: Key metrics to track: - `kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec`: Failed produce requests (often ACL issues) - `kafka.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec`: Failed consume requests - Authorization failure log entries: `org.apache.kafka.common.errors.AuthorizationException` Example alerting rule: ``` ALERT KafkaAuthorizationFailures IF rate(kafka_server_broker_topic_metrics_failed_produce_requests_total[5m]) > 10 SUMMARY "High rate of authorization failures detected" DESCRIPTION "{{ $value }} authorization failures/sec on cluster {{ $labels.cluster }}" ``` Regular ACL audits combined with real-time monitoring create a defense-in-depth authorization strategy. Monitoring authorization failures helps detect misconfigured applications before they impact users. ## Troubleshooting Common ACL Issues Authorization failures often manifest as cryptic error messages in application logs. Understanding common issues accelerates debugging. ### Producer Authorization Errors - **Error**: `org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [payments.completed]` - **Causes**: - Missing Write permission on the topic - Missing Describe permission (always required with Write) - Principal name mismatch (e.g., ACL for `User:payment-service` but client authenticates as `User:CN=payment-service`) **Solution**: ```bash # Verify principal name (check client authentication logs) # Add both Write and Describe permissions kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:payment-service \ --operation Write \ --operation Describe \ --topic payments.completed ``` - **Error**: `org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed` - **Causes**: - Missing IdempotentWrite permission when using idempotent producer - Missing ClusterAction permission for certain admin operations **Solution**: ```bash # Grant IdempotentWrite for idempotent producers kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:payment-service \ --operation IdempotentWrite \ --cluster ``` ### Consumer Authorization Errors - **Error**: `org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [customer-events]` - **Causes**: - Missing Read permission on topic - Missing Describe permission on topic **Solution**: ```bash kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:analytics-service \ --operation Read \ --operation Describe \ --topic customer-events ``` - **Error**: `org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: analytics-group` - **Causes**: - Missing Read permission on consumer group - Consumer group name doesn't match ACL pattern **Solution**: ```bash # Grant Read permission on the consumer group kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:analytics-service \ --operation Read \ --group analytics-group # Or use prefixed pattern for multiple consumer groups kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:analytics-service \ --operation Read \ --resource-pattern-type prefixed \ --group analytics- ``` ### Admin Operation Errors - **Error**: `org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [new-topic]` (when creating topics) - **Causes**: - Missing Create permission on topic or cluster **Solution**: ```bash # Option 1: Grant Create on specific topic (for auto-creation) kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:app-service \ --operation Create \ --topic new-topic # Option 2: Grant Create at cluster level (for admin operations) kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:kafka-admin \ --operation Create \ --cluster ``` ### Debugging Techniques **1. Enable authorization debug logging**: ```properties # In broker log4j.properties log4j.logger.kafka.authorizer.logger=DEBUG ``` This logs every authorization decision with principal, resource, operation, and result. **2. List effective ACLs for a principal**: ```bash # See all ACLs for a specific principal kafka-acls.sh --bootstrap-server localhost:9092 \ --list \ --principal User:payment-service ``` **3. Test with superuser**: ```properties # Temporarily add your test principal as superuser to verify non-ACL issues # In server.properties: super.users=User:test-principal ``` If the operation succeeds as superuser, the issue is definitely ACL-related. **4. Check principal extraction**: Verify how Kafka extracts the principal from authentication. For mTLS, Kafka uses the certificate's Distinguished Name (DN). For SASL, it depends on the mechanism. Example: If your certificate has `CN=payment-service,OU=Engineering,O=Company`, Kafka might extract `User:CN=payment-service,OU=Engineering,O=Company` or just `User:payment-service` depending on configuration. Check broker configuration: ```properties # For mTLS, control principal extraction ssl.principal.mapping.rules=RULE:^CN=(.*?),OU=.*$/$1/,DEFAULT ``` **5. Verify ACL propagation**: In KRaft mode, ACL changes propagate through the metadata log. Verify all brokers have the latest metadata: ```bash # Check metadata log on each broker kafka-metadata-shell.sh --snapshot /var/lib/kafka/data/__cluster_metadata-0/*.checkpoint # Or check broker logs for ACL updates grep "Processing ACL" /var/log/kafka/server.log ``` ### Common Pitfalls - **Forgetting Describe permission**: Almost all operations require Describe in addition to the primary operation. Missing Describe is the #1 ACL mistake. - **Principal name mismatches**: ACL says `User:app` but client authenticates as `User:CN=app` or `User:app@REALM.COM`. Always verify exact principal format. - **Consumer group wildcards**: Using `--group '*'` grants access to ALL consumer groups. Use prefix patterns instead. - **Order of operations**: Create ACLs BEFORE deploying the application. Applications fail fast without retrying authorization failures. - **Case sensitivity**: Kafka principal names are case-sensitive. `User:PaymentService` ≠ `User:paymentservice`. ## Best Practices for Kafka Authorization Successful ACL implementations follow several key principles: - **Start with deny-by-default**: Never disable authorization in production. Configure `allow.everyone.if.no.acl.found=false` to require explicit ACLs for all operations. - **Use service accounts**: Avoid shared credentials. Each application should have its own principal with specific permissions. - **Apply least privilege**: Grant only the minimum permissions required. Wildcard ACLs (`--topic '*'`) should be rare and carefully justified. - **Separate administrative access**: Cluster administration operations (creating topics, modifying configurations) require special permissions. Limit these to a small set of operators or automation systems. - **Document ACL rationale**: Maintain documentation explaining why each ACL exists and which application or team owns it. - **Integrate with identity providers**: For user access, integrate Kafka authentication with corporate identity systems (LDAP, Active Directory, OAuth) to use existing access controls. - **Monitor and alert**: Track authorization failures in metrics and logs. Sudden spikes often indicate misconfiguration or security incidents. - **Test ACL changes**: Validate ACL modifications in non-production environments before applying to production clusters. - **Plan for rotation**: Service credentials should be rotated regularly. Ensure ACLs reference principals in a way that survives credential rotation. - **Use prefixes and patterns**: Kafka supports prefix and pattern matching in ACL rules, reducing the number of individual ACLs needed for related resources. ## Summary Kafka ACLs provide fine-grained authorization control essential for production streaming platforms. Understanding how ACLs work in modern KRaft-based deployments, from the basic four-component structure to StandardAuthorizer evaluation, enables teams to design secure, scalable permission models. Modern authorization extends beyond traditional ACLs to include OAuth 2.0/OIDC integration for cloud-native identity management, Open Policy Agent for policy-based authorization, and sophisticated RBAC/ABAC patterns for large-scale environments. These advanced patterns provide the flexibility and centralization required by enterprise streaming platforms in 2025. Common patterns like team-based access control, environment separation, and service-specific permissions help organizations manage authorization at scale. Resource pattern types (LITERAL, PREFIXED) reduce ACL sprawl, while infrastructure-as-code practices enable version control and peer review of permission changes. Managing ACLs through automation, infrastructure-as-code, and specialized tooling like Conduktor dramatically reduces errors and provides essential audit trails. Conduktor's visual ACL management, validation features, and Conduktor Gateway's authorization testing capabilities simplify operations for platform teams while maintaining strong security governance. Following best practices like least privilege, deny-by-default, and regular audits ensures authorization remains a security strength rather than an operational burden. Combined with comprehensive troubleshooting knowledge and real-time monitoring, these practices create defense-in-depth security for streaming data infrastructure. Effective authorization is not a one-time configuration but an ongoing process that evolves with organizational needs and security requirements. ## Related Concepts - [Kafka Authentication: SASL, SSL, and OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth) - Authentication mechanisms that establish identity before ACL evaluation. - [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments) - Isolation strategies and ACL patterns for shared infrastructure across teams. - [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming) - Broader governance patterns including OPA integration and policy-based authorization controls. ## Sources and References - [Apache Kafka Documentation - Authorization and ACLs](https://kafka.apache.org/documentation/#security_authz) - Official documentation covering Kafka's authorization model, StandardAuthorizer implementation, and KRaft-based ACL storage - [KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum) - The Kafka Improvement Proposal that introduced KRaft mode and changed how ACLs are stored and propagated - [KIP-419: Extend ACL Resource Pattern Types](https://cwiki.apache.org/confluence/display/KAFKA/KIP-419%3A+Extend+ACL+Resource+Pattern+Types) - Introduces PREFIXED resource patterns for more flexible ACL management - [Apache Kafka Security: OAuth 2.0 Authentication](https://kafka.apache.org/documentation/#security_sasl_oauthbearer) - Documentation for integrating OAuth 2.0/OIDC with Kafka for modern authentication flows - [Open Policy Agent (OPA) Documentation](https://www.openpolicyagent.org/docs/latest/) - Policy-based authorization framework for implementing ABAC patterns with Kafka - [Strimzi Kafka OAuth](https://github.com/strimzi/strimzi-kafka-oauth) - Open-source implementation of OAuth 2.0 authentication and authorization for Kafka - [NIST Special Publication 800-162: Guide to Attribute Based Access Control (ABAC)](https://csrc.nist.gov/publications/detail/sp/800-162/final) - Framework for understanding modern access control patterns applicable to distributed systems - [Conduktor Platform Documentation](https://docs.conduktor.io/platform/) - Commercial platform for Kafka ACL management, visualization, and governance - "Kafka: The Definitive Guide" (2nd Edition) by Neha Narkhede, Gwen Shapira, and Todd Palino - Chapter on Securing Kafka provides practical guidance on implementing ACLs in production environments - [Cloud Native Computing Foundation (CNCF) - Kafka Security in Cloud-Native Environments](https://www.cncf.io/) - Community-driven best practices for securing Kafka with modern identity and authorization patterns Source: https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns --- # Kafka Admin Operations and Maintenance While developers focus on producing and consuming data, Kafka administrators ensure the underlying infrastructure remains healthy, performant, and secure. Effective admin operations and maintenance are critical to preventing data loss, minimizing downtime, and maintaining the reliability that streaming applications depend on. This article covers the essential practices for administering Kafka clusters in production environments, from daily monitoring tasks to strategic planning for disaster recovery. ![Kafka admin operations workflow and monitoring layers](https://www.conduktor.io/assets/images/glossary/kafka-admin-operations-and-maintenance-0.webp) ## Understanding Kafka Administration Responsibilities Kafka administrators manage the health and performance of broker clusters, coordinate topic configurations, enforce security policies, and respond to operational incidents. Unlike traditional databases with single-instance management, Kafka's distributed nature requires administrators to think about replication, partition leadership, and cluster-wide consistency. Key responsibilities include: - Monitoring broker health and resource utilization - Managing topic lifecycles and partition assignments - Configuring security policies and access controls - Planning capacity and scaling strategies - Performing upgrades with zero downtime - Responding to production incidents and performance issues The complexity increases with cluster size. A three-broker development cluster is straightforward, but a production cluster with dozens of brokers serving hundreds of topics requires systematic approaches to monitoring and automation. ### KRaft Mode: Modern Kafka Architecture Since Kafka 3.3 (production-ready in Kafka 4.0+), Kafka uses **KRaft mode** (Kafka Raft metadata mode) instead of ZooKeeper for managing cluster metadata. This architectural change simplifies operations by: - Eliminating ZooKeeper dependency, reducing operational complexity - Improving metadata scalability (supporting millions of partitions) - Enabling faster controller failover (sub-second instead of seconds) - Simplifying deployment and reducing infrastructure costs In KRaft mode, Kafka brokers elect controllers using the Raft consensus protocol. Administrators should understand: - **Controller quorum**: A subset of brokers (or dedicated controller nodes) form the metadata quorum - **Quorum sizing**: Typically 3 or 5 controller nodes for production (must be odd for majority voting) - **Metadata logs**: Stored in `__cluster_metadata` topic, separate from data logs For detailed coverage of KRaft architecture, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). For migration from legacy ZooKeeper-based clusters, see [ZooKeeper to KRaft Migration](https://www.conduktor.io/glossary/zookeeper-to-kraft-migration). ## Cluster Health Monitoring Administrators must track metrics across multiple dimensions: broker health, topic performance, consumer lag, and system resources. ### Critical Metrics to Monitor - **Broker-level metrics** include CPU usage, disk I/O, network throughput, and JVM heap memory. High disk utilization can indicate retention policies need adjustment, while consistent CPU spikes may signal undersized brokers or inefficient serialization. - **Topic-level metrics** track message rates (bytes in/out per second), partition count, and replication status. Under-replicated partitions are a critical warning sign, they indicate that some replicas are not keeping up with the leader, creating data durability risks. - **Consumer metrics** focus on consumer lag, the difference between the latest offset and the consumer's current position. Growing lag suggests consumers cannot keep pace with producers, requiring either consumer optimization or scaling. ### Monitoring Tools and Approaches Kafka exposes metrics via JMX (Java Management Extensions), which can be collected by monitoring systems. Modern monitoring stacks (as of 2025) typically include: **Open-source monitoring stack:** - **Prometheus**: Collects Kafka metrics via JMX Exporter or Kafka Exporter - **Kafka Lag Exporter**: Specialized tool for tracking consumer lag across all groups - **Grafana**: Visualizes metrics with pre-built Kafka dashboards - **Alertmanager**: Routes alerts based on configured thresholds **Example monitoring setup:** ```yaml # Prometheus JMX Exporter configuration for Kafka broker rules: - pattern: kafka.server<>Count name: kafka_server_broker_topic_metrics_$1 - pattern: kafka.server<>Value name: kafka_server_replica_manager_under_replicated_partitions ``` For example, an alert for under-replicated partitions exceeding zero for more than five minutes should trigger immediate investigation, as this indicates potential data loss risk if the leader broker fails. **Commercial platforms:** [Conduktor](https://www.conduktor.io) provides unified dashboards that aggregate cluster health metrics, topic statistics, and consumer lag in a single interface, reducing the complexity of monitoring distributed systems. Conduktor includes built-in [alerting](https://docs.conduktor.io/guide/monitor-brokers-apps/alerts), governance, and operational features beyond basic monitoring. [Explore Conduktor's monitoring capabilities](https://docs.conduktor.io/guide/monitor-brokers-apps/index) for comprehensive cluster observability. For comprehensive coverage of Kafka monitoring architecture and best practices, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ## Topic and Partition Management Topics are the primary organizational structure in Kafka, and managing them properly is essential for performance and cost control. ### Topic Creation and Configuration When creating topics, administrators must decide on: - **Partition count**: Higher partition counts enable greater parallelism but increase overhead. A common starting point is one partition per expected consumer in the consumer group. - **Replication factor**: Typically set to 3 for production workloads to survive two broker failures. - **Retention policies**: Time-based (e.g., 7 days) or size-based (e.g., 100GB per partition) retention controls disk usage. - **Compaction settings**: Log compaction is essential for changelog topics in stream processing applications. **Using Kafka CLI tools:** ```bash # Create a topic with 12 partitions and replication factor 3 kafka-topics.sh --bootstrap-server localhost:9092 \ --create \ --topic user-events \ --partitions 12 \ --replication-factor 3 \ --config retention.ms=604800000 \ --config compression.type=lz4 # List all topics kafka-topics.sh --bootstrap-server localhost:9092 --list # Describe a specific topic (shows partition distribution) kafka-topics.sh --bootstrap-server localhost:9092 \ --describe \ --topic user-events # Increase partition count (cannot be decreased) kafka-topics.sh --bootstrap-server localhost:9092 \ --alter \ --topic user-events \ --partitions 24 ``` **Using AdminClient API (Java):** ```java Properties props = new Properties(); props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); try (AdminClient admin = AdminClient.create(props)) { NewTopic topic = new NewTopic("user-events", 12, (short) 3) .configs(Map.of( "retention.ms", "604800000", "compression.type", "lz4" )); admin.createTopics(List.of(topic)).all().get(); System.out.println("Topic created successfully"); } ``` For detailed topic design guidelines, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines). ### Real-World Example: Retention Tuning Consider a logging topic receiving 1TB of data daily. With a default 7-day retention, the topic stores 7TB. If disk capacity is limited, administrators might reduce retention to 3 days or enable compression (e.g., `compression.type=lz4`) to reduce storage by 50-70%. Changing retention on active topics requires careful planning. Reducing retention from 7 days to 1 day immediately makes 6 days of data eligible for deletion, which could impact consumers expecting historical data. ## Security and Access Control Kafka security involves multiple layers: authentication (verifying identity), authorization (controlling permissions), and encryption (protecting data in transit and at rest). ### Authentication and Authorization Kafka supports several authentication mechanisms (as of Kafka 4.0+): - **SASL/PLAIN**: Simple username/password authentication (not recommended for production) - **SASL/SCRAM-SHA-512**: Salted Challenge Response Authentication Mechanism, more secure than PLAIN - **SSL/TLS (mTLS)**: Mutual certificate-based authentication, strongest security - **OAuth 2.0**: Integration with enterprise identity providers (Okta, Azure AD, Keycloak) - **Kerberos**: For enterprise environments with existing Kerberos infrastructure **Example: Managing ACLs via CLI** ```bash # Grant read access to a consumer group for a specific topic kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:analytics-service \ --operation Read \ --topic customer-events \ --group analytics-consumers # Grant write access for a producer kafka-acls.sh --bootstrap-server localhost:9092 \ --add \ --allow-principal User:order-service \ --operation Write \ --topic orders # List all ACLs for a topic kafka-acls.sh --bootstrap-server localhost:9092 \ --list \ --topic customer-events ``` Authorization uses Access Control Lists (ACLs) to define who can produce to topics, consume from topics, or perform admin operations. For example, an ACL might grant the `analytics-team` group read access to `customer-events` but deny write access. For detailed security architecture and best practices, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices), [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns), and [Kafka Authentication (SASL, SSL, OAuth)](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). For mutual TLS implementation, see [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka). ### Audit Logging Tracking who accessed what data and when is critical for compliance and security investigations. Kafka's audit logging capabilities, combined with tools like Conduktor that provide ACL management interfaces, help administrators maintain security without manual command-line ACL creation. Commercial platforms can integrate with SIEM systems for centralized security monitoring. ## Performance Tuning and Optimization Kafka performance depends on proper configuration of brokers, producers, consumers, and the underlying infrastructure. ### Identifying Bottlenecks Common performance bottlenecks include: - **Disk I/O**: Slow disks or insufficient IOPS limit throughput. Using SSDs or provisioned IOPS on cloud platforms significantly improves write performance. - **Network saturation**: Replication traffic and consumer fetch requests consume network bandwidth. Monitoring network utilization helps identify when additional brokers or network capacity is needed. - **Unbalanced partition leadership**: If one broker handles leadership for many high-traffic partitions, it becomes a bottleneck. Running preferred leader election redistributes leadership. ### Configuration Tuning Key broker configurations for performance include: - `num.network.threads`: Handles network requests; increase for high-connection workloads (default: 3, consider 8-16 for large clusters) - `num.io.threads`: Handles disk I/O; typically set to match disk count (default: 8) - `socket.send.buffer.bytes` and `socket.receive.buffer.bytes`: Larger buffers improve throughput for high-latency networks (consider 1MB+ for WAN) - `replica.fetch.max.bytes`: Maximum bytes per fetch request for replication (default: 1MB, increase for high-throughput topics) **Example: Performance-tuned broker configuration** ```properties # server.properties - Performance optimizations num.network.threads=16 num.io.threads=16 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 replica.fetch.max.bytes=5242880 # Enable compression to reduce network and disk I/O compression.type=lz4 # Adjust log segment size for better performance log.segment.bytes=536870912 # Background thread configuration num.replica.fetchers=4 num.recovery.threads.per.data.dir=2 ``` Producer and consumer configurations also matter. Producers using `acks=all` ensure durability but reduce throughput compared to `acks=1`. Batching (`linger.ms`, `batch.size`) trades latency for throughput. **Running preferred leader election** to rebalance partition leadership: ```bash # Trigger preferred leader election for all partitions kafka-leader-election.sh --bootstrap-server localhost:9092 \ --election-type PREFERRED \ --all-topic-partitions ``` For comprehensive performance optimization strategies, see [Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide) and [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning). ## Backup, Recovery, and Disaster Planning While Kafka's replication provides durability within a cluster, administrators must plan for cluster-wide failures, data center outages, or data corruption scenarios. ### Replication Strategies **Within-cluster replication** (via replication factor) protects against individual broker failures. **Multi-datacenter replication** using tools like MirrorMaker 2 (part of Kafka Connect framework) provides disaster recovery across geographic regions. For detailed coverage of replication architecture, see [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability). For cross-cluster replication patterns, see [Kafka MirrorMaker 2 for Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication). ### Backup Approaches and Tiered Storage Kafka's append-only log structure makes backups straightforward conceptually but challenging operationally. Modern approaches (Kafka 3.6+) include: **Tiered Storage (Kafka 3.6+):** Automatically offloads older log segments to object storage (S3, Azure Blob, GCS) while keeping recent data on local disks. This dramatically reduces storage costs and simplifies disaster recovery. ```properties # server.properties - Enable tiered storage remote.log.storage.system.enable=true remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.RemoteLogManager remote.log.manager.task.interval.ms=30000 # Configure S3 as remote storage remote.log.storage.backend=s3 s3.bucket.name=kafka-tiered-storage s3.region=us-east-1 ``` **Traditional backup approaches:** - **Topic snapshots**: Using consumers to read entire topics and store in object storage - **Log segment backups**: Directly copying log segment files from broker disks - **Continuous replication**: Using MirrorMaker 2 to maintain a synchronized secondary cluster **Example: Partition reassignment for disaster recovery** ```bash # Generate partition reassignment plan kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \ --topics-to-move-json-file topics.json \ --broker-list "4,5,6" \ --generate # Execute the reassignment kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \ --reassignment-json-file reassignment.json \ --execute # Verify completion kafka-reassign-partitions.sh --bootstrap-server localhost:9092 \ --reassignment-json-file reassignment.json \ --verify ``` Testing recovery procedures is essential. Many organizations discover gaps in their disaster recovery plans only when attempting actual recovery. For comprehensive disaster recovery planning, see [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters). For tiered storage details, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). ## Troubleshooting Common Issues Effective troubleshooting requires understanding Kafka's architecture and having systematic diagnostic approaches. ### Under-Replicated Partitions When partitions show as under-replicated, check: 1. Broker health: Is the follower broker offline or experiencing issues? 2. Network connectivity: Can followers reach the leader? 3. Disk space: Are brokers running out of disk? 4. Replication lag: Are followers too far behind to catch up? ### Consumer Lag Growing consumer lag indicates consumers cannot keep pace with producers. Solutions include: - Scaling consumer groups by adding consumer instances - Optimizing consumer processing logic - Increasing partition count to enable more parallelism - Checking for network or serialization bottlenecks For detailed coverage of consumer group management, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). For monitoring consumer lag patterns, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). ### Broker Restarts and Leadership Changes When brokers restart, partition leadership changes. If restarts are frequent, investigate underlying causes (memory issues, crash loops) rather than treating symptoms. **Diagnostic commands:** ```bash # Check broker logs for errors kafka-broker-logs.sh --log-dirs /var/lib/kafka/data # Verify replica synchronization status kafka-replica-verification.sh --broker-list localhost:9092 \ --topic-white-list '.*' # Monitor ISR (In-Sync Replica) shrinkage kafka-topics.sh --bootstrap-server localhost:9092 \ --describe \ --under-replicated-partitions ``` ## Kubernetes-Based Kafka Operations Running Kafka on Kubernetes has become increasingly common, leveraging operators like **Strimzi** to automate deployment, configuration, and operational tasks. ### Strimzi Operator Benefits The Strimzi Kafka Operator provides: - Automated broker deployment and scaling - Rolling updates with zero downtime - Automatic TLS certificate management - Integration with Kubernetes monitoring and logging - GitOps-friendly declarative configuration **Example: Kafka cluster definition in Strimzi** ```yaml apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: production-cluster spec: kafka: version: 4.0.0 replicas: 6 # KRaft mode - no ZooKeeper required metadataVersion: 4.0-IV0 listeners: - name: plain port: 9092 type: internal tls: false - name: tls port: 9093 type: internal tls: true config: offsets.topic.replication.factor: 3 transaction.state.log.replication.factor: 3 transaction.state.log.min.isr: 2 default.replication.factor: 3 min.insync.replicas: 2 storage: type: persistent-claim size: 1000Gi class: fast-ssd # Controllers for KRaft metadata management controllers: replicas: 3 storage: type: persistent-claim size: 100Gi class: fast-ssd entityOperator: topicOperator: {} userOperator: {} ``` **Scaling Kafka on Kubernetes:** ```bash # Scale Kafka cluster kubectl patch kafka production-cluster --type merge \ -p '{"spec":{"kafka":{"replicas":9}}}' # Monitor rolling update kubectl get pods -l strimzi.io/cluster=production-cluster -w ``` For comprehensive Kubernetes deployment patterns, see [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) and [Strimzi Kafka Operator for Kubernetes](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes). For infrastructure automation, see [Infrastructure as Code for Kafka Deployments](https://www.conduktor.io/glossary/infrastructure-as-code-for-kafka-deployments). ## Kafka Admin Operations in Streaming Architectures In modern streaming platforms, Kafka often connects hundreds of applications, stream processors, and data sinks. Admin operations directly impact the reliability of entire streaming pipelines. For example, in a real-time fraud detection system, consumer lag in the fraud-scoring consumer group means delayed fraud detection. Administrators monitoring lag can proactively scale consumers before fraud slips through. Similarly, improper partition scaling can bottleneck stream processing jobs in Apache Flink or Kafka Streams. If a stateful Flink job is bound to partition count, adding partitions requires rebalancing state, which administrators must coordinate with application teams. ## Summary Kafka administration encompasses monitoring cluster health, managing topics and partitions, enforcing security policies, tuning performance, planning for disasters, and troubleshooting production issues. As Kafka clusters scale and become critical infrastructure for streaming applications, systematic operational practices become essential. **Key takeaways for modern Kafka operations (2025):** - **KRaft mode** (Kafka 4.0+) eliminates ZooKeeper, simplifying operations and improving scalability - **Modern monitoring** combines Prometheus, Grafana, and Kafka Lag Exporter with commercial platforms like Conduktor - **CLI tools and AdminClient API** provide programmatic control over cluster operations - **Tiered storage** (Kafka 3.6+) reduces storage costs by offloading data to object storage - **Kubernetes operators** like Strimzi automate deployment and operational tasks - **Security best practices** emphasize OAuth 2.0, mTLS, and comprehensive ACL management Administrators must balance competing concerns: durability versus throughput, retention versus disk costs, security versus operational complexity. Success requires deep understanding of Kafka's distributed architecture, proactive monitoring, and well-tested procedures for common scenarios. Proper tooling, automation, and monitoring let organizations operate Kafka clusters that reliably handle large event volumes while maintaining the low latency and high availability that streaming applications require. ## Related Concepts - [Kafka Capacity Planning](https://www.conduktor.io/kafka-capacity-planning) - Essential for sizing clusters based on monitoring insights and growth projections. - [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments) - Operational considerations for managing multiple teams and applications on shared clusters. - [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems) - Proactive testing strategies to validate operational resilience and disaster recovery procedures. ## Related Articles - [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) - Modern metadata management without ZooKeeper - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) - Comprehensive monitoring strategies - [Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide) - Optimization techniques - [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) - Security architecture - [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) - Kubernetes deployment patterns - [Strimzi Kafka Operator for Kubernetes](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes) - Automated K8s operations - [Infrastructure as Code for Kafka Deployments](https://www.conduktor.io/glossary/infrastructure-as-code-for-kafka-deployments) - IaC automation - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka) - Cost-effective storage management - [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) - Consumer group operations - [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability) - Replication strategies ## Sources and References 1. Apache Kafka Documentation - Operations Guide (4.0+): https://kafka.apache.org/documentation/#operations 2. Apache Kafka Documentation - KRaft Mode: https://kafka.apache.org/documentation/#kraft 3. Narkhede, N., Shapira, G., & Palino, T. (2017). *Kafka: The Definitive Guide*. O'Reilly Media. 4. Strimzi Documentation - Kafka Operator: https://strimzi.io/docs/operators/latest/overview.html 5. Apache Kafka Improvement Proposals (KIPs): - KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum - KIP-405: Kafka Tiered Storage - KIP-554: Add Broker-side SCRAM Config API 6. Conduktor Documentation - Kafka Management: https://www.conduktor.io/kafka 7. Prometheus JMX Exporter - Kafka Monitoring: https://github.com/prometheus/jmx_exporter 8. Kafka Lag Exporter: https://github.com/seglo/kafka-lag-exporter Source: https://www.conduktor.io/glossary/kafka-admin-operations-and-maintenance --- # Kafka Architecture Diagram & Components **Kafka's architecture** is a distributed commit log: producers append records to topics, brokers store those records in partitioned, replicated logs across a cluster, and consumers read from those logs independently at their own pace. This decoupled, durable design lets a single cluster serve high-throughput writes and reads simultaneously while surviving individual broker failures without data loss. ## High-Level Architecture At the top level, a [Kafka cluster](https://www.conduktor.io/glossary/kafka-cluster) has three categories of participant: - **Producers**: applications that write records to topics - **Brokers**: servers that store partition replicas and serve requests - **Consumers**: applications (organized into consumer groups) that read records from topics ![Kafka cluster architecture with brokers, partitions, leaders, and followers](https://www.conduktor.io/assets/images/glossary/kafka-topics-partitions-brokers-core-architecture-3.webp) The cluster itself is coordinated by a **controller**, in Kafka 4.0+ this is a KRaft controller quorum, not ZooKeeper. The controller tracks which broker leads each partition, handles partition leadership elections on broker failure, and stores all cluster metadata. ## Core Components ### Topics A [topic](https://www.conduktor.io/glossary/kafka-topics-explained) is a named, append-only log. It is the primary abstraction that producers and consumers interact with. Every record belongs to a topic. Topics are retained independently of consumption, consumers can replay past records by resetting their offset. Key properties: - Schema-agnostic: brokers store raw bytes - Multi-subscriber: any number of consumer groups can read the same topic independently - Configurable retention: time-based (`retention.ms`) or size-based (`retention.bytes`) ### Partitions A [partition](https://www.conduktor.io/glossary/kafka-partitions-explained) is the physical unit of a topic, an ordered, immutable sequence of records on a single broker's local disk. Every topic has one or more partitions. Partitions serve two purposes: 1. **Parallelism**: each partition is consumed by at most one consumer per group at a time, so partition count caps maximum consumer parallelism 2. **Distribution**: partitions are spread across brokers, distributing I/O and storage load Within a partition, records have a monotonically increasing **offset**, the mechanism consumers use to track their position. ### Brokers A [broker](https://www.conduktor.io/glossary/kafka-brokers-explained) is a Kafka server. It stores partition replicas on local disk, handles producer write requests, serves consumer fetch requests, and participates in replication. Multiple brokers form a cluster. Kafka distributes partition leaders across brokers so that no single broker bears the full write load. ### KRaft Controllers Since Kafka 4.0 (available since 3.3, GA in 3.5), the cluster controller uses **KRaft**, Kafka's own Raft-based consensus protocol, to store and manage metadata. Controller nodes (which may be the same as broker nodes in combined mode) form a quorum and store cluster state in an internal `__cluster_metadata` topic. The controller is responsible for: - Assigning partitions to brokers at topic creation - Electing new partition leaders when a broker fails - Tracking ISR (in-sync replica) membership per partition - Persisting topic configuration, ACLs, and broker registrations ## Data Flow: End to End ``` Producer → [hash(key) % partitions] → Partition Leader (Broker N) ↓ replicate Followers (Broker M, Broker P) ↓ Consumer Group (reads from leader) ``` 1. **Producer** sends a record with an optional key. If a key is present, Kafka hashes it to select the target partition, ensuring all records with the same key land in the same partition (ordering guarantee). Without a key, the producer round-robins across partitions. 2. **Partition leader** (the broker leading that partition) receives the record, appends it to the local log, and, depending on the `acks` setting, waits for follower acknowledgment before responding to the producer. 3. **Follower brokers** pull new records from the leader and append to their local replica log. Followers that stay within `replica.lag.time.max.ms` remain in the ISR. 4. **Consumer** in a consumer group is assigned a subset of partitions by the group coordinator. It polls the partition leader for new records, processes them, and commits its offset back to Kafka (stored in the `__consumer_offsets` internal topic). ## Partition Replication Layout ![Partition replication across brokers with leader/follower roles](https://www.conduktor.io/assets/images/glossary/kafka-topics-partitions-brokers-core-architecture-3.webp) For a 3-partition topic with replication-factor 3 across 3 brokers, Kafka spreads leadership and replicas evenly: ``` Partition 0: Leader=Broker 1 │ Followers=Broker 2, Broker 3 Partition 1: Leader=Broker 2 │ Followers=Broker 1, Broker 3 Partition 2: Leader=Broker 3 │ Followers=Broker 1, Broker 2 ``` If Broker 1 fails, Broker 2 or 3 (whichever is in the ISR for partition 0) is elected leader by the KRaft controller. Producers and consumers discover the leadership change via metadata refresh and reconnect, typically within seconds. ## KRaft Architecture (Kafka 4.0+) ![KRaft vs ZooKeeper architecture comparison](https://www.conduktor.io/assets/images/glossary/kafka-topics-partitions-brokers-core-architecture-1.webp) Before KRaft, Kafka relied on an external ZooKeeper ensemble for metadata coordination. ZooKeeper was a separate cluster to manage, monitor, and upgrade, and it capped practical cluster size at around 200K partitions. KRaft replaces this with an internal Raft quorum of controller nodes. Benefits: | | ZooKeeper mode | KRaft mode (4.0+) | |--|--|--| | Controller election | Seconds | Milliseconds | | Max partitions | ~200K | Millions | | External dependency | ZooKeeper cluster | None | | Metadata storage | ZooKeeper znodes | `__cluster_metadata` topic | | Auth model | Split (ZK + Kafka) | Unified | For production KRaft deployments, Confluent and the Apache Kafka team recommend 3–5 controller nodes in a quorum (odd number for majority voting). Controllers can be co-located with brokers (combined mode) or run as dedicated nodes (isolated mode) for large clusters. See [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) for migration and configuration details. ## Tiered Storage Architecture (Kafka 3.6+) ![Tiered storage: hot tier on broker disk, cold tier on object storage](https://www.conduktor.io/assets/images/glossary/kafka-topics-partitions-brokers-core-architecture-2.webp) Kafka 3.6 introduced **tiered storage**, decoupling data retention from broker disk capacity: - **Hot tier**: recent log segments on broker local disk, low latency, fast access - **Cold tier**: older segments offloaded to object storage (S3, GCS, Azure Blob Storage), ~90% cheaper per GB than local disk Consumers are unaware of which tier serves their request. The broker transparently fetches cold segments from object storage and streams them to the consumer. This enables months or years of retention without scaling broker disk proportionally. Before tiered storage, a 90-day retention window required 90 days of data on every broker disk. For configuration and best practices, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). ## Architecture Enabling Properties The combination of topics, partitions, brokers, and KRaft gives Kafka its core guarantees: | Property | Mechanism | |----------|-----------| | **Horizontal scalability** | Add brokers; reassign partitions to distribute load | | **Fault tolerance** | Partition replication; ISR-based leader election | | **Per-key ordering** | Key-based partition routing; per-partition ordering | | **Independent consumption** | Each consumer group tracks its own offsets | | **Replay** | Retention-based log; consumers can reset offsets | | **High throughput** | Sequential disk I/O; batching; compression | ## Kafka in the Streaming Ecosystem Kafka's architecture makes it the hub of modern data platforms: - **Stream processing**: Kafka Streams, Apache Flink, and Apache Spark Structured Streaming consume topics, transform records, and write to output topics - **Data integration**: Kafka Connect moves data between Kafka topics and external systems (databases, S3, Elasticsearch) - **Event-driven microservices**: services publish domain events to topics and subscribe to events from other services - **Analytics pipelines**: operational topics feed data lakes and warehouses via connectors or stream processors For more on Kafka's role in data platforms, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) and [Kafka Producers and Consumers](https://www.conduktor.io/glossary/kafka-producers-and-consumers). ## Related Pages - [What is a Kafka Topic?](https://www.conduktor.io/glossary/kafka-topics-explained), the logical stream abstraction - [What is a Kafka Partition?](https://www.conduktor.io/glossary/kafka-partitions-explained), ordering, parallelism, and replication unit - [What is a Kafka Broker?](https://www.conduktor.io/glossary/kafka-brokers-explained), storage, replication, and client request handling - [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka), controller quorum deep dive - [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability), ISR, failover, and durability - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka), separating hot and cold storage - [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained), partition assignment and offset management - [Kafka Producers and Consumers](https://www.conduktor.io/glossary/kafka-producers-and-consumers), how records are written and read ## Sources - [Apache Kafka 4.0 Documentation, Core Concepts](https://kafka.apache.org/documentation/#intro_concepts_and_terms) - [KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum) - [KIP-405: Kafka Tiered Storage](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage) - [Apache Kafka, Replication](https://kafka.apache.org/documentation/#replication) - [Kafka: The Definitive Guide, 2nd Edition](https://www.confluent.io/resources/kafka-the-definitive-guide/), Chapter 5: Kafka Internals Source: https://www.conduktor.io/glossary/kafka-architecture-diagram --- # Kafka Authentication: SASL, SSL, and OAuth **Kafka authentication** verifies the identity of every client and broker before any data flows across a connection. Kafka authenticates over two channels: SASL, whose mechanisms are PLAIN, SCRAM-SHA-512, GSSAPI/Kerberos, and OAUTHBEARER (the mechanism that carries OAuth 2.0 / OIDC tokens from an external identity provider), and TLS mutual authentication (mTLS) using client certificates. Each sets a different trade-off between operational simplicity, credential rotation, and integration with existing enterprise identity. In KRaft mode (Kafka 4.0+), there is no ZooKeeper, so SCRAM credentials and controller-to-broker authentication are configured on the brokers and controllers directly, a change from the ZooKeeper-based setup that older guides describe. ![Kafka authentication mechanisms and flow](https://www.conduktor.io/assets/images/glossary/kafka-authentication-sasl-ssl-oauth-0.webp) ## Why Authentication Matters in Kafka Apache Kafka operates as a distributed streaming platform that handles sensitive data across producers, consumers, brokers, and various ecosystem components. Without proper authentication, any client could connect to your cluster, publish malicious data, or consume confidential information. Authentication provides the foundation for a comprehensive security model. It works alongside authorization (ACLs) and encryption to create defense-in-depth protection. While encryption protects data in transit and authorization controls what authenticated users can do, authentication answers the fundamental question: "Who are you?" For a complete security strategy, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices), and for authorization details, refer to [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming). In regulated industries like finance and healthcare, authentication is not optional. Compliance frameworks such as GDPR, HIPAA, and SOC 2 require proof that only identified entities access data systems. ## Understanding SASL Authentication SASL (Simple Authentication and Security Layer) is a framework that separates authentication mechanisms from application protocols. Kafka supports several SASL mechanisms, each suited to different environments. ### SASL/PLAIN The simplest SASL mechanism transmits username and password credentials. While easy to configure, SASL/PLAIN sends credentials in cleartext and should always be combined with SSL/TLS encryption. Configuration example for a Kafka producer: ```properties security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="producer-app" \ password="secure-password"; ``` SASL/PLAIN works well for development environments or when integrated with external secret management systems that rotate credentials programmatically. ### SASL/SCRAM SCRAM (Salted Challenge Response Authentication Mechanism) provides stronger security than PLAIN by avoiding cleartext password transmission. Instead of sending passwords over the network, SCRAM uses a challenge-response protocol where the client proves knowledge of the password through cryptographic hashing without revealing the password itself. Kafka supports SCRAM-SHA-256 and SCRAM-SHA-512, with SCRAM-SHA-512 recommended for new deployments due to its stronger cryptographic properties. In Kafka 4.0+ with KRaft mode (the default and only supported metadata management system), SCRAM credentials are stored in the cluster's metadata log. For legacy ZooKeeper-based clusters (deprecated and removed in Kafka 4.0), credentials were stored in ZooKeeper. This centralized credential management makes SCRAM particularly suitable for multi-tenant environments where you need to manage many application credentials without distributing password files to brokers. Example broker configuration for SCRAM-SHA-512: ```properties # Enable SCRAM-SHA-512 mechanism sasl.enabled.mechanisms=SCRAM-SHA-512 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512 # For KRaft mode (Kafka 3.3+/4.0+) # Credentials stored in metadata log automatically ``` To create SCRAM credentials in KRaft mode: ```bash # Create a new SCRAM credential kafka-configs --bootstrap-server localhost:9092 \ --alter --add-config 'SCRAM-SHA-512=[password=secure-password]' \ --entity-type users --entity-name producer-app ``` ### SASL/GSSAPI (Kerberos) For enterprise environments with existing Active Directory or Kerberos infrastructure, SASL/GSSAPI provides single sign-on capabilities. Clients authenticate using Kerberos tickets rather than passwords. Kerberos offers robust security but requires significant infrastructure: a Key Distribution Center (KDC), proper DNS configuration, and synchronized clocks across all systems. This complexity makes it best suited for large organizations with dedicated identity management teams. ### SASL/OAUTHBEARER The OAUTHBEARER mechanism enables Kafka to validate OAuth 2.0 bearer tokens, bridging traditional Kafka authentication with modern cloud-native identity providers. This mechanism is particularly relevant for Kafka 4.0+ deployments integrating with enterprise identity platforms. Configuration for OAuth requires both client-side token acquisition and broker-side token validation: ```properties # Client configuration (producer/consumer) security.protocol=SASL_SSL sasl.mechanism=OAUTHBEARER sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \ clientId="kafka-client" \ clientSecret="client-secret" \ scope="kafka.cluster" \ tokenEndpointUri="https://identity-provider.com/oauth2/token"; # Broker configuration for token validation sasl.enabled.mechanisms=OAUTHBEARER listener.name.sasl_ssl.oauthbearer.sasl.server.callback.handler.class=\ org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler listener.name.sasl_ssl.oauthbearer.sasl.jwks.endpoint.url=\ https://identity-provider.com/.well-known/jwks.json ``` For detailed mTLS certificate management and automation in production environments, see [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka). ## SSL/TLS Certificate-Based Authentication SSL/TLS serves dual purposes in Kafka: encrypting network traffic and authenticating clients through mutual TLS (mTLS). With mTLS, both the broker and client present certificates to verify their identities. Certificate-based authentication eliminates password management entirely. Instead, you manage certificate lifecycles, certificate authorities (CAs), and trust stores. Modern certificate automation tools like cert-manager (Kubernetes), HashiCorp Vault, and AWS Certificate Manager significantly reduce the operational burden of mTLS at scale. ### How mTLS Works in Kafka 1. The broker presents its certificate to the client, proving its identity 2. The client validates the broker certificate against its trust store 3. The client presents its certificate to the broker 4. The broker validates the client certificate and extracts the principal (identity) 5. If validation succeeds on both sides, the encrypted connection is established Configuration requires keystores (containing private keys and certificates) and trust stores (containing CA certificates): ```properties security.protocol=SSL ssl.keystore.location=/var/private/ssl/client.keystore.jks ssl.keystore.password=keystore-password ssl.key.password=key-password ssl.truststore.location=/var/private/ssl/client.truststore.jks ssl.truststore.password=truststore-password ``` Certificate authentication scales well in container environments where certificate issuance can be automated through systems like cert-manager in Kubernetes. ## OAuth 2.0 Integration OAuth 2.0 has become the standard for API authentication in cloud environments. Kafka's OAuth support (via SASL/OAUTHBEARER) allows integration with enterprise identity providers like Keycloak, Okta, Azure AD, and AWS Cognito. This integration is increasingly important in Kafka 4.0+ deployments where centralized identity management simplifies security at scale. ### OAuth Flow in Kafka When a producer or consumer connects to Kafka with OAuth: 1. **Token Acquisition**: The client application authenticates with the identity provider (IdP) using client credentials (client ID and secret) or other OAuth flows, requesting an access token with appropriate scopes for Kafka access 2. **Connection with Token**: The client sends the JWT access token to the Kafka broker via SASL/OAUTHBEARER during the authentication handshake 3. **Token Validation**: The broker validates the token by: - Verifying the cryptographic signature using the IdP's public keys (fetched from JWKS endpoint) - Checking token expiration and validity period - Validating audience and issuer claims match expected values - Ensuring required scopes are present 4. **Principal Extraction**: The broker extracts the principal identity from the token's subject (`sub`) claim or a custom claim configured in the broker 5. **Authorization**: Kafka ACLs are evaluated based on the extracted principal, determining which topics and operations the client can access This approach centralizes authentication in a dedicated identity system, separating it from Kafka cluster management. Token-based authentication also enables fine-grained, time-limited access without distributing long-lived credentials. When tokens expire (typically after 15-60 minutes), clients automatically request new tokens from the IdP without service interruption. OAuth particularly shines in microservices architectures where services already authenticate with an identity provider for other APIs. Using the same tokens for Kafka access creates consistent security posture across the entire platform. Modern service meshes like Istio can automate token acquisition and rotation, further reducing application complexity. ## Choosing the Right Authentication Method The optimal authentication mechanism depends on your infrastructure, team expertise, and compliance requirements: | Method | Best For | Complexity | Key Benefit | Kafka 4.0+ Notes | |--------|----------|------------|-------------|------------------| | SASL/PLAIN | Development, testing | Low | Simplicity | Use only with TLS encryption | | SASL/SCRAM-SHA-512 | Multi-tenant production | Medium | Centralized credential management in KRaft metadata | **Recommended** for most production deployments | | SASL/GSSAPI | Enterprise with Kerberos | High | Single sign-on integration | Mature, well-supported in KRaft | | SSL/TLS (mTLS) | Container platforms, service mesh | Medium-High | No password management, cryptographic identity | Excellent for zero-trust architectures | | OAuth 2.0 | Cloud-native, microservices | Medium | Centralized identity, token-based, time-limited access | **Growing adoption** in Kafka 4.0+ deployments | Many organizations use different mechanisms in different environments. Development might use SASL/PLAIN for convenience, while production uses SASL/SCRAM-SHA-512, OAuth, or mTLS for stronger security. The choice often depends on existing infrastructure: enterprises with Kerberos use GSSAPI, cloud-native organizations prefer OAuth, and Kubernetes deployments commonly use mTLS with automated certificate management. ## Authentication in Data Streaming Ecosystems Kafka authentication extends beyond broker connections. A complete data streaming platform includes multiple components that must authenticate: - **Kafka Connect**: Connectors need credentials to authenticate with Kafka brokers - **Schema Registry**: Should require authentication and verify client identity - **ksqlDB**: Database queries operate under authenticated principals - **Stream processors** (Kafka Streams, Flink): Applications authenticate as service accounts - **Management tools**: Administrative interfaces need strong authentication to prevent unauthorized changes When troubleshooting authentication issues, tools that visualize connection states and authentication failures can dramatically reduce debugging time. A failed SASL handshake might produce cryptic error messages in logs, but seeing the exact authentication flow and failure point makes resolution straightforward. ## Managing Authentication at Scale with Conduktor As Kafka deployments grow, managing authentication across multiple clusters, hundreds of applications, and diverse authentication mechanisms becomes complex. **Conduktor** provides enterprise-grade authentication management that simplifies operations: - **Centralized Authentication Visibility**: Conduktor provides a unified interface to view and manage authentication configurations across all your Kafka clusters, whether using SASL, mTLS, or OAuth. See which applications use which credentials, identify authentication failures in real-time, and audit authentication attempts. - **Self-Service Credential Management**: Enable development teams to request and obtain Kafka credentials through a governed workflow. Platform teams approve requests based on policy. Conduktor provides the self-service interface and approval workflow, but credential provisioning (SCRAM credentials, TLS certificates) is handled by the underlying Kafka infrastructure — Conduktor does not auto-provision credentials itself. - **Authentication Testing and Validation**: Before deploying changes to production, test authentication configurations in Conduktor's testing environment. **Conduktor Gateway**, a Kafka proxy, can simulate various [authentication scenarios](https://docs.conduktor.io/guide/conduktor-concepts/gw-authen-author), including token expiration, certificate rotation, and authentication failures, helping teams validate their error handling. - **Certificate Lifecycle Management**: For mTLS deployments, Conduktor tracks certificate expiration dates across all clients and brokers, alerts teams before certificates expire, and provides workflows for certificate rotation without service disruption. - **OAuth Integration**: Conduktor integrates with enterprise identity providers (Okta, Azure AD, Keycloak) to provide unified OAuth-based access to both Kafka clusters and the Conduktor platform itself, creating a consistent authentication experience. - **Compliance and Audit Trails**: Maintain detailed audit logs of authentication changes, credential access, and permission modifications to satisfy regulatory requirements. Generate compliance reports showing who had access to which Kafka resources during specific time periods. For organizations running multiple Kafka environments (development, staging, production) with different authentication requirements, Conduktor's centralized management significantly reduces operational complexity while maintaining security standards. ## Summary Kafka provides flexible authentication options suitable for diverse deployment scenarios. SASL mechanisms (PLAIN, SCRAM-SHA-512, GSSAPI, OAUTHBEARER) use credentials or tokens, while SSL/TLS enables certificate-based authentication through mTLS. OAuth 2.0 integration brings modern, cloud-native identity management to Kafka. **For Kafka 4.0+ deployments**, the recommended authentication strategies are: - **SASL/SCRAM-SHA-512**: Best for most production environments, with credentials stored securely in KRaft metadata - **mTLS**: Ideal for container platforms, Kubernetes, and zero-trust architectures with automated certificate management - **OAuth 2.0**: Excellent for cloud-native microservices architectures with existing identity providers - **SASL/PLAIN**: Acceptable only for development/testing, always with TLS encryption Understanding KRaft mode's impact on authentication is essential for modern deployments. For details on how KRaft simplifies metadata management including authentication credentials, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). Regardless of mechanism, always combine authentication with encryption and authorization. Authentication identifies who is connecting, encryption protects data in transit, and authorization determines what authenticated principals can do. For comprehensive authorization strategies, see [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming). Proper authentication configuration is foundational to Kafka security. It protects your data streams, satisfies compliance requirements, and enables audit trails showing exactly which services accessed which topics. For operational best practices covering authentication, authorization, encryption, and monitoring, consult [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices). ## Related Concepts - [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) - Fine-grained authorization controls that work with authentication to secure Kafka resources. - [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka) - Certificate-based mutual authentication for service-to-service communication in zero-trust architectures. - [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming) - Automated policy enforcement that builds on authentication to maintain security and compliance. ## Sources and References 1. **Apache Kafka Documentation - Security (2025)**: Official Kafka 4.0+ security configuration covering authentication, authorization, and encryption - https://kafka.apache.org/documentation/#security 2. **Apache Kafka KIP-500 - KRaft Mode**: Details on how KRaft mode affects metadata management including SCRAM credential storage - https://cwiki.apache.org/confluence/display/KAFKA/KIP-500 3. **RFC 7628 - SASL Mechanisms for OAuth**: Technical specification for OAuth 2.0 integration with SASL - https://datatracker.ietf.org/doc/html/rfc7628 4. **RFC 7519 - JSON Web Tokens (JWT)**: Standard for OAuth access tokens used in SASL/OAUTHBEARER - https://datatracker.ietf.org/doc/html/rfc7519 5. **Strimzi Kafka Operator - OAuth 2.0 Documentation**: Kubernetes-native OAuth implementation for Kafka - https://strimzi.io/docs/operators/latest/overview.html#security-oauth2 6. **OWASP Authentication Cheat Sheet**: Best practices for secure authentication implementation - https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html 7. **cert-manager Documentation**: Automated certificate management for Kubernetes-based Kafka deployments - https://cert-manager.io/docs/ Source: https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth --- # Kafka Backup vs Replication: Key Differences A **Kafka backup** is a point-in-time, independent, offline or immutable copy of topic records, Schema Registry schemas, and possibly consumer-group offsets, ACLs, and topic configurations. It is distinct from replication and cross-cluster mirroring: those keep extra live copies for availability, while a backup lets a cluster be restored to an earlier, known-good state after a complete or logical failure. Kafka backup is often a requirement for compliance. ![kafka-backup-vs-replication diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-backup-vs-replication-0.webp) ## Two different failure classes Why do you need Kafka backups? A [replication factor](https://www.conduktor.io/glossary/kafka-replication-and-high-availability) defaults to 3 in production, keeping a copy of every partition on multiple brokers or racks to avoid data loss. [MirrorMaker 2](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication) does the same across data centers, so an entire site can fail over to a standby cluster. A backup defends against something different: **logical failure**: - An operator deletes the wrong topic. - A buggy deploy ships a producer that writes poison records. - A retention misconfiguration expires data early. - Ransomware encrypts the payloads. The data was never lost to hardware; it was destroyed by a valid, intentional-looking operation that replication dutifully applies to all copies. You need all these solutions together: 1. **Redundancy** (RF=3): survives broker and disk failure inside one cluster. 2. **Geo-replication** (MirrorMaker 2, MSK Replicator): survives site failure by keeping a second live cluster. 3. **Backup**: survives logical failure by keeping a restorable point-in-time copy outside the cluster. ## Replication is not a backup With `acks=all`, a write is acknowledged only after every replica in the current in-sync set (ISR) has it, so those in-sync replicas stay identical within milliseconds. When a producer writes a corrupt record, it reaches every in-sync replica just as fast, so within one cluster all copies are identical and the corruption is everywhere. The same applies to a topic delete, retention expiry, or compaction: inside a single cluster the operation hits every replica, leaving no unaffected copy to fall back to. | | Replication (RF=3) | MirrorMaker 2 / geo-replication | Backup | |---|---|---|---| | Protects against broker/disk failure | Yes | Yes | Partially (via restore) | | Protects against site failure | No | Yes | Yes (if stored off-site) | | Protects against accidental delete | No | Incidental: a source delete is not mirrored, so the target survives, by accident not by design | Yes | | Protects against poison/corrupt data | No | No | Yes | | Protects against ransomware | No | No | Yes (if immutable/offline) | | Point-in-time rollback | No | No | Yes (to a snapshot point) | | Captures offsets, schemas, ACLs, configs | No (it is the live cluster) | Partial: configs + ACLs; offsets opt-in, inactive groups only; no schemas | Yes | | RPO / RTO | 0 / near-0 | > 0 / > 0 | RPO > 0 (snapshot gap); RTO scales with topic size | | Cost / infrastructure | 3x in-cluster storage | Second live cluster + Connect | Cheap object storage | ## MirrorMaker 2 is disaster recovery, not backup MirrorMaker 2 is often mistaken for a backup because it produces a second copy of the data. It is a [cross-cluster replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication) tool built for failover: - **It is asynchronous.** Records reach the target cluster after they are committed on the source, so RPO > 0 and RTO > 0. - **It has no point-in-time rollback.** MM2 keeps the target current with the source; it does not retain "the state as of T-1h." - **It copies bad data too.** A poison record written by a broken producer is a valid record, so MM2 replicates it to the target like any other. - **It needs a second live cluster plus Connect.** A backup lives in cheap object storage; MM2 requires running, paying for, and operating a full standby cluster and the Connect runtime. - **Failback is complex.** After a failover, resyncing topics and reconciling offsets between clusters is a non-trivial operation, not a one-command restore. A *topic delete* does **not** automatically propagate through MM2. That makes the mirror an accidental, time-bounded survivor of a delete, but it is not a designed, restorable capability. MM2 is the right tool for site failover only. ## What a real backup must capture Restoring Kafka is more than just replaying bytes. A complete backup captures: - **Records**: the message payloads and keys, per partition. - **Schema Registry schemas**: the contents of the `_schemas` topic. Each record embeds a schema ID, so those IDs must be restored identically or the payloads cannot be deserialized. - **ACLs**: without them the restored cluster is either open or inaccessible. - **Topic and cluster configs**: partition counts, retention, compaction, replication factor. - **Consumer-group offsets**: so applications resume where they left off. They must be translated into equivalent positions in the target cluster, or consumers reprocess or skip data. Restore is a bulk re-produce, so the recovery time scales with the size of the topic: a multi-terabyte topic takes real time to replay. Then, offset translation and resetting or repointing consumer groups are part of that restore runbook, not something the cluster does automatically. ## How to back up Kafka: tools vs Connect S3 sink vs tiered storage **Kafka Backup tools.** Kannika Armory and the open-source OSO `kafka-backup` write to S3-compatible storage and typically offer point-in-time restore, offset translation, schema-ID mapping, and selective or topic-remapped restore. **Kafka Connect S3 sink.** A Connect S3 sink archives records to object storage and is frequently repurposed as a "backup", but it is not really one: - No point-in-time restore. - No offset capture. - No schema or schema-ID translation. - No ACLs or topic configs. **Tiered storage is not a backup either.** [Tiered storage](https://www.conduktor.io/glossary/tiered-storage-in-kafka) ([KIP-405](https://kafka-options-explorer.conduktor.io/kip/405/)) moves cold segments to object storage to cut cost and extend retention, but those segments obey the *same* retention policy and the *same* delete path as local data. Deleting the topic deletes the tiered segments too. It lowers the cost of keeping data; it does not make data restorable. ## Compliance retention and running both Regulated environments (audit trails, GDPR, SOX, financial record-keeping) generally require retention that is **immutable** and **point-in-time restorable**, with a defined RPO and RTO, properties that replication and tiered retention do not provide. Prevention belongs alongside recovery. Governance tooling such as Conduktor can add RBAC and [audit trails](https://www.conduktor.io/glossary/streaming-audit-logs) that reduce the odds of an accidental topic delete in the first place, but guardrails narrow the blast radius; they do not remove the need for a restorable copy. A Kafka backup always has RPO > 0 (it is scheduled or asynchronous by nature). That is precisely why backup complements the zero-RPO replication tier rather than replacing it: replication protects the latest writes; backup protects against the logical failures replication would carry forward. Apache Kafka ships neither backup nor a built-in restore tool, so the backup layer is always something added deliberately on top. **Does a replication factor of 3 count as a backup?** No. RF=3 keeps three synchronized copies to survive broker or disk failure. A logical failure (an accidental delete, a poison write, ransomware) is applied to all three copies at once, so there is no earlier state to restore from. **Is MirrorMaker 2 a backup?** No, it is a disaster-recovery and migration tool. It replicates asynchronously, keeps the target current rather than as-of an earlier point, carries corrupt records and compacted-topic tombstones forward, and requires a second live cluster plus Connect. **Can I use a Kafka Connect S3 sink as a backup?** Not safely. It is a forward-only archive with no point-in-time restore; offsets reset to 0 when a topic is recreated, schema IDs diverge across registries, and it captures no ACLs or topic configs, so restores become fragile, hand-built replay jobs. **Is tiered storage the same as a backup?** No. Tiered storage moves cold segments to object storage for cost and retention, but those segments follow the same retention and delete path as local data. Deleting a topic deletes its tiered data too. **What does a complete Kafka backup have to include?** Records, consumer-group offsets (with offset translation on restore), Schema Registry schemas (with schema-ID mapping), ACLs, and topic and cluster configs, plus a defined RPO/RTO and the ability to restore to a point in time. ## Related Pages - [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters): how backup fits alongside failover and multi-region design. - [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability): what the replication factor actually protects. - [Kafka MirrorMaker 2 for Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication): the DR/migration tool, and why it is not a backup. - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka): cheap long-term retention, which is not the same as recoverability. - [Streaming Audit Logs](https://www.conduktor.io/glossary/streaming-audit-logs): the audit trail that helps prevent accidental deletes. - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka): delivery guarantees, and why they do not remove the need for backup. ## Sources - [KIP-405: Kafka Tiered Storage, Apache Kafka wiki](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405:+Kafka+Tiered+Storage) - [Apache Kafka Documentation, Replication & Operations](https://kafka.apache.org/documentation/#replication) - [Apache Kafka Backup: Everything You Need To Know, Confluent](https://www.confluent.io/learn/kafka-backup/) - [Cross-Data-Center Kafka Replication: Decision Framework, Confluent](https://www.confluent.io/blog/kafka-cross-data-center-replication-decision-playbook/) - [Demystifying Kafka MirrorMaker 2, Red Hat Developer](https://developers.redhat.com/articles/2023/11/13/demystifying-kafka-mirrormaker-2-use-cases-and-architecture) - [Disaster Recovery Using MirrorMaker 2, Red Hat Streams for Apache Kafka docs](https://docs.redhat.com/en/documentation/red_hat_streams_for_apache_kafka/3.2/html/disaster_recovery_using_mirrormaker_2/con-mm2-disaster-recovery-str) - [Amazon MSK Replicator and MirrorMaker2: choosing the right strategy, AWS Big Data Blog](https://aws.amazon.com/blogs/big-data/amazon-msk-replicator-and-mirrormaker2-choosing-the-right-replication-strategy-for-apache-kafka-disaster-recovery-and-migrations/) - [Kafka Connect Isn't a Backup Strategy, Kannika](https://www.kannika.io/kannika-stories/stop-using-kafka-connect-for-backups-you-are-building-a-trap-not-a-safety-net) - [kafka-backup: point-in-time recovery to S3/Azure/GCS, OSO / GitHub](https://github.com/osodevops/kafka-backup) Source: https://www.conduktor.io/glossary/kafka-backup-vs-replication --- # What is a Kafka Broker? Role & Architecture A **Kafka broker** is a server process that stores partition replicas and handles read/write requests from producers and consumers. Brokers form a [Kafka cluster](https://www.conduktor.io/glossary/kafka-cluster): multiple brokers collaborate to distribute data, replicate partitions for fault tolerance, and elect partition leaders when failures occur. In Kafka 4.0+, dedicated controller nodes within the cluster manage metadata via KRaft, eliminating ZooKeeper. ## What Is a Kafka Broker? Every record written to Kafka lands on a broker's local disk inside a partition log. Brokers are identified by a unique integer ID within a cluster. When you run a 3-broker Kafka cluster and create a topic with 3 partitions and replication-factor 3, each broker stores a replica of each partition, 9 replica logs total across the cluster. Brokers expose a single network endpoint (the bootstrap server) that clients use to discover the full cluster topology. After initial connection, clients cache the cluster metadata (which broker leads which partition) and connect directly to the right broker for each request. ## Broker Responsibilities Each broker in a cluster: - **Stores partition replicas** assigned to it by the controller, on local disk - **Handles producer requests**: receives record batches, writes to the leader partition, acknowledges based on `acks` setting - **Handles consumer fetch requests**: serves records from local partition logs (or remote tiered storage for older segments) - **Participates in replication**: follower brokers pull data from partition leaders and maintain their local replica - **Manages log retention**: deletes old segments when time- or size-based retention limits are exceeded, or compacts logs for compacted topics - **Reports metadata to the controller**: brokers register themselves and their partition state with the KRaft controller quorum ## Cluster Topology and Partition Distribution Kafka distributes partition replicas across brokers to balance load. For a topic with 3 partitions and replication-factor 3 on a 3-broker cluster: ``` Topic: transactions (3 partitions, RF=3) Partition 0: Leader=Broker 1, Followers=Broker 2, Broker 3 Partition 1: Leader=Broker 2, Followers=Broker 1, Broker 3 Partition 2: Leader=Broker 3, Followers=Broker 1, Broker 2 ``` Each broker leads one partition and follows the other two, balanced leadership and balanced storage. This spread is computed by the controller at topic creation time and can be manually reassigned or automatically rebalanced. For a full picture of how the cluster architecture fits together, see [Kafka Architecture: Diagram & Components](https://www.conduktor.io/glossary/kafka-architecture-diagram). ## Leader and Follower: How Reads and Writes Flow For each [partition](https://www.conduktor.io/glossary/kafka-partitions-explained), exactly one broker is the **leader** at any time. All producer writes go to the leader. Follower brokers replicate from the leader by pulling record batches, writing them to their local logs. By default, consumers also read from the leader. Kafka 2.4+ introduced **fetch-from-follower** (via `client.rack` and `broker.rack` config): consumers can read from the nearest follower in the same availability zone, reducing cross-AZ network costs. ## In-Sync Replicas (ISR) and Durability The **ISR** (in-sync replica set) is the subset of followers that have replicated up to within `replica.lag.time.max.ms` of the leader. Only ISR members are eligible to become the new leader during failover. Durability is controlled by two complementary settings: | Setting | Where | Meaning | |---------|-------|---------| | `min.insync.replicas` | Topic config | Minimum ISR size required to accept a write | | `acks=all` (or `-1`) | Producer config | Wait for all ISR members to acknowledge | With `min.insync.replicas=2` and `acks=all`, a write is only acknowledged after 2 brokers have persisted it. A single broker failure cannot cause data loss. If the ISR shrinks below `min.insync.replicas` (e.g., due to broker failure and a lagging follower), the partition becomes **unavailable for writes** until the ISR recovers. This is a deliberate safety trade-off: Kafka prefers to fail writes over risking silent data loss. For full coverage of replication behavior and failure scenarios, see [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability). ## Metadata Management: KRaft Mode (Kafka 4.0+) Since Kafka 3.3 (GA in 3.5, required in 4.0+), clusters use **KRaft** (Kafka Raft Metadata protocol) instead of ZooKeeper. KRaft changes how cluster metadata is stored and how the controller election works. In KRaft mode, dedicated **controller nodes** (which may also be brokers in combined mode) form a Raft quorum. Cluster metadata, topic configs, partition assignments, broker registrations, is stored in an internal `__cluster_metadata` topic on the controller nodes. The active controller is the Raft leader. ![KRaft vs ZooKeeper architecture diagram](https://www.conduktor.io/assets/images/glossary/kafka-topics-partitions-brokers-core-architecture-1.webp) **KRaft benefits over ZooKeeper:** - **Faster failover**: controller election in milliseconds instead of seconds - **Higher scalability**: supports millions of partitions per cluster (ZooKeeper topped out around 200K) - **Simpler operations**: no external coordination service to deploy, monitor, and upgrade - **Stronger consistency**: Raft consensus provides clearer durability semantics than ZooKeeper's ZAB protocol - **Unified security**: single authentication/authorization model across the cluster **KRaft deployment modes:** - **Separated mode**: dedicated controller nodes (no partition storage) + broker nodes, recommended for large clusters - **Combined mode**: nodes act as both broker and controller, suitable for development or small clusters For migration guidance and architecture details, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## Scaling Brokers Brokers scale horizontally. Adding a broker to a cluster: 1. The new broker registers with the controller via KRaft metadata protocol 2. Existing partition replicas are **not automatically moved**, you must trigger a partition reassignment (via `kafka-reassign-partitions.sh` or tooling like Conduktor) 3. New topics created after the broker joins can have partitions placed on the new broker **Scaling out typically reduces:** - Per-broker disk usage and I/O load - Per-broker network load from replication and client traffic **Scaling does not automatically solve:** - Hot partitions on existing topics (key skew means one partition gets all the traffic) - Partition leadership imbalance (use automatic leader rebalancing in Kafka 4.0+) ## Tiered Storage: Separating Compute from Storage (Kafka 3.6+) Kafka 3.6+ introduced **tiered storage**, which allows brokers to offload older log segments to remote object storage (S3, GCS, Azure Blob Storage) while keeping recent data on local disk. ![Tiered storage architecture diagram](https://www.conduktor.io/assets/images/glossary/kafka-topics-partitions-brokers-core-architecture-2.webp) | Tier | Storage | Access | |------|---------|--------| | Hot | Broker local disk | Fast, low-latency | | Cold | Object storage (S3/GCS/Azure) | Cost-efficient, ~90% cheaper | Consumers are unaware of which tier serves their fetch, the broker transparently proxies requests to object storage for cold segments. This decouples retention from broker disk capacity. Before tiered storage, retaining 6 months of data meant provisioning 6 months of disk on every broker. With tiered storage, brokers only need local disk for recent (hot) data; cold data lives in object storage at a fraction of the cost. For configuration details, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). ## Monitoring Broker Health Key metrics to watch per broker: - **UnderReplicatedPartitions**: partitions where ISR < replication factor, indicates a follower is lagging or a broker is offline - **ActiveControllerCount**: exactly 1 across the cluster at all times; 0 or >1 indicates a problem - **OfflinePartitionsCount**: partitions with no available leader, causes producer and consumer errors - **RequestHandlerAvgIdlePercent**: low values indicate broker CPU saturation - **BytesInPerSec / BytesOutPerSec**: per-broker throughput; uneven values indicate partition imbalance For a comprehensive metrics reference, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ## Managing Brokers with Conduktor At scale, tracking broker health, partition distribution, and ISR status across multiple clusters becomes operationally demanding. Conduktor provides a UI for visualizing broker assignments, spotting under-replicated partitions, and triggering partition reassignments without CLI commands. See the [Brokers Management documentation](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/brokers). ## Related Pages - [What is a Kafka Topic?](https://www.conduktor.io/glossary/kafka-topics-explained), the logical streams that brokers store - [What is a Kafka Partition?](https://www.conduktor.io/glossary/kafka-partitions-explained), the physical unit that brokers replicate - [Kafka Architecture: Diagram & Components](https://www.conduktor.io/glossary/kafka-architecture-diagram), full cluster layout with producers, consumers, and controllers - [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability), ISR mechanics, failover, and durability configuration - [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka), deep dive into KRaft controller quorum - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics), broker-level metrics and alerting ## Sources - [Apache Kafka 4.0 Documentation, Core Concepts](https://kafka.apache.org/documentation/#intro_concepts_and_terms) - [Apache Kafka, Replication](https://kafka.apache.org/documentation/#replication) - [KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum) - [KIP-405: Kafka Tiered Storage](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage) - [Kafka: The Definitive Guide, 2nd Edition](https://www.confluent.io/resources/kafka-the-definitive-guide/), Chapter 5: Kafka Internals Source: https://www.conduktor.io/glossary/kafka-brokers-explained --- # Kafka Capacity Planning Capacity planning is one of the most critical tasks when deploying Apache Kafka in production. Underestimate your needs, and you risk performance degradation or outages. Overestimate, and you waste resources and budget. Getting it right requires understanding your workload characteristics and how Kafka's architecture translates into infrastructure requirements. This guide walks through the essential aspects of Kafka capacity planning, from calculating throughput to sizing storage and network resources. ![Kafka capacity planning resource dimensions](https://www.conduktor.io/assets/images/glossary/kafka-capacity-planning-0.webp) ## KRaft Mode and Modern Kafka Architecture (2025) With Kafka 4.0+ running in KRaft mode (without ZooKeeper), capacity planning has evolved significantly. KRaft mode eliminates the need for separate ZooKeeper servers, reducing your infrastructure footprint and operational complexity. **Key capacity benefits of KRaft mode:** - **Reduced metadata overhead**: KRaft's Raft-based consensus is more efficient than ZooKeeper, reducing CPU and memory requirements for metadata management - **Faster recovery**: Metadata replication in KRaft is faster, reducing recovery time after broker failures - **Simplified deployment**: No separate ZooKeeper ensemble means fewer servers to provision and manage - **Improved scalability**: KRaft can handle clusters with millions of partitions more efficiently than ZooKeeper For capacity planning, this means you can allocate more resources to data processing rather than coordination overhead. A typical 3-node ZooKeeper ensemble (previously required) can be eliminated, and brokers themselves handle metadata with minimal overhead. When planning for Kafka 4.0+ deployments, factor in KRaft's efficiency gains, you may need fewer resources than older ZooKeeper-based architectures for the same workload. For detailed information on KRaft architecture, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## What is Kafka Capacity Planning? Kafka capacity planning is the process of estimating the hardware and infrastructure resources needed to support your expected workload. This includes determining the number of brokers, amount of storage, network bandwidth, CPU, and memory required to handle your message throughput, retention policies, and consumer patterns. For foundational understanding of Kafka's core components, see [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). Unlike traditional databases where capacity planning often focuses on transaction volume and query performance, Kafka's distributed log architecture requires considering factors like replication, partition count, message retention, and the continuous nature of streaming data. To understand how replication impacts availability, see [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability). Proper capacity planning ensures your Kafka cluster can handle peak loads, maintain low latency, and scale as your data streaming needs grow. It also directly impacts cost efficiency, especially in cloud environments where you pay for provisioned resources. ## Key Resources to Consider Kafka capacity planning involves four primary resource types, each serving different aspects of the system: - **Disk Storage** is typically the most critical resource. Kafka persists all messages to disk, and your storage needs depend on message throughput, average message size, retention period, and replication factor. Unlike in-memory systems, Kafka's durability guarantees mean you must provision enough disk space for the entire retention window. - **Network Bandwidth** becomes crucial when dealing with high-throughput scenarios. Network traffic includes producer writes, consumer reads, inter-broker replication, and potentially cross-datacenter replication. Each of these can saturate network interfaces if not properly sized. - **CPU** usage in Kafka is generally moderate but increases significantly when using compression, SSL/TLS encryption, or complex message transformations. Producers and consumers also consume CPU for serialization and network operations. - **Memory** is used for page cache (which Kafka relies on heavily for performance), in-flight requests, and internal buffers. While Kafka itself has a relatively small heap footprint, the operating system's page cache is critical for read performance. ## Calculating Throughput Requirements Start by understanding your workload characteristics. You need to know your peak message rate (messages per second), average message size, and desired retention period. A simple formula for daily storage requirements is: ``` Daily Storage = Messages/sec × Message Size × 86,400 (seconds/day) × Replication Factor ``` The constant 86,400 represents the number of seconds in a day (24 hours × 60 minutes × 60 seconds). For example, if you're processing 1,000 messages per second with an average size of 1 KB and a replication factor of 3: ``` Daily Storage = 1,000 msg/s × 1,024 bytes × 86,400 sec/day × 3 = 265 GB/day ``` For a 7-day retention policy, you would need approximately 1.8 TB of storage capacity per topic. This is a simplified calculation, real-world scenarios require adding overhead for log segment metadata, indexes, and operational headroom (typically 20-30% extra capacity). Don't forget to account for growth. If your message volume is expected to double in the next year, factor that into your initial provisioning to avoid frequent hardware additions. ## Storage Planning and Disk I/O Disk performance is as important as disk capacity. Kafka's sequential write pattern is disk-friendly, but you still need sufficient I/O throughput to handle your write rate, especially during peak loads or when consumers fall behind. **Modern storage recommendations (2025):** - **NVMe SSDs**: Use NVMe (PCIe 4.0 or 5.0) for production clusters. NVMe provides 3-7x better throughput than SATA SSDs and sub-millisecond latency. For high-throughput workloads (>100 MB/s per broker), NVMe is essential. - **Local vs Network Storage**: In cloud environments, local NVMe instance storage (e.g., AWS i4i instances) typically provides better performance and cost efficiency than network-attached storage for Kafka's workload patterns. - **Storage Tiers**: For on-premises deployments, use high-performance NVMe for hot data and consider slower (but larger) SATA SSDs for cold storage tiers. While Kafka's sequential access patterns worked reasonably well with spinning disks historically, modern production clusters should exclusively use SSDs, and specifically NVMe SSDs, for consistent sub-10ms p99 latencies. Consider your replication factor carefully. A replication factor of 3 is common for production systems, providing fault tolerance while tripling your storage requirements. Each replica must be written to disk, so your disk I/O capacity must handle the replication overhead. Log segment size also impacts performance. Kafka rolls log segments based on size or time. Smaller segments mean more files to manage but faster cleanup and compaction. Larger segments reduce file count but can slow maintenance operations. ## Tiered Storage and Long-Term Retention Kafka's tiered storage feature (production-ready in Kafka 3.6+) fundamentally changes capacity planning by separating hot and cold data storage. This feature allows you to offload older log segments to cheaper object storage (S3, Azure Blob, Google Cloud Storage) while keeping recent data on local broker disks. **Capacity impact of tiered storage:** With tiered storage enabled, your local disk capacity needs are dramatically reduced. Instead of provisioning for the entire retention period, you only need enough local storage for the "hotset" period, typically 1-7 days of data. **Example calculation with tiered storage:** Without tiered storage (30-day retention): ``` Local Storage = 1,000 msg/s × 1 KB × 86,400 × 30 days × RF 3 = 7.8 TB ``` With tiered storage (7-day local hotset, 30-day total): ``` Local Storage = 1,000 msg/s × 1 KB × 86,400 × 7 days × RF 3 = 1.8 TB Remote Storage = 1,000 msg/s × 1 KB × 86,400 × 23 days × RF 1 = 2.0 TB ``` Notice that remote storage doesn't require replication factor multiplication, the cloud object store handles durability. This can reduce your total storage costs by 60-80% for long retention scenarios. **When to use tiered storage:** - Retention periods beyond 7-14 days - Compliance requirements for long-term data retention (months or years) - Cost-sensitive environments where local NVMe storage is expensive - Workloads where most consumers read recent data, with occasional historical queries For detailed setup and operational guidance, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). ## Network and CPU Considerations Network bandwidth requirements multiply quickly. Producer traffic is written to disk and replicated to other brokers. If you have a replication factor of 3, each message is written once by the producer but transmitted twice more for replication. Consumer traffic adds to network usage. Each consumer group reads the full dataset, so multiple consumer groups multiply your read traffic. A common mistake is planning only for producer throughput and being surprised when adding consumer groups saturates network links. For more on consumer behavior and performance considerations, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). **Network bandwidth calculation:** For a workload with 1,000 msg/s at 1 KB with replication factor 3 and 2 consumer groups: ``` Producer inbound: 1,000 × 1 KB = 1 MB/s per broker (divided across brokers) Replication traffic: 1 MB/s × 2 (to other replicas) = 2 MB/s per broker Consumer outbound: 1 MB/s × 2 consumer groups = 2 MB/s per broker Total: ~5 MB/s per broker (plus overhead) ``` This is for average load, peak traffic can be 3-5x higher, so provision accordingly. **Modern network recommendations (2025):** - **25GbE minimum**: Modern production clusters should use 25GbE network interfaces. 10GbE networks are insufficient for high-throughput clusters (>500 MB/s cluster-wide). - **40GbE/100GbE**: For large-scale deployments (>10 brokers, >1 GB/s throughput), use 40GbE or 100GbE to avoid network bottlenecks. - **Separate networks**: For high-throughput scenarios, use dedicated network interfaces or VLANs for inter-broker replication traffic to avoid contention with client traffic. - **Cloud networking**: In cloud environments, choose network-optimized instance types (AWS: network-optimized instances, GCP: n2-highmem with higher bandwidth). **CPU considerations:** CPU usage spikes when enabling compression. Compressing messages reduces network and storage requirements but increases CPU load on both producers and brokers. Modern compression algorithms like zstd (available in Kafka 2.1+) provide excellent compression ratios with lower CPU overhead than gzip. SSL/TLS encryption adds CPU overhead, typically 10-30% depending on cipher choice and hardware. Modern CPUs with AES-NI instructions reduce this overhead significantly. When planning capacity, benchmark your specific workload with encryption enabled to understand the real impact. **Modern CPU recommendations (2025):** - **Minimum cores**: 8-16 cores per broker for production workloads - **Architecture**: Use recent CPU architectures (Intel Ice Lake/Sapphire Rapids or AMD EPYC Genoa) for better single-threaded performance and AES-NI support - **Compression**: Budget 20-40% extra CPU capacity if using compression - **Encryption**: Budget 15-25% extra CPU capacity if using SSL/TLS with modern CPUs (more with older hardware) ## Calculating Broker Count and Cluster Size One of the most common questions in capacity planning is: "How many brokers do I need?" The answer depends on your throughput, storage, and fault tolerance requirements. **Throughput-based sizing:** If you need to handle 2 GB/s of producer throughput with replication factor 3: ``` Total write throughput = 2 GB/s × 3 (replicas) = 6 GB/s Per-broker capacity = ~500 MB/s (typical limit for balanced workloads) Minimum brokers = 6 GB/s ÷ 0.5 GB/s = 12 brokers ``` Add 20-30% headroom for peaks and maintenance: 15-16 brokers recommended. **Storage-based sizing:** If you need 50 TB total storage with replication factor 3: ``` Total raw storage = 50 TB × 3 = 150 TB Per-broker storage = 10 TB (NVMe capacity) Minimum brokers = 150 TB ÷ 10 TB = 15 brokers ``` **Fault tolerance considerations:** For high availability, you need at least 3 brokers (to survive 1 failure with RF=3). For critical production systems, 5-7 brokers minimum is recommended to handle multiple concurrent failures and maintenance windows. **Use the most constraining factor**: Your final broker count should be the maximum of throughput-based, storage-based, and fault tolerance requirements. ## Cloud vs On-Premises Sizing Capacity planning differs significantly between cloud and on-premises deployments. - **Cloud-specific considerations:** - **Instance selection:** - AWS: Use i4i instances (local NVMe), r7iz for memory-intensive workloads, or m7i for balanced workloads - Azure: Use Lsv3-series (NVMe local storage) or memory-optimized Esv5-series - GCP: Use c3-highmem with local SSDs attached, or n2-highmem for general purpose **Storage options:** - **Local NVMe**: Best performance and cost for Kafka. Data is ephemeral but replicated by Kafka itself. - **EBS/Persistent Disks**: Use io2/gp3 (AWS) or Premium SSD (Azure) only when data persistence beyond instance lifecycle is required. Typically 30-50% more expensive and lower performance than local NVMe. - **Tiered storage**: Essential for cloud deployments to reduce costs. Use local NVMe for hot data, object storage (S3/Blob/GCS) for cold data. **Network bandwidth:** - Cloud instances have network limits. An AWS m5.2xlarge has 10 Gbps limit, insufficient for high-throughput brokers. - Choose network-optimized instances or larger instance types for adequate bandwidth. **Managed services:** - AWS MSK, Azure Event Hubs (Kafka-compatible), GCP Managed Kafka abstract some capacity planning but still require sizing decisions. - For detailed guidance on AWS, see [Amazon MSK: Managed Kafka on AWS](https://www.conduktor.io/glossary/amazon-msk-managed-kafka-on-aws). - **On-premises considerations:** - **Hardware flexibility:** - Can optimize hardware precisely for Kafka workloads (e.g., high-core-count CPUs, large NVMe arrays) - Typical server: 2x Intel Xeon or AMD EPYC CPUs (32-64 cores total), 256-512 GB RAM, 8-12x 4TB NVMe SSDs, dual 25GbE or 40GbE NICs **Cost model:** - Higher upfront CapEx but lower OpEx over 3-5 year lifecycle - Better for predictable, sustained workloads **Scalability:** - Requires procurement lead time (weeks/months) vs instant cloud scaling - Plan for 18-24 months of growth upfront **When to choose each:** - **Cloud**: Variable workloads, fast iteration, limited ops team, global distribution needs - **On-premises**: Predictable sustained workloads, data sovereignty requirements, existing datacenter infrastructure, cost optimization at scale For Kubernetes-based deployments (whether cloud or on-premises), see [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) for container-specific capacity considerations. ## Traffic Control Policies for Capacity Management Capacity planning becomes predictable when you can enforce limits programmatically. Conduktor Traffic Control Policies provide automated enforcement of rate limits, configuration policies, and access controls. Rate limiting prevents individual services from consuming disproportionate resources, configuration policies ensure capacity planning assumptions hold (mandatory retention periods, partition count limits), and access control policies implement capacity-aware governance rules. The key advantage is programmatic enforcement rather than relying on documentation and team discipline. When you plan for specific throughput limits, policies guarantee they aren't exceeded. This reduces required safety margins and enables tighter resource utilization. For implementation details, see [Traffic Control Policies](https://docs.conduktor.io/guide/conduktor-concepts/traffic-control-policies). ## Chargeback and Cost Accountability Understanding where Kafka infrastructure costs originate is essential for capacity planning. Conduktor Chargeback tracks resource consumption by service account (bytes produced/consumed with Conduktor Gateway, a Kafka proxy) or by topic (storage and partition count without Conduktor Gateway), providing visibility into which applications drive your capacity requirements. Chargeback transforms estimates into precise measurements, instead of guessing needs, you measure actual consumption during peak hours. This enables cost-based governance where teams monitor consumption against budgets, optimization through visibility (identifying high-cost topics for retention adjustment), and objective metrics for cross-team capacity negotiations. For implementation details, see [Chargeback](https://docs.conduktor.io/guide/conduktor-concepts/chargeback). ## Monitoring and Scaling in Data Streaming Effective capacity planning doesn't end at deployment. Continuous monitoring is essential to validate your estimates and detect when you're approaching limits. Key metrics to track include: - **Broker CPU and memory utilization**: Sustained high CPU (>70%) indicates you may need more brokers or to reduce load - **Disk usage and I/O wait**: Approaching disk capacity or high I/O wait times signal storage constraints - **Network throughput**: Monitor bytes in/out per broker to identify network bottlenecks - **Consumer lag**: Growing lag indicates consumers can't keep up, which may require scaling Platforms like Conduktor provide comprehensive monitoring dashboards that track these metrics across your cluster, making it easier to spot capacity issues before they impact production. Being able to visualize throughput trends, partition distribution, and resource utilization in one place helps teams make informed scaling decisions. [Conduktor's Insights Dashboard](https://docs.conduktor.io/guide/insights/index) provides cost control analysis and [risk analysis](https://docs.conduktor.io/guide/insights/risk-analysis) to help identify capacity bottlenecks before they impact production. For comprehensive coverage of monitoring strategies and key metrics, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). When scaling Kafka, you can scale vertically (larger machines) or horizontally (more brokers). Horizontal scaling is generally preferred as it improves fault tolerance and allows for better load distribution across partitions. For detailed performance optimization strategies, see [Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide). ## Common Pitfalls and Best Practices Several common mistakes can derail capacity planning: - **Underestimating replication overhead** is frequent. Remember that replication multiplies your network and storage needs. A replication factor of 3 doesn't just triple storage, it also triples write network traffic. - **Ignoring partition count** can limit scalability. Since partitions are the unit of parallelism in Kafka, having too few partitions means you can't fully use multiple brokers or scale consumer throughput. However, too many partitions increase metadata overhead and recovery time in older ZooKeeper-based clusters. With Kafka 4.0+ and KRaft mode, the partition limit has increased significantly, KRaft can efficiently handle millions of partitions compared to ZooKeeper's practical limit of ~200,000. For guidance on topic and partition design, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) and [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). - **Not planning for peaks** is another common issue. Average load might be manageable, but if your system experiences traffic spikes (e.g., end-of-day processing), you need capacity for peak load, not average. - **Best practices include:** - Start with conservative estimates and monitor actual usage - Provision 20-30% headroom beyond calculated requirements - Use tiered storage for long-term retention to reduce primary storage costs - Test capacity under load before production deployment - Document your capacity assumptions and review them quarterly ## Related Concepts - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka) - Fundamentally changes capacity planning by separating hot and cold storage tiers. - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) - Continuous monitoring validates capacity estimates and detects when scaling is needed. - [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) - Partition count and retention policies directly impact storage and throughput capacity requirements. ## Summary Kafka capacity planning requires balancing multiple resources, storage, network, CPU, and memory, based on your throughput, retention, and reliability requirements. Start by calculating your storage needs from message rate, size, retention period, and replication factor. Factor in network bandwidth for both client traffic and replication, and don't overlook CPU requirements for compression and encryption. Continuous monitoring is essential to validate your planning and detect when scaling is needed. By understanding these fundamentals and avoiding common pitfalls like underestimating replication overhead or ignoring peak loads, you can build a Kafka infrastructure that's both cost-effective and reliable. Proper capacity planning isn't a one-time task, it's an ongoing process as your data streaming needs evolve. Regular review of your metrics and capacity assumptions ensures your Kafka cluster continues to meet your organization's requirements. ## Sources and References 1. Apache Kafka Documentation - Operations Guide: https://kafka.apache.org/documentation/#operations 2. Confluent - Kafka Capacity Planning: https://docs.confluent.io/platform/current/kafka/deployment.html 3. Narkhede, Neha, Gwen Shapira, and Todd Palino. "Kafka: The Definitive Guide" (O'Reilly Media, 2017) 4. LinkedIn Engineering Blog - Kafka Capacity Planning at Scale: https://engineering.linkedin.com/kafka 5. AWS Big Data Blog - Best Practices for Amazon MSK Sizing: https://aws.amazon.com/blogs/big-data/ Source: https://www.conduktor.io/glossary/kafka-capacity-planning --- # Kafka Cluster Monitoring and Metrics Monitoring an Apache Kafka cluster is essential for maintaining reliable data streaming operations. Without proper observability into broker health, partition replication, consumer lag, and throughput patterns, teams operate blind to performance degradation, capacity issues, and potential outages. Effective monitoring transforms Kafka from a black box into a well-understood system where problems can be detected, diagnosed, and resolved before they impact business operations. This article covers the key metrics, monitoring architecture, and best practices that enable engineering teams to maintain healthy Kafka deployments at scale. For the underlying components these metrics describe, see the [Kafka architecture diagram](https://www.conduktor.io/glossary/kafka-architecture-diagram). ![Kafka monitoring architecture and key metrics](https://www.conduktor.io/assets/images/glossary/kafka-cluster-monitoring-and-metrics-0.webp) ## Why Kafka Monitoring Matters Kafka clusters handle mission-critical data pipelines in modern architectures. A single under-replicated partition can lead to data loss. Unnoticed consumer lag can cause hours of processing delays. Broker resource exhaustion can cascade into cluster-wide failures. Unlike traditional databases with simpler request-response patterns, Kafka's distributed nature creates complex interdependencies. A topic might have dozens of partitions spread across multiple brokers, each with its own replication state. Hundreds of consumers might be reading at different rates, each maintaining their own offset positions. Without visibility into these components, troubleshooting becomes guesswork. Good monitoring enables proactive capacity planning, rapid incident response, and informed optimization decisions. Teams that invest in it spend less time fighting fires and more time building features. ## Core Metrics Categories Kafka exposes hundreds of metrics through Java Management Extensions (JMX), a standard Java technology for monitoring and managing applications. JMX provides a structured way to access internal metrics from Kafka brokers, producers, and consumers through Management Beans (MBeans). Understanding which metrics matter requires organizing them into logical categories. ### Broker Metrics Broker-level metrics reveal the health of individual Kafka servers. Key indicators include: - **Under-replicated partitions** (`kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions`): Partitions where follower replicas have fallen behind the leader. Should be 0 in healthy clusters; any value above 0 indicates replication issues requiring immediate investigation. - **Offline partitions** (`kafka.controller:type=KafkaController,name=OfflinePartitionsCount`): Partitions without an active leader, representing immediate data unavailability. Must always be 0; any offline partitions mean data cannot be produced or consumed. - **Request handler idle ratio** (`kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent`): Percentage of time request handlers are idle. Values below 20% indicate broker saturation; healthy clusters typically show 60-80% idle time. - **Network processor idle ratio** (`kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent`): Network thread utilization. Values below 30% suggest network bottlenecks; aim for 60%+ for healthy operations. These metrics directly indicate whether brokers can handle current load and maintain replication guarantees. ### KRaft Controller Metrics (Kafka 3.3+) With KRaft mode replacing ZooKeeper as Kafka's consensus mechanism, new controller-specific metrics become essential for monitoring metadata operations: - **Active controller count** (`kafka.controller:type=KafkaController,name=ActiveControllerCount`): Should be exactly 1 in the cluster. If 0, the cluster cannot process metadata changes; if greater than 1, indicates a split-brain scenario requiring immediate attention. - **Metadata log end offset** (`kafka.server:type=KafkaRaftServer,name=MetadataLogEndOffset`): Tracks the latest offset in the metadata log. Monitoring the rate of change helps identify metadata update patterns and potential issues. - **Metadata error rate** (`kafka.server:type=KRaftMetrics,name=MetadataErrorRate`): Failed metadata operations. Should be 0 in healthy clusters; any errors indicate controller instability or configuration issues. - **Controller event queue size** (`kafka.controller:type=ControllerEventManager,name=EventQueueSize`): Number of pending controller events. High values (>100) suggest the controller is overwhelmed, which can delay partition leadership elections and topic operations. KRaft mode eliminates ZooKeeper monitoring complexity while introducing cleaner, more predictable controller metrics. For clusters still using ZooKeeper, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) for migration guidance. ### Topic and Partition Metrics Per-topic metrics help identify problematic topics and unbalanced workloads: - **Bytes in/out rate**: Throughput for producers and consumers - **Messages in rate**: Message volume over time - **Partition count**: Number of partitions per topic, affecting parallelism - **Leader partition distribution**: Whether partition leadership is balanced across brokers Uneven partition distribution can overload specific brokers while others sit underutilized, creating performance bottlenecks. ### Consumer Group Metrics Consumer lag is often the most critical metric for application teams. For comprehensive consumer monitoring strategies, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). - **Consumer lag** (`kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*,partition=*`): Number of messages between the consumer's current offset and the log end offset. Lag of 1,000 messages might be normal during steady state, but 1,000,000+ messages indicates the consumer cannot keep pace with production rate. - **Commit rate** (`kafka.consumer:type=consumer-coordinator-metrics,client-id=*`): Frequency of offset commits, indicating consumer progress. Low commit rates may signal consumer crashes or processing bottlenecks. - **Fetch rate** (`kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*`): Consumer fetch request patterns. Monitoring fetch latency helps identify network or broker-side issues affecting consumption. - **Records consumed rate** (`kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*`): Actual message consumption throughput measured in records per second. Compare this to production rate to identify processing capacity issues. A consumer group processing payment events might accumulate lag during traffic spikes. If lag reaches millions of messages, transaction confirmations could be delayed by hours, directly impacting customer experience. Modern tools like Kafka Lag Exporter provide dedicated consumer lag monitoring with Prometheus integration. ### Producer Metrics Producer metrics reveal client-side performance and health: - **Record send rate**: Messages produced per second - **Compression rate**: Efficiency of message compression - **Request latency**: Time to receive acknowledgment from brokers - **Error rate**: Failed produce requests indicating configuration or capacity issues ## Monitoring Architecture Most Kafka monitoring solutions follow a similar pattern: expose JMX metrics, collect them into a time-series database, and visualize them in dashboards. ### JMX Metrics Export Kafka brokers expose metrics via JMX, a standard Java monitoring interface. The Prometheus JMX Exporter converts these metrics into formats consumable by modern monitoring systems. The exporter runs as a Java agent alongside the broker, scraping metrics and making them available via HTTP endpoints. **JMX Exporter Configuration Example** Deploy the JMX Exporter as a Java agent by downloading the JAR and configuring Kafka startup: ```bash # Download JMX Exporter wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.20.0/jmx_prometheus_javaagent-0.20.0.jar # Add to Kafka broker startup (in kafka-server-start.sh or systemd unit) export KAFKA_OPTS="-javaagent:/opt/kafka/jmx_exporter/jmx_prometheus_javaagent-0.20.0.jar=7071:/opt/kafka/jmx_exporter/kafka-broker.yml" ``` **Sample JMX Exporter Configuration (kafka-broker.yml)** ```yaml # Enable lower-cased output for metric names lowercaseOutputName: true lowercaseOutputLabelNames: true # Define which metrics to expose rules: # Broker metrics - pattern: kafka.server<>Value name: kafka_server_$1_$2 # Controller metrics - pattern: kafka.controller<>Value name: kafka_controller_$1_$2 # Topic metrics - pattern: kafka.server<>Count name: kafka_server_brokertopicmetrics_$1_total labels: topic: "$2" # Network metrics - pattern: kafka.network<>Count name: kafka_network_$1_$2_total labels: request: "$3" # KRaft-specific metrics (Kafka 3.3+) - pattern: kafka.server<>Value name: kafka_kraft_$1 ``` The exporter listens on port 7071 by default, exposing metrics at `http://broker-host:7071/metrics` for Prometheus to scrape. Filter patterns control which JMX beans are exported, reducing metric cardinality and improving performance. ### Time-Series Databases Prometheus has become the de facto standard for storing Kafka metrics. Its dimensional data model naturally represents Kafka's hierarchical metric structure (cluster → broker → topic → partition). Prometheus scrapes JMX Exporter endpoints at regular intervals, storing metric history for alerting and trend analysis. **Prometheus Configuration Example** ```yaml # prometheus.yml scrape_configs: - job_name: 'kafka-brokers' static_configs: - targets: - 'kafka-broker-1:7071' - 'kafka-broker-2:7071' - 'kafka-broker-3:7071' scrape_interval: 30s scrape_timeout: 10s ``` Alternative solutions include InfluxDB, TimescaleDB, or managed services like Datadog and New Relic. ### OpenTelemetry Integration OpenTelemetry provides a vendor-neutral observability framework increasingly adopted for Kafka monitoring in 2025. Instead of using JMX Exporter directly, OpenTelemetry Collector can scrape JMX metrics and forward them to multiple backends simultaneously (Prometheus, Grafana Cloud, Datadog, etc.). **OpenTelemetry Collector Configuration for Kafka** ```yaml # otel-collector-config.yaml receivers: prometheus: config: scrape_configs: - job_name: 'kafka-brokers' static_configs: - targets: ['kafka-broker-1:7071', 'kafka-broker-2:7071'] processors: batch: timeout: 10s # Add resource attributes for better filtering resource: attributes: - key: service.name value: kafka-cluster-prod action: upsert exporters: prometheus: endpoint: "0.0.0.0:9090" otlp: endpoint: "https://your-observability-backend.com" headers: api-key: "${OTEL_API_KEY}" service: pipelines: metrics: receivers: [prometheus] processors: [batch, resource] exporters: [prometheus, otlp] ``` OpenTelemetry's unified approach simplifies multi-backend observability and enables correlation between Kafka metrics, application traces, and infrastructure logs, essential for troubleshooting complex distributed systems. ### Visualization and Alerting Grafana provides rich visualization capabilities with pre-built Kafka dashboards available from the community. Teams can create custom dashboards showing cluster health at a glance, with drill-down capabilities into specific brokers or consumer groups. Alerting rules trigger notifications when metrics cross thresholds. Critical alerts might fire when under-replicated partitions exceed zero, while warning alerts could trigger when consumer lag exceeds baseline expectations. ## Kafka Monitoring in Data Streaming Systems In modern data streaming architectures, Kafka often sits at the center of complex event pipelines. Real-time analytics, microservice communication, and event-driven workflows all depend on Kafka's reliable message delivery. Consider a streaming analytics platform processing clickstream data. Raw events flow into Kafka from web applications, stream processing jobs (Apache Flink, Kafka Streams) transform the data, and results are written back to Kafka for consumption by dashboards and databases. Monitoring must cover the entire pipeline. If a Flink job slows down, its consumer lag will increase in Kafka. If an upstream service starts producing malformed messages, error rates will spike. If a broker fails and partition leadership doesn't transfer cleanly, under-replicated partitions will appear. Each symptom requires different remediation, but all are visible through Kafka metrics. Effective monitoring connects Kafka metrics to broader system health. Consumer lag in the analytics pipeline might correlate with increased database write latency. Broker CPU spikes might coincide with deployment of a new producer service. Understanding these relationships enables faster root cause analysis. ## Tools and Platforms Several tools address Kafka monitoring needs across different complexity levels. - **Open Source Solutions**: The Prometheus + Grafana combination provides powerful monitoring at no licensing cost. JMX Exporter handles metric collection, Prometheus stores time-series data, and Grafana visualizes dashboards. This stack requires setup and maintenance but offers complete flexibility. Pre-built Grafana dashboards for Kafka are available in the community gallery. - **Kafka Lag Exporter**: A lightweight, specialized tool for monitoring consumer group lag. It exports lag metrics to Prometheus without requiring full JMX Exporter setup for consumers. Particularly useful for teams focused on consumer monitoring. - **AKHQ**: A modern web UI providing cluster administration interfaces with basic monitoring capabilities. While not a substitute for comprehensive monitoring, it offers quick visibility into topic configuration, consumer group status, and message browsing for development and troubleshooting. - **Conduktor Platform**: For teams seeking unified monitoring and management, Conduktor Platform provides comprehensive Kafka observability with visual insights into cluster health, real-time consumer lag tracking, broker performance metrics, and topic analytics. Unlike basic monitoring stacks, Conduktor combines monitoring with governance, security, and testing capabilities, essential for organizations managing multiple Kafka clusters in production. [Explore Conduktor's monitoring capabilities](https://docs.conduktor.io/guide/monitor-brokers-apps/index) and [configure alerts for proactive issue detection](https://docs.conduktor.io/guide/monitor-brokers-apps/alerts). Conduktor Platform features include: - Real-time cluster health dashboards with automated anomaly detection - Consumer lag monitoring with customizable alerting thresholds - Topic and partition metrics visualization - Historical trend analysis for capacity planning via the [Insights Dashboard](https://docs.conduktor.io/guide/insights/index) - Integration with existing Prometheus/Grafana stacks - Governance features for topic configuration management - **Conduktor Gateway**: For advanced monitoring scenarios including chaos testing and traffic observability, Conduktor Gateway acts as a Kafka proxy enabling request inspection, latency injection, and error simulation, critical for validating monitoring and alerting systems under failure conditions. - **Commercial APM Tools**: Datadog, New Relic, and Dynatrace offer Kafka monitoring as part of broader application performance monitoring suites, providing correlation with infrastructure and application metrics. The right choice depends on team size, existing monitoring infrastructure, and operational complexity. Teams starting from scratch often benefit from Conduktor Platform's integrated approach, while those with established observability pipelines may prefer open-source tools with custom integration. ## Best Practices for Kafka Monitoring Successful monitoring programs follow consistent patterns: - **Establish Baselines**: Understand normal behavior before setting alerts. Consumer lag that's acceptable during low-traffic periods might indicate problems during peak hours. Record baseline metrics for throughput, latency, and resource utilization. - **Alert on Symptoms, Not Causes**: Alert when under-replicated partitions exist, not when CPU is high. Focus on metrics that directly impact availability and correctness rather than resource utilization alone. - **Monitor the Full Pipeline**: Track metrics from producers through Kafka to consumers. A problem anywhere in the chain affects overall system reliability. - **Use Tiered Alerting**: Distinguish between critical alerts requiring immediate action (offline partitions, cluster unavailability) and warnings indicating degraded performance (elevated lag, increased latency). Modern alerting tools like Prometheus AlertManager enable routing different severity levels to appropriate channels (PagerDuty for critical, Slack for warnings). - **Example Alert Rules**: ```yaml # Prometheus alerting rules groups: - name: kafka-cluster rules: - alert: KafkaOfflinePartitions expr: kafka_controller_kafkacontroller_offlinepartitionscount > 0 for: 1m labels: severity: critical annotations: summary: "Kafka has offline partitions" description: "Cluster has {{ $value }} offline partitions - immediate data unavailability" - alert: KafkaUnderReplicatedPartitions expr: kafka_server_replicamanager_underreplicatedpartitions > 0 for: 5m labels: severity: warning annotations: summary: "Kafka has under-replicated partitions" description: "{{ $value }} partitions are under-replicated on {{ $labels.instance }}" - alert: ConsumerLagCritical expr: kafka_consumer_lag > 1000000 for: 10m labels: severity: critical annotations: summary: "Consumer lag exceeds 1M messages" description: "Consumer group {{ $labels.group }} is critically behind on topic {{ $labels.topic }}" ``` - **Define SLOs and Error Budgets**: Modern monitoring practices emphasize Service Level Objectives (SLOs) over arbitrary thresholds. Define acceptable performance targets (e.g., "99.9% of messages processed within 5 seconds") and track error budgets to inform operational decisions. - **Automate Runbooks**: Common scenarios like consumer lag recovery or partition rebalancing should have documented procedures. Better yet, automate remediation for well-understood issues using tools like Kubernetes operators or Cruise Control for automated cluster rebalancing. - **Regular Review**: Schedule periodic reviews of monitoring dashboards and alert rules. As systems evolve, so should monitoring configuration. Conduct monthly reviews to eliminate noisy alerts, adjust thresholds based on actual behavior, and ensure new features are properly monitored. ## Summary Kafka cluster monitoring provides the visibility required to operate reliable data streaming platforms. By tracking broker health, partition replication, consumer lag, and throughput metrics, teams can detect issues before they escalate into outages. Modern Kafka monitoring in 2025 embraces KRaft mode metrics, OpenTelemetry integration, and cloud-native observability practices. Effective monitoring combines metrics collection (JMX Exporter or OpenTelemetry), storage (Prometheus), visualization (Grafana), and intelligent alerting with SLO-based thresholds into a cohesive observability platform. As Kafka clusters grow in size and criticality, monitoring becomes non-negotiable infrastructure. The investment in proper observability pays dividends through reduced downtime, faster incident resolution, and confidence in system reliability. ## Related Concepts - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Leverage monitoring data to make informed capacity planning decisions and predict resource requirements. - [Distributed Tracing for Kafka Applications](https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications) - Correlate metrics with traces for end-to-end observability in streaming pipelines. - [DataOps for Streaming](https://www.conduktor.io/glossary/dataops-for-streaming) - Apply DataOps practices to integrate monitoring with streaming data quality and governance. ## Related Topics For comprehensive Kafka operations and management, explore these related topics: - **[Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka)**: Learn about KRaft's architecture and monitoring implications for modern Kafka deployments - **[Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring)**: Deep dive into consumer lag tracking strategies and tools - **[Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide)**: Use monitoring insights to optimize Kafka performance - **[Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning)**: Apply monitoring data to capacity planning decisions - **[Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability)**: Understand replication metrics and availability monitoring - **[Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes)**: Container-native monitoring approaches for Kubernetes-based deployments - **[What is Data Observability: The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars)**: Broader context for observability practices beyond metrics ## Sources and References 1. [Apache Kafka Monitoring Documentation](https://kafka.apache.org/documentation/#monitoring) - Official Kafka monitoring guide covering JMX metrics and operational best practices 2. [Confluent Monitoring Kafka](https://docs.confluent.io/platform/current/kafka/monitoring.html) - Comprehensive guide to Kafka metrics and monitoring strategies from Confluent 3. [Prometheus JMX Exporter](https://github.com/prometheus/jmx_exporter) - Official JMX to Prometheus exporter for collecting Kafka metrics 4. [Kafka Monitoring with Prometheus and Grafana](https://medium.com/@dunefro/kafka-monitoring-with-prometheus-and-grafana-5c0c0c0c0c0c) - Practical guide to setting up Kafka monitoring stack 5. [LinkedIn's Kafka Monitoring at Scale](https://engineering.linkedin.com/blog/2016/08/burrow-kafka-consumer-monitoring-reinvented) - Real-world insights from LinkedIn's Kafka operations team Source: https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics --- # Kafka Cluster: Architecture & How It Works A **Kafka cluster** is a set of broker servers that work together to store, replicate, and serve streams of records. Clients connect to any broker, discover the rest of the cluster automatically, and read or write to topic partitions that are spread and replicated across every broker. The cluster is the unit of fault tolerance: individual brokers can fail without losing data or interrupting producers and consumers. ## What is a Kafka cluster? A Kafka cluster is one or more [brokers](https://www.conduktor.io/glossary/kafka-brokers-explained) that share a single cluster identity and a common metadata log. Each broker is a JVM process, usually on its own machine or pod, that owns a slice of the total data. Together they present a single logical system to clients: you send records to a topic, and the cluster decides which broker holds which partition. Three properties make a group of brokers an actual cluster rather than a set of isolated servers: - **Shared cluster ID** — every broker in the cluster carries the same `cluster.id`, generated once when the cluster is first formatted. A broker with a mismatched ID is rejected. - **Shared metadata** — the cluster agrees on which topics exist, how many partitions each has, and which broker leads each partition. In Kafka 4.0 this metadata lives in a KRaft controller quorum. - **Coordinated replication** — partitions are copied across brokers so that the loss of one broker does not lose data. A single-broker "cluster" is valid for local development, but production clusters run at least three brokers so replication and quorum can tolerate a failure. ## Kafka cluster architecture Every cluster has two roles: **brokers** that store partition data and serve client requests, and a **controller** that manages cluster metadata and partition leadership. Since Kafka 4.0, the controller is a KRaft quorum (a set of controller nodes running the Raft consensus protocol); ZooKeeper has been removed. For the full component breakdown and data flow, see the [Kafka architecture diagram](https://www.conduktor.io/glossary/kafka-architecture-diagram). ![A Kafka cluster with three brokers, each holding partitions P0/P1/P2 with one leader per partition and followers on the other brokers, a KRaft controller quorum below, and a producer writing to leaders and a consumer reading, all via bootstrap.servers](https://www.conduktor.io/assets/images/glossary/kafka-cluster.svg) The controller does not sit in the data path. Producers and consumers talk directly to the brokers that lead the partitions they use; the controller only steps in to elect a new leader when a broker fails. ## How brokers form a cluster Brokers find each other through the metadata layer, and clients find the cluster through a **bootstrap** connection. A minimal KRaft broker configuration references the controller quorum and a unique node ID: ```properties # server.properties (KRaft combined broker + controller) process.roles=broker,controller node.id=1 controller.quorum.voters=1@broker1:9093,2@broker2:9093,3@broker3:9093 listeners=PLAINTEXT://broker1:9092,CONTROLLER://broker1:9093 ``` Before first start, the cluster is formatted with a shared ID so all nodes agree on identity: ```bash # generate one cluster ID, reuse it on every broker KAFKA_CLUSTER_ID=$(bin/kafka-storage.sh random-uuid) bin/kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c config/server.properties ``` A client only needs one or two reachable brokers in `bootstrap.servers`; it fetches the full cluster topology from there. You can inspect the live controller quorum with the metadata CLI: ```bash bin/kafka-metadata-quorum.sh --bootstrap-server broker1:9092 describe --status ``` ```text ClusterId: abc123-Kq9... LeaderId: 2 CurrentVoters: [1, 2, 3] CurrentObservers: [] ``` The `LeaderId` here is the active controller, not a partition leader. If node 2 dies, the remaining voters elect a new controller within seconds and clients keep operating. ## Partitions, replicas, and leadership The cluster's real job is spreading partitions across brokers and keeping replicas in sync. Each [partition](https://www.conduktor.io/glossary/kafka-partitions-explained) has one leader and a set of followers defined by the replication factor. Producers and consumers only ever talk to the leader; followers copy the leader's log and stand ready to take over. The set of replicas currently caught up with the leader is the **in-sync replica (ISR)** set. A partition stays available for writes as long as enough replicas remain in-sync, which is why replication factor and `min.insync.replicas` are cluster-wide durability decisions rather than per-topic afterthoughts. See [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability) for how ISR and failover behave under load. ## Single cluster vs multiple clusters One cluster can scale to hundreds of brokers, but teams often run several clusters, one per region, environment, or tenant, for isolation, data residency, or blast-radius control. Kafka does not replicate between clusters on its own; you add tooling such as [MirrorMaker 2](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication) to mirror topics across cluster boundaries. Deciding when to add brokers to an existing cluster versus standing up a new one is a capacity and governance question. Broker count, disk, partition density, and throughput headroom all feed that decision, covered in [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning). ## Operating a Kafka cluster A healthy cluster needs continuous visibility into broker health, under-replicated partitions, controller state, and consumer lag. These signals live in JMX metrics that most teams scrape into Prometheus and Grafana, or a dedicated tool. Governance layers such as Conduktor sit in front of one or more clusters to add access control, audit, and a single view across brokers without exposing raw broker internals to every user. The core metrics to watch, and why, are detailed in [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). Common day-two operations, rolling restarts, partition reassignment, adding brokers, all run against the cluster as a whole and rely on the controller to rebalance leadership safely. ## Summary A Kafka cluster is the durable, fault-tolerant backbone of a streaming platform: brokers hold replicated partitions, a KRaft controller quorum tracks metadata and leadership, and clients reach everything through a bootstrap connection. Teams size clusters around partition count and throughput, replicate across clusters with MirrorMaker 2, and monitor broker health continuously. Governance and access-control tools like Conduktor operate on top of the cluster to make it safe to share across many teams. ## Related Pages - [Kafka Architecture Diagram & Components](https://www.conduktor.io/glossary/kafka-architecture-diagram), the full component and data-flow picture - [What is a Kafka Broker?](https://www.conduktor.io/glossary/kafka-brokers-explained), the server that stores and serves partitions - [What is a Kafka Partition?](https://www.conduktor.io/glossary/kafka-partitions-explained), the unit of ordering and parallelism - [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka), the ZooKeeper-free controller quorum - [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability), ISR and failover - [MirrorMaker 2 for Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication), mirroring between clusters - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning), sizing brokers and partitions - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics), what to watch in production ## Sources and References - [Apache Kafka 4.0 Documentation, Core Concepts](https://kafka.apache.org/documentation/#intro_concepts_and_terms) - [KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum) - [Apache Kafka, KRaft Configuration](https://kafka.apache.org/documentation/#kraft) - [Apache Kafka, Replication](https://kafka.apache.org/documentation/#replication) Source: https://www.conduktor.io/glossary/kafka-cluster --- # Kafka Connect: Building Data Integration Pipelines Kafka Connect is a distributed framework for reliably streaming data between Apache Kafka and external systems. Instead of writing custom producers and consumers for every data source and destination, Kafka Connect provides a standardized, scalable approach to data integration. This framework has become essential for data engineers building modern streaming architectures. With Kafka 4.0+ running in KRaft mode (eliminating ZooKeeper), Kafka Connect operates as a fully modern, cloud-native integration platform. The framework handles complex integration patterns including exactly-once delivery, schema evolution, and automatic failover, making it the de facto standard for connecting Kafka to databases, cloud storage, data warehouses, and other systems. ## Understanding Kafka Connect Architecture Kafka Connect operates as a distributed service that runs connector tasks across a cluster of workers. The architecture consists of three main components: - **Workers** are the runtime processes that execute connector tasks. They can run in standalone mode (single process) or distributed mode (multiple processes forming a cluster). Distributed mode is recommended for production environments as it provides fault tolerance and load balancing. - **Connectors** define how data should be copied to or from Kafka. They create and manage tasks that actually move the data. A source connector reads from an external system and writes to Kafka topics, while a sink connector reads from Kafka topics and writes to an external system. - **Tasks** perform the actual work of reading or writing data. Connectors split work into multiple tasks that can run in parallel across different worker nodes, enabling horizontal scalability. ![kafka-connect-building-data-integration-pipelines diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-connect-building-data-integration-pipelines-0.webp) ## Source Connectors: Ingesting Data into Kafka Source connectors bring data from external systems into Kafka topics. They handle the complexity of reading from different data sources while providing features like exactly-once semantics (ensuring each record is delivered exactly once, even during failures), schema management, and offset tracking. Common source connector types include: - **Database connectors** (JDBC, Debezium) that capture changes from relational databases using Change Data Capture (CDC) - **File connectors** that monitor directories and ingest file contents - **Message queue connectors** that bridge other messaging systems to Kafka - **Cloud storage connectors** that import data from S3, GCS, or Azure Blob Storage ### Example: JDBC Source Connector Configuration Here's a practical example of configuring a JDBC source connector to ingest data from PostgreSQL: ```json { "name": "postgres-source-connector", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max": "3", "connection.url": "jdbc:postgresql://postgres:5432/mydb", "connection.user": "${secrets:postgres:username}", "connection.password": "${secrets:postgres:password}", "mode": "incrementing", "incrementing.column.name": "id", "topic.prefix": "postgres-", "table.whitelist": "orders,customers,products", "poll.interval.ms": "1000", "batch.max.rows": "1000" } } ``` This configuration creates three parallel tasks to ingest data from three tables, using incremental mode to capture only new rows. The connector automatically creates topics named `postgres-orders`, `postgres-customers`, and `postgres-products`. ### Understanding Exactly-Once Semantics Kafka Connect supports **exactly-once semantics** (EOS), ensuring that each record is delivered exactly once even during failures, restarts, or network issues. This is critical for financial transactions, inventory management, and other use cases where duplicate or missing data causes problems. For source connectors, exactly-once delivery works by coordinating offset commits with Kafka transactions. The connector writes data to Kafka topics and commits source offsets in a single atomic transaction. If a failure occurs, the transaction rolls back, and upon restart, the connector resumes from the last committed offset. To enable exactly-once semantics in Kafka Connect: ```properties # Worker configuration exactly.once.source.support=enabled # Connector configuration (source connectors) transaction.boundary=poll ``` This requires Kafka 3.3+ and compatible connectors. Note that not all connectors support exactly-once semantics - check the connector documentation. ## Sink Connectors: Delivering Data from Kafka Sink connectors consume data from Kafka topics and write it to external systems. They provide automatic batching, error handling, and delivery guarantees to ensure reliable data delivery. Popular sink connector types include: - **Database sinks** (JDBC, Elasticsearch, MongoDB) for analytical workloads - **Data warehouse sinks** (Snowflake, BigQuery, Redshift) for analytics - **Object storage sinks** (S3, GCS, HDFS) for data lakes - **Search and indexing sinks** (Elasticsearch, Solr) for real-time search ### Example: Elasticsearch Sink Connector Configuration Here's an example of a sink connector that writes Kafka data to Elasticsearch for search and analytics: ```json { "name": "elasticsearch-sink-connector", "config": { "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "tasks.max": "2", "topics": "postgres-orders,postgres-customers", "connection.url": "https://elasticsearch:9200", "connection.username": "${secrets:elastic:username}", "connection.password": "${secrets:elastic:password}", "type.name": "_doc", "key.ignore": "false", "schema.ignore": "false", "behavior.on.null.values": "delete", "batch.size": "1000", "max.buffered.records": "20000", "flush.timeout.ms": "5000", "errors.tolerance": "all", "errors.deadletterqueue.topic.name": "dlq-elasticsearch", "errors.deadletterqueue.context.headers.enable": "true" } } ``` This configuration demonstrates several production best practices: batching for performance (`batch.size`), error handling with a dead letter queue (DLQ) for failed records, and secret management using ConfigProviders (a Kafka Connect feature that externalizes sensitive values). ## Kafka Connect in the Data Streaming Ecosystem Kafka Connect is the integration layer within broader streaming architectures. It complements other streaming components: - **Stream Processing**: While Kafka Streams and ksqlDB transform data within Kafka, Connect handles the integration boundaries. A typical pipeline might use a source connector to ingest database changes, Kafka Streams to enrich and aggregate data, and a sink connector to write results to a data warehouse. For detailed stream processing patterns, see [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) and [ksqlDB for Real-Time Data Processing](https://www.conduktor.io/glossary/ksqldb-for-real-time-data-processing). - **Schema Management**: Connect integrates tightly with Schema Registry to enforce schema validation. Connectors can automatically register schemas for data they produce and validate schemas for data they consume, ensuring data quality across the pipeline. This prevents schema compatibility issues and enables safe schema evolution. For comprehensive schema management strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). - **Change Data Capture (CDC)**: Debezium connectors running on Kafka Connect provide CDC capabilities, capturing row-level changes from databases in real-time. This enables event-driven architectures and keeps multiple data stores synchronized. CDC is essential for building real-time data pipelines and maintaining consistency across distributed systems. For CDC implementation details, see [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) and [Log-Based vs Query-Based CDC Comparison](https://www.conduktor.io/glossary/log-based-vs-query-based-cdc-comparison). - **Governance and Monitoring**: Tools like Conduktor provide visibility into Connect deployments, allowing teams to monitor connector health and enforce governance policies across integration pipelines. This visibility is essential as organizations scale to dozens or hundreds of connectors. See [Conduktor's Kafka Connect management features](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) for managing connectors, monitoring status, and troubleshooting connection issues. For broader monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) and [What is Data Observability: The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars). ## Configuration and Deployment Best Practices Running Kafka Connect reliably requires attention to several operational concerns: - **Distributed Mode Configuration**: Always use distributed mode in production. Configure multiple workers with the same `group.id` to form a Connect cluster. Store connector configurations and offsets in Kafka topics (specified by `config.storage.topic`, `offset.storage.topic`, and `status.storage.topic`) to enable automatic failover. With Kafka 4.0+ running in KRaft mode, these topics are managed without ZooKeeper, simplifying the architecture. For organizations migrating from older Kafka versions, see [ZooKeeper to KRaft Migration](https://www.conduktor.io/glossary/zookeeper-to-kraft-migration). - **Resource Allocation**: Allocate sufficient CPU and memory for connector tasks. Complex transformations and high-throughput connectors require adequate resources. Monitor JVM metrics and adjust heap sizes accordingly. - **Error Handling**: Configure dead letter queues (DLQ) for sink connectors to capture records that fail processing. This prevents connector failures from blocking the entire pipeline. The DLQ pattern writes failed records to a separate Kafka topic where they can be analyzed and reprocessed. - **Security**: Use externalized secrets (via ConfigProviders) instead of hardcoding credentials. Enable SSL/TLS for communication between Connect and both Kafka and external systems. Modern deployments should use OAuth 2.0 for authentication and mTLS (mutual TLS) for service-to-service communication. For detailed security patterns, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) and [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka). - **Monitoring**: Track key metrics including connector state, task status, offset lag, throughput, and error rates. Set up alerts for connector failures and performance degradation. Tools like Conduktor provide centralized visibility into all connectors across your organization, with features for monitoring, governance, and troubleshooting. Learn how to [manage Kafka Connect resources in Conduktor](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) to view connector configurations, restart failed tasks, and monitor throughput in real-time. ## Deploying Kafka Connect on Kubernetes Modern production deployments increasingly run Kafka Connect on Kubernetes for better resource management, scaling, and operational simplicity. The Strimzi operator provides native Kubernetes support for Kafka Connect. ### Strimzi KafkaConnect Custom Resource Here's an example of deploying Kafka Connect using Strimzi: ```yaml apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: my-connect-cluster annotations: strimzi.io/use-connector-resources: "true" spec: version: 3.9.0 replicas: 3 bootstrapServers: my-kafka-cluster-kafka-bootstrap:9093 # Enable KRaft mode (Kafka 4.0+) config: group.id: connect-cluster offset.storage.topic: connect-cluster-offsets config.storage.topic: connect-cluster-configs status.storage.topic: connect-cluster-status offset.storage.replication.factor: 3 config.storage.replication.factor: 3 status.storage.replication.factor: 3 # Install connector plugins build: output: type: docker image: my-registry/kafka-connect:latest plugins: - name: debezium-postgres artifacts: - type: tgz url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/2.7.0.Final/debezium-connector-postgres-2.7.0.Final-plugin.tar.gz - name: elasticsearch-sink artifacts: - type: zip url: https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-elasticsearch/versions/14.0.0/confluentinc-kafka-connect-elasticsearch-14.0.0.zip # Resource limits resources: requests: memory: 2Gi cpu: 1000m limits: memory: 4Gi cpu: 2000m # OAuth authentication (2025 best practice) authentication: type: oauth tokenEndpointUri: https://auth.example.com/token clientId: kafka-connect clientSecret: secretName: kafka-connect-oauth key: client-secret # TLS encryption tls: trustedCertificates: - secretName: kafka-cluster-ca-cert certificate: ca.crt ``` This Kubernetes-native approach provides several advantages: - **Automatic scaling**: Scale Connect workers up or down with `kubectl scale` - **Rolling updates**: Update connector plugins without downtime - **Secrets management**: Integrate with Kubernetes secrets and external secret stores - **Resource isolation**: Kubernetes manages CPU and memory allocation - **Observability**: Native integration with Prometheus and Grafana for metrics For detailed Kubernetes deployment patterns, see [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) and [Strimzi Kafka Operator for Kubernetes](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes). ## Performance Tuning and Monitoring Optimizing Kafka Connect performance requires tuning several key parameters: - **Task Parallelism**: Increase `tasks.max` to parallelize work across multiple tasks. Each task runs independently and can process different partitions or tables concurrently. Monitor CPU and memory usage to find the optimal task count. - **Batching Configuration**: Configure batch sizes to balance latency and throughput. For sink connectors, larger batches (`batch.size`, `flush.timeout.ms`) improve throughput but increase latency. For source connectors, `batch.max.rows` controls how many records are fetched per poll. - **Consumer and Producer Tuning**: Connect uses Kafka consumers and producers internally. Override their configurations using `consumer.*` and `producer.*` prefixes: ```json { "config": { "producer.compression.type": "snappy", "producer.linger.ms": "10", "producer.batch.size": "32768", "consumer.max.poll.records": "1000", "consumer.fetch.min.bytes": "1024" } } ``` **Key Metrics to Monitor**: | Metric | Description | Alert Threshold | |--------|-------------|-----------------| | `connector-state` | Connector status (RUNNING, FAILED, PAUSED) | Alert on FAILED | | `task-state` | Individual task status | Alert on FAILED | | `source-record-poll-total` | Records polled from source | Drops to 0 unexpectedly | | `sink-record-send-total` | Records sent to sink | Drops to 0 unexpectedly | | `offset-commit-skip-total` | Skipped offset commits | Increasing trend | | `task-error-total` | Total task errors | Any increase | For comprehensive monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) and [Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide). ## Single Message Transforms (SMTs) For simpler use cases, **Single Message Transforms (SMTs)** allow inline data transformation without custom connector development. SMTs process records as they flow through Connect, enabling lightweight transformations like field manipulation, filtering, routing, and masking. ### Common SMT Patterns Here are practical examples of frequently used transformations: **Field Masking for PII Protection**: ```json { "name": "postgres-source-with-masking", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max": "1", "connection.url": "jdbc:postgresql://postgres:5432/mydb", "table.whitelist": "users", "mode": "incrementing", "incrementing.column.name": "id", "topic.prefix": "postgres-", "transforms": "maskEmail,maskSSN", "transforms.maskEmail.type": "org.apache.kafka.connect.transforms.MaskField$Value", "transforms.maskEmail.fields": "email", "transforms.maskEmail.replacement": "****@****.com", "transforms.maskSSN.type": "org.apache.kafka.connect.transforms.MaskField$Value", "transforms.maskSSN.fields": "ssn", "transforms.maskSSN.replacement": "***-**-****" } } ``` **Topic Routing Based on Field Value**: ```json { "transforms": "routeByStatus", "transforms.routeByStatus.type": "org.apache.kafka.connect.transforms.RegexRouter", "transforms.routeByStatus.regex": ".*", "transforms.routeByStatus.replacement": "orders-$topic" } ``` **Extracting and Flattening Nested Fields**: ```json { "transforms": "flatten,extract", "transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value", "transforms.flatten.delimiter": "_", "transforms.extract.type": "org.apache.kafka.connect.transforms.ExtractField$Value", "transforms.extract.field": "payload" } ``` **Adding Metadata Timestamp**: ```json { "transforms": "insertTimestamp", "transforms.insertTimestamp.type": "org.apache.kafka.connect.transforms.InsertField$Value", "transforms.insertTimestamp.timestamp.field": "processed_at" } ``` For detailed SMT patterns and use cases, see [Kafka Connect Single Message Transforms](https://www.conduktor.io/glossary/kafka-connect-single-message-transforms). ## Connector Development and Customization While connector repositories like Confluent Hub provide hundreds of pre-built connectors, custom requirements sometimes necessitate developing proprietary connectors. The Kafka Connect framework provides interfaces for building new connectors: **SourceConnector** and **SinkConnector** classes define connector lifecycle and task configuration. **SourceTask** and **SinkTask** classes implement the actual data reading and writing logic. Connector plugins are deployed by placing JAR files in the `plugin.path` directory, which Connect scans at startup. This plugin isolation prevents dependency conflicts between different connectors. ## Summary Kafka Connect provides a powerful, scalable framework for integrating Apache Kafka with external systems. By standardizing how data moves in and out of Kafka through source and sink connectors, it reduces development complexity and operational overhead. The distributed architecture enables fault tolerance and horizontal scalability, while features like schema management, transformations, and error handling ensure reliable data delivery. For data engineers building streaming platforms, Kafka Connect connects databases, data warehouses, object storage, and other systems to the central Kafka event bus. With Kafka 4.0+ running in KRaft mode, modern deployments on Kubernetes via Strimzi, and OAuth/mTLS security, Connect provides a production-ready foundation for data integration. Understanding connector configuration, deployment patterns, and operational best practices is essential for building production-grade data integration pipelines. Whether ingesting database changes through CDC, loading data lakes, implementing PII masking via SMTs, or synchronizing multiple data stores, Kafka Connect simplifies the complexity of modern data integration. ### Related Topics For deeper understanding of Kafka Connect and its ecosystem, explore: - **Architecture Foundation**: [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) for core concepts and [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) for modern deployment - **Data Integration Patterns**: [Streaming ETL vs Traditional ETL](https://www.conduktor.io/glossary/streaming-etl-vs-traditional-etl) and [Streaming Data Pipeline](https://www.conduktor.io/glossary/streaming-data-pipeline) - **Security**: [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices), [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth), and [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka) - **Data Quality**: [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing) and [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework) - **PII Protection**: [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams) - **Lakehouse Integration**: [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables) and [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse) - **Serialization**: [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka) and [Avro vs Protobuf vs JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema) ## Related Concepts - [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) - Learn how to implement Change Data Capture using Debezium connectors running on Kafka Connect for real-time database replication. - [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling) - Understand how to handle failed messages in Kafka Connect sink connectors using dead letter queue patterns. - [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) - Explore how data contracts ensure reliability and compatibility in Kafka Connect integration pipelines. ## Sources and References - [Apache Kafka Documentation - Kafka Connect](https://kafka.apache.org/documentation/#connect) - Official documentation covering architecture, configuration, and operations for Kafka 4.0+ - [Kafka Connect Configuration Reference](https://kafka.apache.org/documentation/#connectconfigs) - Complete reference for worker and connector configuration parameters - [Debezium Documentation](https://debezium.io/documentation/) - Leading CDC connector framework built on Kafka Connect - [Strimzi Documentation - Kafka Connect](https://strimzi.io/docs/operators/latest/deploying.html#deploying-kafka-connect-str) - Kubernetes-native deployment patterns for Kafka Connect - [Kafka Improvement Proposals - KIP-618](https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors) - Exactly-once semantics for source connectors - [Kafka Connect REST API](https://kafka.apache.org/documentation/#connect_rest) - REST API for managing and monitoring connectors - [Confluent Hub](https://www.confluent.io/hub/) - Repository of pre-built, community-contributed Kafka Connect connectors - [Conduktor Platform](https://www.conduktor.io/) - Commercial platform for managing, monitoring, and governing Kafka Connect deployments Source: https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines --- # Kafka Connect Single Message Transforms When building data pipelines with Kafka Connect, you often need to modify data as it flows between systems. Single Message Transforms (SMTs) provide a lightweight mechanism to transform records within the connector itself, without requiring separate stream processing applications. For comprehensive coverage of Kafka Connect fundamentals and architecture, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines). ![Single Message Transforms in Kafka Connect pipeline](https://www.conduktor.io/assets/images/glossary/kafka-connect-single-message-transforms-0.webp) ## What Are Single Message Transforms? Single Message Transforms are pluggable components in Kafka Connect that modify individual records as they pass through a connector. SMTs operate on one message at a time, applying transformations like field manipulation, filtering, or routing during the data movement process. SMTs are configured directly in the connector configuration and execute within the Connect worker's JVM. This makes them simpler to deploy than standalone stream processing applications, but also more limited in capability. They are ideal for straightforward transformations that don't require aggregations, joins, or complex stateful processing. The Kafka Connect framework includes several built-in SMTs, and you can also develop custom transformations by implementing the `org.apache.kafka.connect.transforms.Transformation` interface. ## How SMTs Work SMTs are applied in a chain during the connector's data flow. For source connectors, transformations occur after data is read from the source system but before it's written to Kafka. For sink connectors, SMTs execute after reading from Kafka but before writing to the destination system. Each SMT receives a `SinkRecord` or `SourceRecord` object, applies its transformation logic, and returns a modified record. The transformed record then passes to the next SMT in the chain, creating a pipeline of transformations. Here's how you configure SMTs in a connector: ```properties transforms=maskPII,addTimestamp transforms.maskPII.type=org.apache.kafka.connect.transforms.MaskField$Value transforms.maskPII.fields=ssn,creditCard transforms.addTimestamp.type=org.apache.kafka.connect.transforms.InsertField$Value transforms.addTimestamp.timestamp.field=processing_time ``` **Understanding the Configuration Syntax**: The `transforms` property defines the ordered list of transformation names. Each transformation is then configured using the pattern `transforms..`. The `type` property specifies the Java class implementing the transformation, while other properties are specific to that transformation type. This configuration creates a chain of two transformations: first masking sensitive fields, then adding a timestamp field. **Before and After Example**: To illustrate how SMTs modify data, consider this input record: ```json { "customer_id": 12345, "name": "Jane Doe", "ssn": "123-45-6789", "creditCard": "4111-1111-1111-1111" } ``` After applying the SMT chain above, the output becomes: ```json { "customer_id": 12345, "name": "Jane Doe", "ssn": null, "creditCard": null, "processing_time": 1704067200000 } ``` The sensitive fields are masked (replaced with null), and a timestamp field is added to track when the transformation occurred. ## Conditional Transformations with Predicates Since Kafka Connect 2.6, you can apply SMTs conditionally using predicates. Predicates evaluate each record and determine whether to apply a transformation, enabling more sophisticated data processing logic. ```properties transforms=addRegion,maskPII transforms.addRegion.type=org.apache.kafka.connect.transforms.InsertField$Value transforms.addRegion.static.field=region transforms.addRegion.static.value=us-east transforms.addRegion.predicate=isUSCustomer transforms.maskPII.type=org.apache.kafka.connect.transforms.MaskField$Value transforms.maskPII.fields=ssn transforms.maskPII.predicate=hasSSN transforms.maskPII.negate=false predicates=isUSCustomer,hasSSN predicates.isUSCustomer.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches predicates.isUSCustomer.pattern=us-.* predicates.hasSSN.type=org.apache.kafka.connect.transforms.predicates.HasHeaderKey predicates.hasSSN.name=contains-ssn ``` This configuration adds a region field only for topics matching the `us-.*` pattern and masks SSN fields only when records have a specific header. Built-in predicates include `TopicNameMatches`, `HasHeaderKey`, and `RecordIsTombstone`. You can also create custom predicates for complex business logic. Predicates support negation via the `negate=true` parameter, allowing you to apply transformations when a condition is NOT met. This is particularly useful for filtering or handling exceptional cases. ## Common Built-in SMT Types Kafka Connect provides several useful built-in transformations: - **Field Manipulation**: The `InsertField` SMT adds metadata fields like timestamps or static values. The `ReplaceField` SMT renames or excludes specific fields. The `MaskField` SMT masks sensitive data with null values, useful for PII protection. For comprehensive strategies on handling personally identifiable information in streaming pipelines, see [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). - **Casting and Type Conversion**: The `Cast` SMT converts field types, such as changing strings to integers or timestamps to different formats. The `TimestampConverter` SMT transforms timestamp fields between different formats (string, Unix, Timestamp) and timezones. This helps align data types between source and destination systems. For deeper understanding of data serialization formats in Kafka, see [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka). - **Key and Value Transformations**: The `ValueToKey` SMT copies fields from the record value to the record key, useful for rekeying streams. The `KeyToValue` SMT does the opposite, copying key fields into the value. The `ExtractField` SMT extracts a single field from a complex record structure. - **Filtering and Routing**: The `Filter` SMT drops records based on predicates, while `RegexRouter` and `TimestampRouter` modify topic names dynamically based on record content or timestamps. These are particularly useful for partitioning data across multiple topics or implementing routing logic. - **Schema Modifications**: The `Flatten` SMT converts nested structures into flat records, making complex JSON or Avro data compatible with relational databases. The `HoistField` SMT wraps the entire record value in a struct field. The `SetSchemaMetadata` SMT modifies schema names and versions, useful for schema evolution scenarios. For understanding schema management and evolution strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). ## Real-World Use Cases Consider a typical scenario: streaming customer data from a PostgreSQL database to Amazon S3 for analytics. The source data contains sensitive information that must be masked before storage. ```properties name=postgres-to-s3-connector connector.class=io.aiven.kafka.connect.s3.AivenKafkaConnectS3SinkConnector topics=customer_data transforms=maskSensitive,addTimestamp,flatten transforms.maskSensitive.type=org.apache.kafka.connect.transforms.MaskField$Value transforms.maskSensitive.fields=ssn,credit_card,password transforms.addTimestamp.type=org.apache.kafka.connect.transforms.InsertField$Value transforms.addTimestamp.timestamp.field=ingest_timestamp transforms.flatten.type=org.apache.kafka.connect.transforms.Flatten$Value transforms.flatten.delimiter=_ ``` This pipeline masks PII fields, adds a timestamp field for tracking and partitioning, and flattens nested structures for easier querying. Note that you would typically partition S3 data using the connector's built-in partitioning capabilities rather than SMTs. Another common pattern involves dynamic topic routing. A source connector might read from multiple database tables and route each to a different Kafka topic: ```properties transforms=route transforms.route.type=org.apache.kafka.connect.transforms.RegexRouter transforms.route.regex=(.*) transforms.route.replacement=db_$1_events ``` This pattern is particularly common in Change Data Capture (CDC) scenarios where database changes from multiple tables need to be routed to separate Kafka topics. For detailed CDC implementation patterns, see [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium). ## SMTs vs Full Stream Processing While SMTs provide convenient transformation capabilities, they have important limitations compared to dedicated stream processing frameworks like Kafka Streams or Apache Flink. SMTs operate on individual messages without state or context from other messages. You cannot perform aggregations, windowing, joins between streams, or complex event processing. If your transformation logic requires correlating multiple events or maintaining state over time, you need a stream processing framework. SMTs also lack sophisticated error handling. If a transformation fails, the entire connector typically stops. Stream processing frameworks offer fine-grained error handling, dead letter queues, and retry mechanisms. However, SMTs offer significant advantages for simple transformations. They eliminate the need to deploy and manage separate stream processing applications, reducing operational complexity. They're easier to configure and test, and they keep your data pipeline architecture simpler when complex processing isn't required. Tools like Conduktor Platform help manage this complexity by providing visual configuration and validation of SMT chains through [Kafka Connect management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect), making it easier to build and debug transformation pipelines without manual JSON editing. ## Conduktor Interceptors: An Alternative to SMTs While SMTs operate within Kafka Connect, Conduktor Interceptors provide transformation at the Conduktor Gateway proxy layer, intercepting all Kafka traffic regardless of origin (Connect, producers, or consumers). Unlike SMTs configured per-connector, Interceptors apply globally to all matching traffic, creating consistent transformation behavior across your entire ecosystem. This makes them ideal for universal transformations like PII masking or timestamp standardization. Interceptors also excel at policy enforcement that SMTs cannot address, blocking messages violating data quality rules, enforcing producer settings, or validating schema compliance in real-time. The trade-off is operational complexity: SMTs are simpler for connector-specific transformations, while Interceptors require Conduktor Gateway deployment. For organizations already using Conduktor Gateway, Interceptors provide centralized transformation without per-connector configuration. Learn more at the [Interceptors documentation](https://docs.conduktor.io/guide/conduktor-concepts/interceptors). ## Best Practices and Limitations When working with SMTs, keep transformations simple and focused. Each SMT should perform one clear function. Complex multi-step transformations become difficult to debug and maintain. Be aware of performance implications. SMTs execute synchronously in the connector's data path, so slow transformations directly impact throughput. Avoid expensive operations like external API calls or heavy computation within SMTs. Consider schema compatibility carefully. If you're using a schema registry, ensure your transformations maintain schema compatibility or explicitly version your schemas. The `Cast` and `Flatten` SMTs can inadvertently break schema evolution if not configured properly. Test your SMT configurations thoroughly before production deployment. Since SMTs modify data in flight, errors can corrupt your data pipeline. Use tools that validate configurations and provide test environments for SMT chains. For comprehensive testing strategies for streaming data pipelines, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications) and [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing). Remember that SMTs cannot access external data sources or services. They work only with the data in the current record and connector configuration. If you need to enrich data from external systems, consider using a stream processing framework instead. ## Developing Custom SMTs When built-in transformations don't meet your needs, you can develop custom SMTs by implementing the `org.apache.kafka.connect.transforms.Transformation` interface. Custom SMTs are particularly useful for business-specific logic like data enrichment, validation, or format conversion. Here's a simple custom SMT that adds a custom field: ```java package com.example.transforms; import org.apache.kafka.common.config.ConfigDef; import org.apache.kafka.connect.connector.ConnectRecord; import org.apache.kafka.connect.transforms.Transformation; import org.apache.kafka.connect.data.Schema; import org.apache.kafka.connect.data.SchemaBuilder; import org.apache.kafka.connect.data.Struct; import java.util.Map; public class AddEnvironmentField> implements Transformation { private String environmentValue; @Override public void configure(Map configs) { environmentValue = (String) configs.get("environment"); } @Override public R apply(R record) { if (record.value() == null) { return record; } Struct value = (Struct) record.value(); Schema updatedSchema = SchemaBuilder.struct() .field("environment", Schema.STRING_SCHEMA) .fields(value.schema().fields()) .build(); Struct updatedValue = new Struct(updatedSchema); updatedValue.put("environment", environmentValue); value.schema().fields() .forEach(f -> updatedValue.put(f.name(), value.get(f))); return record.newRecord( record.topic(), record.kafkaPartition(), record.keySchema(), record.key(), updatedSchema, updatedValue, record.timestamp() ); } @Override public ConfigDef config() { return new ConfigDef() .define("environment", ConfigDef.Type.STRING, ConfigDef.Importance.HIGH, "Environment identifier"); } @Override public void close() { // Clean up resources if needed } } ``` **Packaging and Deployment**: Package your custom SMT as a JAR with dependencies, then place it in Kafka Connect's plugin path (typically defined by `plugin.path` in the worker configuration). Connect automatically discovers and loads plugins from this directory. ```bash # Build the custom SMT mvn clean package # Copy to Connect's plugin path mkdir -p /opt/kafka-connect/plugins/custom-smts cp target/custom-smt-1.0.0-jar-with-dependencies.jar \ /opt/kafka-connect/plugins/custom-smts/ ``` **Best Practices for Custom SMTs**: - Keep transformations stateless - SMTs should not maintain state between records - Ensure thread-safety - multiple threads may execute your SMT concurrently - Handle null values gracefully - check for null records, keys, and values - Implement proper error handling - throw `DataException` for data issues - Version your schemas appropriately when modifying schema structure - Test thoroughly with different data types and edge cases - Document configuration parameters clearly in the `ConfigDef` - Consider performance impact - avoid expensive operations in the hot path ## Monitoring SMT Performance SMT performance directly impacts connector throughput since transformations execute synchronously in the data path. Kafka Connect exposes JMX metrics for monitoring transformation performance: **Key Metrics**: - `transformation-time-ms-avg`: Average time spent in transformation per record - `transformation-time-ms-max`: Maximum transformation time - `transformation-record-rate`: Number of records transformed per second - `transformation-record-total`: Total records transformed Access these metrics via JMX or monitoring tools that integrate with Kafka Connect's metrics: ```bash # Example: Query JMX metrics using jmxterm echo "get -b kafka.connect:type=connector-task-metrics,connector=my-connector,task=0 transformation-time-ms-avg" | java -jar jmxterm.jar -l localhost:9999 ``` **Performance Optimization Tips**: - Profile SMT chains to identify bottlenecks - complex transformations may need optimization - Consider batch processing patterns if you control the connector implementation - Use predicates to skip unnecessary transformations on certain records - Monitor memory usage - large transformations can cause GC pressure - Test with production-like data volumes before deploying Tools like Conduktor Platform provide built-in monitoring dashboards for Connect metrics, including SMT performance, making it easier to identify and resolve performance issues without manual JMX querying. ## Summary Single Message Transforms provide a lightweight, convenient way to modify data within Kafka Connect pipelines. They're ideal for simple, per-record transformations like field masking, type conversion, schema flattening, and dynamic routing. SMTs shine when you need straightforward transformations without the operational overhead of separate stream processing applications. They're configured directly in connector properties, execute efficiently within the Connect worker, and use built-in transformations for common patterns. However, SMTs are not suitable for stateful processing, aggregations, or complex event patterns. Understanding when to use SMTs versus full stream processing frameworks is crucial for building efficient, maintainable data pipelines. For simple transformations at the connector level, SMTs offer the right balance of capability and simplicity. For complex processing requirements, consider Apache Kafka Streams, Apache Flink, or similar frameworks that provide comprehensive stream processing capabilities. To understand when to choose stream processing frameworks over SMTs, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink) and [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). ## Related Concepts - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines) - Comprehensive guide to Kafka Connect architecture and connector patterns where SMTs provide lightweight transformations. - [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) - Understand how SMTs interact with schema evolution when transforming message structures and handling schema compatibility. - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) - Learn how SMTs can enforce data quality rules through validation and transformation at the connector level. ## Sources and References 1. Apache Kafka Documentation - "Kafka Connect Transformations": https://kafka.apache.org/documentation/#connect_transforms 2. Confluent Documentation - "Single Message Transforms": https://docs.confluent.io/platform/current/connect/transforms/overview.html 3. Kafka Connect GitHub Repository - SMT Source Code: https://github.com/apache/kafka/tree/trunk/connect/transforms 4. Confluent Blog - "12 Single Message Transformations to Make the Most of Kafka Connect": https://www.confluent.io/blog/kafka-connect-single-message-transformation-tutorial-with-examples/ 5. Apache Kafka Javadoc - Transformation Interface: https://kafka.apache.org/documentation/#connect_transforms Source: https://www.conduktor.io/glossary/kafka-connect-single-message-transforms --- # Kafka Consumer Groups: How They Work A **Kafka consumer group** is a set of consumers that cooperatively read from a topic by splitting its partitions among themselves. Each partition is consumed by exactly one group member, enabling horizontal scaling without duplicate processing. The group coordinator tracks committed offsets and triggers a rebalance whenever members join or leave, reassigning partitions to maintain coverage. For foundational understanding of Kafka's architecture including topics, partitions, and brokers, see [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). ## What Are Consumer Groups? A consumer group is a collection of consumers that work together to consume messages from one or more Kafka topics. Each consumer in the group is assigned a subset of the topic's partitions, ensuring that no two consumers in the same group read from the same partition simultaneously. The key benefit is parallel processing. Instead of a single consumer reading all messages sequentially, multiple consumers can process different partitions concurrently. This approach scales horizontally, add more consumers to handle increased throughput. For a broader overview of how consumers and producers work together, see [Kafka Producers and Consumers](https://www.conduktor.io/glossary/kafka-producers-and-consumers). Kafka tracks which messages each consumer group has processed by storing offsets in a special internal topic called `__consumer_offsets`. Think of this as a ledger that records "Consumer Group A has processed messages up to offset 1000 on partition 0." This allows consumers to resume from where they left off after a restart or failure. In Kafka 4.0+ with KRaft mode (the successor to ZooKeeper), the group coordinator functionality is handled by dedicated controller nodes. This improves scalability and reduces operational complexity compared to the legacy ZooKeeper-based coordination. For more details, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## How Consumer Groups Enable Scalability Partition assignment is at the heart of consumer group scalability. When consumers join a group, Kafka's group coordinator assigns partitions to each consumer using a partition assignment strategy. ```java // Java consumer group configuration example Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("group.id", "analytics-processors"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("enable.auto.commit", "false"); // Manual offset management props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.CooperativeStickyAssignor"); KafkaConsumer consumer = new KafkaConsumer<>(props); consumer.subscribe(Arrays.asList("user-events")); ``` ![kafka-consumer-groups-explained diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-consumer-groups-explained-0.webp) Consider a topic with six partitions and three consumers in the same group. Each consumer might receive two partitions. If you add a fourth consumer, the partitions are redistributed, perhaps three consumers get two partitions each, and one gets none. If you have more consumers than partitions, the extra consumers remain idle. Common assignment strategies include: - **Cooperative Sticky (Incremental Cooperative Rebalancing)**: The modern default since Kafka 2.4+. Allows consumers to continue processing unaffected partitions during rebalancing, minimizing disruption. This is the recommended protocol for all new applications in 2025. - **Sticky**: Minimizes partition movement during rebalancing but requires all consumers to stop (eager rebalancing) - **Range**: Assigns partitions in ranges, which can lead to uneven distribution across multiple topics - **Round-robin**: Distributes partitions evenly in a circular fashion (deprecated in favor of cooperative sticky) Set the assignment strategy using `partition.assignment.strategy` configuration (default is `[CooperativeStickyAssignor]` in Kafka 3.0+). For most applications, use **cooperative sticky**, which provides the best balance of even distribution and minimal disruption during rebalancing. Only consider other strategies if you have specific requirements like maintaining partition locality across topics (range) or need compatibility with older Kafka versions. The group coordinator is one of the Kafka brokers that takes responsibility for managing a specific consumer group. Kafka automatically assigns coordinator duties among brokers to distribute the load. This coordinator tracks which consumers are alive (via heartbeats) and triggers rebalancing when needed. ## The Rebalancing Process Rebalancing occurs when the consumer group membership or topic metadata changes. Common triggers include: - A consumer joins the group - A consumer leaves (gracefully or due to failure) - A consumer is considered dead (missed heartbeat deadline) - Partitions are added to a subscribed topic During a traditional rebalance, all consumers stop processing, release their partitions, and wait for new assignments. This causes a processing pause that can impact latency-sensitive applications. Modern rebalancing protocols like cooperative rebalancing (also called incremental cooperative rebalancing) reduce this impact significantly. Instead of stopping all consumers, only the partitions being reassigned are revoked. Consumers not affected by the change continue processing without interruption. This protocol has been the default since Kafka 2.4+ and is essential for minimizing disruption in production systems. Rebalancing is necessary for fault tolerance and elasticity, but frequent rebalances can hurt performance. Setting appropriate session timeout and heartbeat interval configurations helps balance responsiveness with stability. Set `session.timeout.ms` (default: 10 seconds) higher than your maximum expected processing time for a single message, or process messages asynchronously. Use `max.poll.interval.ms` (default: 5 minutes) to control how long consumers can spend in their processing loop before calling `poll()` again. If consumers don't poll within this interval, Kafka considers them stuck and triggers a rebalance. Typical production settings: - `session.timeout.ms`: 45000 (45 seconds) - `heartbeat.interval.ms`: 3000 (3 seconds, typically 1/3 of session timeout) - `max.poll.interval.ms`: 300000 (5 minutes) ### Static Membership In containerized and cloud environments where consumers may restart frequently, Kafka 4.0+ supports static membership to avoid unnecessary rebalances. By setting a unique `group.instance.id` for each consumer, Kafka preserves partition assignments across restarts within the `session.timeout.ms` window. This is particularly valuable in Kubernetes deployments where pods restart for rolling updates or scaling operations. Static membership reduces rebalancing overhead and improves processing stability. For comprehensive guidance on running Kafka consumers in Kubernetes, see [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes). ```java Properties props = new Properties(); props.put("group.id", "analytics-processors"); props.put("group.instance.id", "consumer-1"); // Unique static ID props.put("session.timeout.ms", "60000"); // 60 seconds for restart window ``` ## Consumer Group Patterns Different applications have different consumption patterns. Understanding common approaches helps you design the right architecture. ### Multiple Consumer Groups The same topic can be consumed by multiple independent consumer groups. Each group maintains its own offsets and processes all messages independently. For example, imagine a topic containing user activity events. One consumer group might run real-time analytics, calculating metrics as events arrive. A second group might feed an alerting system, detecting anomalies. A third group could replicate data to a data warehouse for historical analysis. Each group processes the complete event stream at its own pace. This pattern enables building multiple independent applications from the same data source without coupling their implementations. ### Single Consumer Per Group Some use cases require processing all messages in order or maintaining state across the entire topic. Using a single consumer in a group ensures sequential processing but sacrifices parallelism and fault tolerance. This pattern is rare and typically indicates an architectural concern. Consider whether partitioning your data differently could enable parallel processing while maintaining necessary ordering guarantees within each partition. ## Monitoring Consumer Groups Effective monitoring is critical for operating consumer groups in production. The most important metric is consumer lag, the difference between the latest message offset in a partition and the consumer's committed offset. For example, if a partition's latest offset is 10,000 but your consumer has only committed offset 9,200, your lag is 800 messages. If that number keeps growing, your consumer is falling behind. High lag indicates consumers cannot keep up with incoming messages. This might signal: - Insufficient consumer capacity (need more consumers or instances) - Slow message processing logic - Network or broker issues - Frequent rebalancing disrupting processing Tracking rebalancing frequency and duration helps identify stability issues. Frequent rebalances suggest configuration problems or unstable consumers. Tools like Conduktor provide real-time visibility into consumer group health, displaying lag per partition, rebalancing events, and consumer assignment status. Learn more about [managing consumer groups in Conduktor](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) to monitor lag, track offsets, and troubleshoot rebalancing issues. For open-source monitoring, Kafka Lag Exporter (the 2025 standard for Prometheus-based monitoring) and Burrow provide solid lag tracking and alerting capabilities. For detailed coverage of consumer lag monitoring strategies and metrics collection, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ## Common Pitfalls and Best Practices ### Rebalancing Storms Misconfigured session timeouts can cause cascading failures. If processing a single message takes longer than the session timeout, the consumer is kicked out, triggering a rebalance. When it rejoins, the cycle repeats, a rebalancing storm. Set `session.timeout.ms` higher than your maximum expected processing time, or process messages asynchronously. Use `max.poll.interval.ms` to control how long consumers can spend in processing loops. Consider using static membership (`group.instance.id`) to reduce rebalance sensitivity during restarts. ### Partition Skew Uneven data distribution across partitions can leave some consumers idle while others are overloaded. Use partition keys that distribute load evenly. For detailed strategies on partition key design and distribution, see [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). Monitor per-partition metrics to identify skew. ### Offset Management Kafka provides automatic offset commits, but these can lead to message loss or duplication in failure scenarios. For exactly-once semantics (EOS), configure consumers with `isolation.level=read_committed` to only read transactionally committed messages. Combine this with idempotent producers (`enable.idempotence=true`) and transactional producers to achieve end-to-end exactly-once processing. For more details, see [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). Always commit offsets after successfully processing messages, not before. Processing then committing ensures at-least-once delivery. ### Scaling Limits You cannot have more active consumers than partitions in a consumer group. Plan partition counts based on expected parallelism requirements. Creating topics with too few partitions limits future scaling. For guidance on planning partition counts and topic design, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines). ## Related Concepts - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Essential for tracking consumer group performance and detecting processing bottlenecks. - [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices) - Partition design directly impacts consumer group scalability and performance. - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Combines consumer group offset management with transactional processing for guaranteed delivery semantics. - [Kafka UI: The Complete Guide](https://www.conduktor.io/kafka-ui) - Inspecting consumer groups, offsets, and lag from a UI is table stakes for Kafka Visibility. ## Summary Consumer groups are fundamental to Kafka's scalability model. They enable parallel processing through partition assignment, provide fault tolerance through rebalancing, and allow multiple applications to consume the same data independently. Key takeaways: - Consumer groups distribute partitions among consumers for parallel processing - Cooperative sticky rebalancing (default in Kafka 2.4+) minimizes disruption during membership changes - Static membership prevents unnecessary rebalances in containerized environments like Kubernetes - KRaft mode (Kafka 4.0+) improves group coordinator scalability and operational simplicity - Monitoring lag with tools like Kafka Lag Exporter, Burrow, and Conduktor is essential for production operations - Proper configuration of `session.timeout.ms`, `max.poll.interval.ms`, and `heartbeat.interval.ms` prevents common issues like rebalancing storms - Exactly-once semantics (EOS) requires careful configuration of consumers with transactional producers Understanding consumer groups lets you build scalable streaming applications that can grow with your data volumes and processing requirements. ## Sources and References 1. Apache Kafka Documentation - Consumer Groups: https://kafka.apache.org/documentation/#consumerconfigs 2. Narkhede, N., Shapira, G., & Palino, T. (2017). *Kafka: The Definitive Guide*. O'Reilly Media. 3. Confluent Blog - Apache Kafka Rebalance Protocol: https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/ 4. KIP-429: Kafka Improvement Proposal for Incremental Cooperative Rebalancing: https://cwiki.apache.org/confluence/display/KAFKA/KIP-429 5. Kreps, J. (2013). The Log: What every software engineer should know about real-time data's unifying abstraction: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying Source: https://www.conduktor.io/glossary/kafka-consumer-groups-explained --- # Kafka Data Governance: Definition & Primitives **Kafka data governance** is the layer of policies and controls that determines, for every topic and message in a Kafka estate, who owns it, who can access it, what schema and quality rules apply, what data is sensitive, and how every action is audited. Kafka brokers do not provide governance on their own; it is built on top with a combination of Schema Registry, ACLs or RBAC, IAM, key management, and audit pipelines. ## The Six Primitives of Kafka Data Governance Kafka governance breaks into six independent primitives. With one team you can ignore most of them. Past a handful of teams sharing the same brokers, all six matter. ![Kafka data governance primitives](https://www.conduktor.io/assets/images/glossary/kafka-data-governance-0.webp) | Primitive | What it answers | |---|---| | Schema policy | What schema is allowed on which topic; what evolution rules apply (backward, forward, full); what happens when a producer breaks the contract | | Topic ownership | For every topic, which application and which team is accountable; how orphan topics are detected | | Access control | Who, human or service account, can produce, consume, create, or delete; expressed as roles and groups rather than raw principals | | Encryption and masking | Which fields are sensitive (PII, PHI, secrets); which are encrypted at the field level; which are masked in lower environments; which keys protect them | | Audit and lineage | Who did what, when, from where; queryable and exportable rather than raw broker log lines | | Data quality | What validation rules a message must pass to be accepted; what happens to records that fail (reject, route to DLQ, log) | ## Why Kafka Brokers Don't Provide Governance Apache Kafka is a broker. It serves bytes. The broker has no concept of "team", "topic owner", "sensitive field", "schema contract", or "audit retention". Each governance primitive has to come from somewhere outside the broker: - **Schemas** live in a Schema Registry (Confluent, Apicurio, AWS Glue). The registry stores schemas; on its own it does not block bad-shape produces or enforce ownership of subjects. See [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). - **Ownership** lives in a separate system. Without an application or topic catalog, ownership is a wiki page or a spreadsheet that drifts from reality. - **Access** lives in `kafka-acls.sh` for the broker layer, sometimes layered with RBAC. See [Kafka ACLs](https://www.conduktor.io/kafka-acl) and [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming). - **Encryption** is split between TLS on the wire, KMS-backed keys for field-level encryption, and disk encryption for at-rest data — three layers that need to be configured independently. - **Audit** is broker logs (Log4j authorizer output, request logs) shipped to a SIEM. The brokers do not produce a queryable audit history on their own. - **Data quality** is either enforced at produce time by the application, or at the gateway/proxy layer, or not at all. Multiply that by 20 teams, 500 topics, three clusters, and a compliance reviewer asking *"who has access to topics containing PII?"*, and the missing pieces stop looking like background admin. ## Maturity Levels Most teams pass through four stages, usually in this order: 1. **Ad hoc** — ACLs added during incidents, no Schema Registry, no central audit. One platform engineer knows where everything is. 2. **Discoverable** — Schema Registry deployed, topic naming conventions written down (if not enforced), brokers shipping audit to a SIEM. 3. **Owned** — topics registered against applications and teams, access requests go through a workflow, schema breaking changes blocked at produce time. 4. **Programmable** — governance expressed as code (Terraform, GitOps), policies enforced declaratively, audit and quality rules versioned alongside application code. Stages 3 and 4 are where governance stops being its own workstream and just becomes how the platform behaves. ## Governance vs Security The two overlap but are not the same. **Security** answers *"can the wrong person reach the data?"* — encryption, authentication, authorization. **Governance** answers *"can the right person reach the right data with the right shape?"* — security plus schema, ownership, quality, lineage. See [Kafka Security: The Four Pillars](https://www.conduktor.io/kafka-security) for the security-only frame. ## Implementing Kafka Data Governance In practice, these six primitives live in a platform layered on top of Kafka, not in the brokers. For how Conduktor implements them in one control plane, see the [Kafka governance platform](https://www.conduktor.io/kafka-data-governance) page. ## Related Pages - [Kafka ACLs](https://www.conduktor.io/kafka-acl), how broker-level access control works and where it stops scaling - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management), the contract layer that governance relies on - [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming), patterns for ACLs and RBAC across consumer groups and topics - [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms), shipping broker events to a SIEM - [Automated Data Quality Testing](https://www.conduktor.io/glossary/automated-data-quality-testing), validation rules for messages in flight - [Kafka Security: The Four Pillars](https://www.conduktor.io/kafka-security), the security subset of governance ## Sources - [Apache Kafka Documentation, Security and Authorization](https://kafka.apache.org/documentation/#security) - [Confluent Schema Registry Documentation](https://docs.confluent.io/platform/current/schema-registry/index.html) - [NIST RBAC Standard (INCITS 359)](https://csrc.nist.gov/projects/role-based-access-control) - [DAMA-DMBOK: Data Management Body of Knowledge](https://www.dama.org/cpages/body-of-knowledge), the broader data governance frame Kafka governance fits into Source: https://www.conduktor.io/glossary/kafka-data-governance --- # Kafka Log Compaction Explained Apache Kafka is widely known for its ability to retain message streams for extended periods. While most users are familiar with time-based or size-based retention policies, Kafka offers a powerful alternative called log compaction. This retention mechanism serves a fundamentally different purpose: maintaining the latest state of each record rather than preserving a complete historical timeline. ![Kafka log compaction process and structure](https://www.conduktor.io/assets/images/glossary/kafka-log-compaction-explained-0.webp) ## What is Kafka Log Compaction? Log compaction is a retention policy that ensures a Kafka topic retains at least the last known value for each message key within a partition. Unlike standard deletion-based retention (which removes old data after a specified time or when a size limit is reached), compaction keeps the most recent update for each key indefinitely. For foundational understanding of Kafka's architecture including topics, partitions, and brokers, see [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). This approach transforms a Kafka topic from an append-only log of events into something closer to a table of current states. However, the log structure remains intact, compaction doesn't remove messages immediately, and it preserves message offsets so consumers can still track their position reliably. A critical feature of log compaction is support for tombstone records. When a message with a null value is published for a given key, it signals that the key should be deleted. The compaction process will eventually remove both the tombstone and all previous messages for that key. ## How Log Compaction Works Kafka partitions are divided into segments, immutable files that contain a portion of the log. Each segment typically contains a time-bounded or size-bounded chunk of messages, allowing Kafka to manage, compact, and delete data efficiently. Log compaction operates on these segments using a background cleaner thread. Log compaction works identically in both legacy ZooKeeper-based Kafka and modern KRaft mode (Kafka 4.0+). The compaction process is entirely broker-side and doesn't depend on the metadata management layer, making it a reliable feature across all Kafka deployment modes. The log is conceptually split into two sections: - **Clean section**: Segments that have already been compacted, containing only the most recent value for each key - **Dirty section**: Segments containing potentially duplicate keys that haven't been compacted yet The cleaner thread periodically scans the dirty section, builds a hash map of the latest offset for each key, and creates new clean segments containing only the most recent records. The old segments are then deleted. Several configuration parameters control when compaction runs: - `min.cleanable.dirty.ratio`: The minimum ratio of dirty records to total records before compaction triggers (default 0.5) - `segment.ms`: How long before a segment becomes eligible for compaction - `min.compaction.lag.ms`: Minimum time a message remains uncompacted, ensuring recent messages stay available Importantly, the active segment (currently being written to) is never compacted, ensuring new data is immediately available to consumers. ## Log Compaction vs Time-Based Retention Standard Kafka topics use deletion-based retention, configured with `retention.ms` or `retention.bytes`. Messages older than the retention period are deleted regardless of their content. Log compaction, enabled with `cleanup.policy=compact`, takes a different approach. It retains messages based on key uniqueness rather than age. This means: - **Time-based retention**: Appropriate for event streams where historical order matters (clickstreams, logs, audit trails) - **Log compaction**: Appropriate for state or entity updates where only the current value matters (user profiles, product catalogs, configuration data) You can also combine both policies with `cleanup.policy=compact,delete`, which compacts the log while also removing old segments that exceed retention limits. ## Common Use Cases Log compaction enables several important patterns in data streaming: ### Change Data Capture (CDC) When capturing database changes into Kafka, each message represents a row update keyed by the primary key. Log compaction ensures the topic always contains the latest state of each row without storing the complete change history indefinitely. For a comprehensive introduction to CDC patterns and implementation strategies, see [What is Change Data Capture (CDC) Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). For example, consider a `users` table synced to Kafka: ``` Key: user_123 | Value: {"name": "Alice", "email": "alice@example.com", "updated_at": "2025-01-15"} Key: user_123 | Value: {"name": "Alice", "email": "alice@newmail.com", "updated_at": "2025-02-20"} Key: user_123 | Value: {"name": "Alice Smith", "email": "alice@newmail.com", "updated_at": "2025-03-10"} ``` After compaction, only the final record remains, representing the current state of user_123. ### State Stores and Materialized Views Applications that build local state from Kafka topics benefit from compaction. When restarting or rebalancing, they can rebuild state by consuming only the latest value for each key rather than processing the entire history. ### Configuration Management Distributed systems often use compacted topics to share configuration across services. Each configuration key holds its current value, and services can reconstruct the full configuration by reading the compacted topic. For broader guidance on designing topics for different use cases, including configuration management, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines). ## Configuring Log Compaction To enable log compaction on a topic, set the cleanup policy: ```bash # Using kafka-configs.sh kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type topics \ --entity-name my-compacted-topic \ --alter \ --add-config cleanup.policy=compact # Or specify during topic creation kafka-topics.sh --bootstrap-server localhost:9092 \ --create \ --topic my-compacted-topic \ --partitions 3 \ --replication-factor 3 \ --config cleanup.policy=compact \ --config min.cleanable.dirty.ratio=0.5 ``` Additional important configurations include: - `min.cleanable.dirty.ratio`: Controls compaction frequency (lower values mean more frequent compaction but higher resource usage). Default is 0.5, meaning compaction triggers when 50% of the log contains duplicate keys. - `delete.retention.ms`: How long tombstone records are retained before final deletion (default 24 hours) - `max.compaction.lag.ms`: Maximum time before a message must be compacted - `segment.ms` and `segment.bytes`: Control segment size and rotation, affecting compaction granularity It's important to understand that compaction is not immediate. Messages remain in the log until the cleaner thread runs and processes the segments containing them. ## Log Compaction and Tiered Storage Kafka 3.6+ introduced tiered storage, which allows older log segments to be moved to cheaper object storage (S3, GCS, Azure Blob) while keeping recent data on local broker disks. Understanding how log compaction interacts with tiered storage is critical for 2025 deployments. **Key behaviors with tiered storage enabled:** - **Compaction only applies to local segments**: The cleaner thread only compacts segments stored on broker disks, not segments that have been moved to remote storage - **Remote segments remain uncompacted**: Once a segment is uploaded to tiered storage, it retains all its original messages (including duplicates) indefinitely - **Implications for state reconstruction**: When rebuilding state from a compacted topic with tiered storage, consumers may need to read and deduplicate remote segments, potentially increasing recovery time **Best practices for compacted topics with tiered storage:** 1. **Adjust local retention**: Set `local.retention.ms` high enough to ensure adequate compaction before segments are moved to remote storage 2. **Monitor compaction lag**: Track the dirty ratio and ensure segments are compacted before tiering 3. **Consider compaction-only policies**: For true state store topics, use `cleanup.policy=compact` without tiered storage, or use very high local retention thresholds For detailed coverage of tiered storage architecture and configuration, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). ## Log Compaction in Data Streaming Architectures Log compaction plays a crucial role in stateful stream processing frameworks like Kafka Streams and Apache Flink. ### KTables and Compacted Topics In Kafka Streams, a **KTable** represents a changelog stream where each record is an update to a keyed state. KTables are fundamentally backed by compacted topics, the abstraction directly maps to log compaction semantics: ```java // KTable automatically interprets the compacted topic as a table StreamsBuilder builder = new StreamsBuilder(); KTable usersTable = builder.table( "users-compacted", // Must be a compacted topic Consumed.with(Serdes.String(), userSerde) ); // Updates to the KTable produce to a compacted changelog usersTable .filter((key, user) -> user.isActive()) .toStream() .to("active-users-compacted"); // Also compacted ``` When you materialize a KTable with operations like aggregations or joins, Kafka Streams automatically creates compacted changelog topics to back the state stores. For example: ```java KTable orderCounts = ordersStream .groupByKey() .count(Materialized.as("order-counts-store")); // Creates compacted topic: app-id-order-counts-store-changelog ``` ### State Store Recovery with Compacted Topics Kafka Streams uses compacted changelog topics to back state stores. When a stream processing application updates local state (such as an aggregation or join table), it writes changes to a compacted topic. If the application fails or restarts, it can rebuild its state by consuming this topic. Since the topic is compacted, restoration is faster, only the latest value for each key needs to be processed. This pattern enables exactly-once processing semantics (EOS) and fault-tolerant stateful operations without requiring external databases. The compacted topic serves as both a changelog and a source of truth for application state. **Example: Tombstone deletion in Kafka Streams** ```java // Deleting from a KTable produces a tombstone to the compacted topic usersTable .toStream() .filter((key, user) -> user.shouldDelete()) .mapValues(user -> null) // Produces tombstone (null value) .to("users-compacted"); // Eventually removes the key entirely ``` For detailed coverage of state store architecture and recovery mechanisms, see [State Stores in Kafka Streams](https://www.conduktor.io/glossary/state-stores-in-kafka-streams). ### Event Sourcing and Materialized Views Event sourcing architectures also use log compaction. While the complete event history might be preserved in one topic, a compacted topic can maintain the current projection or materialized view derived from those events. This pattern is common in CQRS (Command Query Responsibility Segregation) architectures where read models are built from event streams. For comparing stream processing frameworks and their approaches to state management, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink). ## Monitoring and Troubleshooting Monitoring log compaction is essential to ensure it's working correctly and not falling behind. ### Key JMX Metrics Important JMX metrics to track: - `kafka.log.LogCleanerManager.max-dirty-percent`: Shows how much of the log is dirty and eligible for compaction - `kafka.log.LogCleaner.cleaner-recopy-percent`: Indicates compaction efficiency (lower is better) - `kafka.log.LogCleaner.max-compaction-delay-secs`: Maximum time any partition waited for compaction - `kafka.server.BrokerTopicMetrics.TotalProduceRequestsPerSec`: Helps correlate compaction lag with write throughput - Segment count and size per partition ### Modern Monitoring with Conduktor In 2025, managing compaction across multiple clusters and environments is significantly easier with platforms like **Conduktor**. Conduktor provides: - **Visual topic configuration inspection**: Quickly identify which topics have `cleanup.policy=compact` across all clusters - **Real-time compaction metrics**: Dashboard views of dirty ratios, cleaner lag, and segment statistics without manually querying JMX - **Alerting on compaction issues**: Get notified when dirty ratios exceed thresholds or compaction falls behind - **Configuration management**: Easily adjust compaction parameters (`min.cleanable.dirty.ratio`, `max.compaction.lag.ms`) through the UI with validation - **Data inspection**: Browse compacted topics to verify latest values and identify tombstone records Conduktor Gateway, a Kafka proxy, also enables testing compaction behavior in development environments by introducing controlled delays or failures in the compaction process. For comprehensive monitoring strategies across your Kafka infrastructure, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ### Common Issues Frequent compaction problems and their solutions: - **Compaction lag**: If the dirty ratio stays high, compaction may not be keeping up. Increase cleaner threads (`log.cleaner.threads`) or adjust `min.cleanable.dirty.ratio` to trigger compaction more aggressively - **Missing keys**: Messages without keys cannot be compacted, they're retained until time/size-based deletion. Ensure all messages include keys when using compacted topics - **Unexpected deletion**: Check for tombstone records (null values) if data disappears unexpectedly. Review producer code for accidental null value writes - **Slow state restoration**: With tiered storage enabled, verify that adequate local retention allows compaction before segments move to remote storage ## Related Concepts - [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) - Guidance on choosing between compaction, time-based retention, and hybrid retention policies for different use cases. - [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams) - Kafka Streams relies heavily on compacted topics for maintaining state stores and changelog topics. - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka) - Understanding how compaction interacts with tiered storage is critical for modern Kafka deployments. ## Summary Kafka log compaction is a powerful retention mechanism that maintains the latest state for each message key rather than preserving a complete time-ordered history. It transforms Kafka topics into durable, fault-tolerant state stores that support use cases like change data capture, configuration management, and stateful stream processing. Understanding the mechanics of compaction, how the cleaner thread operates on segments, when compaction triggers, and how tombstones enable deletion, is essential for using this feature effectively. While compaction isn't appropriate for all use cases, it's invaluable when you need a distributed, scalable way to maintain current state in a data streaming architecture. By configuring compaction correctly and monitoring its performance, you can use Kafka not just as a message broker but as a reliable state management layer. ## Sources and References 1. [Apache Kafka Documentation: Log Compaction](https://kafka.apache.org/documentation/#compaction) - Official Kafka documentation detailing compaction mechanics and configuration 2. [Confluent: Log Compaction Highlights](https://www.confluent.io/blog/kafka-log-compaction-demystified/) - In-depth explanation of how log compaction works under the hood 3. Narkhede, N., Shapira, G., & Palino, T. (2017). *Kafka: The Definitive Guide*. O'Reilly Media. - Comprehensive book covering Kafka internals including log compaction 4. Kleppmann, M. (2017). *Designing Data-Intensive Applications*. O'Reilly Media. - Context on event sourcing and state management patterns 5. [Kafka Streams Documentation: State Stores](https://kafka.apache.org/documentation/streams/developer-guide/processor-api#state-stores) - How Kafka Streams uses compacted topics for state management Source: https://www.conduktor.io/glossary/kafka-log-compaction-explained --- # Kafka MirrorMaker 2: Cross-Cluster Replication **Kafka MirrorMaker 2 (MM2)** is the Apache Kafka project's tool for replicating topics between clusters, built on the Kafka Connect framework. It copies records, topic configurations, and consumer-group offsets across data centers, cloud regions, or providers, which is what makes disaster recovery, cluster migration, and low-latency regional reads possible. It replaces the original MirrorMaker, whose single-consumer design could not keep offsets or configuration in sync. For disaster recovery planning beyond cross-cluster replication, see [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters). Each cluster MM2 mirrors follows the same single-cluster [Kafka architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). ## What is Kafka MirrorMaker 2 MirrorMaker 2 is a Kafka Connect-based replication tool that copies data between Apache Kafka clusters. It is the successor to the original MirrorMaker (now called MirrorMaker 1), addressing several limitations of its predecessor. While MirrorMaker 1 was a simple consumer-producer pair that could replicate topic data, it lacked essential features for production use. MM2 introduces significant improvements including automatic topic creation, consumer group offset synchronization, access control list (ACL) replication, and exactly-once semantics support. These enhancements make MM2 suitable for enterprise-grade replication scenarios where data consistency and operational simplicity matter. MM2 was introduced in Apache Kafka 2.4 through KIP-382 and has become the recommended approach for cross-cluster replication. It builds on the Kafka Connect framework, which means it benefits from Connect's scalability, fault tolerance, and operational characteristics. **Modern Deployment with Kafka 4.0+ and KRaft**: As of Kafka 4.0 and later, MirrorMaker 2 runs on KRaft-based clusters (Kafka's consensus protocol that replaced ZooKeeper). KRaft simplifies operations by eliminating ZooKeeper dependencies, reducing operational complexity and improving cluster startup times. When deploying MM2 with KRaft clusters, the connectors interact with cluster metadata through the KRaft controller layer, maintaining the same replication semantics while benefiting from KRaft's improved metadata management and faster failover capabilities. For organizations migrating from ZooKeeper-based Kafka, MM2 can replicate between mixed environments, making it valuable during transitions. For details on KRaft architecture, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## Architecture and Components MirrorMaker 2 consists of three main connector types that work together to provide comprehensive replication: ![MirrorMaker 2 consists of three main connector types that work together to provide comprehensive replication](https://www.conduktor.io/assets/images/glossary/kafka-mirrormaker-2-for-cross-cluster-replication-0.webp) - **MirrorSourceConnector** handles the core data replication task. It consumes records from topics in the source cluster and produces them to corresponding topics in the target cluster. The connector preserves message keys, values, headers, and timestamps, ensuring data fidelity across clusters. - **MirrorCheckpointConnector** synchronizes consumer group offsets between clusters. This is crucial for disaster recovery scenarios where applications need to fail over from one cluster to another. By tracking which offsets have been replicated, the checkpoint connector enables applications to resume consumption from the correct position after a failover. Understanding how offset translation works is essential for successful failover. When MM2 replicates messages from the source cluster, the target cluster assigns new offsets to these messages (starting from 0 for new topics). The checkpoint connector maintains a mapping between source offsets and target offsets. For example, if a consumer has read up to offset 1000 in the source cluster's `orders` topic, the checkpoint connector calculates the equivalent offset in the target cluster's `us-west.orders` topic (perhaps offset 1000 maps to target offset 998 if two messages failed replication). This translation is stored in the `*.checkpoints.internal` topic on the target cluster. During failover, consumers can query this mapping to determine where to start reading in the target cluster, ensuring continuity without message loss or duplication. For more on consumer offset management, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). **MirrorHeartbeatConnector** monitors replication health by emitting heartbeat messages. These heartbeats help track replication lag and detect connectivity issues between clusters. All three connectors run within the Kafka Connect framework, either in standalone mode for simple setups or distributed mode for production deployments. The distributed mode provides fault tolerance and horizontal scaling, allowing MM2 to handle high-throughput replication across large Kafka deployments. For detailed coverage of Kafka Connect architecture and building integration pipelines, see [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines). One notable aspect of MM2's design is its topic naming convention. By default, replicated topics are prefixed with the source cluster name. For example, a topic named `orders` in a cluster designated as `us-west` would appear as `us-west.orders` in the target cluster. This naming pattern prevents conflicts and makes the data lineage clear, though it can be customized if needed. ## Replication Patterns and Use Cases MirrorMaker 2 supports several replication patterns, each suited to different business and technical requirements: **Active-Passive Replication** is the most common pattern for disaster recovery. A primary cluster handles all production traffic while MM2 continuously replicates data to a secondary cluster in a different region or availability zone. If the primary cluster fails, applications can fail over to the secondary cluster. The checkpoint connector ensures that consumers can resume from the correct offset, minimizing data loss and duplication. For broader disaster recovery strategies including RTO/RPO planning and backup mechanisms, see [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters). ![kafka-mirrormaker-2-for-cross-cluster-replication diagram 2](https://www.conduktor.io/assets/images/glossary/kafka-mirrormaker-2-for-cross-cluster-replication-1.webp) For example, a financial services company might run its primary Kafka cluster in US-East with active replication to a standby cluster in US-West. If the US-East data center experiences an outage, trading applications can quickly switch to the US-West cluster using the synchronized consumer offsets. **Active-Active Replication** involves bidirectional replication where multiple clusters both produce and consume data. This pattern supports multi-region deployments where applications in different geographies need low-latency access to data. However, active-active replication requires careful handling of potential data conflicts and cycles in the replication topology. Understanding cycle detection and conflict resolution is critical for active-active patterns. When you configure bidirectional replication (Cluster A → Cluster B and Cluster B → Cluster A), MM2 must prevent infinite replication loops where messages bounce back and forth between clusters indefinitely. MM2 handles this through header-based cycle detection: when replicating a message, the MirrorSourceConnector adds a header indicating the source cluster. Before replicating, MM2 checks message headers to detect if a message originated from the target cluster, if so, it's skipped to break the cycle. However, cycle detection doesn't solve application-level conflicts. Consider this scenario: An order-processing application in US-East updates order ID 12345 to status "SHIPPED", while simultaneously an application in US-West updates the same order to "CANCELLED". Both updates get replicated bidirectionally, but which update wins? MM2 doesn't provide conflict resolution, it replicates both updates in the order they arrive. Applications must implement their own conflict resolution strategies: - **Timestamp-based resolution**: Use message timestamps to keep the latest update - **Application-level versioning**: Include version numbers in messages and apply optimistic locking - **Regional authority**: Designate specific clusters as authoritative for certain data domains - **Last-writer-wins with CRDTs**: Use Conflict-free Replicated Data Types for commutative operations For critical use cases requiring active-active with strong consistency, consider application-level distributed coordination (like etcd or Consul) or single-region writes with cross-region reads. - **Hub-and-Spoke Aggregation** centralizes data from multiple regional or edge clusters into a central cluster for analytics, reporting, or cross-regional data access. For instance, a global retail company might replicate sales data from clusters in different countries to a central data hub for enterprise-wide analytics. - **Fan-Out Distribution** does the opposite, replicating data from a central cluster to multiple regional clusters. This pattern works well for distributing reference data, configuration updates, or content that needs to be available locally in multiple regions. ## Configuration and Deployment Deploying MirrorMaker 2 requires configuring source and target cluster connections, specifying which topics to replicate, and tuning performance parameters. A basic MM2 configuration defines cluster aliases, connection details, and replication flows. Here's a simplified example: ```properties clusters = source, target source.bootstrap.servers = source-kafka:9092 target.bootstrap.servers = target-kafka:9092 source->target.enabled = true source->target.topics = orders.*, inventory.*, shipments.* replication.factor = 3 offset-syncs.topic.replication.factor = 3 checkpoints.topic.replication.factor = 3 ``` This basic configuration establishes replication from the `source` cluster to the `target` cluster for all topics matching the patterns `orders.*`, `inventory.*`, and `shipments.*`. The replication factor settings ensure that MM2's internal topics are properly replicated for fault tolerance. **Production Configuration with Security and Exactly-Once Semantics**: For enterprise deployments, a comprehensive configuration includes authentication, encryption, and transactional guarantees: ```properties # Cluster definitions clusters = source, target # Source cluster configuration (KRaft-based Kafka 4.0+) source.bootstrap.servers = source-kafka-1:9093,source-kafka-2:9093,source-kafka-3:9093 source.security.protocol = SASL_SSL source.sasl.mechanism = SCRAM-SHA-512 source.sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required \ username="mm2-source-user" \ password="${file:/etc/mm2/source-password.txt:password}"; source.ssl.truststore.location = /etc/mm2/truststore.jks source.ssl.truststore.password = ${file:/etc/mm2/truststore-password.txt:password} # Target cluster configuration (KRaft-based Kafka 4.0+) target.bootstrap.servers = target-kafka-1:9093,target-kafka-2:9093,target-kafka-3:9093 target.security.protocol = SASL_SSL target.sasl.mechanism = SCRAM-SHA-512 target.sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required \ username="mm2-target-user" \ password="${file:/etc/mm2/target-password.txt:password}"; target.ssl.truststore.location = /etc/mm2/truststore.jks target.ssl.truststore.password = ${file:/etc/mm2/truststore-password.txt:password} # Replication flow configuration source->target.enabled = true source->target.topics = orders.*, inventory.*, shipments.* source->target.groups = .* source->target.topics.blacklist = .*[\-\.]internal, .*\.replica, __.* # MirrorSourceConnector configuration source->target.emit.checkpoints.enabled = true source->target.emit.heartbeats.enabled = true source->target.sync.topic.configs.enabled = true source->target.sync.topic.acls.enabled = true # Exactly-once semantics (Kafka 2.5+) source->target.exactly.once.support = enabled source->target.transaction.timeout.ms = 900000 # Performance tuning tasks.max = 8 source->target.producer.compression.type = zstd source->target.producer.batch.size = 32768 source->target.producer.linger.ms = 100 source->target.producer.buffer.memory = 67108864 source->target.consumer.max.poll.records = 2000 source->target.consumer.fetch.min.bytes = 1048576 # Internal topic configuration replication.factor = 3 offset-syncs.topic.replication.factor = 3 checkpoints.topic.replication.factor = 3 heartbeats.topic.replication.factor = 3 config.storage.replication.factor = 3 offset.storage.replication.factor = 3 status.storage.replication.factor = 3 # Offset sync configuration refresh.topics.interval.seconds = 60 sync.topic.configs.interval.seconds = 60 emit.checkpoints.interval.seconds = 30 emit.heartbeats.interval.seconds = 5 # Replication policy (use IdentityReplicationPolicy to avoid prefixes) replication.policy.class = org.apache.kafka.connect.mirror.DefaultReplicationPolicy replication.policy.separator = . ``` This production configuration demonstrates several critical features: - **Security**: Both clusters use SASL/SSL with SCRAM-SHA-512 authentication. Passwords are externalized using Kafka's configuration provider mechanism (the `${file:...}` syntax), preventing credential exposure. For detailed security configuration, see [Kafka Authentication: SASL, SSL, and OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). - **Exactly-Once Semantics**: Setting `exactly.once.support = enabled` activates transactional replication, ensuring that messages are replicated exactly once even during failures. This uses Kafka's transactional API under the hood, writing to the target cluster atomically. Note that exactly-once replication requires both clusters to support transactions (Kafka 2.5+) and adds some latency overhead due to transaction coordination. - **Performance Optimization**: The configuration uses zstd compression (superior compression ratios compared to gzip or snappy in Kafka 4.0+), increased batch sizes, and optimized fetch parameters. The `tasks.max = 8` setting allows parallel processing across partitions. - **Topic Filtering**: The blacklist pattern excludes internal Kafka topics and system topics from replication, preventing unnecessary overhead. Key configuration considerations include topic filtering (using regex patterns), replication factor settings, security credentials (when clusters use authentication), and performance tuning parameters like buffer sizes and number of tasks. For high-throughput environments, increasing the number of tasks allows MM2 to parallelize replication across multiple partitions and topics. MM2 can be deployed as a dedicated cluster or co-located with existing Kafka Connect infrastructure. Many organizations run MM2 on dedicated hardware close to the source cluster to minimize network latency and egress costs. ## Kubernetes Deployment Modern organizations increasingly deploy MirrorMaker 2 on Kubernetes for better resource management, automated scaling, and declarative infrastructure. Kubernetes deployments offer advantages including pod-level isolation, horizontal scaling, health monitoring, and integration with cloud-native observability stacks. **Deploying MM2 with Strimzi Operator**: The Strimzi Kafka Operator provides Kubernetes-native support for running both Kafka clusters and Kafka Connect (which powers MM2) as custom resources. For comprehensive Strimzi guidance, see [Strimzi Kafka Operator for Kubernetes](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes). Here's a Kubernetes manifest for deploying MM2 using Strimzi: ```yaml apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaMirrorMaker2 metadata: name: mm2-cluster namespace: kafka spec: version: 3.8.0 replicas: 3 connectCluster: "target" clusters: - alias: "source" bootstrapServers: source-kafka-bootstrap.source-ns:9093 config: config.storage.replication.factor: 3 offset.storage.replication.factor: 3 status.storage.replication.factor: 3 tls: trustedCertificates: - secretName: source-cluster-ca-cert certificate: ca.crt authentication: type: tls certificateAndKey: secretName: source-mm2-credentials certificate: user.crt key: user.key - alias: "target" bootstrapServers: target-kafka-bootstrap.kafka:9093 config: config.storage.replication.factor: 3 offset.storage.replication.factor: 3 status.storage.replication.factor: 3 ssl.cipher.suites: "TLS_AES_256_GCM_SHA384" ssl.enabled.protocols: "TLSv1.3" ssl.protocol: "TLSv1.3" tls: trustedCertificates: - secretName: target-cluster-ca-cert certificate: ca.crt authentication: type: tls certificateAndKey: secretName: target-mm2-credentials certificate: user.crt key: user.key mirrors: - sourceCluster: "source" targetCluster: "target" sourceConnector: tasksMax: 8 config: replication.factor: 3 offset-syncs.topic.replication.factor: 3 sync.topic.acls.enabled: "true" sync.topic.configs.enabled: "true" replication.policy.class: "org.apache.kafka.connect.mirror.IdentityReplicationPolicy" refresh.topics.interval.seconds: 60 checkpointConnector: tasksMax: 4 config: checkpoints.topic.replication.factor: 3 sync.group.offsets.enabled: "true" emit.checkpoints.interval.seconds: 30 heartbeatConnector: tasksMax: 1 config: heartbeats.topic.replication.factor: 3 topicsPattern: "orders.*|inventory.*|shipments.*" groupsPattern: ".*" resources: requests: memory: 4Gi cpu: 2000m limits: memory: 8Gi cpu: 4000m metricsConfig: type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: mm2-metrics-config key: metrics-config.yml template: pod: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - kafka-mirror-maker2 topologyKey: "kubernetes.io/hostname" ``` This configuration demonstrates several production best practices: - **High availability**: 3 replicas with pod anti-affinity to distribute across nodes - **Security**: TLS encryption and authentication for both clusters - **Performance**: 8 source connector tasks for parallelism across partitions - **Monitoring**: JMX Prometheus exporter integration for metrics - **Resource management**: Appropriate CPU and memory limits - **Topic synchronization**: ACL and configuration replication enabled **Alternative Helm Deployment**: For organizations not using Strimzi, standard Kafka Connect Helm charts can deploy MM2. The Bitnami Kafka Connect Helm chart works well: ```bash helm repo add bitnami https://charts.bitnami.com/bitnami helm install mm2-connect bitnami/kafka \ --set replicaCount=3 \ --set mode=connect \ --set connectConfig.connectorClass=org.apache.kafka.connect.mirror.MirrorSourceConnector ``` Then deploy MM2 connectors via the Connect REST API or Kubernetes ConfigMaps. **Resource Sizing**: For Kubernetes deployments, size MM2 pods based on replication throughput: - **Low throughput** (<100 MB/s): 2 CPU cores, 4GB RAM - **Medium throughput** (100-500 MB/s): 4 CPU cores, 8GB RAM - **High throughput** (>500 MB/s): 8+ CPU cores, 16GB+ RAM, consider vertical pod autoscaling Monitor JVM heap usage and adjust memory limits accordingly. MM2 is memory-intensive when replicating high-partition-count topics. ## Monitoring and Operational Considerations Effective monitoring is essential for maintaining healthy cross-cluster replication. The primary metric to track is replication lag, which measures how far behind the source cluster the target cluster has fallen. High replication lag can indicate network issues, insufficient MM2 capacity, or problems with the target cluster. **Modern Observability with Prometheus and OpenMetrics**: While MM2 traditionally exposes metrics via JMX, modern deployments (especially on Kubernetes) increasingly use Prometheus-based monitoring. The JMX Prometheus Exporter (shown in the Kubernetes example above) converts JMX metrics to Prometheus format. Key metrics to monitor include: ``` # Replication lag metrics kafka_connect_mirror_source_connector_record_age_ms{cluster="source", topic="orders"} > 10000 kafka_connect_mirror_source_connector_replication_latency_ms{cluster="source"} > 5000 # Throughput metrics kafka_connect_mirror_source_connector_byte_rate{cluster="source"} kafka_connect_mirror_source_connector_record_count{cluster="source"} # Error metrics kafka_connect_mirror_source_connector_failed_record_count{cluster="source"} kafka_connect_task_error_total{connector="MirrorSourceConnector"} # Checkpoint lag kafka_connect_mirror_checkpoint_connector_checkpoint_latency_ms > 30000 ``` Set up Prometheus alerts for replication lag exceeding SLA thresholds (e.g., >60 seconds for real-time use cases) and failed record counts. For comprehensive Kafka monitoring patterns, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). **Conduktor Platform for Multi-Cluster Management**: Conduktor provides enterprise-grade visibility and management capabilities specifically designed for multi-cluster Kafka environments. Conduktor's replication monitoring features include: - **Cross-cluster topology visualization**: Visual maps showing replication flows between clusters, making it easy to understand replication topology and identify bottlenecks - **Replication lag dashboards**: Real-time lag tracking with historical trends, alerting when lag exceeds configured thresholds - **Consumer offset tracking**: View source and target offset mappings for failover planning - **Configuration management**: Centralized interface for managing MM2 connector configurations across multiple environments - **Topic comparison**: Compare topic configurations, partition counts, and data between source and target clusters - **Governance and compliance**: Audit trails for replication changes and data movement tracking for regulatory requirements For disaster recovery scenarios, Conduktor's failover simulation capabilities let teams validate that consumer offset translations work correctly before actual outages occur. This is invaluable for building confidence in your DR strategy. **Operational Procedures and Failover**: Coordinating failover procedures requires careful planning. When failing over to a backup cluster, teams must ensure that consumer groups start from the correct offsets and that producers switch to the new cluster atomically to avoid data inconsistencies. A typical failover procedure includes: 1. **Detect primary cluster failure** through monitoring alerts 2. **Verify target cluster health** and replication lag (ideally <1 minute) 3. **Stop producers** writing to the primary cluster (or redirect via DNS/load balancer) 4. **Wait for replication to catch up** to minimize data loss 5. **Query checkpoint topic** to retrieve translated consumer offsets 6. **Reset consumer groups** to translated offsets on target cluster 7. **Start consumers** on target cluster 8. **Redirect producers** to target cluster 9. **Verify data flow** through both consumer and producer metrics Automate this procedure where possible using orchestration tools (Kubernetes operators, Ansible playbooks) to reduce recovery time objectives (RTO). - **Operational Challenges**: Managing topic configuration drift between clusters, handling schema evolution, and coordinating failover procedures remain key challenges. When using Schema Registry, synchronize schemas between clusters using Schema Registry replication or backup/restore procedures. For schema management best practices, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). Topic configuration drift (different retention policies, partition counts, compression settings) can occur over time, regularly audit and reconcile configurations to prevent issues during failover. Understanding Kafka's built-in replication mechanisms helps optimize MM2 configurations; see [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability) for details. - **Network Bandwidth and Cost Optimization**: Replicating high-throughput topics across cloud regions can incur significant egress charges. Organizations should monitor bandwidth usage and consider: - **Compression**: Use zstd compression (Kafka 4.0+) for superior compression ratios - **Topic filtering**: Only replicate business-critical topics, exclude logs and debugging topics - **Dedicated network links**: Use AWS Transit Gateway, Azure ExpressRoute, or GCP Interconnect for predictable costs - **Strategic cluster placement**: Co-locate clusters in the same region when possible, use multi-AZ before multi-region - **Batch vs. real-time**: Consider whether all topics need real-time replication or if hourly/daily batches suffice For large-scale deployments, monitoring bandwidth consumption by topic helps identify cost optimization opportunities. ## MirrorMaker 2 in the Data Streaming Ecosystem Cross-cluster replication is a foundational capability for building resilient real-time data streaming architectures. In modern streaming platforms, data flows continuously through Kafka topics, feeding stream processing applications built with Kafka Streams or Apache Flink, populating real-time dashboards, and triggering automated business processes. When these streaming pipelines span multiple regions or require disaster recovery capabilities, MirrorMaker 2 becomes an essential component. It ensures that the streaming data infrastructure remains available even during regional outages, supports compliance requirements for data sovereignty, and enables global applications to access data with minimal latency. MM2 integrates naturally with the broader Kafka ecosystem. Since it's built on Kafka Connect, it works alongside other connectors that move data in and out of Kafka. Stream processing applications can consume from replicated topics without modification, and schema registries can be synchronized using separate replication strategies to maintain compatibility across clusters. For organizations running complex streaming platforms with multiple Kafka clusters, Connect clusters, stream processing applications, and data pipelines, managing the entire topology becomes challenging. This is where unified platform solutions can help by providing centralized visibility, governance, and operational controls across the distributed streaming infrastructure. ## Summary Kafka MirrorMaker 2 provides enterprise-grade cross-cluster replication capabilities that are essential for modern data streaming architectures. Its Connect-based architecture, support for multiple replication patterns, and features like offset synchronization make it significantly more capable than its predecessor. Whether implementing disaster recovery with active-passive replication, enabling global applications with active-active patterns, or aggregating data from distributed edge clusters, MM2 offers the flexibility and reliability needed for production deployments. Successful MM2 implementations require careful attention to configuration, monitoring, and operational procedures, particularly around failover scenarios. As organizations continue to adopt multi-region, multi-cloud, and hybrid cloud architectures, tools like MirrorMaker 2 become increasingly important for maintaining data availability, consistency, and performance across distributed streaming platforms. ## Related Concepts - [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters) - Comprehensive DR planning including RTO/RPO considerations, backup strategies, and failover procedures. - [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) - Deploy MirrorMaker 2 alongside Kafka clusters in containerized environments for cloud-native replication. - [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments) - Replicate multi-tenant clusters while maintaining isolation and security policies across regions. ## Sources and References 1. [Apache Kafka Documentation - MirrorMaker 2.0](https://kafka.apache.org/documentation/#georeplication) - Official Apache Kafka documentation covering MM2 architecture and configuration 2. [KIP-382: MirrorMaker 2.0](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0) - Kafka Improvement Proposal that introduced MirrorMaker 2 3. [Kafka Connect Documentation](https://kafka.apache.org/documentation/#connect) - Framework documentation for understanding MM2's underlying architecture 4. [Apache Kafka Operations Guide](https://kafka.apache.org/documentation/#operations) - Operational best practices for monitoring and managing Kafka deployments 5. [Strimzi Kafka Operator Documentation](https://strimzi.io/docs/operators/latest/overview.html) - Kubernetes-native Kafka deployment including MirrorMaker 2 support Source: https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication --- # Kafka Partitioning: 5 Strategies Compared **Kafka partitioning** assigns each producer message to one partition within a topic, determining storage placement, consumer assignment, and ordering guarantees. The strategy chosen controls load distribution and whether related messages land together. The five main strategies — key-based, round-robin, sticky, custom, and co-partitioning — each trade off ordering, throughput, and hot-partition risk. ## What is Kafka Partitioning? For foundational understanding of how partitions fit into Kafka's architecture, see [Kafka Topics, Partitions, Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). ![Kafka partitioning strategies and consumer group mapping](https://www.conduktor.io/assets/images/glossary/kafka-partitioning-strategies-and-best-practices-0.webp) ## Partitioning Strategies in Kafka Kafka provides several partitioning strategies that determine how records are assigned to partitions. ### Key-Based Partitioning When a producer sends a message with a key, Kafka uses a hash of the key to determine the partition. The default partitioner uses the murmur2 hash algorithm: `murmur2(key) % number_of_partitions`. This ensures that all messages with the same key always go to the same partition, preserving ordering for that key. **Important:** This hash-to-partition mapping is deterministic but becomes invalid if you add partitions to an existing topic. Messages with the same key may route to different partitions after repartitioning, breaking ordering guarantees. For example, if you're processing user events and use the user ID as the key, all events for user "12345" will be routed to the same partition. This is crucial for stateful processing where you need to maintain user-specific state or ensure events are processed in order. ### Round-Robin Partitioning When no key is provided (null key), Kafka 2.4+ uses a **sticky partitioner** by default. Instead of true round-robin, the sticky partitioner batches multiple messages to the same partition before switching, which improves throughput by enabling better batching and compression. Messages are still distributed evenly across partitions over time, but with better performance characteristics. This strategy maximizes throughput and ensures balanced load distribution but provides no ordering guarantees. ### Custom Partitioners Kafka allows you to implement custom partitioners by extending the `Partitioner` interface. This is useful when you have specific business logic for data distribution. For instance, you might want to route high-priority messages to specific partitions or use composite keys for partitioning decisions. Here's an example of a custom partitioner that routes high-priority messages to partition 0 and distributes normal messages using standard hashing: ```java import org.apache.kafka.clients.producer.Partitioner; import org.apache.kafka.common.Cluster; import org.apache.kafka.common.utils.Utils; public class PriorityPartitioner implements Partitioner { @Override public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) { int numPartitions = cluster.partitionCountForTopic(topic); // Route high-priority messages to partition 0 if (value instanceof PriorityMessage && ((PriorityMessage) value).getPriority() == Priority.HIGH) { return 0; } // Use murmur2 hash for normal messages (skip partition 0) if (keyBytes == null) { return 1 + Utils.toPositive(Utils.murmur2(valueBytes)) % (numPartitions - 1); } return 1 + Utils.toPositive(Utils.murmur2(keyBytes)) % (numPartitions - 1); } @Override public void close() {} @Override public void configure(Map configs) {} } ``` To use your custom partitioner, configure it in the producer: ```java Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, PriorityPartitioner.class.getName()); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class.getName()); KafkaProducer producer = new KafkaProducer<>(props); ``` For more details on producer configuration and message sending patterns, see [Kafka Producers](https://www.conduktor.io/glossary/kafka-producers). ## Key Design Considerations ### Cardinality and Distribution **Cardinality** refers to the number of unique values in your partition key. The cardinality of your partition key significantly impacts performance. If you have too few unique keys (low cardinality), you'll end up with hot partitions that receive disproportionate traffic. If you have too many keys with uneven distribution (skewed cardinality), some partitions may be idle while others are overloaded. Consider an e-commerce system where you partition orders by customer ID. If a small number of customers generate most of the orders (think enterprise clients versus individual consumers), you'll create skewed partitions. A better approach might be to use order ID or a composite key that distributes load more evenly. **Key selection guidelines:** - **High, even cardinality:** User IDs, transaction IDs, device IDs (if evenly distributed) - **Low cardinality (avoid):** Boolean flags, status codes, country codes with uneven distribution - **Skewed cardinality (mitigate):** Tenant IDs where some tenants dominate traffic, consider salting or composite keys ### Ordering Guarantees Kafka only guarantees ordering within a partition, not across partitions. If strict global ordering is required, you must use a single partition, which severely limits scalability. More commonly, you need ordering for a specific entity (user, device, transaction), which key-based partitioning handles well. ### Consumer Parallelism The number of partitions determines the maximum parallelism for consumers. A consumer group can have at most one consumer per partition. If you have 10 partitions, adding an 11th consumer to the group won't increase parallelism. Plan your partition count based on your expected consumer scaling needs. For detailed coverage of consumer group behavior and partition assignment, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). ### Partition Assignment Strategies Kafka consumers use partition assignment strategies to determine which partitions each consumer in a group processes. Since Kafka 2.4+, the default strategy is **CooperativeStickyAssignor**, which enables incremental cooperative rebalancing: - **RangeAssignor** (legacy): Assigns contiguous partition ranges per topic, can cause imbalance across topics - **RoundRobinAssignor**: Distributes partitions evenly across consumers in round-robin fashion - **StickyAssignor** (Kafka 0.11+): Maximizes partition assignment stability during rebalancing, reducing state rebuilding - **CooperativeStickyAssignor** (Kafka 2.4+, default 3.0+): Like StickyAssignor but supports **cooperative rebalancing**, only affected partitions are reassigned, reducing rebalancing impact Cooperative rebalancing is a major improvement over the older "stop-the-world" approach. During rebalancing, consumers continue processing unaffected partitions, significantly reducing latency spikes and consumer lag during group membership changes. ## Common Partitioning Anti-Patterns ### Hot Partitions Hot partitions occur when data distribution is severely unbalanced. This happens with poor key selection (like using a binary flag as a partition key) or when certain keys naturally dominate traffic. Hot partitions create bottlenecks, increase latency, and can cause consumer lag. **Mitigation strategies for hot partitions:** 1. **Key Salting**: Append a random suffix to high-volume keys to distribute load ```java // Instead of: key = "enterprise-customer-123" String salt = String.valueOf(ThreadLocalRandom.current().nextInt(10)); String saltedKey = String.format("%s-%s", originalKey, salt); // Produces: "enterprise-customer-123-7" ``` 2. **Composite Keys**: Combine multiple attributes to improve distribution ```java // Instead of just tenant_id, combine with time window String key = String.format("%s:%s", tenantId, timestamp / 3600000); // hourly ``` 3. **Custom Partitioner with Load Awareness**: Implement partitioners that track and avoid hot partitions (requires external state/metrics) 4. **Time-Based Partitioning**: For time-series data, include time windows in keys to spread load 5. **Testing with Conduktor Gateway**: Use Conduktor Gateway, a Kafka proxy, to simulate partition traffic patterns and test hot partition scenarios before production deployment. For comprehensive information on Gateway capabilities and deployment, see the [Gateway Overview documentation](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/index) ### Choosing Too Few Partitions Starting with too few partitions limits future scalability. While you can add partitions later, this breaks the hash-based key-to-partition mapping, potentially disrupting ordering guarantees. It's better to slightly over-provision partitions initially. ### Choosing Too Many Partitions Excessive partitions increase memory overhead on brokers, slow down leader elections during failures, and can increase end-to-end latency. Each partition requires file handles and memory for buffering. **Kafka 4.0+ partition scaling guidance:** - Modern Kafka (3.0+) with KRaft mode dramatically improves metadata handling and leader election speed - With Kafka 4.0+ and proper tuning, clusters can handle significantly more partitions per broker than older versions - Rule of thumb: Start with under 4,000 partitions per broker for conservative deployments - Well-tuned Kafka 4.0+ clusters can exceed this with adequate resources (CPU, memory, disk I/O) - Monitor these limits: file descriptors (`ulimit -n`), heap memory for metadata, and controller performance **KRaft mode benefits for partitioning:** - Faster leader elections (milliseconds vs seconds with ZooKeeper) - Better metadata scalability for large partition counts - Reduced operational complexity for partition-heavy workloads For more on KRaft mode's improvements, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ### Ignoring Key Null Values Some developers accidentally send null keys, expecting them to maintain some ordering. In reality, null keys trigger sticky partitioner behavior (Kafka 2.4+) or round-robin distribution (older versions), distributing messages across all partitions without ordering guarantees. This can break ordering assumptions in downstream processing. **Best practice:** Be explicit about your partitioning strategy. If you need ordering, always provide a key. If you want maximum throughput without ordering, use null keys intentionally, not accidentally. ## Best Practices for Production Systems ### Partition Count Planning Calculate your initial partition count based on: - **Target throughput**: Modern Kafka (3.0+) can handle 50-100+ MB/s per partition with proper tuning (older estimates of 10-50 MB/s were conservative) - **Expected consumer parallelism**: Current and future needs (each consumer can process one partition max) - **Data retention requirements**: More partitions mean more segment files and storage overhead - **Rebalancing tolerance**: More partitions = more work during consumer group rebalancing - **Formula:** `max(throughput_requirement / per_partition_throughput, expected_max_consumers)` - **Worked example:** - Target throughput: 500 MB/s - Per-partition throughput: 50 MB/s (conservative for modern Kafka) - Expected max consumers: 30 - **Calculation:** `max(500/50, 30) = max(10, 30) = 30 partitions` - **Start conservatively higher:** Recommend 40-50 partitions for this scenario to allow for growth and uneven distribution. - **Note:** Partition throughput depends on message size, compression, replication factor, and disk I/O. Test with realistic workloads to determine your cluster's per-partition capacity. For comprehensive guidance on sizing Kafka clusters and planning partition capacity, see [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning). ### Choosing Partition Keys Select partition keys that: - Have high cardinality with even distribution - Align with your ordering requirements - Match your access patterns (queries often filter by this field) - Remain stable over time (avoid timestamps or ephemeral values) For a payment processing system, `transaction_id` is often better than `merchant_id` if some merchants process far more transactions than others. ### Handling Repartitioning When you must increase partition count, consider: - **New topic migration**: Create a new topic with the desired partition count and migrate consumers - **Dual-write patterns**: Use MirrorMaker 2 or custom producers to write to both old and new topics during transition, then switch consumers - **Kafka Streams repartitioning**: For Kafka Streams applications, use the `repartition()` operation to create intermediate topics with proper partition counts - **Documentation**: Document the partition count change for teams relying on ordering guarantees, keys will map to different partitions after the change **Example with Kafka Streams:** ```java StreamsBuilder builder = new StreamsBuilder(); KStream orders = builder.stream("orders"); // Repartition with custom partition count KStream repartitioned = orders .repartition(Repartitioned.with(Serdes.String(), orderSerde) .withNumberOfPartitions(50) .withName("orders-repartitioned")); ``` **Avoid decreasing partition counts**, Kafka doesn't support this operation. Instead, create a new topic and migrate. ### Testing Partition Distribution Before deploying to production, test your partitioning strategy with realistic data. Check for skew by measuring messages per partition and identifying hot partitions. **Testing approaches:** - **Unit tests**: Verify custom partitioners assign messages to expected partitions - **Load testing**: Use realistic data volumes to identify skew and hot partitions - **Partition monitoring**: Use Conduktor to visualize partition distribution and identify imbalances - **Chaos testing**: Use Conduktor Gateway to simulate partition failures and test rebalancing behavior Conduktor provides partition monitoring dashboards showing records per partition, bytes per partition, and consumer lag per partition to help identify imbalances before they impact production workloads. ## Monitoring and Troubleshooting ### Detecting Partition Skew Monitor these metrics to detect partition imbalance: - **Records per partition over time**: Track via JMX `kafka.log:type=Log,name=Size,topic=*,partition=*` - **Bytes per partition**: Monitor partition size growth rates - **Consumer lag per partition**: Use Kafka Lag Exporter or Conduktor for partition-level lag tracking - **Partition leader distribution across brokers**: Ensure leaders are balanced (affects write performance) - **Partition throughput**: Messages/sec and bytes/sec per partition **JMX metrics for partition monitoring:** ```properties # Partition size kafka.log:type=Log,name=Size,topic=,partition= # Log end offset (total records) kafka.log:type=Log,name=LogEndOffset,topic=,partition= # Partition message rate (via rate of LogEndOffset changes) ``` **Using Kafka Lag Exporter:** ```yaml # Reports partition-level lag as Prometheus metrics kafka_consumergroup_group_lag{partition="0",topic="orders"} 1523 ``` **Using Conduktor:** Conduktor provides partition monitoring dashboards with per-partition consumer lag visualization and alerts for lag threshold violations. Significant variance in these metrics (>20% deviation from mean) indicates skew that may require addressing through key redesign or custom partitioners. For comprehensive monitoring strategies and metrics collection, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). ### Consumer Lag Analysis Consumer lag often manifests differently across partitions when partitioning is suboptimal. If specific partitions consistently show higher lag, investigate whether those partitions receive disproportionate traffic or contain more complex records to process. **Partition-level lag monitoring:** - **Kafka Lag Exporter**: Exports partition-level lag to Prometheus for time-series analysis - **Conduktor**: Provides real-time partition lag visualization with alerts for lag threshold violations - **Kafka Consumer Groups CLI**: Check per-partition lag with `kafka-consumer-groups.sh --describe` ```bash # Check per-partition lag kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --describe --group my-consumer-group # Output shows lag per partition: # TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG # orders 0 12500 12500 0 # orders 1 8200 15800 7600 <- Hot partition # orders 2 12300 12400 100 ``` If partition 1 consistently shows high lag while others don't, this indicates either a hot partition (more data) or uneven processing complexity. Use partition-level monitoring to diagnose whether the issue is partitioning-related or consumer-specific. ### Rebalancing Considerations Partition rebalancing occurs when consumers join or leave a group. Understanding rebalancing behavior is crucial for partition strategy decisions. **Rebalancing metrics to monitor:** - **Rebalance frequency**: How often rebalances occur (should be rare in stable deployments) - **Rebalance duration**: Time to complete rebalancing (milliseconds with cooperative rebalancing, seconds with eager) - **Partitions revoked/assigned**: Track which partitions move during rebalancing **Kafka 2.4+ cooperative rebalancing benefits:** - Only affected partitions are reassigned (not all partitions) - Consumers continue processing unaffected partitions during rebalancing - Dramatically reduces "stop-the-world" pauses and consumer lag spikes **Common rebalancing issues:** - **Frequent rebalancing**: Insufficient partitions, aggressive session timeouts, or unstable consumers - **Long rebalancing duration**: Too many partitions, large state stores (Kafka Streams), or network issues - **Session timeout misconfigurations**: `session.timeout.ms` too low causes false positives **Configuration for stable rebalancing:** ```properties # Use cooperative-sticky assignor (Kafka 2.4+) partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor # Increase session timeout for stability (default: 45s) session.timeout.ms=60000 # Reduce max.poll.interval.ms if processing is fast max.poll.interval.ms=300000 ``` ## Related Concepts - [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) - Comprehensive guidance on designing topics with appropriate partition counts and replication factors. - [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) - Understanding how partition assignment affects consumer group parallelism and rebalancing. - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) - Essential metrics for detecting partition skew and monitoring distribution health. ## Summary Kafka partitioning is a powerful mechanism for achieving scalability and parallelism in distributed streaming systems. The key to success lies in understanding your data characteristics, access patterns, and ordering requirements before choosing a partitioning strategy. Start by selecting partition keys with high cardinality and even distribution, aligned with your ordering needs. Plan partition counts based on throughput requirements and expected consumer parallelism, erring slightly on the side of more partitions. Avoid common pitfalls like hot partitions, null keys, and extreme partition counts. Continuously monitor partition distribution and consumer lag to detect imbalances early. When issues arise, use monitoring tools to diagnose whether partitioning adjustments are needed. Remember that partitioning decisions have long-term implications, so invest time in testing and validation before deploying to production. By following these best practices, you'll build Kafka-based streaming applications that scale efficiently, maintain ordering guarantees where needed, and avoid performance bottlenecks as your system grows. ## Sources and References 1. **Apache Kafka Documentation** - [Kafka Partitioning](https://kafka.apache.org/documentation/#intro_topics) - Official documentation covering partition concepts and configuration 2. **Apache Kafka Documentation** - [Producer Configurations](https://kafka.apache.org/documentation/#producerconfigs) - Producer settings including partitioner configuration 3. **Kafka: The Definitive Guide** by Neha Narkhede, Gwen Shapira, and Todd Palino (O'Reilly, 2017) - Comprehensive coverage of partitioning strategies and production considerations 4. **Designing Data-Intensive Applications** by Martin Kleppmann (O'Reilly, 2017) - Distributed systems patterns including partitioning strategies 5. **Conduktor** - [Kafka Partition Management](https://www.conduktor.io) - Commercial platform for partition monitoring, analytics, and testing with Conduktor Gateway 6. **Kafka Lag Exporter** - [GitHub Repository](https://github.com/seglo/kafka-lag-exporter) - Open-source tool for partition-level lag monitoring Source: https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices --- # What is a Kafka Partition? How They Work A **Kafka partition** is an ordered, immutable sequence of records within a topic. Each partition is an independent log stored on a broker, identified by a zero-based integer. Partitions are the unit of parallelism, distribution, and replication in Kafka: they determine how many consumers can read simultaneously, how data is spread across brokers, and how Kafka achieves fault tolerance. ## What Is a Kafka Partition? Every [Kafka topic](https://www.conduktor.io/glossary/kafka-topics-explained) is split into one or more partitions at creation time. While a topic is the logical stream (e.g., `user-events`), a partition is the physical log (e.g., `user-events` partition 0, partition 1, partition 2). Each record in a partition receives a monotonically increasing integer called an **offset**. Offsets are per-partition, partition 0 has its own offset sequence starting at 0, independent of partition 1. ``` Topic: user-events (3 partitions) Partition 0: [offset 0][offset 1][offset 2][offset 3] Partition 1: [offset 0][offset 1][offset 2] Partition 2: [offset 0][offset 1][offset 2] ``` ![Kafka partition layout diagram](https://www.conduktor.io/assets/images/glossary/kafka-topics-partitions-brokers-core-architecture-0.webp) ## Ordering Guarantees Kafka guarantees strict ordering within a partition. Records written to partition 0 are always read in the same order they were written. There is **no ordering guarantee across partitions** of the same topic. This design choice is intentional: it trades global ordering for horizontal scalability. If your use case requires that all events for a given entity (e.g., a user, an order) are processed in order, route them to the same partition using a consistent key. ## Parallelism: How Partitions Enable Scale Within a consumer group, each partition is assigned to exactly one consumer at a time. This means the number of partitions is the maximum degree of parallelism for consumption: - 3 partitions → at most 3 consumers in a group can read simultaneously - 10 partitions → at most 10 parallel consumers - Adding a 4th consumer to a 3-partition topic leaves the 4th consumer idle **Parallelism for producers** is also partition-based: producers write to specific partitions (by key hash, explicit assignment, or round-robin), and throughput scales with the number of partitions that can be written in parallel. ## How Records Are Assigned to Partitions When a producer sends a message, Kafka determines the target partition using one of three strategies: 1. **Explicit partition**: the producer specifies `partition=2`, the record goes to partition 2 regardless of the key 2. **Key-based routing** (most common): Kafka hashes the record key and maps it to a partition: `hash(key) % numPartitions`. All records with the same key always land in the same partition, guaranteeing ordering per key 3. **Round-robin**: if no key is provided, records are distributed evenly across partitions for maximum throughput ```java // Key-based routing, all events for "user-123" go to the same partition ProducerRecord record = new ProducerRecord<>("user-events", "user-123", eventPayload); // Explicit partition ProducerRecord record = new ProducerRecord<>("user-events", 2, "user-123", eventPayload); ``` WARN: Changing the partition count after a topic is created breaks key-based routing for existing consumers, because `hash(key) % numPartitions` maps the same key to a different partition. Plan partition counts before going to production. ## Leader and Follower Replicas Each partition is replicated across multiple [brokers](https://www.conduktor.io/glossary/kafka-brokers-explained) for fault tolerance. At any point in time, one replica is the **leader** and the rest are **followers**: - Producers always write to the leader - Consumers by default read from the leader (fetch-from-follower is configurable for latency optimization in multi-AZ setups) - Followers continuously replicate from the leader ``` Partition 0 of topic "transactions": Leader: Broker 1 ← producers write here Follower: Broker 2 ← replicates from Broker 1 Follower: Broker 3 ← replicates from Broker 1 ``` ## In-Sync Replicas (ISR) The **in-sync replica set (ISR)** is the subset of replicas that have fully caught up with the leader's log within the threshold defined by `replica.lag.time.max.ms`. Only ISR members are eligible to become the new leader if the current leader fails. This is the mechanism behind Kafka's durability guarantee: configure `min.insync.replicas=2` on the topic and `acks=all` on the producer, and a write is only acknowledged after at least 2 ISR members have persisted it. A single broker failure cannot cause data loss under this configuration. For deep coverage of ISR management and replication strategies, see [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability). ## Choosing the Right Partition Count Partition count is set at topic creation and hard to change safely. Key considerations: **More partitions enable:** - Higher consumer parallelism - Better throughput distribution across brokers - Finer-grained load balancing **Too many partitions cause:** - Longer leader election times during broker failures - Higher memory overhead (each partition has open file handles and in-memory state on the broker) - Increased latency for end-to-end replication with many followers - Higher metadata overhead on the controller **A practical sizing formula:** ``` partitions = ceil(target_throughput_MB_s / single_partition_throughput_MB_s) ``` Single-partition throughput is typically 10–50 MB/s depending on hardware, record size, and compression. For most systems, 6–30 partitions per topic is a reasonable range. Start conservatively; you can add partitions later (with the ordering caveat above). For worked examples and partition key selection patterns, see [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). ## Partition Distribution Across Brokers When a topic is created, Kafka distributes partition replicas across brokers to balance load. A 3-partition, replication-factor-3 topic across 3 brokers looks like: ``` Partition 0: Leader=Broker1, Followers=Broker2, Broker3 Partition 1: Leader=Broker2, Followers=Broker1, Broker3 Partition 2: Leader=Broker3, Followers=Broker1, Broker2 ``` Each broker leads an equal share of partitions. If a broker fails, its partitions elect new leaders from the ISR. Kafka 4.0+ improves automatic partition leadership rebalancing after recovery. ## Partitions and Consumer Groups Consumer group membership is tracked per-partition. When a new consumer joins or an existing one leaves, Kafka triggers a **rebalance**: partitions are redistributed among active group members. During a rebalance, consumption pauses briefly. Cooperative rebalancing (available since Kafka 2.4 via `CooperativeStickyAssignor`) minimizes this pause by only reassigning partitions that need to move, rather than revoking all assignments. See [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) for the full consumer coordination model. ## Related Pages - [What is a Kafka Topic?](https://www.conduktor.io/glossary/kafka-topics-explained), the logical container that partitions belong to - [What is a Kafka Broker?](https://www.conduktor.io/glossary/kafka-brokers-explained), where partitions are physically stored and replicated - [Kafka Architecture: Diagram & Components](https://www.conduktor.io/glossary/kafka-architecture-diagram), how partitions fit into the overall cluster design - [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices), key selection, custom partitioners, and sizing - [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability), ISR, replication factor, and failover behavior - [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained), how partitions are assigned to consumers ## Sources - [Apache Kafka 4.0 Documentation, Core Concepts](https://kafka.apache.org/documentation/#intro_concepts_and_terms) - [Apache Kafka, Replication](https://kafka.apache.org/documentation/#replication) - [Apache Kafka, Topic and Partition Configuration](https://kafka.apache.org/documentation/#topicconfigs) - [Kafka: The Definitive Guide, 2nd Edition](https://www.confluent.io/resources/kafka-the-definitive-guide/), Chapter 5: Kafka Internals Source: https://www.conduktor.io/glossary/kafka-partitions-explained --- # Kafka Performance Tuning: Config Cheatsheet Achieving optimal Kafka performance requires careful tuning of producers, brokers, and consumers. A poorly configured Kafka cluster can suffer from high latency, low throughput, or resource exhaustion, while a well-tuned deployment can handle massive data volumes efficiently. Before tuning, learn to [measure Kafka latency at the p99 and p999 tails](https://www.conduktor.io/blog/kafka-latency) rather than the misleading average. Modern Kafka deployments benefit from significant architectural improvements. Kafka 4.0+ with KRaft mode (removing ZooKeeper dependency) delivers 30-40% faster metadata operations and lower latency. Tiered storage enables cost-effective data retention by offloading cold data to object storage. Understanding these modern capabilities is essential for optimal performance in 2025. This guide covers the key areas for optimizing Kafka performance, from producer batching to broker disk I/O. For foundational understanding, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ![Kafka performance tuning areas and trade-offs](https://www.conduktor.io/assets/images/glossary/kafka-performance-tuning-guide-0.webp) ## Understanding Kafka Performance Fundamentals Kafka performance is measured across three primary dimensions: throughput (messages per second), latency (end-to-end message delivery time), and resource utilization (CPU, memory, disk, network). These metrics often involve trade-offs, maximizing throughput may increase latency, while reducing latency might require more resources. The performance of a Kafka deployment depends on multiple components working together. Producers must efficiently batch and compress messages. Brokers need adequate memory for page cache (the OS-level cache that stores recently accessed data in RAM to avoid disk I/O), properly configured replication, and optimized disk I/O. Consumers must fetch data in appropriate batch sizes and parallelize processing effectively. Understanding your workload characteristics is essential before tuning. Are you optimizing for real-time analytics with low latency requirements, or bulk data transfer where throughput matters most? Does your use case require strict ordering guarantees or can you parallelize consumption across partitions? The answers guide your configuration decisions. **KRaft Mode Performance Benefits**: Since Kafka 3.0 and matured in 4.0+, KRaft mode eliminates ZooKeeper as an external dependency, managing metadata directly within Kafka brokers. This architectural change delivers measurable performance improvements: metadata operations are 30-40% faster, partition leader elections complete in milliseconds instead of seconds, and clusters scale to millions of partitions more efficiently. For detailed coverage, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## Producer Performance Optimization Producers are often the first point of optimization. The `batch.size` parameter controls how many bytes of data are batched before sending to Kafka. The default is 16KB, but increasing to 32KB or 64KB can significantly improve throughput for high-volume producers by reducing network overhead. The `linger.ms` setting introduces a small delay before sending batches, allowing more messages to accumulate. Setting this to 10-100ms can dramatically increase batch sizes and throughput, with minimal latency impact for most use cases. For example, a producer sending 1000 messages per second might see throughput double with `linger.ms=20` while adding only 20ms of latency. Compression reduces network bandwidth and broker disk usage. The `compression.type` parameter supports `gzip`, `snappy`, `lz4`, and `zstd`. In Kafka 3.0+, `zstd` is the recommended default, offering superior compression ratios (20-30% better than `lz4`) with improved CPU efficiency on modern hardware. For latency-sensitive workloads, `lz4` remains excellent for its minimal CPU overhead. Avoid `gzip` in 2025 deployments unless compatibility with legacy systems requires it. For serialization format selection and schema management, see [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka) and [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). The `acks` parameter controls durability versus throughput trade-offs. Setting `acks=1` requires acknowledgment from only the leader broker, offering higher throughput but risking data loss if the leader fails before replication. Using `acks=all` ensures all in-sync replicas (ISR, brokers fully caught up with the leader) acknowledge the write, providing maximum durability at the cost of higher latency. For non-critical data streams, `acks=1` can significantly boost performance. **Producer Idempotence and Exactly-Once Semantics**: Since Kafka 3.0+, producer idempotence (`enable.idempotence=true`) is enabled by default, preventing duplicate messages from retries with minimal performance overhead (typically 1-3% throughput reduction). For transactional guarantees, configure `transactional.id` to enable exactly-once semantics. For detailed producer patterns, see [Kafka Producers](https://www.conduktor.io/glossary/kafka-producers). ## Broker Configuration Tuning Broker performance heavily depends on memory allocation for the operating system page cache. Kafka relies on the OS to cache recently written and frequently read data in RAM, avoiding expensive disk reads. Allocate at least 50% of server memory to page cache (outside the JVM heap) for optimal performance. **JVM and Garbage Collection Tuning**: Configure broker JVM heap between 5-8GB (more rarely improves performance and can increase GC pauses). In Kafka 4.0+ deployments, use G1GC as the default garbage collector with `-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35`. For ultra-low latency requirements, consider ZGC (`-XX:+UseZGC`) available in modern JVMs, which maintains sub-10ms pause times even under heavy load. The `num.replica.fetchers` parameter controls how many threads fetch replication data from leader brokers. Increasing from the default of 1 to 4-8 can improve replication throughput in clusters with high partition counts or cross-datacenter replication. For detailed replication architecture, see [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability). Log segment configuration affects both write performance and retention management. The `log.segment.bytes` parameter defaults to 1GB. Smaller segments (256MB) enable faster log compaction and deletion but create more files. Larger segments (2GB) reduce file system overhead but delay retention policy application. For log compaction strategies, see [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained). Thread pool sizing impacts concurrency. The `num.network.threads` parameter (default 3) handles network I/O, while `num.io.threads` (default 8) processes requests. For high-throughput clusters in 2025, increasing these to 8-12 and 16-24 respectively can reduce request queuing on modern multi-core hardware, though excessive threads cause context switching overhead. **Tiered Storage for Cost-Effective Performance**: Kafka 3.6+ introduces tiered storage, enabling brokers to offload cold data to object storage (S3, Azure Blob, GCS) while keeping hot data locally. Configure `remote.log.storage.system.enable=true` and specify retention policies with `local.retention.ms` and `retention.ms`. This dramatically reduces local disk requirements, a cluster storing 100TB can operate with only 10TB of local storage while maintaining full data access. Tiered storage introduces minimal latency for historical data queries (50-200ms additional latency) while cutting storage costs by 70-80%. For detailed configuration, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). ## Consumer Performance Optimization Consumer throughput depends largely on `fetch.min.bytes` and `fetch.max.wait.ms`. Setting `fetch.min.bytes=100000` (100KB) ensures consumers wait for substantial data before fetching, reducing request overhead. The `fetch.max.wait.ms=500` parameter limits wait time, balancing latency and batch size. Parallelism is critical for consumer performance. Kafka partitions enable parallel consumption, each partition is consumed by exactly one consumer in a group. If you have 12 partitions but only 3 consumers, increasing to 12 consumers can multiply throughput by 4x, assuming adequate processing capacity. For detailed consumer group mechanics, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). The `max.poll.records` parameter controls how many records are returned in a single poll. The default of 500 works for many cases, but batch-oriented processing might benefit from increasing to 2000-5000, while latency-sensitive applications might reduce to 100-200. Session timeouts and heartbeat intervals affect rebalance behavior (the process where consumers redistribute partition ownership). Setting `session.timeout.ms=30000` and `heartbeat.interval.ms=3000` provides stability for most workloads. Longer session timeouts tolerate temporary processing pauses but slow down failure detection, while shorter timeouts enable faster rebalancing at the risk of false positives (incorrectly marking healthy consumers as failed due to brief GC pauses or network delays). **Consumer Lag Monitoring**: Track consumer lag (the difference between the latest produced message and the consumer's position) as the primary indicator of consumption performance. In Kafka 4.0+, the improved consumer protocol reduces rebalance times by 50-70%, minimizing disruption during scaling operations. Use tools like Kafka Lag Exporter for Prometheus integration or Conduktor for comprehensive lag visualization and alerting. ## Network and Infrastructure Optimization Operating system tuning can significantly impact Kafka performance. Increasing TCP socket buffer sizes improves network throughput. Setting `net.core.rmem_max` and `net.core.wmem_max` to 16MB or higher on Linux allows Kafka to use larger buffers for high-bandwidth connections. Disk I/O configuration matters enormously. Kafka performs primarily sequential writes, making it well-suited to both SSDs and properly configured HDDs. Modern deployments (2025) increasingly use NVMe SSDs, delivering 3-5x better throughput and lower latency than SATA SSDs. RAID 10 offers good performance for HDDs with redundancy, while JBOD (Just a Bunch of Disks) configurations work well for Kafka since it handles replication at the application level, eliminating the need for hardware-level redundancy. File system selection affects performance and reliability. XFS is generally recommended over ext4 for Kafka workloads due to better performance with large files and more efficient handling of concurrent writes. The `noatime` mount option eliminates unnecessary metadata updates, improving write performance by 5-10%. Network topology impacts latency and bandwidth. Modern Kafka clusters benefit from high-bandwidth networking, 25Gbps or 100Gbps in large deployments, minimizing bottlenecks during peak traffic. Cross-datacenter replication benefits from dedicated network links and careful tuning of `replica.lag.time.max.ms` to account for network latency. For multi-region deployments, see [Kafka MirrorMaker 2 for Cross-Cluster Replication](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication). ## Monitoring and Measuring Performance Effective performance tuning requires comprehensive monitoring. Key broker metrics include request latency percentiles (p50, p95, p99), bytes in/out rates, under-replicated partitions, and ISR shrink rate (indicating replication lag). Producer metrics like batch size average, compression ratio, and request latency reveal batching effectiveness. Consumer lag is the critical metric for consumption performance. **Modern Monitoring Approaches**: In 2025, Kafka monitoring relies on multiple tools and approaches: - **Native Kafka Metrics**: Kafka 4.0+ includes improved metrics exposure via JMX and native OpenTelemetry support, integrating directly with modern observability platforms - **Prometheus and Grafana**: JMX exporters expose Kafka metrics to Prometheus, with Grafana dashboards visualizing performance trends - **Conduktor Platform**: Purpose-built for Kafka, Conduktor provides specialized monitoring with pre-built dashboards for performance metrics, configuration drift detection, cluster health visualization, and performance bottleneck identification - **Conduktor Gateway**: For performance testing and chaos engineering, Conduktor Gateway acts as a proxy layer enabling sophisticated testing scenarios like network delay injection, partition throttling, and failure simulation For comprehensive cluster monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics). Load testing validates tuning changes. Tools like `kafka-producer-perf-test` and `kafka-consumer-perf-test` generate synthetic workloads to measure throughput and latency under controlled conditions. Testing different configurations helps identify optimal settings for your specific hardware and workload. Conduktor Gateway enables advanced performance testing scenarios including chaos engineering experiments to validate performance under degraded conditions. Profiling real workloads provides insights that synthetic tests cannot. Analyzing actual producer batching behavior, broker request queue times, and consumer processing rates reveals where bottlenecks occur in production scenarios. ## Common Performance Scenarios Consider an e-commerce platform processing clickstream events. Initially, producers send each click individually with `acks=all`, achieving only 5,000 messages per second with 50ms latency. By implementing `batch.size=32768`, `linger.ms=10`, `compression.type=lz4`, and `acks=1` (acceptable for analytics data), throughput increases to 25,000 messages per second with latency under 20ms. Another common scenario involves consumer lag during traffic spikes. A trading platform processes market data through 24 partitions but runs only 8 consumer instances. During high-volume periods, lag grows to millions of messages. Scaling to 24 consumers eliminates the bottleneck, and tuning `fetch.min.bytes=50000` reduces network overhead, bringing lag back to near-zero even during peaks. Over-replication can also degrade performance. A small development cluster configured with `replication.factor=3` across only 3 brokers experiences poor write performance due to network saturation from replication traffic. Reducing to `replication.factor=2` for non-critical topics frees bandwidth and improves throughput by 40%. For comprehensive topic configuration guidance, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) and [Kafka Topics, Partitions, Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). ## Performance Tuning Quick Reference For common performance scenarios, start with these tuning priorities: **High Throughput (Batch Processing)**: - Increase `batch.size` to 64KB-128KB - Set `linger.ms` to 50-100ms - Use `compression.type=zstd` - Enable tiered storage for long retention - Scale consumers to match partition count **Low Latency (Real-Time Processing)**: - Reduce `batch.size` to 8KB-16KB - Set `linger.ms` to 0-5ms - Use `compression.type=lz4` - Tune `fetch.min.bytes` to smaller values - Deploy on NVMe storage with high-bandwidth networking - **Capacity Planning**: Performance tuning works hand-in-hand with capacity planning. For comprehensive guidance on sizing clusters and predicting resource requirements, see [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning). - **Security Considerations**: Performance tuning must balance speed with security. Authentication (SASL, SSL, OAuth) and encryption (TLS) introduce overhead but are essential for production deployments. For guidance on minimizing security-related performance impact, see [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth) and [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices). ## Summary Kafka performance tuning is a multifaceted challenge requiring optimization across producers, brokers, consumers, and infrastructure. Producer tuning focuses on batching and compression to maximize throughput, while broker configuration emphasizes memory allocation, thread pools, and disk I/O. Consumer optimization centers on parallelism and fetch sizing. Modern Kafka deployments (2025) benefit from architectural improvements including KRaft mode for faster metadata operations, tiered storage for cost-effective retention, and enhanced monitoring capabilities. Understanding these 2025 features is essential for building high-performance streaming platforms. The key to successful tuning is understanding your workload requirements, measuring current performance comprehensively, making incremental changes, and validating results through monitoring and load testing. There is no one-size-fits-all configuration, optimal settings depend on your specific use case, hardware, and performance goals. By systematically applying these tuning strategies and continuously monitoring your Kafka deployment, you can build a high-performance streaming platform capable of handling demanding real-time data workloads. ## Related Concepts - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Translate performance requirements into cluster sizing decisions and resource allocation strategies. - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) - Establish monitoring to measure performance improvements and identify new bottlenecks. - [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership) - Balance performance optimization with cost considerations across infrastructure, operations, and maintenance. ## Sources and References 1. Apache Kafka Documentation - "Performance Tuning": https://kafka.apache.org/documentation/#performance 2. Confluent - "Kafka Performance Tuning Guide": https://docs.confluent.io/platform/current/kafka/deployment.html#performance-tuning 3. Jun Rao (LinkedIn) - "How to choose the number of topics/partitions in a Kafka cluster": https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ 4. DataDog - "Monitoring Kafka performance metrics": https://www.datadoghq.com/blog/monitoring-kafka-performance-metrics/ 5. Netflix Tech Blog - "Evolution of the Netflix Data Platform": https://netflixtechblog.com/evolution-of-the-netflix-data-pipeline-da246ca36905 Source: https://www.conduktor.io/glossary/kafka-performance-tuning-guide --- # Kafka Producers and Consumers Apache Kafka is designed as a distributed, fault-tolerant commit log. While Kafka brokers manage storage and replication, the actual transfer of data relies on two critical client components: producers and consumers. These client applications determine the data quality, latency, throughput, and reliability of modern data streaming architectures. Understanding how producers and consumers work, their configuration options, delivery guarantees, and operational characteristics, is essential for building scalable, reliable streaming systems. For foundational architecture concepts, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) and [Kafka Topics, Partitions, Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). ![Kafka producers and consumers architecture and flow](https://www.conduktor.io/assets/images/glossary/kafka-producers-and-consumers-0.webp) ## Producers and Consumers: The Communication Protocol Kafka producers and consumers are core client applications that mediate the flow of data between application layers and the Kafka cluster. They interact using the Kafka protocol, which enables a completely asynchronous and decoupled architecture. A **producer** is an application that serializes and writes records (messages) to a Kafka topic. Producers are responsible for converting application data structures into bytes, determining which partition receives each record, and managing delivery guarantees through configurable acknowledgment mechanisms. A **consumer** is an application that reads records sequentially from Kafka topic partitions. Consumers track their reading position through offsets, can operate independently or as part of a consumer group for parallel processing, and control when to commit their progress back to Kafka. This decoupling is fundamental: producers don't need to know who is consuming the data, and consumers don't need to know who produced it. This separation allows applications to scale independently, evolve at different rates, and recover from failures without affecting each other. ## How Kafka Producers Work: Record Delivery and Guarantees The producer is responsible for reliably packaging, routing, and delivering messages to the appropriate Kafka topic partition. ### Producer Responsibilities - **Serialization**: Producers convert application data structures (JSON objects, Avro records, Protobuf messages) into byte arrays before sending them over the network. A Kafka record consists of a key (optional), value, headers (optional metadata), and timestamp. Common serializers include StringSerializer, ByteArraySerializer, and schema-aware serializers like Avro or Protobuf that integrate with Schema Registry for structured data validation. For detailed coverage of serialization formats and trade-offs, see [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka) and [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). - **Partitioning**: Producers determine which partition receives each record. If a key is provided, Kafka uses consistent hashing to map the key to a partition: ``` partition = hash(key) % number_of_partitions ``` This ensures all records with the same key go to the same partition, preserving ordering for that key. If no key is specified, the default partitioner (since Kafka 2.4+) uses a **sticky partitioning strategy**, filling batches by sending records to the same partition until the batch is full, then switching to another partition. This "sticky" behavior significantly improves throughput by creating larger, more efficient batches compared to round-robin partitioning, which scattered records across all partitions and created many small batches. **Buffering and Batching**: Producers don't send records individually. Instead, they accumulate records in memory buffers and send them to brokers in efficient batches, significantly improving throughput by amortizing network and disk I/O overhead. Key batching configurations include: - `batch.size`: Maximum bytes per batch (default: 16 KB) - `linger.ms`: Maximum time to wait before sending a batch (default: 0) Setting `linger.ms` to 5-10 milliseconds allows the producer to collect more records per batch, increasing throughput with minimal latency impact. Producers can also compress batches using algorithms like gzip, snappy, lz4, or zstd. ### Delivery Semantics and Acknowledgments A critical configuration is the acknowledgment (`acks`) setting, which determines the delivery guarantee: - **acks=0 (At-Most-Once)**: Producer doesn't wait for any acknowledgment from brokers. This provides the fastest throughput but risks data loss if the broker fails before persisting the record. - **acks=1 (At-Least-Once)**: Producer waits for the leader partition replica to acknowledge the write. This guarantees delivery but may result in duplicate records if the producer retries after a network failure and the original write actually succeeded. - **acks=all or -1 (Exactly-Once Foundation)**: Producer waits for acknowledgment from all in-sync replicas (ISRs). Since Kafka 3.0+, idempotent producers are enabled by default (`enable.idempotence=true`), which effectively requires `acks=all` for data safety. Combined with transactional IDs, this enables exactly-once processing, ensuring records are written once and only once, even after failures. For critical data, use `acks=all` combined with `min.insync.replicas=2` to ensure at least one replica has received the data before acknowledgment, protecting against data loss. ### Example Producer Configuration (Java) ```java Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); // Modern production settings (Kafka 3.0+) props.put("acks", "all"); props.put("enable.idempotence", "true"); // Default since 3.0+ props.put("compression.type", "lz4"); props.put("batch.size", 32768); // 32 KB batches props.put("linger.ms", 10); // Wait up to 10ms for batching KafkaProducer producer = new KafkaProducer<>(props); // Send asynchronously with callback producer.send(new ProducerRecord<>("orders", "key123", "order data"), (metadata, exception) -> { if (exception != null) { // Handle send failure } else { // Record sent to partition metadata.partition() at offset metadata.offset() } }); ``` ## How Kafka Consumers Work: Tracking Progress with Offsets Consumers read records from Kafka partitions by subscribing to one or more topics and pulling data from assigned partitions. Unlike push-based messaging systems, Kafka's pull model allows consumers to control their read rate and handle backpressure naturally. For strategies on managing consumer processing rates, see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems). ### Sequential Reading and Offset Management Consumers read records sequentially based on their **offset** within a partition, a monotonically increasing identifier that serves as the consumer's position in the log. Each partition maintains its own offset sequence independently. - **Offset Commitment**: A consumer periodically commits its latest successfully processed offset back to Kafka, specifically to an internal topic named `__consumer_offsets`. This commit serves as a checkpoint, indicating which records have been successfully processed. In Kafka 4.0+ with KRaft mode (Kafka's ZooKeeper-free architecture), offset management remains identical from a client perspective, but the underlying coordination is handled by Kafka's internal Raft-based consensus protocol. For details on KRaft's architecture, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). - **Failure Recovery**: If a consumer instance fails or restarts, it retrieves the last committed offset from `__consumer_offsets` and resumes reading from the next record. This ensures continuous processing, though it may result in reprocessing records if the consumer failed between processing and committing. Offset commits can be: - **Automatic** (`enable.auto.commit=true`): Commits occur at regular intervals configured by `auto.commit.interval.ms` (default: 5 seconds). Simple to implement but may cause duplicate processing if the consumer fails between the automatic commit and actual record processing. - **Manual**: Application explicitly commits after successfully processing records, providing precise control over delivery semantics. ### Example Consumer Configuration (Java) ```java Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("group.id", "order-processing-group"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); // Modern production settings props.put("enable.auto.commit", "false"); // Manual commit for precise control props.put("max.poll.records", 500); // Process up to 500 records per poll props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.CooperativeStickyAssignor"); // Incremental cooperative rebalancing KafkaConsumer consumer = new KafkaConsumer<>(props); consumer.subscribe(Arrays.asList("orders")); while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord record : records) { // Process record processOrder(record.value()); } // Commit offsets after successful processing consumer.commitSync(); } ``` ### Consumer Groups: Scaling Read Concurrency A **consumer group** is a set of consumers that share a common `group.id`. For a given topic, the partitions are divided among the consumer instances in that group, enabling parallel processing while maintaining order within each partition. - **Exclusive Partition Access**: Each partition is assigned to exactly one consumer instance within the group at any time. This prevents duplicate processing within the group and ensures each record is processed by only one consumer. - **Rebalancing**: When a consumer instance joins or leaves a group (due to scaling, failure, or deployment), or if the topic's partitions change, a group rebalance occurs. The partitions are redistributed among active members, ensuring high availability. Since Kafka 2.4+, **incremental cooperative rebalancing** is the default protocol (replacing the older eager rebalancing). With cooperative rebalancing, consumers only give up partitions that need to be reassigned, continuing to process their remaining partitions during the rebalance. This significantly reduces processing interruptions compared to eager rebalancing, where all consumers stopped processing until the rebalance completed. For example, if a topic has six partitions and a consumer group has three consumers, each consumer handles two partitions. Adding a fourth consumer triggers a rebalance, redistributing partitions so each consumer handles 1-2 partitions. However, adding more consumers beyond the partition count provides no additional concurrency, extra consumers remain idle. ## Delivery Semantics: At-Most-Once, At-Least-Once, and Exactly-Once Kafka supports three delivery guarantee models, each representing different trade-offs between reliability, performance, and complexity. ### At-Most-Once Records may be lost but are never duplicated. Achieved by: - Producer: `acks=0` or `acks=1` without retries - Consumer: Commit offsets before processing records Provides maximum throughput and lowest latency but risks data loss. Suitable for non-critical telemetry or monitoring data where occasional loss is acceptable. ### At-Least-Once Records are never lost but may be processed multiple times. Achieved by: - Producer: `acks=all` with retries enabled - Consumer: Commit offsets after successfully processing records Most common semantic in production systems, ensuring no data loss. Applications must implement idempotent processing logic to handle potential duplicates correctly. ### Exactly-Once Records are delivered and processed exactly once, even during failures. Kafka achieves this through: - Idempotent producers (`enable.idempotence=true`): Prevents duplicate writes during retries - Transactional producers: Atomic writes across multiple partitions - Transactional consumers: Read only committed messages (`isolation.level=read_committed`) Exactly-once semantics require careful configuration but provide the strongest guarantees for financial transactions, inventory updates, or billing systems where duplicates or data loss cause incorrect results. For comprehensive coverage of transactional mechanics, see [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). ## Producers and Consumers in Streaming Architectures The producer-consumer model powers real-time data pipelines and advanced stream processing across modern architectures: - **Change Data Capture Pipelines**: Tools like Debezium act as specialized producers (Kafka Connect Source Connectors), continuously capturing database change events and writing them into Kafka topics, enabling real-time replication and analytics. For detailed coverage of CDC patterns, see [What is Change Data Capture (CDC) Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). - **Stream Processing Engines**: Frameworks like Apache Flink and Kafka Streams are built on the consumer API. They read records from topics, perform complex stateful transformations (aggregations, joins, windowing), and write results back using the producer API. - **Sink Connectors and Data Integration**: Kafka Connect Sink Connectors act as specialized consumers, reliably reading data from topics and writing it to final destinations like data warehouses (Snowflake, BigQuery), search indexes (Elasticsearch), or cloud storage (S3). - **Microservices Communication**: Services act as both producers and consumers, publishing domain events to topics and subscribing to events from other services. This event-driven architecture decouples services while maintaining consistency through event ordering and replay capabilities. ## Monitoring, Troubleshooting, and Governance at Scale As Kafka deployments grow, managing producer and consumer behavior becomes increasingly complex. Operational health monitoring must extend beyond Kafka brokers to the client applications themselves. ### Key Operational Challenges - **Consumer Lag**: The difference (in offsets or time) between the latest record written to a partition and the last record committed by a consumer group indicates processing delays. High consumer lag suggests consumers can't keep up, requiring optimization, additional consumer instances, or investigation of processing bottlenecks. Modern tools like Kafka Lag Exporter (with Prometheus/Grafana integration) provide automated lag monitoring and alerting. For comprehensive strategies and best practices, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). - **Producer Throughput and Error Rates**: Monitoring producer metrics like record send rate, batch size, compression ratio, and network errors helps identify performance bottlenecks. Misconfigured batching, serialization failures, or broker unavailability can significantly impact pipeline reliability. - **Consumer Group State**: Tracking group rebalancing frequency is essential, frequent rebalancing indicates instability due to consumer failures, network issues, or configuration problems. Each rebalance temporarily halts processing, impacting end-to-end latency. ### Client Governance and Access Control In enterprise environments, managing access for numerous client applications is a major governance challenge. Kafka uses ACLs (Access Control Lists) to manage permissions, allowing only specific client IDs to write to sensitive topics or limiting which consumer groups can read confidential data streams. For comprehensive security patterns and best practices, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices). Without centralized governance, tracking producer and consumer activity becomes challenging. Questions like "Which applications are producing to this topic?", "Who's consuming this sensitive data?", and "Why is this consumer group lagging?" require manual investigation across application logs and metrics systems. Platforms like **Conduktor** address these challenges by providing unified visibility and controls for Kafka environments. Teams can monitor consumer lag in real-time across all consumer groups, track which applications interact with each topic, enforce schema validation at the producer level, manage ACLs through a visual interface, and audit client access patterns for compliance. Learn how to [manage topics and monitor consumer groups](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) and [configure service accounts with proper ACLs](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/service-accounts-acls) for secure access control. When troubleshooting lag spikes, Conduktor immediately identifies which partitions are falling behind, correlates this with producer throughput changes or rebalancing events, and alerts the responsible team, enabling faster root cause analysis while maintaining a complete audit trail. For advanced testing scenarios like chaos engineering and protocol manipulation, Conduktor Gateway provides proxy-based capabilities to validate producer and consumer resilience. ## Related Concepts - [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) - Details how consumer groups coordinate partition assignment for parallel processing. - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Essential for managing serialization formats used by producers and consumers. - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Critical for detecting when consumers fall behind producers and ensuring pipeline health. ## Summary Kafka producers and consumers are the vital client-side components that implement the asynchronous and decoupled communication central to data streaming. Producers handle serialization, partition selection, batching, compression, and delivery guarantees through configurable acknowledgment mechanisms. Consumers coordinate through consumer groups to process partitions in parallel, manage offsets to track progress, and support multiple delivery semantics from at-most-once to exactly-once processing. The producer-consumer model manages critical functions like message delivery guarantees, sequential processing with ordering preservation, and high-throughput concurrency via consumer groups. This architecture enables the decoupled, scalable patterns that power real-time data platforms, allowing multiple independent consumers to process the same data for different purposes, supporting microservices communication without tight coupling, and connecting directly with stream processing frameworks. Teams focus on tuning client configuration parameters (acks, batch.size, linger.ms, auto.commit settings), implementing proper error handling and retry logic, aggressively monitoring client health, particularly consumer lag and producer error rates, and choosing appropriate delivery semantics based on data criticality. As deployments scale, proactive monitoring of client-side metrics becomes vital for detecting and resolving pipeline slowdowns. Enterprise governance platforms provide the management layer for controlling and auditing which client applications have permission to produce or consume data, maintaining security and compliance. These tools offer centralized monitoring of consumer lag, tracking of producer and consumer activity per topic, schema validation enforcement, and visual ACL management, bridging the gap between Kafka's low-level client mechanics and enterprise operational requirements. ## Sources and References - Apache Software Foundation. [Apache Kafka Documentation: Producer Configs](https://kafka.apache.org/documentation/#producerconfigs) - Apache Software Foundation. [Apache Kafka Documentation: Consumer Configs](https://kafka.apache.org/documentation/#consumerconfigs) - Confluent. [Transactions in Apache Kafka (Exactly-Once Semantics)](https://www.confluent.io/blog/transactions-apache-kafka/) - Confluent. [Apache Kafka Producer and Consumer Fundamentals](https://developer.confluent.io/courses/apache-kafka/producer-consumer-fundamentals/) - Debezium. [Architecture Overview](https://debezium.io/documentation/reference/architecture.html) Source: https://www.conduktor.io/glossary/kafka-producers-and-consumers --- # Kafka Producers A Kafka producer sends records to a topic and routes each record to a partition. The client connects to brokers, fetches metadata, and batches writes for throughput. Producers control delivery policy, retries, compression, and partition choice, which sets latency, cost, and durability. ![Kafka producer architecture and message flow](https://www.conduktor.io/assets/images/glossary/kafka-producers-0.webp) ## API Notes A record carries a topic, optional key, value, and optional headers. The `KafkaProducer` is thread safe, so one shared instance per process often works best. You can send asynchronously with a callback or call `Future.get()` for sync behavior. Set a clear `client.id`, since brokers expose it in logs and metrics and this speeds triage. ### Example: CLI quick start ``` kafka-console-producer --topic orders --bootstrap-server localhost:9092 \ --property parse.key=true --property key.separator=: ``` Each `key:value` line becomes a keyed record. ## Setup and Serialization Producers must serialize keys and values into bytes. A schema keeps changes safe across teams and time. Pick a format and a registry. Avro, Protobuf, and JSON Schema pair well with a Schema Registry that stores versions and enforces compatibility. For detailed serialization guidance, see [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka) and [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). Key properties: * `bootstrap.servers` for initial brokers. * `key.serializer` and `value.serializer` for encoding. * `client.id` for traceability. * `compression.type` to reduce bytes on the wire. * `delivery.timeout.ms`, `request.timeout.ms`, and `retries` for retry policy. ### Example: Java producer with Avro and callback ```java Properties p = new Properties(); p.put("bootstrap.servers", "broker1:9092,broker2:9092"); p.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); p.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer"); p.put("schema.registry.url", "http://registry:8081"); p.put("client.id", "payments-prod-v1"); // Note: enable.idempotence=true is the default since Kafka 3.0+ p.put("acks", "all"); // Ensures durability with in-sync replicas try (Producer producer = new KafkaProducer<>(p)) { ProducerRecord rec = new ProducerRecord<>("payments", "user-42", value); producer.send(rec, (md, ex) -> { if (ex != null) { // log error with client.id, topic, partition } else { // md.topic(), md.partition(), md.offset() } }); producer.flush(); // optional if you close soon after } ``` The try-with-resources block calls `close()`, which flushes in-flight records and releases resources. ## Delivery Guarantees and Acknowledgments Delivery behavior depends on acks, retries, idempotence, and transactions. * **At-most-once** sends do not retry. You can lose data on faults. * **At-least-once** sends retry. You can see duplicates. * **Exactly-once processing** uses idempotent producers and transactions with read-committed consumers to avoid both loss and duplicates in a pipeline. Key settings: * `acks=0` returns before storage. Lowest safety. * `acks=1` waits for the leader write (default). * `acks=all` waits for the leader and in-sync replicas. Use with topic `min.insync.replicas`. Required for exactly-once semantics. * `enable.idempotence=true` (default since Kafka 3.0+) drops duplicates created by retries within a producer session. Enables Producer ID (PID) and sequence numbers per partition. * Transactions use a stable `transactional.id`, `initTransactions`, `beginTransaction`, and `commitTransaction`, with consumers set to `isolation.level=read_committed`. Provides atomic multi-partition writes and exactly-once delivery. ### Transactional producer example For exactly-once semantics across multiple partitions or topics: ```java Properties p = new Properties(); p.put("bootstrap.servers", "broker1:9092"); p.put("transactional.id", "payment-processor-001"); // Stable unique ID p.put("enable.idempotence", "true"); // Required (default since 3.0+) p.put("acks", "all"); // Required for transactions try (KafkaProducer producer = new KafkaProducer<>(p)) { producer.initTransactions(); // One-time initialization try { producer.beginTransaction(); producer.send(new ProducerRecord<>("orders", "order-123", orderJson)); producer.send(new ProducerRecord<>("inventory", "item-456", inventoryUpdate)); producer.commitTransaction(); // Atomic commit across both topics } catch (Exception e) { producer.abortTransaction(); // Rollback on failure throw e; } } ``` Consumers must set `isolation.level=read_committed` to see only committed messages. The `transactional.id` ties the producer to a transaction coordinator; if the producer crashes, a new instance with the same ID can fence out the old one and continue. ### Error handling checklist Treat these as retriable: `NOT_LEADER_FOR_PARTITION`, `NETWORK_EXCEPTION`, `UNKNOWN_TOPIC_OR_PARTITION`, `REQUEST_TIMED_OUT`. Treat these as non-retriable: `INVALID_CONFIG`, `TOPIC_AUTHORIZATION_FAILED`, `RECORD_TOO_LARGE`. Route non-retriable failures to a dead-letter topic or alert the team, then fix the cause. Handle `NotEnoughReplicas` on `acks=all` with `min.insync.replicas` by backing off or failing fast. ### Safe write profile (Kafka 3.0+ defaults) Since Kafka 3.0, the following safe settings are enabled by default: ```properties acks=1 # Change to acks=all for stronger durability enable.idempotence=true # Default since 3.0+ retries=2147483647 # Max int, effectively unlimited max.in.flight.requests.per.connection=5 # Safe with idempotence delivery.timeout.ms=120000 # 2 minutes ``` - **For strongest durability**, explicitly set `acks=all` and ensure topic has `min.insync.replicas=2` or higher. - **For strict per-key ordering during retries**: With idempotence enabled (default), Kafka maintains partition order even with `max.in.flight.requests.per.connection=5`. Only set this to `1` if you observe out-of-order issues with older brokers or need absolute guarantees, accepting lower throughput. ## Keys, Partitions, and Ordering Kafka guarantees order per partition per producer session. Your key choice decides partition placement and thus locality. For partition strategy details, see [Kafka Topics, Partitions, Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). Practical rules: * Use a stable business key, for example `user_id` or `order_id`, to keep related events together. * Records without keys use a sticky strategy that forms larger batches while spreading load. * Hot keys can overload a single partition. Use a composite key, a salted hash for the hottest keys, or a custom partitioner. * Increasing partition count changes key-to-partition mapping. Plan the change during low traffic and confirm ordering needs. ### Quick check for routing Send a few `key:message` pairs with the console producer, then consume with partition info. You should see all records for a key in the same partition, in order. ## Batching, Backpressure, and Throughput Tuning The producer batches per partition to improve throughput. Larger batches raise efficiency and compression ratio at some latency cost. For handling downstream slowness and flow control, see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems). Levers that matter: * `batch.size` sets the target batch bytes (default 16384). * `linger.ms` lets the client wait briefly to coalesce records. Values around 1 to 5 ms help throughput while keeping tail latency reasonable. * `compression.type` trades CPU for bytes. **Zstd** (available since Kafka 2.1+) is recommended for new deployments in 2025, offering 20-30% better compression than gzip with comparable CPU usage. LZ4 works for speed with modest savings. Snappy is deprecated in favor of LZ4 or Zstd. * `buffer.memory` bounds the client buffer (default 32MB). When full, `send()` blocks up to `max.block.ms` or throws. * `max.request.size` (default 1MB) and broker `message.max.bytes` cap record size. Use chunking or external object storage for very large payloads. Watch `record-send-rate`, `record-error-rate`, `request-latency-avg`, p95 or p99 latency, `compression-rate-avg`, average batch size, and `bufferpool-wait-ratio`. Rising retries, timeouts, or buffer waits point to broker load or network issues. Tune, then measure again. ### Modern producer monitoring (2025) - **Kafka 4.0+ improvements**: Kafka 4.0 runs on KRaft (no ZooKeeper), which improves producer metadata fetch times and reduces tail latencies. Enhanced producer metrics expose per-partition batch stats and compression ratios. For KRaft migration details, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) and [ZooKeeper to KRaft Migration](https://www.conduktor.io/glossary/zookeeper-to-kraft-migration). - **Observability stack**: Use **Kafka Lag Exporter** with Prometheus to track producer throughput and errors. Export JMX metrics (`kafka.producer:*`) to Grafana dashboards. Monitor consumer lag to understand producer impact on downstream systems: [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). For commercial environments, **Conduktor** provides unified producer monitoring, real-time error tracking, schema validation, and governance across clusters. See [Conduktor's topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) to view producer metrics, track message throughput, and inspect message content. ## Where It Fits in Streaming Architectures Producers sit at the edge of a streaming stack and shape downstream state and cost. Common placements: * **Microservices** publish domain events that drive other services. See [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing) for safe event emission from databases. * **CDC to Kafka** uses Debezium to read database logs and publish to topics. Stable schemas and keys reduce reprocess churn. See [What is Change Data Capture (CDC) Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). * **Flink** jobs read topics and write results to Kafka with exactly-once sinks that coordinate with Kafka transactions. See [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). * **Kafka Streams** reads and writes topics inside the same process. Idempotent or transactional producers prevent duplicate outputs on recovery. * **Kafka Connect** acts as a managed producer for databases and SaaS sources. * **ETL to lakehouses** publishes to topics that hydrate Iceberg or Delta tables through sinks. See [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables). Design notes: * Pick keys that match downstream grouping, for example session joins on `session_id`. * Use compatibility rules in the registry to protect consumers. See [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). * If the pipeline needs end-to-end exactly-once, combine idempotent producers, transactions, and read-committed consumers. For deep dive, see [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). * For headers usage (metadata, tracing, routing), see [Using Kafka Headers Effectively](https://www.conduktor.io/glossary/using-kafka-headers-effectively). ## Governance and Platform Support The producer is the first gate for safety and reuse. Teams need clear controls on who can write, what shapes are allowed, and how changes get tracked. What to enforce: * **RBAC and ACLs** for write access, for example a service account with write to `orders-*`. See [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming) and [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices). * **Schema policy** for backward or full compatibility, required fields like `event_time` or `tenant_id`, and validation in CI. * **Field protection** with masking or encryption on sensitive fields in transit or at a gateway. See [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). * **Quotas** per client ID to protect shared clusters. See [Quotas and Rate Limiting in Kafka](https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka). * **Audit and lineage** that record who wrote, when, with which schema version, and how downstream tables or features depend on that topic. See [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). * **Standard producer libs** or config bundles to keep defaults consistent across teams. ### Platform support with Conduktor **Conduktor** provides comprehensive producer governance and monitoring for Kafka clusters: * **RBAC and access control**: Manage which service accounts can write to specific topics with fine-grained policies. Learn about [service accounts and ACL management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/service-accounts-acls) in Conduktor. * **Schema validation**: Enforce schema compatibility rules before messages reach brokers, preventing breaking changes * **Data masking**: Apply dynamic field-level masking on sensitive data (PII, credentials) at produce time using [Conduktor interceptors](https://docs.conduktor.io/guide/conduktor-concepts/interceptors) * **Audit trail**: Track all produce operations with full lineage - who wrote what, when, and with which schema version * **Monitoring**: Real-time dashboards showing producer throughput, error rates, latency, and authorization failures across clusters via [topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) * **Testing with Conduktor Gateway**: Use Conduktor Gateway, a Kafka proxy, to inject chaos scenarios (latency, failures, broker outages) to test producer retry logic and resilience. For broader testing strategies, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). * **Impact analysis**: Visualize dependencies between producers, topics, schemas, and downstream consumers This centralized approach keeps producer changes safe, traceable, and compliant at scale without requiring custom tooling. ## Related Concepts - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Ensures producers write messages with validated, compatible schemas that consumers can reliably deserialize. - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Combines idempotent and transactional producers to achieve end-to-end exactly-once delivery guarantees. - [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) - Critical for securing producer authentication, authorization, and data encryption in production environments. ## Summary Producers turn domain events and CDC streams into durable topic data. Their settings for serialization, acks, retries, batching, and keys decide reliability, latency, and cost for the whole stack. **Modern defaults (Kafka 3.0+)** include idempotent producers, making at-least-once delivery safe by default. **Kafka 4.0 with KRaft** improves metadata fetch performance and reduces tail latencies. For 2025 deployments, use **Zstd compression**, monitor with Kafka Lag Exporter and Prometheus, and enforce governance with platforms like **Conduktor**. Streaming engines such as Flink and Kafka Streams depend on producer choices for correct joins, windows, and state. Teams reach stable results with schema-aware encoding, safe acks with idempotence (or transactions for exactly-once), tuned batching, and clear key strategy. Governance platforms add RBAC, schema policy, masking, audit, lineage, quotas, and shared defaults, which keeps producer changes safe and traceable at scale. ## Sources and References * Apache Kafka documentation, Producer API and configuration: [https://kafka.apache.org/documentation/](https://kafka.apache.org/documentation/) * Confluent documentation, Producer configs, idempotence, transactions, and exactly-once semantics: [https://docs.confluent.io/platform/current/clients/producer.html](https://docs.confluent.io/platform/current/clients/producer.html) * Apache Flink documentation, Kafka connector and sinks: [https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/](https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/) * Debezium documentation, Architecture and Kafka topics: [https://debezium.io/documentation/](https://debezium.io/documentation/) * Kafka Improvement Proposals index, producer design background: [https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals) ](http://localhost:4321/articles/kafka-producers-sending-messages-to-topics) Source: https://www.conduktor.io/glossary/kafka-producers --- # Kafka Replication and High Availability Apache Kafka's replication system is what makes it reliable: it ensures data durability and high availability even when hardware fails or network partitions occur. This article covers how Kafka replication works, its high availability features, and practical considerations for building resilient systems. For foundational Kafka concepts, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). To understand Kafka's modern architecture, refer to [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## Understanding Kafka Replication Kafka replication works by maintaining multiple copies of data across different brokers in a cluster. When you create a topic, you specify a replication factor that determines how many copies of each partition will exist. For detailed coverage of topics, partitions, and brokers, see [Kafka Topics, Partitions, Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). ![kafka-replication-and-high-availability diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-replication-and-high-availability-0.webp) For example, with a replication factor of 3, each partition has one leader and two followers. The leader handles all read and write requests, while followers continuously replicate data from the leader. If the leader fails, one of the followers automatically becomes the new leader. This design provides fault tolerance without requiring complex coordination protocols during normal operations. Writers and readers only interact with the leader, which simplifies the programming model while maintaining strong consistency guarantees. ## In-Sync Replicas (ISR) Not all replicas are equal in Kafka's replication model. The system maintains a critical concept called In-Sync Replicas (ISR), which determines which followers are fully caught up with the leader. A follower is considered in-sync if it has fetched messages up to the leader's high watermark within a configurable time window (controlled by `replica.lag.time.max.ms`, typically 10 seconds). Only in-sync replicas are eligible to become leaders during failover. The ISR mechanism balances durability and availability. When `min.insync.replicas` is set to 2 and a producer uses `acks=all`, Kafka guarantees that writes are acknowledged only after at least two replicas (including the leader) have received the data. This prevents data loss even if a broker fails immediately after acknowledging a write. Consider a scenario with three brokers and a replication factor of 3. If one broker goes offline, the partition remains available as long as two replicas stay in-sync. However, if two brokers fail simultaneously, the partition becomes unavailable for writes when `min.insync.replicas=2`, preventing potential data loss. ## Automatic Failover and Recovery Kafka handles failure scenarios well. When a broker fails, the cluster controller (one of the brokers in the KRaft quorum since Kafka 4.0+) detects the failure and triggers leader election for all partitions where the failed broker was the leader. The controller selects a new leader from the ISR set, ensuring that no committed data is lost. Clients experience a brief interruption (typically a few seconds) while the election completes, but the system automatically recovers without manual intervention. When the failed broker comes back online, it rejoins the cluster as a follower for its partitions. It catches up by replicating missing messages from the current leaders, eventually rejoining the ISR once fully synchronized. ### KRaft and Controller Quorum (Kafka 4.0+) Starting with Kafka 4.0, the Apache Kafka project removed ZooKeeper dependency entirely, adopting KRaft (Kafka Raft) as the consensus protocol for cluster metadata management. This architectural shift significantly improves high availability characteristics. In KRaft mode, controller nodes form a quorum using the Raft consensus algorithm. The controller quorum itself is replicated, typically across 3 or 5 nodes, ensuring that metadata operations (like leader elections) can continue even when controller nodes fail. This eliminates ZooKeeper as a potential single point of failure and reduces failover times. **Key KRaft improvements for high availability:** - **Faster Leader Elections**: Metadata changes propagate more quickly through the Raft log, reducing partition unavailability windows from seconds to milliseconds in many cases. - **Controller Quorum Resilience**: With 3 controller nodes, the cluster tolerates 1 controller failure. With 5 controllers, it tolerates 2 failures while maintaining metadata availability. - **Unified Storage Model**: Controllers store metadata in the same log-based format as regular Kafka topics, simplifying backup and disaster recovery procedures. - **No External Dependencies**: Eliminating ZooKeeper reduces operational complexity and removes a potential failure domain from your architecture. For production deployments, configure dedicated controller nodes separate from broker nodes to prevent resource contention. Use `controller.quorum.voters` to specify the controller quorum members and ensure they're distributed across failure domains (availability zones or racks). ```properties # Example KRaft controller configuration (Kafka 4.0+) process.roles=controller node.id=1 controller.quorum.voters=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093 controller.listener.names=CONTROLLER # Distribute controllers across availability zones for resilience broker.rack=us-east-1a # Set appropriately for each controller node ``` ### Unclean Leader Election In extreme scenarios where all ISR members are unavailable, Kafka faces a choice: remain unavailable or elect a non-ISR replica as leader. This trade-off is controlled by `unclean.leader.election.enable`. Setting this to `false` (recommended for most applications) prioritizes consistency over availability. The partition remains offline until an ISR member recovers. Setting it to `true` allows non-ISR replicas to become leaders, restoring availability but potentially losing committed messages. ## Configuration Best Practices These are the key settings for availability and durability: - **Replication Factor**: Use a minimum of 3 for production systems. This allows the system to tolerate one broker failure while maintaining redundancy. Critical data might warrant a replication factor of 5 or higher. - **min.insync.replicas**: Set to `replication_factor - 1` for balanced durability. With replication factor 3, use `min.insync.replicas=2`. This ensures writes are acknowledged by multiple replicas while allowing the system to tolerate one replica being temporarily unavailable. - **acks Configuration**: Producers should use `acks=all` for critical data, ensuring writes are replicated before acknowledgment. Use `acks=1` only for use cases where some data loss is acceptable in exchange for lower latency. ```java // Producer configuration for high durability (Kafka 4.0+) Properties props = new Properties(); props.put("bootstrap.servers", "broker1:9092,broker2:9092,broker3:9092"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); // Critical settings for durability props.put("acks", "all"); // Wait for all in-sync replicas props.put("enable.idempotence", "true"); // Prevent duplicates props.put("retries", Integer.MAX_VALUE); // Retry failed sends props.put("max.in.flight.requests.per.connection", "5"); // Kafka 3.0+ safe with idempotence KafkaProducer producer = new KafkaProducer<>(props); ``` For comprehensive producer configuration guidance, see [Kafka Producers](https://www.conduktor.io/glossary/kafka-producers). **Rack Awareness**: Configure rack awareness (`broker.rack`) to ensure replicas are distributed across physical failure domains like availability zones or data centers. This protects against correlated failures from power outages or network issues. ```properties # Broker configuration for rack awareness (Kafka 4.0+) # broker-1.properties (in availability zone us-east-1a) broker.id=1 broker.rack=us-east-1a # broker-2.properties (in availability zone us-east-1b) broker.id=2 broker.rack=us-east-1b # broker-3.properties (in availability zone us-east-1c) broker.id=3 broker.rack=us-east-1c ``` When rack awareness is enabled, Kafka's replica assignment algorithm ensures that replicas for each partition are distributed across different racks. With a replication factor of 3 and 3 availability zones, each zone hosts exactly one replica, protecting against entire zone failures. **Replica Fetcher Configuration**: Modern Kafka (2.4+) supports reading from follower replicas in the same rack, reducing cross-AZ data transfer costs and improving read latency. Configure `replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector` and set consumer `client.rack` to enable this feature. ```java // Consumer configuration for rack-aware reads (Kafka 4.0+) Properties consumerProps = new Properties(); consumerProps.put("bootstrap.servers", "broker1:9092,broker2:9092,broker3:9092"); consumerProps.put("group.id", "my-consumer-group"); consumerProps.put("client.rack", "us-east-1a"); // Consumer's rack/AZ // Consumer will prefer reading from replicas in us-east-1a when available ``` ## Monitoring and Troubleshooting Key metrics to track: - **Under-replicated Partitions**: Tracks partitions where followers have fallen behind the leader. Monitor the JMX metric `kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions`. Sustained under-replication suggests insufficient broker resources, network issues, or failing hardware. A value of 0 indicates healthy replication. - **Offline Partitions**: Partitions with no in-sync replicas represent actual unavailability and require immediate attention. Monitor `kafka.controller:type=KafkaController,name=OfflinePartitionsCount`. Any non-zero value indicates a production incident requiring immediate investigation. - **ISR Shrink/Expand Rate**: Frequent ISR changes indicate instability. Monitor `kafka.server:type=ReplicaManager,name=IsrShrinksPerSec` and `IsrExpandsPerSec`. Frequent shrinks/expands suggest resource contention, network problems, or GC pauses affecting replication. - **Replication Lag**: The offset difference between leaders and followers helps identify slow replicas before they fall out of ISR. Monitor `kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica`. High lag (>1000 messages) indicates performance issues. - **Controller Status**: In KRaft mode (Kafka 4.0+), monitor controller quorum health with `kafka.controller:type=KafkaController,name=ActiveControllerCount` (should be 1) and `kafka.server:type=raft-metrics,name=state` (should show "Leader" for active controller). ### Monitoring Tools and Setup For production Kafka clusters, use: - **Conduktor Platform**: Provides real-time visibility into replication metrics through visual dashboards, partition topology views, and alerting. Conduktor helps operators quickly identify under-replicated partitions, visualize rack distribution, and diagnose replication issues before they impact availability. - **JMX Exporters**: Use Prometheus JMX Exporter to expose Kafka metrics for time-series monitoring: ```yaml # jmx_prometheus_exporter config for replication metrics (Kafka 4.0+) lowercaseOutputName: true rules: - pattern: kafka.server<>(Value|Count) name: kafka_server_replicamanager_$1 - pattern: kafka.controller<>(Value|Count) name: kafka_controller_$1 - pattern: kafka.server<>Value name: kafka_server_replica_fetcher_max_lag labels: client_id: "$1" ``` **Open-Source Monitoring Tools**: - **kafka-ui**: Lightweight web UI for monitoring Kafka clusters, including replication status - **Redpanda Console** (open-source): Visual monitoring of topics, partitions, and consumer groups - **Kafka Exporter**: Dedicated Prometheus exporter for Kafka lag and partition metrics **Alerting Thresholds**: Configure alerts for: - `OfflinePartitionsCount > 0` (critical - immediate response) - `UnderReplicatedPartitions > 0` for >5 minutes (warning) - `MaxLag > 10000` messages (warning - investigate broker performance) - `IsrShrinksPerSec > 5` sustained rate (warning - cluster instability) For comprehensive monitoring strategies, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). For Kubernetes deployments, refer to [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) and [Strimzi Kafka Operator for Kubernetes](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes). ## Role in Data Streaming Platforms Replication and high availability underpin reliable data streaming platforms. Stream processing frameworks like Apache Flink, Kafka Streams, and ksqlDB depend on Kafka's durability guarantees to maintain processing state and ensure exactly-once semantics. For details on exactly-once semantics, see [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). When a stream processor checkpoints its state to Kafka topics, replication ensures that state survives broker failures. This allows processors to recover and resume from the last checkpoint without data loss or duplicate processing. ### Multi-Datacenter Replication and Disaster Recovery In multi-datacenter deployments, Apache Kafka's MirrorMaker 2 (MM2) extends replication across geographic regions, providing disaster recovery capabilities and enabling active-active architectures for global applications. Kafka 4.0+ includes performance improvements to MM2's replication streams and better handling of metadata synchronization. **MirrorMaker 2 Features for High Availability**: - **Active-Active Replication**: Bi-directional mirroring enables writes to multiple clusters with conflict resolution - **Offset Translation**: Automatically maps consumer offsets between clusters for seamless failover - **Topic Configuration Sync**: Replicates topic configurations, ACLs, and quotas across datacenters (see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) for ACL design) - **Exactly-Once Semantics**: Supports transactional producers for consistent cross-cluster replication **Tiered Storage (Kafka 3.6+)**: Kafka's tiered storage feature offloads older log segments to object storage (S3, Azure Blob, GCS) while maintaining recent data on local disks. This reduces storage costs and improves recovery times by limiting the data that needs replication when brokers restart. While tiered storage doesn't replace replication for high availability, it complements it by reducing the cost of maintaining high replication factors for long retention periods. For detailed information, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). Local replication handles fault tolerance, cross-datacenter replication covers disaster recovery, and tiered storage reduces retention costs. With careful topology design and failover procedures, organizations can reach five-nines (99.999%) availability. ## Related Concepts - [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) - KRaft's modern consensus protocol improves failover speed and simplifies high availability operations. - [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) - Essential monitoring strategies for tracking replication health and detecting availability issues. - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka) - Complements replication by providing cost-effective long-term retention while maintaining availability. ## Summary Kafka replication maintains multiple synchronized copies of data across brokers and handles failover automatically, so systems survive hardware failures and network issues without data loss. The ISR mechanism, leader election, and configurable durability settings give operators control over availability-consistency trade-offs. Proper configuration of replication factors, acknowledgment settings, and rack awareness keeps systems within their reliability targets. With Kafka 4.0+, KRaft consensus eliminates ZooKeeper dependencies and improves failover times. Monitoring replication health metrics lets teams catch and resolve issues early. Tools like Conduktor provide visibility into replication status, partition distribution, and controller health. Combined with geographic replication and tiered storage, these capabilities let organizations build streaming platforms that meet demanding availability and durability SLAs. ## Sources and References 1. Apache Kafka Documentation - "Replication" - Official documentation covering replication design and configuration: https://kafka.apache.org/documentation/#replication 2. Neha Narkhede, Gwen Shapira, Todd Palino - "Kafka: The Definitive Guide, 2nd Edition" (O'Reilly, 2021) - Comprehensive coverage of Kafka internals including replication mechanics 3. Jun Rao - "Intra-cluster Replication in Apache Kafka" (Confluent Blog) - Detailed explanation from Kafka's co-creator: https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/ 4. Jay Kreps - "Benchmarking Apache Kafka: 2 Million Writes Per Second" (LinkedIn Engineering Blog) - Real-world insights into replication performance: https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines 5. Confluent Documentation - "Durability Guarantees" - Production best practices for configuring replication: https://docs.confluent.io/platform/current/kafka/design.html#durability-guarantees Source: https://www.conduktor.io/glossary/kafka-replication-and-high-availability --- # Kafka Security Best Practices As Kafka deployments grow in scale and criticality, securing them becomes essential. A security breach can expose sensitive business data, break compliance requirements, and disrupt critical operations. This guide covers security best practices for Apache Kafka: authentication, authorization, encryption, and operational security. ![Kafka security layers and defense-in-depth architecture](https://www.conduktor.io/assets/images/glossary/kafka-security-best-practices-0.webp) ## Why Kafka Security Matters in Data Streaming Data streaming platforms like Kafka process and store vast amounts of information in real time. Unlike traditional databases with well-established security patterns, streaming platforms present unique challenges. Data flows continuously between producers, brokers, and consumers, often across network boundaries and organizational teams. Without proper security controls, any application with network access to your Kafka cluster can publish or consume any data. This unrestricted access creates significant risks, including unauthorized data access, data tampering, compliance violations, and service disruption through malicious or accidental actions. Modern streaming architectures often involve multiple teams, microservices, and external partners sharing the same Kafka infrastructure. This amplifies security concerns and makes strong access controls essential. ## Authentication: Verifying Client Identity Authentication ensures that only verified clients can connect to your Kafka cluster. Kafka supports several authentication mechanisms, each with different security profiles and operational complexity. For in-depth coverage of authentication mechanisms and implementation patterns, see [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). ### SASL (Simple Authentication and Security Layer) SASL provides a framework for authentication with multiple mechanisms: **SASL/SCRAM** (Salted Challenge Response Authentication Mechanism) offers username/password authentication with credentials stored in KRaft (Kafka 3.3+) or ZooKeeper (legacy). Modern Kafka deployments using KRaft mode benefit from simplified credential management without ZooKeeper dependencies. This mechanism provides strong security without requiring external infrastructure: ```properties # Broker configuration sasl.enabled.mechanisms=SCRAM-SHA-512 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512 ``` - **SASL/PLAIN** transmits credentials in plaintext (albeit over encrypted connections) and should only be used with TLS encryption. While simpler to implement, SCRAM is generally preferred for production environments. - **SASL/GSSAPI (Kerberos)** integrates with enterprise identity systems and provides strong authentication with single sign-on capabilities. However, it requires Kerberos infrastructure and additional operational complexity. ### Mutual TLS (mTLS) Mutual TLS authentication uses client certificates for authentication. Both the broker and client present certificates, providing strong two-way authentication. This approach eliminates password management but requires certificate infrastructure and distribution mechanisms. mTLS works well for service-to-service communication where certificate lifecycle management can be automated, though it can be more challenging for human users and dynamic client environments. For detailed implementation guidance, see [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka). ### OAuth 2.0 and OIDC (Modern Cloud-Native Authentication) OAuth 2.0 with OpenID Connect (OIDC) has become the standard authentication mechanism for cloud-native Kafka deployments in 2025. This approach integrates Kafka with enterprise identity providers like Okta, Auth0, Azure AD, or Keycloak, enabling centralized identity management, token-based authentication, and fine-grained access control. **SASL/OAUTHBEARER** is Kafka's OAuth 2.0 implementation that supports both unsecured JWT tokens (development only) and secured tokens from external identity providers: ```properties # Broker configuration for OAuth 2.0 sasl.enabled.mechanisms=OAUTHBEARER sasl.mechanism.inter.broker.protocol=OAUTHBEARER listener.name.sasl_ssl.oauthbearer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required; listener.name.sasl_ssl.oauthbearer.sasl.login.callback.handler.class=io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler listener.name.sasl_ssl.oauthbearer.sasl.server.callback.handler.class=io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler ``` **Benefits of OAuth 2.0/OIDC**: - **Centralized identity management**: Leverage existing enterprise identity infrastructure - **Token-based authentication**: Short-lived tokens reduce credential exposure - **Integration with SSO**: Users authenticate once across multiple systems - **Dynamic authorization**: Token claims can drive authorization decisions - **Audit trails**: Identity providers maintain comprehensive authentication logs OAuth 2.0 is particularly valuable in multi-tenant environments, microservices architectures, and organizations with established identity management systems. However, it requires additional infrastructure (identity provider) and more complex configuration than SCRAM or mTLS. ## Authorization: Controlling Access with ACLs Authentication identifies who is connecting, but authorization determines what they can do. Kafka uses Access Control Lists (ACLs) to enforce fine-grained permissions on topics, consumer groups, and cluster operations. For comprehensive coverage of authorization patterns and implementation strategies, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). ACLs specify which principals (users or services) can perform which operations on which resources. For example: ```bash # Allow user 'analytics-app' to read from 'customer-events' topic kafka-acls --add --allow-principal User:analytics-app \ --operation Read --topic customer-events ``` ### ACL Best Practices - **Principle of least privilege**: Grant only the minimum permissions required for each application or user. A consumer application typically needs only Read permission on specific topics and Write permission on its consumer group. - **Resource-specific permissions**: Avoid wildcard ACLs in production. Instead of granting access to all topics (`*`), explicitly list permitted topics. This prevents accidental access to sensitive data. - **Team-based access control**: In multi-team environments, prefix topic names with team identifiers (e.g., `marketing.events`, `engineering.logs`) and grant ACLs accordingly. This creates clear ownership boundaries. For strategies on managing shared Kafka infrastructure across teams, see [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments). - **Regular audits**: Review ACLs periodically to remove unused permissions and identify over-privileged accounts. Managing ACLs through command-line tools can become cumbersome as your cluster grows. Governance platforms provide visual ACL management interfaces that make it easier to understand and maintain permission structures across hundreds of topics and users. For broader access control patterns including RBAC and ABAC, see [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming). ## Encryption: Protecting Data in Transit and at Rest Encryption prevents unauthorized access to data as it moves through your infrastructure and while stored on disk. For detailed encryption implementation guidance, see [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka). ### Encryption in Transit (TLS 1.3) TLS encryption protects data as it travels between clients and brokers and between brokers. Without encryption, network traffic can be intercepted and read by anyone with network access. **TLS 1.3 is the modern standard** (2025) and should be enforced for all production deployments. TLS 1.3 provides improved security by removing vulnerable cipher suites and enhancing performance through reduced handshake latency: ```properties # Broker configuration with TLS 1.3 listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093 ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks ssl.keystore.password=your-keystore-password ssl.key.password=your-key-password # Enforce TLS 1.3 (recommended for 2025) ssl.protocol=TLSv1.3 ssl.enabled.protocols=TLSv1.3 # Strong cipher suites for TLS 1.3 ssl.cipher.suites=TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_CHACHA20_POLY1305_SHA256 ``` **TLS 1.2 compatibility**: If you must support legacy clients, allow TLS 1.2 as a fallback but plan migration to TLS 1.3: ```properties ssl.protocol=TLSv1.3 ssl.enabled.protocols=TLSv1.3,TLSv1.2 ``` - **Performance considerations**: TLS 1.3 encryption has minimal overhead on modern hardware. With AES-NI hardware acceleration available in modern CPUs, the throughput impact is typically less than 5%, making encryption cost-effective for all deployments. TLS 1.3's optimized handshake also reduces connection establishment latency by 33% compared to TLS 1.2. - **Certificate management**: Implement automated certificate rotation using tools like cert-manager (Kubernetes) or HashiCorp Vault to prevent certificate expiration incidents. Certificates should be rotated every 90 days or less. ### Encryption at Rest Kafka stores messages in log files on broker disks. Encryption at rest protects this data if physical disks are compromised or improperly decommissioned. While Kafka itself doesn't provide built-in encryption at rest, you can implement it through: - **Filesystem-level encryption**: Tools like dm-crypt on Linux or LUKS encrypt entire partitions where Kafka stores data. - **Cloud provider encryption**: AWS EBS volumes, Azure Disk Encryption, and GCP persistent disk encryption provide transparent encryption. - **Application-level encryption**: Encrypt message payloads before producing them to Kafka, though this prevents brokers from compressing data effectively. ## Network Security and Isolation Beyond authentication and encryption, network-level security provides additional protection layers. - **Network segmentation**: Deploy Kafka brokers in private subnets without direct internet access. Use bastion hosts or VPN connections for administrative access. - **Firewall rules**: Configure firewalls to allow only necessary traffic. Broker-to-broker communication should be restricted to broker IP addresses, and client access should be limited to specific application networks. - **Security groups**: In cloud environments, use security groups or network policies to enforce network-level access control. For example, only allow traffic to Kafka brokers from specific application tiers. - **Inter-cluster replication**: When replicating data between clusters across different network zones or cloud regions, always use encrypted connections and authentication even within private networks. ## Zero Trust Architecture for Kafka Zero Trust security requires continuous verification of every access request regardless of origin, rather than trusting any network location by default. - **Core Zero Trust principles for Kafka**: - **Never trust, always verify**: Authenticate and authorize every client connection, even from internal networks. Don't assume that traffic from your private network is safe, implement authentication and encryption everywhere. - **Least privilege access**: Grant the minimum permissions required for each client to perform its function. A consumer reading from `user-events` should not have access to `payment-transactions`. - **Assume breach**: Design your security architecture assuming attackers may already be inside your network. Use encryption in transit for all connections, including broker-to-broker communication within the same data center. - **Verify explicitly**: Use strong authentication mechanisms (OAuth 2.0, mTLS, SCRAM-SHA-512) rather than relying on network location or IP addresses for access control. - **Monitor continuously**: Track all access patterns, authentication attempts, and authorization decisions to detect anomalies that may indicate compromise or misconfiguration. - **Microsegmentation**: Isolate different workloads and tenants using network policies, ACLs, and separate clusters when necessary. Critical financial data should not share infrastructure with lower-sensitivity workloads. For comprehensive guidance on implementing Zero Trust principles across your streaming infrastructure, see [Zero Trust for Streaming](https://www.conduktor.io/glossary/zero-trust-for-streaming). ## Monitoring and Auditing Security Events Security requires ongoing monitoring, not just initial configuration. - **Audit logs**: Enable Kafka's audit logging to track authentication attempts, authorization decisions, and administrative operations. These logs are crucial for security investigations and compliance reporting. For comprehensive audit logging strategies, see [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). - **Security metrics monitoring** (2025 tooling): Track security-related metrics using modern observability platforms: - **Authentication failures**: Monitor failed login attempts and authentication errors - **ACL violations**: Track unauthorized access attempts to topics or consumer groups - **TLS handshake failures**: Detect certificate issues or version mismatches - **Unusual access patterns**: Identify anomalous consumer or producer behavior Modern platforms like **Conduktor** provide centralized security management interfaces with [visual RBAC editors](https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac), permission auditing, and [audit logging](https://docs.conduktor.io/guide/conduktor-in-production/admin/audit-logs). These tools make it easier to understand your security posture across hundreds of topics, thousands of ACLs, and multiple Kafka clusters. - **Alerting**: Configure alerts for security events like repeated authentication failures, connections from unexpected IP addresses, or attempts to access unauthorized topics. Integration with incident response platforms (PagerDuty, Opsgenie) ensures rapid response to security incidents. - **Security testing**: Use **Conduktor Gateway**, a Kafka proxy, to test security controls through chaos engineering scenarios, simulate authentication failures, certificate expiration, or unauthorized access attempts to verify your security monitoring and alerting work correctly before incidents occur in production. ## Security in Data Streaming Architectures Securing Kafka extends beyond the cluster itself to the broader streaming ecosystem. Apache Flink applications, Kafka Connect connectors, and ksqlDB servers all require proper authentication credentials and authorization to access Kafka topics. - **Credential management**: Use secret management systems like HashiCorp Vault, AWS Secrets Manager, or Kubernetes secrets to store and distribute Kafka credentials rather than hardcoding them in configuration files. - **Schema Registry security**: If using Schema Registry or other schema systems, secure these components separately as they control data formats and evolution. For schema management security considerations, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). - **Connect and ksqlDB**: These components often have elevated privileges to create topics and manage consumer groups. Secure their administrative APIs and limit their Kafka permissions appropriately. - **Policy enforcement**: Implement automated policy enforcement for security standards, data classification, and compliance requirements. For comprehensive policy enforcement strategies, see [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming). ## Virtual Clusters for Multi-Tenant Security Managing security across multiple teams sharing a Kafka cluster traditionally requires complex network segmentation or intricate ACL hierarchies. Conduktor Virtual Clusters provide logical isolation within a single physical cluster, where each tenant operates in their own namespace completely isolated from other tenants' topics and configurations. This scopes all security controls, authentication, authorization, and encryption policies, to individual virtual clusters rather than managing them globally. Virtual Clusters allow different security postures across workloads. A virtual cluster handling payment data can enforce strict mTLS and restrictive ACLs, while a development virtual cluster maintains lighter controls. Compromised credentials affect only one virtual cluster, limiting blast radius. For organizations implementing Zero Trust, Virtual Clusters provide microsegmentation at the Kafka layer. For implementation details, see the [Virtual Clusters documentation](https://docs.conduktor.io/guide/conduktor-concepts/virtual-clusters). ## Summary Securing Apache Kafka requires authentication, authorization, encryption, network isolation, and continuous monitoring. Start with these practices: 1. **Enable authentication** using OAuth 2.0/OIDC, SASL/SCRAM-SHA-512, or mTLS to verify all client identities 2. **Implement ACLs** following the principle of least privilege with regular audits 3. **Encrypt all traffic** with TLS 1.3 to protect data in transit 4. **Adopt Zero Trust principles** with continuous verification and microsegmentation 5. **Isolate networks** to limit exposure and attack surfaces 6. **Monitor security events** continuously with modern observability platforms 7. **Secure the ecosystem** including Schema Registry, Connect, and streaming applications 8. **Automate security testing** using chaos engineering to validate controls Security is not optional for production Kafka deployments. The implementation complexity is worth it given the risks of data breaches, compliance failures, and service disruptions. ## Related Concepts - [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments) - Strategies for securing shared Kafka infrastructure across multiple teams with isolation and access control patterns. - [Data Classification and Tagging Strategies](https://www.conduktor.io/glossary/data-classification-and-tagging-strategies) - How to classify and tag sensitive data to enforce appropriate security controls. - [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) - Comprehensive strategies for tracking security events and maintaining compliance audit trails. ## Sources and References 1. **Apache Kafka Security Documentation** - Official security configuration guide covering authentication, authorization, and encryption: https://kafka.apache.org/documentation/#security 2. **Conduktor Security Documentation** - Platform for managing Kafka security, ACLs, and governance with visual interfaces and testing tools: https://www.conduktor.io/ 3. **OWASP API Security Project** - General API security principles applicable to Kafka's protocol: https://owasp.org/www-project-api-security/ 4. **Kafka: The Definitive Guide** (O'Reilly, 2024) - Chapter on securing Kafka deployments, written by Kafka committers and practitioners 5. **NIST Cybersecurity Framework** - Framework for improving critical infrastructure cybersecurity, applicable to data infrastructure: https://www.nist.gov/cyberframework 6. **OAuth 2.0 for Apache Kafka** - Strimzi OAuth library documentation and best practices: https://github.com/strimzi/strimzi-kafka-oauth 7. **Zero Trust Architecture (NIST SP 800-207)** - NIST standards for implementing Zero Trust security principles: https://csrc.nist.gov/publications/detail/sp/800-207/final Source: https://www.conduktor.io/glossary/kafka-security-best-practices --- # Kafka Queues: Share Groups Explained **Kafka share groups** are a consumption model introduced in [KIP-932](https://kafka-options-explorer.conduktor.io/kip/932/) that lets many consumers cooperatively read one topic with per-record acknowledgement, so multiple members can process records from the same partition at once. This solves a long-standing Kafka limitation by decoupling consumer parallelism from partition count, bringing queue-like semantics to Apache Kafka without changing the underlying partitioned log. ![kafka-share-groups diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-share-groups-0.webp) ## What are Kafka share groups? A classic [consumer group](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) binds each partition to exactly one member at a time. Concurrency is therefore capped at the partition count: ten consumers on a six-partition topic leave four idle. Raising throughput has historically meant adding partitions, which can have negative consequences for the cluster. Share groups remove that coupling. A share group is a named set of consumers/workers where **multiple members can process records from the same partition concurrently**, and each record is acknowledged individually rather than by advancing a partition offset. The broker distributes records, tracks their state, and reclaims any that are not acknowledged in time to redistribute them to other workers. This effectively gives a shared work queue on top of the Kafka log, with two consequences: - Parallelism is no longer tied to partition count. A share group can have more active members than the topic has partitions. - Failed records can be redelivered or discarded per record, without stalling the rest of the stream (avoiding head-of-line blocking). ## Share groups vs consumer groups | Dimension | Classic consumer group | Share group | |---|---|---| | Partition sharing | One consumer per partition | Many consumers per partition | | Parallelism ceiling | Number of partitions | Independent of partition count | | Acknowledgement | Offset commit (high-water mark) | Per record: accept / release / reject | | Ordering | Guaranteed within a partition | Within a delivered batch only; none across batches or members | | Redelivery | Rewind offset (whole group) | Per record, up to a delivery-attempt limit | | Assignor | Client- or server-driven | Share-partition assignor (server-driven) | | Best fit | Ordered streams, high throughput | Job/task queues, slow I/O, AI agents ⭐ | What to remember: share groups exchange strict per-partition ordering for the ability to spread work across arbitrarily many consumers and scale horizontally. ## How per-message acknowledgement works Instead of committing an offset, a share consumer resolves each record individually. Three outcomes: - **ACCEPT**: the record was processed successfully; it's marked acknowledged and will never be redelivered. - **RELEASE**: the consumer cannot process it right now (a transient failure); the record is redelivered to another consumer. - **REJECT**: the record is unprocessable (e.g. an unexpected payload or a critical error); it is archived with no further attempts. A consumer that needs more time (slow I/O, high latency, LLM calls...) can also RENEW the acquisition lock to extend processing without releasing the record. Each record moves through a small state machine per share group: `Available → Acquired → Acknowledged` (ACCEPT), or back to `Available` (RELEASE), or to `Archived` (REJECT, or once retries are exhausted). ![kafka-share-groups diagram 2](https://www.conduktor.io/assets/images/glossary/kafka-share-groups-1.webp) When a consumer acquires a record it holds an **acquisition lock** for [`share.record.lock.duration.ms`](https://kafka-options-explorer.conduktor.io/configuration/?search=share.record.lock.duration.ms#broker) (default 30s). If the record is not acknowledged before the lock expires (a crash, a hang, a slow downstream call) it automatically returns to `Available` for redelivery. There is also built-in protection against poison-pill messages. Every acquisition increments a delivery count; when it reaches [`group.share.delivery.count.limit`](https://kafka-options-explorer.conduktor.io/configuration/?search=group.share.delivery.count.limit#broker) (default 5) the record is archived and no longer delivered, so a single bad record cannot loop forever. ## When share groups help Share groups help most when the bottleneck is the *consumer* itself: - **I/O-bound work.** Each record triggers a slow external call: an HTTP request, a database write, an LLM inference. Consumers spend most of their time waiting, so adding more of them raises effective throughput even on a single partition. - **More consumers than partitions.** You want 50 workers but the topic has 6 partitions. A share group admits all 50 without repartitioning. - **Head-of-line blocking.** In a classic group, one slow record blocks its partition. With per-record acquisition, other members keep draining while the slow one is retried or archived. - **Job and task queues.** Workloads that would otherwise force over-partitioning purely to raise concurrency map cleanly onto a share group. - **AI agents and multi-agent systems.** A single agent step is an LLM call or tool invocation measured in seconds, not milliseconds, so a stream of agent tasks is dominated by consumer wait time rather than broker throughput. When an orchestrator fans work out to specialized sub-agents over Kafka, a share group runs far more agent workers than the topic has partitions, and per-record acknowledgement retries a slow or failed step (an inference timeout, a provider rate limit) without stalling the rest. That makes Kafka a natural backbone for multi-agent workloads, where the bottleneck is model latency, not the log. ## When share groups do not help - **Ordered per-key processing.** [CDC](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) and any state machine that depends on strict in-partition order need a classic consumer group. Share groups give no ordering across batches, members, or redeliveries. - **[Event-driven architecture](https://www.conduktor.io/glossary/event-driven-architecture) and [event sourcing](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka).** Ordering is critical: entity events must be processed in order (create, then update) to form a deterministic state. A share group instead splits records among its members like a work queue and acknowledges them one at a time, so no single member sees the whole ordered stream. - **Raw maximum throughput.** For firehose pipelines where consumers are already CPU- or network-saturated, classic consumer groups with large fetch batches and sequential I/O win. Per-record acknowledgement and lock tracking add overhead that pure throughput workloads don't want to pay. - **Exactly-once transactional chains.** The mature transactional/EOS tooling is built around offset commits, not share-group acknowledgement. ## Status, versions, and enabling share groups Share groups have followed a staged rollout: - **Kafka 4.0**: Early Access, test-only, not upgradeable, wire format not stable. - **Kafka 4.1**: Preview (not wire-compatible with the 4.0 early access). - **Kafka 4.2**: production-ready / GA. ```bash kafka-features.sh --bootstrap-server localhost:9092 describe | grep share # Feature: share.version SupportedMinVersion: 0 SupportedMaxVersion: 1 FinalizedVersionLevel: 1 ``` The share rebalance protocol has to be turned on: [`group.coordinator.rebalance.protocols`](https://kafka-options-explorer.conduktor.io/configuration/?search=group.coordinator.rebalance.protocols#broker) must list `share`. This applies to every cluster, single-broker or not. Without it, share consumers join but receive empty assignments (and therefore zero records): ```properties group.coordinator.rebalance.protocols=classic,consumer,share ``` Share-group state (locks, delivery counts, start offset) lives in a new internal topic `__share_group_state`, managed by a share coordinator. That topic carries its own **replication settings**: ```properties share.coordinator.state.topic.replication.factor=3 # Set to 1 if single broker in dev share.coordinator.state.topic.min.isr=2 # Set to 1 if single broker in dev ``` There is a new Java API: class `KafkaShareConsumer` (not your classic `KafkaConsumer`) and the CLI adds `kafka-console-share-consumer.sh` for consuming and `kafka-share-groups.sh` (`--list`, `--describe`, `--members`, `--state`) for administration. ## Kafka queues vs MQ alternatives Share groups add queue semantics, but they are **not a drop-in RabbitMQ or ActiveMQ Artemis replacement**. Kafka still has many gaps: - **No built-in dead-letter queue.** - **No delayed or backoff retry.** - **Log-retention model, not queue-drain.** So share groups are best understood as *queue-like consumption over a durable log*, closing the biggest gap that pushed teams toward a separate broker (see [Kafka vs RabbitMQ](https://www.conduktor.io/glossary/kafka-vs-rabbitmq)) without adopting RabbitMQ's full feature surface. Tools like Conduktor sit alongside the native CLIs to give operators visibility into share-group state, member assignments, and redelivery behavior across environments. **Do share groups replace consumer groups?** No. They are an additional consumption model for queue-like, unordered work. Ordered per-key streams, high-throughput pipelines, and exactly-once chains still use classic consumer groups. **Can more consumers than partitions be active in a share group?** Yes. That is the point: parallelism is decoupled from partition count, so a share group can run more members than the topic has partitions, all sharing the same partitions. **Are share groups production-ready?** They are GA in Apache Kafka 4.2, gated by the share.version feature flag (finalized to level 1 by default). They were Early Access in 4.0 and Preview in 4.1. **What happens to a record that keeps failing?** Each delivery increments a count; once it hits the limit (default 5, group.share.delivery.count.limit) the record is archived and never redelivered. There is no built-in dead-letter queue yet. **Do share groups guarantee ordering?** Only within a single delivered batch for a share-partition. There is no ordering guarantee across batches, across members, or across redeliveries. **Is Kafka a message queue?** Not by default: a topic is a retained, replayable log, and records are not deleted on read. Share groups (KIP-932) add queue-like consumption on top (per-record acknowledgement, many consumers per partition), so Kafka can serve queue workloads while staying a log. See [Kafka vs RabbitMQ](https://www.conduktor.io/glossary/kafka-vs-rabbitmq). **What is the difference between a Kafka topic and a queue?** A topic is a durable log that many consumer groups each read in full and can replay; a traditional queue distributes each message to one worker and drops it on ack. A share group adds queue-style distribution over a topic, but the log underneath stays retained and replayable. ## Related Pages - [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained): the classic one-consumer-per-partition model share groups extend. - [Kafka Partitions Explained](https://www.conduktor.io/glossary/kafka-partitions-explained): why partition count historically capped consumer parallelism. - [Kafka Topics Explained](https://www.conduktor.io/glossary/kafka-topics-explained): the durable log that share groups consume from. - [Kafka vs RabbitMQ](https://www.conduktor.io/glossary/kafka-vs-rabbitmq): how log-based streaming compares to a traditional message queue. - [Dead Letter Queues for Error Handling](https://www.conduktor.io/glossary/dead-letter-queues-for-error-handling): the DLQ pattern share groups still leave to the application. ## Sources - [KIP-932: Queues for Kafka (Apache Kafka wiki)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka) - [Queues for Kafka (KIP-932), Preview Release Notes](https://cwiki.apache.org/confluence/display/KAFKA/Queues+for+Kafka+(KIP-932)+-+Preview+Release+Notes) - [Apache Kafka 4.2 Group Configs (share.auto.offset.reset, share.isolation.level)](https://kafka.apache.org/42/configuration/group-configs/) - [KafkaShareConsumer API (Apache Kafka Javadoc)](https://kafka.apache.org/42/javadoc/org/apache/kafka/clients/consumer/KafkaShareConsumer.html) - [Let's Take a Look at KIP-932: Queues for Kafka! (Gunnar Morling)](https://www.morling.dev/blog/kip-932-queues-for-kafka/) - [Configure Share Groups for Docker in Confluent Platform](https://docs.confluent.io/platform/current/installation/docker/operations/kafka-queues-docker.html) Source: https://www.conduktor.io/glossary/kafka-share-groups --- # Kafka Streams vs Apache Flink: When to Use What When building real-time data pipelines, two frameworks come up most often: Kafka Streams and Apache Flink. Both are mature and production-ready, but they take fundamentally different approaches. Understanding the differences helps you choose the right tool for your use case. ## What Are Kafka Streams and Apache Flink? - **Kafka Streams** is a client library for building stream processing applications that read from and write to Apache Kafka. Released in 2016 as part of Apache Kafka, it's designed to be lightweight and embedded directly into your Java or Scala applications. As of 2025, Kafka Streams 3.6+ provides mature stateful processing with improved RocksDB performance, enhanced error handling, and seamless integration with Kafka's KRaft mode (the ZooKeeper-free architecture). There's no separate cluster to manage, your application is the stream processor. - **Apache Flink** is a distributed stream processing framework with its own cluster runtime. Originally developed in academia and open-sourced in 2014, Flink provides a complete execution environment with advanced features for complex event processing, batch processing, and machine learning. Flink 1.19+ (current as of 2025) delivers significant improvements in checkpoint performance, unified SQL/Table API capabilities, and mature Python DataStream API support. Applications are submitted to a Flink cluster, which handles scheduling, fault tolerance, and resource management. The fundamental difference: Kafka Streams is a library you embed in your application, while Flink is a framework that runs your application. For in-depth coverage of each framework individually, see [What is Apache Flink? Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) and [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams). ## Architecture and Design Philosophy ### Kafka Streams: The Library Approach Kafka Streams applications are standard Java applications that happen to process streams. You package your application as a JAR file and deploy it like any other microservice. Scaling is horizontal, run more instances of your application, and Kafka Streams automatically redistributes the work based on Kafka partition assignments. ![kafka-streams-vs-apache-flink diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-streams-vs-apache-flink-0.webp) This architecture has significant implications. There's no separate cluster to operate, no JobManager or TaskManager to monitor. Your operations team uses the same tools they already know for deploying and monitoring Java applications. State is stored locally using RocksDB, with changelog topics in Kafka providing durability and recovery. ### Apache Flink: The Framework Approach Flink provides a complete distributed runtime. You submit jobs to a Flink cluster, which consists of a JobManager (coordinator) and one or more TaskManagers (workers). The JobManager handles scheduling, checkpointing, and failure recovery, while TaskManagers execute the actual data processing tasks. ![kafka-streams-vs-apache-flink diagram 2](https://www.conduktor.io/assets/images/glossary/kafka-streams-vs-apache-flink-1.webp) This architecture enables sophisticated features like dynamic resource allocation, savepoints for versioning application state, and unified batch and stream processing. However, it also means operating a separate cluster infrastructure with its own monitoring, scaling, and high availability requirements. ## Deployment and Operations ### Operational Complexity Kafka Streams is simpler to operate. Deploy it as a standard application using Docker, Kubernetes (with operators like Strimzi), or traditional application servers. Scaling means starting more instances. Rolling updates are straightforward, just redeploy your application. With Kafka 4.0's KRaft mode (ZooKeeper-free architecture), the entire stack is simpler to operate. For detailed guidance on Kubernetes deployments, see [Running Kafka on Kubernetes](https://www.conduktor.io/glossary/running-kafka-on-kubernetes) and [Strimzi Kafka Operator for Kubernetes](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes). To understand KRaft mode, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). Flink requires cluster management. You need to provision JobManagers with high availability (using Kubernetes native HA since Flink 1.15+), manage TaskManager resources, and understand Flink-specific operational concepts like checkpoints and savepoints. The Flink Kubernetes Operator (officially supported as of 2025) significantly simplifies deployment and management on Kubernetes, providing automated lifecycle management, autoscaling, and upgrade capabilities. Cloud platforms like Amazon Managed Service for Apache Flink help reduce operational burden, but there's inherently more infrastructure to manage than Kafka Streams. For teams already running Kafka and familiar with microservices deployment, Kafka Streams fits naturally into existing workflows. Streaming management tools like Conduktor can enhance this by providing visibility into Kafka topic health through [topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), consumer lag, and data quality, critical for monitoring Kafka Streams applications. For comprehensive monitoring strategies, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). ### State Management Both frameworks support stateful processing with fault tolerance, but with different approaches and scale characteristics. - **Kafka Streams** uses local state stores backed by changelog topics in Kafka. State is partitioned alongside data, keeping related data and computation together on the same instance. When state changes, updates are written to both the local RocksDB store and a Kafka changelog topic. Recovery involves replaying the changelog topic to rebuild local state. This approach works well for moderate state sizes, typically up to 50-100GB per instance, though Kafka Streams 3.6+ improvements to RocksDB integration push these limits higher. The co-location of state and processing provides excellent locality and low-latency access. - **Flink** uses distributed snapshots (checkpoints) to external storage like S3, HDFS, or distributed filesystems. Checkpoints are coordinated across all parallel tasks, providing consistent global snapshots of the entire application state at specific points in time. Flink 1.19+ delivers significant checkpoint performance improvements, handling incremental and asynchronous snapshots efficiently. This architecture enables much larger state, terabytes or even petabytes across a cluster, and advanced features like savepoints (manual snapshots for application versioning, migration, or A/B testing). The tradeoff is added latency for checkpoint writes and dependency on external storage systems. - **Practical guidance**: If your state fits comfortably in 10s of GBs per partition and you value operational simplicity, Kafka Streams' local storage is efficient. For 100s of GBs to TBs of state, or when you need savepoint capabilities for zero-downtime upgrades, Flink's distributed state backend is the better choice. For detailed implementation guidance on Kafka Streams state management, see [State Stores in Kafka Streams](https://www.conduktor.io/glossary/state-stores-in-kafka-streams). ## Feature Comparison ### Processing Capabilities Both frameworks support core stream processing operations: filtering, mapping, aggregations, joins, and windowing. However, differences emerge in advanced scenarios. - **Windowing**: Both support tumbling (fixed, non-overlapping time windows), sliding (overlapping windows), and session windows (dynamic windows based on activity gaps). Flink additionally offers custom window logic with flexible trigger mechanisms and custom eviction policies, enabling sophisticated time-based analytics. For more on windowing patterns, see [Session Windows in Stream Processing](https://www.conduktor.io/glossary/session-windows-in-stream-processing) and [Windowing in Apache Flink: Tumbling, Sliding, and Session Windows](https://www.conduktor.io/glossary/windowing-in-apache-flink-tumbling-sliding-and-session-windows). - **Joins**: Kafka Streams excels at stream-table and stream-stream joins over Kafka topics, with strong consistency guarantees within Kafka's partitioning model. Flink supports similar joins plus more complex patterns like interval joins (joining events within a time range) and temporal table joins (joining with versioned lookup tables), useful for enrichment scenarios. For comprehensive coverage of join patterns, see [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns). - **Exactly-Once Semantics**: Both provide exactly-once processing guarantees, meaning each record is processed exactly one time even in the presence of failures, critical for financial transactions, billing, and audit scenarios. Kafka Streams achieves this through Kafka's native transaction support, ensuring atomic writes across multiple topics and state stores. Flink uses distributed checkpoints with two-phase commit protocol for external systems, coordinating state consistency across the entire job graph. For implementation details of Kafka transactions, see [Kafka Transactions Deep Dive](https://www.conduktor.io/glossary/kafka-transactions-deep-dive). - **Complex Event Processing (CEP)**: Flink's CEP library enables sophisticated pattern matching over event streams, detecting event sequences, timeouts, and complex conditions like "detect three failed login attempts within 5 minutes followed by a successful login from a different location." This is powerful for fraud detection, security monitoring, and business process tracking. Kafka Streams handles simpler patterns through custom processors but requires significantly more code for advanced CEP scenarios. For practical CEP applications, see [Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming). ### Language Support Kafka Streams is Java-native with Scala support through wrapper libraries. Your application must run on the JVM. This limitation is rarely a problem for teams already using Java/Kotlin/Scala for backend services, but can be a barrier for Python-first organizations. Flink provides native APIs for Java, Scala, and Python. As of Flink 1.15+, the Python DataStream API has matured significantly, bringing near-feature-parity with Java/Scala for most streaming operations. The DataStream API offers fine-grained control over stream processing logic, while the Table API and Flink SQL provide higher-level, declarative abstractions for analytics use cases. Flink SQL in particular has become powerful in 2025, supporting complex streaming queries, temporal joins, and window aggregations with SQL syntax familiar to data analysts. ## When to Use Kafka Streams Choose Kafka Streams when: - **Your data lives in Kafka**: If you're already using Kafka for event streaming, Kafka Streams is the natural choice. No data movement required, and you leverage Kafka's partitioning and consumer groups for scaling. For foundational Kafka concepts, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) and [Kafka Topics, Partitions, Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). - **You prefer microservices architecture**: Kafka Streams applications deploy like any other microservice. They fit cleanly into containerized environments and don't require separate cluster infrastructure. - **State requirements are moderate**: For applications with gigabytes of state per instance, Kafka Streams' local storage model works well. Think fraud detection with recent transaction history, real-time aggregations over short windows, or event-driven microservices. - **Team expertise favors application development over cluster operations**: If your team is comfortable with Java application development but less experienced with distributed systems operations, Kafka Streams reduces operational burden. - **Example**: An e-commerce platform processing clickstream data to generate real-time product recommendations. Each Kafka Streams instance processes a subset of users, maintaining recent browsing history in local state stores. Scaling happens by adding instances, and the existing Kafka infrastructure handles message distribution. ## When to Use Apache Flink Choose Apache Flink when: - **You need advanced processing capabilities**: Complex event processing, ML model serving, iterative algorithms, or sophisticated windowing logic favor Flink's richer API surface. For ML use cases, see [Real-Time ML Inference with Streaming Data](https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data). - **State size is large or unbounded**: Applications with terabytes of state, or state that grows without bound, benefit from Flink's distributed state backend and checkpoint mechanism. - **Multiple data sources**: While Flink works excellently with Kafka, it also connects to databases, filesystems, message queues, and custom sources. If your pipeline integrates diverse systems, Flink's broad connector ecosystem helps. - **Batch and stream unification matters**: Flink's unified runtime lets you write logic once and execute it on both streaming and batch data, useful for backfilling or reprocessing historical data. - **You have cluster operations expertise**: Teams already running Kubernetes, YARN, or other cluster managers can leverage that expertise for Flink deployments. - **Example**: A financial services firm detecting complex fraud patterns across multiple event types (transactions, logins, device changes) with pattern matching over days of history. Flink's CEP library expresses multi-step fraud scenarios naturally, while distributed state handles millions of customer profiles. ## Monitoring and Development Tools Both frameworks benefit from proper tooling for development and operations. For Kafka Streams, monitoring happens through application metrics (JMX), Kafka consumer lag, and state store health. Since Kafka Streams applications are standard Java services, you can use familiar observability tools like Prometheus, Grafana, and application performance monitoring (APM) solutions. However, understanding the full picture requires correlating application metrics with underlying Kafka infrastructure, topic health, partition distribution, consumer group behavior, and data quality. This is where platforms like **Conduktor** excel, providing unified visibility into Kafka topics through [topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), consumer lag, schema evolution, and data governance policies. Conduktor's interface helps teams quickly identify whether issues stem from the Kafka Streams application logic or underlying Kafka infrastructure problems. For Flink, the built-in web UI provides comprehensive job graphs, checkpoint statistics, backpressure indicators, and task manager metrics. Third-party monitoring integrates with systems like Prometheus and Grafana for time-series metrics and alerting. The operational challenge is correlating Flink job metrics with source systems, particularly Kafka topics. **Conduktor** provides visibility into the Kafka topics that Flink reads from and writes to — consumer group lag, topic throughput, schema compliance, and governance policies — critical for debugging data pipeline issues at the Kafka layer. For development and testing, **Conduktor Gateway** enables chaos engineering scenarios on the Kafka side, letting teams test how their Flink jobs handle Kafka failures, latency injection, and schema evolution issues before production deployment. Understanding backpressure behavior is critical for both frameworks, see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) for strategies. For comprehensive testing approaches, refer to [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). ## Summary Kafka Streams and Apache Flink both deliver production-grade stream processing, but serve different needs: - **Choose Kafka Streams** for Kafka-centric pipelines with moderate state, when you value operational simplicity and microservices deployment patterns. It excels at event-driven applications and real-time aggregations within the Kafka ecosystem. - **Choose Apache Flink** for complex processing requirements, large state, multiple data sources, or when you need advanced features like CEP, ML integration, or unified batch/stream processing. Accept the operational complexity of managing a separate cluster in exchange for greater flexibility. Many organizations use both: Kafka Streams for microservices and simple transformations, Flink for complex analytics and ML pipelines. The key is matching the tool to the problem, not choosing one as the universal solution. For teams seeking SQL-based stream processing on Kafka, consider [ksqlDB for Real-Time Data Processing](https://www.conduktor.io/glossary/ksqldb-for-real-time-data-processing) as an alternative that provides declarative stream processing without writing application code. Start with your requirements: data sources, state size, processing complexity, and team capabilities. The right choice follows from there. ## Related Concepts - [Exactly-Once Semantics](https://www.conduktor.io/glossary/exactly-once-semantics) - Understanding exactly-once processing guarantees in distributed stream processing systems - [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) - Critical patterns for managing flow control in stream processing pipelines - [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns) - Comprehensive guide to joining streams and enriching data in real-time - [Kafka Streams vs Flink (hands-on guide)](https://www.conduktor.io/kafka-streams/vs-flink) - Pick by deployment model, with the operational trade-offs spelled out ## Sources and References 1. Apache Kafka Documentation - Kafka Streams Architecture: https://kafka.apache.org/documentation/streams/architecture 2. Apache Flink Documentation - Concepts and Architecture: https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/overview/ 3. Confluent - Kafka Streams vs Apache Flink: https://www.confluent.io/blog/kafka-streams-vs-apache-flink/ 4. Ververica - Stream Processing with Apache Flink: https://www.ververica.com/what-is-stream-processing 5. "Stream Processing with Apache Flink" by Fabian Hueske and Vasiliki Kalavri (O'Reilly Media, 2019) Source: https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink --- # Kafka to Iceberg: Topics as Tables **Kafka to Iceberg** is the pattern of exposing a Kafka topic as an [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg) table, making streaming records directly queryable by engines like Spark, Trino, and Snowflake. There are two types of implementations: - topic-native approaches, where the broker materializes the table from the topic's own storage - copy-based sinks, where an external connector writes a separate Iceberg dataset. ![kafka-to-iceberg-topics diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-to-iceberg-topics-0.webp) ## Why materialize topics as tables A Kafka topic and a table are optimized for opposite access patterns. - A topic is a log: consumers read it sequentially, in offset order, streaming the whole thing from a position forward. - A table is built for selective/random access: a transactional row store answers point lookups and updates, while an analytical table (Iceberg over columnar Parquet) answers aggregations by reading only the columns and files a query touches, through partition pruning and predicate pushdown. Exposing a topic as an Iceberg table is therefore a change of both paradigm and physical data layout, rewriting the same records from the log's sequential, record-at-a-time form into columnar files a query engine can scan selectively. A Kafka topic's records can be exposed as rows in a table. The goal is that analysts and BI tools, which do not speak the Kafka protocol, can query the data in SQL over Iceberg through Spark, Trino, Flink, or Snowflake. Historically the bridge was a simple integration with a connector: a consumer read the topic and wrote the records into an Iceberg table in object storage. It duplicates data and you have a connector to operate. The newer approaches move that write into the broker directly: the bytes Kafka already retains become the Iceberg data files, so a topic surfaces as a queryable table with no external connector to run, and in some designs, without duplicating data at all. ## Copy or Zero-Copy? **(A) Topic-native / broker-managed.** The broker materializes the Iceberg table without a connector. Most builds extend [tiered storage](https://www.conduktor.io/glossary/tiered-storage-in-kafka): segments landing in object storage are written as Parquet and committed to an Iceberg catalog (WarpStream and Confluent Cloud use their own object-storage materialization instead). Two types of implementations: - Zero-copy: Aiven's RemoteStorageManager keeps a single Parquet object that serves both Kafka fetch and Iceberg queries (no second copy) - With copy: Redpanda and Confluent Tableflow keep the log and write distinct Iceberg files **(B) External copy-based sinks.** A separate process like Kafka Connect or Flink consumes the topic and writes an independent Iceberg copy. | Dimension | (A) Topic-native / broker-managed | (B) Copy-based sink | |---|---|---| | Storage footprint | One shared object, no copy (Aiven RSM zero-copy); or a full second copy the broker writes (Redpanda, Confluent, WarpStream), same size as B | Always a full second copy of the topic | | Freshness | Bound to the segment-roll or materialization cadence; no connector interval to tune | Bound to the connector commit interval (default 5 min); fresher means more small files | | Operations | No connector to run; managed by the vendor, or self-hosted in the broker's storage plugin (Aiven) | You run and monitor Kafka Connect or Flink | | Table maintenance | Still required, since streaming writes create small files; automated by managed vendors, but not by Aiven's self-hosted alpha | Yours to schedule: compaction, snapshot expiry, orphan-file cleanup | | Lock-in / runs on | Requires a specific vendor's platform; the output is a standard Iceberg table any engine reads | Runs on any Kafka; open-source connector; the output is a standard Iceberg table too | ![kafka-to-iceberg-topics diagram 2](https://www.conduktor.io/assets/images/glossary/kafka-to-iceberg-topics-1.webp) ## Schema mapping: from Schema Registry to Iceberg schema All methods derive the Iceberg schema from the Avro, Protobuf, or JSON schema managed in [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management), mapping each field to an Iceberg column type. Inferring a schema from raw JSON is possible, but it often produces incorrect or suboptimal types (everything as string, numeric widths guessed), so a registry-backed subject is strongly preferred. Schema evolution is handled in various ways: - Tableflow auto-evolves the table as the subject evolves (adding columns, widening types). - Connect evolves when `iceberg.tables.evolve-schema-enabled` is on. - Aiven's alpha build, by contrast, freezes the table on the first record's schema and does not yet handle evolution. ## Compaction and table maintenance Streaming's continuous writes create a housekeeping burden. Every commit interval or segment roll adds a small Parquet file, so a busy topic can produce thousands of tiny files an hour, and many small files slow reads: a query plans and opens across all of them, and per-file overhead dominates the scan. **Compaction** must run regularly. It rewrites many small data files into fewer large ones (typically a few hundred MB each), which keeps scans fast and metadata small. It also folds the accumulated delete files into the data files so readers no longer reconcile deletes at query time. This is one of several jobs every Iceberg table needs: - **Snapshot expiry** drops old snapshots so history and metadata do not grow without bound. - **Orphan-file cleanup** removes data files that no live snapshot references, left by failed or partial writes. - **Manifest rewrites** keep the manifest lists compact so query planning stays cheap. Managed offerings do it for you. If you run the sink yourself, that maintenance is yours to schedule. See [Maintaining Iceberg Tables](https://www.conduktor.io/glossary/maintaining-iceberg-tables-compaction-and-cleanup) and [Iceberg catalog management](https://www.conduktor.io/glossary/iceberg-catalog-management-hive-glue-and-nessie) for the deeper mechanics. **Does the topic-native path double my storage?** It depends on the design. Aiven's RemoteStorageManager stores a single Parquet object that serves both Kafka replay and Iceberg queries, so there is no duplication. Redpanda and Tableflow materialize broker-managed Iceberg files rather than a copy you operate. Copy-based sinks, by contrast, do store a full second copy you run and maintain. **Do I still need an Iceberg catalog?** Yes. Both families register the table in a catalog (REST, Glue, Nessie, Hive, or a vendor's open catalog). The catalog is what lets Spark, Trino, or Snowflake discover and read the table. **Is exactly-once guaranteed?** On the copy path, the Apache Iceberg Connect sink provides exactly-once via [KIP-447](https://kafka-options-explorer.conduktor.io/kip/447/) (Kafka 2.5+) plus a control topic for commit coordination. On the topic-native path it depends on the design: in the zero-copy approach (Aiven RSM) the analytical view inherits the broker's own durability because there is no separate copy to reconcile, while Redpanda and Tableflow reconcile a broker-managed materialization that the platform commits for you. **Can I backfill historical topic data into Iceberg?** It depends on the implementation. A copy-based sink can consume from the earliest offset. Redpanda, however, does not backfill an existing topic when you enable Iceberg; only records produced afterward appear in the table. **Which approach should I choose?** Choose topic-native for zero extra storage and no connector to operate when you are on a platform that offers it. Choose the copy-based Connect sink when you need an open, portable, engine-agnostic path across catalogs and are willing to run and maintain it. ## Related Pages - [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables): the broader pattern of landing streams as governed lakehouse tables. - [Streaming Ingestion to Lakehouse](https://www.conduktor.io/glossary/streaming-ingestion-to-lakehouse): ingestion architectures that feed lake and warehouse engines. - [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg): the open table format these approaches target. - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka): the KIP-405 mechanism the zero-copy builds extend. - [Iceberg Catalog Management: Hive, Glue, and Nessie](https://www.conduktor.io/glossary/iceberg-catalog-management-hive-glue-and-nessie): how tables are registered and discovered. - [Zero-Copy Data Sharing](https://www.conduktor.io/glossary/zero-copy-data-sharing): the principle of one dataset serving multiple views. - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management): the source of the Iceberg schema in registry-driven paths. ## Sources - [Confluent, Inside Tableflow GA: Real-Time Kafka to Iceberg](https://www.confluent.io/blog/tableflow-ga-kafka-snowflake-iceberg/) - [Confluent Docs, Tableflow: Kafka Topics as Iceberg and Delta Lake Tables](https://docs.confluent.io/cloud/current/topics/tableflow/overview.html) - [Redpanda, 25.1: Iceberg Topics now generally available](https://www.redpanda.com/blog/redpanda-25-1-iceberg-topics-ga) - [Redpanda Docs, About Iceberg Topics](https://docs.redpanda.com/current/manage/iceberg/about-iceberg-topics/) - [Aiven, Iceberg Topics for Apache Kafka: Zero ETL, Zero Copy](https://aiven.io/blog/iceberg-topics-for-apache-kafka-zero-etl-zero-copy) - [Aiven, tiered-storage-for-apache-kafka: Iceberg whitepaper](https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/blob/main/iceberg_whitepaper.md) - [WarpStream Docs, Tableflow](https://docs.warpstream.com/warpstream/tableflow/tableflow) - [Apache Iceberg, Kafka Connect sink docs](https://iceberg.apache.org/docs/nightly/kafka-connect/) - [KIP-405: Kafka Tiered Storage](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405:+Kafka+Tiered+Storage) - [KIP-447: Producer scalability for exactly-once semantics](https://cwiki.apache.org/confluence/display/KAFKA/KIP-447:+Producer+scalability+for+exactly+once+semantics) Source: https://www.conduktor.io/glossary/kafka-to-iceberg-topics --- # Kafka Topic Design Guidelines Kafka topic design affects performance, scalability, and maintainability across your entire streaming architecture. Poor decisions here lead to operational complexity, data inconsistency, and difficulty evolving your system. This article covers key guidelines for well-structured topics. ![Kafka topic design decision framework](https://www.conduktor.io/assets/images/glossary/kafka-topic-design-guidelines-0.webp) ## Why Topic Design Matters Topics are the fundamental unit of organization in Apache Kafka. They define how data flows through your system, how it's partitioned across brokers, and how consumers access it. Unlike traditional databases where schema changes are common, topic design decisions in Kafka are harder to reverse once producers and consumers are deployed. A well-designed topic structure supports multiple teams working independently, enables efficient data processing, and makes governance manageable. Conversely, poorly designed topics can create bottlenecks, increase operational overhead, and make debugging difficult. Modern Kafka (4.0+ with KRaft mode) has removed many historical limitations around partition counts and cluster management, enabling more flexible topic designs. However, the core principles of good topic architecture remain essential for building scalable streaming systems. For foundational understanding of Kafka's architecture, see [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram). ## Naming Conventions Consistent naming conventions make Kafka clusters manageable as they grow. A clear naming scheme helps teams understand data ownership, purpose, and lifecycle at a glance. ### Common Patterns Many organizations adopt hierarchical naming patterns such as: - `..`: `payments.order.created`, `inventory.product.updated` - `..`: `prod.analytics.user-events`, `staging.fraud.alerts` - `.`: `public.metrics`, `confidential.customer-pii` The key is choosing a pattern that reflects your organizational structure and sticking to it consistently. Avoid generic names like `data`, `events`, or `topic1` that provide no context. ### Versioning For topics that will evolve over time, consider including version information in the name: `orders.v2`, `user-events-v3`. This allows you to run multiple schema versions in parallel during migrations without breaking existing consumers. ## Partitioning Strategy Partitions are Kafka's unit of parallelism. The number of partitions determines how many consumers can read from a topic concurrently and affects throughput and latency. ### Determining Partition Count Start with your throughput requirements. If you need to process 100 MB/s and each consumer can handle 10 MB/s, you need at least 10 partitions. Also consider: - **Consumer parallelism**: You can't have more active consumers in a consumer group than partitions (see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained)) - **Broker distribution**: Partitions should be distributed across brokers for fault tolerance - **Future growth**: Adding partitions later affects message ordering guarantees - **Cluster capabilities**: Kafka 4.0 with KRaft mode eliminates the ZooKeeper-era partition scaling limitations, supporting 100,000+ partitions per cluster efficiently **2025 Guidance**: Start with 12-30 partitions for production topics to allow for growth. The old conservative guidance of 6-12 partitions was based on ZooKeeper's metadata limitations, which no longer apply with KRaft. However, don't over-partition unnecessarily, each partition adds overhead for replication and broker management. For capacity planning and partition sizing calculations, refer to [Kafka Capacity Planning and Performance Tuning](https://www.conduktor.io/glossary/kafka-capacity-planning-performance-tuning). ### Partition Key Selection The partition key determines which partition a message goes to. Choosing the right key is critical for maintaining ordering guarantees and load distribution. **Example**: For a user activity tracking system, using `user_id` as the partition key ensures all events for a specific user go to the same partition, maintaining per-user ordering. However, if some users are extremely active, this could create hot partitions. In such cases, consider composite keys like `user_id + session_id` to distribute load more evenly. When combining keys, concatenate them with a delimiter (e.g., `"user123:session456"`) or use a hashing function that combines both values. ```java // Example: Composite partition key using Kafka producer String partitionKey = userId + ":" + sessionId; ProducerRecord record = new ProducerRecord<>( "user-events", partitionKey, // Composite key event ); ``` Avoid using keys that could lead to skewed data distribution, such as timestamps (if most events happen during business hours) or geographic regions (if one region dominates traffic). ## Replication and Durability Replication protects against data loss when brokers fail. The replication factor determines how many copies of each partition exist across the cluster. ### Standard Configurations - **Replication factor of 3**: The industry standard for production systems, providing good fault tolerance without excessive storage costs - **Minimum in-sync replicas (min.isr) of 2**: Ensures writes are acknowledged by at least 2 replicas before confirming success - **acks=all**: Producer configuration ensuring writes succeed on all in-sync replicas These settings balance durability with performance. For non-critical data like metrics or logs, you might use a replication factor of 2. For financial transactions or critical state, maintain factor of 3 or higher. **Configuration Example**: ```properties # Topic-level configuration (topic creation or alter) replication.factor=3 min.insync.replicas=2 ``` ```java // Producer-side durability configuration Properties props = new Properties(); props.put("acks", "all"); // Wait for all in-sync replicas props.put("enable.idempotence", "true"); // Kafka 3.0+ default props.put("max.in.flight.requests.per.connection", "5"); ``` ### Advanced Replication Features (Kafka 3.0+) - **Rack awareness**: Distribute replicas across availability zones to survive zone failures - **Follower fetching (KIP-392)**: Consumers can read from nearby follower replicas to reduce cross-AZ traffic and improve latency - **Tiered Storage Integration**: Replicas in hot tier with historical data in object storage For comprehensive replication architecture, see [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability). ## Retention Policies Retention policies determine how long Kafka keeps messages. Unlike traditional message queues where messages are deleted after consumption, Kafka retains data based on time or size limits. ### Time-Based Retention Common patterns include: - **Short retention (hours to days)**: Real-time event streams, operational metrics - **Medium retention (days to weeks)**: Application logs, user activity events - **Long retention (months to years)**: Audit logs, compliance data, event sourcing - **Infinite retention with tiered storage (2025)**: Historical data in object storage with recent data on local disks **Example**: A topic containing IoT sensor readings might use 7-day retention on local storage for operational monitoring, while a topic storing financial transactions might retain data for 7 years for regulatory compliance using tiered storage. ```properties # Short retention example (7 days on local storage) retention.ms=604800000 # Medium retention (30 days) retention.ms=2592000000 # Tiered storage configuration (Kafka 3.6+) # Keep 7 days locally, indefinite in object storage local.retention.ms=604800000 retention.ms=-1 # Infinite with tiered storage ``` ### Size-Based Retention For topics with unpredictable message rates, size-based retention (`retention.bytes`) prevents uncontrolled storage growth. Combine time and size limits for robust lifecycle management. ```properties # Limit topic size to 100GB per partition retention.bytes=107374182400 ``` ### Compaction Log compaction is useful for changelog topics where only the latest state per key matters. Compacted topics retain the most recent message for each key indefinitely, making them ideal for building materialized views or caching layers. **Example**: A `product-catalog` topic with product IDs as keys. As products are updated, old versions are eventually removed, but the current state of every product remains available forever. ```properties # Enable log compaction cleanup.policy=compact # Hybrid: delete old data AND compact cleanup.policy=compact,delete retention.ms=2592000000 # 30 days before eligible for deletion ``` ### Tiered Storage for Long Retention (Kafka 3.6+) Tiered storage (KIP-405) offloads older segments to cost-effective object storage (S3, GCS, Azure Blob) while keeping recent data on fast local disks. This enables: - **Infinite retention** at low cost (pennies per GB/month vs. dollars for SSD) - **No partition reassignment** when scaling storage - **Time-travel queries** on historical data without separate archival systems **Configuration Example**: ```properties # Enable tiered storage remote.log.storage.enable=true remote.log.storage.system.enable=true # Keep only 3 days locally, rest in object storage local.retention.ms=259200000 local.retention.bytes=107374182400 # Infinite total retention retention.ms=-1 retention.bytes=-1 ``` For detailed tiered storage architecture and use cases, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). ## Schema Design Considerations While Kafka topics don't enforce schemas, planning for schema evolution is essential for long-term maintainability. ### Schema Registry Integration Using a schema registry enables: - **Centralized schema management**: Single source of truth for message formats - **Compatibility enforcement**: Prevents incompatible schema changes - **Version tracking**: Clear history of schema evolution **Popular Schema Registry Options (2025)**: - **Apache Schema Registry implementations**: Open-source, compatible with multiple formats (Avro, Protobuf, JSON Schema) - **Apicurio Registry**: Open-source with REST API and web console - **AWS Glue Schema Registry**: Integrated with AWS services - **Conduktor**: Commercial platform providing visual schema management, compatibility testing, and governance workflows across multiple clusters For comprehensive schema management practices, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). ### Evolution Strategies Design schemas with evolution in mind: - Use optional fields for new attributes - Never remove required fields - Avoid changing field types - Use enums carefully (adding values is safe, removing is not) **Example of Safe vs. Unsafe Evolution**: ```json // Original schema (v1) { "type": "record", "name": "UserEvent", "fields": [ {"name": "userId", "type": "string"}, {"name": "eventType", "type": "string"} ] } // ✅ SAFE: Adding optional field with default { "type": "record", "name": "UserEvent", "fields": [ {"name": "userId", "type": "string"}, {"name": "eventType", "type": "string"}, {"name": "timestamp", "type": ["null", "long"], "default": null} ] } // ❌ UNSAFE: Removing required field breaks old consumers { "type": "record", "name": "UserEvent", "fields": [ {"name": "userId", "type": "string"} // eventType removed - old consumers will fail! ] } ``` Formats like Avro, Protobuf, and JSON Schema support backward and forward compatibility when followed correctly. See [Avro vs Protobuf vs JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema) for format-specific guidance. ## Organizational and Governance Patterns As Kafka deployments grow, governance becomes critical. Without clear ownership and policies, topic proliferation leads to confusion and maintenance burden. ### Topic Ownership Define clear ownership for each topic: - Who approves schema changes? - Who monitors data quality? - Who handles incidents? Metadata systems or topic naming conventions can encode ownership information. Some organizations use topic prefixes to indicate owning teams: `team-payments.orders.created`. ### Access Control Implement fine-grained ACLs (Access Control Lists) to control who can: - Produce to topics - Consume from topics - Create or delete topics - Modify configurations **ACL Configuration Example**: ```bash # Grant producer access to a specific topic kafka-acls --bootstrap-server localhost:9092 \ --add --allow-principal User:app-producer \ --operation Write --topic payments.order.created # Grant consumer group access kafka-acls --bootstrap-server localhost:9092 \ --add --allow-principal User:analytics-service \ --operation Read --topic payments.order.created \ --group analytics-consumer-group ``` **2025 Governance Tools**: - **Conduktor**: Centralized ACL management with visual interface, approval workflows, and audit trails across multiple clusters. For implementing self-service topic creation with governance policies, see the [Self-Service documentation](https://docs.conduktor.io/guide/conduktor-concepts/self-service) and [Topics Management guide](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) - **OPA (Open Policy Agent)**: Policy-as-code for automated authorization decisions - **Kafka KRaft ACLs**: Simplified ACL storage and management without ZooKeeper dependency For comprehensive security practices, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) and [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming). ### Topic Lifecycle Management Establish processes for: - **Creation**: Approval workflows, configuration templates - **Monitoring**: Data quality checks, lag alerts, throughput metrics - **Deprecation**: Migration plans, sunset timelines, archival procedures Document these processes and enforce them through automation where possible. ### Logical Topics: Abstraction Layer for Topic Design Flexibility Conduktor's [Logical Topics](https://docs.conduktor.io/guide/conduktor-concepts/logical-topics) provide an abstraction layer that addresses common topic design challenges. **Alias Topics** solve Kafka's inability to rename topics, teams share topics using business-friendly names while maintaining internal conventions. **Concentrated Topics** dramatically reduce partition counts by co-locating multiple logical topics on a single physical topic (e.g., 4 regional topics with 400 partitions consolidate to 100 partitions, 75% reduction). **SQL Topics** enable filtering at the topic level using SQL WHERE clauses. These abstractions reduce topic proliferation and partition overhead while maintaining logical separation for producers and consumers, particularly valuable for multi-tenant architectures or regional deployments. ## Connecting to Data Streaming Architecture Kafka topics connect the components of modern streaming architectures. They integrate with: - **Stream processing frameworks**: Kafka Streams, Apache Flink, and Spark Structured Streaming consume from and produce to topics. See [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink) for framework selection guidance. - **CDC pipelines**: Change Data Capture tools like Debezium write database changes to topics. See [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) for topic design considerations in CDC scenarios. - **Data lakes and lakehouses**: Connectors stream data from topics to object storage and table formats. See [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables) for integration patterns. - **Microservices**: Services communicate asynchronously through topics using event-driven patterns. See [CQRS and Event Sourcing with Kafka](https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka) for architectural patterns. Proper topic design ensures these integrations remain maintainable as your architecture evolves: - **Well-partitioned topics** enable parallel processing in Flink jobs without consumer lag bottlenecks - **Consistent naming conventions** help data engineers discover relevant data for analytics pipelines - **Appropriate retention policies** ensure downstream systems can replay data during outages or migrations - **Schema enforcement** prevents breaking changes from propagating through dependent systems **Example**: An e-commerce platform might design topics like: - `orders.created` (30 partitions, 7-day retention) → feeds real-time analytics dashboards - `orders.enriched` (30 partitions, 30-day retention + tiered storage) → used by data warehouse ETL jobs - `customer.profile.changelog` (12 partitions, compacted) → maintains customer state for microservices caches ## Summary Effective Kafka topic design requires balancing technical constraints with organizational needs. Key guidelines include: - **Adopt consistent naming conventions** that reflect your domain and team structure - **Choose partition counts and keys** based on throughput requirements and ordering guarantees - Modern Kafka (4.0+ with KRaft) supports 100k+ partitions per cluster - Start with 12-30 partitions for production topics - **Configure replication** for appropriate durability without excessive overhead - Standard: replication factor 3, min.isr 2, acks=all - Use rack awareness and follower fetching for improved availability and performance - **Set retention policies** aligned with data lifecycle and compliance requirements - Leverage tiered storage (Kafka 3.6+) for cost-effective long retention - Use log compaction for changelog topics - **Plan for schema evolution** using schema registries and compatible change practices - Always add optional fields, never remove required fields - Test compatibility before deploying changes - **Establish clear governance processes** for topic ownership, access control, and lifecycle management - Use tooling like Conduktor for centralized governance - Implement ACLs following least-privilege principles **2025 Key Enhancements**: - **KRaft mode** eliminates ZooKeeper-era partition scaling limitations - **Tiered storage** enables infinite retention at low cost - **Follower fetching** improves read performance across availability zones - **Enhanced ACL management** with KRaft's simplified metadata model These principles help build Kafka deployments that scale efficiently and stay maintainable as your architecture grows. ## Related Concepts - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Essential for enforcing schema compatibility rules that prevent breaking changes in topic evolution. - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka) - Modern retention strategy that separates hot and cold data for cost-effective long-term storage. - [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) - Critical for implementing access control and encryption policies at the topic level. ## Related Articles - [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram) - Foundational concepts - [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) - ZooKeeper-free architecture - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka) - Long-term retention strategies - [Kafka Replication and High Availability](https://www.conduktor.io/glossary/kafka-replication-and-high-availability) - Durability and fault tolerance - [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) - Consumer parallelism and partition assignment - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing message schemas - [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) - Safe schema changes - [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) - Authentication and authorization - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Tracking partition consumption ## Sources and References 1. Apache Kafka Documentation - Topic Configuration: https://kafka.apache.org/documentation/#topicconfigs 2. "Kafka: The Definitive Guide, 2nd Edition" by Gwen Shapira, Todd Palino, Rajini Sivaram, and Krit Petty (O'Reilly Media, 2021) 3. KIP-405: Kafka Tiered Storage: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage 4. KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum (KRaft): https://cwiki.apache.org/confluence/display/KAFKA/KIP-500 5. KIP-392: Allow consumers to fetch from closest replica: https://cwiki.apache.org/confluence/display/KAFKA/KIP-392 6. Jay Kreps - "The Log: What every software engineer should know about real-time data's unifying abstraction": https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying Source: https://www.conduktor.io/glossary/kafka-topic-design-guidelines --- # What is a Kafka Topic? Definition & Examples A **Kafka topic** is a named, durable, append-only log to which producers write records and from which consumers read. Topics are the fundamental organizational unit in Kafka: every record belongs to exactly one topic, and every producer and consumer references topics by name. Topics are immutable after write and retained independently of consumption. ## What Is a Kafka Topic? Think of a topic as a category or feed. `user-events`, `payment-transactions`, and `audit-logs` are all topics. Producers publish records to a topic; any number of consumer groups can independently read the same topic without interfering with each other. Topics are **schema-agnostic at the broker level**. Brokers store raw bytes. Producers and consumers agree on a data format (JSON, Avro, Protobuf) externally, typically enforced via a Schema Registry. See [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) for details. ## Structure: The Append-Only Log Each topic is physically implemented as one or more partition logs (see [Kafka Partitions Explained](https://www.conduktor.io/glossary/kafka-partitions-explained)). Within a partition, records are: - **Append-only**: new records are always added to the tail - **Immutable**: once written, a record cannot be updated or deleted - **Offset-indexed**: each record has a monotonically increasing integer offset, unique within its partition ``` Partition 0: [offset 0][offset 1][offset 2][offset 3] … Partition 1: [offset 0][offset 1][offset 2] … Partition 2: [offset 0][offset 1][offset 2] … ``` ![Kafka topic partitions diagram](https://www.conduktor.io/assets/images/glossary/kafka-topics-partitions-brokers-core-architecture-0.webp) Offsets are **per-partition**, not per-topic. A consumer tracks its position in each partition independently. This design enables replay: a consumer can reset its offset to any past position and re-read records as many times as needed. ## Topics vs. Partitions A topic is the **logical** unit; a partition is the **physical** unit. When you create a topic you choose how many partitions it has. Partitions determine: - **Parallelism**: each partition can be consumed by one consumer within a consumer group simultaneously - **Ordering**: ordering is guaranteed within a partition, not across partitions - **Distribution**: partitions are spread across brokers for load balancing For a full treatment of how partitions work, how leaders and followers are elected, and how to choose partition counts, see [What is a Kafka Partition?](https://www.conduktor.io/glossary/kafka-partitions-explained). ## Retention: Data Lives Until Expired, Not Until Consumed Unlike a queue, Kafka does not delete records when they are consumed. Retention is controlled by two independent policies: | Policy | Config key | Default | |--------|-----------|---------| | Time-based | `retention.ms` | 7 days (604 800 000 ms) | | Size-based | `retention.bytes` | -1 (unlimited) | When both are set, whichever limit is hit first triggers deletion of the oldest log segment. This allows multiple consumer groups to read at their own pace, and enables replay for new consumers joining later. **Log compaction** is an alternative retention strategy for changelog-style topics (e.g., a CDC stream for a database table). Kafka retains the latest record per key indefinitely, allowing consumers to reconstruct current state. See [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained). ## Naming Conventions Topic names must match `[a-zA-Z0-9._-]+`. Common conventions in production: - **Dot-separated namespacing**: `payments.transactions.created`, `users.profile.updated` - **Service-prefixed**: `orders-service.order-placed` - **Environment-prefixed**: `prod.payments.refunds` Avoid dots and underscores in the same name, Kafka internally converts dots to underscores in some metrics contexts, causing collisions. Pick one separator and be consistent. ## Common Topic Operations ### Create a topic ```bash # Kafka 4.0+ with KRaft kafka-topics.sh --bootstrap-server localhost:9092 \ --create \ --topic user-events \ --partitions 10 \ --replication-factor 3 \ --config min.insync.replicas=2 \ --config retention.ms=604800000 ``` ### List topics ```bash kafka-topics.sh --bootstrap-server localhost:9092 --list ``` ### Describe a topic ```bash kafka-topics.sh --bootstrap-server localhost:9092 \ --describe \ --topic user-events ``` Output shows partition count, replication factor, leader broker, ISR list, and any non-default configs. ### Alter configuration ```bash kafka-configs.sh --bootstrap-server localhost:9092 \ --entity-type topics \ --entity-name user-events \ --alter \ --add-config retention.ms=86400000 ``` ### Delete a topic ```bash kafka-topics.sh --bootstrap-server localhost:9092 \ --delete \ --topic user-events ``` Deletion is asynchronous. The topic is marked for deletion and removed once all replicas have been cleaned up. ## Multi-Subscriber Model Multiple independent consumer groups can read the same topic simultaneously without any coordination. Each group maintains its own committed offsets. This is one of Kafka's key architectural advantages over traditional message queues: - An analytics pipeline and an alerting system can both consume `payment-transactions` without one blocking or competing with the other - A new consumer group starts at the earliest available offset by default, enabling replay - Consumer groups scale horizontally: add consumers up to the partition count for linear throughput increase See [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) for group coordination, rebalancing, and offset management details. ## Topic Design Guidelines Choosing the right topic granularity matters: - **One event type per topic**: mixing unrelated event types in one topic complicates consumer logic and schema management - **Avoid too many small topics**: Kafka has per-partition overhead; thousands of single-partition topics can strain the controller - **Plan for partition count early**: increasing partitions later breaks key-based ordering guarantees for existing consumers For comprehensive guidance on topic structure, key selection, and sizing calculations, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) and [Kafka Partitioning Strategies and Best Practices](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). ## Managing Topics with Conduktor At scale, managing topics across multiple clusters requires tooling beyond the CLI. Conduktor provides a UI and API for creating, configuring, and monitoring topics across clusters, including visibility into partition distribution, consumer lag per partition, and configuration drift. See the [Topics Management documentation](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) for details. ## Related Pages - [What is a Kafka Partition?](https://www.conduktor.io/glossary/kafka-partitions-explained), how topics are physically split for parallelism and ordering - [What is a Kafka Broker?](https://www.conduktor.io/glossary/kafka-brokers-explained), where topic partitions are stored and served - [Kafka Architecture: Diagram & Components](https://www.conduktor.io/glossary/kafka-architecture-diagram), how topics, partitions, and brokers fit together - [Kafka Producers and Consumers](https://www.conduktor.io/glossary/kafka-producers-and-consumers), how records are written to and read from topics - [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines), practical rules for naming, sizing, and structuring topics - [Kafka Log Compaction Explained](https://www.conduktor.io/glossary/kafka-log-compaction-explained), alternative retention for stateful topics - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka), overview of the Kafka ecosystem ## Sources - [Apache Kafka 4.0 Documentation, Core Concepts](https://kafka.apache.org/documentation/#intro_concepts_and_terms) - [Apache Kafka, Topic Configuration Reference](https://kafka.apache.org/documentation/#topicconfigs) - [Kafka: The Definitive Guide, 2nd Edition](https://www.confluent.io/resources/kafka-the-definitive-guide/), Chapter 3: Kafka Producers Source: https://www.conduktor.io/glossary/kafka-topics-explained --- # Kafka Transactions Deep Dive In distributed systems, processing messages exactly once is hard. Traditional message delivery offers two guarantees: at-most-once (messages may be lost) and at-least-once (messages may be duplicated). For critical applications like financial transactions, inventory management, or payment processing, neither is acceptable. Kafka transactions solve this by enabling exactly-once semantics (EOS) across producers, brokers, and consumers. This article is a technical deep dive into Kafka's transaction implementation. For a broader overview of exactly-once semantics including use cases and simpler explanations, see [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka). For foundational Kafka concepts, refer to [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ![Kafka transaction flow with coordinator](https://www.conduktor.io/assets/images/glossary/kafka-transactions-deep-dive-0.webp) ## Understanding Kafka Transactions Kafka transactions allow producers to send multiple messages to multiple partitions atomically. Either all messages in a transaction are committed and become visible to consumers, or none are. This atomic guarantee extends across multiple topic partitions and even spans multiple Kafka clusters in some configurations. ### The Core Problem Consider a stream processing application that reads from topic A, transforms the data, and writes results to topic B. Without transactions, failures can lead to: - **Duplicate writes**: A producer crashes after writing to topic B but before committing its consumer offset for topic A - **Data loss**: Consumer commits offset before processing completes - **Inconsistent state**: Partial writes when processing multiple output records Transactions solve these issues by coordinating producer writes and consumer offset commits in a single atomic operation. ## How Kafka Transactions Work Kafka's transactional mechanism relies on several key components working together to maintain consistency across distributed operations. ### Transaction Coordinator The transaction coordinator is a broker-side component responsible for managing transaction state. Each producer with transactions enabled is assigned a transaction coordinator based on its `transactional.id`. The coordinator: - Assigns Producer IDs (PIDs) and epoch numbers for idempotency - Maintains transaction logs in the internal `__transaction_state` topic (which stores transaction metadata, state transitions, and participant partitions) - Coordinates the two-phase commit protocol across partitions (a distributed algorithm ensuring all participants agree to commit or abort) - Handles transaction timeouts and recovery In Kafka 4.0+ with KRaft mode (Kafka's ZooKeeper-free architecture), transaction coordinator performance has been significantly improved. KRaft-based transaction coordinators provide faster transaction commits, reduced coordinator overhead, and better scalability compared to the legacy ZooKeeper-based implementation. For more on KRaft's benefits, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ### Transactional Producer Flow When a producer initiates a transaction, the following sequence occurs: 1. **Initialization**: Producer registers with the transaction coordinator using its `transactional.id` 2. **Begin Transaction**: Producer starts a transaction locally 3. **Add Partitions**: As the producer writes to partitions, the coordinator tracks which partitions are involved 4. **Commit/Abort**: Producer sends commit or abort request to coordinator 5. **Two-Phase Commit**: Coordinator writes commit markers to all involved partitions and updates the transaction log ### Control Messages and Consumer Isolation Kafka uses special control messages (`COMMIT` and `ABORT` markers) written to partitions to signal transaction boundaries. Consumers configured with `isolation.level=read_committed` only see messages from committed transactions, while those set to `read_uncommitted` see all messages. ## Transactional APIs and Configuration Implementing transactions requires specific configuration on both producers and consumers. ### Producer Configuration ```java Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("transactional.id", "my-transactional-producer"); // Note: enable.idempotence is default true in Kafka 3.0+, explicit setting optional props.put("enable.idempotence", "true"); props.put("acks", "all"); KafkaProducer producer = new KafkaProducer<>(props); producer.initTransactions(); try { producer.beginTransaction(); producer.send(new ProducerRecord<>("topic-a", "key", "value1")); producer.send(new ProducerRecord<>("topic-b", "key", "value2")); producer.commitTransaction(); } catch (Exception e) { producer.abortTransaction(); } ``` The `transactional.id` must be unique per producer instance and enables the coordinator to identify and fence zombie producers (producers that appear to have failed but resume later). Note that idempotent producers (`enable.idempotence=true`) are the default since Kafka 3.0+, providing automatic duplicate prevention even without explicit transactions. For comprehensive coverage of producer configuration and idempotence, see [Kafka Producers](https://www.conduktor.io/glossary/kafka-producers). ### Consumer Configuration ```java Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("group.id", "my-consumer-group"); props.put("isolation.level", "read_committed"); props.put("enable.auto.commit", "false"); KafkaConsumer consumer = new KafkaConsumer<>(props); ``` Disabling auto-commit is crucial when using transactions, as you'll manually commit offsets within the transaction boundary. For more details on consumer group coordination and offset management, see [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). ### Consume-Process-Produce Pattern The most common transactional pattern combines consumption and production: ```java while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); producer.beginTransaction(); try { for (ConsumerRecord record : records) { // Process and produce ProducerRecord output = transform(record); producer.send(output); } // Commit consumer offsets as part of transaction producer.sendOffsetsToTransaction( getOffsets(records), consumer.groupMetadata() ); producer.commitTransaction(); } catch (Exception e) { producer.abortTransaction(); } } ``` ## Real-World Use Cases ### Financial Payment Processing A payment processing system reads payment requests from a Kafka topic, validates them, debits accounts, and writes confirmation records. Transactions ensure that if any step fails, the entire operation rolls back, preventing partial payments or duplicate charges. For patterns implementing reliable payment processing and financial data flows, see [Streaming Data in Financial Services](https://www.conduktor.io/glossary/streaming-data-in-financial-services). ### Multi-System Data Synchronization When synchronizing data between a database and Kafka, transactions guarantee that database changes and Kafka messages are committed together. For example, an e-commerce platform updating inventory in PostgreSQL while publishing availability events to Kafka can use transactions to maintain consistency. For coordinating transactions across multiple microservices using event-driven patterns, see [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions). For reliable event publishing from databases, the [Outbox Pattern for Reliable Event Publishing](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing) provides complementary guarantees. ### Stream Processing Pipelines Kafka Streams and Flink use transactions internally to provide exactly-once processing guarantees. When aggregating real-time analytics or joining multiple streams, transactions ensure that state stores and output topics remain consistent even during failures. Kafka Streams leverages the `exactly_once_v2` protocol (introduced in Kafka 2.5) for improved performance, and Kafka 4.0+ further optimizes transaction handling in KRaft mode. For a detailed comparison of these frameworks and their transactional capabilities, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink). ## Monitoring and Troubleshooting Transactions Understanding transaction health requires monitoring several key metrics and understanding common failure patterns. ### Key Metrics Monitor these critical transaction metrics to ensure healthy operation: - `transaction-state`: Number of transactions in each state (ongoing, prepare commit, completed) - helps identify stuck transactions - `txn-commit-time-ms`: Time taken to commit transactions - increases may indicate coordinator or network issues - `producer-id-expiration-time-ms`: Time before inactive producer IDs are expired - tracks producer lifecycle - `last-stable-offset-lag`: Gap between log end offset and last stable offset (LSO) - measures the backlog of uncommitted transactions waiting to be read by `read_committed` consumers - **Open-source monitoring**: For Prometheus-based monitoring, Kafka Lag Exporter (the 2025 standard for consumer lag tracking) provides transaction-aware metrics. JMX metrics can be scraped using Prometheus JMX Exporter to track transaction coordinator performance. Tools like Burrow can detect consumer lag issues related to transaction processing delays. - **Commercial platforms**: Tools like Conduktor provide comprehensive visibility into transactional flows through [topic monitoring](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), allowing teams to visualize transaction states, identify hanging transactions, and debug configuration issues. The platform can surface warnings when transaction timeouts are misconfigured or when zombie producers are detected, and provides real-time alerts on transaction performance degradation. Conduktor Gateway, a Kafka proxy, also enables chaos testing of transactional scenarios by injecting faults, latency, and network partitions to validate transaction behavior under failure conditions. For comprehensive testing strategies including transaction testing, see [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications) and [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems). ### Common Issues - **Transaction Timeout**: If `transaction.timeout.ms` is too short for your processing time, transactions will abort automatically. Monitor processing times and adjust accordingly. - **Zombie Fencing**: When a producer is fenced (its epoch is incremented), it can no longer commit transactions. This is by design to prevent split-brain scenarios, but requires proper producer lifecycle management. - **Coordinator Overload**: High transaction volumes can overload the transaction coordinator. Consider partitioning transactional load across multiple `transactional.id` values. ## Best Practices and Considerations ### When to Use Transactions Transactions add latency and complexity. Use them when: - Exactly-once semantics are required for correctness - Processing spans multiple partitions or topics - You're implementing consume-process-produce patterns Avoid transactions for: - Simple logging or metrics where occasional duplicates are acceptable - Ultra-low latency scenarios where the overhead is prohibitive - Single-partition writes where idempotence alone suffices ### Performance Implications Transactions introduce several performance considerations: - **Increased latency**: Two-phase commit adds network round trips - **Throughput impact**: Transaction state management consumes broker resources - **Storage overhead**: Control messages and transaction logs require additional storage Batch messages within transactions to amortize the coordination overhead. A transaction writing 100 messages is much more efficient than 100 transactions writing one message each. ### Idempotence vs. Transactions Kafka's idempotent producer (`enable.idempotence=true`, default since Kafka 3.0+) prevents duplicate writes within a single partition but doesn't coordinate across partitions or provide atomic multi-partition writes. Transactions build on idempotence to provide cross-partition atomicity and exactly-once semantics (EOS). Use idempotence alone when you only need duplicate prevention without multi-partition coordination. For a broader overview of exactly-once semantics including idempotence, transactions, and their relationship, see [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka). ### Transaction Timeout Configuration Set `transaction.timeout.ms` based on your maximum expected processing time, including retries. The default is 60 seconds, but long-running transformations may need higher values. However, longer timeouts delay the detection of failed transactions, so balance carefully. ## Summary Kafka transactions coordinate atomic writes across multiple partitions and topics. The transaction coordinator, two-phase commit protocol, and control messages ensure that either all messages in a transaction are visible to consumers or none are. The transactional APIs enable patterns like consume-process-produce, where reading, transforming, and writing happen atomically. This is essential for financial systems, data synchronization, and stream processing pipelines where consistency is critical. Transactions add latency and complexity, but are necessary for applications that cannot tolerate duplicates or data loss. **Key 2025 updates:** - Idempotent producers are now default in Kafka 3.0+, providing automatic duplicate prevention - Kafka 4.0+ with KRaft mode delivers significantly improved transaction coordinator performance with faster commits and reduced overhead - Modern monitoring tools like Kafka Lag Exporter and Conduktor provide enhanced visibility into transaction health and performance - The `exactly_once_v2` protocol in Kafka Streams (2.5+) combined with KRaft optimizations makes exactly-once semantics more performant than ever ## Related Concepts - [Exactly-Once Semantics in Kafka](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) - Overview of exactly-once guarantees in Kafka - [Kafka Topics, Partitions, and Brokers: Core Architecture](https://www.conduktor.io/glossary/kafka-architecture-diagram) - Foundational architecture enabling transactions - [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions) - Using transactions in distributed workflows ## Sources and References 1. [Apache Kafka Documentation - Transactions](https://kafka.apache.org/documentation/#semantics) - Official documentation covering transaction semantics and configuration 2. [KIP-98: Exactly Once Delivery and Transactional Messaging](https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging) - Original Kafka Improvement Proposal detailing transaction design 3. [Confluent: Transactions in Apache Kafka](https://www.confluent.io/blog/transactions-apache-kafka/) - Technical deep-dive from Kafka creators 4. [Exactly-once Semantics in Kafka Streams](https://www.confluent.io/blog/enabling-exactly-once-kafka-streams/) - How Kafka Streams leverages transactions 5. [Understanding Kafka Transaction Coordinator](https://www.conduktor.io/kafka/kafka-transactions) - Practical guide to transaction internals and troubleshooting Source: https://www.conduktor.io/glossary/kafka-transactions-deep-dive --- # Kafka vs Postgres: When to Use Each **Kafka** is a distributed append-only log built for retention, replay, and many independent consumers. **Postgres** is an ACID transactional table store that can also act as a durable queue via `SELECT ... FOR UPDATE SKIP LOCKED`. The Kafka vs Postgres choice usually comes down to scale, architecture, and ownership of the systems. Most workloads sit well below the scale Kafka is designed for, which raises a fair question: why not just use Postgres? A quick TL;DR: it's rarely about technology; it's about your goal. As a queue or a single application's backend, Postgres is the right call. For **data exchange at scale**, sharing data across many teams, services, or organizations, Postgres is a definitive no-go, for architectural reasons rather than performance. That is the heart of the matter: [why Kafka is the classic choice for organizational data exchange](#why-kafka-is-the-classic-choice-for-organizational-data-exchange). ![kafka-vs-postgres diagram 1](https://www.conduktor.io/assets/images/glossary/kafka-vs-postgres-0.webp) ## What each system actually is - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) is a distributed commit log. Records are appended to partitioned, replicated [topics](https://www.conduktor.io/glossary/kafka-topics-explained) and retained by a policy (time, size, or compaction); they are **not deleted when read**. Consumers pull at their own offset, and multiple consumer groups read the same topic independently. This is what makes replay and fan-out first-class primitives. - Postgres is an ACID transactional database. Its primitives are tables, rows, MVCC snapshots, and the write-ahead log (WAL). It is durable like Kafka, but the durability serves transactions and point-in-time consistency, not a re-readable event history. A queue on Postgres is just a table with a `status` column and the right locking. Both are durable. The difference is the shape of the data: an immutable log to re-read versus a mutable set of rows to claim and remove. Both can act as pub/sub or queues. ## Kafka vs Postgres at a glance | Dimension | Apache Kafka | PostgreSQL (as a queue) | |---|---|---| | Storage model | Distributed append-only commit log | ACID row store (queue table) | | Retention / replay | Retained by time/size; replay any offset | Rows deleted on ack; no replay after delete | | Consumer fan-out | N independent consumer groups, each a full copy | One worker pool per table (competing consumers) | | Throughput ceiling | Millions of msg/s across a cluster | Thousands of msg/s single node; scales with cores | | Ordering | Per-partition (per key) | Per single reader; lost across competing workers | | Delivery | At-least-once; exactly-once with transactions | At-least-once (claim-and-delete) | | Failover | Automatic, replicated brokers | Single writer; needs Patroni or manual promotion | | Ops cost | Cluster + Connect + Schema Registry | One database, very common | | Best fit | High throughput, replay, many readers | Small-to-medium queues, one worker pool | ## Postgres as a queue: SKIP LOCKED, LISTEN/NOTIFY, pgmq The competing-consumers pattern on Postgres relies on `FOR UPDATE SKIP LOCKED`. Each worker locks a batch of `ready` rows; `SKIP LOCKED` makes concurrent workers step over rows another transaction already holds. You will want a partial index to avoid a full table scan on `WHERE status = 'ready' ORDER BY id`: ```sql CREATE INDEX jobs_ready_idx ON jobs (id) WHERE status = 'ready'; ``` You need to claim and mark rows atomically in one statement, with a CTE that selects `FOR UPDATE SKIP LOCKED` feeding an `UPDATE ... RETURNING`: ```sql -- each worker runs, in its own transaction: BEGIN; WITH c AS ( SELECT id FROM jobs WHERE status = 'ready' ORDER BY id FOR UPDATE SKIP LOCKED LIMIT 3 ) UPDATE jobs SET status = 'workerA' FROM c WHERE jobs.id = c.id RETURNING jobs.id; COMMIT; ``` For instance, that would give: ```text Worker A RETURNING: 1,2,3 Worker B RETURNING (returned immediately while A held locks): 4,5,6 SELECT status, count(*), array_agg(id ORDER BY id) FROM jobs GROUP BY status: ready | 4 | {7,8,9,10} workerA | 3 | {1,2,3} workerB | 3 | {4,5,6} ``` Without `SKIP LOCKED`, a plain `SELECT ... FOR UPDATE` from the second worker would **block** on the rows the first transaction holds. You can: - *claim-and-delete* to remove the rows on ack - *claim-and-mark* flips a `status` column Marking is easier to audit but leaves rows in place, which compounds the MVCC bloat covered below. Then there is **crash recovery**. If a worker dies mid-job, its rows stay stuck in the in-flight state and are never reprocessed. Therefore, you may need to add a `claimed_at timestamptz` and a periodic reaper that resets abandoned rows: ```sql -- reaper: reclaim rows a crashed worker never finished UPDATE jobs SET status = 'ready', claimed_at = NULL WHERE status = 'processing' AND claimed_at < now() - interval '5 minutes'; ``` To avoid constant polling, workers can use **LISTEN/NOTIFY** to add push-style wake-ups. A worker runs `LISTEN jobs_channel` and blocks, an insert fires `NOTIFY jobs_channel`: ```sql LISTEN jobs_channel; -- worker session NOTIFY jobs_channel, 'new job'; -- producer, after INSERT ``` This has a big drawback: if no listener is connected when `NOTIFY` fires, the notification is lost (there is no replay) and very high notification rates can contend on an internal lock. **pgmq** is a PostgreSQL extension (originally from Tembo) that packages all of this behind an SQS-like SQL API. It uses `FOR UPDATE SKIP LOCKED` internally and is the recommended path over hand-rolling queue machinery: ```sql SELECT pgmq.send('jobs', '{"id": 1}'); -- enqueue SELECT * FROM pgmq.read('jobs', 30, 1); -- claim with a 30s visibility timeout SELECT pgmq.archive('jobs', 1); -- ack: move to archive table ``` The cost of any of these is **MVCC bloat**. Even `pgmq.read()` performs an `UPDATE` (to move the visibility timeout), which generates dead tuples. On a hot queue table, autovacuum often *cannot* keep pace, so you need aggressive per-table autovacuum settings. ## The real debate: most Kafka carries small data It's common in the community to say "you don't need Kafka, just use Postgres". If you have a workload of 500 KB/s, you don't need a Kafka cluster: a queue table (or pgmq) on a database already in production is simpler to run. Even Aiven and Redpanda confirmed that roughly **50% of deployments ingest below 10 MB/s**. This is equivalent to 1000 msg/s (for an average of 10KB per message). A well-tuned Postgres can easily cover that. ## Where Kafka still earns its place Postgres-as-a-queue stops being enough when you need real scale and fan-out without heavy tuning. Kafka's advantages here are structural, not incremental: - **Replay and log semantics.** A consumer can reset its offset and re-read every retained record. A simple Postgres queue deletes or archives rows on ack, so replay is not built in: it requires explicit retention plus per-consumer position tracking, versus Kafka's retained offsets. Kafka offers both queue-like consumption (via [share groups](https://www.conduktor.io/glossary/kafka-share-groups)) and a durable, replayable log; a queue table gives only the former. - **Fan-out to many independent consumers.** Multiple consumer groups each read the full stream at their own pace, so search indexing, analytics, caches, and other services consume the same events without coordinating. A queue table feeds one worker pool; adding independent readers means extra modeling, such as per-consumer checkpoints, logical-replication publications, or separate read models. - **Specialized, multi-language clients.** Kafka has a mature multi-language client ecosystem: the JVM client maintained by Apache Kafka plus widely used Go, Python, Rust, C/C++, and .NET clients. A consumer subscribes and receives records, with no query to write. Postgres access means SQL plus a driver, and each consumer hand-rolls the claim, ack, and reaper logic (or adopts pgmq). - **No MVCC bloat.** Kafka's log is append-only, so old data ages out by retention rather than by vacuuming dead tuples. A busy Postgres queue churns rows constantly (every claim, and even `pgmq.read()`, is an UPDATE), making autovacuum a permanent operational concern. - **Native horizontal scaling and failover.** Scaling Kafka is vanilla: add partitions and brokers and throughput grows, with automatic failover across replicated brokers. Postgres has built-in streaming replication and read replicas, but write scaling is vertical by default: sharding and automatic failover orchestration (for example Patroni) are not core and rely on external tooling. ## Why Kafka is the classic choice for organizational data exchange A database shared *directly* across teams tends to expose far more of its internal surface than intended: arbitrary reads and writes, joins over internal tables, and a physical schema consumers start to depend on. Postgres can narrow that surface with roles, column privileges, views, row-level security, and read replicas, but each is deliberate work that gets harder to maintain as more teams connect. The table schema becomes a public API: no team can rename a column or refactor a table without risking someone else's query, and every direct reader adds query load and connection pressure to the OLTP path unless reads are offloaded to a replica. **This is the well-known shared-database (integration-database) anti-pattern.** Kafka exposes a deliberately narrow interface: producers append immutable events, consumers subscribe and read, nothing more. That restriction is the point: - **Read-only for consumers.** A subscriber reads events; it cannot write to or run a query against the producer's database. - **Schema as an explicit contract.** The event schema is the interface, versioned with compatibility rules ([data contracts](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines), [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management)). - **Decoupling in time and ownership.** Producers and consumers do not know each other. A new team subscribes without anything to change on the producer side. - **Governance at the right granularity.** [ACLs](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) and quotas grant "subscribe to the orders stream," not "read our database," with per-client audit and rate limits. A shared log is a much cleaner backbone for cross-team and [cross-organization data sharing](https://www.conduktor.io/glossary/cross-organization-data-sharing-patterns), and a foundation of [data mesh](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation). ## They are complementary: Postgres + Debezium CDC into Kafka This is a very common pattern: Postgres stays the ACID source of truth for microservices, and [Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) reads its WAL to turn every insert, update, and delete into [change events](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) on Kafka topics. Downstream, many independent consumers read from Kafka directly (search indexing, analytics, caches, other services) without touching the OLTP database: ![kafka-vs-postgres diagram 2](https://www.conduktor.io/assets/images/glossary/kafka-vs-postgres-1.webp) **Can Postgres replace Kafka for a message queue?** For a single worker pool below a single node's throughput, usually yes. `FOR UPDATE SKIP LOCKED` (or pgmq) gives real competing-consumers semantics on a database you already run. It cannot give you replay or independent fan-out to many consumer groups. **Is SKIP LOCKED safe for concurrent workers?** Yes. Tested on PostgreSQL 16.14, two concurrent workers claimed disjoint row sets ({1,2,3} and {4,5,6}) and the second never blocked. The safe pattern claims and marks rows atomically in one CTE feeding an UPDATE ... RETURNING. **What is the main downside of a Postgres queue?** MVCC bloat. Every claim and even pgmq.read() is an UPDATE that creates dead tuples; 200 reads generated 200 dead tuples in testing. Autovacuum must keep up, and short benchmarks hide the pile-up. **Does Kafka now support queue semantics?** Yes, via KIP-932 share groups: GA in Kafka 4.2, after early access in 4.0 and preview in 4.1. It adds per-message acknowledgement and message-level parallelism. **How do Postgres and Kafka work together?** Postgres stays the transactional source of truth and Debezium captures its WAL into Kafka topics. Tools like [Conduktor](https://www.conduktor.io/kafka-data-governance) then help teams observe and govern those topics as they fan out to many downstream consumers. ## Related Pages - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka): the distributed log at the center of the comparison. - [Kafka vs RabbitMQ](https://www.conduktor.io/glossary/kafka-vs-rabbitmq): streams versus message queues, the sibling comparison. - [Change Data Capture (CDC) Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals): how database changes become event streams. - [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium): capturing the Postgres WAL into Kafka topics. - [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture): the pattern fan-out and replay enable. - [Streaming ETL vs Traditional ETL](https://www.conduktor.io/glossary/streaming-etl-vs-traditional-etl): why a shared log changes the pipeline. ## Sources - [KIP-932: Queues for Kafka, Apache Kafka wiki](https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka) - [Queues for Kafka (KIP-932), Preview Release Notes](https://cwiki.apache.org/confluence/display/KAFKA/Queues+for+Kafka+(KIP-932)+-+Preview+Release+Notes) - [Apache Kafka 4.0.0 Release Announcement](https://kafka.apache.org/blog/2025/03/18/apache-kafka-4.0.0-release-announcement/) - [Kafka Queue Semantics Now GA with Share Consumer API, Confluent](https://www.confluent.io/blog/kafka-queue-semantics-share-consumer-ga/) - [PostgreSQL Documentation: SELECT, The Locking Clause (FOR UPDATE / SKIP LOCKED)](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE) - [Message Queuing Using Native PostgreSQL, Crunchy Data](https://www.crunchydata.com/blog/message-queuing-using-native-postgresql) - ["You Don't Need Kafka, Just Use Postgres" Considered Harmful, Gunnar Morling](https://www.morling.dev/blog/you-dont-need-kafka-just-use-postgres-considered-harmful/) - [Kafka is fast, I'll use Postgres (benchmarks), TopicPartition](https://topicpartition.io/blog/postgres-pubsub-queue-benchmarks) - [Postgres, Kafka and event queues, Kaarel Moppel](https://kmoppel.github.io/2025-11-13-postgres-kafka-and-event-queues/) - [pgmq: A lightweight message queue for PostgreSQL (GitHub)](https://github.com/pgmq/pgmq) - [Debezium Documentation: PostgreSQL connector](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) Source: https://www.conduktor.io/glossary/kafka-vs-postgres --- # Kafka vs Pulsar: Architecture Compared **Apache Kafka** stores streams in a partitioned, replicated log on broker-local disks; brokers own both compute and storage. **Apache Pulsar** separates compute (brokers) from storage (Apache BookKeeper) and uses a segmented log model instead of a monolithic partition file. The core architectural difference: Kafka binds partitions to specific brokers; Pulsar brokers are stateless and storage is disaggregated across a BookKeeper cluster. ## TL;DR | Dimension | Apache Kafka | Apache Pulsar | |---|---|---| | Storage model | Partitioned log on broker-local disk | Segmented log on Apache BookKeeper | | Compute/storage coupling | Coupled (brokers own storage) | Disaggregated (stateless brokers + BookKeeper) | | Partition rebalancing | Requires data movement | Instant (brokers are stateless) | | Multi-tenancy | Manual (naming conventions, quotas) | Native (tenants, namespaces, topics) | | Subscription models | Consumer groups (offset-based) | Exclusive, shared, failover, key-shared | | Message queuing | No built-in queue semantics | Yes (shared subscription) | | Tiered storage | Via plugins or Kafka 3.6+ KIP | Native | | Geo-replication | MirrorMaker 2 or Confluent | Built-in (async replication) | | License | Apache 2.0 | Apache 2.0 | | Ecosystem | Very large | Smaller, growing | | Operational complexity | Moderate (KRaft) | High (Kafka + BookKeeper + ZooKeeper*) | *Pulsar historically depends on ZooKeeper for metadata. Newer versions support pluggable metadata backends (ZooKeeper, etcd, and others depending on deployment). Verify ZooKeeper requirements against your target Pulsar version. ## What is Apache Kafka? [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) is a distributed event streaming platform using a partitioned append-only log. Each [partition](https://www.conduktor.io/glossary/kafka-partitions-explained) is stored as a sequence of segment files on the broker's local disk. [Brokers](https://www.conduktor.io/glossary/kafka-brokers-explained) hold both the data and serve reads and writes for their assigned partition leaders. Kafka 3.3+ uses KRaft (Raft-based consensus) for metadata, eliminating ZooKeeper. See [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## What is Apache Pulsar? Apache Pulsar is a cloud-native distributed messaging and streaming platform developed at Yahoo and open-sourced in 2016. It separates brokers (compute, protocol handling) from storage (Apache BookKeeper). Pulsar brokers are **stateless**: they own no persistent data. All data lives in BookKeeper, a distributed write-ahead log service that stores data as ledger entries across an ensemble of bookies (BookKeeper nodes). Topics in Pulsar are composed of **segments** (BookKeeper ledgers). When a ledger fills, a new one is created. Brokers can be assigned any topic instantly because no data movement is required — the new broker simply points to the existing BookKeeper ledgers. ## Architecture compared ### Storage disaggregation **Kafka**: Each partition is "owned" by a leader broker. That broker's local disk holds the partition data. Rebalancing a partition to a different broker requires copying gigabytes of data across the network. Scaling Kafka storage typically means adding brokers and triggering partition reassignment — an expensive operation. **Pulsar**: Brokers are stateless. Adding a broker instantly gives it capacity to serve topics without any data movement. Scaling storage means adding BookKeeper bookies; scaling compute means adding Pulsar brokers. These dimensions scale independently. The tradeoff: Pulsar's disaggregated architecture adds another layer of operational complexity. You run and monitor three tiers (brokers, bookies, ZooKeeper/metadata store) instead of Kafka's single broker tier. ### Segmented vs partitioned log **Kafka's partitioned log**: A partition is a sequence of segment files. Segments are immutable once rolled. Oldest segments are deleted (or tiered) when retention limits are hit. Data is stored on broker-local SSD/HDD. **Pulsar's segmented log (BookKeeper)**: A topic is a sequence of BookKeeper ledgers. Each ledger is replicated across an ensemble of bookies (configurable write quorum and ack quorum). Ledgers are sealed when a bookie fails or when the broker decides to roll. The broker tracks the ledger sequence in metadata. This model makes Pulsar naturally resilient to bookie failures without partition leadership handoff. ### Subscription models Kafka's model: a consumer group assigns partitions to consumers. One consumer per partition within a group; ordering is guaranteed per partition. Pulsar provides four subscription types: - **Exclusive**: single consumer per subscription (like a dedicated consumer) - **Shared**: multiple consumers share messages round-robin (like a queue / competing consumers) - **Failover**: primary + standby consumers; standby takes over if primary disconnects - **Key-shared**: messages with the same key always go to the same consumer (ordering per key, across consumers) Pulsar's shared and key-shared subscriptions give it native message-queue semantics that Kafka lacks without application-level workarounds. ### Multi-tenancy Kafka multi-tenancy is manual: teams use topic naming conventions (`team-a.orders`, `team-b.events`), quotas per client ID, and ACLs per topic. See [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments). Pulsar has **native multi-tenancy built into the data model**: `persistent://tenant/namespace/topic`. Tenants and namespaces are first-class objects with authentication, authorization, and resource quota policies. This makes Pulsar well-suited for SaaS platforms where strong isolation between tenants is required without naming-convention discipline. ### Geo-replication Kafka uses MirrorMaker 2 or Confluent Replicator for cross-cluster replication. MirrorMaker 2 is operationally significant — it's a separate Kafka Connect cluster. See [Kafka MirrorMaker 2](https://www.conduktor.io/glossary/kafka-mirrormaker-2-for-cross-cluster-replication). Pulsar includes async geo-replication as a built-in feature: namespaces are configured to replicate across geographically distributed Pulsar clusters. No external replication process is required. ## Operational trade-offs **Kafka advantages:** - Simpler operational model: a single process type (broker / controller) — no separate storage tier - Mature ecosystem: Kafka Connect, ksqlDB, Schema Registry, MirrorMaker 2, Conduktor, and thousands of integrations tested in production - KRaft mode eliminates ZooKeeper — Kafka's dependency stack is simpler today - Dominant adoption: most job postings, most cloud managed offerings (MSK, Confluent, Aiven, Upstash), most training material **Kafka disadvantages:** - Partition rebalancing requires data movement — slow and I/O intensive - Multi-tenancy requires convention and tooling rather than native isolation - Tiered storage is newer (KIP-405 in Kafka 3.6+, still maturing) compared to Pulsar **Pulsar advantages:** - Independent scaling of compute and storage - Instant broker scaling without data movement - Native multi-tenancy with tenant/namespace isolation - Built-in geo-replication and tiered storage - Flexible subscription models covering both streaming and queuing use cases **Pulsar disadvantages:** - Significantly higher operational complexity: brokers + bookies + metadata store (ZooKeeper or equivalent) - Smaller ecosystem: fewer connectors, less tooling, smaller community - BookKeeper expertise is scarce and debugging bookie issues is non-trivial - Higher base resource requirements for a functioning cluster (minimum 3 bookies + 3 brokers + 3 ZooKeeper nodes for production) ## When to choose Kafka - Your team has existing Kafka expertise, tooling, and managed-service contracts - You use managed Kafka (MSK, Confluent Cloud, Aiven) — all major clouds run Kafka, not Pulsar - You need the broadest connector and integration ecosystem - Your multi-tenancy requirements are manageable via quotas and ACLs (see [Kafka Quotas and Rate Limiting](https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka)) - You want a single-tier broker architecture with lower baseline operational complexity ## When to choose Pulsar - You need **native multi-tenancy** as a first-class platform feature (SaaS product, shared infrastructure platform) - Independent scaling of compute and storage is architecturally important (bursty compute, low-cost object-storage tiering from day one) - You need both streaming (ordered log) and queuing (competing consumers) from a single system - Built-in geo-replication without external replication infrastructure is a hard requirement - Your team can invest in BookKeeper expertise ## Migration considerations - **No wire compatibility**: Pulsar does not implement the Kafka protocol natively. Pulsar provides a Kafka-on-Pulsar (KoP) protocol handler that maps Kafka clients to Pulsar, but KoP coverage is incomplete — not all Kafka APIs are supported, and performance characteristics differ. - **Data migration**: There is no live migration tooling between Kafka and Pulsar equivalent to MirrorMaker. Dual-write patterns (write to both, drain Kafka readers, cut over) are the typical approach. - **Connector ecosystem gap**: Kafka Connect has thousands of connectors. Pulsar IO supports fewer; verify your specific sources and sinks before committing. - **Consumer offset semantics**: Pulsar uses cursor-based positions (MessageId) rather than integer offsets, which affects consumer migration tooling. See also: [Redpanda vs Kafka](https://www.conduktor.io/glossary/redpanda-vs-kafka) for a Kafka-compatible alternative with a simpler operational model. **Is Pulsar better than Kafka?** Neither is universally better. Pulsar's disaggregated storage excels for independent compute/storage scaling and native multi-tenancy. Kafka excels for ecosystem maturity, managed-service availability, and simpler operations. The right choice depends on your specific scaling requirements and team expertise. **Can I use Kafka clients with Pulsar?** Yes, via the Kafka-on-Pulsar (KoP) protocol handler. KoP translates Kafka protocol calls to Pulsar's native API. Coverage is incomplete — some Kafka APIs (certain transaction endpoints, admin APIs) may not work. KoP performance and compatibility require evaluation against your specific client version. **Does Pulsar require ZooKeeper?** Pulsar historically depends on ZooKeeper for metadata and coordination. Newer Pulsar versions support pluggable metadata backends (ZooKeeper, etcd, and others), but production maturity of ZooKeeper-free deployments varies by version. Verify ZooKeeper requirements against your target Pulsar release before assuming it can be eliminated. **Why don't managed cloud services offer Pulsar?** AWS, GCP, and Azure offer managed Kafka (MSK, HDInsight, Confluent Cloud). None offer managed Pulsar at the same tier. StreamNative offers managed Pulsar/StreamNative Cloud. The operational complexity of the multi-tier architecture makes managed Pulsar more expensive to run than managed Kafka, which reduces cloud provider investment incentive. Source: https://www.conduktor.io/glossary/kafka-vs-pulsar --- # Kafka vs RabbitMQ: Streams vs Queues **Apache Kafka** is a distributed log for stream storage and replay: records are retained durably and consumers pull at their own pace. **RabbitMQ** is a message broker implementing AMQP: messages are pushed to consumers and deleted once acknowledged. The core difference is persistence model — Kafka is a log you can re-read; RabbitMQ is a queue that empties as messages are consumed. ## TL;DR | Dimension | Apache Kafka | RabbitMQ | |---|---|---| | Architecture | Distributed append-only log | Message broker (AMQP / STOMP / MQTT) | | Delivery model | Consumer pulls at own pace | Broker pushes to consumers | | Message retention | Configurable (days, size, forever) | Deleted after acknowledgment | | Ordering | Per-partition ordering guarantee | Per-queue (no cross-queue ordering) | | Replay | Yes — reset consumer offset to any point | No — consumed messages are gone | | Routing | Topic-based (producer specifies topic) | Exchange types: direct, fanout, topic, headers | | Consumer model | Consumer groups (shared offset tracking) | Competing consumers on a queue | | Throughput | Very high (millions of msg/sec per node) | High (tens of thousands/sec per queue) | | Protocol | Kafka binary protocol | AMQP 0-9-1 (primary), STOMP, MQTT | | Language | Scala / Java (JVM) | Erlang | | License | Apache 2.0 | MPL 2.0 | | Best fit | Event streaming, analytics, CDC, audit logs | Task queues, work distribution, RPC | ## What is Apache Kafka? [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) is a distributed event streaming platform. Records are written to named [topics](https://www.conduktor.io/glossary/kafka-topics-explained), stored in partitioned, replicated logs on broker disk, and retained independently of consumption. [Consumer groups](https://www.conduktor.io/glossary/kafka-consumer-groups-explained) track their position (offset) in each partition; multiple groups can independently read the same topic. Records are never deleted on read — only by retention policy (time or size). ## What is RabbitMQ? RabbitMQ is a message broker developed by Rabbit Technologies (now VMware/Broadcom), written in Erlang. It implements AMQP 0-9-1 and supports additional protocols (STOMP, MQTT, HTTP). Producers publish messages to **exchanges**; exchanges route to **queues** based on routing rules. Consumers subscribe to queues and receive messages via push delivery. Once a consumer acknowledges a message, the broker removes it. RabbitMQ 3.9 introduced streams (via the `rabbitmq_stream` plugin), which add an append-only log structure with offset-based consumption closer to Kafka's model. However, this is a separate protocol and not the default queue behavior. ## Architecture compared ### Storage and delivery Kafka stores every message on disk regardless of consumer state. A message written at 9:00 AM is still readable at 9:00 PM (within the retention window). Consumers self-manage their read position (offset). This makes Kafka suitable for: - **Event sourcing**: replay history to rebuild state - **Multiple independent consumers**: analytics, alerting, and replication can all read the same topic - **Audit logs**: immutable record of what happened and when RabbitMQ's queues are transient by nature: messages exist to be delivered and then removed once acknowledged. (RabbitMQ Streams, available since 3.9, behave differently — they are append-only and non-destructive.) Queue depth for classic/quorum queues is bounded by memory/disk limits, not a durability window. This makes RabbitMQ suitable for: - **Task queues**: distribute units of work to worker processes - **RPC patterns**: request-reply with correlation IDs - **Short-lived messages**: where post-delivery persistence adds no value ### Consumer model **Kafka consumer groups**: Partitions are assigned to consumers in a group. Each partition is consumed by exactly one consumer within the group at a time; a single consumer may own multiple partitions, and if there are more consumers than partitions, some consumers sit idle. Adding consumers scales throughput up to the partition count. Multiple independent groups read the same topic without interfering. See [Kafka Consumer Groups Explained](https://www.conduktor.io/glossary/kafka-consumer-groups-explained). **RabbitMQ competing consumers**: Multiple consumers on the same queue share messages round-robin (one consumer per message). This is a classic work-queue pattern but does not allow multiple independent consumers to each receive a copy of the same message without separate queues (fanout exchanges route copies to multiple queues, approximating Kafka's multi-group model). ### Routing RabbitMQ provides rich routing via exchange types: - **Direct**: route by exact routing key - **Fanout**: broadcast to all bound queues - **Topic**: wildcard routing key patterns - **Headers**: route by message header attributes Kafka routes by producer-specified topic name. Filtering within a topic requires consumer-side processing or KafkaStreams/Flink. The simplicity is intentional — Kafka offloads routing logic to producers and consumers. ### Ordering Kafka guarantees ordering within a partition. Messages with the same key always go to the same partition (key-based routing), ensuring ordered delivery per entity (e.g., all events for user ID 123 are ordered). Cross-partition ordering is not guaranteed. RabbitMQ guarantees ordering within a single queue with a single consumer. With competing consumers, message ordering is not preserved (a slow consumer can hold an unacked message while faster consumers process later messages). ## Operational trade-offs **Kafka advantages:** - High throughput at scale — partitioned design allows horizontal scaling without coordination - Durable replay — essential for event sourcing, CDC pipelines, and audit requirements - Multiple independent consumer groups at no extra cost - Native support for stream processing via Kafka Streams or Flink on top **Kafka disadvantages:** - Higher operational complexity — partition management, consumer group rebalancing, offset tracking - No built-in complex routing logic — routing must be in producers or external processors - Messages cannot be selectively deleted (only log compaction per key) - Not suitable as a task queue where you need exactly-one delivery without re-reads **RabbitMQ advantages:** - Rich routing model (exchanges, binding keys) with minimal client code - Lower latency for small message volumes — push delivery is faster for interactive workloads - Message TTL, dead-letter queues, priority queues built-in - Easier to reason about for simple point-to-point or worker-queue patterns - Per-message acknowledgment and requeue semantics **RabbitMQ disadvantages:** - Messages are lost after acknowledgment — no replay, no audit trail by default - Queue depth is memory-bounded; large queues degrade performance - Competing consumers don't scale as cleanly for high-throughput partitioned workloads - No native stream processing integration ## When to choose Kafka - You need **replay**: a new service needs to process historical events without data loss - You have **multiple independent consumers** of the same event stream - You are building **CDC pipelines**, **event sourcing**, or **audit logging** — retention is mandatory - You need **high throughput** (millions of messages/sec) with predictable horizontal scaling - Your data engineering stack uses tools that integrate natively with Kafka (Flink, Spark, dbt, etc.) - You need [exactly-once semantics](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) in a transactional pipeline ## When to choose RabbitMQ - You have a **task queue** pattern: distribute units of work to workers, acknowledge on completion - You need **flexible routing** — topic exchanges with wildcard routing keys, header-based dispatch - Your messages are short-lived and **replay is not a requirement** - You need **per-message TTL** or **dead-letter exchange** routing out of the box - Your stack is polyglot and needs AMQP, STOMP, or MQTT protocol support - You are implementing **RPC or request-reply** patterns with correlation IDs ## Can Kafka and RabbitMQ coexist? Yes — and this is common. A typical pattern: Kafka carries high-volume event streams (user activity, transactions), while RabbitMQ handles internal task distribution (send email, resize image, trigger workflow). They solve different problems and the boundary is usually clear: if you need replay or multiple consumers, use Kafka; if you need flexible task routing with acknowledgment semantics, use RabbitMQ. **Can RabbitMQ replace Kafka?** For most event streaming use cases, no. RabbitMQ does not retain messages after consumption, so replay — essential for event sourcing, CDC, and audit logs — is not possible with standard queues. RabbitMQ Streams (the plugin) adds an append-only log, but it is a separate protocol and lacks Kafka's ecosystem maturity. **Can Kafka replace RabbitMQ?** For task queues and RPC patterns, Kafka is a poor fit. Kafka has no built-in mechanism for per-message acknowledgment with requeue, dead-letter semantics, or per-message TTL. Implementing a task queue on Kafka requires significant application-level logic that RabbitMQ provides out of the box. **Which is faster, Kafka or RabbitMQ?** It depends on the metric. Kafka achieves higher aggregate throughput (millions of messages/sec) due to its batching and sequential I/O. RabbitMQ achieves lower per-message latency for small volumes because push delivery removes the pull polling cycle. Under high load, Kafka scales more predictably. **Does Conduktor work with RabbitMQ?** Conduktor is purpose-built for Apache Kafka (and Kafka-compatible brokers). It does not connect to RabbitMQ. If your stack includes both Kafka and RabbitMQ, Conduktor manages the Kafka side. Source: https://www.conduktor.io/glossary/kafka-vs-rabbitmq --- # ksqlDB for Real-Time Data Processing Building stream processing applications on Kafka traditionally required writing complex Java or Scala code. ksqlDB provides a SQL interface instead, making real-time data transformation accessible to a wider range of developers. ![ksqlDB architecture with streams, tables, and queries](https://www.conduktor.io/assets/images/glossary/ksqldb-for-real-time-data-processing-0.webp) ## What is ksqlDB? ksqlDB is an event streaming database built on Apache Kafka and Kafka Streams. It allows developers to build stream processing applications using familiar SQL syntax instead of writing procedural code. Originally released as KSQL in 2017 and rebranded as ksqlDB in 2019, it has evolved into a complete database for stream processing workloads. As of 2025, ksqlDB 0.29+ includes significant enhancements such as improved pull query performance, enhanced exactly-once semantics, native support for Kafka's KRaft mode (ZooKeeper-free architecture), and better integration with modern schema registries. These updates make ksqlDB a mature, production-ready platform for SQL-based stream processing. Unlike traditional databases that store data at rest, ksqlDB operates on data in motion. It processes continuous streams of events, allowing you to filter, transform, aggregate, and join data as it flows through Kafka topics. This approach enables real-time analytics, data enrichment, and event-driven architectures without the complexity of custom stream processing applications. ksqlDB serves dual purposes: it functions as both a stream processing engine and a materialized view layer. Queries can be transient (returning results and terminating) or persistent (continuously processing events and maintaining state). This flexibility makes it suitable for both ad-hoc analysis and production streaming pipelines. ## Core Concepts and Architecture ### Streams and Tables ksqlDB introduces two fundamental abstractions: streams and tables. A stream represents an unbounded sequence of immutable events, such as user clicks or sensor readings. Each event is independent and append-only. A table, in contrast, represents the current state derived from a stream, where each key has exactly one value that can be updated over time. For example, a stream of user login events becomes a table showing the last login time for each user. This distinction between event log (stream) and current state (table) is crucial for modeling real-world data flows. ### Continuous Queries and Query Types ksqlDB supports multiple query types, each serving different use cases: - **Persistent queries** run continuously, processing new events as they arrive. When you create a persistent query, ksqlDB automatically maintains it across restarts and failures. The query reads from Kafka topics, applies transformations, and writes results back to Kafka topics, creating a complete streaming data pipeline. - **Push queries** (also called continuous queries) stream results to clients as new data arrives. These are ideal for real-time dashboards, monitoring systems, or applications that need to react to changes as they happen. For example, a dashboard displaying live transaction counts would use a push query. - **Pull queries** allow point-in-time lookups against materialized views, similar to traditional database queries. In ksqlDB 0.29+, pull queries offer significantly improved performance and support for high-availability setups, making them suitable for serving user-facing applications that need low-latency access to aggregated state. Materialized views provide queryable state. When you create a table with aggregations, ksqlDB maintains the aggregated results in memory and on disk, allowing you to query the current state at any time using pull queries. This eliminates the need for separate caching layers or batch processing jobs, enabling ksqlDB to serve as both a stream processor and a queryable database. ## Key Features and Capabilities ### SQL-Based Transformations ksqlDB supports standard SQL operations including filtering, projecting, joining, and aggregating data. You can write window-based aggregations (tumbling, hopping, session windows) to analyze time-series data. Complex transformations that would require hundreds of lines of Java code can be expressed in a few lines of SQL. ```sql CREATE STREAM filtered_events AS SELECT user_id, event_type, timestamp FROM raw_events WHERE event_type = 'purchase'; ``` ### Stream-Table Joins ksqlDB enables joins between streams and tables, or between multiple streams. This capability is essential for enriching event data with reference information. For instance, you can join a stream of transactions with a table of user profiles to add customer demographics to each transaction in real-time. For detailed patterns on implementing stream enrichment and various join types, see [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns). ### Scalability and Fault Tolerance Built on Kafka Streams, ksqlDB inherits distributed processing capabilities. Queries are automatically distributed across multiple servers, and state is partitioned and replicated for fault tolerance. If a server fails, ksqlDB automatically redistributes the workload and restores state from Kafka topics. Under the hood, ksqlDB leverages the same state store mechanisms as Kafka Streams, using RocksDB for persistent state and changelog topics for recovery. For more details on how state management works in stream processing, see [State Stores in Kafka Streams](https://www.conduktor.io/glossary/state-stores-in-kafka-streams). ## ksqlDB in Data Streaming Ecosystems ksqlDB operates within the broader Apache Kafka ecosystem, working alongside several complementary technologies. It reads and writes Avro, JSON, and Protobuf formats, integrating seamlessly with Schema Registry implementations to enforce data contracts and schema evolution. For details on schema management patterns, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). ### ksqlDB vs Kafka Streams vs Apache Flink Choosing between ksqlDB, Kafka Streams, and Apache Flink depends on your team's expertise, use case complexity, and operational preferences: **ksqlDB** is ideal when: - Your team prefers SQL over code (accessible to analysts and data engineers) - You need rapid development of streaming transformations and aggregations - Operational simplicity is a priority (no separate cluster to manage) - Use cases involve filtering, aggregations, joins, and windowing that SQL can express - You want to serve queries directly from stream processing (pull queries) **Kafka Streams** is better when: - You need full programmatic control and custom logic - Your application requires complex stateful processing beyond SQL capabilities - You want to embed stream processing directly into microservices - Integration with existing Java/Scala codebases is important **Apache Flink** excels when: - You need advanced features like complex event processing (CEP) or iterative algorithms - State size is very large (terabytes) - You're processing multiple data sources beyond Kafka - Batch and stream unification matters for backfilling or reprocessing For a detailed comparison of Kafka Streams and Flink, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink). For comprehensive coverage of Flink's capabilities, see [What is Apache Flink? Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). For organizations running Kafka, ksqlDB provides a natural path to stream processing without introducing additional infrastructure. It leverages existing Kafka topics as both input and output, maintaining a consistent data platform. Governance and management platforms like Conduktor enhance this experience by providing visual interfaces to develop and monitor ksqlDB queries, view query topology, troubleshoot performance issues, and ensure data quality across the streaming infrastructure. See [Conduktor's ksqlDB management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/ksql-db) to create and manage ksqlDB queries, monitor query performance, and inspect materialized views. Stream processing with ksqlDB complements traditional batch processing. Many organizations implement a lambda architecture where ksqlDB handles real-time processing while batch jobs provide comprehensive historical analysis. Others adopt kappa architecture, using ksqlDB for all processing with different time windows. ## Real-World Use Cases and Examples ### Real-Time Analytics A common use case is calculating real-time metrics from event streams. Consider an e-commerce platform tracking page views: ```sql CREATE TABLE page_view_counts AS SELECT page_url, COUNT(*) as view_count, WINDOWSTART as window_start, WINDOWEND as window_end FROM page_views WINDOW TUMBLING (SIZE 1 HOUR) GROUP BY page_url EMIT CHANGES; ``` This query continuously aggregates page views into hourly windows, providing near-instant insights into popular pages. The materialized table can be queried using pull queries for real-time dashboards: ```sql -- Pull query: Get current counts for a specific page SELECT page_url, view_count FROM page_view_counts WHERE page_url = '/products/bestseller'; ``` Results are also written to a Kafka topic for downstream consumers. For broader context on real-time analytics patterns, see [Real-Time Analytics with Streaming Data](https://www.conduktor.io/glossary/real-time-analytics-with-streaming-data). ### Data Enrichment and Filtering Financial institutions use ksqlDB to enrich transaction streams with account information and filter suspicious activities. A simple fraud detection example: ```sql CREATE STREAM suspicious_transactions AS SELECT t.transaction_id, t.amount, a.account_type, a.risk_score FROM transactions t LEFT JOIN accounts a ON t.account_id = a.account_id WHERE t.amount > 10000 OR a.risk_score > 75; ``` This query joins streaming transactions with an accounts table, then filters for high-value or high-risk transactions. Security teams can monitor the output stream in real-time, enabling immediate fraud investigation. ### Event-Driven Microservices ksqlDB enables microservices to react to events without direct service-to-service calls. Services publish events to Kafka topics, and ksqlDB queries transform and route events to appropriate downstream topics. This decouples services while maintaining real-time data flow. ## Materialized Views and Query Serving One of ksqlDB's most powerful features is its ability to serve as both a stream processor and a queryable database through materialized views. ### Creating Materialized Views When you create a table with aggregations in ksqlDB, you're creating a materialized view, a continuously updated representation of query results: ```sql CREATE TABLE user_activity_summary AS SELECT user_id, COUNT(*) as event_count, MAX(timestamp) as last_activity, COLLECT_LIST(event_type) as recent_events FROM user_events WINDOW TUMBLING (SIZE 5 MINUTES) GROUP BY user_id EMIT CHANGES; ``` This table is continuously updated as new events arrive, maintaining real-time aggregated state. ### Querying Materialized Views **Push queries** stream continuous updates: ```sql -- Stream all changes to the materialized view SELECT * FROM user_activity_summary EMIT CHANGES; ``` **Pull queries** provide point-in-time lookups (ksqlDB 0.29+ significantly improved performance): ```sql -- Get current activity for a specific user SELECT event_count, last_activity FROM user_activity_summary WHERE user_id = 'user_12345'; ``` Pull queries return immediately with the current state, making them suitable for serving user-facing applications, REST APIs, or microservices that need low-latency access to aggregated data. ### High Availability for Pull Queries In production deployments, ksqlDB supports high availability for pull queries through standby replicas. Multiple ksqlDB servers maintain replicas of the same state, allowing queries to be served even if some servers fail. This makes ksqlDB a viable alternative to traditional databases for serving real-time aggregations. ## Best Practices and Considerations ### When to Use ksqlDB ksqlDB is ideal when your team prefers SQL over code, when streaming transformations align with SQL capabilities, and when tight Kafka integration is valuable. It works well for filtering, aggregations, simple joins, and real-time metrics. Teams already using Kafka for messaging gain stream processing capabilities without additional infrastructure. ### Performance Considerations Query performance depends on several factors: partition count affects parallelism, state store configuration impacts memory usage, and window sizes influence latency and resource consumption. Monitor query performance using metrics exposed through JMX and integrate with observability platforms like Conduktor to visualize resource utilization, track consumer lag, and identify bottlenecks. For high-throughput scenarios, ensure adequate partitioning in source topics (aligning with ksqlDB server count), allocate sufficient memory for state stores (RocksDB configuration), and consider scaling horizontally by adding more ksqlDB servers. Testing queries with production-like data volumes before deployment prevents performance surprises. **ksqlDB 0.29+ performance improvements:** - Enhanced pull query performance with better caching and query planning - Improved state store restoration times for faster recovery - Better memory management reducing heap pressure during high-throughput operations - Optimized serialization/deserialization for Avro and Protobuf formats ### Limitations and Alternatives While ksqlDB is powerful, it has limitations compared to lower-level frameworks: - **SQL limitations**: Not all stream processing patterns can be expressed in SQL. Complex stateful operations, intricate multi-way joins, and custom processing logic may be better suited to Kafka Streams (programmatic control) or Flink (advanced features like CEP). - **User-defined functions (UDFs)**: While UDFs extend ksqlDB capabilities, they require Java development and deployment, reducing the SQL-only advantage. - **State size**: For very large state (terabytes), Flink's distributed state backend may be more appropriate than ksqlDB's RocksDB stores. - **Processing guarantees**: While ksqlDB supports exactly-once semantics within Kafka, complex pipelines with external systems may require more control than SQL provides. - **Version compatibility**: Kafka compatibility requires attention during upgrades. As of 2025, ksqlDB 0.29+ is compatible with Kafka 3.4+ and fully supports KRaft mode. Always consult compatibility matrices and test thoroughly in non-production environments. Schema evolution strategies should be planned upfront to avoid breaking changes in production queries. ## Summary ksqlDB provides a SQL interface to Apache Kafka, giving both data engineers and analysts a way to build streaming applications. Streams, tables, and query types (persistent, push, and pull) map naturally to database concepts while operating on data in motion. **Key strengths of ksqlDB:** - **SQL accessibility**: Build stream processing applications without writing Java/Scala code - **Dual-purpose architecture**: Functions as both stream processor and queryable database through materialized views - **Pull query capabilities**: Serve real-time aggregations directly from the stream processor (significantly improved in 0.29+) - **Operational simplicity**: No separate cluster to manage; deploys as a standard application - **Kafka integration**: Leverages existing Kafka infrastructure for durability and scalability - **2025 maturity**: KRaft support, enhanced exactly-once semantics, and production-grade performance ksqlDB works well for filtering, aggregation, enrichment, and joins. Kafka Streams gives more programmatic control, and Flink handles more complex scenarios, but ksqlDB is a good fit for SQL-first teams that want rapid development and operational simplicity. ## Related Concepts - [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink) - Compare ksqlDB's underlying Kafka Streams engine with Apache Flink for architectural decisions - [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns) - Implementing join patterns and data enrichment in ksqlDB - [Real-Time Analytics with Streaming Data](https://www.conduktor.io/glossary/real-time-analytics-with-streaming-data) - Building analytics dashboards and metrics using ksqlDB materialized views - [Kafka Streams vs ksqlDB (hands-on guide)](https://www.conduktor.io/kafka-streams/vs-ksqldb) - The real difference between the SQL layer and the library underneath ## Sources and References 1. Confluent Documentation. "ksqlDB Documentation." Confluent, https://docs.ksqldb.io/. Official documentation covering ksqlDB architecture, syntax, and operations (updated for 0.29+ features). 2. Confluent Blog. "ksqlDB 0.29 Release Notes." Confluent, 2024-2025. Details on pull query improvements, KRaft support, and performance enhancements. 3. Narkhede, Neha, et al. "Kafka: The Definitive Guide, 2nd Edition." O'Reilly Media, 2021. Comprehensive coverage of Kafka ecosystem including stream processing concepts. 4. Apache Software Foundation. "Kafka Streams Documentation." Apache Kafka, https://kafka.apache.org/documentation/streams/. Foundation technology underlying ksqlDB. 5. Kleppmann, Martin. "Designing Data-Intensive Applications." O'Reilly Media, 2017. Theoretical foundations of stream processing and event-driven architectures. 6. Confluent Blog. "Introducing ksqlDB." Confluent, 2019, https://www.confluent.io/blog/intro-to-ksqldb-sql-database-streaming/. Original announcement and overview of ksqlDB capabilities. 7. Confluent Developer. "ksqlDB Tutorials and Recipes." https://developer.confluent.io/tutorials/. Practical examples and patterns for ksqlDB applications. Source: https://www.conduktor.io/glossary/ksqldb-for-real-time-data-processing --- # Log Aggregation with Kafka Log aggregation is a core requirement for operating distributed systems at scale. As applications move from monoliths to microservices, collecting and centralizing logs from dozens or hundreds of services becomes critical for debugging, monitoring, and compliance. Apache Kafka is a popular backbone for log aggregation pipelines, offering durability, scalability, and real-time processing capabilities that traditional approaches lack. For foundational knowledge of Kafka's architecture, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ![Kafka-based log aggregation architecture](https://www.conduktor.io/assets/images/glossary/log-aggregation-with-kafka-0.webp) ## What is Log Aggregation? Log aggregation is the process of collecting log data from multiple sources across a distributed system and centralizing it for analysis, monitoring, and long-term storage. Every application component, whether a web server, database, or microservice, generates logs that contain valuable information about system behavior, errors, performance metrics, and security events. In a traditional monolithic application running on a single server, logs might simply write to local files that administrators can tail or grep. However, modern distributed systems require a more sophisticated approach. Log aggregation systems must handle high throughput (often millions of log events per second), provide reliable delivery guarantees, enable real-time analysis, and support multiple consumers with different processing needs. The goal is not just to collect logs, but to make them actionable. This means routing different log types to appropriate destinations, enabling both real-time alerting and historical analysis, and maintaining enough context to trace requests across service boundaries. Modern log aggregation increasingly integrates with distributed tracing systems (like OpenTelemetry) to correlate logs with traces and metrics, providing complete observability. ## Why Traditional Log Aggregation Falls Short Traditional log aggregation typically relies on agents that tail log files and forward them to a central server or storage system. While this approach works for small-scale deployments, it reveals significant limitations as systems grow. - **Scalability bottlenecks** emerge when a central log server becomes overwhelmed by the volume of incoming logs. Vertical scaling only delays the problem. Additionally, file-based collection introduces latency, logs must be written to disk before being read and forwarded, creating delays that make real-time analysis difficult. - **Reliability concerns** plague traditional systems. If the central log server goes down, logs may be lost or buffered locally until disk space runs out. There's often no guarantee of delivery, and recovering from failures can be complex. - **Inflexibility** becomes apparent when multiple teams need access to the same log data. Traditional systems typically support a single consumer pattern, logs go to one destination. If the security team needs access to audit logs while the operations team wants to feed them into a monitoring system, you end up duplicating log streams or building complex custom routing logic. These limitations led companies like LinkedIn to create Apache Kafka, originally built to solve log aggregation at scale. ## How Kafka Enables Modern Log Aggregation Apache Kafka addresses the shortcomings of traditional log aggregation through its distributed, append-only commit log architecture. At its core, Kafka treats logs as ordered streams of events that can be reliably stored and replayed. With Kafka 4.0+ running in KRaft mode (eliminating the ZooKeeper dependency), modern deployments benefit from simplified operations, faster metadata propagation, and support for millions of partitions. - **Distributed scalability** is built into Kafka's design. Topics are partitioned across multiple brokers (Kafka servers that store and serve data), allowing horizontal scaling of both storage and throughput. As log volume grows, you can add more brokers and partitions to handle the load. Each partition is an ordered log that can be consumed independently, enabling parallel processing. Modern Kafka clusters routinely handle millions of log events per second across thousands of topics. - **Durability and reliability** come from Kafka's replication model. Every partition can be replicated across multiple brokers, ensuring logs survive individual machine failures. Producers receive acknowledgments confirming that data has been safely replicated before considering a write successful. With tiered storage (available in Kafka 3.6+), you can retain logs cost-effectively by moving older data to object storage while keeping recent logs on local disks for low-latency access. - **Multi-consumer support** is native to Kafka. Multiple consumer groups (independent sets of consumers that track their own reading position) can read from the same log topic simultaneously without interfering with each other. The operations team can stream logs to their monitoring system, while the security team sends the same logs to their SIEM, and the data team archives everything to object storage, all consuming from the same Kafka topic at their own pace. - **Decoupling** is another key benefit. Producers (your applications) don't need to know about consumers (your log processing systems). You can add new log consumers, upgrade existing ones, or temporarily take them offline without impacting log production. ## Architecture Patterns for Kafka-Based Log Aggregation A typical Kafka log aggregation architecture consists of several layers, each with specific responsibilities. - **Log producers** are the applications and services generating logs. Rather than writing to local files, applications can use Kafka client libraries to send log events directly to Kafka topics. Alternatively, log shippers like Vector, Filebeat, or Fluentd can tail existing log files and forward them to Kafka, useful when modifying application code isn't feasible. Vector (written in Rust) has emerged as a modern, high-performance alternative with excellent resource efficiency and rich transformation capabilities. For cloud-native environments, OpenTelemetry Collector provides unified log, trace, and metric collection with consistent semantic conventions. - **Topic organization** requires thoughtful design. Common patterns include: - Separate topics per service (e.g., `logs.checkout-service`, `logs.payment-service`) - Topics organized by log level (e.g., `logs.error`, `logs.info`) - Topics based on log type (e.g., `logs.access`, `logs.application`, `logs.audit`) The choice depends on consumption patterns. If different teams consume different services' logs, service-based topics make sense. If you want to route all ERROR-level logs to a paging system regardless of source, level-based topics may be better. For comprehensive guidance on topic design decisions, see [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines). - **Partitioning strategy** impacts both performance and ordering guarantees. Partitioning by service instance ID ensures logs from a specific instance remain ordered, which helps with debugging. Partitioning by user ID or trace ID (a unique identifier that tracks a request across multiple services) can keep related logs together, making distributed tracing more effective. When using OpenTelemetry, the trace context is automatically propagated through log fields, enabling powerful correlation between logs, traces, and metrics. - **Log consumers** process the log streams. Common consumer patterns include: - Stream processors like Apache Flink 1.20+ or Kafka Streams that analyze logs in real-time, detecting anomalies or calculating metrics - Connectors (Kafka Connect) that sink logs to storage backends like Elasticsearch, Grafana Loki, ClickHouse, S3, or databases - OpenTelemetry Collector consuming from Kafka to forward logs to observability platforms - Custom applications that trigger alerts or update dashboards based on log patterns **Storage backends** have evolved significantly. While Elasticsearch remains popular for full-text search, Grafana Loki offers a more cost-effective alternative optimized for log aggregation with excellent Prometheus integration. ClickHouse provides exceptional query performance for log analytics at scale. Many organizations use a hybrid approach: Loki or ClickHouse for recent logs (last 30 days), with older logs archived to S3 via Kafka Connect for long-term retention and compliance. Consider a concrete example: an e-commerce platform with dozens of microservices. Each service sends structured JSON logs to a service-specific Kafka topic using OpenTelemetry SDKs, which automatically include trace and span IDs. A Flink application (Flink 1.20+) consumes all topics, extracts error events using pattern matching, and writes them to an `errors.critical` topic that triggers PagerDuty alerts. Meanwhile, a Kafka Connect S3 sink consumes all logs and archives them to object storage for compliance. A Kafka Connect Loki sink provides full-text search capabilities for debugging sessions with 30-day retention. All three consumers operate independently, at their own pace, from the same source data. ## Practical Implementation Example Here's a simple example of sending structured logs directly to Kafka using the Java producer: ```java import org.apache.kafka.clients.producer.*; import com.fasterxml.jackson.databind.ObjectMapper; import java.util.*; public class KafkaLogProducer { private final Producer producer; private final String topic; private final ObjectMapper mapper = new ObjectMapper(); public KafkaLogProducer(String bootstrapServers, String topic) { Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer"); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer"); props.put(ProducerConfig.ACKS_CONFIG, "all"); // Wait for all replicas props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy"); // Compress logs this.producer = new KafkaProducer<>(props); this.topic = topic; } public void logEvent(String level, String message, Map context) { try { Map logEntry = new HashMap<>(); logEntry.put("timestamp", System.currentTimeMillis()); logEntry.put("level", level); logEntry.put("message", message); logEntry.put("service", "checkout-service"); logEntry.put("context", context); String jsonLog = mapper.writeValueAsString(logEntry); ProducerRecord record = new ProducerRecord<>(topic, level, jsonLog); producer.send(record, (metadata, exception) -> { if (exception != null) { System.err.println("Failed to send log: " + exception.getMessage()); } }); } catch (Exception e) { System.err.println("Error creating log entry: " + e.getMessage()); } } public void close() { producer.close(); } } // Usage Map context = new HashMap<>(); context.put("userId", "user-12345"); context.put("orderId", "order-67890"); context.put("traceId", "trace-abc123"); // For distributed tracing logProducer.logEvent("ERROR", "Payment processing failed", context); ``` For production use with OpenTelemetry integration, consider using the OpenTelemetry Log4j2 or Logback appender, which automatically handles trace context propagation and provides consistent semantic conventions: ```xml kafka-broker:9092 snappy ``` This configuration automatically includes OpenTelemetry trace context in each log entry, enabling correlation with distributed traces. ## Real-World Implementation Considerations Successfully implementing Kafka-based log aggregation requires attention to several practical details. - **Schema design** is crucial. Structured logs (JSON or Avro) enable rich querying and processing compared to unstructured text. Define a consistent schema that includes standard fields like timestamp, service name, log level, message, and trace context. OpenTelemetry provides well-defined semantic conventions for log fields, ensuring consistency across services and compatibility with observability tools. Using Schema Registry (7.0+) helps enforce consistency and enables schema evolution as requirements change. Avro is particularly useful for log data as it provides compact binary encoding, reducing network and storage costs significantly compared to JSON. For details on schema management, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka). - **Retention policies** balance storage costs with debugging needs. Kafka can retain logs for hours, days, or weeks depending on disk capacity. With tiered storage (Kafka 3.6+), you can extend retention dramatically by offloading older log segments to object storage (S3, GCS, Azure Blob) while maintaining transparent access through Kafka's consumer API. Configure retention based on how far back you need to investigate issues, typically 7-30 days in hot storage (local disks) and 90+ days in cold storage (object storage) for compliance and deep analysis. For comprehensive coverage of tiered storage, see [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka). - **Volume management** prevents runaway log production from overwhelming the system. Implement sampling for high-volume debug logs while keeping all errors and warnings. Use quotas to limit how much data individual producers can send, protecting shared infrastructure from misbehaving services. For quota implementation details, see [Quotas and Rate Limiting in Kafka](https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka). - **Monitoring the monitor** is essential. Track metrics like producer send latency, consumer lag, and broker disk usage. Lag in log consumers might mean alerts aren't firing in real-time or logs aren't being archived properly. Tools like Conduktor provide comprehensive visibility into Kafka-based log pipelines, including consumer lag monitoring ([Monitoring Guide](https://docs.conduktor.io/guide/monitor-brokers-apps/index)), topic health ([Topics](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics)), throughput metrics, and schema registry management. This operational visibility is critical for maintaining reliable log aggregation at scale. For detailed strategies on tracking consumer lag, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). - **Security considerations** include encrypting logs in transit and at rest, especially for logs containing sensitive information. Use authentication and authorization to control which services can produce logs to which topics, and which teams can consume them. For comprehensive security guidance, see [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices). Since logs often contain audit information, also consider the patterns in [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). ## Integration with the Modern Observability Ecosystem Kafka-based log aggregation doesn't exist in isolation, it integrates with a broader ecosystem of tools and platforms. - **OpenTelemetry integration** has become the standard approach for modern log aggregation. OpenTelemetry provides SDKs for instrumenting applications to emit logs with consistent semantic conventions, including automatic trace context propagation. The OpenTelemetry Collector can receive logs from applications, process and enrich them, and export to Kafka topics. On the consumption side, collectors can read from Kafka and forward logs to various backends. This creates a vendor-neutral, standardized pipeline that works across cloud providers and observability platforms. - **Stream processing frameworks** like Apache Flink 1.20+ and Kafka Streams enable sophisticated log analysis. Flink's advanced features like exactly-once processing, low-latency stateful operations, and SQL support make it ideal for complex log analysis scenarios. You can build applications that correlate logs across services to detect distributed failure patterns, calculate real-time statistics about error rates or latency, or enrich logs with contextual information from other data sources. For choosing between stream processing frameworks, see [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink), and for detailed Flink capabilities, refer to [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). - **Modern log backends** have diversified beyond Elasticsearch. Grafana Loki provides a Prometheus-inspired approach optimized for log aggregation, indexing only metadata rather than full text content for cost efficiency. ClickHouse offers columnar storage with exceptional query performance for log analytics at massive scale. These backends typically consume logs from Kafka via dedicated connectors or custom consumers. - **Hybrid architectures** remain common during transitions. Logstash can consume from Kafka topics and transform logs before sending to Elasticsearch. Fluentd and Vector support Kafka as both input and output, enabling gradual migration from file-based collection. Vector's transformation capabilities make it particularly useful for normalizing logs from diverse sources before sending to Kafka. - **Observability platforms** increasingly support Kafka as a log source. Many APM and logging SaaS products can consume directly from Kafka topics, or you can use their agents to forward logs through Kafka for centralized control before they reach the vendor platform. This "Kafka in the middle" pattern provides flexibility to change vendors or add new consumers without modifying applications. - **Data governance and operational management** become critical as log pipelines scale. Conduktor provides comprehensive capabilities for managing Kafka-based log infrastructure, including topic discovery, schema validation, consumer lag monitoring, data quality checks, and access control. As log pipelines grow to hundreds of topics and dozens of consumer applications, having centralized visibility and governance prevents operational chaos and ensures teams can troubleshoot issues quickly. ## Summary Log aggregation with Kafka provides a scalable, reliable foundation for observability in distributed systems. By treating logs as event streams, Kafka enables real-time processing, multiple consumers, and flexible routing while maintaining durability and ordering guarantees. The main advantages are horizontal scalability through partitioning, reliability through replication, decoupling of producers and consumers, and support for multiple consumption patterns. KRaft mode (Kafka 4.0+), tiered storage (Kafka 3.6+), and OpenTelemetry integration have made this approach more cost-effective. Good implementation requires decisions about topic organization, partitioning, schema design, and retention policies. Pairing Kafka with OpenTelemetry for collection, Apache Flink or Kafka Streams for processing, and backends like Grafana Loki or ClickHouse for storage produces a complete pipeline for both operational and analytical needs. The 2025 log aggregation landscape emphasizes: - **Standardization** through OpenTelemetry for consistent log formats and trace correlation - **Cost efficiency** through tiered storage and optimized backends like Loki - **Operational visibility** through platforms like Conduktor for monitoring and governance - **Performance** with modern collectors like Vector and processors like Flink 1.20+ Kafka-based log aggregation is particularly valuable when operating microservices architectures, handling high log volumes (millions of events per second), requiring real-time log analysis, or needing multiple teams to consume the same log data independently. For simpler use cases with low volume and a single consumer, traditional approaches may suffice. As systems grow and log data becomes central to operations, security, and business intelligence, Kafka's streaming model scales with your organization. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/apache-kafka) - Distributed streaming platform for log aggregation - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) - Integrate log data with storage and analytics systems - [Consumer Lag Monitoring](https://www.conduktor.io/consumer-lag-monitoring) - Track log consumer health and processing performance ## Related Topics For deeper understanding of related concepts, explore these articles: - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Foundational Kafka architecture and concepts - [Kafka Topic Design Guidelines](https://www.conduktor.io/glossary/kafka-topic-design-guidelines) - Best practices for organizing log topics - [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring) - Tracking log consumer health - [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) - Securing log data in transit and at rest - [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) - Patterns for audit log collection - [Tiered Storage in Kafka](https://www.conduktor.io/glossary/tiered-storage-in-kafka) - Cost-effective long-term log retention - [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink) - Choosing stream processors for log analysis - [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) - Advanced log processing capabilities - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing structured log schemas - [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka) - Choosing log serialization formats - [Quotas and Rate Limiting in Kafka](https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka) - Controlling log volume - [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) - Modern Kafka deployment without ZooKeeper - [What is Data Observability: The Five Pillars](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars) - Broader observability context ## Sources and References 1. [Apache Kafka Documentation - Log Retention and Cleanup](https://kafka.apache.org/documentation/#design_log) 2. [Apache Kafka 4.0 Documentation - KRaft Mode](https://kafka.apache.org/documentation/#kraft) 3. [Apache Kafka 3.6+ Documentation - Tiered Storage](https://kafka.apache.org/documentation/#tiered_storage) 4. [The Log: What every software engineer should know about real-time data's unifying abstraction](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying) - Jay Kreps, LinkedIn Engineering 5. [OpenTelemetry Documentation - Logs](https://opentelemetry.io/docs/concepts/signals/logs/) 6. [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/) 7. [Grafana Loki Documentation](https://grafana.com/docs/loki/latest/) 8. [Apache Flink 1.20 Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.20/) 9. [Vector Documentation](https://vector.dev/docs/) 10. [Martin Kleppmann - Designing Data-Intensive Applications](https://dataintensive.net/) - Chapter on Stream Processing and Log-based Architectures Source: https://www.conduktor.io/glossary/log-aggregation-with-kafka --- # Log-Based vs Query-Based CDC: Comparison Change Data Capture (CDC) enables real-time data synchronization between systems. Choosing the right approach matters as organizations move toward event-driven architectures and real-time analytics. This article compares the two primary CDC strategies: log-based and query-based. For foundational CDC concepts, see [What is Change Data Capture (CDC): Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). For hands-on implementation guidance, refer to [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium). ## Understanding CDC Approaches CDC captures changes made to data sources and propagates them to downstream systems. While the goal remains consistent across implementations, the mechanism for detecting and capturing changes varies significantly between approaches. - **Query-Based CDC** periodically polls the source database using SQL queries to identify changed records. Think of it like checking your mailbox every few minutes to see if new mail arrived, you actively look for changes at regular intervals. It typically relies on timestamp columns, version numbers, or hash comparisons to detect modifications. - **Log-Based CDC** reads changes directly from the database's transaction log (also called write-ahead log or redo log), capturing every committed transaction without querying the source tables. This is like having a doorbell notification, the database tells you immediately when something changes. The transaction log is an internal file that databases maintain for crash recovery and replication, recording every write operation in order. ![log-based-vs-query-based-cdc-comparison diagram 1](https://www.conduktor.io/assets/images/glossary/log-based-vs-query-based-cdc-comparison-0.webp) ## Query-Based CDC: Architecture and Characteristics Query-based CDC operates by executing periodic SELECT queries against source tables to identify new, modified, or deleted records. ### How It Works The process follows this pattern: 1. Store the last synchronization checkpoint (timestamp, ID, or hash) 2. Execute a query filtering records beyond the checkpoint 3. Process captured changes 4. Update the checkpoint for the next iteration **Example: Timestamp-based polling** ```sql -- Initial checkpoint: store max timestamp SELECT MAX(updated_at) AS checkpoint FROM orders; -- Result: 2025-01-15 10:30:00 -- Subsequent polling query (runs every 5 minutes) SELECT order_id, customer_id, total_amount, status, updated_at FROM orders WHERE updated_at > '2025-01-15 10:30:00' ORDER BY updated_at ASC LIMIT 10000; -- Update checkpoint after processing UPDATE cdc_checkpoints SET last_sync = '2025-01-15 10:35:00' WHERE table_name = 'orders'; ``` **Example: Version-based polling** ```sql -- Each row has a monotonically increasing version number CREATE TABLE orders ( order_id BIGINT PRIMARY KEY, customer_id BIGINT, total_amount DECIMAL(10,2), version BIGINT DEFAULT 1, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); -- Trigger to increment version on every update CREATE TRIGGER orders_version_trigger BEFORE UPDATE ON orders FOR EACH ROW SET NEW.version = OLD.version + 1; -- Polling query using version number SELECT order_id, customer_id, total_amount, version FROM orders WHERE version > 1523678 ORDER BY version ASC LIMIT 10000; ``` The checkpoint value (timestamp `2025-01-15 10:30:00` or version `1523678`) is persisted between polling intervals to track progress. ### Advantages - **Simple Implementation**: Requires only database read access and basic SQL knowledge - **Database Agnostic**: Works with any database supporting SQL queries - **No Special Permissions**: Standard read permissions suffice - **Easy Debugging**: Query logic is transparent and traceable ### Limitations - **Performance Impact**: Periodic queries add load to production databases - **Missing Deletes**: Hard deletes cannot be captured without additional tracking tables - **Latency**: Polling interval creates inherent delay (typically minutes) - **Schema Dependencies**: Requires specific columns (timestamps, version fields) - **Incomplete Capture**: Only sees final state, misses intermediate changes within polling intervals ## Log-Based CDC: Architecture and Characteristics Log-based CDC reads changes directly from the database's transaction log, capturing every modification as it occurs. ### How It Works Database transaction logs record every committed change for recovery and replication purposes. Log-based CDC tools parse these binary logs (read and interpret the database's internal format) to extract change events, converting them into structured messages that applications can consume. ### Advantages - **Low Latency**: Near real-time capture (sub-second to seconds) - **Minimal Source Impact**: No queries against production tables - **Complete Change Capture**: Captures all operations including deletes - **Historical Accuracy**: Preserves order and timing of changes - **Schema Flexibility**: No special columns required in source tables ### Limitations - **Complex Setup**: Requires specialized tools and database-specific configuration - **Permission Requirements**: Needs elevated database privileges - **Database Specific**: Implementation varies by database system - **Operational Complexity**: Log retention, parsing errors, and schema evolution require careful management ## Head-to-Head Comparison | Dimension | Query-Based CDC | Log-Based CDC | |-----------|----------------|---------------| | **Latency** | Minutes (polling interval) | Sub-second to seconds | | **Source Impact** | High (periodic queries) | Minimal (log reading) | | **Delete Capture** | Difficult/impossible | Native support | | **Setup Complexity** | Low | High | | **Database Support** | Universal | Database-specific | | **Permissions** | Read access | Replication/log access | | **Resource Usage** | CPU/IO on source DB | Minimal on source | | **Operational Maturity** | Well-understood | Requires specialized expertise | | **Cost** | Lower (simpler tooling) | Higher (specialized tools) | | **Schema Changes** | Manual query updates | Automatic detection | ## Streaming Ecosystem Integration Modern data architectures rely on streaming platforms for event distribution, making CDC integration with streaming ecosystems important. For comprehensive coverage of Apache Kafka's role in CDC pipelines, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ### Kafka and Debezium **Debezium 2.5+** is the leading open-source platform for log-based CDC, providing connectors for MySQL, PostgreSQL, MongoDB, SQL Server, Oracle, Db2, Cassandra, and Vitess. It streams database changes directly to **Apache Kafka 4.0+** (with KRaft replacing ZooKeeper for simplified operations), enabling real-time data pipelines with sub-second latency. Modern Debezium deployments (2025) leverage several advanced features: - **Incremental Snapshotting**: Capture initial table state without locking, enabling safe production deployment - **Signal Tables**: Control connector behavior at runtime (pause, resume, trigger ad-hoc snapshots) - **Schema Evolution**: Automatic handling of DDL changes with schema registry integration - **Exactly-Once Semantics**: When paired with Kafka transactions, ensures no duplicate or lost events - **Multi-Tenant Support**: Single connector instance can replicate multiple databases efficiently A typical Debezium connector configuration for PostgreSQL: ```json { "name": "postgres-orders-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres.example.com", "database.port": "5432", "database.user": "debezium_user", "database.password": "${secret:dbpassword}", "database.dbname": "ecommerce", "database.server.name": "prod-orders", "table.include.list": "public.orders,public.order_items", "plugin.name": "pgoutput", "publication.autocreate.mode": "filtered", "slot.name": "debezium_orders_slot", "transforms": "unwrap", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "false", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "snapshot.mode": "incremental" } } ``` This produces CDC events in Kafka topics following the pattern `{server.name}.{schema}.{table}`: ```json { "before": null, "after": { "order_id": 12345, "customer_id": 67890, "total_amount": 149.99, "status": "pending", "created_at": 1735689600000 }, "source": { "version": "2.5.0.Final", "connector": "postgresql", "name": "prod-orders", "ts_ms": 1735689601234, "snapshot": "false", "db": "ecommerce", "schema": "public", "table": "orders", "lsn": 123456789 }, "op": "c", "ts_ms": 1735689601456 } ``` Key fields: - **op**: Operation type (`c` = create, `u` = update, `d` = delete, `r` = read/snapshot) - **before**: Previous row state (null for inserts, populated for updates/deletes) - **after**: New row state (null for deletes, populated for inserts/updates) - **source**: Metadata including LSN (log sequence number) for ordering and deduplication ### Governance and Observability As CDC implementations scale, governance and visibility become critical. Modern streaming governance platforms like **Conduktor** provide centralized management for Kafka-based CDC pipelines, offering: - **Schema Registry Management**: Track schema evolution across CDC topics with version history - **Topic Catalog**: Browse and document CDC topics with business context, ownership, and schema information - **Connector Health Monitoring**: Real-time visibility into Debezium connector status, throughput, and errors with [Kafka Connect Management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) - **Consumer Lag Tracking**: Identify downstream bottlenecks consuming CDC events - **Quality Monitoring**: Detect data quality issues, lag, and throughput anomalies with [Data Quality tools](https://docs.conduktor.io/guide/use-cases/observe-data-quality) - **Access Control**: Govern who can consume CDC streams and enforce data policies - **Alerting**: Proactive notifications for connector failures, lag spikes, or schema conflicts - **Testing and Chaos Engineering**: Use Conduktor Gateway to inject failures and test CDC pipeline resilience For teams running Debezium at scale, platforms like Conduktor simplify operations by providing: - Single-pane-of-glass view of all CDC connectors across environments - Historical metrics for capacity planning and SLA monitoring - Automated runbooks for common failure scenarios (connector restarts, rebalancing) - Integration with incident management tools (PagerDuty, Slack, etc.) Modern CDC monitoring (2025) also leverages: - **Kafka Lag Exporter**: Prometheus-compatible metrics for consumer lag - **JMX metrics**: Connector-level throughput, error rates, and snapshot progress - **Custom Kafka Streams applications**: Real-time data quality validation on CDC streams ## Operational Considerations Beyond the architectural differences, several operational aspects impact CDC implementation success. ### Initial Snapshots and Bootstrapping Both approaches must handle the initial state before capturing ongoing changes: **Query-Based Snapshot:** ```sql -- Full table scan for initial load SELECT * FROM orders ORDER BY order_id; -- Then switch to incremental polling SELECT * FROM orders WHERE updated_at > '2025-01-15 10:00:00'; ``` Simple but locks tables during read, potentially impacting production. **Log-Based Snapshot:** Debezium 2.5+ offers three snapshot modes: 1. **initial**: Lock-based consistent snapshot (default, safest) 2. **incremental**: Lock-free snapshot using signals (recommended for production) 3. **never**: Skip snapshot, capture only new changes Incremental snapshots avoid table locks by: - Reading data in chunks (configurable size) - Interleaving snapshot reads with live CDC events - Using watermarks to track progress ```json { "snapshot.mode": "incremental", "incremental.snapshot.chunk.size": 1024, "signal.data.collection": "public.debezium_signal" } ``` ### Schema Evolution Handling **Query-Based CDC**: Schema changes require manual intervention: - Update SQL queries to match new column structure - Add mappings for renamed columns - Handle data type changes in application code **Log-Based CDC**: Automatic schema evolution with caveats: - New columns appear in CDC events automatically - Dropped columns stop appearing in `after` block - Renamed columns require manual mapping (SMT - Single Message Transform) - Data type changes may require consumer updates Example SMT for column renaming: ```json { "transforms": "renameField", "transforms.renameField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.renameField.renames": "old_column_name:new_column_name" } ``` ### Transaction Ordering and Guarantees **Query-Based CDC**: No transaction guarantees: - Polling may capture partial transactions - Updates within polling window may be seen out of order - No way to correlate changes from the same transaction **Log-Based CDC**: Strong ordering guarantees: - All changes from a transaction appear in order - Transaction boundaries preserved (when using Debezium's transaction metadata) - LSN (Log Sequence Number) enables deduplication and ordering For exactly-once processing, use Kafka transactions: ```properties enable.idempotence=true transactional.id=debezium-connector-1 transaction.timeout.ms=900000 ``` ### Failure Recovery Patterns **Query-Based CDC**: Simple checkpoint recovery: - Store last successful checkpoint - Resume from checkpoint on failure - Risk: may re-process recent changes (at-least-once) **Log-Based CDC**: Connector offset management: - Kafka Connect stores offsets in internal topics - Connector resumes from last committed offset (LSN/binlog position) - Exactly-once possible with Kafka transactions If connector fails during snapshot: ```json { "snapshot.mode": "recovery", "snapshot.recovery.enabled": true } ``` ## Choosing the Right Approach Select your CDC strategy based on these criteria: ### Choose Query-Based CDC When: - Latency requirements are relaxed (5-15+ minute delays acceptable) - Source system load can accommodate periodic queries - Budget constrains specialized tooling investment - Team lacks database administration expertise - Deletes are rare or handled through soft-delete patterns - Database variety makes log-based tooling impractical ### Choose Log-Based CDC When: - Near real-time data (seconds) is required - Source database performance must be protected - Complete audit trail including deletes is necessary - Integrating with streaming platforms like [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Organization has database administration capabilities - Data consistency and ordering matter - Supporting event-driven architectures (see [CDC for Microservices Event-Driven Architectures](https://www.conduktor.io/glossary/cdc-for-microservices-event-driven-architectures)) ### Hybrid Approaches Some organizations combine both approaches: - **Log-based for critical tables**: High-value, frequently changing data - **Query-based for dimension tables**: Slowly changing, less critical data - **Fallback mechanisms**: Query-based as backup when log access is unavailable ## Summary Log-based and query-based CDC serve the same fundamental purpose but differ significantly in architecture, performance, and operational characteristics. - **Query-based CDC** offers simplicity and universal compatibility, making it suitable for batch-oriented use cases with relaxed latency requirements. Its polling mechanism introduces source database load and inherent delays but requires minimal setup and expertise. - **Log-based CDC** delivers near real-time capture with minimal source impact by reading transaction logs directly. While operationally complex and database-specific, it excels in streaming architectures and event-driven systems where latency and completeness matter. For modern data engineering scenarios involving real-time analytics (see [CDC for Real-Time Data Warehousing](https://www.conduktor.io/glossary/cdc-for-real-time-data-warehousing)), microservices integration, or event streaming platforms like Kafka, log-based CDC with tools like Debezium generally provides superior capabilities. Organizations implementing these solutions should leverage governance platforms to maintain visibility and control as CDC pipelines scale. The choice ultimately depends on your specific requirements: if you need simplicity and can accept delays, query-based CDC suffices. If real-time data, minimal source impact, and complete change capture are priorities, invest in log-based CDC infrastructure. ## Related Concepts - [What is Change Data Capture: CDC Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) - Foundational CDC concepts that apply to both log-based and query-based approaches. - [Implementing CDC with Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) - Hands-on guide to implementing production-grade log-based CDC using Debezium for real-time database replication. - [Streaming Data Pipeline](https://www.conduktor.io/glossary/streaming-data-pipeline) - Understand how CDC integrates into broader streaming data pipeline architectures for real-time data movement. ## Sources and References - **Debezium Documentation**: [https://debezium.io/documentation/](https://debezium.io/documentation/) - **Kafka Connect Documentation**: [https://kafka.apache.org/documentation/#connect](https://kafka.apache.org/documentation/#connect) - **PostgreSQL Logical Decoding**: [https://www.postgresql.org/docs/current/logicaldecoding.html](https://www.postgresql.org/docs/current/logicaldecoding.html) - **MySQL Binary Log**: [https://dev.mysql.com/doc/refman/8.0/en/binary-log.html](https://dev.mysql.com/doc/refman/8.0/en/binary-log.html) - **Martin Kleppmann - "Designing Data-Intensive Applications"**: Chapter 11 on Stream Processing - **Apache Kafka Connect JDBC Connector**: Open-source connector for query-based CDC Source: https://www.conduktor.io/glossary/log-based-vs-query-based-cdc-comparison --- # Low-Latency Pipelines: Achieving Millisecond Response Times In data streaming, latency is the interval between an event occurring and your system responding to it. Low-latency pipelines operate in milliseconds to sub-second ranges, well below what batch processing targets. Fraud detection systems need to block transactions before they go through. High-frequency trading measures competitive advantage in microseconds. Recommendation engines must respond before a user moves on. These applications require low latency, not just benefit from it. A low-latency pipeline typically targets end-to-end latencies under 100 milliseconds, with some critical applications demanding single-digit millisecond responses. Achieving this requires understanding every source of delay and making deliberate architectural choices at every layer of your streaming infrastructure. ## Understanding Latency Sources Before optimizing for low latency, you must identify where time is being spent. Latency accumulates across multiple dimensions of your pipeline: ![Before optimizing for low latency, you must identify where time is being spent. Latency accumulates across multiple dimensions of your pipeline](https://www.conduktor.io/assets/images/glossary/low-latency-pipeline-0.webp) - **Network transmission** often represents the most significant and variable component. Data traveling across regions or poorly configured networks can introduce tens to hundreds of milliseconds. Even within a data center, network congestion and routing decisions impact latency. - **Serialization and deserialization** impose CPU overhead that many developers underestimate. Converting objects to bytes and back isn't free, the choice between JSON, Avro, or Protocol Buffers can mean the difference between 5ms and 50ms per message at scale. - **Processing logic** includes your business rules, transformations, and enrichments. Complex joins, external lookups, or inefficient algorithms create bottlenecks. Even well-written code suffers if it blocks on I/O or waits for external services. - **I/O operations** to disk or external systems interrupt the flow of data. While SSDs have dramatically reduced disk latency, any synchronous write still introduces milliseconds of delay. Remote database queries or API calls can add 10-100ms per operation. - **Garbage collection pauses** in JVM-based systems like Kafka or Flink can cause intermittent spikes. A poorly tuned JVM might pause for hundreds of milliseconds during a major GC cycle, creating latency outliers that violate SLAs. - **Queue waiting times** occur when messages sit in buffers waiting for processing. If your pipeline can't keep pace with incoming data, queues grow and latency increases proportionally. For strategies to manage this, see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems). ## Design Principles for Low-Latency Pipelines Building a low-latency pipeline requires architectural discipline from the start. Several core principles guide effective designs: - **Minimize processing hops.** Each stage in your pipeline adds latency. A message flowing through Kafka to a stream processor to another Kafka topic to a database accumulates delays at each transition. Evaluate whether you truly need each component or if operations can be consolidated. - **Avoid blocking operations.** Any synchronous wait, for a database query, an HTTP call, or a file write, stalls your processing thread. Embrace asynchronous, non-blocking patterns. Use reactive frameworks or async/await patterns to keep threads productive while waiting for I/O. - **Design for asynchronous processing.** Structure your pipeline as a series of independent, asynchronous stages. Each component consumes messages, processes them without waiting, and publishes results immediately. This allows different stages to work in parallel rather than sequentially. - **Prioritize data locality.** Co-locate related processing steps to avoid network hops. If a Kafka consumer must enrich data from a database, place them in the same data center or availability zone. Better yet, cache frequently accessed reference data locally to eliminate the remote call entirely. - **Consider stateless vs stateful trade-offs.** Stateless processing scales easily and introduces minimal latency, but many real-world applications require state (aggregations, joins, pattern detection). When state is necessary, use in-memory state stores and minimize serialization overhead. ## Technology Choices and Trade-offs Your technology stack profoundly impacts achievable latency. Understanding the characteristics of each component helps you make informed decisions: **Apache Kafka** excels at low-latency message delivery when properly configured. Unlike traditional message queues that optimize for guaranteed delivery semantics, Kafka's log-based architecture allows for extremely fast appends and sequential reads. For detailed configuration strategies, see [Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide). With Kafka 4.0+ and KRaft mode (eliminating ZooKeeper overhead), you can achieve sub-5ms producer-to-broker latencies. Critical producer settings for low latency: ```properties # Kafka Producer Configuration for Low Latency linger.ms=0 # Send immediately, no batching delay batch.size=16384 # Small batches for quick sends acks=1 # Leader acknowledgment only (trade durability for speed) compression.type=none # Avoid CPU overhead (or use lz4 if network-bound) buffer.memory=33554432 # Adequate buffer to prevent blocking socket.send.buffer.bytes=131072 socket.receive.buffer.bytes=131072 ``` For latency-sensitive consumers, configure: ```properties # Kafka Consumer Configuration for Low Latency fetch.min.bytes=1 # Don't wait for batches fetch.max.wait.ms=0 # Return immediately with available data max.poll.records=500 # Smaller batches for faster processing ``` KRaft mode (Kafka 4.0+) eliminates controller election overhead and reduces broker communication latency by 20-50% compared to ZooKeeper-based deployments. For fundamentals, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). Traditional message queues like RabbitMQ or ActiveMQ provide different guarantees but often introduce higher latency due to acknowledgment protocols and message routing overhead. They excel in scenarios requiring complex routing or transactional guarantees, not raw speed. **For stream processing,** Apache Flink 1.19+ is purpose-built for low latency with true streaming semantics and event-time processing. It processes events individually as they arrive, making it ideal for millisecond-latency requirements. For in-depth coverage, see [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). Flink's state backend choice significantly impacts latency: - **HashMapStateBackend**: In-memory state for lowest latency (sub-millisecond access), requires sufficient heap - **EmbeddedRocksDBStateBackend**: Disk-backed state for larger state sizes, adds 1-5ms latency per access Spark Structured Streaming, while continuously improving, uses micro-batching (even with continuous processing mode), which introduces inherent latency from batch intervals. For understanding micro-batching trade-offs, see [Micro-Batching in Streaming](https://www.conduktor.io/glossary/micro-batching-streaming). **Serialization formats** present a classic latency-throughput-compatibility trade-off. JSON is human-readable and flexible but verbose and slow to parse. Apache Avro provides schema evolution and compact encoding with good performance. Protocol Buffers offer excellent performance and smaller message sizes but require more rigid schema management. For lowest latency, Protocol Buffers or even more specialized formats like FlatBuffers reduce serialization overhead. For detailed comparison and best practices, see [Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka). ## Optimization Techniques Once your architecture is sound, fine-tuning extracts the final milliseconds of latency: - **Batching strategies** require careful balance. Sending messages one-by-one minimizes latency but sacrifices throughput and increases CPU overhead. Kafka's `linger.ms` parameter controls how long producers wait to batch messages, set it to 0 for minimum latency, but understand this reduces throughput. Monitor your p99 latencies to find the sweet spot for your workload. - **Compression trade-offs** must account for CPU cost. While compression reduces network transfer time, it adds CPU overhead for compression and decompression. For low-latency pipelines, test whether the network savings justify the CPU cost. In well-provisioned networks, uncompressed messages might be faster. If you do compress, use fast algorithms like LZ4 or Snappy rather than heavier options like gzip. - **Partitioning and parallelism** allow horizontal scaling. More Kafka partitions enable more parallel consumers, reducing per-consumer load and latency. However, too many partitions introduce management overhead. Size partitions to match your consumer capacity and scale horizontally rather than vertically when possible. - **Connection pooling and keepalives** eliminate connection establishment overhead. Opening a new database connection or HTTP connection can cost 50-100ms. Maintain warm connection pools and use HTTP keepalive to reuse connections across requests. - **Memory management and GC tuning** for JVM applications requires expertise but pays dividends. Modern garbage collectors (GC) automatically manage memory but can cause pause times that spike latency: - **G1GC** (Garbage-First): Default in Java 11+, targets predictable pause times (10-50ms) - **ZGC** (Z Garbage Collector): Ultra-low latency GC in Java 17+, pauses typically <1ms even with large heaps - **Shenandoah GC**: Another low-latency option with sub-10ms pauses - **Generational ZGC** (Java 21+): Combines ZGC's low latency with generational memory management for better throughput For sub-10ms latency requirements, use ZGC or Shenandoah. Size your heap appropriately, too small causes frequent GCs, too large causes longer pauses. Monitor GC logs and tune based on your specific workload patterns: ```bash # Example JVM flags for low-latency Kafka/Flink applications (Java 17+) -XX:+UseZGC -XX:MaxGCPauseMillis=5 -Xms8g -Xmx8g # Equal min/max prevents heap resizing -XX:+AlwaysPreTouch # Initialize memory at startup ``` ## Monitoring and Measuring Latency You cannot optimize what you don't measure. Effective latency monitoring requires instrumentation throughout your pipeline: - **End-to-end latency tracking** follows individual events from source to sink. Embed timestamps in your messages at origin, then measure elapsed time at each processing stage. This reveals which components contribute most to total latency and where optimization efforts should focus. - **Percentile metrics** matter more than averages. Your median (p50) latency might be 10ms, but if your p99 is 500ms, 1% of users experience terrible performance. Monitor p50, p95, p99, and p99.9 latencies separately. Many SLAs focus on p99 because outliers often indicate systemic issues. For comprehensive lag tracking strategies, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). - **Identifying bottlenecks** requires correlation between latency metrics and system metrics. Is high latency correlated with CPU saturation? Network throughput limits? Garbage collection pauses? Tools like distributed tracing (Jaeger, Zipkin) visualize latency breakdowns across microservices, making bottlenecks obvious. - **Alerting strategies** should trigger on latency percentiles, not averages. Alert when p99 latency exceeds thresholds for a sustained period (avoiding noise from transient spikes). Include latency SLOs in your incident response runbooks. ## Infrastructure and Hardware Considerations Software optimization can only go so far, hardware and infrastructure set fundamental limits: - **Network topology and co-location** dramatically affect latency. Placing Kafka brokers, stream processors, and downstream services in the same availability zone reduces network hops and latency variance. For the most demanding applications, dedicated network links or premium networking tiers reduce jitter. - **Network tuning** involves operating system and TCP parameter optimization. Increase TCP buffer sizes for high-throughput connections. Adjust MTU (Maximum Transmission Unit) to reduce packet fragmentation. Disable Nagle's algorithm by enabling `TCP_NODELAY` (in Kafka: `socket.nagle.enable=false`) to prevent batching delays on small messages. Nagle's algorithm batches small TCP packets to improve efficiency, but this adds 40-200ms of latency, unacceptable for low-latency pipelines. - **Hardware choices** impact latency at every layer. SSDs provide 10-100x lower latency than spinning disks for any stateful operations. Modern NICs with kernel bypass (DPDK) eliminate context switches for ultra-low latency applications. CPU selection matters too, higher clock speeds and better single-thread performance benefit stream processing. - **Cloud vs on-premises** considerations involve trade-offs. Cloud providers offer elasticity and managed services but introduce network variability and multi-tenancy effects. Bare-metal or dedicated instances provide more consistent performance but less flexibility. Premium networking options in cloud environments (AWS Enhanced Networking, Azure Accelerated Networking) narrow this gap. ## The Latency-Throughput-Cost Triangle Low-latency pipelines require balancing three competing concerns: - **Latency vs Throughput:** Optimizing for minimum latency (small batches, no compression, many partitions) often reduces maximum throughput. Conversely, maximizing throughput (large batches, compression, buffering) increases latency. Define your requirements clearly, is it more important to process 100,000 events per second with 50ms latency or 50,000 events per second with 5ms latency? - **Latency vs Cost:** Achieving single-digit millisecond latencies is expensive. It requires premium hardware, over-provisioning for headroom, dedicated networking, and significant engineering effort. For many applications, spending 10x more to reduce latency from 50ms to 5ms doesn't justify the business value. Understand your true latency requirements before optimizing prematurely. - **Governance and Control:** As pipelines become more complex and performance-critical, maintaining quality and compliance becomes challenging. Tools like Conduktor provide governance capabilities for streaming platforms, ensuring that performance optimizations don't compromise data quality, schema compatibility, or access controls. Monitor your topics and validate data quality with [Data Quality tools](https://docs.conduktor.io/guide/use-cases/observe-data-quality), and manage Kafka resources efficiently through the [Topics interface](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics). Conduktor's real-time monitoring can track end-to-end latencies, identify slow consumers, and alert on SLA violations, critical for maintaining predictable low-latency performance. Clean, well-governed streams are essential, schema validation failures or malformed messages introduce latency spikes and recovery overhead. ## Practical Example: End-to-End Configuration Here's a complete example of configuring a low-latency Kafka-to-Flink pipeline: ```yaml # Kafka Broker Configuration (server.properties) # KRaft mode for reduced latency (Kafka 4.0+) num.network.threads=8 num.io.threads=16 socket.send.buffer.bytes=131072 socket.receive.buffer.bytes=131072 socket.request.max.bytes=104857600 replica.lag.time.max.ms=10000 ``` ```java // Java Producer Configuration Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("linger.ms", "0"); // No batching props.put("batch.size", "16384"); props.put("acks", "1"); // Leader ack only props.put("compression.type", "none"); props.put("buffer.memory", "33554432"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer"); KafkaProducer producer = new KafkaProducer<>(props); ``` ```java // Flink Low-Latency Configuration StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setBufferTimeout(0); // Disable buffering for immediate forwarding env.enableCheckpointing(60000); // Checkpoint every minute (balance latency vs fault tolerance) // Use HashMapStateBackend for in-memory state (lowest latency) env.setStateBackend(new HashMapStateBackend()); KafkaSource source = KafkaSource.builder() .setBootstrapServers("localhost:9092") .setTopics("transactions") .setStartingOffsets(OffsetsInitializer.latest()) .setValueOnlyDeserializer(new ProtobufDeserializationSchema<>(Transaction.class)) .setProperty("fetch.min.bytes", "1") // Don't wait for batches .setProperty("fetch.max.wait.ms", "0") .build(); ``` This configuration prioritizes latency over throughput and durability. For production use, adjust based on your specific SLA requirements. ## Use Cases Demanding Low Latency Understanding which applications truly need low latency helps prioritize engineering investment: - **Fraud detection** systems must evaluate transactions in milliseconds before authorizing payment. A 100ms latency budget leaves little room for complex rule evaluation and machine learning inference, driving architectural decisions around pre-computed features and in-memory state. For implementation patterns, see [Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming). - **High-frequency trading** operates at microsecond scales where even minor latency improvements provide competitive advantage. These systems often bypass general-purpose stream processors entirely in favor of custom C++ applications with kernel bypass networking. - **Real-time recommendation engines** power e-commerce and content platforms where user engagement depends on immediate, relevant suggestions. Latencies above 100ms create perceptible lag in user interfaces, degrading experience and conversion rates. For building these systems, see [Building Recommendation Systems with Streaming Data](https://www.conduktor.io/glossary/building-recommendation-systems-with-streaming-data). - **IoT and industrial applications** like autonomous vehicles or manufacturing process control require deterministic latencies to ensure safety and coordination. These systems often use specialized real-time operating systems and dedicated networks. - **Real-time machine learning inference** requires low-latency feature extraction, model serving, and result delivery. For comprehensive patterns, see [Real-Time ML Inference with Streaming Data](https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data). ## Troubleshooting High Latency When latency exceeds your targets, systematically investigate these common causes: **P99 Latency Spikes:** 1. Check GC logs for major collection pauses (>100ms) 2. Review Kafka broker logs for ISR shrinks (replication delays) 3. Examine network metrics for packet loss or retransmission 4. Verify no resource contention (CPU throttling, memory swapping) **Steady High Latency:** 1. Measure producer `batch.queue.time.ms` and consumer `fetch-latency-avg` metrics 2. Check if consumers are keeping up, review lag metrics 3. Profile application code for slow operations (database calls, complex computations) 4. Verify serialization format efficiency and message sizes **Intermittent Latency:** 1. Correlate with batch jobs or scheduled tasks (resource competition) 2. Check for network routing changes or cross-AZ traffic 3. Review deployment/scaling events 4. Monitor for cascading failures from downstream systems Use tools like Conduktor for real-time latency monitoring across your entire Kafka ecosystem, helping identify bottlenecks quickly. ## Related Topics For deeper exploration of related concepts: - **[Apache Kafka](https://www.conduktor.io/glossary/apache-kafka)** - Foundational platform architecture and capabilities - **[Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide)** - Comprehensive configuration optimization strategies - **[What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing)** - Stream processing engine for low-latency applications - **[Message Serialization in Kafka](https://www.conduktor.io/glossary/message-serialization-in-kafka)** - Choosing and optimizing serialization formats - **[Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems)** - Managing flow control and queue buildup - **[Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring)** - Tracking and alerting on processing delays - **[Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka)** - Modern Kafka architecture without ZooKeeper - **[Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming)** - Practical low-latency use case ## Summary Building a low-latency streaming pipeline means eliminating waste at every layer. From technology choices to JVM garbage collection tuning, every decision either adds or removes milliseconds from your critical path. **Key Takeaways:** 1. **Start with requirements** - Not all applications need single-digit millisecond responses. Define your true latency SLAs before optimizing. 2. **Measure comprehensively** - Track p50, p95, p99, and p99.9 latencies separately. Outliers often indicate systemic issues. 3. **Optimize architecture first** - Minimize hops, avoid blocking operations, and embrace asynchronous patterns before tuning parameters. 4. **Leverage modern technology** - Kafka 4.0+ with KRaft, Flink 1.19+, ZGC, and Protocol Buffers provide significant latency improvements. 5. **Balance trade-offs** - Low latency requires investment in hardware, engineering effort, and often compromises on throughput or cost. Low latency is not free. It requires premium hardware, over-provisioning, dedicated networking, and engineering effort. Balance those costs against your application's actual requirements. A pipeline that's fast but unreliable serves no one. Streaming governance ensures optimizations don't introduce risk or break compliance. With careful design, appropriate technology choices, and systematic optimization, Apache Kafka and Apache Flink can deliver millisecond-scale latencies for demanding real-time applications. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - High-performance streaming platform for low-latency pipelines - [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) - Stream processor designed for millisecond-latency requirements - [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) - Managing flow control to maintain low latency ## Sources and References 1. **Apache Kafka Documentation - Performance Tuning** - Official Apache Kafka documentation covering producer and broker configuration for optimal performance and low latency. [https://kafka.apache.org/documentation/#performance](https://kafka.apache.org/documentation/#performance) 2. **Apache Flink Documentation - Low Latency Streaming** - Official documentation on Apache Flink's architecture and configuration for achieving low-latency stream processing. [https://flink.apache.org/](https://flink.apache.org/) 3. **Confluent - Optimizing Kafka for Low Latency** - Best practices and configuration recommendations from Confluent for minimizing end-to-end latency in Kafka deployments. [https://www.confluent.io/blog/configure-kafka-to-minimize-latency/](https://www.confluent.io/blog/configure-kafka-to-minimize-latency/) 4. **Martin Kleppmann - Designing Data-Intensive Applications** - Comprehensive coverage of distributed systems performance, including latency optimization techniques and trade-offs. O'Reilly Media, 2017. 5. **Google - The Tail at Scale** - Research paper on understanding and optimizing tail latency in distributed systems, applicable to streaming architectures. [https://research.google/pubs/pub40801/](https://research.google/pubs/pub40801/) Source: https://www.conduktor.io/glossary/low-latency-pipeline --- # Maintaining Iceberg Tables: Compaction and Cleanup Apache Iceberg's time-travel capabilities and transactional guarantees come with a maintenance cost: small files, expired snapshots, and orphan data files can accumulate over time. Without proper maintenance, these artifacts degrade query performance, inflate storage costs, and complicate metadata management. This article explores the essential maintenance procedures that keep Iceberg tables healthy and performant in production environments. For foundational understanding of Iceberg's architecture and how maintenance relates to metadata layers, see [Iceberg Table Architecture: Metadata and Snapshots](https://www.conduktor.io/glossary/iceberg-table-architecture-metadata-and-snapshots). For broader lakehouse context, refer to [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture). ![Iceberg table maintenance lifecycle](https://www.conduktor.io/assets/images/glossary/maintaining-iceberg-tables-compaction-and-cleanup-0.webp) ## Understanding Iceberg's Maintenance Challenges ### Small File Problem Iceberg tables can accumulate numerous small files through incremental writes, streaming ingestion, or high-frequency updates. Each insert operation typically creates new data files rather than modifying existing ones, following Iceberg's immutable file design. While this approach enables ACID transactions and time travel, it leads to several performance issues: - **Query overhead**: Reading hundreds of small files is slower than reading fewer large files due to I/O overhead and metadata processing - **Planning latency**: Query planning time increases with the number of files the optimizer must evaluate - **Cloud storage costs**: Object storage systems often charge per-request, making small files expensive to read ### Metadata Growth Every commit to an Iceberg table creates a new snapshot, capturing the table's state at that point in time. Each snapshot references manifest files, which in turn reference data files. Over time, this metadata accumulates (for detailed architecture, see [Iceberg Table Architecture: Metadata and Snapshots](https://www.conduktor.io/glossary/iceberg-table-architecture-metadata-and-snapshots)): - Snapshot history grows linearly with commit frequency - Manifest files accumulate faster in tables with frequent schema evolution or partition changes - Metadata JSON files can reach sizes that impact table loading performance ### Orphan Files Orphan files are data files present in table storage but not referenced by any snapshot. They arise from: - **Failed writes**: Transactions that write files but fail before committing metadata - **Concurrent operations**: Race conditions in distributed systems - **Improper cleanup**: Manual interventions or external tools modifying table storage Orphan files waste storage but don't affect correctness since Iceberg never reads unreferenced files. ## Compaction: Consolidating Small Files Compaction merges small data files into larger ones, optimizing file sizes for query performance. Iceberg provides two compaction strategies: **bin-packing** and **sorting**. ### Bin-Packing Compaction Bin-packing groups small files together without changing data order (like packing items efficiently into bins), making it ideal for tables where write order matters or when you want a fast compaction process. ```sql -- Spark SQL: Rewrite small files using bin-packing CALL catalog.system.rewrite_data_files( table => 'db.events', strategy => 'binpack', options => map( 'target-file-size-bytes', '536870912', -- 512 MB target 'min-input-files', '5' -- Only rewrite partitions with 5+ files ) ); ``` ```python # PySpark: Programmatic compaction with filtering from pyspark.sql import SparkSession spark = SparkSession.builder.appName("compaction").getOrCreate() # Compact specific partitions spark.sql(""" CALL catalog.system.rewrite_data_files( table => 'db.events', strategy => 'binpack', where => 'event_date >= current_date() - 7' ) """) ``` ### Sort-Based Compaction Sort-based compaction rewrites data in a sorted order, improving query performance through better data clustering and predicate pushdown. This is particularly valuable for tables with frequent range queries. ```sql -- Compact and sort by frequently queried columns -- Choose sort columns based on your most common WHERE clauses and JOIN keys CALL catalog.system.rewrite_data_files( table => 'db.user_events', strategy => 'sort', sort_order => 'user_id, event_timestamp', -- Sort by filter/join columns options => map('target-file-size-bytes', '536870912') ); ``` ### Fault-Tolerant Compaction (Iceberg 1.6+) For large tables or long-running compaction jobs, Iceberg 1.6+ supports **partial progress mode**, which commits work incrementally to prevent data loss if the operation fails partway through: ```sql -- Enable partial progress for fault-tolerant compaction CALL catalog.system.rewrite_data_files( table => 'db.large_events', strategy => 'binpack', options => map( 'target-file-size-bytes', '536870912', 'partial-progress.enabled', 'true', -- Commit progress incrementally 'partial-progress.max-commits', '10' -- Commit every 10 file groups ) ); ``` Partial progress mode is essential for production environments with: - Tables containing millions of small files - Resource-constrained compaction windows - Cloud environments with spot instances that may be interrupted - Distributed compaction across multiple partitions ### Compaction Best Practices - **Schedule during low-traffic periods**: Compaction is resource-intensive and benefits from dedicated compute resources - **Partition-aware compaction**: Use `where` clauses to compact only recently modified partitions - **Monitor file sizes**: Set target file sizes based on your query patterns (typically 256 MB to 1 GB) - **Combine with snapshot expiration**: Compact first, then expire snapshots to maximize cleanup ## Expiring Snapshots Snapshot expiration removes old snapshots and their associated metadata files, reclaiming storage and preventing unbounded metadata growth. ```sql -- Expire snapshots older than 7 days CALL catalog.system.expire_snapshots( table => 'db.events', older_than => TIMESTAMP '2025-11-30 00:00:00', retain_last => 10 -- Always keep at least 10 snapshots ); ``` ```python # PySpark: Expire snapshots with retention period spark.sql(""" CALL catalog.system.expire_snapshots( table => 'db.events', older_than => current_timestamp() - INTERVAL '7' DAY, retain_last => 5, max_concurrent_deletes => 10 -- Parallel file deletion threads ) """) ``` ### Retention Considerations - **Compliance requirements**: Ensure retention periods satisfy audit and regulatory needs - **Time-travel dependencies**: Don't expire snapshots that downstream consumers rely on for incremental processing - **Snapshot metadata size**: Check metadata directory sizes to determine aggressive expiration schedules ```sql -- Query snapshot history to understand retention needs SELECT snapshot_id, committed_at, operation, summary FROM db.events.snapshots ORDER BY committed_at DESC LIMIT 20; ``` ## Removing Orphan Files Orphan file removal identifies and deletes files not referenced by any valid snapshot. This operation is safe only after ensuring no concurrent writes are occurring. ```sql -- Remove orphan files older than 3 days (safety margin) CALL catalog.system.remove_orphan_files( table => 'db.events', older_than => TIMESTAMP '2025-12-04 00:00:00', location => 's3://bucket/warehouse/db/events' ); ``` ```python # PySpark: Dry-run to preview orphan files result = spark.sql(""" CALL catalog.system.remove_orphan_files( table => 'db.events', older_than => current_timestamp() - INTERVAL '3' DAY, dry_run => true ) """) result.show() ``` ### Safety Guidelines - **Use safety margins**: Only delete files older than your longest-running transaction or write operation - **Run during maintenance windows**: Ensure no active writers exist when removing orphans - **Test with dry-run**: Always preview deletions before executing - **Backup metadata**: Maintain metadata backups before aggressive cleanup operations ## Compacting Manifest Files While data file compaction addresses the small file problem, **manifest files themselves can also accumulate** and slow down query planning. Each write operation creates new manifest files that track data file changes. Over time, tables with frequent writes accumulate hundreds or thousands of small manifest files. ### Understanding Manifest Bloat Manifest files contain metadata about data files (paths, row counts, partition values, column statistics). When query engines plan queries, they must read all relevant manifest files to determine which data files to scan. Too many small manifest files cause: - **Slow query planning**: Reading thousands of small manifest files sequentially - **Metadata overhead**: Storing many small objects inefficiently in cloud storage - **Cache inefficiency**: Limited manifest caching with fragmented metadata ### Manifest Compaction Procedure Iceberg provides `rewrite_manifests` to consolidate small manifest files: ```sql -- Compact manifest files (Iceberg 1.3+) CALL catalog.system.rewrite_manifests( table => 'db.events', use_caching => true -- Cache manifest contents during rewrite ); ``` ```python # PySpark: Manifest compaction with filtering from pyspark.sql import SparkSession spark = SparkSession.builder.appName("manifest_compaction").getOrCreate() # Compact manifests for specific partitions spark.sql(""" CALL catalog.system.rewrite_manifests( table => 'db.events', use_caching => true ) """) ``` ### When to Compact Manifests Monitor manifest file counts and schedule compaction when: - Query planning times increase noticeably - Tables have more than 100 manifest files per snapshot - Frequent small writes create many single-file manifests - After large bulk operations (imports, migrations) ```sql -- Check manifest file count SELECT COUNT(*) as manifest_count FROM db.events.manifests WHERE added_snapshot_id = (SELECT MAX(snapshot_id) FROM db.events.snapshots); ``` Manifest compaction is particularly important for streaming tables with high write frequency, where each micro-batch creates new manifest files. ## Streaming Ecosystem Integration Iceberg maintenance becomes more critical in streaming environments where continuous writes amplify small file and metadata growth. ### Spark Structured Streaming Maintenance ```python from pyspark.sql.streaming import StreamingQuery # Streaming write with periodic compaction trigger query = ( stream_df .writeStream .format("iceberg") .outputMode("append") .option("checkpointLocation", "s3://bucket/checkpoints/events") .trigger(processingTime="5 minutes") .toTable("db.events") ) # Separate maintenance job def maintain_table(): spark.sql(""" CALL catalog.system.rewrite_data_files( table => 'db.events', strategy => 'binpack', where => 'event_hour >= current_timestamp() - INTERVAL 2 HOUR' ) """) # Schedule maintenance every hour from apscheduler.schedulers.background import BackgroundScheduler scheduler = BackgroundScheduler() scheduler.add_job(maintain_table, 'interval', hours=1) scheduler.start() ``` ### Flink Integration Apache Flink 1.18+ provides native Iceberg maintenance actions through the `FlinkActions` API, enabling programmatic compaction integrated with your streaming jobs: ```java // Flink 1.18+: Programmatic maintenance with FlinkActions import org.apache.iceberg.flink.actions.Actions; import org.apache.iceberg.catalog.TableIdentifier; import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; StreamTableEnvironment tableEnv = // ... your table environment // Load the Iceberg table Table icebergTable = catalog.loadTable(TableIdentifier.of("db", "events")); // Configure and execute compaction action Actions.forTable(icebergTable) .rewriteDataFiles() .option("target-file-size-bytes", "536870912") // 512 MB target .option("min-input-files", "5") // Only compact partitions with 5+ files .option("use-starting-sequence-number", "true") // Incremental compaction .execute(); ``` For tables with write-time configuration, you can also set target file sizes in table properties: ```sql -- Configure target file sizes at table creation CREATE TABLE events ( event_id STRING, user_id BIGINT, event_time TIMESTAMP(3), payload STRING ) WITH ( 'connector' = 'iceberg', 'catalog-name' = 'iceberg_catalog', 'write.upsert.enabled' = 'true', 'write.target-file-size-bytes' = '134217728', -- 128 MB target for writes 'commit.manifest.target-size-bytes' = '8388608' -- 8 MB manifest target ); ``` Note that write-time configuration minimizes small files during ingestion but doesn't eliminate the need for periodic compaction as data patterns and partition distributions change over time. ### Governance and Visibility with Conduktor In organizations managing multiple Iceberg tables across streaming pipelines, visibility into the Kafka layer becomes critical. When Kafka streams feed Iceberg tables through Flink or Spark, **Conduktor** provides governance capabilities for the Kafka side of these pipelines: **Kafka-to-Iceberg Pipeline Monitoring:** - **End-to-end latency tracking**: Monitor time from Kafka ingestion through Iceberg commit using [Conduktor's topic monitoring](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics), identifying bottlenecks in streaming writes - **Consumer lag monitoring**: Track Flink/Spark consumer lag to detect when compaction jobs slow down streaming ingestion - **Data quality validation**: Enforce schema contracts and validation rules with [Schema Registry integration](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry) on Kafka messages before they reach Iceberg tables - **Throughput analysis**: Measure messages per second and file creation rates to optimize micro-batch sizes using [Kafka Connect monitoring](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) **Table Health Management:** - **Small file detection**: Alert when Iceberg partitions exceed thresholds (e.g., more than 100 files under 10MB) - **Snapshot growth monitoring**: Track snapshot accumulation rate and alert when retention policies may be insufficient - **Maintenance job observability**: Log all compaction, expiration, and cleanup operations with execution duration and files affected - **Cost tracking**: Correlate cloud storage costs with table file counts and maintenance schedules **Chaos Testing with Conduktor Gateway (a Kafka proxy):** - **Simulate Kafka broker failures**: Test how streaming-to-Iceberg pipelines handle broker outages and consumer rebalancing - **Inject latency**: Validate that compaction jobs don't interfere with time-sensitive streaming ingestion - **Test Kafka-side resilience**: Verify that Flink consumer groups recover correctly from Kafka failures and continue feeding Iceberg tables **Compliance and Auditing:** - **Consumer group monitoring**: Track which applications are consuming from the Kafka topics feeding your Iceberg tables - **Access auditing**: Log which users and applications query specific Iceberg snapshots, critical for GDPR and compliance - **Retention policy enforcement**: Automate snapshot expiration aligned with regulatory requirements This governance layer is essential when multiple teams manage different parts of the streaming-to-lakehouse pipeline, ensuring that developers can iterate quickly while maintaining production reliability. ## Branch-Specific Maintenance (Iceberg 1.5+) Iceberg 1.5+ introduced **branches and tags**, enabling Git-like version management for tables. Each branch maintains its own snapshot lineage and can have **independent retention policies**, making branches ideal for development, testing, and experimental workflows without affecting production data. ### Maintenance Operations on Branches Branches require separate maintenance from the main table, allowing teams to manage staging and production environments independently: ```sql -- Expire snapshots only on the experimental branch CALL catalog.system.expire_snapshots( table => 'db.events', branch => 'experimental', older_than => TIMESTAMP '2025-12-10 00:00:00', retain_last => 3 -- Keep fewer snapshots on experimental branches ); -- Compact data files on a specific branch CALL catalog.system.rewrite_data_files( table => 'db.events', branch => 'staging', strategy => 'binpack', options => map('target-file-size-bytes', '536870912') ); -- Remove orphan files specific to a branch's history CALL catalog.system.remove_orphan_files( table => 'db.events', branch => 'dev', older_than => TIMESTAMP '2025-12-14 00:00:00' ); ``` ### Branch Maintenance Patterns **Development branches** (short-lived, aggressive cleanup): - Expire snapshots older than 24 hours - Retain only last 5 snapshots - Run orphan cleanup after every merge to main **Staging branches** (moderate retention): - Expire snapshots older than 7 days - Retain last 20 snapshots for debugging - Weekly compaction aligned with test cycles **Production (main branch)** (long retention for compliance): - Expire snapshots older than 90 days (or compliance requirement) - Retain last 100 snapshots - Daily compaction during low-traffic windows ### Cleaning Up Merged Branches After fast-forwarding or merging a branch to main, the branch's exclusive snapshots become orphaned. Clean them up explicitly: ```sql -- Drop a branch after merging to main ALTER TABLE db.events DROP BRANCH IF EXISTS experimental; -- This marks branch-exclusive snapshots for expiration -- Run orphan cleanup to reclaim storage CALL catalog.system.remove_orphan_files( table => 'db.events', older_than => current_timestamp() - INTERVAL '3' DAY ); ``` ### Use Cases for Branch-Specific Maintenance - **Isolated testing**: Create a `test` branch, run experiments, compact aggressively, then drop the branch without affecting main - **Cost optimization**: Apply aggressive retention on ephemeral branches to minimize storage costs - **Compliance isolation**: Keep production snapshots for regulatory periods while cleaning up dev/test branches frequently - **Multi-tenant tables**: Different teams manage their own branches with customized maintenance schedules Branch-specific maintenance is essential for organizations adopting data-as-code workflows, where table branches mirror software development branch strategies. ## When to Perform Maintenance Knowing when tables need maintenance prevents both over-maintenance (wasting compute) and under-maintenance (degrading performance). Monitor these signals to trigger maintenance operations: ### Signals for Data File Compaction **Small File Indicators:** ```sql -- Check average file size and count SELECT COUNT(*) as file_count, AVG(file_size_in_bytes) / 1024 / 1024 as avg_size_mb, COUNT(CASE WHEN file_size_in_bytes < 10485760 THEN 1 END) as files_under_10mb FROM db.events.files WHERE file_size_in_bytes > 0; ``` **Trigger compaction when:** - More than 30% of files are under 10MB - Average file size drops below 100MB (for typical analytical workloads) - A single partition contains more than 100 files - Query planning time increases by more than 20% compared to baseline **Cost indicators:** - Cloud storage request costs spike (common with many small files) - Query execution time increases despite same data volume ### Signals for Snapshot Expiration **Snapshot Growth Indicators:** ```sql -- Count snapshots and check age SELECT COUNT(*) as snapshot_count, MIN(committed_at) as oldest_snapshot, MAX(committed_at) as newest_snapshot, DATEDIFF('day', MIN(committed_at), MAX(committed_at)) as retention_days FROM db.events.snapshots; ``` **Trigger expiration when:** - More than 500 snapshots exist (impacts metadata loading) - Oldest snapshot exceeds compliance retention requirements - Metadata directory size exceeds 500MB - Table loading time increases noticeably **Balance considerations:** - Compliance: Regulatory requirements may mandate minimum retention - Time travel dependencies: Downstream jobs may need historical snapshots - Debugging: Recent snapshots help troubleshoot data issues ### Signals for Manifest Compaction **Manifest Bloat Indicators:** ```sql -- Check manifest file count and sizes SELECT COUNT(*) as manifest_count, AVG(LENGTH) / 1024 as avg_size_kb, COUNT(CASE WHEN added_data_files_count = 1 THEN 1 END) as single_file_manifests FROM db.events.manifests WHERE added_snapshot_id = (SELECT MAX(snapshot_id) FROM db.events.snapshots); ``` **Trigger manifest compaction when:** - More than 100 manifest files exist in current snapshot - Over 50% of manifests track only 1-2 data files - Query planning time exceeds 5 seconds consistently - After major bulk operations (imports, schema changes) ### Signals for Orphan File Cleanup **Orphan Indicators:** ```sql -- Compare storage size to tracked files (run via cloud CLI) -- AWS example: aws s3 ls s3://bucket/warehouse/db/events/data/ --recursive --summarize -- Compare to Iceberg's tracked size SELECT SUM(file_size_in_bytes) / 1024 / 1024 / 1024 as tracked_size_gb FROM db.events.files; ``` **Trigger orphan cleanup when:** - Storage size exceeds tracked file size by more than 10% - After failed write operations or job cancellations - Weekly or monthly as a preventive measure - Before major cost optimization reviews ### Monitoring Dashboard Metrics Set up monitoring dashboards tracking: - **File count trends**: Growing file count suggests compaction needed - **Average file size trends**: Declining size indicates small file accumulation - **Query planning time**: Increasing duration signals metadata bloat - **Snapshot count**: Unbounded growth requires expiration - **Storage costs**: Spikes correlate with maintenance gaps **Recommended alert thresholds:** - File count growth rate: >10% per day for 3 consecutive days - Average file size: <50MB for tables with >1000 files - Query planning time: >3 seconds for simple SELECT COUNT(*) queries - Snapshot count: >300 for frequently updated tables ## Maintenance Automation and Scheduling Production Iceberg deployments require automated maintenance schedules to prevent degradation. ### Airflow DAG Example ```python from airflow import DAG from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator from datetime import datetime, timedelta default_args = { 'owner': 'data-platform', 'depends_on_past': False, 'start_date': datetime(2025, 12, 1), 'retries': 2, 'retry_delay': timedelta(minutes=5) } dag = DAG( 'iceberg_maintenance', default_args=default_args, schedule_interval='0 2 * * *', # Daily at 2 AM catchup=False ) compact_task = SparkSubmitOperator( task_id='compact_events_table', application='/path/to/compact.py', conf={ 'spark.sql.catalog.iceberg': 'org.apache.iceberg.spark.SparkCatalog', 'spark.sql.catalog.iceberg.type': 'hive' }, dag=dag ) expire_task = SparkSubmitOperator( task_id='expire_snapshots', application='/path/to/expire.py', dag=dag ) orphan_task = SparkSubmitOperator( task_id='remove_orphans', application='/path/to/orphan_cleanup.py', dag=dag ) compact_task >> expire_task >> orphan_task ``` ### Maintenance Sequence Always perform maintenance operations in this order: 1. **Compaction**: Consolidate small files first 2. **Manifest compaction**: Consolidate manifest files after data compaction 3. **Snapshot expiration**: Remove old snapshots that reference old small files 4. **Orphan cleanup**: Delete unreferenced files after snapshots are expired This sequence ensures maximum storage reclamation while maintaining data integrity. ### Operational Considerations **Duration and Resource Planning:** Maintenance operations consume significant compute and I/O resources. Plan accordingly: - **Compaction duration**: Typically 1-2 minutes per GB of data being rewritten. A 500GB partition may take 8-16 hours depending on cluster size and parallelism. - **Snapshot expiration**: Fast metadata-only operation, usually completes in seconds to minutes regardless of table size. - **Manifest compaction**: Quick metadata operation, typically under 5 minutes even for large tables. - **Orphan cleanup**: I/O intensive, requires listing all files in table storage. Can take hours for tables with millions of files. **Compute Costs:** - Compaction reads and rewrites data, costing 2x I/O (read + write) plus compute time - Use autoscaling clusters or spot instances for cost-effective maintenance - Schedule during off-peak hours to leverage lower cloud pricing - Consider dedicated maintenance clusters to avoid resource contention with production queries **Query Availability During Maintenance:** Iceberg's MVCC (Multi-Version Concurrency Control) architecture allows queries to continue during maintenance: - **Read queries**: Continue unaffected, reading existing snapshots while maintenance creates new ones - **Write queries**: May experience brief contention during snapshot commits but remain available - **Time travel**: Historical snapshots remain queryable until explicitly expired - **Zero downtime**: No need for maintenance windows or table locks **Rollback and Recovery:** If maintenance fails or produces unexpected results: ```sql -- Rollback to a previous snapshot if needed CALL catalog.system.rollback_to_snapshot('db.events', ); -- Check recent snapshots to find the right restore point SELECT snapshot_id, committed_at, operation, summary FROM db.events.snapshots ORDER BY committed_at DESC LIMIT 10; ``` **Best Practices:** - Run dry-run mode first for orphan cleanup - Test maintenance procedures on non-production tables - Monitor job progress and set up alerts for failures - Document baseline metrics (file counts, query times) before maintenance - Keep at least 2-3 snapshots before major maintenance operations for easy rollback ## Summary Maintaining Iceberg tables through compaction, snapshot expiration, manifest compaction, and orphan file cleanup is essential for production deployments. **Data file compaction** addresses the small file problem through bin-packing or sort-based strategies, with Iceberg 1.6+ introducing fault-tolerant partial progress mode for long-running jobs. **Manifest compaction** prevents query planning slowdowns by consolidating metadata files. **Snapshot expiration** prevents unbounded metadata growth while respecting time-travel requirements and compliance needs. **Orphan file removal** reclaims wasted storage from failed writes and concurrent operations. Iceberg 1.5+ **branch-specific maintenance** enables independent retention policies for development, staging, and production environments, aligning data maintenance with software development workflows. Branches allow aggressive cleanup on experimental tables while maintaining long retention for compliance on production data. In streaming environments, maintenance becomes more critical as continuous writes amplify these challenges. Modern integrations like **Flink 1.18+ Actions API** enable programmatic maintenance alongside streaming ingestion. Platforms like **Conduktor** provide governance for the Kafka side of Kafka-to-Iceberg pipelines, monitoring consumer lag, enforcing data quality and schema contracts on Kafka topics, and enabling chaos testing on the Kafka layer to validate pipeline resilience. **When to perform maintenance** is as important as how, monitor file counts, average file sizes, query planning times, and snapshot growth to trigger maintenance proactively. Set up dashboards tracking these metrics and automate maintenance through orchestration platforms like Airflow, triggering operations based on concrete thresholds rather than arbitrary schedules. Understanding **operational considerations**, duration, compute costs, and query availability, ensures maintenance operations run efficiently without disrupting production workloads. Iceberg's MVCC architecture enables zero-downtime maintenance, allowing queries to continue uninterrupted while compaction and cleanup proceed in the background. Regular maintenance, driven by concrete thresholds rather than arbitrary schedules, keeps Iceberg tables performant and cost-efficient at scale. ## Related Concepts - [Data Pipeline Orchestration with Streaming](https://www.conduktor.io/data-pipeline-orchestration-with-streaming) - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) - [Real-Time Analytics with Streaming Data](https://www.conduktor.io/real-time-analytics-with-streaming-data) ## Related Articles - [Iceberg Table Architecture: Metadata and Snapshots](https://www.conduktor.io/glossary/iceberg-table-architecture-metadata-and-snapshots) - Understanding Iceberg's internal architecture - [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg) - Comprehensive overview of Iceberg features - [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture) - Lakehouse fundamentals and ecosystem - [Time Travel with Apache Iceberg](https://www.conduktor.io/glossary/time-travel-with-apache-iceberg) - Advanced time travel and snapshot management - [Schema Evolution in Apache Iceberg](https://www.conduktor.io/glossary/schema-evolution-in-apache-iceberg) - Managing schema changes safely - [Iceberg Partitioning and Performance Optimization](https://www.conduktor.io/glossary/iceberg-partitioning-and-performance-optimization) - Partitioning strategies ## Sources and References - [Apache Iceberg Documentation: Maintenance](https://iceberg.apache.org/docs/latest/maintenance/) - [Apache Iceberg: Spark Procedures](https://iceberg.apache.org/docs/latest/spark-procedures/) - [Apache Iceberg 1.6.0 Release Notes: Partial Progress Support](https://iceberg.apache.org/releases/#160-release) - [Apache Iceberg 1.5.0 Release Notes: Branches and Tags](https://iceberg.apache.org/releases/#150-release) - [Apache Flink 1.18 Iceberg Connector Documentation](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/iceberg/) - [Netflix Tech Blog: Managing Apache Iceberg Tables](https://netflixtechblog.com/apache-iceberg-managing-data-files-at-scale-7465e2f5d9e9) - [Tabular: Iceberg Table Maintenance Best Practices](https://tabular.io/blog/table-maintenance/) - [Apache Iceberg GitHub: Maintenance Actions](https://github.com/apache/iceberg/tree/master/spark/v3.3/spark/src/main/java/org/apache/iceberg/actions) Source: https://www.conduktor.io/glossary/maintaining-iceberg-tables-compaction-and-cleanup --- # Message Serialization in Kafka Serialization is the process of converting data structures or objects into a byte format that can be transmitted over a network or stored on disk. Deserialization is the reverse process, reconstructing the original data structure from bytes. In [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka), every message is stored and transmitted as an array of bytes. This means that before a [producer](https://www.conduktor.io/glossary/kafka-producers) sends a message to Kafka, it must serialize the data. Similarly, when a consumer reads a message, it must deserialize those bytes back into a meaningful data structure. ## What is Serialization and Why Does It Matter? The choice of serialization format affects multiple critical aspects of your streaming architecture: message size, processing speed, schema evolution capabilities, interoperability between systems, and developer productivity. ![Message serialization flow in Kafka](https://www.conduktor.io/assets/images/glossary/message-serialization-in-kafka-0.webp) ## How Kafka Uses Serialization Kafka itself is agnostic to the content of your messages. It treats both keys and values as byte arrays. The responsibility for serialization and deserialization falls on producers and consumers through serializers and deserializers (often called "serdes"). When configuring a Kafka producer, you specify two serializers (often called "serdes", a shorthand for serializers/deserializers): ```java properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); properties.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer"); // For Avro with Schema Registry, use schema-aware serializers: // properties.put("value.serializer", "io.apicurio.registry.serde.avro.AvroKafkaSerializer"); ``` Similarly, consumers specify deserializers: ```java properties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); properties.put("value.deserializer", "org.apache.kafka.common.serialization.ByteArrayDeserializer"); // For Avro with Schema Registry: // properties.put("value.deserializer", "io.apicurio.registry.serde.avro.AvroKafkaDeserializer"); ``` This decoupling allows different applications to use different serialization formats, though consistency across producers and consumers of the same topic is essential to avoid deserialization failures. ## Common Serialization Formats Here's a quick comparison to help you choose the right format: | Feature | String/Primitives | JSON | Avro | Protobuf | |---------|------------------|------|------|----------| | **Message Size** | Small | Large (2-3x binary) | Compact | Most compact | | **Human Readable** | Yes | Yes | No | No | | **Schema Evolution** | None | Limited | Excellent | Excellent | | **Serialization Speed** | Fastest | Slow | Fast | Fastest | | **Schema Registry** | Not needed | Optional | Required | Required | | **Code Generation** | No | No | Optional | Yes | | **Best For** | Simple values | APIs, debugging | Data pipelines | Microservices | | **2025 Compression** | N/A | Zstd | Zstd | Zstd | ### String and Primitive Types The simplest approach is using Kafka's built-in serializers for strings, integers, and other primitive types. This works well for simple use cases but lacks structure and schema enforcement. **Use case**: Log messages, simple metrics, prototype development. ### JSON JSON is human-readable, widely supported, and familiar to most developers. However, it comes with significant drawbacks: larger message sizes due to field name repetition, no built-in schema enforcement, and slower parsing compared to binary formats. **Use case**: External APIs, when human readability during debugging is prioritized, initial development phases. ### Apache Avro Avro is a binary serialization format that separates schema from data. Messages contain only the values, making them compact. Schemas are typically stored in a [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management), and messages reference the schema by ID. Avro supports rich schema evolution rules, allowing you to add fields with defaults, remove optional fields, and maintain backward and forward compatibility. This makes it ideal for long-lived streaming applications where schemas evolve over time. For detailed comparison with other formats, see [Avro vs Protobuf vs JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema). **Use case**: High-throughput data pipelines, analytics platforms, event-driven architectures requiring schema evolution. ### Protocol Buffers (Protobuf) Protobuf is Google's binary serialization format. It offers excellent performance and compact message sizes. Like Avro, it requires schema definition files (.proto files) and supports schema evolution, though with slightly different compatibility rules. **Use case**: Microservices architectures, polyglot environments, systems requiring cross-language compatibility. ## Schema Evolution and Compatibility One of the most challenging aspects of distributed systems is managing schema changes over time. As your business evolves, you'll need to add new fields, deprecate old ones, or change data types. Consider this real-world scenario: You have a `CustomerOrder` event with fields `orderId`, `customerId`, and `amount`. Six months later, you need to add a `discount` field. With Avro, you can add this field with a default value (e.g., 0.0). Old consumers that don't know about the discount field will continue working. New consumers can read both old messages (using the default) and new messages (with the actual discount value). This is **backward compatibility**. **Forward compatibility** means old consumers can read messages produced by new producers, even if those messages have fields the consumer doesn't know about. The [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management) enforces compatibility rules to prevent breaking changes. A Schema Registry typically supports four compatibility modes: - **BACKWARD**: New schema can read old data - **FORWARD**: Old schema can read new data - **FULL**: Both backward and forward compatible - **NONE**: No compatibility checking For comprehensive guidance on managing schema changes safely, see [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). Without proper schema management, you risk runtime errors, data loss, or forcing synchronized deployments across all producers and consumers, a significant operational burden. ## Performance and Operational Considerations Different serialization formats have distinct performance characteristics: - **Message Size**: Binary formats like Avro and Protobuf are significantly smaller than JSON. In a high-throughput system processing millions of messages per second, this translates to reduced network bandwidth and storage costs. A JSON message might be 2-3x larger than its Avro equivalent. - **Compression (2025 Recommendations)**: Modern Kafka deployments should use **Zstd compression** (available since Kafka 2.1+), which provides 20-30% better compression ratios than gzip with similar CPU usage. Combined with binary serialization formats, Zstd offers optimal throughput-to-storage tradeoffs. Configure compression at the producer level with `compression.type=zstd`. - **Serialization/Deserialization Speed**: Binary formats are generally faster to serialize and deserialize because they avoid text parsing. However, the difference may be negligible compared to network I/O in many applications. In benchmarks, Protobuf typically offers the fastest serialization, followed by Avro, with JSON being 3-5x slower. - **CPU Usage**: JSON parsing is CPU-intensive compared to binary formats, which can impact consumer lag during peak loads. For high-throughput applications (>100k messages/sec per consumer), binary formats significantly reduce CPU overhead. - **Developer Experience**: JSON is easier to debug because you can read messages directly. Binary formats require tools to decode messages. Governance platforms like **Conduktor** provide message viewers that automatically deserialize and display messages in human-readable formats regardless of the underlying serialization, bridging the gap between binary efficiency and developer productivity. For managing schemas in Conduktor, see [Schema Registry Management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry). ## Serialization in Data Streaming Ecosystems Message serialization is foundational to the broader data streaming ecosystem. Apache Kafka acts as the central nervous system, but data flows through multiple systems, stream processing frameworks like Apache Flink or Kafka Streams, data warehouses, analytics platforms, and microservices. Choosing a well-supported serialization format ensures interoperability across these systems. Avro's tight integration with [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management) makes it a de facto standard in Kafka-based ecosystems. Protobuf's wide language support makes it popular in polyglot microservices architectures. Stream processing frameworks like Flink support multiple serialization formats and can automatically handle schema evolution when integrated with a Schema Registry. This enables complex transformations while maintaining data consistency. When operating Kafka clusters at scale, managing schemas becomes critical. Governance platforms help teams visualize schema evolution, validate compatibility before deployment, and troubleshoot deserialization errors by providing clear visibility into message formats and schema versions across topics. For example, if a consumer starts failing due to a deserialization error, you can quickly inspect the Schema Registry to identify which schema version changed, compare schemas side-by-side, and determine whether the issue is a compatibility violation or a configuration problem. ## Troubleshooting Serialization Issues Serialization problems are among the most common issues in Kafka deployments. Here are practical approaches to common scenarios: ### Deserialization Failures - **Symptom**: Consumers throw exceptions like `SerializationException` or `InvalidAvroSchemaException`. - **Common Causes**: - Schema version mismatch between producer and consumer - Missing schema in Schema Registry - Incorrect deserializer configuration - Corrupted or malformed messages **Resolution Steps**: 1. Check consumer logs for the specific error and schema ID 2. Verify the schema exists in Schema Registry: `curl http://registry:8081/schemas/ids/{id}` 3. Confirm producer and consumer use compatible serializer/deserializer classes 4. Inspect the problematic message offset using CLI tools or Conduktor's message viewer 5. Validate Schema Registry connectivity from consumer hosts ### Schema Evolution Errors - **Symptom**: Schema Registry rejects new schema versions with compatibility errors. - **Common Causes**: - Breaking changes (removing required fields, changing types) - Incorrect compatibility mode for use case - Missing default values on new fields **Resolution Steps**: 1. Review compatibility mode: `curl http://registry:8081/config/{subject}` 2. Use schema validation tools to test compatibility before deployment 3. Add default values to new required fields 4. Consider using FULL compatibility mode if producers and consumers upgrade independently 5. For breaking changes, create a new topic and migrate consumers gradually ### Performance Degradation - **Symptom**: High consumer lag, increased CPU usage, or slow message processing. - **Common Causes**: - Inefficient serialization format (JSON at high volumes) - Missing compression - Large message payloads - Schema Registry connection issues **Resolution Steps**: 1. Profile serialization/deserialization time in your application 2. Enable Zstd compression if not already configured 3. Consider switching from JSON to Avro or Protobuf for high-throughput topics 4. Implement Schema Registry client-side caching to reduce registry calls 5. For very large messages (>1MB), consider storing payload in object storage and sending references ### Custom Serializers For specialized requirements, you can implement custom serializers by implementing Kafka's `Serializer` and `Deserializer` interfaces. Common use cases include: - **Encryption/Decryption**: Transparent field-level encryption during serialization - **Compression**: Application-level compression before Kafka's transport compression - **Multi-format support**: Detect format based on magic bytes and route to appropriate deserializer - **Validation**: Enforce business rules during serialization **Example custom serializer skeleton**: ```java public class CustomSerializer implements Serializer { @Override public void configure(Map configs, boolean isKey) { // Initialize with configuration } @Override public byte[] serialize(String topic, MyObject data) { // Implement serialization logic // Example: encrypt sensitive fields, add headers, etc. return serializedBytes; } @Override public void close() { // Cleanup resources } } ``` For complex serialization needs, consider using **Conduktor Gateway**, a Kafka proxy that can intercept messages and apply transformations, encryption, or validation rules without modifying application code. ## Serialization and Kafka Headers Kafka message headers provide metadata about messages without affecting the serialized payload. Headers are commonly used with serialization for: - **Schema Version Tracking**: Schema-aware serializers automatically embed schema IDs in message headers, allowing consumers to retrieve the correct schema for deserialization without parsing the payload. - **Content Type Identification**: When topics contain messages with multiple serialization formats, a `content-type` header can indicate the format (e.g., `application/avro`, `application/protobuf`), enabling polyglot consumers to route to the appropriate deserializer. - **Tracing and Lineage**: Headers can track serialization metadata like producer application ID, serialization timestamp, and data lineage information for debugging and governance. - **Encryption Metadata**: For encrypted payloads, headers can store encryption algorithm details, key IDs, or initialization vectors needed for decryption. - **Example with headers**: ```java ProducerRecord record = new ProducerRecord<>("orders", key, serializedValue); record.headers().add("content-type", "application/avro".getBytes()); record.headers().add("schema-version", "v2.1.0".getBytes()); record.headers().add("trace-id", traceId.getBytes()); producer.send(record); ``` For comprehensive guidance on leveraging headers effectively, see [Using Kafka Headers Effectively](https://www.conduktor.io/glossary/using-kafka-headers-effectively). ## Summary Message serialization in Kafka is a foundational decision that affects performance, operational complexity, and system evolution over time. **Key Takeaways**: - Binary formats like Avro and Protobuf offer the best combination of performance, schema evolution capabilities, and ecosystem support for production streaming applications - JSON remains useful for development and debugging but comes with trade-offs in production (2-3x larger messages, slower parsing) - [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management) is essential for managing schema evolution and enforcing compatibility rules - [Schema evolution best practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) prevent runtime errors and enable independent evolution of producers and consumers **2025 Recommendations**: - Use **Zstd compression** (`compression.type=zstd`) for optimal throughput-to-storage ratio - Leverage **Conduktor** for unified message debugging, schema validation, and governance across formats - Implement custom serializers for encryption, validation, or multi-format support when needed - Use Kafka headers for metadata like schema versions, content types, and tracing information **Choosing Your Format**: - **Avro**: High-throughput data pipelines with frequent schema changes ([comparison details](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema)) - **Protobuf**: High-performance microservices requiring strong typing and cross-language support - **JSON**: APIs, debugging, initial development (use JSON Schema for validation) - **Primitives**: Simple values, metrics, logs When choosing a serialization format, consider your specific requirements: throughput needs, schema evolution frequency, team expertise, and ecosystem constraints. The right choice balances technical performance with operational maintainability. Start with Avro for data pipelines or Protobuf for services, and use governance platforms to manage the complexity as your system scales. ## Related Concepts - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Essential for managing serialization schemas centrally and enforcing compatibility rules across producers and consumers. - [Avro vs Protobuf vs JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema) - Detailed comparison of the three major serialization formats to help choose the right one for your use case. - [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) - Learn how to evolve serialization schemas safely over time while maintaining backward and forward compatibility. ## Sources and References 1. **Apache Kafka Documentation - Serialization**: Official Kafka documentation covering serializers and deserializers ([kafka.apache.org](https://kafka.apache.org/documentation/#serialization)) 2. **Apache Avro Specification**: Official specification and documentation for the Avro serialization format, including schema evolution rules ([avro.apache.org](https://avro.apache.org/docs/)) 3. **Protocol Buffers Documentation**: Google's official documentation for Protobuf, including language guides and encoding details ([protobuf.dev](https://protobuf.dev/)) 4. **Apicurio Registry Documentation**: Open-source schema registry supporting Avro, Protobuf, and JSON Schema with comprehensive compatibility mode documentation ([apicurio.io](https://www.apicur.io/registry/docs/apicurio-registry/)) 5. **"Kafka: The Definitive Guide" by Neha Narkhede, Gwen Shapira, and Todd Palino**: O'Reilly book covering Kafka architecture, including detailed chapters on serialization and schema management 6. **Martin Kleppmann - "Designing Data-Intensive Applications"**: Chapter 4 (Encoding and Evolution) provides deep technical analysis of serialization formats and schema evolution patterns Source: https://www.conduktor.io/glossary/message-serialization-in-kafka --- # Metadata Management: Technical vs Business Metadata In modern data architectures, metadata underpins how teams understand, govern, and operationalize data assets. Yet many organizations struggle to distinguish between technical and business metadata, leading to fragmented governance strategies and missed opportunities. This article explores the differences between these metadata types and why both matter for Data Architects and Data Governance Officers. ![Technical vs business metadata layers](https://www.conduktor.io/assets/images/glossary/metadata-management-technical-vs-business-metadata-0.webp) ## Understanding the Metadata Landscape Metadata, data about data, exists in multiple dimensions across your data ecosystem. While all metadata describes characteristics of your data assets, the distinction between technical and business metadata reflects fundamentally different perspectives on the same underlying information. - **Technical metadata** captures the structural, operational, and system-level characteristics of data assets. It answers questions like: What is the schema? How is data stored? What transformations occur? Where does data flow? This metadata is typically generated automatically by systems and tools, reflecting the mechanical reality of your data infrastructure. - **Business metadata** describes the semantic meaning, context, and organizational relevance of data. It addresses questions such as: What does this data represent? Who owns it? What business processes does it support? How should it be interpreted? This metadata connects technical implementation to business understanding, making data accessible to non-technical stakeholders. ### A Concrete Example: Customer Revenue Field Consider a simple field to illustrate both metadata types working together: **Technical Metadata:** - Field name: `customer_revenue_mtd` - Data type: `DECIMAL(10,2)` - Kafka topic: `finance.revenue.events` - Schema version: `v3.2` - Partition key: `customer_id` - Compression: `snappy` - Update frequency: Every 5 minutes **Business Metadata:** - Display name: "Customer Revenue Month-to-Date" - Definition: "Cumulative revenue generated by a customer account from the first day of the current calendar month through the current date" - Owner: Finance Data Team - Data steward: jane.smith@company.com - Business criticality: High (feeds executive dashboards) - PII classification: Non-sensitive (aggregated revenue only) - Related business terms: Revenue Recognition, Monthly Recurring Revenue (MRR) This example shows how technical metadata enables engineers to integrate and process the data, while business metadata ensures analysts and business users understand its meaning and proper use. ### Metadata Types Comparison | Aspect | Technical Metadata | Business Metadata | Operational Metadata | |--------|-------------------|-------------------|---------------------| | **Primary Audience** | Data Engineers, Platform Teams | Data Analysts, Business Users | SREs, Data Ops Teams | | **Generation** | Automated (system-generated) | Manual (human-curated) | Automated (usage tracking) | | **Content Focus** | Structure, format, location | Meaning, ownership, policies | Usage patterns, performance | | **Update Frequency** | Real-time (as systems change) | Periodic (as understanding evolves) | Continuous (runtime metrics) | | **Examples** | Schema, data types, partitions | Definitions, data stewards, tags | Query counts, access patterns | | **Primary Value** | Enables integration & processing | Enables understanding & governance | Enables optimization & monitoring | ## Technical Metadata: The Infrastructure Foundation Technical metadata forms the backbone of data operations and engineering workflows. Data Architects rely on this metadata to design scalable systems, optimize performance, and ensure data quality. ### Core Components - **Schema metadata** defines data structures: column names, data types, constraints, indexes, and relationships. In streaming platforms like Apache Kafka, schema metadata managed through Schema Registry (a centralized service for storing and validating schemas) becomes crucial for ensuring producer-consumer compatibility and enabling schema evolution without breaking downstream applications. - **Lineage metadata** traces data movement across systems, capturing source-to-target mappings, transformation logic, and dependency chains. Understanding that a Kafka topic receives data from three upstream services, undergoes enrichment via a stream processor (a component that processes events in real-time), and feeds into five downstream consumers provides critical insight for impact analysis and troubleshooting. For comprehensive coverage of data lineage implementation, see [Data Lineage: Tracking Data from Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption). - **Operational metadata** tracks runtime characteristics: execution times, data volumes, error rates, resource consumption, and performance metrics. For streaming architectures, this includes lag monitoring (tracking how far behind consumers are from the latest messages), throughput measurements, and partition distribution (how data is distributed across Kafka partitions), metrics that directly impact system reliability. ### Streaming Integration Considerations In streaming architectures, technical metadata takes on additional complexity. Unlike batch systems with discrete execution boundaries, streaming platforms generate continuous metadata streams. Modern governance tools like Conduktor provide comprehensive metadata visibility for Kafka environments, exposing cluster configurations, topic schemas, consumer group positions, and real-time performance metrics in a unified interface. Open-source metadata platforms like DataHub and OpenMetadata also offer robust Kafka integration for metadata discovery and lineage tracking. This visibility becomes critical when debugging issues like consumer lag (when message processing falls behind) or partition skew (uneven data distribution across partitions). Technical metadata reveals that a consumer group is falling behind on specific partitions, while operational metadata shows which transformations are consuming excessive resources, enabling targeted optimization. ## Business Metadata: The Semantic Layer While technical metadata describes how systems operate, business metadata explains what data means and why it matters. For Data Governance Officers, business metadata transforms data catalogs from technical inventories into strategic assets. ### Essential Elements - **Semantic definitions** provide clear, standardized descriptions of data elements. A field named `cust_rev_mtd` might be technically described as `DECIMAL(10,2)`, but business metadata explains it as "Customer Revenue Month-to-Date: The cumulative revenue generated by a customer account from the first day of the current calendar month through the current date, measured in USD." - **Ownership and stewardship** metadata identifies who is responsible for data quality, who can authorize access, and who should be consulted about interpretation. In distributed architectures where data flows across team boundaries, clear ownership prevents ambiguity and establishes accountability. - **Business rules and policies** document constraints, calculations, and governance requirements. These might include data retention policies, masking requirements for PII (Personally Identifiable Information), or business logic for calculating derived metrics. For streaming data, this could specify how late-arriving events (messages that arrive after their expected processing window) should be handled or when windowed aggregations (calculations over time periods) should be materialized. - **Usage context** captures how data supports business processes, which reports or dashboards consume it, and what decisions it informs. This metadata helps prioritize data quality efforts and assess the impact of schema changes. ## The Operational Metadata Bridge A third category, operational metadata, sits at the intersection of technical and business concerns. It captures how data systems are actually being used: query patterns, access frequencies, user interactions, and data consumption trends. For streaming platforms, operational metadata might reveal that a particular Kafka topic thought to be deprecated still has active consumers, or that a specific data field is queried thousands of times daily despite being marked as low-priority in business metadata. These insights drive informed decisions about resource allocation and deprecation strategies. ## Modern Metadata Technologies (2025) The metadata management landscape has evolved significantly with new standards, AI-powered tools, and automated approaches that bridge technical and business perspectives more effectively than ever. ### OpenLineage: The Lineage Standard OpenLineage has emerged as the industry standard for data lineage interoperability. Rather than each tool maintaining its own lineage format, OpenLineage provides a unified specification for capturing and exchanging lineage metadata across heterogeneous systems. With OpenLineage, you can track data flow from Kafka topics through Flink transformations to data warehouse tables using a consistent metadata format. Modern orchestrators (Airflow 2.7+, Dagster 1.5+) emit OpenLineage events natively, enabling end-to-end lineage visualization across your entire data platform without vendor lock-in. ### AI-Powered Metadata Discovery Large language models are transforming metadata generation from manual documentation to intelligent automation. Modern metadata platforms now offer: - **Automated Classification**: LLMs scan column names, sample data, and usage patterns to automatically classify data as PII, financial, operational, or other sensitive categories, dramatically reducing the manual tagging burden for data governance teams. - **Semantic Description Generation**: AI analyzes table and column names, data distributions, and relationships to generate human-readable descriptions. A field named `txn_amt_usd` with values ranging from 0-50000 might be automatically described as "Transaction amount in US dollars, typically representing purchase or payment values." - **Smart Tagging and Relationships**: Machine learning identifies implicit relationships between datasets based on join patterns, shared keys, and usage co-occurrence, surfacing connections that manual documentation might miss. ### Active Metadata: From Documentation to Automation Active metadata represents a paradigm shift from passive documentation to actionable intelligence. Rather than simply describing data, active metadata triggers workflows and enforces policies in real-time. - **Metadata-Driven Data Quality**: Tools like Soda Core read metadata definitions to automatically generate and execute data quality checks. When business metadata specifies that a revenue field "must always be positive," the system generates validation SQL without manual rule coding. - **Dynamic Access Control**: Data classification metadata (PII, PHI, confidential) automatically triggers masking rules or access restrictions. Tag a Kafka topic field as containing email addresses, and governance policies immediately apply appropriate encryption or tokenization. - **Reverse ETL Automation**: Modern platforms use metadata about data freshness and dependencies to automatically trigger downstream syncs when upstream data updates complete, eliminating manual orchestration overhead. ### Column-Level Lineage While table-level lineage shows that `customer_revenue_summary` derives from `raw_transactions`, column-level lineage reveals that the `total_revenue` field specifically comes from aggregating `raw_transactions.amount` where `transaction_type = 'purchase'`. This granularity becomes critical for: - **Impact Analysis**: Understanding exactly which reports break when changing a specific field - **Compliance Auditing**: Proving that sensitive PII fields are properly masked or excluded from analytics tables - **Optimization**: Identifying unused columns that can be safely dropped to reduce storage costs Modern metadata platforms (DataHub 0.12+, OpenMetadata 1.4+) support field-level lineage through SQL parsing, data transformation analysis, and API instrumentation. ### Data Contracts as Metadata Bridges Data contracts formalize the interface between data producers and consumers, combining technical schemas with business guarantees. A data contract for a customer events stream might specify: - **Technical Schema**: Avro schema with required fields and data types - **Business Semantics**: Field definitions, valid value ranges, update frequency - **Quality Guarantees**: Maximum null rates, freshness SLAs, uniqueness constraints - **Lifecycle Policies**: Retention periods, deprecation warnings, breaking change notifications Tools like dbt contracts and Great Expectations enable contracts to be defined in code, tested automatically, and published to data catalogs as integrated metadata, creating a single source of truth that serves both engineering and business needs. For detailed guidance on implementing business terminology and definitions, see [Building a Business Glossary for Data Governance](https://www.conduktor.io/glossary/building-a-business-glossary-for-data-governance). ### Modern Metadata Platform Ecosystem **Open-Source Platforms:** - **DataHub** (LinkedIn/Acryl): Extensible metadata platform with strong Kafka and streaming support, GraphQL API, and rich lineage visualization - **OpenMetadata**: Modern UI, AI-powered discovery, comprehensive connector ecosystem for batch and streaming sources - **Apache Amundsen**: Pioneered metadata-as-a-product approach, strong on data discovery and user collaboration **Specialized Tools:** - **dbt**: Transforms analytics code into rich business metadata through documentation, tests, and lineage graphs - **Soda Core**: Data quality platform that generates metadata about data health, anomalies, and test results - **Conduktor**: Commercial Kafka governance platform providing comprehensive metadata visibility, schema management, and access control for Kafka environments **Semantic Layer Tools:** - **dbt Semantic Layer**: Defines business metrics as metadata, ensuring consistent definitions across BI tools - **Cube**: Headless BI platform using metadata to generate optimized queries and maintain metric consistency ## Implementing Integrated Metadata Management Effective metadata management requires treating technical and business metadata as complementary rather than separate concerns. ### Establishing Bidirectional Linkage Every technical asset should connect to corresponding business metadata. A Kafka topic schema should link to business glossary terms, ownership information, and usage policies. Conversely, business metadata entries should reference their technical implementations, knowing which tables, topics, or streams contain specific business entities. ### Automating Technical Metadata Capture Leverage tools that automatically extract and maintain technical metadata. Schema registries, data catalogs, and platform-specific governance tools continuously harvest metadata from running systems, ensuring accuracy and reducing manual overhead. For streaming architectures, automated metadata capture includes tracking schema versions, monitoring consumer group configurations, and recording partition assignments, information that changes frequently and must remain current. **Example: Programmatic Metadata Enrichment with DataHub** Here's how to automatically enrich a Kafka topic with business metadata using DataHub's Python SDK: ```python from datahub.emitter.mce_builder import make_dataset_urn from datahub.emitter.rest_emitter import DatahubRestEmitter from datahub.metadata.schema_classes import ( DatasetPropertiesClass, OwnershipClass, OwnerClass, OwnershipTypeClass, GlobalTagsClass, TagAssociationClass, ) # Initialize connection to DataHub emitter = DatahubRestEmitter("http://datahub:8080") # Create dataset URN for Kafka topic dataset_urn = make_dataset_urn( platform="kafka", name="production.customer.revenue.events", env="PROD" ) # Add business metadata: ownership ownership = OwnershipClass( owners=[ OwnerClass( owner="urn:li:corpuser:jane.smith", type=OwnershipTypeClass.DATAOWNER, ) ] ) # Add business metadata: description and custom properties properties = DatasetPropertiesClass( description="Real-time customer revenue events emitted when " "transactions are processed. Used for executive " "dashboards and revenue forecasting.", customProperties={ "business_criticality": "HIGH", "data_classification": "INTERNAL", "update_frequency": "Real-time (5 min SLA)", "retention_period": "90 days", } ) # Add business metadata: tags for classification tags = GlobalTagsClass( tags=[ TagAssociationClass(tag="urn:li:tag:PII"), TagAssociationClass(tag="urn:li:tag:Financial"), TagAssociationClass(tag="urn:li:tag:Revenue"), ] ) # Emit metadata to DataHub emitter.emit_mcp( MetadataChangeProposalWrapper( entityUrn=dataset_urn, aspect=ownership, ) ) emitter.emit_mcp( MetadataChangeProposalWrapper( entityUrn=dataset_urn, aspect=properties, ) ) emitter.emit_mcp( MetadataChangeProposalWrapper( entityUrn=dataset_urn, aspect=tags, ) ) print(f"Enriched metadata for {dataset_urn}") ``` This approach enables CI/CD pipelines to automatically publish metadata when new Kafka topics are created, ensuring documentation stays synchronized with infrastructure. ### Crowdsourcing Business Metadata Unlike technical metadata, business metadata often requires human input. Implement workflows that encourage data producers and consumers to document definitions, add tags, and identify sensitive data. Treat metadata enrichment as part of the data development lifecycle rather than an afterthought. ### Creating Unified Discovery Experiences Data catalogs should present both metadata types in context. When a data analyst searches for "customer revenue," they should see technical details (location, schema, refresh frequency) alongside business context (definition, owner, certified status) in a single view. This integration accelerates understanding and reduces time-to-insight. For guidance on building discoverable data platforms, see [What is a Data Catalog? Modern Data Discovery](https://www.conduktor.io/glossary/what-is-a-data-catalog-modern-data-discovery). ## Governance Implications The distinction between technical and business metadata has profound governance implications. Technical metadata enables operational controls, access policies, data quality checks, lineage tracking. Business metadata enables strategic governance, standardized terminology, clear ownership, compliant data handling. Effective data governance requires both. Technical metadata without business context creates compliance blind spots, you might track data lineage perfectly but not recognize which flows contain regulated PII. Business metadata without technical grounding creates enforcement gaps, policies that can't be translated into automated controls remain aspirational rather than operational. ## Summary Technical metadata captures how systems work and data flows. Business metadata explains what data means and why it matters. Operational metadata bridges them, revealing actual usage patterns. Treating metadata management as critical infrastructure rather than documentation overhead is what separates teams that can govern data at scale from those that can't. In streaming architectures where data moves continuously across system boundaries, both technical and business metadata are necessary for building reliable, compliant data platforms. ## Related Concepts - [Data Lineage: Tracking Data from Source to Consumption](https://www.conduktor.io/glossary/data-lineage-tracking-data-from-source-to-consumption) - Understanding how metadata flows through pipelines - [Building a Business Glossary for Data Governance](https://www.conduktor.io/glossary/building-a-business-glossary-for-data-governance) - Creating business context for technical assets - [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) - Managing technical metadata for streaming platforms ## Getting Started: Practical Steps If you're beginning your metadata management journey, follow this phased approach: **Phase 1: Foundation (Weeks 1-4)** 1. **Select a metadata platform**: Evaluate DataHub, OpenMetadata, or Amundsen based on your technology stack and team size 2. **Connect technical sources**: Start with automated metadata extraction from your most critical systems (databases, Kafka clusters, data warehouses) 3. **Establish governance team**: Identify data stewards and owners across business domains 4. **Define metadata standards**: Document required fields for all datasets (minimally: description, owner, update frequency) **Phase 2: Enrichment (Weeks 5-12)** 1. **Add business context**: Have domain experts enrich top 20% of datasets with business metadata (these likely represent 80% of usage) 2. **Implement data classification**: Use automated PII detection tools to tag sensitive data 3. **Document lineage**: Enable lineage capture for key pipelines and transformations 4. **Create business glossary**: Document 20-30 critical business terms with clear definitions **Phase 3: Activation (Weeks 13+)** 1. **Implement metadata-driven policies**: Use classification tags to automatically enforce access controls and masking 2. **Enable self-service discovery**: Train analysts to use data catalog for finding and understanding data 3. **Integrate with CI/CD**: Automatically publish metadata when deploying new pipelines or schemas 4. **Measure adoption**: Track catalog search usage, metadata completeness scores, and time-to-insight metrics **Quick Wins:** - Start with Kafka topics: Schema Registry already provides technical metadata, add business descriptions and ownership - Use existing documentation: Import READMEs, Confluence pages, and data dictionaries as starting business metadata - Focus on pain points: If teams repeatedly ask "what does this field mean?", that's your first enrichment target ## Sources and References - [DAMA-DMBOK: Data Management Body of Knowledge](https://www.dama.org/cpages/body-of-knowledge) - [The Data Catalog: Your Guide to Metadata Management](https://www.oreilly.com/library/view/the-enterprise-data/9781492054771/) - [Apache Kafka Schema Registry Documentation](https://kafka.apache.org/documentation/#schemaregistry) - [Data Governance: The Definitive Guide](https://www.oreilly.com/library/view/data-governance-the/9781492063483/) - [DataHub: Modern Metadata Platform](https://datahubproject.io/) - [OpenMetadata: Open Source Data Discovery and Observability](https://open-metadata.org/) - [OpenLineage: Open Standard for Data Lineage](https://openlineage.io/) Source: https://www.conduktor.io/glossary/metadata-management-technical-vs-business-metadata --- # Micro-Batching: Near-Real-Time Stream Processing Stream processing exists on a spectrum. On one end, traditional batch processing handles large volumes of data at scheduled intervals, hourly, daily, or weekly. On the other end, true streaming processes each event individually as it arrives, achieving latencies measured in single-digit milliseconds. Micro-batching sits squarely in the middle, offering a pragmatic compromise that has powered some of the world's largest streaming applications. Rather than processing events one-by-one or waiting hours for batch jobs, micro-batching collects small groups of events over short time windows, typically ranging from 100 milliseconds to several minutes, and processes them as tiny batches. This hybrid approach unlocks significant architectural advantages while maintaining near-real-time responsiveness for many use cases. ![Micro-batching vs traditional batch vs true streaming comparison](https://www.conduktor.io/assets/images/glossary/micro-batching-streaming-0.webp) ## How Micro-Batching Works At its core, micro-batching applies batch processing semantics to streaming data by dividing the continuous stream into discrete, time-bounded chunks. Every few seconds (or subseconds), the system collects all events that arrived during that interval and processes them together as a single batch. Consider a typical micro-batch pipeline processing clickstream events. With a 5-second batch interval, the system accumulates all clicks received between 12:00:00 and 12:00:05, processes this group together, then moves to the 12:00:05-12:00:10 window. Each micro-batch is independent, executed as a bounded dataset with clear start and end points. This approach leverages the well-understood batch processing model. Each micro-batch runs through the same execution plan: read from source, apply transformations, write to sink. State management becomes simpler because the system only needs to coordinate state within each small batch rather than managing continuously evolving state across individual events. The batch interval becomes the fundamental tuning parameter. Shorter intervals reduce latency but increase overhead from job scheduling and coordination. Longer intervals improve throughput by amortizing overhead across more events but increase end-to-end latency proportionally. ## Why Micro-Batching Exists Micro-batching emerged as a practical engineering solution to three fundamental challenges in stream processing: exactly-once semantics, fault tolerance, and operational simplicity. - **Exactly-once processing** is significantly easier to achieve with micro-batching. Because each batch is atomic and deterministic, the system can replay failed batches without worrying about partial progress or duplicate processing. If batch 1,427 fails halfway through, simply reprocess batch 1,427 from the beginning. The bounded nature of each batch makes idempotent reprocessing straightforward. - **Fault tolerance** builds naturally on batch boundaries. When a node fails, the system only needs to recompute the affected micro-batches, not reconstruct complex in-flight state for thousands of individual events. Checkpointing happens between batches at well-defined points, creating clean recovery semantics. This aligns perfectly with distributed computing frameworks designed around batch operations. - **Operational simplicity** matters enormously in production systems. Developers already understand batch processing. They know how to reason about bounded datasets, optimize batch jobs, and debug processing logic. Micro-batching lets teams apply this existing expertise to streaming problems without mastering the additional complexity of per-record state management, watermarks (timestamp-based markers that track event-time progress in out-of-order streams), and event-time processing that true streaming demands. The trade-off is clear: accept slightly higher latency (measured in seconds rather than milliseconds) in exchange for simpler programming models and more robust exactly-once guarantees. ## Spark Structured Streaming Implementation Apache Spark popularized micro-batching through Structured Streaming (Spark 3.5+), which treats streams as unbounded tables that grow continuously. The programming model is elegant: write the same DataFrame operations you'd use for batch processing, and Spark handles the streaming execution. Here's a basic example processing Kafka events with a 5-second micro-batch interval: ```python from pyspark.sql import SparkSession from pyspark.sql.functions import window, count spark = SparkSession.builder \ .appName("MicroBatchExample") \ .getOrCreate() # Read from Kafka as a stream clicks = spark.readStream \ .format("kafka") \ .option("kafka.bootstrap.servers", "localhost:9092") \ .option("subscribe", "clickstream") \ .load() # Process micro-batches every 5 seconds windowed_counts = clicks \ .selectExpr("CAST(value AS STRING) as click") \ .groupBy(window("timestamp", "1 minute")) \ .agg(count("click").alias("click_count")) query = windowed_counts.writeStream \ .outputMode("update") \ .format("console") \ .trigger(processingTime="5 seconds") \ .start() query.awaitTermination() ``` Structured Streaming offers three trigger modes that control how micro-batches execute: - **Processing time triggers** execute micro-batches at fixed intervals. Setting `trigger(processingTime='5 seconds')` runs a new batch every 5 seconds, regardless of how long the previous batch took. This mode works well when you need predictable intervals and can keep up with the data rate. - **Once triggers** process all available data as a single batch and then stop. This proves valuable for testing streaming jobs or implementing scheduled near-real-time workflows that run periodically rather than continuously. - **Continuous triggers** (introduced in Spark 2.3, refined in 3.5+) attempt to achieve lower latency by using a different execution engine that processes records with minimal delay. Unlike traditional micro-batching which waits for the trigger interval, continuous mode achieves end-to-end latencies as low as 1 millisecond for simple operations by maintaining long-running tasks that continuously process incoming data. However, this mode has limitations, it only supports map-like operations and certain sources/sinks, not complex stateful aggregations. ### Checkpointing and Fault Tolerance Under the hood, Spark uses incremental execution planning combined with write-ahead logs for reliability. Each micro-batch analyzes what new data arrived, generates an optimized query plan, executes it across the cluster, and checkpoints progress to fault-tolerant storage (HDFS, S3, etc.). The checkpoint location stores: - **Offset logs**: Which data has been processed from each source - **State snapshots**: Intermediate aggregation results for stateful operations - **Commit logs**: Confirmation that batches completed successfully If a job fails, Spark reads the checkpoint to determine the last successfully processed offset, then replays subsequent micro-batches. This design makes exactly-once processing straightforward, each batch is idempotent and can be safely reprocessed. ```python # Checkpoint configuration query = windowed_counts.writeStream \ .outputMode("update") \ .format("kafka") \ .option("kafka.bootstrap.servers", "localhost:9092") \ .option("topic", "aggregated_clicks") \ .option("checkpointLocation", "/tmp/checkpoint") \ .trigger(processingTime="5 seconds") \ .start() ``` For organizations managing complex Spark streaming deployments at scale, monitoring and governance become critical. Conduktor provides comprehensive capabilities for [monitoring streaming pipeline health](https://docs.conduktor.io/guide/monitor-brokers-apps/index), enforcing data quality policies, and ensuring compliance as your streaming infrastructure grows. For Kafka-specific details, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ## Latency and Performance Trade-offs Micro-batching's latency characteristics fall into a distinct range. End-to-end latency typically spans from 100 milliseconds to several minutes, depending on batch interval and processing complexity. This makes micro-batching suitable for near-real-time use cases but not for ultra-low-latency scenarios. Several factors determine actual latency: - **Batch interval** sets the minimum latency floor. A 10-second interval means events wait an average of 5 seconds just to be included in a batch, plus processing time. You cannot achieve sub-second latency with minute-long batch intervals. - **Processing time** adds to the batch interval. Consider a concrete example: if your batch interval is 5 seconds and your pipeline receives 100,000 events per batch, processing those events (parsing, transforming, aggregating) might take 2 seconds on a properly sized cluster. This means effective average latency is 5 seconds (wait time) + 2 seconds (processing) = 7 seconds. If processing slows to 6 seconds due to increased load, your effective latency jumps to 11 seconds, and you start falling behind. - **Scheduling overhead** becomes significant at very short intervals. Starting and coordinating a distributed job carries fixed costs, typically tens to hundreds of milliseconds. At sub-second batch intervals, this overhead can dominate, making micro-batching inefficient compared to true streaming. - **Throughput optimization** often conflicts with latency goals. Increasing batch interval improves throughput by amortizing overhead and enabling better compression, but directly increases latency. The sweet spot typically falls between 1-30 seconds for most workloads. Performance tuning focuses on finding the right balance. Increase parallelism to process larger batches faster. Adjust batch intervals to match your latency requirements. Monitor end-to-end latency metrics to ensure batches complete within their interval window, avoiding backlog buildup. ## Micro-Batching vs True Streaming The distinction between micro-batching and true streaming matters for architecture decisions. For deep comparisons, see [Flink vs Spark Streaming: When to Choose Each](https://www.conduktor.io/glossary/flink-vs-spark-streaming-when-to-choose-each) and [Kafka Streams vs Apache Flink](https://www.conduktor.io/glossary/kafka-streams-vs-apache-flink). For foundational Kafka knowledge, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). - **True streaming engines** like Apache Flink (1.19+) process events one-at-a-time through a dataflow graph. This enables single-digit millisecond latencies and more natural event-time semantics. Flink's state backend has matured significantly, RocksDB state backend now supports incremental checkpointing, state TTL (time-to-live) for automatic cleanup, and changelog-based recovery in Flink 1.15+. For comprehensive coverage, see [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing). However, it requires sophisticated state management, complex checkpointing mechanisms, and careful watermark handling to achieve exactly-once guarantees. - **Kafka Streams** (3.0+) takes a different approach to true streaming, embedding stream processing directly into your application as a library. It processes records individually but maintains local state stores and coordinates distributed processing through Kafka's consumer groups. With `processing.guarantee=exactly_once_v2` (introduced in Kafka 2.5, refined in 3.0+), Kafka Streams achieves exactly-once semantics with significantly lower overhead than earlier implementations. This achieves low latency while leveraging Kafka's durability and replication. Here's a Kafka Streams example for comparison to the earlier Spark micro-batching code: ```java Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG, "click-counter"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, "exactly_once_v2"); StreamsBuilder builder = new StreamsBuilder(); KStream clicks = builder.stream("clickstream"); // Process each event individually (true streaming) clicks .groupByKey() .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1))) .count() .toStream() .to("aggregated_clicks"); KafkaStreams streams = new KafkaStreams(builder.build(), props); streams.start(); ``` **Micro-batching frameworks** trade latency for simplicity. By processing small groups of events together, they achieve stronger exactly-once semantics more easily and offer simpler failure recovery. The programming model is more accessible to developers familiar with batch processing. The performance gap has narrowed over time. Spark's continuous mode reduces micro-batch overhead, while Flink's checkpointing has become more efficient. Modern micro-batching can achieve sub-second latencies for many operations, while true streaming has simplified its programming models. Choose based on requirements: if you need millisecond-level latency or complex event-time processing, true streaming excels. If second-level latency suffices and you value operational simplicity, micro-batching often proves more pragmatic. ## Use Cases and When to Choose Micro-Batching Micro-batching shines in specific scenarios where near-real-time processing suffices and operational simplicity matters. - **Aggregations and metrics** work naturally with micro-batching. Computing page views per minute, calculating rolling averages, or generating dashboards updated every few seconds aligns perfectly with batch intervals. The slight delay is acceptable, and batch processing makes aggregations efficient. - **ETL and data synchronization** between systems often tolerates seconds of latency. Moving data from Kafka to a data warehouse, synchronizing databases, or updating search indexes every 5-30 seconds satisfies most business requirements while batching improves throughput. - **Machine learning feature generation** benefits from micro-batching's throughput. Computing features for recommendation systems or fraud detection models can happen in 10-second batches, providing fresh features without requiring millisecond latency. - **Business monitoring and alerting** where response times of 10-60 seconds are acceptable. Detecting anomalies in application logs, monitoring business KPIs, or triggering alerts based on threshold violations work well with micro-batching. Avoid micro-batching when you need sub-second latency for user-facing features, complex event pattern matching requiring precise timing, or financial trading systems where milliseconds matter. In these cases, true streaming engines provide necessary performance. The evolution continues toward hybrid approaches. Modern frameworks increasingly blur the lines, offering both micro-batch and continuous modes. As systems mature, choose based on your specific latency requirements, team expertise, and operational complexity tolerance rather than strict architectural dogma. Micro-batching's strengths (simplicity, fault tolerance, throughput) and limitations (latency floor) make it a good fit for many use cases, and a poor fit for a few others. Know which category you're in before choosing. ## Related Concepts - [Flink vs Spark Streaming: When to Choose Each](https://www.conduktor.io/glossary/flink-vs-spark-streaming-when-to-choose-each) - Comparing micro-batching to true streaming architectures - [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) - True streaming alternative to micro-batching - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Platform commonly used as input source for micro-batch processing ## Sources and References 1. **Apache Spark Structured Streaming Programming Guide** - Official documentation covering micro-batching architecture, trigger modes, and performance tuning in Spark 3.5+. [https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) 2. **Apache Flink Documentation** - Official documentation for Apache Flink's true streaming architecture and state management. [https://flink.apache.org/](https://flink.apache.org/) 3. **Kafka Streams Documentation** - Official guide covering Kafka Streams architecture and exactly-once semantics. [https://kafka.apache.org/documentation/streams/](https://kafka.apache.org/documentation/streams/) 4. **Conduktor** - Comprehensive platform for managing, monitoring, and governing Kafka and streaming data pipelines including Spark Structured Streaming deployments. [https://www.conduktor.io/](https://www.conduktor.io/) 5. **Tyler Akidau et al. - The Dataflow Model** - Research paper introducing streaming concepts and discussing batch vs streaming trade-offs. Google, 2015. Source: https://www.conduktor.io/glossary/micro-batching-streaming --- # Migrating to Apache Iceberg from Hive or Parquet Apache Iceberg is the leading table format for modern [data lakehouses](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture), offering ACID transactions, time travel, schema evolution, and partition evolution that traditional Hive tables and raw Parquet files cannot provide. As of 2025, [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg) 1.7+ provides production-grade migration tools, REST catalog support, and advanced features like branches and tags that make migration safer and more flexible than ever. Migrating to Iceberg unlocks these capabilities while maintaining compatibility with your existing query engines and data infrastructure. This guide explores proven migration strategies, practical conversion techniques, and critical considerations for data engineers and architects planning an Iceberg migration. ![Migration strategies: in-place vs snapshot vs dual-write](https://www.conduktor.io/assets/images/glossary/migrating-to-apache-iceberg-from-hive-or-parquet-0.webp) ## Table of Contents 1. [Understanding the Migration Landscape](#understanding-the-migration-landscape) 2. [Migration Strategies](#migration-strategies) 3. [Migrating Hive Tables to Iceberg](#migrating-hive-tables-to-iceberg) 4. [Converting Parquet Datasets to Iceberg](#converting-parquet-datasets-to-iceberg) 5. [Streaming Integration During Migration](#streaming-integration-during-migration) 6. [Validation and Testing](#validation-and-testing) 7. [Best Practices and Considerations](#best-practices-and-considerations) ## Understanding the Migration Landscape Before initiating a migration, assess your current state and requirements: **From Hive Tables:** - Existing Hive metastore integration - Partition structures and naming conventions - Table statistics and metadata - Query patterns and access frequencies - Downstream dependencies on Hive-specific features **From Raw Parquet:** - File organization and directory structure - Partition schemes (if any) - Schema consistency across files - Metadata availability - Current read/write patterns Iceberg's design accommodates both scenarios with different migration approaches: **in-place migration** for Hive tables (converting metadata without moving data) and **metadata-based adoption** for Parquet datasets (creating Iceberg metadata to track existing files). For comprehensive coverage of Iceberg's metadata architecture, see [Iceberg Table Architecture: Metadata and Snapshots](https://www.conduktor.io/glossary/iceberg-table-architecture-metadata-and-snapshots). ## Migration Strategies ### Strategy 1: In-Place Migration (Hive to Iceberg) In-place migration converts existing Hive tables to Iceberg tables without moving or rewriting data files. This approach offers: - **Minimal downtime**: Metadata conversion happens quickly - **No data movement**: Original Parquet/ORC files remain in place - **Rollback capability**: Can revert to Hive if needed - **Resource efficiency**: No data copying or rewriting required **When to use:** Production Hive tables with stable partition schemes, large datasets where data copying is prohibitive, or scenarios requiring minimal disruption. ### Strategy 2: Snapshot and Migrate Create an Iceberg table and copy data from the source, allowing for optimization during migration: - **Data optimization**: Rewrite files to optimal sizes - **Partition evolution**: Restructure partitioning scheme - **Schema refinement**: Clean up schema inconsistencies - **Incremental migration**: Migrate in batches over time **When to use:** When data reorganization is beneficial, source tables have performance issues, or you want to optimize file layouts during migration. ### Strategy 3: Dual-Write Transition Temporarily write to both old and new formats during transition: - **Zero downtime**: Seamless cutover for readers - **Extended validation**: Verify Iceberg behavior with production workloads - **Gradual migration**: Migrate read traffic incrementally **When to use:** Mission-critical tables where zero downtime is mandatory, or when extensive validation is required before full cutover. ## Migrating Hive Tables to Iceberg ### Using Spark SQL for In-Place Migration Spark 3.5+ provides native support for in-place Hive table migration with Iceberg 1.7+: ```sql -- Migrate a Hive table to Iceberg format in-place (Iceberg 1.4+) CALL system.migrate('db_name.table_name'); -- Verify migration success DESCRIBE EXTENDED db_name.table_name; ``` This command performs the following operations: 1. Reads existing Hive table metadata and partition information 2. Creates Iceberg metadata files (metadata.json, manifest lists, manifest files) 3. Updates the Hive metastore to point to Iceberg table format 4. Preserves all existing data files in their current locations (no data movement) 5. Maintains backward compatibility for readers that haven't been updated yet ### Programmatic Migration with Spark For more control over the migration process (Spark 3.5+ with Iceberg 1.7+): ```scala import org.apache.iceberg.spark.actions.SparkActions // Configure Spark with Iceberg support val spark = SparkSession.builder() .config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog") .config("spark.sql.catalog.spark_catalog.type", "hive") // Using Hive Metastore for catalog .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") .enableHiveSupport() .getOrCreate() // Perform in-place migration with granular control val actions = SparkActions.get() val migrationResult = actions .migrateTable("db_name.table_name") .execute() // Migration result provides detailed statistics println(s"Migrated ${migrationResult.migratedDataFilesCount} files") println(s"Migration completed in ${migrationResult.executionTimeMillis}ms") ``` ### Snapshot Migration Approach For scenarios requiring data rewrite and optimization: ```sql -- Create new Iceberg table with optimized settings (Iceberg 1.7+) CREATE TABLE iceberg_catalog.db_name.new_table USING iceberg PARTITIONED BY (date, region) TBLPROPERTIES ( 'write.format.default' = 'parquet', 'write.parquet.compression-codec' = 'zstd', -- Modern compression (Zstandard) 'write.target-file-size-bytes' = '536870912', -- 512 MB target file size 'write.metadata.compression-codec' = 'gzip' -- Compress metadata files ) AS SELECT * FROM hive_db.legacy_table; -- Verify data integrity SELECT COUNT(*), SUM(bytes_size) FROM iceberg_catalog.db_name.new_table; ``` ### Using Branches for Safe Migration Testing (Iceberg 1.5+) Iceberg's branch feature allows you to test migrations in isolation before committing to production: ```sql -- Create a migration test branch ALTER TABLE iceberg_catalog.db_name.new_table CREATE BRANCH IF NOT EXISTS migration_test RETAIN 7 DAYS; -- Test writes and transformations on the branch INSERT INTO iceberg_catalog.db_name.new_table.branch_migration_test SELECT * FROM hive_db.legacy_table WHERE date = '2025-01-15'; -- Validate data on the branch SELECT COUNT(*), AVG(metric_value) FROM iceberg_catalog.db_name.new_table VERSION AS OF 'migration_test'; -- If validation passes, fast-forward main to the branch CALL iceberg_catalog.system.fast_forward( 'db_name.new_table', 'main', 'migration_test' ); ``` This branching approach provides a safety net for migration testing, allowing you to validate transformations before affecting production queries. ## Converting Parquet Datasets to Iceberg ### Adding Metadata to Existing Parquet Files Iceberg can adopt existing Parquet files without rewriting them using the modern bulk import API (Iceberg 1.6+): ```scala import org.apache.iceberg.spark.actions.SparkActions import org.apache.iceberg.PartitionSpec import org.apache.iceberg.catalog.TableIdentifier // Infer schema from existing Parquet files val schema = spark.read.parquet("s3://bucket/data/events/") .schema // Define partition specification using hidden partitioning val spec = PartitionSpec.builderFor(schema) .day("event_time") // Transforms event_time into daily partitions automatically .identity("region") // Uses region column as-is for partitioning .build() // Create Iceberg table metadata structure (no data movement) spark.sql(""" CREATE TABLE iceberg_catalog.db_name.events ( event_id STRING, user_id BIGINT, event_time TIMESTAMP, region STRING, properties MAP ) USING iceberg PARTITIONED BY (days(event_time), region) LOCATION 's3://bucket/data/events/' TBLPROPERTIES ( 'write.format.default' = 'parquet' ) """) // Bulk import existing Parquet files into Iceberg table metadata (Iceberg 1.6+) SparkActions.get(spark) .rewriteDataFiles( TableIdentifier.of("db_name", "events") ) .option("use-starting-sequence-number", "true") // Preserves file ordering .execute() ``` Note: The `days()` partition function automatically handles date transformations, extracting the date component from the timestamp field for efficient partition pruning. ### Incremental Data Migration For large datasets, migrate in stages to minimize resource consumption and enable parallel processing: ```sql -- Create Iceberg table structure CREATE TABLE iceberg_catalog.analytics.user_events ( user_id BIGINT, event_type STRING, event_time TIMESTAMP, properties MAP ) USING iceberg PARTITIONED BY (days(event_time)); -- Migrate data in daily increments (can be parallelized across multiple Spark jobs) INSERT INTO iceberg_catalog.analytics.user_events SELECT * FROM parquet.`s3://bucket/legacy/events/date=2025-01-01/`; INSERT INTO iceberg_catalog.analytics.user_events SELECT * FROM parquet.`s3://bucket/legacy/events/date=2025-01-02/`; -- Use dynamic SQL for bulk migration across date ranges INSERT INTO iceberg_catalog.analytics.user_events SELECT * FROM parquet.`s3://bucket/legacy/events/` WHERE date BETWEEN '2025-01-01' AND '2025-01-31'; ``` ## Streaming Integration During Migration ### Kafka to Iceberg with Spark Structured Streaming During migration, establish streaming pipelines to keep Iceberg tables updated (Spark 3.5+ with Iceberg 1.7+): ```scala import org.apache.spark.sql.streaming.Trigger // Read from Kafka with modern connector configuration val kafkaStream = spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "localhost:9092") .option("subscribe", "user-events") .option("startingOffsets", "earliest") .option("maxOffsetsPerTrigger", "10000") // Control batch size for backpressure .load() // Parse and transform Kafka messages val parsedStream = kafkaStream .selectExpr("CAST(value AS STRING) as json") .select(from_json($"json", schema).as("data")) .select("data.*") // Write stream to Iceberg with exactly-once semantics (Iceberg 1.7+) parsedStream.writeStream .format("iceberg") .outputMode("append") .trigger(Trigger.ProcessingTime("30 seconds")) .option("checkpointLocation", "s3://bucket/checkpoints/user-events") .option("fanout-enabled", "true") // Distribute writes across workers for parallelism .option("write.format.default", "parquet") // Explicit format specification .option("write.parquet.compression-codec", "zstd") // Modern compression .toTable("iceberg_catalog.analytics.user_events") ``` ### Monitoring Streaming Migrations with Conduktor Managing Kafka-to-Iceberg streaming pipelines during migration requires comprehensive observability and governance. **Conduktor** provides essential capabilities for production-grade migrations: - **Data Quality Monitoring**: Validate message schemas and enforce data contracts before writes reach Iceberg tables, preventing corrupted migrations - **Consumer Lag Tracking**: Monitor streaming job performance in real-time using [topic monitoring](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) to ensure migration keeps pace with incoming data, preventing backlog accumulation - **Topic Management**: Visualize and coordinate multiple Kafka topics feeding into Iceberg tables during phased migrations - **Schema Registry Integration**: Manage schema evolution across both legacy and Iceberg tables with [Schema Registry](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry), ensuring compatibility during the transition - **Pipeline Testing with Conduktor Gateway**: Use Conduktor Gateway, a Kafka proxy, to inject chaos scenarios (network delays, broker failures, partition rebalances) to validate exactly-once semantics and checkpoint recovery before production deployment - **Connector Management**: Manage and monitor Kafka Connect connectors feeding data into Iceberg tables with [Kafka Connect](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) Conduktor's governance features help data engineers identify bottlenecks, validate data consistency, and ensure zero data loss during the transition to Iceberg-based architectures. For mission-critical migrations, testing pipeline resilience with Conduktor Gateway before cutover reduces risk and ensures production stability. ### Handling Late-Arriving Data Iceberg's ACID guarantees make it ideal for handling late data during migration without data loss or duplication: ```sql -- Merge late-arriving historical data (Iceberg 1.4+) MERGE INTO iceberg_catalog.analytics.user_events t USING ( SELECT * FROM parquet.`s3://bucket/late-data/2025-01/` ) s ON t.user_id = s.user_id AND t.event_time = s.event_time WHEN NOT MATCHED THEN INSERT *; ``` ## Validation and Testing ### Data Integrity Validation Post-migration, verify data completeness and correctness: ```sql -- Row count comparison SELECT (SELECT COUNT(*) FROM hive_db.legacy_table) as legacy_count, (SELECT COUNT(*) FROM iceberg_catalog.db.new_table) as iceberg_count, ABS((SELECT COUNT(*) FROM hive_db.legacy_table) - (SELECT COUNT(*) FROM iceberg_catalog.db.new_table)) as difference; -- Sample data comparison (identify records present in source but not in target) SELECT * FROM hive_db.legacy_table LIMIT 100 EXCEPT SELECT * FROM iceberg_catalog.db.new_table LIMIT 100; -- Aggregate consistency check (identify date partitions with mismatches) SELECT date, COUNT(*) as record_count, SUM(amount) as total_amount FROM hive_db.legacy_table GROUP BY date EXCEPT SELECT date, COUNT(*) as record_count, SUM(amount) as total_amount FROM iceberg_catalog.db.new_table GROUP BY date; ``` ### Performance Testing Compare query performance before and after migration to validate improvements: ```scala // Benchmark query execution with statistical analysis def benchmarkQuery(query: String, iterations: Int = 5): Unit = { val times = (1 to iterations).map { _ => val start = System.currentTimeMillis() spark.sql(query).count() System.currentTimeMillis() - start } val avg = times.sum / iterations val min = times.min val max = times.max println(s"Average: ${avg}ms, Min: ${min}ms, Max: ${max}ms") } // Compare performance on typical query patterns benchmarkQuery("SELECT COUNT(*) FROM hive_db.legacy_table WHERE date >= '2025-01-01'") benchmarkQuery("SELECT COUNT(*) FROM iceberg_catalog.db.new_table WHERE date >= '2025-01-01'") // Test partition pruning efficiency benchmarkQuery("SELECT AVG(amount) FROM hive_db.legacy_table WHERE date = '2025-01-15'") benchmarkQuery("SELECT AVG(amount) FROM iceberg_catalog.db.new_table WHERE date = '2025-01-15'") ``` ## Best Practices and Considerations ### Metadata Management Choosing the right catalog implementation is critical for migration success. As of 2025, catalog options include: - **REST Catalog (Recommended for 2025)**: Modern cloud-native catalogs like **Apache Polaris** (Snowflake's open-source catalog) or **Project Nessie** offer vendor neutrality, multi-tenancy, and Git-like versioning. Ideal for new implementations and multi-cloud environments. - **AWS Glue**: Native AWS integration with IAM-based access control. Best for AWS-centric architectures but creates cloud vendor lock-in. - **Hive Metastore**: Legacy option for backward compatibility with existing Hadoop ecosystems. Not recommended for new implementations, consider migrating to REST catalogs. **Metadata Storage Best Practices**: - Store metadata in highly available storage (S3 with versioning, HDFS with replication) - Configure snapshot retention policies to balance time travel capabilities with storage costs - Enable metadata compression for large-scale tables (Iceberg 1.7+) ```sql -- Set snapshot retention policy (Iceberg 1.4+) ALTER TABLE iceberg_catalog.analytics.events SET TBLPROPERTIES ( 'write.metadata.delete-after-commit.enabled' = 'true', 'write.metadata.previous-versions-max' = '100', 'write.metadata.compression-codec' = 'gzip' -- Compress metadata files ); ``` ### Performance Optimization - **File Sizing**: Target 512MB-1GB files for optimal query performance - **Compaction**: Schedule regular compaction for tables with many small files - **Partition Evolution**: Leverage hidden partitioning to avoid partition explosion ```sql -- Compact small files CALL iceberg_catalog.system.rewrite_data_files( table => 'analytics.events', options => map( 'target-file-size-bytes', '536870912', 'min-file-size-bytes', '134217728' ) ); ``` ### Rollback Planning Maintain rollback capabilities during migration: ```sql -- Snapshot Iceberg table before major changes CALL iceberg_catalog.system.create_changelog_view( table => 'analytics.events', options => map('start-snapshot-id', '12345678') ); -- Rollback to previous snapshot if needed CALL iceberg_catalog.system.rollback_to_snapshot( 'analytics.events', 12345678 ); ``` ### Incremental Adoption Don't migrate everything at once: 1. **Start with non-critical tables**: Gain experience with low-risk tables 2. **Validate thoroughly**: Run parallel workloads to compare results 3. **Monitor performance**: Track query latency, throughput, and resource usage 4. **Gather feedback**: Involve data consumers in validation 5. **Scale gradually**: Expand to critical tables after proven success ## Related Concepts - [Streaming ETL vs Traditional ETL](https://www.conduktor.io/streaming-etl-vs-traditional-etl) - [Kafka Connect: Building Data Integration Pipelines](https://www.conduktor.io/kafka-connect-building-data-integration-pipelines) - [Data Pipeline Orchestration with Streaming](https://www.conduktor.io/data-pipeline-orchestration-with-streaming) ## Summary As of 2025, Iceberg 1.7+ provides production-grade migration tooling — REST catalogs, branches and tags, puffin statistics, and improved streaming integrations — making the transition safer and more flexible than earlier versions. In-place migration minimizes disruption, snapshot migration allows optimization, and dual-write enables zero-downtime cutover. Pick the strategy that fits your table size, downtime tolerance, and team capacity. Use branches for isolated testing before committing to production. Start with pilot migrations, validate with row counts and aggregate checks, and scale from there. For deeper coverage of Iceberg's capabilities, see [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg), [Iceberg Table Architecture: Metadata and Snapshots](https://www.conduktor.io/glossary/iceberg-table-architecture-metadata-and-snapshots), and [Introduction to Lakehouse Architecture](https://www.conduktor.io/glossary/introduction-to-lakehouse-architecture). ## Sources - [Apache Iceberg 1.7.x Documentation - Migration Guide](https://iceberg.apache.org/docs/1.7.0/spark-procedures/#migrate) - [Apache Iceberg 1.7.x - Table Migration Strategies](https://iceberg.apache.org/docs/1.7.0/migration/) - [Apache Iceberg 1.7.x - Spark SQL Guide](https://iceberg.apache.org/docs/1.7.0/spark-queries/) - [Apache Iceberg 1.7.x - Configuration and Best Practices](https://iceberg.apache.org/docs/1.7.0/configuration/) - [Apache Iceberg - Branching and Tagging (1.5+)](https://iceberg.apache.org/docs/1.7.0/branching/) - [Apache Polaris - Open-Source Iceberg Catalog](https://www.polaris.io/) - [Project Nessie - Git-like Catalog for Iceberg](https://projectnessie.org/) - [Apache Spark 3.5+ Structured Streaming Guide](https://spark.apache.org/docs/3.5.0/structured-streaming-programming-guide.html) Source: https://www.conduktor.io/glossary/migrating-to-apache-iceberg-from-hive-or-parquet --- # Model Drift in Streaming: When ML Models Degrade in Real-Time Machine learning models powering [real-time applications](https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data) face a unique challenge: the world they model never stops changing. A fraud detection model trained on last quarter's patterns may miss today's attack vectors. A recommendation engine optimized for pre-holiday shopping behavior struggles when consumer preferences shift. This phenomenon, model drift, is the silent degradation of ML performance over time. In streaming architectures, where models consume continuously flowing data and make split-second predictions, drift isn't just a concern, it's inevitable. The question isn't whether your models will drift, but when, how fast, and whether you'll detect it before it impacts your business. ![Model drift types: data, concept, and prediction drift](https://www.conduktor.io/assets/images/glossary/model-drift-in-streaming-0.webp) ## Understanding the Types of Model Drift Model drift manifests in three distinct but often overlapping forms, each with different root causes and detection strategies. - **Data drift** occurs when the statistical properties of input features change. Your fraud model was trained on transactions averaging $50, but suddenly sees a stream dominated by high-value purchases. The distribution has shifted, even though the underlying relationship between features and fraud hasn't changed. In streaming systems, data drift can emerge from upstream schema changes, new data sources joining a stream, or gradual shifts in user behavior. For deeper coverage of data drift detection and monitoring strategies, see [Data Drift in Streaming](https://www.conduktor.io/glossary/data-drift-in-streaming). - **Concept drift** represents a more fundamental problem: the relationship between features and outcomes has changed. A customer churn model learned that "30-day inactivity" strongly predicts cancellation, but after a product redesign, engaged users now check in less frequently. The features are the same, but their meaning has evolved. Streaming amplifies this challenge because business logic, user interfaces, and operational processes continuously evolve, often without triggering model retraining. - **Prediction drift** focuses on the model's output distribution. Even when inputs seem stable, the predictions themselves may shift, 95% of transactions were classified as legitimate last week, but this week it's 87%. This can signal either data drift, concept drift, or emerging model issues, making it a valuable early warning signal in production systems. ## Why Streaming Amplifies Model Drift Batch ML pipelines retrain on a schedule, weekly, monthly, or when performance degrades beyond a threshold. Streaming models don't have this luxury. They operate in continuous time, consuming events as they arrive, with no natural breakpoint for evaluation and retraining. Several streaming-specific factors accelerate drift: - **Continuous feature updates** mean your model sees the latest data immediately. A recommendation model using "trending_topics" as a feature will reflect breaking news within minutes, potentially destabilizing predictions if those topics fall outside the training distribution. - **Upstream schema evolution** is constant in event-driven architectures. When a producer adds a new field, changes an enum value, or subtly alters a timestamp format, downstream models may receive unexpected inputs. Without governance, these changes propagate silently through topics until they surface as prediction anomalies. - **Seasonal and temporal patterns** create cyclical drift. An e-commerce model trained in January will underperform during Black Friday surges unless explicitly designed for seasonality. Streaming's real-time nature means you encounter these patterns as they happen, with no buffering period to prepare. - **Business logic changes** deploy faster than model updates. A product team launches a new feature, marketing runs a campaign, or pricing strategy shifts, all generating event patterns the model has never seen. The model keeps predicting with outdated assumptions while the business has moved on. ## Detection Strategies: Catching Drift Early Effective drift detection in streaming requires continuous monitoring, not periodic batch analysis. Several statistical and operational approaches provide early warning signals. - **Statistical hypothesis testing** compares current data distributions against a reference window from training. The Kolmogorov-Smirnov (KS) test, a non-parametric test measuring the maximum difference between cumulative distributions, detects changes in continuous feature distributions, while chi-square tests catch shifts in categorical features. In a streaming context, these tests run on sliding windows, comparing the last hour's events against baseline statistics. When divergence exceeds a threshold, alerts fire. - **Prediction monitoring** tracks the distribution and confidence of model outputs. A sudden shift in the proportion of positive classifications, a narrowing of confidence scores, or an increase in predictions near decision boundaries all signal potential drift. For models with ground truth labels (the actual correct outcomes, as opposed to model predictions) available after a delay (like fraud detection confirmed after manual investigation), tracking prediction accuracy over time provides direct drift measurement. - **Feature drift detection** monitors individual input features for distribution shifts. In streaming, this means maintaining summary statistics (mean, variance, percentiles) for each feature and comparing recent windows against training baselines. Advanced approaches use embeddings or dimensionality reduction to detect drift in high-dimensional feature spaces. - **Performance degradation monitoring** remains the ultimate drift indicator. If your model has access to labels (even delayed), tracking precision, recall, F1, or AUC over time reveals when predictions degrade. The challenge in streaming is the label delay, fraud labels may arrive days after predictions, creating a detection lag. - **Modern drift detection tooling (2025)** has matured significantly for streaming ML: - **Evidently AI** provides open-source drift detection with pre-built tests for data drift, concept drift, and prediction drift. It integrates with streaming platforms to generate real-time reports comparing current data windows against reference datasets. - **WhyLabs/whylogs** offers lightweight data logging that profiles ML inputs and outputs at scale. Its statistical profiles enable drift detection without storing raw data, making it ideal for privacy-sensitive streaming applications. - **NannyML** specializes in performance estimation without ground truth labels, critical for streaming scenarios where labels arrive with significant delay. It estimates model performance metrics in real-time using confidence-based estimation. - **Alibi Detect** (from Seldon) provides algorithms specifically designed for online drift detection in streaming environments, including outlier detection and adversarial detection for ML security. These tools integrate with streaming platforms like Apache Kafka and Apache Flink to provide continuous drift monitoring without disrupting production inference pipelines. **Example: Real-time drift detection with Evidently AI and Kafka** Here's a practical example of monitoring data drift in a streaming ML pipeline: ```python from evidently.test_suite import TestSuite from evidently.tests import TestColumnDrift, TestShareOfDriftedColumns from kafka import KafkaConsumer import pandas as pd import json # Reference data from training period reference_data = pd.read_parquet('training_data.parquet') # Initialize Kafka consumer for model input stream consumer = KafkaConsumer( 'ml-model-inputs', bootstrap_servers=['localhost:9092'], value_deserializer=lambda m: json.loads(m.decode('utf-8')) ) # Configure drift detection suite drift_suite = TestSuite(tests=[ TestColumnDrift(column_name='transaction_amount'), TestColumnDrift(column_name='user_age'), TestShareOfDriftedColumns(threshold=0.3) # Alert if >30% of features drift ]) # Sliding window for current data current_window = [] window_size = 1000 for message in consumer: current_window.append(message.value) # Run drift detection every 1000 events if len(current_window) >= window_size: current_data = pd.DataFrame(current_window) # Compare current window to reference drift_suite.run(reference_data=reference_data, current_data=current_data) results = drift_suite.as_dict() # Check for drift if not results['summary']['all_passed']: # Publish drift alert to monitoring topic alert = { 'timestamp': message.timestamp, 'drifted_columns': [test['column_name'] for test in results['tests'] if test['status'] == 'FAIL'], 'severity': 'high' if results['summary']['failed_count'] > 2 else 'medium' } print(f"DRIFT DETECTED: {alert}") # Trigger retraining pipeline or send alert # Slide the window current_window = current_window[500:] # 50% overlap ``` This pattern processes streaming ML inputs, maintains a sliding window of recent data, and continuously compares statistical distributions against training baselines. When drift exceeds thresholds, it triggers alerts or automated retraining workflows. **Apache Flink for stateful drift detection** For production-scale drift monitoring, [Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) provides stateful stream processing with exactly-once semantics. Flink's state backends maintain running statistics (mean, variance, percentiles) for each feature across windows, enabling drift detection without recomputing from raw events. Flink ML (introduced in Flink 1.13+) includes built-in operators for online learning and drift adaptation, allowing models to incrementally update as data distributions shift, a more sophisticated alternative to full retraining. ## Prevention Through Data Governance The most effective drift prevention happens upstream, before degraded data reaches your models. This is where data governance and event streaming intersect. - **Schema contracts** define the structure, types, and valid ranges for events in your streams. Governance platforms like Conduktor enforce these [contracts](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) at write time through its [data quality module](https://docs.conduktor.io/guide/use-cases/observe-data-quality), rejecting events that violate expectations. When a producer attempts to send a "user_age" of 500 or omit a required field, the contract blocks it before it corrupts your model's input stream. See [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) for implementation details. - **Schema evolution policies** control how event structures can change over time. Requiring backward compatibility ensures new fields are optional and existing fields maintain their semantics. This governance layer prevents the silent schema drift that destabilizes models. - **Data quality rules** validate not just schema but business logic. A rule might enforce that "transaction_amount > 0" or "timestamp is within the last 5 minutes." These quality gates filter anomalous events before they become model inputs, reducing noise and preventing drift from data quality issues. For comprehensive data quality strategies, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework). - **Topic-level governance** segregates different data quality tiers. Production models consume from gold-tier topics with strict quality enforcement, while experimental models can read bronze-tier streams with raw, unvalidated data. This separation prevents degraded data from reaching critical production models. ## Monitoring and Remediation: Responding to Drift Detection without remediation is just expensive alerting. Streaming ML systems need automated responses to drift events. - **Drift dashboards** provide real-time visibility into model health. Visualizing feature distributions, prediction outputs, and performance metrics over time helps teams distinguish between temporary anomalies and sustained drift. Well-designed dashboards with clear status indicators make it easy to spot issues at a glance. - **Alerting thresholds** trigger when drift metrics exceed acceptable bounds. A two-tier system works well: warnings for moderate drift that requires investigation, and critical alerts for severe drift that demands immediate action. Thresholds should be feature-specific, as some inputs naturally vary more than others. - **Automated retraining triggers** initiate model updates when drift persists beyond thresholds. In a streaming MLOps pipeline, this means kicking off a training job that pulls recent labeled data, retrains the model, validates performance on holdout sets (separate validation data not used during training), and, if validation passes, deploys the updated model to production. The entire cycle happens without manual intervention. - **Model rollback capabilities** provide a safety net when retraining fails or drift proves temporary. If a newly deployed model performs worse than the previous version, automated rollback restores the prior model while teams investigate. This requires maintaining model versions and routing infrastructure to switch between them. - **Feature updates** sometimes address drift better than full retraining. If "time_since_last_login" drifts because users now log in less frequently, adding "weekly_active_sessions" as a complementary feature may stabilize predictions. Feature engineering in response to drift is an ongoing process in streaming ML. ## MLOps Integration: CI/CD for Drifting Models Managing drift at scale requires treating models as continuously evolving artifacts, not static deployments. This is the domain of MLOps, DevOps principles applied to machine learning. - **Continuous integration** for models means automated testing of every model update against drift scenarios. Does the new model handle recent data distributions? Does it maintain performance on edge cases? Integration tests catch regressions before deployment. - **Continuous deployment** automates the path from retraining to production. When a model passes validation, it deploys automatically to staging environments, then production after burn-in periods. Canary deployments (a gradual rollout strategy that routes a small percentage of traffic to the new model version while monitoring for issues) detect problems before full rollout. - **Continuous monitoring** closes the loop, feeding drift signals back into the retraining pipeline. This creates a self-correcting system: drift detection triggers retraining, deployment automation updates production, and monitoring validates the fix. Streaming platforms like [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) (especially Kafka 4.0+ with KRaft mode for lower-latency metadata operations) and [Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing), when paired with governance tools like Conduktor, provide the data infrastructure for this loop. Governed topics ensure clean training data, [streaming feature stores](https://www.conduktor.io/glossary/feature-stores-for-machine-learning) provide consistent feature engineering, and event-driven architectures enable real-time monitoring and response. Kafka 4.0's KRaft architecture eliminates ZooKeeper dependency, reducing operational complexity and improving metadata consistency for ML pipelines. This translates to faster schema updates, more reliable consumer group coordination, and reduced latency in feature serving, all critical for minimizing the window between drift detection and remediation. ## Governance as Drift Prevention Model drift in streaming systems is inevitable, but catastrophic drift is preventable. The key is data governance: enforcing quality at the source, controlling schema evolution, and maintaining contracts between data producers and ML consumers. Upstream governance combined with downstream monitoring lets models adapt to changing conditions without silently degrading. Your models are only as stable as the data contracts protecting them. ## Related Concepts - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/data-quality-dimensions-accuracy-completeness-and-consistency) - Quality dimensions affecting model drift - [Schema Registry and Schema Management](https://www.conduktor.io/schema-registry-and-schema-management) - Prevent schema drift in ML inputs - [Streaming Data Pipeline](https://www.conduktor.io/streaming-data-pipeline) - Build resilient pipelines for ML models ## Sources and References 1. **Google - ML Engineering Best Practices - Monitoring** - Best practices for monitoring machine learning models in production, including drift detection strategies. [https://developers.google.com/machine-learning/guides/rules-of-ml](https://developers.google.com/machine-learning/guides/rules-of-ml) 2. **Evidently AI - Machine Learning Model Monitoring** - Comprehensive guide to detecting and managing different types of model drift in production systems. [https://www.evidentlyai.com/blog/machine-learning-monitoring-data-and-concept-drift](https://www.evidentlyai.com/blog/machine-learning-monitoring-data-and-concept-drift) 3. **AWS - Amazon SageMaker Model Monitor** - Documentation on automated drift detection and monitoring for deployed ML models. [https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html) 4. **Chip Huyen - Designing Machine Learning Systems** - Comprehensive coverage of MLOps practices including drift detection and model retraining strategies. O'Reilly Media, 2022. 5. **Conduktor - Data Governance for Streaming ML** - Best practices for implementing data quality rules, schema enforcement, and topic governance to prevent model drift. [https://www.conduktor.io/](https://www.conduktor.io/) Source: https://www.conduktor.io/glossary/model-drift-in-streaming --- # mTLS for Kafka: Mutual Authentication in Streaming In modern data streaming architectures, machine-to-machine communications need strong authentication. While many systems rely on one-way TLS to encrypt data in transit, regulated industries and zero-trust environments require cryptographic proof of client identity. mTLS extends traditional TLS by requiring both the client and server to present valid certificates during the connection handshake. For Kafka, this means brokers authenticate clients, and clients authenticate brokers, creating a cryptographically verified identity for every participant in the streaming platform. mTLS is one of several authentication mechanisms available in Kafka. For a comprehensive comparison of authentication approaches including SASL/SCRAM, SASL/PLAIN, and OAuth 2.0, see [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth). For broader security context covering authorization, encryption, and operational best practices, refer to [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices). ## Understanding mTLS vs. One-Way TLS Traditional TLS (often called one-way TLS) establishes an encrypted connection where only the server presents a certificate to prove its identity. The client verifies this certificate against a trusted Certificate Authority (CA), ensuring it's connecting to the legitimate server. However, the server accepts any client connection without cryptographic proof of the client's identity. Mutual TLS adds a critical second verification step. After the client verifies the server's certificate, the server requests the client's certificate and validates it against its own trusted CA. Both parties must successfully authenticate each other before any data exchange occurs. While mTLS provides authentication (proving identity), it works in conjunction with encryption in transit. For detailed coverage of how TLS encryption protects Kafka data as it moves through your infrastructure, see [Encryption at Rest and In Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka). ![mtls-for-kafka diagram 1](https://www.conduktor.io/assets/images/glossary/mtls-for-kafka-0.webp) This bidirectional authentication model is crucial for Kafka because: - **Machine-to-machine authentication**: Producers and consumers are typically applications, not human users. Traditional username/password mechanisms are poorly suited for automated systems that need to authenticate thousands of times per second. - **Zero-trust architecture**: In environments where network perimeter security cannot be assumed, every connection must be individually authenticated and authorized. - **Regulatory compliance**: Industries like finance, healthcare, and government often mandate cryptographic authentication for sensitive data streams. ## Certificate Architecture for Kafka Implementing mTLS for Kafka requires a well-designed certificate infrastructure with three primary components: - **Certificate Authority (CA)**: The CA sits at the root of trust, issuing and signing certificates for both brokers and clients. Organizations typically operate a private CA for internal Kafka deployments, though some cloud environments integrate with managed certificate services. - **Server Certificates**: Each Kafka broker receives a server certificate signed by the CA. This certificate contains the broker's identity (typically its hostname or FQDN) and proves to clients that they're connecting to a legitimate broker. The certificate's Subject Alternative Name (SAN) field must match the broker's advertised listener addresses. - **Client Certificates**: Every producer and consumer application requires its own client certificate, also signed by the CA. These certificates identify the specific application or service, forming the basis for Kafka's Access Control Lists (ACLs). The certificate's Distinguished Name (DN) or Common Name (CN) becomes the principal used in authorization policies. For example, a typical client certificate DN might look like: ``` CN=payment-processor,OU=Finance,O=Acme Corp,L=San Francisco,ST=California,C=US ``` Breaking this down: - **CN** (Common Name): `payment-processor` - The application/service identifier - **OU** (Organizational Unit): `Finance` - Department or team - **O** (Organization): `Acme Corp` - Company name - **L** (Locality): `San Francisco` - City - **ST** (State): `California` - State/province - **C** (Country): `US` - Two-letter country code Many organizations use minimal DNs for simplicity: `CN=payment-processor,OU=production,O=Acme` contains sufficient information for most authorization policies. The trust chain works as follows: Clients trust the CA, the CA signs broker certificates, therefore clients trust brokers. The same logic applies in reverse for broker-to-client authentication. ## Implementation: Keystores and Truststores Kafka's mTLS implementation relies on Java's keystore and truststore mechanism: - **Keystores** contain private keys and certificates that identify the entity. Brokers have keystores with their server certificates and private keys. Clients have keystores with their client certificates and private keys. These files must be protected with strong passwords and filesystem permissions, as compromise of a private key allows impersonation. - **Truststores** contain the CA certificates used to verify the other party. Both brokers and clients need truststores containing the CA certificate that signed the certificates they'll be validating. For Kafka brokers, the configuration looks like this: ```properties listeners=SSL://kafka-broker:9093 security.inter.broker.protocol=SSL # TLS 1.3 is the modern standard (2025), offering better performance and security ssl.protocol=TLSv1.3 ssl.enabled.protocols=TLSv1.3 # PKCS12 is the modern keystore format (JKS is deprecated in Java 9+) ssl.keystore.type=PKCS12 ssl.keystore.location=/var/private/ssl/kafka.server.keystore.p12 ssl.keystore.password=keystore-password ssl.key.password=key-password ssl.truststore.type=PKCS12 ssl.truststore.location=/var/private/ssl/kafka.server.truststore.p12 ssl.truststore.password=truststore-password # Enforce mutual authentication (alternatives: 'requested' allows but doesn't require client certs) ssl.client.auth=required ``` The critical parameter is `ssl.client.auth=required`, which enforces mutual authentication by demanding client certificates. Setting this to `requested` would allow both authenticated and unauthenticated clients, while `none` disables client certificate verification entirely. ## mTLS in KRaft Mode Kafka 3.3+ introduced KRaft mode, eliminating ZooKeeper dependencies. KRaft mode introduces additional mTLS considerations for controller-to-controller and broker-to-controller communications. In KRaft deployments, you must configure mTLS for the controller quorum in addition to client-facing listeners: ```properties # Controller listener (separate from client listeners) controller.listener.names=CONTROLLER listeners=CONTROLLER://kafka-controller:9094,SSL://kafka-broker:9093 # Controller inter-communication uses mTLS controller.quorum.voters=1@kafka-controller-1:9094,2@kafka-controller-2:9094,3@kafka-controller-3:9094 # Apply TLS configuration to controller listener listener.security.protocol.map=CONTROLLER:SSL,SSL:SSL ssl.client.auth=required ``` KRaft controllers form a Raft consensus group that stores cluster metadata. Securing this communication with mTLS is critical, as compromised controller access allows complete cluster control. Modern Kafka deployments should treat controller certificates with the same rigor as broker certificates, implementing automated rotation and monitoring. ## Certificate Lifecycle Management Certificates have finite lifespans, and managing their lifecycle presents operational challenges in production Kafka deployments. - **Certificate Rotation**: Industry best practices recommend certificate rotation every 90 days or less. For large Kafka clusters with hundreds of applications, this creates significant operational overhead. Automation becomes mandatory, manual certificate updates across distributed systems are error-prone and don't scale. - **Expiration Monitoring**: Expired certificates cause immediate application failures. Monitoring systems must track certificate expiration dates and alert operators well in advance. Many organizations set alerts at 30, 14, and 7 days before expiration. - **Certificate Revocation**: When private keys are compromised or applications are decommissioned, their certificates must be revoked. Kafka supports Certificate Revocation Lists (CRLs) and Online Certificate Status Protocol (OCSP), though many production deployments rely on short certificate lifespans (24-72 hours) rather than implementing revocation checking. OCSP stapling, where the server provides certificate status during the handshake, is more practical than traditional CRL downloads for distributed Kafka clusters. - **Key Storage Security**: Private keys represent the security foundation of mTLS. They should never be stored in version control, shared via email, or placed in unsecured locations. Hardware Security Modules (HSMs) or cloud key management services provide the highest security, though many deployments use encrypted keystores with strict filesystem permissions. ## Performance Considerations mTLS introduces computational overhead that impacts Kafka performance, particularly for high-throughput deployments. However, modern TLS implementations and hardware acceleration significantly reduce this overhead. **TLS 1.3 Performance Improvements**: TLS 1.3 (recommended for 2025 deployments) offers substantial performance gains over TLS 1.2: - **Reduced handshake latency**: TLS 1.3 completes handshakes in one round-trip instead of two, cutting connection establishment time in half - **0-RTT resumption**: Clients can send encrypted data in the first packet when resuming sessions, eliminating handshake overhead for subsequent connections - **Simplified cipher suites**: TLS 1.3 removes weak cryptographic algorithms, improving negotiation speed and security - **Session Resumption**: Modern Kafka clients should implement TLS session caching. When a client reconnects to a broker it recently communicated with, session resumption allows bypassing the full handshake, reusing previously negotiated keys. This dramatically improves reconnection performance, especially in microservice architectures where applications frequently restart. - **Hardware Acceleration**: Modern server processors include specialized instructions that accelerate cryptographic operations: - **AES-NI**: Intel/AMD instruction set for hardware-accelerated AES encryption (reduces encryption overhead by 80-90%) - **AES-GCM**: Galois/Counter Mode provides both encryption and authentication in a single operation - **AVX-512**: Vector instructions further accelerate cryptographic operations on recent processors - **Connection Pooling**: Applications should maintain persistent connections to brokers rather than creating new connections for each operation. Connection pooling amortizes the handshake cost across thousands of messages. Most Kafka client libraries handle connection pooling automatically. - **Performance Impact**: With TLS 1.3 and modern hardware, mTLS overhead is typically 5-15% CPU increase compared to plaintext connections, significantly better than the 20-30% overhead of TLS 1.2. The exact impact varies with message size, throughput, and hardware capabilities. For most deployments, the security benefits of mTLS far outweigh the modest performance cost. ## Integration with Kafka ACLs mTLS authentication integrates seamlessly with Kafka's authorization system. The certificate's Distinguished Name becomes the principal in ACL rules. For example, a client certificate with `CN=payment-processor,OU=Finance,O=Company` produces the principal `CN=payment-processor,OU=Finance,O=Company`. ACL rules can then grant specific permissions: ```bash kafka-acls --add --allow-principal "User:CN=payment-processor,OU=Finance,O=Company" \ --operation Write --topic payments ``` This tight integration between authentication and authorization creates a robust security model: clients prove their identity with certificates, then ACLs enforce what each authenticated identity can do. Some organizations use certificate Subject Alternative Names (SANs) or custom certificate extensions to encode additional metadata like environment (dev/staging/prod) or team ownership, enabling sophisticated authorization policies. ## Operational Challenges Despite its security benefits, mTLS introduces operational complexity that teams must address. - **Certificate Distribution**: Every client application needs its certificate and keystore. In large organizations with hundreds of microservices, distributing and installing certificates securely becomes a significant challenge. Manual distribution doesn't scale, and automated solutions must ensure certificates reach only their intended recipients. - **Key Management**: Private keys must be protected throughout their lifecycle, during generation, storage, distribution, and eventual destruction. Leaked private keys allow attackers to impersonate legitimate applications. - **Debugging**: Connection failures in mTLS environments can stem from expired certificates, incorrect CA chains, hostname mismatches, or misconfigured truststores. Troubleshooting requires SSL/TLS expertise and careful examination of certificate properties. - **Multi-tenant Environments**: Platforms serving multiple teams or customers must isolate certificates and ensure one tenant cannot use another's credentials. This requires careful certificate namespace design and strict validation of certificate properties. ## Best Practices for Production Successful mTLS deployments in production Kafka environments follow several key practices: - **Automate Certificate Management**: Use tools like cert-manager (Kubernetes), HashiCorp Vault, or AWS Certificate Manager to automate certificate issuance, renewal, and distribution. Manual processes don't scale and create security vulnerabilities. - **Implement Certificate Monitoring**: Deploy monitoring systems that track certificate expiration dates, validate certificate chains, and alert on upcoming expirations or validation failures. - **Use Short Certificate Lifespans**: While operationally more challenging, shorter certificate validity periods (30-90 days) limit the damage from compromised keys and force automation maturity. - **Separate Environments**: Use different CAs or certificate namespaces for development, staging, and production environments. This prevents production certificates from being accidentally deployed to less secure environments. - **Document Certificate Standards**: Establish organizational standards for certificate DNs, including naming conventions for different application types and environments. - **Test Certificate Rotation**: Regularly test certificate rotation procedures in non-production environments to ensure automation works correctly and applications handle certificate updates gracefully. - **Implement Least Privilege**: Combine mTLS authentication with fine-grained ACLs that grant each application only the minimum permissions required for its function. ## Certificate Management Tools Modern infrastructure platforms provide tools that simplify mTLS implementation for 2025 deployments: - **cert-manager (Kubernetes)**: A Kubernetes-native certificate management tool that automates certificate issuance and renewal from various CAs (Let's Encrypt, Vault, AWS Private CA). It integrates seamlessly with Kafka deployments running in Kubernetes, automatically rotating certificates and updating secrets. cert-manager supports PKCS12 keystores and can configure certificates for StatefulSets running Kafka brokers. - **SPIFFE/SPIRE**: The Secure Production Identity Framework For Everyone (SPIFFE) provides workload identity standards, while SPIRE implements these standards. SPIRE automatically issues and rotates short-lived certificates (often with 1-hour lifespans) for cloud-native applications. This approach eliminates most certificate lifecycle management overhead: ```yaml # SPIRE automatically provisions certificates for Kafka clients apiVersion: v1 kind: Pod metadata: name: kafka-producer spec: serviceAccountName: payment-processor # SPIRE injects certificates via volume mount or Unix domain socket volumes: - name: spire-agent-socket hostPath: path: /run/spire/sockets/agent.sock ``` SPIRE's automatic rotation means applications never handle long-lived certificates, dramatically reducing security risk from compromised keys. **HashiCorp Vault PKI**: Provides a PKI secrets engine that acts as an internal CA, issuing short-lived certificates programmatically. Vault's API-driven approach fits automated deployment pipelines and integrates with Kubernetes via the Vault Agent Injector: ```bash # Issue a 24-hour certificate via Vault API vault write pki/issue/kafka-client common_name="payment-processor.prod" ttl=24h ``` - **AWS Certificate Manager Private CA**: Issues certificates for internal workloads and integrates with AWS Secrets Manager for automated distribution. While primarily designed for AWS environments, ACM Private CA supports external certificate signing requests (CSRs) from Kafka clients running anywhere. - **Modern Keystore Formats**: Note that all modern certificate management tools default to PKCS12 keystores rather than legacy JKS (Java KeyStore). PKCS12 is an industry-standard format that works across languages and platforms, while JKS was deprecated in Java 9. When generating keystores for Kafka, always specify `-storetype PKCS12` with keytool. These tools transform mTLS from an operational burden into a manageable, largely automated component of Kafka infrastructure. ## Governance and Compliance Organizations concerned with data governance, particularly those in regulated industries, need comprehensive visibility into their Kafka security posture beyond just implementing mTLS. While mTLS handles authentication and encryption at the connection level, governance platforms provide operational oversight and compliance enforcement. **Conduktor** offers governance capabilities that complement mTLS authentication by providing: - **Centralized security visibility**: Monitor which certificates are being used, track access patterns, and audit authentication attempts across your entire Kafka infrastructure - **Certificate lifecycle tracking**: Dashboard views of certificate expiration dates, rotation schedules, and compliance with organizational policies - **Policy enforcement**: Ensure all clients use approved certificate authorities and detect unauthorized certificate usage - **Compliance reporting**: Generate audit trails showing that only properly authenticated clients accessed sensitive topics, supporting SOC 2, HIPAA, and GDPR requirements For organizations running Kafka in production with strict security requirements, combining mTLS authentication with governance platforms creates a complete security and compliance solution. mTLS provides the cryptographic foundation, while governance tools ensure policies are consistently enforced and auditable. ## Troubleshooting Common mTLS Issues mTLS configuration errors can be cryptic and difficult to diagnose. Here are the most common issues and their solutions: **Certificate Hostname Mismatch** ``` ERROR: Certificate doesn't match broker address ``` - **Cause**: The broker's certificate Subject Alternative Name (SAN) doesn't match the hostname used to connect. - **Solution**: Ensure the certificate SAN includes all advertised listener addresses. Verify with: ```bash openssl x509 -in server.crt -text -noout | grep "Subject Alternative Name" -A1 ``` **Expired Certificates** ``` ERROR: Certificate has expired ``` - **Cause**: Client or server certificate has passed its validity period. - **Solution**: Check certificate expiration dates before failures occur: ```bash openssl x509 -in client.crt -noout -dates ``` Implement automated monitoring with alerts at 30, 14, and 7 days before expiration. **Truststore Issues** ``` ERROR: unable to find valid certification path ``` - **Cause**: The truststore doesn't contain the CA certificate that signed the peer's certificate, or the trust chain is incomplete. - **Solution**: Verify the truststore contains the correct CA: ```bash keytool -list -keystore kafka.server.truststore.p12 -storepass password ``` Ensure intermediate certificates are included if your CA uses a multi-level hierarchy. **Permission Denied After Successful Authentication** ``` ERROR: Not authorized to access topics ``` - **Cause**: mTLS authentication succeeded, but Kafka ACLs don't grant permissions to the certificate's DN. - **Solution**: Verify the principal extracted from your certificate: ```bash openssl x509 -in client.crt -noout -subject ``` Then check ACLs match this exact principal: ```bash kafka-acls --list --principal "User:CN=payment-processor,OU=Finance,O=Acme Corp" ``` **TLS Protocol Mismatch** ``` ERROR: Received fatal alert: protocol_version ``` - **Cause**: Client and server don't support compatible TLS versions. - **Solution**: Ensure both client and broker enable TLS 1.3 or at minimum TLS 1.2. Never enable TLS 1.0/1.1 (deprecated and insecure). - **Debugging Certificate Chains** For complex certificate issues, enable SSL debugging in your Kafka clients: ```properties # Add to producer/consumer properties ssl.debug=all ``` Or set the JVM property: ```bash -Djavax.net.debug=ssl:handshake:verbose ``` This produces detailed logs showing certificate validation steps, helping identify exactly where authentication fails. ## Summary Mutual TLS is the standard for authenticating Kafka clients in enterprise and regulated environments. The operational complexity is real — certificate infrastructure, automation tooling, lifecycle management — but the security guarantees are worth it for organizations handling sensitive data or running zero-trust architectures. The payoff is cryptographically verified identity for every Kafka client: a solid foundation for authorization policies and regulatory compliance. ## Related Concepts - [Kafka Authentication: SASL, SSL, OAuth](https://www.conduktor.io/glossary/kafka-authentication-sasl-ssl-oauth) - Comprehensive comparison of mTLS with other authentication mechanisms like SASL/SCRAM and OAuth. - [Zero Trust for Streaming](https://www.conduktor.io/glossary/zero-trust-for-streaming) - How mTLS fits into a broader zero-trust security architecture for streaming platforms. - [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka) - TLS encryption configuration that complements mTLS authentication. ## Sources and References - [Apache Kafka Security Documentation](https://kafka.apache.org/documentation/#security) - Official Kafka security configuration and SSL/TLS setup - [Confluent Security Tutorial](https://docs.confluent.io/platform/current/security/security_tutorial.html) - Comprehensive guide to Kafka security including mTLS implementation - [NIST SP 800-52 Rev. 2: Guidelines for TLS Implementations](https://csrc.nist.gov/publications/detail/sp/800-52/rev-2/final) - Federal guidelines for TLS configuration and certificate management - [RFC 5280: Internet X.509 Public Key Infrastructure](https://datatracker.ietf.org/doc/html/rfc5280) - Standard for X.509 certificates used in mTLS - [HashiCorp Vault PKI Secrets Engine](https://developer.hashicorp.com/vault/docs/secrets/pki) - Automated certificate management for distributed systems Source: https://www.conduktor.io/glossary/mtls-for-kafka --- # Multi-Tenancy in Kafka Environments As organizations scale their use of Apache Kafka, they often face a critical decision: should each team or application have its own dedicated Kafka cluster, or can multiple tenants safely share the same infrastructure? Multi-tenancy in Kafka environments offers a way to consolidate resources, reduce operational overhead, and lower costs while maintaining the isolation and security that different teams require. ## What is Multi-Tenancy? Multi-tenancy refers to the architectural approach where a single Kafka deployment serves multiple independent users, teams, or applications, each considered a "tenant." The goal is to share infrastructure resources efficiently while ensuring that tenants remain isolated from each other in terms of data access, performance, and operational impact. In a multi-tenant Kafka environment, different business units might share the same cluster. For example, a financial services company could host data streams for their trading platform, risk management system, and customer analytics on a single cluster, with each system operating as a separate tenant. The alternative to multi-tenancy is deploying dedicated clusters for each use case. While this provides maximum isolation, it significantly increases infrastructure costs, operational complexity, and management overhead. For background on Kafka's architecture and capabilities, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ## Multi-Tenancy Models in Kafka Organizations can implement multi-tenancy in Kafka using different models, each with distinct trade-offs: ![Organizations can implement multi-tenancy in Kafka using different models, each with distinct trade-offs](https://www.conduktor.io/assets/images/glossary/multi-tenancy-in-kafka-environments-0.webp) ### Logical Separation (Soft Multi-Tenancy) In this model, all tenants share the same physical Kafka cluster but are separated through logical boundaries. Tenants use different topic namespaces, consumer groups, and access controls. For example, topics might follow a naming convention like `team-a.orders` and `team-b.orders`, with ACLs preventing Team A from accessing Team B's data. This approach maximizes resource utilization and minimizes infrastructure costs. However, tenants still share underlying brokers, network bandwidth, and storage. This means one tenant's behavior can potentially impact others, a problem known as the "noisy neighbor" effect, where one tenant's excessive resource consumption (high throughput, many connections, or disk usage) degrades performance for other tenants sharing the same infrastructure. ### Physical Separation (Hard Multi-Tenancy) Physical separation involves deploying dedicated Kafka clusters for each tenant or group of tenants. This provides strong isolation guarantees and eliminates noisy neighbor problems. Organizations often choose this model when regulatory requirements mandate complete data separation, or when performance SLAs are critical. The downside is increased cost and operational complexity, as each cluster requires separate management, monitoring, and maintenance. ### Hybrid Approach Many enterprises adopt a hybrid model, using logical separation for most use cases while reserving dedicated clusters for high-criticality or regulated workloads. This balances cost efficiency with isolation requirements. **Virtual Clusters for Enhanced Multi-Tenancy**: Conduktor's Virtual Clusters feature provides an abstraction layer that creates isolated logical clusters within a single physical Kafka cluster, combining the cost benefits of soft multi-tenancy with stronger isolation guarantees. For detailed implementation guidance, see the [Virtual Clusters documentation](https://docs.conduktor.io/guide/conduktor-concepts/virtual-clusters). ### KRaft Mode Benefits for Multi-Tenancy (Kafka 4.0+) Kafka's KRaft mode (production-ready in Kafka 4.0+) brings significant advantages for multi-tenant deployments: - **Metadata Performance**: KRaft eliminates ZooKeeper dependency, enabling much faster metadata operations. This is critical in multi-tenant environments where frequent topic creation, ACL updates, and partition reassignments are common across many tenants. - **Higher Partition Scalability**: KRaft clusters can efficiently handle 500,000+ partitions, compared to ~200,000 with ZooKeeper. This allows more tenants to coexist on a single cluster or enables more granular topic partitioning strategies. - **Faster Cluster Operations**: Controller failover, partition leader elections, and configuration changes complete in milliseconds rather than seconds. This reduces the impact of administrative operations on tenant workloads. - **Simplified Operations**: Removing ZooKeeper reduces operational complexity, infrastructure costs, and the failure domains in multi-tenant clusters, one less distributed system to manage. For detailed information on KRaft mode, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ## Isolation Strategies Effective multi-tenancy requires multiple layers of isolation to protect tenants from each other: ### Namespace Isolation The foundation of logical multi-tenancy is namespace organization. Teams should adopt clear naming conventions for topics, consumer groups, and other resources. A common pattern is prefixing resources with team or application identifiers, such as `finance.transactions` or `marketing.events`. This makes it easier to apply access controls and track resource ownership at scale. ### Access Control Lists (ACLs) Kafka's ACL system controls which principals (users or applications) can perform specific operations on resources. A principal is an authenticated identity in Kafka, such as a user account or service account, that can be granted permissions. In a multi-tenant environment, ACLs ensure that tenants can only access their own topics and consumer groups. For example, you might configure ACLs so that the `analytics-team` principal has read and write access to topics matching `analytics.*`, but no access to `finance.*` topics. Governance platforms can simplify ACL management by providing visual interfaces to create, review, and audit permissions across tenants, reducing the risk of misconfiguration. For comprehensive coverage of ACL patterns and best practices, see [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). ### Quotas and Rate Limiting Kafka supports quotas to limit the resources any single client can consume. Quotas can restrict: - **Producer byte rate**: How much data a producer can send per second - **Consumer byte rate**: How much data a consumer can fetch per second - **Request rate**: Number of requests per second (Kafka 0.11+) - **Controller mutation rate**: Rate of metadata changes like topic creation or ACL updates (Kafka 3.3+, KRaft-specific) By setting quotas per tenant, platform teams can prevent one tenant from monopolizing cluster resources. For instance, if Team A starts producing data at 100 MB/s, quotas ensure they don't degrade performance for Team B's applications. ```bash # Modern quota configuration using kafka-configs (dynamic, no broker restart) # Set producer quota for tenant-a user kafka-configs.sh --bootstrap-server localhost:9092 \ --alter --add-config 'producer_byte_rate=10485760,consumer_byte_rate=20971520' \ --entity-type users --entity-name tenant-a # Set controller mutation quota (KRaft mode, Kafka 3.3+) kafka-configs.sh --bootstrap-server localhost:9092 \ --alter --add-config 'controller_mutation_rate=10' \ --entity-type users --entity-name tenant-a # Set request rate quota (Kafka 0.11+) kafka-configs.sh --bootstrap-server localhost:9092 \ --alter --add-config 'request_percentage=50' \ --entity-type users --entity-name tenant-a ``` Modern Kafka (3.0+) allows dynamic quota updates without broker restarts, enabling real-time adjustment based on tenant behavior or SLA changes. For detailed quota implementation patterns, see [Quotas and Rate Limiting in Kafka](https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka). ## Security and Access Control Security is paramount in multi-tenant environments where sensitive data from different business units coexists. ### Authentication Kafka supports multiple authentication mechanisms including SASL/PLAIN, SASL/SCRAM, SASL/OAUTHBEARER (OAuth/OIDC), Kerberos, and mutual TLS (mTLS). Each tenant should have unique credentials, preventing unauthorized access if one tenant's credentials are compromised. Modern deployments (Kafka 2.0+) increasingly use OAuth/OIDC for authentication, enabling integration with enterprise identity providers and centralized access management. For more on securing Kafka with mutual TLS, see [mTLS for Kafka](https://www.conduktor.io/glossary/mtls-for-kafka). ### Authorization Beyond authentication, authorization through ACLs ensures tenants can only perform permitted operations. Role-based access control (RBAC) can simplify this by assigning roles like "producer," "consumer," or "admin" to users. Modern governance platforms like Conduktor provide advanced authorization capabilities, including centralized policy management, visual ACL editors, and audit trails for tracking permission changes across tenants. ### Encryption Data in transit should be encrypted using TLS to prevent eavesdropping. For highly sensitive use cases, consider encryption at rest as well, though this can impact performance. ## Performance and Resource Management Multi-tenancy introduces challenges around resource contention and performance predictability. ### Preventing Noisy Neighbors Without proper controls, one tenant's workload can impact others. Common scenarios include: - A producer sending data faster than the cluster can handle - A consumer creating lag that increases disk usage - Misconfigured applications creating excessive connections Quotas are the primary defense against noisy neighbors, but monitoring is equally important. Platform teams should track metrics like produce/fetch rates, disk usage, and connection counts per tenant to identify problems early. ### Capacity Planning Multi-tenant clusters require careful capacity planning. Consider: - The number of partitions (Kafka 4.0+ with KRaft can handle 500,000+ partitions per cluster, a significant improvement over ZooKeeper-based deployments) - Network throughput requirements across all tenants - Storage growth rates and retention policies - The number of concurrent connections KRaft mode (Kafka 3.0+, production-ready in 4.0+) dramatically improves metadata performance for multi-tenant clusters, enabling faster topic creation, partition rebalancing, and controller operations, critical for environments with many tenants. Tools that provide tenant-level observability can help platform teams understand usage patterns and plan capacity accordingly. For comprehensive capacity planning guidance, see [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning). ## Operational Challenges and Best Practices Running multi-tenant Kafka environments introduces operational complexity: ### Self-Service with Guardrails Teams want autonomy to create topics and manage their data streams, but without governance, clusters can become chaotic. Implementing self-service portals with built-in policy enforcement helps balance autonomy with control. Governance platforms like Conduktor enable teams to request topic creation through a UI while automatically enforcing policies like naming conventions, replication factors, and retention limits. This reduces the burden on platform teams while maintaining standards. For implementing self-service with policy-based governance, see Conduktor's [Self-Service documentation](https://docs.conduktor.io/guide/conduktor-concepts/self-service). **Testing Multi-Tenant Isolation**: Conduktor Gateway provides proxy-based capabilities to test multi-tenant behavior, including: - Simulating quota enforcement and throttling - Testing tenant isolation under chaos conditions - Validating ACL configurations without impacting production - Intercepting and filtering messages at the tenant level This enables platform teams to validate multi-tenancy strategies before deploying to production. ### Monitoring and Observability In multi-tenant environments, monitoring must provide both cluster-wide and tenant-specific views. Platform teams need cluster health metrics, while individual tenants need visibility into their own resource usage and application performance. Tagging resources by tenant and using tenant-aware dashboards helps teams quickly identify and troubleshoot issues without sifting through unrelated data. ### Schema Management When multiple teams produce data to shared clusters, schema management becomes critical. Using a schema registry with permissions aligned to tenants prevents incompatible schema changes from breaking downstream consumers. In multi-tenant environments, the schema registry itself should support namespace isolation, ensuring tenants can only modify schemas for their own topics. For comprehensive coverage of schema management strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). ### Cost Allocation Organizations often want to charge back infrastructure costs to the teams using it. Tracking resource consumption (throughput, storage, partition count) per tenant enables fair cost allocation and encourages efficient resource use. ## Multi-Tenancy in the Streaming Ecosystem Multi-tenancy considerations extend beyond Kafka to the broader streaming ecosystem. Apache Flink, ksqlDB, and other stream processing frameworks must also support tenant isolation when operating on shared infrastructure. For Flink, this might involve namespace separation for jobs, resource quotas per tenant, and isolated state backends. The principles are similar to Kafka: balance shared infrastructure efficiency with the isolation requirements of each workload. Platforms like Conduktor provide comprehensive multi-tenancy features out of the box, including quota management, ACLs, monitoring, and self-service governance portals. Cloud-based options like AWS MSK also offer multi-tenancy capabilities. However, understanding these concepts remains essential for designing secure and performant streaming architectures. ## Summary Multi-tenancy in Kafka environments enables organizations to consolidate infrastructure, reduce costs, and simplify operations while serving multiple teams and applications. Success requires careful implementation of isolation strategies including namespace organization, ACLs, quotas, and security controls. The choice between logical and physical separation depends on your specific requirements around cost, isolation, and compliance. Most organizations benefit from a hybrid approach that uses shared clusters where appropriate and dedicated clusters for critical workloads. Key best practices include: - Enforce clear naming conventions and resource tagging - Implement quotas to prevent noisy neighbor problems - Use authentication and authorization to secure tenant data - Provide self-service capabilities with governance guardrails - Monitor both cluster-wide and tenant-specific metrics - Plan capacity based on aggregate tenant requirements With thoughtful design and proper tooling, multi-tenant Kafka deployments can deliver the benefits of shared infrastructure without compromising on isolation, security, or performance. ## Related Concepts - [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns) - Essential ACL patterns for implementing tenant isolation and access control in multi-tenant environments - [Quotas and Rate Limiting in Kafka](https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka) - Quota mechanisms that prevent noisy neighbor problems and ensure fair resource allocation across tenants - [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming) - Comprehensive access control strategies for securing multi-tenant streaming platforms ## Sources and References 1. [Confluent - Multi-Tenancy Deployment Options](https://docs.confluent.io/platform/current/kafka/deployment.html) - Official documentation on Kafka multi-tenancy patterns and best practices. 2. [Apache Kafka Documentation - Security](https://kafka.apache.org/documentation/#security) - Comprehensive guide to Kafka's authentication, authorization, and encryption capabilities. 3. [Uber Engineering - Scaling Kafka at Uber](https://www.uber.com/blog/kafka/) - Case study detailing Uber's approach to multi-tenant Kafka infrastructure at massive scale. 4. [Kafka Quotas Documentation](https://kafka.apache.org/documentation/#design_quotas) - Technical specification of Kafka's quota mechanisms for resource management. 5. [AWS MSK - Best Practices for Multi-Tenancy](https://docs.aws.amazon.com/msk/latest/developerguide/best-practices.html) - Cloud provider guidance on implementing multi-tenant Kafka deployments. Source: https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments --- # NewSQL Databases: Distributed SQL for Real-Time Applications For decades, organizations faced a difficult choice when selecting databases. Traditional relational databases (RDBMS) like PostgreSQL and MySQL offered strong ACID guarantees and familiar SQL interfaces, but struggled to scale horizontally. [NoSQL databases](https://www.conduktor.io/glossary/nosql-databases-real-time) like Cassandra and MongoDB provided massive scalability, but sacrificed transactional consistency and the powerful querying capabilities of SQL. NewSQL databases address this tradeoff directly: they provide the familiar SQL interface and ACID guarantees of traditional RDBMS alongside the horizontal scalability and fault tolerance of NoSQL systems. As [real-time data streaming](https://www.conduktor.io/glossary/what-is-real-time-data-streaming) becomes essential for modern applications, NewSQL databases provide the transactional guarantees and scale needed to power [event-driven architectures](https://www.conduktor.io/glossary/event-driven-architecture). ## What is NewSQL? NewSQL is a category of modern relational database management systems that provide the same scalable performance as NoSQL systems while maintaining the ACID guarantees and SQL interface of traditional databases. The term was coined around 2011 to describe this new generation of distributed databases. ![NewSQL architecture bridging traditional RDBMS and NoSQL](https://www.conduktor.io/assets/images/glossary/newsql-databases-streaming-0.webp) The core promise of NewSQL is simple: you shouldn't have to choose between consistency and scalability. These systems achieve this through modern distributed systems techniques, including: - **Distributed consensus protocols** like Raft or Paxos (algorithms that coordinate agreement across multiple nodes) for maintaining consistency across nodes - **Automatic sharding** that distributes data across multiple machines transparently - **Global transaction coordination** that ensures ACID properties across distributed data - **Multi-version concurrency control (MVCC)** for high-performance reads without blocking writes. MVCC keeps multiple versions of data simultaneously, allowing readers to access consistent snapshots while writers continue making changes Unlike traditional databases that scale vertically (adding more resources to a single machine), NewSQL databases scale horizontally by adding more commodity servers to the cluster. ## Key NewSQL Characteristics ### Distributed Architecture NewSQL databases are designed from the ground up for distributed deployment. Data is automatically partitioned (sharded) across multiple nodes, with each shard replicated for fault tolerance. If a node fails, the system automatically reroutes requests to healthy replicas without data loss or downtime. ### Strong Consistency Despite being distributed, NewSQL databases maintain strong consistency guarantees. Unlike eventually consistent NoSQL systems where different nodes might temporarily show different values, NewSQL databases ensure that all clients see the same view of the data. This is achieved through distributed consensus protocols that coordinate writes across replicas. ### Horizontal Scalability NewSQL databases can grow by adding more machines to the cluster. This happens transparently, applications don't need to be rewritten when the cluster expands. The database automatically rebalances data across the new nodes and adjusts query routing. ### SQL Compatibility NewSQL databases support standard SQL interfaces, making them compatible with existing tools, ORMs, and developer skills. Most implement large subsets of the SQL standard, including joins, transactions, indexes, and constraints. ## Popular NewSQL Databases ### CockroachDB Built on Google's Spanner design, CockroachDB uses a distributed key-value architecture with a SQL layer on top. It emphasizes survivability, automatically replicating and rebalancing data across geographically distributed nodes. CockroachDB uses a multi-active availability model, meaning all nodes can accept both reads and writes. **2025 Capabilities**: CockroachDB Serverless offers elastic scaling with pay-per-use pricing, ideal for variable streaming workloads. Multi-region SQL capabilities allow fine-grained control over data placement, enabling low-latency reads near stream consumers while maintaining global consistency. The distributed SQL optimizer includes cost-based query planning that adapts to streaming write patterns. ### YugabyteDB YugabyteDB combines PostgreSQL's query layer with a distributed storage engine inspired by Apache HBase. It offers both SQL and Cassandra-compatible APIs. YugabyteDB focuses on PostgreSQL compatibility, making migration from traditional PostgreSQL databases relatively straightforward. **2025 Enhancements**: Improved distributed SQL optimizer reduces latency for complex analytical queries on streaming data. Native support for row-level geo-partitioning enables compliance with data residency requirements while processing global event streams. Point-in-time recovery (PITR) enhancements provide sub-second granularity for recovering from data quality incidents. ### TiDB Developed by PingCAP, TiDB uses a layered architecture with stateless SQL nodes (TiDB) and a distributed transactional key-value store (TiKV). It's MySQL-compatible and particularly popular in the APAC region. TiDB separates storage (TiKV) from compute (TiDB layer), allowing independent scaling. **2025 Features**: Resource Control groups allow prioritization of streaming ingestion workloads versus analytical queries on the same cluster. Placement Policies provide declarative control over data location, useful for processing region-specific event streams. TiFlash (columnar storage extension) enables real-time OLAP on freshly ingested streaming data without ETL. ### Google Spanner The original inspiration for many NewSQL systems, Spanner is Google's globally distributed database service. It uses atomic clocks and GPS to provide external consistency across data centers worldwide. While proprietary to Google Cloud, Spanner's design papers have influenced the entire NewSQL category. ### Emerging Serverless Options - **Neon** provides serverless PostgreSQL with branching capabilities, enabling instant creation of database copies for testing streaming pipelines. Its separation of storage and compute allows elastic scaling that matches streaming ingestion patterns. - **PlanetScale** offers serverless MySQL built on Vitess, with non-blocking schema changes that integrate seamlessly with streaming workflows. Schema changes propagate without downtime, critical for evolving event schemas in production streams. ## NewSQL in Streaming Architectures NewSQL databases play several critical roles in real-time streaming architectures. ### Change Data Capture (CDC) NewSQL databases can serve as source systems for event streams through [Change Data Capture (CDC)](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). Tools like [Debezium](https://www.conduktor.io/glossary/implementing-cdc-with-debezium) can capture row-level changes from databases like CockroachDB and YugabyteDB, publishing them to Kafka topics. Because NewSQL databases support transactions, CDC can capture changes in a consistent, ordered manner that preserves transactional boundaries, meaning if three table updates happen in one database transaction, CDC ensures all three changes are captured as a cohesive unit with the same transaction ID. **Example: Financial Transaction CDC** Consider a money transfer between two accounts within a NewSQL database. The transaction updates two rows atomically: ```sql BEGIN; UPDATE accounts SET balance = balance - 100 WHERE account_id = 'A'; UPDATE accounts SET balance = balance + 100 WHERE account_id = 'B'; COMMIT; ``` Debezium captures both updates from the transaction log and publishes them to Kafka with metadata indicating they belong to the same transaction. Downstream consumers can reconstruct the transactional boundary, ensuring the transfer is processed atomically in the streaming pipeline. For scenarios requiring guaranteed event publishing alongside database writes, consider the [Outbox pattern](https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing), which uses CDC to reliably publish events from an outbox table within the same transaction as business data updates. **CockroachDB CDC Configuration** CockroachDB provides native CDC through changefeeds that stream row-level changes directly to Kafka: ```sql -- Create an enterprise changefeed for CockroachDB CREATE CHANGEFEED FOR TABLE accounts, transactions INTO 'kafka://kafka-broker:9092' WITH updated, resolved = '10s', format = 'avro', confluent_schema_registry = 'http://schema-registry:8081'; ``` This changefeed configuration: - Monitors the `accounts` and `transactions` tables - Publishes changes to Kafka in Avro format - Includes full row data after updates (not just changed columns) - Emits resolved timestamps every 10 seconds for watermark tracking **YugabyteDB CDC with Debezium** For YugabyteDB, which is PostgreSQL-compatible, use the Debezium PostgreSQL connector: ```json { "name": "yugabyte-cdc-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "yugabyte-host", "database.port": "5433", "database.user": "debezium_user", "database.password": "secret", "database.dbname": "ecommerce", "database.server.name": "yugabyte_prod", "table.include.list": "public.orders,public.order_items", "plugin.name": "pgoutput", "publication.autocreate.mode": "filtered", "transforms": "unwrap", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "false" } } ``` The configuration captures changes from the `orders` and `order_items` tables, using YugabyteDB's native PostgreSQL replication protocol. The `ExtractNewRecordState` transform simplifies the CDC payload, extracting just the new row state rather than the full before/after envelope. ### Real-Time OLTP NewSQL databases excel as the operational data store for real-time applications that consume from streams. Consider an inventory management system that processes order events from Kafka. The system needs to: - Update inventory counts transactionally - Enforce constraints (no overselling) - Serve real-time queries about current stock levels - Scale to handle high order volumes A NewSQL database can handle all these requirements while maintaining ACID guarantees that prevent inventory inconsistencies, even under high concurrency. **Example: Kafka Consumer to NewSQL** This Java example shows a Kafka consumer writing to CockroachDB with proper transaction handling and idempotency: ```java import org.apache.kafka.clients.consumer.*; import java.sql.*; import java.util.*; public class InventoryStreamProcessor { private final Connection dbConnection; private final KafkaConsumer consumer; public void processOrderStream() { consumer.subscribe(Collections.singletonList("orders")); while (true) { ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord record : records) { try { // Use Kafka offset as idempotency key processOrderWithIdempotency(record); // Commit offset only after successful DB write consumer.commitSync(Collections.singletonMap( new TopicPartition(record.topic(), record.partition()), new OffsetAndMetadata(record.offset() + 1) )); } catch (SQLException e) { // Handle DB errors with retry or DLQ handleProcessingError(record, e); } } } } private void processOrderWithIdempotency( ConsumerRecord record) throws SQLException { dbConnection.setAutoCommit(false); try { // Check if already processed using offset tracking table String checkSql = """ SELECT 1 FROM processed_offsets WHERE topic = ? AND partition = ? AND offset = ? """; try (PreparedStatement ps = dbConnection.prepareStatement(checkSql)) { ps.setString(1, record.topic()); ps.setInt(2, record.partition()); ps.setLong(3, record.offset()); ResultSet rs = ps.executeQuery(); if (rs.next()) { dbConnection.rollback(); return; // Already processed } } // Deduct inventory with constraint check String updateSql = """ UPDATE inventory SET quantity = quantity - ?, updated_at = NOW() WHERE product_id = ? AND quantity >= ? """; try (PreparedStatement ps = dbConnection.prepareStatement(updateSql)) { ps.setInt(1, record.value().quantity); ps.setString(2, record.value().productId); ps.setInt(3, record.value().quantity); int updated = ps.executeUpdate(); if (updated == 0) { // Insufficient inventory publishToOutOfStockTopic(record.value()); dbConnection.rollback(); return; } } // Record processed offset String insertOffsetSql = """ INSERT INTO processed_offsets (topic, partition, offset, processed_at) VALUES (?, ?, ?, NOW()) """; try (PreparedStatement ps = dbConnection.prepareStatement(insertOffsetSql)) { ps.setString(1, record.topic()); ps.setInt(2, record.partition()); ps.setLong(3, record.offset()); ps.executeUpdate(); } dbConnection.commit(); } catch (SQLException e) { dbConnection.rollback(); throw e; } } } ``` **Key Patterns Demonstrated:** 1. **Idempotency**: The `processed_offsets` table tracks which Kafka messages have been processed, preventing duplicate inventory deductions if the consumer restarts. 2. **Atomic Operations**: All updates (inventory, offset tracking) happen in a single database transaction, ensuring consistency. 3. **Constraint Enforcement**: The `WHERE quantity >= ?` clause in the UPDATE prevents overselling, leveraging NewSQL's transaction guarantees. 4. **Manual Offset Management**: Offsets are committed only after successful database commits, ensuring at-least-once processing without data loss. ### Event Sourcing and CQRS In [event sourcing](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka) architectures, NewSQL databases often serve as the read model (query side) in [CQRS patterns](https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka). Events from Kafka are consumed and projected into queryable tables that support complex analytical queries. The distributed nature of NewSQL databases allows these read models to scale independently of the write-optimized event log. - **Event Sourcing Concept Primer**: In event sourcing, every state change is captured as an immutable event in Kafka. Instead of updating database records directly, you append events to a log. To determine current state, you replay events from the beginning (or from a snapshot). CQRS (Command Query Responsibility Segregation) separates write operations (commands that generate events) from read operations (queries against optimized read models). - **CQRS Read Model with NewSQL** Consider an e-commerce system where all orders are event-sourced in Kafka. The command side writes events (`OrderPlaced`, `PaymentProcessed`, `OrderShipped`), while the read side maintains a queryable view in a NewSQL database: ```python from kafka import KafkaConsumer import psycopg2 # Works with YugabyteDB (PostgreSQL-compatible) import json # Consumer for YugabyteDB read model projection consumer = KafkaConsumer( 'order-events', bootstrap_servers='kafka:9092', value_deserializer=lambda m: json.loads(m.decode('utf-8')), group_id='order-read-model-projector' ) # Connection to YugabyteDB conn = psycopg2.connect( host='yugabyte-host', port=5433, database='ecommerce', user='app_user', password='secret' ) def project_event_to_read_model(event): """ Projects Kafka events into a denormalized read model optimized for queries """ cursor = conn.cursor() if event['type'] == 'OrderPlaced': # Create denormalized order view cursor.execute(""" INSERT INTO order_read_model ( order_id, customer_id, total_amount, status, created_at, updated_at ) VALUES (%s, %s, %s, 'PENDING', %s, %s) ON CONFLICT (order_id) DO UPDATE SET updated_at = EXCLUDED.updated_at """, (event['orderId'], event['customerId'], event['totalAmount'], event['timestamp'], event['timestamp']) ) elif event['type'] == 'PaymentProcessed': # Update order status cursor.execute(""" UPDATE order_read_model SET status = 'PAID', payment_method = %s, updated_at = %s WHERE order_id = %s """, (event['paymentMethod'], event['timestamp'], event['orderId']) ) elif event['type'] == 'OrderShipped': # Update with shipping info cursor.execute(""" UPDATE order_read_model SET status = 'SHIPPED', tracking_number = %s, shipped_at = %s, updated_at = %s WHERE order_id = %s """, (event['trackingNumber'], event['timestamp'], event['timestamp'], event['orderId']) ) conn.commit() cursor.close() # Process events continuously for message in consumer: project_event_to_read_model(message.value) ``` This pattern leverages NewSQL's strengths: - **Complex Queries**: The denormalized `order_read_model` table supports rich analytics (`SELECT status, COUNT(*) FROM order_read_model WHERE created_at > NOW() - INTERVAL '7 days' GROUP BY status`) - **Scalability**: Read models can be partitioned geographically using NewSQL's distributed capabilities - **Consistency**: Each projection update is transactional, preventing partial updates - **Rebuild Capability**: If the read model schema changes, replay all events from Kafka to rebuild it ### Stream Processing State Stores Modern stream processors like [Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams) and [Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) typically use embedded state stores (RocksDB) for optimal performance. However, NewSQL databases serve as external state stores in specific scenarios: **When to Use External NewSQL State**: 1. **Multi-Application Access**: When multiple independent applications need to query the same stateful data. Embedded state stores are isolated to individual stream processors. 2. **Interactive SQL Analytics**: When data scientists or analysts need ad-hoc SQL access to streaming state for debugging or analysis. 3. **Cross-Region State Sharing**: When stream processors deployed across multiple regions need access to consistent, globally distributed state. 4. **State Beyond Stream Processor Lifecycle**: When state must persist independently of stream processor restarts or redeployments. - **2025 Context**: Kafka Streams' RocksDB state stores have become highly optimized with incremental snapshots, compression, and fast recovery. Apache Flink's managed state with incremental checkpoints to object storage (S3, GCS) provides durability without external databases. External NewSQL state stores are now reserved for the specific scenarios above, rather than being a default pattern. - **Trade-offs**: External state stores introduce network latency for every state access. For high-throughput, low-latency stream processing, embedded state remains superior. Use external NewSQL state when queryability and multi-consumer access outweigh per-record latency concerns. ## Streaming Integration Best Practices ### Connection Pooling for High-Throughput Consumers Kafka consumers processing thousands of messages per second need efficient database connection management. NewSQL databases handle connection pooling similarly to traditional databases, but with distributed considerations. **HikariCP Configuration for CockroachDB:** ```java import com.zaxxer.hikari.HikariConfig; import com.zaxxer.hikari.HikariDataSource; public class NewSQLConnectionPool { public static HikariDataSource createPool() { HikariConfig config = new HikariConfig(); config.setJdbcUrl("jdbc:postgresql://cockroachdb-load-balancer:26257/app"); config.setUsername("app_user"); config.setPassword("secret"); // Pool sizing for streaming workloads // Rule of thumb: (num_kafka_consumer_threads * 2) + overhead config.setMaximumPoolSize(20); config.setMinimumIdle(10); // Connection lifetime and timeout config.setMaxLifetime(600000); // 10 minutes config.setConnectionTimeout(5000); // 5 seconds config.setIdleTimeout(300000); // 5 minutes // Distributed database considerations config.addDataSourceProperty("ApplicationName", "kafka-consumer-app"); config.addDataSourceProperty("reWriteBatchedInserts", "true"); config.setLeakDetectionThreshold(60000); // Validation query for CockroachDB health checks config.setConnectionTestQuery("SELECT 1"); return new HikariDataSource(config); } } ``` **Key Considerations:** - **Pool Size**: Match connection pool size to Kafka consumer thread count. Over-provisioning wastes database resources; under-provisioning causes contention. - **Connection Lifetime**: NewSQL databases rebalance nodes dynamically. Periodic connection refresh ensures load distribution across the cluster. - **Geographic Awareness**: For multi-region NewSQL deployments, configure connection pools per region to minimize cross-region latency. ### Idempotency and Exactly-Once Semantics Achieving [exactly-once semantics](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) when writing from Kafka to NewSQL requires careful coordination. The earlier inventory example demonstrated offset tracking tables. Here's an alternative using Kafka transactions with database writes: **Transactional Writes Pattern:** When using Kafka's transactional producer alongside NewSQL, you can implement a two-phase commit coordination: ```java // Producer commits to Kafka only after DB commit succeeds dbConnection.setAutoCommit(false); producer.beginTransaction(); try { // Write to database updateInventory(dbConnection, order); // Send result event to output topic producer.send(new ProducerRecord<>("inventory-updated", key, value)); // Commit both atomically (from consumer perspective) dbConnection.commit(); producer.commitTransaction(); } catch (Exception e) { dbConnection.rollback(); producer.abortTransaction(); throw e; } ``` This pattern works well when the NewSQL database acts as both a sink and source for derived events. For pure sink scenarios, the offset tracking approach (shown in the inventory example) remains more straightforward. ### Schema Evolution Strategies Streaming systems with NewSQL databases face a unique challenge: Kafka schemas (Avro, Protobuf) evolve independently of database schemas. Coordination prevents runtime errors. **Schema Evolution Patterns:** 1. **Backward Compatible Database Changes**: When adding optional columns to NewSQL tables, existing Kafka consumers can continue writing without code changes: ```sql -- Safe: Add nullable column ALTER TABLE orders ADD COLUMN loyalty_points INT DEFAULT 0; -- Safe: Add column with default ALTER TABLE customers ADD COLUMN email_verified BOOLEAN DEFAULT FALSE; ``` 2. **Forward Compatible Stream Schema Changes**: When Kafka message schemas add new fields, consumers using older database schemas must handle gracefully: ```java // Consumer handles new fields not yet in database public void processOrder(OrderEvent event) { // Insert only fields that exist in current DB schema String sql = "INSERT INTO orders (order_id, customer_id, total) VALUES (?, ?, ?)"; // event.getLoyaltyPoints() exists in new schema but ignored until DB migration } ``` 3. **Coordinated Schema Migration**: For breaking changes, use a multi-phase deployment: - **Phase 1**: Deploy consumers that accept both old and new message formats - **Phase 2**: Migrate database schema with dual-write support - **Phase 3**: Update producers to new schema - **Phase 4**: Remove backward compatibility code **Schema Registry Integration**: Use [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management) with compatibility rules that match your database evolution strategy. Set `BACKWARD` compatibility for additive-only changes, `FULL` for bidirectional compatibility during migrations. ## NewSQL vs Traditional Databases and NoSQL | Aspect | Traditional RDBMS | NoSQL | NewSQL | |--------|------------------|-------|---------| | **Scalability** | Vertical (limited) | Horizontal (unlimited) | Horizontal (unlimited) | | **Consistency** | Strong (ACID) | Eventual (typically) | Strong (ACID) | | **Query Language** | SQL | Varies | SQL | | **Transaction Support** | Full ACID | Limited/None | Full ACID | | **CAP Trade-off** | CP (availability suffers during partitions) | AP (consistency suffers) | CP (optimized for availability) | | **Use Cases** | Traditional OLTP | High-volume writes, flexible schema | Distributed OLTP, real-time apps | ### CAP Theorem Considerations The CAP theorem states that distributed systems can provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. NewSQL databases choose consistency and partition tolerance (CP), prioritizing correctness over availability during network splits. However, they use advanced techniques to maximize availability: - Multi-region replication for geographic redundancy - Automatic failover with minimal downtime (typically seconds) - Read replicas for distributing query load without impacting write performance - Sophisticated consensus protocols (Raft, Multi-Paxos) that minimize coordination overhead While they may have slightly lower availability during network partitions compared to AP systems like [Cassandra](https://www.conduktor.io/glossary/nosql-databases-real-time), modern NewSQL databases achieve 99.95%+ availability, sufficient for most production requirements. For comparison with NoSQL trade-offs, see [NoSQL Databases for Real-Time Streaming](https://www.conduktor.io/glossary/nosql-databases-real-time). ## When to Choose NewSQL ### Ideal Use Cases NewSQL databases are particularly well-suited for: - **Financial Services**: Transaction processing that requires strong consistency, such as payment systems, trading platforms, and account management. The combination of ACID guarantees and horizontal scalability makes NewSQL ideal for [financial workloads](https://www.conduktor.io/glossary/streaming-data-in-financial-services) that can't tolerate inconsistency but need to scale globally. For complex multi-service financial transactions, combine NewSQL with the [Saga pattern](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions). - **Inventory and Supply Chain**: Real-time [inventory management](https://www.conduktor.io/glossary/supply-chain-visibility-with-real-time-streaming) where overselling prevention is critical. NewSQL transactions ensure that concurrent order processing doesn't violate stock constraints, while scalability handles peak shopping periods. - **User-Facing Applications**: Social platforms, SaaS applications, and gaming systems that need responsive queries, transactional updates, and the ability to scale with user growth. The SQL interface accelerates development with familiar tools and patterns. - **IoT and Time-Series**: Ingesting sensor data at scale while supporting complex analytical queries. NewSQL databases can handle high write throughput while maintaining the ability to run sophisticated SQL analytics on [IoT streams](https://www.conduktor.io/glossary/iot-data-streaming-architectures). - **Multi-Region Deployments**: Applications serving global users with low-latency requirements across regions. NewSQL databases like CockroachDB and Spanner can place data close to users while maintaining consistency. ### Decision Criteria Choose NewSQL when you need: 1. **Strong consistency** across distributed data 2. **SQL interface** for developer productivity and tool compatibility 3. **Horizontal scalability** beyond what a single database server can provide 4. **High availability** with automatic failover 5. **Complex transactions** involving multiple rows or tables 6. **Regulatory compliance** requiring ACID guarantees and audit trails Consider alternatives when: - Simple key-value access patterns dominate (consider [NoSQL](https://www.conduktor.io/glossary/nosql-databases-real-time)) - Eventual consistency is acceptable (consider NoSQL) - You don't need horizontal scalability (consider traditional RDBMS) - Cost is the primary concern and scale is limited (traditional RDBMS is typically cheaper) ### Governance and Compliance In regulated industries, NewSQL databases offer advantages for [data governance](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities). When NewSQL databases feed streams via [CDC](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals), governance tooling ensures that sensitive database changes are properly classified, masked, and audited as they flow through the streaming platform. **Governance Integration Patterns:** 1. **Schema Governance**: Use [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management) to enforce compatibility rules between CDC event schemas and downstream consumers. 2. **Access Control**: Implement [data access controls](https://www.conduktor.io/glossary/data-access-control-rbac-and-abac) at both the NewSQL database layer (row-level security) and streaming layer (Kafka ACLs, governance tooling). 3. **Data Masking**: Apply [masking and anonymization](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming) to CDC streams containing PII before downstream consumers access sensitive fields. 4. **Audit Logging**: Combine NewSQL transaction logs with [streaming audit logs](https://www.conduktor.io/glossary/streaming-audit-logs) for comprehensive compliance reporting. 5. **Data Quality**: Monitor [data quality dimensions](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) (completeness, accuracy, consistency) as data flows from NewSQL sources through streaming pipelines. The combination of transactional guarantees in NewSQL and streaming governance creates end-to-end data integrity, from the database transaction log through Kafka topics to downstream consumers. Tools like Conduktor provide centralized governance for streaming platforms, offering policy enforcement, audit logging of client connections, and compliance auditing across your entire NewSQL-to-stream-to-consumer pipeline. ## Summary NewSQL databases offer distributed ACID transactions and horizontal scalability together — without the consistency trade-offs of NoSQL. For real-time streaming architectures, they work as systems of record that can source and sink event streams while maintaining strong transactional guarantees. The trade-offs to weigh: slightly higher latency than pure NoSQL, and higher cost than a single-server RDBMS. When correctness and scale are both required, NewSQL is usually the right answer. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - The streaming platform that NewSQL databases integrate with for CDC and event-driven architectures - [NoSQL Databases for Real-Time Streaming: Patterns and Integration](https://www.conduktor.io/glossary/nosql-databases-real-time) - Alternative database architectures favoring eventual consistency and higher throughput - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) - Critical for managing transactional data across distributed systems ## Sources and References - [CockroachDB Architecture Documentation](https://www.cockroachlabs.com/docs/stable/architecture/overview.html) - Detailed architecture of a leading NewSQL database - [Google Spanner: TrueTime and External Consistency](https://research.google/pubs/pub39966/) - Original research paper on Spanner's distributed transaction model - [YugabyteDB Documentation](https://docs.yugabyte.com/preview/architecture/) - Architecture and design of PostgreSQL-compatible NewSQL - [CAP Theorem: Revisiting the CAP Theorem](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/) - Eric Brewer's updated perspective on CAP theorem and modern databases - [Debezium CDC for Change Data Capture](https://debezium.io/documentation/) - Connecting NewSQL databases to streaming platforms Source: https://www.conduktor.io/glossary/newsql-databases-streaming --- # NoSQL Databases for Real-Time Streaming: Patterns and Integration Modern streaming applications generate continuous, high-velocity data that traditional relational databases struggle to handle efficiently. NoSQL databases emerged to address these challenges, offering flexible schemas, horizontal scalability, and optimized write performance that fit well with [streaming workloads](https://www.conduktor.io/glossary/what-is-real-time-data-streaming). When [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) or other streaming platforms deliver thousands of events per second, the downstream database must absorb this volume without bottlenecks while maintaining low-latency access for real-time applications. NoSQL databases work well in this role, providing the throughput and flexibility that [streaming architectures](https://www.conduktor.io/glossary/streaming-data-pipeline) demand. ## NoSQL Database Categories Understanding the main NoSQL categories helps you choose the right database for your streaming use case: ### Choosing the Right NoSQL Database | **Use Case** | **Recommended Database** | **Why** | |-------------|-------------------------|---------| | **Flexible event schemas** | MongoDB, Couchbase | JSON documents accommodate evolving event structures without migrations | | **Ultra-high write throughput** | ScyllaDB, Cassandra | 1M+ writes/sec per node, append-only architecture | | **Sub-millisecond reads** | Redis Stack, Valkey | In-memory storage, perfect for caching and session management | | **Time-series telemetry** | Cassandra, ScyllaDB, TimescaleDB | Optimized for time-based queries, automatic data expiration | | **Real-time analytics** | ClickHouse, Apache Druid | Sub-second analytical queries on streaming data | | **Complex relationships** | Neo4j, Amazon Neptune | Graph queries for fraud detection, recommendations | | **Serverless/variable load** | DynamoDB, MongoDB Atlas Serverless | Auto-scaling with pay-per-request pricing | ### Database Category Details - **Document Stores** like MongoDB 8.0 store data as JSON-like documents, making them ideal for event data with nested structures. Their flexible schema accommodates evolving event formats without migrations. MongoDB Atlas (2025) now offers integrated Stream Processing for real-time aggregations directly within the database, eliminating the need for separate stream processors in many use cases. - **Key-Value Stores** such as Redis Stack and DynamoDB provide the fastest read/write operations through simple lookups. Redis Stack (2025) extends traditional Redis with JSON documents, time-series data structures, and full-text search capabilities. Valkey, the Linux Foundation's Redis-compatible alternative launched in 2024, offers similar performance without licensing concerns. DynamoDB's on-demand v2 pricing model provides true serverless scaling for variable streaming workloads. - **Wide-Column Stores** like Apache Cassandra 5.0 and ScyllaDB organize data in column families, optimized for massive write throughput. ScyllaDB's C++ implementation delivers 10x higher throughput than Cassandra on identical hardware, often exceeding 1 million writes/second per node, making it the preferred choice for ultra-high-volume streaming workloads. Their architecture handles time-series data exceptionally well, making them perfect for IoT and telemetry streams. - **Graph Databases** including Neo4j 5.x and Amazon Neptune model relationships explicitly, enabling real-time fraud detection or social network analysis by querying connected data patterns as events arrive. - **OLAP-Style NoSQL** databases like Apache Druid and ClickHouse specialize in real-time analytics on streaming data. These systems ingest from Kafka while simultaneously serving sub-second analytical queries, bridging the gap between operational NoSQL databases and data warehouses. ## Core Real-Time Characteristics NoSQL databases designed for streaming workloads share several critical characteristics: - **High Write Throughput**: Streaming pipelines generate continuous writes. NoSQL databases achieve this through append-only structures (like Cassandra's commit logs), in-memory buffers, and batching mechanisms. A well-tuned Cassandra cluster can handle hundreds of thousands of writes per second per node. - **Low-Latency Reads**: Real-time applications need instant access to fresh data. NoSQL databases minimize read latency through in-memory caching, optimized indexes, and denormalized data models. Redis, for instance, serves reads in sub-millisecond timeframes by keeping entire datasets in memory. - **Horizontal Scalability**: As stream volume grows, adding nodes should increase capacity linearly. NoSQL databases achieve this through consistent hashing, automatic sharding, and peer-to-peer architectures that eliminate single points of failure. ## NoSQL + Kafka Integration Patterns ### MongoDB + Kafka: Change Streams and Connectors MongoDB's change streams feature transforms the database into an event source. Applications can subscribe to real-time notifications of inserts, updates, and deletes. Combined with [Kafka Connect](https://www.conduktor.io/glossary/kafka-connect-building-data-integration-pipelines)'s MongoDB source connector, you can: - Stream database changes into Kafka topics for downstream processing - Build [event-driven architectures](https://www.conduktor.io/glossary/event-driven-architecture) where database modifications trigger workflows - Implement [CDC (Change Data Capture)](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals) patterns, capturing every modification to database records as events, for data synchronization across systems The MongoDB sink connector allows Kafka to write directly to MongoDB collections, handling [schema evolution](https://www.conduktor.io/glossary/schema-evolution-best-practices) and automatic retries. This bidirectional integration creates powerful data pipelines where MongoDB serves both as a streaming destination and source. ```javascript // MongoDB Kafka Sink Connector Configuration { "name": "mongodb-sink-user-events", "config": { "connector.class": "com.mongodb.kafka.connect.MongoSinkConnector", "topics": "user-events", "connection.uri": "mongodb://mongo-cluster:27017", "database": "analytics", "collection": "user_events", "max.num.retries": 3, "retries.defer.timeout": 5000, "document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy", "document.id.strategy.partial.value.projection.list": "event_id", "document.id.strategy.partial.value.projection.type": "AllowList", "writemodel.strategy": "com.mongodb.kafka.connect.sink.writemodel.strategy.ReplaceOneBusinessKeyStrategy" } } ``` **MongoDB Atlas Stream Processing (2025)**: For simpler use cases, MongoDB Atlas now processes Kafka streams natively without external connectors: ```javascript // Atlas Stream Processing Pipeline - Direct Kafka Integration sp.process( [ { $source: { connectionName: "kafka-prod", topic: "user-events" } }, { $match: { event_type: "purchase" } }, { $group: { _id: "$user_id", total_revenue: { $sum: "$amount" }, purchase_count: { $sum: 1 } }}, { $merge: { into: { db: "analytics", coll: "user_totals" } } } ] ) ``` ### Cassandra + Kafka: High-Volume Ingestion Cassandra's distributed architecture and tunable consistency make it a natural fit for absorbing Kafka streams. Common patterns include: **Time-Series Storage**: IoT sensors produce millions of readings per hour. The Kafka-Cassandra connector writes these events to time-partitioned tables, enabling efficient queries by time range. ```sql -- Cassandra Table Schema for IoT Time-Series Data CREATE TABLE iot_readings ( device_id text, reading_date date, timestamp timestamp, temperature double, humidity double, pressure double, PRIMARY KEY ((device_id, reading_date), timestamp) ) WITH CLUSTERING ORDER BY (timestamp DESC) AND compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_size': 1} AND default_time_to_live = 2592000; -- 30 days TTL -- Query recent readings for a device SELECT * FROM iot_readings WHERE device_id = 'sensor-001' AND reading_date = '2025-12-17' AND timestamp > '2025-12-17 10:00:00' LIMIT 100; ``` ```java // Kafka Connect Cassandra Sink Configuration { "name": "cassandra-sink-iot-readings", "config": { "connector.class": "com.datastax.oss.kafka.sink.CassandraSinkConnector", "topics": "iot-sensor-readings", "contactPoints": "cassandra-node1,cassandra-node2,cassandra-node3", "loadBalancing.localDc": "dc1", "cloud.secureConnectBundle": "/path/to/secure-connect.zip", "topic.iot-sensor-readings.iot_telemetry.iot_readings.mapping": "device_id=value.deviceId, reading_date=value.timestamp, timestamp=value.timestamp, temperature=value.temp, humidity=value.hum, pressure=value.pres", "topic.iot-sensor-readings.iot_telemetry.iot_readings.consistencyLevel": "LOCAL_QUORUM" } } ``` - **Event Logging**: Application logs flow through Kafka into Cassandra's wide-column structure, where they're organized by service, timestamp, and severity. The write-optimized storage handles massive volumes while supporting operational queries. - **Materialized Views**: Stream processors like [Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams), [Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing), or [ksqlDB](https://www.conduktor.io/glossary/ksqldb-for-real-time-data-processing) consume events, compute aggregations, and write results to Cassandra. The database serves these precomputed views with low latency. ```sql -- ksqlDB Materialized View to Cassandra (2025 Pattern) CREATE STREAM user_events ( user_id VARCHAR, event_type VARCHAR, amount DOUBLE, timestamp BIGINT ) WITH ( KAFKA_TOPIC='user-events', VALUE_FORMAT='JSON' ); CREATE TABLE user_stats AS SELECT user_id, COUNT(*) as event_count, SUM(amount) as total_amount, MAX(timestamp) as last_seen FROM user_events GROUP BY user_id EMIT CHANGES; -- Sink to Cassandra via Connector CREATE SINK CONNECTOR cassandra_user_stats WITH ( 'connector.class' = 'com.datastax.oss.kafka.sink.CassandraSinkConnector', 'topics' = 'USER_STATS', 'contactPoints' = 'cassandra-cluster:9042', 'loadBalancing.localDc' = 'dc1', 'topic.USER_STATS.analytics.user_statistics.mapping' = 'user_id=key, event_count=value.event_count, total_amount=value.total_amount, last_seen=value.last_seen' ); ``` ### Redis + Kafka: Caching and Session Management Redis excels at storing transient streaming data: **Session Stores**: User sessions from web applications flow through Kafka into Redis with TTL (time-to-live) settings. The in-memory database provides instant session lookup for authentication and personalization. ```python # Redis Sink with Kafka Consumer (Python Example) from kafka import KafkaConsumer import redis import json # Initialize connections redis_client = redis.Redis(host='redis-cluster', port=6379, decode_responses=True) consumer = KafkaConsumer( 'user-sessions', bootstrap_servers='kafka:9092', value_deserializer=lambda m: json.loads(m.decode('utf-8')), group_id='session-manager' ) # Process session events from Kafka to Redis for message in consumer: session = message.value session_key = f"session:{session['session_id']}" # Store session with 30-minute TTL redis_client.setex( session_key, 1800, # 30 minutes in seconds json.dumps(session) ) # Update user's active session list redis_client.sadd(f"user:{session['user_id']}:sessions", session['session_id']) redis_client.expire(f"user:{session['user_id']}:sessions", 86400) # 24 hours ``` **Real-Time Counters**: Streaming analytics update Redis counters, sorted sets, and HyperLogLog structures. Dashboards query these structures for real-time metrics without touching slower persistent stores. ```python # Redis TimeSeries for Real-Time Metrics (Redis Stack 2025) from redistimeseries.client import Client from kafka import KafkaConsumer import json rts = Client(host='redis-stack', port=6379) consumer = KafkaConsumer('page-views', bootstrap_servers='kafka:9092') # Create time-series for page views per URL for message in consumer: event = json.loads(message.value) # Add data point to time-series rts.add(f"pageviews:{event['url']}", event['timestamp'], 1) # Increment daily counter using HyperLogLog for unique visitors redis_client.pfadd(f"unique_visitors:{event['date']}", event['user_id']) # Update top pages sorted set redis_client.zincrby("top_pages:realtime", 1, event['url']) # Query real-time metrics unique_visitors_today = redis_client.pfcount("unique_visitors:2025-12-17") top_5_pages = redis_client.zrevrange("top_pages:realtime", 0, 4, withscores=True) ``` **Cache Warming**: As Kafka delivers database changes, a stream processor updates corresponding Redis cache entries, ensuring cache consistency with minimal staleness. ### DynamoDB Streams and Integration AWS DynamoDB provides built-in streams that capture item-level modifications. These streams integrate with: - AWS Lambda for serverless event processing - Kinesis Data Streams for cross-region replication - Kafka via Kinesis-Kafka connectors for hybrid architectures DynamoDB's pay-per-request pricing and automatic scaling make it attractive for variable streaming workloads where throughput fluctuates significantly. ## Real-Time Use Cases - **IoT Telemetry Ingestion**: [Connected devices](https://www.conduktor.io/glossary/iot-data-streaming-architectures) generate continuous sensor readings. A typical architecture uses Kafka to buffer and partition data by device type, then writes to Cassandra for long-term storage and Redis for real-time thresholds and alerts. - **User Activity Tracking**: Web and mobile applications emit [clickstream events](https://www.conduktor.io/glossary/clickstream-analytics-with-kafka) through Kafka. Stream processors compute [session boundaries](https://www.conduktor.io/glossary/session-windows-in-stream-processing), user journeys, and conversion funnels, storing results in MongoDB for flexible querying by product teams. - **Session Management**: Authentication services write session tokens to Kafka topics. A consumer updates Redis with session metadata, enabling sub-millisecond session validation across distributed services. - **Event Sourcing**: Applications store all state changes as immutable events in Kafka. Consumers rebuild current state in NoSQL databases (often MongoDB or Cassandra), enabling temporal queries and audit trails. Learn more about [event sourcing patterns](https://www.conduktor.io/glossary/event-sourcing-patterns-with-kafka) and the [CQRS pattern](https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka). ## Consistency and Query Patterns **Eventual Consistency**: Most NoSQL databases default to eventual consistency, where writes propagate asynchronously. For streaming workloads, this typically isn't problematic, events arrive in near real-time, and slight delays (milliseconds to seconds) align with stream processing latencies. *Example*: A user posts a comment through a mobile app. The write goes to Cassandra with consistency level ONE (fastest), immediately returns success to the user, then replicates to other nodes in the background. Analytics dashboards querying this data 100ms later will see the comment, well within acceptable bounds for most use cases. **Strong Consistency**: When required, databases like MongoDB offer strong consistency through replica set read concerns, while Cassandra provides tunable consistency (QUORUM, LOCAL_QUORUM) at the cost of increased latency. ```python # MongoDB Strong Consistency Example from pymongo import MongoClient, ReadConcern, WriteConcern client = MongoClient('mongodb://replica-set:27017') db = client.analytics # Write with majority acknowledgment result = db.user_transactions.insert_one( {"user_id": "user123", "amount": 99.99, "timestamp": 1734451200}, write_concern=WriteConcern(w='majority') # Wait for majority of replicas ) # Read with strong consistency guarantees transaction = db.user_transactions.find_one( {"_id": result.inserted_id}, read_concern=ReadConcern('majority') # Read from majority of replicas ) ``` ```sql -- Cassandra Tunable Consistency -- Write with QUORUM: (replication_factor / 2) + 1 nodes must acknowledge INSERT INTO user_events (user_id, event_id, event_type, timestamp) VALUES ('user123', uuid(), 'purchase', toTimestamp(now())) USING CONSISTENCY QUORUM; -- Read with LOCAL_QUORUM: majority within same datacenter SELECT * FROM user_events WHERE user_id = 'user123' AND timestamp > '2025-12-01' USING CONSISTENCY LOCAL_QUORUM; ``` - **Query Optimization**: Design your data model around query patterns. For time-series data in Cassandra, use time-based clustering keys. In MongoDB, create indexes on frequently filtered fields like timestamp, user_id, or device_id. Avoid full table scans by using partition keys that align with your [Kafka topic partitioning strategy](https://www.conduktor.io/glossary/kafka-partitioning-strategies-and-best-practices). - **Time-Series Considerations**: Many streaming use cases involve time-series data. Use TTL (time-to-live) features to automatically expire old data. Partition tables by time windows (hourly, daily) to isolate queries and simplify data retention policies. ## Scaling Strategies - **Partitioning and Sharding**: Align your database partitioning with Kafka topic partitions. If you partition Kafka topics by user_id, use user_id as the Cassandra partition key. This ensures events for the same entity land on the same database node, enabling local aggregations. - **Write Optimization**: Batch writes when possible. Kafka Connect connectors typically buffer events before writing to the database. Tune these batch sizes based on your latency requirements, larger batches improve throughput but increase end-to-end latency. - **Read Replica Patterns**: Separate operational reads from analytical queries using read replicas. Write streaming data to primary nodes while directing dashboards and reports to replicas, preventing query load from impacting ingestion performance. - **Data Governance**: As your streaming pipelines grow in complexity, maintaining data quality and compliance becomes critical. Platforms like Conduktor provide comprehensive governance capabilities for Kafka ecosystems, ensuring schemas are validated and sensitive data is masked. Conduktor Gateway acts as a transparent proxy that enforces policies, validates data contracts, and captures audit logs of all producer/consumer activity across your streaming pipeline. ## Observability and Monitoring Monitoring NoSQL databases in streaming architectures requires visibility across multiple layers: **Write Performance Metrics**: - **Write latency percentiles** (p50, p95, p99): Track end-to-end time from Kafka consumption to database write acknowledgment - **Write throughput**: Measure events/second ingested, watching for degradation that indicates capacity issues - **Connector lag**: For Kafka Connect deployments, [monitor consumer group lag](https://www.conduktor.io/glossary/consumer-lag-monitoring) to detect ingestion delays - **Batch sizes**: Track the average number of events per batch write, too small hurts throughput, too large increases latency **Read Performance Metrics**: - **Query latency**: Monitor p99 read latencies for user-facing queries; spikes often indicate missing indexes or suboptimal data models - **Cache hit rates**: For Redis/caching layers, track hit/miss ratios to ensure cache effectiveness - **Connection pool saturation**: Exhausted connection pools cause timeouts and request queueing **Database-Specific Metrics**: - **Cassandra**: Compaction backlog, memtable flush queues, read repair frequency - **MongoDB**: Working set size vs available RAM, replication lag, oplog window - **Redis**: Memory fragmentation ratio, eviction rate, keyspace hit ratio - **DynamoDB**: Consumed vs provisioned capacity units, throttled requests **Distributed Tracing**: Implement end-to-end tracing from event production through Kafka to database writes. OpenTelemetry instrumentation reveals where latency accumulates: ![Implement end-to-end tracing from event production through Kafka to database writes. OpenTelemetry instrumentation reveals where latency accumulates](https://www.conduktor.io/assets/images/glossary/nosql-databases-real-time-0.webp) **Alerting Strategy**: - **Critical**: Connector failures, database unavailability, consumer group stuck (no progress for 5+ minutes) - **Warning**: p99 write latency exceeds SLA, consumer lag growing, connection pool >80% utilized - **Info**: Planned maintenance, schema migrations, capacity scaling events ## Anti-Patterns to Avoid - **❌ Not Aligning Partitioning Strategies**: Writing Kafka events partitioned by `user_id` to a Cassandra table partitioned by `timestamp` forces scatter-gather queries across nodes. Always align partition keys between Kafka topics and database tables. - **❌ Treating NoSQL Like Relational Databases**: Attempting complex JOINs, normalization, or ACID transactions across tables defeats NoSQL's strengths. Denormalize data, duplicate where necessary, and embrace eventual consistency for streaming workloads. - **❌ Ignoring Hot Partitions**: Poorly chosen partition keys (e.g., `timestamp` rounded to the hour) concentrate writes on a single node, creating bottlenecks. Use high-cardinality keys like `user_id` or `device_id` to distribute load evenly. - **❌ Synchronous Writes in Stream Processors**: Blocking on each database write kills throughput. Use asynchronous writes, batching, and connection pooling to parallelize database operations. - **❌ No TTL/Data Retention Policy**: Unbounded data growth degrades query performance and inflates costs. Implement TTL at the database level (Cassandra, Redis) or archive/delete old data via scheduled jobs. - **❌ Missing Indexes for Query Patterns**: Cassandra and MongoDB require indexes designed for your specific queries. Full table scans on streaming data with millions of records will timeout. Profile queries and add covering indexes. - **❌ Undersized Connection Pools**: Kafka Connect workers or stream processors with insufficient database connections create queueing and timeouts. Size pools based on concurrency (worker count × max tasks per worker). - **❌ Ignoring Schema Evolution**: Writing evolving JSON events to document stores without schema validation leads to data quality issues. Use [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management) with JSON Schema or Avro to enforce contracts, even with schemaless databases. Implementing [data contracts](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) prevents downstream breakage. - **❌ Lack of Monitoring and Observability**: Operating NoSQL-Kafka pipelines without comprehensive [observability](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars) leads to blind spots during incidents. Implement [distributed tracing](https://www.conduktor.io/glossary/distributed-tracing-for-kafka-applications) and metrics collection from the start. - **❌ Ignoring Backpressure**: When databases can't keep up with stream velocity, unconsumed events accumulate in Kafka, eventually exhausting disk space or causing OOM errors. Implement [backpressure handling](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems) and autoscaling policies. ## Conclusion NoSQL databases and streaming platforms pair well for modern data architectures. By understanding the characteristics of different NoSQL categories and applying proven integration patterns, you can build systems that handle massive scale while keeping the low latency that real-time applications need. Choose document stores for flexible event schemas, key-value stores for caching and session management, wide-column stores for time-series data, and graph databases for relationship-heavy workloads. Align your partitioning strategies with your stream topology, tune consistency levels based on your requirements, and use specialized tools to govern your data pipelines. For complementary storage solutions, explore [NewSQL databases](https://www.conduktor.io/glossary/newsql-databases-streaming) for distributed SQL with strong consistency, or consider [lakehouse architectures](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables) with Delta Lake, Iceberg, or Hudi for analytical workloads requiring historical queryability alongside real-time ingestion. The landscape continues to evolve with managed services, serverless options, and tighter integrations, but the fundamental principles remain: optimize for your write patterns, query what you need efficiently, and scale horizontally as your streams grow. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - The streaming platform at the heart of most NoSQL integration patterns - [NewSQL Databases: Distributed SQL for Real-Time Applications](https://www.conduktor.io/glossary/newsql-databases-streaming) - Alternative database approach combining ACID guarantees with horizontal scalability - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) - Essential for managing data quality across NoSQL-Kafka pipelines ## Sources and References - [Apache Cassandra Documentation](https://cassandra.apache.org/doc/latest/) - Official documentation for wide-column NoSQL database - [MongoDB Architecture Guide](https://www.mongodb.com/docs/manual/core/databases-and-collections/) - Document store design patterns and streaming integration - [Redis Documentation](https://redis.io/documentation) - In-memory data structures for real-time applications - [Kafka Connect](https://kafka.apache.org/documentation/#connect) - Framework for connecting Kafka with external systems including NoSQL databases - [DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) - Change data capture for DynamoDB integration with streaming platforms Source: https://www.conduktor.io/glossary/nosql-databases-real-time --- # On-Prem vs Hybrid Streaming: Multi-Environment Architecture Patterns Organizations building data streaming platforms face a fundamental architectural decision: where should the streaming infrastructure live? While cloud-native solutions offer compelling advantages, many enterprises operate streaming platforms on-premises, in hybrid configurations spanning multiple environments, or across multiple cloud providers. Understanding these patterns and their governance implications is essential for building resilient, compliant streaming architectures. ![Multi-environment streaming architecture patterns](https://www.conduktor.io/assets/images/glossary/on-prem-hybrid-streaming-0.webp) ## Why On-Premises Streaming Still Matters Despite the cloud's dominance, on-premises streaming remains relevant for specific use cases. Financial institutions often run Kafka clusters in their own datacenters to meet strict regulatory requirements around data sovereignty, certain jurisdictions mandate that sensitive data never leave specific geographic boundaries or organizational control. When millisecond-level latency is critical, such as in high-frequency trading or real-time manufacturing systems, on-premises deployments eliminate network hops to external cloud providers. Cost considerations also drive on-prem decisions. Organizations with existing datacenter investments and stable workloads may find that owning and operating infrastructure is more economical than paying ongoing cloud fees, particularly when data volumes are massive and predictable. Additionally, some enterprises prefer the operational control that comes with managing their own hardware, networking, and security stack, especially when integrating with legacy systems that cannot easily move to the cloud. However, on-premises streaming comes with responsibilities: capacity planning, hardware procurement, upgrades, patching, and 24/7 operations fall entirely on internal teams. The operational burden is significant, requiring specialized expertise in distributed systems, storage, and networking. Modern on-premises deployments in 2025 increasingly leverage Kubernetes operators like [Strimzi](https://www.conduktor.io/glossary/strimzi-kafka-operator-for-kubernetes) for automated cluster management and [KRaft mode](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) (Kafka's native consensus protocol that replaced ZooKeeper, now the default in Kafka 4.0+) for simpler operations and better scalability. ## Cloud Streaming: Managed Services and Operational Simplicity Cloud-based streaming platforms like [Amazon MSK](https://www.conduktor.io/glossary/amazon-msk-managed-kafka-on-aws), [Azure Event Hubs](https://www.conduktor.io/glossary/azure-event-hubs-and-kafka-compatibility), and Google Cloud Pub/Sub shift much of the operational complexity to the provider. These managed services handle cluster provisioning, scaling, patching, monitoring, and disaster recovery, allowing teams to focus on building streaming applications rather than managing infrastructure. Elasticity is a key advantage, cloud platforms can scale up during peak demand and scale down during quiet periods, paying only for resources consumed. Geographic distribution becomes simpler, with providers offering multi-region deployments that provide low-latency access to global users and built-in disaster recovery capabilities. Cloud providers also integrate streaming platforms with their broader ecosystems: IAM for authentication, KMS for encryption, CloudWatch or Azure Monitor for observability, and native connectors to storage, analytics, and machine learning services. This integration accelerates development but introduces platform lock-in that can make future migrations costly. The trade-off is reduced control and potential cost unpredictability. Cloud egress fees, charges for data leaving the provider's network, can be substantial for streaming workloads that replicate data across regions or to on-premises systems. Organizations must carefully model costs, especially for high-throughput scenarios where bandwidth charges accumulate quickly. ## Hybrid Streaming: Bridging Multiple Environments Hybrid streaming architectures combine on-premises and cloud deployments, creating unified platforms that span environments. Common motivations include gradual cloud migration (maintaining on-prem systems while building cloud-native capabilities), regulatory compliance (keeping sensitive data on-prem while leveraging cloud for analytics), and business continuity (using the cloud for disaster recovery while running primary workloads on-prem). A typical hybrid pattern involves running Kafka clusters in both environments with bi-directional replication. For example, a financial services firm might process transactions in an on-premises cluster for compliance, replicate sanitized events to a cloud cluster for analytics and machine learning, then stream insights back on-prem for operational systems. Successful hybrid architectures require careful network design. VPN tunnels or dedicated interconnects (like AWS Direct Connect, which provides private fiber connections to AWS, or Azure ExpressRoute for private Azure connectivity) provide secure, low-latency connectivity between environments. However, network latency between datacenters and cloud regions introduces replication lag that applications must tolerate. Bandwidth constraints also limit throughput, requiring prioritization of which topics replicate and in which direction. In 2025, hybrid deployments commonly leverage [tiered storage](https://www.conduktor.io/glossary/tiered-storage-in-kafka) to reduce on-premises storage costs by offloading older data to cloud object storage (S3, GCS, Azure Blob) while maintaining local access to recent data. This approach combines the performance of on-prem with the cost-efficiency of cloud storage. ## Replication Strategies for Multi-Environment Consistency Several technologies enable data replication across streaming environments, each with distinct characteristics: - **MirrorMaker 2.0** is Kafka's native replication tool, supporting active-passive (one cluster receives all writes, others are standby replicas) and active-active (multiple clusters accept writes simultaneously) patterns with topic and consumer group replication. It's open-source and flexible but requires operational expertise to tune for performance and reliability. MirrorMaker 2.0 provides exactly-once semantics (guaranteeing each message is replicated exactly once, preventing duplicates) and can preserve timestamps and offsets for consistent disaster recovery. - **Conduktor Gateway**, a Kafka proxy, offers advanced replication capabilities with built-in governance, enabling policy enforcement, data quality validation, and security controls during cross-environment replication. Unlike basic replication tools, Gateway ensures that governance policies defined centrally apply consistently as data flows between on-prem and cloud clusters, preventing configuration drift and compliance violations. - **Custom Kafka Connectors** can implement specialized replication logic using Kafka Connect, allowing transformation, filtering, and integration with external systems during replication. This approach provides maximum flexibility but requires development and maintenance effort. Choosing the right tool depends on requirements: MirrorMaker suits straightforward replication with failover needs, Conduktor Gateway excels when governance consistency across environments is critical, and custom connectors handle unique transformation or integration requirements. In 2025, organizations increasingly prioritize tools that unify governance across hybrid deployments rather than treating replication as a pure data movement problem. ## Governance Consistency Across Environments One of hybrid streaming's biggest challenges is maintaining consistent governance policies across disparate environments. [Data quality](https://www.conduktor.io/glossary/building-a-data-quality-framework) standards, [access controls](https://www.conduktor.io/glossary/access-control-for-streaming), [schema validation](https://www.conduktor.io/glossary/schema-registry-and-schema-management), and compliance policies must apply uniformly whether data flows through on-prem clusters, AWS, Azure, or Google Cloud. Centralized [governance platforms](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) address this by providing a single control plane for [policy definition and enforcement](https://www.conduktor.io/glossary/policy-enforcement-in-streaming) across all streaming infrastructure. Tools like Conduktor enable teams to define data quality rules, access policies, and compliance requirements once, then apply them consistently to every cluster regardless of location. This prevents configuration drift where security policies are strict on-prem but lax in the cloud, or where schema validation works in one environment but not others. Unified governance also simplifies [auditing and compliance reporting](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). Rather than collecting logs and metrics from multiple systems with inconsistent formats, centralized platforms provide a single view of who accessed what data, when, and from where, critical for GDPR, HIPAA, and other regulatory frameworks. ## Multi-Cloud Streaming: Avoiding Vendor Lock-In Some organizations adopt multi-cloud strategies, running streaming platforms on multiple cloud providers to avoid vendor lock-in, improve resilience, or leverage best-of-breed services. For example, an organization might use AWS for primary workloads, Google Cloud for machine learning analytics, and Azure for integration with Microsoft enterprise tools. Multi-cloud streaming introduces complexity: each provider has different IAM models, networking constructs, and managed services. [Cross-cloud replication](https://www.conduktor.io/glossary/cross-az-traffic-streaming) incurs egress fees from both providers, and network latency may be higher than within a single provider's backbone. However, the architectural flexibility can be valuable, applications can failover between clouds, and data can be processed in whichever environment offers the best price or performance for specific workloads. The governance challenge multiplies: policies must now span not just on-prem and cloud but multiple cloud providers with different primitives. Abstraction layers that provide cloud-agnostic interfaces become essential, whether through [Kubernetes-based deployments](https://www.conduktor.io/glossary/running-kafka-on-kubernetes), infrastructure-as-code templates that work across providers, or governance platforms that treat all clusters uniformly. ## Edge Streaming: Distributed Processing at the Source Edge streaming extends hybrid architectures to the network edge, factory floors, retail locations, vehicles, and IoT devices. These edge nodes run lightweight streaming platforms (like Kafka running on minimal hardware or MQTT, Message Queuing Telemetry Transport, a lightweight pub/sub protocol, brokers for IoT) that process data locally, then selectively replicate relevant events to regional or central clusters. Edge streaming reduces latency for time-critical decisions (a manufacturing robot responds to sensor data in milliseconds without round-tripping to the cloud), minimizes bandwidth usage (preprocessing filters out irrelevant data before transmission), and maintains operation during network outages (local processing continues even when cloud connectivity is lost). In 2025, edge deployments increasingly use lightweight Kubernetes distributions (K3s, MicroK8s) or purpose-built edge orchestration platforms (KubeEdge, Azure IoT Edge) for automated deployment and management. WebAssembly (Wasm) is emerging as a runtime for stream processing at the edge, offering near-native performance with strong sandboxing and portability across diverse hardware architectures. However, edge deployments face unique constraints: limited compute and storage resources, unreliable networking, and difficult physical access for maintenance. Governance becomes especially important, ensuring that edge nodes enforce security policies, validate [data quality](https://www.conduktor.io/glossary/automated-data-quality-testing), and comply with regulations without constant manual intervention requires robust automation and centralized policy management. Modern edge governance platforms can push policy updates from a central control plane to thousands of distributed edge nodes, maintaining consistent security and compliance posture. [Chaos engineering](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems) practices are particularly valuable for edge deployments, testing resilience against network partitions, hardware failures, and other common edge scenarios. ## Security in Hybrid Environments Security consistency is paramount in hybrid streaming. Authentication and authorization must work seamlessly across environments, ideally through identity federation that allows users to authenticate once and access resources everywhere. SAML (Security Assertion Markup Language, an XML-based standard for single sign-on) or OAuth (Open Authorization, a token-based authentication protocol) based SSO (Single Sign-On) integrations enable this, with centralized identity providers (like Okta, Azure AD, or corporate LDAP) authenticating users and issuing tokens valid across all clusters. Encryption must be consistent: TLS for data in transit and encryption-at-rest for stored data should apply uniformly. Key management becomes complex when spanning environments, organizations often use centralized KMS (Key Management Service) solutions or HSMs (Hardware Security Modules, tamper-resistant devices for cryptographic key storage) that work across on-prem and cloud, ensuring keys are rotated, audited, and protected consistently. Network security requires careful segmentation. Hybrid architectures should use private networking (not public internet) for replication, with firewalls restricting which systems can communicate. [Zero-trust principles](https://www.conduktor.io/glossary/zero-trust-for-streaming), verify every access attempt regardless of network location, help prevent lateral movement if one environment is compromised. In 2025, zero-trust network access (ZTNA) has become standard, often implemented through service meshes (like Istio, Linkerd) that provide mutual TLS authentication and fine-grained authorization between services, whether they run on-prem or in the cloud. Service meshes also enable consistent observability, capturing traffic patterns and security events across all environments. ## Cost Considerations and Optimization Hybrid streaming's cost structure combines on-premises capital expenses (hardware, datacenter space, power) with cloud operational expenses (compute instances, storage, network egress). Egress fees are often the hidden cost, replicating terabytes daily from cloud to on-prem can dwarf other expenses. Organizations should model data flows carefully, minimizing unnecessary replication and compressing data where possible. In 2025, FinOps practices (financial operations that optimize cloud spending) have become essential for hybrid architectures. Teams use tools like AWS Cost Explorer, Azure Cost Management, and third-party platforms (CloudHealth, Kubecost) to track costs in real-time, set budgets, and identify optimization opportunities. Common strategies include: - **Reserved capacity**: Committing to long-term cloud resources (1-3 years) for predictable workloads reduces costs by 30-70% versus on-demand pricing - **Spot instances**: Using spare cloud capacity at steep discounts for fault-tolerant streaming workloads like batch replication or analytics - **Tiered storage**: Automatically moving older Kafka data to cheaper object storage (S3 Glacier, Azure Cool Blob) while keeping recent data on fast local disks (see [tiered storage](https://www.conduktor.io/glossary/tiered-storage-in-kafka) for implementation details) - **Cross-region optimization**: Routing traffic to minimize egress fees, such as processing data in the same region where it's generated rather than replicating it elsewhere Compute costs vary by environment: cloud offers elasticity but at a premium, while on-prem requires upfront investment but lower marginal costs. The optimal balance depends on workload characteristics, steady-state workloads favor on-prem, while bursty or unpredictable workloads benefit from cloud elasticity. Management overhead is another consideration. Cloud reduces the engineering effort for infrastructure but increases complexity in governance, cost monitoring, and avoiding lock-in. On-prem requires larger operational teams but provides tighter control. Hybrid architectures often have the highest management overhead, requiring expertise in multiple environments and tooling to provide unified visibility. Understanding your [streaming total cost of ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership) across all deployment models is essential for making informed architectural decisions. For implementing cost visibility and chargeback across hybrid deployments, see [Conduktor's Cost Control Insights](https://docs.conduktor.io/guide/insights/cost-control) which provides unified cost tracking regardless of deployment model. ## Practical Implementation: Infrastructure as Code for Hybrid Deployments Modern hybrid streaming architectures rely on infrastructure-as-code (IaC) to maintain consistency across environments. Here's an example using Terraform to define a hybrid Kafka deployment: ```hcl # On-premises Kafka cluster using Strimzi on Kubernetes resource "kubernetes_manifest" "onprem_kafka_cluster" { provider = kubernetes.onprem manifest = { apiVersion = "kafka.strimzi.io/v1beta2" kind = "Kafka" metadata = { name = "production-cluster" namespace = "kafka" } spec = { kafka = { version = "3.8.0" replicas = 5 config = { "offsets.topic.replication.factor" = 3 "transaction.state.log.replication.factor" = 3 "transaction.state.log.min.isr" = 2 "log.retention.hours" = 168 } storage = { type = "persistent-claim" size = "1Ti" class = "fast-ssd" } } # KRaft mode configuration (no ZooKeeper needed in Kafka 4.0+) kafka = { metadataVersion = "3.8" } } } } # Cloud Kafka cluster (AWS MSK) resource "aws_msk_cluster" "cloud_cluster" { cluster_name = "analytics-cluster" kafka_version = "3.8.0" number_of_broker_nodes = 3 broker_node_group_info { instance_type = "kafka.m5.2xlarge" client_subnets = [ aws_subnet.private_a.id, aws_subnet.private_b.id, aws_subnet.private_c.id, ] storage_info { ebs_storage_info { volume_size = 1000 } } } encryption_info { encryption_in_transit { client_broker = "TLS" in_cluster = true } } } # MirrorMaker 2 for bi-directional replication resource "kubernetes_manifest" "mirror_maker" { provider = kubernetes.onprem manifest = { apiVersion = "kafka.strimzi.io/v1beta2" kind = "KafkaMirrorMaker2" metadata = { name = "hybrid-replication" namespace = "kafka" } spec = { version = "3.8.0" replicas = 3 connectCluster = "cloud-cluster" clusters = [ { alias = "onprem-cluster" bootstrapServers = "production-cluster-kafka-bootstrap:9092" }, { alias = "cloud-cluster" bootstrapServers = aws_msk_cluster.cloud_cluster.bootstrap_brokers_tls config = { "security.protocol" = "SSL" } } ] mirrors = [ { sourceCluster = "onprem-cluster" targetCluster = "cloud-cluster" sourceConnector = { config = { "replication.factor" = 3 "sync.group.offsets.enabled" = true "refresh.topics.interval.seconds" = 60 } } checkpointConnector = { config = { "checkpoints.topic.replication.factor" = 3 "sync.group.offsets.enabled" = true } } topicsPattern = "transactions\\..*, analytics\\..*" groupsPattern = "analytics-.*" } ] } } } ``` This Terraform configuration demonstrates: - **On-prem deployment** using Strimzi operator with KRaft mode (Kafka's native consensus replacing ZooKeeper) - **Cloud deployment** using AWS MSK with encryption - **Replication** via MirrorMaker 2 with topic and consumer group synchronization - **Consistent configuration** managed through code, preventing drift between environments ## Conclusion Choosing between on-premises, cloud, and hybrid streaming architectures is not a one-time decision but an evolving strategy aligned with business needs, regulatory requirements, and technical capabilities. On-premises deployments offer control and can be cost-effective for stable workloads but demand significant operational investment. Cloud platforms provide elasticity and integration but introduce lock-in and cost variability. Hybrid approaches bridge these worlds, enabling gradual migration, compliance, and resilience at the cost of increased complexity. Regardless of architecture, consistent governance across environments is non-negotiable. Unified [policy management](https://www.conduktor.io/glossary/policy-enforcement-in-streaming), centralized [observability](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars), and standardized security controls ensure that data quality, compliance, and [access controls](https://www.conduktor.io/glossary/data-access-control-rbac-and-abac) apply everywhere data flows. As streaming platforms increasingly span multiple clouds, edge locations, and on-premises datacenters, the ability to govern them consistently becomes a competitive differentiator, enabling organizations to move fast without sacrificing control, compliance, or security. Organizations should assess their architectural needs using a [streaming maturity model](https://www.conduktor.io/glossary/streaming-maturity-model) to determine when hybrid complexity is justified versus simpler single-environment deployments. For regulated industries, understanding [data governance frameworks](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) and compliance requirements upfront prevents costly architectural rework later. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka) - Platform architecture supporting on-prem, cloud, and hybrid deployments - [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters) - Planning for business continuity in hybrid environments - [Streaming Total Cost of Ownership](https://www.conduktor.io/glossary/streaming-total-cost-of-ownership) - Comparing costs across deployment models ## Sources and References 1. **Apache Kafka MirrorMaker 2.0 Documentation** - Official documentation on Kafka's native replication tool for multi-datacenter deployments. [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0) 2. **Strimzi Kafka Operator** - Cloud-native Apache Kafka deployment on Kubernetes with automated operations and KRaft support. [https://strimzi.io/](https://strimzi.io/) 3. **KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum (KRaft)** - Apache Kafka's architectural evolution removing ZooKeeper dependency. [https://cwiki.apache.org/confluence/display/KAFKA/KIP-500](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500) 4. **AWS Direct Connect Documentation** - Guide to establishing dedicated network connections between on-premises datacenters and AWS for hybrid architectures. [https://aws.amazon.com/directconnect/](https://aws.amazon.com/directconnect/) 5. **Microsoft Azure ExpressRoute** - Documentation on private connectivity between on-premises infrastructure and Azure cloud services. [https://azure.microsoft.com/en-us/products/expressroute/](https://azure.microsoft.com/en-us/products/expressroute/) 6. **KIP-405: Kafka Tiered Storage** - Apache Kafka's tiered storage architecture for hybrid cloud deployments. [https://cwiki.apache.org/confluence/display/KAFKA/KIP-405](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405) 7. **FinOps Foundation** - Best practices for cloud financial management and cost optimization in hybrid architectures. [https://www.finops.org/](https://www.finops.org/) 8. **CNCF Service Mesh Landscape** - Overview of service mesh technologies for secure hybrid deployments. [https://landscape.cncf.io/card-mode?category=service-mesh](https://landscape.cncf.io/card-mode?category=service-mesh) 9. **NIST Cloud Computing Standards** - Framework for understanding cloud deployment models including hybrid and multi-cloud architectures. [https://www.nist.gov/programs-projects/nist-cloud-computing-program-nccp](https://www.nist.gov/programs-projects/nist-cloud-computing-program-nccp) Source: https://www.conduktor.io/glossary/on-prem-hybrid-streaming --- # Optimizing Delta Tables: OPTIMIZE and Z-ORDER As Delta tables grow in production environments, maintaining query performance becomes increasingly challenging. Small files accumulate from frequent writes, and data layout becomes suboptimal for common query patterns. Delta Lake provides two commands, OPTIMIZE and Z-ORDER, that address these issues through file compaction and intelligent data clustering. ![OPTIMIZE and Z-ORDER transformation workflow](https://www.conduktor.io/assets/images/glossary/optimizing-delta-tables-optimize-and-z-order-0.webp) > **Note**: This guide focuses on Delta Lake optimization. For similar techniques in Apache Iceberg, see [Maintaining Iceberg Tables: Compaction and Cleanup](https://www.conduktor.io/maintaining-iceberg-tables-compaction-and-cleanup). For a broader comparison of table formats, refer to [Apache Iceberg](https://www.conduktor.io/apache-iceberg). ## Understanding the Small File Problem Delta Lake's [ACID transaction](https://www.conduktor.io/delta-lake-transaction-log-how-it-works) support and append-only architecture create a natural tendency toward file fragmentation. Each write operation, whether from batch jobs or [streaming pipelines](https://www.conduktor.io/streaming-to-lakehouse-tables), generates new Parquet files. Over time, this leads to: - **Metadata overhead**: Reading thousands of small files requires significant I/O operations just to process file metadata - **Inefficient parallelization**: Query engines struggle to distribute small files efficiently across executors - **Reduced data skipping effectiveness**: Statistics at the file level become less useful when data is scattered across many files A table with 10,000 files of 10 MB each performs significantly worse than the same data consolidated into 100 files of 1 GB each. The OPTIMIZE command addresses this directly. ## The OPTIMIZE Command: File Compaction The OPTIMIZE command performs **bin-packing** (efficiently grouping items to minimize wasted space) to combine small files into larger, more efficiently sized files. This process rewrites data files while preserving the transaction log integrity. ### Basic Syntax ```sql -- Optimize entire table OPTIMIZE delta_table; -- Optimize specific partition OPTIMIZE delta_table WHERE date = '2025-12-01'; ``` ### Configuration and Performance Metrics The target file size is controlled by the `delta.targetFileSize` table property (default: 1 GB). For streaming workloads with frequent small writes, setting this appropriately is critical: ```sql ALTER TABLE delta_table SET TBLPROPERTIES ( 'delta.targetFileSize' = '134217728' -- 128 MB for streaming ); ``` Real-world performance improvements from OPTIMIZE operations: - **Query latency reduction**: 40-60% faster queries on tables with severe fragmentation - **Metadata processing**: Up to 80% reduction in time spent reading file metadata - **Cloud storage costs**: Fewer API calls to object storage (S3, ADLS, GCS) A production e-commerce table with 15,000 small files (avg 8 MB) was optimized to 150 files (avg 800 MB), reducing typical aggregate queries from 45 seconds to 18 seconds, a 60% improvement. ## Z-ORDER: Multi-Dimensional Clustering While file compaction addresses the quantity of files, Z-ORDER addresses their internal organization. Z-ORDER is a technique that co-locates related data within the same file set, maximizing the effectiveness of Delta Lake's data skipping. ### How Z-ORDER Works Z-ORDER uses a **space-filling curve** (Z-curve or Morton curve) to map multi-dimensional data into a single dimension while preserving locality. Think of it as a path that weaves through multi-dimensional space, visiting nearby points in sequence. When you Z-ORDER by multiple columns, data with similar values across those dimensions is stored physically close together, like organizing a library by both subject and publication date simultaneously, rather than choosing just one. ```sql OPTIMIZE delta_table ZORDER BY (customer_id, event_date); ``` This command reorganizes data so that queries filtering on `customer_id`, `event_date`, or both benefit from enhanced data skipping. ### Data Skipping in Action - **Data skipping** is the technique of avoiding reading files that cannot contain query results, based on min/max statistics stored in file metadata. Delta Lake maintains these statistics automatically for each data file. When a query includes filters, the engine uses these statistics to skip entire files that cannot contain matching data, dramatically reducing I/O and compute. Z-ORDER maximizes this effectiveness: - **Without Z-ORDER:** - Query: `SELECT * FROM events WHERE customer_id = 12345 AND event_date = '2025-12-01'` - Files scanned: 450 out of 500 files - Data skipping effectiveness: 10% **With Z-ORDER on (customer_id, event_date):** - Query: Same query - Files scanned: 45 out of 500 files - Data skipping effectiveness: 91% The performance impact is substantial. A financial services company reduced query times on a 5 TB transaction table from 3 minutes to 22 seconds by implementing Z-ORDER on account_id and transaction_date. ### Choosing Z-ORDER Columns Selecting the right columns for Z-ORDER requires understanding your query patterns: 1. **High cardinality columns**: **Cardinality** refers to the number of unique values in a column. High cardinality columns (e.g., user_id with millions of values) provide excellent data skipping opportunities. Low cardinality columns (e.g., status with 3 values) provide minimal benefit. 2. **Frequently filtered columns**: Columns used in WHERE clauses benefit most 3. **Column order matters**: Place the most selective column first when possible 4. **Limit to 3-4 columns**: Z-ORDER effectiveness diminishes beyond 3-4 dimensions ```sql -- Good: High cardinality, frequently queried together OPTIMIZE transactions ZORDER BY (user_id, transaction_date, merchant_id); -- Poor: Low cardinality, rarely queried together OPTIMIZE transactions ZORDER BY (status, is_refunded); ``` ## Streaming Ecosystem Integration In [streaming architectures](https://www.conduktor.io/streaming-data-pipeline), OPTIMIZE and Z-ORDER play complementary roles to batch optimization strategies. Streaming workloads generate small files continuously, making regular optimization essential. For a comprehensive guide on writing streaming data to Delta Lake, see [Streaming to Lakehouse Tables](https://www.conduktor.io/streaming-to-lakehouse-tables). ### Structured Streaming and Auto-Optimize Delta Lake supports auto-optimization features that work seamlessly with Spark Structured Streaming. **Note**: The configuration examples below use Databricks-specific settings. For open-source Delta Lake, you'll need to run OPTIMIZE commands separately. ```python # Enable auto-optimize for streaming writes (Databricks-specific) spark.conf.set("spark.databricks.delta.properties.defaults.autoOptimize.optimizeWrite", "true") spark.conf.set("spark.databricks.delta.properties.defaults.autoOptimize.autoCompact", "true") streaming_df.writeStream \ .format("delta") \ .outputMode("append") \ .option("checkpointLocation", "/checkpoints/events") \ .trigger(processingTime="5 minutes") \ .start("/delta/events") ``` **Auto-optimize strategies:** - **Optimize Write**: Reduces small file creation during writes by shuffling data before writing - **Auto Compaction**: Runs OPTIMIZE automatically after writes when fragmentation thresholds are exceeded **For open-source Delta Lake (2025):** Delta Lake 3.2+ includes optimized writes by default, automatically sizing files more efficiently during writes without requiring Databricks-specific configuration. ### Scheduled Optimization for Streaming Tables For high-throughput streaming tables, scheduled optimization jobs provide better control: ```sql -- Run hourly for recent partitions OPTIMIZE streaming_events WHERE event_hour >= current_timestamp() - INTERVAL 24 HOURS; -- Run daily with Z-ORDER for historical partitions OPTIMIZE streaming_events WHERE event_date = current_date() - INTERVAL 1 DAY ZORDER BY (user_id, event_type); ``` ### Governance and Visibility Managing optimization across multiple streaming pipelines requires visibility into table health and performance. Modern data governance platforms provide capabilities for [streaming data platforms](https://www.conduktor.io/streaming-data-pipeline), including: - **Table health monitoring**: Track file counts, average file sizes, and fragmentation metrics across Delta tables - **Optimization audit trails**: Monitor when OPTIMIZE operations run and their impact on query performance - **Policy enforcement**: Set organization-wide standards for optimization schedules and Z-ORDER configurations - **Cross-platform visibility**: Unified view of Delta tables across different compute engines (Spark, Flink, Trino) **For Kafka and streaming-centric architectures**, platforms like **Conduktor** provide governance capabilities that extend from Kafka topics through to lakehouse tables, ensuring end-to-end [data quality](https://www.conduktor.io/building-a-data-quality-framework) and observability. Manage your streaming data sources with [Conduktor's topic management](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) and [Kafka Connect](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect) for lakehouse ingestion pipelines. This governance layer becomes essential in enterprises with dozens of data engineering teams managing hundreds of Delta tables. It bridges the gap between streaming ingestion and analytical consumption, ensuring that optimization strategies align with organizational SLAs and cost targets. ## Best Practices and Operational Considerations ### When to Run OPTIMIZE: A Decision Guide Not all tables need optimization, and over-optimizing wastes resources. Use this guide: **Run OPTIMIZE when:** - File count exceeds 1,000 files per partition - Average file size drops below 100 MB - Query performance degrades noticeably - After high-volume ingestion periods (e.g., daily batch loads) **Skip OPTIMIZE when:** - Table is append-only with infrequent queries - Files are already near target size (800 MB - 1 GB) - Table is temporary or short-lived - Write volume is low (< 10 files/day) ### Optimization Scheduling Running OPTIMIZE during peak query hours can impact performance and costs. Recommended approaches: ```sql -- Partition-aware optimization (processes only recent data) OPTIMIZE events WHERE date >= current_date() - INTERVAL 7 DAYS; -- Time-windowed execution (runs during off-peak hours) -- Schedule via cron/Airflow/orchestration tools during 2-6 AM UTC ``` **Scheduling tools**: Use workflow orchestrators like Apache Airflow, Dagster, or cloud-native schedulers (AWS Step Functions, Azure Data Factory) to automate optimization during off-peak hours. ### Monitoring Optimization Impact Track these metrics to measure effectiveness: ```sql -- Check table file statistics DESCRIBE DETAIL delta_table; -- View optimization history DESCRIBE HISTORY delta_table; -- Analyze data skipping effectiveness (Databricks) SELECT * FROM ( SELECT input_file_name() as file, count(*) as records FROM delta_table WHERE date = '2025-12-01' GROUP BY 1 ); ``` ### Cost Considerations OPTIMIZE operations rewrite data, incurring compute and storage costs: - **Compute**: Rewriting 1 TB typically costs $2-5 depending on cluster configuration - **Storage**: Temporary 2x storage until old files are vacuumed via the VACUUM command (removes old file versions) - **ROI**: Query performance improvements and reduced scanning typically offset costs within days A data warehouse team reduced their monthly query costs by $12,000 while spending $1,500/month on scheduled optimization, an 8x return. ### Delta Lake 3.x Optimization Improvements (2025) Recent Delta Lake releases introduce significant optimization enhancements: **Incremental OPTIMIZE (Delta 3.1+):** ```sql -- Only optimizes files that have changed since last optimization OPTIMIZE delta_table WHERE date >= '2025-12-01'; ``` This incremental approach dramatically reduces optimization costs by avoiding re-processing of already-optimized data. **Multi-threaded OPTIMIZE (Delta 3.2+):** Delta Lake 3.2 introduces parallel file rewriting, reducing OPTIMIZE execution time by up to 5x on large tables. Enable with: ```python spark.conf.set("spark.databricks.delta.optimize.maxThreads", "8") ``` **Checkpoint Optimization:** For tables with extensive transaction histories, Delta Lake automatically creates checkpoints every 10 commits, speeding up metadata operations and query planning. Monitor checkpoint health with: ```sql DESCRIBE DETAIL delta_table; -- Check lastCheckpoint column ``` ## Advanced Techniques ### Liquid Clustering (Delta 3.0+) For workloads with evolving query patterns, Delta Lake 3.0 introduces [Liquid Clustering](https://www.conduktor.io/delta-lake-liquid-clustering-modern-partitioning) as an alternative to Z-ORDER: ```sql CREATE TABLE events ( user_id BIGINT, event_date DATE, event_type STRING ) USING DELTA CLUSTER BY (user_id, event_date); ``` **Liquid Clustering vs Z-ORDER:** Liquid Clustering automatically maintains clustering without manual Z-ORDER commands, adapting to query patterns over time. Key differences: | Feature | Z-ORDER | Liquid Clustering | |---------|---------|-------------------| | **Setup** | Manual OPTIMIZE ZORDER BY commands | Declared at table creation | | **Maintenance** | Run manually or scheduled | Automatic during writes | | **Query Pattern Adaptation** | Static - requires re-running with different columns | Dynamic - adapts to actual queries | | **Partitioning** | Works with traditional partitions | Replaces partitioning entirely | | **Best For** | Known, stable query patterns | Evolving workloads, multiple access patterns | - **When to use Z-ORDER:** Stable workloads with well-understood query patterns, especially when you need fine-grained control over optimization timing. - **When to use Liquid Clustering:** New tables, evolving analytics workloads, or tables accessed by multiple teams with different query patterns. See the [Delta Lake Liquid Clustering guide](https://www.conduktor.io/delta-lake-liquid-clustering-modern-partitioning) for migration strategies. ### Bloom Filter Indexes For high-cardinality columns with equality filters (e.g., `WHERE user_id = 12345`), combine Z-ORDER with Bloom filters. **Bloom filters** are space-efficient probabilistic data structures that quickly determine if a value definitely doesn't exist in a file, further enhancing data skipping. **Note:** Bloom filter index syntax varies by platform. The example below uses Databricks syntax: ```sql -- Databricks-specific syntax CREATE BLOOMFILTER INDEX ON delta_table FOR COLUMNS (user_id); OPTIMIZE delta_table ZORDER BY (event_date, region); ``` This provides near-constant lookup performance for user_id while maintaining Z-ORDER benefits for other columns. Bloom filters excel when you have frequent point lookups on high-cardinality columns. ## Summary Optimizing Delta Tables through OPTIMIZE and Z-ORDER is essential for maintaining production performance at scale. OPTIMIZE addresses file fragmentation through intelligent compaction, while Z-ORDER enhances data skipping through multi-dimensional clustering. Together, they can reduce query latency by 50-90% on fragmented tables. Key takeaways: - Run OPTIMIZE regularly on high-write tables to combat fragmentation (aim for files between 800 MB - 1 GB) - Apply Z-ORDER to high-cardinality columns used in common query filters (limit to 3-4 columns) - Leverage **Delta Lake 3.x features** (2025): incremental OPTIMIZE, multi-threaded processing, and automatic checkpoint management - For evolving query patterns, consider **[Liquid Clustering](https://www.conduktor.io/delta-lake-liquid-clustering-modern-partitioning)** instead of Z-ORDER - Enable auto-optimization for streaming workloads or schedule manual optimization during off-peak hours - Monitor table health metrics and optimization history to measure ROI - Use governance platforms like **Conduktor** for enterprise-scale visibility across [streaming data pipelines](https://www.conduktor.io/streaming-data-pipeline) As Delta tables grow from gigabytes to petabytes, these optimization techniques move from optional enhancements to operational necessities. Regular optimization pays off through faster queries, reduced costs, and improved data platform reliability. ## Related Concepts - [Delta Lake Liquid Clustering: Modern Partitioning](https://www.conduktor.io/delta-lake-liquid-clustering-modern-partitioning) - The modern alternative to Z-ORDER for adaptive clustering - [Delta Lake Transaction Log: How It Works](https://www.conduktor.io/delta-lake-transaction-log-how-it-works) - How OPTIMIZE operations are tracked in the transaction log - [Streaming to Lakehouse Tables](https://www.conduktor.io/streaming-to-lakehouse-tables) - Auto-optimization strategies for streaming workloads ## Sources and References - [Delta Lake Official Documentation - OPTIMIZE](https://docs.delta.io/latest/optimizations-oss.html) - [Databricks Z-ORDER Technical Documentation](https://docs.databricks.com/delta/data-skipping.html) - [Delta Lake GitHub Repository](https://github.com/delta-io/delta) - [Research Paper: "The Delta Lake Project: Building Reliable Data Lakes at Scale"](https://databricks.com/research/delta-lake) - [Apache Spark Structured Streaming Guide](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) - [Parquet File Format Specification](https://parquet.apache.org/docs/) - [Z-Order Curve (Morton Code) - Wikipedia](https://en.wikipedia.org/wiki/Z-order_curve) Source: https://www.conduktor.io/glossary/optimizing-delta-tables-optimize-and-z-order --- # Outbox Pattern for Reliable Event Publishing The **outbox pattern** publishes events reliably by writing each event to an outbox table inside the same database transaction as the business data it describes. A separate process then reads that table and forwards the events to a message broker such as Kafka. Because the state change and the event commit together, the two can never disagree, which is what removes the dual-write problem. This pattern is foundational for microservices architectures. For broader context on distributed transaction patterns, see [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions) and [CQRS and Event Sourcing with Kafka](https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka). ## The Dual-Write Problem The dual-write problem occurs when an application needs to perform two separate operations that should succeed or fail together: updating a database and publishing an event to a message broker. Consider an e-commerce order service. When a customer places an order, the service must: 1. Save the order to the database 2. Publish an `OrderCreated` event to notify other services (inventory, shipping, analytics) The naive approach attempts both operations sequentially: ``` saveOrder(order) // Write to database publishEvent(orderEvent) // Write to message broker ``` This creates several failure scenarios: - The database write succeeds, but the event publish fails (network issue, broker unavailable) - The event publish succeeds, but the database write fails (validation error, constraint violation) - Both operations appear to succeed initially, but one is later rolled back These inconsistencies lead to data loss, duplicate processing, and complex recovery logic. Traditional distributed transactions (2PC) can solve this but introduce significant performance overhead and tight coupling between systems. ## How the Outbox Pattern Works The outbox pattern transforms the dual-write problem into a single-write problem by treating event publishing as part of the database transaction. The pattern introduces an "outbox" table within the application's database. Instead of publishing events directly to a message broker, the application writes events to this table in the same transaction as the business data: ![The pattern introduces an "outbox" table within the application's database. Instead of publishing events directly to a message broker, the application writes events to this table in the same transaction as the business data](https://www.conduktor.io/assets/images/glossary/outbox-pattern-for-reliable-event-publishing-0.webp) A separate process then reads events from the outbox table and publishes them to the message broker. This decouples event publishing from the business transaction while maintaining consistency through database ACID guarantees. The key insight is that the database becomes the single source of truth. If the transaction commits, both the business data and the event are persisted. If it rolls back, neither is saved. ## Implementation Approaches There are two primary approaches to implementing the relay process that reads from the outbox table and publishes events. ### Polling-Based Relay The simplest approach uses a scheduled job that periodically queries the outbox table for unpublished events: ``` SELECT * FROM outbox WHERE published = false ORDER BY created_at ``` After successfully publishing each event to the message broker, the relay marks it as published or deletes the record. This approach is straightforward to implement but has limitations: - Polling interval creates latency between database commit and event publication - Inefficient for high-throughput systems (constant database queries) - Requires careful handling of failures and retries ### Change Data Capture (CDC) A more sophisticated approach uses Change Data Capture to stream database changes directly to a message broker. CDC tools monitor the database transaction log, the write-ahead log (WAL) in PostgreSQL or binary log (binlog) in MySQL, which records every committed transaction for durability and replication purposes. For comprehensive coverage of CDC fundamentals, see [What is Change Data Capture (CDC) Fundamentals](https://www.conduktor.io/glossary/what-is-change-data-capture-cdc-fundamentals). Tools like Debezium (version 2.5+ as of 2025) have become the industry standard for CDC-based outbox implementations. Debezium reads these transaction logs and publishes changes to Kafka in near real-time, typically within single-digit milliseconds of the database commit. CDC offers several advantages over polling: - **Near-zero latency**: Events published within milliseconds of database commit - **No polling overhead**: Eliminates constant database queries - **Scales efficiently**: Handles high-throughput scenarios with minimal database impact - **Complete capture**: Captures all changes without application-level filtering - **Ordered delivery**: Maintains transaction order within a single table **Modern Debezium 2.5+ Features (2025)**: Debezium 2.5 and later versions introduce significant improvements for outbox pattern implementations: - **Incremental snapshots**: Capture existing outbox records without locking tables, essential for migrating existing data - **Notification channels**: PostgreSQL implementations can use `LISTEN/NOTIFY` for immediate event detection - **Improved schema history**: Better handling of schema evolution in outbox tables - **Custom routing**: Enhanced SMT configurations for complex routing scenarios - **Metrics and monitoring**: Comprehensive JMX metrics for tracking outbox processing lag **Debezium Outbox Event Router SMT**: Debezium's Outbox Event Router SMT (Single Message Transform) provides specialized support for the outbox pattern. An SMT is a Kafka Connect component that transforms records as they flow from source to Kafka, allowing you to convert raw database change events into clean, routable business events. The SMT extracts specific fields from your outbox table and uses them to: - Route events to different Kafka topics based on event type - Set message keys for proper partitioning - Create clean event payloads without database metadata - Add headers for routing and filtering ## Practical Implementation ### Outbox Table Schema The outbox table design varies slightly between databases but follows a common pattern. Here are production-ready schemas for PostgreSQL and MySQL: **PostgreSQL Outbox Table**: ```sql CREATE TABLE outbox ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), aggregate_type VARCHAR(255) NOT NULL, aggregate_id VARCHAR(255) NOT NULL, event_type VARCHAR(255) NOT NULL, payload JSONB NOT NULL, created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, INDEX idx_outbox_created (created_at) ); ``` **MySQL Outbox Table**: ```sql CREATE TABLE outbox ( id BINARY(16) PRIMARY KEY DEFAULT (UUID_TO_BIN(UUID())), aggregate_type VARCHAR(255) NOT NULL, aggregate_id VARCHAR(255) NOT NULL, event_type VARCHAR(255) NOT NULL, payload JSON NOT NULL, created_at TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6), INDEX idx_outbox_created (created_at) ) ENGINE=InnoDB; ``` **Key Fields**: - `aggregate_type`: Entity type (e.g., "Order", "Customer") for routing - `aggregate_id`: Entity identifier (e.g., order ID) used as the Kafka message key - `event_type`: Specific event (e.g., "OrderCreated", "OrderShipped") determines the Kafka topic - `payload`: Event data in JSON format - `created_at`: Timestamp for ordering and debugging ### Debezium Connector Configuration A complete Debezium PostgreSQL connector configuration using the Outbox Event Router SMT: ```json { "name": "orders-outbox-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres.example.com", "database.port": "5432", "database.user": "debezium", "database.password": "${file:/secrets/db-password.txt:password}", "database.dbname": "orders_service", "database.server.name": "orders", "slot.name": "orders_outbox_slot", "table.include.list": "public.outbox", "tombstones.on.delete": "false", "publication.autocreate.mode": "filtered", "transforms": "outbox", "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter", "transforms.outbox.table.field.event.type": "event_type", "transforms.outbox.table.field.event.id": "id", "transforms.outbox.table.field.event.key": "aggregate_id", "transforms.outbox.table.field.event.payload": "payload", "transforms.outbox.route.topic.replacement": "orders.${routedByValue}", "transforms.outbox.route.by.field": "aggregate_type" } } ``` This configuration: 1. Connects to PostgreSQL and creates a replication slot named `orders_outbox_slot` 2. Monitors only the `outbox` table 3. Uses the Outbox Event Router SMT to transform raw change events 4. Routes events to topics like `orders.Order` based on `aggregate_type` 5. Uses `aggregate_id` as the Kafka message key for proper partitioning 6. Extracts the `payload` field as the message value **Application Code Example**: When creating an order, your application inserts both the business data and the outbox event in a single transaction: ```java @Transactional public void createOrder(Order order) { // Insert business data orderRepository.save(order); // Insert outbox event in same transaction OutboxEvent event = new OutboxEvent( UUID.randomUUID(), "Order", order.getId(), "OrderCreated", jsonMapper.writeValueAsString(new OrderCreatedEvent( order.getId(), order.getCustomerId(), order.getTotalAmount(), order.getItems() )), Instant.now() ); outboxRepository.save(event); // If transaction commits, both order and event are saved // If transaction rolls back, neither is saved } ``` ### Database Prerequisites For CDC to function, your database must have transaction logging enabled and configured correctly: **PostgreSQL Requirements**: - Logical replication enabled: `wal_level = logical` in `postgresql.conf` - Replication slot available (created automatically by Debezium) - Appropriate user permissions: `REPLICATION` and `SELECT` on the outbox table **MySQL Requirements**: - Binary logging enabled: `log_bin = ON` - Row-based replication format: `binlog_format = ROW` - Binary log retention configured: `binlog_expire_logs_seconds` set appropriately - User with `REPLICATION SLAVE` and `REPLICATION CLIENT` privileges ## Connection to Data Streaming The outbox pattern naturally integrates with streaming platforms, particularly Apache Kafka, creating a bridge between transactional databases and event streams. When using CDC with Kafka Connect, the outbox table becomes a reliable source for Kafka topics. Each outbox record is transformed into a Kafka message, enabling downstream consumers to react to business events in real-time. For example, an order service's outbox events might populate these topics: - `orders.created` - New orders for inventory and fulfillment services - `orders.updated` - Order modifications for customer notifications - `orders.cancelled` - Cancellations for refund processing Apache Flink and other stream processors can consume these topics to build real-time analytics, complex event processing, or stateful workflows. The outbox pattern ensures that the foundation of these streaming pipelines is built on reliable, consistent data. **Monitoring and Testing with Conduktor**: Platforms like Conduktor help teams manage and monitor CDC pipelines at scale, providing comprehensive visibility into the outbox pattern implementation: - **Kafka Connect Management**: Monitor Debezium connector health, lag metrics, and throughput - **Topic Inspection**: View outbox events flowing through Kafka topics with schema awareness - **Data Quality**: Validate that outbox events match expected schemas and business rules - **Consumer Lag Tracking**: Ensure downstream consumers keep up with outbox event production Conduktor Gateway, a Kafka proxy, adds powerful testing capabilities specifically valuable for outbox pattern implementations: - **Chaos Engineering**: Simulate database or Kafka Connect failures to test recovery mechanisms - **Latency Injection**: Test how your system handles CDC lag scenarios - **Network Partitioning**: Validate behavior when Debezium loses connectivity to Kafka - **Schema Evolution Testing**: Test schema compatibility across producer and consumer versions For example, you can use Conduktor Gateway to inject artificial delays into the CDC pipeline, validating that your application gracefully handles the eventual consistency window between database commit and event availability in Kafka. This type of testing is critical for production readiness of outbox-based systems. For more on testing distributed systems, see [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems) and [Testing Strategies for Streaming Applications](https://www.conduktor.io/glossary/testing-strategies-for-streaming-applications). ## Benefits and Trade-offs The outbox pattern provides significant benefits for event-driven architectures: - **Guaranteed Delivery**: Events are persisted durably before the transaction commits, eliminating the risk of lost events due to broker failures. - **Exactly-Once Semantics**: Each business operation produces exactly one event (within the database transaction), avoiding duplicate event publishing from retry logic. - **Decoupled Systems**: The application doesn't need direct connectivity to the message broker during transaction processing, improving resilience. - **Audit Trail**: The outbox table provides a complete history of all events, useful for debugging and compliance. However, the pattern introduces trade-offs: - **Increased Complexity**: Additional infrastructure (CDC tools, relay processes) and database schema changes. - **Eventual Consistency**: A small delay exists between transaction commit and event publication, requiring consumers to handle eventual consistency. - **Database Load**: The outbox table adds write volume and storage requirements to the database. - **Operational Overhead**: CDC-based approaches require managing additional connectors and monitoring the event pipeline. ## Real-World Considerations When implementing the outbox pattern in production systems, several practical concerns emerge. - **Message Ordering**: If event ordering matters, design your outbox table and publishing logic to preserve order. CDC tools like Debezium maintain ordering within a single database table but not across tables. Consider using a single outbox table per aggregate root (a domain-driven design concept representing the main entity that owns a transaction boundary, like "Order" or "Customer") or including sequence numbers. Kafka's partitioning strategy ensures that events with the same key (aggregate_id) are delivered to consumers in order. For more on Kafka message ordering, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). - **Schema Evolution**: Event payloads stored in the outbox table require careful schema management as your system evolves. Unlike direct Kafka publishing where Schema Registry enforces compatibility at write time, the outbox pattern stores JSON payloads that may need to support multiple schema versions simultaneously. Best practices for schema evolution with outbox: - Include a `schema_version` field in your event payload to track format changes - Use Schema Registry with Avro or Protobuf for strongly-typed event definitions - Apply schema validation before inserting into the outbox table - Use backward-compatible changes when possible (adding optional fields, not removing fields) - Consider using Kafka Connect's Avro converter to transform JSON outbox events into Avro when publishing to Kafka For example, evolving an `OrderCreated` event: ```json // Version 1 { "schema_version": "1.0", "order_id": "123", "customer_id": "456", "total": 99.99 } // Version 2 (backward compatible - adds optional field) { "schema_version": "2.0", "order_id": "123", "customer_id": "456", "total": 99.99, "currency": "USD" // New optional field with default } ``` For detailed schema evolution strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Schema Evolution Best Practices](https://www.conduktor.io/glossary/schema-evolution-best-practices). - **Cleanup Strategy**: Decide how long to retain published events in the outbox table. Options include immediate deletion after publishing, time-based retention for debugging, or archival to separate storage for compliance. - **Idempotency**: Consumers must handle duplicate event delivery gracefully. While the outbox pattern provides at-least-once delivery, network issues or reprocessing scenarios can cause duplicates. Design consumers with idempotent processing logic using techniques like deduplication keys or idempotency tokens. - **Testing**: Simulate failure scenarios (database rollbacks, broker unavailability, CDC lag) in testing environments to validate the complete event publishing pipeline. ## Outbox Pattern vs. Other Distributed Transaction Approaches The outbox pattern is one of several strategies for managing distributed transactions. Understanding when to use each approach helps you make informed architectural decisions. **Outbox Pattern vs. Saga Pattern**: The outbox pattern solves the dual-write problem within a single service, ensuring atomic database updates and event publishing. The [Saga Pattern](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions) solves the broader problem of coordinating transactions across multiple services. These patterns are complementary: - **Outbox**: Guarantees that a single service reliably publishes events when its local state changes - **Saga**: Coordinates a sequence of local transactions across multiple services, with each service using outbox to publish its events For example, an order fulfillment saga might involve: 1. Order Service creates order (using outbox to publish `OrderCreated`) 2. Inventory Service reserves items (using outbox to publish `InventoryReserved`) 3. Payment Service processes payment (using outbox to publish `PaymentCompleted`) Each service uses the outbox pattern internally, while the saga pattern coordinates the overall workflow. **Outbox Pattern vs. Two-Phase Commit (2PC)**: Traditional distributed transactions using two-phase commit guarantee ACID properties across multiple databases or services, but come with significant drawbacks: - **Outbox + CDC**: Asynchronous, eventual consistency, high availability, loose coupling - **2PC**: Synchronous, strong consistency, reduced availability, tight coupling The outbox pattern trades immediate consistency for better availability and scalability, making it more suitable for modern microservices architectures where services should remain independent. **Outbox Pattern vs. Event Sourcing**: [Event Sourcing](https://www.conduktor.io/glossary/cqrs-and-event-sourcing-with-kafka) stores all state changes as a sequence of events, making the event log the source of truth. The outbox pattern stores business data in traditional tables and uses events for inter-service communication. - **Outbox**: Traditional database model with event notifications - **Event Sourcing**: Event log as the primary storage, state derived from events Choose outbox when you want traditional database patterns with reliable event publishing. Choose event sourcing when you need complete audit history and time-travel capabilities as core requirements. ## Summary The outbox pattern elegantly solves the dual-write problem by leveraging database transactions to ensure atomic updates and event publishing. By persisting events to an outbox table within the same transaction as business data, applications gain guaranteed delivery and exactly-once semantics without the complexity of distributed transactions. Implementation approaches range from simple polling-based relays to sophisticated Change Data Capture solutions like Debezium. CDC-based implementations offer near real-time event publishing and natural integration with streaming platforms like Apache Kafka, enabling reliable foundations for event-driven architectures. While the pattern introduces some complexity and operational overhead, the benefits of data consistency and reliability make it a fundamental pattern for modern distributed systems. When building systems that require both strong consistency and event-driven communication, the outbox pattern is a proven, well-tested approach. ## Related Concepts - [Saga Pattern for Distributed Transactions](https://www.conduktor.io/glossary/saga-pattern-for-distributed-transactions) - Coordinating distributed transactions with reliable event publishing - [Event-Driven Architecture](https://www.conduktor.io/glossary/event-driven-architecture) - Architectural patterns that benefit from reliable event publishing - [Data Contracts for Reliable Pipelines](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) - Ensuring data quality in event-driven systems ## Sources and References 1. Richardson, C. (2024). "Pattern: Transactional Outbox". *Microservices.io*. https://microservices.io/patterns/data/transactional-outbox.html 2. Debezium Project. (2024). "Outbox Event Router". *Debezium Documentation*. https://debezium.io/documentation/reference/transformations/outbox-event-router.html 3. Fowler, M. (2017). "What do you mean by 'Event-Driven'?". *Martin Fowler's Blog*. https://martinfowler.com/articles/201701-event-driven.html 4. Narkhede, N., Shapira, G., & Palino, T. (2017). *Kafka: The Definitive Guide*. O'Reilly Media. 5. Kleppmann, M. (2017). *Designing Data-Intensive Applications*. O'Reilly Media. Chapter 11: Stream Processing. Source: https://www.conduktor.io/glossary/outbox-pattern-for-reliable-event-publishing --- # PII Detection and Handling in Event Streams Personally Identifiable Information (PII) refers to any data that can identify a specific individual. This includes obvious identifiers like social security numbers, email addresses, and phone numbers, as well as less obvious data like IP addresses, device IDs, or combinations of demographic information that could uniquely identify someone. In event streaming systems, PII presents unique challenges. Unlike batch processing where data can be scanned and cleaned before use, streaming data flows continuously and must be processed in real-time. A single event containing unprotected PII can propagate through multiple downstream systems in milliseconds, creating compliance risks and potential data breaches. For comprehensive strategies on preventing unintended PII exposure, see [PII Leakage Prevention](https://www.conduktor.io/glossary/pii-leakage-prevention). Consider an e-commerce platform streaming checkout events. These events might contain customer names, addresses, credit card numbers, and email addresses. Without proper PII handling, this sensitive data could end up in analytics databases, logs, monitoring dashboards, or third-party systems where it shouldn't be accessible. The stakes are high. Regulations like GDPR and CCPA impose strict requirements on how PII must be handled, with significant penalties for violations. Organizations need to detect and protect PII at the point of ingestion, not downstream where the data has already spread. For detailed guidance on GDPR compliance, see [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams). ## PII Detection Techniques in Event Streams Detecting PII in real-time streams requires automated approaches that can keep pace with high-volume data flows. Several techniques are commonly used: - **Pattern Matching and Regular Expressions**: The most straightforward approach uses regex patterns to identify common PII formats. Credit card numbers follow predictable patterns (using the Luhn algorithm, which validates credit card numbers by checking digit sequences), email addresses match standard formats, and social security numbers have consistent structures. While fast and deterministic, this approach can generate false positives and misses PII in unexpected formats. - **Schema-Based Detection**: When using schema registries with formats like Avro or Protobuf, field names and metadata can indicate PII. Fields named "email", "ssn", or "credit_card" are obvious candidates. Schema annotations can explicitly mark fields as containing PII, enabling automated detection and handling policies. For comprehensive schema management strategies, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management) and [Data Classification and Tagging Strategies](https://www.conduktor.io/glossary/data-classification-and-tagging-strategies). - **Machine Learning Classifiers**: More sophisticated systems use ML models trained on labeled PII datasets. Modern 2025 tools include: - **AWS Macie**: Provides automated PII discovery using machine learning, supporting 40+ PII types including credentials, financial data, and health records - **Google Cloud DLP API**: Offers 150+ built-in detectors with customizable detection rules and de-identification templates - **Microsoft Presidio**: Open-source PII detection and anonymization framework with pre-trained models for multiple languages - **Pilar (AWS Labs)**: Lightweight open-source library for detecting PII in structured and unstructured data These models can identify PII based on context and content patterns, even when field names don't reveal the data's nature. However, ML approaches add latency (typically 10-50ms per event) and computational overhead to stream processing. **Contextual Analysis**: Some PII only becomes identifiable in combination with other data. A zip code alone isn't PII, but combined with birthdate and gender, it can uniquely identify individuals. Contextual detection requires analyzing multiple fields together, adding complexity to stream processing logic. ## Handling Strategies: Masking, Redaction, and Tokenization Once PII is detected, it must be protected. Different strategies offer different tradeoffs between security, utility, and reversibility: - **Masking**: Replaces parts of PII with placeholder characters. For example, a credit card number "4532-1234-5678-9010" becomes "4532-****-****-9010". Masking preserves partial utility (the first and last digits might be needed for customer service) while protecting the full value. It's irreversible but maintains some data format. - **Redaction**: Completely removes or replaces PII with generic values. An email "user@example.com" might become "[REDACTED]" or "anonymous@example.com". This provides strong protection but eliminates all utility of the original data. - **Hashing**: Applies one-way cryptographic functions to PII. The same input always produces the same hash, enabling joins and deduplication without revealing the original value. However, hashes of common values (like popular email domains) can be reverse-engineered through rainbow tables (precomputed tables of hash values used to crack password hashes). Use salted hashing with algorithms like SHA-256 or Argon2 to mitigate this risk. - **Tokenization**: Replaces PII with randomly generated tokens stored in a secure token vault (a dedicated, access-controlled database or key management service). The original value can be retrieved when needed (by authorized systems) but the token itself reveals nothing. This preserves data utility while maintaining strong security, though it requires maintaining a token mapping database. - **Encryption**: Applies symmetric or asymmetric encryption to PII. Unlike hashing, encryption is reversible with the proper key. This works well when some downstream systems need access to the original data, but key management becomes critical. For comprehensive encryption strategies, see [Encryption at Rest and in Transit for Kafka](https://www.conduktor.io/glossary/encryption-at-rest-and-in-transit-for-kafka). In streaming architectures, these techniques are typically applied at ingestion time using stream processors or Single Message Transforms (SMTs - reusable transformation components in Kafka Connect that modify messages in flight) in Kafka Connect. For detailed masking and anonymization techniques, see [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming). ### Field-Level Encryption for PII Protection Field-level encryption provides targeted protection for PII while maintaining data utility for non-sensitive fields. Conduktor Gateway, a Kafka proxy, implements field-level encryption through [interceptors](https://docs.conduktor.io/guide/conduktor-concepts/interceptors), enabling schema-based encryption (where PII fields tagged in schemas are automatically encrypted) or list-based encryption (where specific fields are defined in configuration). This approach protects sensitive data like SSN, credit cards, and email addresses while leaving non-sensitive fields available for analytics. Gateway integrates with enterprise Key Management Systems (Vault, Azure Key Vault, AWS KMS, GCP KMS) using envelope encryption for secure key management. Encryption can occur on-produce (PII encrypted before entering Kafka) or on-consume (PII encrypted before delivery to specific consumers), enabling role-based access where different consumer groups receive different levels of PII visibility. Combined with data masking policies, this provides defense-in-depth protection that addresses GDPR, HIPAA, PCI-DSS, and CCPA requirements. For complete implementation guidance, see [Encrypting Kafka with Conduktor](https://docs.conduktor.io/guide/use-cases/encrypt-kafka). ## PII in Data Streaming Architectures Modern streaming platforms provide various mechanisms for PII handling: **Apache Kafka 4.0+ with KRaft**: Kafka 4.0 brings improved security foundations with KRaft mode (removing ZooKeeper dependency), making PII protection implementations more streamlined. The simplified architecture enables better audit trails and consistent policy enforcement. For details on KRaft's benefits, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). Kafka Connect offers SMTs that can mask, redact, or encrypt fields during data movement (see [Kafka Connect Single Message Transforms](https://www.conduktor.io/glossary/kafka-connect-single-message-transforms)). Kafka Streams applications can implement custom PII detection and handling logic within stream processing topologies (see [Introduction to Kafka Streams](https://www.conduktor.io/glossary/introduction-to-kafka-streams)). Here's a practical example using Kafka Connect SMT to mask PII fields: ```json { "name": "customer-events-connector", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "connection.url": "jdbc:postgresql://localhost:5432/customers", "table.whitelist": "customer_events", "mode": "incrementing", "incrementing.column.name": "id", "topic.prefix": "raw-", "transforms": "maskEmail,hashSSN,tokenizeCard", "transforms.maskEmail.type": "org.apache.kafka.connect.transforms.MaskField$Value", "transforms.maskEmail.fields": "email", "transforms.maskEmail.replacement": "***MASKED***", "transforms.hashSSN.type": "com.github.jcustenborder.kafka.connect.transform.hash.Hash$Value", "transforms.hashSSN.fields": "ssn", "transforms.hashSSN.algorithm": "SHA256", "transforms.tokenizeCard.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.tokenizeCard.include": "credit_card", "transforms.tokenizeCard.renames": "credit_card:card_token" } } ``` **Apache Flink 1.18+**: Flink's rich DataStream API enables complex PII detection and transformation logic. The latest versions include enhanced Python support and improved state backends optimized for stateful PII operations. Custom functions can implement ML-based detection, while built-in operators handle filtering and transformation. Flink's state management can maintain tokenization mappings or detection models. For comprehensive Flink guidance, see [What is Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) and [Flink State Management and Checkpointing](https://www.conduktor.io/glossary/flink-state-management-and-checkpointing). Example PII masking with Flink DataStream API: ```java import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.api.common.functions.MapFunction; public class PiiMaskingJob { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream events = env .addSource(new FlinkKafkaConsumer<>( "raw-customer-events", new CustomerEventSchema(), kafkaProps )); // Apply PII masking transformation DataStream maskedEvents = events.map(new MapFunction() { @Override public CustomerEvent map(CustomerEvent event) { // Mask email - show first character and domain only String email = event.getEmail(); if (email != null && email.contains("@")) { String[] parts = email.split("@"); event.setEmail(parts[0].charAt(0) + "***@" + parts[1]); } // Hash SSN with SHA-256 String ssn = event.getSsn(); if (ssn != null) { event.setSsn(hashWithSHA256(ssn).substring(0, 16)); } // Tokenize credit card (simplified - use proper tokenization service) String creditCard = event.getCreditCard(); if (creditCard != null) { event.setCreditCard(generateToken(creditCard)); } return event; } private String hashWithSHA256(String input) { try { MessageDigest digest = MessageDigest.getInstance("SHA-256"); byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8)); return Base64.getEncoder().encodeToString(hash); } catch (NoSuchAlgorithmException e) { throw new RuntimeException(e); } } private String generateToken(String input) { // In production, call tokenization service return "TOKEN-" + UUID.randomUUID().toString(); } }); maskedEvents.addSink(new FlinkKafkaProducer<>( "masked-customer-events", new CustomerEventSchema(), kafkaProps )); env.execute("PII Masking Job"); } } ``` - **Schema Registry Integration**: Schema registries enable schema-level PII marking. Fields can be tagged with sensitivity classifications, and consumers can automatically apply appropriate handling based on these tags. This creates a centralized governance model where PII policies are defined once in schemas rather than in every consuming application. - **Conduktor Gateway**: A modern approach uses Conduktor Gateway, which acts as a transparent proxy between applications and Kafka brokers. Gateway can apply [PII masking policies](https://docs.conduktor.io/guide/conduktor-in-production/admin/data-mask) dynamically based on consumer identity, enabling different data views for different teams without modifying application code. For authentication and authorization, see [Gateway authentication and authorization](https://docs.conduktor.io/guide/conduktor-concepts/gw-authen-author): ```yaml # Conduktor Gateway interceptor configuration interceptors: - type: dataMasking priority: 100 config: rules: - topic: customer-events field: $.email maskingType: EMAIL_DOMAIN applies_to: consumer_groups: [analytics-team, ml-training] - topic: customer-events field: $.ssn maskingType: HASH_SHA256 applies_to: consumer_groups: [analytics-team, support-team, ml-training] - topic: customer-events field: $.credit_card maskingType: REDACT applies_to: consumer_groups: [analytics-team] # Production systems get full access - topic: customer-events field: $.credit_card maskingType: NONE applies_to: consumer_groups: [payment-processor, fraud-detection] ``` **Stream Processing Patterns - The PII Firewall**: A common architectural pattern is the "PII firewall" - a dedicated stream processing job that sits between producers and the main Kafka cluster. All events flow through this firewall, which detects and protects PII before data reaches downstream consumers. This centralizes PII handling and ensures consistent policy enforcement. ![pii-detection-and-handling-in-event-streams diagram 1](https://www.conduktor.io/assets/images/glossary/pii-detection-and-handling-in-event-streams-0.webp) ## Compliance and Governance Considerations PII handling in event streams must align with regulatory requirements: **GDPR and Right to Erasure**: GDPR grants individuals the right to have their personal data deleted. In immutable log systems like Kafka, this is challenging. Modern 2025 solutions include: - **Tombstone Messages**: Send null-value messages with the user's key to Kafka compacted topics. During log compaction, Kafka removes all previous messages with that key, effectively deleting the data. - **Tokenization-Based Erasure**: Store PII in a secure token vault and use tokens in Kafka. When a user requests deletion, remove the token mapping - all downstream systems now see only meaningless tokens. - **Topic Compaction with Custom Deleters**: Use Kafka's log compaction with custom cleanup policies that recognize deletion markers. - **Time-Based Retention**: Configure short retention periods (7-30 days) for PII-containing topics, ensuring old data automatically expires. Example tombstone message for user deletion: ```java // Send tombstone to trigger deletion in compacted topic ProducerRecord tombstone = new ProducerRecord<>( "customer-pii-compacted", "customer-123", // User ID as key null // Null value signals deletion ); // Add deletion metadata in headers tombstone.headers().add("deletion-reason", "GDPR Right to Erasure".getBytes()); tombstone.headers().add("deletion-timestamp", String.valueOf(System.currentTimeMillis()).getBytes()); tombstone.headers().add("deleted-by", "privacy-team".getBytes()); producer.send(tombstone); ``` - **Data Residency**: Some regulations require PII to remain within specific geographic boundaries. Multi-region streaming architectures must prevent PII from flowing to unauthorized regions, potentially using topic-level geo-fencing or region-specific clusters. Modern Kafka deployments use cluster linking with filters to enforce data sovereignty requirements. For multi-region architecture patterns, see [Disaster Recovery Strategies for Kafka Clusters](https://www.conduktor.io/glossary/disaster-recovery-strategies-for-kafka-clusters). - **Audit Trails**: Compliance often requires detailed logs of who accessed PII and when. Streaming systems need to capture access patterns, transformation operations, and downstream consumption for audit purposes. Modern platforms integrate with SIEM systems for real-time audit monitoring. For comprehensive audit strategies, see [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) and [Streaming Audit Logs](https://www.conduktor.io/glossary/streaming-audit-logs). - **Retention Policies**: Different PII types have different retention requirements. Credit card data (PCI-DSS) requires minimal retention after transaction completion, while tax-related PII might need seven-year retention. Topic-level retention policies and time-based deletion strategies help meet these requirements: ```bash # Set retention policy for PII topic (7 days) kafka-configs --bootstrap-server localhost:9092 \ --entity-type topics \ --entity-name customer-pii \ --alter \ --add-config retention.ms=604800000 # Enable log compaction for compacted PII topic kafka-configs --bootstrap-server localhost:9092 \ --entity-type topics \ --entity-name customer-pii-compacted \ --alter \ --add-config cleanup.policy=compact ``` ## Implementing PII Protection in Production Building production-grade PII protection requires careful planning: - **Start with Classification**: Inventory your data streams and classify fields by sensitivity. Not everything is PII, and not all PII requires the same protection level. Create a data catalog that documents what PII exists and where it flows. - **Implement Defense in Depth**: Don't rely on a single protection mechanism. Combine schema-based detection, pattern matching, and access controls. If one layer fails, others provide backup protection. - **Consider Performance Impact**: Real-time PII detection and transformation add latency and computational overhead. Benchmark your approaches under realistic load. Sometimes simpler pattern matching outperforms ML-based detection when latency budgets are tight. - **Use Specialized Tools**: Platforms like Conduktor provide built-in data masking capabilities, schema validation, and audit trails specifically designed for Kafka environments. These tools can accelerate implementation and ensure consistent policy enforcement across teams. For access control patterns that complement PII protection, see [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming) and [Kafka ACLs and Authorization Patterns](https://www.conduktor.io/glossary/kafka-acls-and-authorization-patterns). - **Test Thoroughly**: Validate that PII detection catches actual sensitive data without excessive false positives. Test that masked or tokenized data still enables necessary downstream analytics. Verify that compliance requirements are met through regular audits. - **Monitor Continuously**: Implement alerting for potential PII leaks. Monitor for schema changes that might introduce new PII fields. Track PII-related metrics like detection rates, masking operations, and access patterns. For monitoring strategies, see [Kafka Cluster Monitoring and Metrics](https://www.conduktor.io/glossary/kafka-cluster-monitoring-and-metrics) and [Data Observability](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars). ## Summary PII detection and handling in event streams is a critical requirement for modern data platforms. The real-time nature of streaming systems demands automated, performant approaches to identifying and protecting sensitive data before it propagates through downstream systems. Effective PII protection combines multiple detection techniques - from simple pattern matching to ML-based classification - with appropriate handling strategies like masking, tokenization, or encryption. The choice of technique depends on data utility requirements, performance constraints, and compliance obligations. Streaming platforms like Kafka and Flink provide the building blocks for PII protection through stream processors, SMTs, and schema registries. However, successful implementation requires careful planning, thorough testing, and continuous monitoring to ensure sensitive data remains protected while maintaining the value of real-time data pipelines. Organizations that treat PII protection as a first-class concern in their streaming architectures meet compliance requirements, build trust with customers, and reduce the risk of costly data breaches. ## Related Concepts - [PII Leakage Prevention](https://www.conduktor.io/glossary/pii-leakage-prevention) - Strategies for preventing unintended PII exposure in logs, monitoring systems, and misconfigured streams. - [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming) - Detailed techniques for masking, tokenization, and anonymization in real-time streams. - [GDPR Compliance for Data Teams](https://www.conduktor.io/glossary/gdpr-compliance-for-data-teams) - How PII handling supports GDPR requirements including right to erasure and data minimization. ## Sources and References 1. **GDPR Official Text** - European Commission comprehensive resource on GDPR requirements including PII definitions and processing obligations: https://gdpr.eu/ 2. **NIST Special Publication 800-122** - Guide to Protecting the Confidentiality of Personally Identifiable Information with technical guidance on PII safeguards and de-identification techniques: https://csrc.nist.gov/publications/detail/sp/800-122/final 3. **Apache Kafka Security Documentation** - Official documentation on Kafka security features including encryption and access controls: https://kafka.apache.org/documentation/#security 4. **Apache Flink Security Documentation** - Details on implementing security controls in Flink stream processing: https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/security/ 5. **California Consumer Privacy Act (CCPA)** - California Attorney General official resource detailing consumer privacy rights and business obligations: https://oag.ca.gov/privacy/ccpa 6. **OWASP Top 10 Privacy Risks** - Industry-standard reference for understanding and mitigating privacy risks: https://owasp.org/www-project-top-10-privacy-risks/ Source: https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams --- # PII Leakage Prevention: Protecting Personal Data in Streaming The consequences extend beyond technical issues. Unintended PII exposure violates data protection regulations like GDPR, CCPA, and HIPAA, leading to substantial fines, legal liability, and reputational damage. Organizations must implement comprehensive detection and prevention strategies to protect personal data throughout the streaming pipeline. ## The Hidden Risk of PII Leakage Personally Identifiable Information (PII) leakage represents one of the most serious security risks in modern data streaming architectures. Unlike intentional data breaches, PII leakage occurs through unintended exposure, when sensitive personal data appears in logs, error messages, monitoring systems, or misconfigured data streams where it shouldn't exist. In streaming systems that process millions of events per second, a single misconfiguration can expose vast amounts of personal data before anyone notices. ![PII leakage vectors and detection points in streaming systems](https://www.conduktor.io/assets/images/glossary/pii-leakage-prevention-0.webp) ## Understanding PII Leakage Vectors in Streaming PII leakage in streaming systems occurs through multiple vectors, each requiring specific attention: - **Application Logs and Error Messages**: Developers often log full event payloads during debugging, inadvertently capturing credit card numbers, email addresses, or social security numbers. Exception stack traces may include sensitive data from failed transactions or validation errors. - **Monitoring and Observability Systems**: Metrics collection, distributed tracing, and monitoring dashboards can expose PII when event samples are captured for analysis. A single trace showing a complete message payload might reveal customer information to anyone with dashboard access. - **Misconfigured Topics and Streams**: Data may flow to the wrong Kafka topics, event hubs, or S3 buckets due to routing errors, incorrect access controls, or development configurations accidentally deployed to production. Topics intended for internal analytics might receive raw customer data instead of anonymized aggregates. - **Debug Output and Development Tools**: Schema registry snapshots, consumer group testing tools, and data quality dashboards often display sample records. Without proper masking, these tools become PII exposure points accessible to broad engineering teams. - **Third-Party Integrations**: Data sent to external analytics platforms, monitoring services, or partner systems may contain more PII than necessary for the intended use case. API calls that include full event contexts rather than minimal required fields create unnecessary exposure. ## Detection Approaches and Technologies Effective PII detection requires multiple complementary approaches: **Pattern Matching and Regular Expressions**: The foundational detection method uses regex patterns to identify common PII formats, credit card numbers following Luhn validation (a checksum algorithm that validates card number authenticity), email addresses matching RFC specifications, phone numbers in various international formats, and social security numbers with characteristic digit patterns. While fast and deterministic, pattern matching produces false positives (matches that aren't actually PII) and misses contextual PII like names or addresses without distinctive formats. Here's a practical example of pattern-based PII detection in Kafka Streams: ```java import org.apache.kafka.streams.StreamsBuilder; import org.apache.kafka.streams.kstream.KStream; import java.util.regex.Pattern; import java.util.regex.Matcher; public class PIIDetectionStreams { // Common PII detection patterns private static final Pattern EMAIL_PATTERN = Pattern.compile( "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}"); private static final Pattern SSN_PATTERN = Pattern.compile( "\\b\\d{3}-\\d{2}-\\d{4}\\b"); private static final Pattern CREDIT_CARD_PATTERN = Pattern.compile( "\\b\\d{4}[\\s-]?\\d{4}[\\s-]?\\d{4}[\\s-]?\\d{4}\\b"); public static void buildPipeline(StreamsBuilder builder) { KStream events = builder.stream("raw-events"); // Detect and route messages containing PII events.filter((key, value) -> containsPII(value)) .to("pii-detected-events"); // Route clean messages to standard processing events.filterNot((key, value) -> containsPII(value)) .to("clean-events"); } private static boolean containsPII(String message) { return EMAIL_PATTERN.matcher(message).find() || SSN_PATTERN.matcher(message).find() || CREDIT_CARD_PATTERN.matcher(message).find(); } private static boolean isValidLuhn(String cardNumber) { // Luhn algorithm implementation for credit card validation int sum = 0; boolean alternate = false; for (int i = cardNumber.length() - 1; i >= 0; i--) { int n = Integer.parseInt(cardNumber.substring(i, i + 1)); if (alternate) { n *= 2; if (n > 9) n -= 9; } sum += n; alternate = !alternate; } return (sum % 10 == 0); } } ``` - **Machine Learning Classification**: ML models trained on labeled datasets can identify contextual PII that pattern matching misses. Named entity recognition (NER, Natural Language Processing models that identify and classify named entities like person names, locations, and organizations) models detect person names, locations, and organizations within unstructured text. Classification models learn to identify sensitive fields based on data characteristics, field names, and surrounding context. These approaches reduce false positives but require training data and model maintenance. - **Modern AI-Powered PII Detection (2025)**: Cloud providers and open-source projects now offer sophisticated ML-based PII detection that integrates directly with streaming platforms: - **AWS Macie for Kafka**: Integrates with Amazon MSK to automatically detect PII in streaming data using machine learning models trained on vast datasets of sensitive information patterns - **Google Cloud DLP API**: Provides real-time PII scanning with 150+ built-in detectors for global PII types, callable from stream processors - **Azure Purview with Event Hubs**: Offers ML-based classification for Azure event streams with automatic sensitivity labeling - **Microsoft Presidio**: Open-source PII detection and anonymization framework supporting 20+ languages, deployable as a sidecar service - **AWS Pilar (Pilar Labs)**: Open-source privacy-preserving analytics framework with built-in PII detection Here's an example integrating Google Cloud DLP API with Flink for real-time PII detection: ```python from pyflink.datastream import StreamExecutionEnvironment from pyflink.datastream.functions import MapFunction from google.cloud import dlp_v2 import json class DLPPIIDetector(MapFunction): def __init__(self, project_id): self.project_id = project_id self.dlp_client = None def open(self, runtime_context): # Initialize DLP client once per task self.dlp_client = dlp_v2.DlpServiceClient() self.parent = f"projects/{self.project_id}" def map(self, event_json): event = json.loads(event_json) # Inspect content for PII item = {"value": str(event)} response = self.dlp_client.inspect_content( request={ "parent": self.parent, "inspect_config": { "info_types": [ {"name": "EMAIL_ADDRESS"}, {"name": "PHONE_NUMBER"}, {"name": "CREDIT_CARD_NUMBER"}, {"name": "US_SOCIAL_SECURITY_NUMBER"} ], "min_likelihood": dlp_v2.Likelihood.POSSIBLE }, "item": item } ) # Tag event with detected PII types if response.result.findings: pii_types = [f.info_type.name for f in response.result.findings] event['_pii_detected'] = True event['_pii_types'] = pii_types event['_classification'] = 'CONFIDENTIAL' else: event['_pii_detected'] = False event['_classification'] = 'INTERNAL' return json.dumps(event) # Usage in Flink pipeline env = StreamExecutionEnvironment.get_execution_environment() stream = env.from_source(kafka_source, watermark_strategy, "raw-events") classified_stream = stream.map(DLPPIIDetector(project_id="my-project")) classified_stream.sink_to(kafka_sink) ``` - **Data Scanning and Profiling**: Systematic scanning analyzes data distribution, uniqueness, and statistical properties to flag potentially sensitive fields. High-cardinality string fields with person-name-like characteristics, fields containing consistent geographic patterns, or columns showing birthdate distributions warrant investigation. Profiling complements pattern matching by identifying sensitive data not matching known formats. - **Schema-Level Metadata**: The most reliable detection approach marks sensitive fields directly in schemas. Avro, Protobuf, and JSON schemas can include annotations or tags identifying fields containing PII. This metadata-driven approach enables consistent handling across the pipeline without per-message inspection overhead. For comprehensive schema management patterns, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). ### Comparing Detection Approaches | Approach | Speed | Accuracy | Maintenance | Best For | |----------|-------|----------|-------------|----------| | **Pattern Matching** | Very Fast | Medium (false positives) | Low | Structured PII (SSN, credit cards, emails) | | **ML Classification** | Medium | High | High (model training) | Contextual PII (names, addresses) | | **Cloud AI Services** | Medium | Very High | Low (managed service) | Enterprise deployments, multi-language PII | | **Schema Metadata** | Very Fast | Perfect (if accurate) | Medium (schema governance) | Structured streaming with schema registry | | **Data Profiling** | Slow | Medium | Medium | Data discovery, unknown PII identification | ## Prevention Strategies and Best Practices Prevention strategies operate at multiple levels: **Data Masking and Tokenization**: Replace PII with masked values or tokens before data enters logs, metrics, or lower-security environments. Format-preserving encryption (FPE) maintains data characteristics for testing while protecting actual values, for example, encrypting a 16-digit credit card number into another valid-looking 16-digit number that preserves the format but protects the actual value. Tokenization replaces sensitive fields with random identifiers, storing the mapping in a secure token vault accessible only to authorized systems. For comprehensive masking techniques and implementation patterns, see [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming). Here's a practical tokenization example using Kafka Streams with a token vault: ```java import org.apache.kafka.streams.kstream.KStream; import org.apache.kafka.streams.kstream.ValueMapper; import java.util.Map; import java.util.UUID; import java.util.concurrent.ConcurrentHashMap; public class TokenizationProcessor { // Token vault (in production, use Redis or a dedicated token service) private static final Map tokenVault = new ConcurrentHashMap<>(); private static final Map reverseVault = new ConcurrentHashMap<>(); public static void buildTokenizationPipeline(StreamsBuilder builder) { KStream events = builder.stream("raw-customer-events"); // Tokenize PII fields KStream tokenized = events.mapValues(event -> { CustomerEvent tokenizedEvent = event.copy(); // Tokenize email if (event.email != null) { tokenizedEvent.email = tokenize(event.email, "email"); } // Tokenize SSN if (event.ssn != null) { tokenizedEvent.ssn = tokenize(event.ssn, "ssn"); } return tokenizedEvent; }); tokenized.to("tokenized-customer-events"); } private static String tokenize(String value, String fieldType) { // Check if we've already tokenized this value if (reverseVault.containsKey(value)) { return reverseVault.get(value); } // Generate new token String token = fieldType + "_" + UUID.randomUUID().toString().replace("-", ""); // Store in vaults tokenVault.put(token, value); reverseVault.put(value, token); return token; } // Detokenization for authorized services public static String detokenize(String token) { return tokenVault.get(token); } } ``` **Field-Level Encryption for Highly Sensitive Data**: For data classified as highly sensitive, implement field-level encryption that encrypts specific fields while leaving others in plaintext. This allows analytics on non-sensitive fields while protecting PII: ```python from cryptography.fernet import Fernet from confluent_kafka import Producer import json class FieldLevelEncryptionProducer: def __init__(self, bootstrap_servers, encryption_key): self.producer = Producer({'bootstrap.servers': bootstrap_servers}) self.cipher = Fernet(encryption_key) def encrypt_field(self, value): """Encrypt a single field value""" return self.cipher.encrypt(value.encode()).decode() def send_customer_event(self, customer_data): """Send customer event with PII fields encrypted""" # Encrypt PII fields encrypted_event = { 'customer_id': customer_data['customer_id'], # Not encrypted 'email': self.encrypt_field(customer_data['email']), # Encrypted 'ssn': self.encrypt_field(customer_data['ssn']), # Encrypted 'purchase_amount': customer_data['purchase_amount'], # Not encrypted 'timestamp': customer_data['timestamp'] # Not encrypted } self.producer.produce( 'customer-events', key=customer_data['customer_id'], value=json.dumps(encrypted_event) ) self.producer.flush() # Usage key = Fernet.generate_key() # In production, use KMS (AWS KMS, Azure Key Vault, etc.) producer = FieldLevelEncryptionProducer('localhost:9092', key) customer = { 'customer_id': 'CUST12345', 'email': 'john.doe@example.com', 'ssn': '123-45-6789', 'purchase_amount': 299.99, 'timestamp': '2025-03-15T10:30:00Z' } producer.send_customer_event(customer) ``` - **Schema Enforcement and Validation**: Use schema registries to define and enforce which fields contain PII. Reject messages that include sensitive fields in contexts where they shouldn't appear. Implement schema evolution policies that require approval and documentation when adding PII fields to existing schemas. For comprehensive schema management patterns, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). - **Access Control and Topic Segregation**: Implement fine-grained ACLs ensuring only authorized applications and users can access topics containing PII. Separate sensitive and non-sensitive data into different topics with distinct security policies. Use encryption at rest and in transit for all streams carrying personal data. For detailed access control implementation, see [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming). - **Data Classification and Tagging**: Systematically classify data by sensitivity level and use Kafka headers or schema metadata to tag messages with classification information. This enables automated policy enforcement based on data sensitivity. For comprehensive classification strategies, see [Data Classification and Tagging Strategies](https://www.conduktor.io/glossary/data-classification-and-tagging-strategies). - **Data Minimization**: Design applications to avoid collecting, processing, or transmitting PII unless absolutely necessary. Aggregate data early in the pipeline, removing individual identifiers before downstream processing. Question whether each PII field truly needs to flow through the entire system or could be stripped at ingestion. - **Secure Logging Practices**: Configure logging frameworks to automatically redact known PII patterns. Implement structured logging with explicit field-level control over what gets logged. Never log complete event payloads in production environments, log only specific fields required for debugging. ## Real-Time Detection in Stream Processing Stream processing applications must detect PII in real-time as data flows through the pipeline: - **Inline Scanning**: Deploy scanning logic within stream processors using Kafka Streams, Flink, or Spark Streaming. Each message passes through PII detection functions that apply pattern matching and ML classification. Detected PII triggers immediate actions, masking, routing to quarantine topics (secure holding topics for investigation), or alerting security teams. - **Sidecar Processors**: Run dedicated PII scanning applications that consume from all topics, continuously monitoring for leakage. A sidecar processor is a separate application that runs alongside your main stream processing pipeline, providing cross-cutting concerns like security scanning without modifying business logic. This approach separates security logic from business logic and allows centralized policy management. However, it doubles message processing and introduces detection latency. - **Schema Registry Integration**: Validate messages against schemas annotated with PII metadata. Reject or quarantine messages that contain PII fields when connecting to topics or endpoints that shouldn't receive sensitive data. - **Modern Governance Platforms (2025)**: Advanced data governance platforms now provide comprehensive PII leakage prevention capabilities: - **Conduktor Gateway and Platform**: Conduktor provides multiple layers of PII protection: - **Conduktor Gateway**: Acts as a proxy between applications and Kafka brokers, enforcing [PII masking policies](https://docs.conduktor.io/guide/conduktor-in-production/admin/data-mask) in real-time without modifying application code. Gateway can apply different masking rules based on consumer identity, enabling a single topic to serve multiple teams with different data access levels. - **Real-time Data Masking**: Define masking policies centrally that automatically apply to data streams: ```yaml # Conduktor Gateway PII masking configuration interceptors: - type: safeguard config: rules: - name: mask-customer-pii action: MASK condition: topic == 'customer-events' fields: - path: $.email algorithm: SHA256 # Hash emails - path: $.ssn algorithm: REDACT # Completely redact SSNs - path: $.creditCard algorithm: MASK_KEEP_LAST_4 # Keep last 4 digits appliesTo: - consumerGroup: analytics-team - consumerGroup: ml-training - name: detect-accidental-pii action: BLOCK condition: topic != 'customer-events' && message.contains('ssn') alert: true ``` - **Audit Logs and PII Tracking**: Conduktor Gateway's audit logs capture all producer/consumer connections, providing visibility into which applications interact with which topics and helping identify potential PII exposure points. - **Data Quality Rules**: Define and enforce rules that detect unexpected PII in topics that shouldn't contain it: ```yaml # Data quality rule in Conduktor quality_rules: - name: no-pii-in-analytics topic: analytics-events severity: CRITICAL condition: | message.doesNotMatch('\\b\\d{3}-\\d{2}-\\d{4}\\b') && // No SSNs message.doesNotMatch('[a-z0-9._%+-]+@[a-z0-9.-]+\\.[a-z]{2,}') // No emails action: QUARANTINE alert: channels: [slack, pagerduty] ``` **Kafka 4.0+ with KRaft Mode**: The removal of ZooKeeper in Kafka 4.0 brings improved audit capabilities that enhance PII leakage prevention: - **Enhanced Audit Logging**: KRaft's metadata management provides more detailed audit trails, tracking who accessed PII-containing topics and when - **Faster ACL Enforcement**: Improved authorization performance enables more granular, field-level access controls - **Centralized Metadata**: Simplified architecture makes it easier to implement consistent PII protection policies across the entire cluster For comprehensive policy enforcement patterns, see [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming). **Handling Detected PII**: When PII is detected where it shouldn't exist: - **Quarantine**: Route messages to secure holding topics for investigation - **Alert**: Notify security teams immediately with details about the leakage vector - **Redact**: Automatically mask detected PII before allowing further processing - **Audit**: Log all detections for compliance reporting and trend analysis ## Compliance and Regulatory Context PII protection directly supports regulatory compliance: - **GDPR (General Data Protection Regulation)**: Requires organizations to implement appropriate technical measures to protect personal data. PII leakage prevention demonstrates "privacy by design" principles mandated by Article 25. Data breach notification requirements (Article 33) apply to unintended exposure, making detection speed critical, breaches must be reported within 72 hours of discovery. - **CCPA and CPRA (California Consumer Privacy Act, enhanced by California Privacy Rights Act of 2023)**: Mandates reasonable security procedures to protect personal information. The 2023 CPRA amendments strengthened enforcement, introducing the California Privacy Protection Agency with expanded audit powers. PII leakage into logs or third-party systems may violate consumer privacy rights, particularly the right to know what personal information is collected and shared. Organizations must implement risk assessments and automated decision-making safeguards. - **HIPAA (Health Insurance Portability and Accountability Act)**: Requires safeguards to protect Protected Health Information (PHI). Unintended exposure of PHI through logs, monitoring systems, or misconfigured streams constitutes a breach requiring notification and potential penalties. The 2025 updates to the HIPAA Security Rule emphasize real-time monitoring and automated controls for PHI in streaming systems. - **Documentation and Evidence**: Compliance audits require demonstrating PII protection measures. Maintain documentation of detection mechanisms, prevention policies, scanning results, and incident responses. Regular reporting shows ongoing commitment to data protection. ## Building a Comprehensive PII Protection Program Effective PII protection requires organizational commitment: - **Establish PII Inventory and Classification**: Document what PII exists in your systems, where it flows, and its sensitivity level. Create classification tiers (e.g., high-sensitivity for SSNs and credit cards, medium for names and emails) that determine protection requirements. - **Define Policies and Standards**: Create clear policies specifying where PII can exist, how it must be protected, and who can access it. Standardize masking formats, encryption requirements, and access control patterns. Document exceptions and their approval processes. - **Implement Technical Controls**: Deploy the detection and prevention technologies appropriate for your architecture. Start with pattern matching for high-risk PII like credit card numbers, expand to ML-based detection for contextual PII, and ultimately implement schema-driven protection for comprehensive coverage. - **Monitor and Alert**: Create dashboards showing PII detection metrics, where leakage occurs, which applications are involved, and trending patterns. Set up alerts for unusual PII exposure volumes or new leakage vectors. Review alerts regularly and investigate all incidents. - **Training and Awareness**: Educate developers about PII risks in streaming systems. Provide secure coding guidelines, logging best practices, and testing procedures. Make PII protection part of code review checklists and architecture review processes. - **Testing and Validation**: Regularly test PII protection measures using synthetic data that mimics real PII. Attempt to deliberately leak test PII through various vectors to verify detection systems work correctly. Include PII protection testing in CI/CD pipelines. - **Incident Response**: Prepare documented procedures for handling detected PII leakage, how to assess scope, who to notify, how to remediate, and what to document. Practice incident response through tabletop exercises. ## Tools and Platforms for PII Detection Modern streaming platforms provide PII protection capabilities across multiple categories: **Data Governance and Cataloging Platforms**: - **Conduktor Platform**: Comprehensive stream governance with PII detection, data quality rules, and real-time masking via Conduktor Gateway. Provides unified visibility across Kafka environments through audit logs and policy enforcement. - **OpenMetadata**: Open-source data catalog with native Kafka integration, automated lineage tracking, and ML-based classification suggestions for identifying PII fields. - **DataHub (LinkedIn)**: Real-time metadata management with Kafka Connect integration, enabling classification propagation across streaming and batch systems. **Data Loss Prevention (DLP) Solutions**: - **Enterprise DLP Platforms**: Solutions like Symantec DLP, Forcepoint, and McAfee Total Protection can monitor streaming systems, applying pattern matching and ML detection to identify PII exposure in real-time. These integrate with SIEM platforms for comprehensive security monitoring. - **Cloud-Native DLP**: AWS Macie (with MSK integration), Google Cloud DLP API, and Azure Purview provide managed PII detection that integrates directly with cloud streaming services. **Schema Registries and Management**: - **Open-Source Schema Registries**: Apache Schema Registry, AWS Glue Schema Registry, and Apicurio Registry support schema-level metadata for marking sensitive fields, enabling policy enforcement based on schema annotations. - **Schema Governance**: Tools integrate with CI/CD pipelines to validate that schema changes don't accidentally introduce PII into topics that shouldn't contain it. **Stream Processing Frameworks**: - **Apache Kafka Streams**: Lightweight library for building inline PII detection and masking, running within application JVMs with minimal operational overhead. - **Apache Flink**: Powerful stateful stream processing with advanced PII detection capabilities, including windowed aggregation for differential privacy and k-anonymity. - **Spark Streaming**: Batch-oriented stream processing suitable for higher-latency PII scanning and profiling workloads. **AI-Powered Detection Services (2025)**: - **AWS Macie for Kafka**: ML-powered PII detection integrated with Amazon MSK, supporting 150+ PII types across multiple languages and regions. - **Google Cloud DLP API**: Real-time scanning API with customizable detection rules, callable from any stream processor with sub-100ms latency. - **Azure Purview with Event Hubs**: Microsoft's data governance platform with automated PII classification for Azure streaming services. - **Microsoft Presidio (Open Source)**: Privacy-preserving PII detection and anonymization framework supporting 20+ languages, deployable as a microservice or library. - **AWS Pilar (Open Source)**: Privacy-preserving analytics framework with built-in PII detection, designed for streaming data analysis. **Monitoring and Observability**: - **Data Quality Platforms**: Tools like Great Expectations, Soda Core, and Monte Carlo can monitor for unexpected PII patterns in streams, alerting when sensitive data appears where it shouldn't. For detailed data quality implementation, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework). - **Audit Logging Systems**: Comprehensive audit trails showing who accessed PII data, when, and what operations were performed. See [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) for implementation patterns. ## Conclusion PII leakage prevention requires vigilance, technical controls, and organizational commitment. By understanding leakage vectors, implementing detection approaches, and enforcing prevention policies, organizations protect personal data while maintaining compliance with data protection regulations. The investment in PII protection reduces breach risk, supports regulatory compliance, and builds customer trust. Modern tools in 2025 — from AI-powered detection services to governance platforms like Conduktor — make PII protection more accessible and effective. Kafka 4.0+ with KRaft mode provides improved audit capabilities, while cloud AI services offer ML-based detection with minimal operational overhead. **Getting Started Checklist**: 1. **Inventory PII**: Identify what PII exists in your streaming systems and where it flows 2. **Start with high-risk data**: Implement pattern matching for credit cards, SSNs, and other structured PII 3. **Add schema-level controls**: Use schema registry to mark sensitive fields and enforce policies 4. **Deploy detection**: Choose between inline scanning (Kafka Streams/Flink) or centralized scanning (Conduktor Gateway, sidecar processors) 5. **Implement masking**: Add tokenization or encryption for sensitive fields before data reaches lower-security environments 6. **Monitor and alert**: Set up dashboards and alerts for PII detection events 7. **Test regularly**: Use synthetic PII to validate detection and prevention mechanisms 8. **Expand coverage**: Gradually add ML-based detection for contextual PII and advanced anonymization techniques ## Related Concepts - [PII Detection and Handling in Event Streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams) - Core techniques for detecting PII using pattern matching, ML classification, and schema-based approaches. - [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming) - Implementation patterns for masking, tokenization, and anonymization to prevent PII leakage. - [Policy Enforcement in Streaming](https://www.conduktor.io/glossary/policy-enforcement-in-streaming) - Automated policy enforcement to detect and block PII leakage at zone boundaries. ## Sources and References 1. **GDPR Official Text** - Complete text of the General Data Protection Regulation including technical and organizational measures for data protection. [https://gdpr-info.eu/](https://gdpr-info.eu/) 2. **NIST Privacy Framework** - Comprehensive framework for managing privacy risks including PII identification and protection strategies. [https://www.nist.gov/privacy-framework](https://www.nist.gov/privacy-framework) 3. **AWS Macie Documentation** - Guide to automated PII discovery and classification in data streams and storage using machine learning. [https://aws.amazon.com/macie/](https://aws.amazon.com/macie/) 4. **Google Cloud DLP API** - Documentation on detecting and protecting sensitive data including PII using pattern matching and ML classification. [https://cloud.google.com/dlp](https://cloud.google.com/dlp) 5. **OWASP Top 10 Privacy Risks** - Industry-standard reference for understanding and mitigating privacy risks in software applications. [https://owasp.org/www-project-top-10-privacy-risks/](https://owasp.org/www-project-top-10-privacy-risks/) Source: https://www.conduktor.io/glossary/pii-leakage-prevention --- # Policy Enforcement in Streaming: Automated Governance for Real-Time Data In traditional batch processing systems, data quality checks and governance controls are applied at specific points, before loading, after transformation, during validation phases. But streaming systems operate continuously, with events flowing through pipelines 24/7. This constant flow demands a different approach: **policy enforcement** that operates automatically, in real-time, as data moves through the system. ![Multi-layer policy enforcement in streaming architectures](https://www.conduktor.io/assets/images/glossary/policy-enforcement-in-streaming-0.webp) Policy enforcement in streaming means applying rules, validations, and transformations automatically as events flow from producers to consumers. These policies ensure that every message meets quality standards, complies with security requirements, and follows organizational governance rules — without manual intervention and without slowing down the data pipeline. **A simple example:** A policy might automatically reject any order event where the `order_amount` field is missing or negative, preventing downstream services from processing invalid data. Another policy might automatically mask credit card numbers in payment events before they reach analytics consumers, protecting sensitive data without requiring changes to producing applications. ## Understanding Streaming Policies A streaming policy is an automated rule that evaluates, transforms, or routes events as they flow through the system. Unlike application-level validation that happens in specific services, streaming policies operate at the infrastructure level, affecting all data that passes through enforcement points. - **Core policy categories include:** - **Validation policies** verify that events meet structural and semantic requirements. [Schema validation](https://www.conduktor.io/glossary/schema-registry-and-schema-management) ensures messages conform to expected formats. [Data quality rules](https://www.conduktor.io/glossary/building-a-data-quality-framework) check for null values, invalid ranges, or malformed data. Format enforcement verifies dates, emails, phone numbers, and other typed fields follow conventions. - **Transformation policies** modify events to meet standards or protect sensitive data. [Field masking](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming) obscures [personally identifiable information](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). Standardization rules convert values to canonical formats. Enrichment policies add context or metadata to events as they flow through the system. - **Routing policies** determine where events should go based on their content or metadata. Topic routing directs messages to different destinations based on classification. Filtering policies prevent certain events from reaching specific consumers. [Rate limiting](https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka) controls how many events flow to particular endpoints. - **Security policies** protect data and enforce [access controls](https://www.conduktor.io/glossary/access-control-for-streaming). Encryption policies ensure sensitive data is encrypted in transit or at rest. Authentication rules verify producer and consumer identities. Authorization policies control which services can access specific data streams. ## Policy Enforcement Points Policies can be enforced at different locations in the streaming architecture, each with distinct trade-offs. ### Producer-Side Enforcement Implementing policies in producers, the applications generating events, provides the earliest point of validation. Producers can validate schemas, check data quality, and apply masking before sending events to the streaming platform. This approach prevents invalid data from ever entering the system. Failed validation can trigger immediate retries or error handling in the producing application. However, it requires implementing and maintaining policy logic across every producing service, leading to duplication and inconsistency risks. ### Broker-Side Enforcement Applying policies at the broker level, in the streaming platform itself, centralizes enforcement. Kafka brokers, for example, can validate schemas using Schema Registry integration, enforce quotas to prevent resource exhaustion, and apply retention policies to manage storage. Broker-side enforcement provides a single point of control, ensuring all data flowing through the platform meets standards regardless of producer implementation. But broker-level validation adds processing overhead and can create bottlenecks if not carefully designed. ### Consumer-Side Enforcement Policies enforced in consumers, the applications reading events, provide the final validation layer. Consumers can apply business-specific validation, perform additional transformations, or filter events based on their specific needs. This approach allows different consumers to apply different policies to the same data stream. A analytics consumer might accept approximate data while a billing consumer requires exact values. However, consumer-side enforcement doesn't prevent invalid data from entering the system, and duplicating validation across consumers wastes resources. ### Multi-Layer Strategy Production systems typically combine enforcement points. Critical validations like schema compliance and security controls operate at the broker level, ensuring platform-wide consistency. Business-specific rules execute at consumer level, allowing flexibility for different use cases. Producers perform basic validation to catch obvious errors early. ## Implementation Approaches Several technical approaches enable policy enforcement in streaming systems. ### Interceptors Producer and consumer interceptors are plugins that execute automatically when messages are sent or received. An interceptor can inspect messages, modify them, record metrics, or reject invalid data. Interceptors operate within the client application but execute transparently, application code doesn't need explicit calls. A schema validation interceptor can verify every produced message against Schema Registry before allowing transmission. A masking interceptor can obscure sensitive fields in every consumed message before passing data to application logic. ### Single Message Transforms Kafka Connect uses Single Message Transforms (SMTs) to apply policies during data integration. SMTs are lightweight transformations that execute as data flows through connectors, allowing modification without writing custom code. SMTs can mask fields, rename attributes, route messages to different topics, filter events, or add metadata. A connector pulling customer data from a database can apply a masking SMT to obscure email addresses before writing to Kafka. Another SMT can add a timestamp indicating when the event was captured. **Example SMT configuration for masking sensitive fields:** ```json { "name": "customer-source-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "transforms": "maskEmail,maskPhone", "transforms.maskEmail.type": "org.apache.kafka.connect.transforms.MaskField$Value", "transforms.maskEmail.fields": "email", "transforms.maskEmail.replacement": "***@***.com", "transforms.maskPhone.type": "org.apache.kafka.connect.transforms.MaskField$Value", "transforms.maskPhone.fields": "phone_number", "transforms.maskPhone.replacement": "XXX-XXX-XXXX" } } ``` ### Stream Processing Stream processing frameworks like Kafka Streams, Apache Flink, or Apache Beam provide powerful policy enforcement capabilities through continuous processing topologies. A stream processing application can consume from one topic, apply complex validation logic, enrich events with reference data, filter based on sophisticated rules, and produce to multiple output topics based on routing policies. This approach handles policies requiring stateful processing, joins across streams, or aggregations over time windows. **Example policy implementation in Kafka Streams:** ```java StreamsBuilder builder = new StreamsBuilder(); KStream orders = builder.stream("orders"); // Policy: Validate order amounts and split into valid/invalid streams Map> branches = orders.split(Named.as("branch-")) .branch((key, order) -> order.getAmount() != null && order.getAmount() > 0, Branched.as("valid")) .branch((key, order) -> true, Branched.as("invalid")) .noDefaultBranch(); // Valid orders proceed to processing branches.get("branch-valid").to("orders-validated"); // Invalid orders go to dead letter queue for investigation branches.get("branch-invalid").to("orders-dlq"); ``` Stream processors can also publish policy violations to dedicated topics for monitoring and alerting, creating an audit trail of governance enforcement. ### Schema Registry Integration Schema Registry provides centralized schema management and compatibility checking. When configured with Kafka, it enforces schema policies automatically. Producers serialize data using Schema Registry, which validates that schemas are registered and compatible with previous versions using formats like Avro, Protobuf, or JSON Schema. Consumers deserialize using the same registry, ensuring they can parse messages. Compatibility modes (backward, forward, full) define what schema changes are permitted, preventing breaking changes from reaching production. ### Policy-as-Code with Open Policy Agent Modern streaming governance increasingly adopts **policy-as-code** approaches using dedicated policy engines like Open Policy Agent (OPA) or AWS Cedar. These engines separate policy logic from application code, enabling centralized management and consistent enforcement. **OPA uses the Rego language to express policies declaratively:** ```rego package kafka.authz # Policy: Only payment services can produce to payment topics default allow_produce = false allow_produce { input.principal.name == "payment-service" startswith(input.topic, "payments.") } # Policy: Sensitive topics require encryption violation[{"msg": msg}] { startswith(input.topic, "pii.") not input.encryption_enabled msg := sprintf("Topic %v requires encryption", [input.topic]) } ``` OPA integrates with Kafka through interceptors or proxy layers, evaluating policies on every request. Policy decisions can enforce authorization, validate message structure, or require specific configurations based on topic classification. ### Streaming Data Gateways **Conduktor Gateway**, a Kafka proxy, and similar streaming data gateways provide a modern approach to centralized policy enforcement. Rather than implementing policies in every producer and consumer, a gateway sits between clients and brokers, intercepting all traffic. Gateways enable: - **Centralized policy enforcement** without modifying applications - **Field-level encryption and masking** applied transparently - **Schema validation** before messages reach brokers - **Virtual topics** that apply transformations or filters - **Audit logging** of all data access patterns - **Multi-tenancy isolation** with quota enforcement per team This proxy-based architecture is particularly valuable in 2025 as organizations implement [data mesh patterns](https://www.conduktor.io/glossary/data-mesh-principles-and-implementation) where multiple teams produce and consume data independently but must comply with organization-wide governance policies. ## Schema Governance Policies Schema policies maintain structural integrity across the streaming platform. - **Compatibility rules** prevent breaking changes. Backward compatibility ensures new schemas can read old data, critical when consumers haven't upgraded yet. Forward compatibility ensures old schemas can read new data, important when producers deploy first. Full compatibility requires both, while transitive variants extend compatibility across all historical versions. See [schema evolution best practices](https://www.conduktor.io/glossary/schema-evolution-best-practices) for detailed guidance. - **Required fields** ensure critical data is always present. Marking fields as required in [Avro, Protobuf, or JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema) prevents producers from omitting essential information. - **Naming conventions** standardize field names, avoiding confusion. Policies can enforce camelCase or snake_case, prevent abbreviations, or require specific prefixes for certain field types. - **Documentation requirements** can mandate descriptions for schemas and fields, improving discoverability and understanding across teams. These documentation policies are essential components of [data contracts](https://www.conduktor.io/glossary/data-contracts-for-reliable-pipelines) between producers and consumers. ## Security and Privacy Policies Security policies protect sensitive data as it flows through streaming systems. - **Encryption policies** ensure data protection. Encryption in transit using TLS protects data moving between producers, brokers, and consumers. Encryption at rest protects data stored in broker logs. End-to-end encryption can protect data even from platform administrators by encrypting payloads before producing. - **Field-level masking** obscures sensitive data based on rules. A policy might mask credit card numbers for all consumers except payment processing services, or hash email addresses for analytics while providing plaintext to customer communication systems. - **Field-level encryption** protects sensitive data end-to-end, even from platform administrators. Modern approaches use envelope encryption where data encryption keys (DEKs) encrypt individual fields, and key encryption keys (KEKs) stored in key management services (like AWS KMS or HashiCorp Vault) protect the DEKs. This enables selective decryption, only consumers with appropriate KEK access can decrypt specific fields, while the streaming platform handles encrypted data opaquely. - **Access control policies** define who can produce to or consume from topics. Combined with authentication and authorization mechanisms like SASL (Simple Authentication and Security Layer) and ACLs (Access Control Lists), these policies enforce principle of least privilege, ensuring services access only the data they need. - **Audit logging** records all policy decisions, creating compliance trails. Logs capture what policies were evaluated, whether events passed or failed, and what actions were taken. See [audit logging for streaming platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) for implementation patterns. ## Operational Policies Operational policies maintain platform health and resource efficiency. - **Retention policies** control how long data remains available. Time-based retention deletes events after a specified duration. Size-based retention limits topic storage. Compaction-based retention keeps only the latest value for each key, supporting changelog semantics (where topics represent the current state of entities, similar to database change logs). - **Quota policies** prevent resource exhaustion. Producer quotas limit how much data applications can write per second. Consumer quotas limit read bandwidth. Request quotas prevent any single client from overwhelming broker capacity. - **Replication policies** ensure durability and availability. Minimum in-sync replicas requirements prevent data loss by ensuring sufficient replicas acknowledge writes before considering them successful. ## Policy-as-Code and Governance Integration Modern policy enforcement treats policies as code, applying software engineering practices to governance. - **Version control** stores policy definitions in Git alongside application code. Changes go through code review, creating accountability and knowledge sharing. History tracking shows how policies evolved over time. - **CI/CD integration** validates and deploys policies automatically. Automated tests verify policies work as expected before deployment. Staging environments allow testing policy changes against realistic data before production rollout. - **Declarative policy languages** like Rego (used by Open Policy Agent) or proprietary DSLs express policies as readable, testable specifications rather than scattered imperative code. - **Policy testing frameworks** enable comprehensive validation before production deployment. Tests verify that policies correctly allow valid data, reject invalid data, and handle edge cases: ```bash # Testing OPA policies with sample inputs opa test policy/ tests/ -v # Example test case test_payment_topic_access { allow_produce with input as { "principal": {"name": "payment-service"}, "topic": "payments.transactions" } not allow_produce with input as { "principal": {"name": "analytics-service"}, "topic": "payments.transactions" } } ``` **Policy observability** tracks enforcement in production through metrics, logs, and alerts: - **Enforcement metrics** measure policy evaluation latency, success/failure rates, and throughput impact - **Violation alerts** trigger when policies block invalid data, indicating potential producer issues - **Compliance dashboards** visualize policy coverage across topics and consumer groups - **Audit trails** provide forensic analysis of who accessed what data and which policies were applied - **Governance platforms** like **Conduktor** provide centralized policy management across Kafka environments, allowing teams to define data quality rules, security policies, and operational controls through unified interfaces that automatically enforce them across the streaming infrastructure. These platforms combine policy definition, deployment, testing, and monitoring in a single workflow, ensuring consistency and simplifying auditing for regulatory requirements. Integration with [data governance frameworks](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) ensures alignment with organizational policies and responsibilities. - **Compliance reporting** generates documentation showing policy enforcement status. Automated reports demonstrate that data handling meets regulatory requirements like GDPR (General Data Protection Regulation), HIPAA (Health Insurance Portability and Accountability Act), or SOX (Sarbanes-Oxley). Violation tracking identifies patterns requiring attention. ## Conduktor Traffic Control Policies Modern streaming platforms require policy enforcement mechanisms that operate at scale without requiring changes to applications. Conduktor Traffic Control Policies provide a centralized framework for enforcing rate limits (producer/consumer throughput, connection limits), configuration policies (replication factors, retention periods, schema requirements), and access control policies (read-only access, client ID requirements) through automated rules that intercept traffic at the gateway layer. The key advantage is centralized enforcement without application modification, policies apply across all producers and consumers automatically. Configuration policies shift enforcement from documentation to automated validation, eliminating configuration drift. Traffic Control Policies integrate with policy-as-code workflows through version-controlled configuration deployed via CI/CD. For implementation guidance, see [Traffic Control Policies](https://docs.conduktor.io/guide/conduktor-concepts/traffic-control-policies). ## Balancing Enforcement and Performance Policy enforcement introduces processing overhead. Each validation, transformation, or check consumes CPU cycles and adds latency. Effective policy implementation balances governance needs against performance requirements. - **Selective enforcement** applies expensive policies only where necessary. Critical financial data might undergo extensive validation while operational metrics receive lighter checks. High-volume topics might use simpler policies than low-volume sensitive topics. - **Async validation** separates policy enforcement from the critical path. Events flow through quickly, while a parallel stream processing application validates compliance and raises alerts for violations. This trades immediate enforcement for throughput, appropriate when detecting violations quickly matters more than preventing them entirely. - **Policy optimization** ensures rules execute efficiently. Compiled policies run faster than interpreted ones. Early exit strategies skip remaining checks once a violation is detected. Caching reduces redundant lookups or computations. ## Conclusion Policy enforcement transforms streaming platforms from simple message buses into governed data infrastructures. Automated policies ensure data quality, enforce security requirements, and maintain operational health — all in real-time, without manual intervention. Effective policy enforcement combines technical implementation with organizational process. Policies must be designed thoughtfully, balancing governance needs against system performance. They should be expressed declaratively, versioned like code, and deployed through automated pipelines. Enforcement should happen at appropriate points in the architecture: broker-side for universal rules, producer-side for early validation, consumer-side for specific needs. As streaming systems grow in scale and importance, robust policy enforcement becomes essential. It is the foundation for trustworthy real-time data, enabling organizations to move fast while maintaining control, democratize data access while protecting privacy, and meet compliance obligations without sacrificing agility. ## Related Concepts - [Data Governance Framework: Roles and Responsibilities](https://www.conduktor.io/glossary/data-governance-framework-roles-and-responsibilities) - Organizational structure for policy management - [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) - Recording policy enforcement decisions - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/glossary/data-quality-dimensions-accuracy-completeness-and-consistency) - Quality policies for streaming data ## Sources and References 1. **Apache Kafka Documentation - Security**. Apache Software Foundation. https://kafka.apache.org/documentation/#security - Official documentation covering authentication, authorization, encryption, and security policies in Kafka. 2. **Confluent Schema Registry Documentation**. Confluent, Inc. https://docs.confluent.io/platform/current/schema-registry/index.html - Comprehensive guide to schema governance, compatibility policies, and validation in streaming systems. 3. **Open Policy Agent (OPA) Documentation**. Cloud Native Computing Foundation. https://www.openpolicyagent.org/docs/latest/ - Framework for policy-as-code implementation applicable to streaming data governance. 4. **Stopford, Ben. "Designing Event-Driven Systems."** O'Reilly Media, 2018. Chapters on data governance, schema evolution, and policy enforcement patterns in event streaming architectures. 5. **Kafka Governance Best Practices**. Various platform documentation covering centralized policy management, data quality enforcement, and compliance automation for Kafka environments. Source: https://www.conduktor.io/glossary/policy-enforcement-in-streaming --- # Quotas and Rate Limiting in Kafka Apache Kafka clusters serve multiple applications simultaneously, each with varying workloads and resource demands. Without proper controls, a single misbehaving or resource-intensive client can degrade performance for all users. Quotas and rate limiting provide the mechanisms to prevent this scenario, ensuring fair resource allocation and stable cluster operation. ![Kafka quota enforcement and throttling mechanism](https://www.conduktor.io/assets/images/glossary/quotas-and-rate-limiting-in-kafka-0.webp) Understanding how Kafka implements quotas is essential for anyone operating production clusters, especially in multi-tenant environments where predictable performance is critical. For foundational understanding of Kafka architecture, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). ## What Are Quotas in Kafka? Quotas in Kafka are resource controls that limit the amount of bandwidth or request capacity a client can consume. Rather than allowing clients to use unlimited broker resources, quotas enforce upper bounds on data transfer rates and request processing. When a client exceeds its quota, Kafka doesn't reject the request outright. Instead, it throttles the client by delaying responses, effectively slowing down the client to bring it within its allocated limits. This approach maintains data integrity while preventing resource monopolization. Quotas serve several critical purposes in Kafka deployments: - **Resource protection**: Prevent individual clients from overwhelming broker CPU, memory, or network capacity - **Fair sharing**: Ensure all clients receive predictable service levels in shared clusters - **Cost control**: Limit resource consumption to manage infrastructure costs - **Stability**: Maintain consistent latency and throughput even under heavy load ## Types of Kafka Quotas Kafka supports three primary quota types, each targeting different aspects of client behavior. ### Producer Quotas Producer quotas limit the rate at which clients can publish data to Kafka. These are specified as bytes per second and control the total throughput a producer can achieve across all topics. For example, setting a producer quota of 10 MB/sec means a client cannot exceed this rate when writing to the cluster. If the producer attempts to send data faster, Kafka will throttle it by delaying produce responses. ### Consumer Quotas Consumer quotas limit the rate at which clients can fetch data from Kafka. Like producer quotas, these are expressed as bytes per second and control the total bandwidth a consumer can use when reading from topics. A consumer quota of 15 MB/sec ensures that a single consumer application cannot saturate broker network interfaces or monopolize disk I/O when reading historical data. ### Request Quotas Request quotas limit the percentage of request handler thread time a client can consume on a broker. Rather than measuring bytes, this quota type focuses on broker CPU and processing capacity. For instance, a request quota of 50% means a client cannot use more than half of the broker's request processing capacity. This prevents computationally expensive requests from blocking other clients. Examples of computationally expensive requests include: - **Large metadata requests**: Asking for information about thousands of partitions simultaneously - **Complex offset lookups**: Searching for specific timestamps across many partitions - **High-frequency administrative operations**: Repeatedly querying cluster configuration or partition leadership By limiting request processing time rather than just bytes transferred, request quotas protect brokers from clients that make many small but CPU-intensive requests. ## How Rate Limiting Works in Kafka Kafka's rate limiting mechanism is designed to be non-destructive. When a client exceeds its quota, Kafka calculates the delay needed to bring the client back within limits and holds the response for that duration. The broker tracks each client's resource usage in a **sliding window** - a moving time window that continuously measures consumption over recent intervals, typically 1 second. Unlike a fixed window that resets at specific times (like every minute), a sliding window provides smoother enforcement by considering usage over the most recent period at any moment. When usage exceeds the quota, the broker computes the throttle time based on how much the client has overshot its limit. For example, if a producer has a 10 MB/sec quota but sends 15 MB in one second, the broker delays the produce response by approximately 500 milliseconds. This throttling causes the producer to slow down naturally without losing data. The client receives the throttled response along with a metric indicating how long it was delayed. Well-designed clients can use this information to self-regulate their sending or fetching rates proactively. This approach has several advantages over hard rejection: - Data is never lost due to quota violations - Clients automatically adjust to stay within limits - The cluster remains stable even when clients misbehave - Operators have time to diagnose and address quota violations ## Configuring Quotas in Kafka Kafka provides flexible quota configuration options, supporting both static broker configuration and dynamic runtime updates. ### Static Configuration (Legacy) **Note**: Static quota configuration in `server.properties` has been deprecated since Kafka 2.0. While these properties still work in some deployments, dynamic configuration is the recommended approach for all modern Kafka clusters. For reference, legacy static quotas looked like: ```properties # Legacy - not recommended for Kafka 2.0+ quota.producer.default=10485760 quota.consumer.default=15728640 ``` Static configuration required broker restarts and lacked the flexibility needed for production environments. Modern Kafka deployments should exclusively use dynamic configuration. ### Dynamic Configuration (Recommended) Operators set quotas dynamically using the `kafka-configs` command-line tool. This is the standard approach in modern Kafka deployments (2.0+) and allows quota changes without broker restarts, making it ideal for production environments. To set a producer quota for a specific client: ```bash kafka-configs --bootstrap-server localhost:9092 \ --alter \ --add-config 'producer_byte_rate=10485760' \ --entity-type clients \ --entity-name my-producer-app ``` To set a consumer quota for a specific user: ```bash kafka-configs --bootstrap-server localhost:9092 \ --alter \ --add-config 'consumer_byte_rate=15728640' \ --entity-type users \ --entity-name data-pipeline-user ``` Quotas can be scoped to: - **Client ID**: Applies to all connections using a specific client identifier - **User**: Applies to all connections authenticated as a specific user (requires authentication) - **User and Client ID**: Most specific, applies to a user-client combination This hierarchical approach allows fine-grained control. For instance, you might set generous quotas for trusted internal applications while restricting third-party integrations. ## Modern Quota Features in Kafka 4.0+ Kafka 4.0 and later versions introduce several quota enhancements that improve resource management and operational control, particularly for KRaft-mode clusters. ### Connection Quotas Connection quotas limit the maximum number of concurrent connections a client can establish to the cluster. This prevents connection exhaustion attacks and helps manage broker memory consumption. To set a connection quota limiting a client to 100 concurrent connections: ```bash kafka-configs --bootstrap-server localhost:9092 \ --alter \ --add-config 'connection_creation_rate=100' \ --entity-type clients \ --entity-name high-volume-app ``` Connection quotas are particularly valuable in multi-tenant environments where connection storms from misconfigured clients can impact cluster availability. ### IP-Based Quotas IP-based quotas provide an additional security layer by limiting resource consumption based on source IP addresses, even before client authentication. This helps defend against DDoS attacks and resource exhaustion from unknown actors. Setting IP-based rate limits: ```bash kafka-configs --bootstrap-server localhost:9092 \ --alter \ --add-config 'producer_byte_rate=5242880,consumer_byte_rate=5242880' \ --entity-type ips \ --entity-name 192.168.1.100 ``` IP-based quotas complement user and client-id quotas, creating defense in depth for resource protection. ### KRaft-Native Quota Management With Kafka 4.0's KRaft mode becoming the standard (ZooKeeper support has been removed), quota configuration is now stored directly in the metadata log. This provides: - **Faster propagation**: Quota changes propagate immediately across all brokers through the metadata log - **Better consistency**: No risk of ZooKeeper-Kafka state divergence - **Simplified operations**: Single source of truth for all cluster configuration For clusters running on KRaft, quota changes take effect within milliseconds rather than seconds, improving operational responsiveness. For detailed information on KRaft architecture and migration, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka). ### Client Metrics and Observability (KIP-714) Modern Kafka clients (3.7+) support client-side metrics plugins that push telemetry data directly to brokers. This enables more sophisticated quota monitoring and enforcement: - **Predictive throttling**: Clients can self-regulate before hitting hard limits - **Better visibility**: Operators see client-side metrics alongside broker-side enforcement - **Proactive alerting**: Identify quota violations before they impact users Client metrics plugins integrate with tools like Conduktor to provide comprehensive quota dashboards showing both client intent and broker enforcement. ## Quotas in Multi-Tenant Kafka Environments Multi-tenant Kafka deployments particularly benefit from quotas. When multiple teams or customers share a cluster, quotas ensure fair resource distribution and prevent noisy neighbor problems (situations where one tenant's resource usage negatively impacts others). Consider a SaaS platform running Kafka for multiple customer applications. One customer might run a large batch job that reads historical data at maximum speed. Without quotas, this batch job could saturate broker network interfaces, causing latency spikes for other customers running real-time streaming applications. For comprehensive coverage of multi-tenant patterns and best practices, see [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments). By implementing consumer quotas, the platform operator can limit the batch job to a reasonable throughput, perhaps 20 MB/sec, while ensuring real-time applications maintain low-latency access to their data streams. Quotas also enable tiered service models. A platform might offer: - **Free tier**: 1 MB/sec producer and consumer quotas - **Standard tier**: 10 MB/sec quotas - **Enterprise tier**: 100 MB/sec or custom quotas This approach aligns resource consumption with pricing while maintaining cluster stability. ### Virtual Clusters for Quota Isolation Managing quotas in multi-tenant environments requires careful coordination of client IDs and quota hierarchies. Conduktor Virtual Clusters simplify this by scoping quota enforcement to individual virtual clusters, each operates as an independent quota domain with its own rate limits, completely isolated from other virtual clusters. A team's quota violations in one virtual cluster cannot impact applications in other virtual clusters, providing strong isolation against noisy neighbor scenarios. This architecture enables differentiated resource allocation where platform teams assign throughput ceilings to each virtual cluster (e.g., 100 MB/sec for data engineering, 50 MB/sec for analytics). Teams manage their own internal quotas without cross-tenant coordination. Virtual Clusters also simplify tiered service models, customers upgrade by migrating to higher-capacity virtual clusters rather than modifying quota rules. For implementation details, see the [Virtual Clusters documentation](https://docs.conduktor.io/guide/conduktor-concepts/virtual-clusters). ## Monitoring and Managing Quotas Effective quota management requires visibility into quota usage and violations. Modern Kafka deployments use a combination of broker-side metrics, client-side telemetry, and dedicated monitoring tools. ### Broker-Side Metrics Kafka exposes quota enforcement metrics through JMX: - `kafka.server:type=Request,name=ThrottleTime`: Duration clients spend throttled (milliseconds) - `kafka.server:type=ClientQuotas,name=QuotaViolations`: Count of quota violations by client - `kafka.network:type=RequestMetrics,name=RequestQueueTimeMs`: Request queuing delays indicating saturation For KRaft-mode clusters (Kafka 4.0+), additional metrics track quota configuration propagation and metadata log performance. ### Client-Side Metrics (Kafka 3.7+) Modern Kafka clients support client metrics plugins (KIP-714), providing visibility into client behavior before broker enforcement: - **Client-side throttle awareness**: Applications can detect impending quota violations - **Intent vs. enforcement**: Compare what clients want to do versus what brokers allow - **Proactive tuning**: Adjust client behavior based on observed patterns Client metrics integrate with Conduktor and other observability platforms, creating a complete picture of quota health. ### Quota Management Tools Tools like Conduktor provide visual interfaces for quota management, making it easier to configure quotas, monitor violations, and alert on quota-related issues. Rather than managing quotas through command-line tools and interpreting raw JMX metrics, Conduktor offers: - **Quota dashboards**: Real-time visualization of quota usage across all clients - **Violation alerts**: Automated notifications when clients consistently exceed limits - **Historical analysis**: Track quota patterns over time to inform capacity planning - **Multi-cluster views**: Manage quotas consistently across development, staging, and production environments For hands-on guidance on implementing quotas and rate limiting with Conduktor, see the [Traffic Control Policies documentation](https://docs.conduktor.io/guide/conduktor-concepts/traffic-control-policies). Establishing quota violation alerts ensures operators can respond to issues proactively. If a critical application consistently hits quota limits, this might indicate the need for infrastructure scaling or quota increases rather than simply throttling the client. For strategies on tracking consumer performance and identifying quota-related lag issues, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). ## Quotas and Data Streaming Pipelines Quotas play a vital role in maintaining stable data streaming pipelines. Stream processing frameworks like Kafka Streams and Apache Flink rely on predictable Kafka performance to maintain their processing guarantees. When quotas are properly configured, streaming applications experience consistent latency and throughput. This stability enables accurate watermarking (temporal markers for event-time processing), reduces backpressure in processing pipelines, and helps maintain exactly-once semantics. Conversely, without quotas, a sudden spike in producer traffic can cause broker saturation, leading to increased latency for all consumers. This latency spike propagates through stream processing applications, potentially causing processing delays, state store lag, and missed SLAs. For detailed coverage of managing flow control in streaming systems, see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems). For organizations building real-time analytics or event-driven architectures, quotas are a fundamental requirement for production reliability. ## Troubleshooting Quota Violations When clients hit quota limits, understanding the root cause and implementing appropriate fixes ensures optimal performance without compromising cluster stability. ### Identifying Quota Issues Signs that a client is experiencing quota throttling: - **Increased latency**: Producer send or consumer fetch operations take longer than expected - **Throttle metrics**: Client reports non-zero throttle time in metrics - **Throughput plateaus**: Application cannot increase throughput despite available capacity - **Broker logs**: Entries indicating quota enforcement for specific clients ### Common Causes and Solutions **1. Legitimate Load Growth** If a trusted application consistently hits quota limits due to genuine business growth: - **Increase quotas**: Adjust limits to match new requirements using `kafka-configs` - **Scale infrastructure**: Add brokers or increase broker capacity - **Review tiering**: Move application to higher service tier if using quota-based pricing **2. Inefficient Client Behavior** Poorly designed clients may trigger quotas unnecessarily: - **Batching issues**: Producers sending tiny batches frequently instead of larger batches less often - **Polling patterns**: Consumers fetching data too aggressively with very short poll intervals - **Metadata requests**: Applications repeatedly fetching cluster metadata Solutions include optimizing batch sizes, adjusting fetch configurations, and caching metadata appropriately. **3. Noisy Neighbor Scenarios** In multi-tenant environments, one application's spike impacts others: - **Implement connection quotas**: Prevent connection storms - **Use IP-based quotas**: Add defense layer before authentication - **Review quota hierarchy**: Ensure user+client combinations have appropriate limits **4. Testing and Development Traffic** Test environments accidentally pointing to production: - **Separate clusters**: Use dedicated clusters for testing when possible - **Lower default quotas**: Set conservative defaults for unknown clients - **Network isolation**: Enforce network boundaries between environments ### Validating Quota Configuration Test quota behavior proactively using tools like Conduktor Gateway (a Kafka proxy) for chaos engineering scenarios: - **Simulate quota violations**: Generate controlled load exceeding limits - **Measure throttle behavior**: Verify brokers throttle correctly without dropping data - **Test client resilience**: Ensure applications handle throttling gracefully For comprehensive testing strategies, see [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems). ### Best Practices for Quota Management 1. **Start conservative**: Set initial quotas lower than expected peak load, then increase based on monitoring 2. **Monitor continuously**: Track quota metrics and violations across all clients 3. **Document limits**: Maintain clear documentation of quota policies and tiers 4. **Automate alerts**: Configure proactive alerts before clients hit hard limits 5. **Review regularly**: Periodically audit quota configurations as workloads evolve ## Summary Quotas and rate limiting in Kafka provide essential controls for managing cluster resources and ensuring stable, predictable performance. By limiting producer throughput, consumer fetch rates, request processing capacity, and connections, quotas prevent individual clients from monopolizing broker resources. Kafka's throttling-based approach maintains data integrity while enforcing limits, and the flexible configuration system supports both default cluster-wide quotas and fine-grained client-specific controls. This makes quotas particularly valuable in multi-tenant environments where fair resource sharing is critical. Modern Kafka deployments (4.0+) benefit from enhanced quota features including KRaft-native configuration management, connection quotas, IP-based rate limiting, and client metrics plugins (KIP-714) for comprehensive observability. These improvements provide faster configuration propagation, better defense in depth, and proactive quota violation detection. Effective quota management requires ongoing monitoring of quota metrics and violations, with tools available to simplify configuration and alerting. For data streaming pipelines, properly configured quotas are fundamental to achieving consistent performance and meeting processing SLAs. Whether operating a shared Kafka cluster or managing dedicated infrastructure, understanding and implementing quotas should be a standard part of any Kafka deployment strategy. For additional performance optimization techniques, see [Kafka Performance Tuning Guide](https://www.conduktor.io/glossary/kafka-performance-tuning-guide). ## Related Concepts - [Multi-Tenancy in Kafka Environments](https://www.conduktor.io/glossary/multi-tenancy-in-kafka-environments) - Quota strategies for isolating workloads and ensuring fair resource allocation across teams and applications. - [Kafka Security Best Practices](https://www.conduktor.io/glossary/kafka-security-best-practices) - Combine quotas with authentication and authorization for comprehensive resource protection. - [Kafka Capacity Planning](https://www.conduktor.io/glossary/kafka-capacity-planning) - Use quota configurations to inform capacity planning and prevent cluster saturation. ## Sources and References 1. Apache Kafka Documentation - Quotas: [https://kafka.apache.org/documentation/#design_quotas](https://kafka.apache.org/documentation/#design_quotas) 2. KIP-13: Quotas: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas](https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas) 3. Confluent Documentation - Kafka Quotas: [https://docs.confluent.io/platform/current/kafka/design.html#quotas](https://docs.confluent.io/platform/current/kafka/design.html#quotas) 4. Narkhede, Neha, et al. "Kafka: The Definitive Guide." O'Reilly Media, 2017. 5. KIP-219: Improve quota communication: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-219+-+Improve+quota+communication](https://cwiki.apache.org/confluence/display/KAFKA/KIP-219+-+Improve+quota+communication) Source: https://www.conduktor.io/glossary/quotas-and-rate-limiting-in-kafka --- # Real-Time Analytics with Streaming Data Real-time analytics has become essential for organizations that need to make decisions based on data as it arrives, rather than waiting for batch processing cycles. From detecting fraudulent transactions within milliseconds to monitoring IoT sensor networks, analyzing streaming data in real-time enables use cases that were previously impossible. ![Real-time analytics streaming architecture](https://www.conduktor.io/assets/images/glossary/real-time-analytics-with-streaming-data-0.webp) Unlike traditional batch analytics that processes historical data at scheduled intervals, real-time analytics operates on continuous data streams, producing insights with latencies measured in seconds or even milliseconds. This shift requires fundamentally different technologies, architectures, and thinking patterns. ## Understanding Streaming Data Streaming data is characterized by its continuous, unbounded nature. Events arrive one after another in a never-ending flow, whether they're user clicks on a website, sensor readings from industrial equipment, or financial transactions. The key distinction from batch data is temporal. Batch systems ask "what happened?" while streaming systems ask "what is happening right now?" This temporal dimension introduces challenges around ordering, completeness, and timeliness that don't exist in batch processing. Streaming data typically exhibits three important characteristics: - **Volume and velocity**: Data arrives continuously at rates that can range from dozens to millions of events per second. The system must process events as fast as they arrive to maintain real-time responsiveness. - **Event time vs processing time**: Events carry timestamps indicating when they occurred (event time), which may differ from when the system processes them (processing time). Network delays, system failures, or out-of-order delivery can cause significant differences between these times. - **Unbounded datasets**: Unlike batch jobs that process a finite dataset with a clear beginning and end, streaming datasets have no natural conclusion. Analytics must be designed to operate continuously without accumulating unbounded state. ## Stream Processing Technologies Several technologies have emerged to handle the complexities of real-time analytics on streaming data. - **Apache Kafka** is the foundation for many streaming architectures. It provides a distributed, fault-tolerant platform for ingesting, storing, and distributing streaming data. Kafka's log-based architecture ensures durability and allows consumers to replay historical data when needed. Topics partition data across multiple brokers, enabling horizontal scaling to handle massive throughput. For detailed coverage of Kafka fundamentals, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). Modern Kafka deployments (4.0+) use [KRaft mode](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) instead of ZooKeeper, simplifying operations and improving scalability. - **Apache Flink** provides true stream processing with low latency and exactly-once semantics. Flink excels at complex stateful computations, event-time processing, and sophisticated windowing operations. For an in-depth exploration, see [What is Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). - **Apache Spark** offers Structured Streaming, which provides both micro-batch and continuous processing modes. While historically known for micro-batching, Spark 2.3+ introduced continuous processing for sub-millisecond latencies, making it viable for true streaming use cases. - **Kafka Streams** provides a lightweight library for building stream processing applications that run within your existing JVM applications, eliminating the need for separate cluster infrastructure. - **ksqlDB** brings SQL semantics to stream processing, allowing developers to query Kafka topics using familiar SQL syntax. It's particularly useful for teams without deep stream processing expertise who need to perform transformations, aggregations, and joins using declarative SQL rather than procedural code. - **Modern analytical databases** like Apache Pinot, RisingWave, and Materialize are purpose-built for real-time analytics. Apache Pinot provides OLAP capabilities on streaming data with sub-second query latencies. RisingWave offers PostgreSQL-compatible streaming SQL with materialized views that update in real-time. Materialize maintains incrementally updated views over streams, enabling SQL queries on constantly changing data. These technologies work together in typical architectures: Kafka handles ingestion and distribution, while processing engines like Flink perform the actual analytics and transformations. ## Common Real-Time Analytics Patterns Several patterns appear repeatedly in real-time analytics implementations. ### Windowing Since streaming data is unbounded, analytics must operate on finite subsets defined by time windows. For comprehensive coverage of windowing patterns and implementation details, see [Windowing in Apache Flink](https://www.conduktor.io/glossary/windowing-in-apache-flink-tumbling-sliding-and-session-windows). Three common window types are: - **Tumbling windows**: Fixed-size, non-overlapping windows (e.g., "count events every 5 minutes") - **Sliding windows**: Overlapping windows that advance by a smaller interval than their size (e.g., "calculate the average over the last 10 minutes, updated every minute") - **Session windows**: Dynamic windows based on activity periods, useful for user behavior analysis. For detailed session window patterns, see [Session Windows in Stream Processing](https://www.conduktor.io/glossary/session-windows-in-stream-processing). Here's a practical example using Flink SQL to calculate average transaction amounts in 5-minute tumbling windows: ```sql SELECT user_id, TUMBLE_START(event_time, INTERVAL '5' MINUTE) as window_start, AVG(transaction_amount) as avg_amount, COUNT(*) as transaction_count FROM transactions GROUP BY user_id, TUMBLE(event_time, INTERVAL '5' MINUTE) HAVING AVG(transaction_amount) > 1000; ``` For fraud detection, a real-time system might use ksqlDB to track transaction patterns: ```sql CREATE STREAM transaction_alerts AS SELECT user_id, COUNT(*) as tx_count, SUM(amount) as total_amount FROM transactions WINDOW TUMBLING (SIZE 1 MINUTE) GROUP BY user_id HAVING COUNT(*) > 10 OR SUM(amount) > 10000 EMIT CHANGES; ``` For detailed fraud detection patterns, see [Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming). ### Stateful Processing Many analytics require maintaining state across events. Counting unique users, detecting patterns, or correlating related events all need the system to remember information from previous events. For deep dive into state management, see [State Stores in Kafka Streams](https://www.conduktor.io/glossary/state-stores-in-kafka-streams). Stream processors manage state in memory for fast access while checkpointing to persistent storage for fault tolerance. Modern stream processors like Flink (1.13+) and Kafka Streams (3.0+) have significantly improved exactly-once processing performance, making stateful operations reliable even at high throughput. Here's a Kafka Streams example tracking unique user visits per page: ```java StreamsBuilder builder = new StreamsBuilder(); KTable uniqueVisitors = builder .stream("page-views") .groupBy((key, view) -> view.getPageId()) .aggregate( () -> new HashSet(), // Initialize empty set (pageId, view, visitors) -> { visitors.add(view.getUserId()); return visitors; }, Materialized., KeyValueStore>as( "unique-visitors-store") ) .mapValues(visitors -> (long) visitors.size()); ``` This stateful computation maintains a set of unique user IDs per page, automatically handling state recovery if the application restarts. ### Stream Joins Joining multiple streams or enriching streaming data with reference data is common but complex. For instance, joining a stream of purchases with a stream of inventory updates requires handling time synchronization and dealing with late-arriving events. For comprehensive join patterns and implementation strategies, see [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns). Here's a Flink example joining order streams with customer data: ```java DataStream orders = env.addSource(new FlinkKafkaConsumer<>("orders", ...)); DataStream customers = env.addSource(new FlinkKafkaConsumer<>("customers", ...)); DataStream enrichedOrders = orders .keyBy(Order::getCustomerId) .intervalJoin(customers.keyBy(Customer::getId)) .between(Time.minutes(-5), Time.minutes(5)) .process(new ProcessJoinFunction() { @Override public void processElement(Order order, Customer customer, Context ctx, Collector out) { out.collect(new EnrichedOrder(order, customer)); } }); ``` This interval join ensures orders are matched with customer records that arrive within a 10-minute time window, handling slight timing differences between streams. ## Challenges and Best Practices Real-time analytics introduces operational challenges that require careful consideration. ### Handling Late and Out-of-Order Data Network delays and system failures mean events don't always arrive in order. A transaction that occurred at 10:00:01 might arrive after one from 10:00:05. Stream processors use **watermarks**, special markers that track progress through event time, to determine when to close windows and emit results. Watermarks define how long to wait for late events before considering a window complete. For detailed coverage of watermark strategies and trigger mechanisms, see [Watermarks and Triggers in Stream Processing](https://www.conduktor.io/glossary/watermarks-and-triggers-in-stream-processing). Modern stream processors allow configuring allowed lateness and side outputs for handling extremely late data without losing it. ### Exactly-Once Semantics For many use cases like financial analytics, processing each event exactly once is critical. Neither missing events (data loss) nor processing duplicates (incorrect counts) is acceptable. Modern stream processors like Flink achieve exactly-once semantics through distributed snapshots and transactional writes, but this comes with performance overhead. ### Scalability and Performance As data volumes grow, streaming systems must scale horizontally. This requires partitioning data across multiple processing instances while maintaining ordering guarantees where needed. Proper partitioning strategies based on keys (like user ID or device ID) ensure related events are processed by the same instance. ### Monitoring and Debugging Unlike batch jobs that complete and report final status, streaming jobs run indefinitely. Detecting issues requires continuous monitoring of key metrics: - **Consumer lag**: The difference between the latest message offset and the consumer's current position. Growing lag indicates the consumer can't keep up with incoming data. For detailed lag monitoring strategies, see [Consumer Lag Monitoring](https://www.conduktor.io/glossary/consumer-lag-monitoring). - **Processing latency**: Time between event occurrence and processing completion. Increasing latency signals potential bottlenecks. - **Throughput**: Events processed per second. Declining throughput may indicate resource constraints or inefficient operations. - **Backpressure**: When downstream operators can't keep pace with upstream data rates. For handling strategies, see [Backpressure Handling in Streaming Systems](https://www.conduktor.io/glossary/backpressure-handling-in-streaming-systems). - **Checkpoint duration**: Time required to snapshot application state. Long checkpoint times can impact overall system performance. - **State size**: Memory consumed by stateful operations. Unbounded state growth leads to out-of-memory errors. Platforms like Conduktor provide comprehensive visibility into streaming pipelines, helping teams monitor these metrics, inspect message contents, validate data quality, and debug issues before they impact downstream analytics. Teams can centrally manage and monitor [Kafka topics](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) to ensure healthy streaming operations. For broader data quality concerns, see [What is Data Observability](https://www.conduktor.io/glossary/what-is-data-observability-the-five-pillars). ## Architectural Patterns for Real-Time Analytics Modern real-time analytics architectures typically follow one of two patterns: ### Lambda Architecture Lambda architecture maintains separate batch and streaming pipelines that run in parallel. The batch layer processes complete historical datasets to produce accurate views, while the speed layer processes recent data with lower latency. Results are merged at query time. While this provides both accuracy and real-time responsiveness, maintaining two separate codebases increases operational complexity. ### Kappa Architecture Kappa architecture simplifies operations by using a single streaming pipeline for both real-time and historical data. All data flows through the streaming system, which processes it incrementally. By storing the complete event log (like Kafka's infinite retention), the system can reprocess historical data by simply replaying the stream. This reduces complexity but requires the streaming system to handle both real-time and batch-scale processing efficiently. ### Lakehouse Integration Modern real-time analytics increasingly integrates with lakehouse architectures, writing streaming data to transactional table formats like Apache Iceberg or Delta Lake. This enables: - **Unified batch and streaming**: Query the same tables using both real-time stream processing and batch SQL engines - **Time travel**: Replay historical data or query past states of the data - **ACID transactions**: Ensure consistency between streaming writes and batch operations - **Cost-effective storage**: Hot data in streaming processors, cold data in object storage For detailed patterns, see [Streaming to Lakehouse Tables](https://www.conduktor.io/glossary/streaming-to-lakehouse-tables) and [Apache Iceberg](https://www.conduktor.io/glossary/apache-iceberg). ## Real-World Applications Real-time analytics powers critical systems across industries. - **Fraud detection**: Financial institutions analyze transaction streams in real-time, comparing patterns against historical behavior and known fraud signatures. A credit card transaction that occurs in a different country minutes after a previous transaction can be flagged instantly, preventing fraudulent charges. For implementation details, see [Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming). - **IoT monitoring**: Manufacturing facilities monitor thousands of sensors in real-time to detect equipment failures before they occur. By analyzing temperature, vibration, and pressure readings as they stream in, predictive maintenance systems can alert operators to potential issues, reducing downtime. - **Personalization engines**: E-commerce platforms track user behavior in real-time to adjust recommendations and pricing using clickstream analytics. Every click, view, and purchase updates the user's profile, enabling immediate personalization of the shopping experience. For clickstream patterns, see [Clickstream Analytics with Kafka](https://www.conduktor.io/glossary/clickstream-analytics-with-kafka). - **Gaming analytics**: Gaming companies process millions of player events per second to detect cheating, balance gameplay, and personalize experiences in real-time. For specialized gaming patterns, see [Real-Time Gaming Analytics with Streaming](https://www.conduktor.io/glossary/real-time-gaming-analytics-with-streaming). - **Machine learning inference**: Real-time ML systems analyze streaming data to make instant predictions, from content recommendations to predictive maintenance alerts. For ML patterns, see [Real-Time ML Inference with Streaming Data](https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data). - **Network operations**: Telecommunications companies analyze network traffic patterns in real-time to detect anomalies, optimize routing, and identify potential security threats. Processing millions of events per second allows operators to respond to issues before customers notice degraded service. For security patterns, see [Real-Time Threat Detection](https://www.conduktor.io/glossary/real-time-threat-detection). ## Summary Real-time analytics with streaming data represents a fundamental shift from traditional batch processing, enabling organizations to act on insights as events occur rather than hours or days later. The ecosystem has matured significantly, with Apache Kafka providing data distribution, stream processors like Apache Flink and Kafka Streams handling analytics, and modern analytical databases like Apache Pinot and RisingWave enabling SQL queries on streaming data. Success requires understanding the unique characteristics of streaming data: its unbounded nature, the distinction between event time and processing time, and the challenges of late and out-of-order events handled through watermarks. Common patterns like windowing, stateful processing, and stream joins provide building blocks for implementing analytics, while modern improvements in exactly-once semantics (Flink 1.13+, Kafka 3.0+) have made these systems production-ready at scale. Architectural choices matter: Lambda architecture offers separate batch and streaming paths, Kappa simplifies with streaming-only processing, and lakehouse integration provides unified batch/streaming access through Apache Iceberg or Delta Lake. Comprehensive monitoring of consumer lag, processing latency, backpressure, and state size ensures reliable operations. As more systems generate continuous data streams, the ability to analyze and act on that data in real-time becomes increasingly important. Organizations that master real-time analytics gain advantages through faster decision-making, improved customer experiences, and operational efficiency across domains from fraud detection to personalization to IoT monitoring. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/apache-kafka) - Distributed streaming platform for data distribution in real-time analytics - [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) - Stateful stream processing engine for complex analytics - [Consumer Lag Monitoring](https://www.conduktor.io/consumer-lag-monitoring) - Track stream processing health and performance ## Sources and References 1. Kleppmann, M. (2017). *Designing Data-Intensive Applications*. O'Reilly Media. Chapter 11: Stream Processing. 2. Akidau, T., Chernyak, S., & Lax, R. (2018). *Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing*. O'Reilly Media. 3. Apache Flink Documentation. (2025). "Concepts: Timely Stream Processing." https://flink.apache.org/ 4. Narkhede, N., Shapira, G., & Palino, T. (2017). *Kafka: The Definitive Guide*. O'Reilly Media. 5. Apache Kafka Documentation. (2025). "KRaft Mode and Stream Processing." https://kafka.apache.org/ 6. Apache Pinot Documentation. (2025). "Real-Time Analytics at Scale." https://pinot.apache.org/ Source: https://www.conduktor.io/glossary/real-time-analytics-with-streaming-data --- # Real-Time Data Clean Room: Definition A **real-time data clean room** is a controlled environment where two or more organizations collaborate on live data streams without exposing the raw data behind them. It applies the core clean room guarantee, "you can use my data without seeing all of it", to events as they happen, through access control, field-level masking, and filtering enforced on the stream itself rather than on replicated tables. ![Real-time data clean room: a provider's source stream passes through an enforcement layer applying access control, field-level masking, and filtering, so each partner consumes a different governed view with no raw data exchange](https://www.conduktor.io/assets/images/glossary/real-time-data-clean-room-0.webp) ## What Is a Data Clean Room? A data clean room is a secure environment where multiple parties bring data together for joint analysis while each party keeps control over what the others can see. A retailer and a brand can measure campaign overlap on their combined customer data, for example, without either side handing over its customer list. The concept comes from advertising and analytics. Platforms such as AWS Clean Rooms, Snowflake Data Clean Rooms, and Google Ads Data Hub let participants run approved queries against joined datasets and receive only aggregated or policy-filtered results. These are mature, well-designed products for their purpose: collaborative analytics on datasets at rest. Three properties define the category: - **No raw data exchange.** Each party's data stays under its own governance. Collaborators query or consume a governed view, never the underlying records. - **Purpose-bound access.** What a participant can see is defined by explicit policy: which fields, which rows, which aggregation levels. - **Auditability.** Every access is logged, so each party can prove to regulators and to each other that the rules were followed. ## How a Real-Time Clean Room Differs from a Batch Clean Room Conventional data clean rooms operate on tables. Data is loaded, joined, and queried on a schedule, and the collaboration output is an analytical result: an overlap report, an attribution model, an aggregated segment. This works well when the question is analytical and the answer can wait for the next batch cycle. A real-time data clean room applies the same guarantees to data in motion. Instead of governing queries over replicated tables, it governs consumption of live event streams, typically on a platform such as Apache Kafka. The differences follow from that: | Dimension | Batch data clean room | Real-time data clean room | |---|---|---| | Data shape | Tables and datasets at rest | Live event streams | | Data movement | Data is loaded or replicated into the collaboration environment | No copy; consumers read governed views of the source stream | | Enforcement point | Query engine and result policies | Wire-level protocol filtering, masking, and access control | | Output | Aggregated results, reports, model inputs | A continuous, policy-filtered stream per consumer | | Latency | Batch or scheduled | Seconds or less, as events occur | | Typical question | "What is our audience overlap?" | "Which of these events is this partner allowed to react to, right now?" | The two are complements, not competitors. A batch clean room answers analytical questions on joined historical data. A real-time clean room governs the operational flow of events between parties. Many organizations need both: the same partnership might use an analytical clean room for measurement and a governed stream for live activation. ## Architecture of a Real-Time Data Clean Room The defining component is an enforcement layer that sits between the data owner's streaming platform and each external consumer. In practice this is a protocol-aware proxy or gateway: consumers connect to it as if it were the streaming platform itself, and every policy is applied on the wire, before any byte reaches the partner. Four capabilities make up the guarantee: - **Per-consumer identity and access control.** Each partner authenticates as a distinct principal and is authorized for specific streams only. A partner cannot discover or read anything outside its granted scope. - **Field-level masking and encryption.** Sensitive fields (names, account numbers, precise locations) are masked, tokenized, or encrypted per reader. Two partners consuming the same stream can receive two different projections of the same event. - **Row-level filtering.** Events are filtered by content, so a partner receives only the records relevant to the relationship: its own SKUs, its own shipments, its own merchant category. The rest of the stream never leaves the boundary. - **Audit logging.** Every connection, subscription, and read is recorded, giving the data owner a verifiable trail of exactly what each party consumed and when. Because enforcement happens at the protocol layer, there is no replicated dataset to secure, reconcile, or delete later. The data owner keeps a single source of truth, and each partner sees a purpose-built view of it. This is the streaming counterpart of [zero-copy data sharing](https://www.conduktor.io/glossary/zero-copy-data-sharing): access without duplication. For the building blocks, see [data masking and anonymization for streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming), [access control for streaming](https://www.conduktor.io/glossary/access-control-for-streaming), and [audit logging for streaming platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms). ## Use Cases - **Retail media and brand partnerships.** A retailer shares live purchase and inventory signals with brand advertisers. Each brand receives only events for its own products, with shopper identifiers masked, enabling real-time campaign activation without exposing the retailer's customer data or other brands' performance. - **Banking and fraud consortiums.** Financial institutions exchange transaction signals to detect fraud patterns that no single bank can see alone. Field-level masking strips account holders' PII while preserving the attributes fraud models need, and audit logs document compliance with data-sharing agreements. See [PII detection and handling in event streams](https://www.conduktor.io/glossary/pii-detection-and-handling-in-event-streams). - **Logistics and supply chain visibility.** A shipper streams tracking events to carriers, 3PLs, and retail partners. Each party sees only its own shipments in real time, with commercial fields such as order values filtered out, replacing daily file exchanges with a governed live feed. These patterns extend the broader set of [cross-organization data sharing patterns](https://www.conduktor.io/glossary/cross-organization-data-sharing-patterns) with clean-room-grade guarantees on the stream itself. ## Honest Limits The term "clean room" sets expectations that a real-time implementation only partly meets, so it is worth being precise: - **No secure multi-party computation.** Analytical clean rooms can compute joint results (overlap counts, attribution) such that neither party sees the other's inputs. A real-time clean room does not compute over joined data; it governs what each consumer receives from a stream. If the collaboration question is "what does our combined data say?", a batch clean room is the right tool. - **No differential privacy or aggregation engine.** Masking and filtering protect individual fields and records, but the output is still event-level data, not a privacy-budgeted aggregate. - **Trust concentrates in the enforcement layer.** The guarantee is only as strong as the proxy's policy configuration and the network controls preventing consumers from bypassing it. A more accurate framing: a real-time data clean room is governed real-time data sharing that delivers clean-room-style guarantees (no raw exposure, purpose-bound views, full audit) on streams. It complements analytical clean rooms rather than replacing them, and the same governed streams often feed those analytical environments downstream. ## Implementing Real-Time Clean Room Controls Wire-level enforcement of this kind is what streaming proxies are built for. Tools like [Conduktor Gateway](https://www.conduktor.io/gateway) implement per-consumer masking, filtering, and audit at the Kafka protocol layer, with Partner Zones packaging these controls for external sharing. For a vendor-aware walkthrough of governed stream sharing, see [real-time data sharing](https://www.conduktor.io/solutions/use-case/data-sharing), and for how streaming-based sharing relates to table-based protocols such as Delta Sharing, see [Delta Sharing alternatives for real-time data](https://www.conduktor.io/compare/delta-sharing-alternatives). ## Related Concepts - [Zero-Copy Data Sharing](https://www.conduktor.io/glossary/zero-copy-data-sharing) - Sharing data through logical access instead of physical duplication - [Cross-Organization Data Sharing Patterns](https://www.conduktor.io/glossary/cross-organization-data-sharing-patterns) - Architectural patterns for sharing data across company boundaries - [Data Masking and Anonymization for Streaming](https://www.conduktor.io/glossary/data-masking-and-anonymization-for-streaming) - Field-level protection techniques for data in motion - [Access Control for Streaming](https://www.conduktor.io/glossary/access-control-for-streaming) - Authentication and authorization models for event streams - [Audit Logging for Streaming Platforms](https://www.conduktor.io/glossary/audit-logging-for-streaming-platforms) - Building the verifiable access trail clean rooms depend on ## Sources and References - [AWS Clean Rooms Documentation](https://docs.aws.amazon.com/clean-rooms/latest/userguide/what-is.html) - Collaborative analytics without sharing raw data - [Snowflake Data Clean Rooms](https://docs.snowflake.com/en/user-guide/cleanrooms/introduction) - Clean room collaboration on the Snowflake platform - [IAB Tech Lab: Data Clean Rooms Guidance](https://iabtechlab.com/standards/data-clean-rooms/) - Industry standards and definitions for clean room interoperability - [Delta Sharing Protocol](https://delta.io/sharing/) - Open protocol for sharing tables without copying data - [Apache Kafka Documentation: Security](https://kafka.apache.org/documentation/#security) - Authentication, authorization, and encryption primitives for streams Source: https://www.conduktor.io/glossary/real-time-data-clean-room --- # Real-Time Fraud Detection with Streaming Fraud detection has evolved from batch processing delayed by hours or days to real-time systems that analyze transactions in milliseconds. This shift is critical because fraudulent transactions can cause immediate financial damage, and prevention must happen before transactions complete. Streaming platforms provide the infrastructure to process millions of events per second, enabling organizations to detect and prevent fraud as it occurs. This article explores how streaming technologies enable real-time fraud detection, covering core architectures, detection patterns, implementation examples, and operational challenges. ## The Challenge with Traditional Fraud Detection Traditional fraud detection systems rely on batch processing. Transactions are collected throughout the day, analyzed overnight, and suspicious activity is flagged the next morning. This delay creates a window of opportunity for fraudsters. Consider a stolen credit card scenario. In a batch system, a fraudster could make dozens of purchases before the fraud detection system even processes the first transaction. By the time fraud is detected, significant damage has occurred. Real-time fraud detection solves this by analyzing each transaction immediately. If a card is used in New York at 2:00 PM and then in London at 2:05 PM, the system can flag or block the second transaction instantly because the physical impossibility indicates fraud. ## How Streaming Enables Real-Time Fraud Detection Streaming platforms like [Apache Kafka](https://www.conduktor.io/apache-kafka) and processing engines like [Apache Flink](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) form the backbone of modern fraud detection systems. Here's how they work together: - **Event Streaming with Kafka**: Every transaction becomes an event published to Kafka topics. When a customer swipes a card, submits a payment, or transfers money, that event flows through Kafka in real-time. Kafka's distributed architecture handles millions of transactions per second while guaranteeing event ordering and durability. - **Stream Processing with Flink**: Flink consumes transaction events from Kafka and applies fraud detection logic in real-time. Unlike batch systems that process data in chunks, Flink processes each event individually as it arrives. It maintains stateful computations, tracking customer behavior patterns, transaction history, and risk scores across time windows. - **Low Latency**: Streaming systems process events in milliseconds, not hours. This speed is essential because fraud detection must complete before the transaction authorization responds to the customer. Payment processors typically have 100-200 millisecond windows to approve or decline transactions. The streaming approach turns fraud detection from a reactive report into a preventive control system. ## Key Components of a Streaming Fraud Detection System A complete fraud detection pipeline consists of several interconnected components: - **Data Sources**: Transaction systems, payment gateways, mobile apps, and web applications generate events. These sources publish events to Kafka topics with schemas defining transaction amount, merchant, location, timestamp, customer ID, and device fingerprints. - **Event Processing Layer**: Stream processors like Flink consume events and apply detection logic. This layer performs stateful operations such as aggregating transaction counts per customer over rolling time windows, calculating velocity metrics, and comparing current behavior against historical patterns. - **Feature Store**: Machine learning models require features computed from streaming data. A feature store is a specialized database that stores and serves pre-computed features with extremely low latency (typically single-digit milliseconds). Modern feature stores like Feast (open source), Tecton, or AWS Feature Store provide low-latency access to computed features like "number of transactions in the last 10 minutes" or "average transaction amount over 30 days." These systems solve the critical challenge of feature consistency between training (offline, batch-computed features) and inference (online, real-time features), ensuring that your production models receive the same feature representations they were trained on. - **Decision Engine**: This component combines rule-based logic and ML model predictions to make fraud determinations. It might score each transaction on a 0-100 risk scale, where scores above 80 trigger automatic blocks and scores between 50-80 require additional verification. - **Action System**: When fraud is detected, the system must take action: declining transactions, sending alerts, triggering multi-factor authentication, or flagging accounts for review. These actions often feed back into Kafka as new events. - **State Management**: Stream processors maintain state across millions of customers. Flink uses pluggable state backends to track customer profiles, running aggregations, and model predictions efficiently. RocksDB is the most common choice for large-scale production deployments, providing disk-backed state that can handle terabytes of data. For lower-latency scenarios with smaller state sizes, heap-based state backends store data in JVM memory, while cloud-native state backends like AWS S3 can reduce operational overhead at the cost of slightly higher latency. ## Common Fraud Detection Patterns and Techniques Real-time fraud detection employs several proven patterns: - **Velocity Checks**: These detect unusually high transaction frequency. If a customer typically makes 2-3 transactions per week but suddenly makes 20 transactions in an hour, the system flags this as suspicious. Implementation uses [tumbling or sliding windows](https://www.conduktor.io/windowing-in-apache-flink-tumbling-sliding-and-session-windows) to count events over time periods. - **Geolocation Impossibility**: As mentioned earlier, transactions in geographically distant locations within impossible timeframes indicate fraud. The system calculates distance and time between consecutive transactions to identify violations of physical constraints. - **Anomaly Detection**: Machine learning models learn normal customer behavior patterns and flag deviations that could indicate fraud. Modern fraud detection systems use algorithms like Isolation Forest (efficient for high-dimensional data), LSTM autoencoders (for detecting temporal sequence anomalies), or one-class SVM (for learning normal behavior boundaries). For example, a customer who always makes small purchases under $50 suddenly attempting a $5,000 transaction would score high on anomaly detection. In 2025, transformer-based models and vector embeddings have become increasingly popular, as they can capture complex patterns across multiple features simultaneously and detect subtle fraud signatures that traditional algorithms miss. - **Pattern Matching**: Complex Event Processing (CEP) is a technique for detecting specific sequences of events over time. Flink's CEP library allows you to define patterns declaratively and match them against streaming data. For instance, testing transactions with small amounts followed by large purchases is a common fraud pattern called "card testing." The CEP library can detect patterns like: "three transactions under $1 within 5 minutes, followed by a transaction over $500 within the next 10 minutes." When this pattern matches, the system flags the account for immediate review or blocking. - **Network Analysis**: Graph processing identifies fraud rings where multiple accounts share devices, IP addresses, or shipping addresses. These network patterns are difficult to detect in isolation but become obvious when analyzing relationship graphs. Graph databases like Neo4j and TigerGraph excel at this analysis, storing entity relationships (accounts, devices, addresses, payment methods) and running graph algorithms like community detection, PageRank, and shortest-path analysis to identify suspicious clusters. Streaming graph processing tools can update these graphs in real-time as new transactions flow through Kafka, enabling dynamic fraud ring detection. ## Architecture Example: Building a Fraud Detection Pipeline Here's a concrete architecture for a fraud detection system: ![Here's a concrete architecture for a fraud detection system](https://www.conduktor.io/assets/images/glossary/real-time-fraud-detection-with-streaming-0.webp) The Flink job implements multiple detection strategies: 1. **Velocity Check**: Counts transactions per customer using a 10-minute sliding window 2. **Amount Anomaly**: Compares transaction amount against customer's 30-day average 3. **ML Scoring**: Invokes a deployed model with transaction features 4. **Final Decision**: Combines all signals into a risk score and decision State management is crucial. The Flink job maintains: - Customer transaction counts per time window - Customer average transaction amounts - Last transaction location per customer - Fraud pattern states (for CEP) This state must be fault-tolerant and recoverable. Flink checkpoints state to durable storage, ensuring that even if the processing job crashes, it can resume with accurate state. ## Implementation Example: Velocity Check with Flink Here's a concrete example of implementing velocity-based fraud detection using Flink's DataStream API: ```java DataStream transactions = env .addSource(new FlinkKafkaConsumer<>( "transactions", new TransactionDeserializationSchema(), kafkaProps )); DataStream alerts = transactions .keyBy(txn -> txn.getCustomerId()) .window(SlidingEventTimeWindows.of(Time.minutes(10), Time.minutes(1))) .aggregate(new TransactionCountAggregator(), new VelocityChecker()); alerts.addSink(new FlinkKafkaProducer<>( "fraud-alerts", new FraudAlertSerializationSchema(), kafkaProps )); // Aggregator counts transactions per customer in window public static class TransactionCountAggregator implements AggregateFunction { @Override public TransactionCount createAccumulator() { return new TransactionCount(0, 0.0); } @Override public TransactionCount add(Transaction txn, TransactionCount acc) { return new TransactionCount( acc.count + 1, acc.totalAmount + txn.getAmount() ); } @Override public TransactionCount getResult(TransactionCount acc) { return acc; } @Override public TransactionCount merge(TransactionCount a, TransactionCount b) { return new TransactionCount( a.count + b.count, a.totalAmount + b.totalAmount ); } } // Checker evaluates if velocity exceeds threshold public static class VelocityChecker extends ProcessWindowFunction { @Override public void process(String customerId, Context ctx, Iterable counts, Collector out) { TransactionCount count = counts.iterator().next(); // Alert if customer exceeds 10 transactions in 10-minute window if (count.count > 10) { out.collect(new FraudAlert( customerId, "VELOCITY_EXCEEDED", count.count, count.totalAmount, ctx.window().getStart() )); } } } ``` This example demonstrates: - Consuming transactions from Kafka in real-time - Keying by customer ID to track per-customer behavior - Using sliding windows to continuously evaluate transaction velocity - Aggregating transaction counts and amounts efficiently - Publishing fraud alerts back to Kafka for downstream action For Pattern Matching with Flink CEP: ```java Pattern cardTestingPattern = Pattern .begin("small-tests") .where(new SimpleCondition() { @Override public boolean filter(Transaction txn) { return txn.getAmount() < 1.0; } }) .times(3).within(Time.minutes(5)) .followedBy("large-purchase") .where(new SimpleCondition() { @Override public boolean filter(Transaction txn) { return txn.getAmount() > 500.0; } }) .within(Time.minutes(10)); PatternStream patternStream = CEP.pattern( transactions.keyBy(Transaction::getCustomerId), cardTestingPattern ); DataStream cepAlerts = patternStream.select( new PatternSelectFunction() { @Override public FraudAlert select(Map> pattern) { List tests = pattern.get("small-tests"); Transaction largePurchase = pattern.get("large-purchase").get(0); return new FraudAlert( largePurchase.getCustomerId(), "CARD_TESTING_DETECTED", tests.size(), largePurchase.getAmount(), largePurchase.getTimestamp() ); } } ); ``` This CEP example detects the classic "card testing" pattern where fraudsters make multiple small transactions to verify a stolen card works before attempting a large purchase. ## ML Model Integration Modern fraud detection systems combine rule-based checks with machine learning models. Here's how to integrate ML inference: ```java // Async I/O for non-blocking model inference DataStream scoredTransactions = transactions .keyBy(Transaction::getCustomerId) .process(new AsyncDataStream.unorderedWait( new AsyncMLScoringFunction(), 1000, // timeout TimeUnit.MILLISECONDS, 100 // max concurrent requests )); public class AsyncMLScoringFunction extends AsyncFunction { private transient MLModelClient modelClient; @Override public void open(Configuration parameters) { // Initialize model client (TensorFlow Serving, SageMaker, etc.) modelClient = new MLModelClient("fraud-model-v3", endpoint); } @Override public void asyncInvoke(Transaction txn, ResultFuture resultFuture) { // Build feature vector from transaction FeatureVector features = buildFeatures(txn); // Async call to ML model endpoint CompletableFuture scoreFuture = modelClient.predict(features); scoreFuture.thenAccept(fraudScore -> { resultFuture.complete(Collections.singleton( new ScoredTransaction(txn, fraudScore) )); }); } private FeatureVector buildFeatures(Transaction txn) { return FeatureVector.builder() .add("amount", txn.getAmount()) .add("merchant_category", txn.getMerchantCategory()) .add("hour_of_day", txn.getTimestamp().getHour()) .add("day_of_week", txn.getTimestamp().getDayOfWeek().getValue()) // Additional features from feature store .build(); } } // Combine multiple signals into final decision DataStream decisions = scoredTransactions .keyBy(ScoredTransaction::getCustomerId) .process(new DecisionEngine()); public class DecisionEngine extends KeyedProcessFunction< String, ScoredTransaction, FraudDecision> { private transient ValueState profileState; @Override public void processElement(ScoredTransaction scored, Context ctx, Collector out) { CustomerProfile profile = profileState.value(); double mlScore = scored.getFraudScore(); // Combine ML score with rules String decision; if (mlScore > 0.8 || isGeolocationImpossible(scored, profile)) { decision = "BLOCK"; } else if (mlScore > 0.5) { decision = "REVIEW"; } else { decision = "APPROVE"; } out.collect(new FraudDecision( scored.getTransaction(), decision, mlScore )); // Update customer profile profileState.update(updateProfile(profile, scored)); } } ``` This demonstrates: - Asynchronous ML model inference to avoid blocking stream processing - Building feature vectors from transaction data - Combining multiple fraud signals (ML scores, rule-based checks) - Maintaining customer profile state for context-aware decisions ## Operational Challenges and Best Practices Running production fraud detection systems presents several operational challenges: - **Data Quality**: Fraud detection accuracy depends on clean, consistent data. [Schema evolution](https://www.conduktor.io/schema-registry-and-schema-management), missing fields, or malformed events can cause false positives or missed fraud. Conduktor provides schema validation, data quality monitoring, and issue detection capabilities that help catch problems before they impact fraud detection accuracy. For comprehensive cluster and application monitoring, see the [Conduktor Monitoring Guide](https://docs.conduktor.io/guide/monitor-brokers-apps/index). Conduktor Gateway, a Kafka proxy, acts as a policy enforcement layer between producers and Kafka, validating schemas, enforcing data contracts, and blocking malformed events before they enter your fraud detection pipeline. - **Pipeline Monitoring**: When fraud detection fails, understanding why requires visibility into the pipeline. Are events arriving in Kafka? Is consumer lag growing on the topics that Flink reads from? Conduktor provides monitoring for the Kafka layer — real-time visibility into [consumer lag](https://www.conduktor.io/consumer-lag-monitoring), topic throughput, and message inspection — helping teams identify whether issues originate in the Kafka topics feeding the Flink job. For Flink job internals (checkpoint statistics, backpressure, task manager metrics), use Flink's built-in web UI. - **False Positives**: Overly aggressive fraud detection frustrates legitimate customers. Teams must continuously tune thresholds and models to balance fraud prevention with customer experience. A/B testing different configurations requires careful pipeline management and data lineage tracking. - **Backpressure and Performance**: Fraud detection must keep up with transaction volumes during peak periods. If the processing pipeline falls behind, latency increases and fraud can slip through. [Backpressure](https://www.conduktor.io/backpressure-handling-in-streaming-systems) occurs when downstream fraud detection components can't process transactions as fast as they arrive. Monitoring [consumer lag](https://www.conduktor.io/consumer-lag-monitoring), processing rates, and resource utilization is essential. Flink provides backpressure monitoring built into its web UI, showing which operators are bottlenecks. - **Testing and Validation**: Testing fraud detection logic before production deployment is critical. Replaying historical transaction data through updated fraud detection rules helps validate that changes improve detection without increasing false positives. Kafka's time-travel capabilities and Flink's savepoints enable safe testing by restoring state to specific points in time. Conduktor's data masking features protect sensitive information during testing, allowing teams to work with production-like data while maintaining PII compliance. - **Model Monitoring and Drift Detection**: ML models degrade over time as fraud patterns evolve. In 2025, production fraud detection systems implement continuous model monitoring to detect performance degradation. Track metrics like precision, recall, F1 score, and false positive rate in real-time. When fraud tactics change, model accuracy drops, this is called concept drift. Tools like Evidently AI, WhyLabs, or custom Flink jobs can compute drift metrics by comparing recent predictions against baseline distributions. When drift is detected, trigger model retraining pipelines automatically. Track model versions in production, A/B test new models against current ones, and maintain model lineage for compliance and debugging. - **Compliance and Auditing**: Financial regulations require audit trails showing why transactions were blocked or approved. Streaming systems must capture decision metadata, model versions, and rule configurations for compliance purposes. Store all fraud decisions with explainability data: which rules triggered, what the ML model score was, what features influenced the decision. This audit trail enables regulatory compliance, customer dispute resolution, and continuous improvement of fraud detection logic. ## Summary Real-time fraud detection with streaming platforms transforms fraud prevention from reactive reporting to proactive blocking. By processing transactions as events in systems like [Apache Kafka](https://www.conduktor.io/apache-kafka) and [Apache Flink](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing), organizations can detect fraud in milliseconds rather than hours or days. Key takeaways include: - Streaming enables sub-second fraud detection, essential for preventing fraudulent transactions before they complete - Modern architectures combine Kafka for event streaming with Flink for stateful stream processing - Effective fraud detection uses multiple techniques: velocity checks, anomaly detection, pattern matching, and [ML models](https://www.conduktor.io/real-time-ml-inference-with-streaming-data) - Advanced detection leverages graph databases (Neo4j, TigerGraph) for fraud ring identification and transformer models with vector embeddings for sophisticated pattern matching - Feature stores (Feast, Tecton, AWS Feature Store) solve train-serve skew by maintaining consistent features between model training and real-time inference - Operational excellence requires strong data quality, pipeline monitoring, model drift detection, and testing practices - Conduktor and Conduktor Gateway provide essential visibility, policy enforcement, and data quality monitoring for production fraud detection systems - Continuous model monitoring detects concept drift and triggers retraining as fraud patterns evolve As fraud techniques evolve, the ability to rapidly deploy new detection logic and process events at scale becomes more important. Streaming platforms provide the foundation for building adaptive fraud detection systems that protect businesses and customers alike. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/apache-kafka) - Distributed streaming platform that powers fraud detection pipelines - [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) - Deep dive into stateful stream processing for fraud detection - [Exactly-Once Semantics](https://www.conduktor.io/exactly-once-semantics) - Critical processing guarantee for financial fraud detection systems ## Related Topics - [Real-Time ML Inference with Streaming Data](https://www.conduktor.io/real-time-ml-inference-with-streaming-data) - Learn how to deploy ML models for real-time predictions on streaming data - [Consumer Lag Monitoring](https://www.conduktor.io/consumer-lag-monitoring) - Essential for ensuring fraud detection keeps pace with transaction volumes - [Backpressure Handling in Streaming Systems](https://www.conduktor.io/backpressure-handling-in-streaming-systems) - Strategies for handling load spikes during fraud attacks - [Schema Registry and Schema Management](https://www.conduktor.io/schema-registry-and-schema-management) - Maintaining data quality and preventing schema-related false positives - [What is Apache Flink](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) - Deep dive into stateful stream processing for fraud detection - [Streaming Data in Financial Services](https://www.conduktor.io/streaming-data-in-financial-services) - Broader context of streaming in finance beyond fraud detection ## Sources and References 1. Apache Flink Documentation - "Fraud Detection with the DataStream API" - https://nightlies.apache.org/flink/flink-docs-stable/docs/try-flink/datastream/ 2. Apache Kafka Documentation - "Use Cases: Fraud Detection" - https://kafka.apache.org/uses 3. Stripe Engineering - "Scaling Online Payment Fraud Detection with Streaming" - https://stripe.com/blog/online-payment-fraud-detection 4. Dal Pozzolo, Andrea et al. (2015) - "Learned lessons in credit card fraud detection from a practitioner perspective" - Expert Systems with Applications, Volume 41, Issue 10 5. Amazon Web Services - "Real-Time Fraud Detection Architecture Patterns" - https://aws.amazon.com/solutions/implementations/fraud-detection-using-machine-learning/ 6. Evidently AI - "Monitoring Machine Learning Models in Production" - https://www.evidentlyai.com/ 7. Feast Feature Store Documentation - "Real-Time Feature Serving" - https://docs.feast.dev/ Source: https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming --- # Real-Time Gaming Analytics with Streaming Modern gaming platforms generate enormous volumes of data every second. From player movements and combat actions to in-game purchases and social interactions, these events must be captured, processed, and analyzed in real-time to deliver competitive, engaging experiences. Unlike batch analytics that process data hours or days later, real-time gaming analytics requires streaming architectures capable of handling millions of events per second with sub-second latency. ![Gaming analytics streaming architecture with multi-region support](https://www.conduktor.io/assets/images/glossary/real-time-gaming-analytics-with-streaming-0.webp) For example, when a player fires a weapon in a multiplayer shooter, that action generates an event containing the player's ID, weapon type, target, position, and timestamp. This event must be processed immediately to update kill/death statistics, detect potential cheating (impossible accuracy or reaction times), and adjust matchmaking rankings - all within milliseconds. This article explores how [streaming data platforms](https://www.conduktor.io/glossary/what-is-real-time-data-streaming) like Apache Kafka and Apache Flink enable real-time analytics in gaming, the technical challenges involved, and the architectural patterns that make it possible. ## Understanding Gaming Analytics in Real-Time Gaming analytics encompasses the collection and analysis of all player interactions and system telemetry within a game. This includes player behavior (kills, deaths, movement patterns), game state changes (level completions, achievements), technical metrics (latency, frame rate), and business events (purchases, ad impressions). The real-time aspect is critical because many gaming use cases cannot tolerate delays. Anti-cheat systems must detect suspicious patterns within seconds to prevent damage. Matchmaking algorithms need current player skill ratings to create balanced matches. Live operations teams require instant visibility into server issues affecting thousands of concurrent players. Traditional analytics approaches that batch process data every hour or day are inadequate for these requirements. Streaming architectures that process events as they occur have become the standard for modern gaming platforms. ## The Event Landscape: What Data Gaming Systems Generate A typical multiplayer game generates diverse event types across multiple dimensions: - **Player Actions**: Every button press, movement, ability use, and combat interaction generates an event. A single player in a fast-paced shooter might generate 100-500 events per minute during active gameplay. To put this in perspective: a popular game with 1 million concurrent players could generate 1.7 million to 8.3 million events per minute (28,000-139,000 events per second) from player actions alone. - **Game State Events**: Level completions, quest progress, achievement unlocks, and inventory changes represent higher-level game state. These events are less frequent but carry important context about player progression and engagement. - **Social and Economic Events**: Friend requests, chat messages, clan activities, marketplace transactions, and in-game purchases form a critical data stream for community management and monetization analytics. - **System Telemetry**: Server performance metrics, network latency measurements, client frame rates, and error logs provide operational visibility. This technical telemetry is essential for maintaining service quality. Each event type requires different processing logic, retention policies, and latency requirements. A streaming architecture must handle this heterogeneity while maintaining order guarantees and [exactly-once processing](https://www.conduktor.io/glossary/exactly-once-semantics-in-kafka) semantics where required. Here's an example of a player action event schema using Avro format: ```json { "type": "record", "name": "PlayerAction", "namespace": "com.game.events", "fields": [ {"name": "event_id", "type": "string"}, {"name": "timestamp", "type": "long", "logicalType": "timestamp-millis"}, {"name": "player_id", "type": "string"}, {"name": "session_id", "type": "string"}, {"name": "action_type", "type": {"type": "enum", "name": "ActionType", "symbols": ["MOVE", "SHOOT", "USE_ABILITY", "RELOAD", "INTERACT"]}}, {"name": "position", "type": {"type": "record", "name": "Position", "fields": [ {"name": "x", "type": "float"}, {"name": "y", "type": "float"}, {"name": "z", "type": "float"} ]}}, {"name": "target_id", "type": ["null", "string"], "default": null}, {"name": "metadata", "type": {"type": "map", "values": "string"}}, {"name": "schema_version", "type": "string", "default": "1.0"} ] } ``` This schema provides type safety, enables schema evolution, and includes metadata for tracking schema versions as game features change over time. ## Streaming Architecture for Gaming Analytics Apache Kafka has become the backbone of gaming analytics pipelines due to its ability to handle high-throughput event ingestion, durable storage, and exactly-once processing semantics. Modern deployments (2025) use [KRaft mode](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka), which eliminates the ZooKeeper dependency and reduces operational complexity while improving scalability to millions of partitions. Game servers and clients publish events to Kafka topics, partitioned by player ID or game session to maintain ordering within a player's event stream. This partitioning strategy ensures that all events for a given player are processed in sequence, which is critical for game state reconstruction and detecting temporal patterns. A typical architecture includes several layers: **Ingestion Layer**: Game servers write events to Kafka topics through high-performance producers configured with idempotent writes and transactional guarantees. Events are typically serialized using [Avro, Protocol Buffers, or JSON Schema](https://www.conduktor.io/glossary/avro-vs-protobuf-vs-json-schema) to ensure [schema evolution](https://www.conduktor.io/glossary/schema-evolution-best-practices) compatibility as game features change. [Schema Registry](https://www.conduktor.io/glossary/schema-registry-and-schema-management) validates all events against registered schemas before they enter the pipeline, preventing malformed data from corrupting downstream analytics. ```java // Example: Publishing player action events with idempotent producer Properties props = new Properties(); props.put("bootstrap.servers", "kafka-cluster:9092"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer"); props.put("schema.registry.url", "http://schema-registry:8081"); props.put("enable.idempotence", "true"); // Exactly-once semantics props.put("acks", "all"); props.put("compression.type", "lz4"); KafkaProducer producer = new KafkaProducer<>(props); PlayerAction action = PlayerAction.newBuilder() .setEventId(UUID.randomUUID().toString()) .setTimestamp(System.currentTimeMillis()) .setPlayerId(playerId) .setSessionId(sessionId) .setActionType(ActionType.SHOOT) .setPosition(new Position(x, y, z)) .build(); ProducerRecord record = new ProducerRecord<>("player-actions", playerId, action); producer.send(record, (metadata, exception) -> { if (exception != null) { logger.error("Failed to publish event", exception); } }); ``` **Processing Layer**: [Apache Flink](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing) and Kafka Streams applications consume events from Kafka topics and perform stateful computations. This might include calculating player statistics, detecting anomalous behavior patterns, or aggregating metrics across game sessions. Modern gaming platforms (2025) increasingly use Flink SQL for ad-hoc analytics queries, enabling analysts to query streaming data using familiar SQL syntax. For example, calculating real-time player engagement metrics: ```sql -- Flink SQL: Real-time player engagement metrics SELECT window_start, COUNT(DISTINCT player_id) as active_players, COUNT(*) as total_actions, AVG(session_duration_ms) as avg_session_duration FROM TABLE( TUMBLE(TABLE player_actions, DESCRIPTOR(event_time), INTERVAL '5' MINUTES) ) GROUP BY window_start; ``` Kafka Streams provides lightweight [stateful processing](https://www.conduktor.io/glossary/state-stores-in-kafka-streams) for scenarios requiring embedded stream processing within game services: ```java // Kafka Streams: Real-time player statistics StreamsBuilder builder = new StreamsBuilder(); KStream actions = builder.stream("player-actions"); KTable playerStats = actions .groupByKey() .aggregate( PlayerStats::new, (playerId, action, stats) -> stats.update(action), Materialized.>as("player-stats-store") .withKeySerde(Serdes.String()) .withValueSerde(new JsonSerde<>(PlayerStats.class)) ); // Expose stats via interactive queries for low-latency lookups ReadOnlyKeyValueStore store = streams.store(StoreQueryParameters.fromNameAndType("player-stats-store", QueryableStoreTypes.keyValueStore())); ``` **Storage Layer**: Processed results flow to multiple destinations based on use case. Time-series databases (InfluxDB, TimescaleDB) store metrics for real-time dashboards. Analytical databases (ClickHouse, DuckDB) receive aggregated statistics for business intelligence. Low-latency data stores (Redis, ScyllaDB) cache current player states for matchmaking or personalization systems. The streaming platform serves as the central nervous system, ensuring events flow reliably from source to destination while enabling multiple consumers to process the same event stream for different purposes. ## Key Use Cases: From Anti-Cheat to Player Engagement Real-time streaming analytics enables several critical gaming use cases: **Anti-Cheat Detection**: Modern anti-cheat systems (2025) combine rule-based pattern detection with machine learning models to identify cheating in real-time. Streaming processors analyze player behavior patterns using Complex Event Processing (CEP) and ML inference. For example, a Flink CEP pattern might detect impossible player movements (teleportation), while an ML model evaluates behavioral features extracted from the event stream: ```java // Flink CEP: Detect impossible player movements Pattern teleportPattern = Pattern.begin("start") .where(new SimpleCondition() { @Override public boolean filter(PlayerAction action) { return action.getActionType() == ActionType.MOVE; } }) .next("end") .where(new SimpleCondition() { @Override public boolean filter(PlayerAction action) { return action.getActionType() == ActionType.MOVE; } }) .within(Time.milliseconds(100)); DataStream alerts = CEP.pattern(playerActions, teleportPattern) .select((PatternSelectFunction) pattern -> { PlayerAction start = pattern.get("start").get(0); PlayerAction end = pattern.get("end").get(0); double distance = calculateDistance(start.getPosition(), end.getPosition()); double timeDelta = end.getTimestamp() - start.getTimestamp(); double speed = distance / (timeDelta / 1000.0); // meters per second if (speed > MAX_PLAYER_SPEED) { return new Alert(start.getPlayerId(), "TELEPORT_DETECTED", speed); } return null; }); ``` For ML-based detection, features like headshot percentage, reaction time, aiming patterns, and movement consistency are extracted in real-time and fed to lightweight models (XGBoost, ONNX) deployed within Flink jobs for [real-time ML inference](https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data): ```java // Real-time ML inference for cheat detection DataStream cheatScores = playerActions .keyBy(PlayerAction::getPlayerId) .window(TumblingEventTimeWindows.of(Time.minutes(5))) .process(new ProcessWindowFunction() { private transient Model onnxModel; @Override public void open(Configuration parameters) { // Load ONNX model for inference onnxModel = loadONNXModel("cheat-detection-v2.onnx"); } @Override public void process(String playerId, Context context, Iterable actions, Collector out) { // Extract features from action window FeatureVector features = extractFeatures(actions); // Run ML inference double cheatProbability = onnxModel.predict(features); if (cheatProbability > 0.85) { out.collect(new CheatScore(playerId, cheatProbability, context.window().getEnd(), features)); } } }); ``` This dual approach (rule-based CEP + ML models) provides both immediate detection of obvious cheats and sophisticated detection of subtle behavioral anomalies. When a player's cheat score exceeds the threshold, the system can trigger automatic actions (shadow banning, forced logout) or flag for human review, all within seconds of detecting the suspicious behavior. - **Dynamic Matchmaking**: Modern matchmaking systems consider real-time player skill ratings, latency, and queue wait times. Streaming processors continuously update player rankings based on recent match results, ensuring the matchmaking algorithm uses fresh data to create balanced teams. - **Live Operations and Monetization**: Game developers run limited-time events and sales that require real-time tracking of player participation and spending. Streaming analytics power dashboards showing event engagement as it happens, enabling teams to adjust parameters or extend promotions based on live data. - **Player Retention and Engagement**: By analyzing play patterns in real-time, games can trigger personalized interventions. If a new player shows signs of frustration (repeated deaths, declining session length), the system might offer helpful tips or easier content to improve retention. ## Technical Challenges at Gaming Scale Gaming analytics at scale presents unique technical challenges: - **Volume and Velocity**: Popular games serve millions of concurrent players across global regions. This translates to billions of events per day, with peak loads reaching millions of events per second during new releases or special events. - **Latency Requirements**: Many use cases require sub-second processing. Anti-cheat systems must act quickly. Real-time leaderboards need instant updates. This demands careful tuning of streaming processors, including proper parallelization, state management, and checkpoint intervals. - **Global Distribution**: Players connect from worldwide locations, requiring data pipelines that span multiple regions while respecting data sovereignty regulations. Modern gaming platforms deploy active-active Kafka clusters across regions using MirrorMaker 2 (MM2) for bi-directional replication. For GDPR compliance, player data can be geo-partitioned by region, ensuring European player data remains in EU data centers: ```properties # MirrorMaker 2 configuration for cross-region replication clusters = us-west, eu-central, asia-pacific us-west.bootstrap.servers = kafka-us.example.com:9092 eu-central.bootstrap.servers = kafka-eu.example.com:9092 asia-pacific.bootstrap.servers = kafka-apac.example.com:9092 # Replicate player events across regions us-west->eu-central.enabled = true us-west->eu-central.topics = player-actions-global, game-telemetry us-west->eu-central.replication.factor = 3 # GDPR-protected topics stay regional eu-central->us-west.topics.exclude = player-pii-.* ``` Players are automatically routed to their nearest regional cluster, minimizing latency while maintaining data sovereignty. Analytics jobs aggregate data from regional clusters for global insights. **Schema Evolution**: Games constantly evolve with new features, requiring flexible event schemas that can accommodate changes without breaking existing consumers. Schema Registry enforces compatibility rules (FORWARD, BACKWARD, or FULL compatibility) to ensure safe schema evolution. For example, adding a new optional field to the PlayerAction schema: ```json { "type": "record", "name": "PlayerAction", "namespace": "com.game.events", "fields": [ {"name": "event_id", "type": "string"}, {"name": "timestamp", "type": "long"}, {"name": "player_id", "type": "string"}, {"name": "action_type", "type": "string"}, {"name": "position", "type": "Position"}, {"name": "weapon_id", "type": ["null", "string"], "default": null}, {"name": "new_field_2025", "type": ["null", "string"], "default": null} ] } ``` Schema Registry validates that this change is backward compatible (old consumers can still read new events by ignoring the new field) before allowing deployment. This prevents breaking production analytics pipelines when game features are added. **State Management**: Stateful processing at scale requires careful consideration. A player statistics aggregator maintaining state for millions of players must handle state size, recovery times, and consistent snapshots. ## Implementation Patterns and Best Practices Successful gaming analytics implementations follow several key patterns: - **Event Schema Design**: Use structured formats with explicit schemas. Include event metadata like timestamp, version, and player ID in every event. Partition events by player ID to maintain ordering of a player's action sequence. - **Stateful Stream Processing**: Leverage Flink's managed state for aggregations and pattern detection. For example, maintaining a [tumbling window](https://www.conduktor.io/glossary/windowing-in-apache-flink-tumbling-sliding-and-session-windows) of the last 100 player actions enables detection of suspicious patterns without reprocessing historical data. [Session windows](https://www.conduktor.io/glossary/session-windows-in-stream-processing) are particularly useful for analyzing player behavior within game sessions that have variable lengths. - **Exactly-Once Semantics**: For critical analytics like purchase tracking or achievement awards, configure exactly-once processing to prevent duplicate counting or missed events. - **Monitoring and Observability**: Instrument streaming pipelines with metrics on throughput, latency, consumer lag, and error rates. Dead letter queues capture problematic events for later analysis without blocking the main pipeline. - **Player State Reconstruction**: Gaming platforms must be able to reconstruct complete player state from event logs for debugging, customer support, and regulatory compliance. This requires: - **Event Compaction Strategies**: Use Kafka's log compaction for player profile topics, ensuring the latest state is always available while maintaining historical changes. - **Snapshotting**: Periodically snapshot player state to enable fast recovery and historical analysis. Flink's savepoints provide consistent snapshots of stateful computations. - **GDPR Compliance**: For handling the "right to be forgotten," implement a dual-topic pattern where PII lives in a separate compacted topic that can be tombstoned without losing the rest of the player's anonymized behavioral data: ```java // Separate PII and behavioral events for GDPR compliance ProducerRecord behaviorEvent = new ProducerRecord<>("player-behavior", playerId, behaviorData); ProducerRecord piiEvent = new ProducerRecord<>("player-pii-compacted", playerId, piiData); // On GDPR deletion request, publish tombstone to PII topic ProducerRecord deletionRecord = new ProducerRecord<>("player-pii-compacted", playerId, null); ``` **Serverless Analytics (2025)**: Modern gaming platforms increasingly use serverless functions (AWS Lambda, Google Cloud Functions) as lightweight Kafka consumers for specific analytics tasks: ```yaml # Example: Serverless function triggered by Kafka events functions: achievement-processor: handler: processAchievement events: - kafka: topic: player-actions batchSize: 100 startingPosition: LATEST environment: ACHIEVEMENTS_TABLE: player-achievements NOTIFICATIONS_TOPIC: player-notifications ``` This pattern enables auto-scaling individual analytics workloads without managing Flink or Kafka Streams clusters, reducing costs for low-throughput use cases. ## Managing Streaming Pipelines in Production Operating streaming analytics at gaming scale requires robust operational tooling and gaming-specific monitoring strategies. Teams need visibility into topic configurations, consumer lag, schema evolution, data quality, and game-specific metrics. **Gaming-Specific Monitoring**: Beyond standard Kafka metrics, gaming analytics requires tracking: - **Player Session Reconstruction**: Monitor the completeness of player event sequences. Missing events can indicate network issues, client crashes, or producer failures. Track the ratio of session-start to session-end events, and alert on gaps in player action sequences. - **Event Ordering Violations**: Since game state depends on correct event ordering, monitor out-of-order events within partitions. This can indicate clock skew issues or incorrect producer timestamps. - **Duplicate Event Detection**: Despite exactly-once semantics, monitor for duplicate event IDs. Duplicates can occur due to client retries or network issues, and must be deduplicated to maintain accurate statistics. - **[Consumer Lag](https://www.conduktor.io/glossary/consumer-lag-monitoring) by Use Case Priority**: Different consumers have different latency requirements. Anti-cheat systems require near-zero lag, while batch analytics can tolerate hours of lag. Set alerting thresholds based on use case criticality: ```yaml # Example monitoring thresholds for gaming analytics consumer_groups: anti-cheat-detector: max_lag_seconds: 5 alert_severity: critical matchmaking-ranker: max_lag_seconds: 30 alert_severity: high player-retention-ml: max_lag_seconds: 300 alert_severity: medium daily-analytics: max_lag_seconds: 3600 alert_severity: low ``` **Governance and Management**: Platforms like Conduktor provide centralized management and monitoring capabilities for Kafka clusters, making it easier to track consumer group performance, validate schema changes before deployment, and troubleshoot pipeline issues. For gaming analytics teams managing dozens or hundreds of Kafka topics across multiple environments (development, staging, production), Conduktor's unified interface provides: - Real-time consumer lag monitoring with customizable alerts ([Monitoring Guide](https://docs.conduktor.io/guide/monitor-brokers-apps/index)) - Schema Registry integration for safe schema evolution - Topic configuration management and drift detection ([Topics Guide](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics)) - Data quality monitoring and validation rules - ACL and security policy management for multi-team access This unified visibility reduces operational overhead and accelerates incident response when game launches or events cause traffic spikes. ## Summary Real-time gaming analytics has transformed how game developers understand and optimize player experiences. Streaming data platforms like Apache Kafka and Apache Flink provide the foundation for processing millions of player events per second with low latency. The key requirements include handling diverse event types, maintaining stateful computations across millions of players, and supporting use cases from anti-cheat detection to personalized engagement. While the technical challenges of scale, latency, and global distribution are significant, established architectural patterns and operational best practices make these systems achievable. As games grow in complexity and player bases expand, streaming analytics will remain essential for delivering competitive, engaging, and profitable gaming experiences. ## Related Concepts - [Apache Kafka](https://www.conduktor.io/apache-kafka) - Distributed streaming platform for gaming event processing - [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) - Stateful analytics for player behavior and anti-cheat detection - [Exactly-Once Semantics](https://www.conduktor.io/exactly-once-semantics) - Guarantee accurate player statistics and analytics ## Sources and References 1. Apache Kafka Documentation: "Use Cases: Gaming" - Technical overview of Kafka architectures in gaming platforms (https://kafka.apache.org/uses) 2. Apache Flink: "Stream Processing for Real-Time Analytics" - Documentation on stateful stream processing patterns (https://flink.apache.org) 3. Unity Technologies: "Understanding Game Analytics" - Industry perspective on gaming telemetry and player behavior analysis (https://unity.com/solutions/gaming-analytics) 4. AWS Blog: "Building Real-Time Gaming Analytics at Scale" - Architecture patterns for cloud-based gaming analytics (https://aws.amazon.com/blogs/gametech) 5. Riot Games Engineering Blog: "Data Infrastructure at Scale" - Real-world insights from League of Legends' data platform (https://technology.riotgames.com) 6. Conduktor: "Kafka Governance and Management" - Platform for managing and monitoring Kafka infrastructure at scale (https://www.conduktor.io) Source: https://www.conduktor.io/glossary/real-time-gaming-analytics-with-streaming --- # Real-Time ML Inference with Streaming Data Machine learning inference — the process of making predictions using trained models — traditionally happens in batch mode, processing historical data on fixed schedules. Many modern applications require predictions within milliseconds of receiving new data. Real-time ML inference with streaming data lets organizations act on insights immediately, powering use cases from fraud detection to personalized recommendations. ![Real-time ML inference pipeline with feature enrichment](https://www.conduktor.io/assets/images/glossary/real-time-ml-inference-with-streaming-data-0.webp) This article explores how streaming platforms enable low-latency ML inference, the architectural patterns that make it possible, and the practical considerations for implementation. ## Understanding Real-Time ML Inference Real-time ML inference means generating predictions as data arrives, rather than waiting to accumulate batches. When a credit card transaction occurs, a fraud detection system must evaluate it instantly, typically within 50-100 milliseconds. When a user clicks on a product, a recommendation engine should update suggestions immediately, usually within 200-500 milliseconds. The key distinction is latency. Batch inference might process millions of records overnight with latency measured in hours. Real-time inference handles individual events with latency targets ranging from sub-50ms (fraud detection, high-frequency trading) to a few hundred milliseconds (personalized recommendations, dynamic pricing). This shift requires fundamentally different architecture and infrastructure. For foundational understanding of streaming architectures, see [What is Real-Time Data Streaming](https://www.conduktor.io/glossary/what-is-real-time-data-streaming). Model inference is separate from model training. Training happens periodically using historical data and significant compute resources. Inference uses the trained model to make predictions on new data with minimal computational overhead. In streaming contexts, inference must be lightweight, fast, and highly available. ## Batch vs. Streaming Inference Approaches Traditional batch inference reads data from databases or data lakes, applies models to large datasets, and writes predictions back to storage. This works well for non-time-sensitive use cases like monthly churn prediction or daily inventory forecasting. Streaming inference processes events as they flow through a message broker like Apache Kafka. Each event triggers model evaluation, and predictions are published back to the stream for downstream consumers. This approach reduces latency from hours to milliseconds but introduces complexity in state management, fault tolerance, and scalability. The choice between batch and streaming depends on business requirements. If predictions guide immediate actions, blocking fraudulent transactions, serving personalized content, adjusting prices dynamically, streaming inference is essential. If predictions inform strategic decisions with longer time horizons, batch processing may suffice. ## Streaming Architecture for ML Inference Apache Kafka is the backbone for streaming ML inference. Input events (transactions, clicks, sensor readings) flow into Kafka topics. Inference services consume these events, apply ML models, and publish predictions to output topics. This decoupled architecture allows independent scaling of data ingestion, model serving, and downstream applications. Modern Kafka deployments (2025) use KRaft mode for simplified operations and improved performance, see [Understanding KRaft Mode in Kafka](https://www.conduktor.io/glossary/understanding-kraft-mode-in-kafka) for details on the latest architecture. For comprehensive coverage of Apache Kafka fundamentals, see [Apache Kafka](https://www.conduktor.io/glossary/apache-kafka). Apache Flink excels at stateful stream processing, which is crucial for feature engineering. Many ML models require aggregated features, a user's average purchase amount over 30 days, or the standard deviation of sensor readings in the last hour. Flink can compute these features in real-time using windowing and state management, joining streaming data with reference tables to create complete feature vectors for inference. For deep dive into Flink's capabilities, see [What is Apache Flink: Stateful Stream Processing](https://www.conduktor.io/glossary/what-is-apache-flink-stateful-stream-processing). Feature enrichment patterns are covered extensively in [Stream Joins and Enrichment Patterns](https://www.conduktor.io/glossary/stream-joins-and-enrichment-patterns). Feature stores bridge the gap between offline training and online inference. Systems like Feast, Tecton, Feathr, Featureform, and Hopsworks store pre-computed features that can be retrieved with low latency during inference. Features calculated during training must match those available at inference time, a challenge known as **train-serve skew**. For example, if you train a fraud model using the last 90 days of transaction history but only compute 30-day aggregates at inference time, your predictions will be inconsistent and less accurate. Feature stores help enforce consistency by serving the exact same feature definitions for both training and inference. Data quality matters enormously in streaming contexts. Schema validation ensures events match expected formats before inference. Tools like Conduktor can validate schemas, monitor data quality, and alert on anomalies in Kafka streams, preventing corrupted data from reaching ML models and producing unreliable predictions. For schema management best practices, see [Schema Registry and Schema Management](https://www.conduktor.io/glossary/schema-registry-and-schema-management). To understand the relationship between data quality and observability, see [Data Quality vs Data Observability: Key Differences](https://www.conduktor.io/glossary/data-quality-vs-data-observability-key-differences). ## Implementation Patterns and Frameworks There are three primary patterns for deploying ML models in streaming environments: - **Embedded models**: The model is deployed directly within the stream processing application (Flink job, Kafka Streams app). This minimizes network latency but couples model updates to application deployments. It works well for lightweight models like decision trees or linear regressions. - **Model serving platforms**: Dedicated inference services host models behind APIs. Stream processors call these services over HTTP or gRPC. This separates model lifecycle management from stream processing but adds network overhead. It's ideal for complex models like deep neural networks that require specialized hardware (GPUs). Popular 2025 serving platforms include: - **NVIDIA Triton Inference Server**: Multi-framework support (TensorFlow, PyTorch, ONNX) with GPU optimization - **Ray Serve**: Python-native serving with horizontal scaling and composition - **BentoML**: ML model serving with built-in deployment automation - **TensorFlow Serving**: TensorFlow-specific serving with production-grade features - **Seldon Core/KServe**: Kubernetes-native ML deployment with advanced features - **vLLM**: High-performance LLM inference engine optimized for throughput For GPU-intensive models, consider GPU scheduling strategies to maximize utilization. Batch multiple inference requests together for efficiency, or use dynamic batching features available in Triton and vLLM. - **Stream-native ML frameworks**: Libraries like Apache Flink ML or River enable training and inference within stream processing frameworks. This approach is emerging for online learning scenarios where models continuously adapt to new data. - **Model Optimization for Inference** Before deploying models in production, optimize them for latency and throughput: - **Quantization**: Reduce model precision (FP32 → FP16/INT8) for 2-4x speedup with minimal accuracy loss - **ONNX Runtime**: Convert models to ONNX format for cross-framework optimization - **TensorRT**: NVIDIA's high-performance deep learning inference optimizer - **Model distillation**: Train smaller models that mimic larger ones - **Edge inference**: Deploy lightweight models directly on IoT devices for sub-10ms latency ### Practical Implementation Example Here's a Flink job that consumes transaction events from Kafka, enriches them with features, calls a fraud detection model API, and publishes predictions: ```java StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // Consume from Kafka FlinkKafkaConsumer consumer = new FlinkKafkaConsumer<>( "transactions", new TransactionDeserializationSchema(), kafkaProps ); DataStream transactions = env.addSource(consumer); // Enrich with features from state or external lookups DataStream enrichedTransactions = transactions .keyBy(Transaction::getUserId) .map(new FeatureEnrichmentFunction()); // Call model serving endpoint DataStream predictions = AsyncDataStream.unorderedWait( enrichedTransactions, new AsyncModelInferenceFunction("http://model-service:8080/predict"), 1000, // timeout in milliseconds TimeUnit.MILLISECONDS, 100 // max concurrent requests ); // Publish predictions back to Kafka FlinkKafkaProducer producer = new FlinkKafkaProducer<>( "predictions", new PredictionSerializationSchema(), kafkaProps ); predictions.addSink(producer); env.execute("Real-Time Fraud Detection"); ``` ```java // Async inference function with error handling public class AsyncModelInferenceFunction extends RichAsyncFunction { private transient HttpClient httpClient; private final String modelEndpoint; @Override public void open(Configuration parameters) { // Initialize HTTP client with connection pooling httpClient = HttpClient.newBuilder() .connectTimeout(Duration.ofMillis(500)) .build(); } @Override public void asyncInvoke(FeatureVector features, ResultFuture resultFuture) { CompletableFuture.supplyAsync(() -> { try { HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(modelEndpoint)) .header("Content-Type", "application/json") .POST(HttpRequest.BodyPublishers.ofString(features.toJson())) .build(); HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); return new Prediction(features.getTransactionId(), parseResponse(response.body()), System.currentTimeMillis()); } catch (Exception e) { // Circuit breaker pattern: after N failures, stop calling service temporarily // Return default prediction or route to dead letter queue return new Prediction(features.getTransactionId(), 0.0, System.currentTimeMillis(), true); } }).thenAccept(prediction -> resultFuture.complete(Collections.singleton(prediction))); } } ``` Error handling is critical, timeouts, retries, and **circuit breakers** (which temporarily stop calling failing services after a threshold) prevent cascading failures when model services are slow or unavailable. For testing streaming resilience, see [Chaos Engineering for Streaming Systems](https://www.conduktor.io/glossary/chaos-engineering-for-streaming-systems). ## Challenges and Considerations - **Latency requirements** drive every architectural decision. Sub-100ms latency demands co-located compute and optimized model formats. Second-scale latency permits more complex features and models. Measuring end-to-end latency, from event arrival to prediction availability, requires comprehensive observability. - **Model versioning and deployment** become complex in production. Multiple model versions may run simultaneously for A/B testing or gradual rollouts. Prediction results should include model version metadata for debugging and analysis. **Canary deployments** (where new models receive 5-10% of traffic initially before full rollout) test models in production with limited risk. Include model version, timestamp, and confidence scores in prediction payloads: ```json { "transaction_id": "txn_12345", "fraud_probability": 0.87, "model_version": "fraud-detector-v2.3.1", "inference_timestamp_ms": 1735689600000, "features_used": ["amount", "merchant_category", "30d_avg_amount"] } ``` - **Feature engineering** in real-time is challenging. Some features require historical context (moving averages, trend calculations) that must be computed incrementally. Others need joins with slowly-changing dimension tables. Flink's stateful operators and exactly-once processing guarantees make this feasible but require careful design. - **Monitoring and debugging** streaming ML pipelines requires visibility into multiple layers: data quality, feature accuracy, model performance, and system health. Essential metrics include: - **Latency metrics**: P50, P95, P99 inference latency (track outliers that impact user experience) - **Throughput**: Predictions per second, request queue depth - **Error rates**: Timeouts, model failures, malformed input - **Model performance**: Prediction distribution, confidence score distribution, accuracy drift - **Model staleness**: Time since last model update or training Tools for observability: - **Prometheus + Grafana**: Metrics collection and visualization - **Kafka Lag Exporter**: Monitor consumer lag to detect processing delays - **Conduktor**: Kafka-specific monitoring, schema validation, data quality checks - **MLflow**: Track model versions, performance metrics, and experiment history - **Evidently AI**: Monitor ML model quality and data drift in production When predictions degrade, teams need tools to trace issues through the entire pipeline, from input data quality to model behavior. For comprehensive guidance on building data quality frameworks, see [Building a Data Quality Framework](https://www.conduktor.io/glossary/building-a-data-quality-framework). ## Real-World Use Cases - **Fraud detection** exemplifies real-time ML inference. When a payment transaction occurs, features are extracted (transaction amount, merchant category, user location, recent spending patterns). These features feed a model that outputs a fraud probability within milliseconds. High-risk transactions trigger additional verification or blocking before completion. Companies like PayPal and Stripe process millions of such predictions per second. For detailed architecture patterns, see [Real-Time Fraud Detection with Streaming](https://www.conduktor.io/glossary/real-time-fraud-detection-with-streaming). - **Personalized recommendations** use real-time inference to adapt to user behavior. As users browse products or content, their actions stream into Kafka. Models consume this activity to update recommendations dynamically. Netflix and YouTube continuously refine what content to suggest based on viewing patterns, with models updating predictions as users interact with the platform. For implementation guidance, see [Building Recommendation Systems with Streaming Data](https://www.conduktor.io/glossary/building-recommendation-systems-with-streaming-data). - **Predictive maintenance** in manufacturing monitors sensor data from equipment. Vibration, temperature, and pressure readings stream from industrial IoT devices. ML models detect anomalies or predict failures before they occur, triggering maintenance workflows. This reduces unplanned downtime and extends equipment life. Edge inference is particularly valuable here, deploying lightweight models directly on IoT gateways enables sub-10ms latency even when network connectivity is unreliable. ## Summary Real-time ML inference with streaming data enables organizations to act on insights immediately rather than hours or days later. By integrating ML models with streaming platforms like Apache Kafka and Flink, systems can generate predictions with millisecond latency at massive scale. Key architectural decisions include choosing between embedded models and dedicated serving platforms (like NVIDIA Triton, Ray Serve, or BentoML), managing feature engineering in stream processors, optimizing models for production (quantization, ONNX, TensorRT), and ensuring robust monitoring throughout the pipeline. Data quality validation, schema management, and comprehensive observability are critical for reliable production systems. Modern 2025 best practices emphasize: - Using KRaft-mode Kafka for simplified operations - Implementing circuit breakers and proper error handling - Tracking detailed inference metrics (P95/P99 latency, model staleness) - Including model version metadata in all predictions - Testing resilience through chaos engineering - Optimizing models before deployment (quantization, edge inference) While implementing streaming ML inference introduces complexity in state management, fault tolerance, and monitoring, the business value, preventing fraud in real-time, personalizing user experiences instantly, or predicting equipment failures before they occur, often justifies the investment. As streaming platforms mature and ML frameworks improve, real-time inference is becoming accessible to more organizations. ## Related Concepts - [Streaming Data Pipeline](https://www.conduktor.io/streaming-data-pipeline) - Build robust pipelines for real-time ML inference - [Schema Registry and Schema Management](https://www.conduktor.io/schema-registry-and-schema-management) - Ensure data consistency for ML models - [Data Quality Dimensions: Accuracy, Completeness, and Consistency](https://www.conduktor.io/data-quality-dimensions-accuracy-completeness-and-consistency) - Quality requirements for ML inputs ## Sources and References - [Apache Kafka Documentation - Machine Learning Use Cases](https://kafka.apache.org/documentation/) - Comprehensive guide to using Kafka as the backbone for ML inference pipelines - [Apache Flink ML Library](https://nightlies.apache.org/flink/flink-ml-docs-stable/) - Documentation on Flink's machine learning capabilities for stream processing - [NVIDIA Triton Inference Server Documentation](https://docs.nvidia.com/deeplearning/triton-inference-server/) - Multi-framework inference serving with GPU optimization (2025 standard) - [Ray Serve Documentation](https://docs.ray.io/en/latest/serve/) - Python-native scalable model serving framework - [BentoML Documentation](https://docs.bentoml.org/) - Modern ML model serving with deployment automation - [vLLM: Fast and Easy LLM Inference](https://github.com/vllm-project/vllm) - High-performance LLM inference engine for 2025 generative AI use cases - [Uber's Michelangelo - Machine Learning Platform](https://www.uber.com/blog/michelangelo-machine-learning-platform/) - Real-world case study of building production ML systems with streaming inference - [Feature Stores for ML - Feast Documentation](https://feast.dev/) - Open-source feature store for managing online and offline features - [ONNX Runtime Performance Optimization](https://onnxruntime.ai/) - Cross-framework model optimization for production inference - [Evidently AI - ML Monitoring](https://www.evidentlyai.com/) - Monitor ML model quality and data drift in production (2025) Source: https://www.conduktor.io/glossary/real-time-ml-inference-with-streaming-data --- # Real-Time ML Pipelines: Machine Learning on Streaming Data Traditional machine learning systems operate in batch mode: data is collected, processed overnight or periodically, models are trained on historical data, and predictions are generated in large batches. While this approach works for many scenarios, it fundamentally limits how quickly systems can respond to changing conditions. **Real-time ML pipelines** process streaming data to deliver predictions with millisecond to sub-second latency. Unlike batch systems that might update every 24 hours, real-time ML continuously ingests events, computes features on-the-fly, and serves predictions immediately. Consider [fraud detection](https://www.conduktor.io/real-time-fraud-detection-with-streaming): by the time a batch system identifies suspicious patterns from yesterday's data, fraudulent transactions have already cleared. Real-time ML evaluates each transaction as it occurs, blocking fraud before money moves. ### Real-Time Inference vs Online Learning Two patterns are often conflated under "real-time ML": - **Real-time inference**: Models trained offline (often in batch) serve predictions on streaming data with low latency - **Online learning**: Models continuously update their parameters as new data arrives, adapting without full retraining Most production systems use [real-time inference](https://www.conduktor.io/real-time-ml-inference-with-streaming-data) with periodic model updates. True online learning remains challenging due to stability concerns and infrastructure complexity. ## Architecture Components A complete real-time ML pipeline consists of several interconnected layers: ![A complete real-time ML pipeline consists of several interconnected layers](https://www.conduktor.io/assets/images/glossary/real-time-ml-pipelines-0.webp) ### Feature Engineering Layer Raw events from Kafka, Kinesis, or other [streaming platforms](https://www.conduktor.io/what-is-real-time-data-streaming) rarely match the feature vectors models expect. The feature engineering layer transforms streaming events into ML-ready features through: - **Windowed aggregations**: "user's transaction count in last 1 hour" - **Joins**: enriching events with user profiles, product catalogs (see [stream joins and enrichment patterns](https://www.conduktor.io/stream-joins-and-enrichment-patterns)) - **Derived metrics**: ratios, percentiles, z-scores computed in real-time [Apache Flink](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing), Spark Structured Streaming, and Kafka Streams excel at these transformations, maintaining stateful computations across event streams. ### Feature Stores [Feature stores](https://www.conduktor.io/feature-stores-for-machine-learning) solve the dual-access pattern problem: training needs historical features (batch access), while serving requires latest features (real-time lookup). - **Offline store**: Batch-accessible feature history for training (often Parquet/Delta Lake/Iceberg on S3) - **Online store**: Low-latency key-value lookups for serving (Redis, DynamoDB, Cassandra, ScyllaDB) Streaming feature pipelines write to both stores, ensuring training-serving consistency. Tools like Feast, Tecton, Databricks Feature Store, Hopsworks, and Feathr orchestrate this dual-write pattern. **Modern feature store ecosystem (2025)**: - **Feast 0.38+**: Most popular open-source option with improved streaming support and performance - **Hopsworks**: Enterprise feature store with built-in feature monitoring and versioning - **Feathr**: LinkedIn's open-source feature store optimized for large-scale feature engineering - **Tecton**: Managed feature platform with advanced real-time capabilities ### Model Serving Infrastructure Once features are ready, models must generate predictions within strict latency budgets. Serving patterns include: - **REST/gRPC endpoints**: Synchronous request-response (TensorFlow Serving, Seldon) - **Stream processing**: Predictions written to output topics (embedded models in Flink/Kafka Streams) - **Sidecar containers**: Models deployed alongside application services ### Feedback Loops Real-time ML systems must capture prediction outcomes to detect [model drift](https://www.conduktor.io/model-drift-in-streaming) and retrain: ![Real-time ML systems must capture prediction outcomes to detect model drift and retrain](https://www.conduktor.io/assets/images/glossary/real-time-ml-pipelines-1.webp) Outcomes (was the prediction correct?) feed back into training pipelines, creating continuous improvement cycles. See [data drift in streaming](https://www.conduktor.io/data-drift-in-streaming) for detection strategies. ## Real-Time Feature Engineering Feature engineering accounts for 70-80% of real-time ML pipeline complexity. Streaming frameworks cover common patterns: ### Windowed Aggregations Time windows aggregate streaming events into features. Window types include: - **Tumbling**: Fixed, non-overlapping intervals (e.g., 1-hour buckets: 1pm-2pm, 2pm-3pm). Use for daily summaries, hourly statistics. Example: "total transactions per hour." - **Sliding**: Overlapping intervals (e.g., last 1 hour, updated every 5 minutes). Use for moving averages, recent trends. Example: "average purchase amount in last 60 minutes, updated continuously." - **Session**: Dynamic windows based on activity gaps (e.g., group clicks separated by < 30 minutes). Use for user sessions, activity bursts. Example: "session duration" for recommendation features. ### Sessionization User sessions (sequences of activity separated by inactivity) are powerful features that capture user intent and engagement patterns critical for recommendations and personalization. ### Feature Freshness Guarantees Real-time features have expiration semantics. A "user's 1-hour transaction count" computed at 2:00 PM becomes stale by 3:01 PM. Feature stores track feature **freshness**: - **Timestamp**: When the feature was computed - **TTL**: How long the feature remains valid - **Watermarks**: Latest event time processed Serving layers must handle feature staleness, either by rejecting predictions or falling back to older feature versions. ### Preventing Training-Serving Skew **Training-serving skew** occurs when features computed differently in training vs serving: - Training uses Spark batch aggregations - Serving uses Flink streaming aggregations - Subtle logic differences cause distribution shift This is one of the most critical challenges in production ML. [Feature stores](https://www.conduktor.io/feature-stores-for-machine-learning) are specifically designed to prevent this issue. **Prevention strategies**: 1. **Single codebase**: Same transformation logic for batch and streaming 2. **Backfill validation**: Run streaming pipeline on historical data, compare to batch features 3. **Feature store contracts**: [Schema enforcement](https://www.conduktor.io/schema-registry-and-schema-management) ensures consistent feature definitions ## Model Serving Patterns ### Embedded Models Models run inside stream processing applications (Flink, Kafka Streams). - **Pros**: No network latency, batch predictions - **Cons**: Tight coupling, difficult to update models independently ### Sidecar Pattern Models deploy as sidecar containers alongside application pods. - **Pros**: Language independence, separate scaling - **Cons**: Local communication overhead, resource contention ### Dedicated Model Servers Centralized serving infrastructure (TensorFlow Serving, Seldon Core, KServe, Ray Serve). - **Pros**: Independent scaling, specialized hardware (GPUs), A/B testing, model versioning - **Cons**: Network latency, additional infrastructure - **2025 Ecosystem**: - **KServe** (formerly KFServe): The industry standard for Kubernetes-native model serving with autoscaling, canary deployments, and multi-framework support - **Ray Serve**: Scalable model serving built on Ray, excellent for complex inference workflows and distributed Python applications - **BentoML**: Framework-agnostic model serving with built-in monitoring and adaptive batching ### Latency and Throughput Trade-offs | Pattern | Latency (p99) | Throughput | Use Case | |---------|---------------|------------|----------| | Embedded | <5ms | 100K+ req/s | Ultra-low latency | | Sidecar | 5-20ms | 50K req/s | Moderate latency | | Dedicated | 20-100ms | 10K+ req/s | Complex models, GPUs | Choose based on your latency SLA and model complexity. ## Online Learning vs Real-Time Inference ### Real-Time Inference (Common) Models trained offline periodically, served in real-time: 1. Collect data in feature store (continuous) 2. Train model on historical data (daily/weekly) 3. Deploy new model version 4. Serve predictions on streaming data **Advantages**: Stable models, rigorous validation, mature tooling ### Online Learning (Advanced) Models update continuously as data arrives through incremental learning, a technique where the model updates its parameters with each new batch of data without full retraining. For example, an online linear regression model can update its coefficients after seeing each new data point, adapting to changing patterns in real-time. **Modern online learning frameworks (2025)**: - **River**: Python library designed specifically for online machine learning, supporting incremental versions of popular algorithms (linear models, trees, neural networks) - **Vowpal Wabbit**: High-performance online learning system for large-scale applications - **PyTorch Lightning with streaming datasets**: Enables continuous model updates with proper checkpointing **Challenges**: - **Catastrophic forgetting**: New data erases old knowledge. Example: A recommendation model trained on summer fashion trends may "forget" winter preferences when fall data arrives. Solutions include experience replay (storing representative samples from past data) and regularization techniques. - **Concept drift detection**: When to trust new patterns vs ignore noise. If your fraud model suddenly sees unusual transaction patterns, is this emerging fraud or a temporary anomaly? Statistical tests (KS test, drift scores) help distinguish signal from noise. - **Validation**: How to evaluate continuously updating models. Traditional holdout sets become stale. Approaches include prequential evaluation (test-then-train on each sample) and time-based validation windows. **Use cases**: Ad click prediction, content ranking where patterns shift rapidly, real-time recommendations, dynamic pricing ### A/B Testing and Shadow Deployment Before fully deploying new models: - **Shadow mode**: New model scores traffic but doesn't serve predictions. Both models run in parallel, but only the primary model's predictions are used. This allows comparison before switching. Track metrics like prediction distribution, latency, and feature importance differences. If shadow model predictions differ significantly (e.g., fraud scores vary by >10%), investigate before promotion. - **A/B testing**: Split traffic between model versions, typically starting with a small percentage for the new model (5-10%) and gradually increasing based on performance metrics. Monitor business KPIs (conversion rate, fraud catch rate) and technical metrics (latency, error rate). Example: Route 10% of users to new recommendation model for 7 days, compare click-through rate against control group before full rollout. ## Production Considerations ### Latency Requirements Real-time ML systems must meet strict SLAs: - **p50 latency**: Typical case performance - **p99 latency**: 99th percentile, catches tail latencies - **p999 latency**: Extreme cases, often determines user experience Example targets: - Fraud detection: p99 < 50ms - Recommendations: p99 < 200ms - Search ranking: p99 < 100ms **Latency budget breakdown**: ![**Latency budget breakdown**](https://www.conduktor.io/assets/images/glossary/real-time-ml-pipelines-2.webp) Instrument each component to identify bottlenecks. ### Feature Quality Monitoring Features can degrade silently: - **Upstream data quality**: Source events missing fields, [schema changes](https://www.conduktor.io/schema-evolution-best-practices) - **Computation errors**: Window aggregations incorrect due to late data - **Staleness**: Feature updates delayed due to infrastructure issues **Monitoring strategies**: Validate feature ranges, check for null values, track staleness, and log feature distributions to alert on drift from expected ranges. See [building a data quality framework](https://www.conduktor.io/building-a-data-quality-framework) for comprehensive quality strategies. ### Model Lineage and Auditing In regulated industries (finance, healthcare), you must explain predictions: **Model lineage** tracks: - Training data version and time range - Feature definitions and versions - Hyperparameters and training code commit - Evaluation metrics pre-deployment **Prediction auditing** logs: - Input features used - Model version that generated prediction - Prediction value and confidence - Outcome (if available) This enables reproducing predictions and diagnosing errors months later. ### Data Governance for ML Pipelines As real-time ML systems scale, data governance becomes critical. **Conduktor** provides comprehensive governance capabilities specifically designed for streaming ML pipelines: - **Schema validation**: Ensure streaming events match feature expectations before they enter ML pipelines. Conduktor's [Schema Registry integration](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry) prevents malformed events from corrupting feature computations. - **Data quality gates**: Block corrupt data before it reaches feature pipelines through Conduktor's [data quality module](https://docs.conduktor.io/guide/use-cases/observe-data-quality), which validates ranges, patterns, and business rules on streaming data. - **Audit logs**: Conduktor Gateway captures all producer/consumer connections, giving teams visibility into which applications interact with which topics across ML pipelines. - **Access controls**: Restrict who can modify feature definitions or deploy models through [topic-level ACLs](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) and governance policies. - **Audit logs**: Track all changes to feature pipelines and model deployments, essential for regulated industries. For example, Conduktor can enforce that all events in the `transactions` topic include required fields (`user_id`, `amount`, `timestamp`) with valid data types and ranges before feature engineering begins, preventing silent failures in downstream ML pipelines. This governance layer acts as quality gates between data producers and ML consumers. ## Use Cases and Implementation ### Fraud Detection Systems **Real-time requirements**: Evaluate transactions before authorization (50-100ms) For comprehensive coverage, see [real-time fraud detection with streaming](https://www.conduktor.io/real-time-fraud-detection-with-streaming). ![real-time-ml-pipelines diagram 4](https://www.conduktor.io/assets/images/glossary/real-time-ml-pipelines-3.webp) **Features**: - Velocity: Transaction count in last 1h/24h/7d - Behavioral: Distance from user's typical transaction amount/merchant - Contextual: Device fingerprint, IP geolocation ### Real-Time Recommendations **Real-time requirements**: Personalize content as users browse (100-300ms) For detailed implementation patterns, see [building recommendation systems with streaming data](https://www.conduktor.io/building-recommendation-systems-with-streaming-data). **Features**: - User history: Recently viewed items, categories - Session context: Current session duration, items viewed - Popularity: Trending items in last 1h Recommendations are computed by aggregating session data in real-time, combining it with user profiles, and calling the recommendation model to serve personalized top-K items. ### Dynamic Pricing - **Real-time requirements**: Adjust prices based on demand signals (seconds to minutes) - **Features**: - Demand indicators: Search volume, cart adds in last 15min - Supply: Inventory levels, competitor pricing - External: Time of day, seasonality, events Pricing models consume demand signals (search volume, cart adds) and supply signals (inventory, competitor prices) to compute optimal prices in real-time, publishing updates back to the application database. ### Infrastructure Stack Typical real-time ML infrastructure: **Data Layer**: - **Streaming**: [Apache Kafka](https://www.conduktor.io/apache-kafka) (4.0+ with [KRaft](https://www.conduktor.io/understanding-kraft-mode-in-kafka)), Amazon Kinesis, Redpanda - **Processing**: [Apache Flink](https://www.conduktor.io/what-is-apache-flink-stateful-stream-processing) (1.18+), Spark Structured Streaming (3.5+), Kafka Streams - **Storage**: S3/Delta Lake/[Apache Iceberg](https://www.conduktor.io/apache-iceberg) (offline), Redis/DynamoDB/ScyllaDB (online) **ML Layer**: - **Feature Store**: Feast (0.38+), Hopsworks, Feathr, Databricks Feature Store - **Training**: Spark MLlib, scikit-learn, TensorFlow/PyTorch, XGBoost - **Serving**: KServe, Ray Serve, TensorFlow Serving, Seldon Core, BentoML - **Online Learning**: River, Vowpal Wabbit, PyTorch Lightning **Orchestration & MLOps (2025)**: - **ML Orchestration**: Flyte, Metaflow, Hamilton (declarative dataflow) - **Distributed Compute**: Ray (distributed Python), Dask - **Experiment Tracking**: MLflow 2.x, Weights & Biases, Neptune - **Model Registry**: MLflow Model Registry, BentoML **Observabilit