Insights and trends in data streaming, Apache Kafka, and platform engineering https://conduktor.io/blog en-us Thu, 16 Apr 2026 19:24:59 GMT 11 Kafka behaviors we found helping customers, then validated in the source code. Silent safety bypasses, configs that lie, compression traps. https://conduktor.io/blog/11-kafka-gotchas-production-surprises https://conduktor.io/blog/11-kafka-gotchas-production-surprises Mon, 30 Mar 2026 00:00:00 GMT stephane-derosiaux Schema registry governance gets deferred because the tooling isn't there. We built Schema Registry Proxy to change that. https://conduktor.io/blog/introducing-schema-registry-proxy https://conduktor.io/blog/introducing-schema-registry-proxy Tue, 24 Mar 2026 00:00:00 GMT nicole-bouchard Install Kafka, configure governance, and onboard developers with one AI conversation. Open-source Conduktor skill for Claude Code, Cursor, and 30+ tools. https://conduktor.io/blog/set-up-a-kafka-platform-with-an-ai-agent https://conduktor.io/blog/set-up-a-kafka-platform-with-an-ai-agent Fri, 20 Mar 2026 00:00:00 GMT ron-kapoor Map chaos experiments to DORA, SOC 2, PCI-DSS, and GDPR requirements. Turn your Kafka resilience testing into the compliance evidence regulators expect. https://conduktor.io/blog/chaos-testing-for-kafka-compliance-dr-evidence-that-works https://conduktor.io/blog/chaos-testing-for-kafka-compliance-dr-evidence-that-works Thu, 19 Mar 2026 00:00:00 GMT nicole-bouchard Monitoring tells you what broke. Chaos testing tells you what will. Five Kafka system behaviors that only surface under controlled failure. https://conduktor.io/blog/what-kafka-chaos-tests-actually-teach-you-that-monitoring-doesnt https://conduktor.io/blog/what-kafka-chaos-tests-actually-teach-you-that-monitoring-doesnt Thu, 12 Mar 2026 00:00:00 GMT nicole-bouchard One corrupt Parquet file froze a data lake for 6 hours. How Conduktor Gateway prevents bad data from reaching S3 with CEL validation rules. https://conduktor.io/blog/building-a-data-lake-with-kafka-and-s3 https://conduktor.io/blog/building-a-data-lake-with-kafka-and-s3 Wed, 11 Mar 2026 00:00:00 GMT stephane-derosiaux Five tests that validate your Kafka disaster recovery plan before a real outage does. A practical guide for platform engineering teams. https://conduktor.io/blog/chaos-engineering-for-kafka-testing-recovery-before-you-need-it https://conduktor.io/blog/chaos-engineering-for-kafka-testing-recovery-before-you-need-it Wed, 04 Mar 2026 00:00:00 GMT nicole-bouchard Gateway eliminates Kafka DR coordination overhead. One API call switches all clients to the secondary cluster. Cut execution from hours to under a minute. https://conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes https://conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes Mon, 02 Mar 2026 00:00:00 GMT stephane-derosiaux An open-source Kafka Connect connector for WebSocket streams. Auto-reconnect, auth support, and configurable buffering. Get started in 5 minutes. https://conduktor.io/blog/kafka-connect-websocket-source-connector https://conduktor.io/blog/kafka-connect-websocket-source-connector Tue, 24 Feb 2026 00:00:00 GMT stephane-derosiaux Your Kafka investment is hiding costs, unused capacity, and optimization opportunities that no dashboard can surface. AI changes what's possible to ask. https://conduktor.io/blog/your-kafka-investment-is-hiding-costs-and-opportunities https://conduktor.io/blog/your-kafka-investment-is-hiding-costs-and-opportunities Fri, 20 Feb 2026 00:00:00 GMT andy-allison No more juggling 7 CLI tools. A walkthrough of debugging a Kafka consumer lag incident with Conduktor Console. https://conduktor.io/blog/what-debugging-kafka-should-actually-look-like https://conduktor.io/blog/what-debugging-kafka-should-actually-look-like Tue, 17 Feb 2026 00:00:00 GMT ron-kapoor Three operational phases, six technical areas, and a failover runbook. The complete disaster recovery strategy most Kafka teams are missing. https://conduktor.io/blog/kafka-disaster-recovery-the-complete-strategy-beyond-replication https://conduktor.io/blog/kafka-disaster-recovery-the-complete-strategy-beyond-replication Wed, 11 Feb 2026 00:00:00 GMT nicole-bouchard Enterprise Kafka management needs automated governance, not slow approval processes. Control multi-cluster environments without creating bottlenecks. https://conduktor.io/blog/enterprise-kafka-management https://conduktor.io/blog/enterprise-kafka-management Mon, 09 Feb 2026 00:00:00 GMT stephane-derosiaux AI for Kafka operations makes context instantly available. Reduce MTTR from hours to minutes with natural-language diagnostics and MCP. https://conduktor.io/blog/ai-kafka-operations https://conduktor.io/blog/ai-kafka-operations Sat, 07 Feb 2026 00:00:00 GMT stephane-derosiaux You've accepted that debugging matters more than dashboards. Then you open a terminal and realize Kafka's native tools were never built for this. https://conduktor.io/blog/houston-we-have-7-cli-tools-and-zero-answers https://conduktor.io/blog/houston-we-have-7-cli-tools-and-zero-answers Fri, 06 Feb 2026 00:00:00 GMT ron-kapoor Stream traces, metrics, and logs into Kafka topics for durable storage, replay, and custom analytics. Open source, Apache License 2.0. https://conduktor.io/blog/kafka-connect-opentelemetry-source-connector https://conduktor.io/blog/kafka-connect-opentelemetry-source-connector Fri, 06 Feb 2026 00:00:00 GMT stephane-derosiaux Teams invest in Kafka observability thinking they're prepared. Then an incident hits, and they realize dashboards don't tell you which room is on fire. https://conduktor.io/blog/why-every-kafka-incident-ends-with-restart-it https://conduktor.io/blog/why-every-kafka-incident-ends-with-restart-it Wed, 04 Feb 2026 00:00:00 GMT ron-kapoor Kafka data products need contracts, ownership, SLAs, and discovery. Topics with documentation aren't data products—they're shared state. https://conduktor.io/blog/kafka-data-products https://conduktor.io/blog/kafka-data-products Tue, 03 Feb 2026 00:00:00 GMT stephane-derosiaux Producer timestamps can trigger immediate retention deletion. Configure timestamp validation, use LogAppendTime, or enforce bounds at the gateway level. https://conduktor.io/blog/safeguard-kafka-from-timestamp-based-data-loss https://conduktor.io/blog/safeguard-kafka-from-timestamp-based-data-loss Tue, 03 Feb 2026 00:00:00 GMT stephane-derosiaux Platform health differs from infrastructure health. Observability alone won't cut it. Here are the questions that reveal your Kafka platform's real state. https://conduktor.io/blog/why-kafka-platform-health-requires-more-than-monitoring https://conduktor.io/blog/why-kafka-platform-health-requires-more-than-monitoring Mon, 02 Feb 2026 00:00:00 GMT nicole-bouchard Kafka latency averages hide tail problems. Optimize p99, not p50, and measure end-to-end from producer commit to consumer processing. https://conduktor.io/blog/kafka-latency https://conduktor.io/blog/kafka-latency Fri, 30 Jan 2026 00:00:00 GMT stephane-derosiaux PagerDuty's 9-hour Kafka outage reveals the real DR problem - coordinating 47 services during crisis. Why replication alone won't save you. https://conduktor.io/blog/kafka-dr-why-replication-isnt-the-hard-part https://conduktor.io/blog/kafka-dr-why-replication-isnt-the-hard-part Tue, 27 Jan 2026 00:00:00 GMT stephane-derosiaux Kafka policies in Confluence are suggestions. CEL-based enforcement at the API layer prevents bad configs before they reach production. https://conduktor.io/blog/kafka-policy-enforcement https://conduktor.io/blog/kafka-policy-enforcement Wed, 21 Jan 2026 00:00:00 GMT stephane-derosiaux Stream data from gRPC server endpoints directly into Kafka topics with our open-source Kafka Connect source connector. Zero custom code, just configurat... https://conduktor.io/blog/kafka-connect-grpc-source-connector https://conduktor.io/blog/kafka-connect-grpc-source-connector Mon, 19 Jan 2026 00:00:00 GMT stephane-derosiaux Fix Kafka schema chaos with a 4-step governance framework: visibility, audit, accountability, enforcement. Used by Swiss Post across hundreds of topics. https://conduktor.io/blog/kafka-schema-governance-from-chaos-to-confidence https://conduktor.io/blog/kafka-schema-governance-from-chaos-to-confidence Wed, 14 Jan 2026 00:00:00 GMT nicole-bouchard GDPR, SOC2, HIPAA, and DORA compliance for Kafka infrastructure. Generate audit evidence in minutes with continuous automated controls. https://conduktor.io/blog/kafka-security-compliance https://conduktor.io/blog/kafka-security-compliance Wed, 14 Jan 2026 00:00:00 GMT stephane-derosiaux Your Kafka data plane is fast and resilient. Your control plane—provisioning, access, config—probably runs on scripts and tribal knowledge. https://conduktor.io/blog/kafka-control-plane https://conduktor.io/blog/kafka-control-plane Tue, 06 Jan 2026 00:00:00 GMT stephane-derosiaux Manual Kafka audit prep is security theater. Automate compliance evidence generation so audits take days, not months of scrambling. https://conduktor.io/blog/kafka-audit-automation https://conduktor.io/blog/kafka-audit-automation Tue, 30 Dec 2025 00:00:00 GMT stephane-derosiaux Data contracts prevent breaking changes in Kafka. Enforce schema compatibility, versioning, and migration rules before bad data ships. https://conduktor.io/blog/kafka-data-contracts https://conduktor.io/blog/kafka-data-contracts Tue, 23 Dec 2025 00:00:00 GMT stephane-derosiaux IBM's Confluent acquisition shows why vendor-neutral Kafka matters. Decouple apps from infra to cut costs and stay flexible. https://conduktor.io/blog/the-case-for-vendor-neutral-architecture https://conduktor.io/blog/the-case-for-vendor-neutral-architecture Thu, 18 Dec 2025 00:00:00 GMT andy-allison Kafka alert fatigue is a design problem, not a sensitivity problem. Build tiered alerts where 95% of pages result in action, not silence. https://conduktor.io/blog/kafka-alerting https://conduktor.io/blog/kafka-alerting Tue, 16 Dec 2025 00:00:00 GMT stephane-derosiaux Strategic analysis of IBM's Confluent acquisition. What changes for enterprise Kafka deployments and architecture decisions. https://conduktor.io/blog/ibm-confluent-acquisition-kafka-future https://conduktor.io/blog/ibm-confluent-acquisition-kafka-future Mon, 15 Dec 2025 00:00:00 GMT stephane-derosiaux IBM's acquisition of Confluent validates streaming as critical infrastructure. Here's what changes for organizations running Kafka. https://conduktor.io/blog/ibm-confluent-11-billion-says-streaming-isn-t-optional https://conduktor.io/blog/ibm-confluent-11-billion-says-streaming-isn-t-optional Wed, 10 Dec 2025 00:00:00 GMT ron-kapoor Zero-downtime Kafka rolling restarts. Pre-flight checks, ISR verification, controlled shutdown, and KRaft controller order. https://conduktor.io/blog/rolling-restarts-without-downtime https://conduktor.io/blog/rolling-restarts-without-downtime Wed, 10 Dec 2025 00:00:00 GMT stephane-derosiaux Teams overpay for Kafka 30-50% through over-provisioning and topic sprawl. Measure per-team usage and right-size clusters to cut costs. https://conduktor.io/blog/kafka-costs https://conduktor.io/blog/kafka-costs Tue, 09 Dec 2025 00:00:00 GMT stephane-derosiaux Choose the right Kafka testing strategy. Testcontainers for integration tests, MockProducer for unit tests, TopologyTestDriver for Streams. https://conduktor.io/blog/testing-kafka-testcontainers-embedded-mocks https://conduktor.io/blog/testing-kafka-testcontainers-embedded-mocks Tue, 09 Dec 2025 00:00:00 GMT stephane-derosiaux Stop alert fatigue from consumer lag metrics. Offset vs time-based lag, per-workload thresholds, and rate-of-change detection. https://conduktor.io/blog/kafka-consumer-lag-alerting-thresholds-guide https://conduktor.io/blog/kafka-consumer-lag-alerting-thresholds-guide Mon, 08 Dec 2025 00:00:00 GMT stephane-derosiaux Cross-team Kafka data sharing should be as simple as an API call. Discovery catalogs and approval workflows replace Slack-driven access requests. https://conduktor.io/blog/kafka-data-sharing https://conduktor.io/blog/kafka-data-sharing Tue, 02 Dec 2025 00:00:00 GMT stephane-derosiaux Enterprise AI fails due to data access bottlenecks, multi-hop pipelines, and late governance. SMBC, Marsh McLennan leaders share real-world solutions. https://conduktor.io/blog/the-hidden-architecture-problem-behind-enterprise-ai https://conduktor.io/blog/the-hidden-architecture-problem-behind-enterprise-ai Mon, 01 Dec 2025 00:00:00 GMT stephane-derosiaux Kafka encryption has three layers: TLS in transit, disk encryption at rest, end-to-end for payload protection. Match each to your threat model. https://conduktor.io/blog/kafka-encryption https://conduktor.io/blog/kafka-encryption Tue, 25 Nov 2025 00:00:00 GMT stephane-derosiaux Configure Kafka tiered storage with S3 to reduce storage costs 3-9x. Broker config, common errors, and performance tradeoffs. https://conduktor.io/blog/kafka-tiered-storage-infinite-retention https://conduktor.io/blog/kafka-tiered-storage-infinite-retention Fri, 21 Nov 2025 00:00:00 GMT stephane-derosiaux Kafka automation replaces ticket-based provisioning with policy-based self-service. Cut provisioning tickets 75% and deploy 4x faster. https://conduktor.io/blog/kafka-automation-platform https://conduktor.io/blog/kafka-automation-platform Tue, 18 Nov 2025 00:00:00 GMT stephane-derosiaux Multi-cloud Kafka across AWS, Azure, and GCP needs unified management. One control plane for visibility, policies, and operations everywhere. https://conduktor.io/blog/kafka-multi-cloud-management https://conduktor.io/blog/kafka-multi-cloud-management Sun, 09 Nov 2025 00:00:00 GMT stephane-derosiaux Over-partitioned Kafka topics waste memory, slow failovers, and complicate rebalancing. The actual formula for right-sizing partitions based on throughput. https://conduktor.io/blog/stop-over-partitioning-kafka https://conduktor.io/blog/stop-over-partitioning-kafka Sat, 08 Nov 2025 00:00:00 GMT stephane-derosiaux Most Kafka clusters push less than 1MB/s. The challenge isn't throughput—it's building the control plane that makes streaming usable at enterprise scale. https://conduktor.io/blog/the-new-era-of-streaming https://conduktor.io/blog/the-new-era-of-streaming Thu, 06 Nov 2025 00:00:00 GMT stephane-derosiaux Streaming proxies enable governed self-service and AI-ready data at scale. Why Kafka control planes from Confluent, Kong, and Conduktor are converging. https://conduktor.io/blog/the-great-confluence-of-streaming https://conduktor.io/blog/the-great-confluence-of-streaming Wed, 05 Nov 2025 00:00:00 GMT stephane-derosiaux Self-service Kafka without governance is chaos. Policy-based automation gives developers autonomy while enforcing organizational standards. https://conduktor.io/blog/governed-kafka-self-service https://conduktor.io/blog/governed-kafka-self-service Sun, 02 Nov 2025 00:00:00 GMT stephane-derosiaux Understand Kafka offset commits, auto-commit pitfalls, CommitFailedException, and offset reset behavior. Stop losing messages. https://conduktor.io/blog/kafka-offset-management-consumer-commit-guide https://conduktor.io/blog/kafka-offset-management-consumer-commit-guide Sat, 01 Nov 2025 00:00:00 GMT stephane-derosiaux Kafka ACL sprawl makes permissions unknowable. Application-based access with periodic reviews replaces manual ACL-per-service-account chaos. https://conduktor.io/blog/kafka-access-management https://conduktor.io/blog/kafka-access-management Tue, 28 Oct 2025 00:00:00 GMT stephane-derosiaux Current New Orleans 2025 highlights: OpenAI's StreamLink, Kafka protocol deep-dive, Notion's event stack, KIP-1163 diskless topics, and JPMorgan's strat... https://conduktor.io/blog/five-can-t-miss-sessions-at-current-new-orleans-2025 https://conduktor.io/blog/five-can-t-miss-sessions-at-current-new-orleans-2025 Mon, 27 Oct 2025 00:00:00 GMT stephane-derosiaux Conduktor joins AWS ISV Accelerate Program, enabling secure MSK deployments for SageMaker, Redshift—accelerating AI adoption at Swiss Post and beyond. https://conduktor.io/blog/conduktor-joins-aws-isv-accelerate-program https://conduktor.io/blog/conduktor-joins-aws-isv-accelerate-program Wed, 22 Oct 2025 00:00:00 GMT matthew-scott Kafka security goes beyond TLS encryption. Scalable access control, ACL management, and provable permissions for production environments. https://conduktor.io/blog/kafka-security https://conduktor.io/blog/kafka-security Tue, 21 Oct 2025 00:00:00 GMT stephane-derosiaux Kafka's exactly-once works within cluster boundaries. Idempotent producers, transactions, and the hard limits you need to understand. https://conduktor.io/blog/exactly-once-semantics-when-it-works https://conduktor.io/blog/exactly-once-semantics-when-it-works Wed, 15 Oct 2025 00:00:00 GMT stephane-derosiaux Kafka security best practices fail without enforcement. Automate TLS, ACLs, and encryption policies so compliance is built-in, not bolted-on. https://conduktor.io/blog/kafka-security-best-practices https://conduktor.io/blog/kafka-security-best-practices Tue, 14 Oct 2025 00:00:00 GMT stephane-derosiaux Central teams set guardrails; domain teams self-serve within them. Federation delivers velocity, safety, and scale. https://conduktor.io/blog/federated-governance-solve-the-speed-vs-control-dilemma https://conduktor.io/blog/federated-governance-solve-the-speed-vs-control-dilemma Fri, 10 Oct 2025 00:00:00 GMT william-to Kafka proxies enforce field-level filtering, data masking, and rate limiting that native ACLs can't. Apply governance at the wire level. https://conduktor.io/blog/kafka-proxy https://conduktor.io/blog/kafka-proxy Sun, 05 Oct 2025 00:00:00 GMT stephane-derosiaux Kafka governance fails when policies live in wikis. Automated guardrails enforce naming, retention, and schema rules before incidents reach production. https://conduktor.io/blog/kafka-governance https://conduktor.io/blog/kafka-governance Sat, 27 Sep 2025 00:00:00 GMT stephane-derosiaux Kafka self-service with guardrails: Conduktor Scale enables developer autonomy while preventing data chaos, cost overruns, and compliance risks. https://conduktor.io/blog/who-needs-self-service-you-do https://conduktor.io/blog/who-needs-self-service-you-do Thu, 25 Sep 2025 00:00:00 GMT matt-searle Managing 10+ Kafka clusters via SSH and scripts wastes senior engineering time. Centralized tooling gives visibility and control at scale. https://conduktor.io/blog/kafka-cluster-management https://conduktor.io/blog/kafka-cluster-management Fri, 19 Sep 2025 00:00:00 GMT stephane-derosiaux Stop drowning in 200+ Kafka broker metrics. Focus on the 10 signals that actually predict outages, consumer lag, and cluster instability. https://conduktor.io/blog/apache-kafka-monitoring https://conduktor.io/blog/apache-kafka-monitoring Fri, 12 Sep 2025 00:00:00 GMT stephane-derosiaux Use Kafka compacted topics as a lightweight state store. Log compaction configuration, query limitations, and when to choose a real database instead. https://conduktor.io/blog/kafka-compacted-topics-as-database https://conduktor.io/blog/kafka-compacted-topics-as-database Wed, 10 Sep 2025 00:00:00 GMT stephane-derosiaux Kafka monitoring shows what's broken. Observability shows why. Combine metrics, traces, and context to diagnose root causes in minutes. https://conduktor.io/blog/kafka-observability https://conduktor.io/blog/kafka-observability Fri, 05 Sep 2025 00:00:00 GMT stephane-derosiaux Block malformed JSON before it breaks Kafka pipelines. Conduktor Trust validates schema, types, and fields—no Schema Registry migration needed. https://conduktor.io/blog/bounce-bad-kafka-data-with-json-schema-rules https://conduktor.io/blog/bounce-bad-kafka-data-with-json-schema-rules Thu, 04 Sep 2025 00:00:00 GMT chuck-larrieu-casias Kafka testing means validating schemas, configs, and consumer logic before production. Build confidence with contract tests and local clusters. https://conduktor.io/blog/kafka-testing https://conduktor.io/blog/kafka-testing Fri, 29 Aug 2025 00:00:00 GMT stephane-derosiaux Enforce Kafka data quality at write time with schema validation and policies. Garbage in at the producer means garbage out everywhere. https://conduktor.io/blog/kafka-data-quality https://conduktor.io/blog/kafka-data-quality Tue, 26 Aug 2025 00:00:00 GMT stephane-derosiaux Solve the dual-write problem with the transactional outbox pattern. PostgreSQL setup, Debezium CDC, and idempotent consumers. https://conduktor.io/blog/transactional-outbox-pattern-database-kafka https://conduktor.io/blog/transactional-outbox-pattern-database-kafka Mon, 25 Aug 2025 00:00:00 GMT stephane-derosiaux Kafka performance is 80% understanding your workload, 20% tuning configs. Measure bottlenecks before changing partition counts or batch sizes. https://conduktor.io/blog/kafka-performance https://conduktor.io/blog/kafka-performance Tue, 19 Aug 2025 00:00:00 GMT stephane-derosiaux Conduktor partners with LHP to secure and govern streaming data for automotive and aerospace manufacturers using Kafka for IIoT, AI, and analytics. https://conduktor.io/blog/secure-and-govern-real-time-manufacturing-data-with-conduktor-and-lhp https://conduktor.io/blog/secure-and-govern-real-time-manufacturing-data-with-conduktor-and-lhp Mon, 18 Aug 2025 00:00:00 GMT matthew-scott Build custom Kafka partitioners for geographic routing, priority lanes, and hot key distribution. Java implementation with production-tested patterns. https://conduktor.io/blog/kafka-custom-partitioners-beyond-default-hashing https://conduktor.io/blog/kafka-custom-partitioners-beyond-default-hashing Wed, 13 Aug 2025 00:00:00 GMT stephane-derosiaux 10,000+ Kafka forum posts reveal the truth: connector crashes, auth nightmares, schema failures, and cryptic errors plague production deployments. https://conduktor.io/blog/apache-kafka-what-10-000-forum-posts-reveal https://conduktor.io/blog/apache-kafka-what-10-000-forum-posts-reveal Mon, 11 Aug 2025 00:00:00 GMT stephane-derosiaux Mismatched training and production data cripples AI models. Real story from a UK lender shows why Conduktor Trust prevents ML pipeline failures. https://conduktor.io/blog/the-hidden-data-gap-that-wrecks-ai-projects https://conduktor.io/blog/the-hidden-data-gap-that-wrecks-ai-projects Thu, 07 Aug 2025 00:00:00 GMT matt-searle Kafka bridges the operational-analytical data divide in real time. Eliminate batch ETL lag and deliver fresh streaming data to analytics and AI systems. https://conduktor.io/blog/bridging-gap-between-operational-and-analytical-data https://conduktor.io/blog/bridging-gap-between-operational-and-analytical-data Fri, 01 Aug 2025 00:00:00 GMT stephane-derosiaux Kafka's schemaless flexibility creates data quality issues. Conduktor Trust validates schemaless topics, blocks bad data, and enforces custom rules. https://conduktor.io/blog/the-hidden-pitfalls-of-kafka-s-schemaless-data https://conduktor.io/blog/the-hidden-pitfalls-of-kafka-s-schemaless-data Thu, 24 Jul 2025 00:00:00 GMT johnathan-law Simulate broker failures, latency, and corrupted messages without breaking production. Chaos test Kafka with Conduktor's interceptors. https://conduktor.io/blog/chaos-test-kafka-and-build-resilient-streaming-environments https://conduktor.io/blog/chaos-test-kafka-and-build-resilient-streaming-environments Thu, 10 Jul 2025 00:00:00 GMT jorge-ruiz Avoid Kafka vendor lock-in with portable configurations and abstraction layers. Code examples for MSK, Confluent, and self-managed. https://conduktor.io/blog/kafka-vendor-lock-in-portability-strategies https://conduktor.io/blog/kafka-vendor-lock-in-portability-strategies Tue, 08 Jul 2025 00:00:00 GMT stephane-derosiaux Real-time data sharing drives 3x ROI. Explore Kafka-powered use cases in finance, healthcare, logistics—secure partner data exchange with Conduktor. https://conduktor.io/blog/why-real-time-data-sharing-matters https://conduktor.io/blog/why-real-time-data-sharing-matters Thu, 26 Jun 2025 00:00:00 GMT jorge-ruiz Kafka broker startup failures with real error messages and fixes. Cluster ID mismatch, port conflicts, disk issues, and config errors. https://conduktor.io/blog/broker-wont-start-common-failures-fixes https://conduktor.io/blog/broker-wont-start-common-failures-fixes Thu, 19 Jun 2025 00:00:00 GMT stephane-derosiaux Conduktor Trust provides governance, audit trails, and policy enforcement for AI agents, RAG systems, and real-time operational data pipelines. https://conduktor.io/blog/secure-data-operations-in-the-time-of-ai https://conduktor.io/blog/secure-data-operations-in-the-time-of-ai Thu, 19 Jun 2025 00:00:00 GMT quentin-packard Dash 2025 revealed AI observability risks—from exposed PII to agent telemetry attacks. How Conduktor + Datadog secure LLMs at scale. https://conduktor.io/blog/48-hours-at-dash-security-ai-and-the-observability-paradox https://conduktor.io/blog/48-hours-at-dash-security-ai-and-the-observability-paradox Wed, 18 Jun 2025 00:00:00 GMT quentin-packard Understand Kafka network partition failures, split-brain scenarios, and unclean leader election. ISR shrinkage and data loss prevention. https://conduktor.io/blog/kafka-network-partition-split-brain-failures https://conduktor.io/blog/kafka-network-partition-split-brain-failures Fri, 13 Jun 2025 00:00:00 GMT stephane-derosiaux Low-quality data destroys AI and ML outputs. Fix data quality at the source—before problems travel downstream. https://conduktor.io/blog/proactive-and-reactive-the-two-paths-towards-data-quality https://conduktor.io/blog/proactive-and-reactive-the-two-paths-towards-data-quality Thu, 12 Jun 2025 00:00:00 GMT johnathan-law Current London 2025 insights: data quality failures, migration complexity, developer bottlenecks, and Flink challenges shape Conduktor's roadmap priorit... https://conduktor.io/blog/current-london-2025-learnings-to-drive-our-roadmap https://conduktor.io/blog/current-london-2025-learnings-to-drive-our-roadmap Fri, 06 Jun 2025 00:00:00 GMT james-white Conduktor Trust validates streaming data quality at ingestion—preventing AI drift, bad analytics, and compliance failures with proactive governance. https://conduktor.io/blog/conduktor-launches-trust-to-guarantee-data-quality-at-the-source https://conduktor.io/blog/conduktor-launches-trust-to-guarantee-data-quality-at-the-source Wed, 21 May 2025 00:00:00 GMT james-white Track Kafka costs by team, project, and environment—not just partitions. Conduktor Chargeback maps service accounts to real business units. https://conduktor.io/blog/chargeback-attribute-map-kafka-costs-to-your-business https://conduktor.io/blog/chargeback-attribute-map-kafka-costs-to-your-business Thu, 08 May 2025 00:00:00 GMT julien-chanaud Data silos cost $3.1 trillion annually. Break them with Kafka + Conduktor's self-service, discoverability, and granular access controls. https://conduktor.io/blog/break-your-data-silos-before-they-break-your-business https://conduktor.io/blog/break-your-data-silos-before-they-break-your-business Fri, 02 May 2025 00:00:00 GMT jean-louis-boudart Deploy production-grade Kafka on Kubernetes with Strimzi. Separate node pools, KRaft mode, and the configuration choices that matter. https://conduktor.io/blog/strimzi-kubernetes-zero-to-production https://conduktor.io/blog/strimzi-kubernetes-zero-to-production Sat, 19 Apr 2025 00:00:00 GMT stephane-derosiaux Enterprise leaders tackle AI ROI, GenAI governance, and data controls. Conduktor execs reveal why shift-left alone won't protect LLM inputs. https://conduktor.io/blog/ai-expectations-realities-and-risks https://conduktor.io/blog/ai-expectations-realities-and-risks Thu, 17 Apr 2025 00:00:00 GMT paul-feldman A practical guide to transitioning into Kafka and real-time data roles, with specific technologies, projects, and job search strategies. https://conduktor.io/blog/from-engineer-to-event-streaming-expert-how-to-pivot-roles https://conduktor.io/blog/from-engineer-to-event-streaming-expert-how-to-pivot-roles Wed, 16 Apr 2025 00:00:00 GMT alasdair-ross Schema registries won't save you. Real Kafka challenges at scale: zombie topics, unclear ownership, legacy integrations. https://conduktor.io/blog/five-hidden-kafka-challenges-for-enterprises https://conduktor.io/blog/five-hidden-kafka-challenges-for-enterprises Tue, 15 Apr 2025 00:00:00 GMT james-white Real-time, trusted data is the new AI advantage. Discover why the future belongs to companies that treat data as a product, not a by-product. https://conduktor.io/blog/own-the-stream-power-the-model-monetizing-real-time-data-in-the-ai-era https://conduktor.io/blog/own-the-stream-power-the-model-monetizing-real-time-data-in-the-ai-era Tue, 15 Apr 2025 00:00:00 GMT stephane-derosiaux What software engineers, architects, and platform teams told us about the real challenges of scaling Kafka—straight from the floor at QCon London 2025. https://conduktor.io/blog/engineers-pov-on-streaming-qcon-2025-takeaways https://conduktor.io/blog/engineers-pov-on-streaming-qcon-2025-takeaways Fri, 11 Apr 2025 00:00:00 GMT jorge-ruiz Govern Kafka at scale with Terraform and Conduktor. Automate infrastructure, enforce policies, enable self-service—no chaos, no bottlenecks. https://conduktor.io/blog/govern-kafka-as-code-terraform-x-conduktor https://conduktor.io/blog/govern-kafka-as-code-terraform-x-conduktor Wed, 02 Apr 2025 00:00:00 GMT stuart-mould Scale streaming data with governance and quality controls. Shift left to prevent bad data, ensure AI success, and avoid architectural chaos. https://conduktor.io/blog/streaming-at-scale-don-t-let-data-chaos-derail-your-growth https://conduktor.io/blog/streaming-at-scale-don-t-let-data-chaos-derail-your-growth Thu, 27 Mar 2025 00:00:00 GMT stephane-derosiaux AI agents need real-time context to act autonomously. Discover how Kafka + Flink power adaptive, intelligent systems at Netflix and beyond. https://conduktor.io/blog/ai-agents-at-scale-the-critical-role-of-kafka-and-flink https://conduktor.io/blog/ai-agents-at-scale-the-critical-role-of-kafka-and-flink Tue, 25 Mar 2025 00:00:00 GMT stephane-derosiaux Configure Kafka audit logging for SOC2, HIPAA, and PCI-DSS compliance. Authorizer logging and forensic analysis techniques. https://conduktor.io/blog/kafka-audit-logging-compliance-forensics https://conduktor.io/blog/kafka-audit-logging-compliance-forensics Mon, 24 Mar 2025 00:00:00 GMT stephane-derosiaux Most enterprise data never drives a decision. Real-time data activation transforms idle data into competitive advantage. https://conduktor.io/blog/your-dark-data-is-a-goldmine-if-you-activate-it-in-real-time https://conduktor.io/blog/your-dark-data-is-a-goldmine-if-you-activate-it-in-real-time Fri, 21 Mar 2025 00:00:00 GMT stephane-derosiaux Crypto shredding destroys encryption keys to make Kafka data permanently unreadable. Instant GDPR compliance without topic deletion or costly reprocessing. https://conduktor.io/blog/crypto-shredding-in-kafka-a-cost-effective-way-to-ensure-compliance https://conduktor.io/blog/crypto-shredding-in-kafka-a-cost-effective-way-to-ensure-compliance Tue, 18 Mar 2025 00:00:00 GMT florent-ramiere Enterprise AI Assembly insights: breaking data silos across regions, federated governance, shift-left security, and mainframe modernization challenges. https://conduktor.io/blog/from-promise-to-practice-insights-from-the-millennium-alliance-enterprise-ai-maturity-and-transformation-assembly https://conduktor.io/blog/from-promise-to-practice-insights-from-the-millennium-alliance-enterprise-ai-maturity-and-transformation-assembly Tue, 18 Mar 2025 00:00:00 GMT paul-feldman Master Avro schema evolution with BACKWARD vs FORWARD compatibility modes. Compatibility matrix, curl commands for Schema Registry, and what breaks. https://conduktor.io/blog/schema-evolution-avro-compatibility-guide https://conduktor.io/blog/schema-evolution-avro-compatibility-guide Sat, 15 Mar 2025 00:00:00 GMT stephane-derosiaux Discover how Kafka and Flink power real-time AI agents, enabling autonomous decision-making, adaptive intelligence, and scalable data streaming. https://conduktor.io/blog/how-to-succeed-with-ai-kafka-and-ai-guardrails-are-non-negotiable https://conduktor.io/blog/how-to-succeed-with-ai-kafka-and-ai-guardrails-are-non-negotiable Fri, 14 Mar 2025 00:00:00 GMT stephane-derosiaux Gartner D&A Summit 2025: How Conduktor + Cloudera deliver trustworthy real-time data for AI—RAG, agentic AI, and streaming governance insights. https://conduktor.io/blog/unlocking-ai-s-full-potential-what-we-learned-at-the-gartner-d-a-summit-2025 https://conduktor.io/blog/unlocking-ai-s-full-potential-what-we-learned-at-the-gartner-d-a-summit-2025 Tue, 11 Mar 2025 00:00:00 GMT william-to Implement Kafka dead letter queues to isolate poison pill messages. Java error handling patterns, retry strategies, DLT monitoring, and reprocessing wor... https://conduktor.io/blog/dead-letter-topics-handling-poison-pills https://conduktor.io/blog/dead-letter-topics-handling-poison-pills Sun, 23 Feb 2025 00:00:00 GMT stephane-derosiaux Kafka's commit log differs fundamentally from message queues. The mental model shift that prevents costly anti-patterns. https://conduktor.io/blog/kafka-isnt-a-queue https://conduktor.io/blog/kafka-isnt-a-queue Fri, 21 Feb 2025 00:00:00 GMT stephane-derosiaux Christoph Schubert's journey from teaching in China to Confluent to Conduktor's Head of Sales Engineering, mastering Kafka and customer success. https://conduktor.io/blog/from-academia-to-sales-engineering-a-journey-into-streaming-data https://conduktor.io/blog/from-academia-to-sales-engineering-a-journey-into-streaming-data Thu, 20 Feb 2025 00:00:00 GMT francesca-scantlebury Maria's path from studying languages in Seoul to designing UX/UI at Conduktor shows that design careers don't require a traditional route. https://conduktor.io/blog/a-product-designer-s-journey-to-conduktor https://conduktor.io/blog/a-product-designer-s-journey-to-conduktor Mon, 17 Feb 2025 00:00:00 GMT francesca-scantlebury Share Kafka streams with external partners through isolated access zones—no cluster duplication or complex replication required. https://conduktor.io/blog/partner-zones-secure-kafka-data-sharing-without-duplication https://conduktor.io/blog/partner-zones-secure-kafka-data-sharing-without-duplication Fri, 07 Feb 2025 00:00:00 GMT romain-lecomte Conduktor's alerts system gives teams ownership over their Kafka notifications with webhook integrations and event-driven alerting. https://conduktor.io/blog/enterprise-ready-kafka-alerting-decentralized-control-webhooks-for-multiple-teams https://conduktor.io/blog/enterprise-ready-kafka-alerting-decentralized-control-webhooks-for-multiple-teams Tue, 28 Jan 2025 00:00:00 GMT julien-chanaud Build your first Kafka producer in Python, Java, and Go. Working code examples, common connection errors, and essential configuration for reliable messa... https://conduktor.io/blog/first-kafka-producer-python-java-go https://conduktor.io/blog/first-kafka-producer-python-java-go Tue, 28 Jan 2025 00:00:00 GMT stephane-derosiaux 2025 data streaming strategy: experts from Atyeti, GigaOm discuss data maturity, AI readiness, governance, and modernizing your tech stack. https://conduktor.io/blog/your-2025-data-streaming-strategy-blueprint-insights-from-industry-experts https://conduktor.io/blog/your-2025-data-streaming-strategy-blueprint-insights-from-industry-experts Tue, 28 Jan 2025 00:00:00 GMT quentin-packard Client-side encryption creates ungovernable sprawl. Proxy-layer encryption centralizes control and makes Kafka a dumb pipe. https://conduktor.io/blog/stop-building-kafka-encryption-libraries https://conduktor.io/blog/stop-building-kafka-encryption-libraries Sat, 25 Jan 2025 00:00:00 GMT stephane-derosiaux GenAI adoption insights from Resolve.ai CEO Spiros Xanthos: data quality strategies, strategic use cases, and buy vs. build decisions for AI success. https://conduktor.io/blog/taking-the-genai-bet-what-innovators-have-learned https://conduktor.io/blog/taking-the-genai-bet-what-innovators-have-learned Thu, 02 Jan 2025 00:00:00 GMT quentin-packard Query Kafka directly with SQL using Conduktor Scale. Access real-time insights, troubleshoot issues, and optimize operations without data duplication. https://conduktor.io/blog/sql-over-kafka-transforming-real-time-data-into-instant-insights https://conduktor.io/blog/sql-over-kafka-transforming-real-time-data-into-instant-insights Thu, 19 Dec 2024 00:00:00 GMT james-white Configure G1GC and ZGC for Kafka brokers. Heap sizing, pause time targets, and when to switch collectors in production. https://conduktor.io/blog/kafka-jvm-tuning-g1gc-vs-zgc-production https://conduktor.io/blog/kafka-jvm-tuning-g1gc-vs-zgc-production Tue, 10 Dec 2024 00:00:00 GMT stephane-derosiaux Track Kafka costs by team, domain, or application. Attribute ingress/egress usage to optimize infra and forecast spending. https://conduktor.io/blog/tackling-apache-kafka-cost-attribution-with-chargeback https://conduktor.io/blog/tackling-apache-kafka-cost-attribution-with-chargeback Thu, 05 Dec 2024 00:00:00 GMT matt-searle FIMA Europe 2024: data management beats AI as top priority. Break silos, shift quality upstream, expand real-time streaming. https://conduktor.io/blog/future-of-financial-services-lies-in-data-key-takeaways-fima-europe https://conduktor.io/blog/future-of-financial-services-lies-in-data-key-takeaways-fima-europe Tue, 26 Nov 2024 00:00:00 GMT james-white Master data quality in streaming with proactive monitoring, use-case checks, and real-time alerts. Build trust in data for AI and real-time decisions. https://conduktor.io/blog/mastering-data-quality-in-streaming https://conduktor.io/blog/mastering-data-quality-in-streaming Mon, 18 Nov 2024 00:00:00 GMT quentin-packard Conduktor raises $30M in Series B funding led by RTP Global to expand in the U.S. and meet rising demand for secure, real-time data streaming solutions. https://conduktor.io/blog/conduktor-secures-30-million-series-b-to-lead-the-future-of-streaming-data-management https://conduktor.io/blog/conduktor-secures-30-million-series-b-to-lead-the-future-of-streaming-data-management Tue, 12 Nov 2024 00:00:00 GMT nicolas-orban Step-by-step guide to deploying Conduktor Gateway on Kubernetes with host-based (SNI) routing, TLS certificates, and external Ingress access. https://conduktor.io/blog/deploy-conduktor-gateway-with-kubernetes-and-host-based-routing https://conduktor.io/blog/deploy-conduktor-gateway-with-kubernetes-and-host-based-routing Tue, 05 Nov 2024 00:00:00 GMT chuck-larrieu-casias How transport and logistics use Kafka for route optimization, fleet tracking, and why data governance matters at scale. https://conduktor.io/blog/drive-speed-efficiency-and-revenue-growth-with-streaming-data-for-transport-and-logistics https://conduktor.io/blog/drive-speed-efficiency-and-revenue-growth-with-streaming-data-for-transport-and-logistics Mon, 28 Oct 2024 00:00:00 GMT jorge-ruiz Financial services use Kafka for fraud detection, real-time payments, and compliance. Build secure streaming architectures. https://conduktor.io/blog/how-to-streamline-your-kafka-architecture-and-drive-success-in-financial-services https://conduktor.io/blog/how-to-streamline-your-kafka-architecture-and-drive-success-in-financial-services Mon, 21 Oct 2024 00:00:00 GMT jorge-ruiz Learn how retail organizations can simplify Kafka operations, secure data streaming, and drive growth with Conduktor's enterprise data management platform. https://conduktor.io/blog/how-retail-organizations-can-drive-reliable-accurate-data-streaming https://conduktor.io/blog/how-retail-organizations-can-drive-reliable-accurate-data-streaming Tue, 15 Oct 2024 00:00:00 GMT jorge-ruiz BYOC, decoupling compute from storage, and Shift Left strategies dominate data streaming. Insights from Aiven and Conduktor on AI-driven real-time data. https://conduktor.io/blog/the-biggest-trends-in-data-streaming https://conduktor.io/blog/the-biggest-trends-in-data-streaming Fri, 11 Oct 2024 00:00:00 GMT jorge-ruiz Use Kafka headers for distributed tracing, content-based routing, and audit metadata without modifying payloads. Java and Python code examples included. https://conduktor.io/blog/kafka-headers-metadata-without-polluting-payload https://conduktor.io/blog/kafka-headers-metadata-without-polluting-payload Thu, 10 Oct 2024 00:00:00 GMT stephane-derosiaux Platform teams centralize expertise but bottleneck delivery. Embedded engineers move fast but create chaos. Here's how to choose. https://conduktor.io/blog/kafka-team-structure-platform-embedded https://conduktor.io/blog/kafka-team-structure-platform-embedded Thu, 03 Oct 2024 00:00:00 GMT stephane-derosiaux TLS protects data in transit, but brokers still see plaintext. Field-level encryption protects PII at rest and enables crypto shredding. https://conduktor.io/blog/kafka-field-level-encryption-beyond-tls https://conduktor.io/blog/kafka-field-level-encryption-beyond-tls Sat, 07 Sep 2024 00:00:00 GMT stephane-derosiaux Choose between Kafka Streams and ksqlDB for stream processing. Use case comparison, team skills assessment, deployment models, and operational trade-offs. https://conduktor.io/blog/kafka-streams-vs-ksqldb-choosing https://conduktor.io/blog/kafka-streams-vs-ksqldb-choosing Mon, 02 Sep 2024 00:00:00 GMT stephane-derosiaux Zero-code Kafka encryption: field-level encryption/decryption with Conduktor Gateway—secure PII, role-based access, no application changes required. https://conduktor.io/blog/zero-code-encryption-strategies-for-apache-kafka https://conduktor.io/blog/zero-code-encryption-strategies-for-apache-kafka Wed, 28 Aug 2024 00:00:00 GMT stephane-derosiaux Run SQL queries directly against Kafka topics — filter, join, and aggregate streaming data without moving it to a warehouse. Real-time SQL over Kafka. https://conduktor.io/blog/apache-kafka-sql-topics https://conduktor.io/blog/apache-kafka-sql-topics Wed, 14 Aug 2024 00:00:00 GMT stephane-derosiaux How KRaft replaces ZooKeeper in Kafka 4.0. Controller quorum architecture, broker configuration examples, and step-by-step migration from ZooKeeper. https://conduktor.io/blog/kraft-explained-kafka-without-zookeeper https://conduktor.io/blog/kraft-explained-kafka-without-zookeeper Sat, 27 Jul 2024 00:00:00 GMT stephane-derosiaux Build scalable Kafka tooling with GitOps and governance automation. Avoid common pitfalls in real-time data security, costs, and compliance. https://conduktor.io/blog/real-time-data-with-kafka https://conduktor.io/blog/real-time-data-with-kafka Thu, 18 Jul 2024 00:00:00 GMT james-white Build a Kafka topic naming convention that scales. Structural patterns, ACL alignment, and common mistakes that break governance. https://conduktor.io/blog/kafka-topic-naming-conventions-scale https://conduktor.io/blog/kafka-topic-naming-conventions-scale Sat, 13 Jul 2024 00:00:00 GMT stephane-derosiaux The "we'll add it later" approach costs weeks of debugging and painful migrations. Schema Registry is day-one infrastructure. https://conduktor.io/blog/schema-registry-isnt-optional https://conduktor.io/blog/schema-registry-isnt-optional Sat, 13 Jul 2024 00:00:00 GMT stephane-derosiaux Kafka's default security fails in production. ACLs become unmanageable, PII leaks, encryption is missing. Here's the fix. https://conduktor.io/blog/kafka-encryption-and-security-best-practices-for-protecting-data https://conduktor.io/blog/kafka-encryption-and-security-best-practices-for-protecting-data Thu, 11 Jul 2024 00:00:00 GMT james-white Ensure Kafka data quality at scale: prevent invalid messages, enforce schemas, automate checks without disrupting pipelines. https://conduktor.io/blog/enforcing-kafka-data-quality-at-scale https://conduktor.io/blog/enforcing-kafka-data-quality-at-scale Fri, 05 Jul 2024 00:00:00 GMT stuart-mould Complex infrastructure calls for automation. GitOps streamlines Kafka configuration and scales deployments efficiently. https://conduktor.io/blog/kafka-gitops-for-efficient-scaling https://conduktor.io/blog/kafka-gitops-for-efficient-scaling Thu, 04 Jul 2024 00:00:00 GMT james-white Build event-sourced systems on Kafka. Topic design, aggregate reconstruction, CQRS projections, and the gotchas that break production. https://conduktor.io/blog/event-sourcing-kafka-patterns-pitfalls https://conduktor.io/blog/event-sourcing-kafka-patterns-pitfalls Wed, 03 Jul 2024 00:00:00 GMT stephane-derosiaux A code pattern created 4.2M Kafka producers per hour, crashed the cluster, and silenced alerts for 9+ hours. Dissecting the postmortem. https://conduktor.io/blog/pagerduty-kafka-outage-lessons https://conduktor.io/blog/pagerduty-kafka-outage-lessons Fri, 28 Jun 2024 00:00:00 GMT stephane-derosiaux Kafka adds complexity you may not need. Five scenarios where PostgreSQL LISTEN/NOTIFY, Redis Streams, SQS, or simple webhooks outperform Kafka. https://conduktor.io/blog/when-not-to-use-kafka https://conduktor.io/blog/when-not-to-use-kafka Tue, 21 May 2024 00:00:00 GMT stephane-derosiaux Master Kafka retention—retention.ms, segment.ms, and why messages persist longer than expected. Debug commands included. https://conduktor.io/blog/kafka-retention-policies-when-messages-disappear https://conduktor.io/blog/kafka-retention-policies-when-messages-disappear Mon, 06 May 2024 00:00:00 GMT stephane-derosiaux Handle duplicate Kafka messages gracefully. Database constraints, Redis lookups, and the deduplication patterns that scale. https://conduktor.io/blog/building-idempotent-consumers https://conduktor.io/blog/building-idempotent-consumers Fri, 26 Apr 2024 00:00:00 GMT stephane-derosiaux Handle GDPR Article 17 deletion requests in Kafka's append-only log. Crypto shredding implementation, tombstone patterns, and compliant retention strate... https://conduktor.io/blog/gdpr-kafka-right-to-erasure https://conduktor.io/blog/gdpr-kafka-right-to-erasure Wed, 24 Apr 2024 00:00:00 GMT stephane-derosiaux Conduktor 2.0 expands free tier with debugging, monitoring, and encryption tools—simplifying Kafka for developers, architects, and platform teams. https://conduktor.io/blog/conduktor-2-0-the-new-standard-for-teams-working-with-kafka https://conduktor.io/blog/conduktor-2-0-the-new-standard-for-teams-working-with-kafka Thu, 11 Apr 2024 00:00:00 GMT james-white Share real-time Kafka data with partners without replication or DMZ clusters. Conduktor securely encrypts, isolates, and audits external data access. https://conduktor.io/blog/sharing-data-with-partners https://conduktor.io/blog/sharing-data-with-partners Thu, 04 Apr 2024 00:00:00 GMT stuart-mould Kafka Summit London 2024 highlights: real-time analytics, GitOps self-service, SQL over Kafka, and the shift from batch to streaming data. https://conduktor.io/blog/kafka-summit-london-2024 https://conduktor.io/blog/kafka-summit-london-2024 Fri, 29 Mar 2024 00:00:00 GMT stephane-derosiaux Kafka's exactly-once adds latency and complexity. Most applications work better with at-least-once and idempotent consumers. https://conduktor.io/blog/you-dont-need-exactly-once-kafka https://conduktor.io/blog/you-dont-need-exactly-once-kafka Sat, 23 Mar 2024 00:00:00 GMT stephane-derosiaux How to choose Kafka partition count. Sizing formulas, overhead costs, and why decreasing partitions requires topic migration. https://conduktor.io/blog/partition-count-decision-guide https://conduktor.io/blog/partition-count-decision-guide Fri, 01 Mar 2024 00:00:00 GMT stephane-derosiaux Deploy Conduktor Console on AWS in minutes using CloudFormation and ECS Fargate. Works with MSK, Confluent, Redpanda, and Kafka Connect. https://conduktor.io/blog/step-by-step-deploy-the-most-advanced-kafka-console-on-aws-in-minutes https://conduktor.io/blog/step-by-step-deploy-the-most-advanced-kafka-console-on-aws-in-minutes Mon, 19 Feb 2024 00:00:00 GMT james-white Join Conduktor's February 2024 USA tour in NYC, Boston, Bay Area, and Seattle. Learn about Kafka poison pills, schema compatibility, and data contracts. https://conduktor.io/blog/confluent-conduktor-usa-tour https://conduktor.io/blog/confluent-conduktor-usa-tour Thu, 08 Feb 2024 00:00:00 GMT stephane-derosiaux Framework for evaluating self-managed Kafka vs managed services like Confluent Cloud and MSK. TCO analysis, team skills assessment, and decision criteria. https://conduktor.io/blog/kafka-build-vs-buy-decision-framework https://conduktor.io/blog/kafka-build-vs-buy-decision-framework Fri, 26 Jan 2024 00:00:00 GMT stephane-derosiaux Emergency runbook for Kafka disk full scenarios. Immediate triage commands, safe segment deletion, recovery steps, and retention tuning to prevent recur... https://conduktor.io/blog/disk-full-emergency-recovery https://conduktor.io/blog/disk-full-emergency-recovery Wed, 17 Jan 2024 00:00:00 GMT stephane-derosiaux Transform data chaos with strategic tagging. Enable Data Mesh, ABAC security, and better analytics through hierarchical metadata organization. https://conduktor.io/blog/data-chaos-clarity-tag-everything-everywhere https://conduktor.io/blog/data-chaos-clarity-tag-everything-everywhere Tue, 16 Jan 2024 00:00:00 GMT stephane-derosiaux Conduktor's 2023 journey: launching Console, introducing Gateway proxy, achieving SOC2 Type II, and evolving from developer tool to enterprise platform. https://conduktor.io/blog/conduktor-retrospective-2023 https://conduktor.io/blog/conduktor-retrospective-2023 Thu, 11 Jan 2024 00:00:00 GMT nicolas-orban Compare Kafka DR patterns. MirrorMaker 2 setup, offset translation, conflict resolution, and when each architecture makes sense. https://conduktor.io/blog/multi-region-kafka-active-active-passive https://conduktor.io/blog/multi-region-kafka-active-active-passive Thu, 11 Jan 2024 00:00:00 GMT stephane-derosiaux SOC2 Type II journey: policies, pentesting, CVE management, agent monitoring, and compliance automation with Drata—lessons from Conduktor's certification. https://conduktor.io/blog/what-we-learned-from-soc2-type2-write-what-you-do-do-what-you-write https://conduktor.io/blog/what-we-learned-from-soc2-type2-write-what-you-do-do-what-you-write Tue, 02 Jan 2024 00:00:00 GMT stephane-derosiaux Scale Kafka across teams with federated governance and self-service data platforms. Tackle access management, security, and onboarding bottlenecks. https://conduktor.io/blog/scaling-kafka-securely-for-your-organization https://conduktor.io/blog/scaling-kafka-securely-for-your-organization Fri, 22 Dec 2023 00:00:00 GMT james-white Run Kafka locally with one command. Explore real-time data, encryption, masking, and safeguards without any setup. https://conduktor.io/blog/a-new-getting-started-experience-with-kafka-and-conduktor https://conduktor.io/blog/a-new-getting-started-experience-with-kafka-and-conduktor Mon, 18 Dec 2023 00:00:00 GMT stuart-mould Key insights from Current 2023 and Big Data London: Kafka proxies for encryption, cost optimization through virtualization, and Flink's serverless future. https://conduktor.io/blog/the-future-of-data-streaming-current-2023-big-data-london https://conduktor.io/blog/the-future-of-data-streaming-current-2023-big-data-london Thu, 16 Nov 2023 00:00:00 GMT james-white Test Kafka components with Testcontainers, Streams test-utils, chaos engineering, and Conduktor Console. Ensure data quality and resilience at scale. https://conduktor.io/blog/testing-your-apache-kafka-data-with-confidence-2023 https://conduktor.io/blog/testing-your-apache-kafka-data-with-confidence-2023 Thu, 19 Oct 2023 00:00:00 GMT james-white Run real-time SQL queries on Kafka with PySpark. Stream processing tutorial with JSON transformations, ChatGPT integration, and data generation. https://conduktor.io/blog/getting-started-with-pyspark-and-kafka https://conduktor.io/blog/getting-started-with-pyspark-and-kafka Sun, 17 Sep 2023 00:00:00 GMT stephane-derosiaux Claudiu Danaila shares his path from running a design agency to leading design at Conduktor, and why good design is a business differentiator. https://conduktor.io/blog/meet-the-team-claudiu-danaila https://conduktor.io/blog/meet-the-team-claudiu-danaila Wed, 23 Aug 2023 00:00:00 GMT nicolas-orban Conduktor uses Redpanda for local development because it starts instantly, weighs 130MB, includes Schema Registry, and eliminates ZooKeeper complexity. https://conduktor.io/blog/conduktor-redpanda-best-breed-kafka-experience https://conduktor.io/blog/conduktor-redpanda-best-breed-kafka-experience Thu, 10 Aug 2023 00:00:00 GMT stephane-derosiaux How Conduktor built work-life balance at Series A through hybrid working, transparent feedback, and health benefits—without sacrificing startup velocity. https://conduktor.io/blog/promoting-work-life-balance-in-a-series-a-start-up https://conduktor.io/blog/promoting-work-life-balance-in-a-series-a-start-up Wed, 09 Aug 2023 00:00:00 GMT nicolas-orban Deploy Conduktor on AWS Marketplace for MSK with IAM, Glue Schema Registry support, and seamless ECS/EKS integration in minutes. https://conduktor.io/blog/conduktor-available-on-the-aws-marketplace-to-complement-msk https://conduktor.io/blog/conduktor-available-on-the-aws-marketplace-to-complement-msk Wed, 02 Aug 2023 00:00:00 GMT stephane-derosiaux Master Kafka performance: partition sizing, SerDe optimization, dual-write solutions, tombstones, and producer tuning for robust applications. https://conduktor.io/blog/top-5-tips-to-build-more-robust-and-performant-kafka-applications https://conduktor.io/blog/top-5-tips-to-build-more-robust-and-performant-kafka-applications Wed, 26 Jul 2023 00:00:00 GMT stephane-derosiaux Migrate from Desktop to Console: centralized deployment, RBAC, field-level masking, audit logs, SSO, monitoring—enterprise Kafka UI advantages. https://conduktor.io/blog/why-you-should-consider-migrating-to-conduktor-console https://conduktor.io/blog/why-you-should-consider-migrating-to-conduktor-console Wed, 12 Jul 2023 00:00:00 GMT james-white Meet Aurélie Marcuzzo, Customer Success at Conduktor. From France to London, she onboards customers, tests features, and gives users the love they need. https://conduktor.io/blog/meet-the-team-aurelie-marcuzzo https://conduktor.io/blog/meet-the-team-aurelie-marcuzzo Tue, 27 Jun 2023 00:00:00 GMT nicolas-orban Bypass Kafka's 1MB limit without performance hits. Use S3 cold storage with claim check pattern for PDFs, images, and massive batches—seamlessly. https://conduktor.io/blog/beyond-limits-produce-large-records-without-undermining-apache-kafka https://conduktor.io/blog/beyond-limits-produce-large-records-without-undermining-apache-kafka Mon, 19 Jun 2023 00:00:00 GMT jean-louis-boudart Kafka disaster recovery: set RTO/RPO targets, implement multi-region failover, and automate switchover with Conduktor Gateway to eliminate manual errors. https://conduktor.io/blog/when-disaster-hits-kafka-are-you-ready https://conduktor.io/blog/when-disaster-hits-kafka-are-you-ready Sat, 17 Jun 2023 00:00:00 GMT florent-ramiere Centralized Kafka bottlenecks productivity. Decentralized sprawls costs. Build hybrid streaming platforms with multi-tenancy and self-service. https://conduktor.io/blog/5-practices-for-kafka-leaders-to-build-an-efficient-streaming-platform https://conduktor.io/blog/5-practices-for-kafka-leaders-to-build-an-efficient-streaming-platform Thu, 08 Jun 2023 00:00:00 GMT emma-humber DIY Kafka encryption libraries become multi-language, multi-KMS nightmares. Proxy-layer field-level encryption eliminates tech debt across producers and... https://conduktor.io/blog/how-to-encrypt-data-in-kafka-without-piling-up-tech-debt https://conduktor.io/blog/how-to-encrypt-data-in-kafka-without-piling-up-tech-debt Mon, 05 Jun 2023 00:00:00 GMT florent-ramiere Conduktor Gateway sits between Kafka clusters and apps. Review deployment options, networking patterns, and trade-offs. https://conduktor.io/blog/how-to-deploy-conduktor-gateway https://conduktor.io/blog/how-to-deploy-conduktor-gateway Thu, 01 Jun 2023 00:00:00 GMT francois-teychene Kafka costs exploding? Cut storage, networking, and partition expenses 10x with virtual clusters, compression enforcement, and cold storage. https://conduktor.io/blog/8-new-ways-to-drastically-reduce-your-kafka-costs https://conduktor.io/blog/8-new-ways-to-drastically-reduce-your-kafka-costs Tue, 30 May 2023 00:00:00 GMT florent-ramiere Field-level encryption, multi-tenancy, topic-as-a-service, and RBAC dominated conversations at our booth. Here's what the community cares about most. https://conduktor.io/blog/kafka-summit-london-2023-feedback-from-our-booth https://conduktor.io/blog/kafka-summit-london-2023-feedback-from-our-booth Thu, 25 May 2023 00:00:00 GMT stephane-derosiaux Prevent Kafka outages with proactive safeguarding via Conduktor Gateway. Automate best practices, enforce rules, and eliminate tech debt before it starts. https://conduktor.io/blog/proactive-safeguarding-with-conduktor-gateway https://conduktor.io/blog/proactive-safeguarding-with-conduktor-gateway Wed, 17 May 2023 00:00:00 GMT tom-scott Data mesh decentralizes ownership for agility and scale. Learn core principles, architecture, and how Kafka + Conduktor enable domain-driven data products. https://conduktor.io/blog/what-is-a-data-mesh https://conduktor.io/blog/what-is-a-data-mesh Thu, 04 May 2023 00:00:00 GMT stephane-derosiaux Conduktor's CEO Nicolas Orban shares key learnings from our 2023 offsite: quality over quantity, inclusive activities, and transparent communication. https://conduktor.io/blog/key-learnings-from-our-2023-offsite https://conduktor.io/blog/key-learnings-from-our-2023-offsite Mon, 01 May 2023 00:00:00 GMT nicolas-orban Kafka RBAC with fine-grained permissions for humans and applications. Manage producer, consumer, and admin access without modifying broker-level ACLs. https://conduktor.io/blog/security-for-humans-and-applications-in-apache-kafka https://conduktor.io/blog/security-for-humans-and-applications-in-apache-kafka Sat, 08 Apr 2023 00:00:00 GMT stephane-derosiaux Master data integration for big data success. Explore ETL vs ELT processes, data quality benefits, and automation strategies for business intelligence. https://conduktor.io/blog/data-integration-the-what-why-and-benefits https://conduktor.io/blog/data-integration-the-what-why-and-benefits Thu, 09 Mar 2023 00:00:00 GMT stephane-derosiaux Change Data Capture (CDC) explained: real-time database replication, log-based streaming, push vs. pull methods, and Kafka CDC implementation patterns. https://conduktor.io/blog/what-is-change-data-capture-anyways https://conduktor.io/blog/what-is-change-data-capture-anyways Thu, 09 Mar 2023 00:00:00 GMT stephane-derosiaux Automate Kafka topic governance and ACL management with Topic as a Service—eliminate bottlenecks, enforce standards, free your ops team. https://conduktor.io/blog/topic-as-a-service-automate-governance-decisions-for-apache-kafka https://conduktor.io/blog/topic-as-a-service-automate-governance-decisions-for-apache-kafka Tue, 31 Jan 2023 00:00:00 GMT stephane-derosiaux Schema Registry enforces data contracts in Kafka, preventing schema drift and producer-consumer mismatches. Learn serialization, Avro evolution, and set... https://conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it https://conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it Thu, 24 Nov 2022 00:00:00 GMT stephane-derosiaux Deep dive into Kafka segments, indexing, and log retention policies—optimize storage, compaction, and broker performance for production clusters. https://conduktor.io/blog/understanding-kafka-s-internal-storage-and-log-retention https://conduktor.io/blog/understanding-kafka-s-internal-storage-and-log-retention Tue, 19 Jul 2022 00:00:00 GMT tom-scott