# Stéphane Derosiaux

**CPTO**

Co-founder and CTO of Conduktor. 15+ years in software and data engineering, building large-scale streaming systems.

[LinkedIn profile](https://www.linkedin.com/in/stephane-derosiaux/)

## Articles by Stéphane Derosiaux

- [HIPAA 2026 and Kafka: Securing ePHI Streams](https://www.conduktor.io/blog/kafka-hipaa-compliance) (2026-05-25) — The 2026 HIPAA Security Rule mandates encryption, MFA, and annual asset inventory. What it breaks in a typical Kafka deployment, and how to fix it.
- [Rethinking Kafka Security for HIPAA 2026](https://www.conduktor.io/blog/rethinking-kafka-security-for-hipaa-2026) (2026-05-25) — Rethinking Kafka security for HIPAA 2026. Why streaming breaks the 'at rest' abstraction the rule was written around, and what to fix in the architecture.
- [Conduktor MCP + Skills for AI Agents](https://www.conduktor.io/blog/conduktor-mcp-and-skills-for-ai-agents) (2026-05-17) — How Conduktor MCP and Skills let AI agents work with your live Kafka. Patterns for platform onboarding, dev team self-service, and on-demand insights.
- [librdkafka vs Java Client: 10 Config Defaults That Differ](https://www.conduktor.io/blog/librdkafka-vs-java-client) (2026-05-05) — Python confluent-kafka and Java share config names but not behavior. 10 librdkafka differences tested on a KRaft cluster, with source code and real output.
- [The Kafka data sharing ladder](https://www.conduktor.io/blog/the-data-sharing-ladder) (2026-04-15) — Most teams talk data monetization but can't share a Kafka topic outside their VPC. The real path has four levels, from streaming to partner exchange.
- [Kafka in Production: 11 Pitfalls You Should Avoid](https://www.conduktor.io/blog/kafka-production-pitfalls) (2026-03-30) — 11 Kafka behaviors we found helping customers, then validated in the source code. Silent safety bypasses, configs that lie, compression traps.
- [Building a Data Lake with Kafka and S3 (Without Breaking It)](https://www.conduktor.io/blog/building-a-data-lake-with-kafka-and-s3) (2026-03-11) — One corrupt Parquet file froze a data lake for 6 hours. How Conduktor Gateway prevents bad data from reaching S3 with CEL validation rules.
- [How Conduktor Gateway Reduces Kafka DR from Hours to Minutes](https://www.conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes) (2026-03-02) — Conduktor Gateway cuts Kafka DR coordination overhead. One API call switches all clients to the secondary cluster. Cut execution from hours to minutes.
- [Stream Real-Time WebSocket Data to Apache Kafka with Our New Source Connector](https://www.conduktor.io/blog/kafka-connect-websocket-source-connector) (2026-02-24) — An open-source Kafka Connect connector for WebSocket streams. Auto-reconnect, auth support, and configurable buffering. Get started in 5 minutes.
- [Enterprise Kafka: Control Without Bottlenecks](https://www.conduktor.io/blog/enterprise-kafka-management) (2026-02-09) — Enterprise Kafka management needs automated governance, not slow approval processes. Control multi-cluster environments without creating bottlenecks.
- [AI for Kafka Operations](https://www.conduktor.io/blog/ai-kafka-operations) (2026-02-07) — AI for Kafka operations makes context instantly available. Reduce MTTR from hours to minutes with natural-language diagnostics and MCP.
- [Capture OpenTelemetry Signals in Apache Kafka with Our New OTLP Source Connector](https://www.conduktor.io/blog/kafka-connect-opentelemetry-source-connector) (2026-02-06) — Stream traces, metrics, and logs into Kafka topics for durable storage, replay, and custom analytics. Open source, Apache License 2.0.
- [Kafka Data Products: Beyond Topics](https://www.conduktor.io/blog/kafka-data-products) (2026-02-03) — Kafka data products need contracts, ownership, SLAs, and discovery. Topics with documentation aren't data products—they're shared state.
- [Safeguard Kafka from Timestamp-Based Data Loss](https://www.conduktor.io/blog/safeguard-kafka-from-timestamp-based-data-loss) (2026-02-03) — Producer timestamps can trigger immediate retention deletion. Configure timestamp validation, use LogAppendTime, or enforce bounds at the gateway level.
- [Kafka Latency: Measuring p50, p99, p99.9 in Production](https://www.conduktor.io/blog/kafka-latency) (2026-01-30) — Kafka latency averages hide tail problems. Optimize p99 and p999, not p50, and measure end-to-end from producer commit to consumer processing.
- [Kafka DR: Why Replication Isn't the Hard Part](https://www.conduktor.io/blog/kafka-dr-why-replication-isnt-the-hard-part) (2026-01-27) — PagerDuty's 9-hour Kafka outage reveals the real DR problem - coordinating 47 services during crisis. Why replication alone won't save you.
- [Kafka Policy Enforcement: Code Over Docs](https://www.conduktor.io/blog/kafka-policy-enforcement) (2026-01-21) — Kafka policies in Confluence are suggestions. CEL-based enforcement at the API layer prevents bad configs before they reach production.
- [Ingest gRPC Streaming Data into Apache Kafka with Our New Source Connector](https://www.conduktor.io/blog/kafka-connect-grpc-source-connector) (2026-01-19) — Stream data from gRPC server endpoints directly into Kafka topics with our open-source Kafka Connect source connector. Zero custom code, just configuration.
- [Kafka Compliance: GDPR, SOC2, HIPAA, DORA](https://www.conduktor.io/blog/kafka-security-compliance) (2026-01-14) — GDPR, SOC2, HIPAA, and DORA compliance for Kafka infrastructure. Generate audit evidence in minutes with continuous automated controls.
- [Kafka Control Plane: API-Driven Management](https://www.conduktor.io/blog/kafka-control-plane) (2026-01-06) — Your Kafka data plane is fast and resilient. Your control plane—provisioning, access, config—probably runs on scripts and tribal knowledge.
- [Kafka Audit Automation: Continuous Compliance](https://www.conduktor.io/blog/kafka-audit-automation) (2025-12-30) — Manual Kafka audit prep is security theater. Automate compliance evidence generation so audits take days, not months of scrambling.
- [Kafka Data Contracts: Prevent Breaking Changes](https://www.conduktor.io/blog/kafka-data-contracts) (2025-12-23) — Data contracts prevent breaking changes in Kafka. Enforce schema compatibility, versioning, and migration rules before bad data ships.
- [Kafka Alerting: Fix Alert Fatigue](https://www.conduktor.io/blog/kafka-alerting) (2025-12-16) — Kafka alert fatigue is a design problem, not a sensitivity problem. Build tiered alerts where 95% of pages result in action, not silence.
- [IBM's $11B Confluent Acquisition: What It Means for Kafka Users](https://www.conduktor.io/blog/ibm-confluent-acquisition-kafka-future) (2025-12-15) — Strategic analysis of IBM's Confluent acquisition. What changes for enterprise Kafka deployments and architecture decisions.
- [Rolling Restarts Without Downtime](https://www.conduktor.io/blog/rolling-restarts-without-downtime) (2025-12-10) — Zero-downtime Kafka rolling restarts. Pre-flight checks, ISR verification, controlled shutdown, and KRaft controller order.
- [Kafka Costs: Stop Overpaying by 30-50%](https://www.conduktor.io/blog/kafka-costs) (2025-12-09) — Teams overpay for Kafka 30-50% through over-provisioning and topic sprawl. Measure per-team usage and right-size clusters to cut costs.
- [Testing Kafka Applications: Testcontainers, Embedded Kafka, and Mocks](https://www.conduktor.io/blog/testing-kafka-testcontainers-embedded-mocks) (2025-12-09) — Choose the right Kafka testing strategy. Testcontainers for integration tests, MockProducer for unit tests, TopologyTestDriver for Streams.
- [A Practical Guide to Kafka Consumer Lag Alert Thresholds](https://www.conduktor.io/blog/kafka-consumer-lag-alerting-thresholds-guide) (2025-12-08) — Stop alert fatigue from consumer lag metrics. Offset vs time-based lag, per-workload thresholds, and rate-of-change detection.
- [Kafka Data Sharing Across Teams](https://www.conduktor.io/blog/kafka-data-sharing) (2025-12-02) — Cross-team Kafka data sharing should be as simple as an API call. Discovery catalogs and approval workflows replace Slack-driven access requests.
- [Why Enterprise AI Projects Stall on Data Access, Not Models](https://www.conduktor.io/blog/the-hidden-architecture-problem-behind-enterprise-ai) (2025-12-01) — Enterprise AI fails due to data access bottlenecks, multi-hop pipelines, and late governance. SMBC, Marsh McLennan leaders share real-world solutions.
- [Kafka Encryption: Transit, Rest, End-to-End](https://www.conduktor.io/blog/kafka-encryption) (2025-11-25) — Kafka encryption has three layers: TLS in transit, disk encryption at rest, end-to-end for payload protection. Match each to your threat model.
- [Kafka Tiered Storage: Infinite Retention Without Infinite Disks](https://www.conduktor.io/blog/kafka-tiered-storage-infinite-retention) (2025-11-21) — Configure Kafka tiered storage with S3 to reduce storage costs 3-9x. Broker config, common errors, and performance tradeoffs.
- [Kafka Automation Platform](https://www.conduktor.io/blog/kafka-automation-platform) (2025-11-18) — Kafka automation replaces ticket-based provisioning with policy-based self-service. Cut provisioning tickets 75% and deploy 4x faster.
- [Kafka Multi-Cloud: AWS, Azure, and GCP](https://www.conduktor.io/blog/kafka-multi-cloud-management) (2025-11-09) — Multi-cloud Kafka across AWS, Azure, and GCP needs unified management. One control plane for visibility, policies, and operations everywhere.
- [Stop Over-Partitioning: You Don't Need 100 Partitions](https://www.conduktor.io/blog/stop-over-partitioning-kafka) (2025-11-08) — Over-partitioned Kafka topics waste memory, slow failovers, and complicate rebalancing. The actual formula for right-sizing partitions based on throughput.
- [Kafka's Real Problem Is Governance, Not Speed](https://www.conduktor.io/blog/the-new-era-of-streaming) (2025-11-06) — Most Kafka clusters push less than 1MB/s. The challenge isn't throughput—it's building the control plane that makes streaming usable at enterprise scale.
- [Why Every Streaming Vendor Is Building a Kafka Proxy](https://www.conduktor.io/blog/the-great-confluence-of-streaming) (2025-11-05) — Streaming proxies enable governed self-service and AI-ready data at scale. Why Kafka control planes from Confluent, Kong, and Conduktor are converging.
- [Governed Kafka Self-Service](https://www.conduktor.io/blog/governed-kafka-self-service) (2025-11-02) — Self-service Kafka without governance is chaos. Policy-based automation gives developers autonomy while enforcing organizational standards.
- [Kafka Offset Management: How Consumer Groups Track Progress](https://www.conduktor.io/blog/kafka-offset-management-consumer-commit-guide) (2025-11-01) — Understand Kafka offset commits, auto-commit pitfalls, CommitFailedException, and offset reset behavior. Stop losing messages.
- [Kafka Access Management: Fix ACL Sprawl](https://www.conduktor.io/blog/kafka-access-management) (2025-10-28) — Kafka ACL sprawl makes permissions unknowable. Application-based access with periodic reviews replaces manual ACL-per-service-account chaos.
- [5 Sessions Worth Your Time at Current New Orleans 2025](https://www.conduktor.io/blog/five-can-t-miss-sessions-at-current-new-orleans-2025) (2025-10-27) — Current New Orleans 2025 highlights: OpenAI's StreamLink, Kafka protocol deep-dive, Notion's event stack, KIP-1163 diskless topics, and JPMorgan's strategy.
- [Kafka Security: Access Control That Scales](https://www.conduktor.io/blog/kafka-security) (2025-10-21) — Kafka security goes beyond TLS encryption. Scalable access control, ACL management, and provable permissions for production environments.
- [Kafka Exactly-Once: When It Works and When It Doesn't](https://www.conduktor.io/blog/exactly-once-semantics-when-it-works) (2025-10-15) — Kafka's exactly-once works within cluster boundaries. Idempotent producers, transactions, and the hard limits you need to understand.
- [Kafka Security Best Practices](https://www.conduktor.io/blog/kafka-security-best-practices) (2025-10-14) — Kafka security best practices fail without enforcement. Automate TLS, ACLs, and encryption policies so compliance is built-in, not bolted-on.
- [Kafka Proxy: Beyond ACLs for Governance](https://www.conduktor.io/blog/kafka-proxy) (2025-10-05) — Kafka proxies enforce field-level filtering, data masking, and rate limiting that native ACLs can't. Apply governance at the wire level.
- [Kafka Governance: Automated Guardrails Over Wiki Policies](https://www.conduktor.io/blog/kafka-governance) (2025-09-27) — Kafka governance fails when policies live in wikis. Automated guardrails enforce naming, retention, and schema rules before incidents reach production.
- [Kafka Cluster Management: Beyond SSH](https://www.conduktor.io/blog/kafka-cluster-management) (2025-09-19) — Managing 10+ Kafka clusters via SSH and scripts wastes senior engineering time. Centralized tooling gives visibility and control at scale.
- [Kafka Monitoring: 10 Metrics That Matter](https://www.conduktor.io/blog/apache-kafka-monitoring) (2025-09-12) — Stop drowning in 200+ Kafka broker metrics. Focus on the 10 signals that actually predict outages, consumer lag, and cluster instability.
- [Kafka as a Database: When to Use Compacted Topics for State](https://www.conduktor.io/blog/kafka-compacted-topics-as-database) (2025-09-10) — Use Kafka compacted topics as a lightweight state store. Log compaction configuration, query limitations, and when to choose a real database instead.
- [Kafka Observability: Beyond Monitoring](https://www.conduktor.io/blog/kafka-observability) (2025-09-05) — Kafka monitoring shows what's broken. Observability shows why. Combine metrics, traces, and context to diagnose root causes in minutes.
- [Kafka Testing: Beyond Production](https://www.conduktor.io/blog/kafka-testing) (2025-08-29) — Kafka testing means validating schemas, configs, and consumer logic before production. Build confidence with contract tests and local clusters.
- [Kafka Data Quality: Enforce at Write Time](https://www.conduktor.io/blog/kafka-data-quality) (2025-08-26) — Enforce Kafka data quality at write time with schema validation and policies. Garbage in at the producer means garbage out everywhere.
- [Transactional Outbox: Database-Kafka Consistency](https://www.conduktor.io/blog/transactional-outbox-pattern-database-kafka) (2025-08-25) — Solve the dual-write problem with the transactional outbox pattern. PostgreSQL setup, Debezium CDC, and idempotent consumers.
- [Kafka Performance: Measure Before You Tune](https://www.conduktor.io/blog/kafka-performance) (2025-08-19) — Kafka performance is 80% understanding your workload, 20% tuning configs. Measure bottlenecks before changing partition counts or batch sizes.
- [Custom Partitioners: When Default Hashing Isn't Enough](https://www.conduktor.io/blog/kafka-custom-partitioners-beyond-default-hashing) (2025-08-13) — Build custom Kafka partitioners for geographic routing, priority lanes, and hot key distribution. Java implementation with production-tested patterns.
- [The Real Problems with Apache Kafka: 10,000 Forum Posts Analyzed](https://www.conduktor.io/blog/apache-kafka-what-10-000-forum-posts-reveal) (2025-08-11) — 10,000+ Kafka forum posts reveal the truth: connector crashes, auth nightmares, schema failures, and cryptic errors plague production deployments.
- [Bridging Operational and Analytical Data with Kafka](https://www.conduktor.io/blog/bridging-gap-between-operational-and-analytical-data) (2025-08-01) — Kafka bridges the operational-analytical data divide in real time. Eliminate batch ETL lag and deliver fresh streaming data to analytics and AI systems.
- [Escaping Vendor Lock-In: Kafka Portability Strategies](https://www.conduktor.io/blog/kafka-vendor-lock-in-portability-strategies) (2025-07-08) — Avoid Kafka vendor lock-in with portable configurations and abstraction layers. Code examples for MSK, Confluent, and self-managed.
- [Broker Won't Start: Common Kafka Startup Failures](https://www.conduktor.io/blog/broker-wont-start-common-failures-fixes) (2025-06-19) — Kafka broker startup failures with real error messages and fixes. Cluster ID mismatch, port conflicts, disk issues, and config errors.
- [Kafka Network Partitions and Split-Brain Failures](https://www.conduktor.io/blog/kafka-network-partition-split-brain-failures) (2025-06-13) — Understand Kafka network partition failures, split-brain scenarios, and unclean leader election. ISR shrinkage and data loss prevention.
- [Strimzi on Kubernetes: From Zero to Production Kafka](https://www.conduktor.io/blog/strimzi-kubernetes-zero-to-production) (2025-04-19) — Deploy production-grade Kafka on Kubernetes with Strimzi. Separate node pools, KRaft mode, and the configuration choices that matter.
- [Real-Time Kafka Data Is Your AI Competitive Advantage](https://www.conduktor.io/blog/own-the-stream-power-the-model-monetizing-real-time-data-in-the-ai-era) (2025-04-15) — Real-time, trusted data is the new AI advantage. Discover why the future belongs to companies that treat data as a product, not a by-product.
- [Streaming at Scale: Don't Let Data Chaos Derail Your Growth](https://www.conduktor.io/blog/streaming-at-scale-don-t-let-data-chaos-derail-your-growth) (2025-03-27) — Scale streaming data with governance and quality controls. Shift left to prevent bad data, ensure AI success, and avoid architectural chaos.
- [Kafka and Flink Are the Infrastructure for AI Agents](https://www.conduktor.io/blog/ai-agents-at-scale-the-critical-role-of-kafka-and-flink) (2025-03-25) — AI agents need real-time context to act autonomously. Discover how Kafka + Flink power adaptive, intelligent systems at Netflix and beyond.
- [Audit Logging in Kafka: Who Did What and When](https://www.conduktor.io/blog/kafka-audit-logging-compliance-forensics) (2025-03-24) — Configure Kafka audit logging for SOC2, HIPAA, and PCI-DSS compliance. Authorizer logging and forensic analysis techniques.
- [80% of Enterprise Data Sits Unused. Real-Time Activation Changes That.](https://www.conduktor.io/blog/your-dark-data-is-a-goldmine-if-you-activate-it-in-real-time) (2025-03-21) — Most enterprise data never drives a decision. Real-time data activation transforms idle data into competitive advantage.
- [Avro Schema Evolution: Compatibility Guide](https://www.conduktor.io/blog/schema-evolution-avro-compatibility-guide) (2025-03-15) — Master Avro schema evolution with BACKWARD vs FORWARD compatibility modes. Compatibility matrix, curl commands for Schema Registry, and what breaks.
- [Real-Time AI Requires Kafka and Guardrails](https://www.conduktor.io/blog/how-to-succeed-with-ai-kafka-and-ai-guardrails-are-non-negotiable) (2025-03-14) — Discover how Kafka and Flink power real-time AI agents, enabling autonomous decision-making, adaptive intelligence, and scalable data streaming.
- [Dead Letter Topics: Handling Poison Pills](https://www.conduktor.io/blog/dead-letter-topics-handling-poison-pills) (2025-02-23) — Implement Kafka dead letter queues to isolate poison pill messages. Java error handling patterns, retry strategies, DLT monitoring, and reprocessing workflows.
- [Kafka Isn't a Queue: Stop Designing It Like One](https://www.conduktor.io/blog/kafka-isnt-a-queue) (2025-02-21) — Kafka's commit log differs fundamentally from message queues. The mental model shift that prevents costly anti-patterns.
- [Your First Kafka Producer: Python, Java, and Go Examples](https://www.conduktor.io/blog/first-kafka-producer-python-java-go) (2025-01-28) — Build your first Kafka producer in Python, Java, and Go. Working code examples, common connection errors, and essential configuration for reliable message de...
- [Stop Building Kafka Encryption Libraries](https://www.conduktor.io/blog/stop-building-kafka-encryption-libraries) (2025-01-25) — Client-side encryption creates ungovernable sprawl. Proxy-layer encryption centralizes control and makes Kafka a dumb pipe.
- [JVM Tuning for Kafka Brokers: G1GC vs ZGC in Production](https://www.conduktor.io/blog/kafka-jvm-tuning-g1gc-vs-zgc-production) (2024-12-10) — Configure G1GC and ZGC for Kafka brokers. Heap sizing, pause time targets, and when to switch collectors in production.
- [Kafka Headers: Metadata Without Polluting Your Payload](https://www.conduktor.io/blog/kafka-headers-metadata-without-polluting-payload) (2024-10-10) — Use Kafka headers for distributed tracing, content-based routing, and audit metadata without modifying payloads. Java and Python code examples included.
- [Kafka Team Structure: Platform Team vs Embedded Model](https://www.conduktor.io/blog/kafka-team-structure-platform-embedded) (2024-10-03) — Platform teams centralize expertise but bottleneck delivery. Embedded engineers move fast but create chaos. Here's how to choose.
- [Field-Level Encryption in Kafka: Beyond TLS](https://www.conduktor.io/blog/kafka-field-level-encryption-beyond-tls) (2024-09-07) — TLS protects data in transit, but brokers still see plaintext. Field-level encryption protects PII at rest and enables crypto shredding.
- [Kafka Streams vs ksqlDB: Choosing Right](https://www.conduktor.io/blog/kafka-streams-vs-ksqldb-choosing) (2024-09-02) — Choose between Kafka Streams and ksqlDB for stream processing. Use case comparison, team skills assessment, deployment models, and operational trade-offs.
- [Field-Level Encryption in Kafka Without Code Changes Using Conduktor Gateway](https://www.conduktor.io/blog/zero-code-encryption-strategies-for-apache-kafka) (2024-08-28) — Zero-code Kafka encryption: field-level encryption/decryption with Conduktor Gateway—secure PII, role-based access, no application changes required.
- [Apache Kafka SQL: Query Topics Like a Database in Real Time](https://www.conduktor.io/blog/apache-kafka-sql-topics) (2024-08-14) — Run SQL queries directly against Kafka topics — filter, join, and aggregate streaming data without moving it to a warehouse. Real-time SQL over Kafka.
- [KRaft Explained: Kafka Without ZooKeeper](https://www.conduktor.io/blog/kraft-explained-kafka-without-zookeeper) (2024-07-27) — How KRaft replaces ZooKeeper in Kafka 4.0. Controller quorum architecture, broker configuration examples, and step-by-step migration from ZooKeeper.
- [Kafka Topic Naming Conventions That Scale](https://www.conduktor.io/blog/kafka-topic-naming-conventions-scale) (2024-07-13) — Build a Kafka topic naming convention that scales. Structural patterns, ACL alignment, and common mistakes that break governance.
- [Schema Registry Isn't Optional](https://www.conduktor.io/blog/schema-registry-isnt-optional) (2024-07-13) — The "we'll add it later" approach costs weeks of debugging and painful migrations. Schema Registry is day-one infrastructure.
- [Event sourcing with Kafka: patterns and pitfalls](https://www.conduktor.io/blog/event-sourcing-kafka-patterns-pitfalls) (2024-07-03) — Build event-sourced systems on Kafka. Topic design, aggregate reconstruction, CQRS projections, and the gotchas that break production.
- [PagerDuty's Kafka Outage: Lessons from 4.2 Million Rogue Producers](https://www.conduktor.io/blog/pagerduty-kafka-outage-lessons) (2024-06-28) — A code pattern created 4.2M Kafka producers per hour, crashed the cluster, and silenced alerts for 9+ hours. Dissecting the postmortem.
- [When NOT to Use Kafka: 5 Scenarios Where Simpler Wins](https://www.conduktor.io/blog/when-not-to-use-kafka) (2024-05-21) — Kafka adds complexity you may not need. Five scenarios where PostgreSQL LISTEN/NOTIFY, Redis Streams, SQS, or simple webhooks outperform Kafka.
- [Kafka Retention: When Messages Disappear and Why](https://www.conduktor.io/blog/kafka-retention-policies-when-messages-disappear) (2024-05-06) — Master Kafka retention—retention.ms, segment.ms, and why messages persist longer than expected. Debug commands included.
- [Build Idempotent Kafka Consumers: Patterns That Actually Work](https://www.conduktor.io/blog/building-idempotent-consumers) (2024-04-26) — Handle duplicate Kafka messages gracefully. Database constraints, Redis lookups, and the deduplication patterns that scale.
- [GDPR and Kafka: Right to Erasure](https://www.conduktor.io/blog/gdpr-kafka-right-to-erasure) (2024-04-24) — Handle GDPR Article 17 deletion requests in Kafka's append-only log. Crypto shredding implementation, tombstone patterns, and compliant retention strategies.
- [What Kafka Users Actually Care About in 2024: Kafka Summit London Recap](https://www.conduktor.io/blog/kafka-summit-london-2024) (2024-03-29) — Kafka Summit London 2024 highlights: real-time analytics, GitOps self-service, SQL over Kafka, and the shift from batch to streaming data.
- [When Kafka Exactly-Once Semantics Are Worth the Performance Cost](https://www.conduktor.io/blog/kafka-exactly-once-performance-cost) (2024-03-23) — Kafka exactly-once semantics add latency and complexity. When the cost is worth paying, and when at-least-once with idempotent consumers wins instead.
- [Partition Count: The Decision You Can't Undo](https://www.conduktor.io/blog/partition-count-decision-guide) (2024-03-01) — How to choose Kafka partition count. Sizing formulas, overhead costs, and why decreasing partitions requires topic migration.
- [Conduktor and Confluent Kafka Meetups Across the USA, February 2024](https://www.conduktor.io/blog/confluent-conduktor-usa-tour) (2024-02-08) — Join Conduktor's February 2024 USA tour in NYC, Boston, Bay Area, and Seattle. Learn about Kafka poison pills, schema compatibility, and data contracts.
- [Kafka Platform: Build vs Buy Decision Framework](https://www.conduktor.io/blog/kafka-build-vs-buy-decision-framework) (2024-01-26) — Framework for evaluating self-managed Kafka vs managed services like Confluent Cloud and MSK. TCO analysis, team skills assessment, and decision criteria.
- [Disk Full: Emergency Recovery When Kafka Runs Out of Space](https://www.conduktor.io/blog/disk-full-emergency-recovery) (2024-01-17) — Emergency runbook for Kafka disk full scenarios. Immediate triage commands, safe segment deletion, recovery steps, and retention tuning to prevent recurrence.
- [Hierarchical Tagging for Kafka: Cost Attribution, Ownership, and Governance](https://www.conduktor.io/blog/kafka-hierarchical-tagging) (2024-01-16) — Transform data chaos with strategic tagging. Enable Data Mesh, ABAC security, and better analytics through hierarchical metadata organization.
- [Multi-Region Kafka: Active-Active vs Active-Passive](https://www.conduktor.io/blog/multi-region-kafka-active-active-passive) (2024-01-11) — Compare Kafka DR patterns. MirrorMaker 2 setup, offset translation, conflict resolution, and when each architecture makes sense.
- [SOC2 Type II Certification: One Year of Writing Policies and Proving We Follow Them](https://www.conduktor.io/blog/what-we-learned-from-soc2-type2-write-what-you-do-do-what-you-write) (2024-01-02) — SOC2 Type II journey: policies, pentesting, CVE management, agent monitoring, and compliance automation with Drata—lessons from Conduktor's certification.
- [Real-Time SQL on Kafka with PySpark](https://www.conduktor.io/blog/getting-started-with-pyspark-and-kafka) (2023-09-17) — Run real-time SQL queries on Kafka with PySpark. Stream processing tutorial with JSON transformations, ChatGPT integration, and data generation.
- [Why Conduktor Uses Redpanda for Local Kafka Development](https://www.conduktor.io/blog/conduktor-redpanda-best-breed-kafka-experience) (2023-08-10) — Conduktor uses Redpanda for local development because it starts instantly, weighs 130MB, includes Schema Registry, and eliminates ZooKeeper complexity.
- [Conduktor Now Available on AWS Marketplace for MSK Users](https://www.conduktor.io/blog/conduktor-available-on-the-aws-marketplace-to-complement-msk) (2023-08-02) — Deploy Conduktor on AWS Marketplace for MSK with IAM, Glue Schema Registry support, and seamless ECS/EKS integration in minutes.
- [Kafka Producer & Consumer Config: 5 Performance Settings That Matter](https://www.conduktor.io/blog/kafka-producer-consumer-performance-config) (2023-07-26) — Master Kafka performance: partition sizing, SerDe optimization, dual-write solutions, tombstones, and producer tuning for robust applications.
- [What Kafka Users Actually Asked Us at Kafka Summit London 2023](https://www.conduktor.io/blog/kafka-summit-london-2023-feedback-from-our-booth) (2023-05-25) — Field-level encryption, multi-tenancy, topic-as-a-service, and RBAC dominated conversations at our booth. Here's what the community cares about most.
- [Data Mesh Explained: Decentralized Data Ownership for Scalable Organizations](https://www.conduktor.io/blog/what-is-a-data-mesh) (2023-05-04) — Data mesh decentralizes ownership for agility and scale. Learn core principles, architecture, and how Kafka + Conduktor enable domain-driven data products.
- [Kafka RBAC: Role-Based Access Control for Teams and Applications](https://www.conduktor.io/blog/security-for-humans-and-applications-in-apache-kafka) (2023-04-08) — Kafka RBAC with fine-grained permissions for humans and applications. Manage producer, consumer, and admin access without modifying broker-level ACLs.
- [Data Integration: How Merging Datasets Drives Better Business Decisions](https://www.conduktor.io/blog/data-integration-the-what-why-and-benefits) (2023-03-09) — Master data integration for big data success. Explore ETL vs ELT processes, data quality benefits, and automation strategies for business intelligence.
- [Change Data Capture Explained: How CDC Streams Database Changes in Real-Time](https://www.conduktor.io/blog/what-is-change-data-capture-anyways) (2023-03-09) — Change Data Capture (CDC) explained: real-time database replication, log-based streaming, push vs. pull methods, and Kafka CDC implementation patterns.
- [Automate Kafka Topic Governance with Topic as a Service](https://www.conduktor.io/blog/topic-as-a-service-automate-governance-decisions-for-apache-kafka) (2023-01-31) — Automate Kafka topic governance and ACL management with Topic as a Service—eliminate bottlenecks, enforce standards, free your ops team.
- [Kafka Schema Registry: How It Works & Why You Need It](https://www.conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it) (2022-11-24) — Schema Registry enforces data contracts in Kafka, preventing schema drift and producer-consumer mismatches. Covers serialization, Avro, Protobuf, evolution r...