Apache Kafka has become indispensable infrastructure for real-time data, powering everything from customer-facing applications to analytics to AI across every industry. But operating Kafka at enterprise scale is a different problem from adopting it, and the difference is where most organizations get caught out.
Through conversations with current customers and prospects, architects, product managers, tech leads, and executives across sectors ranging from logistics to finance, Conduktor has seen the same five obstacles surface again and again. They are not exotic edge cases. They are the predictable consequences of running Kafka without the governance, self-service, cost visibility, and integration layer it does not ship with.
5Recurring challenges leaders hit at scale
700Connectors deployed by one team with self-service
2,000Topics synced across those connectors
The through-line: technology is only as good as the people and processes around it. Without the right practices and governance, teams struggle with Kafka, fail to tap its full power, and can end up worse off than before. This ebook walks through each challenge, what it looks like in the field, and the benefits organizations realize when they solve it.
Part 1 covers poor quality data, the challenge that undermines every downstream initiative. The four that follow are gated: slow and overly manual workflows, the disconnect between operational and analytical teams, mystery costs and zombie infrastructure, and the gap between legacy and cloud technologies. A closing section shows how Conduktor addresses all five.
For organizations, every major initiative, including AI, analytics, compliance, and digital transformation, requires trustworthy, accurate data to execute. Unfortunately, many Kafka pipelines ingest poor quality data at scale, introducing inconsistent, misformatted, and missing data into downstream systems, with disastrous results. This produces low-quality outputs, reduces confidence among both internal and external users, impacts mission-critical applications, and can lead to outages or disruptions.
What the challenge looks like
The failure mode is rarely a single broken pipeline. It is the slow erosion of trust that happens when nobody owns data quality and nothing blocks bad data at the door.
- Blurry ownership. At one national postal operator, "blurry borders" around team responsibilities for data quality, schema governance, and data retention left key duties unattended. Owners were nominally responsible for their data meeting specific standards, but due to a lack of accountability and ill-defined requirements, the task was neglected. Downstream teams stopped trusting the data.
- No way to block bad data. A technical lead at a logistics company described how misformatted data fields at ingestion would immediately break something on the consumer side, usually something powering a business-critical function. With no way to block the intake of that data, developers and end users saw only broken applications and blamed Kafka.
- Schema registries are not enough. Even organizations using schema registries, the traditional solution, hit major issues, specifically a lack of visibility and validation. Data scientists at one retailer could only identify issues after they appeared and impacted KPIs, because they had no way to monitor messages within Kafka itself.
Without a way to block the intake of bad data, developers and end users see only broken applications, and Kafka takes the blame for a governance gap.
What solving it delivers
By implementing clear ownership, in-stream observability, and automatic enforcement, teams can stop low-quality data from entering applications in the first place. That shift produces three concrete outcomes.
The payoff of blocking bad data at the door
- Improved trust. High quality data creates high quality outputs, true for everything from AI to analytics. That increases trust among colleagues and customers alike, who can use results and applications with confidence.
- Reduced and reallocated time and money. Preempting bad data saves the hours spent on reactive approaches like migrations or lift-and-shift. Employees pivot to other work, increasing productivity and innovation.
- Meeting budgets, KPIs, and SLAs. Uptime and accuracy are two SLAs and KPIs directly threatened by low-quality data. Blocking it up front improves application reliability, meets key goals, and avoids associated financial penalties.
Developers, analysts, and product teams need data now. If they have to wait, blocked by overly manual processes and unclear governance, the business suffers. Many organizations lack the systems, standards, and processes to help internal users access data rapidly and safely.
What the challenge looks like
Platform engineers are responsible for performance, stability, and security, so they build in guardrails to reduce risk. But without automation or self-service, those guardrails become bottlenecks. When developers and analysts wait too long, productivity drops and inter-team tension builds.
The manual path
- At one national postal operator, requesting operational data to be persisted for consumption took weeks.
- Tickets were filed with the platform team.
- The platform team created and configured connectors and S3 buckets.
- Platform teams then had to validate connector permissions before anyone could consume data.
- The result: an overly manual, frustrating process for everyone involved.
The self-service path
- One platform team at a multinational retailer built a one-click system to move data to end users for analysis.
- With it, they deployed 700 connectors.
- They synced 2,000 topics across those connectors.
- Guardrails stayed in place; the bottleneck did not.
What solving it delivers
By implementing automation, standardization, and self-service, an organization helps analysts and developers move faster while reducing the workload on platform teams.
Benefits of self-service with guardrails
- Faster time-to-value and time-to-market. Analysts and developers moving faster means more rapid results. Products and features get created, tested, and released more quickly, and decision makers access insights to stay ahead of the competition.
- Scale operations without chaos. Self-service with guardrails lets data usage grow without throwing operations into disarray.
- Improved business agility. Reducing platform friction increases developer throughput and lets the business pivot faster on market trends, competitor weaknesses, and changing economic conditions.
- Reduced overhead and employee turnover. Stressed, overworked employees leave. Automating self-service with guardrails reduces administrative overhead, so platform engineers focus on more fulfilling work and stay longer.
- Clear ownership and procedures. Consistent, repeatable workflows save time and money. Teams stop reinventing the wheel and focus on the high-value work they were hired to do.
Organizations continue to struggle with silos, specifically the separation between operational and analytical systems, and the teams that run them. Operational data such as credit card payments, delivery vehicle movements, or IoT sensor data remains locked within technologies such as Kafka or application databases, while analytics teams work in data lakes such as Snowflake, BigQuery, or Databricks.
What the challenge looks like
The divide is technical and human at the same time. Developers and analysts may not share the same governance, procedures, or even understanding of data, which breaks down trust between them.
- Analysts lack a way to explore and access streaming data, and lack knowledge of how it is ingested, leading them to blame developers for not building better, more consistent pipelines.
- Developers, in turn, resent analysts for constant requests for access, quality enforcement, and discoverability.
- Analysts cannot find or get timely data, forcing them to work with stale or incomplete data and risk skewing their results.
- Without the right tools, analysts and data scientists may not even know about missing data in the first place, an unknown unknown that further exacerbates flaws in analytics, AI, and decision making.
To overcome these barriers, organizations need a common domain for different data systems and products: essentially a shared data language and structure that all teams and business units are familiar with and participate in. This also clearly defines owners for specific components, such as pipelines or topics, and controls, such as data quality or encryption.
What solving it delivers
At an organizational level, implementing shared technologies and bridging the breakdown between operational and analytical systems, and teams, brings key benefits.
Benefits of bridging the operational-analytical divide
- Faster, better insights. A company that can use more of its data generates high-volume, high-quality insights more rapidly than competitors, with positive implications for innovation, organizational agility, and revenue.
- Higher trust and cohesion. Removing the obstacles between operational and data teams improves morale and cooperation. Clear accountability builds more efficient workflows and removes old cycles of recrimination and blame.
- Better governance for AI, and more accurate results. AI is only as good as its inputs. Ensuring governance and access maintains the integrity, security, and relevance of training data, leading to more reliable, precise outputs.
- Decreased risk. Improving compliance and auditing, in the form of automatic policy enforcement, RBAC, and user logs, reduces the possibility of violating key legislation such as HIPAA or GDPR.
Kafka usage rarely grows in a logical, planned manner. Instead, it grows organically, with topics and schemas being created, partitioned, used, and abandoned, often without documentation or visibility. When migrations or re-factoring initiatives occur, these excess topics, schemas, and partitions might be forgotten entirely. The problem becomes even more entrenched if teams fail to establish clear ownership and naming standards at the time of creation.
What the challenge looks like
While some topics, partitions, and schemas may be revived, many remain dormant, seeing minimal usage even as they continue to generate costs. This zombie infrastructure often accumulates in legacy or non-production environments, where it remains unnoticed by busy teams. Without any way to identify and remove it, teams waste valuable resources and drive up expenditures, without even knowing why.
How teams respond
Automation and observability go a long way toward resolving this. The teams that get ahead of it apply two moves.
Automated cleanup and tagging
- One team created a seven-day cleanup policy in their dev environment, which was neither production nor customer facing, deleting all unused assets after that period.
- They used improved monitoring to tag usage by environment, project, and team, ensuring better visibility for financial planning.
Standardized guardrails
- To keep expenses down, one organization limited self-service users to 10 partitions each.
- Any additional partitions triggered a request that had to be manually approved.
- Standardizing and implementing policies across an environment keeps drift in check.
What solving it delivers
Ultimately, these solutions result in the same thing for organizations: realizing operational and financial efficiencies.
Benefits of removing zombie infrastructure
- Decreased cloud and platform spend. More detailed visibility into financial data helps leaders understand where money is wasted and operate more efficiently, and improves budgeting and forecasting so resources are allocated more effectively.
- Increased reliability and responsiveness for business-critical applications. Cutting unused, excessive infrastructure boosts performance, with lower latency and fewer incidents arising from clutter or misconfigured assets, so applications and products become faster and more reliable.
- Lowered risk. Zombie infrastructure may contain sensitive data and legacy configurations, making it doubly vulnerable to hacks and leaks. Removing it, or even updating retention policies, reduces organizational exposure and improves compliance postures.
- Higher team velocity and shorter time-to-market. Streamlining Kafka architectures means developers deal with fewer irrelevant assets, so teams onboard and work faster, cutting development cycles.
Kafka is not limited to ingesting streaming data. It has moved beyond that niche, becoming a central nervous system that bridges the divide between legacy systems and modern microservices architectures.
What the challenge looks like
Otherwise obsolete technologies such as Managed Queues (MQ), Managed File Transfer (MFT), and SOAP/XML remain within digital environments for a number of reasons. They may power mission-critical applications or complex workflows, or simply be hidden away from view. Whatever the case, these technologies often create fragile, undocumented dependencies, and removing them could lead to outages or worse.
Kafka is the solution, serving as a strategic integration layer with its rich ecosystem of source and sink connectors. Because it can stream data asynchronously, delivering events from one producer to multiple consumers, it frees up data environments from the limits of point-to-point transmission. This flexibility is ideal for mixed architectures, allowing interoperability without the need to create, configure, and maintain custom connectors or pipelines.
Even so, using Kafka as a strategic backbone presents challenges. Without the right systems and standards, teams duplicate efforts, misuse data, or struggle to find and trust the streams they need. The absence of clear ownership and access policies can lead to bottlenecks or accidental exposure of sensitive data.
What solving it delivers
Assuming organizations can standardize, manage, and govern their Kafka infrastructure, they can fill the chasm between obsolete, holdover technologies and newer ones.
Benefits of Kafka as a governed integration layer
- Future-proofing your environment. Outdated systems hold organizations back from moving forward competitively. Using Kafka as the connective tissue lets teams progress without abandoning vital, if older, infrastructure.
- Migrate and upgrade without disruptions. Streaming data between systems via Kafka buys time for teams to migrate on their own terms, with minimal impact on business-critical or customer-facing applications.
- Unify data across platforms and clouds, and remove vendor lock-in. Sharing data via Kafka enables it to move seamlessly across clouds, platforms, technologies, and ecosystems, removing silos by provider and freeing teams to work with the most optimal solutions.
- Increase efficiency and ROI via reuse and standardization. Using Kafka as connective tissue means teams stop building redundant pipelines per business unit or use case, and instead consume data as needed, centralizing governance and speeding up developer productivity.
Apache Kafka has become essential infrastructure for modern digital environments, and the organizations that run them. But while Kafka moves data exceptionally well, it does not include features for managing that data securely, efficiently, or at scale. Kafka also lacks native tools for governance, cost control, access management, and collaboration, making it risky and expensive for enterprises to use without additional control.
Conduktor fills that gap. As a data hub built specifically for Kafka, Conduktor helps organizations transform Kafka from a technical enabler into a true business asset, making it secure, scalable, and aligned to the goals of the enterprise. Each of the five challenges maps to a capability the platform provides.
| Challenge | How Conduktor addresses it |
|---|
| Poor quality data | In-stream observability and validation to improve data quality and prevent downstream issues. |
| Slow, manual workflows | Standardized self-service abilities, so developers and analysts access data quickly without compromising governance. |
| A disconnect between teams | Centralized catalogs and metadata tagging that make Kafka data easier to find, understand, and use across teams. |
| Mystery costs and zombie infrastructure | Identifies unused topics, excessive partitions, and long-retention policies that quietly accumulate, giving platform teams visibility and levers to clean up without disrupting active pipelines. |
| Fragmented legacy and cloud tech | Enables Kafka to serve as a strategic integration layer across cloud and on-prem environments, replacing brittle point-to-point connections and reducing vendor lock-in through standardized, Kafka-native data sharing. |
The business impact
- Accelerate time-to-value and reduce platform and cloud costs.
- Strengthen compliance and unlock more value from real-time data.
- Move faster, stay secure, and scale Kafka without losing control.
If you are ready to bring clarity, control, and confidence to your Kafka, we would love to show you how Conduktor can help.
Ready to operationalize Kafka at scale?
See how Conduktor brings governance, self-service, cost control, and integration to Kafka in one platform. Book a demo and we will walk through the challenges that apply to your estate and show you how to solve them.
Book a demo
