# When NOT to Use Kafka: 5 Scenarios Where Simpler Wins

I've consulted for teams debugging consumer group rebalances at 3 AM for systems processing 50 messages per hour. Kafka is powerful. It's also frequently overkill.

Teams adopt Kafka because they've heard it handles "big data" or because a blog post made it sound like the obvious choice. Six months later, they're managing a distributed system for traffic a PostgreSQL table could handle.

> *We spent months configuring brokers, Schema Registry, and ZooKeeper for 2,000 messages per day. Then we realized: that's 1.4 messages per minute. A cron job would've worked.*
>
> *Engineering Manager at a consulting firm*

## Scenario 1: Low Volume (Under 10K Messages/Day)

At 2,000 messages/day, you're processing 1.4 messages per minute. The infrastructure overhead of Kafka—brokers, partitions, consumer groups, monitoring—isn't justified.

**Better alternatives:**

```sql
-- PostgreSQL as a queue with advisory locks
SELECT * FROM tasks
WHERE status = 'pending'
  AND pg_try_advisory_xact_lock(id)
ORDER BY created_at LIMIT 1
FOR UPDATE SKIP LOCKED;
```

AWS SQS or GCP Pub/Sub provide managed queuing without infrastructure to maintain.

## Scenario 2: Fire-and-Forget Notifications

Real-time notifications, typing indicators, live dashboards—these don't need durability. If a user misses a "someone is typing" message, nobody cares.

Kafka's durability is wasted: messages written to disk, offsets tracked, retention configured.

**Better alternative:** Redis Pub/Sub is fire-and-forget by design. No persistence, no offset management. Sub-millisecond latency.

| Requirement | Redis Pub/Sub | Kafka |
|-------------|---------------|-------|
| Message persistence | No | Yes |
| Replay capability | No | Yes |
| Operational complexity | Low | High |
| Latency | Sub-millisecond | Milliseconds |

## Scenario 3: Request-Reply Patterns

Service A publishes a request, Service B responds on a different topic, Service A correlates by ID. This works. It's also fighting Kafka's design.

**The problems:** Correlation complexity, batching delays, response timeouts.

**Better alternative:** HTTP/gRPC. Simpler, faster, easier to debug.

## Scenario 4: Simple Batch Processing

A nightly ETL job extracts from a database, transforms, and loads elsewhere. Adding Kafka in the middle adds infrastructure, failure points, and exactly-once configuration for a single consumer.

**What you actually need:**

```python
rows = source_db.execute("SELECT * FROM orders WHERE date = %s", [yesterday])
s3.put_object(Body=rows.to_parquet(), Key=f'orders/{yesterday}.parquet')
```

Signs you're using Kafka as a glorified file system: one producer, one consumer, polling once per hour, waiting for "end of batch" markers.

## Scenario 5: Two-Service Architectures

Two services need async communication. Kafka adds 3+ broker nodes, Schema Registry, monitoring, on-call rotations, partition management.

**Better alternatives:**

- **SQS:** No infrastructure, built-in retries, dead-letter queues, pay-per-message
- **Webhooks:** If Service B exposes an HTTP endpoint, a webhook with retry logic is often enough

## When Kafka IS Right

Kafka genuinely solves hard problems:

- **High throughput:** Hundreds of thousands to millions of events/second
- **Event sourcing:** Replaying history to rebuild state
- **Multi-consumer patterns:** 10 services consuming the same events, each with its own offset
- **Decoupling at scale:** Hundreds of microservices across teams
- **Compliance and audit:** Immutable event logs for regulated industries

## Decision Framework

| Question | If No → Consider |
|----------|------------------|
| > 10K messages/day? | PostgreSQL, SQS |
| Multiple independent consumers? | Direct calls, webhooks |
| Need to replay historical events? | Simple queue + database |
| Need sub-100ms latency? | Redis, direct calls |
| Team has streaming experience? | Managed services |
| System will grow to 10+ services? | Start with queues, migrate later |

## The Real Cost

Kafka's complexity isn't free: broker configuration, partition rebalancing, disk management, Kafka-specific expertise, specialized debugging tools, minimum 3 brokers for production. When Kafka is the right choice, [centralized management](https://docs.conduktor.io/guide) reduces operational overhead significantly.

Kafka is powerful when you need it. The skill is knowing when you don't.

[Book a demo](https://www.conduktor.io/contact/demo) to see how Conduktor Console provides centralized visibility and governance without adding more infrastructure.