# Partition Count: The Decision You Can't Undo

Kafka lets you increase partitions anytime. It does not let you decrease them.

This asymmetry makes partition count one of the most consequential decisions in your architecture. Get it wrong, and you're either leaving performance on the table or facing a painful migration.

> *We defaulted to 100 partitions per topic "for future scale" on a cluster processing 10 MB/s. Two years later, we're still migrating topics to right-sized partition counts.*
>
> *Platform Engineer at a logistics company*

## Why It's Irreversible

Kafka stores messages by partition. Decreasing partitions would require redistributing messages, breaking ordering guarantees.

```text
# Key "user-123" hashes to partition 2 with 6 partitions
hash("user-123") % 6 = 2

# Same key hashes to partition 0 with 4 partitions
hash("user-123") % 4 = 0

# Messages split across partitions. Ordering lost.
```

The only way to reduce partitions: create a new topic, migrate data, switch producers/consumers. A multi-day operation for production topics.

## The Sizing Formula

```text
Partitions = max(Target_Throughput / Producer_Rate, Target_Throughput / Consumer_Rate)
```

**Example:** You need 1 GB/s. Producers achieve 100 MB/s per partition. Consumers process 50 MB/s per partition.

```text
Producer: 1000 / 100 = 10 partitions
Consumer: 1000 / 50 = 20 partitions
Result: 20 partitions minimum
```

Test your actual throughput:

```bash
kafka-producer-perf-test.sh --topic test --num-records 1000000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=localhost:9092 acks=all
```

## Parallelism Ceiling

Each partition can have at most one consumer per group:

```text
Max parallel consumers = Number of partitions
```

6 partitions, 10 consumers → 4 sit idle. Plan for growth: if you expect 20 consumers, start with at least 20 partitions. [Monitoring partition distribution](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) helps identify these imbalances before they cause issues.

## Hidden Costs of Over-Provisioning

Each partition requires:
- 10 MB pre-allocated index files
- Open file handles (3-6 per partition)
- Metadata in the controller

**File handles:** 5,000 partitions per broker = 15,000-30,000 file handles. Most Linux defaults to 1,024. Set `ulimit -n` to 100,000+.

**Controller failover:**

| Partitions | Approx. Failover Time |
|------------|----------------------|
| 1,000 | ~2 seconds |
| 10,000 | ~20 seconds |
| 100,000 | ~3+ minutes |

**Limits:**

| Scope | Soft Limit | Hard Limit |
|-------|------------|------------|
| Per broker | 2,000 | 4,000 |
| Per cluster | 50,000 | 200,000 |

## Rebalancing Impact

Consumer rebalances redistribute partitions. More partitions = longer rebalance duration.

```properties
# Enable cooperative rebalancing (Kafka 2.4+)
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
```

For stable groups, static membership prevents rebalances on restart:

```properties
group.instance.id=consumer-1
session.timeout.ms=300000
```

## Practical Guidelines

**Small cluster (< 6 brokers):** 3 × Broker_Count per topic

**Large cluster (> 12 brokers):** 2 × Broker_Count per topic

**Avoid prime numbers.** Use 6, 8, 12, 16, 24, 48—divisible by common consumer counts.

## When You've Over-Partitioned

Signs: controller elections > 30 seconds, rebalances > 60 seconds, URPs spike during rolling restarts.

**Mitigation:** Add brokers, upgrade to KRaft, tune controller settings.

**Full migration (last resort):**

```bash
kafka-topics.sh --create --topic my-topic-v2 --partitions 12
# Mirror data, switch producers, drain old topic, switch consumers, delete old topic
```

When in doubt, slightly over-provision. Adding partitions is easy; removing them is not.

[Book a demo](https://www.conduktor.io/contact/demo) to see how Conduktor Console shows partition distribution and consumer lag.