# Kafka Network Partitions and Split-Brain Failures

Network partitions are inevitable. Kafka handles them better than most systems, but only if you configure your cluster correctly.

I've seen teams lose data because they didn't understand ISR shrinkage. The failure mode is subtle, and the defaults don't protect you.

> *We lost 500 messages during a network partition. Our topic had `min.insync.replicas=1`. After setting it to 2, we've had zero data loss incidents.*
>
> *Platform Engineer at a fintech company*

## How Kafka Handles Partitions

When a broker is isolated:
1. After `replica.lag.time.max.ms` (default 30s), followers leave ISR
2. If the leader is isolated, a new leader is elected from remaining ISR
3. Producers with `acks=all` block until enough replicas acknowledge

```bash
# Check for degraded replication
kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions
```

Any output means replication is degraded. Alert on this immediately. Set up [proactive alerting](https://docs.conduktor.io/guide/monitor-brokers-apps/alerts) to catch ISR shrinkage before it causes data loss.

## The Split-Brain That Causes Data Loss

Here's the dangerous sequence with `min.insync.replicas=1`:

1. Two followers fall behind; leader shrinks ISR to itself
2. Network isolates the leader
3. Old leader accepts writes before detecting isolation
4. Controller elects a new leader; new leader also accepts writes
5. Partition heals; old leader's writes are **truncated**

```text
[Partition payments-0] Truncating log to offset 15000 (was 15500)
# 500 messages lost
```

**Fix:** Set `min.insync.replicas=2`:

```bash
kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name payments \
  --add-config min.insync.replicas=2
```

**Tradeoff:** With `min.insync.replicas=2` and 3 replicas, losing 2 brokers makes the partition unavailable for writes. You're trading availability for consistency.

## Unclean Leader Election

When all ISR replicas are unavailable, Kafka chooses:
1. Wait for ISR member (no data loss, partition offline)
2. Elect out-of-sync replica (data loss, partition available)

Controlled by `unclean.leader.election.enable` (default: `false` since Kafka 0.11).

**Keep disabled** for financial transactions, audit logs, anything that can't handle gaps.

**Enable** for metrics, telemetry, topics that can be rebuilt.

## Configuration for Durability

```properties
# Topic
replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

# Producer
acks=all
enable.idempotence=true
```

**Why this works:**
- 3 replicas tolerates 1 broker failure
- `min.insync.replicas=2` ensures writes on 2+ brokers
- `acks=all` waits for all ISR replicas
- Idempotent producer prevents duplicates

## Metrics to Alert On

| Metric | Warning | Critical |
|--------|---------|----------|
| UnderReplicatedPartitions | > 0 for 1 min | > 0 for 5 min |
| OfflinePartitionsCount | > 0 | > 0 |
| ActiveControllerCount | != 1 | != 1 |
| IsrShrinksPerSec | > 1/min | > 5/min |

## NotEnoughReplicasException

```text
Messages are rejected since there are fewer in-sync replicas than required.
```

This is Kafka doing its job—refusing writes that can't be safely replicated. Check ISR state and broker health.

Network partition handling is where Kafka's distributed nature becomes visible. The defaults favor safety over availability, which is correct for most workloads.

[Book a demo](https://www.conduktor.io/contact/demo) to see how Conduktor Console provides real-time ISR health and partition status monitoring.