# Kafka Producer & Consumer Config: 5 Performance Settings That Matter

Apache Kafka powers most distributed streaming platforms and real-time data pipelines. Getting these five things right determines whether your Kafka applications scale smoothly or collapse under load.

## How Partition Count Breaks Parallelism and Resource Utilization

Partition count is one of the most impactful decisions you will make. Get it wrong in either direction and you will hit performance walls.

**Too few partitions causes:**

- **Limited parallelism**: Kafka achieves high throughput by allowing multiple consumers to process messages in parallel across partitions. Fewer partitions means fewer consumers can work simultaneously, reducing throughput.
- **Wasted broker resources**: Partitions distribute across broker nodes. With too few partitions, brokers sit idle while others are overloaded.
- **Scaling ceiling**: You cannot have more active consumers than partitions. Too few partitions limits how far you can scale your consumer group.

**Too many partitions causes:**

- **Metadata overhead**: Each partition requires offset tracking, replication status, and leader election coordination. This consumes storage and memory.
- **Higher latency**: More partitions means more coordination overhead during rebalancing and more file handles to manage.
- **Operational burden**: Monitoring and maintaining thousands of partitions becomes error-prone and time-consuming.

The right number depends on your expected throughput, consumer scalability needs, and cluster capacity. Monitor partition metrics and adjust based on actual workload.

Conduktor [enforces topic limits at creation time](https://marketplace.conduktor.io/interceptors/safeguard-create-topic-policy/), preventing partition count mistakes and [reducing infrastructure costs](https://www.conduktor.io/blog/8-new-ways-to-drastically-reduce-your-kafka-costs#how-are-partitions-and-costs-related).

## Why Serialization Format Choice Determines System Maintainability

Kafka treats messages as opaque byte arrays. Your serialization choices determine whether your system stays maintainable as schemas evolve.

**Choose the right format for your constraints:**

- **Space and processing time**: JSON is human-readable but verbose. Avro and Protocol Buffers are compact and fast. Pick based on whether you need debuggability or efficiency.
- **Cross-client compatibility**: Producers and consumers must use compatible serialization libraries. A Python producer and Java consumer both need to understand the same format and version.
- **Schema evolution**: Plan for field additions, removals, and default value changes. Understand backward compatibility (old consumers read new messages) and forward compatibility (new consumers read old messages).

**Optimize for production:**

- Use compact formats like Avro or Protobuf combined with compression (GZIP, Snappy) to reduce network bandwidth.
- Implement robust error handling during serialization. Route malformed messages to a dead letter topic for later recovery.
- Validate incoming messages against expected schemas to catch corrupt data before processing.

Use a schema registry. Conduktor works with all major registries including Confluent, AWS Glue, and Aiven.

![Diagram of a Kafka producer using a schema registry (Confluent, AWS Glue, Aiven) with Avro or Protobuf and compression](https://www.conduktor.io/assets/images/blog/conduktor-schema-registry-confluent-glue-aiven.png)

## The Transactional Outbox Pattern Solves Dual-Write Inconsistency

Dual writes cause data inconsistency. When a service updates a database and publishes to Kafka, partial failures leave systems out of sync. If the database commit succeeds but Kafka publish fails, your systems disagree about reality.

The [Transactional Outbox pattern](https://microservices.io/patterns/data/transactional-outbox.html) solves this by making the database the single source of truth:

1. Write changes to your source table and an Outbox table within a single database transaction
2. A separate process polls the Outbox table and publishes events to Kafka
3. Either both writes succeed or neither does

![Diagram of the Transactional Outbox pattern: a service writes business state and an Outbox row in one DB transaction, then a relay polls and publishes to Kafka](https://www.conduktor.io/assets/images/blog/kafka-transactional-write-two-phase.png)

Change Data Capture tools like [Debezium](https://debezium.io/) implement this pattern by capturing database changes and producing them as Kafka events. This keeps Kafka and downstream systems synchronized.

High-availability scenarios add complexity. See [this detailed analysis of Debezium with PostgreSQL replication](https://www.sderosiaux.com/articles/2020/01/06/learnings-from-using-kafka-connect-debezium-postgresql/#double-replication) for edge cases you will encounter.

Proper tooling for monitoring, troubleshooting, and auto-restarting failed connectors is essential.

![Diagram of Kafka Connect monitoring: detecting failed connectors, alerting and auto-restarting them to keep Debezium streams healthy](https://www.conduktor.io/assets/images/blog/kafka-connect-monitoring-restart-flow.png)

## Tombstones: How Kafka Handles Record Deletion

Log compaction (`cleanup.policy=compact`) retains only the latest value per key. A message with a key and null payload is a **tombstone**, a delete marker.

![Diagram of Kafka log compaction with a tombstone record: a key with null payload removes prior values for that key after compaction](https://www.conduktor.io/assets/images/blog/kafka-log-compaction-tombstone-example.png)

Tombstones remove all prior messages with that key. After a retention period, Kafka removes the tombstone itself to free space.

Your consumers must handle tombstones correctly:

- Remove data from internal stores or caches (Kafka Streams state stores)
- Propagate deletions to downstream systems via their APIs

Configure [min.cleanable.dirty.ratio](https://www.conduktor.io/kafka/kafka-options-explorer/?search=min.cleanable;dirty.ratio) to control how long tombstones remain before compaction. Give consumers enough time to process them before they disappear.

See [Kafka Options Explorer](https://www.conduktor.io/kafka/kafka-options-explorer/?search=cleaner) and [compaction documentation](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction) for configuration details.

## Producer Configuration Settings That Actually Matter

Producer configuration directly impacts throughput, latency, and reliability.

**Use asynchronous sends**: If you call `kafkaFuture.get()` after every send, you are blocking. Asynchronous sends let the producer continue processing without waiting for broker acknowledgment.

**Enable batching and compression**: [batch.size and linger.ms](https://www.conduktor.io/kafka/kafka-producer-batching) control how messages are grouped before sending. Larger batches reduce network roundtrips. [compression.type](https://www.conduktor.io/kafka/kafka-message-compression) (GZIP, Snappy, LZ4) reduces data transfer and storage costs with minimal CPU overhead.

**Handle backpressure**: When brokers cannot keep up with your send rate, you need to detect and respond. Monitor broker response times and adjust send rates accordingly.

**Configure buffer sizes**: [buffer.memory](https://www.conduktor.io/kafka/kafka-options-explorer/?search=buffer.memory) controls how much memory the producer uses for unsent messages. Too small causes blocking. Too large wastes memory.

**Control in-flight requests**: [max.in.flight.requests.per.connection](https://www.conduktor.io/kafka/kafka-options-explorer/?search=max.in.flight.requests.per.connection) limits concurrent requests to brokers. Lower values prevent overwhelming brokers but reduce throughput. See [Alpakka's flow control implementation](https://www.lightbend.com/blog/alpakka-kafka-flow-control-optimizations) for advanced patterns.

Conduktor [enforces producer configuration policies](https://marketplace.conduktor.io/interceptors/safeguard-produce-policy/) across your organization without requiring code changes.

## Summary

These five areas determine Kafka application reliability: partition sizing, serialization format, dual-write handling, tombstone processing, and producer tuning.

Kafka ecosystems grow complex quickly. Conduktor enforces best practices at the infrastructure level, preventing misconfigurations before they cause production incidents.

![Diagram of Conduktor enforcing Kafka best practices at the infrastructure layer across partitions, serialization, producer settings and compaction](https://www.conduktor.io/assets/images/blog/conduktor-best-practices-infrastructure-enforcement.png)

[Contact us](https://www.conduktor.io/contact) to discuss your use cases. [Download Conduktor Gateway](https://docs.conduktor.io/gateway/), our Kafka proxy (open source), and browse our [marketplace](https://marketplace.conduktor.io/) for available features.