# How to send large messages in Apache Kafka

*Learn how to handle large messages in Kafka in 15 minutes*

Apache Kafka has default message size limits that can be configured to handle larger payloads, but there are important considerations and best practices to follow.

**What you'll learn:**
- Default message size limits across Kafka components
- How to configure producers, brokers, and consumers for large messages
- Performance implications of large messages
- Alternative patterns for handling large payloads

## Default message size limits

By default, Kafka has the following message size limits:

| Component | Configuration | Default |
|-----------|--------------|---------|
| Producer | `max.request.size` | 1 MB |
| Broker | `message.max.bytes` | 1 MB |
| Topic | `max.message.bytes` | Inherits from broker |
| Consumer | `max.partition.fetch.bytes` | 1 MB |

![Message size limits chained across Producer max.request.size, Broker message.max.bytes, Topic max.message.bytes, and Consumer max.partition.fetch.bytes, each stage at most the next](https://www.conduktor.io/assets/kafka/diagrams/how-to-send-large-messages-in-apache-kafka--1.svg)

```mermaid
flowchart LR
    P["Producer<br/>max.request.size"] -->|"≤ broker limit"| B["Broker<br/>message.max.bytes"]
    B -->|"≤ topic limit"| T["Topic<br/>max.message.bytes"]
    T -->|"≤ consumer limit"| C["Consumer<br/>max.partition.fetch.bytes"]

    style P fill:#e1f5fe
    style C fill:#e8f5e9
```

## Configure Kafka for large messages

To send messages larger than 1MB, you need to configure multiple components:

### Producer configuration

```properties
# Set maximum request size for producer
max.request.size=10485760  # 10MB
# Increase buffer memory if needed
buffer.memory=67108864     # 64MB
```

### Broker configuration

```properties
# Set maximum message size for broker
message.max.bytes=10485760          # 10MB
# Set maximum replica fetch size
replica.fetch.max.bytes=10485760    # 10MB
# Set maximum response size
socket.receive.buffer.bytes=1048576 # 1MB
socket.send.buffer.bytes=1048576    # 1MB
```

### Topic configuration

```bash
kafka-configs --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name large-topic \
  --add-config max.message.bytes=10485760
```

### Consumer configuration

```properties
# Set maximum fetch size for consumer
max.partition.fetch.bytes=10485760  # 10MB
fetch.max.bytes=52428800            # 50MB
```

## Performance implications

Sending large messages in Kafka has several performance implications:

| Area | Impact | Mitigation |
|------|--------|------------|
| Memory | Higher heap usage, GC pressure | Tune JVM heap sizes |
| Network | More bandwidth, potential timeouts | Adjust buffer sizes |
| Disk I/O | More operations, slower compaction | Use faster storage |
| Throughput | Lower overall message rate | Enable compression |

## Alternative approaches

Instead of sending large messages directly, consider these alternatives:

### 1. External storage pattern

Store large payloads in external systems and send only references:

```json
{
  "id": "message-123",
  "timestamp": "2023-01-01T00:00:00Z",
  "data_location": "s3://bucket/path/to/large-file.json",
  "metadata": {
    "size": 50000000,
    "checksum": "abc123"
  }
}
```

**Benefits:**
- Keeps Kafka messages small and fast
- Allows for separate scaling of storage and messaging
- Enables efficient caching strategies

### 2. Split messages

Break large messages into smaller chunks:

```json
{
  "message_id": "msg-123",
  "chunk_id": "chunk-1",
  "total_chunks": 5,
  "chunk_data": "...",
  "sequence": 1
}
```

**Benefits:**
- Works within default Kafka limits
- Allows for parallel processing
- Provides better error recovery

### 3. Compression

Enable compression to reduce message sizes:

```properties
# Producer compression
compression.type=snappy  # or gzip, lz4, zstd
```

**Benefits:**
- Reduces network bandwidth usage
- Decreases storage requirements
- Often improves throughput

## Best practices

> **Recommendations for large messages**
> 1. **Avoid large messages when possible** - Kafka is optimized for small, high-throughput messages
> 2. **Use external storage** - Store large payloads externally and reference them in Kafka messages
> 3. **Enable compression** - Always enable compression for large messages
> 4. **Monitor memory usage** - Ensure adequate heap sizing for all components
> 5. **Test thoroughly** - Verify performance impact in your specific environment

### Configuration checklist

When configuring for large messages, ensure all these settings are aligned:

- ✅ Producer `max.request.size`
- ✅ Broker `message.max.bytes`
- ✅ Topic `max.message.bytes`
- ✅ Consumer `max.partition.fetch.bytes`
- ✅ Consumer `fetch.max.bytes`
- ✅ Broker `replica.fetch.max.bytes`

### Monitor large messages

Monitor these metrics when working with large messages:

- Memory usage on brokers, producers, and consumers
- Network bandwidth utilization
- Disk I/O patterns and latency
- Garbage collection frequency and duration
- Message throughput and latency

## Large message decision guide

![Decision tree for large payloads: compress first, else store externally (S3/blob), else chunk into split messages, else configure larger limits across all components](https://www.conduktor.io/assets/kafka/diagrams/how-to-send-large-messages-in-apache-kafka--2.svg)

```mermaid
flowchart TD
    Start["Need to send<br/>large payload?"] --> Q1{{"Can data be<br/>compressed?"}}

    Q1 -->|"Yes"| Compress["Enable compression<br/>(snappy, lz4, zstd)"]
    Q1 -->|"No"| Q2{{"Can data be<br/>stored externally?"}}

    Compress --> Check{{"Still > 1MB?"}}
    Check -->|"No"| Done["Use default limits"]
    Check -->|"Yes"| Q2

    Q2 -->|"Yes"| External["Use external storage pattern<br/>(S3, blob storage)"]
    Q2 -->|"No"| Q3{{"Can data be<br/>chunked?"}}

    Q3 -->|"Yes"| Chunk["Implement message splitting"]
    Q3 -->|"No"| Config["Configure larger limits<br/>(all components)"]

    External --> Best["Best for large payloads"]
    Chunk --> Good["Good for moderate sizes"]
    Config --> Test["Test thoroughly"]
```

Large messages can significantly impact Kafka performance. Always test in a staging environment that mirrors your production setup before deploying large message configurations.

> **See it in practice with Conduktor**
> [Conduktor Console](https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics) lets you produce and consume messages while monitoring their sizes. Test your large message configurations and verify all component limits are aligned.

## Next steps

- [Apply naming conventions](https://www.conduktor.io/kafka/kafka-topics-naming-convention) to keep large-payload topics organized
- [Enable compression](https://www.conduktor.io/kafka/kafka-message-compression) to reduce message sizes
- [Configure log segments](https://www.conduktor.io/kafka/kafka-topics-internals-segments-and-indexes) for storage optimization