How to send large messages in Apache Kafka

Kafka's default 1 MB message size limit can be raised — set Kafka max message size with message.max.bytes and max.request.size, plus compression alternatives.

Learn how to handle large messages in Kafka in 15 minutes

Apache Kafka has default message size limits that can be configured to handle larger payloads, but there are important considerations and best practices to follow.

What you'll learn:

• Default message size limits across Kafka components
• How to configure producers, brokers, and consumers for large messages
• Performance implications of large messages
• Alternative patterns for handling large payloads

Default message size limits

By default, Kafka has the following message size limits:

Component	Configuration	Default
Producer	max.request.size	1 MB
Broker	message.max.bytes	1 MB
Topic	max.message.bytes	Inherits from broker
Consumer	max.partition.fetch.bytes	1 MB

Configure Kafka for large messages

To send messages larger than 1MB, you need to configure multiple components:

Producer configuration

# Set maximum request size for producer
max.request.size=10485760  # 10MB
# Increase buffer memory if needed
buffer.memory=67108864     # 64MB

Broker configuration

# Set maximum message size for broker
message.max.bytes=10485760          # 10MB
# Set maximum replica fetch size
replica.fetch.max.bytes=10485760    # 10MB
# Set maximum response size
socket.receive.buffer.bytes=1048576 # 1MB
socket.send.buffer.bytes=1048576    # 1MB

Topic configuration

kafka-configs --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name large-topic \
  --add-config max.message.bytes=10485760

Consumer configuration

# Set maximum fetch size for consumer
max.partition.fetch.bytes=10485760  # 10MB
fetch.max.bytes=52428800            # 50MB

Performance implications

Sending large messages in Kafka has several performance implications:

Area	Impact	Mitigation
Memory	Higher heap usage, GC pressure	Tune JVM heap sizes
Network	More bandwidth, potential timeouts	Adjust buffer sizes
Disk I/O	More operations, slower compaction	Use faster storage
Throughput	Lower overall message rate	Enable compression

Alternative approaches

Instead of sending large messages directly, consider these alternatives:

1. External storage pattern

Store large payloads in external systems and send only references:

{
  "id": "message-123",
  "timestamp": "2023-01-01T00:00:00Z",
  "data_location": "s3://bucket/path/to/large-file.json",
  "metadata": {
    "size": 50000000,
    "checksum": "abc123"
  }
}

Benefits:

• Keeps Kafka messages small and fast
• Allows for separate scaling of storage and messaging
• Enables efficient caching strategies

2. Split messages

Break large messages into smaller chunks:

{
  "message_id": "msg-123",
  "chunk_id": "chunk-1",
  "total_chunks": 5,
  "chunk_data": "...",
  "sequence": 1
}

Benefits:

• Works within default Kafka limits
• Allows for parallel processing
• Provides better error recovery

3. Compression

Enable compression to reduce message sizes:

# Producer compression
compression.type=snappy  # or gzip, lz4, zstd

Benefits:

• Reduces network bandwidth usage
• Decreases storage requirements
• Often improves throughput

Best practices

Recommendations for large messages

1. Avoid large messages when possible - Kafka is optimized for small, high-throughput messages
2. Use external storage - Store large payloads externally and reference them in Kafka messages
3. Enable compression - Always enable compression for large messages
4. Monitor memory usage - Ensure adequate heap sizing for all components
5. Test thoroughly - Verify performance impact in your specific environment

Configuration checklist

When configuring for large messages, ensure all these settings are aligned:

• ✅ Producer max.request.size
• ✅ Broker message.max.bytes
• ✅ Topic max.message.bytes
• ✅ Consumer max.partition.fetch.bytes
• ✅ Consumer fetch.max.bytes
• ✅ Broker replica.fetch.max.bytes

Monitor large messages

Monitor these metrics when working with large messages:

• Memory usage on brokers, producers, and consumers
• Network bandwidth utilization
• Disk I/O patterns and latency
• Garbage collection frequency and duration
• Message throughput and latency

Large message decision guide

Large messages can significantly impact Kafka performance. Always test in a staging environment that mirrors your production setup before deploying large message configurations.

See it in practice with Conduktor

Conduktor Console lets you produce and consume messages while monitoring their sizes. Test your large message configurations and verify all component limits are aligned.

Next steps

• Apply naming conventions to keep large-payload topics organized
• Enable compression to reduce message sizes
• Configure log segments for storage optimization