# Multi-Agent AI Is a Distributed Systems Problem

To build a multi-agent system, you split the work across agents, each with its own tools and memory. An orchestrator hands sub-tasks to specialized agents, usually through direct HTTP calls. Then more agents join, and it feels like déjà vu.

Microservices hit this problem in the 2010s, and Kafka came along: put a log between services and share the data with whoever wants to listen. One difference: agents are slow, and each step can take seconds to minutes. What they need from Kafka is a queue, with a pool of workers that grows with the load.

## Dedicated agents for context, parallel work or tool specialization

A single agent with a good set of tools usually beats a group of agents coordinating badly. A split pays off when you want:

- **Context isolation.** The model reasons worse as tools, history and documents fill its context window.
- **Parallel work.** Sub-agents run independent checks at the same time, and the orchestrator merges the results.
- **Specialization.** The more tools an agent holds, the more often it picks the wrong one, and the larger its attack surface.

Kafka fits an [event-driven architecture with LLM steps inside](https://www.conduktor.io/blog/where-ai-agents-belong-in-a-kafka-architecture), where agents hand work over asynchronously and the pool of workers grows with the load.

## Direct calls make the slowest agent the weakest link

Say an IT helpdesk assistant answers employees' IT problems. The orchestrator calls a Windows agent, which calls a knowledge-base tool, which calls a ticketing agent, all over HTTP. The happy path works, but failures show up under load:

- **The domino effect.** If the ticketing agent is slow, every agent above it waits; if it's down, they fail. Each step is a model call that takes seconds, so the waits add up fast.
- **Nothing absorbs a burst.** A spike of user messages fills the orchestrator's queue in memory, the process dies, and the requests in flight are lost.
- **Nothing is recoverable.** The request behind a bad answer disappeared with its HTTP connection.

A Kafka topic changes that, because it persists every message. The orchestrator produces to Kafka and moves on, and the workers share the load at their own pace. When the ticketing agent slows down, tasks wait in the topic and its callers keep working.

## Put a topic between the orchestrator and its workers

Worker agents consume a task topic (`tools.requests`), run their workflow with their own prompt and tools, and publish the result to a `tools.responses` topic that the orchestrator consumes:

A few rules:

- **Key by `conversation_id`.** A conversation's records then stay ordered on one partition.
- **Commit the orchestrator's output and offset together.** Wrap the task it writes and its offset commit in a [transaction](https://www.conduktor.io/glossary/kafka-transactions-deep-dive), so a crash in between doesn't publish the same task twice. Set `share.isolation.level=read_committed` on the workers' group so they skip aborted tasks, and call the model before the transaction starts: those workers read nothing past an open transaction on the partition, including tasks another orchestrator already committed.
- **Make workers idempotent.** The same task can reach a worker twice, so its side effect must not run twice ([idempotent consumer](https://www.conduktor.io/glossary/idempotent-consumer-pattern)).

## Share groups turn a topic into a pool of agent workers

In a classic [consumer group](https://www.conduktor.io/glossary/kafka-consumer-groups-explained), each partition goes to one consumer, so a topic with six partitions keeps at most six workers busy. Tool calls take seconds to minutes, so a tool pool needs more workers than the topic has partitions. Adding partitions only moves the cap, and a thread pool inside one consumer has to track which offsets are done before it can commit.

[Share groups](https://www.conduktor.io/glossary/kafka-share-groups) (KIP-932, "Queues for Kafka") let several consumers read the same partition:

The broker locks each record for the worker that fetched it, and the worker acknowledges each one:

- **ACCEPT** once the work is done.
- **RELEASE** to hand the record back to any worker in the group.
- **REJECT** to drop it for good.

Share groups have been production-ready since [Kafka 4.2](https://kafka.apache.org/blog/2026/02/17/apache-kafka-4.2.0-release-announcement/) (run 4.2.1 or later, which fixes a deadlock, [KAFKA-20505](https://issues.apache.org/jira/browse/KAFKA-20505)).

Share groups don't keep per-key ordering: two tasks from the same partition can run on two workers at once.

A tool worker's loop, with `share.acknowledgement.mode=explicit`:

```java
while (running) {
    for (var rec : consumer.poll(Duration.ofMillis(500))) {
        try {
            var result = callTool(rec);   // idempotent: the same task can arrive twice
            // wait for the broker before accepting the task
            producer.send(new ProducerRecord<>("tools.responses", rec.key(), result)).get();
            consumer.acknowledge(rec, AcknowledgeType.ACCEPT);
        } catch (NonRetryableException e) {           // your own type: a retry won't help
            consumer.acknowledge(rec, AcknowledgeType.REJECT);
        } catch (Exception e) {
            consumer.acknowledge(rec, AcknowledgeType.RELEASE);   // back to the pool
        }
    }
    consumer.commitSync();   // an ACCEPT that came after the lock expired fails here
}
```

These settings decide how your worker pool behaves:

- **One worker can lock the whole batch.** By default (`share.acquire.mode=batch_optimized`), the broker locks entire producer batches for the worker that fetches them. With `share.acquire.mode=record_limit` ([KIP-1206](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1206:+Strict+max+fetch+records+in+share+fetch)), `max.poll.records` becomes a hard limit instead.
- **A task stays locked for 30 seconds.** The lock lasts `share.record.lock.duration.ms`, then another worker can take the task. To keep it longer, acknowledge it with `RENEW` ([KIP-1222](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1222:+Acquisition+lock+timeout+renewal+in+share+consumer+explicit+mode)) and commit: the next poll returns the task, still locked.
- **Five attempts by default.** Since [4.3](https://www.conduktor.io/blog/kafka-4-3-oauth-share-groups-tiered-bootstrap), a group can set its own `share.delivery.count.limit`. Since 4.4, the broker can also send rejected and exhausted tasks to a dead-letter topic: set `errors.deadletterqueue.topic.name` on the group ([KIP-1191](https://cwiki.apache.org/confluence/x/fApJFg)) and `errors.deadletterqueue.group.enable=true` on the topic.
- **Earliest or latest.** Like a new consumer group, a new share group starts at the end of the topic: set `share.auto.offset.reset=earliest` on the group to process what's already there.

A dead-letter record points back at the original task with headers like these:

```text
__dlq.errors.offset=1 __dlq.errors.delivery.count=1 __dlq.errors.message=Offset rejected by client.
__dlq.errors.offset=0 __dlq.errors.delivery.count=5 __dlq.errors.message=Offset delivery count exceeded the threshold.
```

To keep the whole task in the dead-letter topic, set `errors.deadletterqueue.copy.record.enable=true`.

## The log is the memory

Kafka itself is a memory: every user message, task, tool call and response goes through its topics. Replay them to test a new agent version on past conversations, or run it as a canary next to the current one.

The log feeds different types of memory:

- **Working state**, private to each agent, in a compacted topic.
- **Shared context**, materialized by Kafka Streams.
- **Long-term recall**, in a vector database.

My Data Streaming Summit 2026 talk, [Event-Driven Memory for LLM Agent Swarms](https://www.conduktor.io/assets/pdfs/event-driven-memory-for-llm-agent-swarms.pdf), demonstrates how and why:

Slides (PDF)

## Critical: sandbox every agent

Kafka only bounds the delivery. The agent loop inside it can go rogue, so it has to be contained:

- **Limit calls.** Cap model and tool calls per run, stop when the agent repeats a call, and cap the size of each tool result.
- **Limit tokens and time.** Give each run a token budget, and a time limit under the record lock unless the worker renews it.
- **Define triggers.** When events start agent runs, group a burst of events on the same resource into one run, and never let an agent's own output start it again.
- **Define identity.** Each agent gets its own principal, with permissions checked on every call, so removing them stops the run at its next step.
- **Trace everything.** Record each run's prompts, tool calls, tokens and result under its `conversation_id`.

An illustrative run: it stops at the first limit it hits.

We built the agents in [Conduktor Console](https://www.conduktor.io/console) on these limits, each running as its own identity, never wider than its owner's.

Kafka is already proven infrastructure ([more than 80% of the Fortune 100](https://kafka.apache.org/) use it), and a perfect match for asynchronous communication between agents.

[Install Console free](https://www.conduktor.io/get-started?src=blog-multi-agent-ai-is-a-distributed-systems-problem) to try them, or [talk to us](https://www.conduktor.io/contact/demo?src=blog-multi-agent-ai-is-a-distributed-systems-problem) about running agents under your company's AI program.

---

Related: [Where AI Agents Belong in a Kafka Architecture →](https://www.conduktor.io/blog/where-ai-agents-belong-in-a-kafka-architecture) · [Kafka AI Agents Should Behave Like Kubernetes Controllers →](https://www.conduktor.io/blog/kafka-ai-agents-kubernetes-controllers) · [Kafka Share Groups →](https://www.conduktor.io/glossary/kafka-share-groups)