Recovering a Kafka Connector From a Poison Pill with Conduktor Agents

Ron Kapoor November 18, 2026 8 min read
A line of small wireframe cubes travels along a dashed path above a glowing lime platform on dark teal. One cube glows lime inside a tall glass column, and later cubes sit inside three clear glass shapes.

Restarting a failed Kafka Connect task takes seconds. Figuring out why it keeps failing, and whether it's safe to skip the record that's stopping it, takes more work. Kafka Connect resumes from its committed offset, so if the next record is the problem, another restart brings the task back to the same failure.

This is our second Conduktor Agents use case. In the first, Bob and Priya investigated a leaked Kafka credential. This time, they're trying to get a reporting pipeline running again without losing data nobody can account for.

The pipeline is postgres-sink, which reads the pageviews topic and writes to Postgres. It expects Avro. Jack, on another team, is testing a new pageview event and sends {"viewtime": "yesterday", "userid": "User_9", "pageid": "Page_42"} through Console's Produce tab, leaving its value format at the default, JSON.

Bob's team runs the connector. He can restart it, stop it, and reset its consumer group's offsets, but he can't read the audit log or change the connector's configuration. Priya, on the platform team, can do both. As in the first use case, the fix needs both of them.

What does this post change for your team?ClaudeChatGPT

Bob Restarts the Task. It Fails Again.

Within a minute, postgres-sink has one failed task. The error ends in Unknown magic byte!. Its Avro converter expected a Schema Registry wire-format header and didn't find one. That explains the kind of failure, but not which record caused it or how it got there.

Bob presses Restart Task. It fails again. He tries once more, with the same result. The task keeps returning to the bad record rather than getting past it.

Every film here is live HTML. Click one to pause it and scroll to read every line, or step through it with the bar underneath.

Jack sends the test record, the connector fails, and Bob's restarts land on the same error.

Those restarts also leave a trail. Priya already runs the built-in Connector recovery task as an agent, triggered by connector actions such as restarts. Each time Bob restarts, it escalates instead of proposing another restart, and its answer to the fourth is plain: "A restart cannot fix it: the task will read the same record again and fail again."

Bob Finds the Record, but Not Who Sent It

Bob opens Claude Code in the team's repo and asks: "postgres-sink keeps failing. I restarted its task twice and it failed again both times. What's wrong with it?"

His assistant uses the Conduktor MCP server with Bob's own token, so it can investigate the connector with his permissions. It reads the connector's state, then checks its consumer group, connect-postgres-sink. At partition 0, offset 735233, it finds a plain JSON record sitting between two Avro records. That's the record the task can't deserialize.

The message also has an app_name: Conduktor header. Bob's assistant points out that it looks like something produced by hand through Console, but the header alone doesn't establish who sent it.

Claude Code locates the record at the failing offset, compares it with its neighbours, then reaches the limit of Bob's audit-log permissions.

Bob has three ways forward, and none of them is free:

  • Restart again. It will land on offset 735233 and fail again. Auto-restart would repeat that cycle for him.
  • Skip this one record. Stop the connector, move the consumer group forward to 735234, and resume. The record stays in Kafka until retention removes it, but the sink won't write it to Postgres, so someone should keep a copy before moving on.
  • Change how the sink handles bad records. Configure error tolerance and a dead-letter queue so future incompatible records are set aside instead of stopping the pipeline. That's a broader change to the connector, and Bob doesn't have permission to make it.

He leans toward skipping the record to get reporting moving again. But first he wants to know whether it's a harmless test or real data that someone expects to see in Postgres. His assistant tries to look up the producer in Console's audit log and gets a permission error. Bob asks Priya to check.

Priya Finds the Sender and Reviews the Fix

Priya can see the cluster-wide history Bob can't. She runs a custom agent task for this cluster, Failing sink connector, find the poison pill, which follows the failing connector back to the offending record and produces an approval request rather than making the change itself.

It confirms the same offset and format mismatch, then checks Console's audit log. There it finds a Kafka.Topic.ProduceRecord event on pageviews from jack@conduktordemo.io. What looked like a manually produced test record now has an identifiable sender.

The agent proposes a recovery plan: stop postgres-sink, create pageviews.dlq, copy the bad record there, advance connect-postgres-sink from 735233 to 735234 "and leave everything else as it is", resume the connector, and let Jack know what happened. It never proposes another restart. Its last step is for later: add error tolerance and a dead-letter queue to the connector, so the next bad record doesn't stop it.

Priya opens the two approvals in her inbox, reads the poison-pill record and its sender, and reviews the proposed steps before approving them.

Another built-in agent, Unstable connectors, sees the wider pattern: postgres-sink has failed and been restarted four times in the window, all by Bob. One agent explains why repeating the action won't help; the other shows that this is now a pattern worth investigating.

Bob stops the connector, and Priya creates pageviews.dlq and copies Jack's record into it before anyone moves the offset. Approving the card records her decision, and nothing on the cluster changes because she approved it. Bob can now recover his team's connector without losing track of what it skipped.

Bob Skips One Record Instead of 735,000

With the record copied, Bob opens Reset Offset for connect-postgres-sink. The drawer initially shows Earliest. Its preview puts the offset at 0.

That would be a very different recovery. Instead of skipping one record, the sink would replay roughly 735,000 earlier pageviews into Postgres. Depending on how the sink handles repeated writes, that could mean duplicates or a lot of unnecessary work.

Bob changes the option to Shift by, enters 1, and checks the preview: partition 0 will move from 735233 to 735234. He presses Reset, then Resume. This time the task stays running, and consumer lag drops from 19,616 to 916 in about a minute and a half.

Bob checks the reset preview, shifts forward by one offset, and resumes the sink; an agent then reviews what changed.

The reset triggers another custom agent, Offset reset impact. It reconstructs the sequence from Console's audit log, from Bob's restarts to the stop and the reset. Its finding is specific: connect-postgres-sink moved forward one offset on pageviews, so one record was skipped. It also sizes the damage: one missing pageview in Postgres slightly undercounts the reporting data.

Bob has a working connector. Priya has a preserved copy of the record, an identified sender, and an auditable account of why the offset moved.

The Connector Is Running. The Next Bad Record Is Still a Decision.

Bob diagnosed the failure from his own terminal and made the one-offset change his role allows. Priya used broader visibility to confirm who sent the record and approve a recovery that kept a copy. The agents noticed the repeated failure, proposed the steps, and explained the impact afterward. A person made every change.

Here's where each piece helped:

PieceWhat it doesWhere it showed up
Conduktor MCP serverGives an AI assistant Console's tools within the user's permissionsBob finds offset 735233, but can't read the audit log
Connector recovery (built in)Reacts to connector actions and escalates when retrying won't helpEach restart leads to an escalation instead of another restart
Unstable connectors (built in)Reviews failing and repeatedly restarted connectors on a schedulePriya sees the four restarts as one pattern
Failing sink connector, find the poison pill (custom)Finds the offending record, checks its origin, and proposes steps for approvalPriya gets Jack's audit event and the recovery plan
Offset reset impact (custom)Reviews consumer-group resets and estimates their effectThe one-record skip is recorded and explained
There's still a choice for the next failure. Creating pageviews.dlq and copying this record into it preserved evidence; it didn't configure postgres-sink to send future failures there automatically. Priya could make that separate configuration change, using error tolerance and dead-letter-queue settings so malformed records don't stop reporting again. Or she could keep the sink strict and prevent records like Jack's from reaching the topic in the first place. For clients producing through Conduktor Gateway, the Schema ID required policy is one way to enforce that.

Either way, the agents didn't silently discard a bad record or decide the team's data-loss policy. They gave Bob and Priya the context to make a deliberate one-record exception, get the pipeline moving, and see exactly what changed.

See It Live on 8 December

Register for our webinar on Tuesday, 8 December, at 11:00 AM ET / 8:00 AM PT to see Conduktor Agents work through connector failures like this one live: Leave the Chores to Kafka Agents. Can't make it? Register anyway and we'll send you the recording.


Related: Investigating a Leaked Kafka Credential with Conduktor Agents → · Introducing Conduktor Agents → · Conduktor MCP + Skills for AI Agents → · Bounce Bad Kafka Data with JSON Schema Rules →