Restarting a failed Kafka Connect task takes seconds. Figuring out why it keeps failing, and whether it's safe to skip the record that's stopping it, takes more work. Kafka Connect resumes from its committed offset, so if the next record is the problem, another restart brings the task back to the same failure.
This is our second Conduktor Agents use case. In the first, Bob and Priya investigated a leaked Kafka credential. This time, they're trying to get a reporting pipeline running again without losing data nobody can account for.
The pipeline is postgres-sink, which reads the pageviews topic and writes to Postgres. It expects Avro. Jack, on another team, is testing a new pageview event and sends {"viewtime": "yesterday", "userid": "User_9", "pageid": "Page_42"} through Console's Produce tab, leaving its value format at the default, JSON.
Bob's team runs the connector. He can restart it, stop it, and reset its consumer group's offsets, but he can't read the audit log or change the connector's configuration. Priya, on the platform team, can do both. As in the first use case, the fix needs both of them.
What does this post change for your team?ClaudeChatGPT
Within a minute, postgres-sink has one failed task. The error ends in Unknown magic byte!. Its Avro converter expected a Schema Registry wire-format header and didn't find one. That explains the kind of failure, but not which record caused it or how it got there.
Bob presses Restart Task. It fails again. He tries once more, with the same result. The task keeps returning to the bad record rather than getting past it.
Every film here is live HTML. Click one to pause it and scroll to read every line, or step through it with the bar underneath.
Local Kafkalocal-kafka-clusterKafka Connect/Local Connect/postgres-sink
Search⌘KBB
Task restarted successfully
Jack sends the test record, the connector fails, and Bob's restarts land on the same error.
Those restarts also leave a trail. Priya already runs the built-in Connector recovery task as an agent, triggered by connector actions such as restarts. Each time Bob restarts, it escalates instead of proposing another restart, and its answer to the fourth is plain: "A restart cannot fix it: the task will read the same record again and fail again."
Bob opens Claude Code in the team's repo and asks: "postgres-sink keeps failing. I restarted its task twice and it failed again both times. What's wrong with it?"
His assistant uses the Conduktor MCP server with Bob's own token, so it can investigate the connector with his permissions. It reads the connector's state, then checks its consumer group, connect-postgres-sink. At partition 0, offset 735233, it finds a plain JSON record sitting between two Avro records. That's the record the task can't deserialize.
The message also has an app_name: Conduktor header. Bob's assistant points out that it looks like something produced by hand through Console, but the header alone doesn't establish who sent it.
~/reporting
Claude Codev2.1.325
Opus 5.5
~/reporting
Called conduktor 2 times (ctrl+o to expand)
The task fails with Unknown magic byte!. The connector expects Confluent Avro, so I'll check what's actually on the pageviews topic.
Called conduktor 6 times (ctrl+o to expand)
I found it. A single bad record on pageviews breaks the connector, and restarting can't get past it.
It was written at 15:30:54 today. … That looks like a test message sent by hand from Conduktor Console's producer.
Why restarting didn't help
The connector's last saved position is exactly 735233. Each restart picks up from there, reads the same bad record and fails again.
How to fix it
1.Skip the bad record. The connector's consumer group (connect-postgres-sink) has no active members right now because the task is down, so you can move its position to 735234 and then restart the task:
You can also do this from the consumer group page in Console.
2.Stop one bad record from halting the connector again. Add these settings so bad records go to a separate "dead letter" topic instead of killing the task:
I can apply the error-handling config change for you. It updates the running connector, so I'll wait for your go-ahead. I can also check the audit log to see who sent the test message.
38s
Called conduktor 2 times (ctrl+o to expand)
I couldn't check the audit log because the account I'm connected to Conduktor with doesn't have audit-log access (it's missing the platform.auditlog.read permission). So I can't tell you who sent the record.
A Console admin can look it up, or grant that permission to this account so I can query it. …
Fixing the connector doesn't depend on knowing who sent it. Skipping offset 735233 or adding the dead-letter-topic settings will get postgres-sink running again.
10s
Claude Code locates the record at the failing offset, compares it with its neighbours, then reaches the limit of Bob's audit-log permissions.
Bob has three ways forward, and none of them is free:
Restart again. It will land on offset 735233 and fail again. Auto-restart would repeat that cycle for him.
Skip this one record. Stop the connector, move the consumer group forward to 735234, and resume. The record stays in Kafka until retention removes it, but the sink won't write it to Postgres, so someone should keep a copy before moving on.
Change how the sink handles bad records. Configure error tolerance and a dead-letter queue so future incompatible records are set aside instead of stopping the pipeline. That's a broader change to the connector, and Bob doesn't have permission to make it.
He leans toward skipping the record to get reporting moving again. But first he wants to know whether it's a harmless test or real data that someone expects to see in Postgres. His assistant tries to look up the producer in Console's audit log and gets a permission error. Bob asks Priya to check.
Priya can see the cluster-wide history Bob can't. She runs a custom agent task for this cluster, Failing sink connector, find the poison pill, which follows the failing connector back to the offending record and produces an approval request rather than making the change itself.
It confirms the same offset and format mismatch, then checks Console's audit log. There it finds a Kafka.Topic.ProduceRecord event on pageviews from jack@conduktordemo.io. What looked like a manually produced test record now has an identifiable sender.
The agent proposes a recovery plan: stop postgres-sink, create pageviews.dlq, copy the bad record there, advance connect-postgres-sink from 735233 to 735234 "and leave everything else as it is", resume the connector, and let Jack know what happened. It never proposes another restart. Its last step is for later: add error tolerance and a dead-letter queue to the connector, so the next bad record doesn't stop it.
Agentic
Home
Identities
Tasks
Models
Agents
Scheduled
Event-driven
Output
Reports
Recommendations
Approvals
Direct updates
Activity
Connector action → recovery
Failing sink connector, find the poison pill
Settings
main
Output/Failing sink connector, find the poison pill/ea6846c7-95c4-464d-be37-cb650f8a389f
AskA
Output
Everything your agents hand back, reports to read, recommendations to act on, approvals to decide, and a record of direct updates.
ReportsRecommendationsApprovalsDirect updates
Search by title, task or agentStatusAgent
OutputAgentStatusProduced
Unblock postgres-sink: skip the plain-JSON record at pageviews/0@735233Unblock postgres-sink: skip the plain-JSON record at pageviews/0@735233Failing sink connector, find the poison pillFailing sink connector, find the poison pillPending9 minutes ago
Escalate postgres-sink: non-Avro records on pageviews, restarts not helpingEscalate postgres-sink: non-Avro records on pageviews, restarts not helpingConnector recoveryConnector action → recoveryPending12 minutes ago
Unblock postgres-sink: skip the plain-JSON record at pageviews/0@735233ApprovalFailing sink connector, find the poison pillPendingApproved
Produced 9 minutes ago · agent Failing sink connector, find the poison pill
RejectApprove
OutputTrace
Pending your decision: nothing has been applied
The agent wrote no changes. Approving records your decision for the steps below; a person carries them out.
Approved by admin@demo.dev
a few seconds agoJack test record; copied to pageviews.dlq, Bob skipped it
Finding
Connector:postgres-sink (io.confluent.connect.jdbc.JdbcSinkConnector) on connect cluster local-connect reads topic pageviews. Its only task, task 0 on worker connect:8083, is FAILED.
Error: The task failed in the converter stage, before any transform ran and before anything was written to Postgres. This is not a credentials, network or database problem.
Record: Consumer group connect-postgres-sink is Empty. Its committed offset on pageviews partition 0 is 735233, the end offset is 745865, and lag is 10632. The record at partition 0, offset 735233 has:
timestamp 2026-11-16T15:30:54.158Z
an empty key
plain JSON value {"viewtime":"yesterday","userid":"User_9","pageid":"Page_42"} (format Json, schemaId null, 61 bytes)
header app_name=Conduktor
It has no Schema Registry magic byte or schema id, which is exactly why the AvroConverter rejects it. It is also wrong on content: subject pageviews-value (AVRO, id 3, version 1) expects viewtime to be a long, and this record has the string "yesterday".
Previous record: Offset 735232 (2026-11-16T15:30:54.093Z) is valid. Its key is 7339581, its format is AvroWithSchemaRegistry with Confluent schema id 3, it is 24 bytes, and viewtime is 7339581. It carries the datagen headers task.generation, task.id and current.iteration.
Who produced it: The audit log has one Kafka.Topic.ProduceRecord event on pageviews, by Console user jack@conduktordemo.io at 2026-11-16T15:30:54.100162Z. That is within about 60 ms of the record's timestamp, and the record's app_name=Conduktor header also points to a Console produce.
Dead-letter topic:pageviews.dlq does not exist (get-topic returned "Topic not found"), so it has to be created before the record can be copied there.
Proposed steps
Stop connector `postgres-sink` on connect cluster `local-connect` (cluster `local-kafka-cluster`) so consumer group `connect-postgres-sink` stays inactive while you change its offset.
Create topic `pageviews.dlq` on `local-kafka-cluster`; it does not exist yet.
Copy the record at `pageviews` partition 0, offset 735233 to `pageviews.dlq` without changing it: empty key, value `{"viewtime":"yesterday","userid":"User_9","pageid":"Page_42"}`, header `app_name=Conduktor`. Add headers for the source coordinates (topic=pageviews, partition=0, offset=735233, timestamp=2026-11-16T15:30:54.158Z).
Reset consumer group `connect-postgres-sink` on topic `pageviews` partition 0 from 735233 to 735234. Move this one offset only and leave everything else as it is.
Resume `postgres-sink`. Confirm task 0 is RUNNING on `local-connect` and that the lag of `connect-postgres-sink` on `pageviews` partition 0 keeps falling from 10632. If the task fails again, inspect the record at the new committed offset before skipping anything more.
Tell jack@conduktordemo.io and the team that owns `pageviews` that the plain-JSON record they produced through Console on 2026-11-16 at about 15:30:54Z (pageviews/0@735233, now copied to pageviews.dlq) stopped postgres-sink.
To stop this happening again, add `errors.tolerance=all`, `errors.deadletterqueue.topic.name=pageviews.dlq` and `errors.deadletterqueue.context.headers.enable=true` to `postgres-sink`. As well or instead, reject records that have no valid Schema Registry schema id before they reach `pageviews`, for example by only allowing Avro produces with the schema registry serializer.
Approve this proposal?
Approving Unblock postgres-sink: skip the plain-JSON record at pageviews/0@735233 records your decision. Nothing is executed. The steps the agent proposed are for someone to carry out.
Reason (optional)
Kept with the approval, for whoever reads it later
CancelApprove
Priya opens the two approvals in her inbox, reads the poison-pill record and its sender, and reviews the proposed steps before approving them.
Another built-in agent, Unstable connectors, sees the wider pattern: postgres-sink has failed and been restarted four times in the window, all by Bob. One agent explains why repeating the action won't help; the other shows that this is now a pattern worth investigating.
Bob stops the connector, and Priya creates pageviews.dlq and copies Jack's record into it before anyone moves the offset. Approving the card records her decision, and nothing on the cluster changes because she approved it. Bob can now recover his team's connector without losing track of what it skipped.
With the record copied, Bob opens Reset Offset for connect-postgres-sink. The drawer initially shows Earliest. Its preview puts the offset at 0.
That would be a very different recovery. Instead of skipping one record, the sink would replay roughly 735,000 earlier pageviews into Postgres. Depending on how the sink handles repeated writes, that could mean duplicates or a lot of unnecessary work.
Bob changes the option to Shift by, enters 1, and checks the preview: partition 0 will move from 735233 to 735234. He presses Reset, then Resume. This time the task stays running, and consumer lag drops from 19,616 to 916 in about a minute and a half.
Console
Home
Topics
Schema registry
Consumer groups
Kafka Connect
Service accounts
Analytics
Insights
Governance
Application Catalog
Topic Catalog
Resource policies
Data quality
Settings
main
Local Kafkalocal-kafka-clusterConsumer Groups/connect-postgres-sinkKafka Connect/Local Connect/postgres-sink
Connector postgres-sink has been resumed successfully
Agentic
Home
Identities
Tasks
Models
Agents
Scheduled
Event-driven
Output
Reports
Recommendations
Approvals
Direct updates
Settings
main
Output/Offset reset → business impact/96fb4939-559b-4d17-85e9-2f26217d1274
AskA
Output
Everything your agents hand back, reports to read, recommendations to act on, approvals to decide, and a record of direct updates.
ReportsRecommendationsApprovalsDirect updates
Search by title, task or agentAgent
OutputAgentProduced
connect-postgres-sink moved forward 1 offset on pageviews: 1 record skipped, likely a deliberate poison-pill skipOffset reset impactOffset reset → business impacta few seconds ago
connect-postgres-sink moved forward 1 offset on pageviews: 1 record skipped, likely a deliberate poison-pill skipRecommendationOffset reset impactOpen
Produced a few seconds ago · agent Offset reset → business impact
DismissMark as actioned
OutputTrace
A suggestion, not a changeNothing happens unless you act on it. Dismissing it is a valid answer.
Finding
What happened. At 2026-11-16T15:46:34Z, bob@conduktordemo.io reset consumer group connect-postgres-sink with a ShiftBy n=1 reset on all of its topics.
How much was skipped or replayed. Shifting by +1 on one partition moves the position forward, so exactly 1 record was skipped. Nothing will be replayed.
Why it was likely done. Bob's audit trail shows a typical poison-pill troubleshooting sequence on the postgres-sink connector (on local-connect): two connector restarts at 15:31 and 15:32 (through his bob-editor token), two task restarts at 15:35 and 15:36, a Stop at 15:41, then the +1 offset shift at 15:46. The reset looks deliberate, meant to step over one record the sink couldn't process.
Business impact.pageviews is clickstream data (about 754k messages, 7-day retention, 3 consumer groups). One missing pageview in Postgres slightly undercounts analytics.
Bob checks the reset preview, shifts forward by one offset, and resumes the sink; an agent then reviews what changed.
The reset triggers another custom agent, Offset reset impact. It reconstructs the sequence from Console's audit log, from Bob's restarts to the stop and the reset. Its finding is specific: connect-postgres-sink moved forward one offset on pageviews, so one record was skipped. It also sizes the damage: one missing pageview in Postgres slightly undercounts the reporting data.
Bob has a working connector. Priya has a preserved copy of the record, an identified sender, and an auditable account of why the offset moved.
Bob diagnosed the failure from his own terminal and made the one-offset change his role allows. Priya used broader visibility to confirm who sent the record and approve a recovery that kept a copy. The agents noticed the repeated failure, proposed the steps, and explained the impact afterward. A person made every change.
Here's where each piece helped:
Piece
What it does
Where it showed up
Conduktor MCP server
Gives an AI assistant Console's tools within the user's permissions
Bob finds offset 735233, but can't read the audit log
Connector recovery (built in)
Reacts to connector actions and escalates when retrying won't help
Each restart leads to an escalation instead of another restart
Unstable connectors (built in)
Reviews failing and repeatedly restarted connectors on a schedule
Priya sees the four restarts as one pattern
Failing sink connector, find the poison pill (custom)
Finds the offending record, checks its origin, and proposes steps for approval
Priya gets Jack's audit event and the recovery plan
Offset reset impact (custom)
Reviews consumer-group resets and estimates their effect
The one-record skip is recorded and explained
There's still a choice for the next failure. Creating pageviews.dlq and copying this record into it preserved evidence; it didn't configure postgres-sink to send future failures there automatically. Priya could make that separate configuration change, using error tolerance and dead-letter-queue settings so malformed records don't stop reporting again. Or she could keep the sink strict and prevent records like Jack's from reaching the topic in the first place. For clients producing through Conduktor Gateway, the Schema ID required policy is one way to enforce that.
Either way, the agents didn't silently discard a bad record or decide the team's data-loss policy. They gave Bob and Priya the context to make a deliberate one-record exception, get the pipeline moving, and see exactly what changed.
Register for our webinar on Tuesday, 8 December, at 11:00 AM ET / 8:00 AM PT to see Conduktor Agents work through connector failures like this one live: Leave the Chores to Kafka Agents. Can't make it? Register anyway and we'll send you the recording.