Kafka Connect Sink Connector Generator
The JSON for a sink connector, with a dead letter queue that can actually be
diagnosed. The combination to avoid is tolerance all
with no queue configured.
sink-connector.json
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Enabling a dead letter queue without the context headers
errors.deadletterqueue.context.headers.enable defaults to false, so failed records arrive with no indication of what failed or why. The queue becomes a pile of records nobody can act on.
Instead:Turn the context headers on. They carry the topic, partition, offset and the exception.
Setting errors.tolerance to all with no dead letter queue
Bad records are dropped silently. This is data loss configured on purpose and usually by accident.
Instead:Pair tolerance with a dead letter queue, always.
Setting tasks.max above the topic's partition count
A sink cannot use more tasks than there are partitions, so the extra tasks idle and the number misleads whoever reads it.
Instead:Match tasks.max to partitions.
The dead letter queue settings that decide whether it is useful
There are three ways to configure sink error handling and only one of them is a good idea.
The silent-discard combination
errors.tolerance=all with no dead letter queue configured throws failures away. No task failure, no record kept, and nothing in the log unless errors.log.enable is also set. It reads as a perfectly healthy connector that is quietly losing data, and it is the worst of the three configurations because the other two at least tell you something.
Context headers are what make a DLQ diagnosable
Without errors.deadletterqueue.context.headers.enable, a dead letter record carries the original payload and nothing else, so the queue is a pile of records nobody can explain. With it, the headers name the connector, the task, the original topic, partition and offset, and the exception. It is emitted here for that reason, and it is off by default in Kafka.
The DLQ topic's replication factor can stop the connector
The connector creates the dead letter topic, at the replication factor you configure, which defaults to 3. On a cluster with fewer brokers the creation fails and the connector fails with it, so a queue intended to make the connector resilient prevents it from starting at all.
Task count is capped by partitions
A sink task is a consumer in a group, so a sink cannot be more parallel than the total partition count of its topics. Tasks beyond that sit idle exactly as extra consumers in a group do, and the fix is more partitions rather than more tasks.
auto.create and auto.evolve let a producer change your schema
With them on, a JDBC sink creates and alters tables in the target database from the schema of the records arriving. That means whoever controls the producer controls the production database schema. Both are off here: the failure with them off is explicit and fixable, and with them on it is a silent DDL change nobody reviewed.