Kafka Connect Source Connector Generator
The JSON for a source connector, and the task count you will actually get.
A JDBC source creates one task per table however high
tasks.max is set.
source-connector.json
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Reading tasks.max as the number of tasks
It is a ceiling. A JDBC source with one table produces one task regardless, so the extra capacity is imaginary.
Instead:Match it to the real parallelism: tables, partitions or whatever the connector can split.
Omitting a converter for the key
Key and value converters are separate settings. Leaving the key at the worker default while setting the value produces records with a mismatched key format.
Instead:Set both explicitly on the connector.
Assuming a source connector resumes exactly where it stopped
Offsets are committed periodically, so a crash re-reads from the last commit. Sources are at-least-once unless the sink de-duplicates.
Instead:Make the downstream idempotent.
Three things about a source connector that a config file does not show
The task count, what the polling mode cannot see, and the absence of a dead letter queue.
tasks.max is a ceiling
A JDBC source creates one task per table, so tasks.max=10 against three tables gives three tasks. Debezium reads one replication stream and is always single-task whatever you set. Seeing fewer tasks than you asked for is the expected behaviour and it is the most frequently reported non-bug in Connect. The report tells you the number rather than leaving you to discover it.
No polling mode can see a delete
mode=incrementing only returns rows with a higher id, so an update in place is invisible and so is a delete. mode=timestamp can miss two rows written in the same millisecond, because the query must use a strict comparison to avoid re-reading. timestamp+incrementing is the only pair safe against both, and it still cannot see a delete, because a deleted row is not returned by any query. If deletes matter, change data capture is the tool and this is not.
A source connector has no dead letter queue
errors.tolerance and the errors.deadletterqueue settings only apply to a sink. A source failure happens before the record exists in Kafka, so there is nowhere to route it, and a poison row simply stops the task. This asymmetry surprises people who have configured a DLQ on a sink and expect the same on a source.
A Postgres replication slot fills the database's disk
Debezium's slot holds write-ahead log until the connector consumes it. A connector that is paused, failed or deleted while its slot remains makes the database run out of disk, which is a database outage caused by a stopped Kafka connector. Drop the slot when you remove the connector.
What this cannot see
It generates a plausible connector for a shape rather than for your schema, so the connection URL and the table list are placeholders. It also cannot tell you that a connector reporting RUNNING may have every task FAILED: the status endpoint in the output is the only place that shows task state, and it is worth checking after every create rather than trusting the 200 from the PUT.