Paste a connector configuration, as .properties or as the JSON
body you would POST, and see the pipeline in the order it actually runs.
Transforms execute in the order of the transforms list, not the
order of the blocks in the file.
Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.
The answer appears here
Paste on the left and press Draw it. Nothing leaves this tab.
Nothing else to flag.
No formatting problems, and nothing the rules object to. Worth remembering what that covers: this reads the file you pasted, not the account or cluster it will be applied to.
No finding matches that filter.
Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.
The connector, its transform chain and where records go
{"name":"pg-source","config":{"connector.class":"io.debezium.connector.postgresql.PostgresConnector","tasks.max":"1","transforms":"unwrap","transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState"}}Where failed records go, and the settings that decide whether they are kept at all
{"name":"s3-sink","config":{"connector.class":"io.confluent.connect.s3.S3SinkConnector","tasks.max":"4","topics":"orders","errors.tolerance":"all","errors.deadletterqueue.topic.name":"dlq"}}These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
A JDBC source with one table produces one task no matter what tasks.max says. The extra capacity is not used and the number is misleading.
Instead:Match tasks.max to the real parallelism: partitions for a sink, tables or partitions for a source.
SMTs apply in the order named in the transforms list, and an ExtractNewRecordState after a field-level transform operates on a different record shape than expected.
Instead:Name them in the order they should run, and test with a real record.
A JsonConverter reading Avro produces a deserialization error on every record, and the connector fails fast rather than skipping.
Instead:Match key and value converters to what is actually on the topic, which are separate settings.
This is the fact the picture exists for. A source runs its transforms and then the converter; a sink runs the converter and then its transforms. An SMT chain does not transplant between the two unchanged, and nothing in the config says so.
On a source connector the transform runs on the connector's own record, before the converter turns it into bytes, so it sees the connector's schema and types. On a sink connector the converter runs first, so the transform sees a record that has already been deserialised from the wire. A transform that reaches into a field by name works in both cases; one that depends on the type of that field often does not, because the converter is where the type is decided.
transforms is a comma-separated list of aliases, and that list is the execution order. The transforms.<alias>.* blocks below it can appear in any order at all, and frequently do, because people group them by what they configure rather than by when they run. A chain that reads correctly top to bottom can run in a completely different order, and the only place the real order appears is that one line.
transforms=unwrap,mask,route
transforms.mask.type=...MaskField$Value
transforms.route.type=...RegexRouter
transforms.unwrap.type=...ExtractField$Valuepredicates defines them and transforms.<alias>.predicate attaches one to a single transform. When it does not match, that transform is skipped and the record carries on through the rest of the chain untouched. With negate=true the gate is inverted. A predicate defined and never attached does nothing whatsoever and raises no error, which is why this page reports one.
It only exists for sink connectors. It only receives anything when errors.tolerance is all. And it only catches converter and transform failures, never a failure inside the connector itself. Setting the topic and leaving the tolerance at its default none is the commonest way to end up with an empty DLQ and a stopped task, because the first bad record fails the task instead of being routed.
errors.tolerance=all
errors.deadletterqueue.topic.name=dlq
errors.deadletterqueue.context.headers.enable=trueWhether the connector class exists or what its own settings mean, because those are the connector's business and there are hundreds of them. Whether a transform's configuration is valid for its type, for the same reason. The worker's converter defaults, so a config that sets no converter is drawn as inheriting one. It reads the direction from the class name, which is a convention rather than a rule, and says so when the name does not follow it.