Kafka Connect SMT Chain Builder
A single message transform chain, in the order it actually runs. That order
is the transforms list and nothing else, which is
the most common SMT mistake.
transforms.properties
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Assuming transforms apply in the order written in the file
They apply in the order named in the transforms list. The property order in the file is irrelevant.
Instead:Read the list, not the file layout.
Chaining a transform that changes the record shape
A later transform sees the record as the earlier one left it, so a field reference valid at the start can be missing by the time it runs.
Instead:Test the whole chain against a real record rather than each transform alone.
Using a predicate expecting it to filter
A predicate decides whether the TRANSFORM applies. Records that do not match pass through unchanged rather than being dropped.
Instead:Use the Filter transform with a predicate to actually drop records.
Transforms fail quietly, which is what makes them hard
Almost every SMT mistake produces a different record rather than an error.
The order is the list, not the blocks
Connect runs the chain in the order of the transforms property. Moving a block in the file changes nothing; reordering the list changes the record. Reviewing somebody else's connector means reading the list first, and it is worth doing because the blocks are usually written in a different order from the one they run in.
Filter with no predicate drops everything
It is a valid configuration, the connector reports healthy, and the entire stream is discarded. The symptom is a sink that consumes the whole topic and writes nothing. A predicate block is emitted with any Filter for that reason.
A narrowing Cast overflows silently
int64 to int32 on a value above two billion produces a wrong number with no error and nothing downstream able to detect it. Widening casts are safe. This is the transform to be most careful with, because the failure is a plausible-looking value rather than an exception.
ReplaceField's include list is allow-only
A field added to the upstream schema later is silently not forwarded, because it is not in the include list. That is data loss caused by somebody else's change, appearing weeks after the connector was written. Prefer the exclude list when the intent is to remove specific fields.
$Key and $Value are different classes
Every transform except RegexRouter and Filter has both forms, and choosing the wrong one is not an error: the transform runs against the half of the record you did not mean and usually does nothing at all. RegexRouter and Filter act on the record, so they have no suffix, and adding one names a class that does not exist.