Kafka Connect Config Validator

Check a Kafka Connect connector config for the mistakes that do not raise an error: a Filter with no predicate, a transformation missing from the transforms list, errors.tolerance=all with nowhere for bad records to go, and a literal password destined for the config topic.

Partitions cap a sink's useful tasks.max; leave it 0 to skip that check. The broker count matters for the dead letter topic, whose replication factor defaults to 3 and fails to create on a smaller cluster.

Paste below, or drop a file anywhere on this panel

Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.

The answer appears here

Paste on the left and press Validate. Nothing leaves this tab.

Examples

Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.

More tasks than partitions

tasks.max above the topic's partition count, so the extra tasks do nothing

{"connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector","tasks.max":"12","topics":"orders"}

A source connector

The settings a source needs, and the ones only a sink uses

{"name":"pg-source","connector.class":"io.debezium.connector.postgresql.PostgresConnector","tasks.max":"1","database.hostname":"pg"}

Common mistakes

These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

  1. Setting tasks.max above the available parallelism

    A sink cannot use more tasks than the topic has partitions, and a source is limited by what it can split.

    Instead:Match tasks.max to partitions for a sink.

  2. Omitting the name field

    A connector posted without a name is rejected, and the error is about the request body rather than the field.

    Instead:Include name at the top level, outside config, when posting to the REST API.

  3. Leaving errors.tolerance at none

    One bad record stops the connector entirely, which for a stream of mixed data is an outage.

    Instead:Set errors.tolerance: all with a dead letter queue, so bad records are kept rather than dropped.

The Connect misconfigurations that do not raise an error

Connect validates the settings it owns and accepts a great deal that then does nothing, or does something destructive. Those are the ones worth checking before a deploy.

A Filter with no predicate deletes everything

The Filter transformation drops the records its predicate matches. With no predicate it matches every record, so the connector runs, reports RUNNING, consumes the topic and writes nothing. There is no error and no dead letter record, because from Connect's point of view nothing failed. This is the most destructive Connect misconfiguration that produces no signal at all, and it is one missing line away from a working config.

transforms=dropTest
transforms.dropTest.type=org.apache.kafka.connect.\
  transforms.Filter
# no transforms.dropTest.predicate
#   -> every record dropped, connector healthy

A transformation not in the transforms list never runs

The transforms list is both the set and the order. Settings for an alias that is not in it are perfectly valid, perfectly inert, and completely silent: the config is accepted, the connector runs, and the transformation you configured does nothing. It usually happens when an alias is renamed in one place and not the other, and it looks identical to a transformation that is running and having no effect.

errors.tolerance=all without a dead letter queue is silent data loss

With tolerance all and no dead letter topic, a record that fails conversion or a transformation is discarded and the task carries on. The connector stays RUNNING, no metric moves in a way anyone alerts on, and the only trace is a log line if errors.log.enable happens to be on. Pair the tolerance with a dead letter topic and context headers, and give that topic a short retention: it is a diagnostic buffer, not a store.

JsonConverter wraps everything in a schema envelope by default

schemas.enable defaults to true, which means JsonConverter expects and produces {"schema": {...}, "payload": {...}} rather than plain JSON. A sink reading ordinary JSON fails to deserialize; a source writes records that look wrong to every other consumer of the topic. The default is the surprising direction, and the setting has to be repeated per side: setting it for value does nothing for key.

A literal password here is a password in a Kafka topic

Connect stores connector configs in its config topic in plain text and serves them from the REST API to anything that can reach the worker. A password in a connector config is therefore in a Kafka topic, in every backup of it, and in the topic's history even after the connector is deleted. Config providers exist for this: a ${file:...} reference is resolved by the worker at task start and never stored. This page names the settings that look like credentials and never prints their values.

What this cannot check

Only the framework keys. A connector declares its own settings, so validating those needs the plugin installed, and anything unrecognised here is reported as belonging to the plugin rather than as a mistake. Connect's own endpoint does the rest: PUT /connector-plugins/<class>/config/validate against a worker that has the plugin returns per-setting errors and recommended values.

More kafka tools

Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka

Elsewhere on the site

Terraform Validator & Formatter Kubernetes YAML Generator Kubernetes YAML Validator AWS IAM Policy Validator & Analyzer