Kafka Connect Worker Config Generator
A distributed Connect worker configuration, with the security block emitted on all three connection paths rather than only the worker's own.
connect-distributed.properties
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Configuring security once
A worker needs it three times: for the worker's own client, with a producer. prefix for sink tasks, and with a consumer. prefix for source tasks. Missing any one leaves those tasks failing to connect.
Instead:Set all three prefixes.
Running distributed mode with a unique group.id per worker
Workers with different group ids form separate one-node clusters that silently do not share connectors.
Instead:The group.id must be identical across the workers in a cluster, and distinct from any consumer group.
Using a replication factor of 1 for the internal topics
config.storage, offset.storage and status.storage hold the cluster's entire state. Losing them loses every connector configuration.
Instead:Set them to at least 3 in production, and remember config.storage must have exactly one partition.
Connect needs its security configuration three times
A worker makes three kinds of connection, and each reads its settings from a different prefix.
Unprefixed, producer. and consumer.
The worker's own connections read the unprefixed settings. The producers it runs on behalf of source connectors read producer.security.protocol and the rest of the producer. prefix. The consumers it runs for sink connectors read the consumer. prefix. Setting only the unprefixed block gives a worker that starts, joins its group and reports healthy, and then fails on every connector task with an authentication error that names the connector rather than the missing prefix.
The three internal topics are the cluster's configuration
Connect stores connector definitions, source connector positions and task status in Kafka topics named after group.id. Losing them loses every connector in the cluster and every source connector's place in its source system. They are compacted rather than time-retained for that reason, and they deserve at least the replication factor of your ordinary data.
group.id must not collide with a consumer group
Connect uses the group membership protocol for its workers, so a consumer group elsewhere with the same name interferes with worker rebalancing. The symptom is connectors restarting for no apparent reason, which is very hard to trace back to a name collision.
The converter decides the message shape, and changing it changes the data
value.converter.schemas.enable=false emits plain JSON. With it true, every record is wrapped in a schema and payload envelope, which is a completely different shape on the wire. Flipping it on a running connector changes the format with no warning and breaks consumers on the next record. Avro with a registry sends a four byte schema id instead of an inline schema, which is why it is smaller and why it needs the registry to be readable at all.
What this cannot see
It generates the worker configuration and not the connectors, which are posted to the REST API as JSON. It also cannot tell you that the REST port has no authentication by default: anyone who can reach it can create a connector, and a connector can read any topic the worker's credentials allow. That belongs behind a network policy at minimum.