Kafka Streams Config Generator
A Kafka Streams configuration. application.id is
four things at once, which is why changing it is a new application rather
than a rename.
streams.properties
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Treating application.id as a name
It is the consumer group id, the internal topic prefix, the state directory name and the transactional id prefix. Changing it orphans every internal topic and rebuilds all state.
Instead:Choose it once and treat it as permanent.
Running with the default state directory in a container
It defaults to a temporary path, so every restart rebuilds state from the changelog, which on a large store is slow and hammers the cluster.
Instead:Mount a persistent volume and point state.dir at it.
Setting num.standby.replicas to zero and expecting fast failover
With no standby, a failed instance's state is rebuilt from the changelog before processing resumes.
Instead:Set at least one standby where recovery time matters.
application.id is four things, and that is the whole difficulty
One string is the group id, the internal topic prefix, the state directory name and the transactional id prefix.
Changing it is a new application
A different application.id has no committed offsets, so it reprocesses from auto.offset.reset. It has no state, because the state directory and the changelog topics are both named after the old one. And it needs new ACLs, because those are granted on the topic prefix. None of that is a rename, and none of it produces an error: the application starts cleanly and behaves as though it had never run.
state.dir has to survive a restart
Every state store is backed by a changelog topic, and the local copy in state.dir is what makes reads fast. Lose it and Streams replays the entire changelog to rebuild, which on a large store turns a rolling deploy into minutes of the application appearing to hang. An emptyDir volume or a plain container filesystem defeats the whole mechanism, and this is why a Streams application is a StatefulSet with a PersistentVolume.
Standby replicas are what make a failover fast
With num.standby.replicas=0 a failover rebuilds the state store from its changelog on the new instance, from the beginning, while that partition is unprocessed. That is the failure case, which is exactly when you can least afford it. One standby keeps a warm copy and turns minutes into seconds, at the cost of the extra disk and the changelog traffic to maintain it.
exactly_once_v2 is the only exactly-once value left
exactly_once and exactly_once_beta were both removed in Kafka 4.0, so a config naming either does not start. Under v2 the commit interval defaults to 100 ms rather than 30 s, which is a three hundredfold increase in transaction volume and the real cost of the guarantee. It also needs the input topics, the output topics and the changelogs on one cluster, because a transaction cannot span clusters.
What this cannot see
It does not know your topology, so it cannot tell you the task count or which operations create a repartition topic. The Streams internal topic predictor on this site derives the topic names from application.id so they can be created in advance, and the ACL generator emits the prefixed grant a Streams application needs to create them at all.