Internal topics
The changelog and repartition topics a Streams application creates on the cluster
order-totals KSTREAM-AGGREGATE-STATE-STORE-0000000003
Kafka Streams Internal Topic Predictor
The internal topics a Streams application will create, derived from its
application.id. Create them in advance and you
choose their durability rather than inheriting the broker defaults.
Every internal topic name starts with this. It is also the consumer group id, the state directory name and the prefix your ACLs are granted on, which makes it a security boundary as well as an identity.
Paste below, or drop a file anywhere on this panel
Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.
The answer appears here
Paste on the left and press Predict. Nothing leaves this tab.
Formatted
Results
Nothing else to flag.
No formatting problems, and nothing the rules object to. Worth remembering what that covers: this reads the file you pasted, not the account or cluster it will be applied to.
No finding matches that filter.
Examples
Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.
A repartition
A key-changing operation before an aggregation, and the topic it forces
KSTREAM-KEY-SELECT-0000000002-repartition
Common mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Assuming internal topics are cleaned up
Changelog and repartition topics survive the application. Deleting the app leaves them on the cluster with their retention.
Instead:Run the application reset tool, then delete the internal topics deliberately.
Ignoring changelog retention
A changelog is compacted, so it keeps the latest value per key forever. On a large keyspace that is real and permanent disk.
Instead:Size for it, and set a retention where the store is a windowed one.
Changing the application id casually
It is part of every internal topic name, so changing it orphans the old topics and rebuilds every store from scratch.
Instead:Treat it as permanent.
Streams creates topics at runtime, with the broker defaults
The names are fully determined by application.id, so they can be created in advance instead. That is the difference between choosing their durability and inheriting it.
The names are a formula
A changelog is {application.id}-{store}-changelog and a repartition topic is {application.id}-{name}-repartition. Nothing about either is random or version dependent, which means they can be created before the application starts, with the replication factor, cleanup policy and ACLs you chose rather than whatever the broker defaults to.
A changelog is the durable copy of your state
Every write to a state store goes to its changelog topic first, and that topic is what a new instance replays to rebuild the store after a failover. So a changelog at replication factor 1 means the state has one copy, and losing that broker loses it permanently. No number of standby replicas helps, because every standby reads from the same changelog. This is the reason to create them yourself rather than letting the defaults decide.
The two cleanup policies are opposite
A changelog must be compacted, because it has to keep the latest value for every key indefinitely: created with cleanup.policy=delete instead, the state silently loses whatever aged out and a rebuild produces a store missing older keys. A repartition topic must not be compacted, because it is a transient shuffle read once and then worthless, and compacting one keeps every key forever for no reason at all.
The 249 character limit is on the derived name
Not on application.id. A 230 character application.id is comfortably legal and still produces a changelog topic name over the limit, and Streams fails when it tries to create it, at startup, after the application has already connected successfully. The failure names the topic rather than explaining that the prefix is too long.
What this cannot see
It does not read your topology, so you have to name the state stores and repartition points yourself: one per line, with a trailing exclamation mark for a repartition rather than a store. Which operations create a repartition is a topology question, and the short version is that any key-changing operation followed by an aggregation or a join does. For the ACLs these topics need, the ACL generator on this site emits the prefixed grant a Streams application requires.
Related kafka tools
Kafka Streams Topology Viewer Count the repartitions Kafka CLI Command Builder kcat is librdkafka, not Kafka Kafka Streams Config Generator application.id is four things at once Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka Topic Regex Subscription Tester Kafka matches the whole name
All 73 kafka tools