Paste the output of topology.describe() and see the shape of it.
The number that matters is the sub-topology count: each boundary is a full
round trip through the broker, and the description states it only as a
heading.
Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.
The answer appears here
Paste on the left and press Draw it. Nothing leaves this tab.
Nothing else to flag.
No formatting problems, and nothing the rules object to. Worth remembering what that covers: this reads the file you pasted, not the account or cluster it will be applied to.
No finding matches that filter.
Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.
Sub-topologies, sources and sinks read out of describe() output
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [input])
--> KSTREAM-FILTER-0000000001
Processor: KSTREAM-FILTER-0000000001 (stores: [])
--> KSTREAM-SINK-0000000002
<-- KSTREAM-SOURCE-0000000000
Sink: KSTREAM-SINK-0000000002 (topic: output)
<-- KSTREAM-FILTER-0000000001An aggregation, its state store, and the changelog topic it creates on the cluster
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [orders])
--> KSTREAM-AGGREGATE-0000000001
Processor: KSTREAM-AGGREGATE-0000000001 (stores: [order-totals])
<-- KSTREAM-SOURCE-0000000000These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Each sub-topology is scheduled independently, and tasks are assigned per sub-topology. A repartition splits one topology into two that no longer share state or ordering.
Instead:Read the sub-topology boundaries as the real parallelism unit.
Any key-changing operation followed by an aggregation creates an internal repartition topic, which is real network traffic, real disk and real retention on the cluster.
Instead:Check what the topology creates before deploying it. The topic predictor on this site lists them.
Stores are backed by changelog topics that are compacted and kept forever by default. Deleting the application without cleanup leaves them on the cluster.
Instead:Run the reset tool, and delete the internal topics deliberately.
Everything else in a topology description is detail. The number of sub-topologies is how many times each record is written back to Kafka and read again, and it is stated only as a heading number nobody reads as a cost.
Streams splits a topology wherever the key changes, because the data has to be redistributed before the next stage can group or join on it. That split is a repartition: the record is serialised, produced to an internal topic, replicated, fetched back and deserialised. Two sub-topologies means one round trip per record; four means three. It is almost always the largest cost in a Streams application and it is invisible in the description unless you already know to count the headings.
selectKey, map, groupBy and a join on a different key all tell Streams the key may have changed, and Streams believes them. mapValues and flatMapValues promise the key is untouched, so no repartition is inserted. Swapping one for the other is often a one-line change that removes an entire round trip, and it is the first thing to look for when this page shows more sub-topologies than you expected.
// forces a repartition
stream.map((k, v) -> KeyValue.pair(k, f(v)))
// does not
stream.mapValues(v -> f(v))Streams creates one task per sub-topology per input partition, and a task is the unit of assignment. Multiply the sub-topology count by the partition count and that is the maximum number of threads that can ever do work. Instances beyond it hold no tasks and sit idle, which looks like a scaling problem and is a partitioning one.
Every store is backed by a compacted changelog unless it was explicitly built with logging disabled. That is broker disk proportional to the state, and it is what has to be replayed to rebuild the store when a task moves to another instance. Long stalls after a rebalance are usually restore, and standby replicas trade memory for cutting that time rather than cutting the disk.
Partition counts, so it can state the task-count formula and not the number. Throughput, so it can say a boundary is expensive and not how expensive. Whether a store has logging disabled, because the description does not record it. It also cannot see what your processors do: a custom Processor that writes to an external system is a black box here and shows as a node with no successors.