More consumers than partitions
Consumers that will sit idle, because a partition goes to exactly one member and the topic has six
consumer-1 consumer-2 consumer-3 consumer-4 consumer-5 consumer-6 consumer-7 consumer-8
See which partitions each consumer in a group would get, under each of Kafka's three built-in assignors. The algorithms are the real ones, so the assignment is what the coordinator would actually hand out.
All three are computed whichever you pick, so the comparison is always in the output. Member ids are sorted the way the coordinator sorts them, which is what makes range's block boundaries land where they do.
Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.
The answer appears here
Paste on the left and press Assign. Nothing leaves this tab.
Nothing else to flag.
No formatting problems, and nothing the rules object to. Worth remembering what that covers: this reads the file you pasted, not the account or cluster it will be applied to.
No finding matches that filter.
Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.
Consumers that will sit idle, because a partition goes to exactly one member and the topic has six
consumer-1 consumer-2 consumer-3 consumer-4 consumer-5 consumer-6 consumer-7 consumer-8
Three consumers over six partitions, which is what range and roundrobin agree on
consumer-1 consumer-2 consumer-3
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
A partition goes to exactly one consumer in a group, so the extras idle and add rebalance cost.
Instead:Scale consumers up to the partition count, no further.
range, roundrobin, sticky and cooperative-sticky produce different assignments, and range in particular skews across multiple topics.
Instead:Set partition.assignment.strategy deliberately.
The eager protocol stops every consumer on any membership change. Cooperative only moves the partitions that need moving.
Instead:Use cooperative-sticky unless something prevents it.
Kafka's assignors are implemented here, not approximated, so the assignment shown is the one the group would get. The comparison is the point: the default is rarely the best of the three.
Range sorts the members, then for each topic separately hands out contiguous blocks starting from the first member. Subscribe to five topics of three partitions each with three members and member 0 gets partition 0 of all five. When partition 0 is hot, which on a keyed topic with a skewed key space it often is, that skew lands on the same consumer every time. It is also the default for the plain consumer, which is why this shape is so common.
3 members, topics orders=3 payments=3
range m0: orders-0 payments-0
m1: orders-1 payments-1
m2: orders-2 payments-2
# fine here. With uneven partition counts the
# remainders pile onto the early members instead of
# averaging out across topics. It lays every partition of every subscribed topic into one list and deals them round robin. The balance across multiple topics is better than range. The cost is that any membership change shifts the deal by one, so nearly every partition moves to a different consumer, which for a stateful consumer means rebuilding nearly every cache.
With no previous assignment to preserve, the sticky assignors aim purely for balance and produce the same shape as round robin. Their advantage shows up on the second assignment, where they keep members on the partitions they already had. This tool computes assignments from scratch, so it shows you the balance and not the stickiness, and it says so rather than implying otherwise. Balance is only half of why CooperativeStickyAssignor is the right default; the other half is that it revokes only the partitions that move.
Range and round robin both use the eager protocol: every partition is revoked from every member before reassignment, including the ones coming straight back. CooperativeStickyAssignor revokes only what moves and pays two rounds for it. The rebalance duration estimator on this site quantifies the difference; this page shows what each one assigns.
Partitions are the ceiling on consumer parallelism. An extra consumer is assigned nothing, and it still joins every rebalance and still counts toward the join barrier, so it makes rebalances slower while consuming nothing. Partitions can be added and never removed, so the count is worth deciding up front.