Kafka Rebalance Duration Estimator
How long a rebalance takes, how much of it is just noticing a dead consumer, what a rolling restart costs, and what static membership removes.
kafka-rebalance.txt
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Restarting brokers faster than the cluster recovers
Each restart leaves partitions under-replicated until the replica catches up. Restarting the next broker before then can drop below min.insync.replicas and stop writes.
Instead:Wait for UnderReplicatedPartitions to return to zero between brokers.
Ignoring controlled shutdown
Without it, leadership fails over reactively and clients see errors. With it, leadership moves before the broker stops.
Instead:Leave controlled.shutdown.enable on and allow time for it.
Estimating from broker count alone
The cost is dominated by the data each broker must re-replicate, which depends on partition count and retention, not on how many brokers there are.
Instead:Estimate from bytes to move and the replication throughput available.
What a rebalance actually costs
Detection, a barrier that waits for the slowest member, and then assignment. Most of the time is in the first two, and both are configurable.
A clean close costs nothing to detect, a kill costs the session timeout
A consumer that calls close() sends LeaveGroup and the coordinator reacts immediately. A consumer that is killed sends nothing, so it is only noticed when its session expires, which at the 45 second default is roughly a hundred times longer than the rebalance itself. Most reports of slow rebalances are really reports of consumers being SIGKILLed, and the usual cause is a container termination grace period shorter than the shutdown hook needs. That is a deployment setting, not a Kafka one.
The barrier is the slowest member, not the average
Assignment cannot happen until every member has sent JoinGroup, and a member only notices the rebalance when it next polls. So the whole group waits for whichever consumer is deepest into processing its current batch. One member with a slow downstream call sets the rebalance time for everyone. Reducing max.poll.records reduces this directly, and it is more effective than any timeout change.
Static membership removes rebalances rather than shortening them
With group.instance.id set, a member that returns within session.timeout.ms takes its own partitions back and no rebalance happens at all. A rolling restart of twenty consumers goes from twenty rebalances to zero, provided each restart completes inside the session timeout. This is the single largest available win on consumer availability and it costs one config line, with one condition: the deploy has to be faster than the session timeout, so a slow image pull turns the win back into a rebalance.
Eager stops everything, cooperative stops what moves
The eager protocol revokes every partition from every member, including the partitions that are about to be assigned straight back to the same consumer. Cooperative revokes only what actually moves and pays for it with a second round. The difference is not the rebalance duration so much as what stops consuming during it. Switching needs a rolling restart with both assignors listed before the old one is removed, which is a two-deploy operation rather than a config change.
Exceeding max.poll.interval.ms is a loop, not a warning
If processing between polls outlasts max.poll.interval.ms, the coordinator evicts the consumer and rebalances. The evicted consumer then reprocesses the same records, takes just as long, and is evicted again. That is the group that rebalances continuously and never commits an offset, and raising the timeout usually postpones it rather than fixing it. Smaller batches fix it.
What this cannot see
It does not know your assignor's actual computation time, your network, or how long your consumers take to warm a cache after being assigned a new partition, which for a stateful consumer can dwarf everything modelled here. The SyncGroup figure is a small linear estimate rather than a measurement. Treat the outputs as the shape of the cost and where it sits, not as a prediction to the millisecond.