Kafka Partition Count Calculator
A partition count from throughput, per-partition capacity and consumer group size, with the constraint that actually binds named. Partitions can be added and never removed, so this is a number worth getting right once.
kafka-partitions.txt
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Choosing a partition count you can grow into later
Partitions can be added and never removed, and adding them breaks per-key ordering for existing keys permanently.
Instead:Size for expected growth up front. Over-provisioning slightly is far cheaper than resharding.
Adding partitions to fix consumer lag
It only helps if the consumers are already at the partition count and the lag is even. If lag is concentrated on a few keys, more partitions changes nothing.
Instead:Check per-partition lag first.
Ignoring the per-broker partition ceiling
Each partition costs file handles, memory and controller work. Tens of thousands per broker degrades recovery time badly.
Instead:Keep the total in mind, not only the per-topic number.
Choosing a partition count you will not regret
Three constraints set the floor and only one of them binds. The reason to think about it carefully is that the number is effectively permanent.
Partitions are the unit of parallelism, and the unit of ordering
A partition is one ordered log with one leader. Throughput scales by adding partitions, and so does consumer parallelism, because a partition goes to exactly one consumer in a group. Ordering is guaranteed only within a partition, which means the same property that gives you parallelism is the one that takes away total ordering. Everything else follows from that trade.
The binding constraint is usually the consumer
Producer throughput per partition is generally high and mostly a function of batching. Consumer throughput per partition is whatever your processing allows, and if each record involves a database write or an HTTP call it can be two orders of magnitude lower. The report names which of the three constraints is actually setting the number, because tuning the other two moves nothing.
Adding partitions later rehashes your keys
This is the reason to overshoot. `murmur2(key) % partitionCount` is the default assignment, so increasing the count changes where existing keys land. Any consumer relying on per-key ordering sees that ordering break across the change, and the fix is not a redeploy, it is accepting the gap or rebuilding the topic. If your messages are keyed, treat the count as permanent.
More partitions is not free
Each replica is a set of open files on its broker, each leader is a fetch target, and each partition in a consumer's assignment gets its own fetch buffer, so consumer memory scales with the assignment. The figure to watch is replicas per broker, which is partitions times replication factor divided by brokers, and it is what makes a broker restart slow rather than what makes throughput low.
What this cannot see
It does not know your key distribution, and a hot key defeats any partition count: if 40% of your traffic shares one key then 40% goes to one partition regardless of how many exist. It also does not know your message size or your retention, which is what the disk and retention calculator on this site is for. Measure the per-partition figures on your own hardware with your own payload before trusting a count derived from them.