Kafka Cluster Sizing Calculator
How many brokers, and which of network, disk or the replication floor decides it. Includes the replication traffic that no producer or consumer metric ever shows.
kafka-cluster-sizing.txt
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Sizing from producer throughput alone
Replication multiplies write traffic by the replication factor, and every consumer group multiplies read traffic. A cluster carrying 100 MB/s of produce can be moving a gigabyte.
Instead:Include replication and consumer fan-out in the network budget.
Ignoring the page cache
Kafka reads from the page cache when consumers are caught up and from disk when they are not. A cluster sized on disk throughput alone looks fine until one consumer falls behind.
Instead:Leave most of the machine's memory to the OS, and size disk for the lagging case.
Planning storage without the replication factor
Retention is per partition and per replica. A topic with 1 TB of retention and RF 3 occupies 3 TB across the cluster.
Instead:Multiply by RF, then add headroom for a broker being down.
What actually decides how many brokers you need
Three constraints, computed separately, because which one binds changes completely with the shape of the workload.
Replication traffic is the figure people forget
Produce traffic is what clients report. Replication traffic is that multiplied by the replication factor minus one, it crosses the network on every write, and it appears in no client metric at all. At replication factor 3 it is twice the produce traffic, which means the majority of the bytes moving through the cluster are copies. This is also why a cluster that looks comfortable on producer metrics can be saturated.
Consumer fanout multiplies egress
Each consumer group that reads the whole stream sends another full copy out. Three groups reading everything is three times the produce traffic leaving the brokers, on top of replication. Groups are cheap to add and they are not free to serve, and a new group backfilling from the beginning reads the entire retention window as fast as the brokers will allow.
High volume is network bound, long retention is disk bound
A firehose with an hour of retention needs brokers for their network interfaces. A trickle kept for a year needs them for their disks. Most real clusters are somewhere between, and the useful output is not a single number but which of the two is closer, because that is what a bigger machine should be bigger at.
The replication factor sets a hard minimum
Two replicas of one partition cannot share a broker, so the replication factor is a floor no amount of hardware removes. Three brokers is the practical minimum for replication factor 3, and it is worth pairing with min.insync.replicas=2, which then means exactly one broker can be down while writes continue. Four brokers is the first count where a broker can be down and the cluster is still able to lose another without stopping writes.
What this cannot see
It does not know your partition distribution, whether your leaders are balanced, how much CPU your TLS termination and compression cost, or how big your page cache is. It assumes traffic spreads evenly across brokers, which is what a balanced cluster does and what an unbalanced one does not, and an unbalanced cluster fails at the busiest broker rather than at the average. For the disk figure it takes as input, use the disk and retention calculator on this site.