Kafka Disk and Retention Calculator

The disk a topic needs, and the retention.bytes to set for it. Both figures are given, because the setting is per partition and the intention almost never is.

Traffic

One topic. For a whole cluster, add the topics up, or run this once per topic and sum the results.

Retention
Topic and cluster shape

kafka-disk-sizing.txt

updates as you type

    Common mistakes

    These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

    1. Reading retention.bytes as a topic limit

      It is per partition. A topic with 12 partitions and retention.bytes of 1 GB holds up to 12 GB per replica.

      Instead:Multiply by partitions, then by the replication factor.

    2. Sizing disks to the retention figure

      Compaction, segment rolling and a broker being down all need headroom, and a full disk takes the broker offline hard.

      Instead:Leave substantial free space. Kafka does not degrade gracefully on a full disk.

    3. Ignoring compression when estimating

      Retention limits apply to the stored, compressed bytes, so the number of records held depends on how well the data compresses.

      Instead:Measure the ratio on real data rather than assuming.

    Where Kafka disk actually goes

    Four multiplications and two traps. The multiplications are obvious and the traps are what make the estimate wrong by a factor rather than a percentage.

    Replication multiplies everything

    replication.factor=3 means three full copies of every byte, and it is the first thing forgotten when somebody sizes a cluster from a throughput figure. It multiplies disk directly and it multiplies inter-broker network traffic by the factor minus one, which is a separate cost covered by the cluster sizing calculator on this site.

    retention.bytes is per partition

    This is one word in the Kafka documentation and it is worth more than any other sentence here. The setting is a per-partition budget, so the topic holds it multiplied by the partition count, and the cluster holds that multiplied by the replication factor. A 100 GB intention on a 50 partition topic at replication factor 3 becomes 15 TB of disk. Both retention.ms and retention.bytes apply, and whichever is reached first wins.

    A segment is only deleted once it is closed

    Retention does not delete records, it deletes segments, and only after they are closed. An active segment keeps growing until it reaches segment.bytes or segment.ms, so each partition carries up to one extra segment beyond what retention permits, on each replica. Lowering segment.bytes reduces the slack and increases the file count; the report gives both so the trade is visible.

    Size for the peak, because retention holds it

    Retention keeps the busiest hour for the whole retention window, so an average-based estimate is wrong for as long as the window lasts. If traffic doubles for an hour a day and retention is a week, that hour is on disk seven times over.

    What this cannot see

    It does not know your topic count, your actual compression, whether compaction is enabled, or how much of your disk the operating system and the page cache want. A compacted topic behaves differently: its size depends on the number of distinct keys rather than on retention, so these figures are for a delete-policy topic. For the cluster-wide question of how many brokers, and for the replication traffic that comes with the replication factor, the cluster sizing calculator answers the other half.

    More kafka tools

    Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka

    Elsewhere on the site

    Terraform Validator & Formatter Kubernetes YAML Generator Kubernetes YAML Validator AWS IAM Policy Validator & Analyzer