Kafka Key to Partition Mapper

Paste your message keys and a partition count, and see exactly which partition each one lands on: murmur2 with seed 0x9747b28c over the UTF-8 key bytes, masked and taken modulo the count, which is what the Java producer does. Then write 6 -> 12 to see how many of those keys a resize would move, and what that costs you.

Paste below, or drop a file anywhere on this panel

Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.

The answer appears here

Paste on the left and press Map keys to partitions. Nothing leaves this tab.

Examples

Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.

Adding partitions

Why increasing the partition count breaks per-key ordering for existing keys

partitions: 6 -> 12

order-1
order-2
order-3

Null keys

Records with no key do not hash, they round-robin, so they have no ordering guarantee

partitions: 6

null
null
order-1

Common mistakes

These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

  1. Adding partitions to a keyed topic

    Partition assignment is hash(key) mod partitions, so changing the count remaps existing keys and breaks per-key ordering permanently for records already written.

    Instead:Size partitions ahead of growth, or create a new topic and migrate if ordering matters.

  2. Expecting ordering from null-keyed records

    Records with no key are distributed round-robin, so there is no ordering guarantee between them at all.

    Instead:Supply a key for anything whose order matters relative to other records.

  3. Assuming every client hashes keys the same way

    The Java client uses murmur2. librdkafka defaults to a CRC32 based partitioner, so the same key can land on a different partition from a different language.

    Instead:Set the partitioner explicitly where producers in different languages write to one topic.

What it works out

The partition is arithmetic, so it can be computed exactly rather than guessed at. The parts worth having are the ones that explain a skewed topic or a broken ordering guarantee, both of which look like a mystery in a dashboard and are obvious once the keys are laid out.

The exact partition, per key

murmur2 with seed 0x9747b28c over the UTF-8 key bytes, then the 0x7fffffff mask and the remainder. Implemented here and checked against the six vectors in Kafka's own UtilsTest, so the number matches what your Java producer computes rather than approximating it.

What a resize would move

Write a partitions line of the form 6 -> 12 and every key is mapped twice, before and after, with the ones that move marked. The predicted share is gcd(old, new) / max(old, new), and the observed share is printed next to it.

Where the skew comes from

The distribution is drawn to scale, one lane per partition, so an empty partition is visible as an empty lane. Skew is only reported when it is beyond what chance explains at that sample size: uneven counts on five keys are arithmetic, not a finding.

Key cardinality

A key is pinned to one partition, so the number of distinct keys is a hard ceiling on how many partitions can ever receive data. Three tenant ids across twelve partitions leaves nine of them permanently empty, and raising the count cannot help.

The byte-level traps

A leading or trailing space is part of the key and changes the partition, so nothing is trimmed. A non-ASCII key is flagged because the serializer's encoding decides the bytes. An all-numeric key set is flagged because LongSerializer does not write digits.

What a null key really does

It is not hashed and it is not partition 0. The current default is the built-in sticky partitioner from KIP-794, which sends a whole batch to one partition, and that is the most common cause of skew that no key can explain.

How does Kafka decide which partition a key goes to?

For a record that has a key, the Java producer does not ask the broker and does not keep any state. It hashes the serialised key bytes with murmur2, clears the sign bit, and takes the remainder modulo the partition count. That is the whole algorithm, which is why the same key always lands on the same partition, and why the answer changes the moment the count does.

The one line that decides it

This is the body of the built-in partitioner. murmur2 is a non-cryptographic hash, seeded with 0x9747b28c, which is Kafka's own seed and not the murmur2 reference seed. toPositive is a bit mask and not an absolute value: it clears bit 31 so the remainder cannot be negative. Everything downstream, including the per-key ordering guarantee, falls out of these two operations.

Utils.toPositive(Utils.murmur2(serializedKey)) % numPartitions

  key            "foobar"
  UTF-8 bytes    66 6f 6f 62 61 72
  murmur2        -790332482          (signed 32-bit)
  & 0x7fffffff    1357151166          clears the sign bit
  % 12                     6          the partition

note: 1357151166, not 790332482. A mask, not abs().

The bytes are what is hashed, not the characters

The partitioner runs after the serializer, on whatever bytes it produced. For a StringSerializer that is UTF-8, so any character outside ASCII is more than one byte and the hash sees all of them. Change the encoding and you change the partition, with nothing in either producer or consumer reporting a problem. The same applies to a key that is not a string at all: LongSerializer writes eight bytes big-endian, so hashing the digits of the number gives an unrelated answer.

key.serializer = StringSerializer   (default encoding UTF-8)

  "café"   ->  63 61 66 c3 a9      5 bytes, 4 characters
  "café"   ->  63 61 66 e9         4 bytes, if ISO-8859-1

  same characters, different bytes, different partition

key.serializer = LongSerializer

  1001     ->  00 00 00 00 00 00 03 e9    8 bytes, big-endian
  "1001"   ->  31 30 30 31                4 bytes, as text

Adding partitions re-maps almost every key

Because the partition is a remainder, changing the divisor changes the answer. The share of keys that keep their partition is exactly gcd(old, new) divided by the larger of the two: doubling keeps half of them, and any pair of coprime counts keeps one key in the new count. Kafka never moves messages that are already written, so a moved key has its history on one partition and its new records on another, read by two consumers in parallel. Nothing errors. The ordering guarantee is simply gone.

keys that keep their partition, gcd(old, new) / max(old, new)

   6 ->  7      1 in 7      14.3%
   6 ->  8      1 in 4      25.0%
   6 -> 12      1 in 2      50.0%
   6 -> 18      1 in 3      33.3%
  10 -> 15      1 in 3      33.3%

there is no count that keeps most of them

A null key does not hash at all

With no key there is nothing to feed murmur2, so the partitioner chooses on its own and the choice is not reproducible from the record. It is also not partition 0, which is the usual guess. The behaviour has changed twice: the old DefaultPartitioner round-robined every record, UniformStickyPartitioner filled one batch before switching, and since Kafka 3.3 both are deprecated by KIP-794 and partitioner.class defaults to null, which enables the built-in adaptive sticky partitioner. It sticks to one partition until the batch is full or linger.ms elapses, and prefers brokers that are keeping up.

partitioner.class unset  (Kafka 3.3 and later)

  keyed record     murmur2(key) & 0x7fffffff % partitions
  null key         sticky: one partition per batch,
                   biased towards brokers with less backlog

  partitioner.ignore.keys=true
                   keys are ignored entirely, so per-key
                   ordering is given up on purpose

A short burst of keyless records lands almost entirely on
one partition. On a graph that is indistinguishable from
key skew, and it is the more common cause.

Other clients do not agree with Java

murmur2 with this seed is a Java client detail, not a protocol rule. The broker stores whatever partition the producer chose, so two producers in different languages can write the same key to two different partitions and neither of them is wrong. librdkafka, which is the C library under the Python, Go, .NET and Node Confluent clients as well as kcat, defaults to a CRC32 scheme called consistent_random. Sarama, the other common Go client, defaults to FNV-1a. Both are configurable, and only one of the two settings below actually matches Java.

client                       default partitioner

Java (org.apache.kafka)      murmur2
kafka-python                 murmur2, matches Java
librdkafka family            consistent_random  (CRC32)
  confluent-kafka-python
  confluent-kafka-go
  confluent-kafka-dotnet
  node-rdkafka, kcat
Sarama (Go)                  FNV-1a 32-bit

to align librdkafka with Java:
  partitioner=murmur2_random

Sarama needs a custom Partitioner implementing murmur2.

Adding partitions is a breaking change

kafka-topics --alter --partitions 12 is a one line command, it succeeds instantly, and it is the most consequential thing you can do to a keyed topic. The partition a key goes to is the hash modulo the count, so the moment the count changes almost every key gets a different answer.

Kafka does not move the messages that are already there. It has no mechanism to: partitions are append-only logs with their own offsets, and rewriting them would invalidate every committed offset in every consumer group. So a key that used to live on partition 3 now has its history sitting on partition 3 and its new records arriving on partition 9. Two consumer instances read those two partitions independently and in parallel.

Nothing reports this. There is no warning at alter time, no consumer error, and no metric that goes red. The failure shows up later as a state machine that processed an update before the create, or a balance that briefly went negative, and it will be blamed on everything except the topic resize that happened three weeks earlier.

The share of keys that stay put is exactly gcd(old, new) / max(old, new). Doubling keeps half of them, which is the best case and is still a coin flip per key. Going from 6 to 7 keeps one in seven. Paste your keys above with partitions: 6 -> 12 and you get the exact list, per key, before you run the command rather than after.

More kafka tools

Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka

Elsewhere on the site

Terraform Validator & Formatter Kubernetes YAML Generator Kubernetes YAML Validator AWS IAM Policy Validator & Analyzer