CRC32C against CRC32
Kafka uses CRC32C (Castagnoli) since 0.11, and the two differ for the same bytes
hello world
Kafka CRC32C Calculator
Compute CRC32C and CRC32 over any bytes, or paste a whole record batch and have its stored checksum checked. When it does not match, the report names which mistake produced the number you have.
Hex is tried before base64, because a hex string is often valid as both
and nobody pasting 0a1b2c3d means base64.
Separators, newlines and 0x prefixes are
stripped. The bytes decide the checksum, so reading four text characters
rather than two bytes is a different and equally valid answer to a
different question.
Paste below, or drop a file anywhere on this panel
Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.
The answer appears here
Paste on the left and press Checksum. Nothing leaves this tab.
Formatted
Results
Nothing else to flag.
No formatting problems, and nothing the rules object to. Worth remembering what that covers: this reads the file you pasted, not the account or cluster it will be applied to.
No finding matches that filter.
Examples
Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.
A longer payload
The same two checksums over a realistic record value
{"orderId":1001,"status":"shipped","total":42.5}Common mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Using CRC32 where Kafka uses CRC32C
Kafka moved to CRC32C (Castagnoli) with the v2 record batch format in 0.11. The polynomials differ, so the checksums differ for the same bytes.
Instead:Use CRC32C for anything touching modern Kafka batches.
Computing the CRC over the whole batch
The batch CRC covers the bytes AFTER the CRC field, not the whole batch. Including the header bytes gives a value that will never match.
Instead:Start from the attributes field, which is where the broker starts.
Why a Kafka checksum disagrees
Four causes, and only one of them is actually damaged data. The other three are all mistakes in how the number was computed.
There are two algorithms, and they changed at the same time as the format
Message formats v0 and v1 use CRC32, the zlib polynomial that `gzip`, `png` and almost every command line `crc32` tool computes. Format v2, from Kafka 0.11 onward, uses CRC32C, the Castagnoli polynomial. A library that hardcodes one of them produces a checksum the broker rejects, and the failure reads as corruption. This page computes both every time, so a value that matches the other one identifies the bug immediately.
The v2 checksum does not cover the whole batch
It covers from the attributes field at offset 21 to the end. In front of it sit the base offset, the batch length and the partition leader epoch, and then the checksum field itself. The reason is that the broker rewrites some of those: the partition leader epoch changes on a leader election and the base offset is assigned on append, so a checksum covering them would be invalidated by the broker's own normal behaviour. Anything that covers them was computed over the wrong range.
A truncated paste is the most likely cause of all
A v2 batch declares its own length in the four bytes at offset 8, counting everything after that field. If the buffer is shorter than the batch says it should be, nothing is wrong with the bytes that are there and there is simply less of them, which explains the mismatch on its own. If it is longer, that is usually several batches concatenated and each has to be checked separately. This page reports the difference rather than leaving it to be guessed at.
A checksum is not a signature
CRC32C detects accidental change: a flipped bit on a wire, a bad disk sector, a truncated write. It detects nothing deliberate, because anyone who alters the bytes can recompute it in a microsecond. A matching checksum means the batch survived transport, and says nothing at all about where it came from. If you need that, you need authentication, not a checksum.
What this cannot see
It reads bytes and checks arithmetic. It does not parse the records inside a batch, decode their keys or values, or validate anything about the compression codec named in the attributes field, which means a batch whose payload is compressed is checksummed as the compressed bytes it is, exactly as the broker does. For decoding a payload, the Confluent wire format decoder on this site reads the five bytes in front of a Schema Registry message.
More kafka tools
Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro Binary Decoder Wrong schema, no error Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka