Kafka Avro Binary Decoder

Decode an Avro payload against its schema, byte by byte with an offset per field. The wrong schema does not fail, so the signals that do exist are the verdict: bytes left over, a union index that cannot exist, a length that runs past the end.

The payload accepts base64, hex or a decimal byte array. A short all-hex string is genuinely valid as both hex and base64, and the two give different bytes, so the reading that produces a plausible first byte is taken and the other one is reported.

Paste below, or drop a file anywhere on this panel

Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.

The answer appears here

Paste on the left and press Decode. Nothing leaves this tab.

Examples

Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.

Ambiguous encoding

Bytes that could be raw Avro or Confluent wire format, and why the tool refuses to guess

0c48656c6c6f

A record payload

An int and a string decoded against the schema, field by field

9601 0a 68 65 6c 6c 6f

Common mistakes

These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

  1. Feeding Confluent-framed bytes to a raw Avro decoder

    The first five bytes are a magic byte and a schema id. Decoding from byte zero either fails or produces nonsense that looks like corruption.

    Instead:Strip five bytes, or let the tool detect the framing.

  2. Assuming Avro binary is self-describing

    It is not. The bytes carry no field names and no types; without the writer schema they are unreadable.

    Instead:Keep the schema with the data, or use a registry.

  3. Reading a long as if it were fixed width

    Avro encodes integers as zigzag varints, so a value's byte length depends on its magnitude.

    Instead:Decode with the schema rather than by offset.

Avro binary carries no type information at all

A record is its fields back to back, with no names, no lengths and no tags. That is what makes it small, and it is why decoding with the wrong schema does not fail: it returns plausible garbage.

Bytes left over is the strongest signal there is

A correct schema consumes the buffer exactly. Anything remaining means the writer had fields this schema does not, or that more than one record was pasted. Since the values themselves will look entirely reasonable either way, the leftover count is the verdict on this page rather than a note underneath one. It is evidence and not proof in the other direction too: two schemas of the same shape decode the same bytes into different field names, and nothing in the encoding distinguishes them.

The five bytes in front, from the Confluent serializer

A value produced by the Confluent Avro serializer starts with a zero magic byte and a four-byte big-endian schema id, and the Avro datum begins at byte 5. Decoding from byte 0 reads the magic byte as the first field and everything after it is misaligned. This page detects the header and removes it, and reports the id so you can fetch the exact schema it was written with.

curl -s http://registry:8081\
  /schemas/ids/42 | jq -r .schema

The refusals, which exist so the tab keeps responding

A random byte read as an array count becomes a request for billions of items, and an array of nulls costs nothing per item, so a naive decoder loops rather than failing. A block count past a hundred thousand is refused here instead of attempted. Same for a variable-length integer past ten bytes, which cannot encode a 64-bit value and is always a misaligned read rather than a large number.

Longs are decoded exactly, past where a double stops being able to

Avro's long is 64 bits and a JavaScript number holds integers exactly only to 2^53. A timestamp in microseconds is already past 2^50. Anything larger is decoded as an arbitrary-precision integer rather than rounded, because the point at which a decoder starts quietly lying is not a good thing to find out from a reconciliation report.

What this cannot see

Which schema the payload was written with. If it came from the Confluent serializer the id is in the header and the registry has the answer; otherwise there is nothing in the bytes to go on. It also does not verify a checksum, because an Avro datum has none: the CRC lives on the record batch, one level up, and the CRC tool on this site reads that.

More kafka tools

Kafka Confluent Wire Format Decoder The five junk bytes in front of your payload Kafka Key to Partition Mapper Which partition does this key land on? Kafka Topic Name Validator Legal, risky, or 249 characters too long? Kafka Replication Safety Checker How many brokers can you lose Kafka Producer Config Linter Will it start, and will it lose a record? Kafka Message Payload Decoder The first five bytes are usually not data Kafka Connect Source Connector Generator tasks.max is a ceiling, not a count Kafka Connect Sink Connector Generator A dead letter queue with no context headers is a pile of records Kafka Connect SMT Chain Builder The order is the transforms list Kafka MirrorMaker 2 Config Generator It renames every topic by default Kafka Partition Reassignment Generator The throttle is not optional Strimzi Kafka Resource Generator Without the cluster label, nothing happens Kafka mTLS Config Generator The certificate is the identity Kafka Schema Registry Config Generator The compatibility direction is your deployment order Kafka Exactly-Once Config Generator Half of it is worse than none Kafka Broker and KRaft Config Generator The internal topics that break a one-broker cluster Kafka Quota Generator Byte rates are per broker, not per cluster Kafka Streams Config Generator application.id is four things at once Kafka Connect Worker Config Generator Security three times, or the tasks fail Kafka Retention and Unit Converter log.retention.hours does not take milliseconds Kafka Timestamp Converter Two sentinels and two meanings Kafka .properties to YAML Converter Dotted keys stay flat Kafka Streams Internal Topic Predictor Create them before Streams does Kafka ACL Generator The grant you forgot is on another resource type Kafka Topic Config Generator min.insync.replicas is the one that matters Kafka client.properties Generator The file every CLI tool asks for Kafka Producer Config Generator No password field, on purpose Kafka Consumer Config Generator The commit mode decides the semantics Kafka Disk and Retention Calculator retention.bytes is per partition Kafka Partition Count Calculator The number you can never reduce Kafka Cluster Sizing Calculator The traffic no client metric shows Kafka Consumer Lag Catch-Up Calculator Whether it ever clears, not just when Kafka Producer Batching Calculator linger.ms=0 still batches Kafka Segment and Index Sizing Why retention.ms is a lower bound Kafka Rebalance Duration Estimator What a rolling restart really costs Kafka Cost Estimator Your rates, so nothing goes stale Kafka Config Explorer by Version The answer depends on the release Kafka Default Config Reference What moved under a config you never edited Kafka OAuth Bearer Token Decoder Will Kafka accept it, and can it refresh Kafka Record Header Viewer Headers are a list, not a map Kafka Topic Regex Subscription Tester Kafka matches the whole name Kafka ACL Permission Matrix Viewer DENY beats every ALLOW Kafka Connect Config Validator The mistakes that raise no error Kafka Consumer Group Id Validator Which broker coordinates the group Kafka Partition Assignment Visualizer Leadership is the load, not replicas Kafka Consumer Assignment Visualizer The three assignors disagree Kafka ZooKeeper to KRaft Config Converter The authorizer class nobody changes Kafka Config to Strimzi Half of it belongs elsewhere Kafka Docker Compose Generator (KRaft) Reachable from inside and outside Kafka JAAS Config Decoder The line that stops SASL working Kafka CRC32C Calculator Which CRC, over which bytes Kafka Config Upgrade Checker What breaks when you upgrade Kafka Kafka Config Diff Which change actually changed something Kafka Consumer Config Linter Why the group rebalances, and where the records went Kafka Avro Schema Validator The defaults Avro accepts and rejects Kafka Schema Compatibility Checker What the registry will say, before you ask it Kafka Avro Schema Diff Which direction each change breaks Kafka Compression Comparison Measured on your bytes Kafka Delivery Semantics Exactly-once has a consumer half Kafka ksqlDB Query Builder It looks like SQL and the rules are not Kafka Connect SMT Predicate Tester negate reads backwards Kafka Streams Topology Viewer Count the repartitions Kafka Connect Pipeline Visualizer The order things really run in Kafka Protobuf Binary Decoder Works without the .proto Kafka Protobuf JSON Converter Why your JSON does not round-trip Kafka Protobuf to Avro Schema What does not survive the conversion Kafka Avro JSON Converter Why the console producer rejects your line Kafka Avro Sample Data Generator Records that actually serialize Kafka JSON to Avro Schema What JSON cannot tell you Kafka JSON Schema to Avro What does not survive the conversion Kafka SASL JAAS Generator One login module, four syntaxes Kafka CLI Command Builder kcat is librdkafka, not Kafka

Elsewhere on the site

Terraform Validator & Formatter Kubernetes YAML Generator Kubernetes YAML Validator AWS IAM Policy Validator & Analyzer