Kafka Confluent Wire Format Decoder

The format, in full

One byte of framing, four bytes of schema id, then the payload. The id is a plain 32-bit integer in network byte order, which is big-endian. It identifies a schema inside one registry, and it says nothing about which subject or which version that schema belongs to.

00  00 00 04 d2  06 41 64 61 ...
^   ^~~~~~~~~~~  ^~~~~~~~~~~~~~~
|   |            payload: Avro, protobuf or JSON, decided by the schema
|   schema id 1234, 4-byte big-endian
magic byte, always 0x00

Why your deserializer says the data is corrupt

A plain Avro reader starts at byte 0 and reads 0x00 as the first field of your record. Everything after that is off by five bytes, so it reports an unexpected type, a negative string length, or a value that decodes but is nonsense. A JSON parser sees a null byte and gives up on line 1 column 1. Confluent's own deserializer is the one component that says something useful, and even then only if the magic byte is wrong: the message is literally 'Unknown magic byte!', which tells you the framing is missing rather than that the framing exists.

# what people see, in order of how confusing it is
org.apache.avro.AvroRuntimeException: Malformed data. Length is negative
com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 0))
google.protobuf.message.DecodeError: Error parsing message
org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

Protobuf hides a sixth byte, and sometimes more

This is the part that costs an afternoon. A schema id names a whole .proto file, which can declare several message types and nest them, so the id alone does not say which message was serialised. Confluent's protobuf serialiser writes a message-index array after the id to settle it: a ZigZag varint count, then that many ZigZag varint indexes. There is one special case, and it covers most files: a single 0x00 byte means the indexes are [0], the first message type. Avro and JSON Schema have no equivalent, so the offset your payload starts at is 5 for those two and 5 plus the array for protobuf.

Avro and JSON Schema
  00 | 00 00 04 d2 | payload ...                    payload at offset 5

Protobuf, the first message type in the file
  00 | 00 00 04 d2 | 00 | payload ...               payload at offset 6
                     ^^ the [0] shorthand

Protobuf, the second message type
  00 | 00 00 04 d2 | 02 02 | payload ...            payload at offset 7
                     ^^ count 1, ZigZag
                        ^^ index 1, ZigZag

Protobuf, a message nested inside the second one
  00 | 00 00 04 d2 | 04 02 00 | payload ...         payload at offset 8
                     ^^ count 2
                        ^^ ^^ indexes 1 then 0

The schema id is not a version, and it is not global

Two things people assume and neither is true. The id is not the version: version 1 of your subject might be id 47, and version 2 might be id 51, because ids are handed out per schema across the whole registry while versions count within one subject. And the id is only meaningful inside the registry that issued it, so the same four bytes mean a different schema in staging than in production. Copying a payload between environments and expecting it to deserialise is how that gets discovered.

curl -s http://registry:8081/schemas/ids/1234
{"schemaType":"AVRO","schema":"{\"type\":\"record\", ... }"}

curl -s http://registry:8081/schemas/ids/1234/versions
[{"subject":"orders-value","version":3}]

# schemaType is how you learn whether to strip 5 bytes or 6

Reading the id yourself

Five lines in any language, and worth having in a debugging script rather than reaching for a consumer with a registry client configured. Two details to keep right: the integer is big-endian, and Java reads it signed, so a payload with a corrupted top byte asks the registry for a negative id and gets a 404 rather than the large positive number you might expect.

# Python
import struct
magic, schema_id = struct.unpack(">bI", value[:5])
assert magic == 0
payload = value[5:]                  # Avro and JSON Schema

// Java, which is what the Confluent deserializer does
ByteBuffer buffer = ByteBuffer.wrap(value);
if (buffer.get() != 0x0) throw new SerializationException("Unknown magic byte!");
int schemaId = buffer.getInt();      // signed, big-endian

# shell, for one message off a topic
kcat -C -t orders -c 1 -e -f '%s' | xxd | head -1

Keys are framed the same way, with their own id

If a producer registers schemas for both, the key carries its own 5-byte header and its own schema id, usually pointing at a different subject: orders-key rather than orders-value under the default TopicNameStrategy. A key that is a plain string, which is common, carries no framing at all. So a key and a value off the same topic can legitimately disagree about whether the first byte is 0x00, and that is not a bug.

Other registries frame differently

The magic byte 0x00 is Confluent's. Apicurio Registry's default id handler writes an 8-byte long where Confluent writes a 4-byte int, behind the same 0x00, which means a Confluent-shaped read of an Apicurio payload reports schema id 0 with the real id sitting in the next four bytes. AWS Glue Schema Registry does not use 0x00 at all: it writes a header version byte 0x03, a compression byte, then a 16-byte schema version UUID, so eighteen bytes of framing and an identifier that is not a number. Both produce exactly the same symptom as a Confluent mismatch and neither has the same fix.

Confluent   00 | 00 00 04 d2                      | payload
Apicurio    00 | 00 00 00 00 00 00 00 37          | payload
AWS Glue    03 | 00 | 8f 4c 1d 2e ... (16 bytes)   | payload
                 ^^ compression: 00 none, 05 zlib