Kafka Protobuf Binary Decoder

One wire type, four possible readings, and the bytes do not choose

Wire type 2 is shared by string, bytes, every embedded message and every packed repeated field. The three bytes 08 96 01 are both the message {1: 150} and a perfectly ordinary byte string, and nothing in the encoding distinguishes them. This page lists every reading a field admits instead of picking one, because picking one is what produces the confident nonsense that other decoders are known for. Wire type 0 has the same problem more mildly: int32, int64, uint32, uint64, sint32, sint64, bool and every enum share it, so the value is shown read every way at once.

The five bytes in front, and the array after them

A value from Confluent's serialiser starts with a zero magic byte and a four-byte big-endian schema id. For protobuf, and only for protobuf, a message-index array follows that, so the payload begins at byte 5 plus the array rather than at byte 5. Reading from byte 5 leaves the array in front of the payload, which is exactly the decode that comes out garbage by a few bytes. This page detects both and reports the id so the registry can be asked for the schema.

curl -s http://registry:8081\
  /schemas/ids/42 | jq -r .schema

Sign extension is where hand-decoding goes wrong

An int32 holding -1 is widened to 64 bits before encoding, so it occupies ten bytes rather than the five somebody counting bits expects. A buffer truncated inside one of those runs is the usual cause of a last field that reads as nonsense. Values are decoded as arbitrary-precision integers rather than as JavaScript numbers, because a uint64 runs to 18446744073709551615 and a double stops being exact at 2^53.

Stopping early is the diagnosis, not a crash

When the walk cannot read a byte it reports what it read, where it stopped and how much was left, because that is usually the answer: a length that runs past the end of the buffer means either the payload is truncated or the offset it was read from is wrong. Fields decoded before the fault are kept rather than discarded.

What this cannot see

Which message these bytes are. Protobuf payloads are not self-describing at the message level, so with several messages in a .proto nothing in the buffer says which one applies. It also cannot tell an absent field from one set to its default: proto3 does not write defaults, so they are the same bytes. There is no checksum here either, because a protobuf message has none.