Kafka ksqlDB Query Builder
Build a ksqlDB statement with the rules attached. Push against pull,
co-partitioning, the mandatory WITHIN on a stream-stream join,
and the KEY_FORMAT default that catches everybody, each stated
next to the choice that triggers it.
query.sql
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Creating a STREAM over a topic that should be a TABLE
A stream is an append-only log of events and a table is the latest value per key. Choosing wrongly gives either duplicate rows or lost history, and the fix is a rewrite.
Instead:Events are a stream. State keyed by an identifier is a table.
Joining a stream to a stream with no window
A stream-stream join requires a WITHIN clause, because without one the join has unbounded state.
Instead:Specify the window, and size it against the real skew between the two streams.
Expecting a persistent query to survive a schema change
A CREATE ... AS SELECT query is bound to the schema at creation. An incompatible change to the source breaks it, and it must be dropped and recreated.
Instead:Plan schema evolution around the queries, and use compatible changes where possible.
It looks like SQL, and the rules that matter have no SQL equivalent
Nearly every ksqlDB mistake is a rule a database does not have. Six of them decide whether a statement runs at all, and none of them is visible in the syntax.
Push and pull are the same keyword away from each other
EMIT CHANGES makes a query a push query: it never completes, it streams results for as long as the client is connected. Without it the query is a pull query, which returns rows and finishes. A pull query can only read a materialised TABLE, and only by key: a stream has no state to read, so there is no way to make one answer a pull query. A CREATE ... AS SELECT is always a push query, which is why the statement below always carries EMIT CHANGES.
A join needs the same key and the same partition count
Co-partitioning is two conditions and ksqlDB only checks one of them. The keys have to match, which it verifies. The partition counts have to match too, which it does not: a join between a six-partition stream and a twelve-partition table produces missing rows and no error at all. That is the hardest ksqlDB bug to find, because the query runs and the output looks plausible.
A stream-stream join must be windowed
There is no unbounded stream-stream join, because holding every record from both sides forever is not a thing a system can do. WITHIN is mandatory and it also decides how much state each instance keeps, so a wide window is a memory and disk cost paid continuously rather than a setting.
PARTITION BY is a repartition, which is the most expensive thing here
Changing the key means the data has to be redistributed, so ksqlDB writes it to a new topic and reads it back. That is a full round trip through the broker per record: serialise, produce, replicate, fetch, deserialise. It is often necessary and it is never cheap, and it is worth knowing you asked for it.
KEY_FORMAT is not VALUE_FORMAT
They are separate settings and the key defaults to KAFKA, which means a raw serialised primitive with no schema at all. Somebody who set VALUE_FORMAT='AVRO' and expects the key to be Avro too gets a key that a schema-aware deserialiser cannot read. The statement below always writes KEY_FORMAT explicitly, so the choice is visible rather than inherited.
What this cannot see
Your schemas, so it cannot tell you a column does not exist. Your partition counts, so it can state the co-partitioning rule and not whether you satisfy it. Whether the output topic already exists, which matters because PARTITIONS and REPLICAS are ignored silently when it does. It checks the statement's structure, which is what can be checked without a server.