When you use ksqlDB to join streaming data, you must ensure that your streams and tables are co-partitioned, which means that input records on both sides of the join have the same configuration settings for partitions. The only exception is foreign-key table-table joins, which do not have any co-partitioning requirement.
To join two data sources, streams or tables, ksqlDB needs to compare their records based on the joining column. To ensure that records with the same join column are co-located on the same stream task, the join column must coincide with the column that the sources are partitioned by.
Partitioning streams and tables is especially important for stateful or otherwise intensive queries. For more information, see Parallelization.
Tables are always partitioned by their
PRIMARY KEY, and ksqlDB doesn't allow repartitioning
of tables, meaning you can only use a table's primary key as a join column.
Streams don't have primary keys, but they do have an optional
KEY column, when present, defines the partitioning column.
Streams allow joins on expressions other than their key column. When the join criteria differ
KEY column, ksqlDB internally repartitions the stream, which implicitly defines the
correct key and partitioning.
Kafka guarantees the relative order of any two messages from one source partition only if they are both in the same partition after the repartition. Otherwise, Kafka is likely to interleave messages. The use case will determine if these ordering guarantees are acceptable.
The following example shows a
users table joined with a
on the click's
userId column. The
users table has a correct primary key,
id, of the same SQL type. The
clicks stream doesn't have a defined key,
so ksqlDB internally repartitions it on the joining column (
to assign the key before performing the join.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
When you use ksqlDB to join streaming data, you must ensure that your streams and tables are co-partitioned (except for foreign-key table-table joins), which means that input records on both sides of the join have the same configuration settings for partitions.
- The input records for the join must have the same key schema.
- The input records must have the same number of partitions on both sides.
- Both sides of the join must have the same partitioning strategy.
When your inputs are co-partitioned, records with the same key, from both sides of the join, are delivered to the same stream task during processing.
Records Have the Same Key Schema¶
For a join to work, the keys from both sides must have the same SQL type.
For example, you can join a stream of user clicks that's keyed on a
user id with a table of user profiles that's also keyed on a
STRING user id.
Records with the exact same user id on both sides will be joined.
If the schema of the columns you wish to join on don't match, it may be possible
CAST one side to match the other. For example, if one side of the join
INT userId column, and the other a
LONG, then you may choose to cast
INT side to a
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Tables created on top of existing Kafka topics, for example those created with
CREATE TABLE statement, are keyed on the data held in the key of the records
in the Kafka topic. ksqlDB presents this data in the
PRIMARY KEY column.
Tables created inside ksqlDB from other sources, for example those created with
CREATE TABLE AS SELECT statement, will copy the key from their source(s)
unless there is an explicit
GROUP BY or
JOIN clause, which can change what the table
is keyed on.
ksqlDB automatically repartitions a stream if a join requires it, but for stream-table and table-table joins, ksqlDB rejects a join on a (right) table's column that is not the primary key. This is because repartitioning a table's topic has the potential to reorder events and misinterpret tombstones, which can lead to unintended or undesired side effects.
If you are using the same sources in more than one join that requires the data to be repartitioned, you may prefer to repartition manually to avoid ksqlDB repartitioning multiple times.
To repartition a stream, use the PARTITION BY clause. Be aware that Kafka guarantees the relative order of any two messages from one source partition only if they are also both in the same partition after the repartition. Otherwise, Kafka is likely to interleave messages. The use case will determine if these ordering guarantees are acceptable.
If the PARTITION BY expression evaluates to NULL, the resulting row is produced to a
random partition. You may want to use COALESCE to wrap
the expression and convert any NULL values to a default value, for example,
PARTITION BY COALESCE(MY_UDF_THAT_MAY_FAIL(Col0), 0).
For example, if you need to re-partition a stream to be keyed by a
field, and keys need to be distributed over 6 partitions to make a join work,
use the following SQL statement:
1 2 3 4 5
Records Have the Same Number of Partitions¶
The input records for joins must have the same number of partitions on both sides, except for foreign-key table-table joins.
ksqlDB checks this part of the co-partitioning requirement and rejects any join where the partition counts differ.
DESCRIBE <source name> EXTENDED command in the CLI to determine the
Kafka topic under a source, and use the SHOW TOPICS command in the CLI to list
topics and their partition counts.
If the sides of the join have different partition counts, you may want to change the partition counts of the source topics, or repartition one side to match the partition count of the other.
The following example creates a repartitioned stream, maintaining the existing key, with the specified number of partitions.
1 2 3
Records Have the Same Partitioning Strategy¶
Records on both sides of the join must have the same partitioning strategy. If you use the default partitioner settings across all applications, and your producers don't specify an explicit partition, you don't need to worry about the partitioning strategy.
But if the producer applications for your records have custom partitioners specified in configuration, the same custom partitioner logic must be used for records on both sides of the join. The applications that write to the join inputs must have the same partitioning strategy, so that records with the same key are delivered to same partition number.
This means that the input records must be in the same partition on both
sides of the join. For example, in a stream-table join, if a
key with the value
alice123 is in Partition 1 for the stream, but
alice123 is in Partition 2 for the table, the join won't match, even
though both sides are keyed by
ksqlDB can't verify whether the partitioning strategies are the same for both join inputs, so you must ensure this.
The DefaultPartitioner class implements the following partitioning strategy:
- If the producer specifies a partition in the record, use it.
- If the producer specifies a key instead of a partition, choose a partition based on a hash of the key.
- If the producer doesn't specify a partition or a key, choose a partition in a round-robin fashion.
Custom partitioner classes implement the
and are assigned in the producer configuration property,
For example implementations of a custom partitioner, see Built for realtime: Big data messaging with Apache Kafka, Part2 and Apache Kafka Foundation Course - Custom Partitioner.