StreamFromKafkaSpec: + https://iceberg.apache.org/docs/latest/reliability/ Reading messages from Kafka - should be written to Iceberg + Given an Iceberg created with: CREATE TABLE polaris.my_namespace.StreamFromKafkaSpec ( id int, label String, partitionKey long, date Date, timestamp Timestamp ) USING iceberg TBLPROPERTIES ( 'format-version' = '2', 'sort-order' = 'partitionKey ASC NULLS FIRST', 'write.distribution-mode' = 'none' ) PARTITIONED BY (partitionKey); + And a Kafka topic called polaris.my_namespace.StreamFromKafkaSpec1747154279615 + When we send 1024 records to Kafka + And read from the Kafka topic writing to the Iceberg table + Then the table is populated with data + And the number of data files (8) is the same as the number of Spark partitions, not the number of Kafka partitions (24) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +