v0.73.md 2.4 KB


title: "Materialize v0.73" date: 2023-10-18

released: true

v0.73.0

[//]: # "NOTE(morsapaes) v0.73 shipped the ASSERT NOT NULL option for sinks behind a feature flag."

Sources and sinks

  • Private preview. Allow propagating comments in materialized views to the Avro schema of Kafka sinks, as well as manually specifying comments using the new [KEY|VALUE] DOC ON [TYPE|COLUMN] <identifier> connection option.

    Example:

    CREATE TABLE t (c1 int, c2 text);
    COMMENT ON TABLE t IS 'materialize comment on t';
    COMMENT ON COLUMN t.c2 IS 'materialize comment on t.c2';
    
    CREATE SINK avro_sink
      IN CLUSTER my_io_cluster
      FROM t
      INTO KAFKA CONNECTION kafka_connection (TOPIC 'test_avro_topic')
      KEY (c1)
      FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY CONNECTION csr_connection
      (
        DOC ON TYPE t = 'top-level comment for avro record in both key and value schemas',
        KEY DOC ON COLUMN t.c1 = 'comment on column only in key schema',
        VALUE DOC ON COLUMN t.c1 = 'comment on column only in value schema'
      )
      ENVELOPE UPSERT;
    

    Key schema:

    {
      "type": "record",
      "name": "row",
      "doc": "this is a materialized view",
      "fields" : [
        {"name": "a", "type": "string", "doc": "this is column a"},
        {"name": "b", "type": "string"}
      ]
    }{
      "type": "record",
      "name": "row",
      "doc": "top-level comment for avro record in both key and value schemas",
      "fields": [
        {
          "name": "c1",
          "type": [
            "null",
            "int"
          ],
          "doc": "comment on column only in key schema"
        }
      ]
    }
    

    Value schema:

    {
      "type": "record",
      "name": "envelope",
      "doc": "top-level comment for avro record in both key and value schemas",
      "fields": [
        {
          "name": "c1",
          "type": [
            "null",
            "int"
          ],
          "doc": "comment on column only in value schema"
        },
        {
          "name": "c2",
          "type": [
            "null",
            "string"
          ],
          "doc": "materialize comment on t.c2"
        }
      ]
    }
    

Bug fixes and other improvements

  • Allow the value of the SASL MECHANISM option for Kafka connections to be specified in any case style. Previously, Materialize only accepted uppercase case style (as required by librdkafka).