Spark: configurable integration test#2755@pawel-big-lebowski Provides command line tool capable of running Spark integration tests that can be created without Java.
Spark: OpenLineage Spark extension interfaces without runtime dependency hell#2809#2837@ddebowczyk92 New Spark extension interfaces without runtime dependency hell. Includes a test to verify the integration is working properly.
Spark: support latest versions 3.4.3 and 3.5.1.#2743@pawel-big-lebowski Upgrades CI workflows to run tests against latest Spark versions: 3.4.2 -> 3.4.3 and 3.5.0 -> 3.5.1.
Spark: add extraction of the masking property in column-level lineage#2789@tnazarew Adds extraction of the masking property during collection of dependencies for ColumnLineageDatasetFacet creation.
Spark: collect table name from InsertIntoHadoopFsRelationCommand#2794@dolfinus Collects a table name for INSERT INTO command for tables created with USING $fileFormat syntax, like USING orc.
Spark, Flink: add PostgresJdbcExtractor#2806@dolfinus Adds the default 5432 port to Postgres namespaces.
Spark, Flink: add TeradataJdbcExtractor#2826@dolfinus Converts JDBC URLs like jdbc:teradata/host/DBS_PORT=1024,DATABASE=somedb to datasets with namespace teradata://host:1024 and name somedb.table.
Spark, Flink: add MySqlJdbcExtractor#2825@dolfinus Handles different formats of MySQL JDBC URL, and produces datasets with consistent namespaces, like mysql://host:port.
Spark, Flink: add OracleJdbcExtractor#2824@dolfinus Handles simple Oracle JDBC URLs, like oracle:thin:@//host:port/serviceName and oracle:thin@host:port:sid, and converts each to a dataset with namespace oracle://host:port and name sid.schema.table or serviceName.schema.table.
Spark: configurable test with Docker image provided#2822@pawel-big-lebowski Extends the configurable integration test feature to enable getting the Docker image name as a name.
Spark: Support Iceberg 1.4 on Spark 3.5.1.#2838@pawel-big-lebowski Include Iceberg support for Spark 3.5. Fix column level lineage facet for UNION queries.
Spec: add example for change in #2756#2801@Sheeri Updates the customLineage facet test for the new syntax created in #2756.
Spark: fallback to spark.sql.warehouse.dir as table namespace#2767@dolfinus In cases when a metastore is not used, falls back to spark.sql.warehouse.dir or hive.metastore.warehouse.dir as table namespace, instead of duplicating the table's location.
Spark: fix Glue naming format#2766@dolfinus Changes the AWS Glue namespace to match Glue ARN documentation.
Spark: fix Iceberg dataset location#2797@dolfinus Fixes Iceberg dataset namespace: instead of file:/some/path/database.table uses file:/some/path/database/table. For dataset TABLE symlink, uses warehouse location instead of database location.
Spark: fix NPE and incorrect comment#2827@pawel-big-lebowski Fixes an error caused by a recent upgrade of Spark versions that did not break existing tests.
Spark: convert scheme and authority to lowercase in JdbcLocation#2831@dolfinus Converts valid JDBC URL scheme and authority to lowercase, leaving intact instance/database name, as different databases have different default case and case-sensitivity rules.