Databricks open-sources declarative ETL framework powering 90% faster pipeline builds

Today, at its annual Date + you have a summit,, Databricks announced that it was being opened its basic declarative framework as a Spark Decarative Pipelines, making it at the disposal of the whole Apache Spark community in an upcoming version.

Databricks launched the frame as Delta Live Tables (DLT) in 2022 and a since extended to Help teams to create and operate reliable evolving data pipelines from start to finish. This decision to the opening strengthens the company’s commitment to open ecosystems while marking an effort for a snowflake rival, which recently launched its own OpenFlow service for data integration – a crucial data engineering component.

Snowflake’s offer approves Apache Nifi to centralize all data from any source in its platform, while Databricks makes its engineering technology of open internal pipelines, allowing users to execute it anywhere Apache Spark is supported-and not only on its own platform.

Declare the pipelines, let the spark manage the rest

Traditionally, data engineering has been associated with three main points of pain: the creation of complex pipelines, manual general costs and the need to maintain separate systems for workloads by lots and streaming.

With Spark declarative pipelines, engineers describe what their pipeline should do using SQL or Python, and Apache Spark manages the execution. The frame automatically follows the dependencies between the tables, manages the creation of the table and the evolution and manages operational tasks such as parallel execution, control points and attempted production.

“You declare a series of data and data flow sets, and Apache Spark includes the right execution plan,” said Michael Armbrust, distinguished software engineer at Databricks, in an interview with Venturebeat.

The framework supports data, streaming and semi-structured, including object storage systems like Amazon S3, ADLS or GCS, out of the box. Engineers must simply define the treatment in real time and periodic via a single API, with validated pipeline definitions before execution to take problems early – no need to maintain separate systems.

“It is designed for the realities of modern data such as modification data flows, message buses and real -time analyzes that feed AI systems. If Apache Spark can process them (data), these pipelines can manage them, “said Armbrust. He added that the declarative approach marks the latest efforts of Databricks to simplify Apache Spark.

“First of all, we did functional distributed computers with RDDS (resilient distributed data sets). Then we made an execution of the declarative request with Spark SQL. We brought this same model to streaming with structured streaming and made storage of the transactional cloud with Delta Lake. Now we are taking the next editorial leap from pipelines from start to finish, “he said.

High -scale

Although the declaration pipeline frame is determined to be engaged in the Spark code base, its prowess is already known by thousands of companies that used it as part of the Databricks Lakeflow solution to manage workloads ranging from the daily report by lots to subsecond streaming applications.

The advantages are quite similar in all areas: you waste much less time to develop pipelines or on maintenance tasks and get much better performance, latency or cost, depending on what you want to optimize.

The Block financial services company used the framework to reduce the development time by more than 90%, while Navy Federal Credit Union reduced the maintenance time for 99%pipelines. The structured SPark structured streaming engine, on which the declarative pipelines are constructed, allows the teams to adapt the pipelines for their specific latencies, to real -time streaming.

“As engineering director, I like the fact that my engineers can focus on what matters most for the company,” said Jian Zhou, principal director of engineering at Navy Federal Credit Union. “It is exciting to see this level of innovation now open source, which makes it accessible to even more teams.”

Brad Turnbaugh, a senior data engineer at 84.51 °, noted that the frame has “facilitated the management of lots and streaming without sewing separate systems” while reducing the amount of code that his team must manage.

Different approach from snowflake

Snowflake, one of the biggest databricks rivals, also took action during his recent conference to meet data challenges, making the start of an ingestion service called OpenFlow. However, their approach is a little different from that of databricks in terms of scope.

OpenFlow, built on Apache Nifi, mainly focuses on data integration and movement in the Snowflake platform. Users must still clean, transform and aggregate data once they arrive in Snowflake. The Spark declarative pipelines, on the other hand, go beyond the source to usable data.

“Spark declarative pipelines are designed to allow users to run end -to -end data pipelines – focusing on simplifying data processing and complex pipeline operations that underlie these transformations,” said Armbrust.

The open source nature of Spark declarative pipelines also differentiates it from proprietary solutions. Users do not need to be Databricks customers to take advantage of technology, aligning the company’s history of contributing to major projects such as Delta Lake, MLFlow and a unit catalog to the open source community.

Availability time

Apache Spark declarative pipelines will be attached to the Apache Spark code base in an upcoming version. The exact calendar, however, remains uncertain.

“We are enthusiastic about the prospect of the opening of our declarative pipeline frame since we have launched it,” said Armbrust. “In the past 3 years, we have learned a lot about the models that work best and have corrected those who needed a fine setting. Now it is proven and ready to prosper in the open air.”

The open source deployment also coincides with the general availability of Databricks Lakeflow Databricks pipelines, the commercial version of technology that includes features and additional business support.

Data data + Summit AI takes place from June 9 to 12, 2025