by Angela Guess

According to a new press release, “Databricks, the company founded by the creators of the popular Apache Spark project and providers of the leading Spark-based cloud platform for data science, today announced an edition of its cloud platform optimized specifically for data engineering workloads called Databricks for Data Engineering. The new offering enables more cost-effective data engineering using Spark while empowering data engineers to easily combine SQL, structured streaming, Extract, Transform, Load (ETL), and machine learning workloads running on Spark to rapidly and securely deploy data pipelines into production. Databricks for Data Engineering will complement the company’s existing cloud platform by providing all enterprises with a unified data analytics platform that fosters seamless collaboration to accelerate data-driven decisions across the organization.”

The release goes on, “Most organizations today encounter a variety of challenges in building systems on and around Spark to meet the needs of data engineering. Specifically, data engineers perform mission-critical data cleansing, transformations, and manipulations, to make business use cases such as real-time dashboards or fraud detection possible. As a result, for companies that set their sights on making data-driven decisions or automating business processes with intelligent algorithms, mastering data engineering is an essential step. ‘The expansion of our product portfolio to meet the needs of data engineering workloads is a major step in our journey to make big data simple for very complex data problems,’ said Ali Ghodsi, CEO and Co-founder at Databricks. ‘Databricks for Data Engineering will offer organizations a unified environment for data science and data engineering users alike, while optimizing Apache Spark performance — all with the reliability of an enterprise data and analytics platform at an efficient price’.”

Read more at Databricks.com.

Photo credit: Databricks