Data mesh is a type of organization and architectural paradigm, or – in simpler words – a distributed architecture for Data Management. The goal is to provide accessibility by distributing the data in the way that the end user needs it. It allows teams to embrace a self-serve data architecture design where they can look at data as a product and have control over their data pipelines.
Zhamak Dehghani, a ThoughtWorks consultant, created the concept of data mesh. The idea is that enterprises need a more decentralized and domain-driven approach. This approach ensures ease in accessibility, managerial functions, and shareability of data.
The decentralized aspect of a data mesh is based on the ubiquity of data. Data teams do not have to worry about transporting the data to data lakes or warehouses. Instead, users can access it wherever it’s available, saving time and unnecessary effort.
As for various stakeholders within the enterprise, a data mesh serves to be an immensely useful resource. The domain-centric design makes it easier for users to access relevant data whenever required. The self-serve model allows teams to retain ownership of domain capability independently.
Data meshes are a paradigm shift in how enterprises look at data. Here, data producers and users are directly linked, which creates more visibility and helps in aligning analytics and organizational teams. As the gap between different teams narrows, each can focus on more productive tasks like identifying how that data can be used for their purpose.
Other Definitions of Data Mesh Include:
- “A decentralized organizational and technical approach in sharing, accessing, and managing data for analytics and ML. Its objective is to create a sociotechnical approach that scales out getting value from data as the organization’s complexity grows and as the use cases for data proliferate and the sources of data diversify.” (Thoughtworks)
- “A new approach to thinking about data based on a distributed architecture for data management. The idea is to make data more accessible and available to business users by directly connecting data owners, data producers, and data consumers.” (Oracle)
- “A distributed method of data management and analytics that emphasizes domain expertise while easing the burden on centralized teams having to deal with exponentially growing amounts of data.” (TechTarget)
Use Cases Include:
- JP Morgan Chase leveraged AWS Cloud to create a data mesh infrastructure aligned with their business objectives. Considering that they were adopting a cloud-first approach, the data mesh helped them create separate lake formation accounts for different wings of their business. While it looks like a centralized approach, the various business wings can manage their data as they see fit.
- An American investment bank leveraged the data mesh approach to create a data infrastructure aligned with its product strategy. By working with Mesh AI, they realized that by using AWS, they could create data products (with respect to whom they serve) and organize their existing data lakes around them. It gives each business wing more control over their respective data products and makes sharing and accessing these resources easy.
- Zalando, an e-commerce fashion store, worked with Databricks to create a mesh infrastructure within their existing data lake to consolidate data in one place yet make it available to the different departments within their company. While the data lake works as the first ingestion point, the data mesh allows the curated data to be accessed company-wide.
Benefits of Data Mesh Include:
- Establishing inter-departmental communication pipelines
- Reducing operational costs, time, and effort
- Increasing transparency with enterprise-wide access to datasets
- Promoting autonomous teams with independent access to data
- Providing improved data security and reducing any data latency
- Giving stakeholders increased control over their data (data governance)
- Increasing team productivity and promoting scalability
Image used under license from Shutterstock.com
