Advertisement

Testing and Monitoring Data Pipelines: Part Two

In part one of this article, we discussed how data testing can specifically test a data object (e.g., table, column, metadata) at one particular point in the data pipeline. While this technique is practical for in-database verifications – as tests are embedded directly in their data modeling efforts – it is tedious and time-consuming when end-to-end data […]

What Are Data Quality Tools?

Data Quality (DQ) tools are software solutions that aid in cleaning data, improving data integrity, and ensuring the accuracy of information. Quality management is a key component of business intelligence and big data. Through Data Quality management, you can ensure that your organization’s data is accurate and relevant. These tools are frequently found within master data […]

Data Lineage Use Cases

Data lineage can be described as a historical map of data’s journey within an organization. Use cases, in general, provide an example of how services or techniques can be used, and data lineage use cases are situations in which a form of data lineage can be used. Data lineage tools make tracking the data’s lineage […]

AI and Privacy: Navigating the Intersection of Technology and Personal Data

The rapid adoption of generative AI in the business world is transforming the way we work and improving our ability to engage with customers, streamline internal processes, and drive cost savings while introducing unprecedented challenges to the protection of individual privacy. As AI technologies continue to evolve and permeate various aspects of our lives, concerns about data […]

Understanding Data Mesh Principles

ThoughtWorks consultant Zhamak Dehghani defines data mesh as a “decentralized sociotechnical approach to sharing, accessing, and managing analytical data in complex and large-scale environments – within or across organizations.” This type of Data Architecture continues to generate interest among corporations, and data professionals will need to become familiar with data mesh architectures, such as those with data lakes or warehouses. […]