Click to learn more about author Ibrahim Surani

Every organization integrates and shares data with multiple systems, processes, and individuals. With data being generated and consumed at unprecedented levels, the challenge is ensuring its integrity before it’s processed or fed into reporting and analytics databases.

Consider a scenario, where you have to decide whether to invest next quarter’s marketing budget in campaign A or B. Imagine making that decision based on data riddled with inaccuracies. It could dramatically impact your company’s bottom line goals.

Business decision-making, whether it’s related to marketing, finance, customers, or sales, is often dependent on data extracted from heterogeneous systems. Ensuring the integrity of data when collecting and processing disparate information can guarantee trusted data and, by extension, viable decisions.

For businesses to leverage data, it must be:

• Original — Unique and accessible in its original form from the source
• Accurate — Free of errors and inconsistencies
• Comprehensible — Data should be legible by humans and computers
• Attributable — It should demonstrate who recorded the data and when

A report by KPMG revealed that only 35 percent of high-level executives have trust in their business data. 92 percent of the people surveyed believed that bad data could negatively impact branding and reputation.

Understanding the origin of bad data can help businesses take protective measures to preserve the accuracy and quality of data.

What Impacts Data Integrity?

Here are some of the threats that compromise the integrity of data during the integration process:

• Migration Errors: Unintended changes, such as format issues or duplications, can occur when data is replicated from one system to another. To resolve these issues, the data needs to be massaged and transformed before being loaded into the destination system.
• Security Failure: Security misconfigurations, cyber threats, and software bugs can corrupt data and make it useless for organizations. Data integration tools with security features, like user authentication and authorization, can restrict access to datasets and protect them from getting compromised.
• Human Errors: Errors in programming or entering data in a specific field can corrupt data and skew the results. Multi-environments, such as databases, are more prone to human errors as multiple people can access and alter the information.

How to Minimize Integrity Risks During Integration

Here are the measures that can be incorporated into the data integration pipeline to retain the integrity of data:

1. Data Cleaning

Data cleaning accounts for 30–80 percent of data preparation efforts. Ideally, a data cleaning initiative should satisfy two objectives:

• It should eliminate errors, inconsistencies, null values, and extra spaces, letters, or numerals from the source data.
• The process should be standardized for all incoming data to reduce the manual effort in applying the same rules repetitively for streaming data.

2. Data Profiling

Data profiling is used to get an in-depth breakdown of the source data to understand its structure, content, error probability, and descriptive statistics, such as max, min count, and interrelationships. Profiling data makes it easier to identify integrity issues that can be fixed before data is loaded into the target system.

3. Data Quality Rules

Data Quality rules specify which data is accurate and trustworthy, based on custom business rules. For example, the marital status field in employment forms must specify single, married, or widowed. The health insurance policies for all three categories are different. Therefore, by applying Data Quality rules to the source data, the business can ensure that the marital status field isn’t left empty.  

Using these initiatives ensures that the integrated data has integrity, and can be further fed into other applications for analysis, reporting, and other user-facing applications.