Click to learn more about author Kevin W. McCarthy.

Just in time for the holidays, I put together my first annual “Data Quality: Naughty or Nice” list.

For your average person, Data Quality can be a really dry topic. When you start talking about consolidating customer records (checking that Jon Smith and J Smith are the same account) or validating and standardizing data elements, most people, who aren’t Data Quality geeks like me, start to glaze over. However, over the past year, I have gathered many anecdotes from companies whose Data Quality efforts have earned them a spot on the naughty or nice list. For a typically dry topic, it’s amazing the interesting, funny and sometimes scandalous situations that can come about from Data Quality processing.

Let’s start with the nice list (the naughty list is better for dessert!)

Helping the Kids: Data Quality can seem so far removed from actual real-world impact, but the truth is, it can make a big difference. Take the case of child welfare in the United States. There is a new regulation, CCWIS (Comprehensive Child Welfare Information System), which mandates Data Quality as an integral part of new state-level IT systems. Data Quality processes check that each child is being provided the optimal (and accurate) amount of services – ensuring no children without case workers are assigned, there are no interruptions in services over the defined time periods, and that case workers are not assigned more children than they can productively service. There have been issues in the past, so CCWIS is making Data Quality a cornerstone regulation to ensure kids are getting the most out of these programs.

Child Safety: On a Federal level, there are agencies dedicated to child safety. Too often today, there are news broadcasts about missing or exploited children around the country. Data Quality and, in particular, probabilistic and deterministic matching methods are being used to ensure that children listed as missing or exploited are not being taken out of the country. Border agents, using limited contact information, can check these lists in real-time and have used this information to prevent abductions and reunite children with their families. It’s these kinds of stories that make a Data Quality geek like me proud!

Now that we’ve seen some examples of “nice,” it’s time to change the tone and get into the naughty list: 

Condiment Chaos: The Simpsons made a joke once about “Wasting food – the new trend sweeping the nation,” but this one is a little ridiculous! A major hotel chain had significant headaches with food services inventory, especially overordering perishable items. When reviewing their product data, they soon discovered why.  They were double-ordering tomato sauce, because it was in their systems twice as “ketchup” and “catsup.” A simple mistake ended up costing the hotel tens of thousands of dollars!

Insurance Shell Game: A large insurance company decided to create an incentive program for their agents to provide bonuses each month for opening accounts over a certain threshold. Over time, they noticed a few agents that were consistently receiving bonuses. Upon investigation, the data revealed that while these agents had indeed been opening enough accounts to be eligible for the bonuses, they were not really new accounts. Agents were calling existing customers, telling them they were doing maintenance on their account, and subsequently closing that account and opening a new one. All the customer numbers on the new accounts were different from the old accounts, but the names, addresses, and SSNs were the same. Contact matching among the open and closed records exposed the scam. Naughty agents, but the implementation of Data Quality was nice (at least for the Insurance company!).

Single Cheaters View: A large bank implemented customer matching technology to, among other things, improve their ability to market and up-sell to households. They set up rules to create a “Golden Record,” taking into account the most common occurring address information (households might have different addresses based on home/work addresses or “snowbird” addresses from people that summer in one location and winter in another), the name from the most recently opened account, etc. One day after this new matching was implemented, their largest client called saying he was pulling all his accounts from the bank. It so happens that he had recently opened a checking account for his secretary, with whom he was having an affair. His wife got a mailing at their house, with the secretary’s name on it, thanking her for opening the new checking account and wanting to know if she’d now like to open a savings account. As her husband did not have sensible answers for why he would open a checking account for his secretary, she subsequently threw him out of the house. This is a case where the Data Quality business rules did all the things that were requested, but no one could have predicted the results (least of all, the rich guy now living in a hotel!).

It’s interesting to see how Data Quality has an impact in real-life scenarios, and how Data Quality can be used to segregate the naughty from the nice.  Let’s face it: “Santa” is just one Data Quality anagram away from “Satan,” and it’s those kinds of fat-fingered flubs that may create the next story on the Naughty or Nice list!