Click to learn more about author James Briers.

“More than 80% of data migration projects run overtime and/or over budget. Cost overruns average 30%. Time overruns average 41%.” – Bloor Group.

Data migration can often seem like a simple task: move data from one place to another. How hard can it be?

A staggering number of data migrations continue to fail, or significantly exceed budgets. According to recent research by Gartner, 83% of all data migration projects either fail or exceed their budgets and schedules.

Business change in any way often drives the need for data migration strategies and projects, which can include any one of the below:

• New system implementations (ERP, CRM, Loyalty, HR, etc.)
• Large system upgrades (ERP, CRM, HR, etc.)
• Mergers & acquisitions
• Divestitures
• Migrating to the cloud
• A new (or expanded) master data management initiative
• A new (or expanded) business intelligence/data warehousing project
• A big data initiative
• System rationalization/retirement
• Responding to new regulations
• New management

The common thread across these disparate initiatives is a strong need for data.

Unfortunately, many times people think that data migrations sound like a problem for IT”, and then quickly move on from them with a“what could possibly go wrong?” mindset.

Given my experiences in data migrations throughout several industries, I am keen to share some of my own insights into what can commonly trip you up when migrating data and how you can look to avoid these pitfalls.

Don’t Wait Until the Target is Ready to Get Started

Often when migrating data, people wait until the target is ready to get started. But this is a mistake because a large part of the work in migrating data involves the careful planning and scoping of the project. Your plan should include:

• Gather requirements and agree on metrics for success
• Plan scheme mapping, data mapping, and backup and recovery plans
• Include security and go-live plans

Each of these steps takes considerable time. And once you’ve done all that planning, the work of cleansing and normalizing the data needs to happen to get the source data ready to be moved. If you wait until the target is ready to go, you may be behind schedule before you even start.

Avoid Surprises in Your Data

Part of your planning should include an assessment of your sources and their dependencies. You need to perform an inventory of all your data assets and the associated applications to find dependencies.

Pay close attention to the upstream and downstream applications affected by your data migration.

A complex project may have anywhere between 60 and 80 different data objects coming in from a hundred or so different applications. When you discover new source data or dependencies late in the game, it can throw off your migration timeline and add complexity to your project that you really don’t need.

Don’t Skip Data Cleansing

It often seems easier just to move data and clean it once it is moved to the target.

But the time to clean your data is before you move it.

The scenario I often use here is if you were moving to a new house, would you take the contents of your garbage or rubbish can with you? Probably not! So why would you move your bad data? If you move the data without cleansing it, you’ll perpetuate the problems that existed in the source data.

And before you move your data, you should take the time to perform a data profile. A data profile is a thorough examination of your existing data.

Profiling your data will also help you to really understand your data – for example, if there are blank or null values, if the data is unique or duplicated, or if the data patterns and values fall into a range you expect. After you perform a thorough data profile, you should perform data mapping to plan how the source types will correlate to the source types in the target.

Next, cleanse and validate your data. This involves removing extraneous data, filling in missing data, normalizing data (making it conform to a pattern that is compatible with other data), and masking sensitive data.

You may also need to transform and enrich your data. Data transformation is the process of converting data from one format or structure into another format or structure. Some of these processes must be done before you extract the data, while others can be done after extracting the data but before loading it to the target.

Hire Experts

As I touched on earlier, the perception of a data migration project often is that it is a “shift and lift” operation. This perception leads project leaders and stakeholders to not budget appropriately when hiring or assigning people and teams to the project.

The process of migrating data takes an understanding of the complexities of data profiling, data cleansing, and security requirements, among other key things. It is easy to underestimate just how complex and challenging data migrations can become, and spending less on these resources can cost you significantly in the long run in terms of results, budget, and schedule.

If you move bad data, or if you neglect security and compliance, you can end up with poor data quality or worse, a security breach and suddenly you’re non-compliant.

Have a Rollback Plan

Sometimes when you are migrating data, there is a lot of pressure to keep moving forward and not look back. It might seem tempting to push your changes to the target and fix any issues after you have moved the data.

A better way to handle this is to have a rollback plan for various stages of the project. This involves performing checks at various stages and having backups configured if you need to roll back changes. While this may seem more time-consuming and tedious, it will absolutely save you costly headaches down the road.