The list of issues that can derail a data migration can be scary.
This is why, as far back as 2009, and as recently as 2021, Gartner has released statistics detailing the prevalence of failed, delayed, and over budget data migration projects.
In 2009, 83% of data migrations failed or exceeded their budgets and schedules. And through 2024, Gartner predicts that the pressure to migrate quickly to the cloud will cause 60% of organizations to encounter public cloud cost overruns.
If that wasn’t intimidating enough, there’s also the prospect of outages caused by data migrations gone awry. Like when a data migration project caused an outage at Dexcom and a mountain of ensuing customer outrage.
But of course, there are data migration projects that go smoothly.
And even though every migration is unique, there are universally applicable data migration principles and best practices, which we’ll list below. First, though, let’s take a step back and examine three main types of data migration.
Types of Data Migration
Data migrations move data from one system to another. So we break down types of data migration based on the nature of the systems involved.
- Storage migrations involve transferring data from one storage location to another. This type of migration includes data transference from, to and between on-premise or cloud storage systems.
- Database migrations usually occur when an organization is upgrading to a new database. These migrations are typically more complicated than storage migrations since data may need to be formatted differently.
- Application migrations often involve both database and storage migrations because applications use their own databases and storage systems.
Regardless of the type of data migration project you face, here are five best practices to ensure it goes as smoothly as possible.
5 Best Practices to Make Any Data Migration Go Smoothly
- Know Your Data – The processes, technologies, and people you need depend in large part on the type, volume, format, and expected use of your data. The reason your data has so many downstream effects is that the way your source system operates is likely much different than the way your destination system operates. So to begin scoping out your data migration initiative, you need to audit your data. It’s only with a deep audit of your system’s existing data that you can determine how data must be transformed, consolidated, and otherwise processed before (or as) it is moved to the new system.
- Identify the Systems That Will Be Impacted – Identifying systems that will be impacted should be part of the data audit described above. But determining what systems may be impacted by a migration is so critical that it’s worth its own best practice. Rarely does a data migration project only impact the source and destination of the data migration. More often, there are a variety of systems that rely on the data you plan to migrate. Failing to understand these dependencies and connections can be a significant source of cost overruns and project delays.
- Build, and Test the Data Migration Process – With your data and systems fully audited, the next step is designing the migration process and validating the hardware and software requirements. In some cases, this might involve pre-validation testing to ensure everything functions as planned. Generally speaking, there are two approaches to mapping out the migration process flow. Either you can recreate the schema that exists in your source system and adapt it to your destination system. Or, you can automate much of this process with a data integration tool that automates multi-table updates. Once you’ve built your data migration process, it’s time to test it in a sandbox environment. After everything’s been ironed out in staging, the data migration goes live.
- Conduct Continuous Data Verification – After the migration, the best next step is to verify that the migrated data is complete, transformed correctly, and working as it should in the new system. The simplest way to verify this is with a parallel run of the source and destination system. Any disparities between the two systems can then be pinpointed to identify or anticipate data loss. It’s also best practice to generate a data migration report that identifies the status of the migration, information on any issues, and other details relevant to the migration.
- Leverage Automation for Data Migrations – Data migration projects are rarely small or simple, so removing manual work is always welcome. Not only does it reduce the cost of the project, but automation also mitigates the risk of human error. Of course, what you can automate depends on your project, but popular use cases of data migration automation include:
- Cleaning up stale data using an automated retention policy
- Migrating sensitive data with automated quarantine rules
- Automatically re-permissioning data in the destination
- Handling data drift automatically
Big Bang vs Trickle Data Migration
While we’re on the subject of data migration best practices, it’s worth noting the two primary strategic approaches to data migrations.
Big Bang Data Migration
The “big bang” data migration is an approach in which the organization starts and finishes the migration in a short timeframe. In a big bang migration, the system being migrated will go offline while the data is moved to the new system.
If the migration is scheduled strategically, you can accomplish a big bang migration with minimal or no interruption to customers. But there is relatively low margin for error. Should the migration fail, the downtime can become very costly, very quickly.
Trickle Data Migration
You’re typically better off using a trickle migration approach. Using this data migration strategy, the organization completes the migration in smaller chunks of data and workloads over a longer time period. The source system is not taken offline and instead runs parallel to the new system throughout the migration.
This approach reduces the total downside risk associated with any one aspect of the migration. And it eliminates the need for downtime and disruptions that are associated with a big bang migration.
Inject Agility into Data Migrations with a DataOps Approach
The superiority of trickle data migrations over big bang migration plans underscores a larger point. That is, modern organizations can access many of the unfulfilled promises of data analytics with a DevOps approach to their data operations.
We call this DataOps. And our platform was designed with the guiding principles of DataOps in mind.
DataOps seeks to operationalize data management to ensure resiliency and agility in the face of constant change. And in the world of data migrations, decoupling sources and destinations from data pipelines is one way in which you can operationalize your data management.
StreamSets helps you accomplish this by enabling data migration to many different cloud data lakes, including those provided by Databricks, Google, AWS, and Azure.
In fact, you can mine our library of sample data pipelines to build off of and enable a seamless data migration. Even better, once you build a smart data pipeline for your data migration, you can duplicate it to any cloud data lake without rewrites.