In part one of my post, I highlighted the things you should consider as part of your data migration strategy. In part two, I have highlighted the IDEAL methodology that should be considered in your data migration projects. I have used this on many projects that I have been responsible for in the past several years and refined it for my own work.
Below are some key considerations that you should try to use at the onset of your next project and incorporate the best practices into your design:
- Identify the business processes involved.
- Ensure there is key business ownership and documented processes on how to manually create and maintain master, transactional, and historical data sets.
- Determine, select and locate data to migrate at the outset.
- Know what data you are migrating, where it resides, what form it’s in and document the form it will need to take when it arrives at its destination.
- Determine what technical tools (like an ETL) are needed. Does the source or target system also have facilities to export and import data.
- Do you understand the layout of data in both systems and can you create a logical mapping between the two? Have you created a record manually in the new system and recorded all the fields and data types for example?
- Extract, clean, transform and de-duplicate the data.
- All data has problems. Use data migration as an opportunity to clean up the data and test business rule adherence.
- Audit and document the process.
- Regulatory compliance requires you to document each stage of the data migration process and to preserve a clear audit trail of who did what to which data and when.
- Leverage the movement of the data in a systematic execution through enforcement of data migration policies.
- For example, restrict data migrations to overnight hours when network usage is low and won’t interfere with your project.
- Test the migrated data to ensure it’s accurate and in the expected format. Without testing and validating the migrated data, you can’t be confident in its integrity. (Reference Tech Target for additional information.)
There are several factors that go into ensuring a successful data migration strategy can be executed. I have not covered every detail in this short write-up but the first step discussed in this briefing is the enablement of that strategy. It is an important first step because it starts discussion and promotes critical thinking about an often-ignored part of a technical systems project, and it gets people engaged data-centrically.