Data Warehouse Database Re-platforming on the raise!

In addition to the existing Data Warehouse (DW) database, recently many organizations have started to look for an alternate DW database platform.

What is Data Warehouse Database Re-platforming? The process by which an existing DW database is being completely replaced or being complemented by a new database solution.

What are the factors driving DW database Re-platforming?

• DW systems that have been in use for a longer time like about  8+ years or more  are not being able to perform to the expectation, due to reasons like
  • Increase in the data volume
  • Increase in the user base
  • Need to deliver more volume of reports
  • Data not stored and structured in way to handle large queries
• Need of independent ‘Sandbox’ data marts for Business to search and explore data. ‘Sandbox’ data marts replicate a subset of the data ‘AS IS” from an existing DW.
• Need for very high performance databases for specific functions like search or mining
• Need to leverage MPP architectures for higher performance
• Need to reduce the data volume through compression options
• Need to leverage in-memory data storage options for higher performance
• Consolidation in the database products  through acquisitions like Sybase by SAP, Netteza by IBM
• Need to consolidate from multiple data marts and infrastructure to a single DW database infrastructure
• Benchmark results put by many new database platform vendors give a very attractive proposition to try out at least a proof of concept

Also in many of the DW systems the existing performance issues are attributed to the way large volume of data is stored and delivered to the reporting platforms, hence database re-platforming is seen as a natural choice for performance improvement.

There are two ways of re-platforming

• ‘AS IS’ Migration
• New System Design and Migration

For ‘AS IS’ Migration, the key considerations will be like

• ‘AS IS’ model based on the existing DW, but eliminate redundant and unwanted data structures
• Apply the new physicalization features supported like partitions, compression, row-columnar data storage options, in-memory data storage etc
• Migrating the data from existing DW (minimal transformations) to these new structures
• Changing(minimal) the existing external interfaces like reporting  to understand the new data model
• Defining the data archival and maintenance processes

For New System Design and Migration, the key considerations will be like

• Building a new data model based on the existing data structures
• Migrating the data from existing DW by transforming to these new structures
• Changing the existing external interfaces like reporting  to understand the new data model
• Defining the data archival and maintenance processes

Thanks for reading, let me know whether you are observing an increase in Data Warehouse database re-platforming initiatives …