Business Intelligence Utopia - Enabler 2: Real Time Data Integration

Posted by Karthikeyan Sankaran
Comments (2)
August 16th, 2007

Business Intelligence practitioners tend to have lot of respect and reverence for transaction processing systems (OLTP), for without them the world of analytical apps simply does not exist. That explains my previous blog in introducing the first enabler for BI Utopia – The Evolution of OLTP systems to support Operational BI.

In this post, I introduce the second enabler in the “Power of Ten” – Real Time Data Integration

Data Integration in the BI sense, is all about, extracting data from multiple source systems, transforming them using business rules and loading it back into data repositories built to facilitate analysis, reporting, etc.

Given that the raw data has to be converted to a different form more amenable for analysis & decision-making, there are 2 basic questions to be answered:

  1. From a business standpoint, how fast should the ‘data-information’ conversion happen?
  2. From a technology standpoint, how fast can the ‘data-information’ conversion happen?

Traditionally, BI being used more for strategic decision-making,  batch mode of data integration with periodicity of a day or later, was acceptable. But increasingly, businesses demand that the conversion has to happen much faster and technology has to support it. This leads to the concept of “Real Time BI” or more correctly “Right Time BI”.

Since the answer to the first question “How Fast” is fast becoming “as fast as possible”, the focus has shifted to the technology side. One area where I foresee a lot of activity, from a Data Warehouse architectural standpoint, is in the close interaction of messaging tools like IBM Websphere MQ etc. with data integration tools. At this point in time, though the technology is available, there aren’t too many places where messaging is embedded into the BI architectural landscape.

Bottom-line is that there is significant value gained by ensuring that raw business data is transformed to information by the BI infrastructure, as fast as possible – the limits being prescribed by business imperatives. The best explanation I have come across to explain the value of information latency is the article by Richard Hackathorn .

Active Data Warehousing is another topic closely related to Real Time Data Integration and you can get some perspective on it thro’ the blog on Decision management by James Taylor:

Comments (2)

Karthikeyan Sankaran - September 3rd, 2007

Thanks David for your comments and the related link to milestonegroup. It is an interesting thought to look at Right-Time BI interms of the weakest link in the chain, which is typically a "Assembly-line" way of looking at things. From that perspective, we can probably start applying some of the well known concepts like Theory of Constraints to the BI scenario also. Thanks again and please do keep reading this space.

David - August 28th, 2007

Interesting topic and a difficult one for many IT departments to facilitate. The whole notion of Real Time or Right Time is to me a proper and inevitable evolution in processing but one that will initially be constrained by the weakest link in the chain. This weakest link is not the BI applications but those applications feeding it and further by the information and timeliness of the information being received from external parties. There is an interesting company out of Australia that is making inroads into this whole area of what you term BI..see