When I joined my current project in 2005 I didn't know much about Service Oriented Architectures (SOA). I knew it was something to do with transferring information via standard protocols and making it available to consumers but that was about it. My background is Business Intelligence (BI) which is all about it collecting data for storing in a data warehouse and I didn't see how these two seemingly incompatible architectures could work together. The trouble was, that is exactly what our customer wanted them to do.
Prior to my arriving on the project a number people far cleverer than me had already put much thought into devising an architecture that incorporated SOA and BI. They called it Service Oriented Business Intelligence (SoBI).
My take on SoBI, based on working with it for most of the past 18 months, can be succinctly summarised as:
- Data is made available to consumers via services. We have termed our services "service facades".
- The aim of the service facade is to make data available as a business entity as opposed to the rowsets that flow through a traditional ETL solution. Thus the provided data is for more likely to be understood by the data consumers.
- Data can be extracted from one or more Systems of Record (SoRs) in order to provide data to a single service facade
- Data can be cached above the service facades in order to service traditional BI requirements. That cache is invariably a relational data store and is considered a customer/consumer of the service facades.
- Data consumers communicate with facades using standard and agreed entity identifers (more on this in a future post)
- A SoBI solution does not own any data. SoRs are considered the data owners and the service facades provide an interface to the inherent data.
- In SoBI, good quality of data is a pre-requisite whereas in traditional BI bad data is accounted for by data scrubbing and data cleansing
You may be wondering why we bother. I have to be honest, for a long time I thought the same myself. I couldn't see the advantage of doing this as I'd been spoonfed data warehouses for years and they always worked for me. The disadvantages were only too evident to me as the ETL developer on the project - data serialised as XML is a killer when performance is a key factor in your ETL processes. As time went on though I began to see the advantages of SoBI:
- Data quality can't be hidden behind cleansing and scrubbing in the ETL phase. One of the tenets of SoBI is the owners of the SoRs are required to "get their house in order" and the data consumers are compelled to work with the data owners to achieve this. Long term this can only be of benefit to the business.
- Data can be shared between disparate data consumers in the business. This is key on the project that we have been working on and was one the main drivers towards service orientation.
- Real-time BI can be realised (with the caveat that SLAs can be met by the SoRs)
- The SoRs stay as the data owners so we don't have more than one copy of data prevelent throughout the enterprise
- We can drive towards standard identifiers for business entities and the relationships between them (more about that in a future post).
- SoRs can be upgraded and/or replaced with minimal impact
- Non-enterprise-class data stores (e.g. spreadsheets) are eliminated
- Transformation logic can be reused by different systems
I won't say its been easy. Theory is all well and good but turning that theory into deliverables is a little more difficult. As we carry on learning we'll carry on sharing it on http://blogs.conchango.com
This is a very high level view of what SoBI is so if you are interested in knowing more check out the Further Reading section below. If you have any questions then feel free to ask either Mick or myself. In the further posts in this series I'll talk a little more about some of the enabling initiatives that we have utilised, for better or worse.
Lastly, If you wondering who the "far cleverer people" are that I referred to at the top of this article, they are:
Sean Gordon (Microsoft)
Rob Grigg (Conchango)
Mick Horne (Conchango)
Simon Thurman (Microsoft)
In my next post I'll talk about the data model that we used for the BI cache that sits above the facades.
-Jamie
Further Reading/Viewing: