blogs.conchango.com

welcome to the conchango blogging site
Welcome to blogs.conchango.com Sign in | Join | Help
in Search

SSIS Junkie

Conchango are busy and need top level Technical Architects for Microsoft & Open Source platforms in and around London. Interested? Email me or send me a message

Some Analysis Services stuff!

Its a while since I spoke about Analysis Services on here but today I was getting back into the SSAS groove in a couple of ways.

 

Firstly, I received an email from Microsoft Connect in reply to one of my old suggestions which I opened back in June 2006. The submission went something like this:

It would be nice to have "dynamic" partitions. i.e. A new member appears in a dimension and a partition is automatically created for it.
An easy explanation for this being useful is having a partition for each year in teh time dimension. Whenever a new year appears - a new partition is created.
I also think it would be nice to have a logical "grouping" of partitions within a measure group so that we could place all these auto-partitions into a group and say "Process all of those please".

The reply I got was:

Regretfully, we were not able to add this capability for the current release. We still think this is a great idea, and will revisit this during planning for the next release.

Well, given that this obviously wasn't going to appear in SQL Server 2008 at this late stage I take this as good news (assuming this isn't a canned response). I do think that this functionality is sorely missing in SSAS2005 so I hope they make good on their promise to revisit this.

 

 

The second noteworthy item from today isn't to do with Analysis Services per se but definitely has a knock-on effect. I'm currently on a training course for Microsoft's Master Data Management (MDM) product that will be shipping with the next version of Office. Those in the know will be aware that the guts of this product is a rebadged version of Stratature which Microsoft purchased in June of last year.

During the training I've been very impressed by the hierarchy management features and the way that those features are placed in the hands of users rather than ETL developers; these hierarchy management capabilities far outweigh anything else that Microsoft has and that includes what Analysis Services has to offer. This is definitely treading new turf for Microsoft's BI landscape, up until now the type of logic that this product provides was hidden in the murky depths of ETL routines - putting that capability into the hands of end users is a compelling thought and it will be interesting to see how they execute on that vision.

The obvious next step as I see it is how Stratature's hierarchy management capabilities can be leveraged in SSAS. The typical usage scenario for integrating hierarchical data in SSAS today would involve a nightly ETL routine to get the data from MDM into the relational model and then (assuming we're not using ROLAP) a process of my cube.

The obvious problems here are

  1. latency

  2. a reliance on ETL to do the grunt work

The ideal scenario is thus:

  1. User updates their hierarchy in MDM
  2. Hierarchy changes are instantly propagated to SSAS cubes (no ETL involved)

  3. Aggregations are recalculated on the fly

  4. User browses his/her reports in whatever reporting tool they are using and the hierarchy changes that they committed just moments earlier are automatically reflected in the reports

This is MDM/SSAS nirvana...but I hope that the guys in the SSAS team are thinking in these terms.

 

There are also some very obvious (and some not so obvious) synergies with SSIS but I'll talk about those at a later date.

-Jamie

Published 30 January 2008 04:15 by jamie.thomson

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Chris Webb said:

I've not looked at MDM yet, but I agree that it would be useful for users to at least be able to test-drive their cube with an updated hierarchy even if their changes don't get put into production until later (I can imagine there might be some confusion if hierarchies were allowed to change at any time of day). Perhaps the feature to link up with here is dimension writeback (http://msdn2.microsoft.com/en-us/library/ms174540.aspx) which is not very widely known and not supported in most client tools.

January 30, 2008 08:29
 

Chris Webb said:

BTW, have you seen the automated partition creation functionality that is in the Analysis Services Stored Procedure Project?

http://www.codeplex.com/ASStoredProcedures/Wiki/View.aspx?title=Partition&referringTitle=Home

It's not quite what you requested, but it is very useful for prototyping. Come to think of it, it probably wouldn't be that hard to add to this functionality to get to what you do want. Hmmm....

January 30, 2008 08:44
 

DP said:

It does feel that this imight be going full circle. A lot of OLAP started on the desks of business users and possibly Analysis Services more than anything else moved it to the IT dept. But I personally would like to see it happen.

January 30, 2008 17:18
 

Thomas Ivarsson said:

Technically this is interesting. Like Chris says you already have something close to this like dimension writeback, however it requires a developer to manage this.

Problem: This will require that you will manage slowly changing dimensions with type one? And this will mean that end users will see something like the missing data wizard each time the open a report, or changes in dimension structures that will be hard to trace.

Speed is good but it has a cost as well

January 30, 2008 19:47
 

jamie.thomson said:

Thomas,

Very good point. This industry is all about trade-offs, right? At least there's an opportunity for us consulting types to help customers with this :)

Type 1/type 2 did occur to me as a complication. I certainly don't have an answer to that - this is going to take some real thinking about!!!

-Jamie

January 30, 2008 19:56
 

Thomas Ivarsson said:

Jamie! I think that we will need two approaches for this.

Build solutions where users do not care about history, a sort of a dynamic collection of data that never passes a data warehouse. This is more or less how finacial markets works. I have also seen this requirement in areas like delivery accuracy in manufacturing.

The other is the traditional way with history and a DW. In sales you have a longer time span from order to delivery and possible claims before you can actually estimate the profitability of an order/customer.

It should also be possible to mix these approaches. I agree with you "What is BI" blog entry that we will have to review what we are building and develop our approaches.

January 30, 2008 20:43
 

jamie.thomson said:

Thomas,

here's another scenario. In MDM you can define subsets of hierarchies (called collections) and define permissions on them so that only certain people can see those subsets.

Collections map pretty well to named sets in SSAS, but then you have to ask....how would you push permissions across as well?

The more and more I think about this the more scenarios are coming to my mind. Really really really interesting stuff but I'm glad its not me that has to solve it.

-Jamie

January 30, 2008 22:43
 

Thomas Ivarsson said:

Keep on writing more about what MS is doing in MDM Jamie!

Collections of secured hierarchies is a new aspect of one version of the truth.

Quick but wrong information sometimes drives the currency and stockmarkets. I have a degree in economics and know that statistics for unemployment, inflation rates and economic growth are not correct when they are published, every month.

Still most markets react as this is the truth.

Sometimes quick and accurate information is the best, if you can collect that information.

Quick but wrong should never be better than slower but good information.

January 31, 2008 20:05
 

JustinS707 said:

I have been in the business since 2000 and have learned that for a BI System to be effective you need (among other things):

1.  Master Data Management - I would go so far as to say that you need someone assigned to the management of the contextual data on a full-time basis.  If done properly, this should reduce the need for complex ETL processes like "Fuzzy Lookups" or "History Restatements".  I cringe at the idea of letting the end users manipulate the "Master" data in form of Dimension Writebacks.  With the exceptions of Forecasting and Budget Planning, end users should never be allowed to write to the Cube or Dimensional data.  The data source is the only place changes should be made to contextual data and it should only be done by the Master Data Manager(s).

I don't believe there is a need for a Master Metric Data Manager, but there is definitely a need for Metric Data Definitions for use throughout the company.  Unlike the Master Data Managment for contextual data, this should not require a FTE once the definitions are established.

2.  ETL - You need a fast ETL process that will catch and handle "Unknown" contextual data and alert the Master Data Manager(s) of these conditions.  

3.  OLAP - You need an OLAP system that will aggregate the metrical data at the appropriate levels of the contextual data to deliver instant query response times and without impacting OLAP processing times.

4.  Security - Your OLAP system should provide adequate security mechanisms to prevent end users from accessing contextual data without the proper permissions.  The process of defining the security needs for the contextual data could include the Master Data Manager, but why?  

February 7, 2008 15:29

Leave a Comment

(required) 
(optional)
(required) 
Submit

This Blog

Syndication

News

Powered by Community Server (Personal Edition), by Telligent Systems