blogs.conchango.com

welcome to the conchango blogging site
Welcome to blogs.conchango.com Sign in | Join | Help
in Search

SSIS Junkie

Conchango are busy and need talented consultants in and around London. Interested? Email me or send me a message

SSIS: Blurring the lines between ETL (SSIS) and EAI (BizTalk)

I have just been asked by someone, "what is the recommanded way to implement the splitter patterne with an SSIS DataFlow", asked in the context of this article: http://www.enterpriseintegrationpatterns.com/Sequencer.html which talks about splitting an incoming message so as to send the resultant message fragments to different systems.

I guess you have to think about this in SSIS terms. That is, a message is a recordset coming from a source adapter. Each row in that recordset can be considered a constituent part of of the recordset, just as in the example above each order line is a constituent part of the order. Experienced SSIS developers will immediately tell you that the SSIS Conditional Split transformation maps very well to this situation. It can divert an incoming row to any one of a number of sources based upon attributes of that row.

OK, so that's done but it poses a question (which in fact the questioner above also asked me), "I'm a little bit confused with the features overlapping with Biztalk and SSIS."

Well clearly there is an overlap. There is definately a point where the lines between a BizTalk solution and a SSIS solution start to blur. I don't know much about BizTalk I must confess but if you want to make a decision about which technology to utilise then you need to answer some questions about your requirements and use those answers to decide which technology to plump for:

  • Do you require real-time or batch processing of data?
  • Are you likely to receive many small packets of data or a few large data chunks?

Any more???

If your requirement is characteristically real-time with many smaller data packets then you're probably going to plump for BizTalk. If its batched and you're going to be handling much bigger datasets then its probably going to be SSIS.

-Jamie

Published 20 September 2005 08:13 by jamie.thomson

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

JAson said:

Indeed, it does tend to get blurry on whether to use Biztalk or SSIS! We (company) started out using BTS02 for all our data integration processes, but as we evolved (company split into two) into a separate entity, so did the technological specs. We then focused on remodeling our data integrations processes targeting BTS04. However, there was an immediate collision when we started to transform 100+mb files and we were getting out of memory exceptions. Furthermore, we realized that the performance and productivity on BTS04 had decreased when compared to BTS02. Well, after several passes and meetings with MS, we concluded that what we really needed was an ETL type solution and best suited using SSIS. BTS04 was more so targeted toward business rules and long running processes -- If you have a transactional process that needs to go through several steps (such as an invoice) then BTS is well suited, you can take advantage of already in place techno's such as the msgbox.

My personal/final conclusion is make sure to gather all your requirements before jumping the gun, if you feel the project suits more toward business rules/processes then it does ETL, then start running bench tests in BTS04 to find out exactly how performance is going to be, because remember, even if you try and transform a legacy file that is not of XML type, it will still be transformed to canonical (XML) form by going through the pipeline and storing in msgbox. This will have a HUGE performance impact when dealing with large file sizes and an abundant amount of files.

I have other pros/cons posts out on web. Search for BTS04 limitations and you should find them. My 2 cents.
September 26, 2005 15:43

Leave a Comment

(required) 
(optional)
(required) 
Submit

This Blog

Syndication

News

Powered by Community Server (Personal Edition), by Telligent Systems