blogs.conchango.com

welcome to the conchango blogging site
Welcome to blogs.conchango.com Sign in | Join | Help
in Search

SSIS Junkie

SSIS: In response to Daniel Read

Update: Daniel has posted a detailed comment to this below!

Those of you who read Daniel Read's blog will have read (3 different uses of the character sequence r-e-a-d in the space of 7 words - amazing) his excellent post today regarding eventhandlers in SSIS - http://www.developerdotstar.com/community/node/334.

Daniel raised a few points which I was going to leave open to conjecture but I'm afraid I just can't help myself!!

First of all its worth emphasizing what Daniel is talking about here because it cannot be underestimated. He has realised that the eventhandlers provide a fantastic mechanism for extending the inherent SSIS log providers. Sure, you can write your own custom log providers but more and more I'm coming to think that you simply don't have to - eventhandlers are a more than satisfactory substitute, but with an added bonus. The reason that eventhandlers can be substituted for log providers is that they basically do the same thing. They catch events thrown by the package. All the things that get logged by the log providers are simply events thrown up by the package's container hierarchy - in some ways a log provider is just a different implementatiton of an eventhandler. So in essence if you use the eventhandlers you're not doing anything different than the log providers - its just a heck of alot easier to customise. Daniel terms this customisation "domain specific logging" and I think that is a good description.

He also has a great idea to link his domain specific logging with SSIS's own log providers by providing the ExecutionInstanceGUID in his domain specific logging to make it easy to tie back to the SSIS log provider. If you are using the SSIS log provider for text files then you could even include the ExecutionInstanceGUID in the name of your log file and therefore generate a new log file for each execution - and that is very easy to do.

Daniel states he doesn't like the fact that eventhandlers don't interrupt the flow of execution. I can see what he means however I'm happy that it does not do this. The flow of execution can be controlled using OnFailure precedence constraints and that is a better fit to the TRY...CATCH block analogy that he speaks of. Perhaps my lack of a development background serves me well here!!

Cannot debug script tasks in event handlers - I noticed this myself yesterday and just assumed that it was something I had done wrong so didn't pursue it; so its heartening to know that someone else has had the same problem. Let's hope this gets fixed soon.

Events fire too often - Daniel complains that an eventhandler will catch events thrown by the container that it is scoped to AND all descendant containers of that container. I can see why this would be a bad thing but on the other hand it can also be a godsend - it means you only have to apply a generic logging mechanism in one place rather than throughout the package (or all packages - more on this later). Daniel has used a very good workaround to stop eventhandlers catching events raised by child containers that involves the System::SourceName and System::PackageName variables and indeed I have employed this logic in alot of my packages in order to only execute a task in an eventhandler if the event was raised by the package. I prefer to use System::SourceID and System::PackageID admittedly but its basically the same thing and you must be aware of the inherent problem in doing this - Daniel mentions the problem in his post.

There is another workaround however. All eventhandlers have a variable called System::Propogate scoped to them. If you set this variable to FALSE in an eventhandler scoped to the child container then events of that particular type will not be handled by an eventhandler scoped to an ancestral container. Did that gibberish make any sense at all? Probably not.

One other minor thing to note is that Daniel mentions having to put an empty Script Task into an eventhandler in order to conditionally execute a task that does some work. This is a bit of an irritation I have to admit however common convention is to use an empty Sequence Container to do this rather than Script Task.

Daniel talks about a problem with DisableEventhandlers property not having any effect for eventhandlers on ancestral containers. Well I actually think it makes sense that this is the case. The DisableEventHandlers property on a container stops the eventhandlers scoped to that container from executing. It does NOT stop the event from being thrown however and it does not stop eventhandlers scoped to other containers from executing and THAT is why eventhandlers of ancestral containers will still execute. Perhaps this could be fixed using System::Propogate however I do not know if setting DisableEventhandlers=TRUE will also disable the effect of System::Propogate. I suspect not but frankly its too late in the day and I'm too tired to go and investigate it now!!!

So on that note, its time I called it a night. One thing that I will say though is something that I've been meaning to state for  few weeks now - and state it much more clearly than I'm about to. If you have a multi-package solution then I strongly recommend you consider implementing it in such a way that all those packages are called from a single "master" package which contains all your custom logging functionality. I won't elaborate on why for now - but think it over!!

All that remains to do is thank Daniel for a really thought provoking post - you really should have a read of it because he explores some great SSIS concepts.

And with that I shall bid you adieu! I was going to blog about File System versus SQL Server deployments this evening but that really isn't very interesting and therefore can wait until another day!

-Jamie

 

 

Published 19 January 2006 23:06 by jamie.thomson

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Daniel Read said:

Hey, Jamie! Thanks for a great response to my post. I appreciate your taking the time.

It's a very interesting (and mostly fun) process that I've been going through in learning SSIS. I probably come at this tool from a different standpoint than some people might, for two reasons: one, I have a very developer-oriented mindset (though I am a very data-oriented developer, and very experienced with database-centric matters), and two, the somewhat complex, multi-package system I am building is not a BI-related ETL tool. It will be much easier for people from different backgrounds coming to SSIS, which offers a new programming paradigm, once some good books come out that can help people get over the hump to "Thinking in SSIS." It's an adjustment.

Jamie, I agree with you that it's convenient that a higher-level event handler can catch events for lower-level executables, but I wish I had more control over this at the level of the event-handler itself. It would be great to have a way to have a setting like Propagate, but in reverse. This points to something I've had to fight around design-wise in more than one area: as far as I can tell there is no way to set a package-wide default for container- and task-specific properties such as FailPackageOnFailure and MaximumErrorCount. I have more than once considered an otherwise viable design option that is marred by the fact that to make it work me and every other developer on this system would have to remember to set those properties for every container and task from now until forever.

Regarding this from your post: "Daniel states he doesn't like the fact that eventhandlers don't interrupt the flow of execution." It wasn't really event handlers in general that I felt this way about, rather only the OnError event handler. In general, as you point out, an event system is meant to be loosely coupled and asynchronous, so it's generally architecturally impossible for an event to stop the main flow of execution.

I've come to terms with how OnError works, however, in that I can use two other design tactics to achieve my end goal: rule one, keep the default MaximumErrorCount value of 1 for all containers and tasks, and rule two, ensure that everything that flows downstream from anything else is based on a Success constraint. In special cases exceptions can be made, no pun intended, to these two rules.

Regarding multi-package solutions, after much consideration I decided not to drive my packages from a "master" package. This post on my blog explains why in great detail (read all the way to the end):

http://www.developerdotstar.com/community/node/310

But long story short, in my particular mixture of packages some may run for a long time, some may run for only a few minutes, and many packages will need to run more than once per day. Since the Execute Package task launches packages synchronously, and since a package is not a state machine that operates in a loop, like a scheduling tool such as SQL Agent operates (is there something to do now? is there something to do now? is there something to do now? etc.), long running packages would block subsequent runs of short-running packages.

As explained in the above-linked post, I ultimately decided to externalize both a) the launching of packages, and b) the "job management" intelligence to know, for instance, when a job should *not* run, or when to retry a job that failed the last time, and how long to wait before letting a job run again. Feedback welcome.

I'll have more posts coming soon. I haven't even gotten into doing an real data moving yet. Thanks for reading.

Dan
January 20, 2006 01:28
 

Matthew Herndon said:

Hi, Jamie. I started working with SSIS a couple weeks ago and I've found your blog to be an invaluable resource. Right now I'm dealing with an issue that is slightly different than Dan's, basically, I want less logging. My project consists of one base package that calls many sub packages. There are numerous packages starting and finishing every minute and SSIS logs each of these events to the Event Log. This is making the log very crowded and obscuring other things I'd like to see, I've tried to turn this off but nothing I do seems to stop it. I have set LoggingMode = Disabled for the base package and all of the containers inside it, and made sure that nothing in the "Configure SSIS Logs" window is checked, but it is still logging each and every start and finish. I'm at my wits end, and I think it might be a bug in SSIS, then again maybe I'm just missing something. I figured if anyone would know it would be you.

Regards,

--Matt
January 30, 2006 20:10
 

Langston said:

Hi Matt,

I was having the same issue you were. Here's how I fixed it:

1) Open the SSIS Logging dialog box

2) Select the topmost container (the package) in the tree view on the left and if there are any log providers grayed-out AND checked (which I infer that it's disabled), you still must uncheck them. That's the trick; even though the checkbox is grayed-out, if it's checked, the package will still log start and finish times.

3) So, you can either delete the log provider from the list, or you have to re-check the container in the tree view on the left, uncheck the log provider, AND once again uncheck the container in the tree view.

There's probably a couple of other ways to fix this, but I found this to be effective, so I just stuck with it for now.

Regards

August 30, 2007 00:33
 

Nathan said:

Just a short one for those who get confused.  The variable for disabling "bubbling" of errors to parent containers is "System::Propagate", not "System::Propogate".  Took me a bit longer than i'd like to figure that out :)

Still havent got it to WORK yet, but it helps using the right variable name

Cheers,

Nathan

November 9, 2007 02:28
 

Dotnet Fellow said:

Well the instructions above for turning off logging will just get you so far.

I have yet to find a way to turn off the start and finish entries in the event viewer being spawned from the package.

June 21, 2008 01:00
 

Alex Weatherall said:

DotNet Fellow: I've found a way. It's a registry hack though : it involves adding a new custom event log key in the registry and moving the SQLISPackage Event Source into the custom log. The events are still written, but not to the Application log. Please see my reply to the post in the MSDN forums on this subject http://forums.microsoft.com/msdn/ShowPost.aspx?PostID=4014791&SiteID=1&mode=1

Hope this helps,

Thanks

Alex Weatherall

www.teleware.com

October 17, 2008 14:56

Leave a Comment

(required) 
(optional)
(required) 
Submit

This Blog

Syndication

News

Powered by Community Server (Personal Edition), by Telligent Systems