Welcome to blogs.conchango.com Sign in | Join | Help

Welcome to blogs.conchango.com

SSIS Junkie

SSIS Nugget: Setting expressions

Whilst reading posts on the SSIS forum of late it has become evident that people are unaware of the power that expressions can provide within an SSIS package. This is understandable - alot of people are starting to use SSIS now but expressions aren't the most obvious feature.

So what are they? Well the most generic explanation is that expressions allow your packages to be dynamic by setting properties of objects at runtime. What are they used for? Well there are a number of scenarios, the most common being:

  • Dynamically setting the connection string of external connections
  • Conditional precedence constraints
  • Setting the location of raw files
  • An alternative to parameterised SQL statements in the Execute SQL Task or a source adapter

The reason that expressions aren't the most visually obvious feature is they're usually hidden away in the properties pane. Lets demonstrate that in one of the scenarios already mentioned - dynamically setting the connection string of external connections.

Imagine the following scenario:

  • You have a data-flow that loads data from a CSV file.
  • That data-flow is within a ForEach loop that iterates over a collection of CSV files.
  • The ForEach Loop populates a variable with the name and location of the file

 In other words your package looks something like this:

20060311PackageScr.JPG

On each loop iteration @[FileNameAndLocation] will, not surprisingly, store the name and location of the current file.

Now, we need to make sure that the data-flow is going to process that file. We do that by dynamically setting the ConnectionString property of "FlatFileConnectionManager" using an expression. To do that select "FlatFileConnectionManager" and press F4. The properties pane will appear in which you will be able to see the entry point to the connection manager's expression collection:

20060311Properties.JPG

Clicking the ellipsis will display the Property Expressions Editor dialog from which we select the ConnectionString property.

20060311PropertyExpressionsEditor.JPG

Thereafter we go through to the Expression Editor which is where we set up an expression that will dynamically set the ConnectionString property. In this case we use the @[FileNameAndLocation] variable:

20060311ExpressionEditor.JPG

As you can see from the evaluated value, the ConnectionString property will be set to the name of the variable. This means that on every iteration of the ForEach loop the file that the "FlatFileConnectionManager" will be pointing to to the currently iterated file.

 

This is only one use of expressions but hopefully through this simple scenario we've demonstrated how expressions are set and how they can be used.

Remember that expressions are not only available to be set on connection managers, they can be set on variables, task properties and precedence constraints as well. You'll find that as you begin to build ever more complicated solutions you will start to use expressions more and more.

-Jamie

Published 11 March 2006 09:24 by jamie.thomson

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Professional Association for SQL Server (PASS) SIG said:

March 15, 2006 01:51
 

Antony Clark said:

How would you set the connection string dynamically for a destination flat file? Say you had a recordset that contained a column value that dictated what the output destination file should be for that record. I.e. Value 0 in col1 mean output record to c:\output0.txt , Value 1 in col1 means output to record to c:\output1.txt etc.
The tough part is that if the possible values are variable and unknown at design time you cannot create the connection manager(s) before hand, so how do you set the connection string dynamically depending on the value of some data in the recordset?
March 20, 2006 19:00
 

jamie.thomson said:

Hi Antony,
Interesting problem that.

Take a read if this post: http://blogs.conchango.com/jamiethomson/archive/2005/12/04/2458.aspx

It doesn't give you a solution for your problem but it does explain how you can go about setting the name of the destination dynamically. You would have to modify it to base the name of the file on some charateristic of your data. I hope that gives you something to go on and if it doesn't, please let me know.

Its also worth pointing out that you could get all the distinct IDs and push them into a recordset using the recordset destination. Then loop over that recordset and use the currently iterated value to pull out the records that you are interested in.

I dare say you'll be using raw files at some point along the way as well.

Hope that has given you some pointers!

-Jamie

March 20, 2006 20:41
 

Tristan B said:

Jamie,

  I'm having an issue dynamically updating a DB connection with an expression.  The connection type is OLE DB and in the Expression window of the Execute SQL Task I'm setting the connection to the string variable @[User::DBConn].

User::DBConn is being set in an earlier step to :

Provider=SQLNCLI;Server=testDB;Database=Test_60_SLP_DEV;UID=sa;PWD=password;


When I execute the packe I receive the following

Error: 0xC00291EB at EST_Gather Corp Ids, Execute SQL Task: Connection manager "Provider=SQLNCLI;Server=testDB;Database=Test_60_SLP_DEV;UID=sa;PWD=password;" does not exist.


 Do you have any suggestions?  I would really appreciate any info you may have

  Thanks much,

Tristan
April 10, 2006 16:46
 

jamie.thomson said:

Tristan,
I assume you have checked that that is the correct ConnStr by pasting it directly into the ConnStr property at design-time?

-Jamie
April 10, 2006 16:57
 

Tristan B said:

Jamie,

 Got it now... Thanks!!

Tristan
April 10, 2006 18:47
 

Jamie Thomson - SSIS Jibber Jabber said:

For reasons that I'll save until another post, I always deploy my packages as files rather than to SQL...
August 18, 2006 11:31
 

SSIS Junkie said:

Here's a little tip for you. Some database pros are skeptical about the arrival of the XML data type

January 16, 2007 18:09
 

Christina Spain said:

Hi Jamie,

i have read this and as i said in the previous post (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1146555&SiteID=1) i have used the expressions option to this.

I will try to explain it again ( i know my english is not very good and i suppose this is a problem ;-P ).

I have a foreach loop container, in this, i have configured the "transactionoption" property within a user variable using the expressions option. This variable gets value at the beginning of my dts (though a script task) because of its value depends of a validation.

My problem is the transacion property of my foreach loop container takes the value by default  of the user variable but not the value assigned on run time. I have checked that my script informs this variable and this variable takes the new value, but it seems the transactionoption property is not refreshing this value.

I hope you can help me.

Regards,

Thx.

January 24, 2007 16:19
 

Michael Hamilton said:

Excellent. You saved me hours.

July 27, 2007 08:30
 

dynamically set the Connection String said:

Jaime,

I am trying to set the conenction strign dynamically via Expression. I am getting some non fatal errors with that. can you please what i was doing wrong?

ConnectionType=OLE DB

Connection String in the Expression "Provider=SQLNCLI;Server=" +  @[User::DBServer] + ";Database=GHR_Raw;Integrated Security = True; "

After evaluating expression is

Provider=SQLNCLI;Server=localhost;Database=myDB;Integrated Security = True;

I want to use this by changing the DBServer in test server and production server with Configuration values.

It is not recongnizing this dynamically build string.

can you please help?

I am using this Execute SQL Task

August 1, 2007 22:39
 

jamie.thomson said:

What do you mean its not recognising it?

Do you get an error message? What is it? Is it not failing but still not doing what you want?

Does the problem occur at design-time or execution-time?

More info please.

August 1, 2007 22:50
 

Mythreyi said:

I am getting this error when ever I try to use the Expression to build the connection string dynamically.

The connection "Data Source=usdallnac4;Initial Catalog=GHR_Raw;Provider=SQLNCLI.1;Integrated Security=SSPI;Auto Translate=False;" is not found. This error is thrown by Connections collection when the specific connection element is not found.

I really need that so that on different servers I can set the expression value via configuration file.

Your help is much appreciable.

Thanks

Mythreyi

August 14, 2007 20:21
 

Cynthia said:

Fabulous learned a lot from your Blogs

September 20, 2007 18:12
 

Paul said:

Hi Jamie

Great SSIS info has really helped on many occasions.

In one of the expression builder screenshots above, you show the evaluated value being populated with ‘c:\temp\myfile.csv’. I have used the ‘for each’ container successfully many times but don’t get an evaluated value – am I missing something in my setup?

November 27, 2007 10:33
 

jamie.thomson said:

Paul,

Hard to say from the info provided here. Could you post a question to the SSIS forum (http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=80&SiteID=1) with some extra info?

-Jamie

November 27, 2007 14:45
 

Arbon said:

Hi Jamie - maybe you have some insight - I'm using expression builder to build a connection string with SSPI security.  If I set up the connection just by entering the information properly in the "Edit..." page it works well.

If I use the Expression builder to build an exact copy of the automatically generated connection string (and verify it by pressing "evaluate BLOCKED EXPRESSION, I get the following error and the package fails.

[OLE DB Destination] SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONAMANAGER.

and

[DTS.Pipeline] Error: Component "OLE DB Destination" (315) failed validation and returned error code 0xC020801C

and

"Login failed for user ''.  The user is not associated with a trusted SQL server connection."

Any insight?

February 21, 2008 16:12
 

Paul said:

Jaime,

I have never considered myself a programmer, rather more of an analyst, however now am taked with learning SSIS which is far more program leveraging than I am accustom, this said, please bear with me.  I am trying to build a connection string as you are above to loop through some CSV files located in different folders on a share drive.  I have a table in my db outlining sources and destinations for each of the files paths to the sql table.  I am having difficulty as when I created my variable @[User::SourcePath] being "Select Distinct left(path_name,charindex(@User::UniqueDivision,path_name))" i recieve only the expression back not the result that I anticipate.  What am I doing wrong?

Thank you,

May 1, 2008 22:31
 

jamie.thomson said:

Paul,

You need to set:

EvaluateAsExpression=TRUE

Expression=<your expression>

Don't just put your expression into the 'Value' property.

Is that the problem?

-Jamie

May 2, 2008 10:22
 

sandy said:

hello Jammie,

what i need to do is capture path of the file during each iteration and pass it to a script task. please can you suggest me a way to acheive this.

thank u

sandy

June 1, 2008 02:44
 

jamie.thomson said:

Sandy,

it tells you above. Store it in a variable and then it'll be available to you in a script task.

-Jamie

June 2, 2008 14:31
 

Mahesh said:

Hello,

I am not able to configure the connection string dynamically.

I have tried building expression

"Data Source=" + @[User::m_strServer]  + ";User ID= "+ @[User::m_strUserId] + ";Password= "+ @[User::m_strPassword]  + ";Initial Catalog= "+ @[User::m_strDatabase] + ";Provider=SQLNCLI.1;Auto Translate=False;"

When I execute the package following error occurs.

[Execute SQL Task] Error: Failed to acquire connection "OTINSISRCEFDB1\SFA_INT_ST". Connection may not be configured correctly or you may not have the right permissions on this connection.

Please let me know what is wrong?

June 5, 2008 08:28
 

jamie.thomson said:

Mahesh,

Try using SQL Profiler to determine why the connection attempt failed.

-Jamie

June 5, 2008 09:34
 

Rama said:

Hello Techies

I am using 2 variables to set my connection string of a flat file connection.

When i evaluate the expression it give right path. But when i consume same ocnnection in a flat file source component, I get run time error. The path comes wrongly there.

e.g I am using below expression.

@[MessageChannel::WorkingFolder] +  @[Argument::InputFileName]

it evaluates \\localhost\temp\path\test.txt

but at runtime it shows

\localhost\temp\path\localhost\temp\path\test.txt

test.txt

thus i get error. Can anyone please help me in resolving this?

June 23, 2008 08:33
 

Jules said:

Hi Jamie,

I have a slightly different problem. I would like to get the connection string from the connection manager in a script component under a data flow task. I would like to do this to pass the connection string to a DLL component I have created that aggregates the data grabbed by SSIS.

Currently I simply pass in a string value (I could change this to a variable but it wouldn't achieve what I want), but I have to change it when I copy the script to a new server, i.e. UAT / PROD. I would like to simply update the configuration file and forget about the code during deployment.

There must be a way of accessing the connection settings via the script component, so it can be read and then passed to my DLL object.

Any ideas?

Jules

June 23, 2008 12:09
 

jamie.thomson said:

Jules,

There's some code here: http://blogs.conchango.com/jamiethomson/archive/2005/10/10/SSIS-Nugget_3A00_-Verify-a-data-source-before-using-it.aspx that explains how to grab the connectionstring from a connecion manager from within script.

-Jamie

June 23, 2008 12:21
 

Jules said:

Hi Jamie,

Thanks for your help - I can use it to populate a connectionString variable and then access it from anywhere in my SSIS package.

However I was hoping to access the connection string data in a script transformation object within a data flow task. To do this it looks like I have play around with the IDTSConnectionInfo90 and IDTSConnectionManager90 interfaces.

Probably simpler to populate a variable.

Jules

June 24, 2008 09:45
 

rubyA said:

Hello Jamie,

       Is it possible to change the connection Type in execute sql task. LIke from ODBC to ADO.net etc?

Thanks

July 16, 2008 11:32
 

NewBee said:

Hi Jamie,

I have created a variable in which I store a SQL Server connection string. Another variable named table name. I am trying to use them in the BULK INSERT task. [In the expression boxes, setting the property of Destination Connection and Destination table to the variables.] I get failures here -

The Connection .... is not found. This error is thrown by Connections collection when the specific connection element is not found.

But things are working fine when I use the Connection Managers in the Connection page of BULK INSERT.

I am very new to SSIS, so taking a lot of time to deal with the errors.

Please let me know what is the approach needed.

Thanks

NewBee

October 21, 2008 11:36
 

Jonas Carlsson said:

Hi

I try to set the connection string dynamically and it works fine for "execute SQL tasks" inside i "Foreach Loop Container" but when i try to use the same connection manager in a "Data Flow Task" for an OLE DB Source i fails with a validation error:

Validation error. Data Flow Task: OLE DB Source [1]: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER.  The AcquireConnection method call to the connection manager "manager" failed with error code 0xC0202009.  There may be error messages posted before this with more information on why the AcquireConnection method call failed.

Any clues on what Iam doing wrong?

/Jonas

November 27, 2008 13:31

Leave a Comment

(required) 
(optional)
(required) 
Submit

This Blog

Syndication

Powered by Community Server (Personal Edition), by Telligent Systems