Ayende @ Rahien

Unnatural acts on source code

SSIS: I know better than you do

If you haven't guess it by now, I am not fond of SSIS. The latest installment is probably a "feature". Assume the following, I develop an SSIS package on my local machine, testing it against a local database.

Now, I want to run the package against a remote database. I did the Right Thing and put the connection string in a data source, so I change that and run the package from Visual Studio. Imagine my surprise when Visual Studio does a full cycle, including reporting the number of rows that it copied. Everything seems to be fine, until I checked the database itself.

About half the tables where empty, and I am still not sure why. The best guess that I can make is that it is caching the previous database credentials, and writing to that, since I found the data in the local database. Argh!!

Comments

Brandon Beacher
04/15/2007 03:32 PM by
Brandon Beacher

Oren,

Are you using package configurations yet?

I had similar issues when deploying my packages to production. I solved them by enabling package configurations and putting then connection string into an xml config file, .dtsConfig.

When testing locally, the dtsConfig file points to my local installation of SQL Server.

After I am done testing, I double-click a batch file with a Robocopy command which copies the package to a separate Production folder while excluding the dtsConfig file. The separate Production folder has its own dtsConfig which points to the production SQL Server.

Now in this folder I can double-click the package, select the production dtsConfig file, and run it using DTSExecUI. It runs outside VS 2005, so you don't get the pretty green, yellow and red boxes, but you only need those while testing, right? :) Plus, DTSExecUI probably uses less memory than when it is running in debug mode in VS 2005.

Anyways, I've used Brail on several Monorail projects, so I hope that helps you out.

  • Brandon
Ayende Rahien
04/15/2007 04:10 PM by
Ayende Rahien

Brandon, thanks for the tip.

Is it me, or does it seems like a HUGE endevour for such a small and common task?

Vitaly
04/17/2007 02:45 AM by
Vitaly

Yes, I had the same issues when dealing with SSIS. The config mgmt is quite a PITA, and I never managed to get it to the point where it was very seamless.

BIDS doesn't cache db connections -- it just applies either (1) the hardcoded connection info in the data sources or db conn managers or (2) loads the conn info from a variable/config file. Your best bet is to export all conn managers (if you use File Conns, export them as well) to the config file, and modify the conn string there. In order for this to work, though, BIDS (or dtexec.exe) will need to be able to find your config file at runtime, or else it uses the default conn string.

Also, if you are exporting conn strings to a config file (or some other store) and you have "Don't Save Sensitive" set on the package, then the password will not be exported -- you'll need to add it manually after the conn string is exported.

By the way, SSIS is pretty extensible, so you can always roll your own components/tasks (and even provide nice designers for them)! :) (hey, you did say you wanted to write code didn't you? :)).

I usually used dtexec.exe to run packages that were nontrivial since it runs much faster than BIDS (less VS overhead, obviously).

I think MS took the right step with SSIS (the fact that it's a .NET library with extension points is the biggest plus, in my opinion), but it being v1.0, there're quite a few unpleasant quirks with it.

Ayende Rahien
04/17/2007 03:28 AM by
Ayende Rahien

SSIS is pretty extensible, so you can always roll your own components/tasks

And if I was doing Complex stuff, I would. When I am trying to do things that are 101, I am very annoyed that it doesn't work, or have unexpected issues.

I have no issues with having to do it in code, I have issues with realizing that I could replace the entire SSIS process here with code that would take SO MUCH LESS time to deal with.

Vitaly
04/17/2007 03:52 AM by
Vitaly

In this specific case, if you were to roll your own impl, then you'd still need to dynamically set the conn string, and that info would need to be stored somewhere (i.e. config file). In other words, you'd be doing the same thing as SSIS is doing for you.

Mind you, I agree that SSIS is a bit funky, esp when you initially start working with it, but I did find that it solves simple cases pretty well once you get the "hang of it". You're just paying your dues! :)

Ayende Rahien
04/18/2007 08:40 PM by
Ayende Rahien

That still seems to be a funky way to do things.

Comments have been closed on this topic.