http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660There are many other open source ETL tools that you might want to have a look at first, for instance: http://kettle.pentaho.org/ http://www.cloveretl.org/clover-etl/ http://www.ketl.org/ http://www.enhydra.org/tech/octopus/index.html http://www.glassfishwiki.org/jbiwiki/Wiki.jsp?page=ETLSE http://scriptella.javaforge.com/ Some of these are quite powerful. Not as fun as writing your own tool/DSL of course, but great if you want to abandon SSIS quickly. :-) http://ayende.com/2645/test-driving-rhino-etl#comment4http://ayende.com/2645/test-driving-rhino-etl#comment4Fri, 20 Jul 2007 20:16:32 GMTOh, forgot to ask. Will you be including any "common" data cleaning features in your ETL solution? In fact, do you even think that they have a place in such a library? Common ones I've written recently are things like... * Remove commas from numbers * Trim and convert empty string to null * Reformat UK postcodes * Make title case * Remove blank rows * Remove repeated column headers in data * Derive title from name and drop into column 'n' * Unpivot repeated groups onto new rows, Unpivot( startCol, colsPerGroup, numberOfGroups) * Format dates * Remove illegal dates You get the idea :-) http://ayende.com/2645/test-driving-rhino-etl#comment3http://ayende.com/2645/test-driving-rhino-etl#comment3Fri, 20 Jul 2007 08:10:13 GMTLooks sweet, is that so you can redirect sources & targets at run time? I haven't really dug into it, but I was scanning the ActiveWarehouse Ruby project the other day which has it's own way of dong things. Might be good for inspiration! http://activewarehouse.rubyforge.org/etl/. It uses control files to orchestrate sources, transforms etc much like your own does. See example: http://viewvc.rubyforge.mmmultiworks.com/cgi/viewvc.cgi/etl/trunk/test/delimited.ctl?root=activewarehouse&view=markup I'm working on an ETL project right now, which was started by someone else as a bespoke VB.NET solution. I asked him if he's considered using any ETL tools such as SSIS or Kettle etc, but he didn't feel that they gave him enough control. I think a lot of developers feel this way. I only used SSIS briefly, but found that it encouraged far too much mouse clicking, and that annoys me because it's not easily automated. Well, it probably is, but I didn't hang around long enough to find out how! Anyway, I won't rant on. I'I look forward to giving the Ayende flavour of ETL a spin! http://ayende.com/2645/test-driving-rhino-etl#comment2http://ayende.com/2645/test-driving-rhino-etl#comment2Fri, 20 Jul 2007 07:13:56 GMTCan't wait... whens it going in the repo? http://ayende.com/2645/test-driving-rhino-etl#comment1http://ayende.com/2645/test-driving-rhino-etl#comment1Fri, 20 Jul 2007 01:24:46 GMT