Ayende @ Rahien

It's a girl

Rhino ETL & FileHelpers Integration

Well, this image really excites me. It excites me because I got this after integration FileHelpers into Rhino ETL. This image is the result of an ETL script that joined a file against a table, did some additional processing on it and push it to a table in a third database.

I must say that FileHelpers has made this ridiculously easy to do.  All I have left to do is figure out how to test such a thing effectively.


For reference, here is the complete script:

class Customers:
      CustomerID as string
      CompanyName as string
      ContactName as string
      ContactTitle as string
      Address as string
      City as string
      Country as string


      ConnectionType: SqlConnection,
      ConnectionString: "Data Source=localhost;Initial Catalog=ETL_Test; Integrated Security=SSPI;"

      ConnectionType: SqlConnection,
      ConnectionString: "Data Source=localhost;Initial Catalog=Northwind; Integrated Security=SSPI;"


source CustomersFile:
            for customer in Read(typeof(Customers)).From("""Files\CustomersTab.txt"""):
                  SendRow( Row
                        CustomerId: customer.CustomerID
                        CompanyName: customer.CompanyName,
                        ContactName: customer.ContactName


source OrdersFromDatabase, Connection="Northwind":
      Command: "SELECT * FROM Orders"


join CustomersAndOrders:
      if Left.CustomerID == Right.CustomerID:
            Row.OrderID = Right.OrderID
            Row.CompanyName = Left.CompanyName
            Row.ContactName = Left.ContactName


 destination Final, Connection="DestinationDB":
      Command: "INSERT INTO OrdersWareHousing VALUES(@OrderID, @CompanyName,@ContactName)"


pipeline OrdersWareHouse:
      CustomersFile >> CustomersAndOrders.Left
      OrdersFromDatabase >> CustomersAndOrders.Right
      CustomersAndOrders >> Final


 target default:


Markus Zywitza
08/09/2007 06:47 PM by
Markus Zywitza

That's marvellous!

Is there any documentation on Rhino ETL other than your blog entries? If not, is there a good starting point in the source code to look at?


Ayende Rahien
08/09/2007 07:27 PM by
Ayende Rahien

Markus, no documentation yet.

The best place to start look is in the integration tests, I'll upload soon the changes required to support FileHelpers

08/09/2007 09:02 PM by

Wow nice to see the seamless integration !! =)

Thanks a lot for your words Ayende, them are a great motivation to continue developing and releasing the FileHelpers

There are large hours enhacing the API make it more and more usable, mostly thanks to the CLR core team with his:

Designing .NET Class Libraries


And with the Excellent BOOK: Framework design guidelines.

In the company that I work for Im the responsable for our base framework to generate code and so. So I love Framework and LIbrary design, also to design winforms control.

I wish to found time soon to help with a Rhino ETL GUI or some examples

You are doing a OUTSTAND WORK DUDE !!!

About the code you need to add the [FieldQuoted(OptionalForBoth)] because without quotes some values can contains a tab and so fail to load.

We are working in a class level attribute to set all the fields as quoted to make it easy.

Best Regards

Talk to you soon

Jeff Brown
08/12/2007 09:32 AM by
Jeff Brown

Just curious. Why is the syntax for declaring connections different from the rest of the pipeline? Is it side-effecting?

Ayende Rahien
08/12/2007 10:03 AM by
Ayende Rahien

No particular reason, actually, and a good point for improvement.

Comments have been closed on this topic.