Rhino ETL & FileHelpers Integration
Well, this image really excites me. It excites me because I got this after integration FileHelpers into Rhino ETL. This image is the result of an ETL script that joined a file against a table, did some additional processing on it and push it to a table in a third database.
I must say that FileHelpers has made this ridiculously easy to do. All I have left to do is figure out how to test such a thing effectively.
For reference, here is the complete script:
CustomerID as string
CompanyName as string
ContactName as string
ContactTitle as string
Address as string
City as string
Country as string
ConnectionString: "Data Source=localhost;Initial Catalog=ETL_Test; Integrated Security=SSPI;"
ConnectionString: "Data Source=localhost;Initial Catalog=Northwind; Integrated Security=SSPI;"
for customer in Read(typeof(Customers)).From("""Files\CustomersTab.txt"""):
source OrdersFromDatabase, Connection="Northwind":
Command: "SELECT * FROM Orders"
if Left.CustomerID == Right.CustomerID:
Row.OrderID = Right.OrderID
Row.CompanyName = Left.CompanyName
Row.ContactName = Left.ContactName
destination Final, Connection="DestinationDB":
Command: "INSERT INTO OrdersWareHousing VALUES(@OrderID, @CompanyName,@ContactName)"
CustomersFile >> CustomersAndOrders.Left
OrdersFromDatabase >> CustomersAndOrders.Right
CustomersAndOrders >> Final
Is there any documentation on Rhino ETL other than your blog entries? If not, is there a good starting point in the source code to look at?
Markus, no documentation yet.
The best place to start look is in the integration tests, I'll upload soon the changes required to support FileHelpers
Wow nice to see the seamless integration !! =)
Thanks a lot for your words Ayende, them are a great motivation to continue developing and releasing the FileHelpers
There are large hours enhacing the API make it more and more usable, mostly thanks to the CLR core team with his:
Designing .NET Class Libraries
And with the Excellent BOOK: Framework design guidelines.
In the company that I work for Im the responsable for our base framework to generate code and so. So I love Framework and LIbrary design, also to design winforms control.
I wish to found time soon to help with a Rhino ETL GUI or some examples
You are doing a OUTSTAND WORK DUDE !!!
About the code you need to add the [FieldQuoted(OptionalForBoth)] because without quotes some values can contains a tab and so fail to load.
We are working in a class level attribute to set all the fields as quoted to make it easy.
Talk to you soon
Just curious. Why is the syntax for declaring connections different from the rest of the pipeline? Is it side-effecting?
No particular reason, actually, and a good point for improvement.