Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 5,971 | Comments: 44,508

filter by tags archive

Rhino ETL & FileHelpers Integration


Well, this image really excites me. It excites me because I got this after integration FileHelpers into Rhino ETL. This image is the result of an ETL script that joined a file against a table, did some additional processing on it and push it to a table in a third database.

I must say that FileHelpers has made this ridiculously easy to do.  All I have left to do is figure out how to test such a thing effectively.

image

For reference, here is the complete script:

[DelimitedRecord("\t")]
class Customers:
      CustomerID as string
      CompanyName as string
      ContactName as string
      ContactTitle as string
      Address as string
      City as string
      Country as string

     

connection
      "DestinationDB",
      ConnectionType: SqlConnection,
      ConnectionString: "Data Source=localhost;Initial Catalog=ETL_Test; Integrated Security=SSPI;"
      )

connection
      "Northwind",
      ConnectionType: SqlConnection,
      ConnectionString: "Data Source=localhost;Initial Catalog=Northwind; Integrated Security=SSPI;"
      )    

     

source CustomersFile:
      execute:
            for customer in Read(typeof(Customers)).From("""Files\CustomersTab.txt"""):
                  SendRow( Row
                        CustomerId: customer.CustomerID
                        CompanyName: customer.CompanyName,
                        ContactName: customer.ContactName
                        ))

 

source OrdersFromDatabase, Connection="Northwind":
      Command: "SELECT * FROM Orders"

     

join CustomersAndOrders:
      if Left.CustomerID == Right.CustomerID:
            Row.OrderID = Right.OrderID
            Row.CompanyName = Left.CompanyName
            Row.ContactName = Left.ContactName

           

 destination Final, Connection="DestinationDB":
      Command: "INSERT INTO OrdersWareHousing VALUES(@OrderID, @CompanyName,@ContactName)"

           

pipeline OrdersWareHouse:
      CustomersFile >> CustomersAndOrders.Left
      OrdersFromDatabase >> CustomersAndOrders.Right
      CustomersAndOrders >> Final

     

 target default:
      Execute("OrdersWareHouse")


Comments

Markus Zywitza

That's marvellous!

Is there any documentation on Rhino ETL other than your blog entries? If not, is there a good starting point in the source code to look at?

-Markus

Ayende Rahien

Markus, no documentation yet.

The best place to start look is in the integration tests, I'll upload soon the changes required to support FileHelpers

Marcos

Wow nice to see the seamless integration !! =)

Thanks a lot for your words Ayende, them are a great motivation to continue developing and releasing the FileHelpers

There are large hours enhacing the API make it more and more usable, mostly thanks to the CLR core team with his:

Designing .NET Class Libraries

http://msdn.microsoft.com/netframework/programming/classlibraries/

And with the Excellent BOOK: Framework design guidelines.

In the company that I work for Im the responsable for our base framework to generate code and so. So I love Framework and LIbrary design, also to design winforms control.

I wish to found time soon to help with a Rhino ETL GUI or some examples

You are doing a OUTSTAND WORK DUDE !!!

About the code you need to add the [FieldQuoted(OptionalForBoth)] because without quotes some values can contains a tab and so fail to load.

We are working in a class level attribute to set all the fields as quoted to make it easy.

Best Regards

Talk to you soon

Jeff Brown

Just curious. Why is the syntax for declaring connections different from the rest of the pipeline? Is it side-effecting?

Ayende Rahien

No particular reason, actually, and a good point for improvement.

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. Paying the rent online - about one day from now

There are posts all the way to Aug 03, 2015

RECENT SERIES

  1. Production postmortem (5):
    29 Jul 2015 - The evil licensing code
  2. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
  3. API Design (7):
    20 Jul 2015 - We’ll let the users sort it out
  4. What is new in RavenDB 3.5 (3):
    15 Jul 2015 - Exploring data in the dark
  5. The RavenDB Comic Strip (3):
    28 May 2015 - Part III – High availability & sleeping soundly
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats