Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 6,124 | Comments: 45,471

filter by tags archive

Comments

release candidate

gzip all files and then read the compressed stream at once?

Max
Max

Run it in the profiler.

csokun

merge file

Ryan

Compensate for the seek time somehow? Assuming you were on spinning metal drives

Falhar

I would say to merge the files. Makes it easier for subsequent work.

But this merge itself would take some time. But this is really good place for asynchronous IO.

Duckie

Batching Disable indexing (POST /admin/stopindexing )

Duckie

I found some earlier talk about this in google groups 8 feb. Looking a github, you added Self optimizing on batch-sizes:

https://github.com/ravendb/ravendb/commit/294c2134c5fa7b0b95d0297dfac38cb9ab9acd38

Felice Pollano

My two cents: Use IO completion ports ( async ) for reading the files, but mantain the reading strictly sequential. In the "done" function publish an job and consume it by another ( more than one? thread(s) ) pushing data into the target, this will at least compensate the seek time by doing something useful, and possibly if somethime the push opertaion is delayed ( I guess it can be, but maybe I'm wrong ) you can even keep scanning the hard drive.

flukus

Move the files to SSD?

Pure Krome

Handball the problem over to Itamar?

Sam
Sam

+1 for Itamar. Handballing is an extremely efficient operation.

Roger Helliwell

Like flukus said... or even better, mounted a ram disk and filled it with your 3.1 million files.

Itamar

Yeah, he tried with no luck. I agreed to providing mental support at 1AM instead, though.

:-)

SPATEN

SSD

Dmitry

You had done a regular data import and RavenDB did its magic.

Josh

Increased the file chunk size to match the NTFS storage format chunk size

Martin

Put the kettle on?

Every problem is a magnitude simpler when tackled with a hot cup of coffee.

Bundermuft

As you probably already had SSD, you did nothing

Joe
Joe

Loaded them into RavenFS ;-)

Nick

You told the freedb team to get their shit together and clean their mess up?

Bordev

Defrag

Tom Robinson

Disabled your anti-virus?

Harry

Enjoyed a cup of coffee

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. RavenDB 3.5 whirl wind tour: You want all the data, you can’t handle all the data - about one day from now
  2. The design of RavenDB 4.0: Making Lucene reliable - 3 days from now
  3. RavenDB 3.5 whirl wind tour: I’ll find who is taking my I/O bandwidth and they SHALL pay - 4 days from now
  4. The design of RavenDB 4.0: Physically segregating collections - 5 days from now
  5. RavenDB 3.5 Whirlwind tour: I need to be free to explore my data - 6 days from now

And 14 more posts are pending...

There are posts all the way to May 30, 2016

RECENT SERIES

  1. RavenDB 3.5 whirl wind tour (14):
    29 Apr 2016 - A large cluster goes into a bar and order N^2 drinks
  2. The design of RavenDB 4.0 (13):
    28 Apr 2016 - The implications of the blittable format
  3. Tasks for the new comer (2):
    15 Apr 2016 - Quartz.NET with RavenDB
  4. Code through the looking glass (5):
    18 Mar 2016 - And a linear search to rule them
  5. Find the bug (8):
    29 Feb 2016 - When you can't rely on your own identity
View all series

RECENT COMMENTS

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats