http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Brian, Oh, I can certainly do that, no need for C++, you can do that with PInvoke. Problem is that I can't force .NET to allocate on that heap.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment23http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment23Wed, 13 Jun 2012 09:43:09 GMTSo, why can't you create a managed C++ helper DLL to call HeapCreate() [& other low level functions] and manage what you need from C++? If I'm not mistaken, you can link up managed C++ DLLs in C#.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment22http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment22Wed, 13 Jun 2012 07:56:04 GMTI do not buy in that you do not know how much memory you can allocate. Under 64 bit there is virtually no limit until the page file is full. But you do not want to page out the managed heap because each GC is needing nearly all pages again. For 32 bit there is the a limit around 1.1-1.3 GB of private bytes. You should set a memoryfailpoint in this region to stay clear of bad surprises. Your current approach assumes that you are faster if you can consume more memory. But this assumption breaks down as soon as the system is spending large amounts of time paging data into the page file and data out of the page file to complete a full GC. It could be as easy to check Memory:Page Faults/sec to see the system already swapping too much. At this point of time I would throttle my activity and not continue to consume more memory. This will serve you much better than to exhaust the complete page file. You can then treat an OOM as really bad error and stop working. You could continue working if the OOM does happen on an pure indexing thread and you know which data is corrupt. On any other thread you should terminate your app. http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment21http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment21Tue, 12 Jun 2012 19:20:27 GMTThis is a very interesting problem. I wonder if it would be useful to have a community standardized API (akin to CommonServiceLocator) with which to query the resource consumption of the process, with OS-specific implementations? Regarding your current solution, it does seem like a time-bomb as any allocation may throw (as in a method call that needs to allocate memory in which to JIT code).http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment19http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment19Tue, 12 Jun 2012 00:42:20 GMTRafal, No, just the places where I know that are likely to cause the issue. Other than that, it is wrapped in standard exception handling and that can take care of that.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment18http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment18Mon, 11 Jun 2012 15:04:04 GMTHow do you make sure that you handle OOM exceptions in all RavenDB threads? Do you wrap everything in try {} catch(OOM)? http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment17http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment17Mon, 11 Jun 2012 14:58:50 GMTHmmm.. I wonder what Ricky Leeks would think about this. :) http://www.dotnetrocks.com/default.aspx?ShowNum=760 http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment16http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment16Mon, 11 Jun 2012 14:26:00 GMTJesse, Same AppDomain, the cost of marshaling across app domains is too expensive.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment15http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment15Mon, 11 Jun 2012 12:27:22 GMTAyende, so do these ADF's run in the main application memory or in a separate AppDomain?http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment14http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment14Mon, 11 Jun 2012 12:14:16 GMTI remember a presentation from the Chief Architect of Myspace (a couple of years back) where they ran into memory issues in their caching solution. So they build a custom memory management system where they allocated a number of huge binary arrays and then they just serialized/deserialized the objects to and from those huge arrays. It sounded crazy at the time, but they were on a crazy scale of course.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment13http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment13Mon, 11 Jun 2012 12:12:45 GMTPaul, Not really. I can tell how much a document is on disk, but I don't know how much it is in memory (not the same), and I don't have any way of measuring the amount of data that the UDF is using, nor how much mem is consumed in the actual indexing. http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment12http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment12Mon, 11 Jun 2012 12:03:29 GMTYou can try to use a MemoryFailPoint estimating the amount of memory that you require for the task. It will fail with a InsufficientMemoryException if you try to allocate more than expected during the lifetime of the MemoryFailPoint object. http://msdn.microsoft.com/en-us/library/system.runtime.memoryfailpoint.aspxhttp://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment11http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment11Mon, 11 Jun 2012 11:57:36 GMTSo you _could_ know the document size when they request the document to index? You could also build up a heuristic of the amount of memory required for their function by comparing the amount of memory consumed before/after for a given document size. I agree it wouldn't be a perfect measure, but as an order of magnitude it could work.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment10http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment10Mon, 11 Jun 2012 11:53:34 GMTPaul, Simple scenario, I need to index some number of documents. I don't know what their size is (I don't know how many, for that matter), and I need to execute a user supplied function over them, and index the results. I don't know what the results of the user supplied function would be, or what would be the cost of indexing things (is it a single term, analyzed, etc).http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment9http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment9Mon, 11 Jun 2012 11:37:59 GMTAyende, Interesting point; how do you not know how much memory you need, at least within an order of magnitude? You can make multiple calls to MemoryFailPoint and reading the MSDN topic it is about throttling the amount of memory you allocate. I clearly don't understand the context of your reason for doing this, but have bad experience of doing what you are doing and the process becoming corrupt, so maybe that reflects my dislike of the construct.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment8http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment8Mon, 11 Jun 2012 11:28:55 GMTCynic, I thought about this, but this get complex REALLY fast, and considering that we want to be able to run on IIS, Mono, etc, I don't think it is worth it.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment7http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment7Mon, 11 Jun 2012 10:03:09 GMTPaul, I would love to be able to do that, but I don't know in advance how much memory I'll be using.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment6http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment6Mon, 11 Jun 2012 10:01:54 GMTHave you tried custom CLR hosting for your preferred (HeapCreate etc.) solution?http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment5http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment5Mon, 11 Jun 2012 09:52:59 GMT"In 99.9% of the cases, if you see this, it is a mistake" There are all kinds of "best practices" and "never use" guidelines on the internet simply because people fail to explain "the why", "the how" or "the what" and simply limit themselves to saying "just don't do that". Cathing OOM exceptions, is indeed dangerous if not done right, it's one of the more dangerous things to do if not done properly. But like many other things considered dangerous, it can be verry helfull in certain situations. The point: "I realy hate mindles guidelines and rules" that simply state don't do that is bad. http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment4http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment4Mon, 11 Jun 2012 09:48:42 GMTGenerally speaking recovering from asynchronous exceptions in .NET is not recommended - depending on where they happen they can corrupt the app domain or process. This is discussed here: http://msdn.microsoft.com/en-us/magazine/cc163298.aspx The better approach (also discussed in the article) is to use the MemoryFailPoint class (http://msdn.microsoft.com/en-us/library/system.runtime.memoryfailpoint.aspx) to predict if an allocation will fail. This is not guaranteed because of race conditions etc, but is better than just running up to the limit and then failing.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment3http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment3Mon, 11 Jun 2012 09:44:59 GMTSimon, Yes, there are minimum boundaries under which we will not go down.http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment2http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment2Mon, 11 Jun 2012 09:25:29 GMTDo you try to scale the memoryconsumption up again? If not couldn't you run into what could be considered memory-starvation? Say I once in a while run a longish-running very memoryconsuming task on my ravendb-server - if that coincides with ravendb doing this optimization couldn't the memory-usage by ravendb be scaling back too much? or do you have a bare minimum you use to ensure reasonable performance?http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment1http://ayende.com/156161/code-crimes-because-even-the-law-needs-to-be-broken#comment1Mon, 11 Jun 2012 09:23:25 GMT