Reviews

Responding to Effectus commentary

Ayende Rahien — Fri, 08 Jan 2010 10:00:00 GMT

Jose was kind enough to post a review of my sample Effectus application. This post is a reply to that.

The first thing that Jose didn’t like was the fact that I didn’t put an abstraction layer in front of NHibernate’s usage in my application. There are several reasons for that, the first, and simplest, is that I was trying to explain how to use NHibernate, and for that, I wanted to deal with the lowest common denominator, not show off a particular wrapper implementation. Showing NHibernate usage directly means that even if you are using another library with it, you would still be able to take advantage of the information that I give you.

Second, and quite important, is that by using NHibernate directly I can take advantage of NHibernate features explicitly meant to support certain scenarios, but which are not likely to be expose when libraries wrap NHibernate. A good example of that is in Jose’s sample code, which make use of an ISession instead of IStatelessSession to load the data for the main screen. As explained in the article, the difference between the two is important, and in the context where it is used it introduce what is effectively a memory leak into Jose’s implementation, as well as the chance for some really interesting errors down the road if the session will run into an error.

Third, Jose bring up the following:

This presenter is GLUED to NHibernate. And this means for instance that you can’t test it without NHibernate, and this means that you can’t change your persistence layer.

Yes, and that is intentional. That is the very task that those presenters are for. Trying to abstract that away just means that I put an additional layer of abstraction that does absolutely nothing. For that matter, let us look at Jose’s implementation using an abstraction layer compared to mine.

Here is my implementation:

private void LoadPage(int page)
{
    using (var tx = StatelessSession.BeginTransaction())
    {
        var actions = StatelessSession.CreateCriteria<ToDoAction>()
            .SetFirstResult(page * PageSize)
            .SetMaxResults(PageSize)
            .List<ToDoAction>();

        var total = StatelessSession.CreateCriteria<ToDoAction>()
            .SetProjection(Projections.RowCount())
            .UniqueResult<int>();

        this.NumberOfPages.Value = total / PageSize + (total % PageSize == 0 ? 0 : 1);
        this.Model = new Model
        {
            Actions = new ObservableCollection<ToDoAction>(actions),
            NumberOfPages = NumberOfPages,
            CurrentPage = CurrentPage + 1
        };
        this.CurrentPage.Value = page;

        tx.Commit();
    }
}

And here is Jose’s:

private void LoadPage(int page)
{
    var actions = _toDoActionsDao.RetrieveAll()
                                 .Skip(page * PageSize)
                                 .Take(PageSize).ToList();

    var total = _toDoActionsDao.RetrieveAll().Count();
    
    NumberOfPages.Value = total / PageSize 
                        + (total % PageSize == 0 ? 0 : 1);
    
    Model = new Model
    {
        Actions = new ObservableCollection<ToDoAction>(actions),
        NumberOfPages = NumberOfPages,
        CurrentPage = CurrentPage + 1
    };
    
    CurrentPage.Value = page;
}

The code is still doing the exact same thing, in other words, we haven’t moved any logic away, and most of the code is dealing with data access details. My approach for testing this is to make use of an in memory database, which result in a single test that make sure that the code works, instead of a series of tests that verifies that each piece work independently and then my single test. I find it much more effective in terms of time & effort.

As for the idea of changing your persistence layer, forget about it. It isn’t going to happen in any real world application without a lot of effort, so you might as well save yourself the effort of working with the lowest common denominator and take full advantage of the framework. Just to note, when I ported the NerdDinner code to NHibernate (an application that make use of a single table), I had to make major changes to the application.

Jose didn’t like this method:

public void OnSave()
{
    bool successfulSave;
    try
    {
        using (var tx = Session.BeginTransaction())
        {
            // this isn't strictly necessary, NHibernate will 
            // automatically do it for us, but it make things
            // more explicit
            Session.Update(Model.Action);

            tx.Commit();
        }
        successfulSave = true;
    }
    catch (StaleObjectStateException)
    {
        var mergeResult = Presenters.ShowDialog<MergeResult?>("Merge", Model.Action);
        successfulSave = mergeResult != null;

        ReplaceSessionAfterError();
    }

    // we call ActionUpdated anyway, either we updated the value ourselves
    // or we encountered a concurrency conflict, in which case we _still_
    // want other parts of the application to update themselves with the values
    // from the db
    EventPublisher.Publish(new ActionUpdated
    {
        Id = Model.Action.Id
    }, this);

    if (successfulSave)
        View.Close();
}

Because:

ReplaceSessionAfterError, to much responsibility for a presenter.
Session/Transaction again.
There is a BUG, the publish mechanism work in sync with the rest of the code. This means… that this windows is not going to close until others have finished handling the event. For the given example, the Edit windows is not going to close until the main window finish to refresh the current page.
Too much logic for this method = hard to test.

I fully agree that this is a complex method, and Jose’s refactoring for that is an improvement indeed.

[PersistenceConversation(ConversationEndMode = EndMode.End)]
public virtual void OnSave()
{
    _toDoActionsDao.Update(Model.Action);

    EventPublisher.Enlist(new ActionUpdated
    {
        Id = Model.Action.Id
    }, this);

    View.Close();
}

public override void OnException(Exception exception)
{
    if(exception is StaleEntityException)
    {
        Presenters.ShowDialog<MergeResult?>("Merge", Model.Action);

        EventPublisher.Enlist(new ActionUpdated
        {
            Id = Model.Action.Id
        }, this);

    }
}

Jose has implemented the following changes:

I’ve defined a new convention, if a public method in the presenter throw an exception, the OnException method will be called. This is done by a castle interceptor, and this is the last line of defense for unhandled exceptions.
I’m using “StaleEntityException” rather than “StaleObjectStateException”, this is MY exception. This is easily done by a CpBT artifact.
I’m not calling “EventPublisher.Publish” anymore, this code use EventPublisher.Enlist. Here, I’ve split the “Publish” code in two different methods one for Enlist and other for Raise. The enlisted events will be raised right after the OnSave method is called and thus after the windows is closed.
Also, notice that here is the conversation per business transaction pattern with all its splendor. The two former methods are conversation participants, with EndMode equals to continue. This means that the NH Session will remain opened. The OnSave method has EndMode equals to End, this means that right after the method finished, CpBT internally will flush the Unit of Work and close it.

This is better than the original implementation, but I think it can be made better still. First, OnException as a generic method is a bad idea. For the simple reason that the exception logic for different method is different, I would probably define a [MethodName]Error(Exception e) convention instead, which would make it easier to separate the error logic for different methods.

Again, I don’t find any usefulness in abstracting the underlying framework, I haven’t seen a single case where it was useful, but I have seen a lot of cases where it was hurting the team & the project.

The idea about splitting the publication and raising is really nice, I agree.

However, there is a problem in the code with regards to session handling in cases of error. There is a pretty good reason why I introduced the ReplaceSessionAfterError method. In general, I want to keep my session alive for the duration of the form, because I get a lot of benefits out of that. But, if the session has run into an error, I need to replace it and all the objects associated with it. Closing the session no matter what is going on is not a good solution, and you can’t really solve the problem in a generic way without calling back to the presenter that generated the error.

Git commits as code review?

Ayende Rahien — Thu, 15 Oct 2009 04:40:00 GMT

I just had to go through a code base where I had a bunch of of comments.

Instead of going with the usual route of just noting the changes that I think should be done, I decided to do something else. I fixed each individual change, and commit them individually.

This is how it looks like, each commit is usually less than a single screen of changes (diff mode).

I wonder if it is something that I can apply more generally.

Reviewing NerdDinner: The Select N+1 pitfall

Ayende Rahien — Thu, 30 Jul 2009 20:55:00 GMT

During my review of NerdDinner, I came across the following piece of code:

And I knew that there is a Select N+1 problem here. This is quite similar to the problem that I described here. RSVPs is a lazily loaded collection. As such, calling Any() on it will result in a database query.

Now the question was whatever or not it is used somewhere in a loop. ReSharper will help us figure that one out:

The first usage location is here:

And this certainly doesn’t look like it can cause a Select N+1, right. Just to be sure, I checked where it is called from the client side as well:

This looks all right, we will have a query, but it is only one per request.

Great, now let us take a look at the second usage. You can tell that this is going to be a problem just by the ascx suffix. The code is pretty simple:

I have a moral issue with the view generating a call to the database, but at least it isn’t being done in a loop.

I am certain, however, that we are going to find this .ascx file in a grid, which will cause a loop. This .ASCX file is being used in Details.aspx and is used like this:

However, it is not actually being used in a loop.

There is not a Select N+1 issue, which renders this entire post moot.

However, while it is not an active Select N+1, it is a dormant one. It would be very easy to have a new requirements that would make use of the RSVPStatus.ascx file in a loop and cause the issue.

Reviewing NerdDinner

Ayende Rahien — Thu, 30 Jul 2009 20:19:00 GMT

Scott Hanselman has asked me to see if I can port NerdDinner to NHibernate, now that Linq for NHibernate has made RTM. I thought that I might as well take the time to look at the code and do a code review.

I would like to say a few things before actually even starting to look at the code. I am not going to review the code as a sample application, I am going to review it as if it was a standard production project. I am assuming that (remember, haven’t seen the code yet) there are things there that are done that way explicitly because they are easier to explain as a sample app than a more complex project would be.

The first thing that I opened was HomeController, which I am going to show you in full.

There are two things that I don’t like here.

First, the empty actions (return View() is empty as far as I am concerned) resulting in what is, for all intents and purposes, an empty controller (one that contains empty actions). In other words, the only reason that this controller exists is because it is the way ASP.Net MVC want it to. A brief overview of the codebase show me that there are more than a few of method like that.

I would deal with something like that completely on the view side, having a routing action that would check the view directly and just render the view, rather than creating empty actions and controllers.

Second, [HandleErrorWithELMAH], such things should not be on controllers (and that attribute appears on all controllers). It should be on a base controller, because right now, it is a violation of DRY, and it is easy to forget.

Then there are the tests for HomeController, which are… less than useful, shall we say?

Next, let us take a look at the following interface:

I have a serious problem with the Save() method. I had to look at the implementation to figure out what it is doing. It should be called FlushToDatabase, or SubmitChanges. Save() is a totally unexpected name for the method who implementation is:

You are not, actually, saving anything.

As an aside, I don’t like seeing things like this one:

Linq to SQL actually have a way of handling cascading deletes, and it should have been used.

For that matter, coming back to the tests, there aren’t any persistence tests. Since there are some complexity involved in the queries (geo located queries), I would have liked to see at least those queries being covered.

I don’t want to rehash the poor man’s IoC story, but this is really pissing me off:

I mean, if you want to do poor man’s IoC, go ahead. But please don’t create this bastard child.

This is more about code style than hygiene, but I don’t like it:

Those classes should be in separated files, they should not be in AccountController.

Overall, I don’t see too many issues with the codebase. The only thing that pissed me off was the Mad Man IoC.

Review: C# in Depth

Ayende Rahien — Tue, 12 May 2009 10:10:00 GMT

C# in Depth has a very different focus from most “Learn language X” books. Starting from the premise that you are already am familiar with the basic language syntax (for 1.0, or maybe you are a Java or C++ programmer), it focus entirely on the new additions to the language and platform.

Its stated goal is to take C# 1.0 developers and give them all the changes that happened to the language in the C# 2.0 and 3.0 versions. And it most certainly deserves the “in Depth” part of the name.

I consider myself a fairly proficient developer, and I believe that I have adequate knowledge in both C# 2.0 and 3.0, but I still found myself learning new things. More to the point, as someone who do know much of the material in the book, I was quite impress with the quality of the material, the depth of the discussion and the level in which it is being presented.

I think that Jon has managed to capture a lot of the complexities of the language in a way that is approachable, easy to understand and complete.

I have been recommending that book for clients ever since I read it, and only recently realized that I have never actually posted about it. I kept intending to, but that doesn’t seem to put words on the blog, unfortunately (otherwise I would blog even more).

The complexity of the C# is a personal worry of mine, mostly because I see how hard it is for people to bridge the gap when moving to the newer versions of the language and having to face the explosion of possibilities. I think that this book is a big step in closing that gap.

Perhaps the best compliment that I can give to the book is that I fully intend to use the 2nd edition as the text to read to get into C# 4.0 when it is out. No reason not to let Jon do all the hard work :-)

Ode to ReSharper

Ayende Rahien — Wed, 22 Apr 2009 17:34:48 GMT

I am using R# 4.5, and I have to say, for a release that is supposedly all about perf, they managed to squeeze in some really nice new smarts:

Figuring out all the implementations that use or do not use parameters.

Killing off BDD :-)

Reviewing NChord: A codebase that makes me twitch

Ayende Rahien — Tue, 07 Apr 2009 01:41:09 GMT

I was referred to NChord by one of the comments, and it looked very interesting on the surface, so I downloaded the code and took a peek.

It is not typically my style to criticize code openly, especially after a very brief glance, but I thought that it might be a good idea to point some of the things that made me decide that the codebase isn't worth the time to pursue in details.

Single Responsibility Principle doesn't apply to the file level

What you see here are no less than 11(!) files that are all parts of the same class. That is.. bad, from a whole lot of reasons, not least of which is that you have to jump through a lot of files to understand what is going on, and that ChordInstance is at least a demi-god, rapidly moving upward.

Then we have things like this:

Now, this is a matter of taste more than anything else, but it strongly hints that there is lack of threading experience going on here. Indeed, a brief look at the code shows numerous potential race conditions.

Code like this really make me twitch:

All of that trend toward making me doubt the maturity and stability of the codebase.

In short, after a very brief scan, I find numerous issues that I really didn't like in the code base. And I am not willing to put up the effort to try to make headway into this with so many early warning signs already in place.

First Steps with Post Sharp

Ayende Rahien — Thu, 09 Oct 2008 17:52:25 GMT

PostSharp is an AOP framework that works using byte code weaving. That is, it re-writes your IL to add behaviors to it. From my point of view, it is like having the cake (interception, byte code weaving) and eating it (I haven't even looked at the PostSharp source code, just used the binary release).

My initial spike with it went very well. Here it is:

[Serializable]
public class Logger : OnFieldAccessAspect
{
    public override void OnGetValue(FieldAccessEventArgs eventArgs)
    {
        Console.WriteLine(eventArgs.InstanceTag);
        Console.WriteLine("get value");
        base.OnGetValue(eventArgs);
    }

    public override InstanceTagRequest GetInstanceTagRequest()
    {
        return new InstanceTagRequest("logger", new Guid("4f8a4963-82bf-4d32-8775-42cc3cd119bd"), false);
    }

    public override void OnSetValue(FieldAccessEventArgs eventArgs)
    {
        int i = (int?)eventArgs.InstanceTag ?? 0;
        eventArgs.InstanceTag = i + 1;
        Console.WriteLine("set value");
        base.OnSetValue(eventArgs);
    }
}

This is an aspect that run on each field access. It is not really useful, but it helps to show how things works. A couple of things that are I think are insanely useful:

Aspects are instantiated at compile time, allowed time to set themselves up, then serialized to an resource in the assembly. At runtime, they are de-serialized and ready to run. The possibilities this give you are amazing.
InstanceTag is a way to keep additional data per aspect.

Now, let us assume that I want to add the aspect to this code:

[Logger]
public class Customer
{
    public string Name { get; set; }
}

Note, there is no field. (Well, there is, it is generated by the compiler). Now we compile and run the PostSharp post compile step. With that, we can now investigate what is going on.

As you can see, we are deserializing the attribute and storing it in a field that we can now access. Let us check the Customer implementation now:

We have the logger field, which is used for something, but we also have the ~get~<Name>k__Backingfield and ~set~<Name>k__BackingField. <Name>k__BackingField (and I would love to hear the story behind that) is the compiler generated field that was created for us. The ~get~... and ~set~ are generated by PostSharp. Before we look at them, we will look at the implementation of Name.

Where it used to call the field directly, now it is doing this via a method call. And now we can look at those method calls.

There is a lot going on here. We create a new field access event arg, call the aspect method, and return the value. Note that the state (instance tag) is stored in the object as well, for each field access.

It looks very well done.

Reading Eralng: CouchDB Streams

Ayende Rahien — Sun, 05 Oct 2008 23:37:58 GMT

A question to my dear readers, do you find this series valuable? Do you consider it interesting? It is a significant departure from my usual set of topics.

In my recent post I wondered about the concept of summary streams, and how they related to the way CouchDB works. I couldn't figure out what they were doing. As it turn out, there is a good documentation for them in this post. There were two things that mislead me. The first was the notion of summary. I think that this is a misnomer, because this is the only place in CouchDB where that term is used. It is not a summary, it is the actual document. As a matter of fact, I think that it is a term that was carried over from Notes.

The second misleading clue was stream. I am used to think about streams in the classic sense, as a way to access a stream of bytes. The CouchDB notion of stream is quite different. It is a way to optimize disk access, it seems. I wondered about that, because the nature of CouchDB append only file seems certain to cause a lot of issues with regards to internal fragmentation in the file (which would require a lot of seeks, which are slow).

Let me see if I can deconstruct what is going on in couch_stream, first, we have the structure declaration:

This seems to be pretty reasonable. The write_stream is defining a reserved space in the file, note the next_alloc field. It looks like we are allocating memory (or disk space). Should be interesting. The stream structure just hold the process and the file description, and isn't really interesting.

The initialization of a stream is interesting in itself:

Here we just create a new write stream and copy the initialization values to the state. There seems to be a 1 to 1 mapping between a stream and an erlang process. Let us examine how we write the data. First, we have the stop condition of running out of data to write:

You can learn a lot about the a function in erlang just from its declaration. In this case, <<>> means a binary with no items in it, hence, we run out of things to write.

And now we come to the function clause that deals with the issue of running out of room to write it.

The first part of the function is using variable binding to extract the values out of the stream. It goes against the grain, I know, to see CurrentPos being "set" when it is on the right side of the assignment, but that how it works. (Well, actually it isn't being set, it is being matched, or bound, but that is another issue).

Next we find what is the next size that we have to allocate, and ask the file to expand. I don't think that I have seen this before, let us take a look:

We first get a the end of the file, and then we write a single byte at the end of the file plus the expansion value. In other words, we increase the length of the file by however big Num is. For .NET, this is the equivalent of the SetLength call, and it is important to create continuos files, with as little fragmentation as possible.

Going back to write_data, we have this line:

The syntax isn't really nice, in my opinion, but what we have here is basically: Write to the file Fd at position CurrentPos the value of NewPos (with pack to FILE_POINTER_BITS and then the value of NewSize (packed to STREAM_OFFSET_BITS).

It is important to note where this is written. Go and take a look at the case statement in which this expression is. First, we setup enough size for the data we want to write and for the next allocation. When we come to the end of the current chunk, we create a new one (by expanding the file) and then write the address of the new chunk into the end of the chunk, following that by a move to the new chunk.

The last part of write_data is very simple:

We start by figuring out how many bytes we have to write, and then split the binary data by that. We write what we can to the file, and then recursively call ourself (which will either exit (nothing to write) or create a new chunk of the file and continue writing it.

Elegant, short and to the point. It take more time to describe how it works than it is to write the code. The code for reading is just as sweet:

The stop condition is when we have no more data to read. The second clause is when we have run out of data to read in the current stream, and we need to read about the next one. Again, erlang's pattern matching is useful here, because it allow to easily unpack the values from the file to in memory representation.

The last clause is where things are actually happening. We select the number of bytes to read, offset is a really misleading term here, it is not the offset from the beginning of the chunk (like most of us would think), it is the amount of bytes remaining in the current chunk.

We read the data to memory, update the Sp (stream position?) and then call the function that we were passed, to find out if we should read more or stop.

Now, how are those streams used?

From reading the code, it looks like there are two streams used in CouchDB. The first is the document stream (called the summary_stream). And the second (actually, the seconds) is a stream for all the binary attachments for a document (a stream per a set of document attachments).

And with this, we conclude the reading of CouchDB persistence architecture. Next topic, views, and how they are used.

Erlang Reading: CouchDB - Digging Down to Disk

Ayende Rahien — Sat, 04 Oct 2008 12:32:46 GMT

Jan has corrected a misconception that I had here, where I assumed that PUT was create and POST was update. Apparently this is not quite correct. A PUT request will update/create a new document with a user supplied name, while a POST request will create a new document with a server generated name.

That narrows down my search for "create new document" code path again. Here is the method in couch_httpd_db that handles a POST request on a document.

It parse the form, get the revision of the document, and then it calls to open_doc_revs, which we haven't explored so far. I feel confident that I understand what this is doing, however:

I'll not show the open_doc_revs_int, because it is too big, but it contains some references that so far I haven't encountered. Specifically, couch_key_tree, which I don't think that I seen before, and to get_full_doc_infos, which looks like this:

Oh, now we are on a more familiar territory. Lookup performs a search on a binary tree, and we have gone over it in depth before. However, we can now see that the we pass to it a fulldocinfo_by_id_tree... I don't think we have explored how CouchDB make use of the btree, so let us head to the db structure and see what else we have there.

Here is what I know about this structure at the moment. db_header container the data that is actually written to the file, while db is the in memory structure. Not sure what write_version is all about, update_seq looks like the the internal sequence number for CouchDB. I don't think that I have even seen the concept of stream before, but it is there, and look like most stream implementations that I am already familiar with.

get_state on that returns a {Position, BytesRemaining}, since I don't understand how this is used at the moment, I don't understand why this is important. fulldocsinfo_by_id_btree_state contains the root of the btree, and the same holds for docinfo_by_seq_btree_state, I assume that local_docs_btree_state is also the same.

purge_seq is a counter for the last purge (physical delete), and it is used in conjunction with the views in order to ensure that there are no purged documents in the view. purged_docs keeps track of the documents that were last purged.

db, the runtime structure, is considerably more interesting. It took me a while to figure out where it is created, but I finally found the init_db function in couch_db_updater:

The first part is to open the summary_state stream, still not sure what this is about, though. We then define a comparison function for use when opening the fulldocsinfo btree, and we pass several additional functions when we create the btree. Reduce is usually used when we compact the tree, and to get information about the btree (such as count of documents). The join and split also prepare the data to be written to disk in a consistent fashion.

I don't like the name join/split. The internal names, assemble and extract seems to be much more accurate.

Finally, we get to the actual creation of the db structure itself. An interesting thing to note is that only the update_pid is filled. Since CouchDB implements the many readers / single writer, I assume that this is how we ensure the single reader, because only a single process ever handle writes to the DB. I am assuming for now that main_pid is for readers, and compactor_pid if for compacting (obviously). I'll track them next.

Well, that took some doing, the main_pid is set in the couch_db_updater:init (the same function that calls the init_db function):

I had hard time tracking this because I couldn't figure out what who was calling couch_db_updater:init (and what is the MainPid). Usually, in gen_server processes, there is a start_link function that you call, but in this case, there wasn't. Some digging revealed that the couch_db process is the one that actually call start_link on the couch_db_updater. I wonder if this is a common pattern in erlang, because it sure did caught me by surprise.

So main_pid is the couch_db process, and update_pid is the couch_db_updater. It actually goes a lot further than that. couch_db_updater doesn't actually have a public interface. It looks like couch_db is the one that exposes the interface, and send synchronous messages to the couch_db_updater process:

That is the way in which CouchDB ensures that we have only a single write, I presume. Compaction is handled in couch_db_updater as well, the implementation is interesting:

We create a new process and return before we finish compacting. It is interesting because it is the first example of explicit concurrency that I saw. Let us dig a bit better into start_copy_compact_int, which is pretty straight forward:

This just create a new file, init a database and call_compact_docs, finishing by notifying the updater that it finished compacting. copy_compact_docs is interesting in itself:

We start by defining the EnumBySeqFun, which batches documents to write to disk until should_flush returns true. should_flush is funny, it check memory thresholds and flush to disk whenever there is about 10 MB of data ready to be flushed.

The usage of foldl here is a good reason to go back and examine it in more details:

And fold is defined as:

So we have this stream_node, whatever that is, let us take an additional look.

We are already familiar with get_node(), but adjust_dir and stream_kp_node and stream_kv_node are new. The first is easy:

Let us explore stream_kv_node, which is likely to be easier than stream_kp_node.

This is slightly complex. We start by defining the drop off point (smaller / greater from the current key). The important bit is in the last function, the call to our function and the recursion back into the function. As a reminder, here is the function that we pass:

I hope that this make it clear how this works. We simply stream all the items to the function, and it copies it. Note that an important subtlety is that we have several btree over the same data (we will touch down in a bit) and in this case we are only copying the items that are greater than the update_seq in the current DB. Still not sure what this update_seq is all about. We will go all the way back to copy_compact_docs_int's last line, where it notify the updater that it finished the current compaction. This message arrive to this method:

This looks complex, but it really isn't. It is simply opening the new database, and checking if the current update_seq equal to the latest in the new database. If it isn't, we restart the process. If it is, then we copy the local documents (not sure why they get special treatment, and it looks like it is expected to have very few of them), move the files and move on with our lives.

Now, it looks like update_seq is updated whenever someone make a change to the database, but let us verify this, shall we?

Looking at couch_db_updater:update_docs_int, we can see that the update_seq is updated for each new document that is modified (see merge_rev_trees).

Okay, that was interesting. But I still don't know how we save a document to disk. Let us go back to couch_db:update_docs, and see what is going on here. This is pretty complex, so we will break it apart to discrete pieces.

We start by generating a new revision for the documents. Note that in the case local revisions, we just increment the revision, if it is a standard document, we generate a random revision number. I think that this is done in order to support farm wide revisions, so if I update document #1 in server A and in server B, I get two different revisions, which would be detected as conflicts.

We extract all the new revisions, group all the documents by id and extract all the ids. So far, so good. Let us see the rest of the function:

We get the current existing documents, and then we merge the existing with the new bucket. That requires some explanation.

The first function clause uses list comprehensions in order to ensure that if we couldn't find a document in the DB, we didn't specify a previous version. The second is more complex and require us to understand about key trees. CouchDB defines them as:

Based on that and the dict:from_list arguments, we can deduct what the structure of a revision is. It should be noted that it looks like there is an expectation that the number of revisions wouldn't be too big. This make sense, CouchDB does keep previous versions, but it make no guarantees about their life spans. Also, we should note that we have both revs and rev_tree, I am not quite sure why.

The last line in the zipwith function is prepare_doc_for_new_edit, which is called for each document, let us take a look at it.

This is an example of how condense functional code can get. Let us see what it does:

Get the new revisions and all the previous revisions
If the previous revisions has a value, it perform a lookup based on the revision to the revision tree, to get the revision information. (Which answers the question of why we need both revs and rev_tree. revs is an ordered list of revision keys. rev_tree is apparently the metadata).
Now, the LeafRevsDict contains only the most recent one, so failing to find it in the dictionary means that we trying to update a non current version, so we error.
I am not sure what a stub is, but the rest of it seems pretty simple.

The last part is interesting, however, we check that we aren't trying to create a new document with the same id as existing one (although we allow it if that document is deleted).

After the call to zipwith(), we have a call to doc_flush_binaries, which we totally ignored so far. It flushes all the documents to disk, it is not really interesting, it is dealing with writing the attachments to disk. I am not quite sure what is going on (it is complex), mostly because I don't follow to which file it is writing. Or, to be rather more exact, I am not following how the attachment can be written to a different file, which is what the code seems to imply.

Anyway, the code there is pretty clear about what is going on, we append to the current file (except if we write to a different file, again, not following that) and update the in memory structure of the document to point to the range of data that each attachment contains.

Let us look at the last part of the couch_db:update_docs:

And is looks like we have hit the place where it is actually happening. We call to the couch_db_updater in order to do the actual update, and we make be asked to retry (this will require more study), in which case we do try.

Let us look at update_docs at the couch_db_updater.

Sigh, I can feel it in my waters, update_docs_int is going to be complex... once more, we will try to divide and conquer.

We start by extracting variable from the Db, then we split the list based on whatever this is local or not. NonRep documents (local) are not replicated, have only a single version and are expected to have very few of them. I still don't know what is the use case for those, however. It looks like DocsList is a list of lists of docs, probably this is the buckets that we had to deal with earlier.

We finish the snipper by extracting the ids from the documents, what comes next?

We lookup the existing documents, and then we merge them into a file info (note that we create a new full_doc_info with just the id if this is a new file). We then create the new doc infos by calling merge_rev_trees. We will ignore the implementation for now.

Finally, we gather all the sequences that we would like to remove. Moving on, we see that we start by updating the local docs and then flushing the trees. Both of which looks promising for my quest to understand how the data is stored on the disk:

update_local_docs just modify the in memory btree, so I am not going to show it, flush_trees should be more interesting, based on just the comment.

This is interesting. What we see here is that we write the attachment references and the document body to the summary stream. That is interesting, because I am still not sure what the summary stream is. Note that here we have the issue of writing to a file during compaction, so we ask the higher level code to retry that again.

We still have to see how the indexes are maintained, but that just update the in memory structure, the real bit happens next:

Here we see our good old friend the btree, and we add/remove the new index to the file. We have seen how this works here. We finish by updating the Db structure with the new values and committing the data.

I now feel that I have a much better understanding on how CouchDB file handling works. Except that I still don't know what this summary stream is. That is a topic for later, though.