http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660I agree with Jan that documenting early stage code is useless. The beauty of Erlang as a functional language is that one writes a lot less code. So it may be dense but in some ways it's easier to get your head around. For example couch_btree in the latest from trunk is only ~900 loc. You may want to have a look at Distel, it makes emacs into a full-featured IDE http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment13http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment13Wed, 12 Nov 2008 12:23:19 GMTFirst, thanks for the effort to write this up. I became entranced with Erlang not so long ago. Selling it as an enabling tech (where appropriate) to my company is harder. More info and explanations of key products helps. It also helps me as well! @Mark: That sounds like a very interesting and worthwile project. I don't know of anything that pulls it all together in just such a way. There's been some pieces of it - trac, whatever the PHP online reference manual is done with (which I wish I knew), and things like that. But they are just parts, seems like you are seeing the whole. http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment12http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment12Thu, 09 Oct 2008 13:18:03 GMTHi, you try to increase understanding of the CouchDB codebase by presenting, commentating and explaining it. This is great! The feedback here indicates, that others are interested as well. For long I think that such commentation is necessary for more parts of the Erlang ecosystem, particulary the Erlang/OTP release itself, if we want to ever have a couple of core contributors outside the Ericsson team. My gut feeling is that such efforts should be supportable by a web application. (Possibly one that makes use of CouchDB :-) I still struggle to understand what features and processes such a system should have. Usually I first try to identify what stuff is already there, that could be used for this. My list so far contains: * The Talmud - it features a unique graphical representation of a core text and systems of comments around it * Lions Commentary on Unix , an example of a longer technical commentary, alas in print, so no feedback * Blog articles - like your blog article, where you manually present part code, part explanation and other people make comments via the blog comment function * Donald Knuth's literate programming - it presents part code, part comment, but derived it from a combined source text I would love to combine elements from all of these into a web application. It should work along side a source code repository (svn, git) and allow a group of developers to collectivly work on an explanation, with annotations, articles, feedback. If anyone would be interested to help developping such, please drop me a mail. http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment11http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment11Fri, 26 Sep 2008 07:20:17 GMTI have not yet read through the whole article but nevertheless I'd like to thank you; explaining real-world Erlang code to others looks to me like a great step at promoting Erlang and showing its power to the newcomers. Keep up the good work; probably some day I'll do something similar :) http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment10http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment10Thu, 25 Sep 2008 09:42:07 GMT> In the first case, I am no sure why we need io:format("~s~n", [Msg]) That's a print function. http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment9http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment9Thu, 25 Sep 2008 09:19:11 GMT"CouchDB" http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment8http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment8Wed, 24 Sep 2008 12:13:43 GMTHm, it seems that there is some interest in .NET world for the scalability provided by the CoachDB. Although it is a bit different from what the mainstream devs are used to. http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment7http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment7Wed, 24 Sep 2008 12:13:01 GMTThanks for the write up! You did a lot better than me with your first attempt at groking the code. http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment6http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment6Wed, 24 Sep 2008 10:09:07 GMTJan, I also have: [ayende.com/.../...couchdb-reading-btreelookup.aspx](http://ayende.com/Blog/archive/2008/09/24/more-couchdb-reading-btreelookup.aspx) [ayende.com/.../...b-reading-btreequery_modify.aspx](http://ayende.com/Blog/archive/2008/09/24/more-couchdb-reading-btreequery_modify.aspx) Regarding reduce_stream_kv_node2, I didn't like the 2 prefix. > Since the caller passes in the state at the request and requests are isolated, concurrent requests don't interfere with each other and mess up the state. But don't you need to handle the global state in some fashion? For example, let us say that you have two processes trying to register to be notified at the same time, how do you handle that? You need to either serialize the requests or to have isolated processes that don't have all the information. I am going to follow up on the lucene part on the mailing list. http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment5http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment5Wed, 24 Sep 2008 09:54:38 GMTHi, this is an impressive walkthrough! I have added a few note: > There is a set of lookup functions that I am not sure that I can understand. That seems to be a problem in general with the CouchDB source code, there is very little documentation, and I don't think that it is just my newbie status at Erlang that is making this difficult. This is intentional. a) At this point the code is way too much in flux to make proper documentation not a waste of time and b) we are trying to keep the newbies out :) If you can figure out the source, your patches are likely to be of a higher quality :) > I got up to get_node (line 305) before I realized that this isn't actually an in memory structure. This is backed by couch_file:pread_term, whatever that is. And then we have write_node. I am skipping ahead and see things like reduce_stream_kv_node2, which I am pretty sure are discouraged in any language. What's discouraged about it? `kv` is likely to be key/value and IMHO a valid abbrev. in CouchDB code since k/v-pairs are all over the place. and numbering function names is not per-se bad. See below. > In the first case, I am no sure why we need io:format("~s~n", [Msg]), that seems like a no op to me. io:format("~s~n", [Msg]), is equivalent to printf("%s\n", Msg); in C > On the other hand, it is fairly complex, for something that procedural code would handle with extreme ease and be more readable. I tend to agree :) > The last line is interesting. Because servers needs to maintain state, they do it by always returning their state to their caller. It is the responsibility of the Erlang platform (actually, I think it is OTP specifically) to maintain this state between calls. The closest thing that I can think of is that this is similar to the user's session, which can live beyond the span of a single request. > However, I am not sure how Erlang deals with concurrent requests to the same process, since then two calls might result in different state. Maybe they solve this by serializing access to processes, and creating more processes to handle concurrency (that sounds likely, but I am not sure). Since the caller passes in the state at the request and requests are isolated, concurrent requests don't interfere with each other and mess up the state. > couch_ft_query is very interesting. It is basically a way to shell out to an external process to do additional work. This fits very well with the way that Erlang itself works. What seems to be going on is that full text search is being shelled elsewhere, and the result of calling that javascript file is a set of doc id and score, which presumably can be used later to get an ordered list from the database itself. > > This seems to indicate that there is a separate process that handles the actual indexing of the data (Lucene / Solar seems to be a natural choice here). It isn't currently used, which maps to what I know about the current state of CouchDB. It is interesting to note that Lucene is also a document based store. See: [svn.apache.org/.../lucene-search/](http://svn.apache.org/viewvc/incubator/couchdb/branches/lucene-search/) and: [http://github.com/davisp/couchdb/tree/external](http://github.com/davisp/couchdb/tree/external) http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment4http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment4Wed, 24 Sep 2008 09:17:49 GMTI enjoy your guided tours of codebases. The C# translations made the content consumable. http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment3http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment3Wed, 24 Sep 2008 06:40:50 GMTYou're welcome! Actually I've been interested in Erlang and its premises, but haven't investigated it thoroughly. It seems a language designed for concurrency from the start is and will be more and more important to us as we go further into the multi-core world. That said, wasn't the whole reason ProLog/LISP/Scheme never became popular was because how difficult it is to read? It seems like Erlang suffers the same drawbacks... I suppose that's expected being as its based on ProLog syntax. Half of programming is reading others work. If that's hard then it doesn't matter that the program was quick to write. Still... as you said, that's a small amount of code for what it's doing. Maybe I'm just suffering from my lack of familiarity with Erlang and other functional languages. http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment2http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment2Wed, 24 Sep 2008 04:53:55 GMTI'm glad to see you are digging into the source. Erlang can be a little hard to read, and couch_btree is one of the toughest parts of CouchDB to grok. couch_httpd.erl is probably the best place to start understanding, simply because is mirrors the well-documented HTTP API. You raise a good question: How does CouchDB store key/value pairs in a BTree structure, while also maintaining an append-only file structure? It's one I've been wondering about lately, and I plan to ask Damien next time I get the opportunity. http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment1http://ayende.com/3607/reading-erlang-inspecting-couchdb#comment1Tue, 23 Sep 2008 23:39:06 GMT