Okay, you probably noticed by now that I'm talking a lot about the amount of queries that a page makes. Right now I managed to drop it from 91 to 72 (didn't test in the process, and wasted tons of time) from 72 to 63 (tested in the process, took minutes). It was all about putting lazy in the right places (and fixing the tests that assumed that the data is always there).
I then tried to find out exactly where most of the traffic was, the first part was an easy target, just finding out that I made a couple of dozen calls for the exact same data was a big relief. I used the ASP.Net Cache, and that dropped the number of queries to 41.
Then I was stumped. I knew what was going on, but I didn't know how to fix it. I'd the following problem: Blog->Posts->Comments. Where I need to do some processing on blogs that I thought was also involving the comments. The result of that was two statements to load the blog and the posts, and another 38 statements to load each post comments. Looking at the code, I realized that I didn't really need the comments for that particular process. (I likely will never need them for a blog-wide process). Marking the Comments as lazy dropped the number of queries to three. If I was really desperate, I could have gone to two (which is minimal amount of statements that I could produce manually). I don't choose to do that because… well, that would mean that the UI would have to make decisions about how the data would be loaded.
I rather leave that decision outside of the UI layer at the moment. It's not a problem to add this feature, but the difference between two or three queries is not big enough to justify adding responsibilities to the UI that don't belong there.
FireFox 1.5 is nice, but it has a very serious problem for me.
It doesn't support Hebrew well. It can display it, but it doesn't support mixing it with English or with numbers.
Here is how a page looks like in FireFox 1.5:
Here is how it looks like in IE 6: (This is correct).
I once had to implement printing a mixed document of Hebrew, English and numbers. The amount of stuff that you need to do it is staggering. And you are never going to get it truly right.
It's interesting to note that FireFox 1.0 did support it.
Paul Graham has an article about Great Hackers that is worth reading. (I listened to the podcast, and it's much better when you can hear him speaks than just read the text). Joel has a very good article about Leaky Abstractions (although no podcast about that).
Put those things together, and you can get a fairly good idea about why leaky abstractions are so annoying for developers. If you read my blog you're aware of much hair pulling and gnashing of teeth in regards to ASP.Net that has been going on lately. Most of the problem is that I don't really understand what is going on under the hood. On the other side, I've had a similar experience with NHibernate and ActiveRecord.
I had a page that had to display a list of records from the database. Currently the database holds ~5 rows. The page loaded very quickly, but I took a look at the profiler to see what kind of SQL statements were executed and I nearly had a scream fit about it*. 18 statements for loading 5 rows? All of them in the same table? I got into a short argument (with myself, this time) about the usefulness of an ORM that generate such lousy code. I then put a couple of dozens of Lazy = true in various places in my code, and watch the page load again. There was only one SQL statement this time, and while it wouldn't pass the code police (horrible names), it did the work. For a moment, though, I had a very real sense of what would've happened if another developer, who isn't familiar with NHibernate or ActiveRecord would have with the same problem.
The second issue that I had was a matter of cascading. I thought that I had information, because I created it and then saved the root object to the database. But when I tried to get the information, I couldn't find it. The issue is (probably) that I didn't configure the cascades properly. I think that I fixed this by adding the proper cascade (I'm writing this post in the breaks that I have to take because of the long compile times for ASP.Net). Adding cascades broke the tests, which require some fixing, but it was very straightforward, most of the time.
Okay, I usually try to have a point with my posts, so let's try to give this one at least moderate one. The issue here is about control; understanding and how leaky is the abstraction. NHibernate is great, and I think that I cut in more than half the time that I need to spend thinking about data access. In fact, it's so good that I don't think about it most of the time.
Implementing the Session-Pre-Request and Session-Per-Test is very straightforward using ActiveRecord, and it is a joy to work with. I had written some very bad application in the past (and I'm sure that I'll say it again in the futureJ), and I distinctly remember the amount of thought that I had to dedicate to building any non-trivial queries. Keeping an eye on what is going on under the covers is good, but it is so good not having it in your face all the time.
* One SQL statement was so horrendous it was 5KB(!!) and had 11 joins! I couldn't follow it, and I was too afraid to see what the execution plan looked like.
I find that I much rather have a solution where I need to write a little code to make the hard parts work, than have a complex framework that tries to do anything (look at XSD for an example, if you need one.)
I would like to dispute his other claim, that DB portability is not important. It's not important for an in-house project, yes. But the moment you need to integrate a project into an existing infrastructure, you'll soon find that it's not going to be that simple. Some people use SQL Server, some favor MySQL, some like Oracle, etc. You probably wants to sell to all of them. DB portability is an important feature in most applications that are going outside.
The game is called bugs and it's addictive.
I warned you, now check it out. Cute.
Yes, I know that I said that I'm not going to post today. But Jeff fixed a problem with Partial Mocks (couldn't call a method that had parameters). It's 0:45AM, and I'm releasing software... I think that I could be arrested for this. :-)
Thanks Jeff, and it really is turning into a daily build project :-D
Okay, last post for today.
In my current project, I've a class that is the center point for the rest of the system. It's not a Blob, but rather delegate to a lot of other classes and methods to do the work. It's sort of like a facade over the system, but for the system itself. I now find myself in the position where I need to provide some caching support for data that I pull from the database.
The issue is that while the application itself is a web application, I'm trying very hard to make sure that it doesn't depend on the web itself to do its work. But the caching that I need is for something that I need only on certain requests, but then definately need it (currently it makes 21 calls to load the same information). When the request end, I want the cache to go away.
This is very easy on the web, since I can use the current request to store the items, but this cause problems with both non-web usages (which is possible) and while using tests. I'm currently thinking of using some sort of per thread LRU hash table. I'm not very happy about it, though, so any ideas would be welcome.
I have 2Gb on this machine! Whoopee! I already run into issues with MSN Search & Outlook because of it, but I hope to get the patch tomorrow.
Beyond that, life is good, I've so much memory that the OS doesn't know what to do with it, so it leaves most of it alone. That will change when I start doing development again on this machines.
I feel a burning in my eyes, a pain in my temples. What can it be? Perhaps it is this code that I just read? Unfortantely I can't post it, but it's a TheDailyWTF candidate for sure. The authors carefuly considered each and every best practice in .Net, and gave them a very careful, very... interesting twist. For instance, on the subject of business objects vs. datasets, the code implements ORM on top of DataSets, Argh! And that isn't the least of it!
So, I'm not going to write any more code today in protest for WTF code everywhere. I actually had a sort of morbid enjoyment watching this code work. I counted a WPM (WTF Per Minute) of ~15, which I believe to be a world record.
Okay, I'm getting curious about stuff, so I keep the SQL profiler open, and I just saw a single page issue 91(!) SQL statements. Granted, this is a complex page, but I think that I can do it manually with maybe 4 SQL statements (and ~5,000 Line of code to sort it out later).
There are some parts of the code that I really don't know what to do. On the one hand, lazy loading some of the information make sense most of the time, but that may bring me to the horrendous Select N+1 problem.
Right now I'm deferring the decision, I added some lazy attributes, and that reduced the problem somewhat to 72(?!) statements. Still way too high, but this require some thought before acting. I'll let it lie in peace now, while I consider what to do. I also made a huge mistake by not running the test after I changed each relationship to lazy. I got so many errors when I finally did run them that it took me way too long to fix them.
One other thing I discovered, NHibernate assumes that accessing the properties is always safe, and it breaks it pretty badly if you start to mess with uninitialized collections during the initialization process. I reverted back to the old behavior (yes, I'm inconsistent, I know.)
Okay, I just run into some head scratching bugs when using NHibernate.Generics. The issue was the well documented edge case, which turn to be a bitch when testing. I might have been able to fix this, but the whole thing pissed me too much when I noticed that it would work in the debugger, but not when running outside of it.
This is a crime that I won't tolerate. If I debug a piece of code, it should bloody well work, and not behave differently. I'll remove it from the code tonight, and hopefully I'll also remember to update the documentation for it. This would make NHibernate.Generics behave identically to the usual NHibernate' collections, which is how it should be.
So I was listening to DotNetRocks #115 (with Jackie Goldstein), and part of what they were talking about was the migration wizard that was suppose to migrate VB6 to VB.Net code. One of the thing that they brought up in the talk was that people expected that the wizard will take the code and turn it into a best-practice-using code. One of the examples that they gave was turning VB6 IO to .Net Streams IO (which the migration wizard doesn't do).
I couldn't help thinking about the poor developer who would run this hypothetical wizard and suddenly all his code is transformed from the familiar VB6 code to incomprehensible VB.Net code. I can just feel the anguish of that developer, trying to get the entire framework at once, stumbling along in a suddenly unfamiliar code base.
That is if the wizard was 100% correct, all the time. Occasionally I decide to break away from the base practice in a certain area, usually because I understand what I'm doing, and I know what needs to be done. I would hate to see what a wizard would do to this code.
This time I actually did nothing for this release. Jeff Brown added support for Out & Ref parameters for Dynamic Proxy, and all I had left to do is hit the build button a couple of times and upload the stuff to the server.
This closes the next to last big problem with Rhino Mocks. The very last one has to do with generic methods, which I hope to be able to fix this week.
I'm getting another 1Gb of RAM tomorrow. My main machine has 1Gb of RAM, and I constantly has to monitor the memory usage because I use a lot of applications simultaneity. Right now I'm using 967Mb:
- Outlook is the usual culprit, with 91Mb.
- RssBandit add another 68Mb to the pressure.
- NAnt is currently compiling Castle, so it's happily eating 110Mb.
- WBEditor (which I use to write this post) is taking a hefty 59Mb to allow me to do that
- Firefox is no mincemeat with 55Mb taken.
- Winword is active because I'm writing a mail message, add another 48Mb to the cake.
- SQLServer is a modesy 46Mb
- Explorer is another 30Mb
I'm reporting only the current mem usage, there is also the peak mem usage to consider, since I don't see it being returned the the free memory pool until the process is closed. And I've not even talked about the amount of memory Visual Studio feels free to grab. While I wrote this post the memory usage went up to 1037Mb, the peak memory usage for this machine is 1.5Gb.
I'm pretty pissed that I need to do that, even though the last upgrade I made on this computer was over a year ago, I expect 1Gb to last, damn it. The one thing that I can't stand is when I'm thinking faster than the computer can follow. The problem is that the moment that this starts, I can feel the thoughts going away, and I know that I'm going to lose them while I wait the computer to finish whatever it is that it is doing.
Swapping is a great idea, but it make the computer totally unusable for the user seating in fornt of it. It breaks every assumtion that you have with regard to cheap/costly operations. To everyone who says that memory is cheap, consider this: I'm probably going to break Outlook 2003, since it's not capable of handling more than 1Gb of RAM (but it has a hotfix).
What is the most useless, annoying, brainless thing that an application can do? It is to shout: "Error occured". Tell me you haven't ever sat in front of the computer, staring at a completely helpless error, knowing that the program had the information that you need, but it just won't tell you.
I just revised an exception message four times, and on each time, I added more and more information to it.
Here are the iterations:
- Property must be virtual on classes with lazy loading enabled - This one explain the problem, but doesn't really say what happened to caused it.
- Property must be virtual on classes with lazy loading enabled [ActiveRecord(Lazy=true)] - This one gives a little more info, now even someone without intimate knowledge of the way ActiveRecord works can just search for it.
- Property DemoProp must be virtual because the class is lazy loaded [ActiveRecord(Lazy=true)] - Better, now I've information that says where the error happened. I still don't know in which class this happened, and if this is a common name, I'm still in a problem.
- Property DemoProp must be virtual because the class DemoClass is loaded loaded [ActiveRecord(Lazy=true)] - Much better, now I know the name of the property and the name of the class, I can easily find it, and fix it.
Any horror stories about silent exceptions? Any way to improve the exception?
BTW, there is another side to the coin, the verbos exception, but that isn't nearly that much of a problem. You can reduce information, but you can create it. On the other hand, the misleading exception is the worst beast of them all.
It has been a week since I released any software, and I begin to itch :-)
Okay, this one is a bug fix release for NHibernate.Generics, it's meant to fix an error that I run into where NHibernate.Generics was modifying a collection when NHibernate was iterating over it. The download is here.
Okay, the bug I was tracking? The one that I needed to work so hard to track? It was in my own code (not surprising, I found NHibernate to be rock solid.) It has to do with NHibernate.Generics and Cascades. I'm pretty sure that it doesn't interest you, but there will be a fix in the site later tonight.
I really hate it when my own code make me work so hard.
The issue is not that the code isn't good or that it's complex or that I don't understand it. The issue is that I have to manually move all the references to the complied assembly to the project, and when I'm done, I need to reset all of them back. I'm not using NHibernate alone, but several other libraries that depend on it (ActiveRecord and NHibernate.Generics), which I also need to add to the project, and I need to change their references, etc.
I can't even think about keeping them there permanently, I have to track a bug into NHibernate about once a month or so, and the build times are going up when I add all those additional projects. I thought about writing a macro to do this for me, but I don't know enough (or care enough) about VS internals to do it.
Okay, I'm currently uninstalling the 210 build of ReSharper. And for the first time, it's not because it is unusable. The memory it consume is huge, and I really can't afford to dedicate 300 – 400 MB of RAM just to the IDE. Performance improvement should start next week, so I expect to see a severe drop in the memory it needs.
In the meantime, I'm hitting the download page every day or so, just to see if a newer build is out.
The last blog entry was written in Outlook 12, and I took advantage of some of the nicest stuff it has to offer, easy visual selection. I was wondering how it would transfer this to the web. Now I know.
It simply took the text box with the code and generated an image for it. Nice for making sure that things looks the same, I guess. Not so nice when you're sending to a blog. I'm going to keep it as a memento.
Unfortunately, I just run into this "issue", check this out. In the context of an ASP.Net page, put this code;
This code run for a couple of minutes before I noticed something wrong. Suddenly my machine was very slow an unresponsive. At first I suspected the devenv.exe process, which is getting pretty heavy lately (mainly because of ReSharper, to be honest.) When I managed to load the TaskManager, I found the aspnet wp.exe process was consuming over 700Mb of memory, and was swapping like mad. It had over a million page faults, and was advancing fast to oblivion. The machine has 1Gb memory, and it used 1.8Gb, just to give you an idea about the pain this machine had suffered.
This tiny piece of code managed to:
Eat 100% CPU.
Push all other memories from memory.
Push itself out of memory, so each loop iteration caused several page faults.
Clear the pre fetched cache, so any new applications that I load will load much slower.
Pretty impressive, all in all. And all I forgot is a single line of code that would advance the current time.
Some random facts that Google has to say about the readers of my site:
- 97% windows users, but I also have Linux, Mac & Unix readers.
- 82% of the windows users has Windows XP, to the 5 of your that are still using Windows 98 & ME, switch. Now!
- 56% of you are using IE, but 37% are using Firefox, and ~5% are using Opera. Miniscule amounts are using the other browsers (Safari, Mozilla, Camino, etc).
- I get about 200 unique visitors a day, and it seem to be slowly increasing.
- I get readers from all over the world, mostly from the US (33%), but also from Germany, UK, Canada, Israel, China, etc.
Apparently I'd a glaring typo in my blog title.
I can usually get away with that, but this time the typo was in my own name*.
Big thanks for Gunnlaugur for finding it out and letting me know.
*well, nickname, you know the story.