Trusting the benchmark

time to read 2 min | 224 words

imageI was scanning this article when I read noticed this bit.  A benchmark showing superior performance to another dynamic proxy implementation.  

I should mention in advance that I am impressed. I have built one from scratch, and am an active member of DP2 (and DP1, when it was relevant). That is not something that you approach easily. It is hard, damn hard. And the edge cases will kill you there.

But there are two problems that I have with this benchmark. They are mostly universal for benchmarks, actually.

First, it tests unrealistic scenario, you never use a proxy generation framework with the caching off. Well, not unless you want out of memory exceptions.

Second, it tests the wrong thing. DP2 focus is not on generation efficiency (it is important, but not the main focus), the main focus are correctness (DP2 produces verifiable code) and runtime performance. You only generate a proxy type once, this means that if you can do more in the generation phase to make it run faster at runtime, you do it.

Again, it is entirely possible that the LinFu proxies are faster at runtime as well, but I have no idea, and I don't mean to test those. Benchmarks are meaningless most of the time, you need to test for performance yourself.