Ayende @ Rahien

Damon Wilder Carr commented on Full speed ahead, and damn the benchmarks

Tue, 15 Apr 2008 06:58:40 GMT

Ayende, This is one of the most common issues I run into when leading teams in the areas we both love to play in... You hit all the salient points, however I would simply add that (adding a layer of abstraction - grin) IN GENERAL if you are predicting the future, you are likely in a place that is creating waste (typically almost complete waste). The case of 'regression unit tests' that try to 'predict' performance issues to me is simply an instance of the more general 'predicting the future' problem that millions of people and BILLIONS of dollars are still wasted on. This is how I present is usually. Most people instinctively know that we cannot predict the future, then turn around and ask for a non-statistical project plan! Systems are complex, and even 'predicting the future' and saying 'we understand the problem domain' before we are finished is usually silly (even when the domain seems fairly trivial, I am always astounded at how much the experts learn). A core argument I make in the damn book I hope to finish someday is just how much software engineering is a 'wicked problem' in the formal sense as Rittel(?) around 1973 defined it. A critical path through a plan somebody dreamed up (again predicting) is not a singular thing any more then one 'theoretical future' is. Each managed task element should (at least if people really want to try to wrestle this monster) have (in my experience) a statistical confidence (to the week is about all u can do), a recording of the net impact for a 'bad occurance' where indeed say the 80% confidence has fallen in reality into the 20% fail category (2 tasks at 80% might have drastically different net 'problems' created by their failure even though they have the same duration -- work around the holidays is a common but perhaps overly simplistic example). I almost hate to say the word 'Agile' as it is so misunderstood, but Agile is meant to provide the real-time feedback for a 'living project plan' that evolves as we evolve what the hell we are doing. Anything else is a plan to fail (as I know u know). Anyway, keep up the awesome work.. I'd love to hear your take on these ideas as they have come to frame how I conceptualize and execute solutions. In fact all the design/technology decisions almost become self-evident when these truths are embraced in very compelling ways... Thanks, Damon Carr

Paul Hatcher commented on Full speed ahead, and damn the benchmarks

Mon, 14 Apr 2008 14:20:28 GMT

I think a lot of this stuff does boil down to problems with premature optmization, but there are things that don't count as premature optmization, the primary one is algorithm choice - which will be dictated by business/scale requirements. Say you pick an algorithm that happens to have O(n^2) performance - unless you know that that is the performance driver, you might spend a lot of time tweaking the edges to improve it without realizing that it's a fundamentally approach. My general approach is to work out/select algorithms that have the appropriate big O complexity, and then profile the rest.

Jeremy Gray commented on Full speed ahead, and damn the benchmarks

Sat, 12 Apr 2008 21:09:40 GMT

@Ayende - buzzword is right. You'd probably be surprised just how many people misunderstand the term, at least within the environments in which I usually find myself working. "But I can't change how it does anything! I can only make the code look a bit different!" and things like that. They seem to latch onto the size of each change, which in reality is only intended to keep things safe and make individual refactorings readily applicable across a range of situations. They latch onto how little a single refactoring does, and forget that the entire point is to progressively make small refactoring after small refactoring. They latch onto the fact that existing functionality mustn't change, forgetting that it only mustn't change when seen from specific viewpoints (ie. the UI or an externally-exposed integation point) that have to be established on a situation by situation basis. It usually takes a while to bring these people around, but even then some of them can't quite get their head around it. Luckily, most do, even if you have to pair with them and walk through a whole series of incremental refactorings resulting in an overall obvious improvement.

Ayende Rahien commented on Full speed ahead, and damn the benchmarks

Sat, 12 Apr 2008 15:23:54 GMT

Frans, My definition of refactor is changing without modifying external functionality. Changing the code is not that. That is changing the code. Refactor became somewhat of a buzzword

Frans Bouma commented on Full speed ahead, and damn the benchmarks

Sat, 12 Apr 2008 14:41:01 GMT

"Another issue is that I can refactor for performance, but (by definition), I can't refactor to be correct." That depends on what you define under 'refactor' If 'refactor' means: change working code without changing its functionality how it works, then, yes you can't refactor to correctness. However, if 'refactor' means: changing the code so it's something different, you can refactor to become correct. I think a lot of people will see 'refactoring' as the action 'change code'.

Ayende Rahien commented on Full speed ahead, and damn the benchmarks

Sat, 12 Apr 2008 14:36:14 GMT

There are hotspots in most applications that could benefit from optimization efforts. Those optimizations sometimes will lead you to remove abstractions in order to enhance performance. Those hotspots tends to be rare, in my experience. Most often, perf can be had by changing the impl of the abstraction. Replacing LockingQueue with lock free queue, replacing the impl of IGetData with CachingGetData, etc. I am not arguing that there aren't places to do so, I am arguing that it is infrequent to need this, and when you do, in a good system, you'll have the ability to go into one place and reduce the level of abstraction that you have.

Jeremy Gray commented on Full speed ahead, and damn the benchmarks

Sat, 12 Apr 2008 14:16:18 GMT

@Sasha - re: "And if you did couple everything to Queue, then how exactly are you going to refactor your code for performance when you discover that you need a lock-free queue?" - I think that without realizing it you just accidentally _made_ the case _for_ decoupling. Using your own IPlaceOrder example, the decoupling allows one to optimize for performance once any bottlenecks have been concretely measured in a nearly-complete system.

Sasha Goldshtein commented on Full speed ahead, and damn the benchmarks

Sat, 12 Apr 2008 09:33:10 GMT

Oren, thanks for resuming this interesting discussion. There is another point that has to be made. When designing code to be testable, you'd be looking for dependency injection, inversion of control, interface- and component-based approaches, and many other techniques of that ilk. Essentially, the primary obstacle to testability is coupling. On the other hand, the primary obstacle to getting good performance is decoupling! For example, assume I was writing the .NET framework and my current assignment was to implement floating-point numbers addition. If I were to design a truly decoupled approach with no internal dependencies, I would have an IAddFloatingNumbers interface which would then be implemented by the framework, but could be mocked oh so easily. But I can't afford having an interface call for floating point addition, right? Quite vice versa - this addition is an intrinsic IL instruction which is recognized by the JIT and compiled to the most efficient code possible on the platform, using SSE, SSE2 and whatnot. Which is very coupled. Which is very NOT mockable. And guess what? It's the only way to implement addition unless you're writing a toy language. And the exact same argument applies to a framework you or I are building. Right, if you're designing a PlaceOrder component, it's perfectly fine to hide it behind a mockable IPlaceOrder interface and forget about performance. But what if you're designing a messaging infrastructure for intra-application communication? It is no longer fine to hide it behind a mockable IQueue interface because that interface call is more expensive than the enqueue operation! And if you did couple everything to Queue, then how exactly are you going to refactor your code for performance when you discover that you need a lock-free queue? I think we disagree on the meaning of "refactoring" if that's refactoring.

Christian Thuv commented on Full speed ahead, and damn the benchmarks

Sat, 12 Apr 2008 08:46:05 GMT

If there are measurable non-functional requirements, you should have tests for it. For example "Creating an invoice should take less than 2 seconds". Pre-optimization would be to try to make the operation take much less time, say 0.5 seconds. In that way performance tests keep optimization just at the correct level.