There is a point where a project goes beyond the pale, where the complexity goes so far out of line that it is simply ludicrous.
I had such a point today. I had enough with SSIS and decided that I want to replace it with something better. I wrote an ETL tool to handle that in a few hours.
Why is this relevant? Because I have already build an ETL tool. Rhino ETL.
It is quite telling when the author of a tool decide that he doesn't want to use it.
I was decidedly proud of Rhino ETL for a while, then the problems started to creep in. The problems were not in the code per se, the entire architecture of the code was overly complex. In order to handle this complexity, I had resorted to throwing code at the problem, and then more code, and more code yet again.
At the moment, the current code base has two "minor" problems, exception handling and threading. The bigger problem is that I don't want to have to wade into this stinking pile and try to figure out what is going on there. I tried to be clever, and it is clever, in a horrible sort of way.
I don't have the time or patience to decipher code at the best of time, and at this point, it has gotten simply too complex. The project right now is at ~9,000 lines of code, so it is not that it is big, it is simply complex.
From the architecture perspective, I have made one huge mistake, I exposed the threading model to the application code. You can say that this stand in the root of my problems. I actually re-wrote this once already, moving from a home grown threading solution to using Retlang for threading. I made the same mistake and exposed the threading model to the application itself. Can you say: Big mistake!
From the point of view of the project itself, I started by defining the DSL syntax, and then built the project around that. It turns out that this has the usual "let us build the whole layer at a time". It also meant that a lot of the code had deep assumptions about the way it is called, making it unusable for using in other ways. This is excusable if we are talking about the DSL mapping layer, but not for the core code base itself.
Anyway, I am ranting and I should stop.
I spend six to eight hours today rewriting it from scratch. It doesn't do threading, and it doesn't have a DSL interface yet, but it does pretty much everything that the old project did, in a quarter of the lines of code, and in a way that is much safer and easier to handle than what we are using currently.