On Friday, I had an encounter with the sewers at my house. On Call team has came and fixed the issue, which was a plugged pipe somewhere down the street very fast, so no harm was done (although checking the sewers just after taking a shower is not my idea of fun).
That did, however, let me learn a bit about the way sewers are constructed.
Broadly, the sewer is based on the idea of pipes and pools ( at least, those are the Hebrew terms, translated ). The idea is that the sewer flows in the pipes, toward where ever it goes. But if we have the pipes, why do we need the pools as well?
The pools are there for capacity control. They help to ensure that even if there is increase demand for the sewer, it won't back fire on you (literally, yuck!) when used.
If the downstream pipe cannot accept the sewer that we have locally, it is gathered in the pool, and will trickle downstream based on whatever "bandwidth" it will get.
Usually, you have several such pools, each of which can take quite a bit of sewer water, so you tend not to have issues with the sewer unless it is a real problem. Reduced capacity for the sewer is not noticed, because the pools absorbe the spike in usage and moderate it.
In my house, we had the following issue:
Three pools down from the actual drain (about 100 meters from my house, actually), we finally found the culprit, a semi plugged pipe that wasn't draining properly. The interesting thing here is that it is likely that it wasn't functioning correctly for a long time (weeks, at least).
Now, last I checked, this blog was mainly technical stuff, and while I am sure that there are Sewer Geek somewhere, I am not part of that particular subculture. Why am I telling you that?
Because there are a lot of parallels with software. Specifically, with building software that can withstand the harsh environment called production. The idea of pools for the sewers closely match the idea of queued messaging for communication, for example.
And I have seen application survive any number of transient failures as a result of this type of architecture.
So, this was a trip in the sewers, hope you held your noses.