+972 52-548-6969
Nice process, but what about the engineering bits?
Software processes has always been a popular topic of discussion in our industry. Those can get quite heated, with advocates of the “stable / stale” Waterfall method pointing fingers toward “rapid / rabid” Agile methods, with the CMMI people throwing documents around and Lean people standing on the sidelines muttering about Waste.
This isn’t a post about a specific software process, I’ll defer that to another day. Instead, I want to focus on a flaw in the basic building blocks in many* software building processes.
They ignore the actual building the software.
That may sound ridiculous on the face of it, after all, how can a software process ignore the act of building software. But take a look at the following diagrams:
If you’ll pay attention, you’ll notice that those processes talk about everything except how to actually build software. They talk about people, about requirements, about managing customers, about a whole lot of things, but not about the part where you have people sitting down and writing code. In most of those, in fact, that part is usually defined as one of those:
Why is that a problem? After all, isn’t there a big distinction between software engineering (we know what to do, now let us do it) and project management (getting to know what we need to do, and verifying that we did it right). Those processes deal primarily with project management and leave the engineering part to be defined in a way that fit that particular project. Surely that is better, right? In theory, it might be. But there is a big problem when you have a software process that ignore the software engineering aspects of building software.
The problem is that that in many cases, there are hidden assumptions that are going to hammer you down the road if you use a certain process with engineering practices that doesn’t fit it. Take a look at the following chart, showing a team velocity over time, does this look familiar?
The term I heard used for this is Scrum Wall, but I have seen similar results in other processes as well. The best description for that is Allan Kelly’s:
You hit the Scrum wall when you adopt Scrum and everything goes well, then, after a few Sprints things don’t work any more - to use an English expression, they go pear shaped. You can’t keep your commitments, you can’t release software, your customers get annoyed and angry, it looks like Scrum is broken.
This is what happens when you adopt Scrum without technical practices such as Test Driven Development, continuous integration and refectoring. When teams adopt the Scrum process, they go faster, show progress, things look good... and then the quality becomes a problem. Now the team are fighting through quick sand.
The code quality is poor and developers are expected to continue to make progress. Maybe the Scrum Master/Project Manager reverts to past behavior and demands overtime and weekend working. Maybe the team start busting a gut to keep their commitments. Either way the team is heading for burn-out.
The major issue is in focusing so much effort and time on project management with what amounts to willful ignorance of the technical and engineering practices will inevitably leads to disaster. The process of building software is intractably linked to the engineering practices involved in building the software. Moreover, some technical practices are actively harmful in some scenarios and life savers in others.
Many Agile and post-Agile processes focus on short cycles, each of them producing something with a distinct value to the customer. That may be an iteration, a commit or a feature, where the goal is to increase the velocity over time so we can provide as much value to the customer in as short a time as possible. What those processes ignore are things like technical debt, large scale refactoring and non functional efforts. Oh, you see those things mentioned on the edge, but they aren’t something that is dealt with heads on, as a core issue to consider.
There is a bit more to that, actually. The software engineering practices and the project management strategies are linked and of paramount importance when the time comes to decide how the software should actually be built. No, this is not tautology. We just need to take into account Conway’s law and expand on it a bit.
Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure.
Part of the design process of a project should include design the team(s) structure, the project management strategy and the software engineering practices in order to align the end result with what is desired. Ignoring this leads to imbalance in the project, and if that imbalance is big enough, and goes on for long enough, the project is going to rip itself apart.
* Nitpicker corner: I said many, not all. Don’t bother to list me software process that deals with it. I had a reason to explicitly list the processes that I did.
Comments
A lot of teams that adopt scrum/agile/whatever no doubt do it because they -- or their ... management -- see it as a panacea. Adopt the most hyped development process around and quality software will follow, right? Er, no.
I totally agree that If you don't have the people or the engineering practices in place, scrum is just a band-aid. It will replace the ugly truth with thin veneer of order. Strangely enough, some management folks seem to prefer the 'truth-lite' version of proceedings.
A few years ago, we were absolutely screwed. Now we're a lot less screwed, and it's nothing to do with adopting scrum, because we were doing scrum when everything was at its worst, too. All of the positive changes have been brought about by a change in personnel and promoting practices that, on the whole, are beneficial. Like having better quality code with fewer hacks, not abusing language features, creating more thorough tests (or... some tests at all), a non-crap CI with nicely written build scripts etc.
That door definitely swings both ways. Contracting for a number of organizations I've seen the effects of project management dominance, where an estimate of 6 weeks for two developers (roughly 3 calendar weeks) gets handed to the client as a quote for nearly 6 months. This is thanks to "processes" and lots of wonderful template-based project management documentation. 3 weeks, 2 guys, somehow the project plan ended up with 5 development iterations. In that case, the PM documentation was ignored and we delivered in 5 weeks. (3 weeks development, 2 weeks acceptance testing which resulted in less than 2 days rework.)
On the other hand is virtually no project management. Often this the where projects are over-engineered, or effort is spent on features that definitely don't matter to the end user.
Software development can be a bit frustrating because those that have gone through the pain points seek to educate others to their concept of the "right" and "efficient" way to do things. but the message often comes out like "Do what I say, not what I did." The process for actually building the software becomes having some "technical architect" plan things out with minions to implement it. Then when it still goes pear-shaped, they appease themselves with thoughts of the mythical re-write.
Excellent post. Dealing with a prime example of Conway's Law manifested right now, actually - the organization has been re-org'd into silos - UI, Platform/Services, QA, etc... and then silos per features within those groups. And guess what? Utter crap came out the ass-end of our "process" - death march to the end, but we're agile, right? All those modules the UI guys developed in isolation from one another - yeah, they're all calling the same backend services repeatedly to get the same data, etc... to the extent that the new version of the product now has 1/5 the capacity of the previous version. Rockin' and rollin', lemme tell ya.
The only process that I can talk about with any authority is Scrum - and anyone who knows their stuff when it comes to Scrum will tell you that Scrum is a 'product' development methodology and not a 'software' development methodology. Scrum specifically distances itself from the engineering practices and a knowledgeable Scrum practitioner will hammer home the importance of adopting sound engineering practices to protect from technical debt. Many Scrum teams adopt XP like engineering practices to complement the Scrum process. Scrum has two key components that should work together to iteratively drive the team towards appropriate engineering practices - the definition of 'Done' and the Sprint Retrospective. There is no excuse for a well run Scrum team to have problems with their engineering practices.
You've hit the nail on the head with this blog post. This is exactly the sort of thing we've experienced on projects where the need for velocity has out-weighed programmers concerns.
A very good post. I'd say that the real issue here comes from lack of understanding. Management do not understand software engineering and the focus has always been on how the engineers should understand the business. Very seldom we talk about the opposite, that the business need to understand engineering.
This leads to processes that fit business requirements but that the job we do is different from the job they do.
So in short: no matter what the process is, if you have sucky programmers you get sucky software.
Gee, who would have thought! ;)
There's nothing wrong with adapting a methodology, any methodology, but you have to stick with it and that means the people in the team have to fit the methodology, otherwise they start working they way THEY think the project has to be done, which means the team doesn't use the methodology well.
A methodology is never a way to make things work, it has to match the people you have. This is also true for tools used: a bunch of table/sql oriented people won't work well with o/r mappers for example, no matter how hard you try.
Very well written Ayende.
And I agree the lack of understanding is a problem, but we shouldn't look too much to the 'managers' alone. It is also our job to convince them and our colleagues of the need of good software engineering. We have been changing our way of working & technologies too much in the past. So it's not that strange we lost our credibility.
Software Development has been too long driven by amateurism, trial and error. I think it is one of the phew industrial sectors where the 'science' part is ignored or not existing (at least, I don't know of it and most software developers don't). Most of the time we did what we thought was good, or felt good or what we've had read from some guru.
For a while now, we have come to the point where we have learned from our mistakes by trial and error and (I hope) now we know what to do.
At least, that's how I see it..
@Steven Willems: to quote Dijkstra:
"The required techniques of effective reasoning are pretty formal, but as long as programming is done by people that don't master them, the software crisis will remain with us and will be considered an incurable disease. And you know what incurable diseases do: they invite the quacks and charlatans in, who in this case take the form of Software Engineering gurus. "
(source: www.cs.utexas.edu/.../EWD1305.html )
Unless we as a profession actively put effort in learn what computer science has produced as knowledge and wisdom in the past 20+ years, this won't change.
Who used a well known algorithm over a home-grown one lately?
This method includes a lot of focus on "construction" along with guidelines about how much of what to do in earlier vs. later phases. (E.g. early on get something working end to end.)
It can be using in an agile fashion if you so desire, it's pretty flexible that way.
Well put!
Bad quality will get you into more trouble by going faster. That's why you need to start by focus on the quality of your code.
Totally agree.
Part of the problem is that it's tempting to think of Agile as a way to replace individual engineering skill and virtues by the structure of a process.
Good engineers are hard to find, expensive, and "do not scale well". It's hard to build a predictably successful development organization when you rely on such soft factors.
Agile methods include at least some engineering techniques such as TDD, builds, and repeated contact to the real world of the product owner. But it's just like a car loaded with all kinds of electronic gizmos to keep it on track: the perceived security lures people to faster driving.
I'm one of those "managers". I completely agree that management must make an effort to understand the art of building good software. I need to understand what the teams are talking about.
To me one of the pitfalls is the refactoring part. Although we always discuss it at every planning session we often find that we did not manage to reafactor enough to keep technical dept low.
I try to compensate for this by asking "Any crap we need to clear before we continue?", and by adding blocks of "unknown refactoring" to the product backlog.
I'll talk about Scrum which is what I know best. It is not intended to be a full software process in itself, but only for the project management part of it, as you said. Does it mean it is wrong? No, it just means it is not meant to solve all your problems. For the engineering part, XP (which you didn't mention at all) is a very well suited partner, and often recommended by Scrum coaches.
The fact that some people ignore that part when adopting it, doesn't make Scrum bad.
It is a tool, use it for what it is good, but don't intend to build a house with just that one tool.
Hmm that diagram looks at lot like "Lean" to me.
Where do we learn how to "build software" now? When I went to college (10 years ago),my first job was a jump from having learned a programming language at University to actually try to build something. You could say I learned against the "wall". My point is: If processes ignore the actual building of software, I hope at least Universities don't so that new developers don't find the same gap programmers of my age did.
Nice post!
Oh my god - you just described my company in one go (hence anonymous posting).
Our trigger for this is as follows - look for these signs...
1/ massive management power struggle between useless management staff
2/ a lot of hacks piled on top of a clean architecture because the clients don't want a rewrite to move even a single pixel
3/ most of the specs and planning done via powerpoint and THEN handed to the tech team as fully complete with a delivery date.
They call it a process, I call it shit.
IMHO, eXtreme Programming address both management and engineering problems. Why didn't you mention this?
Something to keep in mind...
Some processes are better suited than others for allowing time for team members to improve their engineering practices.
In more prescriptive environments, where the team spends significant time in project management related work, by definition, you will have less time available for engineering practice education.
If you want developers to have the opportunity to improve their dev practices, then use a process that frees them to do that.
Very well written. I agree you must have both project management and engineering management in place to be successful. Engineering management goes beyond writing good quality code. An organization must have in place engineering best practices to allow long term maintenance and growth of the project. Small projects / companies may succeed without these. At a certain size the weight of not having good engineering or project management in place becomes an issue. You will spend less time delivering features to your customers and more time in the viscous cycle that is your process. Organizations need project management processes to prevent waste, provide quality checks, and prioritize resources. Organizations need engineering management for the same reasons. The easy one to grasp for organizations is the project management best practices. These come more naturally as they apply to the whole organization and generally you don't get very far with out them. Mature organizations grasp the need for engineering best practices. These are a bit more difficult, and are often over-looked because, unfortunately, you can have some initial success without them. Eventually you pay the price and the organization matures....or it stagnates and eventually ceases to exist.
Hire a good software process engineer and you won't have that problem with your processes. What you're showing is a development life cycle, not a software process. My software process descriptions are a lot more complete than those shown here, and include requirement reviews, design and code reviews, walkthroughs, developing test procedures (I require whoever develops the code module to write a testing procedure for that module), initial testing (by the developer according to their procedure), software build process, content management systems, obtaining customer feedback, etc, etc...
You can save yourself a lot of time and trouble by hiring a good software process engineer. And having management buy-in to your processes. This is why good companies develop ISO procedures -- and follow them.
You are NOT describing SCRUM, you are describing a broken software process.
SCRUM is not about the software development cycle, but about the project cycle.
I see a lot of people agreeing with you, unfortunately they have all misunderstood SCRUM, SCRUM is not about someone who tells others what to do. SCRUM takes all developers into the process, before each sprint the team (max.9 persons) meats with the product owner/ project manager and together they discuss what the goals should be in this sprint (spring backlog). No one pushes anyone to late night work etc., SCRUM is about working together and not having a project manager/ software architect (from Allan Kelly's blog) that doesn't understand how to built a good product.
If the team doesn't succeed in finalizing/ reports the product backlog "done" when the sprint is due, the product shouldn't be declared done! and the team should learn from their mistakes and creating smaller backlogs.
It is NOT the product owner that reports the product "DONE", nor the SCRUM-master!
It's only the team, the whole team and nothing but the team so help me Hirotaka LOL
As Frans Bouma wisely mentioned:
"So in short: no matter what the process is, if you have sucky programmers you get sucky software. "
You mentioned process frameworks, not processes. No wonder, they have some gaps to fill while being implemented.
Software processes aren't purposed to directing engineering techniques. That diagram of a Lean virtuous circle presumes that engineering functions are well in-hand. It's got no business dictating engineering bits. It's out of scope.
Nice article. I think you have hit on something that all experienced (and good at their job) developers have issues with, but is never spoken about in process design.
Very often the job of building a team is taken out of the hands of the people who know the difference between a programmer and a good programmer. Failure to build a team with good people is the #1 reason for project failures in my experience.
Companies will often build a large team of cheap programmers because it looks like they will save money, when in fact it will end up costing more than a smaller team of good (and expensive) programmers would.
I think you are correct in saying that in general the frameworks of the Agile movement do not specifically focus on "Sound Engineering" practices. Things like the agile manifesto and Scrum focus in on People and Process - As in you have to have a process that everyone understands and you have to have all the players involved at various points. What should be noted along the way though is - scrum in particular - tries to allow people to come to the conclusion that they can not be faster by continuing to ignore fundamental things about how they develop software ... this sort of thing should come out of the retrospectives of each and every sprint in scrum.
The typical engineering gripe is "I don't have time to do it right", I incur technical debt and am never allowed to go back and fix it. Scrum provides mechanisms to attempt to make these types of short falls 'obvious' and once obvious provide a way to get them accomplished. Scrum - contrary to belief does not directly make things better or faster - it makes the bad things more obvious and attempts to slow things down so that there is TIME to do them right. Definition of DONE in scrum for an organization ... should include unit testing, functional testing etc. etc. Once all the 'right things' are in place and the organization is CRAWLING along, then whats obvious is that non of the good stuff is automated/fast. Continuous and unrelenting improvement is at the core of Scrum and a fair number of the agile processes... so while not specifically called out in the 'processes' the thoughts are their for those that care to see them for what they are.
Good post, good comments!
Ayende, indeed your blog has generated quite a bit of response. Let me address scrum as that is what we employ in our projects. Scrum is neither a development nor an engineering process. It is a collaborative project management / cycle process. So you have to couple it with engineering techniques and principles. The idea of scrum is that you can use pretty much any software engineering technique or discipline you choose in a sprint development effort. You can use TDD, BDD, Exploratory Testing, etc. or even traditional development coding techniques if you choose to. Fact is, any management process you use in software development will need to be coupled with the software engineering techniques. What is important is that these should be compatible so that neither diminishes the effectiveness of the other. If IT management does not understand this then yes they missed the point of being agile and the idea of agile development. But even worst is if us developers don't say anything about it.
At OutSystems ( www.outsystems.com) we use scrum and use the OutSystems Agile Platform to augment development techniques like TDD and BDD. We've solve many technical issues agile teams face with builds, CI, collaborative development, etc. by having a development platform that helps us manage these.
Comment preview