http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660Ayende, indeed your blog has generated quite a bit of response. Let me address scrum as that is what we employ in our projects. Scrum is neither a development nor an engineering process. It is a collaborative project management / cycle process. So you have to couple it with engineering techniques and principles. The idea of scrum is that you can use pretty much any software engineering technique or discipline you choose in a sprint development effort. You can use TDD, BDD, Exploratory Testing, etc. or even traditional development coding techniques if you choose to. Fact is, any management process you use in software development will need to be coupled with the software engineering techniques. What is important is that these should be compatible so that neither diminishes the effectiveness of the other. If IT management does not understand this then yes they missed the point of being agile and the idea of agile development. But even worst is if us developers don't say anything about it. At OutSystems ( [www.outsystems.com](http://www.outsystems.com)) we use scrum and use the OutSystems Agile Platform to augment development techniques like TDD and BDD. We've solve many technical issues agile teams face with builds, CI, collaborative development, etc. by having a development platform that helps us manage these. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment28http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment28Tue, 02 Mar 2010 15:30:55 GMTGood post, good comments! http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment27http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment27Mon, 22 Feb 2010 19:19:57 GMTI think you are correct in saying that in general the frameworks of the Agile movement do not specifically focus on "Sound Engineering" practices. Things like the agile manifesto and Scrum focus in on People and Process - As in you have to have a process that everyone understands and you have to have all the players involved at various points. What should be noted along the way though is - scrum in particular - tries to allow people to come to the conclusion that they can not be faster by continuing to ignore fundamental things about how they develop software ... this sort of thing should come out of the retrospectives of each and every sprint in scrum. The typical engineering gripe is "I don't have time to do it right", I incur technical debt and am never allowed to go back and fix it. Scrum provides mechanisms to attempt to make these types of short falls 'obvious' and once obvious provide a way to get them accomplished. Scrum - contrary to belief does not directly make things better or faster - it makes the bad things more obvious and attempts to slow things down so that there is TIME to do them right. Definition of DONE in scrum for an organization ... should include unit testing, functional testing etc. etc. Once all the 'right things' are in place and the organization is CRAWLING along, then whats obvious is that non of the good stuff is automated/fast. Continuous and unrelenting improvement is at the core of Scrum and a fair number of the agile processes... so while not specifically called out in the 'processes' the thoughts are their for those that care to see them for what they are. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment26http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment26Mon, 22 Feb 2010 16:49:30 GMTNice article. I think you have hit on something that all experienced (and good at their job) developers have issues with, but is never spoken about in process design. Very often the job of building a team is taken out of the hands of the people who know the difference between a programmer and a good programmer. Failure to build a team with good people is the #1 reason for project failures in my experience. Companies will often build a large team of cheap programmers because it looks like they will save money, when in fact it will end up costing more than a smaller team of good (and expensive) programmers would. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment25http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment25Mon, 22 Feb 2010 14:42:11 GMTSoftware processes aren't purposed to directing engineering techniques. That diagram of a Lean virtuous circle presumes that engineering functions are well in-hand. It's got no business dictating engineering bits. It's out of scope. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment24http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment24Mon, 22 Feb 2010 04:05:54 GMTYou mentioned process frameworks, not processes. No wonder, they have some gaps to fill while being implemented. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment23http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment23Sun, 21 Feb 2010 20:18:16 GMTYou are NOT describing SCRUM, you are describing a broken software process. SCRUM is not about the software development cycle, but about the project cycle. I see a lot of people agreeing with you, unfortunately they have all misunderstood SCRUM, SCRUM is not about someone who tells others what to do. SCRUM takes all developers into the process, before each sprint the team (max.9 persons) meats with the product owner/ project manager and together they discuss what the goals should be in this sprint (spring backlog). No one pushes anyone to late night work etc., SCRUM is about working together and not having a project manager/ software architect (from Allan Kelly's blog) that doesn't understand how to built a good product. If the team doesn't succeed in finalizing/ reports the product backlog "done" when the sprint is due, the product shouldn't be declared done! and the team should learn from their mistakes and creating smaller backlogs. It is NOT the product owner that reports the product "DONE", nor the SCRUM-master! It's only the team, the whole team and nothing but the team so help me Hirotaka *LOL* As Frans Bouma wisely mentioned: "So in short: no matter what the process is, if you have sucky programmers you get sucky software. " http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment22http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment22Sun, 21 Feb 2010 15:46:15 GMTHire a good software process engineer and you won't have that problem with your processes. What you're showing is a development life cycle, not a software process. My software process descriptions are a lot more complete than those shown here, and include requirement reviews, design and code reviews, walkthroughs, developing test procedures (I require whoever develops the code module to write a testing procedure for that module), initial testing (by the developer according to their procedure), software build process, content management systems, obtaining customer feedback, etc, etc... You can save yourself a lot of time and trouble by hiring a good software process engineer. And having management buy-in to your processes. This is why good companies develop ISO procedures -- and follow them. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment21http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment21Sun, 21 Feb 2010 03:26:57 GMTVery well written. I agree you must have both project management and engineering management in place to be successful. Engineering management goes beyond writing good quality code. An organization must have in place engineering best practices to allow long term maintenance and growth of the project. Small projects / companies may succeed without these. At a certain size the weight of not having good engineering or project management in place becomes an issue. You will spend less time delivering features to your customers and more time in the viscous cycle that is your process. Organizations need project management processes to prevent waste, provide quality checks, and prioritize resources. Organizations need engineering management for the same reasons. The easy one to grasp for organizations is the project management best practices. These come more naturally as they apply to the whole organization and generally you don't get very far with out them. Mature organizations grasp the need for engineering best practices. These are a bit more difficult, and are often over-looked because, unfortunately, you can have some initial success without them. Eventually you pay the price and the organization matures....or it stagnates and eventually ceases to exist. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment20http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment20Sat, 20 Feb 2010 22:00:43 GMTSomething to keep in mind... Some processes are better suited than others for allowing time for team members to improve their engineering practices. In more prescriptive environments, where the team spends significant time in project management related work, by definition, you will have less time available for engineering practice education. If you want developers to have the opportunity to improve their dev practices, then use a process that frees them to do that. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment19http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment19Sat, 20 Feb 2010 21:52:34 GMTIMHO, eXtreme Programming address both management and engineering problems. Why didn't you mention this? http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment18http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment18Sat, 20 Feb 2010 19:54:50 GMTOh my god - you just described my company in one go (hence anonymous posting). Our trigger for this is as follows - look for these signs... 1/ massive management power struggle between useless management staff 2/ a lot of hacks piled on top of a clean architecture because the clients don't want a rewrite to move even a single pixel 3/ most of the specs and planning done via powerpoint and THEN handed to the tech team as fully complete with a delivery date. They call it a process, I call it shit. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment17http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment17Sat, 20 Feb 2010 19:21:49 GMTWhere do we learn how to "build software" now? When I went to college (10 years ago),my first job was a jump from having learned a programming language at University to actually try to build something. You could say I learned against the "wall". My point is: If processes ignore the actual building of software, I hope at least Universities don't so that new developers don't find the same gap programmers of my age did. Nice post! http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment16http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment16Sat, 20 Feb 2010 17:25:13 GMTHmm that diagram looks at lot like "Lean" to me. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment15http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment15Sat, 20 Feb 2010 17:23:39 GMTI'll talk about Scrum which is what I know best. It is not intended to be a full software process in itself, but only for the project management part of it, as you said. Does it mean it is wrong? No, it just means it is not meant to solve all your problems. For the engineering part, XP (which you didn't mention at all) is a very well suited partner, and often recommended by Scrum coaches. The fact that some people ignore that part when adopting it, doesn't make Scrum bad. It is a tool, use it for what it is good, but don't intend to build a house with just that one tool. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment14http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment14Sat, 20 Feb 2010 15:46:55 GMTI'm one of those "managers". I completely agree that management must make an effort to understand the art of building good software. I need to understand what the teams are talking about. To me one of the pitfalls is the refactoring part. Although we always discuss it at every planning session we often find that we did not manage to reafactor enough to keep technical dept low. I try to compensate for this by asking "Any crap we need to clear before we continue?", and by adding blocks of "unknown refactoring" to the product backlog. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment13http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment13Sat, 20 Feb 2010 13:09:33 GMTPart of the problem is that it's tempting to think of Agile as a way to replace individual engineering skill and virtues by the structure of a process. Good engineers are hard to find, expensive, and "do not scale well". It's hard to build a predictably successful development organization when you rely on such soft factors. Agile methods include at least some engineering techniques such as TDD, builds, and repeated contact to the real world of the product owner. But it's just like a car loaded with all kinds of electronic gizmos to keep it on track: the perceived security lures people to faster driving. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment12http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment12Sat, 20 Feb 2010 12:50:33 GMTWell put! Bad quality will get you into more trouble by going faster. That's why you need to start by focus on the quality of your code. Totally agree. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment11http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment11Sat, 20 Feb 2010 12:27:31 GMTThis method includes a lot of focus on "construction" along with guidelines about how much of what to do in earlier vs. later phases. (E.g. early on get something working end to end.) It can be using in an agile fashion if you so desire, it's pretty flexible that way. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment10http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment10Sat, 20 Feb 2010 12:24:20 GMT@Steven Willems: to quote Dijkstra: "The required techniques of effective reasoning are pretty formal, but as long as programming is done by people that don't master them, the software crisis will remain with us and will be considered an incurable disease. And you know what incurable diseases do: they invite the quacks and charlatans in, who in this case take the form of Software Engineering gurus. " (source: [www.cs.utexas.edu/.../EWD1305.html](http://www.cs.utexas.edu/users/EWD/transcriptions/EWD13xx/EWD1305.html) ) Unless we as a profession actively put effort in learn what computer science has produced as knowledge and wisdom in the past 20+ years, this won't change. Who used a well known algorithm over a home-grown one lately? http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment9http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment9Sat, 20 Feb 2010 10:49:33 GMTVery well written Ayende. And I agree the lack of understanding is a problem, but we shouldn't look too much to the 'managers' alone. It is also our job to convince them and our colleagues of the need of good software engineering. We have been changing our way of working & technologies too much in the past. So it's not that strange we lost our credibility. Software Development has been too long driven by amateurism, trial and error. I think it is one of the phew industrial sectors where the 'science' part is ignored or not existing (at least, I don't know of it and most software developers don't). Most of the time we did what we thought was good, or felt good or what we've had read from some guru. For a while now, we have come to the point where we have learned from our mistakes by trial and error and (I hope) now we know what to do. At least, that's how I see it.. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment8http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment8Sat, 20 Feb 2010 09:38:10 GMTSo in short: no matter what the process is, if you have sucky programmers you get sucky software. Gee, who would have thought! ;) There's nothing wrong with adapting a methodology, any methodology, but you have to stick with it and that means the people in the team have to fit the methodology, otherwise they start working they way THEY think the project has to be done, which means the team doesn't use the methodology well. A methodology is never a way to make things work, it has to match the people you have. This is also true for tools used: a bunch of table/sql oriented people won't work well with o/r mappers for example, no matter how hard you try. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment7http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment7Sat, 20 Feb 2010 09:36:50 GMTA very good post. I'd say that the real issue here comes from lack of understanding. Management do not understand software engineering and the focus has always been on how the engineers should understand the business. Very seldom we talk about the opposite, that the business need to understand engineering. This leads to processes that fit business requirements but that the job we do is different from the job they do. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment6http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment6Sat, 20 Feb 2010 09:14:25 GMTYou've hit the nail on the head with this blog post. This is *exactly* the sort of thing we've experienced on projects where the need for velocity has out-weighed programmers concerns. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment5http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment5Sat, 20 Feb 2010 09:05:45 GMTThe only process that I can talk about with any authority is Scrum - and anyone who knows their stuff when it comes to Scrum will tell you that Scrum is a 'product' development methodology and not a 'software' development methodology. Scrum specifically distances itself from the engineering practices and a knowledgeable Scrum practitioner will hammer home the importance of adopting sound engineering practices to protect from technical debt. Many Scrum teams adopt XP like engineering practices to complement the Scrum process. Scrum has two key components that should work together to iteratively drive the team towards appropriate engineering practices - the definition of 'Done' and the Sprint Retrospective. There is no excuse for a well run Scrum team to have problems with their engineering practices. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment4http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment4Sat, 20 Feb 2010 07:08:49 GMTExcellent post. Dealing with a prime example of Conway's Law manifested right now, actually - the organization has been re-org'd into silos - UI, Platform/Services, QA, etc... and then silos per features within those groups. And guess what? Utter crap came out the ass-end of our "process" - death march to the end, but we're agile, right? All those modules the UI guys developed in isolation from one another - yeah, they're all calling the same backend services repeatedly to get the same data, etc... to the extent that the new version of the product now has 1/5 the capacity of the previous version. Rockin' and rollin', lemme tell ya. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment3http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment3Sat, 20 Feb 2010 06:20:58 GMTThat door definitely swings both ways. Contracting for a number of organizations I've seen the effects of project management dominance, where an estimate of 6 weeks for two developers (roughly 3 calendar weeks) gets handed to the client as a quote for nearly 6 months. This is thanks to "processes" and lots of wonderful template-based project management documentation. 3 weeks, 2 guys, somehow the project plan ended up with 5 development iterations. In that case, the PM documentation was ignored and we delivered in 5 weeks. (3 weeks development, 2 weeks acceptance testing which resulted in less than 2 days rework.) On the other hand is virtually no project management. Often this the where projects are over-engineered, or effort is spent on features that definitely don't matter to the end user. Software development can be a bit frustrating because those that have gone through the pain points seek to educate others to their concept of the "right" and "efficient" way to do things. but the message often comes out like "Do what I say, not what I did." The process for actually building the software becomes having some "technical architect" plan things out with minions to implement it. Then when it still goes pear-shaped, they appease themselves with thoughts of the mythical re-write. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment2http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment2Sat, 20 Feb 2010 02:36:29 GMTA lot of teams that adopt scrum/agile/whatever no doubt do it because they -- or their ... management -- see it as a panacea. Adopt the most hyped development process around and quality software will follow, right? Er, no. I totally agree that If you don't have the people or the engineering practices in place, scrum is just a band-aid. It will replace the ugly truth with thin veneer of order. Strangely enough, some management folks seem to prefer the 'truth-lite' version of proceedings. A few years ago, we were absolutely screwed. Now we're a lot less screwed, and it's nothing to do with adopting scrum, because we were doing scrum when everything was at its worst, too. All of the positive changes have been brought about by a change in personnel and promoting practices that, on the whole, are beneficial. Like having better quality code with fewer hacks, not abusing language features, creating more thorough tests (or... some tests at all), a non-crap CI with nicely written build scripts etc. http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment1http://ayende.com/4403/nice-process-but-what-about-the-engineering-bits#comment1Sat, 20 Feb 2010 02:29:18 GMT