Meeting the Joel Test 2.0
I run into this post, which updates the (by now) venerable Joel Test to our modern age. I remember reading the Joel Test (as well as pretty much anything by Joel) at the beginning of my career and I’m pretty sure that it influenced the way I choose employers and designed software. Seeing this post, I decided to see how Hibernating Rhinos would rank on this test today. I put both the original and updated version, and my comments are below.
Original | Updated | |
1 | Do you use source control? | |
2 | Can you make a build in one step? | Can you build and deploy your software in one step? |
3 | Do you make daily builds? | Do you build on every commit? |
4 | Do you have a bug database? | |
5 | Do you fix bugs before writing new code? | |
6 | Do you have an up-to-date schedule? | Do you measure your progress in terms of value delivered? |
7 | Do you have a spec? | Do you have a runnable spec? |
8 | Do programmers have quiet working conditions? | Does your environment foster collaboration? |
9 | Do you use the best tools money can buy? | |
10 | Do you have testers? | Is testing everyone's responsibility? |
11 | Do new candidates write code during their interview? | |
12 | Do you do hallway usability testing? |
- Source control – Yes, thankfully, I think that the days of anyone not using source control for anything but a scratch project are behind us. Now the arguments are which source control.
- Build & deploy in one step – Yes, the build process runs on a Team City server, and while it sometimes require some TLC (I’m looking at you , nuget), it pretty much runs without us having to pay much attention to it.
- Build & verify on every commit – No. But yes. What we do is have the build server run the full suite on every Pull Request, which is our unit of integration. Commits are far less important, because we break them apart to make them easier to review.
- Bug database – Yes, but see also the next topic. We mostly use it for bugs we find, and features / improvements, not so much for customers bugs.
- Do you fix bugs before writing new code – No. But yes. The reason this is complex to answer is how you define bugs. A customer issue is typically handled from A to Z on the spot. We have a rotating function of support engineer that handle such scenarios, and they prioritize that over their routine work.
- Do you have a schedule / do you measure progress in term of value – We have a rough schedule, with guidelines about this is hard deadline and this is a nice deadline. Hard deadline is about meeting outside commitments, typically. Nice deadlines are about things we would like to do, but we won’t kill ourselves doing them. We do have a sense of what is important and what isn’t. By that I mean is that we have a criteria for “we should be chasing after this” and “this is for when we run out of things to do”.
- Do you have a (runnable) spec? - Yes, we have a spec. It isn’t runnable, and I’m not sure what a runnable spec for a database would be. The spec outline thinks like the data format and how we do data fetches for indexes, architectural considerations and rough guidelines into where we are going. It isn’t detailed to the point of being runnable, and I don’t like the idea very much.
- Developers have quite working conditions / environment encourage collaboration – The typical setup we have is a separate office for every two developers. I typically see people move around the offices and collaborate on all sort of stuff. If it bugs the other dev in the room, they usually have headphones to deal with it, but that isn’t happening enough to be a major problem. A common issue for people who leave their workstation unattended and use headphones is that by the time they get back and put the headphones, the music has been changes to something suitably amusing, such as this one.
- Best tools that money can buy – Procurement in Hibernating Rhinos is a process, it involves sending an email with “I need this tool”, and you must include the link to the tool. Then you have to wait anything between 15 minutes to 24 hours (depending on when you asked), and you’ll get the license. I have seen far too many stupid decisions of “oh, we don’t have a budget for this 200$ tool but we’ve no problem paying the 2000$ that it would cost us in time” to suffer that.
- Testers / everyone is a responsible – Yes. Every single dev is writing tests, and every single PR is sent after tests has been run locally, and then on the build server.
- Candidates write code in interview – Yes, oh yes they do.
- Hallway usability testing – See below, too complex to answer here.
RavenDB has multiple level of “user interface”. The most obvious one is the RavenDB studio, but the one that we spend the most time on is the external (and internal) APIs. For the API, we have a review process in place to make sure that we are consistent and make sense. Most of the time we are doing things that follow the same design line as before, so there is not much to think about. For big things, we typically also solicit feedback from the community, to make sure that we aren’t looking into with colored glasses.
For our actual user interface, the Studio, we used to just have the other devs look at the new functionality. But that led to a lot of stuff that worked, but the amount of attention we actually paid to the UI used to be really variable. Some features we would iterate over for multiple weeks, getting them just right (the most common operations, as we see them). But other stuff was just “we need to expose this functionality, let us do this”, which led to almost one to one mapping of the server side concept to the UI, which isn’t always helpful for the users.
We have started with a full UX study of the RavenDB Studio, and we are going to be doing full design analysis on each of our views with an eye to improve it significantly by 4.0.
Comments
Perhaps add one: Do you manage technical debt? (I consider a bug database unsuitable for that)
Sebastiaan, We use issues for that, yes.
A very important metric is "How different is your Joel Score from that of your subordinates?". A manager may think he has a spec but the team members may only see micro-tasks or generic wish lists. A manager may think he is creating a quiet environmet by forcing people to use online tools instead of just talking to each other, forcing team member to talk after hours. The concept of "the best tool money can buy" can mean different things to managers and programmers.
If the manager's score is 25% higher than the team or more, something fishy is going on
"– Yes, oh yes they do.".... hahaha Why do I get the feeling an interview here would leave one close to tears?
@Oren, would you consider making some of your tests available, just for curiosity's sake?
Quintonn, I posted a whole bunch of them over the years.
Here are a few:
https://ayende.com/blog/163394/new-interview-question
https://ayende.com/blog/170402/interview-question-fix-the-index
https://ayende.com/blog/174049/the-low-level-interview-question
https://ayende.com/blog/174305/the-high-level-interview-question
https://ayende.com/blog/3659/yet-another-good-interview-test
I feel like "having dedicated testers" and "every developer is a tester" are different things. A developer works until the program runs - the tester works until it crashes. These are opposite mindsets, and I would not expect anyone to have both mindsets to the same product at the same day.
I have two more, and one nit to pick:
Additionally, I want to re-iterate what Karsten said: Testers are not optional, even if you have full unit-test coverage. Someone who comes up with all the things that users do, but developers don't expect. Like entering phone numbers without a country code, or ordering -3 beers.
Just came in as a consultant to a company and immediately made the head of IT order me a better mouse for 25 $. He ordered it immediately without discussion or comment and I had it the next day. The other employees had their mouth open. It never occurred to them that the mouse was a tiny expense compared to their salaries and they also could pretty much have what they want if they only asked.
I'd like to see the test include more about developers being empowered to make change in their environment:
e.g. ... 3. Are developers able to deploy new services and applications both internally and to production 4. How many people does a developer have to persuade in order to trial a new tool? ...
I've always been confused about the shift in attitude toward testing, from "Do you have testers?" to "Is testing everyone's responsibility?". For example, I'm on a team building a cloud service provider for enterprise scale applications. It's a fairly large project. We have 3 developers, a solutions architect, operations specialist, and quality assurance manager (create/review completed tests). So we effectively have 3 developers.
The project must meet stringent FDA requirements for testing and quality assurance. This means hundreds of functional tests. For a stretch of about 2 months we were so bogged down with tests that we barely had time to develop features that were requested and paid for by our customers. We've been requesting full-time functional testers for some time now but management refuses and suggests we work harder. "Testing is everyone's responsibility". It's not that we aren't delivering product, but it's really demotivating for our small development team. In fact, it was so demotivating that one developer left.
Shouldn't point 10 be conditional on the size of the team, or the amount of testing required? I think "Testing is everyone's responsibility" really oversimplifies the solution. Look. We all run unit tests on our code, and rebuild after every test, but perhaps point 10 should read "Is testing everyone's responsibility and do you have at least one full-time functional tester?"... Maybe we're an edge case.
Carmack, That really depend on what kind of tests you are talking about. I would expect the people building a feature to build the unit tests to it. At the same time, depending on how valuable a feature is, I would also have additional people using those features with the intend to break them.
That can be performance benchmarks, random input, longevity, etc.
That is a separate and dedicated task, not something that you do on the fly.
Comment preview