testjutsu

Jeff Atwood recently wrote a post about phases of software development/release. He wondered where the terms come from and why we rarely use ‘gamma’ - I wondered if these terms actually have any relevance to what actually goes on in software development anymore.

To me, these terms are used to broadly group stages of software stability as simplifiers to people who don’t necessarily understand the finer details of software development (or don’t particularly want to). These labels seem to equate these days to marketing wank for shops that still do big design up front. It’s easier to say ‘we’re at alpha release’ than saying ‘we’re two-thirds feature complete with no current showstoppers, have a bug trendline that indicates a find-rate approaching zero and we’re satisfied that testing to date has adequately mitigated the risks identified so far’.

My issue with these terms is that they oversimplify a process that should be viewed as difficult. Alpha/Beta/Gamma (or whatever other pre-release signifiers you like) imply a linear, production-line progression of software. Anyone that has worked on any project of significance knows that the development of software is anything but. I worry that this oversimplification helps tie us in to a flawed mental model that invites us to fail over and over again.

There is no guarantee that half way through ‘beta testing’ your testing team won’t find a showstopper that requires you to go back to pre-alpha. How well does your ‘A->B->Ship it’ model stack up then? It seems that because we have this simplified mental model of development, decision makers think that a) we can reliably estimate up-front the time and budget required to put together complex systems and b) when, close to completion neither of these estimates prove even remotely accurate, makes them insist on kludging their way through so that the immutable ship date can be met.

The problem with this mental model is that while it probably works well for automatons that churn out identical widgets time and again, software development is not that. Software development is about human beings turning out solutions to difficult problems in a non-uniform manner. Human beings each with their own individual strengths, weaknesses, passions, neuroses and everything else that makes us human. The fact that these people are not producing identical widgets day in, day out means that the production line model is entirely inappropriate, as is the model of big design (and estimation) up front.

This is a can of worms that you could (and some have) write a book about. Entwined with development is software testing. Another very challenging craft done by humans. Testing is an oft-misunderstood profession. The alpha/beta/ship it model doesn’t help. I worry that this model locks people into thinking that testing is something that happens once the developers have finished doing what they do, at which point the testers are left to find whatever bugs they are able to in between that point and the shipping date.

Then, depending on the nature of the project and the people they’re working with, they’re possibly blamed for finding too many bugs, or flaws that, if properly fixed, would extend the length of the project well beyond the date where the PM gets his nice fat bonus for bringing this thing in on time. The PM shoots the messenger because what other choice is there at that point?

This is a stupid way to make software. This is a stupid way to think about making software.

Scrum appears to be intended to make transparent the human issues that crop up in software development and it sounds great in theory, but I wonder how many software shops are prepared to invest in the up-front discomfort of changing over and are prepared to have (on a daily basis) the open, honest and frank dialogue necessary for a framework like that to work. My guess is not many. This is perhaps a subject for another post.

If you want to be an extremely effective software tester, I highly recommend you do something else. Really. I’m not talking about being able to complete Halo3 in world-record time. Specifically I mean you should find something that you are passionate about; that takes practice and perseverance to become proficient in, and then master it.

I’m not talking about ‘I have nothing more to learn from anyone’ arrogance. Anyone who has mastered anything knows there is always more to learn. Always. What I mean is that you should do it at least until you can confidently teach someone else without fear of leading them astray.

Things like playing a musical instrument, speaking a second language, ikebana, playing golf, it doesn’t need to be remotely testing related or even related to computers. It may be better if it isn’t.

There’s something interesting that happens when you master an art. You can recognise that same mastery of something in others, even when what they do is completely different. When you understand the sort of effort required to reach a certain proficiency, you tend to appreciate it wherever else you find it. There is recognition too, of the many stages of learning one goes through on the way to mastering something. You see it in others because you yourself have been there.

How do you recognise that same thing in others? Because once you have mastered something, it reveals itself in everything that you do. Your experience cannot help but inform you about other things, and influence the way you take action.

How will that make you a better tester?

If nothing else, it will make it easier for you to play the simile game. X is like Y. It’s an effective way to learn something new - relate it to something you already know well. It can give you a different way of thinking about things. Paradoxically, any art worth mastering will have concepts that are unique to that art, that don’t directly translate into anything else. Opening your mind to new concepts can help you look at testing problems from angles you may not otherwise have considered.

Recognising various stages in learning allows you to become a better teacher. If you can recognise where someone else is along the path, you can more effectively tailor your guidance to be more meaningful. There’s no point talking about concepts they’re going to have no clue about if what they need is a simple, straightforward explanation that leads them to a deeper understanding.

So I maintain that if you really want to become a great software tester, be great at something else.

In testing, I think there is a tendency to confuse repetition for the purpose of learning with repetition as the application of skill. I have heard that some expound using scripts as a means of skilling up unskilled staff. I believe this thinking is fundamentally flawed. The reason being is that there is no clear delineation between the repetition of an action (i.e., a script) and actual software testing, which is far, far more than simply following a recipe.

In kendo, a beginner starts by learning the basic moves. A student might practise a single cut 10,000 times or more. Time and again they are made to demonstrate correct footwork, body position, striking and movement. At this point, they have no clue about the application of these techniques, only how to execute them. They can make a cut, but it doesn’t mean they can consider themselves skilled in kendo.

Once they have the basic mechanics down, they are ready to strap on armour and face a real opponent. Unlike before, where they had specific instructions about what do do, how and when, they now face a situation where the application of their techniques may or may not work and it is up to them to work out why. Moreover, their opponent is attacking them, shouting at them, trying to unsettle them, make them nervous and throw them off their game. Are you software testers out there seeing parallels?

The difference here is that the delineation in kendo between learning basic technique and application of that technique is quite explicit. In software testing, especially where scripting is involved, this is not always so. Herein lies the danger of using scripts as learning tools. A script may help you learn to execute a technique. It will not teach you to be a tester.

So I’m back from CAST. Yes, I know I haven’t finished writing up my notes from the CITCON event. I had every intention of doing so on the trip home, but 30 hours stuck in planes and transit lounges with screaming children, people with questionable personable hygiene and vapid narcissists who need to share with everyone within earshot is not conducive to cogent writing. The good news is that I’m not in prison for visiting bloody ruin upon any of the aforementioned human vermin, and hope to return you to your (ir)regular programming soon.

So, CAST rocked. Toronto was a cool place. The local people were, if not friendly then at least not aggressively disinterested (apart from a bartender at the Imperial, who was a total douche). The attendees of the conference were amazing human beings, if for no other reason than their eyes not glazing over when I was talking about testing. Some really bloody smart people there doing some very cool stuff.

I took the day-long tutorial with Jerry Weinberg on communicating as a tester. The cost of the course was probably worth it for this alone. There were keynote speeches by other big names in testing, which were also very good. More importantly though I think were the people I got to hang out with. I think it takes a particular brand of weirdness to be a good tester. There’s a certain pattern to the way they speak that gives you an insight into how they think. Testers aren’t afraid to probe, or question either so I also enjoyed the distinct lack of listening along politely.

I have more material to write up than I know what to do with. I have a busy few weeks coming up on the kendo front, so I’m not going to get to it as soon as I want to, but it’s coming. In the meantime, check out the CAST site conference archive. If you’re considering going to next year’s, it’s in Colorado Springs. I highly recommend it. I shall be doing everything in my power to make it there. If you have any sort of interest in software testing, I urge you to do the same.

So CITCON is done and dusted. I met a lot of really interesting people, some of them doing some very cool stuff. Saturday turned out to be a long-ish day and quite a fruitful one. I’m going to break down some of the stuff I picked up in the five sessions that I attended in another post, but I thought I’d post a couple of things while they’re fresh in my mind. There was quite a heavy automation focus, I suppose as you would expect at conference for continuous integration. I hadn’t spent a lot of time around people who do more automation than manual testing. Some of the opinions I heard were enlightening.

In a session on testing web apps, there was a demonstration using grails to quickly build a testing framework from which to build automated tests, as well as building a domain-specific language from the method level up to complete actions in the end-user’s natural language. One tester in the group said ‘this stuff blows my mind - are you guys testers or developers?’ - it became clear that the majority of the people in the room were developers. One of them said ‘we wouldn’t expect our testers to get this stuff. We might present them with a domain specific language to help them test it’ - and there was general assent from the other developers gathered.

I didn’t say anything at the time, but I thought it was a real shame that they felt the need to ‘dumb it down’ for the testers. There appeared to be the assumption that testers have not the skill to be able to handle something of this complexity. It made me wonder how widespread the assumption that ‘testing is a poor cousin of development’ goes. Of course, there is merit in being able to group such actions if testing those grouped actions were what you were trying to test, but there just seemed to be this implication that they deemed their testers incapable of doing anything more.

In another session, one participant wondered aloud how to get developers to do testing when testing was like doing the dishes. Developers were the cooks of the analogy and testers apparently the kitchen hands. I put my $0.02 in at this point and suggested that a) they ditch the ‘washing dishes’ analogy and b) that until the developers understood that testing was a highly cerebral task requiring a different mind set, then they probably wouldn’t be all that motivated to learn more about it.

It saddened me a little to think that there was such a wide comprehension gap between what some developers seem to think testing is, and what I understand testing to be. In my experience, good developers like to create things. They like to solve problems. They like the challenge of coming up with an elegant solution to something non-trivial.

Testing is not so dissimilar. We like to be creative too. We may have a different set of problems to create solutions for, but software testing is no less mentally taxing, or challenging. Developers tend to be thinking largely along the lines ‘how do I make this work’, whereas a tester’s mindset is more along the lines of ‘what could go wrong? What might happen that is unexpected and/or unwanted?’.

If you as a developer don’t see the merit in writing or performing tests, try asking yourself what might possibly go wrong with the code you’ve written. Ask a tester to read it - ask them what they think could go wrong with it. See what they come up with that you don’t. Oh, and if the answer you get is ‘Wow, I wouldn’t even know where to begin testing that’, then it might be time to do some refactoring

There has been an interesting thread going on in the yahoo software testing group to which I subscribe. The topic began as a discussion about software development as art, but was quickly shoehorned into a discussion about productivity. Jared Quinert blogged a response to this recently which got me thinking again about tester advocacy. I haven’t made much progress on the code of conduct front (though the principles for the practise of craft resonate strongly with me), but I think being proactive in the education of others is up there on the list, and the discussion around the meaning of productivity helps to highlight it.

For those that are too time-poor, a potted summary of Jared’s post might read ‘productivity - appropriately defined - is the key to defensible testing. Problems arise where productivity is seen as synonymous with output / effort as opposed to the gathering of helpful information’.

I agree with this, and particularly with the qualification ‘appropriately defined’. I think the danger of (mis)applying the term ‘productivity’ to testing is that the audience expects a ‘product’ at the end. Some sort of artefact that they can point to as proof that testing time was time well spent.

In my experience, the test report is often the thing by which tester productivity is measured. Not necessarily the testing that the report represents, but the report itself. Is it then not incumbent upon the tester to ensure the content of the reporting clearly displays the value of the test effort? Both the actual interaction with the code as
well as the thought and planning that goes around it.

In assisting the creation of a software product, we are generally measured by our productivity towards that end. If the people we are reporting to have a misdirected sense of what productivity means in relation to testing, I believe the responsibility is ours to correct it. How that happens really depends on who you’re reporting to and what they’re looking for.

If the thinking goes ‘a developer produces code, then what does a tester produce?’ - then the tester in my opinion is probably being set up for an unfair comparison. If I were forced to give an answer to that question it might be ‘information’. I want to make damned sure
that the information they get out of the testing I have done is valuable to them in some way. If that makes me a producer of information in their eyes, so be it.

Another way to look at it is to ask ‘If there were no testers, what else would you not have?’ If your audience can’t answer that, then I think your problems are more fundamental than their understanding of what you produce.

I hesitate to say that I’ve been sitting on this post for a while. Let’s just say I’ve been giving it some thought.

At my current place of work, because we host our product online the testers, beyond testing the code pre-release necessarily have to have to support this code once it goes live.

This leads to a situation where the more that is deployed, the greater the scope of support testing becomes. Pile on top of this the joys and complexities of feature requests and bug reports that inevitably come in along with a weekly deployment cycle and you have a recipe for what recruiters like to call a ‘fast paced work environment’.

Our testing team grew organically, but rapidly from a single person to two, three, then five. Each tester would join the team, skill up on our products, environment and processes and then gravitated to their own areas that they were particularly good with. We had the tendency to do work on those areas fairly exclusively. Having testers who were product area experts was good in that we turned requests for testing around quickly, and were able to troubleshoot problems at lightning speed.

Over time however, problems started to arise. Testers started to miss problems, sometimes very obvious ones. When one of the team was away, we occasionally struggled to do the work that was normally assigned to them. We knew the products we were working with, but new additions and changes might mean that the product was significantly different if we hadn’t looked at that area for a couple of months.

With around sixteen different product areas to support (the majority of which are still being actively developed) it was also difficult to make sure that all these areas were getting the attention they required.

I spent a lot of time thinking about how to solve these issues. Up to that point, the testing team was seen as an highly efficient unit. Of course it only takes a few small issues before people begin asking pointed questions. I was asked some pointed questions and felt some sense of urgency to overcome the situation.

When I broke down the problems into their components, the answer became obvious. Issues that I expected testers to be picking up were being missed due to fatigue through repetition. If you look at something frequently enough, there’s a tendency to skip over things.

It’s like driving a car. When you learn to drive, you are painfully aware of everything. Tyre check, clutch, gearstick, accelerator, speedometer, indicator, head checks, and so on. As you become more accustomed, changing gears goes from a precariously balanced set of bodily maneuvers to a smooth singular action. Same with changing lanes, or overtaking.

You can become blase about it though. Miss the clutch changing a gear - grind the gearbox. Miss the speedo when on the freeway, cop a speeding fine. Miss a head check when changing lanes, kill everyone in your vehicle.

Okay, maybe a little dramatic, but you get the point. Repetition and subtle change don’t go together if that subtle change needs to be noticed immediately.

Another issue was over-specialisation. While we had people who could troubleshoot and test their area of specialty, if they weren’t there, then the testing process took noticeably longer because the other testers had become unfamiliar with that part of the product.

Lastly, because people gravitated toward what they were good at doing, and because the amount that we had to support was growing steadily, there were little things that started to slip. There wasn’t a clear enough delineation of responsibilities for the testers to follow.

When I thought about how to solve the problem of over-familiarity and under-familiarity, and slippage, it seemed that a rotation system was the obvious solution. Rotate the testers through all of the product areas on a regular basis.

There were a few things in terms of logistics to get right. How long should each rotation be? How do you reach an equitable split in the workload between testers? Should each area of the rotation contain related products, or should there be a distribution of different products within each area?

We experimented with a few different options before arriving at a solution that worked well. With our rotation split into four areas, we found that having related items grouped into a single rotation worked better than splitting them up. I suspect that this is because each rotation is quite different, as opposed to having four rotations that are variations of each other.

Four weeks per rotation was too long. Two weeks was too short. We finally settled on three weeks per rotation and this has worked well for us. Of course YMMV depending on how complex the animals you’re testing are and how actively they’re being changed.

It hasn’t all been smooth sailing. There is occasionally confusion for people interacting with testers being thrown by who is in which area of the rotation, but that’s more of an information flow thing, really.

The rotation relies on the testers being able enough to work autonomously and being diligent enough to put their hand up when their workload requires assistance. When testers are absent, it is a reasonably simple matter to distribute their workload amongst the testers either side of that rotation.

It has turned out to be an excellent training tool. With such a vast array of product areas to learn, the rotation breaks it down into manageable chunks that one can get to grips with before moving on.

We review the layout of the rotations every six months or so and occasionally make adjustments where the workload of one has increased at the expense of the other. There are also occasions where projects that finish create an influx of tickets for a particular area. We keep track of the feedback/bug reporting rate around these releases to try and determine whether it looks like being a long-term increase in work load or not.

As rotation time rolls around, the testers brief their replacements on where the product is at, what changes have taken place and what the incoming tester(s) may need to be aware of. Any near-complete tasks are kept by the tester that started them, rather than bringing someone else up to speed.

Overall it has been a massive help to the team, and allows me as a manager to more easily keep tabs on the testers’ workloads. I don’t know how well a rotation would work for shorter projects, but if you’re struggling with some of the issues I have described above for long-term support projects then you might like to consider it.

I’m going to be attending CITCON coming up in Melbourne at the end of June.

There will be a bunch of people from all different tech areas to discuss CI and the testing thereof. It’s an ‘OpenSpace‘ format, so I’m looking forward to seeing what sort of things people want to discuss and hear about.

It’s a free seminar and I have it on good authority from people who attended last year’s event that it is highly worthwhile. The event is limited to 150 places and I believe many of them have already been filled, so if you’re considering attending then I’d act sooner rather than later.

I hope to see you there.

After blogging about tester advocacy I began to wonder whether I could come up with a code of conduct that testers could use to guide their actions and decisions. I don’t think you’re likely to get a one-size-fits-all code, so perhaps I should say a collection of heuristics by which a tester can guide his own actions.

I’m not talking so much about the day-to-day activities of a tester, though that certainly plays a part. I’m more concerned by the actions in his relationships with his peers, his employer and the people whom his testing may represent. I was hoping that with some careful consideration, I’d be able to put together a concise list of points.

I started listing the sort of things that are handy to remind myself about, but I quickly realised that they aren’t really that useful in all situations.

I listed things like

I will pro-actively educate myself in the art of testing.
I will find out who matters and test and report accordingly.
I will use all the resources at my disposal
I will not be the gatekeeper of the decision to ship/not ship

The problem with these is that while they work for me right now, they may not work for testers in other places, or with differing levels of experience, different duties, different responsibilities. The other problem with a list like the one above is that you could easily expand it to a hundred points or more.

I don’t want a laundry list, or a bunch of ‘thou shalt’ commandments. I am also wary of locking the art of testing into some sort of cookie-cutter formula/recipe to be followed by rote, rather something that is applicable in all situations that any tester can use to guide their thinking and actions in a positive way. This appears to me no small task.

I put it to the MAST list to see what they had to say about it. There was some interesting and spirited discussion around the term ‘trusted advisor’, but what I took away from the conversation was that Paul Szymkowiak’s suggestion the exercise of attempting to define such a code of conduct is perhaps as important (or more so) than the results.

It is still something that I am spending time thinking about, but if the journey turns out to be as important as the destination, then perhaps someone else out there might gain something if I document my endeavours here. To that effect, I shall do my best to post updates here as and when I make progress on this front.

I have been reading Matt’s blog for a while now. Since October last year, Matt has been periodically expanding and evolving his thinking about ‘technical debt’ - the small, seemingly inconsequential short-cuts and hacks that we tend to implement or let slide in order to hit deadlines or save us doing more work (at least in the short term). Like most bad habits (such as the gradual accrual of fiscal debt, say, on a credit card), these things tend to lead to large amounts of pain down the track.

Rather than regurgitate what Matt has been saying, let me point you at his work. I think it is worthy of consideration for anyone who has had to interact with another human being in a professional capacity, ever.

• Technical Debt #1
• Technical Debt #2
• Technical Debt #3
• Technical Debt #4
• Technical Debt #5
• Technical Debt #6

If you’ve skipped to this bit of text because you’re not up for six essays on an unfamiliar concept, let me recommend that you at least read #6 before checking out details on Matt’s upcoming Technical Debt Workshop.

If you’re in the vicinity and have something you can contribute, then why not do it? If you’re not, then keep an eye on Matt’s blog. I’ve no doubt there will be some very interesting outcomes from the workshop.