testjutsu

Jonathan Kohl beat me to the punch on this blog entry by a good margin, but since I was already working on this thing, I’d like to go ahead and basically say exactly what he said in probably a far less cogent fashion.  A lot has been written about the good and bad things about Exploratory testing and the writing and (mis)usage of test cases.

I don’t see a lot written about harmonious use of both. I’m sure people do it. I know I do. Is it like airing dirty laundry? Do context driven testers not want to admit to using test cases in case James Bach reads their blog and decides to tear them a new one? Do factory testers (sorry – what do you BDUF testers call yourselves anyway? Factory tester brings to mind a grey-clad downtrodden peon, which I suppose is appropriate, but not very flattering) – anyway, do you lot not want to admit to doing exploratory testing in case your overseers beat your for not following the script? Am I missing something? Is this just not a big deal?

Johnathan advocates using a blend of both and uses the terms prescriptive and descriptive testing, which quite appeals to me. Michael Bolton’s recent highlighting of the difference between testing and checking as something we should be more conscious of helped me to frame this better in my mind.

There are things that you can plan ahead for. There are risks you can identify and tests that you can design to mitigate against them. I see no reason why you shouldn’t use test cases for these if that is what you are comfortable with.

One of the issues I have with test cases is that some people see them as a magic artefact that covers all possible problems in a given area, because one or more tests have been written that mention it.  They fail to see that many test cases are highly focused on verifying a very specific item somewhere, and just because we go through a bunch of steps to get there does not mean that those steps have also been verified.

This leads me to wonder if I have been thinking about scripts all wrong.

I recently read Michael Hunter’s work on setting up test frameworks. One of the things that was a lightbulb for me was separating the verification from the action. Very very obvious in hindsight, but if you think about it – this is basically a use case and a checklist.

We have a series of steps from point A to point B

We have a list of things that we want to make sure are as we expect them.  Why do we need to mash them together? Why do we need to repeat the list of steps over and over and over so that we can list a different verification point next to it?

An argument I hear for test cases is that if you write them well enough, then anyone can execute them. I really hate the way that thinking goes, mostly because I read it as ‘anyone can do testing, you just follow the scripts’. That being said, a clear set of instructions and a checklist could mean that you could get a non-tester to do checking, even of mission-critical information. That, I might hold with.

Checklists and Check scripts? I think this is something worth exploring further.

Part of the work I’m doing just now is helping developers to increase their testing skills to handle the bandwidth that my team can’t. I asked Yuka Horino, one of my colleagues to translate James Bach’s Heuristics of Software Testability into Japanese as one of the things to distribute to our dev group. (Thanks Horino-san!)

This sort of thing is probably useful to a wider group than just the people I work with, so here it is. Enjoy.

Heuristics of Software Testability (Japanese)

So a while back I said I’d be posting translations from the Japanese version of Lessons learned in software testing. Basically, I pulled the trigger way too early on that. There have been a bunch of things all demanding my time, so this was a side-project that I just wasn’t able to get to before now.

So the usual disclaimers apply – I have permission from the original authors of this tome, but not from the publishers. To the best of my knowledge this falls within fair use, but given the geography of this blog and myself coupled with the fact that I’m not a lawyer, I could be dead wrong. So if any of the relevant publishing bods happen across this and have a problem with it, please get in contact.

My Japanese ability is so-so. I get by in day-to-day conversation, but I am by no means fluent right now. The object of publishing these translations is to see what differences I can find between the two books and to see if there is anything that seems to fundamentally change the meaning of the original text. I’m going to start off with something reasonably short and see how it goes. For those people out there with Japanese abilities better than mine, feel free to let me know if I’ve made some glaring error in my translation.

I’ll post the Japanese first, with my own translation following each line. At the end I will post the original text.

Lesson 25
モデリングはテストの決め手だ。
Modeling is a deciding factor in testing

テスト設計時には、頭の中で全体の構成を思い描いているだろう。
When test planning, you probably paint a mental picture of the test domain in your head.

機能一覧や関連図を駆使することもある。
You may also make use of a list of functions or graphical representations.

ユーザは誰で、どういったことを気にするのか、といった想定も必要だ。
Hypothesising about who the user is and what is important to them is also necessary.

こういったものを、総称して「モデル」と呼んでいる。
These sorts of things are generally called models.

実際、テストを設計するとき、元になるのは頭に描いた製品のモデルであり、新しいモデリング技法を学べば製品に対して新しい視点を持つことができる。
In practical terms, when test planning the origin (of your tests) is the model you have drawn in your head and learning new modelling techniques can give you a new perspective of the product.

このために、モデリング技法を勉強しよう。身に付けば、テストも上手くなる。
For this reason you should study modelling techniques. If you acquire this knowledge, your testing will also improve.

これには要求分析とソフトウエアアーキテクチャの教科書やセミナーが役に立つだろう。
You might find a textbook or seminar on requirements analysis or software architecture useful.

モデリングの技術全般を身に付けるのによい方法はシステムシンキング(Systems Thinking) を勉強することだ。
A good way of improving your technique in modeling overall is study of systems thinking.

An introduction to general systems thinking: silver anniversary edition (Weinberg 2001) を参照されたい。
Refer to …(reference in English as per original text)

Original text:

All testing is based on models.

You may have a mental picture in your mind when you design tests. Or, you may be working with a list of features or a diagram of some kind. All of these are models. No matter what, your tests will be based primarily on your models of the product, not the actual product. A flawed model results in flawed tests. Learning a new way to model a product is like learning a new way to see it.

Study modeling. You will test better as you become more skillful in the art of modelling. Textbooks and classes about requirements analysis and software architecture can help. A wonderful way to gain skill in all kinds of modeling is to study systems thinking. See An Introduction to General Systems Thinking Silver Anniversary Edition. (Weinberg 2001).

It looks to me like the translators have done their best to keep the meat of the content, but I’m curious about their choice to drop ‘A flawed model results in flawed tests’. Did they not think that was important information? I certainly do.

Translation is a tough gig. I’m not necessarily having a go at the translators, but these choices can change the tone and sometimes the meaning of the text. The sentence about the origin of tests was also simplified. Implicit in the original text is the fact that the relationship between the tester and the product is through their ability to model it effectively.

I’m curious to translate more and see what other differences I can find.

Jonathan Kohl is at least partially to blame for a new tool to assist testers doing session-based testing called (creatively enough) Session Tester. It’s been in the pipes for a while now and is finally seeing the light of day. It’s still in beta, but looking very promising. They’re also taking change request suggestions, so if you’re doing SBT and want a tool that suits your needs – go take a test drive and make suggestions.

James Bach recently wrote a post entitled Quality is Dead.

It brought to mind this particular gem

I’ll let you decide which software development role is analogous to the players in the sketch.

Testing isn’t dead yet, but it’s generally not being done any favours by anyone around it. The reasons are legion and of course they’re different from project to project. Fundamentally though I think it comes down to human beings with differing and often conflicting priorities.

I work in Japan where the customer is God and quality is paramount. In terms of software development however, some of the work practises I have seen here were old before Tokugawa Ieyasu came to power (and they haven’t aged well). I know where I would like to take quality focus, but the people who generally have the power to make it happen tend not to bother talking to grunts like me.

The brass jump up and down about quality for a while when customers or shareholders squeeze their balls about something going wrong. Said ball-squeezing is transferred down the management food chain until it gets to the QA division at which point one tends to hear about how we need to make things work better.

Of course, when you attempt to actually make changes for the better, you run into the aforementioned problem of human beings with conflicting priorities and you have zero power to change it.

What I would very much like to see happen is a bunch of people who have at one time or another been very skilled software testers assume CIO/CEO roles of large companies, understand what quality means to the people spending money on their product (and to the people not spending money on their product who may otherwise become customers) and make that priority one.

If that means the stock price suffers a little while you spend the necessary money, so be it. Until you get someone in a position with enough power to actually make focus on improving quality a must, then what you will inevitably end up with is a growing number of skilled testers who find themselves out of work when either a) their company outsources their job to a cheaper, less skilled option, or b) when they take an ethical stand and find themselves replaced by someone who is happy to take their place on a production line turning out broken toys.

Some of you that frequent this place might remember that I did a series of loosely-related posts on Tester Advocacy last year. I decided I was going to come up with a tester’s code of conduct by which a tester might conduct oneself.

Part 1
Part 2
Part 3

A change of job, a change of country – a change of many things actually, has given me pause to think about this once again. I think I’ve come up with a list that works.

Rule (of thumb) #1

Be useful.

That’s it. That’s my list. That’s my tester’s code of conduct. At all times do your level best to be useful.

How? Who to? What do you mean by useful?

I mean make it your mission to be as useful to the people you report to (the people who matter) as you possibly can. What can you do right now to be the most effective you can be in the way that your handler(s) need you to be?

Sometimes that is going to be doing what they ask you to do (even if you don’t like it). Sometimes it’s going to be doing something other than what they’ve asked because you are 100% certain that what you are doing is more useful to them. Sometimes it’s test execution, sometimes it’s teaching, sometimes (often, I hope) learning, sometimes it’s doing boring paperwork.

Be useful.

As a software tester, what is the most useful thing you can be doing right now? At the end of the day, I want to know that the actions I took have contributed to improving the quality of the product I worked on, or the company I work for. I want to know that I have improved the effectiveness of the people who work for me, or that other areas of the company received something of value from the testing team.

Be resourceful, be enthusiastic (or not), be proactive, be whatever you want as long as it’s also being useful. Note – being useful might not be appreciated. It might get you fired. Refusing to do something can sometimes be the most useful thing you can do. It might not be the wisest career-move (at least within the current company). If you can’t be useful, then maybe you need to be somewhere else.

Edit: Also – Happy new year, everybody.

For those of you (us) whose Japanese reading skills aren’t comprehensive, you might find some valuable assistance with a firefox plugin called pera-pera kun. It is an extension of the also excellent rikai-chan.

It allows you to hover over Japanese words and phrase fragments and get a popup with dictionary / grammatical meanings. I couldn’t do without it.

You can find the plug-in here
and the required Japanese dictionary here

I believe there is also a Chinese dictionary out there, don’t think they do hangul (sorry Jared   )

For those that are studying Japanese (or anything where flashcards are useful, really), check out Anki.
I am creating my own study deck, but if you want to dive right in, there are a bunch of pre-made decks available and plugins if you like that sort of thing.

I wish I’d found both of these years ago. Seriously. Hopefully they’re handy for some of you.

Clearly I’ve had it far too easy for the past 10 years. Working for internet-based companies with unfettered access to install whatever I like on my machine, all the internets I could eat, complete control over my testing environment(s).

I have a question for all of you who have in the past experienced complete machine lockdown in your role as a tester (or developer or similar). Imagine the following completely hypothetical situation:

You need to install stuff on your local box, but can’t because you don’t have privileges. Getting stuff installed on your box requires going through a cubic buttload of bureaucracy and can literally take 3 weeks. No (instant or semi-instant) comms with the outside world, disabled USB ports, the works. Does this hypothetical situation sound familiar to anyone?

Does one

a) Politely request that ones machine (and the machines of one’s team mates) be unshackled and cite decreased productivity as a reason (and possibly offer to instead abide by a set of guidelines that are reasonable)?
b) Rant against the concept of treating intelligent adults like naughty children or criminals?
c) Say nothing, but add a couple of weeks per app one may need to install to project test estimation?
d) None of the above, but something awesome that I have neglected to mention here?

Teh Google shows a lot of hits for people who are in favour of lockdown. Guess it’s okay when you’re on the other side of the fence. I found a few articles that were in favour of trusting users – I enjoyed reading through the comments here. I think it comes down to finding (begging for?) balance. I know not every user out there is savvy enough to stay safe on the internets, but there are plenty that are. Why should they suffer?

I’m interested to hear how people have handled this situation and what the results were.

It’s funny how one word can have multiple meanings. I’ve been thinking about this a lot lately in relation to learning Japanese, and especially Japanese grammar where identical grammatical structures can have quite different meanings depending on context.

There’s also an English word that’s been bugging me a bit lately. The word ‘just’ can be an awesome word. It can mean ‘fair and honourable’, or ‘precise or exact’, but I’m not so much of a fan when it is used in the following sentence fragment – ‘You’re just a tester…’

It often is accompanied by phrases or questions like ‘you wouldn’t understand’, ‘you don’t need to worry about that’, ‘what are you asking about that for?’, ‘you don’t need access to that’ or ‘what do you mean you should be paid as much as a developer?’

As grating as the assumption that you’re not techie enough, or that anyone can do your job (or similar ignorance) may be, my advice is to not take it personally.

I could choose to feel insulted, but I don’t. It generally means to me that I have to help educate someone. I’ve heard it from developers, project managers, salespeople, executives, all sorts of people who think they have a handle on what it takes to be a tester. It may be that their previous experience has been a negative one with low-skilled testers. Maybe they have no experience with testers at all and are going on assumptions and second or even third-hand information. Whatever the reason, it is clear that their understanding of software testing is framed differently to mine.

Sometimes, a short conversation on what our differences in understanding are is all it takes, but if someone has the idea firmly in their head that testers are (just) monkeys that click on stuff toward the end of a project, you’re probably going to have to be a little more hands-on in demonstrating your value.

Find out what they think your limitations are, then find a way to add value that goes above and beyond what they’re expecting. At the start of a project, analyse the design and put together a risk analysis (along with what can be done – not just testing – to mitigate them). Are there unit tests or other automated tests you can review (and possibly improve)? Have developers put their money where their mouth is with Mike Kelly’s 5 bugs in 5 minutes challenge (PDF link – see page 5).

You might demonstrate the value of exploratory testing by conducting a paired testing session with them. What you do will of course depend on your own situation. Be proactive about it though. You can point them at blog posts and pdf links all you like, but until you actually demonstrate to them the value of a tester in a way that they’re going to notice or care about, you’re just going to have to deal with being thought of as ‘just a tester’. You might not have created the situation, but if there are people (who matter) whose point of view needs adjustment then the onus is on you.

Jeff Atwood recently wrote a post about phases of software development/release. He wondered where the terms come from and why we rarely use ‘gamma’ – I wondered if these terms actually have any relevance to what actually goes on in software development anymore.

To me, these terms are used to broadly group stages of software stability as simplifiers to people who don’t necessarily understand the finer details of software development (or don’t particularly want to). These labels seem to equate these days to marketing wank for shops that still do big design up front. It’s easier to say ‘we’re at alpha release’ than saying ‘we’re two-thirds feature complete with no current showstoppers, have a bug trendline that indicates a find-rate approaching zero and we’re satisfied that testing to date has adequately mitigated the risks identified so far’.

My issue with these terms is that they oversimplify a process that should be viewed as difficult. Alpha/Beta/Gamma (or whatever other pre-release signifiers you like) imply a linear, production-line progression of software. Anyone that has worked on any project of significance knows that the development of software is anything but. I worry that this oversimplification helps tie us in to a flawed mental model that invites us to fail over and over again.

There is no guarantee that half way through ‘beta testing’ your testing team won’t find a showstopper that requires you to go back to pre-alpha. How well does your ‘A->B->Ship it’ model stack up then? It seems that because we have this simplified mental model of development, decision makers think that a) we can reliably estimate up-front the time and budget required to put together complex systems and b) when, close to completion neither of these estimates prove even remotely accurate, makes them insist on kludging their way through so that the immutable ship date can be met.

The problem with this mental model is that while it probably works well for automatons that churn out identical widgets time and again, software development is not that. Software development is about human beings turning out solutions to difficult problems in a non-uniform manner. Human beings each with their own individual strengths, weaknesses, passions, neuroses and everything else that makes us human. The fact that these people are not producing identical widgets day in, day out means that the production line model is entirely inappropriate, as is the model of big design (and estimation) up front.

This is a can of worms that you could (and some have) write a book about. Entwined with development is software testing. Another very challenging craft done by humans. Testing is an oft-misunderstood profession. The alpha/beta/ship it model doesn’t help. I worry that this model locks people into thinking that testing is something that happens once the developers have finished doing what they do, at which point the testers are left to find whatever bugs they are able to in between that point and the shipping date.

Then, depending on the nature of the project and the people they’re working with, they’re possibly blamed for finding too many bugs, or flaws that, if properly fixed, would extend the length of the project well beyond the date where the PM gets his nice fat bonus for bringing this thing in on time. The PM shoots the messenger because what other choice is there at that point?

This is a stupid way to make software. This is a stupid way to think about making software.

Scrum appears to be intended to make transparent the human issues that crop up in software development and it sounds great in theory, but I wonder how many software shops are prepared to invest in the up-front discomfort of changing over and are prepared to have (on a daily basis) the open, honest and frank dialogue necessary for a framework like that to work. My guess is not many. This is perhaps a subject for another post.