testjutsu

I am making an appeal to those out there who are managing testers, or are in a position where their decisions influence testers.

I loosely follow several testing groups to see what interesting conversations are going on. I don’t often participate, mostly because I can’t commit to having the time to finish a conversation. If I can drop a comment I think will be helpful, then I will.

Then there are questions or statements that make me a little upset. I try not to respond because I tend to say things that are unkind. There are some people out there that need help and seem to be unaware of it. If you manage a testing team and you don’t have a lot of experience as a tester, you may be one of them.

The questions that get me irritated are mostly stupid questions asked by people that (according to their listed job title) should know better. Maybe stupid is not the right word. Lazy, ill-informed, ignorant – these are probably better adjectives.

Recently I saw the following questions asked:

How can testing be monitored? How can we measure the performance of testing?

That was it. No context, no qualification, no further explanation. I assert that becasuse of this, they are stupid questions. I suspect what this person meant is ‘how do I gauge the efficiency and effectiveness of the work that my testers are doing?’, but that’s one of many different possible interpretations of the original questions.

Not to mention the fact that I suspect that what this person actually wants is a bunch of numbers they can use to reassure themselves that all is well, or possibly to threaten perceived underperformers with.

I don’t really want to get into a rant on metrics. It’s been done to death by far more qualified people than I (see ‘meaningful metrics’). What I do want to say is that if you are managing software testers and you are asking questions like this, you really should know better. Stop being lazy and educate yourself. I feel sorry for the people working for you.

For people who are managing testers – especially if you do not have experience in software testing, let me pose the following questions to you.

How do you measure the effectiveness of a systems administrator? a business analyst? How do you know when you have someone who is ineffective in these roles?

Both of these roles require quite specific knowledge. There is also a massive amount of variation within the definition of these roles. A sysadmin that works on webservers is likely to have different strengths than one that specialises in SMTP admin. Of course there will be overlap in their skills, but they are different animals. Applying any sort of blanket measurement to them because they share the same title is stupid.

One thing these jobs do have in common (and in common with testing) is that they are knowledge and service related. If the people they are providing a service to are not satisfied, then you have an issue. You don’t go setting up metrics to count how many shell scripts and admin has written. You don’t judge a BA on how many use cases they write. You gauge their effectiveness by how their work is received by their peers and their clients.

Why would it be any different with testing? Testing requires specialised knowledge. How do you gauge the effectiveness of the people doing it? Look at the information they produce. Consult the people they produce it for. Speak with the testers themselves. Is the information useful, or not? Why/Why not? There you go. That’s how you measure the effectiveness of your testing.

Without the qualitative information these people can convey, you can collect all the metrics and numbers you like. You can draw all the conclusions from them you like, but those numbers alone are not enough to make any meaningful distinctions.

Worse still, if you’re judging the effectiveness of a testing effort during a project by things like bug counts and test case coverage, you’re potentially ignoring important information. A simple example – we have X open bugs. So what? What does that mean? So you’ve covered 100% of test cases – what does that mean? What other testing have you done? What did that find? What does that mean?

If you are using numbers as a barbiturate so that you can feel reassured about how the project is going – if you look at the numbers and and nothing else, and you accept them at face value, then you are being lazy. If you don’t have time to dig deeper then that’s a bigger problem, not an excuse.

If I sound a bit defensive, it’s because I am. I’m sick of seeing lazy people give testers a bad name because they don’t understand testing, aren’t interested in learning about testing and yet want to have some way to control testing because that’s how they think it works.

If you manage testers but you don’t trust them to be able to do their job, and if you aren’t willing to learn about the difference between effective testing and ineffective testing then have the stones to step aside from someone that is. You are worse than useless in your current role.You are a hindrance and are in all likelihood making people deeply unhappy.

If you are willing to learn, do try and think it through. What is it you are trying to achieve? Are you doing it because it’s genuinely useful, or because that’s what everyone else seems to be doing? Do some research. Plenty has been written down about it. At least that way when you ask a question, it will be an informed one. When I read questions like the above, here’s how it comes across:  ‘o hai, I ar new manager. Plz hope me teh metriks with xamplez kthxbai’.

I think that as managers of testers, it his high time we raised the bar. We have a duty of care to understand what they do and the challenges they face. To support their work and to maximise the effectiveness of their efforts.If you are not constantly seeking sources to help you do this, then you are doing them a disservice.

If you have your heart set on measurement, then please, at the very least learn the difference between first, second and third order measurement and how to apply them.

Additionally, it’d be a bonus if you understood more about testing and how testing fits into software development (please do your testers a favour and read this book).

Hopefully for you this is the tip of a very large iceberg. If you do these things, you will look back at questions like the ones posed above and wonder how you could ever have been that naieve. If you can cultivate a habit of curiosity and the desire to learn then you stand good chance of becoming the sort of manager your testers need you to be.

I wanted to tackle something a little heftier than the last post to see if content that was less straightforward to explain was also challenging to translate. I went with lesson 28 – Exploring requires a lot of thinking.

The translation had a few things that were interesting, including the title. I’ll put up the translation, let you formulate your own opinions (and give Japanese readers the chance to tell me that I’ve got it all wrong), then explain what I took form it.

Lesson 28

探求のためには三つの思考法がある。
When it comes to exploration, there are 3 ways of thinking

探求は探偵と同じだ。調査の結果、何が出てくるか全く想像も出来ない。探求は空間内の移動だと思えるとよい。思考法には前向き、後向き、水平がある。
Exploration is the same as investigation. What comes out of the results of investigation is impossible to imagine. You can think of searching as movement through space. Ways of thinking are forward, backward and laterally.

前方思考：既知から未知へ、目の前の事実からまだ見ぬものを探索する思考法である。既知の行為による結果や影響を探索するのだ。
例）　プリントを見る。クリックしたら、何が起こるのだろう？
Forward thinking: From the known to the unknown, from the immediate facts search for the things you have not seen before. Seek out the results or effects of known actions.
e.g.) Look at the print menu. If you click it, what do you think happens?

後方思考：問題や想像から既知へとたどり、推測の正否を確認する思考法である。
例）　文書を印刷する方法があるだろうか。メニューに印刷の項目があるか確かめよう (Solow 1990)。
Backward thinking: from questions or conjecture get to a place that is known. Check the correctness of your guess.
e.g.) document print functionality probably exists. Make sure there is a printing option in the menu.

水平思考：思考の寄り道をしてみよう。思いつきで脇道を探索し、また本筋に戻る (de Bono 1970)。
例）　おもしろいグラフィックだな。よし、何か複雑なグラフィックを印刷してどうなるか見てみよう。
Lateral thinking: Try taking detours in your thinking. Thinking of an idea, investigate side-roads and return to the main thread.
e.g.) That’s an interesting graphic. Right, let’s try printing complex graphics and see what happens.

テストする製品なくても探求プロセス使える。こうした思考プロセスで、一連の文書を探求したり、プログラマにインタビューしたりすればよい。より豊かにより観念化されたモデルを創り上げることによって進歩するのだ。また、モデルを使えば効果的なテストを設計することができる。
Even if you don’t have a product to test, you can still use an exploratory process. Using this thought process is good for exploring a series of documents, or interviewing a programmer. By creating enriched conceptual models we make progress. Also, if you create a model, you can design more effective tests.

Original text

Exploring involves a lot of thinking.

Exploring is detective work. It’s an open-ended search. Think of exploration as moving through a space. It involves forward, backward and lateral thinking.

Forward thinking. Work from what you know to what you don’t know; what you see toward what you haven’t yet seen. Seek ramifications and side effects. Example: I see a print menu item. I’ll click on it and see what happens.

Backward thinking. Work from what you suspect or imagine back toward what you know, trying ti confirm or refute your conjectures. Example: I wonder if there’s a way to print this document? I’ll look through the menus and see whether there’s a print item. (Solow 1990)

Lateral thinking. Let your work be distracted by ideas that pop into your head, exploring tangents and then returning to the main thread (de Bono 1970). Example: That’s an interesting graphic. Hey, I think I’ll print some complex graphics and see what happens.

The exploratory process works even if you don’t have a product to test. You can explore a set of documents or interview a programmer, using the same thought processes. You make progress by building richer, better mental models of the product. These models then allow you to design effective tests.

I got the feeling that the tanslators wanted to make this a little more ‘concrete’ than the original. Not saying they editorialised, but there do seem to be some liberties taken.

My own take is that the original lesson describes one possible way to organise your thinking but isn’t necessarily looking to lock the reader into 3 distinct ways of thinking. There are other ways you could choose to organise your thinking. This translation has echoes of the book title ‘lessons learned’ vs. ‘immutable laws’ – one feels like a coversation (or an argument), the other a decree.

Okay, so I’m projecting a little bit, but I do think the renamed title takes something important away from the lesson. The words are there, the same structure is there, but the meaning I take from the Japanese version is different from the one I take from English.

As an aside, the word they chose for ‘exploration’ was 探求 (tankyuu). The two dictionaries I consulted listed this as ‘quest, search, pursuit’. When I looked for ‘explore’ I got 探検 (tanken). They share the same first character (tan), which means to search for or look for. 求 (kyu) means ‘want, demand, require’. 検 means ‘investigate or examine’. Based on the characters, I’d have thought 探検 was a more fitting choice for ‘exploration’, but perhaps the general usage of the word has connotations of an outdoors expedition. I’m not sure (I’ll ask around).

I think I’ll see what else I can find, maybe try a few lesson titles and see what the differences are there.

Here’s my latest foray into re-translation. Actually, this seems to be a pretty good translation of the original text. I suspect what I need to do is find some of the more difficult passages and see what I can find around those. I’ve been looking at the shorter entries to this point because hey – they’re short and my kanji reading is still really slow.

I’ll look to get something a little meatier in my next post on the topic.

Lesson 35

結局、テスト結果は製品に対する「印象」に過ぎない。
After all, test results are only an impression of the product

製品の品質について知っていることは、すべて製品に対する推測に過ぎない。
With respect to quality, what you know about the entire product is just conjecture.

どんなに支持されても正しいとは言い切れない。
However much support is available, you cannot simply declare it correct.

だから製品の品質状況を報告するときは、どんなテストしたか、テストプロセスから考えてどんな限界があったかといった条件を必ずレポートに付けるべきなのである。
Because of this, when reporting the condition of quality, you should add to the report qualifications such as the kind of testing that was done and any limits to the test process.

Original text:

In the end, all you have is an impression of the product.

Whatever you know about the quality of the product, it’s conjecture. No matter how well supported, you can’t be sure you’re right. Therefore any time you report the status of the product quality, you should qualify that report with information about how you tested and the known limitations of your test process.

Jonathan Kohl beat me to the punch on this blog entry by a good margin, but since I was already working on this thing, I’d like to go ahead and basically say exactly what he said in probably a far less cogent fashion.  A lot has been written about the good and bad things about Exploratory testing and the writing and (mis)usage of test cases.

I don’t see a lot written about harmonious use of both. I’m sure people do it. I know I do. Is it like airing dirty laundry? Do context driven testers not want to admit to using test cases in case James Bach reads their blog and decides to tear them a new one? Do factory testers (sorry – what do you BDUF testers call yourselves anyway? Factory tester brings to mind a grey-clad downtrodden peon, which I suppose is appropriate, but not very flattering) – anyway, do you lot not want to admit to doing exploratory testing in case your overseers beat your for not following the script? Am I missing something? Is this just not a big deal?

Johnathan advocates using a blend of both and uses the terms prescriptive and descriptive testing, which quite appeals to me. Michael Bolton’s recent highlighting of the difference between testing and checking as something we should be more conscious of helped me to frame this better in my mind.

There are things that you can plan ahead for. There are risks you can identify and tests that you can design to mitigate against them. I see no reason why you shouldn’t use test cases for these if that is what you are comfortable with.

One of the issues I have with test cases is that some people see them as a magic artefact that covers all possible problems in a given area, because one or more tests have been written that mention it.  They fail to see that many test cases are highly focused on verifying a very specific item somewhere, and just because we go through a bunch of steps to get there does not mean that those steps have also been verified.

This leads me to wonder if I have been thinking about scripts all wrong.

I recently read Michael Hunter’s work on setting up test frameworks. One of the things that was a lightbulb for me was separating the verification from the action. Very very obvious in hindsight, but if you think about it – this is basically a use case and a checklist.

We have a series of steps from point A to point B

We have a list of things that we want to make sure are as we expect them.  Why do we need to mash them together? Why do we need to repeat the list of steps over and over and over so that we can list a different verification point next to it?

An argument I hear for test cases is that if you write them well enough, then anyone can execute them. I really hate the way that thinking goes, mostly because I read it as ‘anyone can do testing, you just follow the scripts’. That being said, a clear set of instructions and a checklist could mean that you could get a non-tester to do checking, even of mission-critical information. That, I might hold with.

Checklists and Check scripts? I think this is something worth exploring further.

Part of the work I’m doing just now is helping developers to increase their testing skills to handle the bandwidth that my team can’t. I asked Yuka Horino, one of my colleagues to translate James Bach’s Heuristics of Software Testability into Japanese as one of the things to distribute to our dev group. (Thanks Horino-san!)

This sort of thing is probably useful to a wider group than just the people I work with, so here it is. Enjoy.

Heuristics of Software Testability (Japanese)

So a while back I said I’d be posting translations from the Japanese version of Lessons learned in software testing. Basically, I pulled the trigger way too early on that. There have been a bunch of things all demanding my time, so this was a side-project that I just wasn’t able to get to before now.

So the usual disclaimers apply – I have permission from the original authors of this tome, but not from the publishers. To the best of my knowledge this falls within fair use, but given the geography of this blog and myself coupled with the fact that I’m not a lawyer, I could be dead wrong. So if any of the relevant publishing bods happen across this and have a problem with it, please get in contact.

My Japanese ability is so-so. I get by in day-to-day conversation, but I am by no means fluent right now. The object of publishing these translations is to see what differences I can find between the two books and to see if there is anything that seems to fundamentally change the meaning of the original text. I’m going to start off with something reasonably short and see how it goes. For those people out there with Japanese abilities better than mine, feel free to let me know if I’ve made some glaring error in my translation.

I’ll post the Japanese first, with my own translation following each line. At the end I will post the original text.

Lesson 25
モデリングはテストの決め手だ。
Modeling is a deciding factor in testing

テスト設計時には、頭の中で全体の構成を思い描いているだろう。
When test planning, you probably paint a mental picture of the test domain in your head.

機能一覧や関連図を駆使することもある。
You may also make use of a list of functions or graphical representations.

ユーザは誰で、どういったことを気にするのか、といった想定も必要だ。
Hypothesising about who the user is and what is important to them is also necessary.

こういったものを、総称して「モデル」と呼んでいる。
These sorts of things are generally called models.

実際、テストを設計するとき、元になるのは頭に描いた製品のモデルであり、新しいモデリング技法を学べば製品に対して新しい視点を持つことができる。
In practical terms, when test planning the origin (of your tests) is the model you have drawn in your head and learning new modelling techniques can give you a new perspective of the product.

このために、モデリング技法を勉強しよう。身に付けば、テストも上手くなる。
For this reason you should study modelling techniques. If you acquire this knowledge, your testing will also improve.

これには要求分析とソフトウエアアーキテクチャの教科書やセミナーが役に立つだろう。
You might find a textbook or seminar on requirements analysis or software architecture useful.

モデリングの技術全般を身に付けるのによい方法はシステムシンキング(Systems Thinking) を勉強することだ。
A good way of improving your technique in modeling overall is study of systems thinking.

An introduction to general systems thinking: silver anniversary edition (Weinberg 2001) を参照されたい。
Refer to …(reference in English as per original text)

Original text:

All testing is based on models.

You may have a mental picture in your mind when you design tests. Or, you may be working with a list of features or a diagram of some kind. All of these are models. No matter what, your tests will be based primarily on your models of the product, not the actual product. A flawed model results in flawed tests. Learning a new way to model a product is like learning a new way to see it.

Study modeling. You will test better as you become more skillful in the art of modelling. Textbooks and classes about requirements analysis and software architecture can help. A wonderful way to gain skill in all kinds of modeling is to study systems thinking. See An Introduction to General Systems Thinking Silver Anniversary Edition. (Weinberg 2001).

It looks to me like the translators have done their best to keep the meat of the content, but I’m curious about their choice to drop ‘A flawed model results in flawed tests’. Did they not think that was important information? I certainly do.

Translation is a tough gig. I’m not necessarily having a go at the translators, but these choices can change the tone and sometimes the meaning of the text. The sentence about the origin of tests was also simplified. Implicit in the original text is the fact that the relationship between the tester and the product is through their ability to model it effectively.

I’m curious to translate more and see what other differences I can find.

Jonathan Kohl is at least partially to blame for a new tool to assist testers doing session-based testing called (creatively enough) Session Tester. It’s been in the pipes for a while now and is finally seeing the light of day. It’s still in beta, but looking very promising. They’re also taking change request suggestions, so if you’re doing SBT and want a tool that suits your needs – go take a test drive and make suggestions.

James Bach recently wrote a post entitled Quality is Dead.

It brought to mind this particular gem

I’ll let you decide which software development role is analogous to the players in the sketch.

Testing isn’t dead yet, but it’s generally not being done any favours by anyone around it. The reasons are legion and of course they’re different from project to project. Fundamentally though I think it comes down to human beings with differing and often conflicting priorities.

I work in Japan where the customer is God and quality is paramount. In terms of software development however, some of the work practises I have seen here were old before Tokugawa Ieyasu came to power (and they haven’t aged well). I know where I would like to take quality focus, but the people who generally have the power to make it happen tend not to bother talking to grunts like me.

The brass jump up and down about quality for a while when customers or shareholders squeeze their balls about something going wrong. Said ball-squeezing is transferred down the management food chain until it gets to the QA division at which point one tends to hear about how we need to make things work better.

Of course, when you attempt to actually make changes for the better, you run into the aforementioned problem of human beings with conflicting priorities and you have zero power to change it.

What I would very much like to see happen is a bunch of people who have at one time or another been very skilled software testers assume CIO/CEO roles of large companies, understand what quality means to the people spending money on their product (and to the people not spending money on their product who may otherwise become customers) and make that priority one.

If that means the stock price suffers a little while you spend the necessary money, so be it. Until you get someone in a position with enough power to actually make focus on improving quality a must, then what you will inevitably end up with is a growing number of skilled testers who find themselves out of work when either a) their company outsources their job to a cheaper, less skilled option, or b) when they take an ethical stand and find themselves replaced by someone who is happy to take their place on a production line turning out broken toys.

I was wondering whether or not I would make it to CAST this year. After having such a fantastic time last year, I really wanted to attend this one also. This year though, my other passion will be taking precedence. I am going to Brazil at the end of August to attend the world kendo championships.

For you software testers who are going to be in the States (or close by) mid-July – check out details of the CAST conference. It’s in Colorado this year. Well worth attending. For anyone that’s going to be in Sao Paulo at the end of August, buy an Australian flag and come support the Aussie kendo team

Some of you that frequent this place might remember that I did a series of loosely-related posts on Tester Advocacy last year. I decided I was going to come up with a tester’s code of conduct by which a tester might conduct oneself.

Part 1
Part 2
Part 3

A change of job, a change of country – a change of many things actually, has given me pause to think about this once again. I think I’ve come up with a list that works.

Rule (of thumb) #1

Be useful.

That’s it. That’s my list. That’s my tester’s code of conduct. At all times do your level best to be useful.

How? Who to? What do you mean by useful?

I mean make it your mission to be as useful to the people you report to (the people who matter) as you possibly can. What can you do right now to be the most effective you can be in the way that your handler(s) need you to be?

Sometimes that is going to be doing what they ask you to do (even if you don’t like it). Sometimes it’s going to be doing something other than what they’ve asked because you are 100% certain that what you are doing is more useful to them. Sometimes it’s test execution, sometimes it’s teaching, sometimes (often, I hope) learning, sometimes it’s doing boring paperwork.

Be useful.

As a software tester, what is the most useful thing you can be doing right now? At the end of the day, I want to know that the actions I took have contributed to improving the quality of the product I worked on, or the company I work for. I want to know that I have improved the effectiveness of the people who work for me, or that other areas of the company received something of value from the testing team.

Be resourceful, be enthusiastic (or not), be proactive, be whatever you want as long as it’s also being useful. Note – being useful might not be appreciated. It might get you fired. Refusing to do something can sometimes be the most useful thing you can do. It might not be the wisest career-move (at least within the current company). If you can’t be useful, then maybe you need to be somewhere else.

Edit: Also – Happy new year, everybody.