testjutsu

I wanted to tackle something a little heftier than the last post to see if content that was less straightforward to explain was also challenging to translate. I went with lesson 28 – Exploring requires a lot of thinking.

The translation had a few things that were interesting, including the title. I’ll put up the translation, let you formulate your own opinions (and give Japanese readers the chance to tell me that I’ve got it all wrong), then explain what I took form it.

Lesson 28

探求のためには三つの思考法がある。
When it comes to exploration, there are 3 ways of thinking

探求は探偵と同じだ。調査の結果、何が出てくるか全く想像も出来ない。探求は空間内の移動だと思えるとよい。思考法には前向き、後向き、水平がある。
Exploration is the same as investigation. What comes out of the results of investigation is impossible to imagine. You can think of searching as movement through space. Ways of thinking are forward, backward and laterally.

前方思考：既知から未知へ、目の前の事実からまだ見ぬものを探索する思考法である。既知の行為による結果や影響を探索するのだ。
例）　プリントを見る。クリックしたら、何が起こるのだろう？
Forward thinking: From the known to the unknown, from the immediate facts search for the things you have not seen before. Seek out the results or effects of known actions.
e.g.) Look at the print menu. If you click it, what do you think happens?

後方思考：問題や想像から既知へとたどり、推測の正否を確認する思考法である。
例）　文書を印刷する方法があるだろうか。メニューに印刷の項目があるか確かめよう (Solow 1990)。
Backward thinking: from questions or conjecture get to a place that is known. Check the correctness of your guess.
e.g.) document print functionality probably exists. Make sure there is a printing option in the menu.

水平思考：思考の寄り道をしてみよう。思いつきで脇道を探索し、また本筋に戻る (de Bono 1970)。
例）　おもしろいグラフィックだな。よし、何か複雑なグラフィックを印刷してどうなるか見てみよう。
Lateral thinking: Try taking detours in your thinking. Thinking of an idea, investigate side-roads and return to the main thread.
e.g.) That’s an interesting graphic. Right, let’s try printing complex graphics and see what happens.

テストする製品なくても探求プロセス使える。こうした思考プロセスで、一連の文書を探求したり、プログラマにインタビューしたりすればよい。より豊かにより観念化されたモデルを創り上げることによって進歩するのだ。また、モデルを使えば効果的なテストを設計することができる。
Even if you don’t have a product to test, you can still use an exploratory process. Using this thought process is good for exploring a series of documents, or interviewing a programmer. By creating enriched conceptual models we make progress. Also, if you create a model, you can design more effective tests.

Original text

Exploring involves a lot of thinking.

Exploring is detective work. It’s an open-ended search. Think of exploration as moving through a space. It involves forward, backward and lateral thinking.

Forward thinking. Work from what you know to what you don’t know; what you see toward what you haven’t yet seen. Seek ramifications and side effects. Example: I see a print menu item. I’ll click on it and see what happens.

Backward thinking. Work from what you suspect or imagine back toward what you know, trying ti confirm or refute your conjectures. Example: I wonder if there’s a way to print this document? I’ll look through the menus and see whether there’s a print item. (Solow 1990)

Lateral thinking. Let your work be distracted by ideas that pop into your head, exploring tangents and then returning to the main thread (de Bono 1970). Example: That’s an interesting graphic. Hey, I think I’ll print some complex graphics and see what happens.

The exploratory process works even if you don’t have a product to test. You can explore a set of documents or interview a programmer, using the same thought processes. You make progress by building richer, better mental models of the product. These models then allow you to design effective tests.

I got the feeling that the tanslators wanted to make this a little more ‘concrete’ than the original. Not saying they editorialised, but there do seem to be some liberties taken.

My own take is that the original lesson describes one possible way to organise your thinking but isn’t necessarily looking to lock the reader into 3 distinct ways of thinking. There are other ways you could choose to organise your thinking. This translation has echoes of the book title ‘lessons learned’ vs. ‘immutable laws’ – one feels like a coversation (or an argument), the other a decree.

Okay, so I’m projecting a little bit, but I do think the renamed title takes something important away from the lesson. The words are there, the same structure is there, but the meaning I take from the Japanese version is different from the one I take from English.

As an aside, the word they chose for ‘exploration’ was 探求 (tankyuu). The two dictionaries I consulted listed this as ‘quest, search, pursuit’. When I looked for ‘explore’ I got 探検 (tanken). They share the same first character (tan), which means to search for or look for. 求 (kyu) means ‘want, demand, require’. 検 means ‘investigate or examine’. Based on the characters, I’d have thought 探検 was a more fitting choice for ‘exploration’, but perhaps the general usage of the word has connotations of an outdoors expedition. I’m not sure (I’ll ask around).

I think I’ll see what else I can find, maybe try a few lesson titles and see what the differences are there.

Here’s my latest foray into re-translation. Actually, this seems to be a pretty good translation of the original text. I suspect what I need to do is find some of the more difficult passages and see what I can find around those. I’ve been looking at the shorter entries to this point because hey – they’re short and my kanji reading is still really slow.

I’ll look to get something a little meatier in my next post on the topic.

Lesson 35

結局、テスト結果は製品に対する「印象」に過ぎない。
After all, test results are only an impression of the product

製品の品質について知っていることは、すべて製品に対する推測に過ぎない。
With respect to quality, what you know about the entire product is just conjecture.

どんなに支持されても正しいとは言い切れない。
However much support is available, you cannot simply declare it correct.

だから製品の品質状況を報告するときは、どんなテストしたか、テストプロセスから考えてどんな限界があったかといった条件を必ずレポートに付けるべきなのである。
Because of this, when reporting the condition of quality, you should add to the report qualifications such as the kind of testing that was done and any limits to the test process.

Original text:

In the end, all you have is an impression of the product.

Whatever you know about the quality of the product, it’s conjecture. No matter how well supported, you can’t be sure you’re right. Therefore any time you report the status of the product quality, you should qualify that report with information about how you tested and the known limitations of your test process.

Jonathan Kohl beat me to the punch on this blog entry by a good margin, but since I was already working on this thing, I’d like to go ahead and basically say exactly what he said in probably a far less cogent fashion.  A lot has been written about the good and bad things about Exploratory testing and the writing and (mis)usage of test cases.

I don’t see a lot written about harmonious use of both. I’m sure people do it. I know I do. Is it like airing dirty laundry? Do context driven testers not want to admit to using test cases in case James Bach reads their blog and decides to tear them a new one? Do factory testers (sorry – what do you BDUF testers call yourselves anyway? Factory tester brings to mind a grey-clad downtrodden peon, which I suppose is appropriate, but not very flattering) – anyway, do you lot not want to admit to doing exploratory testing in case your overseers beat your for not following the script? Am I missing something? Is this just not a big deal?

Johnathan advocates using a blend of both and uses the terms prescriptive and descriptive testing, which quite appeals to me. Michael Bolton’s recent highlighting of the difference between testing and checking as something we should be more conscious of helped me to frame this better in my mind.

There are things that you can plan ahead for. There are risks you can identify and tests that you can design to mitigate against them. I see no reason why you shouldn’t use test cases for these if that is what you are comfortable with.

One of the issues I have with test cases is that some people see them as a magic artefact that covers all possible problems in a given area, because one or more tests have been written that mention it.  They fail to see that many test cases are highly focused on verifying a very specific item somewhere, and just because we go through a bunch of steps to get there does not mean that those steps have also been verified.

This leads me to wonder if I have been thinking about scripts all wrong.

I recently read Michael Hunter’s work on setting up test frameworks. One of the things that was a lightbulb for me was separating the verification from the action. Very very obvious in hindsight, but if you think about it – this is basically a use case and a checklist.

We have a series of steps from point A to point B

We have a list of things that we want to make sure are as we expect them.  Why do we need to mash them together? Why do we need to repeat the list of steps over and over and over so that we can list a different verification point next to it?

An argument I hear for test cases is that if you write them well enough, then anyone can execute them. I really hate the way that thinking goes, mostly because I read it as ‘anyone can do testing, you just follow the scripts’. That being said, a clear set of instructions and a checklist could mean that you could get a non-tester to do checking, even of mission-critical information. That, I might hold with.

Checklists and Check scripts? I think this is something worth exploring further.