testjutsu

落花枝に
帰ると見れば
胡蝶哉

A haiku written by Moritake Arakida which can be translated as ‘Did I see blossom return to the branch? (No, it was) a butterfly.

When I first read this, it struck a chord with me for some reason. Perhaps it was that someone found mistaking a butterfly for a blossom or a leaf profound enough to write a poem about, or that four hundred years later it would be around for the likes of me to read and enjoy. Perhaps it’s because the nature of cherry blossoms is so very fleeting, I liked the idea of a single flower bucking the system. In any case, it’s one of my favourites.

From a testing point of view, it serves to remind me to challenge my preconceptions. Because I think something is so, doesn’t necessarily mean that it is. The map is not the territory. Have you ever edited code, and run it only to find zero change? Done variations on that for hours before realising it was the wrong file? I have, but only because I didn’t challenge my presumption that I was editing the right file.

If something isn’t working as you expect - especially if it’s one of those really excruciatingly difficult things to find, try taking a step back and examining if you can, what your most basic assumptions are - then challenge them. After all, when you really get down to it, if you rip the wings off a butterfly, it’s just another bug.

Most testers I know love brain-teasers and mind games. I’m no exception (I just happen to be really crap at them). When news surfaced that the new Nine Inch Nails album ‘Year Zero’ had an alternate reality game (ARG) attached to it, I was initially skeptical. Probably some bullshit marketing hype for the emo fanboys to spend their pocket money on.

I didn’t bother looking into it too much initially, but after attending their recent concerts in Melbourne, I thought I’d check it out. Turns out that it’s less (overtly) about the marketing, and more about well, the album itself I suppose - anti-government resistance from 15 years in the future are sending back forbidden images and data to us here in the past.

Checkout ninwiki.com for what has been found so far. If you’re a fan of sleuthing out obscure references in texts and interpreting that to find hidden messages in things, then you’ll probably enjoy this. If you already enjoy Trent Reznor’s music, you’ll probably like it more.

In any case, I think it’s a great exercise to keep a tester’s mind sharp. I might throw it to my test team and see how they do.

I was at a good friend’s wedding several nights ago, which was held at one of the more well-to-do hotels in Melbourne. The night itself was an immense success. The bride and groom looked fantastic, I caught up with old friends from far away and a great time was had by all. It struck me though how deep testing seems to run in my veins - I just can’t seem to switch off (even when there’s an open bar).

I arrived on foot and my first impression of the place was the apparent haphazardness of the valet parking - there were a lot of cars and all of them were parked on the footpath leading to the lobby. My better half and I picked our way to the front doors, taking care not to scratch the paintwork of some reasonably expensive machines.

Once inside, there was the matter of finding a cloakroom. There was nothing in the immediate vicinity, so I asked at the concierge. ‘Oh, it’s downstairs’ said they. The stair case was about fifty meters away on the other side of the hall.

So not only was the cloakroom not located close to the main entrance, but the process of asking after it left you about as far away from it as it was possible to be. I headed down to the cloakroom and was asked ‘are you here for the ball, or the wedding?’ and when I responded with the latter, was directed back upstairs to the bus service desk outside the main entrance.

Somewhat incredulous, I headed back upstairs. The first thing the chap at the bus desk said when I presented my coat was ‘oh, you’ll have to go down to the cloak room’.

Not the most wonderful first impression of what is supposed to be one of the city’s finest hospitality establishments. The place was certainly decked out ostentatiously. They hadn’t spared expense on materials, but whoever designed the place, didn’t have the end-user in mind.

Things on the usability front didn’t improve that much over the course of the night. In three of the four places I was in that night (lobby included), the doors to the toilets were built to look like the rest of the wall and were nigh-on invisible. I observed at least twenty people ask the staff where they were.

I can understand not wanting to have pink flashing neon lights that say ‘here be the dunny’, and I get the whole ‘wanting to be subtle’ thing. It’s nice, but surely there is a way to have the entrance to the water closet be unobtrusive and yet visible when needed. A door sign that reads ‘ladies’ or ‘gents’ is not going to spoil the ambience. That and not being asked to point them out is likely to be less grating to the staff.

The wedding reception was in a bar on the lobby level. It had a spiral staircase that opened onto a wide area with all sorts of nooks and crannies in which small groups of people could hide and chat. Which is great when you’ve ordered five drinks and have no idea where the people you just ordered them for have disappeared to.

When the time came to leave, we discovered that because the place was easy to get into, it didn’t necessarily follow that it was easy to leave. The staircase itself finished in one corner and was blocked off from one side such that it looked accessible from both sides until you actually went to use it. Not conducive to use by a room full of people with access to an open bar. At the top of the stairs, opaque glass walls confronted us, and led the eye around to the kitchens - and nothing else.

Closer inspection revealed that one of the glass walls was actually a door, but again, not really designed with the end user in mind. My overall impression of the place was that it was a very large, very expensive place that was designed to make you feel uncomfortable and dumb as often as possible. Probably not what you really want from a hotel.

An oft touted phrase by martial artists is ‘enzan no metsuke’, which translates roughly as ‘looking at the far mountain’ - essentially the gaze you would use to view something very distant. If I am fighting an opponent and I focus too keenly on their sword, or the movements of their body only, then I am likely to miss movement elsewhere, the way they breathe, the way they distribute their weight, or lose the insight I would gain by observing that movement as part of the whole. An opponent who focuses too keenly on one particular area is easily distracted and hence defeated.If instead I observe my opponent’s eyes as though observing a distant mountain, I can use my peripheral vision to observe not only what the eyes themselves reveal, but everything else that is going on. With practise, I gain an wholistic understanding of my opponent’s movements and through that, their intent.

This is, in effect, exploratory testing. I don’t know much about my opponent, so I will observe everything I possibly can in order to expose a weakness, and then having done so, take full advantage. As rapidly as possible, you must determine what the appropriate action is.

Knowing one thousand tests does not make you a good tester, or even a tester at all. Anyone can learn to perform tests robotically. In any given situation, you might happen to run a series of tests, and you may even find bugs, but if you don’t understand why running the tests you did was appropriate, then you are not really testing. Likewise, a kendoka may strike an opponent, he may be victorious, but if he does not understand why his cut was successful, then he has improved neither his kendo nor himself.

When testing a new product, you’re looking not only to ensure that functionality works as described/expected (assuming for the sake of the example that this is what you are doing), but you are also on the lookout for the slightest inclination that something is amiss, or presents an opportunity to be taken advantage of.

You need to be aware that any action you take, and any action that the program makes in response to you can reveal information that you can use, even if you don’t understand why or how at the time. Initially this might simply be to use the product as it was intended, ensuring that the response is within acceptable parameters. Even if this doesn’t reveal errors, doing this should give you a number of ideas about what to try next.

You probably have a range of go-to tests - the usual suspects when testing certain types of things. In Japanese these are called your toku-ii waza (your best techniques). You take all the potentially vulnerable areas you suspect and go to work on them with your toku-ii waza. Because they’re techniques you know, you also know how things react to them, both when they’re working and when they’re not.

Either way, this is more information to go on with. If your toku-ii waza do not bear fruit, you might try instead to disguise your intent. Make one attack appear like another. If I look at my opponent’s head as though I intend to strike there, and then launch an attack, I may induce him to raise his hands in defence, at which point I can strike the wrists or abdomen instead.

The testing equivalent might be something as simple as renaming the extension of one file to another and opening it, or uploading it. It might be using regular expressions or executable script in the url or input form of an online form. If you abstract the concept, there are any number of ways one test can be made to appear as another.

Again, each response to each test you conduct tells you something more about the code under test. You must have enough focus that you are able to observe and take in the information you see, but enough detachment that you can correctly interpret it in a way that allows you to learn more. What I have described above is but one of many possible paths you might choose. The importance is the consciousness of that choice being appropriate to the information that precedes it.

One of the things I really appreciate about kendo is that we don’t wear coloured belts to denote rank. If I have never fought someone before, I have only what I can read from them to go by, be it how they carry themselves in the dojo, how they dress or the information I can gather from them when we cross swords. I am unfettered by any preconceptions I might have if I was able to see the colour of their belt. Because of this, I am free to attack them with full spirit and I am challenged to use my understanding of kendo to gauge their skill and fight them accordingly.

Testing is no different. When you are asked to test a piece of software you have never seen before, you should form your own opinion of the program’s abilities and complexity. Sure you may have heard a bunch of stuff about the program, but you should be prepared to discard or disregard it, especially if you suspect it may hinder the freedom with which you approach your testing. If you have been told that a piece of software is horribly complex and will do your head in, your response may be to try and perform tests of equal complexity or greater in an attempt to maximise coverage. You might also overlook the fundamentals of your craft - the simple things that can give you greater insights if not immediate results.
As an example, you may have been asked to compromise the security of an online login interface. ‘It’s super secure. You’ll never break in’, says the client. You might go to work with dictionaries and all sorts of wonderous script-kiddy play toys, but if you go back to basics you might learn something interesting. For instance, you might try logging in with ‘guest’ as a user/pass combo. If the test team is security lax, you could try combinations of ‘test / 1234′, or even ‘ or ” = ‘ to see what the db behind it will do. You could use the names and common details of the company itself, the company’s owner, executive team and or board members. That information is easy to come by. I’m not advocating you go try this stuff on the first online login you find, merely saying that simple is good. Fundamental is good. If it doesn’t give you results, perhaps it gives you something to work with.

Observe for yourself. Use what you have heard to supplement your understanding, not as a substitute for understanding. If in doubt, go back to basics.

At the other end of the scale, your preconceptions might lead you to believe that a relative novice will be a walkover. Sure they might only have a white belt on, but unbeknownst to you, they may have been training for years in several other martial arts. Looks can be deceiving. If you assume that someone will be a walkover, you might be right 99% of the time, but for that one per cent you are wrong, you *will* get your arse handed to you.

The same goes with testing. Something may at first appear very simple, take for instance a three radio button interface to a code profiler that switches between active, passive and coverage modes. Nothing too complex there. It looks like a white-belt, acts like a white belt - you switch modes, run the profiler, switch mode again, run again, observe, repeat - nothing too brain busting about that right?
Well, it depends. Maybe it is that simple. Maybe it’s just toggling a bit somewhere to tell the code to use different options. As soon as you assume that this is so, you disregard the possibility that the interface uses a very different style of code-fu for each option. If you don’t adapt your testjutsu to account for that possibility (and it’s 1 per cent time), there may be any number of people lined up to hand your arse to you.

When one talks about martial arts, inevitably the question of rank comes up. People love to know what rank you’ve achieved. One problem with this is that rank is somewhat irrelevant as an indicator to someone’s ability. There are varying standards of ability within a given rank for a given martial art, and there are often misconceptions of what that rank means by people unfamiliar with martial arts (not to mention confusion and devaluation caused by McDojos handing out black belts after a 2 week course).
Average joe non-martial-artist, when told that person x is a black belt (shodan), has a tendency to ooh and aah about how great they must be. There is no shortage of martial arts movies where people kowtow or are otherwise beaten up by the dreaded ‘black belt’. Anyone that has achieved a shodan worth holding will tell you that the only thing it means is that beyond a tenuous grasp of the basics, you have a burgeoning awareness that you know five eighths of bugger all. The vast majority of the journey still lies ahead.

It is with this context firmly in mind that I would describe myself as a software testing shodan. I have an understanding of the basics and I can see just how much I have to learn - and it’s a whole lot. It is my hope that this blog serves as a way to document my understanding and lessons learned about the real-world application of testing. I am not saying I have all the answers, or even that I am trying to teach. More than anything, I am trying to learn. You need to think critically about anything you read here. Further to this, I have a favour to ask of anyone reading this. At some point, I will probably say something that is completely and utterly nonsensical. If I do, please, please call me on it. I genuinely want to know.