My regular reader might have noticed that I (re)read a book called ‘Excession‘ recently. Penned by a gent called Iain Banks, who in my opinion, can excrete pure gold.
It is a great read. For those of you already familiar with his work, it probably needs no introduction. For those yet to experience it – if you have any time for sci-fi novels at all, then your life is not complete until you have read this one as well as ‘The Player of Games‘ and better yet ‘Use of Weapons‘.
In Excession, he coins the term ‘Outside Context Problem’ that he describes thus:
An Outside Context Problem was the sort of thing most civilisations encountered just once, and which they tended to encounter rather in the same way a sentence encountered a full stop. The usual example given to illustrate an Outside Context Problem was imagining you were a tribe on a largish, fertile island; you’d tamed the land, invented the wheel or writing or whatever, the neighbours were cooperative or enslaved but at any rate peaceful and you were busy raising temples to yourself with all the excess productive capacity you had, you were in a position of near-absolute power and control which your hallowed ancestors could hardly have dreamed of and the whole situation was just running along nicely like a canoe on wet grass… when suddenly this bristling lump of iron appears sailless and trailing steam in the bay and these guys carrying long funny-looking sticks come ashore and announce you’ve just been discovered, you’re all subjects of the Emperor now, he’s keen on presents called tax and these bright-eyed holy men would like a word with your priests.
It even has its own wikipedia entry.
I’m sure that any tester that has been around the block once or twice has experienced a situation where they were blindsided by a problem (quite possibly after the system under test went to production) that made them think something along the lines of ‘Wow – never in a million years would I have thought of that as a possibility’.
The Outside Context Problem (OCP) is a little different to the SEP heuristic, in that rather than mentally deleting problems that appear to be in someone else’s jurisdiction, OCP describes those things that are nigh on impossible to factor into your risk analysis simply because you have no concept of it prior to it happening.
Of course this has particular relevance to software testing – we’re supposed to be the ones who can think outside the box; who can come up with the oddball issues that the developers and the stakeholders and the analysts didn’t think of. So how do we factor in OCPs to software testing?
Personally, I don’t know that there’s a definitive answer to that question. You can factor in extra testing time for that X factor, you can look at similar projects and products where applicable, to see if there were things that they didn’t account for, you can get advice from your peers, you can do all sorts of end-user testing to gauge how they might (mis)use the product and so on, but depending on what it is you’re testing you might have one, all or none of these things at your disposal. They still won’t guarantee you coverage of an OCP.
We encounter OCPs often when we are still learning and hopefully less and less over time, but you can’t ever completely eliminate the possibility of another one coming along. It’s why a one-size-fits-all standard simply won’t work for the software testing industry.
Beyond there being a multitude of known differences to factor in from one project to another, you have any number of unknowables that no amount of planning will allow you to say ‘I’ve covered all the bases’.
In terms of the OCP, perhaps the best you can do is cover your arse. If you can list the risks you have identified and present them to your audience, get sign off that the set of identified risks is acceptably comprehensive, then as long as you have done the best you can with the resources you have at hand, there is not much more that you can do.
That’s probably cold comfort for people working with virulent strains of bacteria or virii, or any sort of system where lives depend on getting it right the first time. You might be able to legally cover yourself, but you still have to sleep at night and I wonder how many people following a catastrophe (that they didn’t see coming) do not wonder ‘Is there something else I could have done?’