testjutsu

Jared Quinert pointed out to me that I missed an opportunity to work in a good point about testing in my entry supporting the WGA. I’ve recently been struggling with the task of coming up with meaningful stats to report to my handlers to help justify the existence of my team and anyone with a brain cell would have noticed that talking about prodco execs massaging stats of WGA writers would have been the perfect segue into this. Apparently I still suck at the blogging thing. I’m sorry.

Let’s pretend for a moment that I was clever enough to put two and two together. Here’s what I might have said.

You can make statistics say pretty much whatever you want. This is potentially dangerous because in my experience, more often than not, your audience is passive won’t care at how you arrived at your final figures as long as the layman’s explanation sounds reasonable.

A tester worth their salt is different. They’re conditioned to ask ‘why’ to almost everything, but I have a sneaking suspicion that there are many people out there that accept any given number at face value as long as it’s not obviously outrageous, maybe because they’re afraid of numbers, maybe because it simply doesn’t occur to them to question why.

Let’s take the following statement on the AMPTP’s website:

According to WGAw, 4,434 of its working film and television members earned a combined $905.8 million in 2006. The average member earned $204,295 and over half earned at least $104,750. The WGA noted that these numbers are based on earnings reported for dues purposes and thus do not fully reflect above-scale payments.

In case you’re wondering where they got their information, they got it from a report like this one.

It would be very easy to read that statement and conclude that at least half of screenwriters earned over 100K last year - not a bad paypacket in most people’s book. Certainly the AMPTP seem quite content to leave it at that.

Let’s have a closer look though. One key word that jumps out at me is ‘working’. ‘4,434 of its working film and television members…’ - meaning that there is an unstated number of non-working film and television guild members. What is the guild’s total membership? According to the report I linked to above, it’s 7313 +/- 400.

So when they say ‘The average member earned $204,295‘ they mean 910m divided by 4,434, which they use to arrive at their average figure of around 200k. Let’s factor in the conservative figure of total membership - 6913 (7313 - 400).

910m / 6913 = $131,636

Very different to their figure of 205K. Still, 131K is still a pretty good wicket to be on, right? Maybe, but remember, we’re averaging wages earned by all across a large number and at least 2,479 of those people (36%) did not work at all during the fiscal year stated.

Let’s add a little more contextual information. As a writer, you might work steadily for a season on TV, or be paid for a script during one year, and then not work again for 12 months or more. The 36% of writers that didn’t work this year are predominantly not going to be the same people who don’t work next year (assuming the strike is resolved by then).

Taking our average figure of 131.6K and stretching that across two years, we’re suddenly looking at 65K and this is before we factor in tax. An agent will typically take 10 percent which further eats away at your nest egg. Suddenly the princely sum the AMPTP are touting is not looking nearly so princely.

Even if you were to say that the average writer works 64% of the time, then you come to a sum of $84,224, less agents fees - $75,800, less tax (let’s be generous and tax them 33%) and you arrive at almost exactly $50,000 - and remember I’ve been using conservative figures.

Are the AMPTP being deceitful? I’ll leave that for you to conclude. They’re definitely not giving us the entire picture, that’s a certainty.

As software testers, we’re taught to look for multiple explanations for problems. The map is not the territory. How often has your first conclusion turned out to be incorrect. If you’re anything like me, the answer is very.

I don’t think examining stats is very different in this regard. You need to look closely and think critically, whether or not there’s a neat little summary next to the numbers that tell you what you’re supposd to think. What the numbers don’t say may be just as important as what they do.

What contextual information may be missing? Can you drill further into the numbers and look more closely? Is there a heavy skew and a long tail hidden by a mean average? What does that mean?

Abstracting a little more, you can look at who has put the statistics together and why they might have chosen to present it the way they did. Do they have a particular bias or motive that may make them want to present statistics in a particular light? Someone with a KPI bonus for keeping bugs low might present a set of figures very differently than someone with a bonus based on the number of bugs found.

I am not at all surprised at the number of war stories one hears about statistics being misused and abused. It’s too easy to accept numbers at face value when the numbers look as good as you want them to, or present a slam-dunk answer to an argument.

I am going to do my best however, to make a point of being the guy who, when presented with stats, does dig deeper and shakes things to see what falls out. It might be a bit more effort, but it might also become a lot more interesting, not to mention valuable.

Writing for the screen has many similarities with software development. It is a highly collaborative affair, and one that requires patience, skill, clear communication and quite possibly a penchant for the masochistic.

A screenwriter has only 120 pages in which to tell their story. It sounds like a lot, but when you can write only what can be seen or heard, it rapidly becomes very challenging. Every word must drive the story forward.

The story has to be one that works on film. If the story depends on internal states (what people are thinking and feeling), it may be something that is better suited to a novel. If it’s dialogue-heavy, you might have a stage play on your hands. Not every story lends itself to being on film.

A writer of spec scripts (scripts that are unsolicited) is, whether they know it or not, writing for at least four audiences. Firstly the production company’s readers.

Readers are paid a pittance to wade through massive piles of screenplays that (literally) tens of thousands of other writers submit to Hollywood every year. They’re the ones who sift the gold from the shit (and 99.9% is shit). They’re hoping that the next 120 pages that they pick up is going to be the nugget they can plonk on a producer’s desk and say ‘we have to make this’, and of course they’re generally disappointed. Poor bastards.

As a screenwriter, you have to grab their attention from the first sentence - the first word if you can, then hold it. Keep them wanting to turn the pages even if it’s the 30′th script they’ve read that day and their eyes are red raw. A spec script has to be that good.

After that, the script has to grab the imagination of a director. If you take a look at a novice screenplay, it’s generally full of camera direction and references to music and such. This is a no-no for writing on spec.

It is the director’s job to call the shots, not the writers. Okay, so a spec script with this sort of direction will be binned by a reader before it ever gets to the director, but the point is, screenplay writing is collaborative and you need to know what other people’s jobs are, and let them do it. That doesn’t mean you can’t capture their mind’s eye and suggest implicitly what the right shot might be.

Let’s say for instance you have an action movie where you have a hostage strapped to an operating table about to be operated on, and your next scene is your protagonist tending his wounds after a battle. If the last sentence of your operating theatre scene ends with the hostage being anesthetised, and the first sentence of your protagonist scene is him having a bullet wound stitched shut, you’re not explicitly telling the director to superimpose the needles for the scene transition, but the inference is reasonably clear.

You’re not telling the director how to do his job, but there’s a suggestion there if he wants to use it.

The screenwriter needs to write for the actors as well. If you don’t have characters that test the range of an actor, that challenges them, they’re less likely to want the part. You need to know how to build tension and create conflict and show them the opportunity to display their acting chops.

The novice writer probably starts out with the movie-going audience in mind and of course they’re important too. You need to have a story that works on screen, that grips the interest and imagination of the viewer, holds it for 2 hours and leaves them feeling that the time they spent was worthwhile.
As a software tester reporting bugs, your role is not so terribly different (although hopefully on a smaller scale in terms of length). You also have several audiences and several relationships to manage.

You need to report your findings clearly and concisely with enough detail to adequately convey the bug and the environment and context surrounding it. Make it easy for the person reading your report. If there is some heavy lifting you can do for them, do it.

Is your bug worthy of reporting? Relate the problem to risk. Why is the bug worth fixing? What happens if we ship without it being fixed? Who benefits from it being fixed (or who is it detrimental to if not). Create a compelling case for the resolution of your bug.

Don’t tell a programmer how to do their job. If you have theories about the cause, or suspicions about a more serious underlying issue, then note it down, but preceding it with ‘you need to fix’ or ‘I would have written it like this…’, or ‘the whole thing needs rearchitecting’ is unlikely to win you many friends.

Like a screenwriter, once you submit your bug, it is out of your hands. You want to send it into the world with the best possible chance of success. A screenwriter wants their play to entertain (and also come back with a large cheque attached), and build their reputation as someone who produces quality work. You want your bug to come back resolved and build the same kind of reputation.

Understand who your audience is, what they are looking for, and make a point of providing it to them.

Cem Kaner has an excellent document on his site about bug advocacy. Well worth the read.

For any budding screenwriters in the audience, if you haven’t already devoured the wordplayer.com columns written by Ted Elliot and Terry Rossio, then your next eight hours are in this link.

Another non-software testing / non kendo post, but for what it is worth, I want to lend my support to some people that deserve it.

If you watch t.v. or see movies, you probably know there’s a strike on at the moment. I have a few friends who are participating. Your favourite television series might be delayed a good while because of it, but if you’re blaming those overpaid, lazy writers, then your ire is misdirected. What these people are asking for is not unreasonable (more detail here).

The AMTPT would have you believe that they’re the ones being hard done by; that because writers are paid for paid downloads and online pay-per-view, that not paying them for streaming video online is not ripping them off. They’d have you believe that the average working writer is taking home over $200K a year, based on some sums they did on numbers from the WGAw.

This ain’t necessarily so. I am led to understand that this is a more representative explanation of how screenwriters are paid. It looks to me like the AMPTP is massaging the facts in order to put a very particular spin on events. I’ll admit, my viewpoint is not unbiased. I want my friends to be better off.

I’m tempted to draw parallels between this and the recording industry where artists are beginning to break away from the draconian Establishment and distribute on their own terms. Unfortunately for the screenwriters their relationship with the prod co’s is a symbiotic one, as there are many more moving parts involved in putting a movie together. The AMPTP seems a little hazy on the difference between symbiotic and parasitic.

I do hope for their sake, but more particularly that of the people on the picket line that they work it out soon.

It’s easy to tell yourself that you’re training hard when in actual fact you’re slacking off. I find myself doing it far too often for my liking. If in the kendojo I’m fighting beginners, I might fend off attacks in order to get them to try and work out why they are unsuccessful. Nothing necessarily wrong with that, but if that’s all I do, then I’m slacking. More effective for the both of us would be if I tried to fight at a level slightly above theirs and encourage them to rise to that standard.

If I’m not really trying, I notice myself falling into bad habits. I let my feet move move lazily, I block without trying to counter, if the first strike fails, I may not immediately make a second or third effort (all of these rob you of the ability to take advantage of opportunities) - these things are physical manifestations of mental laziness. When I see others being lazy, I take them to task for it. When I find myself doing it, it annoys me. It makes me feel like a hypocrite.

If I don’t put in the effort, then I don’t learn anything, but possibly worse is that my training partner isn’t learning anything, and if you think they don’t pick up on the fact that you’re not putting in 100%, think again. I don’t want to be the person that teaches others that a half-hearted effort is okay. Moreover, by doing this, you are saying ‘you have nothing to teach me’.

You can learn from anyone. This has been proven to me time and again, both in the software testing field and in kendo. Insights can come from the most unlikely sources. Think about the questions people ask (before you rattle off a rote answer). Observe what they do and ask them why.

It’s not easy to alway make the extra effort, but it is something you can train yourself to do. Like anything, there’s that initial force of will required while your brain rewires itself to a new standard, but the more you do it, the more you’re able to do it. Doing it consistently in practise means it becomes second nature when it counts.

Practise doesn’t make perfect, perfect practise makes perfect. Don’t know who coined that one, but they were right.

There are always things you can find to work on. If you find yourself with the luxury of time, or your domain knowledge outstrips the challenge at hand, look for ways to challenge yourself. What is it you are trying to achieve? What can you do to make that achievement something that helps you to grow also?

Not necessarily easy questions to answer. The answers you come up with may not be easy to implement either, but if you can identify something challenging in what would otherwise be routine, then surely it’s worth it.

Forgive me if I’m being overly abstract. I shall try for something more practical in my next entry.