testjutsu

There has been an interesting thread going on in the yahoo software testing group to which I subscribe. The topic began as a discussion about software development as art, but was quickly shoehorned into a discussion about productivity. Jared Quinert blogged a response to this recently which got me thinking again about tester advocacy. I haven’t made much progress on the code of conduct front (though the principles for the practise of craft resonate strongly with me), but I think being proactive in the education of others is up there on the list, and the discussion around the meaning of productivity helps to highlight it.

For those that are too time-poor, a potted summary of Jared’s post might read ‘productivity - appropriately defined - is the key to defensible testing. Problems arise where productivity is seen as synonymous with output / effort as opposed to the gathering of helpful information’.

I agree with this, and particularly with the qualification ‘appropriately defined’. I think the danger of (mis)applying the term ‘productivity’ to testing is that the audience expects a ‘product’ at the end. Some sort of artefact that they can point to as proof that testing time was time well spent.

In my experience, the test report is often the thing by which tester productivity is measured. Not necessarily the testing that the report represents, but the report itself. Is it then not incumbent upon the tester to ensure the content of the reporting clearly displays the value of the test effort? Both the actual interaction with the code as
well as the thought and planning that goes around it.

In assisting the creation of a software product, we are generally measured by our productivity towards that end. If the people we are reporting to have a misdirected sense of what productivity means in relation to testing, I believe the responsibility is ours to correct it. How that happens really depends on who you’re reporting to and what they’re looking for.

If the thinking goes ‘a developer produces code, then what does a tester produce?’ - then the tester in my opinion is probably being set up for an unfair comparison. If I were forced to give an answer to that question it might be ‘information’. I want to make damned sure
that the information they get out of the testing I have done is valuable to them in some way. If that makes me a producer of information in their eyes, so be it.

Another way to look at it is to ask ‘If there were no testers, what else would you not have?’ If your audience can’t answer that, then I think your problems are more fundamental than their understanding of what you produce.

I hesitate to say that I’ve been sitting on this post for a while. Let’s just say I’ve been giving it some thought.

At my current place of work, because we host our product online the testers, beyond testing the code pre-release necessarily have to have to support this code once it goes live.

This leads to a situation where the more that is deployed, the greater the scope of support testing becomes. Pile on top of this the joys and complexities of feature requests and bug reports that inevitably come in along with a weekly deployment cycle and you have a recipe for what recruiters like to call a ‘fast paced work environment’.

Our testing team grew organically, but rapidly from a single person to two, three, then five. Each tester would join the team, skill up on our products, environment and processes and then gravitated to their own areas that they were particularly good with. We had the tendency to do work on those areas fairly exclusively. Having testers who were product area experts was good in that we turned requests for testing around quickly, and were able to troubleshoot problems at lightning speed.

Over time however, problems started to arise. Testers started to miss problems, sometimes very obvious ones. When one of the team was away, we occasionally struggled to do the work that was normally assigned to them. We knew the products we were working with, but new additions and changes might mean that the product was significantly different if we hadn’t looked at that area for a couple of months.

With around sixteen different product areas to support (the majority of which are still being actively developed) it was also difficult to make sure that all these areas were getting the attention they required.

I spent a lot of time thinking about how to solve these issues. Up to that point, the testing team was seen as an highly efficient unit. Of course it only takes a few small issues before people begin asking pointed questions. I was asked some pointed questions and felt some sense of urgency to overcome the situation.

When I broke down the problems into their components, the answer became obvious. Issues that I expected testers to be picking up were being missed due to fatigue through repetition. If you look at something frequently enough, there’s a tendency to skip over things.

It’s like driving a car. When you learn to drive, you are painfully aware of everything. Tyre check, clutch, gearstick, accelerator, speedometer, indicator, head checks, and so on. As you become more accustomed, changing gears goes from a precariously balanced set of bodily maneuvers to a smooth singular action. Same with changing lanes, or overtaking.

You can become blase about it though. Miss the clutch changing a gear - grind the gearbox. Miss the speedo when on the freeway, cop a speeding fine. Miss a head check when changing lanes, kill everyone in your vehicle.

Okay, maybe a little dramatic, but you get the point. Repetition and subtle change don’t go together if that subtle change needs to be noticed immediately.

Another issue was over-specialisation. While we had people who could troubleshoot and test their area of specialty, if they weren’t there, then the testing process took noticeably longer because the other testers had become unfamiliar with that part of the product.

Lastly, because people gravitated toward what they were good at doing, and because the amount that we had to support was growing steadily, there were little things that started to slip. There wasn’t a clear enough delineation of responsibilities for the testers to follow.

When I thought about how to solve the problem of over-familiarity and under-familiarity, and slippage, it seemed that a rotation system was the obvious solution. Rotate the testers through all of the product areas on a regular basis.

There were a few things in terms of logistics to get right. How long should each rotation be? How do you reach an equitable split in the workload between testers? Should each area of the rotation contain related products, or should there be a distribution of different products within each area?

We experimented with a few different options before arriving at a solution that worked well. With our rotation split into four areas, we found that having related items grouped into a single rotation worked better than splitting them up. I suspect that this is because each rotation is quite different, as opposed to having four rotations that are variations of each other.

Four weeks per rotation was too long. Two weeks was too short. We finally settled on three weeks per rotation and this has worked well for us. Of course YMMV depending on how complex the animals you’re testing are and how actively they’re being changed.

It hasn’t all been smooth sailing. There is occasionally confusion for people interacting with testers being thrown by who is in which area of the rotation, but that’s more of an information flow thing, really.

The rotation relies on the testers being able enough to work autonomously and being diligent enough to put their hand up when their workload requires assistance. When testers are absent, it is a reasonably simple matter to distribute their workload amongst the testers either side of that rotation.

It has turned out to be an excellent training tool. With such a vast array of product areas to learn, the rotation breaks it down into manageable chunks that one can get to grips with before moving on.

We review the layout of the rotations every six months or so and occasionally make adjustments where the workload of one has increased at the expense of the other. There are also occasions where projects that finish create an influx of tickets for a particular area. We keep track of the feedback/bug reporting rate around these releases to try and determine whether it looks like being a long-term increase in work load or not.

As rotation time rolls around, the testers brief their replacements on where the product is at, what changes have taken place and what the incoming tester(s) may need to be aware of. Any near-complete tasks are kept by the tester that started them, rather than bringing someone else up to speed.

Overall it has been a massive help to the team, and allows me as a manager to more easily keep tabs on the testers’ workloads. I don’t know how well a rotation would work for shorter projects, but if you’re struggling with some of the issues I have described above for long-term support projects then you might like to consider it.

Mad, I tell you. Well, maybe not, but I’m making an effort to get out and about.

I’m going to be in Japan for a couple of weeks in early July, so when I heard about the CAST conference happening around the same time in Toronto, I figured that since I was sort of in the area (well, closer than I would be if I stayed in Melbourne), I might hop across the pond and check it out.

I’ve not been to Toronto before, so that’ll be fun and I hear all the cool kids have been invited.

I did find something interesting while signing up.

The link offering to tell me why 123signup.com wanted my email address up-front instead gave me a nice little stacktrace. Now maybe I’m being a bit of a wet blanket, but I tend to hope that the people I give my credit card details to know enough about how to configure a webserver to turn the error display off.

I didn’t have time to poke around any more than the screengrab that I took, but hey why not sign up. If you have time to be reading this blog, you probably have time to poke around a bit during the signup process and let’s face it, enough disposable $ to jet off to Canada.

If you do sign up, I hope to see you there.

I’m going to be attending CITCON coming up in Melbourne at the end of June.

There will be a bunch of people from all different tech areas to discuss CI and the testing thereof. It’s an ‘OpenSpace‘ format, so I’m looking forward to seeing what sort of things people want to discuss and hear about.

It’s a free seminar and I have it on good authority from people who attended last year’s event that it is highly worthwhile. The event is limited to 150 places and I believe many of them have already been filled, so if you’re considering attending then I’d act sooner rather than later.

I hope to see you there.

After blogging about tester advocacy I began to wonder whether I could come up with a code of conduct that testers could use to guide their actions and decisions. I don’t think you’re likely to get a one-size-fits-all code, so perhaps I should say a collection of heuristics by which a tester can guide his own actions.

I’m not talking so much about the day-to-day activities of a tester, though that certainly plays a part. I’m more concerned by the actions in his relationships with his peers, his employer and the people whom his testing may represent. I was hoping that with some careful consideration, I’d be able to put together a concise list of points.

I started listing the sort of things that are handy to remind myself about, but I quickly realised that they aren’t really that useful in all situations.

I listed things like

I will pro-actively educate myself in the art of testing.
I will find out who matters and test and report accordingly.
I will use all the resources at my disposal
I will not be the gatekeeper of the decision to ship/not ship

The problem with these is that while they work for me right now, they may not work for testers in other places, or with differing levels of experience, different duties, different responsibilities. The other problem with a list like the one above is that you could easily expand it to a hundred points or more.

I don’t want a laundry list, or a bunch of ‘thou shalt’ commandments. I am also wary of locking the art of testing into some sort of cookie-cutter formula/recipe to be followed by rote, rather something that is applicable in all situations that any tester can use to guide their thinking and actions in a positive way. This appears to me no small task.

I put it to the MAST list to see what they had to say about it. There was some interesting and spirited discussion around the term ‘trusted advisor’, but what I took away from the conversation was that Paul Szymkowiak’s suggestion the exercise of attempting to define such a code of conduct is perhaps as important (or more so) than the results.

It is still something that I am spending time thinking about, but if the journey turns out to be as important as the destination, then perhaps someone else out there might gain something if I document my endeavours here. To that effect, I shall do my best to post updates here as and when I make progress on this front.

I have been reading Matt’s blog for a while now. Since October last year, Matt has been periodically expanding and evolving his thinking about ‘technical debt’ - the small, seemingly inconsequential short-cuts and hacks that we tend to implement or let slide in order to hit deadlines or save us doing more work (at least in the short term). Like most bad habits (such as the gradual accrual of fiscal debt, say, on a credit card), these things tend to lead to large amounts of pain down the track.

Rather than regurgitate what Matt has been saying, let me point you at his work. I think it is worthy of consideration for anyone who has had to interact with another human being in a professional capacity, ever.

• Technical Debt #1
• Technical Debt #2
• Technical Debt #3
• Technical Debt #4
• Technical Debt #5
• Technical Debt #6

If you’ve skipped to this bit of text because you’re not up for six essays on an unfamiliar concept, let me recommend that you at least read #6 before checking out details on Matt’s upcoming Technical Debt Workshop.

If you’re in the vicinity and have something you can contribute, then why not do it? If you’re not, then keep an eye on Matt’s blog. I’ve no doubt there will be some very interesting outcomes from the workshop.

The company I work for has been using subversion for several years now and it suits our needs reasonably well. It’s generally pretty easy to use; it’s certainly an improvement on CVS, but on the other hand there are some things that aren’t all that great. The way externals were implemented leaves much room for improvement and svn tends not to play nicely with mixed user permissions (such as when files are written alongside or over checked-out code by www-data). Subversion has a relationship with posix ACLs that is rather less cordial than the one described in The Pogues’ Fairytale of New York, but these are perhaps war stories for another time. These errors at least have the decency to fail with the requisite amount of melodrama.

More recently we’ve come across some more subtle and potentially nasty issues that subversion barely bothers to notify you about, if at all.

The way our shop works, we have a single trunk from which branches are forked. Changes are made on-branch before being tested and re-integrated into the trunk. Problems can arise when you make changes to a file on one branch (A), and move the same file on another (B).

Subversion treats moving files as a ‘delete’ and an ‘add’ (delete the old file, add a new file with the same contents and a different name).  If you integrate branch A and then branch B, the newer changes from branch A will be lost (as the file is deleted) and the branch B version will be instated with its new name. SVN does not warn you that a newer instance has been overwritten - charming.

If on the other hand, you merge branch B first, then branch A, then because subversion can’t find the the file to merge with branch A (because it was renamed in B), it simply reports that the file is ’skipped’. No mention that changes to that file could not be merged as the file was not found just ’skipped <filename>’.

Easy enough to miss when you have several hundred files scrolling past you at a rapid rate of knots.

Apparently, the best you can do is be very, very careful when integrating branches that contain deleted or renamed files. I’ve taken to pointing any sizeable svn output to a file and grepping the hell out of it.

A similar thing happens when you use nested branches.  This one is perhaps more understandable, but the level of reporting is just as woeful. If you branch from the trunk, then make a second branch from the branch then care must be taken if you attempt to merge the second tier branch directly back to the trunk. If you have added files in the second tier branch, they too will simply be ’skipped’ if you attempt such a merge.

Merging back to the first tier branch is no problem, and is probably what the system was designed for. We discovered this issue when a developer created the nested branch to make newer updates to a very old branch. A better solution would have been to create a fresh branch from the trunk and merge the changes from the old branch into it, before making the required modifications.

Hopefully these are useful to some of you other SVN users out there.

I used to think that as a software tester, ‘if you’re doing your job well, then no one notices’. This sort of thinking was something I was perpetuating by misunderstanding my role as (merely) someone who performs test execution. I could see the benefit of the work I was doing and assumed that because no one else seemed really interested, they didn’t care.

From my experience, there is a tendency for testers to be thought of as second-class dev-team citizens, if not overtly then subconsciously. A developers job is perhaps easier to visualise than that of a tester. I’m not saying their job is easier, but for some reason a coder implementing programming algorithms by writing code seems easier to picture than a tester implementing testing algorithms by… ‘you guys just click on stuff to make sure it works, right?’.

I spent a lot of time arguing for recognition of testers and questioning why other members of the dev team seemed not to understand how important testing was. It was frustrating to me. It seemed like more often than not, when people took notice it was because something had gone pear-shaped because it was my fault (I’m the quality guy, right?).

It slowly dawned on me that it wasn’t because they weren’t interested, but because the way I was conveying the how, what and why of testing was not compelling. My colleagues knew development. They understood development, but their experience with software testing was limited. It’s not that they didn’t value the test team, but the way I was reporting my testing wasn’t as relevant to them because it was more or less a binary ‘Yes I have found problems’ or ‘no I haven’t’. This had a dangerous side effect of making me a defacto deployment gatekeeper.

They valued the ‘yes’ or ‘no’ a great deal, but because that’s all I was providing, and because that was their experience with testing, that’s all that was expected. I was thinking hard about testing and how the product might break and what else might go wrong, but I wasn’t really focused on how to get that across to the people that mattered. They assumed that when I said I was done, I knew enough to know that was true. To be honest, at that point, I didn’t know enough to argue otherwise.

As my experience in testing grew, I realised that test execution is a small (but of course important) facet of a tester’s role. I began to see the testers role as multifaceted and I became part devils advocate, part psychopomp and possibly large amounts of the damned from various circles of Dante’s inferno :). I pushed to be included earlier in the project lifecycle. I asked questions, made suggestions, asked for documentation and information and made sure the people I was working with were across the thinking I was doing about risk and risk mitigation. I started talking in terms of how the testing we were doing was covering the risks we’d raised (and sometimes how it wasn’t). I didn’t always go the most diplomatic way about it, but gradually things began to change for the better.

The role of a tester to me now is to really be a central information hub, and a representative for those who may not have a voice (such as the end user). Reporting the test effort should be a matter of knowing who your audience is (or more broadly, who matters) and what matters to them. The development group I work with now have come to expect assessments of risk, analysis of proposed solutions - the thought behind the effort is as important as the effort itself.

I suspect that my early experiences are not uncommon. I have heard and read many accounts of testers feeling undervalued and misunderstood, being blamed for errors that go into production, being the gatekeeper for the decision to deploy. I don’t think testing is nearly as understood as a profession as coding is and it’s something I’d like to change.

I think the onus is on the software tester to be proactive in promoting their effort in such a way that it adds value to those that matter; so that the people they are supporting understand exactly what the tester’s role is. I think there is a need for education both of non-testers, to show them the value of software testing, as well as education of software testers so they are able to convey a compelling story of what their testing is and does and why it matters.

My situation is of course going to differ from other testing shops out there, both in the style of work and the relationship with developers in the development or project group. There may be an element of ‘picking your battles’, or making change in small increments over time. Testers might face extreme reluctance or hostility to change. I think at that point it becomes a matter of the tester being clear on the sort of behaviour they are willing to commit/adhere to, and those that they will not. More on that soon.

I am reasonably certain that somewhere in the world, a tester is being blamed for not finding this.

I am a little pressed for time right now, but I have been thinking a lot lately about the perception of testers by the people they support and what that can mean in the context of the projects they work on, the companies they work for and testers as individuals. I think there’s a rich vein of blogworthy material to mine here, which I promise to attempt just as soon as time permits.

Jared Quinert pointed out to me that I missed an opportunity to work in a good point about testing in my entry supporting the WGA. I’ve recently been struggling with the task of coming up with meaningful stats to report to my handlers to help justify the existence of my team and anyone with a brain cell would have noticed that talking about prodco execs massaging stats of WGA writers would have been the perfect segue into this. Apparently I still suck at the blogging thing. I’m sorry.

Let’s pretend for a moment that I was clever enough to put two and two together. Here’s what I might have said.

You can make statistics say pretty much whatever you want. This is potentially dangerous because in my experience, more often than not, your audience is passive won’t care at how you arrived at your final figures as long as the layman’s explanation sounds reasonable.

A tester worth their salt is different. They’re conditioned to ask ‘why’ to almost everything, but I have a sneaking suspicion that there are many people out there that accept any given number at face value as long as it’s not obviously outrageous, maybe because they’re afraid of numbers, maybe because it simply doesn’t occur to them to question why.

Let’s take the following statement on the AMPTP’s website:

According to WGAw, 4,434 of its working film and television members earned a combined $905.8 million in 2006. The average member earned $204,295 and over half earned at least $104,750. The WGA noted that these numbers are based on earnings reported for dues purposes and thus do not fully reflect above-scale payments.

In case you’re wondering where they got their information, they got it from a report like this one.

It would be very easy to read that statement and conclude that at least half of screenwriters earned over 100K last year - not a bad paypacket in most people’s book. Certainly the AMPTP seem quite content to leave it at that.

Let’s have a closer look though. One key word that jumps out at me is ‘working’. ‘4,434 of its working film and television members…’ - meaning that there is an unstated number of non-working film and television guild members. What is the guild’s total membership? According to the report I linked to above, it’s 7313 +/- 400.

So when they say ‘The average member earned $204,295‘ they mean 910m divided by 4,434, which they use to arrive at their average figure of around 200k. Let’s factor in the conservative figure of total membership - 6913 (7313 - 400).

910m / 6913 = $131,636

Very different to their figure of 205K. Still, 131K is still a pretty good wicket to be on, right? Maybe, but remember, we’re averaging wages earned by all across a large number and at least 2,479 of those people (36%) did not work at all during the fiscal year stated.

Let’s add a little more contextual information. As a writer, you might work steadily for a season on TV, or be paid for a script during one year, and then not work again for 12 months or more. The 36% of writers that didn’t work this year are predominantly not going to be the same people who don’t work next year (assuming the strike is resolved by then).

Taking our average figure of 131.6K and stretching that across two years, we’re suddenly looking at 65K and this is before we factor in tax. An agent will typically take 10 percent which further eats away at your nest egg. Suddenly the princely sum the AMPTP are touting is not looking nearly so princely.

Even if you were to say that the average writer works 64% of the time, then you come to a sum of $84,224, less agents fees - $75,800, less tax (let’s be generous and tax them 33%) and you arrive at almost exactly $50,000 - and remember I’ve been using conservative figures.

Are the AMPTP being deceitful? I’ll leave that for you to conclude. They’re definitely not giving us the entire picture, that’s a certainty.

As software testers, we’re taught to look for multiple explanations for problems. The map is not the territory. How often has your first conclusion turned out to be incorrect. If you’re anything like me, the answer is very.

I don’t think examining stats is very different in this regard. You need to look closely and think critically, whether or not there’s a neat little summary next to the numbers that tell you what you’re supposd to think. What the numbers don’t say may be just as important as what they do.

What contextual information may be missing? Can you drill further into the numbers and look more closely? Is there a heavy skew and a long tail hidden by a mean average? What does that mean?

Abstracting a little more, you can look at who has put the statistics together and why they might have chosen to present it the way they did. Do they have a particular bias or motive that may make them want to present statistics in a particular light? Someone with a KPI bonus for keeping bugs low might present a set of figures very differently than someone with a bonus based on the number of bugs found.

I am not at all surprised at the number of war stories one hears about statistics being misused and abused. It’s too easy to accept numbers at face value when the numbers look as good as you want them to, or present a slam-dunk answer to an argument.

I am going to do my best however, to make a point of being the guy who, when presented with stats, does dig deeper and shakes things to see what falls out. It might be a bit more effort, but it might also become a lot more interesting, not to mention valuable.