katte, kabuto no o wo shimeyo.
When you win, tighten your helmet strings.
This saying is a reminder to remain ever vigilant. In battle, victory over an opponent should not be a signal to let your guard drop. If anything, it should serve to remind you to be even more wary. Defeating one is no guarantee that there aren’t more around.
Have you ever signed off on a product that you were confident was sufficiently tested only to find out there was something you missed that you might have caught if only you’d thought about it just a little bit more (or just a bit differently)? Sure you’re never going to find all the defects, but the ones that burn are the ones that you didn’t find but know you could have.
Time and circumstances permitting, when you’re ready to sign off on your test effort, stop and think about it for a little while. Ask questions, either of colleagues, or of someone completely unrelated – you’d be surprised how often someone with a fresh perspective points out something obvious. After you’ve been testing something for a while, there’s a tendency to do the go-to tests, look at the usual suspects – to maybe coast a little. Especially if your test suite has been coming back green. It’s really at this point you need to tighten your helmet strings.
Are there any new risks you can think of? Can you check the reliability of your oracles? Have you covered the important areas? Sufficiently? How do you know? (and on and on)
I’m not trying to give you the answers. I am saying that when things look like they’re going swimmingly, you need to be asking yourself the hard questions – and making sure you have answers that stand up.