This morning a colleague came up with an interesting question.
They had just finnished a testing round for a client. They ran 100 acceptance test cases and got a whopping 99% pass rate.
But the story doesn’t end there. While running the tests, they found 14 relevant bugs outside the scope of the original test cases.
Now my colleague faced a dilemma. If they wrote 14 new test cases based on bugs found, would everything then be ok?
My first impulse to most testing questions is “How would I know?”. Most questions are context dependent anyways.
Yes, I admit that this is the easy way out of the question. If you’ve read this far, you deserve a better answer too.
Based on my experience in testing, the first thing to consider in most situations is the outcome that the testing team wants to produce. Test cases, after all, are just a tool like any other. And tools are only a means to an end.
In terms of the project we were discussing, there were three outcomes to go for.
Option #1 – Produce evidence: If testing is to produce evidence about how a 100 specific things work, then the set of test cases already serves its purpose perfectly, and should not be changed.
Option #2 – Hunt new bugs: If testing is to find new bugs, then it seems to be a good idea to ignore test cases all together and focus testing to the area outside the scope of at least 99 passing cases.
Option #3 – Observe and prevent: If testing is to observe that things get fixed and never appear again, then it may be a good idea to add 14 new cases to the original set of test cases.
For you, there may be other possible outcomes too. Have you ever mapped the options with your project and your responsibility area?
Learn to guide both your own and the teams focus to the outcome you aim for. And if you want to take things deeper, learn to use the Navigator Principle!
Always begin with the end in mind. What is the primary outcome you need right now? After asking this, you’re more likely to find your answer to the original question too.
P.S. Are you a professional committed to excellence in software testing? If you are, I wrote this for you -> The Storytools of Testing -book gives you epic tools lead your clients, colleagues and bosses towards better software testing! And the best thing is that it’s on -50% discount until end of month here: prove.guru/thebook