Does Anyone Know What Testing Is?

(With apologies to Chicago for the title) I'm a big fan of good design which should be no surprise to those who know me. It can irritate my wife when I analyze a building and then start describing all the different ways in which you can tell it was poorly designed. I do rant when I see great design but those are much more rare in these "I don't care if it's a half-baked piece of crap, we need it ready now!" times. I apologize for any management flashbacks that caused.

In any event, I'm also a big fan of testing. I see testing scenarios around me during the day. This story just happens to be one that is easy to tell. The mens room where I work has three sinks, each with a soap dispenser. There was one sink that more heavily used than the others based on the fact that there was never any soap in the dispenser. You'd push on the plunger a couple of times and when nothing came out you'd shrug and move to another sink. I always figured that it was running out quickly since I would see the buildings cleaning crew in the bathroom a couple times a week. Surely they were refilling the dispensers right? After a few weeks of the "soapless sink shuffle" I mentioned the problem to the cleaning guy. I got his attention and showed him how no soap came out. He looked confused and said something to the effect of "it should work because it has plenty of soap". As I looked under the counter where he was examining the half-full bottle of soap it was clear that lack of soap wasn't the problem. I left him to his work, figuring that maybe the tube from the plunger was disconnected; plus, he didn't need me looking over his shoulder.

Upon reflection, I realized that his 'test' for being out of soap was looking under the counter to see if any of the containers were empty. Our 'test' for being out of soap was trying to get soap from the dispenser. So even though his test passed, it still was failing the end user because he wasn't verifying the expected result; the assumption being that the only reason you wouldn't get soap was if the container was empty.

The bottom line is to always be aware of what your tests are testing. Just because you're tests all pass does not guarantee everything is working as long as you aren't testing the way the system/device is used in real life. So, when you look at your test plan ask yourself, am I checking that we are out of soap, or am I checking whether I can get soap out of the dispenser? It might seem like a small difference but it can be difference between a cleanly running system and one that leaves you all wet.