Testing and the Parable of the Painter

One of the many stories from the real world is having a coworker who doesn't understand the idea of how to test an application.  When asked to write a test program, they think that means writing a second program with the same logic - which is redundant of course - and any other approach makes no sense.

Here is a story to tell these folks.

There were two painters who kept busy painting houses around town. The painters were brothers named Jerry and Dale. They'd come by the house to be painted and the owner would tell them what color to use, the people in town didn't have eclectic tastes which made it easy for the men, the choices were: white, blue, red, or yellow. They had both been painting for many years and made a comfortable living this way.

A retired sea captain moved to town and bought an older house that was in dire need of a paint job.  He hired Dale to paint his house with instructions to paint the house ocean blue. The captain had bent Dale's ear about how the sea was his first love and she showed many different faces and that you could encounter waters of midnight blue during a storm or nearly turquoise near a shallow lagoon on a sunny afternoon.

Dale goes back to his shop, picks up his blue paint, telling himself that blue is blue, and heads off to do the job. When he's done the captain comes around, takes one look at his house and exclaims, "I've never seen the wild ocean look like that! I said ocean blue, not sky blue!".  The captain dismissed Dale and refuses to pay.

Dale returns home and grumbles to his brother Jerry about the whole affair, "If he wanted a specific shade of blue he should have said so before I painted the whole house!"

The next day, Jerry is surprised to find the very same captain at his door asking to have his house painted a, "proper shade of ocean blue."  Jerry, with the hindsight of his brothers experience, takes the captain to his paint shed and quickly mixes several different shades of blue until the captain smiles and says, "That one reminds me of calm seas off the coast on a sunny day, I'll take it.".  Jerry splashes the chosen color onto a thin peice of wood and the two men agree on a price. The captain directs Jerry to get started the next day while he is away at the docks helping a friend repair a fishing boat.

The work goes quickly and when the house is done Jerry goes to fetch the captain to ask him to come see the finished product. On the way to the docks the sky darkens and starts to fill with ominous clouds, a storm is headed this way. Jerry finally reaches the docs and finds the captain in a sour mood, the captain snaps, "It had better be the right color, I'm not going to live in a house that doesn't remind me of the fickle sea."  Upon seeing his house the captain throws a fit, "That is not the color of the ocean! What kind of painter are you? You're not going to get one cent out of me for such a lousy job". He continues his verbal abuse until Jerry pulls out the peice of wood with paint splattered over it and holds it up saying, "You recognize this?  This is the color you agreed to before we started.  When I hold it next to your house, you can clearly see that they are the same color.  You might have changed your mind but I've held up my end of the bargin and expect to be paid fair-and-square.  If you want it painted some other color, I'll be happy to add as many coats as you have money to buy, but you'll have to pay for this one first."   

The moral of the story? Agreeing on the outcome beforehand is the only way to prevent any arguments over what 'done' means at the end.

Discontinuous Integration

Refactoring is becoming a commonplace term thrown around these days. The expectation is building that refactoring is something that all good developers do and if you don't, you must be in one of "those" types of companies. You know the ones, they don't use a version control system or have a repeatable build, i.e. Continuous Integration, much less a well-defined deployment procedure.

If you are lucky enough to be building the first version of a product, you can refactor to your hearts content. There is no existing code to worry about breaking. If you are changing an existing system refactoring isn't always simple or straightforward. Even if you have the most automated CI and 100% code coverage of unit tests you'll still face the practical issues of deployment and business dependencies.

Let's say you operated the worlds most modern airline reservation system. It's written in the latest language and has all the buzzwords, highly available, highly reliable, highly scalable, etc. It has tons of automated tests so you are 100% confident that nothing will break. Now, since you have business partners like travel agencies and customers looking to book a flight or retrieve flight information, they have a stake in your deployment. The customers may tolerage a "Down for maintenencem, come back later." message but your partners will take a dim view of your downtime. They'll insist you tell them in advance so they can prepare any necessary changes in their systems or alert their support staff in case there are issues after the deployment.

Another example would be a large back-office accounting system. Whatever level of confidence you have in your code. you still have to deal with the myriad inputs and outputs of the system and disrupting the processing schedule for a code migration. No amount of automated testing/build/migration will ever solve these types of complexities.

The point is that considerations like these limit the opportunities to deploy code changes and increase the cost of change even when that change is guaranteed to work.

Refactoring is good but there are practical considerations why it's not done to the degree it could be in some shops. Keep that in mind next time you want to pass judgement on the perceived lack of refactoring somewhere.

An Estimate By Any Other Name

Terri Vajdi, a colleague of mine, sent me this response to my post on estimating. She always had the most thorough estimates I've seen in my career, so I have a lot of respect for what she has to say.

Since I drill down to assume the details of the requirements when it's missing, I equate this to all the missed requirements to a certain percentage. I read the requirements and try to envision the amount of code, junits, number of iterations, and unit testing for each iteration, Code Review and implementing the changes based on the Code Review, etc...
So, I come up with an estimate for each of these and then multiply it by the percentage/factor for the missing pieces. Add this the estimate for the missing pieces to my estimate to come up with the estimate that I turn over to my manager. In majority of cases, I break even and come close to the estimate that I have given. Of course, this is with putting a lot of overtime to meet my estimate. Remember we do not live in and ideal world and there are a lot of mishaps. This is just my two cents. Of course, I'm a quality oriented individual and I do not like to revisit my code once it's in Production. I figure that number of iterations after it has gone to Production is very very expensive. I'd rather spent it up front. So far, I've been able to stick to my methodology and have been able to spend my time wisely on other assignments once the code is in Production.

Sometimes I'm too eager and go light on my estimates which would mean even more work later and for which I've been criticized when the unknown issues would crop up and the project would run late. Terri has the right idea. She passed along this link:
http://www.apl.jhu.edu/Classes/635431/felikson/Publications/No_Silver_Bullet_Brooks.html

Anyone else have "Estimation Stories" to share?

Does Anyone Know What Testing Is?

(With apologies to Chicago for the title) I'm a big fan of good design which should be no surprise to those who know me. It can irritate my wife when I analyze a building and then start describing all the different ways in which you can tell it was poorly designed. I do rant when I see great design but those are much more rare in these "I don't care if it's a half-baked piece of crap, we need it ready now!" times. I apologize for any management flashbacks that caused.

In any event, I'm also a big fan of testing. I see testing scenarios around me during the day. This story just happens to be one that is easy to tell. The mens room where I work has three sinks, each with a soap dispenser. There was one sink that more heavily used than the others based on the fact that there was never any soap in the dispenser. You'd push on the plunger a couple of times and when nothing came out you'd shrug and move to another sink. I always figured that it was running out quickly since I would see the buildings cleaning crew in the bathroom a couple times a week. Surely they were refilling the dispensers right? After a few weeks of the "soapless sink shuffle" I mentioned the problem to the cleaning guy. I got his attention and showed him how no soap came out. He looked confused and said something to the effect of "it should work because it has plenty of soap". As I looked under the counter where he was examining the half-full bottle of soap it was clear that lack of soap wasn't the problem. I left him to his work, figuring that maybe the tube from the plunger was disconnected; plus, he didn't need me looking over his shoulder.

Upon reflection, I realized that his 'test' for being out of soap was looking under the counter to see if any of the containers were empty. Our 'test' for being out of soap was trying to get soap from the dispenser. So even though his test passed, it still was failing the end user because he wasn't verifying the expected result; the assumption being that the only reason you wouldn't get soap was if the container was empty.

The bottom line is to always be aware of what your tests are testing. Just because you're tests all pass does not guarantee everything is working as long as you aren't testing the way the system/device is used in real life. So, when you look at your test plan ask yourself, am I checking that we are out of soap, or am I checking whether I can get soap out of the dispenser? It might seem like a small difference but it can be difference between a cleanly running system and one that leaves you all wet.

Process Paranoia

Is your manager part of the problem or the solution and how can you tell?

Allow me to tell a story.

You are a developer on a large project that involves a large portion of the IT department not to mention multiple external stakeholders and has lots of moving parts and many upstream project dependencies. There are a large cadre of contractors working on a bug-list that keeps growing. The test environment is so complex that features can't be tested outside of certain windows due to the difficulties in getting all the moving parts built and installed correctly (no disrespect here, I mean it's that complex of a beast). The powers-that-be freeze check-ins to the development environment to reduce the number of moving parts during a weekly (yes, not daily) build/package/deploy to test. We won't get into the easy ways to handle this like tagging or snapshots, the management, which includes the technical leads, of the project have been working extra hours for weeks and don't have mental bandwidth to discuss theories. The large influx of developers who are supposed to help fix bugs only further tax the original developers with requests to help setup dev environments, understand the database, or request additional access to various resources.

This is not new, Brooks wrote about it decades ago so why am I writing about it now? With a project like the one that I've described, change management becomes a key tool in maintaining sanity and enabling success. By predefining a process and a set of criteria by which change requests will be evaluated, management has the power to be in control of the project schedule. One such example would be "Once the project has begun User Acceptance Testing, no more change requests will be accepted without approval by the CIO." The level of approval needed would of course vary by the priority of the project. There can be many such rules so that the threshold for permitting a change request rises as the project gets further into the development cycle.

If 'Analysis Paralysis' is the tendancy to continue to analyze past the point of diminishing returns, 'Process Paranoia' is the tendancy to reject efforts to define processes from a misguided attempt to forstall Analysis Paralysis. How is this connected to my tale of development woe? One of the upstream projects wants to provide a function for their customers which otherwise would require said customer to log into a second system to perform a query and then use the results in the upstream system. Normally this type of integration is commendable and encouraged. Why not now? When the managers allow change requests to be considered on an ad-hoc basis, it becomes highly probable that some request will trigger a fire drill of effort in order to evaluate the feasibility or ramifications when that decision should have been simply a matter of saying "According to our change management process, no change requests are accepted even for evaluation once we enter the testing phase without CIO approval." End of story. To do otherwise is almost guaranteeing that a project that was chugging down the tracks will eventually become a large trainwreck waiting to happen.

So the question is, does your management believe/understand/support proper change management processes or just pay them lip service? The answer to this last question goes a long way to answering the initial question of this post.

Blood, Sweat, and Tears

Showing a fellow developer how to create JUnit tests for Java code, I came up with a saying. "Refactoring is a whetstone with which you can hone your design skills." Maybe I can come up with another good phrase that includes unit testing.

Being a Codewright is like being a blacksmith. While casting something in iron is relatively straightforward, it also requires much more prior planning. It also assumes that we want to make copies of an existing design. Anything custom means pounding it out by hand the hard way.

With much sweat and toil in the crucible of coding, we put code into the forge of the code review to make it malleable, we use unit tests like an anvil to support our efforts as we bring to bear the hammer of refactoring to shape the code to our will all the while evaluating how much work is needed to achieve the desired design.

So, when you hear your team banging away at the code ask yourself, is the noise they're making that of peices being moved around with fervent hope that "maybe *this* will work!" or are they methodically improving the code through the use of the forge, hammer, and anvil?

Thomas Jefferson on Software

Thomas Jefferson on Software
Software Development lessons from our Founding Fathers

Who knew that Washington and Jefferson had experience with Software Development? When they were discussing the methods (methodology?) by which the people should be governed, they split the power that government can wield into three branches. In doing so, the founding fathers tapped into a very fundamental division of responsibility. By forcing each division, or branch, to work *with* the other branches sometimes and *against* at other times, they intended to balance the power of any one branch and work to check the abuse of power by all branches.

*From the back row* "What does that have to do with software? The man's gone daft!"
Hey, I'd give your money back but this is a free lecture.

Ok, maybe the civics lesson is boring so I'll summarize:

Legislative - Decides on and Defines law
Executive - Implements and enforces the law
Judicial - Interprets the law and decides whether the implementation follows the law as defined

It didn't take much of a stretch to come up with the following roles and responsibilities:

Customer - Decides on and Defines features
Developer - Implements requested feature
Tester - Interprets the feature, Decides whether the implementation follows the definition.

For those who just woke up, here are the Clif-Notes:
Legislative (Customer)
Executive (Developer)
Judicial (Tester)


Again from the back row: "If software development has three branches does that mean that it also has 'Checks and Balances'?"
I'm glad you asked that question! Of course it does, give that man a cigar! Let's use the tool we get all our information from, the idiot box.

*static*
[CLICK]
And now for a word from our sponser!
It's lah---og, it's lah---og,
It's brown! It's round! it's wood!
[CLICK]
*This is a story. *
*About a man named Brady.*
[CLICK]
"And now back to 'Jefferson on Software' on the History channel!"

...For simplicity's sake, I've come up with several scenarios to help describe how the different branches relate.

Checks and Balances


Feature Definition:
- Customer requests a feature

Scenario 1 (signs law)
- Developer accepts feature

Scenario 2 (veto)
- Developer vetoes a feature request(feasibility, priority, poor definition, not testable, etc.)
- Customer does not overrides the veto (low priority, nice-to-have)

Scenario 3 (override veto)
- Developer vetoes a feature request(feasibility, priority, etc.)
- Customer overrides the veto(raising the priority, accepting constraints, etc.)


Feature Development:

Scenario 1 (default)
- Developer implements feature
- Developer submits feature for testing
- Tester validates feature

Scenario 2 (improper implementation)
- Developer fails to implement and/or submit feature for testing
- Tester declares feature fails test by default
- Tester notifies Customer of incomplete feature

Scenario 3 (renegade implementation)
- Developer implements unwanted feature
- Tester decides not to include it in the application
- Customer cuts funding (power of the purse)

Feature Testing:

- Customer provides a testable feature request
- Developer agrees to the test definition
- Developer implements feature

Scenario 1 (feature passes test/upheld)
- Tester declares feature as working as requested

Scenario 2 (feature fails test/struck down)
- Tester declares feature as not working as requested
- Feature returned to developer for further work

Scenario 3 (feature passes test but Customer disagrees)
- Tester declares feature as working as requested
- Customer disagrees and thinks the feature is not complete
- Customer submits a refined new request with an updated feature definition (new legislation)


Without the Testing branch, there can be no checks and balances between what was requested and what was delivered. It is a peer to the other two and should not be seen as subservient.

For all those non-coders, pretend you are back in Speech 101 class. They teach you to tell them what you are about to say (state the requirement), say it (code the feature), then tell them what you just said (test the feature)?