Wednesday, April 29, 2009

An Estimate By Any Other Name

Terri Vajdi, a colleague of mine, sent me this response to my post on estimating. She always had the most thorough estimates I've seen in my career, so I have a lot of respect for what she has to say.

Since I drill down to assume the details of the requirements when it's missing, I equate this to all the missed requirements to a certain percentage. I read the requirements and try to envision the amount of code, junits, number of iterations, and unit testing for each iteration, Code Review and implementing the changes based on the Code Review, etc...
So, I come up with an estimate for each of these and then multiply it by the percentage/factor for the missing pieces. Add this the estimate for the missing pieces to my estimate to come up with the estimate that I turn over to my manager. In majority of cases, I break even and come close to the estimate that I have given. Of course, this is with putting a lot of overtime to meet my estimate. Remember we do not live in and ideal world and there are a lot of mishaps. This is just my two cents. Of course, I'm a quality oriented individual and I do not like to revisit my code once it's in Production. I figure that number of iterations after it has gone to Production is very very expensive. I'd rather spent it up front. So far, I've been able to stick to my methodology and have been able to spend my time wisely on other assignments once the code is in Production.
Sometimes I'm too eager and go light on my estimates which would mean even more work later and for which I've been criticized when the unknown issues would crop up and the project would run late. Terri has the right idea. She passed along this link:
http://www.apl.jhu.edu/Classes/635431/felikson/Publications/No_Silver_Bullet_Brooks.html

Anyone else have "Estimation Stories" to share?
Delicious Bookmark this on Delicious

Thursday, April 23, 2009

Realistic Irrationality

When faced with management's abdication of responsibility for their own decisions, the pragmatic developer will be faced with a decision: to live in the manager's fantasy world, which may involve constantly beating their head against a brick wall or looking for greener pastures, full of craftsmen wanting to deliver software on-time, that works, and is maintainable. This must be closely related to the management style which is unable to prioritize. The hallmark of this occurs during the negotiation desperation of a late project; trying to reign in project scope and attempting to decide which features to cut from the current iteration. When the question is, 'Which features are the highest priority? ' an answer of 'They're all top priority' is not only suspect but probably irrational. The trade offs between better, faster, cheaper are not to be trifled with; this article at softwarereality.com has a good take on the subject. If only this was a tall tale meant to scare young developers; sadly I've experienced this one myself. Feel free to send me your 'Irrational Management' stories, they make good telling at those Agile 'standups' like ghost stories around the campfire...

http://thedailywtf.com/Articles/Deploy!--Deploy!-Deploy!.aspx
Delicious Bookmark this on Delicious

Wednesday, April 15, 2009

In My Estimation

There I was at my first real programming job and my team lead comes by and drops a bomb. "How long will it take to make the change?", he asks. I hope I kept the panic from my face. "Wait a minute!", I was thinking, "Did I miss the class on estimation in college?" No, I didn't. They didn't have one. Hmmm, what do I tell my boss, "sorry, can't help you!", "I have no idea", or even better "they didn't teach that in school"? If I close my eyes and pull a number out of my estimation orifice I could look like a ill-trained idiot programmer by guessing too high or try to please by low-balling it which could easily backfire. Why did I want to be a programmer again, someone remind me. It surely wasn't to look incompetent to a guy that hasn't coded in over 20 years.

Why is estimation still so hard and what are we doing about it?
Delicious Bookmark this on Delicious

Monday, April 06, 2009

Programming as a Social Activity

What do we imagine when we consider the individual programmer? There are many stories that liken the programmer to the cowboy who finds himself in a small remote town rife with lawlessness and misery. The cowboy grudgingly accepts the job as sheriff, to right wrongs and otherwise do deeds of much heroism; riding off into the sunset, when the crisis has been averted, to the wails of the local townfolk who think they are losing their only defense against the chaos that comes with frontier life. Is this how individual programmers actually work or is this a delusion used to supplicate the ego of socially inadequate developers?

This continues my review of "The Psychology of Computer Programming" by Gerald M. Weinberg. The previous chapter Chapter 3 - How Can We Study Programming discusses the difficulty of studying behaviors which are hard to isolate, repeat, or tranlate into an experimental setting. Chapter 2 - What Makes a Good Program evaluates the subjective and objective criteria for judging programs. The beginning of the series of reviews is Chapter 1 - Reading Programs. Techrepublic.com has a review of the same book which reinforces this book's classic status.

In Part 2 of "The Psychology of Computer Programming", Weinberg describes three different ways that programmers are organized: the group, the team, and the project. Chapter 4 is titled "The Programming Group". He starts with the observation that when programmers work together in the same environment, they inevitably seek each other out for help or advice and expresses the premise that the lone programmer is at a serious disadvantage.


Formal and Informal Organization

The first lesson is that in any assemblage of people there are formal and informal groups. Formal hierarchies like an organization chart might be nice for the manager types but do not begin to describe the various interactions between individuals in a group. One good story is from back in the batch-scheduled era when it was difficult to know whether your job had run and its output was available, which was accomplished by posted completed jobs on a board outside the computer area. Programmers in remote offices might waste half an hour coming over just to find their results were not ready. As it happened, there was secretary who sat in a location which allowed her to see the result board. When a programmer who was waiting for a job to complete happened to call her for a date. When the conversation turned back to work, she mentioned that he must have work to get back to. He told her that he was waiting for a job to complete. She volunteered that she could see the board from where she was sitting and asked him in which job he was interested. This began an informal service which quickly became known to the outlying programmers. The service became popular enough to disrupt the work of the secretary and need the attention of the administration. Wisely acknowledging that the service could not just be cut off, they decided to implement a sort of job posting hot-line, even using the secretary's original phone number, where a recording would give the latest job completion posting. Here an informal and inefficient solution was replaced with a formal but efficient one.

This contrasts with a second story where a university computing center provided a common room for working on programming problems and an adjoining room with graduate students available to answer questions ("consulting services" was the term used in the book), and at one end of the common area were vending machines. When students complained about the noise from the crowd of students around the vending machines, the administration removed the offending machines. Not long after the removal of the vending machines, another delegation came to the administration with a complaint, there wasn't enough consulting services. As it turns out, the students who were making all the noise were actually discussing their programs and helping each other with them, answering all sorts trivial questions in an informal setting. The administration, unable to believe that the removal of the vending machines was the actual problem tried to increase the number of consultants available with little success as fewer graduate students were willing to deal with the ever increasing number of mundane and trivial problems. The formal solution was not nearly as effective as the old informal one.


Physical Environment and Social Organization

Does the environment in which you work affect your productivity? Companies are all about getting the most for the least so it is no surprise that in devising the seating arrangements of programmers, minimum cost usually trumps maximum productivity. There are exceptions like early Microsoft and their "every programmer gets an office with a door" mantra, to more recent examples like Fog Creek, who also take a programmer-first approach. In Weinburg's words,

"Someone should definitely study the depressing effect that the all-too-common half partitions have on programmer productivity. They manage to cut off all useful communication while permitting all disturbing sound and movement to penetrate."
Another example was the building that replaced the old manual elevator with an automatic one. This was unfortunate for the programmers because the old elevator operator had run an informal pickup and delivery service for them between their floor -- the eighth-- and the machine room -- the basement. They saved the cost of the elevator operator but cost their programmers much time every day going back and forth to the basement of the building.


Error and Ego
"If asked, most programmers would probably say they preferred to work alone in a place where they wouldn't be disturbed by other people"

"The ideas expressed in the preceding paragraph are possibly the most formidable barrier to improved programming that we shall encounter."
Programmers tend to get invested in the programs they write, attached to them we'll say. This causes behaviors that are counter-productive. What's wrong with "owning" programs, after all artists "own" paintings; authors "own" books; architects "own" buildings. When we go to the bookstore we can get a good idea about the quality of a book if it by a familiar author. Wouldn't the same hold true for programs and their authors?
"The admiration of individual programmers cannot lead to an emulation of their work, but only to an affectation of their mannerisms. This is the same phenomenon we see in "art colonies," where everyone knows how to look like an artist, but few, if any, know how to paint like one."
Another problem with the idea of ownership happens when we start to evaluate or judge a work. Art is to a large degree subjective so the artist can dismiss the opinions of critics. Can a programmer dismiss the judgement of a computer? If we took the "ownership" association to its logical end we would reason something like this:
"This program is defective. This program is part of me, an extension of myself, even carrying my name. I am defective."
Obviously that does not happen. Programmers do not quit just because of a single bug or logic flaw but they will tend to overlook problems and strive to prove the program is correct even in the face of mounting evidence to the contrary. Proving the program is wrong turns into a minefield of self-worth where bugs become the buried mines which will injure the pride of the programmer who can't help but react in defense of his own sense of self-worth.

"Thus, if we are going to attack the problems of making good programs, and if we are going to start at the fundamental level of meeting specifications, we are going to have to do something about the perfectly normal human tendency to believe that ones "own" program is correct in the face of hard physical evidence to the contrary."

This explains the unconscious motivations behind the resistance to code-reviews, pair-programming, or any other effort to get them to loosen their hold on "their" code. If we saw this type of behavior in a toddler, we would immediately recognize it as childish, as in "James doesn't want to share his code with the others." Will James feel like he is losing something if he has to let others play with his code? It would go a long way to explain things I've seen in my programming career.


Egoless Programming

So, what do we do? The manager could insist that the coders work harder at finding the bugs in their code. He could even go around and ask them to show him the bugs in their code. Applying even the smallest amount of psychology would tell us that this will backfire. The average person will feel he is on trial and very easily take it personally. If we can change the social environment of the group, we have a chance to affect change in the values of the programmers of that group. This is possible, we know of egoless programming groups since the very early days of programming.

"John von Neumann himself was perhaps the first programmer to recognize his inadequacies with respect to examination of his own work. Those who knew him have said that he was constantly asserting what a lousy programmer he was, and that he incessantly pushed his programs on other people to read for errors and clumsiness. Yet the common image today of von Neumann is of the unparalleled computing genius--flawless in his every action. And indeed, there can be no doubt of von Neumann's Genius. His very ability to realize his human limitations put him head and shoulders above the average programmer today."
Weinburg tells another story of a programmer from the early days of space tracking systems. "Bill" was confident in a loop and gave it to "Marilyn" to informally review, as was common practice in the group. Marilyn found seventeen bugs in only thirteen statements. Bill chalked it up to a 'bad programming day' and became more and more amused as each bug was found. He took it as proof that it just wasn't his day and used the rest of the day giving everyone a laugh at his expense. Marilyn, on the other hand, did not have any false confidence in her own ability and took the code to a third person. She knew that she had looked at it so much she wasn't going to be able to see any more errors. She needed fresh eyes. With others help, she found three more errors before the day was over. The epilogue was that no more errors were found in the code despite diabolical testing and being in production in over a dozen installations even after nine years.

Why aren't there more groups like this? Weinburg asserts that such groups are not an isolated case. They regard their practices as valuable proprietary information, implying they think other would copy them if given a chance. These groups are very satisfied and stable which reduces the opportunities for programmers to have encountered these groups. Also, gypsy programmers - who never settle in one place - encourage the myth that the best programming is the product of genius, to justify ever-increasing consulting rates.

The last word of this section is on how there is an advantage to making your code as readable as possible for other programmers for that means that those who come after will have a chance to learn from such code without feeling the need to have it deciphered or translated. So, the best code isn't the stuff that is hardest to read but the easiest to read. Easier to test, maintain, understand, and learn from; who could argue with that?


Creating and Maintaining the Programming Environment

One danger programming environments face is the tendency to fixate or lock into some behavior, habit, or tool. Programming language is a common example. Even if another language is better suited for a problem, the language of choice will be preferred. More people know it, there are more tools available (to the group), more library routines, etc. In other words, it's "how we do things here" which makes choosing a different language like paddling upstream, lots of resistance and little to show for a great deal of effort.

Another fixation is the social environment which either encourages or discourages egoless programming. If the new guy is ridiculed when asking another for help or advice, he is less likely to seek assistance the next time. If, on the other hand, someone pays him the implied compliment by asking him to look over a program, he is more likely to feel secure enough to seek advice.
"To a large extent, we behave the way we see people behaving around us, so a functioning programming group will tend to socialize new members to its philosophy of programming."
On the subject of egoless programming and management, Weinburg astutely describes the culture shear between common management and uncommon programming groups. Remember that this book came out almost 40 years ago; you can draw your own conclusions about how much things have changed.
"Managers tend to select themselves from the 'aggressive' component of society and have difficulty appreciating the fact that other people do not completely share their goals of money and prestige. They are especially at a loss to understand the smooth functioning of a programming group based on mutual respect for individual talent and cooperation in the common cause. Instead, the tend to view people as working for money or under threat -- as they themselves do."
Another example given is of a group that was particularly success and producing a new system. The achievement was noticed by the company's management who decided to give them a cash award. Typically, they wanted to give the award to the groups manager but were bewildered when the manager would not accept it unless the whole group got the award. The groups manager understood that the group acted as a team and that responsibility for success could not be attributed to a single individual. Other managers thought he wanted more money, others thought he was trying to setup a "prima donna" group. The company decide to force him to take it and break up the group -- which seemed to have unhealthy ideas. For those doubters out there, I have personally witnessed this same scenario in my career. The moral of the story? Again, Weinburg says it as well as anyone,
"Had the management been more aware of what this group had to offer, and had they been more flexible, they might have worked out a solution that would have permitted this group to influence the work of other groups in a favorable way. But this is not easily done by managers, who tend to feel that when work gets done it is the direct result of the actions of some leader of outstanding ability. Even when the manager appreciates the work of the group, it is not consistent with his own philosophy to see the productivity of the group as a property of the group, not as a sum of the contributions of the individual members."

Summary

To understand how programmers perform in a group setting, you must consider formal and informal structures, the physical environment, and even the social interactions in play between members of the group. These are lessons which are still worth learning today. The hard question we must ask ourselves is why we keep searching for new solutions when we have not addressed the issues identified four decades ago. I maintain that the Agile Manifesto or the Manifesto for Software Craftsmanship, is but a few of the efforts to awaken the courage of individual programmers to affect change around them, and cause a groundswell of change which might have the momentum to disturb the inertia of the status-quo.


Sample questions:

For managers: What is your honest opinion of people who are not trying to "move up" in your organization, but who seem satisfied with the kind of work they do and the amount of money they get? To what extent is your view influenced by your own feelings for yourself?

For programmers: Do you refer to your work as "my" program? Try passing one week without using the personal possessive in reference to programs, and take notes on the effects you observe.


p.s - These chapters are large enough that I may break them down so each section is its own post.
Delicious Bookmark this on Delicious