Context as a commodity

How much context do you need to do your job?  Can we quantify context in any way?

Jeff Atwood has mentioned that he doesn’t shut his machine down every night.  He wants his environment to be as he left it at the end of the previous coding session.  Getting all the apps, tools or other windows opened to the right place takes not only CPU time but mental energy, aka context. 

Some jobs require no context, for example a bank teller can walk in to work on Tuesday and not need to remember any of the transactions that took place on Monday.  Contrast that with a novelist in the middle of writing a new book, let’s say writing the next chapter not editing anything.  They need to remember all the characters, their personalities, the existing story, and in what ways the plot threads are to interact in the new material.

Another good contrast is a professional athlete.  A pro baseball player doesn’t need to remember what happened in yesterday’s game in order to pitch a strike, hit a home-run, or execute a double-play.  The limited context that does appear has to do with the optimizing performance.  A pitcher does need to know the preferences and history of the batter who is at the plate, just like the batter needs to know if the pitcher has a wicked slider. 

Companies want to be “green” these days and have instituted policies to shutdown computers overnight.  This may be pennywise and pound foolish.  Here’s why.  A consultant gets paid $50/hr and it takes 30 minutes to get their workspace ready after an overnight shutdown lasting from 5pm to 8am the next day, 15 hours.  A conservative energy estimate would be if electricity costs $0.25 a kilowatt hour multiplied by 15 hours for a cost of $3.75.  So $25 is spent trying to save $3.75.  That’s motivation to use the hibernate feature at a minimum. It also is a very simple way to quantify the cost of context.

Can we describe jobs by how much context is required?  If so, is there a relationship between  the amount of context needed and the average salary?  Those are interesting questions, maybe I’ll cover those in a later post. 

Of Rockstars and Bricklayers

Today we're going to start with a thought experiment.  You are one of the better coders in your company.  All the proof I need is that you're reading this.  Ok, that may be a little self-serving but it is true that the better coders I know are always reading something about coding.  It might be a new language, or a new technique, a programming question and answer site, or even a site that likes to rag on bad code. 

Based on how many Code Monkeys you know, it's easy to figure yourself in the top half of programmers, a 'good' coder if you will. An assumption that is easy to make is that the higher your productivity the higher your pay.  Is this true?  If you base it on simple experience, the answer is no.  It's too easy to get a job right out of college and get mediocre raises and then find yourself interveriewing candidates for openings in your own group where you have to offer 'market rates' to attract anybody.  If you've been there for 5+ years, given a company desperate to hire warm bodies, it doesn't take much for the new guy to be making 15-25% more than you even if he is a two-finger typist and can't spell SQL if you spotted him the 'S' and the 'Q'. 


Now, I'm happy where I am but it didn't take much to trigger my bad habit of thinking up 'what-ifs'.  The Stackoverflow podcast has been a good one for this sort of pondering.  Jeff and Joel have enough real-world experience to bring a ton of wisdom to the discussion on programming as it's own topic and in episode #42 discuss underperforming programmers.  I'd been reading about how star actors can command huge salaries based on their past work and the expected effect on the box-office if their name appeared on the marquee.  That major-league sports can have individuals who make 10-times what the lowest paid athlete makes on a single team, it doesn't take much to ask why programming doesn't see the same effect. 

I submitted this as a question to the Stackoverflow podcast which they spent a good 10 minutes on in episode 77. It didn't fit so well as a question on Stackoverflow itself, I suspect the culprit is the lack of code snippets. Interestingly enough I discovered the podcast when they were at episode 65 and decided to start listening back at #1 and was up to around #42 when I decided to contribute my question.  So anyway, when I saw the summary for #77, I didn't have the audio handy so jumped over to the transcript wiki for #77 and read the transcript.  It was a pleasant surprise at how engaged they were and that the question wasn't dismissed out-of-hand.  When I downloaded the audio, to hear their discussion it only reinforced my reaction.

The range of talent can be mighty wide even if there is some disagreement on the empirical numbers.  Joel Spolsky talks about the difficulty in finding great programmers, and uses similiar metaphores in a comparison with opera singers. Jeff has a similar take on Coding Horror.  Finding great talent is hard because there are only so many people who can do what you need done.

There have been some good discussions that has surfaced.  John Cook has a good argument based on lack of meaningful metrics.  It's a question that has been around for awhile.  Frank Wiles makes the claim that hiring average (or below) programmers actually costs your company money (followup post). Obviously when a company says they only hire the to 10%, they are deluding themselves or at least paying lip-service to wanting good programmers.  Joel does a great job of shattering that particular illusion. 

What about sales? In some fields top salesmen can bring home seven-figure incomes so we know that the wide range of incomes aren't limited to Hollywood and the big leagues.  James Kwak picks up on John's post and supports Joel and Jeff contention that paying the top performers much more than the average, outside of easily measured areas like sales and movies, has a corrosive effect on the group and the best coders end up leaving for greener pastures while the dregs hang on for dear life or are eventually pushed out.  I can't disagree.  It seems that if you are underpaid you have to ask yourself Ann Lander's question, "Am I better off leaving or staying put?"  Just realize that it's no use complaining to the company, the status quo works for them.

Life's Too Short To Work For Clowns.

One of the cliche's of our culture is the little boy who rebels against his parents and dreams of running away to join the circus. His mind is filled with the glamor of the Big Top, the exotic animals, the independence of wandering the country from one town to the next. He hasn't really thought about what it means to work in a circus. The Ringmaster did many menial jobs before arriving in the spotlight. The animals have to be cared for; double the time it takes to feed them when you consider both ends of the animal. Traveling from town to town may sound romantic until you realize it means you spend more time traveling than in one place, lots of lonely nights on the road regardless of the weather or climate because 'the show must go on" after all. The life of a circus is not just popcorn, parades, and pretty girls.

I ran across a quote in Christopher Duncan's, "The Career Programmer: Guerilla Tactics for an Imperfect World" that made me both laugh and sigh.

You need to consider one additional thing when trying to convince management the current approach to developing software needs improvement. What do you do if non of this sways them and they insist on continuing with chaotic development approaches that consistently create disasters and stress? Update your resume. Life's too short to work for clowns.

So the question becomes, do you work for clowns? Does the development process at your company make a circus look like a well-oiled machine? You might work for a clown.

I plan on putting together a complement to The Joel Test where instead of a list of things a company does before you consider working for them, The Clown Test would be a list of things that indicate you might work for clowns.

This might describe your boss,

• if he thinks an estimate is a promise,
• if he thinks developers are interchangeable with no effect on the schedule,
• if he thinks its ok for the lead developer to publicly berate a junior developer,
• if he thinks requirements documents are a waste of time,
• if he thinks the only place for designers is at a fashion show,
• if he thinks he just needs one good programmer who can code anything,
• if he thinks that people whine too much about teamwork,
• if he believes in Silver Bullets,

he might be a Tech Clown.

My apologies to Jeff Foxworthy.

When you run across a good example of a Tech Clown, send 'em my way and I'll add it to the list for every one's enjoyment.

The Myth of the Aging Developer

I question that I've heard throughout my career is, "What happens to older software developers?" The danger with this question is when the discussion turns to the implied version which is "The lack of older developers means that it is a young persons game." Nothing could be further from the truth..

My wife and I are both developers. We have both been working as developers for 15+ years. One thing we noticed was that there were a lot of people working as developers who really shouldn't be, they didn't have the aptitude, interest or sometimes even the background. It would be easy to dismiss the problem as a lack of proper education but that doesn't explain the developers who lack a CS degree but who can code circles around veteran devs. Those folks usually decide that programming is not for them and move on to different job. Some go into management, some become analysts, while others change careers completely.

What's left? The ones who are passionate about development and are interested in doing it right. They love learning new stuff and don't gripe about how things used to be in "The good ole' days." They might be grumpy but they usually know their stuff.

The main problem would be in finding management who doesn't have a pre-conceived notion about what makes a good developer. If they feel that only the younger ones have the stamina to keep up with the pace, I'd wonder whether those same employers were the same ones who believe that the 'Deathmarch' style of project management is standard operating procedure. They'd rather grind through fresh-meat developers (Code-Monkeys anyone?) than listen to the experienced ones and change their development process much less change their management style.

Lastly, I've noticed that doing nothing but coding holds less business value than being able to communicate with customers, users, management, and vendors. One reason you start coding less is because finding answers to ambiguous requirements takes more experience and provides more payback to the business.

My advice, to those who worry about their coding future, would be to find a place where they need experienced developers to help with the overall task of creating or maintaining software, that also gives you the opportunity to code regularly. That way you can have the time to keep up with industry changes while taking advantage of your understanding of software development.

Discontinuous Integration

Refactoring is becoming a commonplace term thrown around these days. The expectation is building that refactoring is something that all good developers do and if you don't, you must be in one of "those" types of companies. You know the ones, they don't use a version control system or have a repeatable build, i.e. Continuous Integration, much less a well-defined deployment procedure.

If you are lucky enough to be building the first version of a product, you can refactor to your hearts content. There is no existing code to worry about breaking. If you are changing an existing system refactoring isn't always simple or straightforward. Even if you have the most automated CI and 100% code coverage of unit tests you'll still face the practical issues of deployment and business dependencies.

Let's say you operated the worlds most modern airline reservation system. It's written in the latest language and has all the buzzwords, highly available, highly reliable, highly scalable, etc. It has tons of automated tests so you are 100% confident that nothing will break. Now, since you have business partners like travel agencies and customers looking to book a flight or retrieve flight information, they have a stake in your deployment. The customers may tolerage a "Down for maintenencem, come back later." message but your partners will take a dim view of your downtime. They'll insist you tell them in advance so they can prepare any necessary changes in their systems or alert their support staff in case there are issues after the deployment.

Another example would be a large back-office accounting system. Whatever level of confidence you have in your code. you still have to deal with the myriad inputs and outputs of the system and disrupting the processing schedule for a code migration. No amount of automated testing/build/migration will ever solve these types of complexities.

The point is that considerations like these limit the opportunities to deploy code changes and increase the cost of change even when that change is guaranteed to work.

Refactoring is good but there are practical considerations why it's not done to the degree it could be in some shops. Keep that in mind next time you want to pass judgement on the perceived lack of refactoring somewhere.

In Search of Wisdom

The other day I was reading StackOverflow, a forum for developers to ask and answer technical questions, and ran across a post from a discouraged student. The responses had one thing in common: programmers know that they must keep learning. Tomorrow's hot new technology is just as soon to be yesterday's lost and forgotten relic. The best programmers are not the experts in a given niche but the ones who are comfortable with learning as a way of life. They know that they do not know and accept it. This makes them resistant to dictated dogma; a good example being Joel Spolsky's "Emperor's New Clothes" treatment of C-style strings. They are then freed from the ups and downs of the technological tide and are not battered by the constant waves of the newest, latest fad, nor do they feel deserted when some favored tech is washed away into the sea or wrecked upon the shoals.

Socrates is said to have walked the streets of Athens looking for a wise man. Wisdom is usually associated with knowledge but since it is impossible to know everything, true Wisdom is the awareness that you don't know. You can know a little about some things or a lot about a few. Those who believe they do know are blinded by their own ego and are thus willing to dismiss all doubts and set a course powered by a self-fulfilling sense of certainty. This describes how many a tribe of men met their tragic fate. Technology is not immune from the siren's song of certainty hence the never-ending search for the always-elusive "Silver Bullet".

When it comes to Wisdom, what sets apart the Codewright from the Code Monkey? A healthy dose of self-doubt. Doubt enough to realize that risks should not be dismissed solely because they are improbable. Doubt enough to allow the most skillful to help do the work, the most far-sighted to help guard the course, the most cynical to help figure the schedule. Doubt enough to keep unjustified self-confidence from enabling acts of hubris. This is just as true for teams as well. The best teams are the ones who understand both their strengths and their weaknesses. A group of individual superstars who can't learn to play together isn't always guaranteed success while a bunch of unknown amateurs who work has a cohesive unit can perform miracles.

Software development as a field naturally attracts those who are willing to accept the reality that there will always be more to learn. It is a good sign, one that indicates our chosen field will never stagnate. There will always be something to discover, a constantly flowing river of new challenges to overcome and puzzles to solve. The search for Wisdom is not for a desired destination, but of a fulfilling journey.

What Did You Say?

This says it all. Most of the time research focuses on if new stuff is possible. The real 'Silver Bullet' would be to find a way to solve the communication problem. Click on the image to embiggen.

Why Extreme Programming Fails

In the decades of the young life of software development, we've discovered many behaviors, processes, and techniques which have nothing to do with software yet continue to hold back progress. We ask time and time again why the mythical silver bullet is still searched for by those who have already agreed that it is a fools errand. Why are death-marches allowed to form when the futility is obvious to all but the oblivious? Why are proven techniques not mandated by management or agreed to by the workers who constantly complain about the current poor practices? This is true whether we're discussing XP or Agile or SCRUM.

We can use rational thought to decipher the mystery, attempting to achieve clarity. However much success is possible in finding rational explanations, there comes a time when we must recognize the difference between clarity and understanding. We can state things extremely clearly and succinctly and still fail to understand why people continue their unproductive habits or why change is viewed by those in charge with such disdain.

Submitted here for your edification, The 48 Laws of Power.

Why does that developer refuse to pair with someone? There are lots of reasons, i.e. Law #1 - Never Outshine the Master or law #30 - Make your Accomplishments Seem Effortless, are good places to start. Why is Continuous Integration scoffed at? See law #11 - Learn to Keep People Dependent on You

The words 'politics' or 'management' may seem like pejoratives but they serve as simple placeholders for the understanding of human behavior. At their worst, they can be weapons in a Machiavellian conflict while at their best they can be the protective shield against those who see your efforts as a challenge to their plans. Is every use of these laws an instance of plotting and scheming? No, never forget that human nature operates below the level of conscious thought; these behaviors are ancient artifacts used for thousands of years to survive in a violent and irrational world. Disagree with them all you'd like but ignoring them can easily cost you your job.

The Myth of Code Construction

Ever work with a developer who avoided letting anyone see their code, refused to program in pairs or participate in a code review? They say "This is my code" as if they have sole right of refusal on the final product regardless of who else may be paying the bill.

Great writers still have editors. No matter how great a writer you are, you still have to submit your work to an editor before it will be published. Let's examine the idea of putting code into production as a form of publication.

Before we jump into a new analogy, we should look at the current state of affairs. If engineering were a family construction would be the successful older brother leaving software as the immature younger brother, never quite living up to the accolades of the elder sibling yet endlessly compared nonetheless.

In general, the phases in software are: analysis, design, construction, test, deployment, and training. When you want to build a building of any size, let's use a downtown skyscraper as an example. If we continue the comparison we'd start with analysis which includes many factors like the target location or how many square feet are required. The location can impose limitations to the footprint which then dictates how many floors would be needed to achieve the required square footage. Such trade-offs are examined and decided upon during the design phase. Construction is all about execution with limited opportunity for design changes. How do you test a building, what does that even mean? You might might test the subsystems, like elevators, fire alarms, etc. but how do you test the entire building? The closest we can come in scale might be the opening and closing the retractable roof of an almost complete stadium like the new Cowboy's stadium (this being written in May of 2009). How do you deploy a building? You don't, you open it for business. There is no pull of the lever where what did not exist suddenly does. As for iterations, the closest construction can come is building a new wing on an existing building, adding a room to the house at hand. The cost and effort of remodeling can end up being a significant ratio to the original price of the house.

Now let's discuss a comparison of software development with writing a book. A publisher decides the market is ready for a new book on ant farming. They either have recent submissions by authors interested in writing such a book or they solicit existing authors who have experience in the subject matter. In any event, they request an outline of the book with maybe a sample of one chapter to gauge the material matches what the publish wants. The author or authors work at creating the manuscript all the while knowing that editors will read the book and suggest changes. Editors who are not experts in the content but are experts in the publishing of books. Even the highest-paid authors - Mary Higgins-Clark, Stephen King, or J. K. Rowlings to name a few - will all submit their work to the editors pen.

When the manuscript is complete and the editors happy, there are still other contributors to the final product, researchers, artists, designers, and the like who do things like create illustrations, populate an index, check facts, or plan the visual layout of the chapters. All these activities are coordinated by the publisher who has a target date for when the book will be printed, planning a marketing blitz to coincide with the publish date. Getting the manuscript done is important but still is a small part of the whole effort and if an author cannot produce on time, a lot of investment by the publisher goes to waste; a good way to dissuade the publishing company to find an author who can deliver. The key is that it doesn't have to be perfect but does have to be complete. If there is enough demand, it is easier to publish a new edition rather than hold off on the initial printing. Testing is as easy as a small test run where the final product can be validated by all contributors before scaling up into production-sized runs; images replaced, a new cover chosen, typos fixed. Printing becomes like running the build with a target of production. This means that "build" isn't a phase, it's an activity; unlike any in construction. Deployment becomes a straightforward, if not simple matter, of distributing copies of the book to various retail book-sellers or their distributors.

My suggestion is for the software developers of the world realize that allowing our craft to be constantly compared to the construction industry is to remain the little boy, wanting to live up to the expectations of those who came before, never earning the respect or acceptance they so long for. Look instead to the world of publishing, where writing is still an art and yet the businesspeople have found a way to make money without rejecting the fundamental nature of writing, be that a novel, the latest tech book, why not the latest edition of your favorite word processor?

p.s. Kudos to the wife for the idea.

An Estimate By Any Other Name

Terri Vajdi, a colleague of mine, sent me this response to my post on estimating. She always had the most thorough estimates I've seen in my career, so I have a lot of respect for what she has to say.

Since I drill down to assume the details of the requirements when it's missing, I equate this to all the missed requirements to a certain percentage. I read the requirements and try to envision the amount of code, junits, number of iterations, and unit testing for each iteration, Code Review and implementing the changes based on the Code Review, etc...
So, I come up with an estimate for each of these and then multiply it by the percentage/factor for the missing pieces. Add this the estimate for the missing pieces to my estimate to come up with the estimate that I turn over to my manager. In majority of cases, I break even and come close to the estimate that I have given. Of course, this is with putting a lot of overtime to meet my estimate. Remember we do not live in and ideal world and there are a lot of mishaps. This is just my two cents. Of course, I'm a quality oriented individual and I do not like to revisit my code once it's in Production. I figure that number of iterations after it has gone to Production is very very expensive. I'd rather spent it up front. So far, I've been able to stick to my methodology and have been able to spend my time wisely on other assignments once the code is in Production.

Sometimes I'm too eager and go light on my estimates which would mean even more work later and for which I've been criticized when the unknown issues would crop up and the project would run late. Terri has the right idea. She passed along this link:
http://www.apl.jhu.edu/Classes/635431/felikson/Publications/No_Silver_Bullet_Brooks.html

Anyone else have "Estimation Stories" to share?

Realistic Irrationality

When faced with management's abdication of responsibility for their own decisions, the pragmatic developer will be faced with a decision: to live in the manager's fantasy world, which may involve constantly beating their head against a brick wall or looking for greener pastures, full of craftsmen wanting to deliver software on-time, that works, and is maintainable. This must be closely related to the management style which is unable to prioritize. The hallmark of this occurs during the negotiation desperation of a late project; trying to reign in project scope and attempting to decide which features to cut from the current iteration. When the question is, 'Which features are the highest priority? ' an answer of 'They're all top priority' is not only suspect but probably irrational. The trade offs between better, faster, cheaper are not to be trifled with; this article at softwarereality.com has a good take on the subject. If only this was a tall tale meant to scare young developers; sadly I've experienced this one myself. Feel free to send me your 'Irrational Management' stories, they make good telling at those Agile 'standups' like ghost stories around the campfire...

http://thedailywtf.com/Articles/Deploy!--Deploy!-Deploy!.aspx

In My Estimation

There I was at my first real programming job and my team lead comes by and drops a bomb. "How long will it take to make the change?", he asks. I hope I kept the panic from my face. "Wait a minute!", I was thinking, "Did I miss the class on estimation in college?" No, I didn't. They didn't have one. Hmmm, what do I tell my boss, "sorry, can't help you!", "I have no idea", or even better "they didn't teach that in school"? If I close my eyes and pull a number out of my estimation orifice I could look like a ill-trained idiot programmer by guessing too high or try to please by low-balling it which could easily backfire. Why did I want to be a programmer again, someone remind me. It surely wasn't to look incompetent to a guy that hasn't coded in over 20 years.

Why is estimation still so hard and what are we doing about it?

Programming as a Social Activity

What do we imagine when we consider the individual programmer? There are many stories that liken the programmer to the cowboy who finds himself in a small remote town rife with lawlessness and misery. The cowboy grudgingly accepts the job as sheriff, to right wrongs and otherwise do deeds of much heroism; riding off into the sunset, when the crisis has been averted, to the wails of the local townfolk who think they are losing their only defense against the chaos that comes with frontier life. Is this how individual programmers actually work or is this a delusion used to supplicate the ego of socially inadequate developers?

This continues my review of "The Psychology of Computer Programming" by Gerald M. Weinberg. The previous chapter Chapter 3 - How Can We Study Programming discusses the difficulty of studying behaviors which are hard to isolate, repeat, or tranlate into an experimental setting. Chapter 2 - What Makes a Good Program evaluates the subjective and objective criteria for judging programs. The beginning of the series of reviews is Chapter 1 - Reading Programs. Techrepublic.com has a review of the same book which reinforces this book's classic status.

In Part 2 of "The Psychology of Computer Programming", Weinberg describes three different ways that programmers are organized: the group, the team, and the project. Chapter 4 is titled "The Programming Group". He starts with the observation that when programmers work together in the same environment, they inevitably seek each other out for help or advice and expresses the premise that the lone programmer is at a serious disadvantage.

Formal and Informal Organization

The first lesson is that in any assemblage of people there are formal and informal groups. Formal hierarchies like an organization chart might be nice for the manager types but do not begin to describe the various interactions between individuals in a group. One good story is from back in the batch-scheduled era when it was difficult to know whether your job had run and its output was available, which was accomplished by posted completed jobs on a board outside the computer area. Programmers in remote offices might waste half an hour coming over just to find their results were not ready. As it happened, there was secretary who sat in a location which allowed her to see the result board. When a programmer who was waiting for a job to complete happened to call her for a date. When the conversation turned back to work, she mentioned that he must have work to get back to. He told her that he was waiting for a job to complete. She volunteered that she could see the board from where she was sitting and asked him in which job he was interested. This began an informal service which quickly became known to the outlying programmers. The service became popular enough to disrupt the work of the secretary and need the attention of the administration. Wisely acknowledging that the service could not just be cut off, they decided to implement a sort of job posting hot-line, even using the secretary's original phone number, where a recording would give the latest job completion posting. Here an informal and inefficient solution was replaced with a formal but efficient one.

This contrasts with a second story where a university computing center provided a common room for working on programming problems and an adjoining room with graduate students available to answer questions ("consulting services" was the term used in the book), and at one end of the common area were vending machines. When students complained about the noise from the crowd of students around the vending machines, the administration removed the offending machines. Not long after the removal of the vending machines, another delegation came to the administration with a complaint, there wasn't enough consulting services. As it turns out, the students who were making all the noise were actually discussing their programs and helping each other with them, answering all sorts trivial questions in an informal setting. The administration, unable to believe that the removal of the vending machines was the actual problem tried to increase the number of consultants available with little success as fewer graduate students were willing to deal with the ever increasing number of mundane and trivial problems. The formal solution was not nearly as effective as the old informal one.

Physical Environment and Social Organization

Does the environment in which you work affect your productivity? Companies are all about getting the most for the least so it is no surprise that in devising the seating arrangements of programmers, minimum cost usually trumps maximum productivity. There are exceptions like early Microsoft and their "every programmer gets an office with a door" mantra, to more recent examples like Fog Creek, who also take a programmer-first approach. In Weinburg's words,

"Someone should definitely study the depressing effect that the all-too-common half partitions have on programmer productivity. They manage to cut off all useful communication while permitting all disturbing sound and movement to penetrate."

Another example was the building that replaced the old manual elevator with an automatic one. This was unfortunate for the programmers because the old elevator operator had run an informal pickup and delivery service for them between their floor -- the eighth-- and the machine room -- the basement. They saved the cost of the elevator operator but cost their programmers much time every day going back and forth to the basement of the building.

Error and Ego

"If asked, most programmers would probably say they preferred to work alone in a place where they wouldn't be disturbed by other people"

"The ideas expressed in the preceding paragraph are possibly the most formidable barrier to improved programming that we shall encounter."

Programmers tend to get invested in the programs they write, attached to them we'll say. This causes behaviors that are counter-productive. What's wrong with "owning" programs, after all artists "own" paintings; authors "own" books; architects "own" buildings. When we go to the bookstore we can get a good idea about the quality of a book if it by a familiar author. Wouldn't the same hold true for programs and their authors?

"The admiration of individual programmers cannot lead to an emulation of their work, but only to an affectation of their mannerisms. This is the same phenomenon we see in "art colonies," where everyone knows how to look like an artist, but few, if any, know how to paint like one."

Another problem with the idea of ownership happens when we start to evaluate or judge a work. Art is to a large degree subjective so the artist can dismiss the opinions of critics. Can a programmer dismiss the judgement of a computer? If we took the "ownership" association to its logical end we would reason something like this:

"This program is defective. This program is part of me, an extension of myself, even carrying my name. I am defective."

Obviously that does not happen. Programmers do not quit just because of a single bug or logic flaw but they will tend to overlook problems and strive to prove the program is correct even in the face of mounting evidence to the contrary. Proving the program is wrong turns into a minefield of self-worth where bugs become the buried mines which will injure the pride of the programmer who can't help but react in defense of his own sense of self-worth.

"Thus, if we are going to attack the problems of making good programs, and if we are going to start at the fundamental level of meeting specifications, we are going to have to do something about the perfectly normal human tendency to believe that ones "own" program is correct in the face of hard physical evidence to the contrary."

This explains the unconscious motivations behind the resistance to code-reviews, pair-programming, or any other effort to get them to loosen their hold on "their" code. If we saw this type of behavior in a toddler, we would immediately recognize it as childish, as in "James doesn't want to share his code with the others." Will James feel like he is losing something if he has to let others play with his code? It would go a long way to explain things I've seen in my programming career.

Egoless Programming

So, what do we do? The manager could insist that the coders work harder at finding the bugs in their code. He could even go around and ask them to show him the bugs in their code. Applying even the smallest amount of psychology would tell us that this will backfire. The average person will feel he is on trial and very easily take it personally. If we can change the social environment of the group, we have a chance to affect change in the values of the programmers of that group. This is possible, we know of egoless programming groups since the very early days of programming.

"John von Neumann himself was perhaps the first programmer to recognize his inadequacies with respect to examination of his own work. Those who knew him have said that he was constantly asserting what a lousy programmer he was, and that he incessantly pushed his programs on other people to read for errors and clumsiness. Yet the common image today of von Neumann is of the unparalleled computing genius--flawless in his every action. And indeed, there can be no doubt of von Neumann's Genius. His very ability to realize his human limitations put him head and shoulders above the average programmer today."

Weinburg tells another story of a programmer from the early days of space tracking systems. "Bill" was confident in a loop and gave it to "Marilyn" to informally review, as was common practice in the group. Marilyn found seventeen bugs in only thirteen statements. Bill chalked it up to a 'bad programming day' and became more and more amused as each bug was found. He took it as proof that it just wasn't his day and used the rest of the day giving everyone a laugh at his expense. Marilyn, on the other hand, did not have any false confidence in her own ability and took the code to a third person. She knew that she had looked at it so much she wasn't going to be able to see any more errors. She needed fresh eyes. With others help, she found three more errors before the day was over. The epilogue was that no more errors were found in the code despite diabolical testing and being in production in over a dozen installations even after nine years.

Why aren't there more groups like this? Weinburg asserts that such groups are not an isolated case. They regard their practices as valuable proprietary information, implying they think other would copy them if given a chance. These groups are very satisfied and stable which reduces the opportunities for programmers to have encountered these groups. Also, gypsy programmers - who never settle in one place - encourage the myth that the best programming is the product of genius, to justify ever-increasing consulting rates.

The last word of this section is on how there is an advantage to making your code as readable as possible for other programmers for that means that those who come after will have a chance to learn from such code without feeling the need to have it deciphered or translated. So, the best code isn't the stuff that is hardest to read but the easiest to read. Easier to test, maintain, understand, and learn from; who could argue with that?

Creating and Maintaining the Programming Environment

One danger programming environments face is the tendency to fixate or lock into some behavior, habit, or tool. Programming language is a common example. Even if another language is better suited for a problem, the language of choice will be preferred. More people know it, there are more tools available (to the group), more library routines, etc. In other words, it's "how we do things here" which makes choosing a different language like paddling upstream, lots of resistance and little to show for a great deal of effort.

Another fixation is the social environment which either encourages or discourages egoless programming. If the new guy is ridiculed when asking another for help or advice, he is less likely to seek assistance the next time. If, on the other hand, someone pays him the implied compliment by asking him to look over a program, he is more likely to feel secure enough to seek advice.

"To a large extent, we behave the way we see people behaving around us, so a functioning programming group will tend to socialize new members to its philosophy of programming."

On the subject of egoless programming and management, Weinburg astutely describes the culture shear between common management and uncommon programming groups. Remember that this book came out almost 40 years ago; you can draw your own conclusions about how much things have changed.

"Managers tend to select themselves from the 'aggressive' component of society and have difficulty appreciating the fact that other people do not completely share their goals of money and prestige. They are especially at a loss to understand the smooth functioning of a programming group based on mutual respect for individual talent and cooperation in the common cause. Instead, the tend to view people as working for money or under threat -- as they themselves do."

Another example given is of a group that was particularly success and producing a new system. The achievement was noticed by the company's management who decided to give them a cash award. Typically, they wanted to give the award to the groups manager but were bewildered when the manager would not accept it unless the whole group got the award. The groups manager understood that the group acted as a team and that responsibility for success could not be attributed to a single individual. Other managers thought he wanted more money, others thought he was trying to setup a "prima donna" group. The company decide to force him to take it and break up the group -- which seemed to have unhealthy ideas. For those doubters out there, I have personally witnessed this same scenario in my career. The moral of the story? Again, Weinburg says it as well as anyone,

"Had the management been more aware of what this group had to offer, and had they been more flexible, they might have worked out a solution that would have permitted this group to influence the work of other groups in a favorable way. But this is not easily done by managers, who tend to feel that when work gets done it is the direct result of the actions of some leader of outstanding ability. Even when the manager appreciates the work of the group, it is not consistent with his own philosophy to see the productivity of the group as a property of the group, not as a sum of the contributions of the individual members."

Summary

To understand how programmers perform in a group setting, you must consider formal and informal structures, the physical environment, and even the social interactions in play between members of the group. These are lessons which are still worth learning today. The hard question we must ask ourselves is why we keep searching for new solutions when we have not addressed the issues identified four decades ago. I maintain that the Agile Manifesto or the Manifesto for Software Craftsmanship, is but a few of the efforts to awaken the courage of individual programmers to affect change around them, and cause a groundswell of change which might have the momentum to disturb the inertia of the status-quo.


Sample questions:

For managers: What is your honest opinion of people who are not trying to "move up" in your organization, but who seem satisfied with the kind of work they do and the amount of money they get? To what extent is your view influenced by your own feelings for yourself?

For programmers: Do you refer to your work as "my" program? Try passing one week without using the personal possessive in reference to programs, and take notes on the effects you observe.


p.s - These chapters are large enough that I may break them down so each section is its own post.

Review - The Psychology of Computer Programming

I got a "new" book in the mail a few weeks ago and finally had time to crack it open, "The Psychology of Computer Programming" by Gerald M. Weinberg published in 1971. There is a Silver Anniversary edition that recently came out but I wanted a cheap copy in hardback so mine came from an online used book site. Why read a computer programming book that is almost as old as myself? Because it addresses the core issue of producing computer programs, the people who read, write, edit, maintain, or otherwise cuss at (other people's of course), code. People are the primary factor in all programs; before hardware, language, specifications, anything. If you don't understand how humans go about creating these abstract entities called programs, you will constantly be mystified by missed deadlines, frustrated by missing functionality, and stymied by endless scope creep. Enough of my pontificating, let's talk about the book.

Table of Contents

1. Programming as Human Performance




1. Reading Programs


2. What Makes a Good Program?


3. How Can We Study Programming?




2. Programming as a Social Activity




4. The Programming Group


5. The Programming Team


6. The Programming Project




3. Programming as an Individual Activity




7. Variations in the Programming Task


8. Personality Factors


9. Intelligence, or Problem-Solving Ability


10. 1Motivation, Training, and Experience




4. Programming Tools




11. Programming Languages


12. Some Principles for Programming Language Design


13. Other Programming Tools




5. Epilogue

"Computer programming is a human activity." A pretty bold thesis from the intro to Part I. Is there a mystique to programming? "Either you can program or you cannot. Some have it; some don't." Both quotes give you a good idea as to what is in this book, tackled expertly by Mr. Weinberg.

Chapter 1 - Reading Programs

Some years ago, when COBOL was the great white programming hope, one heard much talk of the possibility of executives being able to read programs. Weith the perspective of time, we can see that this claim was merely inteded to attract the funds of exectutives who hoped to free themselves from bondage to their programmers. Nobody can seriously have believed that executives could read programs. Why should they? Even programmers do not read programs.

I hear a similar story from when assemblers were introduced - as in, "With the development of assemblers, we won't need programmers anymore!" I believe similar statements have been propagated by hordes of 4th generation language and CASE salesmen.

"Programming is, among other things, a kind of writing."

This is not a very mainstream view in the programming world even though we work with constructs which are literally "languages".

"We shall need a method of approach for reading programs, for, unlike novels, the best way to read them is not always from beginning to end. They are not even like mysteries, where we can turn to the penultimate page for the best part -- or like sexy books, which we can let fall open to the most creased pages in order to find the warmest passages. No, the good parts of a program are not found in any necessary places -- although we will later see how we can discover crucial sections for such tasks as debugging and optimization. Instead, we might base our reading on a conceptual framework consisting of the origin of each part. In other words, as we look at each peice of code, we ask ourselves the questions, 'Why is this piece here?'"

The author begins examining a section of PL/I code showing how certain machine, language, and human limitations influence a program. Machine limitations like lack of memory to hold the entire problem set in memory at the same time causing the use of two loops instead of one. Language limitations like the lack of an end-of-file indicator which forces the operators to include a special character (or card in the 1970's punch-card centric world) and causes the programmer to account for the special character in code. Programmer limitations like not really understanding array notation of the language, in this instance PL/I. As a program is modified over time, machines change, languages are updated, and programmers come and go.

"And so, some years later, a novice programmer who is given the job of modifying this program will congratulate himself for knowing more about PL/I than the person who originally wrote this program. Since that person is probably his supervisor, an unhealthy attitude may develop -- which, incidentally, is another psychological reality of programming life which we shall have to face eventually."

Some programs have inscrutable logic like the use of special characters which are ordinarily invalid; "magic numbers" used for interim states for some long-forgotten problem.

"Once the patch was made, it worked so well that everyone forgot about it -- more psychology -- and there it sat until unearthed many years later by two archeologist programmers."

To different versions of the same PL/I program are compared, the first includes many limitations of which we've already spoken, the second much improved by removal of the previous limitations. Of the comparison,

"When we look at the difference between Figures 1-1 and 1-4, we might begin to believe that very little of the coding that is done in the world has much to do with the problems we are trying to solve"

Would we be any better off if we could use the latest code (Figure 1-4) as our spec?

"Specifications evolve together with programs and programmers. Writing a program is a process of learning -- both for the programmer and the person who commissions the program. Moreover, this learning takes place in the context of a particular machine, a particular programming language, a particular programmer or programming team in a particular working environment, and a particular set of historical events that determine not just the form of the code but also what the code does!

In a way, the most important reason for stufying the process by which programs are written by people is not to make the programs more efficient, more compact, cheaper, or more easily understood. Instead, the most important gain is the prospect of getting from our programs what we really want -- rather than just whatever we can manage to produce in our fumbling, bumbling way"

Hopefully the reason to study a 30+ year-old book is apparent; those who fail to learn from history are bound to repeat it.

Let's study historical mistakes so we can make our own new mistakes instead of repeating mistakes of programmers long-retired.

The rest of the book is just as rich in lessons we can still take to heart. By approaching the book chapter by chapter, I'm hoping to increase my chances of getting all the way through without writing a 50-page article all at once which would never get finished.

Here is the link for the next chapter: Chapter 2 - What Makes a Good Program

The Management Delusion

A good friend of mine sent me this:

Subject: New Stroustrup quote

"The companies are complaining because they are hurting. They can't
produce quality products as cheaply, as reliably, and as quickly as they
would like. They correctly see a shortage of good developers as a part
of the problem. What they generally don't see is that inserting a good
developer into a culture designed to constrain semi-skilled programmers
from doing harm is pointless because the rules/culture will constrain
the new developer from doing anything significantly new and better."

Those companies are in a bed of their own making. They want software written on a schedule which means hiring warm bodies who have the minimum training, enough to run a compiler, and can crank out code on a consistent basis; consistently low quality and in a consistently long time. When the pain of low quality software finally registers through their management-induced fog, they look for answers. What keeps these "Leaders" from admitting their own contribution to the problem?

Leadership sometimes means making hard choices, almost guaranteed to make someone unhappy. Leadership sometimes means suspending self-doubt to enable making those hard choices without the burden of guilt. Consider car company executives contemplating closing a factory - let all the employees down through bankruptcy or let a smaller set of employees down through a layoff? The best example of this would be President Truman's decision to use the A-bomb on Japan. Good luck trying get them to second-guess themselves - to do so would mean letting self-doubt out of the tiny box in which they hide it. Now, for the leader of the free world, we can understand why self-doubt is a luxury they can't afford.

What about the corporate world? Well, the upper-level executives may catch a break when they have to make truly gut-wrenching decisions like the car company example above. Middle and line-level managers do not have those reasons to hide behind so what's their story? Even without earth-shattering, bet-the-company decisions to make, a manager faces the same dilemma, how to isolate their own sense of self-worth from being damaged by potentially bad decisions. One way of doing this is by never looking back, never re-examining prior decisions, not entertaining the idea that they made a bad choice which is, by the way, popular with leaders at any level.

This may keep them from being paralyzed with self-doubt or wracked with guilt but is also makes them oblivious to their own short-comings. It takes a strong leader to learn from their mistakes, to emerge from self-examination stronger because they own up to their mistakes and learn from them.

Here then is the key. Allow me to extend the quote to the next logical step. Companies that are complaining about the lack of good developers are in a state of denial. What they are really complaining about is how few good developers want to work for them. There can be many reasons from an environment that accepts mediocrity to the unwillingness to pay the good ones what they're worth. In the first case it could be the red-tape just to get their development workstation setup, "What do you mean I have to get each Eclipse plugin on the approved software list!?!?" In the second case it could be failing to attract good talent; after all its only fair for a programmer that is 10x better than average to get paid 10x the average, right? Describing your workplace as "We have reasonable development policies and reasonably good salaries" you should not be surprised that you have reasonably talented programmers. To deny this is effectively admitting to self-delusion.

So, when your boss complains about the lack of good developers, they are really trying to deny their own reality and shift the blame away from themselves.

This reminds me of a story about 10 hiring managers who all insist they hire only the top 10% but I'll save that for next time.