Programming as an Individual Activity

I've decided to come back to Weinberg's classic 'The Psychology of Computer Programming' since it is at the heart of why I took up writing about programming in the first place.  At my first job oh so long ago, it was obvious that we were nowhere close to a solution to programming-in-the-large.  With applications small enough to be written by a single developer we could manage to deliver software that works, maybe it was inefficient or clunky but it worked well enough.  Sure large systems were built using the waterfall approach but they would invariably overrun both their budget and their schedule.  The industry was by no means young, even though the PC revolution was in full swing, the underpinnings had already been discussed ad nauseum. You could dismiss the waterfall method but would still be left with the same questions, 'what do they want?', 'how do we build it?', 'how long will it take?', and so forth.  The Agile or Lean movements have been able to change the discussion to ask 'what is the smallest amount of work that we need to address at one time' and go from there.

I believe that we have been avoiding the most important questions - which are uncomfortable to contemplate - like, "what is the largest single aspect that controls a software schedule?" or "How do we plan a project when the members of the team are not pluggable resources?"

This brings me back to one of the biggest unresolved question of our field, "Why can't we engineer software like we engineer buildings or spacecraft?"  I ask this not for serious comparison since the differences are legion, but rather to point at the single most important advancement in the field of engineering since the advent of the first simple tools (the lever, the wheel, etc), calculus. 

Why calculus you ask? because it gives us a way to plan for infinite variability and still have confidence in the result.  Planning a trip to the moon would be impossible without being able to calculate how much fuel is needed since how much fuel is needed depends upon the weight of the rocket which creates a recursive dependency.

Software will have no equivalent for calculus because the variability comes from the people participating in the project.  Until we accept that people are a first-order component of any software project we will continue to look to silver-bullets, the next development process fad, or the latest new-fangled technology for relief.

Pattern Ignorance

I've been in graduate school for a few years now and keep asking myself how we could engender good habits in students - like any good Codewright - during their education, like usage of version control or refactoring.

My database class provides a good example of the potential and the obstacles. The first project involves creating a schema where the tables have contraints that prevent the tables to be created all at once and those same constraints make it impossible to load the sample data all at once. You don't need the actual schema, I'm just describing a bit of complexity where it influences the assignment.

So the data is provided in text files in a comma seperated form and the approach I decided upon is simply parsing the values and then generate SQL statements in the right order.  First off, I thought about using an ORM framework, like Hibernate, but have no experience with one.  I did find a CSV library to parse the data which cut down the code I had to write but will complicate the grading because the TA will need to download and extract the library in order to run what I submit so I'll have to include those details in a readme file.

So, processing a single file is fairly straightforward; parse the file into a list of lines, loop over the lines, generate the appropriate SQL statement for the data from that file. I soon found myself repeating code for looping over the lines since the files for each table needed slightly different SQL statements.  In rejecting the ORM approach, avoiding the definition of domain objects as simple containers for data -- I also didn't really need domain objects for each file because there was not going to be any behaviour, not to mention the lack of a data framework -- I continued the straight text approach.  Looping over the lines in the first file (Employee), it was easy to directly call a routine to generate SQL rows.  This caused a problem when I wanted to reuse the looping structure for the next file (Departments).  I could leave the loop and add an argument but that would require defining an interface and implementing a distinct class for each file. Avoiding data objects puts me on the slippery slope of avoiding other patterns too; what I keep telling myself is it's only for this one project, it's due in two days, and the TA isn't going to give bonus points for using nifty patterns assuming he's ever been exposed to them.

In short, there is no incentive to produce well-written code.  The assignment is graded by evaluating the results without regard for the methods used to produce those results. 

SharePoint 2010 Custom IFilter development update

The previous post on IFilter development for Windows and SharePoint, I included several links to resources we used to code a custom IFilter; the hope being they would be useful to anyone looking to write their own, seeing as how difficult it was to find accurate information on the subject.

Alex Culp did an outstanding job with the IFilter, so much so that Microsoft asked him to write a MSDN article on the subject of writing a custom IFilter.  The custom file type wasn't the issue, it was all the complexities of C++ and IFilter development combined with registry entries, properties, 32bit/64bit details and so forth, like how the 64-bit version of Visual Studio creates a 32-bit installer by default which then didn't install the correct version of the .Net dlls.  Another bit-ness nugget is to pay attention to which Regedit you run when setting registry entries; 64-bit entries are not visible to the 32-bit executable.

Mysteries of IFilter Development Revealed

Alex Culp and I worked on a IFilter for a customer file type.  His page on writing an ifilter is down while he works on publishing an article on writing ifilters for Microsoft. [Update: His MSDN article on IFilter development has been published, see my post, http://codewright.blogspot.com/2011/12/sharepoint-2010-custom-ifilter.html]  In the meantime, I figured I'd share my list of IFilter development links.  Considering all the other technology out there and the age of the ifilter interface, I was surprised how difficult it was to find useful information.

A good start is this MSDN page: http://msdn.microsoft.com/en-us/library/ms916793.aspx. It's somewhat old but does cover lots of the details involved. We had the hardest time finding out how to create a custom property or property set for the ifilter to use when it found metadata.  We ended up using an existing category, Basic, and creating property ID's above the range of existing properties, i.e. we started at 200 since the existing Basic properties didn't get above 100.

Here's info on debugging IFilters in MOSS. We did have trouble getting all the registry entries needed to properly register the IFilter with the server. Keep a log of the entries, there are quite a few and in confusingly similiar places, especially since GUIDs are involved.

Here is another page on ifilter development.  Keep reading through the comments, lots more details are discussed there.

IFilter explorer. Shows information about the installed ifilters.

Good article series from Anne Stenberg on crawled properties and how to find the one you want to map. This is useful in understanding more about a crawled property, like when you want to make your own.

This post mentions a dev guide to writing your own SP2007 protocol handler. Might be a good resource.

Martin Fowler, Alistair Cockburn, and Optimism

The effort to make Software Engineering actually live up to the 'Engineering' part has been greatly helped by Martin Fowler's book on refactoring. Martin recently posted about his decision to not participate in an effort by others in the industry, not lacking in optimism, to define a common process for software development.

Martin Fowler declined to get involved in a similar effort and for the same reason.  The key quote is,

“Why this is so was primarily crystallized for me by Alistair Cockburn who explained that since people are the central element in software development, and people are inherently non-linear and unpredictable - such an effort is fundamentally doomed.”

While my previous post about Alistair's paper covers much of the same ground I felt it was worthy of an update.  There will always be souls who crave consistency just as there will always be salesmen willing to offer the illusion of it.  Facing our own nature is one of life's hardest lessons. Some of those who ignore the human factor are optimists seeking to improve their chances of consistency, which has driven many towards the latest and greatest geegaw, a new language, a new GUI, a new design tool. Others are simply deluding themselves in the hopes that if only people would sit up and fly right (read this to mean 'do it their way') all our problems would be solved. 

Studying the process of computer programming has been around for decades.   Even so, we still have trouble internalizing human foibles - people aren't predictable and follow "standards" about as well as a random-walk.  So the most successful way to get people to be consistent is to lead by example, blaze a trail ahead and show them the way.  In this manner Extreme Programming was its own worst enemy and its own best cheerleader. By showing how a set of practices work in situ, it created a much-needed focus. The "Waterfall" model was the favorite whipping-boy, essentially dead, and the iterative approach was in its infancy.  So XP served as a lightning-rod for discussions on the craft of software development.  It provided an example of practices that could be emulated and provided an easy target for the doubters, disbelievers, and denialists; no doubt the negativity coming from the critics was channeling  their innate understanding of the darkside of human behavior.

While Extreme Programming hasn't become the standard development model, that doesn't mean it failed.  When the history of Software Development is written, XP will be given credit for re-introducing the most important factor; not tools nor process, but people.

Life's Too Short To Work For Clowns.

One of the cliche's of our culture is the little boy who rebels against his parents and dreams of running away to join the circus. His mind is filled with the glamor of the Big Top, the exotic animals, the independence of wandering the country from one town to the next. He hasn't really thought about what it means to work in a circus. The Ringmaster did many menial jobs before arriving in the spotlight. The animals have to be cared for; double the time it takes to feed them when you consider both ends of the animal. Traveling from town to town may sound romantic until you realize it means you spend more time traveling than in one place, lots of lonely nights on the road regardless of the weather or climate because 'the show must go on" after all. The life of a circus is not just popcorn, parades, and pretty girls.

I ran across a quote in Christopher Duncan's, "The Career Programmer: Guerilla Tactics for an Imperfect World" that made me both laugh and sigh.

You need to consider one additional thing when trying to convince management the current approach to developing software needs improvement. What do you do if non of this sways them and they insist on continuing with chaotic development approaches that consistently create disasters and stress? Update your resume. Life's too short to work for clowns.

So the question becomes, do you work for clowns? Does the development process at your company make a circus look like a well-oiled machine? You might work for a clown.

I plan on putting together a complement to The Joel Test where instead of a list of things a company does before you consider working for them, The Clown Test would be a list of things that indicate you might work for clowns.

This might describe your boss,

• if he thinks an estimate is a promise,
• if he thinks developers are interchangeable with no effect on the schedule,
• if he thinks its ok for the lead developer to publicly berate a junior developer,
• if he thinks requirements documents are a waste of time,
• if he thinks the only place for designers is at a fashion show,
• if he thinks he just needs one good programmer who can code anything,
• if he thinks that people whine too much about teamwork,
• if he believes in Silver Bullets,

he might be a Tech Clown.

My apologies to Jeff Foxworthy.

When you run across a good example of a Tech Clown, send 'em my way and I'll add it to the list for every one's enjoyment.

The Myth of the Aging Developer

I question that I've heard throughout my career is, "What happens to older software developers?" The danger with this question is when the discussion turns to the implied version which is "The lack of older developers means that it is a young persons game." Nothing could be further from the truth..

My wife and I are both developers. We have both been working as developers for 15+ years. One thing we noticed was that there were a lot of people working as developers who really shouldn't be, they didn't have the aptitude, interest or sometimes even the background. It would be easy to dismiss the problem as a lack of proper education but that doesn't explain the developers who lack a CS degree but who can code circles around veteran devs. Those folks usually decide that programming is not for them and move on to different job. Some go into management, some become analysts, while others change careers completely.

What's left? The ones who are passionate about development and are interested in doing it right. They love learning new stuff and don't gripe about how things used to be in "The good ole' days." They might be grumpy but they usually know their stuff.

The main problem would be in finding management who doesn't have a pre-conceived notion about what makes a good developer. If they feel that only the younger ones have the stamina to keep up with the pace, I'd wonder whether those same employers were the same ones who believe that the 'Deathmarch' style of project management is standard operating procedure. They'd rather grind through fresh-meat developers (Code-Monkeys anyone?) than listen to the experienced ones and change their development process much less change their management style.

Lastly, I've noticed that doing nothing but coding holds less business value than being able to communicate with customers, users, management, and vendors. One reason you start coding less is because finding answers to ambiguous requirements takes more experience and provides more payback to the business.

My advice, to those who worry about their coding future, would be to find a place where they need experienced developers to help with the overall task of creating or maintaining software, that also gives you the opportunity to code regularly. That way you can have the time to keep up with industry changes while taking advantage of your understanding of software development.

If Gulliver Were a Designer

The term 'Design' gets thrown around a lot in software developer circles. There are constant flamewars over favorite techniques and battles over what good design really entails. When those arguments start getting too ad hominem figure out whether the speaker is a Code Monkey before giving them much credence. Since I've already braved the waters, see "Have Design Will Travel" or "The Age of the Design Review", instead of safely avoiding the issue, I'm going to bravely (or foolishly) join the fray.

Realize that there is more than one type of 'Design'. Design in the large turns into architecture and is something that really only comes with experience. Designing at the lower levels, say at the individual class/object level is easier to cover. Today I'm writing about design in the small vs the large; Lilliput vs. Brobdingnag.

The issues with designing a class is the same regardless of platform or language. The key is whether an object should be autonomous or whether it is better for any given behavior to be spread out among objects with limited scope and distributed responsibility.

For each class the answer might be different. We end up with a spectrum along which we can place classes based upon the Density of Responsibility.

             (Level of responsibility for behavior)
        Autonomy - - - - - - - - - - -  - - Dependence
     High
 C      -   [god object]                [spaghetti]
 l      -
 a      -
 s      -                                
 s      -           
        -                  
 s      -
 i      -
 z      -
 e      -  [template]                   [framework]
      low

Let's say you favor letting the class perform all the behaviors itself, or as many as you can. Starting on the left side of this graph, when you make your class more autonomous, the size of the class will grow unless you continuously refactor it to make it more generic. This leads to a template. If no refactoring is done, the tendency is for the class to become more "god-like" because if there is some behavior it needs, it has a method for that - it can do anything and everything. The number of fields and methods grow and soon become both unmanageable and unavoidable. Since the class already does so much, coders would rather add to the monstrosity than try to piece it apart and cut the Gordian knot.

The right side of the graph has classes that depend on other classes to a large degree. If the dependency level is high but the individual class is small, that is a sign of a framework; each class doesn't do much and requires lots of dependent classes to accomplish some function. On the other hand, a highly-dependent class that also has a large amount of code is a sign that the class is full of spaghetti.

The key to this question is to determine where you feel more comfortable on the graph. In any event, individual classes will end up spread out on the graph unless some organizational principle is applied, which is how you can achieve the results of Template or Framework.

Having just written that, I would say that there is a correlation between class size and degree of organization. Robert C. Martin (or "Uncle Bob") covers similar ground with package dependencies in his very thorough paper on Design Principles and Design Patterns[pdf]. JDepend is an implementation of the ideas behind the graph on page 26 and complements static analysis tools such as Checkstyle and PMD.

Chapter 5 - The Programming Team

This continues my review of "The Psychology of Computer Programming" by Gerald M. Weinberg.

So far we've had individual programmers either working solo or in an environment where other programmers work but on separate tasks. As long as a single person can maintain the conception of the program, conflict resolution involves the thinking of a single individual, regardless how much advice is available or heeded. When a problem is too large for one person to hold in their mind at once, the scale of the problem goes up due to the increasing communication needs but is now one of a different kind. In this second case,

"conflicting technical demands are translated into potential interpersonal conflicts, and a social mechanism must be formed to resolve them"

This is part of the territory. Two people with solutions that are mutually exclusive. Programmers are proud of their intelligence and are more than willing to argue their side.

How A Team Forms

"As a rough rule, three programmers organized into a team can do only twice the work of a single programmer of the same ability"

I'm no longer as convinced of this as I once was. It really depends on how well defined the separation is between the modules.

Establishing And Accepting Goals

"Social psychologists have verified in other contexts that failure of one or more members to share the group goals affects the group performance -- not only through that member's share but through a reduced performance on the part of the others, for they invariably perceive the division within the group or the indifference of one of the members."

A non-programming example of this is when football teams get a new coach; the TV pundits like to use the phrase "They've bought into the new system" meaning that the players have taken the coaches goals for their own. A common programming example you might see in a development shop is where the current toolset and technology stack used is from Microsoft and there is that one guy who keeps muttering under his breath that "Microsoft sucks". His lack of cooperation will be felt by the rest of the team. The result may be that they avoid him instead of sharing the new trick they learned to solve some daily annoyance. No one wants to pair-program with him. The price being paid, at a minimum, is the reduced productivity of the frustrated Microsoft-hater.

The Team In Crisis

"The replacement of a leader--or any team member--is probably the most frequent and typical crisis in the life of a team. The effect of adding or removing a member depends quite sensitively on the group structure, and many a manager has been unpleasantly surprised at the change in performance resulting from the removal or addition of an 'insignificant' group member. "

If we need proof that programming is a social activity, we need look no further. Unless you are the sole programmer working at a company or on a project, there will always be a social aspect to what you do. I'll go even farther and say that unless you are the only person who will use the application, there will inevitably be people involved, from the person who asked you to write the app to the person who will be using the app.

"A member who is competent but who does not get along with the others can be an even more serious problem for the democratic group than an out-and-out incompetent. In an authoritarian group, such a member would not have much contact with others on a working basis anyway, so as long as he gets along adequately with the leader, he presents no particular problem. Indeed, some programmers prefer to work under a strong, centralized leader in order that they do not have to socialize with their fellow workers. But in a democratic team, an antisocial member cuts lines of communications and is a constant impediment to consensus in team meetings."

Sometimes we are our own worst enemy. If we want to stay our of our own way, we need to remember in software development, people are the biggest cause of failure.

Summary

Questions

For Managers:

- Are your hiring practices such that you get more uniformity on teams that you would like? When making up teams, do you try to see that a good "mix" of people is on each one, or do you strive in the opposite direction?

- Do you ever do things to try to inflate the appearance of your technical competence in front of the people who work for you? Describe some of these incidents, and also some incidents in which it was discovered that your technical competence was in at least one respect inferieor to one of the people who work for you. What were the consequences of that discovery, and do they justify attempting to cover up?

- In setting your own working goals, what part is set by what is passed down from above, and what part is set by what comes up from below? Are you satisfied with this arrangement, or would you like to alter it in some ways?


For Programmers:

- What part of programming work do you do best? Are you permitted to contribute that best part of your work to your team, and is it generally recognized that it is the best part of your work?

- Has your manager ever done anything to make you doubt his honesty? If so, describe the incident, what ultimately happened to your doubt, and how your work was affected.

Peanut Butter Programming

When I first started messing with dynamic web pages it was using Perl and CGI. It was fun at first getting a page to respond based on some algorithm. It was also easy to overlook the growing mound of HTML strings embedded in your program. It soon became apparent that this approach doesn't scale very well. Keeping track of which tag needed to be closed and when was too much like keeping track of dynamic memory allocations in C.

Java came along with Java Servlet Pages (JSPs) and it seemed like a huge improvement. By inverting the relationship between code and HTML, you didn't have to have your HTML all strung out in little bits in the program. When you wanted to include some program logic in your page, you had two choices: write an extension so your logic could be expressed like an HTML tag, or you could embed it as a code-snippet directly in with the HTML. While there were plenty of examples of how to create your own tags, the overwhelming amount of JSP code I've seen took the other approach. To the extreme. When some behavior was needed in multiple places, sometimes it refactored into a common file and then referred to from multiple pages. Other times (which translates to whenever possible), it was copied and pasted around with the justification of "I just need it to work". The result was a file with little bits of program strung out through your HTML. Sometimes mixing two things can prove to be a good thing. remember the the TV commercial for peanut butter cups? "You have your peanut butter in my chocolate! No, you have your chocolate in my peanut butter!" Unlike that TV commercial, mixing HTML and code is a recipe for heartburn. Peanut butter may be tasty but will stick to everything it touches and you'll end up with a greasy mess to clean up if you're not careful. That's why I call this "Peanut Butter Programming".

Why do developers keep using approaches like 'cut-n-paste' even though they usually know better? When the choice is between doing the expedient thing and the right, people typically want to do the right thing but only to a point. If 'cut-n-paste' is simpler or easier than doing the right thing, the playing field becomes so tilted that some other motivation needs to be strong enough to deter choosing the short-term solution. Code standards are one example of that kind of motivation. They express a value judgment that the increasing maintenance costs of supporting bad code is not worth the time saved by taking short-cuts. If the programmer knows he'll have to face his peers and explain why he took a short-cut, his choice becomes "I can do it quick-and-dirty (QaD) in 5 minutes instead of the right way which would take 15 minutes. But if I do it QaD, I'll have to spend 30 minutes arguing at the code-review. It's not worth the argument so I'll just do it the right way and save myself the trouble." This sort of balancing act shows up naturally during pair-programming where having the second coder around makes it harder to justify writing messy code on the first pass.

Recently, I went to a class on Microsoft's ASP.NET technology. Setting aside any feelings for the company, the way they approached the problem of mixing code and HTML was to make it easier to do the right thing. They introduced the concept of a separate 'Code Behind' file which contains the C# or VB code for a given page. By making it easy to manipulate objects on the page and respond to events while keeping the code separate, they've made the right way also be the easiest way.

Even experienced developers will justify taking a short-cut, however short-sighted, if doing the right thing is hard enough to do relative to the short-cut. Different people have different tolerance level for short-cuts. Understanding why people do the things they do isn't easy but gives us the perspective needed to address the problem.

Software as Literature

Alternative title: Sonnets to CPUs

Have you ever noticed that code follows some basic patterns? JUnit uses Setup(), Test(), Teardown().

I’ve playfully mixed writing and coding below.

---

//[Play about A Monarch's Destiny, in three acts]
Public KingdomFuture tellTheTaleOfMyRule(Kingdom landMyFatherLeftMe)
{
   //[1st Act:  Build Drama]
   Vassal mySubjects = SoliloquyAboutMyKingdom.getMyPeasants();
   Boolean mySubjectsLoyalty = testOfLoyalty(mySubject);

   Vassal theCaptainOfTheGuard = SoliloquyAboutMyKingdom.getMyTrustedRightHandMan();
   Boolean myTrustedServantsLoyalty = testOfLoyalty(theCaptainOfTheGuard);

   KingdomFuture historysTaleOfMyRule = KingdomFuture.UNDECIDED;

   //[2nd Act: Drama plays out]
   if ( mySubjectsLoyalty == TRUE 
        && myTrustedServantsLoyalty == TRUE)
   {
       HeapPraiseUpon(theCaptainOfTheGuard);
       historysTaleOfMyRule = ExpandKingdomByForce(mySubjects);
   }
   if ( mySubjectsLoyalty == FALSE 
        && myTrustedServantsLoyalty == FALSE)
   {
       ArrestTraitor(theCaptainOfTheGuard);
       historysTaleOfMyRule = ImposeDispicableLaws(mySubjects);
   }

   //[3rd Act: resolve drama]
   if ( deceptionHiddenSuccessfully(theCaptainOfTheGuard) )
   {
       return oldRulerOverthrown;
   }
   else
   { 
      return historysTaleOfMyRule;
   }
}

Tower of Babbling

When I started coding at my first job almost 20 years ago, it was at an OS/2 shop using C and Presentation Manager. I had several different languages in college so I wasn't too phased by the 1-2 week C training course we got. What I wasn't prepared for was memory management, C style string handling, message-driven applications, just to name a few.

I began my dislike of C early with the insanity called string handling. I was pleased when I found out that Joel agrees with me. Although there are plenty of people who will defend how 'intuitive' the string functions are, I knew there were much simpler ways to manipulate strings as long as you didn't need one larger than 255 characters, in which case using a char* buffer would be fine. Writing apps in something other than C was dismissed offhand with statements like, "C is much closer to the metal", "You don't have the run-time bloat of other languages", or "You have so much more control with C" which almost sounds like a masculinity test or something. Think lumberjacks and "my chainsaw is bigger so only the strongest can use it" type argument. Nowadays, this type of behavior would remind me of The Code Monkey.

Even 2 decades ago, it was possible to call code written in one language to call code written in another; in school we called assembly graphics routines from within Ada code for one class assignment. Notable exceptions are running compiled code from within scripts or executing a system command from within compiled code; but rarely do we write a library in C++ to use in our Java app. When Java first appeared and it was still slow, I never understood the resistance to writing the main app in Java and moving the bottleneck code into some C/C++ dll and called using JNI. It's like we use the "language X sucks at a, b, and c so we can't use it" as a shield that keeps us from having to design multi-language applications.

So, after all these years, what keeps us from writing multi-language apps?

An Apprentice of His Very Own

In my career as a developer, I reached a point where the amount of work that needed to be done was much greater than I had time to do myself. Budget realities aside, there was a lot of coding tasks like writing more unit tests or keeping various diagrams up-to-date that would be tedious but easy to accomplish as long as a basic understanding of development was present. Wouldn't it be nice to be able to afford to hire your own apprentice? Someone maybe still in college who had some exposure to coding but was still wet behind the ears enough to keep quiet and learn by osmosis.

At the time, I had visions of a helper who could take rough UML diagrams, like from scratch paper or from a white-board session, and update the project documentation. As each change came through, they would create new versions and archive old ones like a file clerk from decades ago.

First off, a fresh-from-college developer might consider that type of work as beneath him or her. I'd argue that learning the processes is just as valuable as learning how to code. Did they learn about version control, shared builds or managing a large code-base in school? Of course not, but experience with those types of processes are what most junior developers lack.

Secondly, if your job is consistently repeatable enough to explain to an apprentice, why are you there in the first place? The apprentice is much cheaper and easier to replace. More than likely, you job is one of those ones which is hard to explain because it relies on your experience and training to understand how to approach the problem and the possible/practical solutions.

Usually I stay away from comparing software with construction but in this respect, iron workers have a similar problem: how to bring on new blood without getting anyone hurt or jeopardizing the project. Even the most junior iron worker needs to understand how to do his small part and stay safe. The head iron worker needs to trust that his men won't need explicit instructions for every task and that when a crisis erupts, they'll understand what the priorities are without having to stop and have a meeting.

So, what you end up needing is someone who wants to learn, has the patience to listen, the intelligence to follow directions, the determination to keep doing hard work even when they don't know the importance of their contribution, and the wisdom to understand that not all knowledge comes from a book. The next step is to think about how to carve up your work so someone can help you with it. That will have to wait for another day.

We Have Met The Enemy

In the previous post "Why Extreme Programming Fails"I attempted to shine a light on a dirty little secret of software development. The biggest obstacle was, is, and shall be, the people writing software. Do I condone the 48 Laws of Power, of course not. But to ignore the realities of human behavior in building software would be as foolish as building an elaborate sandcastle on a beach below the high-tide line and expect to hold back the water from destroying all your hard work. Programming is a social activity. See my review of Weinberg's book at http://codewright.blogspot.com/2009/04/programming-as-social-activity.html.

Sun Tzu wrote that you should know yourself and your opponent to have the best chance of victory. Understanding your team is the best place to start to ward off non-technical issues. Admitting that people have hidden motives is not to say those motives are good but to realize how situations should be approached or to understand why a given developer is so hard to work with. The 48 Laws are stated in an almost malevolent way in an effort to illustrate how people act at the extreme. Law #1, "Never Outshine the Master" can be taken either as a command for those who want to curry favor or it can be taken as a reason why a long-time developer may not take it well when showed up on a coding task by a new hire. At their foundation, the laws show behavior issues where ego, mistrust, control, and laziness are the motivations instead of loyalty, honesty, or integrity. We don't have the luxury of assuming everybody in the workforce is on the up-and-up or motivated to do a good job; once we learn that lesson we can begin to steer our own course, taking detours around those who put obstacles in our way instead of being at the mercy of those whose desires are as capricious as they are destructive.

Why Extreme Programming Fails

In the decades of the young life of software development, we've discovered many behaviors, processes, and techniques which have nothing to do with software yet continue to hold back progress. We ask time and time again why the mythical silver bullet is still searched for by those who have already agreed that it is a fools errand. Why are death-marches allowed to form when the futility is obvious to all but the oblivious? Why are proven techniques not mandated by management or agreed to by the workers who constantly complain about the current poor practices? This is true whether we're discussing XP or Agile or SCRUM.

We can use rational thought to decipher the mystery, attempting to achieve clarity. However much success is possible in finding rational explanations, there comes a time when we must recognize the difference between clarity and understanding. We can state things extremely clearly and succinctly and still fail to understand why people continue their unproductive habits or why change is viewed by those in charge with such disdain.

Submitted here for your edification, The 48 Laws of Power.

Why does that developer refuse to pair with someone? There are lots of reasons, i.e. Law #1 - Never Outshine the Master or law #30 - Make your Accomplishments Seem Effortless, are good places to start. Why is Continuous Integration scoffed at? See law #11 - Learn to Keep People Dependent on You

The words 'politics' or 'management' may seem like pejoratives but they serve as simple placeholders for the understanding of human behavior. At their worst, they can be weapons in a Machiavellian conflict while at their best they can be the protective shield against those who see your efforts as a challenge to their plans. Is every use of these laws an instance of plotting and scheming? No, never forget that human nature operates below the level of conscious thought; these behaviors are ancient artifacts used for thousands of years to survive in a violent and irrational world. Disagree with them all you'd like but ignoring them can easily cost you your job.

The Myth of Code Construction

Ever work with a developer who avoided letting anyone see their code, refused to program in pairs or participate in a code review? They say "This is my code" as if they have sole right of refusal on the final product regardless of who else may be paying the bill.

Great writers still have editors. No matter how great a writer you are, you still have to submit your work to an editor before it will be published. Let's examine the idea of putting code into production as a form of publication.

Before we jump into a new analogy, we should look at the current state of affairs. If engineering were a family construction would be the successful older brother leaving software as the immature younger brother, never quite living up to the accolades of the elder sibling yet endlessly compared nonetheless.

In general, the phases in software are: analysis, design, construction, test, deployment, and training. When you want to build a building of any size, let's use a downtown skyscraper as an example. If we continue the comparison we'd start with analysis which includes many factors like the target location or how many square feet are required. The location can impose limitations to the footprint which then dictates how many floors would be needed to achieve the required square footage. Such trade-offs are examined and decided upon during the design phase. Construction is all about execution with limited opportunity for design changes. How do you test a building, what does that even mean? You might might test the subsystems, like elevators, fire alarms, etc. but how do you test the entire building? The closest we can come in scale might be the opening and closing the retractable roof of an almost complete stadium like the new Cowboy's stadium (this being written in May of 2009). How do you deploy a building? You don't, you open it for business. There is no pull of the lever where what did not exist suddenly does. As for iterations, the closest construction can come is building a new wing on an existing building, adding a room to the house at hand. The cost and effort of remodeling can end up being a significant ratio to the original price of the house.

Now let's discuss a comparison of software development with writing a book. A publisher decides the market is ready for a new book on ant farming. They either have recent submissions by authors interested in writing such a book or they solicit existing authors who have experience in the subject matter. In any event, they request an outline of the book with maybe a sample of one chapter to gauge the material matches what the publish wants. The author or authors work at creating the manuscript all the while knowing that editors will read the book and suggest changes. Editors who are not experts in the content but are experts in the publishing of books. Even the highest-paid authors - Mary Higgins-Clark, Stephen King, or J. K. Rowlings to name a few - will all submit their work to the editors pen.

When the manuscript is complete and the editors happy, there are still other contributors to the final product, researchers, artists, designers, and the like who do things like create illustrations, populate an index, check facts, or plan the visual layout of the chapters. All these activities are coordinated by the publisher who has a target date for when the book will be printed, planning a marketing blitz to coincide with the publish date. Getting the manuscript done is important but still is a small part of the whole effort and if an author cannot produce on time, a lot of investment by the publisher goes to waste; a good way to dissuade the publishing company to find an author who can deliver. The key is that it doesn't have to be perfect but does have to be complete. If there is enough demand, it is easier to publish a new edition rather than hold off on the initial printing. Testing is as easy as a small test run where the final product can be validated by all contributors before scaling up into production-sized runs; images replaced, a new cover chosen, typos fixed. Printing becomes like running the build with a target of production. This means that "build" isn't a phase, it's an activity; unlike any in construction. Deployment becomes a straightforward, if not simple matter, of distributing copies of the book to various retail book-sellers or their distributors.

My suggestion is for the software developers of the world realize that allowing our craft to be constantly compared to the construction industry is to remain the little boy, wanting to live up to the expectations of those who came before, never earning the respect or acceptance they so long for. Look instead to the world of publishing, where writing is still an art and yet the businesspeople have found a way to make money without rejecting the fundamental nature of writing, be that a novel, the latest tech book, why not the latest edition of your favorite word processor?

p.s. Kudos to the wife for the idea.

An Estimate By Any Other Name

Terri Vajdi, a colleague of mine, sent me this response to my post on estimating. She always had the most thorough estimates I've seen in my career, so I have a lot of respect for what she has to say.

Since I drill down to assume the details of the requirements when it's missing, I equate this to all the missed requirements to a certain percentage. I read the requirements and try to envision the amount of code, junits, number of iterations, and unit testing for each iteration, Code Review and implementing the changes based on the Code Review, etc...
So, I come up with an estimate for each of these and then multiply it by the percentage/factor for the missing pieces. Add this the estimate for the missing pieces to my estimate to come up with the estimate that I turn over to my manager. In majority of cases, I break even and come close to the estimate that I have given. Of course, this is with putting a lot of overtime to meet my estimate. Remember we do not live in and ideal world and there are a lot of mishaps. This is just my two cents. Of course, I'm a quality oriented individual and I do not like to revisit my code once it's in Production. I figure that number of iterations after it has gone to Production is very very expensive. I'd rather spent it up front. So far, I've been able to stick to my methodology and have been able to spend my time wisely on other assignments once the code is in Production.

Sometimes I'm too eager and go light on my estimates which would mean even more work later and for which I've been criticized when the unknown issues would crop up and the project would run late. Terri has the right idea. She passed along this link:
http://www.apl.jhu.edu/Classes/635431/felikson/Publications/No_Silver_Bullet_Brooks.html

Anyone else have "Estimation Stories" to share?

Programming as a Social Activity

What do we imagine when we consider the individual programmer? There are many stories that liken the programmer to the cowboy who finds himself in a small remote town rife with lawlessness and misery. The cowboy grudgingly accepts the job as sheriff, to right wrongs and otherwise do deeds of much heroism; riding off into the sunset, when the crisis has been averted, to the wails of the local townfolk who think they are losing their only defense against the chaos that comes with frontier life. Is this how individual programmers actually work or is this a delusion used to supplicate the ego of socially inadequate developers?

This continues my review of "The Psychology of Computer Programming" by Gerald M. Weinberg. The previous chapter Chapter 3 - How Can We Study Programming discusses the difficulty of studying behaviors which are hard to isolate, repeat, or tranlate into an experimental setting. Chapter 2 - What Makes a Good Program evaluates the subjective and objective criteria for judging programs. The beginning of the series of reviews is Chapter 1 - Reading Programs. Techrepublic.com has a review of the same book which reinforces this book's classic status.

In Part 2 of "The Psychology of Computer Programming", Weinberg describes three different ways that programmers are organized: the group, the team, and the project. Chapter 4 is titled "The Programming Group". He starts with the observation that when programmers work together in the same environment, they inevitably seek each other out for help or advice and expresses the premise that the lone programmer is at a serious disadvantage.

Formal and Informal Organization

The first lesson is that in any assemblage of people there are formal and informal groups. Formal hierarchies like an organization chart might be nice for the manager types but do not begin to describe the various interactions between individuals in a group. One good story is from back in the batch-scheduled era when it was difficult to know whether your job had run and its output was available, which was accomplished by posted completed jobs on a board outside the computer area. Programmers in remote offices might waste half an hour coming over just to find their results were not ready. As it happened, there was secretary who sat in a location which allowed her to see the result board. When a programmer who was waiting for a job to complete happened to call her for a date. When the conversation turned back to work, she mentioned that he must have work to get back to. He told her that he was waiting for a job to complete. She volunteered that she could see the board from where she was sitting and asked him in which job he was interested. This began an informal service which quickly became known to the outlying programmers. The service became popular enough to disrupt the work of the secretary and need the attention of the administration. Wisely acknowledging that the service could not just be cut off, they decided to implement a sort of job posting hot-line, even using the secretary's original phone number, where a recording would give the latest job completion posting. Here an informal and inefficient solution was replaced with a formal but efficient one.

This contrasts with a second story where a university computing center provided a common room for working on programming problems and an adjoining room with graduate students available to answer questions ("consulting services" was the term used in the book), and at one end of the common area were vending machines. When students complained about the noise from the crowd of students around the vending machines, the administration removed the offending machines. Not long after the removal of the vending machines, another delegation came to the administration with a complaint, there wasn't enough consulting services. As it turns out, the students who were making all the noise were actually discussing their programs and helping each other with them, answering all sorts trivial questions in an informal setting. The administration, unable to believe that the removal of the vending machines was the actual problem tried to increase the number of consultants available with little success as fewer graduate students were willing to deal with the ever increasing number of mundane and trivial problems. The formal solution was not nearly as effective as the old informal one.

Physical Environment and Social Organization

Does the environment in which you work affect your productivity? Companies are all about getting the most for the least so it is no surprise that in devising the seating arrangements of programmers, minimum cost usually trumps maximum productivity. There are exceptions like early Microsoft and their "every programmer gets an office with a door" mantra, to more recent examples like Fog Creek, who also take a programmer-first approach. In Weinburg's words,

"Someone should definitely study the depressing effect that the all-too-common half partitions have on programmer productivity. They manage to cut off all useful communication while permitting all disturbing sound and movement to penetrate."

Another example was the building that replaced the old manual elevator with an automatic one. This was unfortunate for the programmers because the old elevator operator had run an informal pickup and delivery service for them between their floor -- the eighth-- and the machine room -- the basement. They saved the cost of the elevator operator but cost their programmers much time every day going back and forth to the basement of the building.

Error and Ego

"If asked, most programmers would probably say they preferred to work alone in a place where they wouldn't be disturbed by other people"

"The ideas expressed in the preceding paragraph are possibly the most formidable barrier to improved programming that we shall encounter."

Programmers tend to get invested in the programs they write, attached to them we'll say. This causes behaviors that are counter-productive. What's wrong with "owning" programs, after all artists "own" paintings; authors "own" books; architects "own" buildings. When we go to the bookstore we can get a good idea about the quality of a book if it by a familiar author. Wouldn't the same hold true for programs and their authors?

"The admiration of individual programmers cannot lead to an emulation of their work, but only to an affectation of their mannerisms. This is the same phenomenon we see in "art colonies," where everyone knows how to look like an artist, but few, if any, know how to paint like one."

Another problem with the idea of ownership happens when we start to evaluate or judge a work. Art is to a large degree subjective so the artist can dismiss the opinions of critics. Can a programmer dismiss the judgement of a computer? If we took the "ownership" association to its logical end we would reason something like this:

"This program is defective. This program is part of me, an extension of myself, even carrying my name. I am defective."

Obviously that does not happen. Programmers do not quit just because of a single bug or logic flaw but they will tend to overlook problems and strive to prove the program is correct even in the face of mounting evidence to the contrary. Proving the program is wrong turns into a minefield of self-worth where bugs become the buried mines which will injure the pride of the programmer who can't help but react in defense of his own sense of self-worth.

"Thus, if we are going to attack the problems of making good programs, and if we are going to start at the fundamental level of meeting specifications, we are going to have to do something about the perfectly normal human tendency to believe that ones "own" program is correct in the face of hard physical evidence to the contrary."

This explains the unconscious motivations behind the resistance to code-reviews, pair-programming, or any other effort to get them to loosen their hold on "their" code. If we saw this type of behavior in a toddler, we would immediately recognize it as childish, as in "James doesn't want to share his code with the others." Will James feel like he is losing something if he has to let others play with his code? It would go a long way to explain things I've seen in my programming career.

Egoless Programming

So, what do we do? The manager could insist that the coders work harder at finding the bugs in their code. He could even go around and ask them to show him the bugs in their code. Applying even the smallest amount of psychology would tell us that this will backfire. The average person will feel he is on trial and very easily take it personally. If we can change the social environment of the group, we have a chance to affect change in the values of the programmers of that group. This is possible, we know of egoless programming groups since the very early days of programming.

"John von Neumann himself was perhaps the first programmer to recognize his inadequacies with respect to examination of his own work. Those who knew him have said that he was constantly asserting what a lousy programmer he was, and that he incessantly pushed his programs on other people to read for errors and clumsiness. Yet the common image today of von Neumann is of the unparalleled computing genius--flawless in his every action. And indeed, there can be no doubt of von Neumann's Genius. His very ability to realize his human limitations put him head and shoulders above the average programmer today."

Weinburg tells another story of a programmer from the early days of space tracking systems. "Bill" was confident in a loop and gave it to "Marilyn" to informally review, as was common practice in the group. Marilyn found seventeen bugs in only thirteen statements. Bill chalked it up to a 'bad programming day' and became more and more amused as each bug was found. He took it as proof that it just wasn't his day and used the rest of the day giving everyone a laugh at his expense. Marilyn, on the other hand, did not have any false confidence in her own ability and took the code to a third person. She knew that she had looked at it so much she wasn't going to be able to see any more errors. She needed fresh eyes. With others help, she found three more errors before the day was over. The epilogue was that no more errors were found in the code despite diabolical testing and being in production in over a dozen installations even after nine years.

Why aren't there more groups like this? Weinburg asserts that such groups are not an isolated case. They regard their practices as valuable proprietary information, implying they think other would copy them if given a chance. These groups are very satisfied and stable which reduces the opportunities for programmers to have encountered these groups. Also, gypsy programmers - who never settle in one place - encourage the myth that the best programming is the product of genius, to justify ever-increasing consulting rates.

The last word of this section is on how there is an advantage to making your code as readable as possible for other programmers for that means that those who come after will have a chance to learn from such code without feeling the need to have it deciphered or translated. So, the best code isn't the stuff that is hardest to read but the easiest to read. Easier to test, maintain, understand, and learn from; who could argue with that?

Creating and Maintaining the Programming Environment

One danger programming environments face is the tendency to fixate or lock into some behavior, habit, or tool. Programming language is a common example. Even if another language is better suited for a problem, the language of choice will be preferred. More people know it, there are more tools available (to the group), more library routines, etc. In other words, it's "how we do things here" which makes choosing a different language like paddling upstream, lots of resistance and little to show for a great deal of effort.

Another fixation is the social environment which either encourages or discourages egoless programming. If the new guy is ridiculed when asking another for help or advice, he is less likely to seek assistance the next time. If, on the other hand, someone pays him the implied compliment by asking him to look over a program, he is more likely to feel secure enough to seek advice.

"To a large extent, we behave the way we see people behaving around us, so a functioning programming group will tend to socialize new members to its philosophy of programming."

On the subject of egoless programming and management, Weinburg astutely describes the culture shear between common management and uncommon programming groups. Remember that this book came out almost 40 years ago; you can draw your own conclusions about how much things have changed.

"Managers tend to select themselves from the 'aggressive' component of society and have difficulty appreciating the fact that other people do not completely share their goals of money and prestige. They are especially at a loss to understand the smooth functioning of a programming group based on mutual respect for individual talent and cooperation in the common cause. Instead, the tend to view people as working for money or under threat -- as they themselves do."

Another example given is of a group that was particularly success and producing a new system. The achievement was noticed by the company's management who decided to give them a cash award. Typically, they wanted to give the award to the groups manager but were bewildered when the manager would not accept it unless the whole group got the award. The groups manager understood that the group acted as a team and that responsibility for success could not be attributed to a single individual. Other managers thought he wanted more money, others thought he was trying to setup a "prima donna" group. The company decide to force him to take it and break up the group -- which seemed to have unhealthy ideas. For those doubters out there, I have personally witnessed this same scenario in my career. The moral of the story? Again, Weinburg says it as well as anyone,

"Had the management been more aware of what this group had to offer, and had they been more flexible, they might have worked out a solution that would have permitted this group to influence the work of other groups in a favorable way. But this is not easily done by managers, who tend to feel that when work gets done it is the direct result of the actions of some leader of outstanding ability. Even when the manager appreciates the work of the group, it is not consistent with his own philosophy to see the productivity of the group as a property of the group, not as a sum of the contributions of the individual members."

Summary

To understand how programmers perform in a group setting, you must consider formal and informal structures, the physical environment, and even the social interactions in play between members of the group. These are lessons which are still worth learning today. The hard question we must ask ourselves is why we keep searching for new solutions when we have not addressed the issues identified four decades ago. I maintain that the Agile Manifesto or the Manifesto for Software Craftsmanship, is but a few of the efforts to awaken the courage of individual programmers to affect change around them, and cause a groundswell of change which might have the momentum to disturb the inertia of the status-quo.


Sample questions:

For managers: What is your honest opinion of people who are not trying to "move up" in your organization, but who seem satisfied with the kind of work they do and the amount of money they get? To what extent is your view influenced by your own feelings for yourself?

For programmers: Do you refer to your work as "my" program? Try passing one week without using the personal possessive in reference to programs, and take notes on the effects you observe.


p.s - These chapters are large enough that I may break them down so each section is its own post.

What Makes A Good Program?

This is a continuation of the review of "The Psychology of Programming". You can find the post on Chapter 1 here.

Chapter 2 - What Makes A Good Program?

If we plan to study programming as a human activity, we are going to have to develop some measures of programming performance. That is, we are going to have some idea of what we mean when we say that one programmer is better than another, or one program is better than another. Although we all have opinions on these questions, we shall find that the answers are not as simple as we might wish. For programming is not just human behavior; it is complex human behavior.

What do we mean by complex human behavior? This article at betterexplained.com does a good job highlighting the false dichotomy between simplicity and powerful. Another explanation my good friend Scott sent me used simple, complicated, and complex. I'll let Dave Nicolette speak for himself.

The spectrum goes from easy to understand and predict to extremely hard to understand and predict (but possible with enough effort), to impossible to understand and predict regardless of the level of effort.

In our studies of the psychology of programming, we shall be hampered by our inability to measure the goodness of programs on an absolute scale.

Plenty of effort has been wasted trying to discover that mythical absolute scale; lines-of-code, lines per day, runtime, memory usage, the list goes on and on. The only thing that is agreed it would be that there is no single metric which can be applied to all programs.

Looking honestly at the situation, we are never looking for the best program, seldom looking for a good one, but always looking for one that meets the requirements.

This might be the hidden reason for why so much lip-service is paid to software quality. Crappy code that ready right now always beats higher-quality, however defined, that is late. Unless there is a business reason which drives a requirement connected to quality (like uptime, throughput, etc.) where time-to-market is trumped by proper outcomes, managers will revert to turning a blind-eye to quality. Risk deferred is risk ignored.


Specifications

Of all the requirements that we might place on a program, first and foremost is that it be correct. In other words, it should give the correct outputs for each possible input. This is what we mean when we say that a program "works," and it is often and truly said that "any program that works is better than any program that doesn't."

Here follows a good story about a system for a car maker to handle all the options which prospective car purchasers could order. When the programmer arrives on the scene of a disaster in the making, the long-settled approach used was not only overly complicated but didn't produce the correct results. When the programmer exclaimed that the Emporer had no clothes, he was condemned for being uncooperative. While on the way home, he can't stop thinking about the problem. In a fit of insight, he realizes that a workable approach was within reach. Upon returning to the customer site, he describes his solution. Naturally the audience gave him a cool reception but they listened without questions - until the original create of the old system raised his hand and asked, "And how long does _your_ program take?" to which the response was, "That varies with the input but on average, about ten seconds per card" (an indication of how old this story is). "Aha," was the triumphant reply. "But _my_ program takes only one second per card." This seemed to settle the issue with the audience until our protagonist observes,

"But your program doesn't work. If the program doesn't have to work, I can write one that takes one millisecond per card - and that's faster than our card reader."

Here is where the sarcastic could say "We just redefine what 'done' means", or "that isn't a bug, it's an undocumented feature"

In effect, then, there is a difference between a program written for one user and a piece of "software." When there are multiple users, there are multiple specifications. When there are multiple specifications, there are multiple definitions of when the program is working. In our discussions of programming practices, we are going to have to take into account the difference between programs developed for one user and programs developed for many. They will be evaluated differently, and they should be produced by different methods.

Imagine a scene from "I Love Lucy" where Ricky Ricardo and Fred Mertz are trying to rearrange furniture at the behest of Lucy and Ethel. "Move that painting to the other wall", Lucy commands. When she then turns her attention to having the men move the couch "a teensy bit" closer to the wall, Fred moves his side an inch while Ricky moves his side a foot; after all, how many "teensies" are in a foot? Ethel, meanwhile, sneaks over to the picture while everyone else is busy with the couch and moves the painting back to its original location, "It looked better that anyway", she thinks to herself.

However comical this might be, when this happens in software it can be many, many times worse. With multiple stakeholders, multiple upstream and downstream dependencies, and a large group of developers trying to write code, it should be easy to realize how requirements (and their management) can become a nightmare.

Schedule

One of the recurring problems in programming is meeting schedules, and a program that is late is often worthless.

Most people prefer to wait a fixed ten minutes for the bus each morning than to wait one minute on four days and tewenty-six minutes once a week. Even though the average wait is six minutes in the second case, the derangement caused by one long and unexpected delay more than compensates for this. This is where a psychological study would be rewarding. Project management is not a science and only partially is controlled by mathematics. Knowing the psychological landscape in which the project functions can make a big difference in its success.

Adaptability

Few programmers of any experience would contradict the assertion that most programs are modified in their lifetime. Why, then, when we are forced to modify programs do we find it such a Herculean task that w4e often decide to throw them away and start over? Reading programs gives us some insight, for we rarely find a program that contains any evidence of having been written with an eye to subsequent modification. But this is only a symptom, not a disease. Why, we should ask, do programmers, who know full well that programs will inevitably be modified, write programs with no thought to such modifications? Why, indeed, their programs sometimes look as if they had been devilishly contrived to resist modification - protected like the Pharaohs' tomb against all intruders?

This really goes to the heart of the quality software movement. If it's shown time and again that the majority of software cost is in maintenance, why do we fail so miserably to factor in maintainability while writing software?

Adaptability is not free. Sometimes, to be sure, we get a program that happens to be adaptable as well as satisfactory in all other ways, but we generally pay for what we get - and often fail to get what we pay for.

It's as if the manager figures that maintenance doesn't come out of his budget and she's only being measured by whether the project comes in on time even if the resulting code will cost twice as much to maintain.

Fisher's Fundamental Theorem states - in terms appropriate to the present context - that the better adapted a system is to a particular environment, the less adaptable it is to new environments.

This has large effects on requirements. If one requirement is to have a database that runs on a specific system, a hidden cost is now attached where if that vendor goes out of business, the code can be next to impossible to move to another platform. You satisfied the requirement but paid a huge price. "An ounce of prevention is worth a pound of cure."

However, the same managers who scream for efficiency are the ones who tear their hair when told the cost of modifications. Conversely, the managers who ask for generalized and easily modified programs are wont to complain when they find out how slow and spacious there programs turn out to be. We must be adult about such matters: neither psychology nor magic is going to help us to achieve contradictory goals simultaneously. Asking for efficiency and adaptability in the same program is like asking for a beautiful and modest wife. Although beauty and modesty have been known to occur in the same woman, we'll probably have to settle for one or the other. At least that's better than neither.

Be carefule what you wish for, you just might get it...

Efficiency

Moreover, with the cost per unit of computation decreasing every year and cost per unit of programming increasing, we have long since passed the point where the typical installation spends more money on programming than it does on production work. [like the cost of machine time, ed.] This imbalance is even more striking when all the work improperly classified as "production" is put under the proper heading of "debugging."

Even 30 years ago, the cost of development was the largest cost in the field of programming. If you don't have a manager who understands the challenges and complexities of software development, you are facing a two-front war, 1) the challenge of writing software that works in the time required, using the resources allotted and, 2) a struggle to keep a clueless, however well-meaning, boss from making things worse through ignorance much less incompetence.

Summary

Questions to ponder:
1. Does the program meet specifications? Or, rather, how well does it met specifications?
2. Is it produced on schedule, and what is the variability in the schedule that we can expect from particular approaches?
3. Will it be possible to change the program when conditions change? How much will it cost to make the change?
4. How efficient is the program, and what do we mean by efficiency? Are we trading efficiency in one area for inefficiency in another?


I can here the pointy-haired-boss now, "Why are you asking so many (implied -stupid-) questions? I don't understand why you're making it so hard, just go write it already!"

This is a good place to point out that repeating the same behavior and expecting a different outcome is one of the definitions of insanity.

Next is Chapter 3 - How Can We Study Programming

Where do you go from here, and where is here exactly?

Developers have a love / hate relationship with version control systems. A related post is this one on 'Scar Tissue'. On one hand they either don't do something you desperately need regardless or any pleading or begging. On the other hand they do something that 'conveniently' adds to your workload and refuses to have it any other way. However maddening a VCS can be, software development would be a nightmare without one. Most shops have a preferred tool which is a polite way of saying "Use it or else!" That being said, developers typically learn to use the preferred tool and leave the administration to someone else.

Installing a VCS for my personal work has been on my todo list for a while now but always got pushed onto the back-burner. In the past I've actually worked in a serious development shop that didn't have any versioning system for the first few years I was there. They didn't collaborate much or have the need for branching and since it was my first gig out of college it didn't have that "working without a net" feeling that it would these days. That shop did finally move to Sorcerer's Apprentice before it was bought by black-hole of software known as Borland. After that was PVCS which was expertly configured and taught me a great deal about the benefits of a rationally conceived and executed version control system. Using promotion groups in an orderly way to move code changes along a path from 'only-seen-by-me' to 'ready-for-final-testing' was an arrangement that I wouldn't see again. From there was Visual Source Safe which was serviceable, which means I wasn't the admin, but wasn't very robust. Then there was RCS/CVS which work better than nothing but still leave something to be desired. At this point I realized that the thing I was searching for was not to be found in a tool but in the common understanding of a group of developers who share a development environment and work together to get the most out of the VCS and make each others jobs easier. There is a lot to be said for sitting down as a development team and hashing out a commonly understood development model.


Where am I going with all this? The company for which I work has a policy of buy over build which was served them well for the last 10 years. They've kept their IT costs down while still getting applications deployed for their internal customers. The vendor was trying to write applications it could turn around and sell to other companies in the same market and my employers got a discount on the work; everyone was happy. This arrangement went south when the development partner decided to get out of the business of writing and selling software. They parted ways without rancor but now my company is left with application code for which they have no experience in managing. That's where I come in, to help them with managing software, to advise them when vendors try to close a sale with marketing smoke and mirrors, and to detect when those vendors could be hiding important details about their product or service.

So, I ask for the code and am given a large ZIP file, I kid you not. Evidently the off-shore group of the development shop used by the vendor uses Visual Source Safe locally but it isn't used by anyone here in my office. That led to another nightmare when I researched how code was deployed to production but I'll save that story for another time. I've used VSS before but since most of the code is Java and we're not using Visual Studio, it didn't make sense to use VSS. After poking around on the Internet, I found some really good reasons not to use VSS. Evidence about corrupted databases, well documented issues, and stories from the trenches made it easy to decide to use some other tool.

My first thought was to install Subversion so I downloaded it at home and started learning how to manage a repository. Someone at work mentions a new VCS called 'Git' which is fairly new on the scene, so when I checked it out (there I go again) and at first figured we didn't need the multiple author patch model, where a change has an author and a submitter which is useful for accepting patches through email. After more thought the realization came that the strength of the tools like PVCS or IBM's Rational Clearcase were that they understood branches even if you didn't use them all that much. Subversion doesn't have a great track record with branching. Since the budget was tight that means sticking to free products which came down to Subversion or Git. I'm leaning towards Git.

Off into the wild blue yonder I go. If anyone has stories and/or advice on setting up a VCS for the first time, I'm all ears.