XSLT - The Modern Day Snake Oil

Technology seems to attract all kinds, no surprise these days when everybody and their dog are high-tech consumers. But those of us who earn their living in that fast-paced world every day can't help but get a bit jaded seeing the same mistakes repeated time and time again.  From the desperate or possibly Machiavellian managers who want to ignore common sense and decades of history and continually chase the proverbial silver bullet, to the greedy vendors selling the latest 'enterprisey' tool with promises which sound more like a late-nite comedy skit, "Not only is it a floor wax, it's a tasty dessert topping!".  In software it is, "Without writing a single line of code". Why do people fall for these pitches? You can blame ignorance, malice, or greed; in any event, it is up to each of us to keep our wits about us when evaluting claims for the latest-and-greatest doohickey the next salesdroid offers.

There is one phrase that I've seen many times over the last twenty years and whenever I hear it, I immediately think of snake oil.  Whenever you hear a consultant say anything to the effect of, "easy to use by non-programmers!", smile and quickly look for the exit.  How trite is it? Well, early in my college days I remember hearing stories from the early mainframe days when assemblers were first coming out, "It works so well you won't need programmers anymore!"  At my first job at a large company, the 4GL vendors would use the same line on the managers, distracting their mark from the large price tag with promises of savings from lower staffing.  The reality was nobody but programmers could figure out how to actually use the immense packages, assuming they could get them installed correctly in time to use on a project. Even modern tools like Requisite Pro fall, which I really liked, prey to this. They are supposed to improve productivity but the learning curve is so large that the only way to properly use it on a large project is to have someone who already knows it configure it for you.

Recently I ran accross the claim in an old book, "Professional Java XML Programming with Servlets and JSP" by Alexander Nakhimovsky and Tom Myers published in 1999 (page 526 to be exact) which attempts to summarize the ream of paper constituting the previous chapters.  In a section appropriately but belatedly (considering where it appears) titled "What's XSLT for?" the authors explain,

We are rapidly approaching the summary and conclusions to this final chapter of the book In the meantime, the main conclusion about XSLT that we hope you will draw from this chapter is this: XSLT is best described as a general-purpose mini-language, intended to make many common operations on XML documents easy to express and accessible to non-programmers.

Is the book old and out-dated? Of course. The age of the book only matters though if the authors would express a different attitude given a chance to update their book. Granted, right after the offending paragraph they attach some conditions,

Allows non-programmers to do some transformations:
- without tools, only simple things
- with tools, more complex things

When I read the above, I couldn't help thinking, "Have you guys _looked_ at what you've written lately? If it takes a book 500 pages long to explain it, you're never going to get non-programmers to use it"  Why is programming hard, I don't know and do not want to venture a guess.  What I do know is when competent programmers who are at least trained - if not adept - at imperative object-oriented languages face a significant learning curve when working with a declarative language like SQL much less a functional one like Erlang.  XSLT is like a combination of web templates with an embeded declarative language which also include functional aspects, although the authers probably disagree saying, "As of today [1999], JSP is a tool for a prgrammer while XSLT is a tool for a Competent User who is NOT a programmer."

Am I trying to call out XSLT or XML? No, they are useful tools (like automatically formatting your log4j config file) and at this point we couldn't get rid of them if we tried. The issue is quite old and better men than myself have already weighed in.  What I do want is for people who should know better, like said authors, to refrain from propagating the myth that if only we had better tools or languages, non-programmers would rush in to join the computerized fun.
...

SharePoint 2010 Custom IFilter development update

The previous post on IFilter development for Windows and SharePoint, I included several links to resources we used to code a custom IFilter; the hope being they would be useful to anyone looking to write their own, seeing as how difficult it was to find accurate information on the subject.

Alex Culp did an outstanding job with the IFilter, so much so that Microsoft asked him to write a MSDN article on the subject of writing a custom IFilter.  The custom file type wasn't the issue, it was all the complexities of C++ and IFilter development combined with registry entries, properties, 32bit/64bit details and so forth, like how the 64-bit version of Visual Studio creates a 32-bit installer by default which then didn't install the correct version of the .Net dlls.  Another bit-ness nugget is to pay attention to which Regedit you run when setting registry entries; 64-bit entries are not visible to the 32-bit executable.

Computer Science is an Art

Of the many eternal debates that spiral around software development blogs the one that seems the most intransigent is whether the practice of programming is an art or a science.

Chefs have a similar discussion over cooking and baking.  Cooking is an art while baking is a science.  Why?  Because when you put something in the oven, you had better get it right the first time.  Mistakes mean starting over.  Cooking, on the other hand, allows the chef the leeway to make changes as the dish progresses. The chef can take action to keep the dish from heading off course. For example, when making gravy you keep adding flour until it is the right consistency and if you add too much flour you can add some milk to thin it down.  It is incremental by nature.

Martin Fowler, Alistair Cockburn, and Optimism

The effort to make Software Engineering actually live up to the 'Engineering' part has been greatly helped by Martin Fowler's book on refactoring. Martin recently posted about his decision to not participate in an effort by others in the industry, not lacking in optimism, to define a common process for software development.

Martin Fowler declined to get involved in a similar effort and for the same reason.  The key quote is,

“Why this is so was primarily crystallized for me by Alistair Cockburn who explained that since people are the central element in software development, and people are inherently non-linear and unpredictable - such an effort is fundamentally doomed.”

While my previous post about Alistair's paper covers much of the same ground I felt it was worthy of an update.  There will always be souls who crave consistency just as there will always be salesmen willing to offer the illusion of it.  Facing our own nature is one of life's hardest lessons. Some of those who ignore the human factor are optimists seeking to improve their chances of consistency, which has driven many towards the latest and greatest geegaw, a new language, a new GUI, a new design tool. Others are simply deluding themselves in the hopes that if only people would sit up and fly right (read this to mean 'do it their way') all our problems would be solved. 

Studying the process of computer programming has been around for decades.   Even so, we still have trouble internalizing human foibles - people aren't predictable and follow "standards" about as well as a random-walk.  So the most successful way to get people to be consistent is to lead by example, blaze a trail ahead and show them the way.  In this manner Extreme Programming was its own worst enemy and its own best cheerleader. By showing how a set of practices work in situ, it created a much-needed focus. The "Waterfall" model was the favorite whipping-boy, essentially dead, and the iterative approach was in its infancy.  So XP served as a lightning-rod for discussions on the craft of software development.  It provided an example of practices that could be emulated and provided an easy target for the doubters, disbelievers, and denialists; no doubt the negativity coming from the critics was channeling  their innate understanding of the darkside of human behavior.

While Extreme Programming hasn't become the standard development model, that doesn't mean it failed.  When the history of Software Development is written, XP will be given credit for re-introducing the most important factor; not tools nor process, but people.

Programmings Best Kept Secret

Software development is often compared to engineering or construction which has a hidden and debatable assumption that the source code is the product.  One camp has declared that source code is the design .   Other camps declare that programming as a branch of applied mathematics; see Dijkstra's "uncomfortable truth" quotes. Extreme Programming as a movement may have failed, it at least served as the flashpoint for a new examination of the nature of programming. Instead of arguing about engineering we can brag about how Agile we are or that we are Software Craftsmen.

Programmers are not engineers, nor are they artists.  Even when they are compared to craftsmen,  the comparison is always with woodworkers or blacksmiths. Programmers are nothing of the sort, programmers are writers.  Danny Thorpe discusses how you might tell what sort of literature your app may be.

Danny is on the right track but when it comes to software, there is an elephant in the room that needs to be discussed in the open.  Developers do not produce software, they write code.  This means that sometimes they need the logic skills of an engineer but just as often they need the communication skills of a poet.  Reading and writing, thinking about how to achieve a desired affect through the written word, those are the qualities of a writer.  It just so happens that the one of the chosen audiences is a single-minded overly-literal pendantic polyglot idiot-savant, otherwise known as a computer.  The other audience is other humans that may or may not have to actually follow the instructions, they do have to understand them well enough to tell if the instructions are correct.


It would be useful to compare programming to other fields and see how the comparison holds up.  It is not a coincidence that the games industry has adopted the filmmaking model for the production of games. Good choices would be publishing a book, producing a movie, or creating a recipe.


Programming as publishing


Programmers are the authors.  The compiler is the printing press.  Publishers are the companies that agree to fund the process in hopes of turning a profit.  Self publishing would be like creating training manuals for internal use; no profit is involved but the medium is appropriate for other reasons.  The book still needs to be edited, proofread, typeset, marketed, etc. I've covered this ground before but there is still much to contemplate.


Programming as filmmaking

Filmmaking is in interesting comparison because it deals with translating the written word into another medium. Programmers are the writers of the scripts and screenplays.  The producer is the stakeholder who desires the software. The director is the project lead.  Instead of actors and crew  we have the compiler which takes our instructions and executes whatever script of commands we provide; call it a screenplay or call it an application, the actions and actors my be different but it's still a set of commands given to actors to be performed.



Programming as cooking

Programmers create a recipe for a given dish.  The recipe is not the final product, the result of following the recipe is.  You don't buy a cookbook for the fabulous chocolate cake recipe, you buy it because you want to eat the cake.  In this case the compiler would be the tools available to the chef; pots and pans, utensils, and the stove/oven.  Having the right tools really helps but even the best oven won't correct a recipe that calls for 2 cups of vinegar for the sugar cookies.  Eventually software may survive the patent wars and finally be considered on par with a recipe; instructions to a cook in order to produce the desired product.

Summary

Software is  actually a mix of disciplines, all trying to contribute towards a common goal. The game industry may have it right, evidenced by them being to busy producing games and not endlessing debating whether their process is broken or seeking the latest silver bullet.  They use directors, producers, art and sound specialists, etc.  The compiler is their camera, used to capture the intent of the contributors; the CPU the silver screen for which to project the final product for consumption by the audience.

Of Rockstars and Bricklayers

Today we're going to start with a thought experiment.  You are one of the better coders in your company.  All the proof I need is that you're reading this.  Ok, that may be a little self-serving but it is true that the better coders I know are always reading something about coding.  It might be a new language, or a new technique, a programming question and answer site, or even a site that likes to rag on bad code. 

Based on how many Code Monkeys you know, it's easy to figure yourself in the top half of programmers, a 'good' coder if you will. An assumption that is easy to make is that the higher your productivity the higher your pay.  Is this true?  If you base it on simple experience, the answer is no.  It's too easy to get a job right out of college and get mediocre raises and then find yourself interveriewing candidates for openings in your own group where you have to offer 'market rates' to attract anybody.  If you've been there for 5+ years, given a company desperate to hire warm bodies, it doesn't take much for the new guy to be making 15-25% more than you even if he is a two-finger typist and can't spell SQL if you spotted him the 'S' and the 'Q'. 


Now, I'm happy where I am but it didn't take much to trigger my bad habit of thinking up 'what-ifs'.  The Stackoverflow podcast has been a good one for this sort of pondering.  Jeff and Joel have enough real-world experience to bring a ton of wisdom to the discussion on programming as it's own topic and in episode #42 discuss underperforming programmers.  I'd been reading about how star actors can command huge salaries based on their past work and the expected effect on the box-office if their name appeared on the marquee.  That major-league sports can have individuals who make 10-times what the lowest paid athlete makes on a single team, it doesn't take much to ask why programming doesn't see the same effect. 

I submitted this as a question to the Stackoverflow podcast which they spent a good 10 minutes on in episode 77. It didn't fit so well as a question on Stackoverflow itself, I suspect the culprit is the lack of code snippets. Interestingly enough I discovered the podcast when they were at episode 65 and decided to start listening back at #1 and was up to around #42 when I decided to contribute my question.  So anyway, when I saw the summary for #77, I didn't have the audio handy so jumped over to the transcript wiki for #77 and read the transcript.  It was a pleasant surprise at how engaged they were and that the question wasn't dismissed out-of-hand.  When I downloaded the audio, to hear their discussion it only reinforced my reaction.

The range of talent can be mighty wide even if there is some disagreement on the empirical numbers.  Joel Spolsky talks about the difficulty in finding great programmers, and uses similiar metaphores in a comparison with opera singers. Jeff has a similar take on Coding Horror.  Finding great talent is hard because there are only so many people who can do what you need done.

There have been some good discussions that has surfaced.  John Cook has a good argument based on lack of meaningful metrics.  It's a question that has been around for awhile.  Frank Wiles makes the claim that hiring average (or below) programmers actually costs your company money (followup post). Obviously when a company says they only hire the to 10%, they are deluding themselves or at least paying lip-service to wanting good programmers.  Joel does a great job of shattering that particular illusion. 

What about sales? In some fields top salesmen can bring home seven-figure incomes so we know that the wide range of incomes aren't limited to Hollywood and the big leagues.  James Kwak picks up on John's post and supports Joel and Jeff contention that paying the top performers much more than the average, outside of easily measured areas like sales and movies, has a corrosive effect on the group and the best coders end up leaving for greener pastures while the dregs hang on for dear life or are eventually pushed out.  I can't disagree.  It seems that if you are underpaid you have to ask yourself Ann Lander's question, "Am I better off leaving or staying put?"  Just realize that it's no use complaining to the company, the status quo works for them.

Programmers - Poets or Mathematicians?

Programming software is much like writing a story. Any given story can be told in a near-infinite number of ways and those who practice the craft will spend just as much time, if not more, arguing over the style of a work, as if their subjective opinion can suddenly become objective fact. Usually this leads to a contest of "If you don't agree with me, I'll just keep talking until you see reason" or otherwise known as talking the subject to death.

I thought I'd shine some light on an example of this issue in programming. There is a never-ending debate in programming circles about the 'proper' way to write code that creates some true/false decision. Stackoverflow has a good example of this.

My personal pet peeve (petty but my teeth grind everytime I see it) is verbosely setting booleans, e.g.

bool isValid;
if (percentage >= 0 && percentage <= 100)
  isValid = true;
else
  isValid = false;

whats wrong with

bool isValid = percentage >= 0 && percentage <= 100;

It's soooooo much more succinct and easier on the eye

Even though the single line version isn't the most complex, different people read through code in different ways. A single expression can be easily parsed by people who have a math background or are experienced coders. That's not the point. The expanded version reads more like a sentence than a formula. It's easy to forget that there are plenty of coders without a background in math. It really comes back to the preferred style of the reader. Do they prefer reading expressions and formulae or something more like human speech.

Homer's The Illiad and The Oddessy are ancient works dedicated to expressions of speech. They were originally poems that were passed down in an oral tradition and meant to be sung instead of read.

Euclid's Elements is a work dedicated to expressions of logic. It covers objective proofs and unambiguous facts and is the beginning of the mathematical tradition, if you will, of rigorous scientific thinking.

The problem we face in software is code is a form of human expression, with all the imprecision that entails, attempting to communicate with the utterly logical CPU; programmers will be forever trying to translate "An Ode to the Machine" into the Pythagorean Theorem.