The Art and War of Software Releases
Releasing software is a black art. It takes a little luck and a little magic. Doing it on time is a major headache.
“Any day now, any day now,
I shall be released.” - Bob Dylan, Before the Flood
To quote Orson Wells in a famous advertising campaign for Paul Masson, “We will sell no wine before it’s time.”
Software doesn’t work that way. While there are perfectionists in every walk of life, wrinkling out the very last .01% of bugs in any software will take 99.99% of its total development time. It’s the modern software industry’s dirty little secret: all software has bugs. It’s a constant war between product and time. You either have 1) a fixed deadline with a not ready for prime-time software product or 2) a slipped deadline with a slightly less than not-ready-for-primetime product. The battle’s always the same, but a good software operations manager knows when to throw in the towel and wait for the next round. As Arthur Dent says in Hitchhiker’s Guide to the Galaxy, “Always know where your towel is.”
In fact everything I learned about production software releases can probably be traced back to the Hitchhiker’s Guide. More specifically, the 3rd book in the series, “Life, the Universe, and Everything” (New York: Harmony Books, 1982, Douglas Adams). The most relevant lesson is Bistromathematics. It can be summed up this way:
- Numbers written on restaurant bills within the confines of restaurants do not follow the same mathematical laws as numbers written on any other pieces of paper in any other parts of the Universe.
- The amount owed on the check is never the same amount as the sum of the amounts owed as calculated by the individuals.
At the end of the meal, there’s bargainings, discussions, negotiations, written calculations, rationalizations, justifications, and prognostications. In the micro-seconds before the waiter approaches the table to collect the bill, all of the numbers and participants magically reconcile themselves without anyone having to run to the ATM, run to the cash register to break a $20, or feel unfairly imposed upon by having to contribute an extra buck or two.
Our last software release was like that. We had a fixed deadline, multiple streams of development going on, multiple sub-projects, bugs, fixes, personnel changes, late checkins, and no clear view of how we were ever going to pull it all together in time. Just when we thought it was going to be infinitely improbable we were going to finish on time–poof–something happened. We ran out of bugs to fix and everything magically fell into place.
“It’s done.”
“It’s done?”
“It’s done.”
“Wow. It’s done.”
In the last release of our software, we had 42 release notes. Coincidentally, “42″ is also the answer in the Hitchhiker’s Guide about Life, the Universe, and Everything. Somehow when we we weren’t paying attention, ripping out the guts of the old platform and replacing it, refactoring the code, optimizing the performance and adding new features they all just disappeared.
We all stood back, sighed a breath of relief and marvelled at the magic.
Everyone else on the other hand said, “Oh, you just got lucky on that release. Let’s see you do that again with the next one!”
Little luck. Little magic.
Greg
