What I Found in Panama

I feel a guilty need to read my share of books which are outside my area of expertise (I have friends who are substantially more prolific readers than I – they make me feel guilty).

Once in a while, I read a book with unexpected overlap with my professional experience (which, of course, is part of the point). 

I’d like to share one I read recently. 

Path Between The Seas

panama

The Path Between The Seas by David McCullough describes a good portion of the history of the construction of the Panama Canal.

It covers the early motivation to build the canal, efforts to build consensus, and stories of success and failure. 

There are additional fascinating details related to political, social and medical issues surrounding the endeavor.

When it comes to large engineering projects, there are few that top the Panama Canal.  But I’d like to distill a few details which seem to recur in projects regardless of scale.

Evangelizing

220px-Ferdinand_de_LessepsFerdinand de Lesseps, having developed the Suez Canal, was instrumental in evangelizing support for the Panama Canal.

However, there’s some evidence that Lesseps’ evangelism was less affected by actual project status than it could have been.

There are some stories of him reporting progress substantially more favorably than he had observed firsthand, and also collecting estimates from expert subordinates and “adjusting” the estimates downward.

Hmm.

Lesson: charisma is an amplifier; projects need evangelists, but to be effective they need to be pointed in the right direction

Getting It Going

150px-John_Frank_StevensJohn Stevens was an early chief engineer during the second major phase of canal construction.  He was very hands-on and beloved by his workers for that among other reasons. 

Stevens really seemed to Get The Thing Going, and had the foresight to improve the industrial and medical infrastructure to the point that canal construction was actually feasible.  He pragmatically stated

… the problem is one of magnitude and not miracles.

Lesson: every project needs someone to turn the dream into reality

Building It

220px-George_Washington_GoethalsGeorge Goethals followed John Stevens on the project.  He was not particularly well liked at first, but ultimately earned deep respect from his workers. 

Joseph Bishop, a New York reporter, recalled of Goethal’s relations with workers that

They were treated like human beings, not brutes, and they responded by giving the best service within their power

He brought organization, efficiency, and discipline to the project, and ultimately was the one who Built The Damn Thing.

Lesson: every project needs someone who can Build The Damn Thing

Simplicity

The construction of the Panama Canal featured advancements in engineering and medical science.  It was an early example of electrical control systems (and a major project for General Electric).

I’d like to highlight one feature that caught my attention from a work flow perspective.  It involves the central control panel for the canal.

From the book, pp 602-603

The genius of the system, however, was in the elaborate racks of interlocking bars concealed from view beneath the board.  For not only was the operator able to see the entire lockage process in miniature and in operation before him, but the switches were interlocking-mechanically.  Each had to be turned in proper sequence, otherwise it would not turn.

In brief, the control board implemented a mechanical state machine which rendered operator errors impossible. 

That is, designed to be inherently simple, bug-free and foolproof – I like it.

Lesson: it takes hard work to make things simple, yet is often worth the effort

 

This blog post is a reprise of an article originally written in September 2009.

Posted in Books, Leadership, Simplicity | Leave a comment

Scoping for 90%

90 percentI’ve been working at Zynga since early 2010. 

One of the challenges of running an online game (like Café World, on which I’ve spent most of my time) involves releasing new features on a regular, predictable cadence.

As a result, our team has devoted resources to collecting metrics on feature development and release timing.  We use that data as a feedback mechanism to adjust our scoping and scheduling guidelines.

The approach outlined below describes a process for constructing schedules with relatively high confidence.  It also gives a statistical foundation for justifying “padding” which might otherwise raise the eyebrows of skeptical managers. 

Driving to Work

driving-to-workTo the left is a histogram of the time required to drive 18 miles to work, over many dozens of trips.  The data is not perfect, but roughly models a route that I drove frequently – a mix of highway and local roads.

Most days, the trip took about the same amount of time, around 30 minutes. 

An occasional traffic accident would hold things up, maybe requiring 45 minutes.  Sometimes bad weather would slow things down; an accident and bad weather together would be even worse – requiring perhaps an hour.

Rarely, with unusually clear traffic and cooperative traffic lights, I could race home in only 20 minutes.  But never faster.

Schedules Are Subjective

If you asked me “Steve, how long does it take you to drive to work?” I could sensibly answer

“About 25-35 minutes”

reflecting the average, and most common trip times.  And about half the time, I’d be right.

TrafficHowever if I had a 9am meeting with the CEO, you could be certain that I would leave at 8am or earlier, in order to have a nearly 100% chance of being there on time. 

If you wanted to pin me down to something certain, my answer would be something closer to

“An hour or less”

When the stakes are high, a schedule with 50% confidence just doesn’t cut it.

Variability In Driving

Relative to software development, driving to work is a technically non-complicated task (apologies to all you professional drivers out there).

Driving to work is easily understood, and easily repeatable.  In my example, the route is executed each time by the same person, in the same way.  Only the environmental conditions vary, yet they substantially impact the actual execution.

In spite of the many constants, we observe a remarkable 3:1 variation in performance (20 to 60 minutes). 

That is fairly surprising for this non-complicated task.

Tossing Coins

coin-tossesImagine you are constructing a schedule comprised of 10 tasks which must be executed in sequence.  Each has an independent 50% chance of requiring 1 day to execute, and a 50% chance to require 2 days to execute.

Clearly the project will require 10 to 20 days to complete, but what would be a safe estimate?  Would the midpoint of 15 days be a reasonable schedule commitment?

coins-annotatedThis admittedly simplistic example can be modeled as a series of coin tosses, with a probability distribution as shown in the figures above and to the right.

Because the distribution is symmetric, there is only a 50-50 chance that the tasks will be completed in 15 or fewer days. 

Scheduling for 15 days is not a winning strategy for a high-confidence commitment.

In the figure to the above right, the same distribution is shown with the rightmost bins annotated with the likelihood that the completion time will be fewer than the indicated number of days.

coin-tosses-90We can use this information to construct a schedule with relatively high confidence.

The figure to the right shows a bounding box around 90% of the schedule possibilities. 

A schedule estimate between 17 and 18 days will be too short only 1 in 10 times.

Statistical Variation is the Norm

Even if our estimates are perfect (and they are not), environmental factors intrude.  The network or VPN can be down, source code control can be broken, or the build might not work.  A colleague may have gifted you with an unusually complicated branch merge.

These things happen all the time, yet we often pretend they do not.  Software estimation, even when done well, is a highly stochastic process.

Scoping for 90%

I encourage our teams to scope and schedule for a 90% on time rate, since the consequences of schedule misses are fairly severe.

When iterating over a large number of feature iterations of similar size and complexity, it is possible to accumulate statistical evidence to help drive this process in a compelling manner.

For tasks which are highly novel, unusually complicated, or having strong external dependencies, variances are naturally much higher and statistical categorization can remain elusive.

But it’s always worth tracking the original estimates, the actual execution, and their historical variations.

Posted in Management, Software, Zynga | 2 Comments

Quadratic Songs and Books

quad-ballsMy son is 2 1/2 years old, and loves stories.  His favorite book at the moment is A Fly Went By.

In the book, a boy sits lazily by the lake and observes a fly passing.  The fly in turn is pursued by a frog, who is in turn chased by a cat, and so on.

Each animal introduced is pursued by yet another, and a pattern emerges in the story.  For each new animal

  • The animal is introduced
  • All the prior animals are again enumerated

In this manner, following the introduction of the dog, the story goes:

The fly ran away in fear of the frog
who ran from the cat, who ran from the dog …

Cumulative Stories

This style in song form is called cumulative, and other examples include the familiar Twelve Days of Christmas, for which on day three we have

On the third day of Christmas, my true love gave to me…
3 French Hens
2 Turtle Doves
And a Partridge in a Pear Tree.

Yet another similar example is the song There Was And Old Lady Who Swallowed a Fly.  After swallowing a bird, spider and fly, the song goes

There was an old lady who swallowed a bird.
How absurd to swallow a bird.
She swallowed the bird to catch the spider,
that wiggled and wiggled and tickled inside her.
She swallowed the spider to catch the fly.
I don’t know why she swallowed the fly.
I guess she’ll die.

Each has a similar pattern.

Quadratic Behavior

With some amusement one evening while reading A Fly Went By, I realized that this style exhibits quadratic behavior.  That is, the length of the song/story increases with the square of the number of actors/stanzas. 

This is evident due to the number of actors/stanzas enumerated in each of the N segments in sequence:

1
2
3

N

Arranged with an equivalent number of objects (dots, for example) on each line, the sequence forms a half-triangle pattern, as pictured at the top of this article.

If you accumulate all the sizes of all the segments, we have the numerical series

1 + 2 + 3 + … + N  =  N (N+1) / 2

which, for large N, is dominated by the term N2.  In computer science, an accumulation like this is frequently an indication of non-ideal algorithm design.

What’s The Bottom Line?

In summary, if you double the number of actors / stanzas, the resulting book / song will be not twice, but four times as long.  “24 days of Christmas” would be an excruciating 4 times as tedious to listen to, relative to the (already tedious) 12 days.

It isn’t very surprising, then, that these songs and books generally tend to call it quits after 8-12 segments.

In the future, my interview questions may include children’s books.

Posted in Software | Leave a comment

Klink’s Hierarchy of Train Wrecks

train-wreckYou’ve heard of Maslow’s hierarchy of needs?  I’ve got Klink’s hierarchy of train wrecks.

I was talking to a colleague recently about basic expectations for staff responses in adverse situations, based on their level of experience. 

The further up the ladder you are, the higher the expectations for proactive observation and corrective action.

Hiearchy of Train Wrecks

I have in mind something like this

  • passive

That was nasty. What happened?

  • observation

I was in a train wreck!

  • communication

I inform others that a train wreck occurred, and describe the details.

  • analysis

I can describe the root cause of the train wreck, and find ways to avoid the same in the future.  My analysis is objective and dispassionate, of course.

  • proactivity

I can see conditions shaping up for a future train wreck.  I communicate that to others and actively work to avoid the accident.

  • prevention

I accept that trains wrecks, while regrettable, are inevitable.  I actively work to reduce their frequency and impact.  This may involve a combination of training, process and technological improvements, political maneuvering, and so on.

  • extinction

I implement a process or technology shift (riding bicycles for example) which eliminates train wrecks as a class of problems.

The goal is to spot organizational train wrecks early, communicate that they are forthcoming, mount resources to correct them and (ideally) factor out their possibility of occurrence.

Posted in Leadership, Management | Leave a comment

Dahon Number Three

001There should be a Dahon “Frequent Buyers’ Club.”  I’d be a member, somewhat reluctantly.

Due to some unwise choices parking my bicycles outside, I’ve had two stolen in the past 18 months.  As a result, I’m on my 3rd Dahon Speed D7 in as much time (so far I’ve owned one 2010 model and two from 2011).

I’ve been happy overall with the Speed D7 from a price / value point of view.  Each time I’ve looked over other models (and considered their likelihood of being stolen!).  Some of the lighter, more expensive models are intriguing, but I’ve settled back on the D7 again.  After all, I’m mostly using to commute short distances.

I mail ordered the 2010 model from SafetyCycle, which was an easy enough experience.  On arrival the bike required some minor tuning to the rear derailleur – nothing surprising.

I purchased the two 2011 models from REI, because they were in stock, and following the thefts I was pressed for time to obtain a replacement.  After the member refund, the price isn’t that much different from a mail order.

The 2nd model I purchased (now stolen) was the smoothest so far – no significant adjustments needed.  The most recent D7 had a problem with the cassette.  The rear derailleur was properly adjusted, but the 7th gear kept skipping.  REI replaced it the first week under warranty.

(And yes, I have since purchased a better lock)

Posted in Biking, Commuting | 3 Comments

Joel On Software

joel-on

After a bit of a delay, I’ve got another book to recommend.

Joel Spolsky has kept a popular blog in the past decade, archived on his web site and in several books including Joel On Software.

Joel borrows ideas from sources like Peopleware and Mythical Man Month, as well as his experiences developing software for several companies including his own.  He’s a thought-provoking and entertaining writer, making him an easy and worthwhile read.

A Few Sections

I have a few sections to recommend, if you’re short on time

  • Guerilla Interviewing
  • Painless Functional Specifications
  • Painless Software Schedules
  • Get Crash Reports From Users – Automatically!
  • Human Task Switches Considered Harmful
  • Getting Things Done When You’re Only A Grunt

But the book is sprinkled with gems.

Summary

A few caveats.  I don’t always agree with Joel, but generally find his writing worth reading.  As with the other books I’ve mentioned, there is some dated material (ten years later, dot com references are sounding somewhat quaint to me).  Last, there’s quite a bit of material focused on Microsoft technologies – a reflection of Joel’s background.

Highly recommended.

Posted in Software | Leave a comment

The Mythical Man Month

Mythical Man Month The Mythical Man Month is another of my favorite books on software and team productivity (this wikipedia page briefly covers the key ideas).

The Mythical Man Month is a classic book which has profoundly affected thinking about software development processes.  Written by Fred Brooks, it is based on his management of the IBM System 360 series of computers.  This was a large software project (hundreds of developers), and some of his observations are most applicable to projects which are large, or of long duration.

The Mythical Month

The mythical man month is one of the key concepts in the book: progress on a software project will not scale linearly with the staff count, and may in some cases may have an inverse relation (adding people to a late project can cause it to fall further behind).

Brooks doesn’t simply toss this out as a hypothesis; he backs it up with analysis and examples.

Task Partitioning

Some of the ideas he introduces are gems for their concise and memorable nature.  Regarding the impact of intercommunication on the mythical man month:

Men and months are only interchangeable commodities only when a task can be partitioned among many workers with no communication between them. This is true of reaping wheat or picking cotton; it is not even approximately true of systems programming.

and this classic observation on the sequential nature of certain tasks:

When a task cannot be partitioned because of sequential constraints, the application of more effort has no effect on the schedule.  The bearing of a child takes nine months, no matter how many women are assigned.

Runny Omelets

One of my favorites is this analogy on the sense of urgency being independent of the actual time required:

Observe that for the programmer, as for the chef, the urgency of the patron may govern the scheduled completion of the task, but it cannot govern the actual completion.  An omelet, promised in two minutes, may appear to be progressing nicely.  But when it has not set in two minutes, the customer has two choices – wait or eat it raw […]

The cook has another choice; he can turn up the heat.  The result is often an omelet that nothing can save – burned in one part, raw in another.

(That is, don’t be surprised by the result if you don’t know much about cooking and impose unreasonable demands on the chef.)

Summary

Some caveats.  The book has a relative focus on observations from large software projects.  Some of the core ideas may not apply directly to your Web-2.0-rapid-prototyping-team-of-3-people.  The technological references are, of course, often dated – the book was originally published in the in mid-1970s.  The book has an academic feel to it (heavier on deep analysis than lightweight advice).

However the essentials remain relevant three decades later, and I highly recommend the book.

Posted in Software | Leave a comment

Peopleware

Peopleware I recently wrote about several of my favorite books on software and team productivity.

Peopleware is a classic book on team building, productive and “jelled” teams, individual and team productivity, and work place environment. It is thought-provoking and should be must-read material for anyone who ends up working in software development.

A few topics/chapters are particularly memorable to me

  • Quality If Time Permits – knowledge workers want to produce high quality products; management and silly deadlines conspire to defeat this.
  • Hiring a Juggler – you wouldn’t hire a juggler without asking him to juggle; why would you hire a programmer before asking him to code?
  • Teamicide – behaviors to avoid if you want to avoid killing off your team (this chapter also introduces one of my favorites – the concept of the “quality reduced product”)

There’s a relatively strong concentration on workplace environment (offices, cubes, windows, interruptions, …), with an emphasis and data to suggest that inefficient workplaces can destroy productivity.  Workers and managers don’t always have easy ways to influence these factors, but they’re very much worth pondering.

The book gets some criticism for its focus on negative behaviors to be avoided, rather than establishing positive behaviors to emulate.  Portions of  the material are also now somewhat dated (references to intercoms and 1970s pop music come to mind). 

However criticizing this book for being dated misses the point; the fundamentals remain valid years later and this remains a classic which has influenced thinking for several decades.

Overall a great read.

Posted in Software, Zynga | Leave a comment

Three Good Books

books I’ve been working on the Cafe World team at Zynga for just over a year now.  It’s a great ride – we have an energetic and talented team who have built a wonderful game.

Zynga executes software projects in highly compressed time frames (features in days or weeks rather than weeks or months), which presents its own unique scheduling challenges.  This has forced me to think hard about how to optimize software development while moving at what we affectionately call “Zynga Speed”.

As a result, I’ve been spending a lot of time recently pondering software schedules and project management issues.  This has brought me back to some of the books and ideas that influenced me over the years.

I’ll cover these in a little more detail in subsequent posts, but the following are three good sources that stuck with me long after I read them

While some or all of the material in each of these books may not apply to the situation you’re in, each is a thought-provoking and worthwhile read.

 

(Thanks for reading – Steve Klinkner)

 

PS To be fair, the full name of Joel’s book is

Joel on Software: And on Diverse and Occasionally Related Matters That Will Prove of Interest to Software Developers, Designers, and Managers, and to Those Who, Whether by Good Fortune or Ill Luck, Work with Them in Some Capacity

Posted in Software, Zynga | 3 Comments

Cool Geek Toy: AR Drone

AR Drone A colleague at Zynga recently brought in his new AR Drone – I now have Christmas Present Envy.  AR Done is a product of Parrot.

Videos

Although I don’t (yet) own one, I’ve had my eye on these for a while.  The Drone seems like a candidate for the top geek toy of 2010.  You can view some videos on their web site, or YouTube.

What’s Cool

I think what’s cool about the the AR Drone is its integration of several innovations

  • self-stabilizing

It has feedback systems (accelerometer, gyroscope, ultrasound, video) allowing it to be self-stabilizing.  Turn it on, and it rises up several feet, and (eerily) just sits there, like a bumblebee.

  • iPhone controller

It’s WIFI-enabled, enabling it to be controlled from an iPhone app.

  • web cam

WIFI capability allows it to stream video back to the phone.

The combination works together nicely. 

Good fun.

 

(Thanks for reading – Steve Klinkner)

Posted in gadgets | 2 Comments