Here I explain what we did at Zynga with anonymous surveys, to objectively assess team health and make improvements.

This topic is especially on my mind as friends and colleagues discuss recent suggestions that working in the gaming industry is just supposed to be insane.

Café World

When I started at Zynga in 2010, I joined the team running the game Café World.

Café World had an incredibly bright and hard working set of folks who were struggling to get releases out on time and hit business targets.  Things got much better over time, though it took some focused effort.

My General Manager (GM) wrote a report each week, summarizing the state of the game from a business perspective.  Part of that report featured a green / yellow / red team health assessment, with a short explanation of the reason for the status.

So once a week the GM would ask me how the team was doing, and I would respond with something like

I think green – bunch of folks took last week off for Christmas and are feeling pretty happy right now

or on another week

Flaming red, we had outages twice in the middle of the night last week, half the team was pulled into it and a couple of guys are ready to quit because they’ve haven’t slept since last Tuesday

Yellow, and more Yellow

In spite of my feedback, the team health in the weekly reports almost always trended to yellow.  This happened on other teams, too. When I inquired about this, I heard something like this

If I mark the team health RED, I get feedback that I’m pushing the team too hard and burning them out.  If I mark the team  GREEN, I get criticized for not pushing them hard enough.  So I usually just mark it YELLOW.

I called BS on this and suggested that we start an anonymous team health survey, and mark the team health objectively based on the survey data.

Thus was born the weekly health survey.

One Minute Weekly Health Survey

The weekly health survey was created with the following goals

• Lightweight (to encourage participation – less than 1 minute to complete)
• Objective measure (anonymous collection)
• Regular cadence for prompt follow-up (weekly collection)
• Responses support grouping into green / yellow / red categories
• Only measure vital signs – are we burning people out? (Not a comprehensive satisfaction survey, keep it simple)

To this end, the survey featured 4 simple questions focused on

• Hours Worked (objective measure of time in office)
• Tiredness (self-reported state of mind, independent of hours worked)
• Efficiency (perceived effectiveness, is process working or causing frustration?)
• Comments (optional section for free-format feedback)

The survey also requested that the individual identify their current team and (optionally) their current role (engineer, product manager, artist, …).

Objective Results

This image shows how the survey questions might be partitioned into green / yellow / red categories.

This part is, of course, subjective, but once decided upon leads to consistent evaluations and apples-to-apples comparisons between teams.

Running The Surveys

Although the surveys originated on the Café World team, they rolled out over time to include many other teams, eventually covering the majority of teams at Zynga.

We collected data for each team, as well as the company as a whole (an aggregate average), and sent summary results back to business leaders for their team, as well as the company averages.

Result Of Surveys

What results did we see from running the surveys?

The immediate result of the surveys was that we could spot obvious hot spots (directly from the multiple choice questions, or indirectly from the free-format comments). This created an opportunity for prompt follow up and corrective action with the affected teams.

However in a broader sense two trends were observed over time

• Reduction in variation between teams
• Improvement in average scores overall

In rolling the survey out to multiple teams, it became easier to spot obvious outliers.  No business leader wanted to be the one killing their team with overwork.  On the other hand, no one wanted to be the one whose team was working less than everyone else, either.

As a result, the first trend was a reduction in variation between teams.

Once the results became more uniformly consistent, a second trend emerged – the average results improved, in general.

The second trend it is less obviously a result of the surveys, but it was pleasing to see nonetheless.

Without additional context, the slide read something like this:

If we can get all our penguins, birds and chicks in alignment, we’ll be able to move forward smoothly.

It was a classic WTF moment; confused glances were exchanged around the room.

Welcome Back, Kotter

Turns out the senior leaders had recently held an offsite where they became acquainted with the work of John Kotter, including the book Our Iceberg Is Melting.  Kotter is regarded as an authority on leadership and change.

I no longer recall what exactly they were hoping to change, but do recall being intrigued – what about this book was (apparently) so inspiring?

I vowed to give it a thorough read, and did.  I also took time to acquire and read the (somewhat related) book Who Moved My Cheese?

Neither impressed me very much.

Capsule Summaries

In the bird book, an observant and forward-thinking penguin concludes that his community’s home iceberg is melting.  Initially ridiculed, avian opinion eventually sways in favor of relocating the colony, as the penguin overcomes skeptics and enlists the support of key allies.

The rodent book deals with residents living in a maze, reliant on a steady supply of cheese.  One day the cheese disappears, and they deal with the consequences, with varying degrees of denial and proactive behavior.

While the penguin book does offer a few strategies for leverage (recruit respected experts, build support from key leaders), I didn’t find either book particularly compelling. 

Nonetheless Inspiring

I shouldn’t downplay the overall usefulness of the literature, however. 

These books were uplifting in an indirect way, as they provided nearly endless material for satirical Photoshop modifications and whiteboard drawings (see book cover above right, and the picture to the right). 

So it would be a lie to imply that I did not find these books inspiring, though not for the reasons intended.

What’s the bottom line?  I found these books unhelpful or objectionable primarily for these reasons

• The fable format is patronizing
• A predetermined outcome is featured
• The struggle for change is not nuanced; it is for life or death

Patronizing

Both books employ a fable format, which I found patronizing.  You are not a human being pondering reasonable real life trade-offs; instead you are a bird or (worse) a rodent.

At least one review on Amazon compares the penguin book favorably to Animal Farm, which to me is puzzling.  In Animal Farm the animals are delightfully symbolic and ironic.  I’m unaware of penguin characteristics that lend themselves well to climate analysis or adaptability (excepting swimming and extreme tolerance to cold).  I will grant that rodents are symbolically opportunistic, though not in a particularly positive way.

In the end, I don’t feel the fable format lends value.  Rather the animals provide an unnecessary layer of abstraction from what might otherwise be a compellingly pragmatic story.

Predetermined Outcome

To complete the fable, the stories need a moral lesson.

As such, the outcomes are predetermined: the animals successfully adapt to change.  There is no room for ambiguity; to succeed, you must change.  Those who change, win.

What if instead one of the rodents ran ragged through the maze, failing to find cheese, eventually collapsing in despair and starvation in a dark, forgotten corner?  What if the cheese supply returned in its original location, better than ever before (the supplier had been on a brief trip to Europe, collecting artisanal cheeses)?

Perhaps the penguins’ dilemma is rooted more deeply than a single iceberg.  Global warming may doom their new home to the same fate, slightly delayed.  Just maybe, their selected strategy was misguided and even worse than staying put.

These possibilities are not part of the story.  In these books, change must happen, somehow.

In reality, it’s the nuanced consequences, and their associated risk, which makes implementing change challenging. 

For example, employees who gave up on Apple in 2004 to join the promising start-up that went belly up may well have believed at the time they were successfully implementing change management in their financial opportunities.

Life or Death

The message of these books is clear: you must change, or you will die.

Not only are the outcomes in these stories predetermined, but the consequences are as large as they can get – life or death.  Move or freeze.  Search or starve.

This of course is a ploy to raise the stakes to the greatest possible levels, inhibiting the possibility for contrary considerations. 

You’d have to be crazy to avoid changing, if you risk death as a result.  Everyone should get on board with that, right?

In reality, the change you’re likely looking to implement involves something less exciting like convincing a team to convert from C++ to Java.  Or you’re trying figure out whether to join the hot startup or let your remaining 25% of stock continue to vest.  No easy life-or-death scenarios here.

One can even imagine the manager thinking …

“I want my team to change, but they’re pushing back.  I’ll have them read this book, then they’ll realize they must change.  Or else they’ll die.  Maybe I’ll even kill them myself … “

Go Change It!

If you need to effect change, don’t hand out books – go make it happen! 

Impress on your colleagues that change is needed, and why.  Propose an improvement and explain why it’s better.  Get folks to agree to it, and follow up to make sure it happens.

If you’re looking to change the semantics of an optional field in JIRA, then an 8-step change process is almost certainly overkill.  But those guidelines are fairly sound and intuitive overall.

But you don’t need to read a silly animal book to understand them.

I’ve previously compared software estimation to variations in the time required to drive to work.

Dispatching delivery trucks is my mental model for comparison to changes in project direction.  This can help others to understand the impact when changes happen too frequently.

A Random Walk

Using an example from the Bay Area, suppose I have a delivery truck stationed in San Francisco.

I call up the driver and ask him to make a delivery in Sacramento.  For reference, Google tells me it takes an hour and a half or so to drive from San Francisco to Sacramento.

Suppose that, 30 minutes into the trip, I call my driver and tell him to make the delivery to San Jose instead.  He’d likely be near San Pablo, only a third of the way to Sacramento. 

The driver dutifully turns around, and starts the one hour trip to San Jose.

After another half an hour, I call him up and instruct him to drive to Tracy.  From somewhere near San Leandro (and short of reaching San Jose) he would start the one hour drive to Tracy.

And so on.

What Did You Accomplish Today?

After a day of this, I recall my driver back to the home office and inquire

“How did it go?  How many deliveries did you make today?”

Exasperated, my driver informs me that, in spite of driving hundreds of miles, he failed to complete a single delivery.

(As a manager, I then express shock and outrage, and we proceed to have a conversation in which perhaps I learn the error of my ways.)

Thrashing

In computer science thrashing refers to situations in which the resources of a computer are largely (or wholly) consumed by forms of overhead, with the result of severely reduced productive output.

The concept can apply to other activities, including projects.  Changes in direction (whether literal or figurative), can happen too frequently with the result that productivity is impacted (and in the worst case, entirely eliminated).

Make sure the driver has a chance to drop off the package.

One of the challenges of running an online game (like Café World, on which I’ve spent most of my time) involves releasing new features on a regular, predictable cadence.

As a result, our team has devoted resources to collecting metrics on feature development and release timing.  We use that data as a feedback mechanism to adjust our scoping and scheduling guidelines.

The approach outlined below describes a process for constructing schedules with relatively high confidence.  It also gives a statistical foundation for justifying “padding” which might otherwise raise the eyebrows of skeptical managers. 

Driving to Work

To the left is a histogram of the time required to drive 18 miles to work, over many dozens of trips.  The data is not perfect, but roughly models a route that I drove frequently – a mix of highway and local roads.

Most days, the trip took about the same amount of time, around 30 minutes. 

An occasional traffic accident would hold things up, maybe requiring 45 minutes.  Sometimes bad weather would slow things down; an accident and bad weather together would be even worse – requiring perhaps an hour.

Rarely, with unusually clear traffic and cooperative traffic lights, I could race home in only 20 minutes.  But never faster.

Schedules Are Subjective

If you asked me “Steve, how long does it take you to drive to work?” I could sensibly answer

“About 25-35 minutes”

reflecting the average, and most common trip times.  And about half the time, I’d be right.

However if I had a 9am meeting with the CEO, you could be certain that I would leave at 8am or earlier, in order to have a nearly 100% chance of being there on time. 

If you wanted to pin me down to something certain, my answer would be something closer to

“An hour or less”

When the stakes are high, a schedule with 50% confidence just doesn’t cut it.

Variability In Driving

Relative to software development, driving to work is a technically non-complicated task (apologies to all you professional drivers out there).

Driving to work is easily understood, and easily repeatable.  In my example, the route is executed each time by the same person, in the same way.  Only the environmental conditions vary, yet they substantially impact the actual execution.

In spite of the many constants, we observe a remarkable 3:1 variation in performance (20 to 60 minutes). 

That is fairly surprising for this non-complicated task.

Tossing Coins

Imagine you are constructing a schedule comprised of 10 tasks which must be executed in sequence.  Each has an independent 50% chance of requiring 1 day to execute, and a 50% chance to require 2 days to execute.

Clearly the project will require 10 to 20 days to complete, but what would be a safe estimate?  Would the midpoint of 15 days be a reasonable schedule commitment?

This admittedly simplistic example can be modeled as a series of coin tosses, with a probability distribution as shown in the figures above and to the right.

Because the distribution is symmetric, there is only a 50-50 chance that the tasks will be completed in 15 or fewer days. 

Scheduling for 15 days is not a winning strategy for a high-confidence commitment.

In the figure to the above right, the same distribution is shown with the rightmost bins annotated with the likelihood that the completion time will be fewer than the indicated number of days.

We can use this information to construct a schedule with relatively high confidence.

The figure to the right shows a bounding box around 90% of the schedule possibilities. 

A schedule estimate between 17 and 18 days will be too short only 1 in 10 times.

Statistical Variation is the Norm

Even if our estimates are perfect (and they are not), environmental factors intrude.  The network or VPN can be down, source code control can be broken, or the build might not work.  A colleague may have gifted you with an unusually complicated branch merge.

These things happen all the time, yet we often pretend they do not.  Software estimation, even when done well, is a highly stochastic process.

Scoping for 90%

I encourage our teams to scope and schedule for a 90% on time rate, since the consequences of schedule misses are fairly severe.

When iterating over a large number of feature iterations of similar size and complexity, it is possible to accumulate statistical evidence to help drive this process in a compelling manner.

For tasks which are highly novel, unusually complicated, or having strong external dependencies, variances are naturally much higher and statistical categorization can remain elusive.

But it’s always worth tracking the original estimates, the actual execution, and their historical variations.

I was talking to a colleague recently about basic expectations for staff responses in adverse situations, based on their level of experience. 

The further up the ladder you are, the higher the expectations for proactive observation and corrective action.

Hiearchy of Train Wrecks

I have in mind something like this

• passive

That was nasty. What happened?

• observation

I was in a train wreck!

• communication

I inform others that a train wreck occurred, and describe the details.

• analysis

I can describe the root cause of the train wreck, and find ways to avoid the same in the future.  My analysis is objective and dispassionate, of course.

• proactivity

I can see conditions shaping up for a future train wreck.  I communicate that to others and actively work to avoid the accident.

• prevention

I accept that trains wrecks, while regrettable, are inevitable.  I actively work to reduce their frequency and impact.  This may involve a combination of training, process and technological improvements, political maneuvering, and so on.

• extinction

I implement a process or technology shift (riding bicycles for example) which eliminates train wrecks as a class of problems.

The goal is to spot organizational train wrecks early, communicate that they are forthcoming, mount resources to correct them and (ideally) factor out their possibility of occurrence.

It’s a great series full of wonderful moments – Shelby Foote stealing the show, the pan-and-scan effect, and many excellent narrative readings.

(Viewing the series this time around, I was struck by several interview scenes in which the subject lighting was much more harsh than I originally recalled.  Ken reports that, back in 1989, they were under a tight budget & schedule.  Some things never change.)

Leaders

In this viewing, I paid special attention to the details attributed to the generals.  Policies & politics aside, they were a set of fascinating characters.

• George McClellan organized and trained the Army of the Potomac, yet displayed indecisive leadership against a smaller more agile adversary.  Lincoln famously summarized “If he can’t fight himself, he excels in making others ready to fight.”
• Robert Lee led the Army of Northern Virginia, in spite of being personally opposed to secession.  In spite of some strategic blunders, he commanded brilliantly against larger forces.
• US Grant struggled at civilian endeavors but ultimately led the Union army to victory.

All great stuff.

On Being Dusty

One sequence from the series that stuck with me is this description of Grant from the Vicksburg Campaign

… the men knew they were cut loose from their base of supplies, but Grant himself gave them confidence.  They believed Grant knew what he was doing, and one great encouragement for their believing that was quite often on the march, whether at night or in the daytime, they’d be moving along a road or over a bridge and right beside the road would be Grant on his horse – a dust-covered man on a dust-covered horse – saying “Move on, close up.”  So they felt very much that he personally was in charge of their movement and it gave them an added confidence.

I like this because from a leadership perspective it reinforces the principles of being seen often, engaged in the activities of the team, and up to your elbows in the details (dust-covered).

A leader who stays engaged with day-to-day activities instills confidence and builds credibility.  If the folks on your team are working long hours and late nights, it’s important to be there with them.

If they are dust-covered, it’s hard to explain not being dust-covered yourself.

(Thanks for reading – Steve Klinkner)