Here I explain what we did at Zynga with anonymous surveys, to objectively assess team health and make improvements.

This topic is especially on my mind as friends and colleagues discuss recent suggestions that working in the gaming industry is just supposed to be insane.

Café World

When I started at Zynga in 2010, I joined the team running the game Café World.

Café World had an incredibly bright and hard working set of folks who were struggling to get releases out on time and hit business targets.  Things got much better over time, though it took some focused effort.

My General Manager (GM) wrote a report each week, summarizing the state of the game from a business perspective.  Part of that report featured a green / yellow / red team health assessment, with a short explanation of the reason for the status.

So once a week the GM would ask me how the team was doing, and I would respond with something like

I think green – bunch of folks took last week off for Christmas and are feeling pretty happy right now

or on another week

Flaming red, we had outages twice in the middle of the night last week, half the team was pulled into it and a couple of guys are ready to quit because they’ve haven’t slept since last Tuesday

Yellow, and more Yellow

In spite of my feedback, the team health in the weekly reports almost always trended to yellow.  This happened on other teams, too. When I inquired about this, I heard something like this

If I mark the team health RED, I get feedback that I’m pushing the team too hard and burning them out.  If I mark the team  GREEN, I get criticized for not pushing them hard enough.  So I usually just mark it YELLOW.

I called BS on this and suggested that we start an anonymous team health survey, and mark the team health objectively based on the survey data.

Thus was born the weekly health survey.

One Minute Weekly Health Survey

The weekly health survey was created with the following goals

• Lightweight (to encourage participation – less than 1 minute to complete)
• Objective measure (anonymous collection)
• Regular cadence for prompt follow-up (weekly collection)
• Responses support grouping into green / yellow / red categories
• Only measure vital signs – are we burning people out? (Not a comprehensive satisfaction survey, keep it simple)

To this end, the survey featured 4 simple questions focused on

• Hours Worked (objective measure of time in office)
• Tiredness (self-reported state of mind, independent of hours worked)
• Efficiency (perceived effectiveness, is process working or causing frustration?)
• Comments (optional section for free-format feedback)

The survey also requested that the individual identify their current team and (optionally) their current role (engineer, product manager, artist, …).

Objective Results

This image shows how the survey questions might be partitioned into green / yellow / red categories.

This part is, of course, subjective, but once decided upon leads to consistent evaluations and apples-to-apples comparisons between teams.

Running The Surveys

Although the surveys originated on the Café World team, they rolled out over time to include many other teams, eventually covering the majority of teams at Zynga.

We collected data for each team, as well as the company as a whole (an aggregate average), and sent summary results back to business leaders for their team, as well as the company averages.

Result Of Surveys

What results did we see from running the surveys?

The immediate result of the surveys was that we could spot obvious hot spots (directly from the multiple choice questions, or indirectly from the free-format comments). This created an opportunity for prompt follow up and corrective action with the affected teams.

However in a broader sense two trends were observed over time

• Reduction in variation between teams
• Improvement in average scores overall

In rolling the survey out to multiple teams, it became easier to spot obvious outliers.  No business leader wanted to be the one killing their team with overwork.  On the other hand, no one wanted to be the one whose team was working less than everyone else, either.

As a result, the first trend was a reduction in variation between teams.

Once the results became more uniformly consistent, a second trend emerged – the average results improved, in general.

The second trend it is less obviously a result of the surveys, but it was pleasing to see nonetheless.

Our game is now live, you can find it at http://apps.facebook.com/chefville/

Here are a few articles about the game.

• Games.com – ChefVille: All about the food, its ingredients and how you can make it
• TheNextWeb.com – Inside Zynga and the making of its next hit, ChefVille

I hope you enjoy it.  Congratulations to the whole ChefVille team.

It’s nice to have the game announced publicly so I can say a little more about it going forward.

ChefVille has a Facebook page where you can find a trailer video and some early screen shots.

You can get a rough idea of the game from the trailer and screen shots.  It’s got wonderful fresh art, cute animations, and fun game play.  I’m proud to be part of the awesome team we’ve assembled to create and support the game.

Look forward to additional news on ChefVille, including a launch … soon.

Hidden Chronicles is a “hidden object” game in which you search for all the objects hidden within an illustration presented on the game screen (like the one to the right).  It also has mini-games and puzzles.

You can read more details in this VentureBeat article.

You can try Hidden Chronicles here

http://apps.facebook.com/hidden-chronicles/

Enjoy!

Pecha Kucha

At NetApp we employed a similar format for “outrageous opinion” talks in engineering which I really enjoyed.  So I happily accepted a recent opportunity to perform a Pecha Kucha presentation at Zynga.

We used a slightly modified format (20 slides of 15 seconds each – 5 minutes total), but the principle is the same.  I chose to transform a longer presentation on scoping for 90% into the shortened format. 

For a technical presentation, that turned out to be a fun challenge. Here’s some things I learned from that experience.

Cadence

The timed format means that your presentation will be chugging along regularly, even if you are not.  If you haven’t mastered the cadence of your slides, the timing of your presentation will sound awkward to your audience.

To address this, you’ll need to know your slides end to end. 

That is, you have to have the sequence of your presentation in your head, so that for each slide you know which is coming next, to ensure a smooth transition.

You should be talking about the next slide in sequence before it shows up, otherwise your audience will wait through an uncomfortable silence.

Overall this is tricky to master.  I recommend several trial runs through your slide deck, with the last run ideally performed shortly prior to your presentation.

In other words, practice.  Watching some examples will help as well.

Visual Simplicity

Shoot for visual simplicity.  Uncomplicated pictures, graphs, smallish text snippets and callouts trump paragraphs of text.  Avoid big gobs of text and long lists of bullet items.

Swap graphs and clip art in place of text, whenever you can meaningfully replace the content.

With any presentation, strong reinforcing visuals are important.  In Pecha Kucha, they are golden.  The format is more friendly to purely visual presentations (design, art, architecture, …), and this fact needs to be leveraged for technical content as well.

5 Seconds

Your slides should take around 5 seconds for the average viewer to read and understand.  Any longer, and they won’t be able to listen to you, or might still be reading as you transition to the next slide.

If your presentation includes a slide with 20 bullet items and lots of text, you’ve got a problem.

Cheat: Repeat

If you’ve got a key point that requires more than the allotted time to present or reinforce, simply copy / paste the slide for a second repeated time slice.

In my talk, I used a slide in order to present a more complicated topic (for 15 seconds) and then repeated the slide with animated annotations to explain key points (for another 15 seconds).

Animate, Carefully

Animations can effective but require careful planning due to the compressed time format.

Powerpoint does a nice job by default of spacing the animations out over the time interval, if your presentation is on an auto timer. 

If your presentation cadence is really on the mark, the animations can appear just in time to support the points you’re discussing with the audience.

Wash, Rinse, Spin

To maximize impact, I arranged my talk in iambic pentameter.

Just kidding. I divided my talk into three sections

• Introduction, introduce the problem (5 slides)
• Concept and reinforcing content (14 slides)
• Conclusion (1 slide)

I limited the reinforcing content to two primary examples and one shorter example.  That was a little too complicated – if I did the talk again, I would limit to 1-2 examples. 

Limit the number of key concepts so the audience can absorb them in the short time allotted.

And good luck.

One of the challenges of running an online game (like Café World, on which I’ve spent most of my time) involves releasing new features on a regular, predictable cadence.

As a result, our team has devoted resources to collecting metrics on feature development and release timing.  We use that data as a feedback mechanism to adjust our scoping and scheduling guidelines.

The approach outlined below describes a process for constructing schedules with relatively high confidence.  It also gives a statistical foundation for justifying “padding” which might otherwise raise the eyebrows of skeptical managers. 

Driving to Work

To the left is a histogram of the time required to drive 18 miles to work, over many dozens of trips.  The data is not perfect, but roughly models a route that I drove frequently – a mix of highway and local roads.

Most days, the trip took about the same amount of time, around 30 minutes. 

An occasional traffic accident would hold things up, maybe requiring 45 minutes.  Sometimes bad weather would slow things down; an accident and bad weather together would be even worse – requiring perhaps an hour.

Rarely, with unusually clear traffic and cooperative traffic lights, I could race home in only 20 minutes.  But never faster.

Schedules Are Subjective

If you asked me “Steve, how long does it take you to drive to work?” I could sensibly answer

“About 25-35 minutes”

reflecting the average, and most common trip times.  And about half the time, I’d be right.

However if I had a 9am meeting with the CEO, you could be certain that I would leave at 8am or earlier, in order to have a nearly 100% chance of being there on time. 

If you wanted to pin me down to something certain, my answer would be something closer to

“An hour or less”

When the stakes are high, a schedule with 50% confidence just doesn’t cut it.

Variability In Driving

Relative to software development, driving to work is a technically non-complicated task (apologies to all you professional drivers out there).

Driving to work is easily understood, and easily repeatable.  In my example, the route is executed each time by the same person, in the same way.  Only the environmental conditions vary, yet they substantially impact the actual execution.

In spite of the many constants, we observe a remarkable 3:1 variation in performance (20 to 60 minutes). 

That is fairly surprising for this non-complicated task.

Tossing Coins

Imagine you are constructing a schedule comprised of 10 tasks which must be executed in sequence.  Each has an independent 50% chance of requiring 1 day to execute, and a 50% chance to require 2 days to execute.

Clearly the project will require 10 to 20 days to complete, but what would be a safe estimate?  Would the midpoint of 15 days be a reasonable schedule commitment?

This admittedly simplistic example can be modeled as a series of coin tosses, with a probability distribution as shown in the figures above and to the right.

Because the distribution is symmetric, there is only a 50-50 chance that the tasks will be completed in 15 or fewer days. 

Scheduling for 15 days is not a winning strategy for a high-confidence commitment.

In the figure to the above right, the same distribution is shown with the rightmost bins annotated with the likelihood that the completion time will be fewer than the indicated number of days.

We can use this information to construct a schedule with relatively high confidence.

The figure to the right shows a bounding box around 90% of the schedule possibilities. 

A schedule estimate between 17 and 18 days will be too short only 1 in 10 times.

Statistical Variation is the Norm

Even if our estimates are perfect (and they are not), environmental factors intrude.  The network or VPN can be down, source code control can be broken, or the build might not work.  A colleague may have gifted you with an unusually complicated branch merge.

These things happen all the time, yet we often pretend they do not.  Software estimation, even when done well, is a highly stochastic process.

Scoping for 90%

I encourage our teams to scope and schedule for a 90% on time rate, since the consequences of schedule misses are fairly severe.

When iterating over a large number of feature iterations of similar size and complexity, it is possible to accumulate statistical evidence to help drive this process in a compelling manner.

For tasks which are highly novel, unusually complicated, or having strong external dependencies, variances are naturally much higher and statistical categorization can remain elusive.

But it’s always worth tracking the original estimates, the actual execution, and their historical variations.

Peopleware is a classic book on team building, productive and “jelled” teams, individual and team productivity, and work place environment. It is thought-provoking and should be must-read material for anyone who ends up working in software development.

A few topics/chapters are particularly memorable to me

• Quality If Time Permits – knowledge workers want to produce high quality products; management and silly deadlines conspire to defeat this.
• Hiring a Juggler – you wouldn’t hire a juggler without asking him to juggle; why would you hire a programmer before asking him to code?
• Teamicide – behaviors to avoid if you want to avoid killing off your team (this chapter also introduces one of my favorites – the concept of the “quality reduced product”)

There’s a relatively strong concentration on workplace environment (offices, cubes, windows, interruptions, …), with an emphasis and data to suggest that inefficient workplaces can destroy productivity.  Workers and managers don’t always have easy ways to influence these factors, but they’re very much worth pondering.

The book gets some criticism for its focus on negative behaviors to be avoided, rather than establishing positive behaviors to emulate.  Portions of  the material are also now somewhat dated (references to intercoms and 1970s pop music come to mind). 

However criticizing this book for being dated misses the point; the fundamentals remain valid years later and this remains a classic which has influenced thinking for several decades.

Overall a great read.

Zynga executes software projects in highly compressed time frames (features in days or weeks rather than weeks or months), which presents its own unique scheduling challenges.  This has forced me to think hard about how to optimize software development while moving at what we affectionately call “Zynga Speed”.

As a result, I’ve been spending a lot of time recently pondering software schedules and project management issues.  This has brought me back to some of the books and ideas that influenced me over the years.

I’ll cover these in a little more detail in subsequent posts, but the following are three good sources that stuck with me long after I read them

• Peopleware: Productive Projects and Teams, Tom DeMarco and Timothy Lister
• The Mythical Man-Month: Essays on Software Engineering, Frederick Brooks
• Joel On Software, Joel Spolsky

While some or all of the material in each of these books may not apply to the situation you’re in, each is a thought-provoking and worthwhile read.

(Thanks for reading – Steve Klinkner)

PS To be fair, the full name of Joel’s book is

Joel on Software: And on Diverse and Occasionally Related Matters That Will Prove of Interest to Software Developers, Designers, and Managers, and to Those Who, Whether by Good Fortune or Ill Luck, Work with Them in Some Capacity

You can check out CityVille at

http://cityville.com

Build the city of your dreams.

(Thanks for reading – Steve Klinkner)