Measure thou not in vain.

Improve business analytics with gamification data

Peer deeper into vanity metrics.

Measure thou not in vain.



Every business has "table-stakes" metrics that must be reported on but don't necessarily yield profound insights. By way of analogy, no one would think about their spending without monitoring their checking account balance; but it doesn't have anything to say about where the money is going. Web site page view reports are my favorite example in the software world. Page views is a vanity metric. It looks good if it goes up, but by itself it's too simplistic to say much about the health of your traffic.

Thankfully, you can turn vanity metrics into valuable reporting by faceting them into cohorts. If you're using gamification, you can base the breakdown on your games, which will let you understand gamification ROI by seeing how it impacts business outcomes.

In this follow-up to part one of my series on monitoring gamification, I'll cover a couple more examples of reports that lead to meaningful ROI insights. Let's return to my hypothetical food-related web site, "Food Unicorn".

Scenario #1

My favorite vanity metric is up first. Web properties monitor page views and we're no different. Here's our graph:

page view graph, up and to the right

Up and to the right! Great! But let's say that 6 months ago, we decided we wanted to increase social sharing (that is, people sharing links to our stuff via Twitter, Facebook, etc.). To incentivize that sharing, we award points when a user posts content to social destinations, and users earn badges at certain intervals. Naturally, the boss wants to know whether that's working. We can report on level attainment (and it's a good idea), but that only tells us something about the health of the game, not the desired business outcome.

So, let's facet our vanity metric using cohorts based on the sharing badge and see if we can turn it into something useful:

the same page view graph with cohorts for deeper insights

No, "bronze/silver/gold" isn't the greatest level naming scheme, since it sets the game up for premature extinguishment… but I'll leave that to a future post.

Better! It's the same up-and-to-the-right trend line, but now you can say conclusively that users playing the sharing game are engaging with the site more—page views from non-sharers are only up slightly, so the stellar growth is actually coming from badge-holders.

(This is just meant to be illustrative; I would suggest that rewarding users based on the inbound visitors that can be attributed to their shared links is much more meaningful to do, and report on.)

Scenario #2

a friendly unicornFood Unicorn, fueled by investor enthusiasm over that fantastic page view growth analysis, introduced user-generated recipes a few months ago. Being really smart, and knowing that we'll want to weed out low-value UGC, we introduced the ability for users to rate other users' recipes. Being gamifiers, we award "reputation points" to recipe authors when their content is rated highly, and deduct points when it's not liked.

This is good gamification (even shamification): as long as it's exposed to end-users, the point system will improve the overall usefulness of content by itself. But these reputation points also give us a better way to understand changes in the quality of our UGC over time.

I won't beat up on the need for cohorts again; let's just skip to the good chart. Using some unicorn-related mathematics, we can break up our users into groups from strongly positive to strongly negative reputation, then count the contributions in these bands:

graph of recipe contributions, by cohort

This chart tells us not just that the volume of contributions is increasing nicely, but that the ratio of good stuff to bad is getting better all the time. Contributions from users whose overall reputation score puts them in the two negative bands are tapering away, just as we want it to.

By combining a vanity metric—contributions over time—with gamification cohorts, we can tell a very strong story about the success of our reputation-based incentive program.

Have a comment (or a unicorn)?

How are you proving the return on your investment in particular gamification programs? Let us know in the comments below.