Types of Metrics, Political Wrangling
Jul 29, 03:32 AM
Joe Ludwig pointed me to an article on Andew Chen’s blog about developing metrics in the context of a startup.
While the article is mostly referring to metrics for web startups, I really like some of the points he makes about the political stance you have to take to adopt metrics at a small company. Some of his advice is universal no matter what kind of company you’re running.
He talks about developing metrics with a “layers of onion” type of approach, which is one that I always recommend our clients take. For example, if you’re developing an MMORPG, the first thing you want to collect metrics on is basic population data. During character creation, record name, race, class, maybe a basic character buildout, whatever is relevant for your game. Maybe throw in some basic advancement metrics (time-to-level, etc) and that is your first pass on metrics.
Once you’ve tackled that problem, add some metrics that tie into that: for example, now that you know what level everyone is, you can tie that in to PvP combat metrics to get a sense of whether there are certain character builds that easily defeat characters of an unacceptably higher level.
Actually, that ties into Chen’s characterization of “Operational reports versus Investigative reports.” Using his characterization, I was just discussing some Investigative reports: you’re wondering how well game system X really works and you’re putting in metrics to find out. Operational reports for a traditional subscription MMO might be churn-related, or a measure of content burn: how many hours of play does it take for players to exhaust all of our content? In a free-to-play MMO you might be more concerned with conversion rates and item sales.
On the other hand, in a heavily PvP-based game, a report on PvP balance very well ought to be considered Operational since that’s the crux of the entire game. If your PvP sucks, you lose money, end of story.
So perhaps the divide between Operational and Investigative is less clear than Chen draws it.
And while Chen may be right that to have completely ideal analytics you’d require about 20% of your engineering resources focused on analytics, it’s important to note that you can still get a TON of value from even a small amount of metrics.
Speaking in terms of politics inside a company, the layered onion approach is also important in terms of allocating engineering resources. If you put one engineer on part time to get a small amount of the right metrics, then people will see those reports and clamor for more, starting a snowball effect that leads to more metrics resources. Before you know it you’re buried in data, in a good way! (Contrariwise, if you put that engineer on a small amount of useless metrics, you can kiss your data goodbye. That will “prove” that metrics are a waste of time and you’ll never see resources put into metrics ever again.)
