When Metrics are Distracting

I was thinking about my post yesterday because I'm spending a lot of time thinking about metrics, KPIs, analytics in general these days. In fact, I'm spending a disproportionate amount of my time thinking about metrics. Related to this is A-B and multivariate testing: relentlessly tweaking copy, imagery, and click paths to eek out a percent here and there. I hear stories "you won't believe how big a difference just changing one word makes!", but at the stage we're in, is this really the best use of our resources?

When I say "resources", I'm referring to the 10's of thousands of dollars on an analytics package, management's time scouring reports, my time taking the derivative of reports for management, the developers time parsing log files to get me the data and "tweaking" a word or image here or there. I'll be the first to admit these numbers, and this analysis is valuable, but if I were to represent it like this.

A Hypothetical Retention Graph:

Where I spend my time between week 1 and week 8 making tweaks. Of course, I can paint this hypothetical scenario in a way to prove my point, but what if each variant costs me and the rest of the team 10 units of effort per week? Then. between weeks 1 and 8, I spent 70 units of effort.

Now, what if the reason my hypothetical retention is low isn't because my copy is bad, or my hero image sucks, but because I have a core product issue?

Not to pick on Dogster (from my last post), but their issue wasn't copy, it was that their feature mix wasn't right. It took looking at their metrics to uncover the issue, but they didn't use their metrics to SOLVE the issue.

If you assume the effort to re-architect the site between weeks 8 and 9 also cost me 10 units, then there's a clear ROI given the results (my retention # significantly improved). But, even if the re-architecture cost me 100 units, I would argue it's the right thing to do:

• If you have a core product issue, then the product isn't healthy, and probably isn't a sustainable/ viable/ optimized business in it's current state. You'll need to take the plunge eventually.
  
• There's an opportunity cost to delaying the "hard decision" for 8 weeks. In those 8 weeks, retained users could be adding content, buying subscriptions, and clicking on ads!
  

When Sociality Takes a Back Seat

At SXSW last week, one of the best panels was "Start Up Metrics for Pirates": Dave McClure - 500 Hats (moderator), Ted Rheingold - Dogster, Todd Vernon - Lijit Hiten Shah - CrazyEgg Lance Tokuda - RockYou. I'd seen Dave McClure's presentation previously at Ignite Seattle, but having the other folks on the panel to provide applied examples of the metrics made the topic more relevant.

Ted Rheingold from Dogster provided his account of Dogster's loyalty issues. To quickly summarize:

• They were spending a tonne of money on Google Adwords
• The drove a tonne of traffic, and they were pumped.
  
• When they later looked at the loyalty numbers (the number of newly registered users that were returning) they were disturbed to find that most weren't returning.
• They did an audit of their Google Adwords keywords, and they noticed that the highest converting keywords were informational NOT social in nature. (no real surprise, people were searching for "boston terrier breed information" not "dog friends").
  
• Dogster redesigned their site to accommodate the information searcher. They de-emphasized their social networking component, and emphasized breed and animal ailment information.
  
• The result: they stopped spending money on Google Adwords and were driving traffic because the site was SEO-friendly (relevant-information-dense).
  

What Dogster and Livemocha have in common:

• Niche Social Networks: That is, unlike Facebook and Myspace which address a non-specific target audience, Dogster is a social network for dogs (and their humans), and Livemocha is a social network for language learning.
  
• Social Graph Comprised of Common Interests, not "Real World" Relationships: To put another way, just because I'm learning French, doesn't mean my friends are AND just because I have a boston terrier, doesn't mean my friends do.
  

Are common interests sufficient to drive strong online social ties? Clearly in the case of Dogster, it wasn't. It dawned on me during the talk that this was a perfect example of Josh Porter's Del.icio.us Lesson: Personal Value Precedes Network Value.

At Livemocha, we dodged this bullet because we built the social network as a feature to support our underlying value proposition. We looked at our goal (to help people to become communicative in a new language) in terms of a "language acquisition curve" that started with self-study (not particularly social) lessons, but gradually worked the user into the community as it benefits them.

... and, yes, in case you're wondering, here's Sousa's profile.