When Metrics are Distracting

I was thinking about my post yesterday because I'm spending a lot of time thinking about metrics, KPIs, analytics in general these days. In fact, I'm spending a disproportionate amount of my time thinking about metrics. Related to this is A-B and multivariate testing: relentlessly tweaking copy, imagery, and click paths to eek out a percent here and there. I hear stories "you won't believe how big a difference just changing one word makes!", but at the stage we're in, is this really the best use of our resources?

When I say "resources", I'm referring to the 10's of thousands of dollars on an analytics package, management's time scouring reports, my time taking the derivative of reports for management, the developers time parsing log files to get me the data and "tweaking" a word or image here or there. I'll be the first to admit these numbers, and this analysis is valuable, but if I were to represent it like this.

A Hypothetical Retention Graph:

Where I spend my time between week 1 and week 8 making tweaks. Of course, I can paint this hypothetical scenario in a way to prove my point, but what if each variant costs me and the rest of the team 10 units of effort per week? Then. between weeks 1 and 8, I spent 70 units of effort.

Now, what if the reason my hypothetical retention is low isn't because my copy is bad, or my hero image sucks, but because I have a core product issue?

Not to pick on Dogster (from my last post), but their issue wasn't copy, it was that their feature mix wasn't right. It took looking at their metrics to uncover the issue, but they didn't use their metrics to SOLVE the issue.

If you assume the effort to re-architect the site between weeks 8 and 9 also cost me 10 units, then there's a clear ROI given the results (my retention # significantly improved). But, even if the re-architecture cost me 100 units, I would argue it's the right thing to do:

• If you have a core product issue, then the product isn't healthy, and probably isn't a sustainable/ viable/ optimized business in it's current state. You'll need to take the plunge eventually.
  
• There's an opportunity cost to delaying the "hard decision" for 8 weeks. In those 8 weeks, retained users could be adding content, buying subscriptions, and clicking on ads!