Putting metrics in place is hard. Some hard-earned lessons to get you measuring
Hey designers, while you're consumed with finding the right metric, you should know a few pragmatic realities of putting a metric in place. Here's what I've learned over the years.
You should know the difference between measures and metrics.
Measures are numbers that give life to metrics. An Error Rate might be 10%, but did you know it has three measures?
# of total possible errors
# number of participants
# errors for each participant
You can't know the error rate until you have these these measures first.
You have to baseline your measures first before you can derive a metric.
Why? Because you want to feel confident that the measure is relevant and real, not that you just got lucky. You want your measures to have business relevance AND statistical significance.
Business relevance is all about measures that connect to a business outcome. Lots of the article you read about the right metrics are only talking about business relevance. This is only half the equation. Relevant measures that aren't real is guessing and guessing is risky!
For statistical significance, you need to think about two things:
The size of the sample of people
The variation in the people you sample with
#2 is where a lot of teams go sideways. When teams only sample from every type of customer, larger sampling errors occur.
Once you baseline, you have to choose a new target to achieve
Your target is really the outcome your experiment is trying to achieve or relate to. Think about it as an if/then statement. "If we reduce errors, then expect to reach our target."
All of the above does not happen over night
This is perhaps the biggest reality to overcome. Getting a metric in place that you can rely on will take more than a 3-week sprint. You will have to make difficult trade-off decisions to just get a baseline measure in place.
Getting a metric in place is more than a business problem to solve
It's also solving people and process problems. You will have to find new ways to get measures in place while your colleagues or stakeholders are worried about slowing down or missing shipping dates. It's these additional factors that we, as an industry, don't talk about a lot and it's a disservice to us all. The more we argue about what the right metrics are, the more we delay having actual measures and metrics in place. We can only learn from them once they're there.
TLDR: You can't choose where to go without knowing where you are. Knowing where you are takes time and energy. There are no right metrics. There is only learning from imperfect measures and metrics to know what are more right. IME, the benefits of doing all this work is you will have MUCH more confidence in your decisions after you have imperfect measures and metrics in place.
You just have to let go that the right metric is there before you start.
