Analysis of Product Metrics is Key to MVP Success

Before diving into the depths of product metrics, let’s begin with a short story.

Back in 2005, one podcasting startup was born. It was going to be a directory where people could find, download, and create audio content. The idea seemed fresh. The team worked hard on implementing it and checking it with early users. But soon the team started realizing that they just didn’t find it helpful enough. People weren’t listening to podcasts as much as they thought they would be. As one of the lead team members put it: “We built it, we tested it a lot, but we never used it.” They decided that the startup’s future was not in podcasting and they had to look for new directions instead. Along with this, the team was hit by the news that Apple was planning to launch iTunes with an embedded podcasting platform. This only proved their findings. The company started holding hackathons, where employees worked on potentially disruptive projects to switch their focus.

One of the projects revolved around the idea of simplicity and was envisioned as an SMS-based social network. It should have been so simple that the user didn’t even think about what they were doing. All they had to do is just type something and send it. It didn’t take much time to build an MVP of the platform and validate it with friends and family. Two months after the release, the platform had 5,000 registered users. Now, it has 1.3 billion. And it’s called Twitter.

What is this story about? About failure? Or, success?

Success, no doubt. This is exactly what an MVP must do — i.e. give you enough data about whether your product idea is viable or not and steer product development on the right course. Even if the post-launch data proves that no one is interested in your MVP product, it is a positive outcome because you still have the resources to check other hypotheses as well. This is the foundation for iterative, or lean, development.

Already a business classic, The Lean Startup has paved the way for a new method to launch successful products — build-measure-learn. It follows the principle of continuous improvement dictated by data about customer interactions with the product. This data-driven methodology switches focus from expert intuition to knowledge and certainty, and thus minimizes the risk that comes with impulsive gut-triggered decisions. The result? — A polished sophisticated product that everyone wants.

Yet still, not all product data is valuable. Data only makes sense when it gives you insights upon which you can base your decisions about the product’s further development.

When setting benchmark metrics of success, it’s worth aiming at those of similar products. And when we say ‘similar’, we mean those matching your product’s characteristics in terms of a niche, market size, audience, user scenarios, and even geographical location. Say you are planning to release an MVP for a peer-to-peer micro-lending platform in the Philippines. It makes no sense to base your success metrics upon the performance indicators of popular fintech products globally. It’s like comparing apples to oranges. The end result will be very misleading. Try to keep your focus as narrow as possible — that’s the only way to get the metrics that are meaningful for your product success.

Moreover, you should also keep in mind that you set performance metrics for an MVP, and not for a ready product. It is a whole different beast. Normally, minimum viable products have only one core function — hence, they are minimum. In order to capture relevant data and measure the right things, you must adapt the conversion funnels and user flows to that very function. By doing this, you’ll cut off all vanity metrics and focus on what really matters for your product’s growth at the current stage.

In order to set the MVP product success metrics right, we suggest sticking to tried-and-tested frameworks. One of them is the AARRR framework introduced by David McClure, an entrepreneur, angel investor, and founder of the 500 Startups accelerator.

AARRR metrics of success

AARRR metrics are funnel-based, meaning they account for user interactions at every touchpoint — from only trying to use your MVP to converting into a paying customer. This gives you an understanding of how people use the product over the course of their journey, what they value most, and where the weak link in your chain is. These product metrics provide you with just enough data to inform your decisions.

Acquisition

Product metrics: traffic, cost per click, customer acquisition cost, click-through rate, bounce rate

Practical insights

A good practice is tracking metrics for each channel. That way, you will see which channels work better than others. Remember the 80/20 rule? It says that 80% of results come from 20% of the effort. If we refer to this particular case, it means that 20% of channels will generate 80% of leads. So why spend money on the remaining 80% of channels that do not work? It’s a waste. Better re-invest it in the 20% of those that have proved effective.

Activation

Product metrics: conversion rate, visitors to registration value, time on the page

Practical insights

If a large percentage of users give up at this stage, there can be several common reasons for this: 1) wrong audience, 2) bad user experience, or 3) no value. With the wrong audience, it’s clear — you need to get back to the user research stage and refine user personas. To enhance user experience, you need to know at what point people drop off. For this, look into heatmaps that show how users actually interact with your MVP. Based on our experience, the root cause of failed activation is often a challenging registration or lack of onboarding. So it’s worth directing the UX improvement efforts to these actions. And finally, if the user doesn’t see the value in your MVP, it’s a warning sign of the product’s irrelevance, indicating that you need to pivot. But, there’s also a possibility that the user hasn’t had a chance to grasp the MVP’s value yet because the path to it is too bumpy. Then again, you need to reconsider the UX design and simplify it.

Retention

Product metrics: repeat visits, daily and monthly active users, churn

Practical insights

If the user retention is low, it means that people just don’t want to use your MVP because 1) it doesn’t deliver on promises, 2) it’s a bad value for money, or 3) it’s too complex. What you need to do is optimize the MVP’s core function, reconsider the pricing, or improve UX and add customer support, respectively. There’s also one more interesting scenario. Chances are some users haven’t intended to buy your product at all — they may only be attracted by the freebies you offer. Then, it makes sense to switch focus to paying users instead, meaning you need to define the target segment and invest in acquiring such leads.

Referral

Product metrics: referrals, mentions, viral coefficient, viral cycle time, ratings & reviews

Practical insights

If you have noticed that people readily use your MVP and even recommend it (including online reviews and shares), consider implementing an affiliate program that rewards both existing users and those they attract. This will imply that you care and also encourage them to further fuel your product’s viral cycle. By doing that, you can significantly reduce the acquisition costs.

Revenue

Product metrics: average revenue per user, customer lifetime value, monthly recurring revenue

Practical insights

We advise you to again resort to the 80/20 rule. Here, it reads like this: only 20% of customers are responsible for 80% of the revenue. By analyzing the collected data, determine who that 20% are, what they have in common, and what they value in your MVP most. That’s your sweet spot. Invest in attracting leads that fall within this category and make sure you provide them with added value by optimizing their experience with your product.

Yet, chances are that you may still have an unclear picture of your product performance even with the proper data analytics in place. If this is the case, your quantitative analysis should be supplemented with the qualitative one — i.e. customer feedback by means of interviews, surveys, and questionnaires. We often apply it at the stages of activation to understand whether the user finds an MVP valuable, and retention — to clarify what functionality improvements they expect. Mind, however, that customer feedback should not be treated as the only success metric for a product. Surveys are a tool that is used to understand the customers’ satisfaction level. They add to your data analytics effort, and not replace it.

Which data analytics tools can you use? Google Analytics, Amplitude, Mixpanel, Hotjar (for heatmaps).

There’s one simple truth you should stick to. No matter how scrupulous you are in building an MVP, all your work can be vain if you haven’t embedded analytics in it. It’s like driving in a foreign country without a GPS or a map — you simply don’t know where it will take you and whether you can get to the destination at all before you run out of petrol.

When clients turn to CXDojo for consulting, we always emphasize that measuring an MVP must be accounted for as early as at the stage of planning. Before building it, you need to know for sure what will be regarded as product success. So think about what you will measure and how you will do it from the very outset. And if you have trouble with it, we are here to guide you.