Finding out in time that you darling idea sucks, or your beloved baby is actually ugly, altering course in the right direction, is the secret potion to product success. Corporates and Startups struggle with un-cracking the elusive product-to-market-fit nut. Product teams can get so obsessed with their baby - the solution, and ignore customers problems. Being receptive, indeed looking for those signals and telling them apart is important and messy to navigate. The process is ridden with pitfalls, sending teams down snake holes, causing debilitating delays and even compromising competitive advantages. So how does a product team stay on track and know what to listen to, what to ignore? Leveraging data to draw the right conclusions on the one hand, while leveraging the expectations and opinions of stakeholders on the other can be a daunting task. Project managers juggle cost, time and resources, to steer the team towards the fastest possible valuable result, while keeping team motivation in check. How value itself is defined varies again on whom you ask - from the HIPPO (Highest Paid Persons Opinion), to the investors, the team members - everyone has an opinion about what should be done and how best to do it. What then, is the right measure of success? How do you know if your baby is ugly when you’re increasingly falling in love with it? On a very broad level, it has to do with how well we are able to listen to people and data.
Early metrics - customer discovery
Listen Now
Do it yourself. This is your baby, the most important work is to discover if customers like your idea or not. It is not outsource-able. You as a founder have to do it, or whoever it is that is capable of changing strategy.
Never ask ‘Do you like it?’. Your baby is always beautiful. Ask ‘How do you <solve said problem> presently?’ Learn how to do this right, don’t speak to your customers looking for confirmation, speak with the intent to listen. Dig deep using the 5 why’s technique for root cause analysis.
Make it contextual. Recreate realistic scenarios. Ask about situations, how they had reacted, what problems they faced, what jobs they were trying to get done. Don’t ask about future behaviour - buying preferences, feedback on your potential solutions. Watch out for social desirability bias, most people lie in everyday conversations to appear likeable and competent. Restrain from asking shaming questions like ‘why don’t you take your medication regularly?’ Ask ‘when was the last time you took your medication?’
Observe your potential customers. Watch what they do, not only what they say. Observe interaction with existing solutions. One of Christensen’s canonical examples of ‘hiring a milkshake’ tells the story of a fast-food restaurant chain that wanted to improve its milkshake sales. Responding with a range of new flavours that customers ‘said’ they would like, resulted in no improvements. By observation and digging deeper, it was learnt that commuters were the major consumers, who were in a hurry, they were wearing work clothes, and had (at most) one free hand and wanted something to do on the boring commute. It turned out thinner straws and thicker milkshakes with more fruit were the answer for commuters, it made the drink last longer and more interesting.
Strap the scope of conversations, repeat the scope with at least 8 - 20 people. Document the things you have learnt, note down patterns you have noticed. Analyse opportunities, extract meaning, prioritise. Especially those that are product strategy relevant. Adapt and go on to the the next 20.
When do you stop doing interviews and start writing code? When the learning curve on discovery flattens out. But you don’t ever stop getting feedback from customers. It merely shifts from discovery to building, testing and shaping the solution.
Related: When to stop customer interviews; Why product managers don't build what their users ask for
Measure later
Focusing on hits, page-views, visits and events at early stage can be very misleading. A landing page that gives you 20 hits or 200 hits, must be explainable in terms of business model or problem insights - if you don’t get it, if it is not actionable, it means nothing. Study the competition.
Conducting desktop research can be very educational towards learning about present solutions, their market share, metrics around their features and offerings - what they’re getting right, what they’re getting wrong. What unique opportunity or gap could it open for you. Avoid getting drowned in a myriad of non-actionable data points so early on (or ever for that matter), just to have some sort of proof for the decision makers.
Products metrics - MVP (Minimum viable product) and beyond
Retention before growth
Identify and understand what’s happening in the customer development. Data is great at identifying problems for further investigation. Better data doesn't necessarily lead to better products, unless you understand why it’s happening, e.g. why only 60 of the 100 users that downloaded your app become registered users. Qualifying what deterred the other 40% is important. Observational usability is a good way to learn if the registration process is broken, but also talking to some of those 40% to learn what drove them away.
First impressions count. Startups and SaaS solutions often invest heavily on early users, offering free trials and on-boarding support. Watch these metrics closely, this could be the entire basis for product success! Reach out to the users that are getting disengaged, offer them extended trials with closer interaction through things like ‘shadowing’. Understand what kind of users they are and how their behaviour could predict and prevent future usage issues.
Measuring quality engagement. What features are users engaged with most, how often are they used? Don’t focus on granularity (likes, comments) decide why a particular usage metric is indicative of the products performance. Hitlist, an app to help people find good travel deals tracked how many cities added were added per user. Although this metric was high and seemed very encouraging in terms of usage, it was quite irrelevant in terms of product success. Instead, measuring users that were clicking through to booking was much more relevant.
Keep your good customers. Acquiring a new customer, according to Harvard Business Review, is 5 - 25 times more expensive than to retain an existing one. Investors and stakeholders tend to push hard to see growth. As the wise Tarun Mitra has said, ‘If you invest in growth before you have retention, you’re renting users, not acquiring them’. Cross and up-selling to customers where trust has already been established is also vastly easier than selling to someone new. It is as important to filter the relevant early customers from the rest by looking at the data more closely. MySugr, a diabetes management app, was famously heralded to be a ‘beautiful solution’ by early customers, many of whom were not diabetics.
Driving growth is done best by happy satisfied customers, incentivised to share their experience through recommendations and referrals.
Study cohorts
Segment users into cohorts. Measure how many and what kind of users keep coming back to your product or app and understand what they do, relative to last week, last month (DAU, WAU, MAU). What feature sets tend to be used by some people, why? Measuring things like averages, hide the pattern behind the data, e.g. revenues drop as customers age. Follow the what happens to users who find your product through Facebook Ads and Google Ads, measure conversion and how many converted users are engaged, retained (or churned) with time. Investigate outliers, if there are a few exceptional high frequency users, double down on what exactly is happening there. Uber learnt that special attention had to be paid to high value users, which lead the company to invest in specialised focused research of day-in-the-life of high frequency users, in their homes with a team of 5, providing video footage on insights for product teams.
A/B tests are a great way to measure what happens with a change vs. whats happens without the change to test which was better for your product. Even when running these tests on different batches of people it is important to know what is happening, to know the change is for the right reasons - was A buggy and hence won against B? Also make sure you have’t created more/other problems.
Related: Focus on metrics that matter
Revenue Metrics - Stakeholder interests
Money talks. Optimising on the above metrics, is what goes to ensure a stabile flow of revenue. Solving a problem, acquiring and retaining satisfied users is the foundation for the expected outcome on revenue. Referrals come out of delivering value to customers and has a great effect of knocking down acquisition costs. The most straight forward way to success is to reduce the customer acquisition cost (CAC) and increase the amount of revenue you earn from a customer during the lifetime as a customer of your product (CLV).
With careful nurturing and due attention to the signals in between the noise, your baby will mature into a valued and beautifully independent adult solution.
Read also: