Metrics help companies to achieve goals and transform business by pinpointing areas for improvement. But sometimes, metrics can actually be harmful and counterproductive. Excessive focus on metrics can cause product teams to neglect customers’ needs.

To find out how metrics can rob value and what helps to balance between business and users’ needs, we interviewed experienced product managers from different companies. We asked them three questions:

  • Why do product managers lose sight of the needs of real users behind the metrics? 
  • How can metrics be deceptive? What are the symptoms of this kind of substitution?
  • How can we apply metrics while still keeping users’ needs in mind?

We would like to thank the product managers who shared their experience with us and helped us answer these important questions:

→ Test your product management and data skills with this free Growth Skills Assessment Test.

→ Learn data-driven product management in Simulator by GoPractice.

→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.

Q: Why do product managers lose sight of the needs of real users behind the metrics?

Sometimes company leadership, and the product managers and teams reporting to them, become obsessed with quantitative metrics like revenue, installs, and downloads. This excessive focus on numbers distracts PMs from the needs of real users. Other common causes, according to the experts we spoke to, include: 

  • Focus on vanity metrics and lack of qualitative research
  • No clear goal or product vision
  • Outdated solutions 
  • Priority on investors, not customers
  • Сommunication issues and lack of autonomy to make independent decisions

Paola Reyes (Senior Product Manager at Zalando Payments) 

Product managers lose the needs of real users behind the metrics when they expect metrics to tell the whole story about their product. But quantitative metrics don’t tell us why users are behaving in a particular way. That’s why combining quantitative insights with qualitative research is so important.

As a product manager working on fintech products, qualitative user testing in the form of interviews or prototype testing has helped me understand, for example, if and why users might feel unsafe when providing a particular piece of personal data. I would not have gotten these insights by looking at the conversion rate of that data field alone. By having these qualitative insights, it was easier to develop more empathy for my users and to better understand their needs when working with my team on designing a solution.

Keshav Sharma (Senior Product Manager at Salesforce) 

Companies usually lose sight of real users for two reasons:

1. Leadership is obsessed with quantitative metrics such as revenue, GMV, installs, downloads, etc not to improvide value, but to please external stakeholders over their customers—for public companies the stock market, for private companies—the investors. When this happens, organizations overoptimize on metrics instead of truly understanding their customers. Besides, this obsession provokes similar incentive-based behavior from product teams.

2. The product team stops investing in the user research team and processes that help a company get qualitative data. Usually, the excuse I hear for this is that user-research is too “expensive” or “time-consuming”. 

Ammro Hussein (Product Manager at Instagram)

Steve Blank said in his book, The Startup Owner’s Manual, “Rule No. 1: There are no facts inside the building, so get out.” Unfortunately, most product managers get caught up in reporting team progress daily or weekly and lose sight of how their users are really doing.

The founders have a vision of how the world around them should function, and they try to bring it to reality. So the product team becomes an execution squad with no autonomy to make independent decisions. The team’s main customer is the founder, not the end user. These things don’t incentivize the product teams to follow Steve Blank’s advice by going outside to talk to users. Users can be contacted in a variety of ways, such as surveys, focus groups, monitoring online reviews, and feedback. When you lose the voice of the customer, the only voice left is the metric movement. Despite metrics being useful for measuring progress and indicating leading or lagging issues with a product, they will never take the place of the voice of the customer.

Rocio Del Moral (Lead Product Manager at Google)

It’s easy to get distracted if there is no clear goal or product vision. I have seen situations like these: 

1) Old solutions that worked great no longer do. I have seen PMs (and sometimes the team at large) fall in love with a solution that perhaps worked very well in the past but no longer. User needs and problems evolve. When you stop talking to customers, or don’t have the right mechanisms to keep track of changes in user needs, then what was perhaps a cash cow in the past, will start declining.

What to do: Be flexible and always stay in touch with your users. That can take many shapes and forms, from constant experimentation to user research, in-depth interviews, etc. Make sure that you have mechanisms in place to understand this evolution. 

2) Focus on vanity metrics over key metrics. “Vanity metrics” are metrics that seem nice but are not connected to your mission and goals either at the company or product level.

What to do: Start from users and see how you can solve their problems with your company mission in mind. Then it will be easier to identify key metrics that are relevant even before you launch anything. 

3) Communication issues across the team. Even if you know what the user problem is, if you don’t communicate effectively to your team, or if you don’t include your key team members from the start, then you’re bound to fail to understand the user as a team. At that point, metrics become meaningless, since your team doesn’t understand the user problem.

What to do: Inspire and influence your team. This most certainly requires effective communication.

Q: How can metrics be deceptive? What are the symptoms of this kind of substitution?

Metrics are helpful but in some cases, they can be misleading. You should avoid using “vanity metrics”, which are flashy, attention-getting numbers that don’t give actual insight into what is happening inside your product. There are many ways a metric can be deceptive:

  • It doesn’t measure the outcome correctly.
  • It doesn’t reflect actual user behavior.
  • You focus on the short-term results instead of focusing on wider progress.
  • You don’t truly understand how your input metrics map to your overall output metric.

Here are more scenarios and examples of how this happens:

Ammro Hussein (Product Manager at Instagram)

Metrics can be misleading if they do not accurately measure the users’ actions. This means the numbers we are looking at are not reflecting the actual user behavior.

Also the metric can be deceptive if it’s trending as the product team is expecting (upward or downward) while the product is suffering from cannibalization or lack of engagement depth (meaningful interactions). A good example of deceptive metrics can be found in many of the 2-sided marketplaces, where measuring DAU or clickthrough rate (CTR) doesn’t necessarily reflect that users are getting the best out of the platform. It is possible for users/buyers to browse products on the platform, but not to purchase or interact with sellers.

Neeraj Mishra (Lead Product Manager at Google)

Once we were trying to maximize the active user base in order to increase the overall revenue for an e-commerce platform. While we saw an increase in the DAU/MAU over a period of time driven by multiple user engagement and activation campaigns, we didn’t see any significant uptick in the revenue even though we had found strong correlation between both. 

Digging deep into the root cause, we found that even though we had increased the traffic numbers, the majority of those new users were discount seekers (students, early-stage professionals, etc.) who only purchased items on sale. This increased the overall transaction volumes but didn’t lift the revenue as expected. By taking a deeper look at the data and by interacting with the users, we were able to better understand their motivations and realigned product offerings. As a result we’ve maximized repeat purchases at lower discount buckets along with targeting a newer untapped user base.

Paola Reyes (Senior Product Manager at Zalando Payments) 

Let’s imagine you are working on a platform for selling used cars and you are trying to understand if the recommendations are relevant for your customers. If you are looking at time spent only you might end up with a recommendation engine that leads customers to click through less relevant listings and results in an increase in time spent. Instead, better metrics might be the average amount of time it takes the customer to purchase a car through your site and the share of customers purchasing a car. Going back to the ultimate goal you are trying to accomplish—customers buying a car, in this case—helps you avoid choosing a deceptive metric.

Similarly if you were working on improving your site’s checkout experience and you were optimizing only for the checkout rate, you might design a checkout experience that is so efficient that customers are accidentally checking out before they are ready to make their purchase. This could result in frustrated users who may not return to your site or a spike in your return rate. One solution might be to look at the checkout rate after returns are factored in. Identifying and keeping track of health metrics, such as return rate, in addition to product metrics can help paint a more accurate picture about your product’s performance. Also it is important to be mindful about possible time lags. Metrics can be deceiving if they measure a trend that requires time to develop and you are looking only at a shorter time interval.  

Keshav Sharma (Senior Product Manager at Salesforce) 

A classic scenario is when a PM over optimizes for the primary metric and does not pay enough attention to the secondary metrics. This is usually true when the organization does not take a long-term view. Let’s consider a early-staged startup. One of the primary goals for companies at this stage is growth. Growth could mean different things. However, for the sake of this example let’s consider the goal of an organization is to grow the number of new users. While the number of new users grows, this does not guarantee long-term success for the business. Since retention plays a key role in long term business and can be more important than customer acquisition. 

Another example is when the team is unclear about the reasoning behind the success of a metric.

Your overall engagement product metrics—for example, time spent performing an action in the app—can paint a false picture of success. Metrics come in two forms: input metrics and output metrics. Time spent performing an action is an output metric and hence a lagging indicator of success. Without truly understanding how your input metrics map to your overall output metric, product teams cannot make good decisions.

Let’s extend the example above. Say the startup grows the number of users and the session length of each user. Seeing that the product metrics are performing well, the startup decides to continue doing what they have been doing so far. However, this might lead to a worse product. Why? Because in hindsight one or more input metrics were not given as much attention as they needed. And this was eventually reflected in the poorer performance. 

Rocio Del Moral (Lead Product Manager at Google)

Beware of situations where your customer or user base is shrinking, but this is masked because revenue is steady. You should keep a close eye on the health of your user base and look closer when there’s a dip in this number. What is causing users to cancel or stop using your product? This of course will depend on the nature of your product and any recent features you might have launched. Thus, it’s important for you to develop a way to look closely at the effects any new feature launch has on the overall business. The success of a new product or feature cannot come at the expense of the user, company, or mission. 

Q: How can we apply metrics while still keeping users’ needs in mind?

Try to start by choosing an appropriate North Star metric and then defining other product metrics you want to measure. It’s important to have a product vision and set metrics for success during the planning phase. The experts also mentioned some other key guidelines: 

  • Use more than just metrics to make product decisions.
  • Cite or identify the user challenges and insights that back those decisions.
  • Evaluate features and launches both qualitatively and quantitatively.
  • Create a culture where your team truly cares about how your product is adding value for its customers.

Rocio Del Moral (Lead Product Manager at Google)

It’s important to have a product vision and established metrics of success at the planning phase. Looking at just revenue or just certain metrics that perhaps were established after launch is shortsighted. It misses out on a lot of key details. So, when you’re sitting down with your team to identify the user problem, also remember to set the metrics of success early on. These are the indicators of whether your solution is actually solving the right problem or not. These metrics might evolve over time as you launch new things or the business changes, that’s valid. Just make sure to establish mechanisms to review, triage, and tweak metrics of success to adjust to new realities over time. 

Neeraj Mishra (Lead Product Manager at Google)

Here are some tips to ensure that you don’t lose sight of your users when building features and products:

1. While identifying a challenge or an opportunity, always factor in direct user feedback on your product. You can do this by running user research, speaking directly with your target customer segments, or working closely with your customer support teams to understand the issues your users face.

2. Break down your users into meaningful user segments when analyzing the product metrics to get more insights on how different sets of users use your product. You will be surprised how the use cases vary and how diverse and unrelated their challenges are.

3. When evaluating the success of a launch, please factor in direct user feedback in addition to results from A/B tests and post-launch impact to identify where the opportunities for improvement lie. You can do this by asking for feedback from the feature launch right after they experience it within product nudges or via a separate survey sent to the users later on. Another thing that you can do to get a better sense of how users perceive your feature is to closely monitor customer support interactions after the launch for specific feedback or complaints associated with your launch.

Paola Reyes (Senior Product Manager at Zalando Payments) 

You should ensure that you understand how the product metrics are connected to the business metric you ultimately want to improve. You should then prioritize optimizing those product metrics with the strongest impact on your business metric. Of course, it will require testing and analysis to uncover the correct and strongest business metrics.

If your company’s main KPI is not moving in the desired direction, micro conversions indicate where users are dropping off or experiencing points of friction. However, you should not optimize just for those micro conversions without connecting them to your main business metric. One challenge here is that in some companies different teams own different parts of the user journey and so they will optimize for the last step in the journey their team owns. This can be risky because, for example, it can result in building your product for the wrong type of user: those who convert in the upper funnel but not in the lower funnel. Connecting your product metrics to your business metrics is key.

Ammro Hussein (Product Manager at Instagram)

A North Star metric needs to be defined based on the current goal that the team is working toward in this stage of the product lifecycle. In a newly launched marketplace, user adoption is a primary goal, and an immediate secondary goal is engagement. So that means that the North Star metrics have to measure the success of achieving those goals. As the product moves through the different stages of the product lifecycle, the team has to evolve their North Star metric and how they measure its success.

To avoid measuring the wrong thing or obsessing over the North Star metrics, here comes the importance of guardrail metrics and counter metrics. The product teams must carefully select counter metrics to protect other aspects of the product that are outside their scope of work. This way they apply guardrails to ensure that their primary metric is not deceptive or giving false positives or negatives.

Keshav Sharma (Senior Product Manager at Salesforce) 

Emphasizing user research and actively making time to participate in user interviews is key. It’s also important to understand your metrics deeply and know how your input metrics are influencing your output metric. Beyond all, if you are in a leadership role, create a culture where your team truly cares about how your product is adding value to its customers.