Data drives product management. Product managers rely on data for everything from creating an easy buying experience to determining if a product should even be developed. And while data is used to inform all aspects of managing a product, are there different considerations for using data for long-term strategy vs. daily decisions? 

It turns out that the answer is yes. While there are many similarities, like needing to ensure success is clearly defined, for strategic decisions more context, expertise, and qualitative data is required. Data used for strategic decisions tends to be higher quality, take longer to collect, and be more expensive than data used for everyday decisions. Data mistakes, such as bias or incorrect benchmarking, are also more costly to the organization when they happen in the context of creating a strategy.

Even if a product manager has the best data to inform strategy, without an efficient process to utilize it in the organization, it will be worthless. To ensure data is guiding the strategy properly, getting agreement on goals, building the proper infrastructure, and utilizing data governance are keys to making sound long-term decisions. 

For more information on this topic, we asked a group of seasoned product managers the following questions:

  • What types of data are most useful to effectively inform long-term product strategy, key priorities, and planning? 
  • What mistakes do companies and teams make when using data to inform product strategy? How can they be avoided?

Many thanks for these product management experts for their advice:

→ Test your product management and data skills with this free Growth Skills Assessment Test.

→ Learn data-driven product management in Simulator by GoPractice.

→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.

Q. What types of data are most useful to effectively inform long-term product strategy, key priorities, and planning? 

To inform long-term product strategies, key priorities, and planning, both quantitative and qualitative data are needed. 

Just data is not enough, however; the data must be accompanied by brainstorming sessions and team prioritization to build the strategy. Product managers must also thoroughly understand the goals of their organizations because this determines the type of data used. The type of data that is most useful will also highly depend on what industry, product, and even feature is being considered.

The kind of data most useful also depends on what phase the product is in, though both quantitative and qualitative are useful. In the early stages, product managers need data to show in which market segments they can win. Most data will come from external sources like customer research, competitive analysis, and industry characteristics. As the product progresses, a mix of external and internal data to define a long-term product strategy should be used. 

Examples of data used to inform product strategy according to our experts are:

  • Buyer behavior research
  • Competitive insights
  • Customer interviews and surveys
  • Financial data 
  • Google Analytics reports
  • Google Trends
  • Industry metrics
  • Market share
  • NPS data
  • OKR movement
  • Regulations
  • Sales targets

Mehmet Yalcin (Senior Product Manager at Amazon)

The type of data that drives product strategy in the early phases of the product would be generated by external sources such as customer research (qualitative and quantitative), competitor landscape, and macro factors (the maturity of the industry, regulations, and funding). 

It would require a mix of external and internal data to define a long-term product strategy if the product is at its growth or maturity phase. 

On the other hand, internal data would become the driving source in the later stages. You can’t take a step back or go forward without data analysis.

I was leading the consumer security product portfolio for Vodafone Group and my direction was to launch a cybersecurity service for families. The idea was simple, expanding the existing cybersecurity services to a new customer segment with a little configurational adaptation. I was only able to understand the real needs and pain points of families after completing customer research. Parents were not concerned about viruses, but they were concerned about the content that their children might see and the amount of time they spent online. I needed to pivot the original idea in order to solve a real customer problem.

Koye Sodipo (Senior Product Manager at Microsoft)

A key question as part of a strategy is where to play (i.e. what markets, industries, verticals, stages of supply chain, etc.) and how to win in those chosen market segments. If you have an existing product, paying close attention to where users spend the most time is a very good indicator of what ideas and features to invest in.

If you’re scoping out a market, it’s unlikely you’d have hard, quantitative data. However, industry reports do show market share and trends. You can also check Google Trends to see the important trends people search for within a particular geography or timeframe. Conducting user surveys is also a great way to get accurate, actionable data on a topic.

Kimberly Komal Agarwal (Senior Product Manager at LinkedIn)

Planning your long-term strategy is best informed by a combination of qualitative and quantitative data. 

For example, when I was planning our long-term strategy for the TikTok Creative Exchange marketplace, we dove into swaths of data, ranging from qualitative (customer interviews, feedback from sales, support tickets, etc.) as well as quantitative (funnel data, ad performance, etc.). We also took a close look at the strengths and weaknesses of our competitors and workarounds for the ad creation process, essentially putting together a SWOT analysis. 

Tying all of this research together required a team effort. We conducted several intensive brainstorming sessions with all cross-functional teams to build alignment around what we learned so far and what opportunities lie ahead of us. 

From these brainstorming sessions, the product team was able to prioritize and define which changes we needed to make to the product over the next 1–2 years. With the understanding that things can change very rapidly, the team then goes back and revises this long-term strategy on a regular basis. 

Ankit Chaudhary (Senior Product Manager at Indeed.com)

There are different types of data which we can use to understand user behavior for strategy; the data is both quantitative and qualitative in nature. A few examples are:

OKR movement

Understanding the movement against the OKRs provides a good sense of how the company and product are performing against expectations. 

Market share

We also keep tabs on things like external market share to understand how we are faring in the outside world. It is good to refer to industry accepted benchmarks. 

Competitive insights

It’s good to do brand perception surveys and understand how customers feel about your product and what are the key product gaps and areas where the product is better than the competition. 

NPS data

Further, it is good to continuously measure things like CSAT (customer satisfaction) and NPS (Net Promoter Score) for existing customers to get a sense of how the customers feel about the product and the broken things which need to be addressed. 

Sujeet Mathew Jose (Senior Product Manager at Zalando SE)

A strong product strategy (followed by key priorities and execution planning) should provide a strong value proposition to customers. This strategy should be cognizant of where the market is moving and should also tie to the business goals of the company. To ensure that the long term strategy is actually doing this, we need to understand the our target customer, their problems/opportunities, competitor trends, market trends and business impact of the strategies and the priorities. This calls for a 360 degree examination of the current situation and target situation requiring user research (quantitative & qualitative), market trends (expert interviews and desk research), competitors tracking, etc.

In my current position as the senior product manager of the online beauty shopping experience in Zalando in the European market I have to define the strategy of the digital beauty experience. We used inputs such as (a) qualitative and quantitative market research to identify the target customer segment (b) competitor benchmarking studies, user research and Google analytics data to identify key customer problems and (c) financial metrics, order economic data to identify business potential of each opportunity area. Using these inputs, we arrived on the product strategy and roadmap for the next couple of years.

Omar H. Sallam (Senior Product Manager at Booking.com)

To determine the priorities and product needs that dictate a product strategy, you always need to focus on data and metrics that are measuring impact relevant to direct user experiences or strategic targets. These metrics are unique to every feature, product or industry. You also need to make sure that the metrics you pick are relevant and specific to the action you are trying to measure. 

For Booking.com, most of the technical or business metrics have to be directly or indirectly tied through a proxy to the key metric that the whole business is striving for which is “booking conversions per day.” 

In my previous role at Delivery Hero, working within the user support scope meant that most of our metrics were related to the customer satisfaction index “CSAT” and several other customer service performance metrics.

Eric Lippincott (Senior Product Manager at Expedia Group)

Competitive research is one of the most valuable datasets you can utilize, but it’s secondary. Primary research and data like user surveys will always be the most valuable for strategy and planning.

Eric Chen (Senior Product Manager at Yahoo, Platform)

While a portion of my time each week is spent on optimizing marketing performance week on week, I am constantly listening to my internal customers to better understand how our business units are evolving. As business units make key decisions around resourcing, opportunities ebb and flow. These opportunities can really only be taken advantage of if the platform engineering/integration teams have the capacity to develop the necessary capabilities. In summary, I am constantly surveying the information landscape to understand my internal customers’ pain points. This helps align my efforts towards maximizing our platform’s current and future capabilities for the firm.

The platform I manage is just one solution in the field that Yahoo could use. I must be constantly aware of capabilities that other firms (outside of Yahoo) and other teams (inside of Yahoo) are enabled for. This research and anecdotes are funneled back to our platform product team so we can have candid conversations internally and throughout the firm that ultimately help our firm to compete in the global market.

Q. What mistakes do companies and teams make when using data to inform product strategy? How can they be avoided?

Probably one of the biggest mistakes in terms of using data to inform strategy is not defining what success is, which in turn causes teams not to know what to measure. 

The finest data in the world cannot make up for a team that hasn’t agreed on their goals. Concepts like OKRs or North Star metrics can be used to guide all of the teams in an organization to success. Relying on vanity metrics, which are simple measurements like page views, rather than on metrics that measure actual user engagement can cause problems for product managers. 

And while not knowing what product success looks like is a recipe for disaster, confirmation bias, changing goals or metrics to support a hypothesis, and comparing against competitors incorrectly can skew data analysis and harm the product. Product managers need to sanity check data analysis with qualitative data from customer interviews or surveys. Product managers also need to rely on their expertise if metrics are not making sense or if external factors can impact the numbers.

On the operational side, neglecting data requirements in the product will result in the ability to collect data or result in a high number of errors in data collection. Not taking data governance seriously, where there is a person or team in charge of keeping the data in order and understanding what it means, results in skewed data and unreliable results.

In summary, these are top mistakes our experts have seen when using data to inform high-level decisions:

  • Not defining success
  • Vanity metrics
  • Confirmation bias
  • Benchmarking incorrectly
  • Changing goals too soon
  • Not validating results with qualitative data
  • Solely relying on data and disregarding expertise
  • Failing to prioritize data governance
  • Neglecting data requirements

Kimberly Komal Agarwal (Senior Product Manager at LinkedIn)

One mistake I have noticed is that product teams select vanity metrics (especially in the early stages) to show product success (e.g. followers, page views, downloads). 

These metrics might help product teams look good for a short time. However, teams can very quickly start facing problems as soon as they try to use those vanity metrics to inform their long-term strategy. 

A good rule of thumb when evaluating metrics is to ask a few questions:

  • Is this result repeatable (do I know what to do to make this happen again)? 
  • Is this result actionable (do I know what to do next)? 
  • Is this result informative (does it tell me something critical about my business)?

If you can’t say yes to each of these three questions, you’re probably dealing with a vanity metric. Instead, I would encourage teams to choose metrics that show some level of product engagement, such as monthly active users, since these metrics can help you gauge product health and satisfaction as they fluctuate over time.

Ankit Chaudhary (Senior Product Manager at Indeed.com)

The biggest mistake is when there are no OKRs at the company level and different teams end up going in different directions. 

Further, if the OKRs are not well defined and well balanced, e.g. teams just going after customer acquisition or monetization and not caring for customer experience or retention, it may lead to an imbalance in the product strategy and flawed execution. 

Eric Chen (Senior Product Manager at Yahoo, Platform)

When working with data to inform product strategy, I see two categories of mistakes:

Interpretation bias

Interrogate and surface biases you may have about your product that might influence the way you use and share data when it comes to long-term product strategy decisions. What may be good for the business ultimately may not be best for your product or team. While that may feel uncomfortable, consider that one of your roles is to help your firm achieve its goals. There is merit in the exercise of surfacing objective data for the sake of the business for higher level leadership decisions.

External vs. internal impact

Invariably there will be externalities that affect your product’s performance. When interpreting data, it is important to keep in mind if the end use of your data needs to consider external forces. For example our platform’s performance will be affected by the market conditions of consumer media consumption. For temporal reasons, we may need to factor out market conditions in order to get a more objective sense of our platform’s performance. Consider who your audience is and what the goal of your conversation will be.

Sujeet Mathew Jose (Senior Product Manager at Zalando SE)

Confirmation bias 

I think the biggest mistake that companies make while using data is confirmation bias. They have a hypothesis in mind and use the data to prove that hypothesis. For example if the customer acquisition power of a strategic direction is less, some product teams will cite the customer retention or NPS of the feature as a proof point for moving into that direction. This can be avoided by moving stage-wise. First we start with the data and check what insights it throws out. Based on the data insights, we should identify problems and then move into the solutions.

Changing goal posts

Many product teams are ready to easily change the goal posts when it comes to success metrics. In this case, when the primary success metrics do not show enough time, they communicate that the initial metrics, or benchmarks or time frames were incorrectly selected. This is due to the immense pressure on product managers in certain organizations to prove that their decision was the right one. We can avoid this by having a strong verification process of the success metrics selected before the development of the product, having a strong steering mechanism.

Benchmarking incorrectly

In many cases, product teams benchmark against non-comparable competitors. Any competitor can have a different customer base, business model, supply, or user experience. We cannot expect to match the competitors on key metrics for any feature. The way to avoid this is by identifying the key competitors upfront and then looking at the ways that we are similar or different. Based on this, we can then set benchmarks.

Koye Sodipo (Senior Product Manager at Microsoft)

One common mistake I see is asking what customers will do in surveys. It’s much more reliable to ask them what they have done in the past as past behavior is a stronger indicator of future behavior.

Another thing to be cautious about is data skewing. I once was at a hackathon where we tried to build a model to predict who would cancel a hotel reservation. The issue is, only 1% of people cancel reservations. So our models, which are tuned to maximize for accuracy, began predicting that nobody at all will cancel. Unsurprisingly, the model was 99% accurate. Did we ever need such a naïve model at all?

Lastly, data isn’t the end-all-be-all. The data that was right yesterday may not be right, or even actionable, tomorrow. Product strategy should involve a more holistic approach including data fundamentals, human experiences and first principles.

Mehmet Yalcin (Senior Product Manager at Amazon)

One fundamental problem is neglecting data requirements. This usually leads to complete data work towards the end of your delivery which creates a high margin for errors. Data requirements (triggering event, collection, storage, usage, removal) should be part of the product team’s routine meetings and should be considered as important as the customer experience.

Eric Lippincott (Senior Product Manager at Expedia Group)

As a product manager, your charge is to know your product and customers better than anyone.

This means that there will be times that your intuition does not fully align with the data you see. I’ve seen data before and thought, “I understand the customer journey very well, and I understand the job to be done, and something with this data doesn’t seem right. We need to interview some customers and dig deeper.”

We’d like to thank Stephanie Walter for incredible help in creating this article.