A quantitative data-driven culture is generally seen as very positive. It’s great for understanding performance and tweaking strategy. Quantitative data is useful for reducing risk and can be used for decisions that need a high degree of certainty. These situations are usually high enough of a priority that adequate time is allotted to use data to verify various experiments before the decision is made. 

But there can be very real downsides. Data alone isn’t particularly useful for big transformations or concepts that don’t exist in the market. Henry Ford, founder of Ford Motor Company, famously said, “If I had asked my customers what they wanted they would have said a faster horse.” Ford’s customers actually wanted a faster way to travel, but lacked the language and vision to describe an automobile. Ford had to make the leap from “faster horse” to “car.” Data cannot do that.

The cost of using data can also outweigh the benefits. Acquiring and preparing data, making sure it’s reliable and accurate, and making it available to the team all requires effort. Performing analysis on the data also takes time and skill, which can delay decisions. These downsides must be carefully weighed as an organization decides how, when, and why to use quantitative data.

Previously we wrote about both the pros and cons of a data-driven culture, which we recommend as a good introduction to this topic. In this follow up, we dig deeper into the potential risks so product managers can be fully aware of the disadvantages. We talked with product management experts in the field and asked these questions:

  • To what extent should data-driven companies rely on data? What perils for the product come with a data-driven approach?
  • Are there situations when it’s better not to use data while making a decision?
  • What alternatives are there to data for insights on product strategy or customer behaviour? Where can you find deeper answers that data cannot provide?
  • Can you provide an anecdote or example when breaking the rule of ‘being data-driven’ was good for a company, product or individual?

Many thanks to these product managers for their insights:

  • Igor Akimov (Head of AI Solutions at Wrike)
  • Edward Castaño (Principal Product Manager at LinkedIn) 
  • Elaine Chao (Senior Product Manager at Adobe – Creative Cloud)
  • Jon Linch (Senior Product Manager, Technical – Alexa Shopping at Amazon)
  • Joel Polanco (Manager of Customer Experience – Health, Education, and Consumer Verticals at Intel Corporation)
  • Kumar Samanvaya Misra (Product Manager, Battery Recycling Platform at BASF)

→ Test your product management and data skills with this free Growth Skills Assessment Test.

→ Learn data-driven product management in Simulator by GoPractice.

→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.

To what extent should data-driven companies rely on data? What perils for the product come with a data-driven approach?

While companies rely on data to make sound business decisions, there are limitations. A quantitative data-driven approach works well to understand an organization’s performance trends and make adjustments to strategy. It can also be used to reduce risk, especially when making an irreversible decision. Mature companies with a large volume of data, especially with B2C products, can use data to verify experiments to produce the best outcomes.

But having too much data can be as problematic as not having enough. Teams can get overwhelmed analyzing large data sets which leads to analysis paralysis. The analysis can take such a long time that decisions are delayed, causing missed deadlines or opportunities. Teams can also get sloppy and fail to properly set up experiments or forgo testing the results.

Data also can’t create a strategy. Things like improving customer value and increasing retention need customer interviews, expert opinions, and innovation. There also might not be enough data on which to base a decision, or the data may be coming from unreliable sources. Even if the data is reliable, there’s a risk of misinterpreting the results, measuring the wrong thing, or confusing correlation with causation. And data use should always be ethical. Violating user privacy to collect better measurements or get positive results is wrong.

Igor Akimov (Head of AI Solutions at Wrike)

A data-driven approach works well at the stage of attracting and activating users, but in terms of customer value, increasing retention and strategic breakthroughs, you can’t make an impact without customer interviews, observations, communication with experts, analysis and faith. This is especially true for AI functionality, the capabilities of which are often either underestimated or very overestimated. It’s very hard to use data to confirm the need for complex functionality, such as AI Recommended Tasks, which works in Wrike. This need was identified in conversations with company and team leaders and only after several iterations of feedback the feature was released to anyone, receiving excellent feedback.

Edward Castaño (Principal Product Manager at LinkedIn) 

In a rapidly moving and dynamic marketplace, anchoring solely on quantitative data can be a recipe for analysis paralysis. User research, market research, voice of the field, qualitative data, and quantitative data are all valid sources of insights to deliver delightful customer experiences. Savvy companies draw insights from all available sources. 

Experienced product managers consider the value of a data-driven approach relative to the cost. A culture of slow decision-making and false precision develops when companies are over reliant on quantitative data. The best companies recognize that every decision is about tradeoffs and sometimes it’s best to acknowledge the limitations of data-driven decision making and leverage the other tools in their arsenal. 

Elaine Chao (Senior Product Manager at Adobe – Creative Cloud)

Making decisions with data can be fraught with peril. Some of the challenges come from looking at data without context, misinterpreting or drawing conclusions from data that mix up correlation with causation, and treating data as a number to be driven instead of seeing it as a measurement of the choices you make as a product team. 

Another common misstep is measuring the wrong thing to gauge success. A few years ago, I had a workflow that was reported through the initial action, but this top-level metric didn’t represent the ongoing nature of utilization. This led our leaders to believe that our feature was being underutilized. I partnered with one of our data scientists to ensure that we instead reported on a metric that more accurately represented the ongoing pattern of use, which helped us understand the true level of engagement our customers had with this feature. 

Jon Linch (Senior Product Manager, Technical – Alexa Shopping at Amazon)

Data is a critical element to making informed decisions, but collecting and analyzing data  comes with an opportunity cost. When the cost of obtaining and analyzing additional data outweighs the benefit, this is the point at which more data exhibits diminishing returns. Some metrics are more expensive to measure while others are so easy to measure that they become distractions when executing on the north star product vision.  

When using  data, it’s important to first define your product’s goals and then select metrics and counter-metrics that measure the progress towards those goals. Having too much data can be just as challenging as having too little data. Sometimes metrics synthesized from large datasets can be overly complex, while other times the plethora of data can lead to analysis paralysis where the team spends too much time analyzing and doesn’t execute in a timely manner. This results in missed opportunities.

Joel Polanco (Manager of Customer Experience – Health, Education, and Consumer Verticals at Intel Corporation)

If you are a data-driven company and executing your data strategy, meaning you’ve set up a strong data foundation (strong data sources, data definitions, and databases), you should increasingly rely on your data to make decisions. Perils? There are many. The one that hits home for me is despite having a strong data foundation, a team will fail to properly set up their experiments. In fact, some teams will make observations and claim success without even having tested! This is the equivalent of shooting an arrow and then drawing the bullseye around it after the fact. Data-driven does not equal data-informed.

Kumar Samanvaya Misra (Product Manager, Battery Recycling Platform at BASF)

It depends on what the company is trying to achieve. Excess of anything is bad. In an ideal world, mature companies should understand where to draw the line. Moreover, sometimes doing things without questioning the appropriateness of the process or technology is ineffective. In my opinion, data can be utilized to:

  • Minimize the risk of making an irreversible decision
  • Prevent negative outcomes from happening
  • Understand current business trends to adjust the overall company strategy

Things become counterproductive when data is used to exploit grey areas. Not respecting privacy to reach the most appropriate target audience for a marketing campaign is an example.

Are there situations when it’s better not to use data while making a decision?

While some argue that it’s always better to use data, there can be exceptions. You may not have enough data to be able to make a decision; this is typically the case in early stage startups or projects. There just isn’t enough data collected. If you do have data volume, your data and its sources must be reliable. If your data is not trusted, you should not use it. Context is also important because numbers alone don’t explain user behavior. Data collected on past behaviors cannot identify the big bets or strategy needed to grow your business. And there is a limit to what data can identify. Sometimes it’s not worth the expense to acquire more data and more analysis.

Timing and priority are also considerations when using data. When a decision needs to be made quickly and either the stakes are low or the decision can be rolled back easily, a data-driven approach is not a good choice. Experimentation may be needed to get the data to make the optimal final decision.

To summarize, in these situations it’s better not to rely solely on data for decision-making:

  • Not enough data
  • Unreliable or untrustworthy data
  • Explaining customer behavior
  • Developing big bets, vision, or strategy
  • The cost of further data capture and analysis outweigh the benefit of the results
  • Urgency is high and stakes are low
  • Decision can be rolled back easily
  • Experimentation is needed to make the decision

Igor Akimov (Head of AI Solutions at Wrike)

First, you shouldn’t use data when you don’t trust it. Your entire data collection and analytics system must be as reliable as clockwork. Second, I think the system of evaluating users’ opinions in the form of a survey is often flawed and brings either pre-packaged answers or even more uncertainty. Third, users can get used to certain behaviors, even broken ones, and until you change the behavior, you won’t get appropriate feedback. 

An example is Wrike’s automatic creation of subtasks from text based on artificial intelligence (AI Subtask Creation). The users never said anything about it in interviews, there were no customer requests or mention in UMUX-Lite or NPS surveys, but we saw a significant waste of human time on UX studies. We identified a fix with the help of NLP (Natural Language Processing). When we released the feature, we received hundreds of positive reviews. All of this can only be revealed by synthesizing all the information from different sources, and this is art, not science.

Edward Castaño (Principal Product Manager at LinkedIn) 

A data-driven approach works when the urgency is low and the stakes are high. Purely data-driven decision-making is ideal in a space where data is plentiful, time pressures are modest, and the consequences of getting the decision wrong are high. Think rocket science. 

Correspondingly, a data-driven approach fails when the urgency is high and the stakes are low. All other scenarios should optimize for a balance between data-driven decision making and user-informed decision-making driven by qualitative insights, experience and intuition. 

Elaine Chao (Senior Product Manager at Adobe – Creative Cloud)

Because data reflects what is, it’s easy to identify levers that might push toward incremental growth. However, it’s much more difficult to identify the big bets that can truly revolutionize a product or industry. Seeing opportunities requires a deeper understanding of your customer than just data will provide alone, which is why I say that data informs business strategy, but shouldn’t control it.

Jon Linch (Senior Product Manager, Technical – Alexa Shopping at Amazon)

Product managers need to identify one-way and two-way door decisions. One-way door decisions are those that can’t be undone and two-way door decisions can easily be rolled back. One-way door decisions always require data and rigorous analysis. Good product managers know how to turn one-way door decisions into two-way door decisions to limit risk and be more nimble when possible. In situations where speed is of the essence, and decisions can be rolled back quickly and easily, the team can launch without rigorous analysis and iterate.

Joel Polanco (Manager of Customer Experience – Health, Education, and Consumer Verticals at Intel Corporation)

While I love data, it can be very misleading. At one point, we couldn’t understand why a key product category was dropping like a rock despite having years of data on its sales. We came to find that we were being disrupted by a new product category which had launched a few months earlier. In these situations, you will want to collect data, but that data will likely be messy, unstructured, and qualitative. A great source of information in this case would be your sales team; don’t forget to reach out to them from time to time.

Kumar Samanvaya Misra (Product Manager, Battery Recycling Platform at BASF)

Data is largely helpful in detecting a pattern and changes in that pattern. But it does not tell you what causes them in the first place. I can think of situations where a data-driven approach can be a bit of over engineering like:

  • You are a small business and you do not have enough data. When you are a small business there are two things which are missing: velocity and veracity of data. This combines with a growth mindset to overcomplicate things. 
  • Using a data-driven approach to replace business sense while growing a company. By its very nature, a data-driven approach performs best in cases of optimization-type problems. They may be good at identifying anomalies but not that great at highlighting the root cause behind the anomalies. This is where people need to rely on business sense and intellect.
  • Understanding everything about your data. During my days as a co-founder of Replique, I was trying to find explanations for all of our data. But I soon realized it’s a rabbit hole; the more you want to answer the data the more you need to implement data collection points and data governance. And for an entrepreneurial venture with pressure to generate revenue to raise the first round of capital, it starts to distract you from the core business. 

What alternatives are there to data for insights on product strategy or customer behaviour? Where can you find deeper answers that data cannot provide?

If quantitative data is not available or unusable, a product manager still has options. A lot of insights can be gained from qualitative techniques, experience, research, and intuition. The practice of design thinking can help provide qualitative insights using techniques such as interviews, usability studies, focus groups, and user action observations. 

A product manager can also generate needed data through customer interviews and surveys. Secondary research, such as competitive analysis and cohort analysis are also useful tools. An industry analysis is useful to compare similar products and customer expectations of features, purchasing, price points, and more. But remember that even if something can’t be measured, there is value in determining what should be measured if the data existed. This stimulates team brainstorming to assess risks and mitigations.

Igor Akimov (Head of AI Solutions at Wrike)

Qualitative research methods from the social sciences have been developing for centuries and in my opinion have been undeservedly neglected in favor of metrics and KPIs. That said, for early-stage startups that have not yet fully defined the problem they are solving, user interviews, expert interviews, usability studies, focus groups, observations of users’ actions (for example, with a web viewer or analytics), and other research methods help answer the main question of WHY users do things this way. A good collection of such tools can be found in the Design Thinking Methodology.

Edward Castaño (Principal Product Manager at LinkedIn) 

A user-informed approach driven by qualitative insights, experience and intuition are antidotes to long measurement cycles, slow decision-making, and false precision. Start by acknowledging the tradeoffs between speed, precision, and timely decision-making. Agree with stakeholders to a timeline for the decision and use that to inform the balance between a data-informed and user-informed decision making. 

I once had to evaluate thousands of calls to the support center during a critical platformation migration. Unfortunately, all of the call data was completely unstructured. I used excel to process the raw data into actionable insights using keywords to identify themes as well as positive or negative sentiment. I was able to distill thousands of conversations into insights that helped build confidence in our approach and drive quick decision-making. 

Lean on your experience to build confidence. While you may not have the exact data to prove this is the right decision, you can leverage your experience on a previous project to build a strong case for making a decision. Data can lie. Intuition is sometimes just a gut feel, but it’s usually based on core assumptions, strong beliefs, and sound logic. Being explicit about these assumptions with the group can help speed decisions in a dearth of data. 

Elaine Chao (Senior Product Manager at Adobe – Creative Cloud)

Customer insight can come from many different sources and can bring a more nuanced understanding of customer behavior. Customer interviews provide insight into motivations for specific behaviors that might seem confusing when looking at data itself. However, two additional tools can be used. One of them is competitive analysis, where you look at the landscape of competitors in your specific market to help understand the addressable market. An honest evaluation of your competitors can provide insight into your customers’ needs. The second tool is cohort or anthropological analysis, which is a type of analysis that involves deep customer interviews about customer behaviors and relationships. These often surface power structures, level of trust, and the underlying customer intent, focusing more on the context of the user. These insights can then be used to find approaches that are sensitive to the user’s core needs, which in turn make a more appealing product. 

Jon Linch (Senior Product Manager, Technical – Alexa Shopping at Amazon)

As a product manager, you represent the voice of the customer. This means that you should know your product inside and out. Perform a bug bash with your team and find problems or potential areas of concern if data doesn’t exist. Engage in an in-depth industry analysis to see how similar products have evolved over time and what customers expect in terms of features, performance, the product life cycle, price points, and more. Look at competing and/or substitute products to try to leverage comparative insights from the market. 

Alternatives to data might also include a trial and error approach in tandem with conducting behavioral interviews with your UX research team. Estimates supported by a margin of error can help when making decisions on product strategy and customer behavior. Even if you can’t measure something, simply knowing what you want to measure can help the team brainstorm potential risks and mitigation steps if estimates turn out to be incorrect.

Joel Polanco (Manager of Customer Experience – Health, Education, and Consumer Verticals at Intel Corporation)

To build a product strategy and understand customer behavior you have to be curious and have a growth mindset. I personally am a huge fan of triangulating data from multiple sources: attitudinal (surveys), behavioral (web, product), and transactional (sales). Sometimes those sources are not good enough and you just have to get out there and watch your customers use your product, especially in tech and especially when you are building software! In lean, the idea of “gemba” (Japanese term for “the actual place”) is that problems are visible and the best way to discover them is through observation.

Kumar Samanvaya Misra (Product Manager, Battery Recycling Platform at BASF)

It would be great to reach a state where we have all the data to get answers to all business questions. However most of the time the data is incomplete or non-conclusive. In case of low data my approach is to seek the support of the data to identify the most probable next steps and then try to validate and prioritize among them. The interesting question is, “What can we do when the data is almost non-existent?” Businesses were still running smoothly when data in its current form (e.g. CRM or analytics) was not available. In such cases we can:

Generate Data

Customer Interviews
If we know the customers, it is best to approach them in the absence of data. It is important to focus on the jobs that they are trying to get done. Think backwards if the product strategy makes sense to deliver the outcomes the customers are looking for.

Surveys
Surveys are amazing for a 0-to-1 product. When we don’t have the data and we know little about the customers, surveys are a very good mode of making informed decisions in the context of product strategy.

Observe

User behavior
Observing user behavior is the primary goal of user analytics data. However, when we don’t have that data it’s best to empathize with the user and understand their pain points to discover a compelling product problem.

Field research
Delve into market research and secondary research to generate key hypotheses to have a starting point to solve the customer problem.

Can you provide an anecdote or example when breaking the rule of ‘being data-driven’ was good for a company, product, or individual?

Success won’t be achieved if the goal of an organization is simply to be data-driven rather than to do what is best for the customer. Remember that the company’s vision is integral to its success, and it is not a data-driven concept. Data cannot tell an organization what its values and culture should be, and it cannot perfectly predict the future.

Relying purely on data for radical changes does not work well. Someone with a vision and the ability to interpret the data is needed to generate the idea for a market disruptor. Most people didn’t know they wanted an iPhone until they saw it. It took Steve Jobs’ deep understanding of his market, human psychology, and technology to create truly unique and desirable products.

When a company is focused on being data-driven at all costs, it can encourage unethical, or even illegal, behavior. When employees lose focus of doing what’s best for the customer, and are only measured on performance metrics, they are incentivized to prioritize the numbers rather than value. This leads to undesirable products, dissatisfied customers, a lack of trust, and potentially dire consequences for the organization. 

Igor Akimov (Head of AI Solutions at Wrike)

The main example that comes to mind is vanity metrics like number of downloads, number of views, or leads. They do not reflect the quality of those views or users, and they are quite easy to hack by spending more money on advertising or by agreeing to increase them at any cost. This eventually leads to situations like Wells Fargo, where employees opened customer accounts without permission to carry out the business strategy, resulting in fines and erosion of customer trust. So don’t let the numbers cloud your view, check everything through the filter of humanity and common sense, and talk to people more than look at charts and reports. I’m sure you’ll succeed.

Edward Castaño (Principal Product Manager at LinkedIn) 

One of the most important values at LinkedIn is “members first.” We frequently disregard data that tells us something would be good for business if we intuitively know it will not be good for members. In this sense, product managers are stewards of the customer experience. We make decisions, not just for tomorrow, but in service of our long-term vision. A vision statement is the most important and most enduring concept at a company, and yet it is the least data-driven. This should give us a strong cue that being data-driven is subservient to higher-level and largely unquantifiable values.

Elaine Chao (Senior Product Manager at Adobe – Creative Cloud)

Before Adobe’s Creative Cloud was a subscription model, users often had to live without an update for years, which meant that creative collaboration across companies involved a huge amount of overhead. Now, Creative Cloud allows for people to stay up-to-date with the latest and greatest features immediately, lowering the overall friction to cross-organizational workflows. This move wasn’t requested by our users explicitly, but the value to our customers increased as a result of this strategic investment. Fast forward to today, this strategic shift, which was a large business investment at the time, has transformed our business and opened doors for additional products and services that are released year-round, which in the end, serves our customers by allowing for more rapid improvements to our entire portfolio of products.

Jon Linch (Senior Product Manager, Technical – Alexa Shopping at Amazon)

It’s necessary to act swiftly and without data in emergency situations where imminent harm to  users may occur. In some cases, ignoring the data entirely and taking conflicting measures is required. One example might include a search engine that doesn’t recognize self-harm or illegal queries and surfaces inappropriate information to users. While the data may indicate that the search engine is performing optimally, these sensitive edge cases require immediate action which may involve surfacing a helpline rather than self-harm content. 

Another  example might be the inappropriate use of PII (personal identifiable information) to power a feature. The PII data may provide a supercharged and highly personalized experience for users, but if it was collected inappropriately, then immediate data wipeout actions must commence regardless of the performance data.

Joel Polanco (Manager of Customer Experience – Health, Education, and Consumer Verticals at Intel Corporation)

My favorite example is when an individual is deciding what to do next in their career path. When I first started my career I had spreadsheets and formulas where I weighed out my opportunities and offers. What I’ve come to learn is that career paths are messy and sometimes you just have to go with your gut. Despite being a highly data-driven individual, I’ve found that certain forward-looking decisions require some inner reflection and despite what the data is telling you, your gut may just be right. You simply cannot predict what the future has in store so trust your gut from time to time, it was put there for a reason.

Kumar Samanvaya Misra (Product Manager, Battery Recycling Platform at BASF)

Back in the early days of my career, when I was in the advertising technology domain, we used data to optimize the ad creatives. At some point we became so obsessed with that data that we started treating our data as a holy grail. Every new information we generated we would use to optimize our creatives both in terms of design and text. The conversion rate of the ads kept on improving. And when we showed these ads to the customers they were very upset because the creatives looked ugly and they did not follow the brand guidelines. That was the point when I realized that being blindly data-driven is also bad.

My key learnings from the experience were:

  1. It’s important to be clear about the problem you want to solve with the data.
  2. Being data-driven should be just one of the many perspectives in a creative domain.
  3. Next time when you have a lot of data, instead of just taking the data at its face value it’s important to look at the data and understand what questions you can generate by looking at this data.

We’d like to thank Stephanie Walter for incredible help in creating this article.