Quantitative vs qualitative data: what is the difference and when should you use one instead of the other

Product managers use data at the heart of their decision making. There are two types of data that they rely on: quantitative and qualitative. Quantitative data refers to numerical data that can be measured; examples include number of clicks, number of users, and monthly recurring revenue. Qualitative data cannot be counted, but instead describes traits or features such as ease of use, user likes and dislikes, and motivations behind user actions. 

Most companies are collecting huge amounts of quantitative data. Behemoths like Netflix and Amazon harness big data and analytics to drive customer satisfaction and run their businesses. This quantitative data can also be used to generate machine learning models to make predictions about the future. However, combining quantitative data with qualitative data leads to powerful insights that can take your organization to the next level.

Ideally product managers should use a mix of quantitative and qualitative data to get a holistic view of their products and businesses. While some organizations rely more heavily on quantitative data, others utilize qualitative data more. But how does each type of data apply to the day-to-day work of product management? And how do you use them together to get the best results? 

To understand how to optimize the usage of both types of data, we’ve reached out to this expert group of product managers:

• Seema Bansal (Product Manager II, Microsoft Teams)
• Anurag Chitlangia (Payments Product, Uber | Coach & Crew, ProductAcademy)
• Jonathan Kahati (Product Manager, Microsoft Excel)
• Vasiya Krishnan (Senior Product Manager, Azure Data, Microsoft)
• Pranjal Tripathi (Senior Product Manager, Salesforce)

And we asked them the following questions:

• What are the key differences in the ways product managers use quantitative vs. qualitative data? What are the limitations of each type of data?
• What kinds of questions can typically be answered by qualitative data? In which situations is qualitative data especially useful?
• Which type of questions would be better answered by quantitative data? When is quantitative data most helpful?
• When is it best to combine qualitative and quantitative data?

Read through their advice below to learn best practices on quantitative vs. qualitative data in product management. Many thanks to these product managers for sharing their knowledge with us.

→ Test your product management and data skills with this free Growth Skills Assessment Test.

→ Learn data-driven product management in Simulator by GoPractice.

→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.

Q. What are the key differences in the ways product managers use quantitative vs. qualitative data? What are the limitations of each type of data?

Both types of data are needed to make good decisions. Quantitative data is an indicator of successes or issues in a product or business. Quantitative data shows that the conversion rate is underperforming, but can’t say why. Qualitative data can give the reason. And for both types of data, their sources should be kept anonymous, both for privacy reasons and also to avoid bias in your findings.

Quantitative data can come from a data feed and is great at highlighting usage trends. For quantitative data to be useful, it needs a large volume of data and infrastructure. Data deficiency can occur if the sample size is too small and can provide incorrect results. Quantitative data is limited in that it cannot tell you why or how.

Qualitative data uncovers user behavior. It is used to supplement quantitative data or when a large volume of data is unavailable, like when a startup is in its early stages. It’s also important when working on B2B products to understand enterprise customers. Sources include user research, usability testing, and surveys. However, qualitative data is expensive, which means that sample sizes are usually small. Unconscious bias can also seep in, and it takes experience to interpret results that can be unclear and indeterminate.

To summarize, the differences between the two types of data are:

• Quantitative data measures successes or problems; qualitative data indicates why success or problems occur.
• Quantitative data requires a large volume of data; qualitative data usually has a small sample size.
• Quantitative data can be relatively cheap to obtain; qualitative data is expensive to collect.
• Sources of quantitative data include data feeds, product metrics; sources for qualitative data include user research, usability testing, and surveys.
• Quantitative measurements are straightforward and less prone to bias; interpretation of qualitative data is more difficult and bias more prevalent.

Example scenario

Let’s say you’re working on a product called fitWorkout, which is a fitness app that allows users to track different health metrics and stream workouts from their favorite coaches. Quantitative data could tell us what workouts are the most popular, what time of day users do workouts and how often, and how many users have a complete profile set up in the app. What if you notice that only 25% of active app users have set up their profile? That’s where qualitative data comes in. User research could indicate that your users don’t understand the value of setting up a profile, can’t find the profile page easily, or have data privacy concerns. Knowing these reasons can lead to solving the issues in the product.

Vasiya Krishnan (Senior Product Manager, Azure Data, Microsoft)

Quantitative data refers to any information that can be expressed as a numerical value and can be measured. This is what is generally referred to as structured data. Qualitative data is non-statistical. This is semi-structured or unstructured data that you collect in the form of interviews, case studies, focus groups, etc.

Limitations of quantitative data: Quantitative research at times will fail to answer the “why” and “how” questions during the product development phase. Quantitative results can provide incorrect direction when the sample size is skewed. Finally, lack of enough data, known as data deficiency, can result in metrics or findings that aren’t statistically significant.

Limitations of qualitative data: When a product manager tries to record interviews, focus group conversations, or case studies you may run into issues where it doesn’t provide accurate or useful information. Some of the common pitfalls include small sample sizes (niche product segment), unconscious bias, selection bias, extremely controlled environments while collecting feedback, and the quality of the information collected. One should also keep in mind the quality of questions asked during the interviews.

Seema Bansal (Product Manager II, Microsoft Teams)

Product managers use quantitative data as the primary (or sometimes the only) input for the best path forward. Quantitative data could be an incoming data feed from trend research reports or telemetry dashboard. An issue with this type of data is that you may not have telemetry in place to source the type of information most useful to you, like whether a certain feature would be a delight factor for our users.

Pranjal Tripathi (Senior Product Manager, Salesforce)

Since qualitative data is interpretation-based descriptive data, product managers use it to collect the general idea or opinion about a product or feature. Qualitative data is also used to provide generic customer feedback or customer concerns. 

The restriction for qualitative data is that it can be vague, uncertain, and unstructured. It can provide information but it cannot provide any statistics as it’s very difficult to run any statistical analyses or apply any data science techniques such as machine learning. The restriction for quantitative data is that it can provide the wrong results in analysis if the sample size of the data is not statistically significant (i.e. small sample size) or if there is a bias or outlier in the data.

Anurag Chitlangia (Payments Product, Uber | Coach & Crew, ProductAcademy)

Quantitative techniques need a volume of qualified data in order to forecast. Quantitative data also needs infrastructure to be able to collect such a large volume of data. A/B testing is one way to validate decisions by quantitative means.  

Qualitative data is used to ask selective questions to a very small set of users to learn insights. Qualitative data is expensive to collect, but it’s needed when you have a new problem for which it is not easy to get a bulk of data. 

Jonathan Kahati (Product Manager, Microsoft Excel)

For both qualitative and quantitative data, it’s important to keep the data as anonymous as possible. User privacy is extremely important and highly regulated, so you need to make sure you do not expose the data. In addition, anonymity helps you avoid biases and identify user segments by usage patterns and not specific characteristics, which is a more modern way of looking at usage data.

Q. What kinds of questions can typically be answered by qualitative data? In which situations is qualitative data especially useful?

Qualitative data is especially useful to deeply understand customer usage, motivation, and pain points; it can show what people “hire” your product to do. It’s great when you need opinions, descriptions, or suggestions. It can also help you to determine a more accurate customer journey and understand the market, segments, and customer behaviors in a comprehensive way. This helps both the product manager, and product team in general, build user empathy that they can’t get from quantitative data.

Another situation where qualitative data really shines is in the discovery phase or design of a new product or feature. At the beginning of a new product, or even in the beginning stages of a startup, you most likely won’t have massive amounts of quantitative data to work with. Using qualitative data can uncover ideas or suggestions that you wouldn’t be able to find otherwise.

In summary, qualitative data is best for:

• User behavior and JTBD (Jobs to Be Done)
• Customer journeys
• Suggestions, recommendations, or ideas for improvement 
• Market, segments, and competition
• When quantitative data is unavailable 

Example scenario

Let’s say that you introduced a new feature in fitWorkout where users can connect with coaches live and also give feedback during pre-recorded workouts, based on feedback that users want the app to be more interactive. The problem is that less than 10% of users are taking advantage of the new features. The usage data from the app doesn’t tell us why this is happening, so it’s time to do user research. You schedule a few user interviews over video chat, but the responses you get back are all very positive. You must be missing something. You decide to observe users as they actually perform a workout using fitWorkout.

You immediately see that the users are having a difficult time finding a way to connect to a coach because the mechanism doesn’t appear until after the workout has started. After you show the users how to connect, you continue to notice difficulties. It’s not easy for the user to tap on a phone screen while they are exercising, even though they want to. One participant even asks if they can use voice controls instead. It took actually observing the users with the app to figure out the problem, but now you know what you have to fix. This is the beauty of qualitative data.

Jonathan Kahati (Product Manager, Microsoft Excel)

Qualitative data is usually expressed in terms of feelings, rather than numerical values. It tends to be more elaborate than just numbers, and it tries to describe the reasons behind the numerical data. Therefore, the idea is not to measure or count it, but to understand trends or significant contributions to various scenarios.

The purpose of qualitative data is to answer the “why” or “how” questions, to help supply reasoning for the numerical analysis. The data is usually an outcome of open-ended studies or surveys, where participants can express their feelings and explain their actions. This makes qualitative data extremely useful when the numbers are not clear enough, or when the numbers indicate something is happening and you want to know why. Understanding this data can help you understand your customers better, and helps you be extremely customer-driven.

Seema Bansal (Product Manager II, Microsoft Teams)

Qualitative data can be in the form of direct user feedback on the potential new features that the team is planning on the product roadmap. In a situation where the team is investing in building a new innovative feature set or when designing a new product or service offering, they tend to source early adopters’ feedback to learn and iterate early on.

Anurag Chitlangia (Payments Product, Uber | Coach & Crew, ProductAcademy)

One big use case of qualitative data is that it is not about numbers, but real life problems. Getting the data from a direct source, there is no better way to understand customer insights, user pain points, and usability issues. This helps to build empathy in your team for the end user. When your customer service inbound increases by a few basis points after the launch of a feature, your developers will not understand the severity of it as much as when they will learn from qualitative data into real customer problems, in their own language. That’s how one can build a persona of the user. These techniques are needed for the discovery phase, MVP (minimum viable product), and the viability of an idea.

Vasiya Krishnan (Senior Product Manager, Azure Data, Microsoft)

Qualitative research is beneficial during the initial phases of product development. It helps product managers get a temperature check on customers’ emotions and reactions. It plays a key role in understanding the market demographics, customer segments and customer behaviors. Overall, qualitative research helps to capture detailed insights on the customer, market, and competitors.

Pranjal Tripathi (Senior Product Manager, Salesforce)

Qualitative data can typically provide answers to the questions where you need opinions, descriptions, and pain points. Qualitative data is useful in below situations:

• When you need suggestions, recommendations, or ideas for improvement for your product.
• When you are documenting the pain points of a customer during the customer journey of the product.
• When you need someone’s opinion on a feature or product.
• When you are writing requirements or user stories for a feature or product.
• When you are documenting the user journey to create a customer defect.

Q. Which type of questions would be better answered by quantitative data? When is quantitative data most helpful?

Quantitative data is great for identifying patterns and monitoring performance. It can tell you how your product is doing in the market, how users are adopting a new feature, and if your business goals are on track. It can alert you to areas of your product or business that need attention, like specific geographies that are underperforming or what are the top product features according to usage. Quantitative data is perfect for understanding how product changes affect user behavior through A/B testing or cohort analysis

Quantitative data is also useful to understand development progress. It can be used to determine the status of a feature and project when it will be completed. It can also uncover quality issues and improvements in certain features based on defects reported.

Example scenario

In the fitWorkout app scenario, you monitor the performance and usage patterns of the fitness classes. During a KPI performance review, you see that the 20 minute HIIT workout classes have the highest monthly views by far, so you consider adding more. Additionally you see that Coach Jen is the most popular coach so you may want to add more of her classes and also ensure she’s incentivized to keep coaching. You notice that participation in the rowing classes is down 60% over the past three months; you need to investigate why this is happening and possibly de-invest in this type of class. Quantitative data leads us to discovering these patterns and indicators so you can make smart business decisions.

Anurag Chitlangia (Payments Product, Uber | Coach & Crew, ProductAcademy)

Typically your product should have KPIs which are easy to measure. These are the kind of data points that use quantitative data to help make the decision of launching a feature or even rolling back an experiment.

Jonathan Kahati (Product Manager, Microsoft Excel)

Quantitative data can be presented in many ways: the rate of product adoption (a percentage), conversions (a number), or page load speed (a unit of time). For example, in the context of a hotel booking website, quantitative data could be how many customers ordered a certain room.

Quantitative data tries to answer questions like “what,” “how many,” and “how often.” This type of data is frequently used for almost any feature in the product, whether it is measures usage or health. It allows you to see what is happening and make data-driven decisions.

Vasiya Krishnan (Senior Product Manager, Azure Data, Microsoft)

Quantitative data gives a really good understanding of how our product is doing in the market. It helps get (almost unbiased) trustworthy, objective insights which in turn helps identify patterns. Structured quantitative research helps in conducting large scale surveys and gives product managers a clear understanding of product health. At times, it does help in breaking a bigger problem into smaller chunks, for example by targeting specific metrics to improve product health.

Pranjal Tripathi (Senior Product Manager, Salesforce)

Quantitative data can typically provide answers to questions such as :

Product development status

• What is the percentage of completion? 

• How many tasks are completed? 

• How many days are needed to finish this product or feature?
  

Product usage

• What is the usage of the feature or product? 

• What is the top used feature? 

• Who are our top five customers in terms of usage?
  

Customer votes

• How many liked the idea vs. how many disliked the idea? 

• What is the top customer rated request?

Quantitative data is useful in almost all situations. Below are some of the situations in the product development lifecycle where we use quantitative data:

• Stakeholders’ votes for new feature implementation and prioritization (top voted feature)
• Development effort estimation for a feature or product delivery (# of days needed to develop)
• Feature development completion status (% of completion)
• Health monitoring of a product by counting its defect statistics (# of defects by feature)
• Feature improvements like customer defects and customer requests prioritization (top requested feature, top defect to resolve)

Q. When is it best to combine qualitative and quantitative data?

While it would be ideal to base all decisions on both qualitative and quantitative data, given expense and time limitations it’s not feasible to do so. Product managers should reserve the use of both types of data to situations where they’ll get the most benefit. One of these situations is when you are making a “big bet” or large investment in a new product or upgrade. You’ll want to make sure that you have all of the information you can so that the launch will be a success.

Another time you’ll want to use both types of data is when something doesn’t make sense or you don’t know the reason behind the numbers. For example, if customer surveys indicate poor feedback on some features, you may want to run an analysis on feature usage, defects, and customer path completion rates. Or if you are concerned that the quantitative data you’re using has a small sample size or is biased, shifting to quantitative data can help give a more comprehensive understanding.

Example scenario

These situations occur throughout the product life cycle. Coming back to the fitWorkout app, let’s say that quantitative data indicates that 25% of users who log in never try a fitness class. And your quantitative data shows that 85% of users who never try a fitness class do not renew their yearly subscription. But that’s all the quantitative data can tell you. You don’t know why users aren’t participating in classes. To get to the bottom of this problem, you need qualitative data.

Because losing these users means a big hit to your renewals and revenue, you decide to invest resources in user surveys and interviews to understand what is going on. You send the surveys to users who have not participated in a class and interview current and potential users. After a lot of work, you find that the top reasons users don’t try the classes are that they forget about the app once they sign up and they feel unmotivated to exercise. You can take these reasons to the product team and come up with a solution to address these issues. Once you deploy a solution, you can use quantitative data to monitor if you improved the number of users trying a first fitness class, and therefore improve renewal rates. Strategically using both quantitative and qualitative data throughout the product life cycle results in great products.

The table below details common use cases for quantitative data, qualitative data, or both. It’s not an exhaustive list, but a summary of the most frequently used scenarios.

Seema Bansal (Product Manager II, Microsoft Teams)

When a product team is making a larger investment bet (in terms of launching a new product or making significant product upgrades), this calls for comprehensive analysis which comes from the combination of both qualitative data and quantitative data. It’s the combination of all streams of information available to the product team that culminates into defining product strategy.

Jonathan Kahati (Product Manager, Microsoft Excel)

Qualitative and quantitative data provide crucial insights to understanding the answers you are looking for. Therefore, combining them should deliver significant benefits, enabling you to validate results and gain much deeper knowledge of how your product is being used. For example, you can validate a hypothesis, gain an understanding of it (through qualitative research), then widen your scope to get statistical data (quantitative), before testing a solution through further qualitative exploration. This creates a beneficial cycle, helping you move fast and driving constant improvement. Another scenario is when the numbers don’t make sense, or don’t explain themselves. You have to combine the qualitative data to understand the full, holistic picture.

Vasiya Krishnan (Senior Product Manager, Azure Data, Microsoft)

Quantitative and qualitative data have their own strengths and weaknesses. As a product manager, I find combining both is extremely helpful throughout the product life cycle.

Product/feature development phase: I use my qualitative research (focus groups, customer interviews, case studies, competitive research) to formulate/identify areas of investigation/user scenarios. I would then use quantitative methods (product telemetry, experimental results, correlational studies) to further expand on the user scenarios that have been identified, assess user preferences and finally come up with the product roadmap.

Post product/feature launch: Quantitative data plays a key role in this phase as it will give you the complete picture around adoption, activation, retention, referral, and revenue metrics. Post launch, measuring qualitative feedback is like getting your customer’s pulse. I would spend time collecting both internal and external feedback. Internal feedback comes from your marketing and sales teams. External feedback comes from current or future customers through initial reactions (positive or critical feedback).

Pranjal Tripathi (Senior Product Manager, Salesforce)

Ideally, we should combine qualitative and quantitative data for every task. This makes all of our tasks tedious and time consuming. So, we generally combine qualitative and quantitative data only when we need to remove the limitations of either qualitative data or quantitative data. As we know, the limitation of qualitative data is that it is descriptive, vague, and unstructured data which creates difficulty in performing any type of data analysis. And the limitation of quantitative data is it can be biased if there is a small sample size. In any of these above situations, we should combine both qualitative and quantitative data. For example: 

• If we have qualitative data and we believe that we don’t have certain decision points or we want to run additional analysis, we should also look into the quantitative data.
• If we have quantitative data and we have also performed the data analysis but we are concerned about the small sample size or biasing of the data then we should utilize qualitative data.

We’d like to thank Stephanie Walter for incredible help in creating this article.