Teams that don’t use experiments usually think they know their product, its users, and what they should do to achieve their intended results. In contrast, teams that use experiments acknowledge that they know very little about their product and users. This way of thinking presents the team with a unique opportunity to improve. Our alumni Anton Rifco dived deeper into the topic to tell our readers more about it.
Let’s start with the basics.
Product experimentation is a process of testing hypotheses aimed at creating improvements for the product, its users, and the company itself. To identify what works best, the product team designs experiments and measures the impact on key metrics. For example, a team behind a video streaming service can experiment with different types of subscription tiers on a certain market thus testing a hypothesis that the local audience would be willing to pay a premium to access the content without ads.
To experiment, you need a hypothesis. A hypothesis is basically a prediction, an educated guess that promises to improve a product. There are various sources of hypotheses for a product team, including user feedback, data and user activity analysis, competitor and market research, new technologies, and many more.
Product experiments are important because they are the driving force of a product’s success. No product is perfect from the get-go. That’s why product teams constantly iterate in optimizing, improving, and adapting products. And to chart and maintain the right course, teams draw their knowledge from the experiments.
So, how can we design product experiments the right way?
***
Successful product work is very much like scientific discovery: you go from hypotheses to experiments and draw conclusions based on your data. Without experiments, creating products will be like placing trust in unverified (and often false) hypotheses. Your hard work is likely to fail. But with the right experiment designs, you can better understand your users and product, quickly discover faulty logic, correct your path, and steer your team’s efforts in the right direction. Here’s an example that can help better understand the importance of scientific discovery and how it relates to modern product work.
Long ago, people had different ideas about how the Earth worked. Some groups believe that the world hangs on a large tree or mountain. Meanwhile, others think that the Earth is flat and there are three elephants carrying it high (Sometimes, elephants are replaced with snakes or turtles).
Later, the Greek philosophers introduced the concept of geocentric astronomy, this way of thinking paved the way for future enlightenment. But in this model, the Earth is still considered the center of the universe, and the Sun and other planets revolve around it.
In the 16th century, during the Renaissance era, the conception of Earth’s place in the universe changed again. This time, astronomer and mathematician Nicolaus Copernicus placed the Sun at the center of his model of the universe, with the other planets, including Earth, revolving around it. He also proved that the Earth rotates on its axis.
Then came the era of Sir Isaac Newton, with his discovery of the law of universal gravitation, followed by the development of mechanical systems and the formulation of the laws of dynamics. These were discoveries that formed the basis of classical physics. Newton asserted the three principles of the Laws of Motion, outlining the theory originally formed by Johannes Kepler and stating that the universe is infinite.
In 1687, his “Mathematical Principles of Natural Philosophy“ summarised the work of his predecessors and formulated a method for modeling the universe using mathematical analysis.
Understand the Difference between Hypothesis and Fact
When a team develops a new product, their initial ideas about the market and user needs are very similar to the balance of the planet behind the giant turtle: Usually far from reality.
Gradually, the team works and understands more deeply various aspects of users and markets, obtains new information, and then updates its product conception. The process is very similar to how humanity’s perception of Earth evolves, leading to new models.
Just like kids playing with their stuff, they need to try many things at first, and along the way, they accumulate new information. Initially, they make assumptions about how things work, then they get the actual responses as facts.
Knowing the difference between a hypothesis and a fact is one of the most important skills in product management.
If you base your product development activities on unverified hypotheses, then all conclusions and plans are at risk. Unproven hypotheses that are assumed to be facts can be very dangerous, especially if they are fundamental in nature. For example, if you don’t test the value proposition of your product, you’re risking everything you have (money, time, people, reputation) on unproven assumptions.
→ Test your product management and data skills with this free Growth Skills Assessment Test.
→ Learn data-driven product management in Simulator by GoPractice.
→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.
→ Learn to apply generative AI to create products and automate processes in Generative AI for Product Managers – Mini Simulator.
→ Learn AI/ML through practice by completing four projects around the most common AI problems in AI/ML Simulator for Product Managers.
Creating a new product is a very risky activity. That is why the product team must turn all major hypotheses into facts to eliminate major risks as early as possible. If you do it right, your confidence in the product will also continue to increase.
If there are some important hypotheses behind your product, you should turn them into facts before moving on to further development stages. Doing this helps to increase the chances of your product’s success.
Some Patterns of Information that are not Facts
Here are things that are often mistaken for facts:
- Similar feature information from other products. What works for one product may not necessarily work for another.
- Information from a blog article. You may have read articles claiming statements like: “Adding emoji to Push Notifications increases click-thru rate by 53%.” Statements like these sound like facts, but they may not necessarily be true in your case.
- Any statement that is not supported by accurate data
However, if there is information that seems relevant, you can add it to the experiment list. This is the only way to prove the hypothesis.
Experimenting to study a product
Experiments are the only way to turn hypotheses into facts. When you work on a new product, you encounter a whole new world, with unknown rules and formulas.
Maybe you can apply the simple model you have in mind for this product, but it won’t help much. Often, the reality is very different from the original theory. Thus, the only way to learn more is to formulate a hypothesis, test it, and learn from the results.
Designing an Experiment
All experiments always start with a hypothesis. The team formulates assumptions about the theory that is expected to get the desired results.
The next step in this process is to determine the specific method for testing the hypothesis. You need to understand what the experiment will actually run on. Then, you must also select the users to be involved in the experiment. You need to divide users into 2 groups, Test and Control. Users in the Test category will run a series of experiments, while those in the Control category will get the standard features. Make sure that the distribution is random.
Finally, you need to determine the metrics to use in order to measure the impact of the change. After the experiment is over, you’ll also need to find out the detailed causes of how the changes affected those metrics.
In essence, the concept of the hypothesis must be thought out carefully at the beginning. This is useful for estimating the expected effect of an experiment. You also need to determine the expected rate of change for the success threshold. For example, if the change in metric X is less than N%, the experiment is considered a failure and the feature is not worth adding.
The following list of questions can be used as a reference for your experiment design:
- What are your hypotheses? (Explain the theory of how we can achieve the desired result!)
- What features are changed in the product? (Explain how the features differ between the test and control groups!)
- What User Segments will be included in the test?
- What are the main metrics that determine the outcome of the experiment? (determine how you will measure the change)
- What sample size is needed to measure the expected impact?
- Explain the action plan after you get the experimental results! (if the result is X, then we will do Y, if we get A, then we will do B)?
Determining the right number of user samples for an Experiment
In order to see results with the right statistical significance, you need to estimate the sample size needed to test your hypothesis. By knowing the sample size, you can determine the time and resources needed to achieve reliable results. It is very important to make such an estimate. Given the importance of experimental results in the decision-making process, you need to do this calculation carefully
For that, you can use the sample size calculator in this tool.
To estimate the sample size:
- Enter a base conversion rate
(For example 17.4%) - Enter the minimum change effect we want to see
(For example, 17.4% ± 2%) - Set the desired level of confidence.
(For example, let’s say we want to have a 95% confidence level)
Note that in this calculator, instead of the confidence level, we set the same Alpha parameter with the difference between 100% and the confidence level, i.e. 100–95 = 5%
Well, now you understand how important it is to run experiments to better understand the product you are developing.
It’s time to take your product to the next level!
