Category posts
Tests and experiments

Learn how to maximize the efficiency of your A/B tests and experiments. How to avoid costly mistakes while designing and analyzing the results?
Author:
Oleg Ya
Experiments where you make your product worse – the most underrated product manager tool

Let’s start with a practical task.

Say a company’s management wants to allocate significant resources to the development of infrastructure that would increase their app’s speed. The hypothesis is that increasing the speed of the app will have a positive effect on the user experience and the key metrics.

Think of an experiment (an A/B test) to validate this hypothesis.

Test your product management and data skills with this free Growth Skills Assessment Test.

Learn data-driven product management in Simulator by GoPractice.

Experiments where you make your product worse – the most underrated product manager tool
(more…)
Author:
Oleg Ya
Why your A/B tests take longer than they should

When conducting experiments, teams usually include all the active users in their tests, or sometimes they tend to add all the new users who join the app during this test. So when calculating the metrics for different test groups, all the data from the moment the A/B test kicks off is taken into account.

Today I’ll talk about how you can reduce the time required to get the signal on the change you are testing in a product. You can do that by changing the process of adding users to the A/B test, and in this essay I will show you how you can do it.

Test your product management and data skills with this free Growth Skills Assessment Test.

Learn data-driven product management in Simulator by GoPractice.

Why your A/B tests take longer than they should
(more…)
Author:
Oleg Ya
Peeking problem – the fatal mistake in A/B testing and experimentation

You can make many mistakes while designing, running, and analyzing A/B tests, but one of them is outstandingly tricky. Called the “peeking problem,” this mistake is a side effect of checking the results and taking action before the A/B test is over.

An interesting thing about the peeking problem is that even masters of A/B testing (those who have learned to check if the observed difference is statistically significant or not) still make this mistake.

Test your product management and data skills with this free Growth Skills Assessment Test.

Learn data-driven product management in Simulator by GoPractice.

Peeking problem – the fatal mistake in A/B testing and experimentation
(more…)
Author:
Oleg Ya
Mistakes in A/B testing: guide to failing the right way

Failing fast and often will help you learn from your mistakes sooner rather than later. This is an advice you hear often from successful product managers. But what you hear less often is that not every failure is a successful learning experience.

A product’s success is largely dependent on coming up with a hypothesis and designing the right tests. Without those elements, you might draw the wrong conclusions and steer your project in the wrong direction.

In his guest post for the GoPractice blog, Ethan Garr, VP of product at TelTech.co, shares some hard-earned experience in product testing. Through concrete case studies, Ethan shows us how to avoid key pitfalls when designing tests for hypotheses.

Test your product management and data skills with this free Growth Skills Assessment Test.

Learn data-driven product management in Simulator by GoPractice.

Mistakes in A/B testing: guide to failing the right way
(more…)
Author:
Editorial
Designing product experiments: template and examples

Teams that don’t use experiments usually think they know their product, its users, and what they should do to achieve their intended results. In contrast, teams that use experiments acknowledge that they know very little about their product and users. This way of thinking presents the team with a unique opportunity to improve. Our former student Anton Rifco dived deeper into the topic to tell our readers more about it.

Test your product management and data skills with this free Growth Skills Assessment Test.

Learn data-driven product management in Simulator by GoPractice.

Designing product experiments: template and examples
(more…)