Category posts
Tests and experiments
Oleg Ya
Let’s start with a practical task.
Say a company’s management wants to allocate significant resources to the development of infrastructure that would increase their app’s speed. The hypothesis is that increasing the speed of the app will have a positive effect on the user experience and the key metrics.
Think of an experiment (an A/B test) to validate this hypothesis.
→ Test your product management and data skills with this free Growth Skills Assessment Test.
→ Learn data-driven product management in Simulator by GoPractice.
→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.
→ Join our discussion on LinkedIn. New topics to talk about every week.

Oleg Ya
When conducting experiments, teams usually include all the active users in their tests, or sometimes they tend to add all the new users who join the app during this test. So when calculating the metrics for different test groups, all the data from the moment the A/B test kicks off is taken into account.
Today I’ll talk about how you can reduce the time required to get the signal on the change you are testing in a product. You can do that by changing the process of adding users to the A/B test, and in this essay I will show you how you can do it.
→ Test your product management and data skills with this free Growth Skills Assessment Test.
→ Learn data-driven product management in Simulator by GoPractice.
→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.
→ Join our discussion on LinkedIn. New topics to talk about every week.

Oleg Ya

You can make many mistakes while designing, running, and analyzing A/B tests, but one of them is outstandingly tricky. Called the “peeking problem,” this mistake is a side effect of checking the results and taking action before the A/B test is over.
An interesting thing about the peeking problem is that even masters of A/B testing (those who have learned to check if the observed difference is statistically significant or not) still make this mistake.
(more…)→ Test your product management and data skills with this free Growth Skills Assessment Test.
→ Learn data-driven product management in Simulator by GoPractice.
→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.
→ Join our discussion on LinkedIn. New topics to talk about every week.
Oleg Ya
Failing fast and often will help you learn from your mistakes sooner rather than later. This is an advice you hear often from successful product managers. But what you hear less often is that not every failure is a successful learning experience.
A product’s success is largely dependent on coming up with a hypothesis and designing the right tests. Without those elements, you might draw the wrong conclusions and steer your project in the wrong direction.
In his guest post for the GoPractice blog, Ethan Garr, VP of product at TelTech.co, shares some hard-earned experience in product testing. Through concrete case studies, Ethan shows us how to avoid key pitfalls when designing tests for hypotheses.
→ Test your product management and data skills with this free Growth Skills Assessment Test.
→ Learn data-driven product management in Simulator by GoPractice.
→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.
→ Join our discussion on LinkedIn. New topics to talk about every week.

Editorial

Teams that don’t use experiments usually think they know their product, its users, and what they should do to achieve their intended results. In contrast, teams that use experiments acknowledge that they know very little about their product and users. This way of thinking presents the team with a unique opportunity to improve. Our alumni Anton Rifco dived deeper into the topic to tell our readers more about it.
(more…)→ Test your product management and data skills with this free Growth Skills Assessment Test.
→ Learn data-driven product management in Simulator by GoPractice.
→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.
→ Join our discussion on LinkedIn. New topics to talk about every week.
Other content series
that you might find useful
- Addressing user pain points vs solving user problems better
- Product manager skills: evolution of a PM role and its transformation
- Product metrics, growth metrics, and added value metrics
- Customer retention levers: task frequency and added value
- How to measure the added value of a product
- Should a product be 10 times better to achieve product/market fit?
- Product/market fit can be weak or strong and can change over time
- Two types of product work: creating value and delivering value
- What is the difference between growth product manager, marketing manager, and core PM
- When user activation matters and you should focus on it
- User activation is one of the key levers for product growth
- The dos and don’ts of measuring user activation
- How “aha moment” and the path to it change depending on the use case
- How to find “aha moment”: a qualitative plus quantitative approach
- How to determine the conditions necessary for the “aha moment”
- Time to value: an important lever for user activation growth
- How time to value and product complexity shape user activation
- Product-level building blocks for designing activation
- When and why to add people to the user activation process
- Session analysis: an important tool for designing activation
- CJM: from first encounter to the “aha moment”
- Designing activation in reverse: value first, acquisition channels last
- User activation starts long before sign-up
- Value windows: finding when users are ready to benefit from your product
- Why objective vs. perceived product value matters for activation
- Testing user activation fit for diverse use cases
- When to invest in optimizing user onboarding and activation
- Optimize user activation by reducing friction and strengthening motivation
- Reducing friction, strengthening user motivation: onboarding scenarios and solutions
- How to improve user activation by obtaining and leveraging additional user data
- Tax/benefit framework for analyzing user activation
- How well do you articulate value during user activation? Check with the value communication framework
- How product teams get the “aha moment” wrong
- Slack vs Teams vs Workplace: the intriguing dynamics of the work messenger market
- How the “Slack vs Microsoft Teams” race evolves as the world switches to remote work
- How Revolut Trading was built. The importance of industry expertise and the balance of conservative and new approaches
- The values and principles of Wise. Key ideas from the Breakout Growth Podcast by Sean Ellis
- How to calculate customer Lifetime Value. The do’s and don’ts of LTV calculation
- Guide to ARPU: formula, calculation example, LTV vs ARPU
- How to calculate unit economics for your business
- Experiments where you make your product worse – the most underrated product manager tool
- Why your A/B tests take longer than they should
- Peeking problem – the fatal mistake in A/B testing and experimentation
- Mistakes in A/B testing: guide to failing the right way
- Designing product experiments: template and examples
- To reduce your product’s churn rate, first find out why users stay
- What is product/market fit and how to measure PMF
- How engagement metrics can be misleading
- How to forecast key product metrics through cohort analysis
- Cohort analysis. Product metrics vs growth metrics
- Correlation and causation: how to tell the difference and why it matters for products
- How product habits are formed and what dopamine has to do with it
- Hook Model: encouraging a product habit to improve retention
- Not every product is habit-forming, but all products can have loyal users
- How to design and run JTBD research interviews: guide and templates
- How to move from marketing to product management?
- Key data skills for product managers: experienced PMs sharing their thoughts
- Data mistakes to know and avoid as a product manager
- Data cherry-picking to support your hypothesis. What is it? Why is it bad?
- How can PMs encourage more teammates to use data?
- Metrics to focus on before and after product/market fit. How to better understand your product at different stages?
- How to establish effective collaboration between product managers and data analysts
- How to move from engineering to product management?
- Losing sight of real users and their needs behind the metrics. How can product teams avoid this?
- Quantitative vs qualitative data: what is the difference and when should you use one instead of the other
- Pros and cons of a data-driven culture
- Data-driven, data-informed, and data-inspired product decisions. What are the differences and when should you use each one?
- Using data to understand competitive and market dynamics
- Moving from a startup to an enterprise as a product manager
- The downsides of a data-driven culture
- Using data for strategic decisions
- Product sense demystified. The importance behind the buzzword
- Common mistakes made by junior product managers and how to overcome them
- Is product management the right choice for you? This is your checklist
- How to move from product analytics to product management?
- How to increase the effectiveness of your product analysts
- Why every team member should know the key product metrics
- How to move from marketing to product management?
- Key data skills for product managers: experienced PMs sharing their thoughts
- Product manager skills: evolution of a PM role and its transformation
- What is the difference between growth product manager, marketing manager, and core PM
- How to move from engineering to product management?
- Product growth, reinvented: what growth hacking is (and isn’t)
- Moving from a startup to an enterprise as a product manager
- Product manager interview: real questions plus guide for employers and candidates
- Rolling retention, Day N retention, and the many facets of the retention metric
- Long-term retention—the foundation of sustainable product growth
- Retention: how to understand, calculate, and improve it
- Errors in calculating ROI and unit economics. Impact of attribution models and incrementality on the ROI calculation of marketing channels
- Traffic attribution models. Why attribution models need to change along with growth channels, product, business objective and external environment