Product teams love measuring changes in engagement. Usually with metrics like time or actions “per active user”. But such metrics can be misleading.
Here’s a simple example why.
→ Test your product management and data skills with this free Growth Skills Assessment Test.
→ Learn data-driven product management in Simulator by GoPractice.
→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.
→ Learn to apply generative AI to create products and automate processes in Generative AI for Product Managers – Mini Simulator.
→ Learn AI/ML through practice by completing four projects around the most common AI problems in AI/ML Simulator for Product Managers.
Your team’s goal is to increase the time spent per active user.
One day, the team rolls out an update aimed at increasing user engagement.
You measure engagement with the following metric: time spent per daily active user.
As more users update the app, the time spent per active user keeps going up and up.
Conclusion: The change had a positive impact.
However, after a few weeks, the time spent per active user returns to what it was before the update.
What went wrong?
Turns out that on the day of update, the app was also featured on the App Store.
This caused a significant inflow of new users. And new users tend to spend more time in a product than old ones.
The increase in the % of active users who are new to the product was the actual reason for the improvement in the engagement metric. The update itself had no impact at all.
Engagement metrics often are calculated with the number of active users in the denominator.
An active audience is a mix of diverse users: old and new, from different countries and of different ages.
When this mix changes, it impacts the engagement metrics as well.
So you have two levers affecting engagement metrics: product changes and changes in the mix of active users. It is hard to differentiate between the two.
That’s why product teams should be careful whenever their engagement metrics involve dividing by the number of active users.
