It is common to evaluate your product performance and the impact of changes you make by using engagement metrics with active audience in the denominator. Examples of these metrics include the time spent per active user, the occurrence of certain actions (messages sent, levels played, chapters read, etc) per active user, or ratio metrics (what percent of active users perform a specific action) .
In most situations, these engagement metrics will be helpful. But in some cases, they can be misleading. And it is important to understand why and when this can happen, and what you can do about it.
→ Test your product management and data skills with this free Growth Skills Assessment Test.
→ Learn data-driven product management in Simulator by GoPractice.
→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.
→ Learn to apply generative AI to create products and automate processes in Generative AI for Product Managers – Mini Simulator.
→ Learn AI/ML through practice by completing four projects around the most common AI problems in AI/ML Simulator for Product Managers.
Example: How time spent per active user can be misleading
“Time spent per active user” is a widely used metric that helps you understand how engaged your audience is. To measure it, we add up the time spent in the app by all users over a certain period and divide the sum by the number of active users in the same period (total time spent by all users / number of users). You can measure this metric with daily, weekly or monthly granularity.
Social media app companies regularly use this metric to evaluate user engagement. For example, Facebook users spend an average of 58 minutes per day on the platform, Instagram comes second at 53 minutes, and YouTube at 40 minutes per day. [source] Time spent per active user gives you a rough idea of which apps have the most engaged users. Another good example is how YouTube considered the television as one of its main competitors and used time spent watching YouTube vs time spent watching TV as a way to measure its progress.
Now here’s where the widely used time spent metric can fail you. Imagine you are working on a mobile product and your team has chosen to focus on growing time spent per active user.
One day you release an update with a big change aimed at increasing user engagement, so you expect it to increase time spent per active user. Fortunately, the App Store likes that change and decides to feature your app. So on the day you roll out the update, you enjoy a huge influx of new users.
The following day, you check the dashboard and see that time spent per active user has gone up. The team is happy – the investment made in the update has paid off. But over the next month, the increase in time spent per user goes away and the metric returns to its previous level. What happened? Was it novelty effect? Or maybe something else?
This is a theoretical case, but let me show you another scenario where the product change had no impact on the product’s performance, and the increase in time spent per active user metric happened due to a different reason.
Imagine the product you are working on has the following dynamics:
- Users spend more time in the product in their first days, mainly because they are exploring its features. In the following weeks, their time spent in the app reduces considerably.
- Behavior in the product varies across users who have been using the product for different periods of time.
As long as the number and structure of daily active users of the product you work on is stable, the time spent per active user metric describes user engagement quite well. If you improve your product and make it more engaging, this metric will reflect it.
But at times where you see a sudden spike in the number of new users (or another factor changes the structure of daily active audience), the very same metric that worked well before can become misleading.
Since new users tend to spend more time in the app in the first few days, you will see an increase in time spent per active user even if the product has not changed.
