Improve Retention by using ML/AI: what to do and what not to do

Retention is a key metric for many products. That’s why teams that start implementing AI/ML in their products want to know how these technologies can help to better retain users.

One of the most common ways to use AI/ML for improving retention is predicting user churn. By identifying users who are highly likely to leave, we can offer them discounts, bonuses, and other perks to keep them engaged with the product.

However, in reality, such projects rarely bring value to the business. Often, teams train a model that accurately predicts user churn, but they can’t prevent churn or retention-focused efforts are not financially viable.

In this article, we will discuss an alternative approach to improving retention using AI/ML that makes more efficient use of your budget — uplift modeling.

The fundamentals of churn prediction

Let’s start by diving into the specifics of churn prediction to better understand the differences in approaches.

How training a model for user churn prediction works? We gather historical data about users who have left and those who continue to use the product.

↓

We calculate features that might contain signals indicating whether a user will leave or stay.

For example, potential features for a marketplace could include: how long the user has been on the service, when they made their last purchase, when they last logged into the app, what their last review was, and so on.

Detailed examples of solutions can be found in these articles:

• “From Big Data to business analytics: The case study of churn prediction”
• “Customer churn prediction using real-time analytics”
• “On Analyzing Churn Prediction in Mobile Games”

↓

We use the data to train an ML model that predicts user churn based on user features.

How the churn prediction model is used:

• After training, the model is applied to active users of the service.
• For users with a high risk of churn, a budget can be allocated to maintain their loyalty through bonuses, discounts, push notifications, or other incentives.
• Incentives are a necessary intervention to prevent user churn.
• It is important to ensure that the costs of intervention are less than the profit obtained from the retained users.

The problem with this approach

In the described approach, the retention mechanisms are applied to all users who are likely to leave. However, not all of these users are the same.

We can divide churn candidates into the following categories:

• Users who will leave in any case (no actions on your part will affect their decision).
• Users who can be influenced, but the cost of retention is higher than the future potential profit (such users can be retained, but it is very expensive and is not economically justified).
• Users who can be influenced and keep the retention cost will be below future profits (such users can be retained, and it is economically beneficial).

Why is it important to distinguish such user groups?

Retaining users from the first and second categories makes no sense – it will be too costly and only result in losses. For effective and economically justified influence on the Retention metric, it makes sense to work only with the third group.

In classic churn prediction, it is assumed that all departing users are engaged equally. However, in the context of uplift modeling, the main focus is specifically on the third group.

If you want to retain users in your product, build a user retention system not based on churn prediction but on uplift modeling.

The fundamentals of uplift modeling

To train a model within the framework of uplift modeling, it is necessary to conduct the following A/B test:

• A portion of active product users is randomly divided into two groups: test and control.
• Chosen retention mechanism (bonuses, discounts, special communication) is applied to all users in the test group.
• The experience of users from the control group does not change.

This test will enable us to collect data on how motivation and retention mechanisms affect user behavior and overall product metrics.

Consequently, we may observe that the retention mechanics:

• Do not work at all. In such a case, there is no point in creating ML models, and it is necessary to investigate the reasons for the low effectiveness of the retention mechanisms.
• Works, but does not pay off. In such a case, you can try to achieve profitability through more selective application of retention mechanisms.
• Works and pays off. In this case, you can move forward and achieve even greater profitability through smarter application of retention mechanisms.

In the second and third cases, it makes sense to move on to the next step of model training.

Uplift modeling involves training two ML models that will predict user churn (as discussed earlier):

• A model that predicts whether a user will leave if no interventions are applied. To train this model, data from the control group of the experiment should be used.
• A model that predicts whether a user will leave even if interventions are applied. To train this model, data from the test group of the experiment should be used.

How to apply the models:

• For each active user, we will apply both trained models and compare their predictions.
• Next, we calculate the ratio of the probability that a user will leave with interventions over the probability that a user will leave without interventions.
• The smaller this value, the stronger the influence of retention mechanisms on the user’s decision.
• The final step is to select a threshold for including users in the motivation program, ensuring that the retention project is cost-effective by comparing the total retention costs with the benefits of retaining users.

The main challenge of uplift modeling is running the A/B tests. These tests often require a sufficiently large user base. This is expensive but essential to gather enough data for training.

What is important to remember

1. If you want to retain your users, use uplift modeling. It allows for the efficient use of budgets by targeting users on whom the proposed intervention will have a genuinely positive impact.

2. Classic churn prediction is best used as an analytical tool to identify factors associated with users’ decisions to leave a product.

3. Uplift modeling can be used not only in user retention projects but also for making marketing campaigns more effective by choosing a suitable target audience. For example, a marketplace can offer coupons only to users who would not make purchases without them, and not offer them to users who would make purchases even without coupons.

AI/ML Simulator for Product Managers by GoPractice will help you effectively apply artificial intelligence and machine learning technologies.