A good friend of ours—someone we’ve collaborated with on many projects—worked with a team running a content platform that offered a mix of free and paid content. Their goal was to convert free-tier users into paid subscribers. Together, they built a machine-learning–powered recommendation system that showed previews of premium content to free users in order to encourage upgrades. Along the way, several important lessons emerged.

  1. The initial ML system relied on content similarity to recommend articles and pieces. While it achieved strong click-through rates, it did not lead to a meaningful increase in paid subscriptions.
  2. Because the available data offered limited insight, the team turned to interviews and surveys. They discovered that the audience was not homogeneous: some users were looking for timely, news-driven content, while others preferred in-depth, educational material. Even when engaging with the same content, these segments expected very different follow-up recommendations (for example, related news stories versus deeper educational dives). This highlighted that a one-size-fits-all approach was ineffective.
  3. To better capture user intent, an onboarding step was added, asking users how they discovered the platform and what goals they had. This enabled much more granular user segmentation.
  4. With this finer segmentation, the team faced a choice:

    — build a single, complex model capable of capturing detailed content features and nuanced user preferences, or
    — develop separate, simpler models for each user segment.

    A unified model would have been easier to manage long-term and more adaptable to changes, but it required a much larger dataset than was available. A segmented approach allowed simpler, more interpretable models that worked with fewer training examples, at the cost of flexibility. They chose the segmented approach initially, with the explicit plan to migrate to a single model once more data was collected.

Result

The initial rollout increased conversion rates by 15%. More importantly, it set the stage for further improvements:

  1. The team invested more heavily in qualitative data through surveys and interviews. For example, they learned that offering limited free trials of paid content helped engage users and improved conversion. These insights also revealed gaps in the content catalog, pointing to new content opportunities that could improve retention and overall experience.
  2. They deepened their understanding of user goals by building an ML system that monitored changes in preferences and behavior over time. When a user deviated significantly from their usual patterns, a short survey was triggered to understand whether their goals had changed (for instance, due to a career transition or a new learning objective). Combining explicit feedback with behavioral signals proved crucial for refining both user segments and content strategy.

By combining quantitative data with qualitative insights, the team further improved the recommendation system and achieved an additional 10% increase in conversion over six months. They also saw a noticeable influx of new users through word-of-mouth recommendations from satisfied subscribers.

To enhance your skills in working on AI/ML products, you can benefit from: