Who are the Best Adopters? User Selection Model for Free Trial Item Promotion

Shiqi Wang, Chongming Gao, Min Gao, Junliang Yu, Zongwei Wang, Hongzhi Yin

TL;DR

The paper introduces SMILE (user Selection Model wIth poLicy gradiEnt), a reinforcement learning framework designed to identify the optimal "adopters" for free trial promotions. By utilizing a balanced hierarchical tree structure to navigate massive user spaces, SMILE efficiently selects users who maximize the exposure of "long-tail" items while actually improving the overall accuracy of the recommendation system.

What Problem Does This Paper Address?

In the world of E-commerce, recommender systems often suffer from popularity bias. Algorithms naturally favor items with high transaction volumes, creating a "rich-get-richer" cycle that leaves niche or new products (the "long tail") hidden. While these low-exposure items often have higher marginal profits and can surprise users, getting them in front of the right people is difficult.

Free trials are a classic marketing solution, but the "who" matters immensely. Traditional methods—like picking users based on simple demographics or high activity levels—are static and often fail to capture personalized, dynamic preferences. Furthermore, from a technical perspective, selecting the "best" users from a pool of millions creates a massive action space problem for standard machine learning models, leading to slow training and poor performance.

Methods

The authors propose SMILE, which treats user selection as a Markov Decision Process (MDP) solved via Reinforcement Learning. The architecture consists of three main modules:

  1. State Tracker: Uses a Simple Recurrent Unit (SRU) to encode the history of previous selections and their resulting rewards into a low-dimensional vector. This allows the model to "remember" what worked in earlier rounds of the promotion.
  2. User Selector: This is the core innovation. To handle the massive number of potential users, the authors reformulate the action space into a Balanced Hierarchical Clustering Tree. Instead of picking one user out of 10,000, the agent makes a series of small, top-down decisions at each node of the tree.
  3. User Filter: A secondary step that uses Matrix Factorization to ensure the selected user is actually likely to enjoy the item (rating > 3.5), preventing negative feedback that could hurt the brand.
  4. Reward Calculator: Measures the "Page View" (PV) exposure of the promoted items. If the selected user’s interaction causes the recommender system to show the item to more people, the agent receives a higher reward.

Datasets

The model was evaluated on three widely-used public datasets:

  • Movielens100K: 943 users, 1,682 movies.
  • Movielens1M: 6,040 users, 3,900 movies.
  • Ciao: 7,935 users, 16,200 items (from a social media site).

To simulate the "long-tail" problem, the authors took popular items, deleted 95% of their interaction data to make them "low-exposure," and then tasked SMILE with recovering their visibility through trial selection.

Key Insights and Ideas

  • The Inactivity Paradox: One of the most surprising findings was that inactive users often make better trial adopters than highly active ones. Active users already have dense profiles that are hard to shift; inactive users have "sparser" profiles, meaning a single high-quality interaction from a free trial can more easily influence the recommendation algorithm to promote that item to others.
  • The "Sweet Spot" of Promotion: The authors found that increasing the number of trial users does not linearly increase exposure. There is a peak, after which adding more users actually reduces the marketing effect. This is likely because "forcing" items onto users who don't truly want them violates authentic preference distributions, causing the recommendation engine to eventually suppress the item.
  • Tree Structure as a Prior: The hierarchical tree doesn't just provide speed; because it is built using PCA-based clustering, it groups similar users together. This provides a structural "hint" to the RL agent, helping it learn collaborative patterns more effectively.

Key Results

  • Superior Exposure: SMILE significantly outperformed all static baselines (Random, Activity-based, etc.). On the Movielens100K dataset, SMILE achieved an average reward of 138.3, compared to just 11.57 for the best-performing baseline (Inactivity).
  • Massive Speedup: The hierarchical tree reduced the time complexity of sampling an action from $O(N)$ to $O(d \times c)$. In practical terms, on the Ciao dataset, sampling time dropped from ~26.7 seconds (without the tree) to just 0.29 seconds (with the tree).
  • Win-Win for the Platform: Crucially, the free trials didn't "pollute" the recommendation data. In fact, SMILE improved recommendation accuracy (Precision@10) by up to 21.4% on the Ciao dataset, proving that smart trial selection alleviates data sparsity.

Limitations and Future Work

The authors acknowledge a few areas for growth:

  • Social Diffusion: The current model treats users as individuals. In reality, "best adopters" are often influencers who trigger word-of-mouth. Future versions could incorporate social graph data to model how a trial for one person affects their entire network.
  • Offline RL: Training currently requires a simulated environment. Moving toward Offline Reinforcement Learning would allow the model to learn directly from historical logs without needing a "live" simulator.
  • Advanced RL Algorithms: The study used standard Policy Gradient (REINFORCE); the authors suggest that more stable algorithms like PPO (Proximal Policy Optimization) could further improve convergence.

Summary updated

Abstract

With the increasingly fierce market competition, the free trial has been widely applied as an effective incentive strategy to attract users and promote products. By providing opportunities to experience goods without charge, a free trial offers adopters more direct contact with the products and thus raises their willingness to buy. However, as the key point in the promotion process, how to select proper adopters is rarely explored. Empirically winnowing users by their static demographic attributes is feasible but less effective due to the lack of consideration of personalized demands. In this work, we propose SMILE – a tailored free trial user selection model for finding the best adopters promptly. Based on the reinforcement learning (RL) technique, SMILE can consider long-run profits and rely on user-item interactions to suggest actions. Besides, since selecting adopters from the large user candidates set is time-consuming, we design a balanced tree structure that reformulates the user action space. The experimental analysis on three datasets demonstrates the proposed model's superiority and elucidates why reinforcement learning and tree structure can improve performance. Our study shows technical feasibility of constructing a more robust and intelligent user selection model and guides for investigating more marketing promotion strategies.
BibTeX
@article{wang2023who,
  author = {Shiqi Wang and Chongming Gao and Min Gao and Junliang Yu and Zongwei Wang and Hongzhi Yin},
  title = {{Who are the Best Adopters? User Selection Model for Free Trial Item Promotion}},
  journal = {IEEE Trans. Big Data},
  year = {2023},
  url = {https://openreview.net/forum?id=B05taOYb5X},
}
Figure