An Unbiased Data Collection and Content Exploitation/Exploration Strategy for Personalization

Liangjie Hong, Adnan BozarXiv 2016 · arXiv · OpenReview

Abstract

One of missions for personalization systems and recommender systems is to show content items according to users' personal interests. In order to achieve such goal, these systems are learning user interests over time and trying to present content items tailoring to user profiles. Recommending items according to users' preferences has been investigated extensively in the past few years, mainly thanks for the popularity of Netflix competition. In a real setting, users may be attracted by a subset of those items and interact with them, only leaving partial feedbacks to the system to learn in the next cycle, which leads to significant biases into systems and hence results in a situation where user engagement metrics cannot be improved over time. The problem is not just for one component of the system. The data collected from users is usually used in many different tasks, including learning ranking functions, building user profiles and constructing content classifiers. Once the data is biased, all these downstream use cases would be impacted as well. Therefore, it would be beneficial to gather unbiased data through user interactions. Traditionally, unbiased data collection is done through showing items uniformly sampling from the content pool. However, this simple scheme is not feasible as it risks user engagement metrics and it takes long time to gather user feedbacks. In this paper, we introduce a user-friendly unbiased data collection framework, by utilizing methods developed in the exploitation and exploration literature. We discuss how the framework is different from normal multi-armed bandit problems and why such method is needed. We layout a novel Thompson sampling for Bernoulli ranked-list to effectively balance user experiences and data collection. The proposed method is validated from a real bucket test and we show strong results comparing to old algorithms.

Introduction

Recommender systems often fall into a "feedback loop" trap: the system displays items it predicts a user will like, the user interacts with a subset of those items, and the resulting data is used to train the next iteration of the model. This creates significant selection bias, as the system only learns about the items it chose to show, remaining blind to the rest of the content pool. This biased data negatively impacts downstream tasks like user profiling and ranking refinement.

While uniform random sampling provides unbiased data, it severely degrades user engagement. This paper addresses the challenge of finding a "user-friendly" exploration strategy that satisfies immediate user interests while fulfilling the system's long-term need for high-quality, unbiased data.

Methods

The authors propose a framework based on Contextual Multi-Armed Bandits (cMAB), specifically utilizing Thompson Sampling (TS). In a standard Bernoulli TS setup, the system maintains a Beta distribution representing the belief about an item's success rate. The paper introduces three key innovations:

  1. Stochastic Ranked-Lists: Instead of a deterministic top-N selection, the system uses a multinomial distribution to sample items stochastically, providing a clear probability for unbiased evaluation.
  2. Unbiased Offline Evaluation: By logging the selection probability ($p_a$) for each item, the authors can use an estimator similar to Inverse Propensity Scoring to evaluate new policies offline as if they were tested on unbiased data.
  3. Efficiency Approximation: To handle millions of candidates, the authors use a fast approximation ($\theta'_i = \lambda \theta_i$, where $\lambda$ is a uniform random variable) that maintains mathematical consistency with probabilistic goals while significantly reducing computation time.

Experimental Results

The proposed framework was evaluated via a live bucket test on the Yahoo France Homepage. The test compared a new Thompson Sampling algorithm (Bucket 203) against a General-Available control (Bucket 684) and a deterministic chronological baseline.

Engagement and Data Quality

The live test demonstrated that the new algorithm successfully balanced exploration with user experience. As shown in Figure 1, engagement metrics—including Dwell Time Per Depth (S1), Dwell Time Per User, and Clicks per PageView—showed marked improvements, approaching the performance of the highly optimized control bucket.

Engagement metrics over time for the test (203) and control (684) buckets.

Beyond raw engagement, the quality of the data collected for future training was significantly higher. The system successfully spread its "attention" across a wider variety of items, reducing the skewness of the resulting data distributions.

Comparison of distribution metrics between the new and old algorithms.

As detailed in Figure 2, the new algorithm reduced View Skewness and CTR Skewness by approximately 30% compared to the old algorithm. Additionally, it proved more effective at discovering new content; the percentage of articles seen within their first 30 minutes rose from 66% to 81%, effectively mitigating the item cold-start problem.

Conclusion

This research demonstrates that a stochastic ranking approach based on Thompson Sampling can effectively collect unbiased data without sacrificing immediate user engagement. By logging selection probabilities, the system enables more accurate offline evaluation and provides a more balanced dataset for downstream modeling. Future work will explore the impact of training ranking and profiling models directly on this unbiased data to quantify the long-term benefits of the exploration strategy.

Summary updated

Abstract

One of missions for personalization systems and recommender systems is to show content items according to users' personal interests. In order to achieve such goal, these systems are learning user interests over time and trying to present content items tailoring to user profiles. Recommending items according to users' preferences has been investigated extensively in the past few years, mainly thanks for the popularity of Netflix competition. In a real setting, users may be attracted by a subset of those items and interact with them, only leaving partial feedbacks to the system to learn in the next cycle, which leads to significant biases into systems and hence results in a situation where user engagement metrics cannot be improved over time. The problem is not just for one component of the system. The data collected from users is usually used in many different tasks, including learning ranking functions, building user profiles and constructing content classifiers. Once the data is biased, all these downstream use cases would be impacted as well. Therefore, it would be beneficial to gather unbiased data through user interactions. Traditionally, unbiased data collection is done through showing items uniformly sampling from the content pool. However, this simple scheme is not feasible as it risks user engagement metrics and it takes long time to gather user feedbacks. In this paper, we introduce a user-friendly unbiased data collection framework, by utilizing methods developed in the exploitation and exploration literature. We discuss how the framework is different from normal multi-armed bandit problems and why such method is needed. We layout a novel Thompson sampling for Bernoulli ranked-list to effectively balance user experiences and data collection. The proposed method is validated from a real bucket test and we show strong results comparing to old algorithms
BibTeX
@misc{hong2016unbiased,
  author = {Liangjie Hong and Adnan Boz},
  title = {{An Unbiased Data Collection and Content Exploitation/Exploration Strategy for Personalization}},
  year = {2016},
  eprint = {1604.03506},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.03506},
}
Figure