Individual Persistence Adaptation for User-Centric Evaluation of User Satisfaction in Recommender Systems

Nozomu Onodera, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

TL;DR

Standard metrics for evaluating recommender systems often assume every user browses a list of suggestions with the same level of patience. This paper introduces a framework that personalizes the "persistence parameter"—the likelihood a user keeps scrolling—for every individual, significantly improving how well offline metrics actually correlate with real-world user satisfaction.

What Problem Does This Paper Address?

In the world of Recommender Systems (RS), the "gold standard" for success is user satisfaction. However, measuring satisfaction directly is expensive and slow, requiring constant surveys. To get around this, researchers use mathematical proxies like Rank-Biased Precision (RBP) or Discounted Cumulative Gain (DCG).

These metrics rely on a "persistence parameter" ($\alpha$), which models a user's browsing behavior: a high $\alpha$ means the user is a "deep scroller" who looks at many items, while a low $\alpha$ means they only care about the top few results. Historically, the RS field has treated $\alpha$ as a constant (e.g., 0.9 for everyone). The authors argue this is a major flaw. Because users have different personalities and goals, a one-size-fits-all parameter creates a "discrepancy gap" between what the math says and how the user actually feels.

Methods

The authors propose a user-centric evaluation framework that adapts the persistence parameter ($\alpha_u$) for each individual user ($u$).

  1. Metric Adaptation: They took three standard IR (Information Retrieval) metrics—nDCG, nRBP, and ERR—and modified them to accept an individual persistence parameter instead of a global one.
  2. Parameter Estimation: To find the right $\alpha_u$ for a specific person, they used regression models based on the user's "continuous exploration probability."
    • For metrics with logarithmic penalties (like DCG), they used logarithmic regression.
  3. For metrics with exponential penalties (like RBP and ERR), they used exponential regression.
  4. Validation: They built a movie recommendation system using the BPR-MF (Bayesian Personalized Ranking) algorithm to test this framework against real human feedback.

Datasets

  • MovieLens25M: Used as the base dataset to train the recommendation algorithm, specifically focusing on ~18,000 movies produced after 1980.
  • Primary Subject Data: The authors conducted their own experiment with 28 subjects (9 Chinese, 19 Japanese). They collected explicit data on:
  • Interest/preference for 50 recommended movies.
  • Self-reported satisfaction levels (every 10 movies).
  • Likelihood to continue browsing (to estimate the individual persistence parameter).

Key Insights and Ideas

  • Persistence as Personality: The core "aha" moment is treating scrolling depth not as a system property, but as a personality trait. A user who is easily satisfied or quickly bored needs a different "yardstick" for evaluation than a power user who digs through dozens of pages.
  • Breaking the Stereotype: The paper challenges the "stereotype" in RS research that evaluation must be quantified equally for all users. By personalizing the evaluation itself, we get a more honest look at system performance.
  • The "Slimmer" Distribution: When plotting metric scores against actual satisfaction, the personalized approach produced much tighter, more linear clusters (slimmer scatter plots) than the fixed-parameter approach. This suggests the personalized metric is a much more reliable predictor of human emotion.

Key Results

The researchers used Spearman’s rank correlation coefficient ($\rho$) to measure how well their metrics matched the users' reported satisfaction.

  • Higher Correlation: The "adaptive" framework consistently outperformed conventional fixed-parameter models. For example, at a list depth of 50 (top-N=50), the adaptive nRBP achieved a correlation of $\rho = 0.3930$, whereas the best fixed-parameter version only reached $\rho = 0.2194$.
  • Statistical Significance: In the comparative (fixed) frameworks, the $p$-values were often greater than 0.05, meaning the correlation wasn't even statistically significant. In contrast, the adaptive framework's correlations were statistically significant, proving it captures a real relationship that standard models miss.

Limitations and Future Work

  • Small Sample Size: The study relied on 28 subjects. While the results are statistically significant, a larger, more diverse pool would further validate the findings.
  • Explicit vs. Implicit Data: Currently, the framework requires users to answer a questionnaire to estimate their persistence. The authors acknowledge that future work should focus on estimating $\alpha_u$ using implicit behavior, such as actual scroll depth, dwell time, or click patterns, to make the system fully automated.
  • Single Algorithm: The study only used one recommendation algorithm (BPR-MF). Testing the framework across various algorithms (e.g., Deep Learning-based RS) would confirm its generalizability.

Summary updated

Abstract

The primary objective of a recommender system (RS) is to enhance user satisfaction, which serves as the gold standard for evaluation. In order to support the advancement of RS, it is crucial to study how to accurately measure user satisfaction. This paper proposes a novel evaluation framework that leverages user-centric evaluations to achieve a more precise measurement of satisfaction. User-centric evaluations involve metrics that model the user’s behavior toward the recommended ranking list. These metrics incorporate a persistence parameter that captures the user’s personality traits. However, most conventional studies in the RS field utilize a predefined persistence parameter for all users, neglecting the individual differences among users. In contrast, our work focuses on optimizing this parameter individually for each user, which, to the best of our knowledge, is the first attempt in the RS context. To demonstrate the effectiveness of our framework in measuring satisfaction, we conducted innovative subject experiments to collect satisfaction data. Through these experiments, we obtained valuable insights into user satisfaction, which we used for quantitative correlation analysis. The results of this analysis provide empirical evidence that our framework is well-suited for RS evaluation, showing a stronger alignment with user satisfaction compared to conventional approaches.
BibTeX
@article{onodera2024individual,
  author = {Nozomu Onodera and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Individual Persistence Adaptation for User-Centric Evaluation of User Satisfaction in Recommender Systems}},
  journal = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2024.3360693},
}
Figure