TL;DR
Standard metrics for evaluating recommender systems often assume every user browses a list of suggestions with the same level of patience. This paper introduces a framework that personalizes the "persistence parameter"—the likelihood a user keeps scrolling—for every individual, significantly improving how well offline metrics actually correlate with real-world user satisfaction.
What Problem Does This Paper Address?
In the world of Recommender Systems (RS), the "gold standard" for success is user satisfaction. However, measuring satisfaction directly is expensive and slow, requiring constant surveys. To get around this, researchers use mathematical proxies like Rank-Biased Precision (RBP) or Discounted Cumulative Gain (DCG).
These metrics rely on a "persistence parameter" ($\alpha$), which models a user's browsing behavior: a high $\alpha$ means the user is a "deep scroller" who looks at many items, while a low $\alpha$ means they only care about the top few results. Historically, the RS field has treated $\alpha$ as a constant (e.g., 0.9 for everyone). The authors argue this is a major flaw. Because users have different personalities and goals, a one-size-fits-all parameter creates a "discrepancy gap" between what the math says and how the user actually feels.
Methods
The authors propose a user-centric evaluation framework that adapts the persistence parameter ($\alpha_u$) for each individual user ($u$).
- Metric Adaptation: They took three standard IR (Information Retrieval) metrics—nDCG, nRBP, and ERR—and modified them to accept an individual persistence parameter instead of a global one.
- Parameter Estimation: To find the right $\alpha_u$ for a specific person, they used regression models based on the user's "continuous exploration probability."
- For metrics with logarithmic penalties (like DCG), they used logarithmic regression.
- For metrics with exponential penalties (like RBP and ERR), they used exponential regression.
- Validation: They built a movie recommendation system using the BPR-MF (Bayesian Personalized Ranking) algorithm to test this framework against real human feedback.
Datasets
- MovieLens25M: Used as the base dataset to train the recommendation algorithm, specifically focusing on ~18,000 movies produced after 1980.
- Primary Subject Data: The authors conducted their own experiment with 28 subjects (9 Chinese, 19 Japanese). They collected explicit data on:
- Interest/preference for 50 recommended movies.
- Self-reported satisfaction levels (every 10 movies).
- Likelihood to continue browsing (to estimate the individual persistence parameter).
Key Insights and Ideas
- Persistence as Personality: The core "aha" moment is treating scrolling depth not as a system property, but as a personality trait. A user who is easily satisfied or quickly bored needs a different "yardstick" for evaluation than a power user who digs through dozens of pages.
- Breaking the Stereotype: The paper challenges the "stereotype" in RS research that evaluation must be quantified equally for all users. By personalizing the evaluation itself, we get a more honest look at system performance.
- The "Slimmer" Distribution: When plotting metric scores against actual satisfaction, the personalized approach produced much tighter, more linear clusters (slimmer scatter plots) than the fixed-parameter approach. This suggests the personalized metric is a much more reliable predictor of human emotion.
Key Results
The researchers used Spearman’s rank correlation coefficient ($\rho$) to measure how well their metrics matched the users' reported satisfaction.
- Higher Correlation: The "adaptive" framework consistently outperformed conventional fixed-parameter models. For example, at a list depth of 50 (top-N=50), the adaptive nRBP achieved a correlation of $\rho = 0.3930$, whereas the best fixed-parameter version only reached $\rho = 0.2194$.
- Statistical Significance: In the comparative (fixed) frameworks, the $p$-values were often greater than 0.05, meaning the correlation wasn't even statistically significant. In contrast, the adaptive framework's correlations were statistically significant, proving it captures a real relationship that standard models miss.
Limitations and Future Work
- Small Sample Size: The study relied on 28 subjects. While the results are statistically significant, a larger, more diverse pool would further validate the findings.
- Explicit vs. Implicit Data: Currently, the framework requires users to answer a questionnaire to estimate their persistence. The authors acknowledge that future work should focus on estimating $\alpha_u$ using implicit behavior, such as actual scroll depth, dwell time, or click patterns, to make the system fully automated.
- Single Algorithm: The study only used one recommendation algorithm (BPR-MF). Testing the framework across various algorithms (e.g., Deep Learning-based RS) would confirm its generalizability.
Summary updated