Take a Fresh Look at Recommender Systems from an Evaluation Standpoint

Aixin Sun

TL;DR

This paper argues that current offline evaluation methods for recommender systems often fail to simulate real-world scenarios by ignoring the global timeline of user interactions. This leads to misleading results, data leakage, and a disconnect between academic research and industry practice. The authors propose a timeline-aware evaluation scheme to address these issues and advocate for a deeper understanding of user decision-making contexts.

What Problem Does This Paper Address?

Recommender systems (RecSys) are ubiquitous, but their evaluation in academic research often falls short of reflecting real-world usage. A key issue is the neglect of the "global timeline" – the chronological order of user-item interactions. Standard offline evaluation practices, such as random train/test splits or leave-one-out, often disregard this temporal aspect. This can lead to several problems:

  • Ill-defined baselines: The popular "Popularity" baseline, which typically ranks items by their frequency in the training set, becomes static and doesn't reflect the dynamic nature of real-world popularity.
  • Data leakage: Models might inadvertently learn from future interactions that wouldn't be available at prediction time, leading to overly optimistic performance estimates.
  • Oversimplified user preference modeling: Ignoring the temporal context of user interactions can lead to models that don't accurately capture how user preferences evolve or how decision-making contexts change over time.

These shortcomings hinder the transferability of academic findings to production systems and can lead to counter-intuitive observations, such as users with more interactions receiving worse recommendations.

Methods

The paper's core contribution lies in its critique of existing evaluation methodologies and its proposal for a more realistic approach.

  • Critique of Data Splitting: The authors analyze common data splitting schemes (Settings 1-5 from [12]) and highlight how most of them (Settings 4 and 5) fail to observe the global timeline. A case study of papers from the ACM RecSys conference (2020-2022) reveals that a significant majority (59.1%) use these timeline-agnostic methods.
  • Revisiting Popularity Baseline: The paper demonstrates how the standard definition of the Popularity baseline (based on total training interactions) is "ill-defined" when the global timeline is ignored, leading to a static ranking that doesn't reflect real-world dynamic popularity.
  • Timeline-Aware Evaluation Scheme: The authors propose a "timeline scheme" for offline evaluation. This involves:
  • Arranging all user-item interactions chronologically.
  • Splitting the timeline into time windows of a configurable size (W).
  • Evaluating models window by window, where for each window, the model can learn from training data within that window and all preceding windows. This approach aims to better simulate how a recommender would operate in a real-time setting.
  • User Preference Modeling: The paper suggests moving beyond simply modeling user-item interactions (the "results" of decision-making) to also consider "decision contexts." This involves understanding the environment in which a user makes a choice, which is inherently time-dependent. The availability of impression data in newer datasets is highlighted as a valuable resource for this.

Datasets

The paper does not introduce new datasets but relies on existing ones for its analysis and examples.

  • MovieLens-25M: Used to illustrate data leakage and the temporal nature of item releases and user interactions.
  • Yelp, Amazon-music, Amazon-electronic: Mentioned as datasets where counter-intuitive observations have been made.
  • MIND, Taobao, FINN.NO Slates, ContentWise Impressions: Mentioned as datasets that provide impression data, which can be valuable for modeling decision contexts.

Key Insights and Ideas

  • The "Global Timeline" is Crucial: Ignoring the chronological order of user interactions in offline evaluation is a fundamental flaw that leads to unrealistic results and hinders progress.
  • Popularity is More Complex Than It Seems: Real-world popularity is dynamic and time-sensitive. The common academic practice of using a static popularity baseline is a poor proxy.
  • Data Leakage is Pervasive: Many standard evaluation practices inadvertently allow models to "see the future," leading to inflated performance metrics.
  • Context Matters for User Preference: User choices are influenced by the context in which they are made. Simply looking at what a user interacted with is insufficient; understanding when and why is critical.
  • Academic vs. Industry Disconnect: Industry practitioners often implicitly handle recency due to data volume and retraining schedules, while academic research, using large historical datasets, often fails to account for this, creating a gap.

Key Results

The paper is primarily a perspective piece and a call for a change in evaluation methodology, rather than a report of quantitative experimental results with new models.

  • Prevalence of Timeline-Agnostic Evaluation: A case study showed that 59.1% of papers in ACM RecSys conferences (2020-2022) used data splitting schemes that do not observe the global timeline.
  • Potential for Improved Popularity Baseline: The paper cites a study [17] showing that using a real-world popularity definition (time-aware) can increase the performance of the popularity method by at least 70% on the MovieLens dataset compared to the ill-defined baseline.
  • Qualitative Arguments: The paper provides strong qualitative arguments for why timeline-aware evaluation is necessary, illustrating data leakage and the limitations of static baselines with examples and figures.

Limitations and Future Work

  • Data Leakage in Proposed Scheme: The proposed timeline scheme still suffers from data leakage, though the amount is controlled by the window size. A more rigorous approach would involve not allowing models to learn from future windows when evaluating the current one.
  • Complexity of Real-time Simulation: Exactly mimicking real-time evaluation (Setting 1) is complex and less practical for academic research. The proposed timeline scheme is a relaxed version.
  • Data Sparsity: Incorporating time into evaluation can exacerbate data sparsity issues, making cold-start problems more pronounced. The authors suggest that richer datasets (including attributes) could help.
  • Standardization and Adaptation: The authors call for the adaptation and standardization of the timeline evaluation scheme.
  • Understanding User Behavior: Further research is needed to deeply understand the relationship between user behavior and recommendation, particularly in modeling user decision-making contexts.
  • Impact on Existing Conclusions: The extent to which current conclusions about model effectiveness would change with timeline-aware evaluation needs further investigation.

Summary updated

BibTeX
@inproceedings{sun2023take,
  author = {Aixin Sun},
  title = {{Take a Fresh Look at Recommender Systems from an Evaluation Standpoint}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3539618.3591931},
}
Figure