Methodological Issues in Recommender Systems Research (Extended Abstract)

Maurizio Ferrari Dacrema, Paolo Cremonesi, Dietmar JannachIJCAI 2020 · DOI

TL;DR

This paper argues that the recommender systems research community is often misled by "phantom progress" due to issues with reproducibility, weak baselines, and arbitrary experimental setups. By re-evaluating recent top-tier recommender system papers, the authors demonstrate that many complex deep learning models are outperformed by simpler, established methods, raising serious questions about the actual advancement in the field.

What Problem Does This Paper Address?

The field of recommender systems is a vibrant area of research, with countless new algorithms proposed each year. To demonstrate progress, researchers typically rely on offline experiments, comparing their novel methods against existing "state-of-the-art" baselines. However, this paper highlights a critical problem: the accepted practices for these offline evaluations may be leading to an illusion of progress, or "phantom progress." This occurs because:

  • Reproducibility is low: It's often difficult to replicate the results reported in published papers.
  • Baselines are weak: The "state-of-the-art" baselines used for comparison are sometimes not well-optimized or are conceptually simpler methods that are more effective.
  • Experimental configurations are arbitrary: The choice of datasets, evaluation metrics, and cutoff lengths (the number of recommended items) is often not well-justified, leading to evaluations that might not reflect real-world performance.

The authors aim to quantify the extent of these issues by analyzing recent research in top recommender systems conferences.

Methods

The authors conducted an empirical study by analyzing 18 recent research papers on recommender systems. Their methodology involved:

  1. Reproducibility Check: Attempting to reproduce the results of the selected papers using the provided code and data.
  2. Benchmarking: For reproducible papers, they re-evaluated the proposed methods against a set of conceptually simpler, well-established baselines (e.g., non-personalized popularity, user/item-based nearest neighbors, and simple graph-based methods).
  3. Systematic Optimization: Crucially, they ensured that all baselines were systematically optimized for the specific datasets and experimental settings used in the original papers, a step they argue is often neglected.

Datasets

The paper mentions the use of several standard datasets for evaluation, including:

  • CiteULike-a
  • Pinterest
  • Epinions
  • MovieLens (ML100K, ML1M, ML20M)
  • LastFM
  • Yelp
  • Netflix
  • HetRec
  • Amazon Video
  • MSD

The authors emphasize that their evaluation used the same datasets and train-test splits as reported in the original papers, but they also introduced additional baselines for comparison.

Key Insights and Ideas

  • The Illusion of Progress: The most striking insight is that many complex, recent deep learning models are not significantly outperforming simpler, older methods when evaluated rigorously. This suggests that the field might be overstating its progress.
  • Reproducibility is a Major Hurdle: A significant portion of the analyzed papers were not reproducible, making it difficult to build upon existing work or verify claims.
  • Weak Baselines Undermine Comparisons: The choice of baselines is critical. If baselines are not properly optimized or are inherently weak, even a modest improvement from a new method can appear substantial.
  • Arbitrary Experimental Design: The lack of clear justification for dataset choices, metrics, and cutoff lengths makes it hard to understand the true generalizability of proposed methods.
  • Deep Learning's Computational Cost: While not the primary focus, the authors note that the computational demands of tuning deep learning models can be a barrier to thorough experimentation.

Key Results

  • Reproducibility Rate: Only 7 out of 18 analyzed papers (39%) were reproducible with reasonable effort.
  • Baseline Performance: For the reproducible papers, 6 out of the 7 complex algorithms were outperformed by at least one simpler heuristic method (like nearest neighbors) when evaluated using the same experimental setup and with optimized baselines.
  • Specific Examples:
    • The Collaborative Memory Network (CMN) was outperformed by simpler baselines on two datasets.
  • MCRec was outperformed by item-based nearest neighbors on the MovieLens dataset.
  • CVAE was outperformed by simple hybrid methods in most configurations.
  • Spectral Collaborative Filtering (SpectralCF) performed poorly when evaluated with properly constructed test sets.
  • Neural Collaborative Filtering (NCF) was sometimes outperformed by item-based nearest neighbors or the SLIM method.
  • Mult-VAE Exception: The paper identifies Mult-VAE as one method that consistently outperformed simpler baselines, even showing improvements over the SLIM method.

Limitations and Future Work

Limitations:

  • Extended Abstract: This paper is an extended abstract, meaning it provides a high-level overview of a larger study. Detailed experimental results and analyses are likely in the full publication.
  • Focus on Top-N Recommendation: The analysis primarily focuses on top-N recommendation tasks.
  • Limited Scope of Baselines: While they included several simpler baselines, the universe of potential baselines is vast.

Future Work:

  • Stricter Publication Requirements: The authors advocate for stricter requirements for reproducibility and more rigorous experimental design in published papers.
  • Increased Awareness: Raising awareness among researchers about these methodological pitfalls is crucial.
  • Beyond Offline Evaluation: The paper strongly suggests moving beyond purely offline evaluations. Future work should more frequently incorporate user studies and consider the human-computer interaction aspects of recommender systems to assess true value.
  • Addressing Over-reliance on Offline Metrics: The field needs to question whether small improvements in offline accuracy metrics truly translate to better user experiences or business value.

Summary updated

BibTeX
@inproceedings{dacrema2020methodological,
  author = {Maurizio Ferrari Dacrema and Paolo Cremonesi and Dietmar Jannach},
  title = {{Methodological Issues in Recommender Systems Research (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/650},
}
Figure