A Re-visit of the Popularity Baseline in Recommender Systems

Yitong Ji, Aixin Sun, Jie Zhang, Chenliang LiSIGIR 2020 · DOI · OpenReview

Abstract

Popularity is often included in experimental evaluation to provide a reference performance for a recommendation task. To understand how popularity baseline is defined and evaluated, we sample 12 papers from top-tier conferences including KDD, WWW, SIGIR, and RecSys, and 6 open source toolkits. We note that the widely adopted MostPop baseline simply ranks items based on the number of interactions in the training data. We argue that the current evaluation of popularity (i) does not reflect the popular items at the time when a user interacts with the system, and (ii) may recommend items released after a user's last interaction with the system. On the widely used MovieLens dataset, we show that the performance of popularity could be significantly improved by 70% or more, if we consider the popular items at the time point when a user interacts with the system. We further show that, on MovieLens dataset, the users having lower tendencies on movies tend to follow the crowd and rate more popular movies. Movie lovers who rate a large number of movies, rate movies based on their own preferences and interests. Through this study, we call for a re-visit of the popularity baseline in recommender system to better reflect its effectiveness.

Introduction: The "Straw Man" Baseline

In recommender systems (RecSys) research, the "Most Popular" (MostPop) method is the simplest possible baseline: it recommends items that have the highest number of total interactions in the training set. It is non-personalized and intended to provide a performance floor. However, this standard implementation is often a "straw man" because it is static. In a standard offline evaluation, the system often recommends items based on their popularity across the entire training period, leading to temporal mismatch and temporal impossibility (e.g., recommending a movie that hadn't been released yet at the time of the user's interaction).

Methods

The authors propose two time-aware versions of the popularity baseline to address these flaws:

  • RecentPop: Ranks items based on popularity within a specific window (e.g., one month) immediately preceding the user's interaction time.
  • DecayPop: Applies an exponential decay function ($e^{-t_m}$) to interactions over the last six months, giving higher weight to very recent trends.

User Behavior Analysis

A key insight of this research is how different user segments interact with popular content. By partitioning users into groups based on their activity levels, the researchers found that casual users (those with fewer ratings) are significantly more likely to follow the crowd. In contrast, "power users" who rate a large number of movies tend to follow their own niche interests.

User activity vs. popularity preference

As shown in the analysis, the percentage of popular movies in a user's history drops as their total number of ratings increases. This suggests that popularity is a much stronger signal for the "cold start" or casual user segment than for dedicated enthusiasts.

Key Results

The quantitative results on the MovieLens 20M dataset show a massive jump in accuracy when time is considered. Simply making the popularity baseline time-aware improves Hit Rate (HR) and NDCG by 70% to 83% compared to the standard MostPop.

Performance comparison of popularity baselines

DecayPop achieved the best results in most measures, closely followed by RecentPop. These results suggest that many complex, personalized models may not be providing as much "lift" as previously thought if they are only compared against a static, unrealistic popularity baseline.

Conclusion

This study highlights the necessity of using temporally consistent baselines in recommender system evaluation. By ignoring the dimension of time, researchers may be underestimating the effectiveness of popularity and overestimating the relative gains of complex neural models. Future work should involve re-evaluating state-of-the-art models against these stronger, time-aware baselines.

Summary updated

Abstract

Popularity is often included in experimental evaluation to provide areference performance for a recommendation task. To understand how popularity baseline is defined and evaluated, we sample 12 papers from top-tier conferences including KDD, WWW, SIGIR, and RecSys, and 6 open source toolkits. We note that the widely adoptedMostPop baseline simply ranks items based on the number of interactions in the training data. We argue that the current evaluation of popularity (i) does not reflect the popular items at the time when a user interacts with the system, and (ii) may recommend items released after a user's last interaction with the system. On the widely used MovieLens dataset, we show that the performance of popularity could be significantly improved by 70% or more, if we consider the popular itemsat the time point when a user interacts with the system. We further show that, on MovieLens dataset, the users having lower tendencies on movies tend to follow the crowd and rate more popular movies. Movie lovers who rate a large number of movies, rate movies based on their own preferences and interests. Through this study, we call for a re-visit of the popularity baseline in recommender system to better reflect its effectiveness.
BibTeX
@inproceedings{ji2020re,
  author = {Yitong Ji and Aixin Sun and Jie Zhang and Chenliang Li},
  title = {{A Re-visit of the Popularity Baseline in Recommender Systems}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  doi = {10.1145/3397271.3401233},
}
Figure