Understanding Diversity in Session-Based Recommendation

Qing Yin, Hui Fang, Zhu Sun, Yew-Soon OngACM Trans. Information Systems 2022 · DOI · OpenReview

TL;DR

This paper dives deep into the often-overlooked aspect of diversity in session-based recommender systems (SBRSs). It reveals that accuracy-focused SBRSs can surprisingly achieve good diversity, and the relationship between accuracy and diversity is more complex than a simple trade-off, sometimes being positively correlated. The study identifies key factors influencing diversity beyond model design, offering insights for building more balanced recommenders.

What Problem Does This Paper Address?

Session-based recommender systems (SBRSs) are designed to predict a user's next action based on their current, short-term interactions. While these systems have become adept at maximizing recommendation accuracy, there's been a relative neglect in understanding and improving recommendation diversity. This is problematic because over-optimizing for accuracy can lead to "filter bubbles," where users are repeatedly shown similar items, hindering discovery and potentially impacting user satisfaction and business goals. Furthermore, the assumed "trade-off" between accuracy and diversity has been increasingly questioned, and a comprehensive understanding of this relationship, especially in the context of SBRSs, is lacking. This paper aims to fill this gap by holistically examining the performance of various SBRSs concerning both accuracy and diversity.

Methods

The authors conduct a comprehensive empirical study comparing a range of SBRSs. They categorize these models into three groups:

  • Traditional Non-neural Methods: Including POP, S-POP, Item-KNN, BPR-MF, and FPMC.
  • Deep Neural Methods: Such as GRU4Rec, NARM, STAMP, SR-GNN, GC-SAN, and GCE-GNN.
  • Deep Diversified Methods: Specifically MCPRN, NARM+MMR, and IDSR.

To ensure a fair comparison, they use standardized experimental setups, including representative datasets, evaluation metrics (accuracy: HR, MRR, NDCG; diversity: ILD, Entropy, DS; comprehensive: F-score), and hyper-parameter optimization using Bayesian TPE of Hyperopt. They also perform in-depth analysis of learned item embeddings using UMAP to understand the underlying reasons for observed performance differences.

Datasets

The study utilizes four publicly available datasets:

  • Diginetica: An e-commerce dataset from CIKM Cup 2016.
  • Retailrocket: An e-commerce dataset collected over 4.5 months.
  • Tmall: An e-commerce dataset from the IJCAI-15 competition.
  • Nowplaying: A music dataset.

These datasets vary in item category information (e-commerce datasets) and artist information (music dataset), and also differ in session diversity.

Key Insights and Ideas

  • Accuracy-Oriented Models Can Be Diverse: Surprisingly, models not explicitly designed for diversity (like POP, S-POP, and GRU4Rec) can achieve competitive or even superior diversity performance compared to some diversified methods. This challenges the notion that diversity must be explicitly engineered.
  • Complex Accuracy-Diversity Relationship: The "trade-off" is not the only story. The paper demonstrates that accuracy and diversity can be positively correlated (win-win scenarios), especially when user preferences are well-mined or datasets have certain characteristics.
  • Item Embeddings Tell a Story: The visualization of learned item embeddings reveals how model architectures (RNNs, Attention, GNNs) influence category separation. Mixed and inseparable embeddings (like in GRU4Rec) correlate with higher diversity but lower accuracy, while distinct embeddings (like in NARM) can lead to better accuracy but poorer diversity.
  • Dataset Characteristics Matter: Beyond model design, factors like the granularity of item categorization (more categories lead to potentially higher diversity metrics) and the inherent session diversity of the dataset significantly influence recommendation diversity.
  • Diversity Metrics Have Biases: Metrics like ILD and Entropy are influenced by the length of recommendation lists, while DS offers a more robust measure for variable-length sessions.

Key Results

  • Accuracy: Deep neural methods generally outperform traditional non-neural methods, with GCE-GNN achieving the best accuracy.
  • Diversity: IDSR (with a lower $\lambda$) consistently performs well on diversity. POP, S-POP, and GRU4Rec also show strong diversity performance.
  • Comprehensive Performance (F-score): NARM+MMR and GCE-GNN achieve strong overall performance.
  • Accuracy-Diversity Relationship: The paper provides evidence for both trade-off (negative correlation) and same-trend (positive correlation) relationships across different models and datasets. For instance, on Diginetica and Retailrocket, S-POP exhibits a win-win scenario.
  • Influential Factors:
    • Item Categorization Granularity: Finer granularity generally leads to higher diversity metric scores.
  • Session Diversity of Datasets: Higher session diversity in datasets correlates positively with recommendation diversity performance.
  • Length of Recommendation Lists: ILD and Entropy increase with list length, while DS tends to decrease.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that while IDSR aims to balance accuracy and diversity, its performance can be sensitive to the hyper-parameter $\lambda$.
  • The paper notes that some models (like GCE-GNN) achieve high accuracy but lower diversity, highlighting the ongoing challenge of balancing these aspects.
  • The influence of item embedding granularity is noted as a factor that can sometimes mask true model performance differences.
  • Future Work:
    • The authors propose designing advanced diversified methods for session-based recommendation based on their findings.
  • They suggest exploring user-perceived diversity, which is more aligned with real-world user experience than purely metric-based evaluations.
  • The paper introduces novel approaches (PCP and NCP) to constrain item embeddings for better diversity and accuracy, suggesting further investigation into these techniques.

Summary updated

Abstract

Current session-based recommender systems (SBRSs) mainly focus on maximizing recommendation accuracy, while few studies have been devoted to improve diversity beyond accuracy. Meanwhile, it is unclear how the accuracy-oriented SBRSs perform in terms of diversity. Besides, the asserted "trade-off" relationship between accuracy and diversity has been increasingly questioned in the literature. Towards the aforementioned issues, we conduct a holistic study to particularly examine the recommendation performance of representative SBRSs w.r.t. both accuracy and diversity, striving for better understanding the diversity-related issues for SBRSs and providing guidance on designing diversified SBRSs. Particularly, for a fair and thorough comparison, we deliberately select state-of-the-art non-neural, deep neural, and diversified SBRSs, by covering more scenarios with appropriate experimental setups, e.g., representative datasets, evaluation metrics, and hyper-parameter optimization technique. Our empirical results unveil that: 1) non-diversified methods can also obtain satisfying performance on diversity, which might even surpass diversified ones; and 2) the relationship between accuracy and diversity is quite complex. Besides the "trade-off" relationship, they might be positively correlated with each other, that is, having a same-trend (win-win or lose-lose) relationship, which varies across different methods and datasets. Additionally, we further identify three possible influential factors on diversity in SBRSs (i.e., granularity of item categorization, session diversity of datasets, and length of recommendation lists).
BibTeX
@article{yin2022understanding,
  author = {Qing Yin and Hui Fang and Zhu Sun and Yew-Soon Ong},
  title = {{Understanding Diversity in Session-Based Recommendation}},
  journal = {ACM Trans. Information Systems},
  year = {2022},
  doi = {10.1145/3600226},
}
Figure