TL;DR
This paper dives deep into the often-overlooked aspect of diversity in session-based recommender systems (SBRSs). It reveals that accuracy-focused SBRSs can surprisingly achieve good diversity, and the relationship between accuracy and diversity is more complex than a simple trade-off, sometimes being positively correlated. The study identifies key factors influencing diversity beyond model design, offering insights for building more balanced recommenders.
What Problem Does This Paper Address?
Session-based recommender systems (SBRSs) are designed to predict a user's next action based on their current, short-term interactions. While these systems have become adept at maximizing recommendation accuracy, there's been a relative neglect in understanding and improving recommendation diversity. This is problematic because over-optimizing for accuracy can lead to "filter bubbles," where users are repeatedly shown similar items, hindering discovery and potentially impacting user satisfaction and business goals. Furthermore, the assumed "trade-off" between accuracy and diversity has been increasingly questioned, and a comprehensive understanding of this relationship, especially in the context of SBRSs, is lacking. This paper aims to fill this gap by holistically examining the performance of various SBRSs concerning both accuracy and diversity.
Methods
The authors conduct a comprehensive empirical study comparing a range of SBRSs. They categorize these models into three groups:
- Traditional Non-neural Methods: Including POP, S-POP, Item-KNN, BPR-MF, and FPMC.
- Deep Neural Methods: Such as GRU4Rec, NARM, STAMP, SR-GNN, GC-SAN, and GCE-GNN.
- Deep Diversified Methods: Specifically MCPRN, NARM+MMR, and IDSR.
To ensure a fair comparison, they use standardized experimental setups, including representative datasets, evaluation metrics (accuracy: HR, MRR, NDCG; diversity: ILD, Entropy, DS; comprehensive: F-score), and hyper-parameter optimization using Bayesian TPE of Hyperopt. They also perform in-depth analysis of learned item embeddings using UMAP to understand the underlying reasons for observed performance differences.
Datasets
The study utilizes four publicly available datasets:
- Diginetica: An e-commerce dataset from CIKM Cup 2016.
- Retailrocket: An e-commerce dataset collected over 4.5 months.
- Tmall: An e-commerce dataset from the IJCAI-15 competition.
- Nowplaying: A music dataset.
These datasets vary in item category information (e-commerce datasets) and artist information (music dataset), and also differ in session diversity.
Key Insights and Ideas
- Accuracy-Oriented Models Can Be Diverse: Surprisingly, models not explicitly designed for diversity (like POP, S-POP, and GRU4Rec) can achieve competitive or even superior diversity performance compared to some diversified methods. This challenges the notion that diversity must be explicitly engineered.
- Complex Accuracy-Diversity Relationship: The "trade-off" is not the only story. The paper demonstrates that accuracy and diversity can be positively correlated (win-win scenarios), especially when user preferences are well-mined or datasets have certain characteristics.
- Item Embeddings Tell a Story: The visualization of learned item embeddings reveals how model architectures (RNNs, Attention, GNNs) influence category separation. Mixed and inseparable embeddings (like in GRU4Rec) correlate with higher diversity but lower accuracy, while distinct embeddings (like in NARM) can lead to better accuracy but poorer diversity.
- Dataset Characteristics Matter: Beyond model design, factors like the granularity of item categorization (more categories lead to potentially higher diversity metrics) and the inherent session diversity of the dataset significantly influence recommendation diversity.
- Diversity Metrics Have Biases: Metrics like ILD and Entropy are influenced by the length of recommendation lists, while DS offers a more robust measure for variable-length sessions.
Key Results
- Accuracy: Deep neural methods generally outperform traditional non-neural methods, with GCE-GNN achieving the best accuracy.
- Diversity: IDSR (with a lower $\lambda$) consistently performs well on diversity. POP, S-POP, and GRU4Rec also show strong diversity performance.
- Comprehensive Performance (F-score): NARM+MMR and GCE-GNN achieve strong overall performance.
- Accuracy-Diversity Relationship: The paper provides evidence for both trade-off (negative correlation) and same-trend (positive correlation) relationships across different models and datasets. For instance, on Diginetica and Retailrocket, S-POP exhibits a win-win scenario.
- Influential Factors:
- Item Categorization Granularity: Finer granularity generally leads to higher diversity metric scores.
- Session Diversity of Datasets: Higher session diversity in datasets correlates positively with recommendation diversity performance.
- Length of Recommendation Lists: ILD and Entropy increase with list length, while DS tends to decrease.
Limitations and Future Work
- Limitations:
- The study acknowledges that while IDSR aims to balance accuracy and diversity, its performance can be sensitive to the hyper-parameter $\lambda$.
- The paper notes that some models (like GCE-GNN) achieve high accuracy but lower diversity, highlighting the ongoing challenge of balancing these aspects.
- The influence of item embedding granularity is noted as a factor that can sometimes mask true model performance differences.
- Future Work:
- The authors propose designing advanced diversified methods for session-based recommendation based on their findings.
- They suggest exploring user-perceived diversity, which is more aligned with real-world user experience than purely metric-based evaluations.
- The paper introduces novel approaches (PCP and NCP) to constrain item embeddings for better diversity and accuracy, suggesting further investigation into these techniques.
Summary updated