Accuracy meets Diversity in a News Recommender System

Shaina Raza, Syed Raza Bashir, Usman NaseemCOLING 2022

TL;DR

This paper introduces a novel two-tower deep neural network architecture for news recommender systems that explicitly balances accuracy and diversity. By incorporating rich news item side information and a category loss function, their approach effectively addresses the challenge of recommending relevant yet varied news content to users with evolving interests.

What Problem Does This Paper Address?

In the age of information overload, news recommender systems (NRS) are crucial for helping users navigate the vast landscape of online news. However, these systems face a fundamental challenge: users' interests are dynamic and can shift rapidly. Furthermore, simply recommending the most relevant articles can lead to a "filter bubble" effect, where users are only exposed to content that confirms their existing views, neglecting the need for diverse perspectives. Existing NRS often prioritize accuracy (relevancy) over diversity, or struggle to effectively incorporate rich contextual information about news items and user behavior. This paper aims to bridge this gap by developing an NRS that can provide both accurate and diverse recommendations, acknowledging the imbalanced nature of news categories and the dynamic evolution of user preferences.

Methods

The core of this work is a novel two-tower deep neural network architecture.

  • Two-Tower Architecture: This standard approach uses separate "towers" to learn representations for users (query tower) and items (news item tower).
  • Rich Side Information: Unlike many existing models that rely solely on IDs, this architecture enriches the news item tower with side information such as news body, category, and title. The query tower incorporates user context, like time and place of interaction.
  • Augmented Vectors: To model the interaction between the towers, augmented vectors are created by concatenating user/item features with information about positive interactions. These augmented vectors are then processed through fully connected layers with ReLU activation and L2 normalization to produce query ($p_u$) and item ($p_v$) representations.
  • Category Loss Function: To explicitly promote diversity, a category loss function is introduced during training. This function aims to align the representations of news items from various categories, addressing the issue of imbalanced category distribution where some categories might dominate recommendations.
  • Loss Function: The overall loss function is a combination of:
  • A binary cross-entropy loss for classifying positive and negative query-item pairs.
  • Mean squared error losses ($l_u$, $l_v$) to fit augmented vectors to query/item embeddings.
  • A category loss ($l_{cate}$) to encourage diversity across news categories.
  • Scoring: The final recommendation score is calculated as the dot product of the query and item embeddings ($s(u, v) = <p_u, p_v>$).

Datasets

The proposed model was evaluated on two datasets:

  • MIND (Microsoft News Dataset): A benchmark dataset consisting of anonymized behavior logs from Microsoft News.
  • NYTimes: News articles and anonymized reader interactions collected using the NYTimes API.

The authors generated training samples from click histories and impression logs, following the format of the MIND paper.

Key Insights and Ideas

  • Accuracy-Diversity Trade-off is Inherent: The paper confirms a well-established observation: as recommendation list length (top@k) increases, accuracy generally rises, while diversity tends to decrease. This highlights the fundamental challenge in balancing these two crucial aspects.
  • Side Information is Crucial for Accuracy and Diversity: Including rich side information (news body, category, title) significantly improves both accuracy and diversity metrics, demonstrating its importance in capturing nuanced user interests and item characteristics.
  • Random Negative Sampling Aids Balance: The study found that using random negative sampling (Rand-100) for evaluation, as opposed to popularity-based sampling, provides a better balance between accuracy and diversity. This suggests that a more uniform sampling strategy can lead to more robust evaluation of both aspects.
  • Category Imbalance is a Key Diversity Hurdle: The paper explicitly addresses the issue of imbalanced news categories, which can lead to recommendations skewed towards dominant categories. Their category loss function is designed to mitigate this.

Key Results

  • Superior Performance: The proposed model achieved the highest accuracy (F1-score, NDCG) and diversity (Gini index) scores across both datasets compared to several state-of-the-art baselines.
  • Balanced Trade-off: The model also demonstrated the best trade-off score, indicating its effectiveness in simultaneously optimizing for accuracy and diversity.
  • Impact of Side Information: When news body information was excluded, accuracy metrics (F1@10, F1@20, F1@50) dropped significantly (e.g., F1@50 from 62.10% to 15.20%), while diversity metrics (Gini@10, Gini@20, Gini@50) also decreased, underscoring the value of rich features.
  • Top@k Impact: Increasing the recommendation list length (top@k) improved accuracy but decreased diversity, as expected.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the learning efficiency of some methods can be limited in high-dimensional parameter spaces.
  • Some existing methods assume only binary feedback, which can be a limitation when direct reward observation is not possible.
  • The study primarily focuses on news body as side information, with other contextual features not fully explored due to space constraints.
  • Future Work:
    • Conducting experiments on larger, real-world news datasets.
  • Developing a deeper neural network architecture.
  • Evaluating the approach using more advanced diversity metrics, such as normalized topic coverage and novelty.
  • Incorporating more user feedback signals, like click-through rates.

Summary updated

BibTeX
@inproceedings{raza2022accuracy,
  author = {Shaina Raza and Syed Raza Bashir and Usman Naseem},
  title = {{Accuracy meets Diversity in a News Recommender System}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure