Neural News Recommendation with Multi-Head Self-Attention

Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, Xing Xie

Introduction

Personalized news recommendation is a critical task for digital platforms, requiring both an accurate representation of news content and a deep understanding of user interests. Traditional approaches have largely relied on Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs). However, these methods have inherent limitations. CNNs focus on local context, often failing to capture long-range dependencies between words in a title. RNNs, while capable of modeling sequences, are computationally expensive and process data sequentially, which hinders real-time performance.

Figure 1: Illustration of long-range dependencies in news titles and user browsing history.

As illustrated in Figure 1, a headline like "Rockets trade Carter-Williams to the Bulls" contains key entities ("Rockets" and "Bulls") that are physically separated but semantically linked. Similarly, a user's browsing history may contain disparate articles that collectively signal a core interest, such as sports, even if they are interspersed with unrelated clicks like weather forecasts.

The NRMS Framework

To address these challenges, the authors propose the Neural News Recommendation with Multi-Head Self-Attention (NRMS) framework. NRMS utilizes a dual-layered attention mechanism to model both news content and user behavior.

Figure 2: The architecture of the NRMS framework, including news and user encoders.

The framework, detailed in Figure 2, consists of three primary modules:

  1. News Encoder: This module uses Multi-Head Self-Attention (MHSA) to allow every word in a news title to interact with every other word, effectively capturing long-range semantic dependencies. This is followed by an additive attention layer that assigns weights to words based on their informativeness, ensuring the final representation focuses on key terms.
  2. User Encoder: Similar to the news encoder, the user encoder applies MHSA to the sequence of news articles a user has browsed. This captures the relatedness between different articles in the history. An additive attention layer then weights these articles, prioritizing those most representative of the user's core interests over incidental clicks.
  3. Click Predictor: The final recommendation is made by calculating the dot product between the user representation and a candidate news representation to estimate the probability of a click.

The model is trained using negative sampling, learning to distinguish between articles a user clicked and a set of non-clicked candidate articles.

Key Insights and Results

A major innovation of NRMS is the symmetry of its attention mechanism. By applying the same hierarchical structure to both words (content) and articles (behavior), the model mirrors the way humans process information. Furthermore, the use of self-attention allows for massive parallelization, making NRMS significantly faster than sequential models like GRUs.

In evaluations using a real-world dataset from MSN News (comprising 10,000 users and nearly 500,000 click samples), NRMS consistently outperformed baseline methods. It achieved an AUC of 0.6275, surpassing GRU-based (0.6102) and DKN (0.6032) models. Beyond accuracy, NRMS demonstrated superior ranking performance in metrics like nDCG@10 and Mean Reciprocal Rank (MRR). Efficiency-wise, NRMS is highly scalable, encoding one million users in just 6.7 minutes, compared to 148 minutes for a GRU-based approach.

Conclusion and Future Work

NRMS represents a significant step forward in news recommendation by leveraging multi-head self-attention to capture complex dependencies efficiently. While currently focused on news titles, future iterations could incorporate full article bodies, temporal information (timestamps), and positional encodings to further refine its predictive power. Addressing the "cold start" problem for new users remains an open area for exploration.

Summary updated

BibTeX
@inproceedings{wu2019neural,
  author = {Chuhan Wu and Fangzhao Wu and Suyu Ge and Tao Qi and Yongfeng Huang and Xing Xie},
  title = {{Neural News Recommendation with Multi-Head Self-Attention}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1671},
}
Figure