Revisiting Language Models in Neural News Recommender Systems

Yuyue Zhao, Jin Huang, David Vos, Maarten de Rijke

TL;DR

Scaling up language models in neural news recommender systems does not guarantee better recommendation accuracy, and larger models require significantly more computational resources and stricter hyperparameter tuning. However, larger models provide a major advantage for "cold-start" users with limited interaction history by shifting the system's reliance from sparse click data to rich content semantics.

What Problem Does This Paper Address?

In news recommendation, understanding the textual content of an article (like its title or abstract) is critical because news cycles move incredibly fast, and new articles lack a long history of user interactions. To handle this, modern neural news recommender systems (RSs) integrate language models (LMs) to encode articles into vector representations.

While the common assumption is that transitioning from older, shallow language models (SLMs like GloVe) to pre-trained language models (PLMs like BERT) or large language models (LLMs like Llama) automatically boosts performance, recent literature has shown conflicting results. In some benchmarks, advanced PLMs actually perform worse than simple GloVe embeddings. This paper systematically unifies and revisits these comparisons to understand when, why, and if scaling up LMs actually improves news recommendations.

Methods

The authors evaluate the impact of eight different language models across three classic neural news recommendation architectures: NAML, NRMS, and LSTUR. These architectures generally follow a standard pipeline consisting of a news encoder, a user encoder, and a click prediction module.

Fig. 1: The typical structure of neural news recommendation methods.

The LMs evaluated span three generations of text modeling:

  1. SLM: GloVe
  2. PLMs: BERT (tested across 5 sizes: tiny, mini, small, medium, and base) and RoBERTa-base
  3. LLM: Llama 3.1-8B

The authors evaluate these LMs in two distinct modes: non-fine-tuned (frozen) and fine-tuned.

Fig. 2: SLMs and PLMs as building blocks of news encoders. Each LM can be used either in its non-fine-tuned form, shown in the left plots, or in its fine- tuned form, shown in the right plots. The parameters/embeddings in the blue “ice” section are fixed, while those in the red “flame” section are fine-tuned.

For the LLM (Llama 3.1-8B), direct fine-tuning within the recommender loop is computationally prohibitive. Instead, they leverage a two-step process: first, they fine-tune Llama on the news data using a generative prompt format via LoRA (Low-Rank Adaptation), and second, they freeze the model to extract token embeddings for the recommendation task.

Fig. 3: Fine-tuning LLMs as news encoders. In step 1, the LLMs are fine-tuned on news data presented in a natural language format. In step 2, the fine-tuned LLMs generate news embeddings, which are used for the recommendation task.

Datasets

All experiments were conducted using the MIND-small dataset (Microsoft News Dataset), a widely adopted real-world benchmark collected from Microsoft News logs. It contains behavior logs from approximately 94,000 users and metadata for over 65,000 news articles.

Key Insights and Ideas

  • Bigger is not always better out-of-the-box: Without fine-tuning, the shallow GloVe model frequently outperforms smaller BERT variants and performs comparably to the massive Llama 3.1-8B. This is likely because GloVe was pre-trained on massive web crawl data, making its static word representations highly robust for general news vocabulary compared to raw, un-tuned PLM embeddings.
  • Fine-tuning is a hyperparameter minefield: For PLMs like BERT, simply fine-tuning the entire model isn't always optimal. The ideal number of layers to fine-tune varies drastically depending on the recommender framework used (e.g., NAML vs. NRMS).
  • The Cold-Start Savior: The true superpower of larger LMs (like Llama) isn't boosting accuracy for highly active users, but rather rescuing performance for "cold-start" users (those with very short click histories). Larger models extract deeper semantic context from the text, allowing the system to make high-quality content-based recommendations even when user interaction data is virtually non-existent.

Key Results

  • Overall Accuracy: In the NAML framework, Llama 3.1-8B achieved the highest overall Area Under the ROC Curve (AUC) at 68.39%, followed closely by BERT-base at 67.30%, while GloVe achieved 66.29%.
  • Fine-Tuning Impact: Fine-tuning is critical for modern architectures. For instance, fine-tuning RoBERTa within the NRMS framework resulted in a massive +9.11% relative improvement in AUC compared to keeping it frozen.
  • User Engagement Levels: When breaking down users into groups based on click history length, Llama provided the most substantial performance gains over GloVe in "Group 1" (the coldest users, averaging only ~4 prior clicks), proving its efficacy in data-sparse scenarios.

Limitations and Future Work

The authors note several limitations to their study:

  • Dataset Constraints: Due to computational limits, experiments were restricted to the English MIND-small dataset. The dynamics might change on larger scales or in non-English languages.
  • LLM Scale: The study capped LLM evaluation at an 8-billion parameter model (Llama 3.1-8B). Larger models (e.g., 70B+) might yield different trade-offs.
  • Recommendation Paradigms: The study focused strictly on content-based neural recommenders. They did not explore ID-based collaborative filtering methods, though they note that 32.9% of the news articles in the test set never appeared in the training set, making ID-based methods fundamentally poorly suited for this domain.

Future work includes testing these dynamics on multilingual datasets and exploring how to optimize larger LMs to better capture evolving, dynamic user interests over time.

Summary updated

Abstract

Neural news recommender systems (RSs) have integrated language models (LMs) to encode news articles with rich textual information into representations, thereby improving the recommendation process. Most studies suggest that (i) news RSs achieve better performance with larger pre-trained language models (PLMs) than shallow language models (SLMs), and (ii) that large language models (LLMs) outperform PLMs. However, other studies indicate that PLMs sometimes lead to worse performance than SLMs. Thus, it remains unclear whether using larger LMs consistently improves the performance of news RSs. In this paper, we revisit, unify, and extend these comparisons of the effectiveness of LMs in news RSs using the real-world MIND dataset. We find that (i) larger LMs do not necessarily translate to better performance in news RSs, and (ii) they require stricter fine-tuning hyperparameter selection and greater computational resources to achieve optimal recommendation performance than smaller LMs. On the positive side, our experiments show that larger LMs lead to better recommendation performance for cold-start users: they alleviate dependency on extensive user interaction history and make recommendations more reliant on the news content.
BibTeX
@inproceedings{zhao2025revisiting,
  author = {Yuyue Zhao and Jin Huang and David Vos and Maarten de Rijke},
  title = {{Revisiting Language Models in Neural News Recommender Systems}},
  booktitle = {ECIR},
  year = {2025},
  doi = {10.1007/978-3-031-88717-8\_12},
}
Figure