TL;DR
Scaling up language models in neural news recommender systems does not guarantee better recommendation accuracy, and larger models require significantly more computational resources and stricter hyperparameter tuning. However, larger models provide a major advantage for "cold-start" users with limited interaction history by shifting the system's reliance from sparse click data to rich content semantics.
What Problem Does This Paper Address?
In news recommendation, understanding the textual content of an article (like its title or abstract) is critical because news cycles move incredibly fast, and new articles lack a long history of user interactions. To handle this, modern neural news recommender systems (RSs) integrate language models (LMs) to encode articles into vector representations.
While the common assumption is that transitioning from older, shallow language models (SLMs like GloVe) to pre-trained language models (PLMs like BERT) or large language models (LLMs like Llama) automatically boosts performance, recent literature has shown conflicting results. In some benchmarks, advanced PLMs actually perform worse than simple GloVe embeddings. This paper systematically unifies and revisits these comparisons to understand when, why, and if scaling up LMs actually improves news recommendations.
Methods
The authors evaluate the impact of eight different language models across three classic neural news recommendation architectures: NAML, NRMS, and LSTUR. These architectures generally follow a standard pipeline consisting of a news encoder, a user encoder, and a click prediction module.
The LMs evaluated span three generations of text modeling:
- SLM: GloVe
- PLMs: BERT (tested across 5 sizes: tiny, mini, small, medium, and base) and RoBERTa-base
- LLM: Llama 3.1-8B
The authors evaluate these LMs in two distinct modes: non-fine-tuned (frozen) and fine-tuned.
For the LLM (Llama 3.1-8B), direct fine-tuning within the recommender loop is computationally prohibitive. Instead, they leverage a two-step process: first, they fine-tune Llama on the news data using a generative prompt format via LoRA (Low-Rank Adaptation), and second, they freeze the model to extract token embeddings for the recommendation task.
Datasets
All experiments were conducted using the MIND-small dataset (Microsoft News Dataset), a widely adopted real-world benchmark collected from Microsoft News logs. It contains behavior logs from approximately 94,000 users and metadata for over 65,000 news articles.
Key Insights and Ideas
- Bigger is not always better out-of-the-box: Without fine-tuning, the shallow GloVe model frequently outperforms smaller BERT variants and performs comparably to the massive Llama 3.1-8B. This is likely because GloVe was pre-trained on massive web crawl data, making its static word representations highly robust for general news vocabulary compared to raw, un-tuned PLM embeddings.
- Fine-tuning is a hyperparameter minefield: For PLMs like BERT, simply fine-tuning the entire model isn't always optimal. The ideal number of layers to fine-tune varies drastically depending on the recommender framework used (e.g., NAML vs. NRMS).
- The Cold-Start Savior: The true superpower of larger LMs (like Llama) isn't boosting accuracy for highly active users, but rather rescuing performance for "cold-start" users (those with very short click histories). Larger models extract deeper semantic context from the text, allowing the system to make high-quality content-based recommendations even when user interaction data is virtually non-existent.
Key Results
- Overall Accuracy: In the NAML framework, Llama 3.1-8B achieved the highest overall Area Under the ROC Curve (AUC) at 68.39%, followed closely by BERT-base at 67.30%, while GloVe achieved 66.29%.
- Fine-Tuning Impact: Fine-tuning is critical for modern architectures. For instance, fine-tuning RoBERTa within the NRMS framework resulted in a massive +9.11% relative improvement in AUC compared to keeping it frozen.
- User Engagement Levels: When breaking down users into groups based on click history length, Llama provided the most substantial performance gains over GloVe in "Group 1" (the coldest users, averaging only ~4 prior clicks), proving its efficacy in data-sparse scenarios.
Limitations and Future Work
The authors note several limitations to their study:
- Dataset Constraints: Due to computational limits, experiments were restricted to the English
MIND-smalldataset. The dynamics might change on larger scales or in non-English languages. - LLM Scale: The study capped LLM evaluation at an 8-billion parameter model (Llama 3.1-8B). Larger models (e.g., 70B+) might yield different trade-offs.
- Recommendation Paradigms: The study focused strictly on content-based neural recommenders. They did not explore ID-based collaborative filtering methods, though they note that 32.9% of the news articles in the test set never appeared in the training set, making ID-based methods fundamentally poorly suited for this domain.
Future work includes testing these dynamics on multilingual datasets and exploring how to optimize larger LMs to better capture evolving, dynamic user interests over time.
Summary updated