Abstract
Transformers4NewsRec is a unified Python framework designed to streamline the development and evaluation of transformer-based news recommendation systems. By introducing a novel "concatenation-based batching" method, the framework addresses the computational inefficiencies of traditional zero-padding, achieving a staggering 100%+ increase in training and evaluation speeds without sacrificing accuracy. This modular pipeline supports a wide array of encoders and provides tools for both performance tracking and qualitative explanation generation.
Introduction
Personalized news recommendation is a complex balancing act: systems must capture the semantic nuances of articles using Pre-trained Language Models (PLMs) like BERT while simultaneously modeling intricate user behaviors and historical interactions. Despite the proliferation of high-performing models, the field has lacked a unified, flexible framework to compare diverse architectures—ranging from traditional Deep Learning to Graph Neural Networks—under identical experimental conditions.
Furthermore, existing implementations frequently rely on zero-padding to handle varying sequence lengths in news titles or user histories. This approach creates significant computational overhead, as hardware resources are wasted processing "empty" zeros rather than meaningful data. Transformers4NewsRec was developed to solve these structural and efficiency challenges.
Framework Architecture
The authors introduce Transformers4NewsRec as a modular pipeline built upon the Transformers and PyTorch libraries. The framework is organized into four core functional modules to ensure flexibility and reproducibility.
- Data Module: Manages raw input ingestion and transformation into structured formats suitable for various model types.
- Model Module: Supports a diverse library of encoders, including GloVe, BERT, CNNs, GRUs, and advanced graph-based models like GLORY.
- Evaluation Module: Tracks standard ranking metrics such as AUC, MRR, and nDCG, while also facilitating the generation of qualitative explanations for recommendations.
- Utility Tools: Provides essential infrastructure, including a specialized
Transformers Trainerfor fine-tuning and automated hyperparameter optimization powered by Optuna.
Concatenation-based Batching
The primary technical innovation within the framework is Concatenation-based Batching. Traditional methods use zero-padding to ensure all sequences in a batch match the length of the longest item. In contrast, this new method concatenates sequences and utilizes specialized indexing to bypass redundant computations.
This optimization is particularly effective when employing separate encoders for users and news items, as it allows the hardware to focus exclusively on active data points, significantly increasing throughput.
Experimental Evaluation
The framework was rigorously tested using the industry-standard Microsoft News Dataset (MIND), including both the MIND-SMALL subset for rapid iteration and the MIND-LARGE dataset for full-scale validation. Models were evaluated across three primary feature configurations: news titles (GloVe), news titles and bodies (GloVe), and news titles (BERT).
Key Results
- Significant Speedup: The concatenation-based batching method improved training speed ($S_{train}$) by 111.77% and evaluation speed ($S_{eval}$) by 197.72% for the NRMS model.
- Superior Performance of Graph Models: Results consistently indicated that GLORY (a graph-based model) outperformed text-only models. On the MIND-LARGE dataset using BERT/Title features, GLORY achieved a top-tier AUC of 71.05 and an nDCG@10 of 45.04.
- Feature Impact: Incorporating the news body rather than just the title provided a consistent performance boost across all models, highlighting the importance of deep content representation.
- Explainability: By utilizing models like BATM-NR, the framework provides "Glass-Box" AI capabilities, generating topic-centric explanations (e.g., linking a recommendation to a user's interest in specific film directors).
Conclusion
Transformers4NewsRec demonstrates that significant gains in recommendation system performance can be achieved through data-handling optimizations rather than just architectural complexity. By doubling throughput while maintaining state-of-the-art accuracy, the framework provides a robust foundation for future research. Future work will focus on expanding the framework to support multilingual datasets and addressing "cold-start" scenarios where historical user data is limited.
Summary updated