Introduction
Modern news recommendation systems have increasingly relied on heavy Pretrained Language Models (PLMs) like BERT to capture the semantic nuances of news articles. While these models improve recommendation accuracy, they introduce a massive "redundancy problem" during training. In a typical end-to-end training cycle, a single news article often appears in the reading histories of thousands of different users. Analysis of the popular Microsoft News Dataset (MIND) reveals that a single article is processed approximately 1,818 times per training batch on average.
When using massive Transformer-based encoders, this redundant processing leads to astronomical computational costs and energy consumption. This paper proposes the OLEO (Only Encode Once) framework, a "Green AI" alternative that decouples news encoding from the recommendation task to eliminate this waste without sacrificing performance.
The OLEO Framework
The core innovation of OLEO is shifting the architecture from a coupled, end-to-end system to a two-stage decoupled process. This allows the system to treat news encoding as a one-time preprocessing step rather than a repetitive part of the training loop.
- News Representation Learning (NRL): In this stage, a news encoder is trained once using self-supervised learning on the news corpus. Once trained, the encoder is "frozen," and all news articles are processed a single time. Their resulting vector representations are then stored in a cache.
- Downstream Recommendation Training: The recommendation model, consisting of the user encoder and interaction module, is trained using these cached vectors. Instead of re-running a heavy language model for every user-item interaction, the system simply performs a high-speed vector lookup. This makes content-based recommenders as efficient as simple ID-based models.
Multi-field Transformer (MFT)
To ensure that the frozen news representations are sufficiently rich for recommendation tasks, the authors introduced the Multi-field Transformer (MFT). Unlike standard BERT, which processes text as a flat sequence, MFT is designed to handle the structured nature of news articles, which typically include fields like Title, Abstract, and Category.
MFT is trained using two specialized self-supervised tasks:
- Masked Token Prediction (MTP): Similar to standard language modeling, the model learns to predict hidden tokens based on the surrounding context across all fields.
- Field Alignment (FA): This task requires the model to detect whether a specific field (such as a category label) correctly corresponds to the rest of the article content. This helps the model capture vital inter-field relationships and semantic consistency.
Experimental Results
The framework was evaluated using the MIND (Microsoft News Dataset), including both the small and large-scale versions. The results demonstrate that OLEO-MFT variants achieve competitive or even superior accuracy compared to traditional "pretrain-finetune" (PLMNR) approaches, which are far more computationally expensive.
Environmental and Efficiency Gains
The most striking result is the reduction in environmental impact. OLEO reduced carbon emissions (CO2E) by up to 13 times compared to standard PLM-finetuning. To quantify this "efficiency-accuracy" trade-off, the authors introduced the ApC (AUC per Carbon Emission) metric. OLEO-MFT consistently outperformed baselines on this metric, providing significantly more "bang for the buck" in terms of environmental cost.
Furthermore, because the news encoding is done once, training times are drastically reduced. This allows researchers to iterate faster and conduct large-scale experiments on more modest hardware, such as a single consumer-grade GPU (RTX 3090).
Conclusion
The OLEO framework proves that high-performance news recommendation does not have to come at a high environmental cost. By decoupling representation learning from the recommendation task and utilizing specialized architectures like MFT, OLEO provides a scalable, efficient, and "green" path forward for content-based recommendation systems.
Summary updated