Only Encode Once: Making Content-based News Recommender Greener

Qijiong Liu, Jieming Zhu, Quanyu Dai, Xiao-Ming WuarXiv 2023 · DOI · arXiv

Introduction

Modern news recommendation systems have increasingly relied on heavy Pretrained Language Models (PLMs) like BERT to capture the semantic nuances of news articles. While these models improve recommendation accuracy, they introduce a massive "redundancy problem" during training. In a typical end-to-end training cycle, a single news article often appears in the reading histories of thousands of different users. Analysis of the popular Microsoft News Dataset (MIND) reveals that a single article is processed approximately 1,818 times per training batch on average.

When using massive Transformer-based encoders, this redundant processing leads to astronomical computational costs and energy consumption. This paper proposes the OLEO (Only Encode Once) framework, a "Green AI" alternative that decouples news encoding from the recommendation task to eliminate this waste without sacrificing performance.

The OLEO Framework

The core innovation of OLEO is shifting the architecture from a coupled, end-to-end system to a two-stage decoupled process. This allows the system to treat news encoding as a one-time preprocessing step rather than a repetitive part of the training loop.

The OLEO framework shifts from end-to-end training to a two-stage decoupled process, caching news representations to eliminate redundant encoding.
  1. News Representation Learning (NRL): In this stage, a news encoder is trained once using self-supervised learning on the news corpus. Once trained, the encoder is "frozen," and all news articles are processed a single time. Their resulting vector representations are then stored in a cache.
  2. Downstream Recommendation Training: The recommendation model, consisting of the user encoder and interaction module, is trained using these cached vectors. Instead of re-running a heavy language model for every user-item interaction, the system simply performs a high-speed vector lookup. This makes content-based recommenders as efficient as simple ID-based models.

Multi-field Transformer (MFT)

To ensure that the frozen news representations are sufficiently rich for recommendation tasks, the authors introduced the Multi-field Transformer (MFT). Unlike standard BERT, which processes text as a flat sequence, MFT is designed to handle the structured nature of news articles, which typically include fields like Title, Abstract, and Category.

The Multi-field Transformer (MFT) architecture utilizes Masked Token Prediction (MTP) and Field Alignment (FA) to learn rich, structured news representations.

MFT is trained using two specialized self-supervised tasks:

  • Masked Token Prediction (MTP): Similar to standard language modeling, the model learns to predict hidden tokens based on the surrounding context across all fields.
  • Field Alignment (FA): This task requires the model to detect whether a specific field (such as a category label) correctly corresponds to the rest of the article content. This helps the model capture vital inter-field relationships and semantic consistency.

Experimental Results

The framework was evaluated using the MIND (Microsoft News Dataset), including both the small and large-scale versions. The results demonstrate that OLEO-MFT variants achieve competitive or even superior accuracy compared to traditional "pretrain-finetune" (PLMNR) approaches, which are far more computationally expensive.

Environmental and Efficiency Gains

The most striking result is the reduction in environmental impact. OLEO reduced carbon emissions (CO2E) by up to 13 times compared to standard PLM-finetuning. To quantify this "efficiency-accuracy" trade-off, the authors introduced the ApC (AUC per Carbon Emission) metric. OLEO-MFT consistently outperformed baselines on this metric, providing significantly more "bang for the buck" in terms of environmental cost.

Furthermore, because the news encoding is done once, training times are drastically reduced. This allows researchers to iterate faster and conduct large-scale experiments on more modest hardware, such as a single consumer-grade GPU (RTX 3090).

Conclusion

The OLEO framework proves that high-performance news recommendation does not have to come at a high environmental cost. By decoupling representation learning from the recommendation task and utilizing specialized architectures like MFT, OLEO provides a scalable, efficient, and "green" path forward for content-based recommendation systems.

Summary updated

Abstract

Large pretrained language models (PLM) have become de facto news encoders in modern news recommender systems, due to their strong ability in comprehending textual content. These huge Transformer-based architectures, when finetuned on recommendation tasks, can greatly improve news recommendation performance. However, the PLM-based pretrain-finetune framework incurs high computational cost and energy consumption, primarily due to the extensive redundant processing of news encoding during each training epoch. In this paper, we propose the ``Only Encode Once'' framework for news recommendation (OLEO), by decoupling news representation learning from downstream recommendation task learning. The decoupled design makes content-based news recommender as green and efficient as id-based ones, leading to great reduction in computational cost and training resources. Extensive experiments show that our OLEO framework can reduce carbon emissions by up to 13 times compared with the state-of-the-art pretrain-finetune framework and maintain a competitive or even superior performance level. The source code is released for reproducibility.
BibTeX
@misc{liu2023only,
  author = {Qijiong Liu and Jieming Zhu and Quanyu Dai and Xiao-Ming Wu},
  title = {{Only Encode Once: Making Content-based News Recommender Greener}},
  year = {2023},
  eprint = {2308.14155},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.14155},
}
Figure