MPGraf: a Modular and Pre-trained Graphformer for Learning to Rank at Web-scale (Extended Abstract)

Yuchen Li, Haoyi Xiong, Linghe Kong, Zeyi Sun, Hongyang Chen

Abstract

Learning to Rank (LTR) is fundamental to web search, yet integrating local text semantics with global graph-based relationships at scale remains challenging. We present MPGraf, a modular framework that combines Transformers and Graph Neural Networks (GNNs). MPGraf utilizes a "link rippling" strategy for graph construction and a "surgical" fine-tuning approach to mitigate distribution shifts. Evaluated on both public benchmarks and large-scale commercial search data, MPGraf demonstrates significant improvements in ranking performance, particularly for long-tail queries.

Introduction

In modern search engines, Learning to Rank (LTR) determines the relevance of webpages to user queries. Two dominant architectures excel in different areas: Transformers are adept at capturing fine-grained query-page text interactions, while Graph Neural Networks (GNNs) excel at modeling broader contextual relationships within query-page bipartite graphs. However, combining these approaches at "web-scale" is difficult due to differing data structures, computational overhead, and distribution shifts between pre-training and target datasets.

Methodology: The MPGraf Framework

MPGraf addresses these challenges through a three-stage modular pipeline.

The complete MPGraf pipeline: (1) Graph Construction, (2) Representation Learning, and (3) Surgical Fine-tuning.

To handle sparse interaction data, MPGraf employs "link rippling." This involves "Query-centered Expanding Ripples" to identify high-relevance pairs and "Webpage-centered Shrinking Ripples" for effective negative sampling. This ensures a robust graph structure even with limited human annotations.

2. Hybrid Graphformer Architecture

The framework is modular, supporting two integration strategies:

  • Stacking Graphformer: GNN outputs are fed directly into a Transformer.
  • Parallelizing Graphformer: GNN and Transformer modules run concurrently, with their features fused in later layers. This allows the model to simultaneously leverage local semantic content and global graph context.

3. Surgical Fine-tuning

To combat distribution shifts, a multi-stage "surgical" fine-tuning strategy is used. By selectively freezing and unfreezing modules, the model preserves general knowledge from pre-training while adapting to specific downstream tasks.

The surgical fine-tuning process, alternating between training and freezing specific modules to maintain stability.

Experimental Evaluation

MPGraf was evaluated on standard benchmarks (MSLR-Web30K, MQ2007, MQ2008) and a large-scale commercial dataset from Baidu (15,000 queries, 770,000+ pairs).

Key Results

  • Offline Performance: MPGraf consistently outperformed state-of-the-art baselines like XGBoost and LightGBM. The "Stacking" variant improved NDCG@10 by up to 1.74% on commercial data.
  • Online Impact: Live deployment on a major search engine showed significant gains, particularly for "long-tail" queries (up to 7.50% improvement in GSB).
  • Data Efficiency: The model remained robust even when labeled data was reduced to 5%, demonstrating its ability to "borrow" information across the graph.

Conclusion

MPGraf successfully bridges the gap between Transformers and GNNs for web-scale ranking. While global graph construction remains an engineering challenge, the modular architecture and surgical fine-tuning provide a scalable path forward. Future work will explore integrating Large Language Models (LLMs) to further enhance semantic understanding within this framework.

Summary updated

BibTeX
@inproceedings{li2024mpgraf,
  author = {Yuchen Li and Haoyi Xiong and Linghe Kong and Zeyi Sun and Hongyang Chen},
  title = {{MPGraf: a Modular and Pre-trained Graphformer for Learning to Rank at Web-scale (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=zi6g0E33KQ},
}
Figure