Abstract
Both Transformer and Graph Neural Networks (GNNs) have been used in learning to rank (LTR), however, they adhere to two distinct yet complementary problem formulations, i.e., ranking score regression based on query-webpage pairs and link prediction within query-webpage bipartite graphs, respectively. Though it is possible to pre-train GNNs or Transformers on source datasets and fine-tune them subject to sparsely annotated LTR datasets separately, the source-target distribution shifts across the pairs and bipartite graphs domains make it extremely difficult to integrate these diverse models into a single LTR framework at a web-scale. We introduce the novel MPGraf model, which utilizes a modular and capsule-based pre-training approach, aiming to incorporate regression capacities from Transformers and link prediction capabilities of GNNs cohesively. We conduct extensive experiments to evaluate the performance of MPGraf using real-world datasets collected from large-scale search engines. The results show that MPGraf can outperform baseline algorithms on several major metrics. Further, we deploy and evaluate MPGraf atop a large-scale search engine with realistic web traffic via A/B tests, where we can still observe significant improvement. MPGraf performs consistently in both offline and online evaluations.
Introduction
Learning to Rank (LTR) is a critical component of modern search engines. Historically, researchers have approached LTR through two primary lenses: Transformers for regression on query-webpage pairs (focusing on textual and contextual features) and Graph Neural Networks (GNNs) for link prediction within bipartite graphs (focusing on structural relationships). While these approaches are complementary, integrating them at a web-scale is challenging due to massive distribution shifts between large-scale pre-training data and sparse, task-specific target data.
MPGraf (Modular and Pre-trained Graphformer) bridges this gap. It is a unified architecture designed to capture both structural "wisdom of the crowd" and specific query intent. By employing a novel "Link Rippling" graph construction and a "surgical" fine-tuning strategy, MPGraf scales hybrid models to web-scale search, significantly improving ranking performance for both head and long-tail queries.
Methodology
The MPGraf framework follows a three-step pipeline: graph construction, representation learning, and surgical fine-tuning.
1. Graph Construction with Link Rippling
To address data sparsity in LTR datasets, MPGraf uses two "Link Rippling" techniques to build informative graphs:
- Query-centered Expanding Ripple: This self-tuning approach propagates labels from annotated pairs to unlabeled ones, generating high-quality "pseudo-links."
- Webpage-centered Shrinking Ripple: A negative sampling method that identifies irrelevant queries for a specific webpage, ensuring a balanced and informative bipartite graph.
2. Hybrid Graphformer Architecture
The core of MPGraf is the Hybrid Graphformer, which integrates GNN and Transformer modules. These can be composed in two primary ways:
- Stacking Graphformer: The GNN first extracts structural features, which are then passed into the Transformer's self-attention layers.
- Parallelizing Graphformer: Both modules extract features simultaneously, with their outputs fused before final ranking score calculation.
3. Surgical Fine-tuning
Standard fine-tuning often fails in hybrid models due to distribution shifts. MPGraf introduces a "surgical" approach to overcome this:
- Phased Updates: Instead of naive joint training, the model undergoes a phased approach. For instance, tuning the GNN module while freezing others, followed by joint fine-tuning of the entire stack.
- Regularization: This selective freezing prevents the generalizable structural knowledge learned during pre-training from being "washed out" by the noisy signals of smaller, specific LTR datasets.
Experimental Evaluation
MPGraf was evaluated using a combination of public benchmarks and commercial data:
- Public Benchmarks: MSLR-Web30K, MQ2007, and MQ2008.
- Commercial Dataset: 15,000 queries and over 770,000 query-webpage pairs from a major search engine.
- Online A/B Testing: Real-world traffic evaluation to measure live performance metrics like $\Delta_{AB}$ and $\Delta_{GSB}$.
Results and Discussion
The experimental results demonstrate the effectiveness of the modular Graphformer approach:
- Offline Gains: On commercial data, the Stacking MPGraf (using NeuralNDCG loss) outperformed standard MLP baselines by up to 1.74% in NDCG@10.
- Online Impact: In live A/B tests, the model achieved a 0.45% improvement in $\Delta_{AB}$ and a significant 7.50% improvement in $\Delta_{GSB}$ for long-tail queries.
- Robustness: Performance gains remained stable even when labeled data was reduced to 5%, highlighting the strength of the pre-training strategy.
A key insight is the synergy between structural and contextual signals. GNNs capture how different queries relate to the same pages, while Transformers capture the specific intent of a single query. This is particularly valuable for "long-tail" queries (appearing <10 times/week), where historical data is sparse but structural signals are rich.
Conclusion
MPGraf successfully unifies GNNs and Transformers for web-scale LTR. While the model shows significant gains, future work could explore more complex heterogeneous graphs (including user behavior or knowledge graph entities) and automated "auto-surgical" policies for fine-tuning. Detailed analysis of inference latency overhead in real-time pipelines also remains an area for further investigation.
Summary updated