GS2P: a generative pre-trained learning to rank model with over-parameterization for web-scale search

Yuchen Li, Haoyi Xiong, Linghe Kong, Jiang Bian, Shuaiqiang Wang, Guihai Chen, Dawei Yin

Abstract

Learning to Rank (LTR) is a fundamental component of modern web search engines. Despite the success of deep learning, LTR models often suffer from labeled data scarcity and suboptimal generalization on structural data. This paper presents GS²P, a novel three-stage framework that integrates semi-supervised learning, generative pre-training, and over-parameterization. By utilizing tree-based models for pseudo-labeling and Transformer-based denoising autoencoders for representation learning, GS²P captures complex data distributions. Furthermore, it leverages the "double descent" phenomenon through Random Fourier Features to achieve superior generalization in the interpolating regime. Empirical results on the Web30K benchmark and a large-scale commercial dataset from Baidu demonstrate that GS²P significantly outperforms existing baselines, particularly in low-resource and long-tail query scenarios.

Introduction

In web-scale search, LTR models are the engines that determine document relevance. However, they face two primary challenges: data scarcity, where only a fraction of query-webpage pairs are labeled, and representation failure, where deep models struggle to generalize from structural data as effectively as traditional statistical learners like Gradient Boosted Decision Trees (GBDTs). GS²P (Generative Semi-Supervised Pre-trained) bridges this gap by creating a deep learning model that generalizes better than traditional methods while scaling to the massive demands of modern search engines.

The GS²P Framework

The GS²P framework operates in three distinct stages designed to maximize data utility and model robustness.

The three-stage GS²P framework: (1) Semi-supervised Pseudo-Label Generation, (2) Self-attentive Representation via Denoising Autoencoding, and (3) LTR via Over-parameterized MLP.

1. Semi-supervised Pseudo-Label Generation (SPLG)

To address the shortage of human-annotated data, GS²P employs a co-training procedure. Diverse tree-based LTR models (e.g., LightGBM) are trained on small labeled datasets to predict "pseudo-labels" for massive unlabeled datasets, effectively expanding the training pool.

2. Self-attentive Representation Learning (SRLDA)

The model uses a Transformer-based encoder to learn query-webpage representations. It is trained with a dual loss: a discriminative loss for ranking and a generative loss for reconstructing original data from noisy inputs. This denoising task forces the model to understand the fundamental structure of the data beyond simple label matching.

3. Over-parameterized Learning to Rank (LTROM)

GS²P utilizes Random Fourier Features (RFF) to map learned representations into a high-dimensional space. By intentionally over-parameterizing the model, the system moves past the "overfitting zone" into the "interpolating regime." This leverages the mathematical phenomenon of double descent, where generalization error decreases as model complexity increases significantly.

System Integration and Deployment

GS²P is designed to function within a high-throughput search architecture, processing candidates retrieved from the index and providing refined rankings before final results are served to the user.

Integration of the GS²P-based LTR model within a web-scale search engine's retrieval and ranking architecture.

Experimental Results

The authors evaluated GS²P on the public Web30K dataset and a massive commercial dataset from Baidu Search. Key findings include:

  • State-of-the-Art Performance: GS²P consistently outperformed baselines like XGBoost, LightGBM, and standard MLP models.
  • Low-Resource Resilience: In scenarios with only 5% labeled data, GS²P achieved a 3.6% improvement in NDCG@4 over standard models.
  • Real-World Impact: Online A/B tests at Baidu showed a 6.5% improvement in GSB (Good vs. Same vs. Bad) scores for long-tail queries, confirming the model's ability to handle niche search intents.

Conclusion

GS²P demonstrates that embracing model complexity through over-parameterization and generative pre-training can overcome traditional LTR limitations. By successfully navigating the double descent curve, the framework provides a robust solution for web-scale search ranking. Future work will focus on end-to-end approaches incorporating raw multimodal data directly into the pre-training phase.

Summary updated

BibTeX
@inproceedings{li2023gs,
  author = {Yuchen Li and Haoyi Xiong and Linghe Kong and Jiang Bian and Shuaiqiang Wang and Guihai Chen and Dawei Yin},
  title = {{GS2P: a generative pre-trained learning to rank model with over-parameterization for web-scale search}},
  booktitle = {DSAA},
  year = {2023},
  doi = {10.1109/dsaa60987.2023.10302487},
}
Figure