A Unified Pretraining Framework for Passage Ranking and Expansion

Ming Yan, Chenliang Li, Bin Bi, Wei Wang, Songfang HuangAAAI 2021 · OpenReview

Abstract

Pretrained language models have recently advanced a wide range of natural language processing tasks. Nowadays, the application of pretrained language models to IR tasks has also achieved impressive results. Typical methods either directly apply a pretrained model to improve the re-ranking stage, or use it to conduct passage expansion and term weighting for first-stage retrieval. We observe that the passage ranking and passage expansion tasks share certain inherent relations, and can benefit from each other. Therefore, in this paper, we propose a general pretraining framework to enhance both tasks with Unified Encoder-Decoder networks (UED). The overall ranking framework consists of two parts in a cascade manner: (1) passage expansion with a pretraining-based query generation method; (2) re-ranking of passage candidates from a traditional retrieval method with a pretrained transformer encoder. Both the two parts are based on the same pretrained UED model, where we jointly train the passage ranking and query generation tasks for further improving the full ranking pipeline. An extensive set of experiments have been conducted on two large-scale passage retrieval datasets to demonstrate the state-of-the-art results of the proposed framework in both the first-stage retrieval and the final re-ranking. In addition, we successfully deploy the framework to our online production system, which can stably serve industrial applications with a request volume of up to 100 QPS in less than 300ms.

Introduction

Modern search engines typically utilize a two-stage pipeline to balance efficiency and accuracy. The first stage involves a fast retrieval of a few hundred candidate documents from a massive corpus, often using term-matching algorithms like BM25. The second stage employs a more complex re-ranking model, such as BERT, to precisely score these candidates.

However, this traditional approach faces two significant challenges. First is the vocabulary mismatch problem, where a user's query and a relevant document use different terms for the same concept (e.g., "car" vs. "automobile"). Passage expansion—generating potential queries to enrich the document's index—is a common solution. Second, most existing systems treat passage expansion and re-ranking as isolated tasks using separate models. Researchers from Alibaba Group argue that these tasks are inherently related and that a unified approach can exploit shared semantic knowledge more effectively.

Comparison between the prevalent separate-model framework and the proposed unified pretraining framework (UED).

The Unified Pretraining Framework (UED)

To address these limitations, the authors introduce the Unified Encoder-Decoder (UED) network. This framework leverages a single Transformer-based architecture to handle both passage expansion (via the decoder) and re-ranking (via the encoder).

Two-Stage Training Regimen

The UED model is developed through a sophisticated training process designed to capture deep semantic relationships:

  1. Unsupervised Pretraining:
    • Encoder Training: Beyond standard Masked Language Modeling (MLM), the authors introduce Sentence Relation Prediction (SRP). Unlike the basic "Next Sentence Prediction" in BERT, SRP requires the model to distinguish if segments are consecutive, reversed, or unrelated, forcing a more robust understanding of context.
  2. Decoder Training: The decoder is trained on Next Sentence Generation (NSG), learning to generate relevant text spans based on long-range context.
  3. Joint Fine-tuning: The model is fine-tuned on labeled search data using a joint training objective. The encoder learns to perform pointwise re-ranking while the decoder simultaneously learns to generate potential queries for passage expansion. This synergy allows the model to improve its ranking capability by understanding how queries are generated and vice versa.
The UED training pipeline, showing the two-stage pretraining (MLM, SRP, and NSG) followed by joint fine-tuning for re-ranking and query generation.

Experimental Results

The UED framework was evaluated on two major Information Retrieval benchmarks: MS MARCO Passage Retrieval and the TREC 2019 Deep Learning Track.

  • MS MARCO: The framework achieved an MRR@10 of 43.6, setting a new record at the time and outperforming standard BERT-Large implementations.
  • TREC 2019: The model reached an NDCG@10 of 77.2, a significant improvement over the previous state-of-the-art of 75.0.

These results confirm that the joint training of ranking and expansion tasks provides a substantial performance boost over using independent models.

Industrial Deployment and Optimization

For real-world application, latency is a critical factor. To deploy the UED framework within Alibaba's production systems, the researchers utilized knowledge distillation. They compressed the 12-layer encoder into a 6-layer "tiny" model. This optimization allowed the system to maintain high accuracy while serving up to 100 Queries Per Second (QPS) with a latency of less than 300ms.

Conclusion

The UED framework demonstrates that passage ranking and expansion are mutually beneficial tasks. By unifying them within a single encoder-decoder architecture and employing a specialized pretraining strategy, the researchers achieved state-of-the-art results that are both effective and efficient enough for large-scale industrial use. Future work may explore extending this unified approach to dense retrieval methods or refining the query generation process to further reduce noise in expanded terms.

Summary updated

Abstract

Pretrained language models have recently advanced a wide range of natural language processing tasks. Nowadays, the application of pretrained language models to IR tasks has also achieved impressive results. Typical methods either directly apply a pretrained model to improve the re-ranking stage, or use it to conduct passage expansion and term weighting for first-stage retrieval. We observe that the passage ranking and passage expansion tasks share certain inherent relations, and can benefit from each other. Therefore, in this paper, we propose a general pretraining framework to enhance both tasks with Unified Encoder-Decoder networks (UED). The overall ranking framework consists of two parts in a cascade manner: (1) passage expansion with a pretraining-based query generation method; (2) re-ranking of passage candidates from a traditional retrieval method with a pretrained transformer encoder. Both the two parts are based on the same pretrained UED model, where we jointly train the passage ranking and query generation tasks for further improving the full ranking pipeline. An extensive set of experiments have been conducted on two large-scale passage retrieval datasets to demonstrate the state-of-the-art results of the proposed framework in both the first-stage retrieval and the final re-ranking. In addition, we successfully deploy the framework to our online production system, which can stably serve industrial applications with a request volume of up to 100 QPS in less than 300ms.
BibTeX
@inproceedings{yan2021unified,
  author = {Ming Yan and Chenliang Li and Bin Bi and Wei Wang and Songfang Huang},
  title = {{A Unified Pretraining Framework for Passage Ranking and Expansion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=erR4vrxU3CY},
}
Figure