A Self-supervised Joint Training Framework for Document Reranking

Xiaozhi Zhu, Tianyong Hao, Sijie Cheng, Fu Lee Wang, Hai LiuNAACL 2022 · DOI

Abstract

Document reranking models, particularly those based on Pre-trained Language Models (PLMs) like BERT, often struggle to capture the deep semantic "bridge" between a specific query and a document. While "further pre-training" on domain-specific data can help, it is computationally expensive and focuses on internal document structure rather than query-document relationships. We propose the Self-supervised Joint Training Framework (SJTF), which introduces a Masked Query Prediction (MQP) task performed simultaneously with standard ranking fine-tuning. By using relevant documents as context to reconstruct masked query terms, SJTF enhances semantic dependency learning. Our framework achieves significant performance gains on MS MARCO and TREC Robust04 without increasing inference latency.

Introduction

In modern information retrieval, document reranking is the critical second stage where a model predicts a relevance score for a query-document pair. While BERT-based models have set new benchmarks, they face two primary hurdles: the high computational cost of domain-specific pre-training and a "context gap" where standard pre-training tasks do not explicitly model how a query relates to its target document.

Methodology: Self-supervised Joint Training

The core of our approach is the integration of a self-supervised task directly into the fine-tuning process.

Masked Query Prediction (MQP)

Unlike standard Masked Language Modeling (MLM) which masks document tokens, MQP masks a random word in the query. The model is then tasked with predicting this missing word using the positive (relevant) document as the primary context. This forces the encoder to learn that a relevant document must contain the semantic information necessary to reconstruct the query.

The SJTF Framework

SJTF performs MQP and the standard ranking task simultaneously. The total loss is defined as a weighted sum: $L = L_{rank} + \alpha \cdot L_{MQP}$, where $\alpha$ is a hyper-parameter controlling the influence of the self-supervised task.

The SJTF architecture: A shared encoder processes query-passage pairs for both relevance scoring and masked query prediction.

The architecture utilizes a shared BERT-style encoder. During training, a lightweight linear decoder is used for MQP, which is discarded during inference to maintain efficiency. This ensures the model gets smarter during training but remains just as fast as a standard BERT model during actual use.

Experimental Evaluation

We evaluated SJTF on two major benchmarks: the MS MARCO Passage Ranking dataset (8.8 million passages) and TREC Robust04, a challenging news-based collection used to evaluate model generalization.

Key Results

SJTF consistently improved performance across various base architectures, including BERT-base, CEDR-KNRM, and PARADE.

Performance comparison of SJTF against baseline models and further pre-training (FP) on MS MARCO and TREC Robust04.

On MS MARCO, BERT+SJTF achieved an MRR@10 of 0.362, outperforming both the standard BERT baseline (0.345) and the more expensive "further pre-trained" (FP) version (0.355). On TREC Robust04, the framework showed strong generalization, with CEDR-KNRM+SJTF reaching an NDCG@20 of 0.5565, a significant jump from the baseline's 0.5388.

Discussion and Insights

  • Contextual Importance: Ablation studies revealed that using only positive documents for MQP is crucial. Including negative (irrelevant) documents introduces noise that degrades performance because they provide the wrong context for the query.
  • Task Alignment: Joint training proves more effective than general domain adaptation (further pre-training). By aligning the self-supervised signal with the specific ranking task, the model learns more relevant semantic features efficiently.
  • Efficiency: Because the MQP decoder is removed after training, SJTF provides "smarter" models with zero additional overhead during document retrieval.

Conclusion

The Self-supervised Joint Training Framework offers a powerful yet efficient method for enhancing document reranking. By leveraging the document as context for query reconstruction, models develop a deeper understanding of relevance. Future work will explore more sophisticated masking strategies and the application of this framework to cross-modal tasks, such as using images as context for caption prediction.

Summary updated

BibTeX
@inproceedings{zhu2022self,
  author = {Xiaozhi Zhu and Tianyong Hao and Sijie Cheng and Fu Lee Wang and Hai Liu},
  title = {{A Self-supervised Joint Training Framework for Document Reranking}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.79},
}
Figure