Abstract
Document reranking models, particularly those based on Pre-trained Language Models (PLMs) like BERT, often struggle to capture the deep semantic "bridge" between a specific query and a document. While "further pre-training" on domain-specific data can help, it is computationally expensive and focuses on internal document structure rather than query-document relationships. We propose the Self-supervised Joint Training Framework (SJTF), which introduces a Masked Query Prediction (MQP) task performed simultaneously with standard ranking fine-tuning. By using relevant documents as context to reconstruct masked query terms, SJTF enhances semantic dependency learning. Our framework achieves significant performance gains on MS MARCO and TREC Robust04 without increasing inference latency.
Introduction
In modern information retrieval, document reranking is the critical second stage where a model predicts a relevance score for a query-document pair. While BERT-based models have set new benchmarks, they face two primary hurdles: the high computational cost of domain-specific pre-training and a "context gap" where standard pre-training tasks do not explicitly model how a query relates to its target document.
Methodology: Self-supervised Joint Training
The core of our approach is the integration of a self-supervised task directly into the fine-tuning process.
Masked Query Prediction (MQP)
Unlike standard Masked Language Modeling (MLM) which masks document tokens, MQP masks a random word in the query. The model is then tasked with predicting this missing word using the positive (relevant) document as the primary context. This forces the encoder to learn that a relevant document must contain the semantic information necessary to reconstruct the query.
The SJTF Framework
SJTF performs MQP and the standard ranking task simultaneously. The total loss is defined as a weighted sum: $L = L_{rank} + \alpha \cdot L_{MQP}$, where $\alpha$ is a hyper-parameter controlling the influence of the self-supervised task.
The architecture utilizes a shared BERT-style encoder. During training, a lightweight linear decoder is used for MQP, which is discarded during inference to maintain efficiency. This ensures the model gets smarter during training but remains just as fast as a standard BERT model during actual use.
Experimental Evaluation
We evaluated SJTF on two major benchmarks: the MS MARCO Passage Ranking dataset (8.8 million passages) and TREC Robust04, a challenging news-based collection used to evaluate model generalization.
Key Results
SJTF consistently improved performance across various base architectures, including BERT-base, CEDR-KNRM, and PARADE.
On MS MARCO, BERT+SJTF achieved an MRR@10 of 0.362, outperforming both the standard BERT baseline (0.345) and the more expensive "further pre-trained" (FP) version (0.355). On TREC Robust04, the framework showed strong generalization, with CEDR-KNRM+SJTF reaching an NDCG@20 of 0.5565, a significant jump from the baseline's 0.5388.
Discussion and Insights
- Contextual Importance: Ablation studies revealed that using only positive documents for MQP is crucial. Including negative (irrelevant) documents introduces noise that degrades performance because they provide the wrong context for the query.
- Task Alignment: Joint training proves more effective than general domain adaptation (further pre-training). By aligning the self-supervised signal with the specific ranking task, the model learns more relevant semantic features efficiently.
- Efficiency: Because the MQP decoder is removed after training, SJTF provides "smarter" models with zero additional overhead during document retrieval.
Conclusion
The Self-supervised Joint Training Framework offers a powerful yet efficient method for enhancing document reranking. By leveraging the document as context for query reconstruction, models develop a deeper understanding of relevance. Future work will explore more sophisticated masking strategies and the application of this framework to cross-modal tasks, such as using images as context for caption prediction.
Summary updated