Enhancing Document Ranking with Task-adaptive Training and Segmented Token Recovery Mechanism

Xingwu Sun, Yanling Cui, Hongyin Tang, Fuzheng Zhang, Beihong Jin, Shi Wang

Abstract

This paper introduces DR-BERT, a novel ranking model designed for large-scale document retrieval. DR-BERT addresses key challenges in applying BERT to ranking tasks, including the discrepancy between training objectives and ranking goals, semantic noise from sub-tokenization of out-of-vocabulary (OOV) words, and label noise in large datasets. The proposed solution features a multi-stage task-adaptive training pipeline—transitioning from pointwise to listwise fine-tuning—and a Segmented Token Recovery Mechanism (STRM) to preserve word-level semantics. Evaluated on the MS MARCO passage ranking benchmark, DR-BERT achieved an MRR@10 of 0.419, securing a top-3 position on the leaderboard.

Introduction

The advent of BERT has significantly advanced natural language processing, yet its direct application to document retrieval (DR) remains non-trivial. Standard BERT-based rankers often struggle with the "ranking gap," where models trained on pointwise or pairwise objectives fail to optimize for the final listwise ranking task. Furthermore, the use of WordPiece tokenization can lead to "OOV mismatching," where sub-tokens of unrelated words overlap, causing false positive matches. Finally, large-scale datasets like MS MARCO are inherently noisy, containing many unlabeled relevant documents. This paper presents DR-BERT, which integrates a two-phase training strategy and a specialized attention mechanism to overcome these hurdles.

Methodology

The DR-BERT framework relies on two primary innovations: Task-adaptive Training and the Segmented Token Recovery Mechanism (STRM).

Task-adaptive Training

The training process is structured into three distinct stages to progressively align the model with the retrieval task. First, domain-adaptive pre-training is performed on the MS MARCO corpus itself. Since the vocabulary distribution of web search queries differs significantly from Wikipedia (BERT's original training ground), this step helps the model adapt to the specific language of the task.

Following pre-training, the model enters a two-phase fine-tuning process. Phase 1 involves pointwise fine-tuning, where the model learns to predict the relevance of individual query-document pairs while considering query types (e.g., location vs. description). During this phase, a "Noise Reduction" step is implemented to identify and exclude "noisy" negative samples—documents labeled as negative but scored highly by the model. Phase 2 shifts to listwise fine-tuning, where the model is trained to rank a set of documents simultaneously, directly optimizing the ranking order.

The two-phase task-adaptive training pipeline for DR-BERT, including the noise reduction step.

Segmented Token Recovery Mechanism (STRM)

To address OOV mismatching, STRM modifies the standard self-attention mechanism. In traditional BERT, sub-tokens attend to each other without considering word boundaries. STRM enforces a hierarchy where the final sub-token of an OOV word acts as a representative. A custom masking matrix ensures that while this representative sub-token can perceive the full context, intermediate sub-tokens are restricted, thereby "recovering" the intended word-level meaning and preventing spurious matches based on coincidental sub-token overlaps (e.g., preventing "bogus" from matching a query for "bogue").

Evaluation

The model was rigorously tested on the MS MARCO (Microsoft Machine Reading Comprehension) passage ranking dataset, which comprises over 1 million queries and 8.8 million passages extracted from web documents. The primary metric for evaluation is Mean Reciprocal Rank at 10 (MRR@10).

Results

DR-BERT demonstrated state-of-the-art performance, achieving an MRR@10 of 0.419 on the official evaluation set. This result placed the model among the top 3 globally at the time of submission.

Ablation studies were conducted to quantify the contribution of each component. As shown in the results, the inclusion of STRM significantly boosted performance from the standard BERT baseline. The transition to listwise fine-tuning and the incorporation of query type features provided further incremental gains, confirming that the multi-stage approach is more effective than any single optimization alone.

Ablation study results showing the impact of various components on the MRR@10 metric.

Discussion and Conclusion

The success of DR-BERT highlights the importance of aligning training objectives with the final task and the need for robust handling of sub-token semantics. The listwise training approach proved superior to pointwise methods by enabling the model to learn comparative features across document sets. Additionally, STRM effectively mitigated the risks associated with WordPiece tokenization.

Future work may explore increasing the number of negative samples in the listwise phase, which was limited by hardware constraints in this study. Furthermore, automating the noise reduction threshold selection could enhance the model's robustness across different datasets.

Summary updated

Abstract

In this paper, we propose a new ranking model DR-BERT, which improves the Document Retrieval (DR) task by a task-adaptive training process and a Segmented Token Recovery Mechanism (STRM). In the task-adaptive training, we first pre-train DR-BERT to be domain-adaptive and then make the two-phase fine-tuning. In the first-phase fine-tuning, the model learns query-document matching patterns regarding different query types in a pointwise way. Next, in the second-phase fine-tuning, the model learns document-level ranking features and ranks documents with regard to a given query in a listwise manner. Such pointwise plus listwise fine-tuning enables the model to minimize errors in the document ranking by incorporating ranking-specific supervisions. Meanwhile, the model derived from pointwise fine-tuning is also used to reduce noise in the training data of the listwise fine-tuning. On the other hand, we present STRM which can compute OOV word representation and contextualization more precisely in BERT-based models. As an effective strategy in DR-BERT, STRM improves the matching perfromance of OOV words between a query and a document. Notably, our DR-BERT model keeps in the top three on the MS MARCO leaderboard since May 20, 2020.
BibTeX
@inproceedings{sun2021enhancing,
  author = {Xingwu Sun and Yanling Cui and Hongyin Tang and Fuzheng Zhang and Beihong Jin and Shi Wang},
  title = {{Enhancing Document Ranking with Task-adaptive Training and Segmented Token Recovery Mechanism}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.289},
}
Figure