Abstract
This paper introduces DR-BERT, a novel ranking model designed for large-scale document retrieval. DR-BERT addresses key challenges in applying BERT to ranking tasks, including the discrepancy between training objectives and ranking goals, semantic noise from sub-tokenization of out-of-vocabulary (OOV) words, and label noise in large datasets. The proposed solution features a multi-stage task-adaptive training pipeline—transitioning from pointwise to listwise fine-tuning—and a Segmented Token Recovery Mechanism (STRM) to preserve word-level semantics. Evaluated on the MS MARCO passage ranking benchmark, DR-BERT achieved an MRR@10 of 0.419, securing a top-3 position on the leaderboard.
Introduction
The advent of BERT has significantly advanced natural language processing, yet its direct application to document retrieval (DR) remains non-trivial. Standard BERT-based rankers often struggle with the "ranking gap," where models trained on pointwise or pairwise objectives fail to optimize for the final listwise ranking task. Furthermore, the use of WordPiece tokenization can lead to "OOV mismatching," where sub-tokens of unrelated words overlap, causing false positive matches. Finally, large-scale datasets like MS MARCO are inherently noisy, containing many unlabeled relevant documents. This paper presents DR-BERT, which integrates a two-phase training strategy and a specialized attention mechanism to overcome these hurdles.
Methodology
The DR-BERT framework relies on two primary innovations: Task-adaptive Training and the Segmented Token Recovery Mechanism (STRM).
Task-adaptive Training
The training process is structured into three distinct stages to progressively align the model with the retrieval task. First, domain-adaptive pre-training is performed on the MS MARCO corpus itself. Since the vocabulary distribution of web search queries differs significantly from Wikipedia (BERT's original training ground), this step helps the model adapt to the specific language of the task.
Following pre-training, the model enters a two-phase fine-tuning process. Phase 1 involves pointwise fine-tuning, where the model learns to predict the relevance of individual query-document pairs while considering query types (e.g., location vs. description). During this phase, a "Noise Reduction" step is implemented to identify and exclude "noisy" negative samples—documents labeled as negative but scored highly by the model. Phase 2 shifts to listwise fine-tuning, where the model is trained to rank a set of documents simultaneously, directly optimizing the ranking order.
Segmented Token Recovery Mechanism (STRM)
To address OOV mismatching, STRM modifies the standard self-attention mechanism. In traditional BERT, sub-tokens attend to each other without considering word boundaries. STRM enforces a hierarchy where the final sub-token of an OOV word acts as a representative. A custom masking matrix ensures that while this representative sub-token can perceive the full context, intermediate sub-tokens are restricted, thereby "recovering" the intended word-level meaning and preventing spurious matches based on coincidental sub-token overlaps (e.g., preventing "bogus" from matching a query for "bogue").
Evaluation
The model was rigorously tested on the MS MARCO (Microsoft Machine Reading Comprehension) passage ranking dataset, which comprises over 1 million queries and 8.8 million passages extracted from web documents. The primary metric for evaluation is Mean Reciprocal Rank at 10 (MRR@10).
Results
DR-BERT demonstrated state-of-the-art performance, achieving an MRR@10 of 0.419 on the official evaluation set. This result placed the model among the top 3 globally at the time of submission.
Ablation studies were conducted to quantify the contribution of each component. As shown in the results, the inclusion of STRM significantly boosted performance from the standard BERT baseline. The transition to listwise fine-tuning and the incorporation of query type features provided further incremental gains, confirming that the multi-stage approach is more effective than any single optimization alone.
Discussion and Conclusion
The success of DR-BERT highlights the importance of aligning training objectives with the final task and the need for robust handling of sub-token semantics. The listwise training approach proved superior to pointwise methods by enabling the model to learn comparative features across document sets. Additionally, STRM effectively mitigated the risks associated with WordPiece tokenization.
Future work may explore increasing the number of negative samples in the listwise phase, which was limited by hardware constraints in this study. Furthermore, automating the noise reduction threshold selection could enhance the model's robustness across different datasets.
Summary updated