Enhancing Document Ranking with Task-adaptive Training and Segmented Token Recovery Mechanism
Lists
Only you see this
Abstract
In this paper, we propose a new ranking model DR-BERT, which improves the Document Retrieval (DR) task by a task-adaptive training process and a Segmented Token Recovery Mechanism (STRM). In the task-adaptive training, we first pre-train DR-BERT to be domain-adaptive and then make the two-phase fine-tuning. In the first-phase fine-tuning, the model learns query-document matching patterns regarding different query types in a pointwise way. Next, in the second-phase fine-tuning, the model learns document-level ranking features and ranks documents with regard to a given query in a listwise manner. Such pointwise plus listwise fine-tuning enables the model to minimize errors in the document ranking by incorporating ranking-specific supervisions. Meanwhile, the model derived from pointwise fine-tuning is also used to reduce noise in the training data of the listwise fine-tuning. On the other hand, we present STRM which can compute OOV word representation and contextualization more precisely in BERT-based models. As an effective strategy in DR-BERT, STRM improves the matching perfromance of OOV words between a query and a document. Notably, our DR-BERT model keeps in the top three on the MS MARCO leaderboard since May 20, 2020.
BibTeX
@inproceedings{sun2021enhancing,
author = {Xingwu Sun and Yanling Cui and Hongyin Tang and Fuzheng Zhang and Beihong Jin and Shi Wang},
title = {{Enhancing Document Ranking with Task-adaptive Training and Segmented Token Recovery Mechanism}},
booktitle = {Conference on Empirical Methods in Natural Language Processing},
year = {2021},
doi = {10.18653/v1/2021.emnlp-main.289},
}| Longformer for MS MARCO Document Re-ranking Task | arXiv · 2020 |
| Multi-Stage Document Ranking with BERT | arXiv · 2019 |
| Document Ranking with a Pretrained Sequence-to-Sequence Model | arXiv · 2020 |
| Learning to Rank in Generative Retrieval | arXiv · 2023 |
| Learning-to-Rank with BERT in TF-Ranking | arXiv · 2020 |
| Simple Applications of BERT for Ad Hoc Document Retrieval | arXiv · 2019 |