PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval

Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, Ji-Rong Wen

Abstract

Recently, dense passage retrieval has become a mainstream approach to finding relevant information in various natural language processing tasks. A number of studies have been devoted to improving the widely adopted dual-encoder architecture. However, most of the previous studies only consider query-centric similarity relation when learning the dual-encoder retriever. In order to capture more comprehensive similarity relations, we propose a novel approach that leverages both query-centric and PAssage-centric sImilarity Relations (called PAIR) for dense passage retrieval. To implement our approach, we make three major technical contributions by introducing formal formulations of the two kinds of similarity relations, generating high-quality pseudo labeled data via knowledge distillation, and designing an effective two-stage training procedure that incorporates passage-centric similarity relation constraint. Extensive experiments show that our approach significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions datasets.

Introduction

Dense Passage Retrieval (DPR) has emerged as a dominant paradigm in information retrieval, utilizing dual-encoders to map queries and passages into a shared vector space. Traditionally, these models are optimized using a query-centric similarity relation (QSR), ensuring a query $q$ is closer to its positive passage $p^+$ than to negative passages $p^-$. However, this approach often fails to adequately discriminate between positive passages and "hard negatives" that share semantic similarities.

Comparison of query-centric (a) and passage-centric (b) similarity relations.

To address this, we propose PAIR (PAssage-centric sImilarity Relations), which introduces a passage-centric perspective. By enforcing that a positive passage $p^+$ should be closer to its corresponding query $q$ than to any negative passage $p^-$, we create a more robust and discriminative embedding space.

Methodology

The PAIR framework introduces three primary innovations:

1. Passage-Centric Similarity Relation (PSR)

In addition to the standard query-centric loss, we introduce a loss function that treats the positive passage as the anchor. This constraint, $s(p^+, q) > s(p^+, p^-)$, ensures that the neighborhood around a positive passage is cleared of irrelevant content, effectively acting as a regularizer during training.

2. Shared Parameter Dual-Encoder

To ensure that passage-to-passage comparisons are semantically valid, PAIR utilizes a single encoder (based on ERNIE-2.0) with shared weights for both queries and passages. This ensures both modalities are projected into the exact same semantic manifold.

3. Knowledge Distillation and Two-Stage Training

Since high-quality labeled data is scarce, we employ a "Cross-Encoder" teacher to generate pseudo-labels for millions of unlabeled pairs. Training proceeds in two stages:

  • Pre-training: The dual-encoder is trained on pseudo-labeled data using both QSR and PSR losses.
  • Fine-tuning: The model is refined on the target task using only the QSR loss to align with the final retrieval objective.

Experimental Results

We evaluated PAIR on the MSMARCO and Natural Questions (NQ) benchmarks. MSMARCO contains approximately 500k training queries and 8.8 million passages, while NQ consists of roughly 58k queries and 21 million Wikipedia passages.

Performance comparison of PAIR against state-of-the-art models on MSMARCO and Natural Questions.

As shown in the results above, PAIR achieves new state-of-the-art performance. On MSMARCO, it reached an MRR@10 of 37.9, surpassing previous leaders like RocketQA. On NQ, it achieved a Top-5 retrieval accuracy of 74.9% and Top-100 accuracy of 89.1%, setting a new benchmark for the field.

Discussion and Conclusion

The success of PAIR highlights the importance of "symmetry" in retrieval: relevance should be a two-way street where a passage "prefers" its query over other passages. Ablation studies indicate that while PSR is vital for building a high-quality vector space during pre-training, task alignment is best served by focusing on query-centric objectives during final fine-tuning. Future work will explore more complex ranking functions and the application of PAIR to complex downstream tasks like multi-hop reasoning.

Summary updated

Abstract

Recently, dense passage retrieval has become a mainstream approach to finding relevant information in various natural language processing tasks. A number of studies have been devoted to improving the widely adopted dual-encoder architecture. However, most of the previous studies only consider query-centric similarity relation when learning the dual-encoder retriever. In order to capture more comprehensive similarity relations, we propose a novel approach that leverages both query-centric and PAssage-centric sImilarity Relations (called PAIR) for dense passage retrieval. To implement our approach, we make three major technical contributions by introducing formal formulations of the two kinds of similarity relations, generating high-quality pseudo labeled data via knowledge distillation, and designing an effective two-stage training procedure that incorporates passage-centric similarity relation constraint. Extensive experiments show that our approach significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions datasets.
BibTeX
@inproceedings{ren2021pair,
  author = {Ruiyang Ren and Shangwen Lv and Yingqi Qu and Jing Liu and Wayne Xin Zhao and Qiaoqiao She and Hua Wu and Haifeng Wang and Ji-Rong Wen},
  title = {{PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=HuOric-h3i},
}
Figure