xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question Answering

Nan Yang, Furu Wei, Binxing Jiao, Daxing Jiang, Linjun YangACL 2021 · DOI

Introduction

Open-Domain Question Answering (ODQA) requires systems to retrieve relevant information from massive corpora, such as Wikipedia, to answer user queries. Modern approaches typically employ Dense Passage Retrieval (DPR), utilizing dual-encoders to map questions and passages into a shared vector space. The effectiveness of these models hinges on contrastive learning, which requires comparing a positive passage against a large set of negative samples.

While Momentum Contrast (MoCo) has been highly successful in computer vision for maintaining a large queue of negatives, its application to ODQA is non-trivial due to the inherent asymmetry between questions and passages. In standard MoCo, only one encoder receives direct gradient updates, which in a QA setting leads to "passage insensitivity" and poor retrieval performance.

The xMoCo Framework

To address these challenges, the authors propose xMoCo (Cross Momentum Contrastive Learning). This architecture introduces a symmetric "cross" design to handle the distinct nature of questions and passages.

Comparison of the standard MoCo architecture (a) and the proposed xMoCo architecture (b), highlighting the dual fast/slow encoder pairs and cross-matching objective.

The xMoCo framework utilizes two pairs of encoders:

  1. Question Encoders: A "fast" encoder ($E_q^{fast}$) updated via gradients and a "slow" encoder ($E_q^{slow}$) updated through a momentum-based moving average.
  2. Passage Encoders: A corresponding fast/slow pair ($E_p^{fast}, E_p^{slow}$) for passages.

The "secret sauce" of xMoCo is its Cross-Matching Objective. Instead of a single matching task, the model jointly optimizes two directions: matching fast-encoded questions against a queue of slow-encoded passages ($L_{qp}$), and matching fast-encoded passages against a queue of slow-encoded questions ($L_{pq}$). This dual optimization ensures that gradients flow into both the question and passage fast encoders, preventing the "drift" seen in standard MoCo.

Experimental Results

The researchers evaluated xMoCo on five standard benchmarks: Natural Questions (NQ), TriviaQA, WebQuestions (WQ), CuratedTREC, and SQuAD v1.1. The model consistently outperformed the standard DPR baseline.

Retrieval performance (Top-20 and Top-100 accuracy) of xMoCo compared to BM25 and DPR across five benchmarks.

Key results include:

  • Superior Accuracy: On Natural Questions, xMoCo achieved a Top-20 accuracy of 82.3%, a significant jump from DPR's 78.6%.
  • Scaling Negatives: xMoCo efficiently maintains a negative pool of up to 16,384 samples. Performance generally improves with queue size, though it eventually plateaus as the "staleness" of older vectors begins to impact learning.
  • Inference Efficiency: A major advantage of xMoCo is that the "slow" encoders are discarded after training. The final deployed model remains a standard dual-encoder, incurring no additional computational cost during inference.

Discussion and Future Work

Ablation studies confirmed that maintaining separate encoders for questions and passages is critical; "tying" the encoders (using the same model for both) resulted in a performance drop, such as falling from 82.3% to 75.4% Top-20 accuracy on the NQ dataset.

While xMoCo provides a massive volume of negatives, future work could focus on improving the "hardness" of these samples. Combining xMoCo with BM25-based hard negative mining or implementing more sophisticated filtering to remove false negatives (where a "negative" in the queue is actually a valid answer) are promising avenues for further boosting retrieval performance.

Summary updated

BibTeX
@inproceedings{yang2021xmoco,
  author = {Nan Yang and Furu Wei and Binxing Jiao and Daxing Jiang and Linjun Yang},
  title = {{xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.477},
}
Figure