Introduction
Open-Domain Question Answering (ODQA) requires systems to retrieve relevant information from massive corpora, such as Wikipedia, to answer user queries. Modern approaches typically employ Dense Passage Retrieval (DPR), utilizing dual-encoders to map questions and passages into a shared vector space. The effectiveness of these models hinges on contrastive learning, which requires comparing a positive passage against a large set of negative samples.
While Momentum Contrast (MoCo) has been highly successful in computer vision for maintaining a large queue of negatives, its application to ODQA is non-trivial due to the inherent asymmetry between questions and passages. In standard MoCo, only one encoder receives direct gradient updates, which in a QA setting leads to "passage insensitivity" and poor retrieval performance.
The xMoCo Framework
To address these challenges, the authors propose xMoCo (Cross Momentum Contrastive Learning). This architecture introduces a symmetric "cross" design to handle the distinct nature of questions and passages.
The xMoCo framework utilizes two pairs of encoders:
- Question Encoders: A "fast" encoder ($E_q^{fast}$) updated via gradients and a "slow" encoder ($E_q^{slow}$) updated through a momentum-based moving average.
- Passage Encoders: A corresponding fast/slow pair ($E_p^{fast}, E_p^{slow}$) for passages.
The "secret sauce" of xMoCo is its Cross-Matching Objective. Instead of a single matching task, the model jointly optimizes two directions: matching fast-encoded questions against a queue of slow-encoded passages ($L_{qp}$), and matching fast-encoded passages against a queue of slow-encoded questions ($L_{pq}$). This dual optimization ensures that gradients flow into both the question and passage fast encoders, preventing the "drift" seen in standard MoCo.
Experimental Results
The researchers evaluated xMoCo on five standard benchmarks: Natural Questions (NQ), TriviaQA, WebQuestions (WQ), CuratedTREC, and SQuAD v1.1. The model consistently outperformed the standard DPR baseline.
Key results include:
- Superior Accuracy: On Natural Questions, xMoCo achieved a Top-20 accuracy of 82.3%, a significant jump from DPR's 78.6%.
- Scaling Negatives: xMoCo efficiently maintains a negative pool of up to 16,384 samples. Performance generally improves with queue size, though it eventually plateaus as the "staleness" of older vectors begins to impact learning.
- Inference Efficiency: A major advantage of xMoCo is that the "slow" encoders are discarded after training. The final deployed model remains a standard dual-encoder, incurring no additional computational cost during inference.
Discussion and Future Work
Ablation studies confirmed that maintaining separate encoders for questions and passages is critical; "tying" the encoders (using the same model for both) resulted in a performance drop, such as falling from 82.3% to 75.4% Top-20 accuracy on the NQ dataset.
While xMoCo provides a massive volume of negatives, future work could focus on improving the "hardness" of these samples. Combining xMoCo with BM25-based hard negative mining or implementing more sophisticated filtering to remove false negatives (where a "negative" in the queue is actually a valid answer) are promising avenues for further boosting retrieval performance.
Summary updated