RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering

Srinivasan Iyer, Sewon Min, Yashar Mehdad, Wen-tau YihNAACL 2021 · OpenReview

Abstract: Srinivasan Iyer, Sewon Min, Yashar Mehdad, Wen-tau Yih. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2021.

Introduction

Open-Domain Question Answering (ODQA) systems typically rely on a "Retrieve-then-Read" pipeline, where a retriever identifies relevant documents and a reader extracts the answer. However, these systems often struggle with precision when multiple semantically similar candidates appear in the retrieved text. This paper introduces RECONSIDER, a re-ranking framework that employs span-focused cross-attention to significantly improve answer selection accuracy.

The Precision Bottleneck in ODQA

While modern Machine Reading Comprehension (MRC) models are effective at identifying the general vicinity of an answer, they frequently fail to distinguish the exact correct span among several plausible candidates. For instance, a question about the collapse of the Soviet Union might retrieve passages mentioning both Mikhail Gorbachev and Boris Yeltsin. A standard reader might incorrectly prioritize a high-confidence but contextually incorrect name.

RECONSIDER resolves ambiguity by focusing on specific spans within retrieved passages.

RECONSIDER addresses this by adding a "second look" phase that re-evaluates the top candidates identified by the base model, ensuring the final answer aligns more precisely with the question's constraints.

Methodology

The core of RECONSIDER is a re-ranking model that performs intensive evaluation of candidate spans through three primary innovations:

  1. Span-Focused Encoding: Instead of processing a raw passage, RECONSIDER receives a passage where the candidate span is explicitly marked with special tokens (e.g., [A] Mikhail Gorbachev [/A]). This forces the BERT-based cross-attention mechanism to concentrate its modeling capacity on the relationship between the question and that specific span.
  2. Coarse-to-Fine Training: Unlike models trained on random negatives, RECONSIDER is trained on "hard negatives"—the high-confidence errors made by the base MRC model. This specialized training allows the re-ranker to learn the subtle distinctions necessary for error correction.
  3. Efficient Cross-Attention: Although full cross-attention is computationally expensive, RECONSIDER maintains efficiency by only re-ranking the top 5–10 candidates. This adds minimal overhead to the overall inference time compared to the base model's initial processing.

Experimental Results

RECONSIDER was evaluated on four major benchmarks: Natural Questions (NQ), TriviaQA, WebQuestions (WebQ), and CuratedTREC. The model achieved new state-of-the-art (SOTA) results across all datasets.

Performance comparison of RECONSIDER against state-of-the-art models on four ODQA benchmarks.

Key findings include:

  • Natural Questions: Achieved 45.5% Exact Match (EM), surpassing the previous DPR baseline.
  • TriviaQA: Reached 61.7% EM, showing significant improvement over existing methods.
  • Comparison to Generative Models: RECONSIDER outperformed Retrieval-Augmented Generation (RAG) models, suggesting that improved extraction can be more effective than generation for many QA tasks.

Discussion and Limitations

The study highlights that marking the span is the most critical factor in the model's success, enabling "span-focused reasoning." However, limitations remain. The model's performance plateaus when re-ranking beyond the top 5 candidates, suggesting it cannot yet "rescue" answers ranked very low by the base model. Additionally, RECONSIDER occasionally demotes correct answers (approx. 4.3% of cases), and its extractive nature limits its application to tasks requiring multi-passage synthesis or generative reasoning.

Summary updated

BibTeX
@inproceedings{iyer2021reconsider,
  author = {Srinivasan Iyer and Sewon Min and Yashar Mehdad and Wen-tau Yih},
  title = {{RECONSIDER: Improved Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=W9vVEVuuAoGB},
}
Figure