MS-Ranker: Accumulating Evidence from Potentially Correct Candidates for Answer Selection

Yingxue Zhang, Fandong Meng, Peng Li, Ping Jian, Jie ZhouarXiv 2020 · arXiv

Introduction: The Lexical Gap in Answer Selection

Answer Selection (AS) is a critical component of modern question-answering systems. Given a question, the goal is to rank a set of candidate answers such that the most relevant ones appear at the top. Traditionally, models treat each candidate independently, calculating a similarity score between the question ($Q$) and a single candidate ($C$).

However, this independent paradigm suffers from a significant limitation known as the lexical gap. A correct answer might be phrased using terminology that differs from the question, making it difficult for a model to identify it in isolation. Conversely, multiple correct answers often share overlapping information. If a model can "remember" a previously identified easy-to-match answer, it can use that information as evidence to identify more subtly phrased correct answers.

An example of the lexical gap where Candidate C2 is relevant but lacks direct keyword overlap with the question.

The MS-Ranker Framework

To address the lexical gap, the authors propose MS-Ranker, a reinforcement learning (RL) based model that treats answer selection as a sequential decision task. Unlike traditional models, MS-Ranker accumulates evidence from previously examined candidates to inform the ranking of subsequent ones.

The system operates in two main stages:

  1. Pre-Ranker: A standard matching network provides an initial ranking. This ensures that high-confidence candidates (the strongest potential "evidence") are processed early in the sequence.
  2. RL Agent: This agent iterates through the ranked list, re-evaluating each candidate. It consists of a Matching Module—which uses both Question-Candidate (QC) and Evidence-Candidate (EC) attention—and an Evidence Module.
The sequential architecture of MS-Ranker, showing how evidence is accumulated across steps.

Evidence Accumulation and Gating

The core innovation is the Evidence Module, which acts as the system's memory. When the agent identifies a candidate as likely correct, a gating mechanism determines how much of that candidate's information should be integrated into a global evidence vector. This prevents incorrect candidates from "polluting" the memory, which would otherwise lead to a cascade of ranking errors.

The model is trained using the REINFORCE algorithm with a listwise reward based on the change in Average Precision (AveP). This directly optimizes the model for the final ranking objective rather than a proxy classification task.

Experimental Results

MS-Ranker was evaluated on two widely used benchmarks: WikiQA (clean data from Wikipedia) and SemEval-2016 CQA (noisy community forum data).

  • State-of-the-Art Performance: MS-Ranker achieved top-tier results among models not relying on massive external knowledge bases. On SemEval-2016, it outperformed the previous best baseline (MVFNN) by 0.86% in MRR and 0.66% in MAP.
  • Synergy with Transformers: The evidence accumulation strategy is highly complementary to pre-trained language models. When integrated with BERT-Large, MS-Ranker reached a MAP of 84.14 and MRR of 84.98, significantly boosting the performance of the base transformer model.

Discussion and Limitations

While MS-Ranker offers significant performance gains, it introduces a sequential bottleneck. Because the RL agent processes candidates one by one, inference time scales linearly with the number of candidates. Additionally, the model's success is heavily dependent on the Pre-Ranker; ablation studies showed a sharp decline in performance when the initial ordering was randomized, highlighting the importance of processing high-quality evidence early in the sequence.

Conclusion

MS-Ranker demonstrates that viewing candidate answers as sources of evidence rather than isolated targets can significantly improve ranking performance. By bridging the lexical gap through sequential decision-making and gated memory, it provides a robust framework for complex answer selection tasks. Future work may extend this "cross-candidate evidence" approach to broader retrieval and recommendation domains.

Summary updated

Abstract

As conventional answer selection (AS) methods generally match the question with each candidate answer independently, they suffer from the lack of matching information between the question and the candidate. To address this problem, we propose a novel reinforcement learning (RL) based multi-step ranking model, named MS-Ranker, which accumulates information from potentially correct candidate answers as extra evidence for matching the question with a candidate. In specific, we explicitly consider the potential correctness of candidates and update the evidence with a gating mechanism. Moreover, as we use a listwise ranking reward, our model learns to pay more attention to the overall performance. Experiments on two benchmarks, namely WikiQA and SemEval-2016 CQA, show that our model significantly outperforms existing methods that do not rely on external resources.
BibTeX
@misc{zhang2020ms,
  author = {Yingxue Zhang and Fandong Meng and Peng Li and Ping Jian and Jie Zhou},
  title = {{MS-Ranker: Accumulating Evidence from Potentially Correct Candidates for Answer Selection}},
  year = {2020},
  eprint = {2010.04970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.04970},
}
Figure