Introduction: The Lexical Gap in Answer Selection
Answer Selection (AS) is a critical component of modern question-answering systems. Given a question, the goal is to rank a set of candidate answers such that the most relevant ones appear at the top. Traditionally, models treat each candidate independently, calculating a similarity score between the question ($Q$) and a single candidate ($C$).
However, this independent paradigm suffers from a significant limitation known as the lexical gap. A correct answer might be phrased using terminology that differs from the question, making it difficult for a model to identify it in isolation. Conversely, multiple correct answers often share overlapping information. If a model can "remember" a previously identified easy-to-match answer, it can use that information as evidence to identify more subtly phrased correct answers.
The MS-Ranker Framework
To address the lexical gap, the authors propose MS-Ranker, a reinforcement learning (RL) based model that treats answer selection as a sequential decision task. Unlike traditional models, MS-Ranker accumulates evidence from previously examined candidates to inform the ranking of subsequent ones.
The system operates in two main stages:
- Pre-Ranker: A standard matching network provides an initial ranking. This ensures that high-confidence candidates (the strongest potential "evidence") are processed early in the sequence.
- RL Agent: This agent iterates through the ranked list, re-evaluating each candidate. It consists of a Matching Module—which uses both Question-Candidate (QC) and Evidence-Candidate (EC) attention—and an Evidence Module.
Evidence Accumulation and Gating
The core innovation is the Evidence Module, which acts as the system's memory. When the agent identifies a candidate as likely correct, a gating mechanism determines how much of that candidate's information should be integrated into a global evidence vector. This prevents incorrect candidates from "polluting" the memory, which would otherwise lead to a cascade of ranking errors.
The model is trained using the REINFORCE algorithm with a listwise reward based on the change in Average Precision (AveP). This directly optimizes the model for the final ranking objective rather than a proxy classification task.
Experimental Results
MS-Ranker was evaluated on two widely used benchmarks: WikiQA (clean data from Wikipedia) and SemEval-2016 CQA (noisy community forum data).
- State-of-the-Art Performance: MS-Ranker achieved top-tier results among models not relying on massive external knowledge bases. On SemEval-2016, it outperformed the previous best baseline (MVFNN) by 0.86% in MRR and 0.66% in MAP.
- Synergy with Transformers: The evidence accumulation strategy is highly complementary to pre-trained language models. When integrated with BERT-Large, MS-Ranker reached a MAP of 84.14 and MRR of 84.98, significantly boosting the performance of the base transformer model.
Discussion and Limitations
While MS-Ranker offers significant performance gains, it introduces a sequential bottleneck. Because the RL agent processes candidates one by one, inference time scales linearly with the number of candidates. Additionally, the model's success is heavily dependent on the Pre-Ranker; ablation studies showed a sharp decline in performance when the initial ordering was randomized, highlighting the importance of processing high-quality evidence early in the sequence.
Conclusion
MS-Ranker demonstrates that viewing candidate answers as sources of evidence rather than isolated targets can significantly improve ranking performance. By bridging the lexical gap through sequential decision-making and gated memory, it provides a robust framework for complex answer selection tasks. Future work may extend this "cross-candidate evidence" approach to broader retrieval and recommendation domains.
Summary updated