Abstract
Answer sentence selection has been widely adopted recently for benchmarking techniques in Question Answering. Previous proposals for the task are essentially general solutions taking the form of neural networks that measure semantic similarity. In contrast, the present paper describes a simple technique to take advantage of such general-purpose tools for dealing with questions and answer sentences without changing the base system. The technique involves replacing wh-words in input questions with a word denoting the prototype of all answers. These transformed questions are passed as input to an existing neural network built for measuring semantic similarity. This technique is evaluated on two different neural network architectures over two datasets: TrecQA and WikiQA. Results of our experiments show improvement in overall accuracy across most question types we are interested in: 'who', 'when' and 'where'-type questions.
Introduction
In the task of Answer Sentence Selection, a system must identify the correct sentence containing an answer from a list of candidates given a specific question. Modern approaches typically employ deep learning architectures, such as Convolutional Neural Networks (CNNs) or Long Short-Term Memory (LSTM) networks, to calculate the semantic similarity between the question and candidate sentences.
However, a significant challenge arises from the nature of "wh-words" (e.g., Who, When, Where). Most models utilize pre-trained word embeddings like GloVe, where word representations are derived from their contexts in large corpora. In these contexts, wh-words often function as relative pronouns rather than interrogatives, leading to "fuzzy" mathematical representations. This makes it difficult for a neural network to match a vague "Who" against a specific entity like a person's name in a declarative answer sentence.
Proposed Method: Answer Prototypes
The core contribution of this work is a "plug-and-play" pre-processing technique that replaces abstract wh-words with concrete "prototypes"—specific named entities that represent the ideal answer type. For instance, the question "Who is the mayor of Chicago?" is transformed into "David is the mayor of Chicago?" before being processed by the similarity model.
By substituting the wh-word with a concrete anchor like "David," the question's vector representation becomes more aligned with the declarative structure of potential answer sentences. This method is model-agnostic and can be applied to any existing architecture without retraining.
Prototype Selection Strategies
The researchers developed two primary methods for identifying optimal prototypes using the Stanford Named Entity Recognizer (NER):
- Short Answer-Based (R+S): This method identifies the entity in the training data that exhibits the highest average cosine similarity to all known correct short answers.
- Wh-word Substitution-Based (R+W): This approach tests various entities within the question and selects the one that maximizes the similarity score with the correct answer sentence across the training set.
Experimental Setup
The proposed technique was evaluated on two benchmark datasets:
- TrecQA: A large dataset widely used for answer selection.
- WikiQA: A dataset that includes specific short-answer labels, facilitating the testing of the R+S method.
The transformed questions were fed into two established neural architectures: the attention-based LSTM model by Tan et al. and the similarity cube model by Rao et al.
Results and Discussion
Experimental results demonstrate consistent performance gains across both datasets and architectures. On the TrecQA dataset, applying the method to the Rao et al. model improved the Mean Reciprocal Rank (MRR) from 0.849 to 0.870.
The impact was particularly pronounced for specific question types. On WikiQA, using the Tan et al. model, the MRR for 'who' questions increased significantly from 0.685 to 0.783. The 'where' and 'when' categories also saw notable improvements. The Wh-word Substitution-Based (R+W) method proved more stable across different datasets compared to the Short Answer-Based (R+S) approach.
Conclusion
This research highlights that simple linguistic transformations can significantly enhance the performance of complex neural models. By bridging the semantic gap between interrogative wh-words and declarative answer entities through prototype substitution, the system achieves higher accuracy in answer sentence selection without requiring architectural changes. Future work may explore extending this technique to more complex 'why' and 'how' questions and reducing dependency on external NER tools.
Summary updated