TL;DR
This paper investigates how paraphrasing user questions can significantly improve the performance of Question Answering (QA) systems, which are often brittle to variations in phrasing. By leveraging machine translation to generate paraphrases and then using a learned model to select the best one, they demonstrate a potential 35% increase in Mean Reciprocal Rank (MRR).
What Problem Does This Paper Address?
Most current Question Answering (QA) systems are highly sensitive to the exact wording of a user's question. Even minor changes in phrasing can lead to drastically different results, sometimes returning incorrect answers or failing to find relevant information. This brittleness makes QA systems less reliable and user-friendly. The authors aim to address this by exploring whether automatically generating and selecting paraphrases of a user's original question can lead to more robust and accurate answers.
Methods
The core of the paper's methodology involves a three-stage process:
- Paraphrase Generation: They use machine translation (MT) engines (specifically Babelfish and Google MT) to translate questions into multiple intermediate languages and then back to English. This process generates semantically equivalent, but lexically and syntactically distinct, paraphrases of the original question.
- Feature Extraction: For each generated paraphrase, a set of features is computed by comparing it to the original question. These features include the sum of Inverse Document Frequency (IDF) scores of terms, the number of query terms, the IDF-weighted cosine distance between question vectors, and whether the predicted answer types overlap. These features aim to capture how well a paraphrase might match the QA system's capabilities.
- Paraphrase Selection: A supervised rule learner (JRip) is trained to classify paraphrases into "worse," "same," or "better" categories relative to the original question. To ensure high precision and avoid degrading performance, they employ a two-step cascade of binary classifiers with specific biases, and a strict confidence threshold for selecting a "better" paraphrase.
Datasets
- TREC QA track datasets (TREC 9 and 10): These datasets are used for training and evaluating the paraphrase selection module.
- AQUAINT corpus: This is the target corpus used by the PIQUANT QA system for evaluation.
Key Insights and Ideas
- QA systems are surprisingly brittle: The paper provides compelling examples of how even semantically identical questions can yield vastly different results from various QA systems, highlighting the problem's ubiquity.
- MT is a powerful tool for paraphrase generation: Leveraging existing MT engines, even with their inherent limitations, provides a readily available and effective way to generate a diverse set of paraphrases.
- Paraphrase selection is crucial: Simply generating many paraphrases isn't enough; a robust mechanism is needed to identify which paraphrases are likely to improve performance. The authors emphasize the need for high precision in this selection process.
- The "answer the question you wish they had asked" paradigm: The core idea is that by rephrasing a question in a way that better aligns with the QA system's strengths, one can effectively "ask the question you wish they had asked" to get a better answer.
Key Results
- Potential for significant improvement: An "oracle" run, which always picks the best paraphrase, shows a potential Mean Reciprocal Rank (MRR) of 0.48, compared to a baseline MRR of 0.345 on original questions. This represents a substantial potential increase.
- Learned selection yields marginal improvement: Their conservative paraphrase selection algorithm, trained using the described features, achieved an MRR of 0.347. While this is only a slight improvement over the baseline, it's important to note that only 2 out of 14 questions saw improved performance, with the rest staying the same. This highlights the difficulty of consistently finding "better" paraphrases.
- Random selection is detrimental: Randomly selecting paraphrases significantly decreased performance to an MRR of 0.156, underscoring the importance of the selection module.
Limitations and Future Work
- Low quality of paraphrases: While MT generates diverse paraphrases, many are ungrammatical or nonsensical, requiring filtering.
- Conservative selection: The authors acknowledge that their selection algorithm is conservative to avoid performance degradation. This means they might be missing opportunities to select truly "better" paraphrases.
- Future Work: The authors plan to develop more effective filtering techniques to reduce the candidate set, experiment with state-of-the-art paraphrasers, and explore using paraphrasing to improve the stability of QA systems.
Summary updated