TL;DR
This research introduces a novel method for improving Community Question Answering (cQA) by automatically extracting and ranking query paraphrases. By leveraging search engine click logs, the authors identify which paraphrases lead to the best answers and train a machine learning model to predict the most effective search terms for a given information need.
Problem Statement: The Vocabulary Mismatch
Community Question Answering platforms like Yahoo! Answers and Quora are vast knowledge repositories, yet finding specific answers is often hindered by the "vocabulary mismatch" problem. A user's search query (e.g., "migraine headaches low iron") may not match the wording of the ideal answer (e.g., "Can low hemoglobin cause headaches?"). While previous work focused on generating paraphrases, this paper addresses the gap in ranking them: not all linguistically similar phrases are equally effective at retrieving high-quality results.
Methodology: A Multi-Stage Pipeline
The authors propose a system that ranks paraphrases based on their actual retrieval effectiveness, measured by Recall and Mean Reciprocal Rank (MRR).
The pipeline consists of four main stages:
- Paraphrase Extraction: Mining Yahoo! Search logs for (Query, Answer Title) pairs where users clicked on specific Yahoo! Answers pages. This creates natural clusters of paraphrases.
- Ground Truth Labeling: Indexing 32 million answers using Lucene and testing each paraphrase to calculate its actual Recall and MRR, turning the problem into a supervised learning task.
- Feature Engineering: Extracting syntactic (Bag-of-Words, POS tags), semantic (WordNet relations), and string similarity features.
- Learning to Rank: Training SVMRank models optimized for both Recall (finding any relevant answer) and MRR (finding the best answer quickly).
Datasets
The study utilized a massive real-world dataset from Yahoo!:
- Initial Collection: 155 million queries linked to 26 million pages.
- Final Corpus: 32 million answers across 6 million pages after filtering for English and removing noisy data.
- Evaluation: Specialized "Recall" and "MRR" collections were used for five-fold cross-validation.
Key Insights
- The Power of Proper Nouns: The presence of Proper Nouns (NNP) was the most impactful feature, showing a strong correlation ($r = 0.89$) with search effectiveness by narrowing the semantic range.
- Two-Step Retrieval: The findings suggest a hybrid strategy: using a Recall-optimized ranker to gather candidates, followed by an MRR-optimized ranker to surface the best answer.
- Specialization Matters: Performance improved significantly when models were trained on specific question types (e.g., "Who" vs. "When"). For example, "Where" questions saw a 27% improvement when using a specialized model.
Key Results
The proposed SVMRanker significantly outperformed Bag-of-Words (BoW) baselines across various metrics and question types.
The system achieved an 8.90% improvement in MRR and a 4.69% improvement in Recall over the baseline. While the system reached approximately 26% of the theoretical "Oracle" MRR, this represents a major advancement in handling the informal and often ungrammatical nature of web queries.
Limitations and Future Work
Current limitations include the sensitivity of NLP tools to the lowercase nature of search logs and difficulties in capturing verb-based nuances. Future research will explore "ensembles of experts" to route queries to specialized rankers and methods to better handle noise in cQA archives.
Summary updated