Answering Complex Questions via Exploiting Social Q&A Collection

Youzheng Wu, Chiori Hori, Hisashi Kawai, Hideki KashiokaIJCNLP 2011 · OpenReview

This paper, presented at IJCNLP 2011, explores how the massive repositories of social Q&A sites like Yahoo! Answers and Baidu Zhidao can be mined to improve automated systems that answer complex, non-factoid questions.

The Challenge of Complex QA

While automated systems excel at answering "factoid" questions (e.g., "Who is the president of France?"), "complex" questions remain a significant hurdle. These questions require a list of information "nuggets" to provide a complete answer, such as "What are the hazards of global warming?" or "What are the functions of the UN?"

The primary bottleneck for complex Question Answering (QA) is the lack of labeled training data. Manually producing such data for every possible question type is prohibitively expensive. Social Q&A sites offer a vast alternative, containing millions of human-generated answers. However, this data is "noisy," characterized by slang, off-topic remarks, and inconsistent formatting. The authors propose a method to extract meaningful linguistic signals from this noise to improve answer recognition.

Methodology: Question-Type-Specific Method (QTSM)

The researchers developed the Question-Type-Specific Method (QTSM), which identifies "cue expressions"—phrases that signal high-quality answers for specific question types. The system was trained on approximately 40 million Q&A pairs crawled from Chinese social sites like Baidu Zhidao. The workflow involves:

  1. Identifying Question Focus: Using Conditional Random Fields (CRFs), the system identifies a "focus" word (e.g., "hazards," "reasons") that defines the question type.
  2. Bootstrapping Training Data: The system mines social Q&A pairs sharing the same or synonymous focuses (using the Hownet semantic knowledge base).
  3. Mining Cue Expressions: It extracts n-grams and dependency patterns frequent in "best answers" for a focus.
  4. Noise Filtering: An SVM classifier trained on these cues filters candidate sentences from a news corpus, discarding those that do not match the expected answer pattern for that question type.
Performance metrics (F3, Nugget Recall, Nugget Precision) improve as lexical and syntactic features are added to the model.

As shown in the ablation study above, performance increases significantly with the addition of bigrams and trigrams. However, the inclusion of complex syntactic dependency patterns yielded diminishing returns, likely due to the difficulty of learning clean patterns from noisy social media text.

Experimental Results

The authors compared QTSM against two baselines: the Question-Specific Method (QSM), which looks for nearly identical questions, and the Monolingual Translation-based Method (MTM), which uses statistical word associations.

QTSM outperformed both across multiple metrics. It achieved a 6.0% improvement in F3 score over QSM and a 5.8% improvement over MTM. In terms of "Nugget Recall" (the ability to find all components of a complex answer), QTSM reached 40.22%, compared to the baseline's 27.64%.

Comparison of QSM and QTSM performance across various question focuses, demonstrating QTSM's superior generalization.

The breakdown by question focus reveals that QTSM consistently provides better precision across a wide variety of topics, from "Hazard" to "Casualty." This suggests that treating the "focus" word as the question type is an effective way to scale QA systems without manual taxonomies.

Conclusion

The study demonstrates that social Q&A data is a superior resource to simple web snippets for training complex QA systems. By hitting the "sweet spot" of granularity—learning patterns for types of questions rather than specific questions or general word associations—the QTSM approach significantly advances the state of the art in non-factoid question answering.

Summary updated

BibTeX
@inproceedings{wu2011answering,
  author = {Youzheng Wu and Chiori Hori and Hisashi Kawai and Hideki Kashioka},
  title = {{Answering Complex Questions via Exploiting Social Q\&A Collection}},
  booktitle = {IJCNLP},
  year = {2011},
  url = {https://openreview.net/forum?id=oOj9I9fe6dA},
}
Figure