Handling Biographical Questions with Implicature

Donghui Feng, Eduard H. HovyEMNLP 2005 · OpenReview

TL;DR

This paper introduces a "cascading guidance strategy" to improve question answering (QA) systems by specifically addressing questions with hidden assumptions or "implicatures." By first identifying if a question implies a non-existent answer or multiple answers, the system can avoid wasted effort and improve accuracy, particularly for biographical fact extraction.

What Problem Does This Paper Address?

Traditional question answering systems typically parse a question, search for relevant documents, and then extract an answer. This straightforward approach falters when questions contain implicatures – unstated assumptions that can lead to incorrect answers or wasted computational resources. For example, asking "When did Beethoven get married?" implies he did get married, leading a naive system to search for a date when the truth is he never married. Similarly, asking "Who was Ronald Reagan's wife?" might lead a system to only find Nancy Davis, missing the other correct answer, Jane Wyman, because the question implicitly assumes only one marriage. This paper tackles the challenge of identifying and handling these implicatures in biographical questions.

Methods

The core contribution is a novel cascading guidance strategy. This strategy is inserted before the main answer extraction process. It consists of two successive classification engines designed to:

  1. Identify potential "no answer" traps: Determine if the question's premise is false (e.g., Beethoven never married).
  2. Recognize multiple answers: Detect if a question implicitly assumes a single answer when multiple exist (e.g., Reagan's wives).

For feature extraction, the system queries a search engine (Google) for the person mentioned in the question and retrieves top N documents. It then identifies relevant sentences using keyword matching and extracts N-grams within a fixed-length window around the topic keywords. These N-grams, combined with their conditional probabilities and frequencies, form the features for the classifiers.

The paper evaluates four popular classification approaches for this cascading strategy:

  • Decision Tree: A simple, intuitive method that breaks down decisions based on N-gram features.
  • Naïve Bayes: Utilizes Bayes' rule for classification.
  • Support Vector Machine (SVM): Employs kernel tricks for maximum-margin hyperplanes.
  • Maximum Entropy (ME): Directly estimates posterior probabilities.

Datasets

The authors used two corpora of biographies:

  • infoplease.com: Approximately 24,975 biographies.
  • biography.com: Approximately 24,345 biographies.

From these, they manually extracted and categorized individuals based on whether they had zero, one, or multiple spouses/children. This data was used to create training and testing sets for the cascading classifiers. The paper does not introduce a new benchmark dataset but rather uses existing biographical data to test their novel approach.

Key Insights and Ideas

  • Implicatures are a significant challenge for QA: Traditional QA systems are ill-equipped to handle unstated assumptions, leading to errors and inefficiency.
  • A pre-processing "guidance" step is crucial: Identifying implicatures before attempting to find an answer is more effective than trying to correct errors post-extraction.
  • N-grams are effective features for implicature detection: The combination of N-gram probabilities and frequencies within a relevant window captures important linguistic patterns for identifying these hidden assumptions.
  • Trigram features offer a good balance: While 4-grams and higher can be too sparse, trigrams provide sufficient evidence for classification without excessive noise.
  • More data doesn't always mean better results for web-based QA: For web-sourced data, increasing the number of retrieved documents beyond a certain point (around 50-70) showed diminishing returns or even slight performance degradation, likely due to increased noise.

Key Results

The cascading guidance strategy, when integrated into the PowerBio QA system, demonstrated improved performance in handling biographical questions with implicatures.

  • Child Classification: Naïve Bayes achieved the highest precision at 83.00% for the first-level classification (has child or not) and 59.46% for the second-level (one or multiple children).
  • Spouse Classification: Naïve Bayes also performed well, with 78.72% precision for the first-level (has spouse or not) and 54.05% for the second-level (one or multiple spouses).
  • Feature Performance: Trigram features generally outperformed unigrams and bigrams, especially when considering a larger number of top N documents.
  • Qualitative Example: The system successfully identified that Sophia Smith "did not marry" (no spouse) and that John Ritter "had children" and "multiple values" (multiple children).

Limitations and Future Work

  • Handling Null Values: The paper acknowledges that explicitly stating negative facts (e.g., "never attended any school") is rare in text, making it harder to identify null values compared to positive facts. This is a significant challenge for certain types of biographical information.
  • Complex Reasoning: For some biographical facts, more complex reasoning and inference might be needed, potentially incorporating external knowledge or leveraging inherent relationships between facts (e.g., marriage and children, birth place and childhood home) for cross-validation.
  • Data Sparsity: While trigrams were effective, higher-order N-grams (4-grams and above) suffered from sparsity, limiting their usefulness.
  • Future Directions: The authors plan to investigate more complex reasoning, incorporate external knowledge, and explore cross-validation using biographical fact relationships. They also aim to address the difficulty of identifying null values in text.

Summary updated

Abstract

Traditional question answering systems adopt the following framework: parsing questions, searching for relevant documents, and identifying/generating answers. However, this framework does not work well for questions with hidden assumptions and implicatures. In this paper, we describe a novel idea, a cascading guidance strategy, which can not only identify potential traps in questions but further guide the answer extraction procedure by recognizing whether there are multiple answers for a question. This is the first attempt to solve implicature problem for complex QA in a cascading fashion using N-gram language models as features. We here investigate questions with implicatures related to biography facts in a web-based QA system, Power-Bio. We compare the performances of Decision Tree, Naive Bayes, SVM (Support Vector Machine), and ME (Maximum Entropy) classification methods. The integration of the cascading guidance strategy can help extract answers for questions with implicatures and produce satisfactory results in our experiments.
BibTeX
@inproceedings{feng2005handling,
  author = {Donghui Feng and Eduard H. Hovy},
  title = {{Handling Biographical Questions with Implicature}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
  url = {https://openreview.net/forum?id=HyWaTzbObH},
}

Loading related content...

Figure