Abstract
Hiroshi Kanayama, Yusuke Miyao, John Prager. Proceedings of COLING 2012.
2012.
Introduction
Traditional Question Answering (QA) systems excel at "factoid" questions—queries seeking specific entities like "Who was the president of France in 2000?" However, yes/no questions, which require validating a proposition, present a significant challenge. Information sources like Wikipedia rarely contain explicit negative evidence (e.g., "Chirac was not the president of Germany"). This paper introduces Question Inversion, a method that transforms validation tasks into retrieval tasks by leveraging high-performance factoid QA systems.
Methodology: The Question Inversion Pipeline
The authors propose a pipeline that decomposes a declarative proposition into multiple factoid questions to verify its constituent facts.
- Question Conversion (Inversion): The system identifies "pivot terms" (typically proper nouns) and replaces them with abstract placeholders based on their category. For example, "Chirac was the president of Germany" generates "Who was the president of Germany?" (Pivot: Chirac) and "Chirac was the president of this country?" (Pivot: Germany).
- Factoid QA Subsystem: These questions are processed by IBM’s DeepQA engine (the architecture behind Watson), which returns candidate answers with confidence scores.
- Answer Aggregation: The system compares the "anticipated answer" (the original pivot term) with DeepQA's top results. A match with high confidence supports a "True" verdict, while a high-confidence mismatch (e.g., returning "France" instead of "Germany") suggests the proposition is "False."
- Reliability Scoring: Formulas are applied to penalize ambiguous or indirect questions, ensuring the final decision relies on the most robust evidence.
Key Insights: Triangulation and False Triggers
A core strength of this approach is triangulation. By generating multiple inversions for a single statement, the system increases the likelihood of uncovering discrepancies. If a statement is true, all pivots should ideally be recovered by the QA system. If false, at least one inversion is likely to reveal a "false trigger"—the specific incorrect detail in the proposition.
As shown in the table above, for the proposition regarding Xuanzang, different pivots ($q_1$ through $q_4$) allow the system to verify the country, the person, the destination, and the time period independently. This method is more practical than traditional Textual Entailment (RTE) because it does not require pre-selected "gold standard" text snippets for comparison.
Evaluation
The system was evaluated using World History exams from the Japanese National Center Test for University Admission. These exams are notoriously difficult because "wrong" options are designed to be plausible, often containing only one subtle error.
- Datasets: The 2009 exam (104 propositions) served as the development set, and the 2007 exam (92 propositions) as the test set.
- Performance: On the 2007 test set, the system achieved 79.3% accuracy in binary true/false classification. In the 4-way multiple-choice format of the actual exam, it reached 65% accuracy, significantly outperforming random guessing (25%) and previous RTE-based approaches (54%).
Limitations and Future Work
While successful, the system faced challenges with temporal reasoning (e.g., specific centuries) and semantic ambiguity (e.g., distinguishing between different "February Revolutions"). Future work aims to automate the "Question Inversion" step—which was partially manual in this study to establish performance ceilings—using NLP techniques like Wikipedia redirect mining to create a fully autonomous validation engine.
Summary updated