TruthTeller: Annotating Predicate Truth

Amnon Lotan, Asher Stern, Ido DaganNAACL 2013

TL;DR

This paper introduces TRUTHTELLER, a novel tool and annotation scheme for identifying the truth value of predicates (actions or relations) within sentences. By integrating various semantic phenomena like negation, modality, and presupposition, TRUTHTELLER aims to provide a more nuanced understanding of textual inference, achieving high accuracy on its own.

What Problem Does This Paper Address?

Understanding the truthfulness of statements is fundamental to natural language understanding. While previous work has tackled aspects of this, such as negation or modality, they often did so in isolation or embedded within larger systems. This paper addresses the need for a unified approach to annotating predicate truth values, considering a broader range of semantic phenomena that influence entailment and inference. Specifically, it aims to capture whether a predicate's action is positively inferred (e.g., "Gal sold her shop"), negatively inferred (e.g., "Gal did not sell it"), or unknown. This is crucial for downstream NLP tasks like Question Answering, Information Extraction, and textual entailment.

Methods

TRUTHTELLER employs a novel algorithm that operates on dependency parse trees. The core of the method involves a multi-stage annotation process:

  1. Predicate Implication Signature (sig): Each predicate is assigned a signature based on its lexical properties, indicating how it entails or presupposes its complements in positive and negative contexts. This is done using extensive lexicons of implicative and factive verbs.
  2. Negation and Uncertainty (NU): Modifiers like "not," "never," "maybe," and "probably" are identified to determine if a predicate is negated or uncertain.
  3. Clause-Truth (CT): This annotation indicates whether the clause headed by a predicate is entailed, contradicted, or unknown relative to the sentence. It's calculated recursively, considering presupposition constructions (like adverbial clauses and WH-arguments) and relative clauses.
  4. Predicate Truth (PT): The final truth value of a predicate is determined by the binary product of its NU and CT values.

The system combines pattern-based annotation rules with a recursive algorithm inspired by natural logic.

Datasets

The paper evaluates TRUTHTELLER's accuracy on 50 sentences sampled from the RTE5 and RTE6 Textual Entailment datasets. These datasets are widely used for textual inference benchmarks. The authors created a gold standard by manually annotating 153 predicates across these sentences. No new datasets were introduced.

Key Insights and Ideas

  • Unified Semantic Phenomena: The paper highlights the importance of integrating various semantic phenomena (negation, modality, presupposition, implicativity) to accurately determine predicate truth values, a gap in previous research.
  • Recursive Truth Calculation: The recursive approach to calculating Clause-Truth (CT) based on governing predicates and clause structures is a key methodological innovation.
  • Lexicon-Driven Approach: The reliance on extensive lexicons for predicate implication signatures is crucial for the system's coverage and accuracy.
  • Handling Unknown Truth Values: TRUTHTELLER explicitly annotates "unknown" truth values, which constitute a significant portion of the gold standard and are often overlooked.

Key Results

TRUTHTELLER demonstrates high accuracy across its four annotation types:

  • Signature: 89.5% accuracy
  • NU: 98% accuracy
  • CT: 90.8% accuracy
  • PT: 89% accuracy

These results significantly outperform a simple "most frequent value" baseline for all annotation types. Manual error analysis suggests that many remaining errors are due to parser inaccuracies rather than the TRUTHTELLER algorithm itself. Precision for CT- and PT- annotations was measured at 78% and 83% respectively, with recall estimates of 47.5% for CT- and 74% for PT-.

Limitations and Future Work

The authors acknowledge that the system's performance is dependent on the accuracy of the underlying dependency parser. Future work includes integrating TRUTHTELLER into a state-of-the-art RTE system to assess its impact on end-to-end performance and exploring the incorporation of monotonicity calculations for more sophisticated inference. They also note the challenge of ensuring that other system components interact effectively with TRUTHTELLER's output without masking its contributions.

Summary updated

BibTeX
@inproceedings{lotan2013truthteller,
  author = {Amnon Lotan and Asher Stern and Ido Dagan},
  title = {{TruthTeller: Annotating Predicate Truth}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure