Generating Discourse Inferences from Unscoped Episodic Logical Formulas

Gene Kim, Benjamin Kane, Viet Duong, Muskaan Mendiratta, Graeme McGuire, Sophie Sackstein, Georgiy Platonov, Lenhart SchubertACL 2019

TL;DR

This paper introduces a novel method for generating discourse inferences from Unscoped Episodic Logical Forms (ULFs), a semantic representation that bridges syntax and semantics. By leveraging a small set of interpretable inference rules and transformations, the system can automatically infer natural language statements from ULF representations, achieving a 68.5% acceptance rate in human evaluations.

What Problem Does This Paper Address?

Understanding and generating natural language requires inferring information that isn't explicitly stated. This is particularly challenging in conversational contexts where inferences are crucial for maintaining coherence and context. While previous work has explored logical inference, it often relies on fully resolved, canonicalized semantic representations that are distant from the surface language. This disconnect makes it difficult to directly map intuitions about language to inference rules and to evaluate inferences in a way that's natural for humans. The authors aim to bridge this gap by using Unscoped Episodic Logical Forms (ULFs), which retain more syntactic information and are closer to English, to generate discourse inferences.

Methods

The core of the paper lies in defining a set of inference rules and transformations that operate on ULFs.

  • Unscoped Episodic Logical Form (ULF): ULF is a semantic representation that captures predicate-argument structure but leaves operator scope, anaphora, and word sense unresolved. This "underspecification" is key, as it allows the representation to remain closer to the original sentence's syntax.
  • Interpretable Predicates and Transformations: The authors define high-level predicates (e.g., lex-pronoun?, request?) that capture semantic and syntactic features of ULFs. They also introduce named tree transformations (e.g., negate-vp!) that perform natural semantic modifications while respecting ULF's looser syntactic constraints.
  • Inference Rules: These are defined as if-then relations, where the antecedent uses the defined predicates and the consequent uses the transformations. The system uses TTT (a tree transduction tool) to implement these rules.
  • ULF-to-English Generation: A multi-stage pipeline translates inferred ULFs back into English sentences. This involves analyzing ULF types, incorporating morphological inflections, filtering logical operators, and mapping symbols to surface forms.
  • Rewriting Rules: To handle semantic equivalence, a rewriting module allows for alternative representations of the same meaning (e.g., clausal restructuring, merging inferences).

Datasets

The authors constructed a dataset for training and evaluation by combining several existing sources:

  • Tatoeba: A crowd-sourced translation dataset, providing fluent and varied sentences.
  • Discourse Graphbank: A corpus annotated for discourse structure, used for sentence delimiting.
  • Project Gutenberg: A repository of public domain texts, filtered for standard writing styles.
  • UIUC Question Classification: A dataset of questions from the TREC competition, used for question-based inferences.

To address the infrequency of certain inference types (like requests), they employed pattern-based filtering using augmented regular expressions to identify candidate sentences. The final annotated dataset consists of 698 inferences from 406 sentences, categorized into four types: clause-taking verbs, counterfactuals, requests, and questions.

Key Insights and Ideas

  • ULF's Sweet Spot: The underspecified nature of ULF proves beneficial for inference generation. By retaining syntactic structure, it allows for more direct mapping of linguistic intuitions to inference rules, unlike highly canonicalized representations.
  • Naturalness of Inferences: The paper emphasizes that the targeted inferences (requests, counterfactuals, etc.) are common and spontaneous in human conversation, highlighting the importance of this line of research for dialogue systems.
  • Bridging Logic and Language: The method demonstrates a practical way to bridge formal logic (Episodic Logic) with natural language processing, allowing for evaluation directly in English rather than through an abstract interface.
  • The Challenge of Ambiguity: The paper acknowledges that ULFs retain ambiguities (scope, anaphora, word sense). Inferences generated from them will also carry these ambiguities, requiring later resolution or tolerance.

Key Results

  • Automatic Evaluation: The system achieved a recall of 18% on a test set, with a maximum edit distance threshold of 3 for matching system-generated inferences to human-annotated gold inferences.
  • Human Evaluation: A majority of human judges found 68.5% of the generated inferences to be reasonable discourse inferences. 16.5% were deemed unreasonable, and 15% were context-dependent or had disagreements.
  • Grammaticality: The ULF-to-English translation system demonstrated strong grammaticality, with 78% of output sentences judged as grammatical by a majority of judges.
  • Rewriting Rule Validation: 91% of system-inferred sentences were judged as valid rewrites of raw inferences, indicating that the rewriting module does not introduce significant semantic changes.

Limitations and Future Work

  • Struggles with Certain Inference Types: The system showed particular difficulty with counterfactual and clause-taking verb constructions, attributed partly to the sampling procedures and the inherent ambiguity in human judgments for these cases.
  • Brittle Rules: Some inference rules were found to be brittle, especially when dealing with arbitrary terms instead of reified sentences or verbs in the antecedent.
  • Ambiguity Handling: The system does not resolve ambiguities inherent in ULF, which can affect inference accuracy and requires further handling.
  • Future Work: The authors suggest improving filtering for counterfactual and clause-taking verbs, gathering a larger and more robust dataset, and conducting further experiments with this larger dataset. They also note the need for more robust ULF macros and verb conjugations in the ULF-to-English system.

Summary updated

BibTeX
@inproceedings{kim2019generating,
  author = {Gene Kim and Benjamin Kane and Viet Duong and Muskaan Mendiratta and Graeme McGuire and Sophie Sackstein and Georgiy Platonov and Lenhart Schubert},
  title = {{Generating Discourse Inferences from Unscoped Episodic Logical Formulas}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}

Loading related content...

Figure