Who Did You Blame When Your Project Failed? Designing a Corpus for Presupposition Generation in Cross-Examination Dialogues

Maria Francis, Julius Steuer, Dietrich Klakow, Volha PetukhovaCOLING 2024

Abstract

Presuppositions are implicit assumptions embedded within language that remain true even under negation. Identifying these nuances is critical in high-stakes environments like legal cross-examinations. This paper introduces the Presupposition as Natural Language Inference (PNLI) task and two new datasets, PECaN and PGen, derived from courtroom transcripts and augmented via rule-based and LLM-driven methods. Our evaluation shows that fine-tuned generative models like T5 significantly outperform discriminative models like BERT, achieving up to 93% accuracy in identifying presuppositions.

Introduction: The Challenge of Loaded Questions

In a courtroom, a lawyer might ask: "Who did you blame when your project failed?" This question contains two presuppositions: that you blamed someone and that your project failed. Unlike standard logical entailments, presuppositions are "taken-for-granted" information. If a witness answers "Nobody," they have implicitly conceded that the project failed.

Current Natural Language Processing (NLP) models are proficient at standard Natural Language Inference (NLI)—determining if a premise entails, contradicts, or is neutral toward a hypothesis. However, they often fail to capture the pragmatic subtext of presuppositions. This research addresses the lack of high-quality benchmarks for Presupposition Generation, a task essential for AI to truly comprehend human dialogue.

Methodology and Data Construction

The researchers introduced the Presupposition as Natural Language Inference (PNLI) task. To build the necessary datasets, they employed a multi-faceted approach:

  1. Manual Annotation: Experts labeled transcripts from actual legal cross-examinations.
  2. Rule-Based Extraction: Using the spaCy library, the team identified linguistic "triggers"—specific words or structures that signal a presupposition.
  3. LLM Prompting: GPT-3.0 was used to generate potential presuppositions, though it frequently conflated them with simple entailments.
  4. Data Augmentation: The team leveraged the "projection property"—the fact that presuppositions persist through negation or questioning—to expand the training set.
Table of linguistic triggers used to identify potential presuppositions, including factive verbs and quantifiers.

The PECaN and PGen Datasets

Two primary datasets were developed:

  • PECaN (Presupposition, Entailment, Contradiction, and Neutral): A classification dataset extending the traditional NLI framework with a fourth "Presupposition" category.
  • PGen (Presupposition Generation): A generative dataset where models must output all presuppositions carried by an input sentence.

Key Insights: Learning the Logic of Subtext

A major finding was the effectiveness of the "Projection" property for training. By creating adversarial examples where the sentence structure changes (e.g., negation) but the presupposition remains constant, models were forced to learn deeper semantic patterns rather than relying on surface-level cues.

However, the researchers noted that high lexical overlap between premises and presuppositions can be a "double-edged sword." While it aids accuracy, it may also allow models to use word-matching heuristics instead of true semantic understanding.

Results: T5’s Generative Advantage

The study compared two prominent architectures: BERT (encoder-only) and T5 (encoder-decoder). The results clearly favored the generative approach.

Confusion matrices for BERT and T5 on the PNLI and NLI tasks, showing T5's superior performance in identifying presuppositions (P).

On the PNLI classification task, the fine-tuned T5 model achieved 84.35% overall accuracy, compared to BERT’s 71.85%. Most notably, T5 reached 93% accuracy when specifically identifying presuppositions, whereas BERT managed only 73%. In the generative PGen task, T5 demonstrated "emerging proficiency" with a ROUGE-L score of 45.62, producing grammatically sound and semantically valid presuppositions, even if the lists were not always exhaustive.

Limitations and Future Directions

Despite these gains, two primary challenges remain:

  1. Completeness: Models often fail to generate all possible presuppositions. In legal contexts, missing a single hidden assumption can have significant consequences.
  2. Evaluation Metrics: Standard metrics like ROUGE, which focus on word overlap, are insufficient for evaluating logical validity. The authors advocate for more semantically-aware metrics or human-in-the-loop evaluations.

Future research will explore the transferability of these courtroom-trained models to broader domains, such as general-purpose conversational AI and virtual assistants.

Summary updated

BibTeX
@inproceedings{francis2024who,
  author = {Maria Francis and Julius Steuer and Dietrich Klakow and Volha Petukhova},
  title = {{Who Did You Blame When Your Project Failed? Designing a Corpus for Presupposition Generation in Cross-Examination Dialogues}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure