Abstract
Presuppositions are implicit assumptions embedded within language that remain true even under negation. Identifying these nuances is critical in high-stakes environments like legal cross-examinations. This paper introduces the Presupposition as Natural Language Inference (PNLI) task and two new datasets, PECaN and PGen, derived from courtroom transcripts and augmented via rule-based and LLM-driven methods. Our evaluation shows that fine-tuned generative models like T5 significantly outperform discriminative models like BERT, achieving up to 93% accuracy in identifying presuppositions.
Introduction: The Challenge of Loaded Questions
In a courtroom, a lawyer might ask: "Who did you blame when your project failed?" This question contains two presuppositions: that you blamed someone and that your project failed. Unlike standard logical entailments, presuppositions are "taken-for-granted" information. If a witness answers "Nobody," they have implicitly conceded that the project failed.
Current Natural Language Processing (NLP) models are proficient at standard Natural Language Inference (NLI)—determining if a premise entails, contradicts, or is neutral toward a hypothesis. However, they often fail to capture the pragmatic subtext of presuppositions. This research addresses the lack of high-quality benchmarks for Presupposition Generation, a task essential for AI to truly comprehend human dialogue.
Methodology and Data Construction
The researchers introduced the Presupposition as Natural Language Inference (PNLI) task. To build the necessary datasets, they employed a multi-faceted approach:
- Manual Annotation: Experts labeled transcripts from actual legal cross-examinations.
- Rule-Based Extraction: Using the
spaCylibrary, the team identified linguistic "triggers"—specific words or structures that signal a presupposition. - LLM Prompting: GPT-3.0 was used to generate potential presuppositions, though it frequently conflated them with simple entailments.
- Data Augmentation: The team leveraged the "projection property"—the fact that presuppositions persist through negation or questioning—to expand the training set.
The PECaN and PGen Datasets
Two primary datasets were developed:
- PECaN (Presupposition, Entailment, Contradiction, and Neutral): A classification dataset extending the traditional NLI framework with a fourth "Presupposition" category.
- PGen (Presupposition Generation): A generative dataset where models must output all presuppositions carried by an input sentence.
Key Insights: Learning the Logic of Subtext
A major finding was the effectiveness of the "Projection" property for training. By creating adversarial examples where the sentence structure changes (e.g., negation) but the presupposition remains constant, models were forced to learn deeper semantic patterns rather than relying on surface-level cues.
However, the researchers noted that high lexical overlap between premises and presuppositions can be a "double-edged sword." While it aids accuracy, it may also allow models to use word-matching heuristics instead of true semantic understanding.
Results: T5’s Generative Advantage
The study compared two prominent architectures: BERT (encoder-only) and T5 (encoder-decoder). The results clearly favored the generative approach.
On the PNLI classification task, the fine-tuned T5 model achieved 84.35% overall accuracy, compared to BERT’s 71.85%. Most notably, T5 reached 93% accuracy when specifically identifying presuppositions, whereas BERT managed only 73%. In the generative PGen task, T5 demonstrated "emerging proficiency" with a ROUGE-L score of 45.62, producing grammatically sound and semantically valid presuppositions, even if the lists were not always exhaustive.
Limitations and Future Directions
Despite these gains, two primary challenges remain:
- Completeness: Models often fail to generate all possible presuppositions. In legal contexts, missing a single hidden assumption can have significant consequences.
- Evaluation Metrics: Standard metrics like ROUGE, which focus on word overlap, are insufficient for evaluating logical validity. The authors advocate for more semantically-aware metrics or human-in-the-loop evaluations.
Future research will explore the transferability of these courtroom-trained models to broader domains, such as general-purpose conversational AI and virtual assistants.
Summary updated