TL;DR
This paper investigates how Natural Language Inference (NLI) systems interpret the conjunction "and," specifically focusing on the pragmatic inference where "A and B" implies a temporal or causal relationship. The authors introduce a new dataset and find that current NLI models exhibit inconsistent and often non-human-like interpretations of "and," highlighting a gap between computational models and human reasoning.
What Problem Does This Paper Address?
Understanding language involves more than just literal meaning; it requires inferring implied information. This paper tackles the challenge of how NLI systems handle "pragmatic inferences," which are common but not explicitly stated meanings. A prime example is the conjunction "and." While logically "A and B" is commutative (meaning "B and A" is equivalent), humans often infer a temporal sequence (e.g., "John fixed the car and it started" implies John fixed it before it started) or a causal link ("He fell and broke his leg" implies he broke his leg because he fell). This phenomenon, known as "conjunction buttressing," is a type of Generalized Conversational Implicature (GCI). Existing NLI systems, trained on datasets like MNLI, may struggle to capture these nuanced pragmatic interpretations, potentially defaulting to a purely logical, commutative understanding. The paper aims to evaluate and understand this behavior by creating a targeted dataset and analyzing system performance.
Methods
The core of the methodology involves creating a new dataset specifically designed to test interpretations of "and" in NLI tasks. This dataset was constructed by:
- Extracting and Modifying Sentences: Starting with sentences that explicitly used temporal or causal connectives (like "before," "because"), these connectives were replaced with "and."
- Manual Annotation: The premise-hypothesis pairs were manually annotated with two label sets: one reflecting a pragmatic hypothesis (where "and" implies temporal succession or causality) and another reflecting a logical hypothesis (where "and" is treated as commutative).
- Controlling for Context: To ensure the systems weren't simply defaulting to a pragmatic interpretation, pairs were also created where "and" should not imply temporal succession (e.g., using static events or creating contradictory temporal relations like precedence).
- System Evaluation: Three Transformer-based NLI systems (BERT, RoBERTa, and DeBERTa), already fine-tuned on the large MNLI dataset, were evaluated on this new dataset without further training. This allows for an assessment of their generalization capabilities.
- MNLI Analysis: To investigate potential biases in the training data, the authors analyzed the MNLI dataset itself to see how often annotators used pragmatic vs. logical interpretations of "and."
- Human Behavioral Study: A subset of the new dataset was used in a behavioral study to compare system performance against actual human intuitions.
Datasets
- UkWaC: A large-scale, web-derived corpus used for extracting initial sentences for dataset creation.
- English Penn Discourse Treebank: Used to ensure the discourse function of extracted connectives was unambiguous.
- MultiNLI (MNLI): The primary dataset used for fine-tuning the NLI systems (BERT, RoBERTa, DeBERTa). The paper also includes an analysis of MNLI to understand annotator biases regarding "and."
- Newly Introduced Benchmark Dataset: A custom dataset of premise-hypothesis pairs specifically designed to evaluate pragmatic and logical interpretations of "and." It contains 3,470 instances across various interpretations (temporal succession, causal, temporal precedence, temporal synchronous, commutative dynamic, commutative static).
Key Insights and Ideas
- "And" is More Than Just Logic: The research strongly suggests that while formal logic treats "and" as commutative, human language use imbues it with rich pragmatic meaning, particularly temporal and causal relationships.
- NLI Systems Mimic Pragmatics, But Inconsistently: The tested NLI models show a tendency to generalize pragmatic interpretations of "and" (like temporal succession and causality) but do so inconsistently, often making inferences that contradict each other or human judgments.
- Training Data Matters (and Might Be Biased): The analysis of MNLI reveals that pragmatic interpretations of "and" are far more common among annotators than logical ones, potentially leading NLI systems to over-rely on these pragmatic patterns, even when inappropriate.
- Divergence from Human Intuition: A significant finding is the lack of correlation between the NLI systems' confidence scores for entailment and the proportion of human participants who interpret the sentences pragmatically. Systems are confident in their (often incorrect) pragmatic inferences, while humans show more variability.
Key Results
- High Pragmatic Accuracy, Low Logical Accuracy: For temporal succession and causal interpretations, NLI systems achieved high "pragmatic accuracy" (close to 1), indicating they correctly identified these pragmatic inferences. However, their "logical accuracy" for commutative interpretations was often low.
- Inconsistent Patterns: All three tested systems (BERT, RoBERTa, DeBERTa) exhibited inconsistent inference patterns, sometimes assigning different interpretations to the same pair of conjuncts.
- Temporal Precedence Ambiguity: Performance varied significantly across systems for temporal precedence, with some systems correctly identifying it and others contradicting it.
- MNLI Bias: The MNLI dataset analysis showed that pragmatic interpretations of "and" (especially temporal succession) were used by annotators in 26 out of 500 examples, while commutative interpretations were used in only 2.
- Human Agreement: In the behavioral study, human participants largely agreed with pragmatic interpretations for causal scenarios (over 0.8 proportion in most cases), but showed more variable responses for commutative scenarios.
Limitations and Future Work
- Limited Scope of "And": The study focuses on "and" and its pragmatic enrichment. Other types of implicatures and connectives are not covered.
- Dataset Specificity: The results are based on a specific dataset. It's unclear if these findings generalize to systems trained on other NLI datasets.
- Alternative Implicature Theories: The paper acknowledges that alternative accounts of implicatures exist (e.g., implicature strength varying with context), which could offer a more nuanced evaluation.
- Future Work: The authors advocate for stronger convergence between neural systems and theories of GCIs. They suggest incorporating information about stereotypical event chains (derived from texts) into systems to improve their understanding of pragmatic inferences. Further research could explore implicature strength and test systems on diverse NLI datasets.
Summary updated