TL;DR
This paper investigates why large language models struggle with negation in Natural Language Inference (NLI) tasks. The authors propose using ELECTRA's Replaced Token Detection (RTD) objective with new pretraining datasets to improve negation understanding, showing significant gains on specific NLI benchmarks.
What Problem Does This Paper Address?
Despite impressive advancements, state-of-the-art language models still falter when interpreting negation cues in Natural Language Inference (NLI). This is a critical issue because negation fundamentally alters the meaning of a sentence, and its misinterpretation can lead to incorrect logical conclusions. Previous research has shown that negation is underrepresented in pretraining datasets, making it difficult for models to learn its nuances. While some methods have attempted to address this through finetuning or additional pretraining, they often face challenges like catastrophic forgetting or limited success on complex datasets like MNLI. This paper aims to understand the root causes of this struggle and propose a more effective pretraining strategy.
Methods
The core of this work lies in leveraging ELECTRA's Replaced Token Detection (RTD) objective for continued pretraining. Unlike BERT's Masked Language Modeling (MLM), ELECTRA's RTD trains a discriminator to distinguish between original tokens and those replaced by a generator. The authors hypothesize that this RTD approach can more directly teach models the semantic impact of negation.
They introduce two new datasets:
- Expanded NLI: This dataset combines existing NLI data with curated negation examples from RTE, MNLI, and SNLI, reformatted for ELECTRA's RTD task. It also includes large amounts of Wikipedia and Books corpus data to prevent overfitting.
- Expanded LAMA: This dataset is built upon the LAMA (Language Model Analysis) and Negated LAMA benchmarks, also adapted for RTD. It includes various configurations of negated and non-negated sentence pairs.
The pretraining process involves continuing ELECTRA-Small's discriminator training on these new datasets, specifically focusing on the RTD objective. This is followed by finetuning on standard NLI benchmarks.
Datasets
- Training Data:
- Expanded NLI (newly created)
- Expanded LAMA (newly created)
- Wikipedia and Books corpus (for preventing overfitting)
- Evaluation Data:
- RTE (Recognizing Textual Entailment) datasets
- MNLI (Multi-NLI)
- SNLI (Stanford NLI)
- Held-out negated data from Hossain et al. (2020)
Key Insights and Ideas
- RTD is more effective than MLM for negation: The paper suggests that ELECTRA's RTD objective, which forces the model to identify replaced tokens, is better suited for learning negation's impact than MLM, which primarily relies on co-occurrence. RTD can directly highlight how negation flips sentence polarity.
- Negation is a "hard" problem, but solvable with the right approach: While negation is notoriously difficult for LMs, the study shows that targeted pretraining with RTD can yield significant improvements, especially on benchmarks like Negated RTE.
- "Neutral" labels are a major hurdle in MNLI: A key finding is that MNLI's "neutral" label category significantly hinders models' ability to learn negation. When neutral examples are removed, even a baseline ELECTRA-Small model performs well on negated MNLI tasks.
- Data scarcity in SNLI is problematic: The extreme underrepresentation of negation in SNLI (1.19% of examples) makes it difficult for models to learn negation cues, leading to overfitting on non-negated patterns.
- Pretraining is better than finetuning for generalization: The authors advocate for continued pretraining over finetuning for negation learning, as finetuning can lead to catastrophic forgetting of general negation knowledge.
Key Results
- Negated RTE: Continued pretraining with Expanded LAMA led to a substantial 19.9% increase in accuracy on the Negated RTE evaluation dataset.
- Standard RTE: Pretraining with Expanded LAMA showed a modest +1.1% improvement on standard RTE.
- MNLI: Performance on MNLI (both standard and negated) saw little to no improvement with the proposed pretraining methods, attributed to the "neutral" label issue.
- SNLI: Negated SNLI performance dropped by 4.6% with Expanded LAMA pretraining, likely due to severe underrepresentation of negation cues and model overfitting.
- Manual analysis of MNLI: When neutral examples were removed from MNLI, models achieved high accuracy (e.g., 88.92% for Base on Negated MNLI without neutrals), demonstrating that the issue was specific to the neutral category.
Limitations and Future Work
- Limited scope of negation: The study primarily focused on "not," "never," and "n't," and verb/noun phrases containing them. Future work should incorporate a wider variety of negation cues, such as affixes (e.g., "-less").
- Dataset size imbalance: The significant difference in size between the Expanded NLI (1,448 examples) and Expanded LAMA (140,000 examples) datasets might be a confounding factor. Future work could create more comprehensive negated NLI examples.
- Neutral examples in MNLI: The success on MNLI after removing neutral examples highlights the need for future pretraining tasks that can effectively handle neutral-labeled NLI examples with negation.
- English-only datasets: The current datasets are English-specific. Future research should extend these to multiple languages to explore cross-lingual transfer of negation knowledge.
Summary updated