TL;DR
This paper tackles the persistent challenge of Large Language Models (LLMs) understanding negation, a crucial aspect of logical reasoning. They introduce new multilingual datasets and demonstrate that larger LLMs significantly improve negation handling, contrary to some prior findings, and that this improvement is influenced by language and premise specificity.
What Problem Does This Paper Address?
Understanding negation is fundamental for accurate language comprehension and logical reasoning. However, LLMs often struggle with negation, leading to errors and "hallucinations." This "negation blindness" is a significant hurdle for their reliability, especially in nuanced communication and complex reasoning tasks. While previous work has explored this, the authors aim to delve deeper into the roots of this problem by investigating how model scale, language, and dataset characteristics influence LLM performance on negation.
Methods
The core of the methodology involves creating new datasets and evaluating existing LLMs.
- Dataset Creation: The authors extended existing Natural Language Inference (NLI) datasets (FEVER and SNLI) by creating paired hypotheses that differ only in the presence or absence of negation. This was done across four languages: English, Czech, German, and Ukrainian. They employed a rule-based Python module for initial negation generation, followed by manual verification to ensure correctness and handle edge cases.
- Model Evaluation: A suite of state-of-the-art LLMs (Llama 3, Qwen 2.5, and Mistral) with varying parameter sizes were evaluated on these new datasets. The evaluation focused on textual entailment, where models predict if a hypothesis logically follows from a premise. They specifically measured accuracy on hypotheses with and without negation to quantify the impact of negation.
Datasets
- NoFEVER-ML: A new multilingual dataset derived from FEVER, containing premise-hypothesis-negated_hypothesis triplets across English, Czech, German, and Ukrainian.
- NOSNLI-ML: A new multilingual dataset derived from SNLI, similarly structured with negated hypotheses across the four languages.
- FEVER-NLI & SNLI (Original): Used as the base for creating the new datasets.
Key Insights and Ideas
- Model Size Matters (A Lot): Contrary to some previous research suggesting model size doesn't help with negation, this study strongly indicates that larger LLMs are significantly better at handling negation. This is a crucial finding for future LLM development.
- Language Nuances: While LLMs perform better on English, the impact of negation varies across languages, suggesting that linguistic features like word order flexibility and phenomena like negative concord (in Slavic languages) play a role.
- Premise Specificity is Key: The authors found that datasets with more factually specific premises (like FEVER) lead to more robust negation handling compared to common-sense reasoning datasets (like SNLI). This hints that LLMs might rely on learned semantic patterns in common-sense tasks, making them more susceptible to negation errors.
- Negation is More Than Just a Token: The study implies that the challenge isn't just about recognizing the "not" token, but about the deeper semantic and logical implications of negation, which larger models seem better equipped to grasp.
Key Results
- Positive Correlation with Model Size: A strong positive correlation was found between model size and both relative accuracy change and negation sensitivity across datasets. Larger models consistently showed better performance and less accuracy drop when negation was introduced. For instance, the Spearman correlation coefficient for averaged relative accuracy change was 0.867.
- Language Differences: A statistically significant difference in relative accuracy change was observed across languages, with models performing worse on the Czech dataset.
- Dataset Impact: The FEVER dataset, with its factually specific premises, showed a stronger positive correlation between model size and negation handling compared to the SNLI dataset.
- Negation Sensitivity: Larger models were less likely to assign the same entailment label to pairs of hypotheses that differed only by negation, indicating improved robustness.
Limitations and Future Work
- Limitations:
- The study focuses on lexical negation (e.g., "not") and does not cover more complex forms like negation scope or double negation.
- The semi-automated translation process might introduce biases, although manual checks were performed.
- The study is limited to Germanic and Slavic languages, constraining generalizability.
- Future Work:
- Investigating which specific model architectures are responsible for negation handling.
- Using tools like LLM-microscope to identify tokens that play decisive roles in negation processing.
- Expanding the evaluation to more diverse languages and more complex negation phenomena.
Summary updated