Introduction
While Large Language Models (LLMs) have demonstrated remarkable capabilities in knowledge extraction and natural language understanding, they frequently falter when encountering basic logical negation. Prompt-based methods, which leverage natural language templates to query Pre-trained Language Models (PLMs), are particularly susceptible to what researchers call logically negative bias. This bias leads models to treat the word "not" as a simple keyword for "false" or "contradiction" rather than a logical operator that inverts a truth value.
The Negation Bottleneck
The core issue is that PLMs often fail to process negation as a logical operation. Analysis across various first-order logic (FOL) expressions reveals a significant performance gap. While models maintain reasonable accuracy (around 65%) for conjunctions, disjunctions, and implications, their performance plummets to near-random levels (approximately 45%) when logical negation is introduced.
This suggests that models rely on spurious correlations: the presence of "not" is biased toward a "Contradiction" label, while its absence is biased toward "Entailment." This prevents prompt-based methods from achieving reliable results in complex reasoning tasks.
The NAND Framework
To address these shortcomings, the authors introduce NAND (Negation Augmenting and Negation Debiasing), a zero-shot, plug-and-play framework designed to "unlock" the latent reasoning capabilities of models like BERT and RoBERTa without requiring additional training.
The NAND framework consists of two primary components:
- Negation Augmenting (NA): For every proposition $x$ (e.g., "Is Bob green?"), the system generates its negation $\neg x$ ("Is Bob not green?"). By evaluating the model's response to both versions, the inherent keyword bias associated with "not" is effectively neutralized.
- Negation Debiasing (ND): In Open World Assumption (OWA) settings, where a statement can be True, False, or Neutral, the bias is more nuanced. NAND introduces an empirical offset ($\gamma$) to the "Neutral" label's probability, helping the model distinguish between logical falsehood and lack of provability.
Experimental Results
NAND was evaluated on three major benchmarks: RuleTaker, ProofWriter, and LogicNLI. The results demonstrate substantial improvements across the board:
- Accuracy Gains: On LogicNLI, accuracy rose from 52.1% to 61.4% for BERT. On ProofWriter, gains of 10 to 15 percentage points were observed.
- Consistency: Beyond raw accuracy, NAND significantly reduces the variance caused by different prompt templates, making prompt engineering more stable.
- Zero-Shot Power: In complex reasoning tasks (e.g., depth-5 inference), the zero-shot NAND method approached the performance of models specifically fine-tuned on the task data.
Conclusion
NAND provides a mathematically grounded and computationally efficient solution to a fundamental flaw in language model reasoning. By treating negation as a logical operation rather than a linguistic indicator, it enables more reliable and consistent performance in complex logical tasks. Future work may explore the application of NAND to even larger models like GPT-4 to see if similar debiasing benefits persist at scale.
Summary updated