Introduction: The Logic Trap
As Large Language Models (LLMs) are increasingly tasked with complex reasoning, researchers have focused on improving their ability to follow formal logical rules. However, human reasoning is inherently "knowledge-informed"; we do not follow symbolic rules in a vacuum but rather check them against our understanding of the world.
This paper identifies a significant "blind spot" in current LLMs: a tendency to over-rigidly apply formal logic even when it leads to conclusions that contradict factual reality. For instance, consider the premises: "Anne is either in Stockholm or somewhere in Sweden. Anne is not in Sweden." While strict Disjunctive Syllogism dictates the conclusion "Anne is in Stockholm," a human recognizes this as impossible because Stockholm is located within Sweden. LLMs, however, often fall into this "logic trap."
The RULEBREAKERS Dataset
To systematically study this phenomenon, the authors introduced RULEBREAKERS, a large-scale dataset consisting of 25,600 instances. The dataset is built upon two fundamental logical rules:
- Modus Tollens (MT): If P, then not Q; Q; therefore, Not P.
- Disjunctive Syllogism (DS): P or Q; Not Q; therefore, P.
The content is drawn from WikiData (geographical pairs like country/city) and ConceptNet (categorical pairs like "bird/goose"). To ensure failures were due to reasoning rather than ignorance, the researchers pre-screened models to confirm they possessed the underlying factual knowledge before testing them on the logical application.
Methodology: Paired Evaluation
The core of the study relies on Paired Evaluation. Each "rulebreaker" instance (where logic leads to a factual contradiction) is paired with a "non-rulebreaker" instance (where logic and facts align). A model is only considered successful if it handles both scenarios correctly. This prevents models from appearing competent simply by blindly following rules or by guessing based on factual priors alone.
Key Results: A Significant Performance Gap
The results reveal a stark divergence between machine logic and human-like reasoning. While most models achieved near-perfect accuracy on non-rulebreaker tasks, their performance plummeted when faced with rulebreakers.
High-performing models like GPT-4o and Gemma-2-27B were ironically the most susceptible to these errors. They appear so well-trained to follow logical patterns that they consistently accept logically valid but factually absurd conclusions. In contrast, Meta-Llama-3-8B-Instruct emerged as the most "human-like," achieving a paired accuracy of approximately 0.6, suggesting a better balance between rule-following and knowledge integration.
Analysis: Why Models Fail
Mechanistic analysis using feature attribution and attention weights provided several insights into this failure mode:
- Attention Deficit: Models failing on rulebreakers tend to assign less attention to the specific premise that creates the factual contradiction, focusing instead on the "if-then" structural tokens.
- Familiarity Bias: Models are more likely to reason correctly when they have high internal probability (familiarity) with the entities involved. When a model is "unsure" about a fact, it defaults to rigid symbolic logic.
- Fine-Tuning Backfire: Interestingly, models specifically fine-tuned on logical reasoning datasets performed worse on RULEBREAKERS. This suggests that current training methods for "improving" reasoning may inadvertently strip away a model's ability to use context and common sense.
Conclusion
The RULEBREAKERS challenge serves as a vital reminder that true intelligence requires the ability to know when rules no longer apply. As the field moves toward more robust LLMs, training objectives must evolve to help models balance formal symbolic logic with dynamic, real-world knowledge. Without this balance, models risk following logic "off a cliff," producing outputs that are technically valid but practically nonsensical.
Summary updated