TL;DR
While researchers are increasingly using formal logic to "fix" LLM reasoning, this paper reveals a significant side effect: models are becoming over-rigid "logic-bots." The authors introduce RULEBREAKERS, a dataset showing that state-of-the-art models (including GPT-4o) often fail to reject conclusions that are logically valid but factually impossible, highlighting a growing divergence between symbolic logic and human-like common sense.
What Problem Does This Paper Address?
In the quest to make Large Language Models (LLMs) better at reasoning, many researchers have turned to formal logic—either by fine-tuning models on logical datasets or by using "neuro-symbolic" approaches that translate language into code for a solver.
However, formal logic is "content-blind." It only cares about the structure of an argument (e.g., "If P then Q; P; therefore Q"). Humans, by contrast, use "knowledge-informed" reasoning. If the premises of a logically valid argument contradict our basic understanding of the world, we don't just blindly accept the conclusion; we recognize the contradiction.
The authors argue that current LLMs are falling into a "logic trap." They are becoming so proficient at following symbolic rules that they ignore the factual contradictions right in front of them. This paper aims to quantify this "blind spot" and understand why it happens.
Methods
The researchers developed a systematic framework to generate reasoning problems where formal logic and world knowledge collide.
- Rulebreaker vs. Non-rulebreaker Pairs: They created pairs of problems with identical logical structures. In "non-rulebreakers," the facts and logic align. In "rulebreakers," the logical conclusion factually contradicts one of the premises (e.g., "Anne is in Stockholm or Sweden. Anne is not in Sweden. Therefore, Anne is in Stockholm"—logically sound, but factually impossible since Stockholm is in Sweden).
- Paired Accuracy: To pass, a model must correctly accept the conclusion in the non-rulebreaker case and reject it in the rulebreaker case. This prevents models from "cheating" by simply having a bias toward saying "Yes" or "No."
- Mechanistic Analysis: The authors didn't just look at the final answer. They used feature attribution (Input x Gradient) and attention weights to see if the models were actually "looking" at the factual premises. They also analyzed internal neuron activations to see if the models internally distinguish between these scenarios even when they output the wrong answer.
Datasets
The authors introduced RULEBREAKERS, a large-scale dataset of 25,600 instances.
- Logical Structures: Based on Modus Tollens (MT) and Disjunctive Syllogism (DS).
- Entity Sources: They pulled real-world data from WikiData (geographical pairs like city/country) and ConceptNet (categorical pairs like "trumpet/brass instrument").
- Robustness: Each instance was permuted across 10 different phrasing variations (e.g., "Does the conclusion follow?" vs. "Can the conclusion be deduced?") to ensure results weren't just "prompt engineering" artifacts.
Key Insights and Ideas
- The "Over-Generalization" Trap: High-performing models like GPT-4o and Gemma-2-27b are actually worse at this task because they are too biased toward rigid logical application. They achieve near-perfect scores on non-rulebreakers but fail rulebreakers nearly 100% of the time.
- Familiarity Breeds Contempt (for Logic): Models are more likely to spot a rulebreaker if they are highly "familiar" with the entities involved (e.g., they spot the error more easily with "Paris/France" than with "Funafuti/Tuvalu").
- Attention Blindness: Analysis showed that when models fail, they tend to assign significantly less attention to the "factual" premise that creates the contradiction, focusing instead on the "logical" template.
- The Logic-Enhancement Backfire: The authors tested a model specifically fine-tuned on logic datasets. It performed worse on RULEBREAKERS than the base model, suggesting that current "logic-boosting" methods may actually be making LLMs less human-like in their reasoning.
Key Results
- Mediocre Overall Performance: Most models struggled, with "Paired Accuracy" scores generally below 0.6.
- Top Performer: Meta-Llama-3-8B-Instruct achieved the highest paired accuracy (~0.61), showing a better balance between logic and world knowledge than much larger models.
- GPT-4o's Blind Spot: GPT-4o performed surprisingly poorly on certain subsets, scoring near 0.0022 on MT categorical-entity pairs. It was so intent on following the logical rule that it almost never rejected the factually impossible conclusion.
- Latent Ability: Interestingly, models showed higher "confidence" (in terms of token probability) for correct answers in non-rulebreaker cases than for incorrect answers in rulebreaker cases, suggesting they "know" something is wrong internally, even if they can't express it in the final output.
Limitations and Future Work
- Single-Step Reasoning: The dataset currently focuses on single-step logical inferences. Real-world human reasoning often involves complex, multi-step "rulebreaking" scenarios.
- Template Constraints: While large, the dataset is still template-based. Future work could explore more "in-the-wild" natural language contradictions.
- Balancing Act: The paper highlights a critical research direction: how do we build models that can follow formal logic when appropriate (like in math or coding) but switch to knowledge-informed reasoning for natural language?
Paper Citation: Chan, J., Gaizauskas, R., & Zhao, Z. (2025). RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning. Proceedings of the 42nd International Conference on Machine Learning (ICML).
Summary updated