RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning

Jason L. Chan, Robert Gaizauskas, Zhixue Zhao

TL;DR

While researchers are increasingly using formal logic to "fix" LLM reasoning, this paper reveals a significant side effect: models are becoming over-rigid "logic-bots." The authors introduce RULEBREAKERS, a dataset showing that state-of-the-art models (including GPT-4o) often fail to reject conclusions that are logically valid but factually impossible, highlighting a growing divergence between symbolic logic and human-like common sense.

What Problem Does This Paper Address?

In the quest to make Large Language Models (LLMs) better at reasoning, many researchers have turned to formal logic—either by fine-tuning models on logical datasets or by using "neuro-symbolic" approaches that translate language into code for a solver.

However, formal logic is "content-blind." It only cares about the structure of an argument (e.g., "If P then Q; P; therefore Q"). Humans, by contrast, use "knowledge-informed" reasoning. If the premises of a logically valid argument contradict our basic understanding of the world, we don't just blindly accept the conclusion; we recognize the contradiction.

The authors argue that current LLMs are falling into a "logic trap." They are becoming so proficient at following symbolic rules that they ignore the factual contradictions right in front of them. This paper aims to quantify this "blind spot" and understand why it happens.

Methods

The researchers developed a systematic framework to generate reasoning problems where formal logic and world knowledge collide.

  1. Rulebreaker vs. Non-rulebreaker Pairs: They created pairs of problems with identical logical structures. In "non-rulebreakers," the facts and logic align. In "rulebreakers," the logical conclusion factually contradicts one of the premises (e.g., "Anne is in Stockholm or Sweden. Anne is not in Sweden. Therefore, Anne is in Stockholm"—logically sound, but factually impossible since Stockholm is in Sweden).
  2. Paired Accuracy: To pass, a model must correctly accept the conclusion in the non-rulebreaker case and reject it in the rulebreaker case. This prevents models from "cheating" by simply having a bias toward saying "Yes" or "No."
  3. Mechanistic Analysis: The authors didn't just look at the final answer. They used feature attribution (Input x Gradient) and attention weights to see if the models were actually "looking" at the factual premises. They also analyzed internal neuron activations to see if the models internally distinguish between these scenarios even when they output the wrong answer.

Datasets

The authors introduced RULEBREAKERS, a large-scale dataset of 25,600 instances.

  • Logical Structures: Based on Modus Tollens (MT) and Disjunctive Syllogism (DS).
  • Entity Sources: They pulled real-world data from WikiData (geographical pairs like city/country) and ConceptNet (categorical pairs like "trumpet/brass instrument").
  • Robustness: Each instance was permuted across 10 different phrasing variations (e.g., "Does the conclusion follow?" vs. "Can the conclusion be deduced?") to ensure results weren't just "prompt engineering" artifacts.

Key Insights and Ideas

  • The "Over-Generalization" Trap: High-performing models like GPT-4o and Gemma-2-27b are actually worse at this task because they are too biased toward rigid logical application. They achieve near-perfect scores on non-rulebreakers but fail rulebreakers nearly 100% of the time.
  • Familiarity Breeds Contempt (for Logic): Models are more likely to spot a rulebreaker if they are highly "familiar" with the entities involved (e.g., they spot the error more easily with "Paris/France" than with "Funafuti/Tuvalu").
  • Attention Blindness: Analysis showed that when models fail, they tend to assign significantly less attention to the "factual" premise that creates the contradiction, focusing instead on the "logical" template.
  • The Logic-Enhancement Backfire: The authors tested a model specifically fine-tuned on logic datasets. It performed worse on RULEBREAKERS than the base model, suggesting that current "logic-boosting" methods may actually be making LLMs less human-like in their reasoning.

Key Results

  • Mediocre Overall Performance: Most models struggled, with "Paired Accuracy" scores generally below 0.6.
  • Top Performer: Meta-Llama-3-8B-Instruct achieved the highest paired accuracy (~0.61), showing a better balance between logic and world knowledge than much larger models.
  • GPT-4o's Blind Spot: GPT-4o performed surprisingly poorly on certain subsets, scoring near 0.0022 on MT categorical-entity pairs. It was so intent on following the logical rule that it almost never rejected the factually impossible conclusion.
  • Latent Ability: Interestingly, models showed higher "confidence" (in terms of token probability) for correct answers in non-rulebreaker cases than for incorrect answers in rulebreaker cases, suggesting they "know" something is wrong internally, even if they can't express it in the final output.

Limitations and Future Work

  • Single-Step Reasoning: The dataset currently focuses on single-step logical inferences. Real-world human reasoning often involves complex, multi-step "rulebreaking" scenarios.
  • Template Constraints: While large, the dataset is still template-based. Future work could explore more "in-the-wild" natural language contradictions.
  • Balancing Act: The paper highlights a critical research direction: how do we build models that can follow formal logic when appropriate (like in math or coding) but switch to knowledge-informed reasoning for natural language?

Paper Citation: Chan, J., Gaizauskas, R., & Zhao, Z. (2025). RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning. Proceedings of the 42nd International Conference on Machine Learning (ICML).

Summary updated

Abstract

Formal logic enables computers to reason in natural language by representing sentences in symbolic forms and applying rules to derive conclusions. However, in what our study characterizes as "rulebreaker" scenarios, this method can lead to conclusions that are typically not inferred or accepted by humans given their common sense and factual knowledge. Inspired by works in cognitive science, we create RULEBREAKERS, the first dataset for rigorously evaluating the ability of large language models (LLMs) to recognize and respond to rulebreakers (versus non-rulebreakers) in a human-like manner. Evaluating seven LLMs, we find that most models, including GPT-4o, achieve mediocre accuracy on RULEBREAKERS and exhibit some tendency to over-rigidly apply logical rules unlike what is expected from typical human reasoners. Further analysis suggests that this apparent failure is potentially associated with the models' poor utilization of their world knowledge and their attention distribution patterns. Whilst revealing a limitation of current LLMs, our study also provides a timely counterbalance to a growing body of recent works that propose methods relying on formal logic to improve LLMs' general reasoning capabilities, highlighting their risk of further increasing divergence between LLMs and human-like reasoning.
BibTeX
@inproceedings{chan2025rulebreakers,
  author = {Jason L. Chan and Robert Gaizauskas and Zhixue Zhao},
  title = {{RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=uqpML2nbIz},
}
Figure