Introduction: The "Shortcut Reasoning" Problem
In recent years, Large Language Models (LLMs) have achieved scores on logical reasoning benchmarks like ReClor and LogiQA that rival human performance. However, these high scores may be deceptive. Researchers suspect that models often employ "shortcut reasoning"—relying on superficial statistical patterns in the text to identify the correct answer without genuinely understanding the underlying logical structure.
Standard multiple-choice evaluations only measure what a model chooses, failing to probe why it made that choice. To exhibit true critical reasoning, a system must not only identify the correct conclusion but also understand the specific rationales that make alternative options incorrect. This paper introduces RULE (Rationale Understanding for Logical Reasoning Evaluation), a benchmark designed to probe this "elimination process" and expose the gap between superficial performance and deep logical understanding.
Methodology: The RULE Dataset
The authors developed a rigorous four-stage pipeline to construct the RULE dataset, starting from 1,200 questions in the ReClor dataset.
- Rationale Writing: Qualified crowdworkers wrote free-text rationales for every option—one explaining why the correct answer is right (Selective), and three explaining why each distractor is wrong (Eliminative).
- Rationale Validation: A separate group of workers performed alignment tests to ensure rationales accurately matched their respective options.
- Sub-question Generation: Using validated rationales, the researchers employed InstructGPT to generate multiple-choice "sub-questions." These questions ask the model to identify the logical basis for accepting or rejecting specific options.
- Human Validation: The final dataset consists of 3,003 human-verified sub-questions associated with 943 main questions.
The primary metric used is Consistency, which requires a model to answer both the main question and all its associated sub-questions correctly. This ensures the model isn't just guessing the main answer correctly while failing to understand the underlying logic.
Key Results: The Reasoning Gap
The evaluation of state-of-the-art models, including LLaMA 2, InstructGPT, and Flan-UL2, reveals a significant "reasoning gap" between AI and humans.
The "Elimination" Blind Spot
Models perform significantly worse on "Eliminative" sub-questions (explaining why distractors are wrong) compared to "Selective" sub-questions (explaining why the correct answer is right). While models can often identify the correct answer, they struggle to articulate the specific logical flaws in "near-miss" distractors.
Right for the Wrong Reasons
The most striking finding is the low consistency scores. Even the best-performing models (e.g., InstructGPT) achieved consistency scores between 18.2% and 23.1%, whereas humans achieved 52.9%. This suggests that in the majority of cases where a model answers a main question correctly, it does so without a full grasp of the logical rationale.
Human vs. AI Rationales
Human-written rationales were found to be superior to model-generated ones in 70% of cases. Human explanations are typically more detailed and better at capturing the implicit logic—the "warrants"—that are not explicitly stated in the text but are crucial for the argument.
Conclusion and Future Directions
The RULE benchmark demonstrates that current LLMs still lag significantly behind humans in critical reasoning, particularly in the ability to eliminate incorrect options based on logical principles. The study highlights that high accuracy on standard benchmarks does not necessarily imply deep understanding.
Future work should focus on improving the ability of LLMs to generate and utilize high-quality, implicit rationales. Bridging this "reasoning gap" is essential for developing AI systems that can provide reliable and explainable logical deductions in complex real-world scenarios.
Summary updated