Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension

Akira Kawabata, Saku SugawaraEMNLP 2023 · DOI

Introduction: The "Shortcut Reasoning" Problem

In recent years, Large Language Models (LLMs) have achieved scores on logical reasoning benchmarks like ReClor and LogiQA that rival human performance. However, these high scores may be deceptive. Researchers suspect that models often employ "shortcut reasoning"—relying on superficial statistical patterns in the text to identify the correct answer without genuinely understanding the underlying logical structure.

Standard multiple-choice evaluations only measure what a model chooses, failing to probe why it made that choice. To exhibit true critical reasoning, a system must not only identify the correct conclusion but also understand the specific rationales that make alternative options incorrect. This paper introduces RULE (Rationale Understanding for Logical Reasoning Evaluation), a benchmark designed to probe this "elimination process" and expose the gap between superficial performance and deep logical understanding.

Methodology: The RULE Dataset

The authors developed a rigorous four-stage pipeline to construct the RULE dataset, starting from 1,200 questions in the ReClor dataset.

  1. Rationale Writing: Qualified crowdworkers wrote free-text rationales for every option—one explaining why the correct answer is right (Selective), and three explaining why each distractor is wrong (Eliminative).
  2. Rationale Validation: A separate group of workers performed alignment tests to ensure rationales accurately matched their respective options.
  3. Sub-question Generation: Using validated rationales, the researchers employed InstructGPT to generate multiple-choice "sub-questions." These questions ask the model to identify the logical basis for accepting or rejecting specific options.
  4. Human Validation: The final dataset consists of 3,003 human-verified sub-questions associated with 943 main questions.
Example of a main logical reasoning question and a corresponding sub-question from the RULE dataset.

The primary metric used is Consistency, which requires a model to answer both the main question and all its associated sub-questions correctly. This ensures the model isn't just guessing the main answer correctly while failing to understand the underlying logic.

Key Results: The Reasoning Gap

The evaluation of state-of-the-art models, including LLaMA 2, InstructGPT, and Flan-UL2, reveals a significant "reasoning gap" between AI and humans.

The "Elimination" Blind Spot

Models perform significantly worse on "Eliminative" sub-questions (explaining why distractors are wrong) compared to "Selective" sub-questions (explaining why the correct answer is right). While models can often identify the correct answer, they struggle to articulate the specific logical flaws in "near-miss" distractors.

Right for the Wrong Reasons

The most striking finding is the low consistency scores. Even the best-performing models (e.g., InstructGPT) achieved consistency scores between 18.2% and 23.1%, whereas humans achieved 52.9%. This suggests that in the majority of cases where a model answers a main question correctly, it does so without a full grasp of the logical rationale.

Performance comparison of various LLMs and humans on the RULE benchmark, highlighting the gap in consistency and eliminative reasoning.

Human vs. AI Rationales

Human-written rationales were found to be superior to model-generated ones in 70% of cases. Human explanations are typically more detailed and better at capturing the implicit logic—the "warrants"—that are not explicitly stated in the text but are crucial for the argument.

Conclusion and Future Directions

The RULE benchmark demonstrates that current LLMs still lag significantly behind humans in critical reasoning, particularly in the ability to eliminate incorrect options based on logical principles. The study highlights that high accuracy on standard benchmarks does not necessarily imply deep understanding.

Future work should focus on improving the ability of LLMs to generate and utilize high-quality, implicit rationales. Bridging this "reasoning gap" is essential for developing AI systems that can provide reliable and explainable logical deductions in complex real-world scenarios.

Summary updated

BibTeX
@inproceedings{kawabata2023evaluating,
  author = {Akira Kawabata and Saku Sugawara},
  title = {{Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.9},
}
Figure