JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in LLMs

Michael K. Chen, Xikun Zhang, Dacheng TaoOpenReview 2025 · Submission to NeurIPS 2025 Datasets and Benchmarks Track · Rejected · OpenReview

Abstract

Logical reasoning is a critical component of Large Language Models (LLMs), and substantial research efforts in recent years have aimed to enhance their deductive reasoning abilities. However, existing deductive reasoning benchmarks, which are crucial for evaluating and advancing LLMs, suffer from fundamental limitations that severely restrict their utility, i.e., the lack of task complexity, the presence of prior knowledge as a confounder, and superficial error analysis. To address these deficiencies, we introduce JustLogic, a synthetically generated benchmark designed for rigorous evaluation of LLMs. JustLogic is (i) highly complex, capable of generating a diverse range of linguistic patterns, vocabulary, and argument structures; (ii) prior knowledge independent, eliminating the advantage of models possessing prior knowledge and ensuring that only deductive reasoning is used to answer questions; and (iii) capable of in-depth error analysis on the heterogeneous effects of reasoning depth and argument form on model accuracy. Our experimental results on JustLogic reveal that (i) state-of-the-art (SOTA) reasoning LLMs perform on par or better than the human average but significantly worse than the human ceiling, and (ii) SOTA non-reasoning models still underperform the human average. All code and data are available at https://github.com/michaelchen-lab/JustLogic
BibTeX
@misc{chen2025justlogic,
  author = {Michael K. Chen and Xikun ZHANG and Dacheng Tao},
  title = {{JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in LLMs}},
  year = {2025},
  url = {https://openreview.net/forum?id=PK5py0eznd},
  howpublished = {OpenReview},
  note = {Submitted to NeurIPS 2025 Datasets and Benchmarks Track},
}
Figure