Abstract
We introduce SLR, an end-to-end framework for systematic evaluation and training of Large Language Models (LLMs) via Scalable Logical Reasoning. Given a user's task specification, SLR enables scalable, automated synthesis of inductive reasoning tasks with precisely controlled difficulty. For each task, SLR synthesizes (i) a latent ground-truth rule, (ii) an executable validation program used by a symbolic judge to deterministically verify model outputs, and (iii) an instruction prompt for the reasoning task. Using SLR, we create SLR-Bench, a benchmark comprising over 19k prompts spanning 20 curriculum levels that progressively increase in relational, arithmetic, and recursive complexity. Large-scale evaluation reveals that contemporary LLMs readily produce syntactically valid rules, yet often fail at correct logical inference. Recent reasoning LLMs do somewhat better, but incur substantial increases in test-time compute, sometimes exceeding 15k completion tokens. Finally, logic-tuning via SLR doubles Llama-3-8B accuracy on SLR-Bench, achieving parity with Gemini-Flash-Thinking at a fraction of computational cost. SLR is fully automated, requires no human annotation, ensures dataset novelty, and offers a scalable environment for probing and advancing LLMs' reasoning capabilities.
Introduction
While Large Language Models (LLMs) have demonstrated remarkable capabilities in deductive reasoning—solving problems where rules are explicitly provided—inductive reasoning remains a significant challenge. Inductive reasoning requires the model to infer general rules from specific examples, a core component of human intelligence. Current evaluation benchmarks for this capability often suffer from data leakage, subjective judging by other LLMs, and a lack of granularity in difficulty control.
To address these limitations, we present the Scalable Logical Reasoning (SLR) framework. SLR leverages symbolic logic to generate an infinite stream of unique, verifiable, and increasingly complex reasoning puzzles, providing a robust environment for both evaluation and training.
The SLR Framework
The SLR framework treats inductive reasoning through the lens of Inductive Logic Programming (ILP). It automates the entire pipeline from task specification to evaluation.
The framework consists of three primary stages:
- Task Specification: Users define a language $\mathcal{L}$ (vocabulary of predicates and constants) and a configuration $\Theta$ (rule length, sampling distributions, etc.).
- Automated Synthesis: The system automatically generates a latent ground-truth rule, a set of positive and negative examples, and an executable Prolog validation program.
- Symbolic Judging: Instead of relying on inconsistent LLM judges, SLR uses the generated Prolog program to deterministically verify if the model's hypothesized rule correctly classifies all provided examples. This provides a "verifiable reward" suitable for Reinforcement Learning (RL).
SLR-BENCH and Curriculum Learning
Using this framework, we developed SLR-BENCH, a massive benchmark consisting of 19,000 tasks organized into a 20-level curriculum. These levels scale across four tiers: Basic, Easy, Medium, and Hard. This structured approach allows for precise identification of the "complexity wall" where a model's logical reasoning fails.
To test generalization, we also introduced SLR-Homes, an out-of-distribution (OOD) dataset with entirely new vocabularies, and evaluated models on standard downstream benchmarks like MMLU, CLUTRR, and LogiQA.
Key Results and Insights
Our evaluation of modern LLMs reveals several critical insights:
- The Complexity Wall: There is a sharp performance cliff as logical complexity increases. Even advanced reasoning models like OpenAI’s o1 see significant accuracy drops at the "Medium" and "Hard" tiers.
- Syntax vs. Semantics: LLMs excel at generating syntactically correct Prolog code but frequently fail at the underlying logical inference, producing rules that are logically unsound despite looking correct.
- Curriculum Learning Efficacy: Training models on a structured path from simple to complex logic is significantly more effective than random difficulty exposure.
As shown in the results, fine-tuning a Llama-3-8B model using the SLR curriculum doubled its accuracy on SLR-BENCH, matching the performance of much larger models like Gemini-Flash-Thinking. Furthermore, these gains generalized to downstream benchmarks, including a +17 point improvement in MMLU High School Statistics.
Conclusion and Future Work
SLR provides a scalable, automated method for probing and advancing LLM reasoning. While currently focused on single-rule induction in clean symbolic environments, the authors identified failure modes like "Shortcut Behavior" (reward hacking) where models bypass logic by memorizing prompt constants. Future work will explore multi-rule systems and the incorporation of natural language noise to better simulate real-world data. By providing verifiable intelligence tests, SLR paves the way for training models that truly understand logical principles.
Summary updated