TL;DR
The paper introduces SLR (Scalable Logical Reasoning), an open-source framework that automatically synthesizes inductive reasoning tasks complete with natural language prompts, latent ground-truth rules, and executable validation programs. By evaluating models on a new 19k-task curriculum benchmark (SLR-BENCH), the authors demonstrate that while current LLMs excel at syntax, they fail at complex logical induction. Crucially, they show that curriculum learning can double the accuracy of smaller models like Llama-3-8B, matching heavy-hitting reasoning models at a fraction of the computational cost.
What Problem Does This Paper Address?
Most existing LLM evaluation benchmarks focus heavily on deductive reasoning—asking models to draw specific conclusions from given premises (e.g., math word problems or logic puzzles). However, inductive reasoning—the ability to infer general rules or patterns from specific examples—remains highly underexplored.
Furthermore, current evaluation frameworks suffer from three major flaws:
- Subjectivity and Gaming: They often rely on multiple-choice formats or use other LLMs as judges, which can be easily gamed or misinterpret logical validity.
- Data Contamination: As training datasets grow, benchmarks increasingly overlap with pre-training data, turning true reasoning tasks into mere exercises in memorization.
- Lack of Granular Control: There is no easy way to precisely scale the logical complexity of a task to see exactly where an LLM's reasoning breaks down.
To bridge this gap, the authors developed SLR to automate the generation of unique, uncontaminated inductive logic tasks with verifiable, symbolic ground truths.
Methods
The SLR framework operates on the Learning-from-Entailment paradigm from Inductive Logic Programming (ILP). Given a user's high-level task specification, the pipeline automatically executes three main stages:
- Task Specification: Defines the vocabulary (predicates and constants) and grammar constraints.
- Automated Synthesis: A task synthesizer generates a latent ground-truth rule ($R^*$), samples background knowledge facts ($B$), and uses a Prolog engine to deterministically label examples as positive ($E^+$) or negative ($E^-$).
- Verifiable Logic Rewards (The Symbolic Judge): Instead of relying on exact text matching or an LLM judge, SLR uses a symbolic judge. It executes the LLM's generated hypothesis against a validation program. If the hypothesis correctly entails all positive examples and refutes all negative ones, it passes—allowing for semantically correct but syntactically unique solutions.
Datasets
The authors introduced two primary datasets using SLR:
- SLR-BENCH: A static benchmark of 19,000 tasks organized into a 20-level curriculum across four difficulty tiers (basic, easy, medium, hard). It scales complexity by adjusting rule length, problem size, and introducing advanced logical structures like recursion and arithmetic. It is inspired by the classic V-LoL trains domain.
- SLR-Homes: An out-of-distribution (OOD) dataset containing 500 house-classification tasks across 5 difficulty levels, used specifically to test cross-domain generalization.
Additionally, the authors evaluated their SLR-tuned models on several established downstream benchmarks, including MMLU, CLUTRR, LogiQA/LogiQA2, GPQA, ARC, and HellaSwag.
Key Insights and Ideas
- Syntax $\neq$ Semantics: Standard LLMs are highly proficient at generating syntactically valid Prolog code, but they consistently fail to capture the correct underlying logic as complexity scales.
- The Steep Cost of "Thinking": Scaling test-time compute (e.g., OpenAI's o1/o3 or DeepSeek-R1) significantly improves inductive reasoning performance. However, it hits a harsh plateau on "hard" tasks and incurs massive financial costs (exceeding $300 for just 1,000 prompts).
- Curriculum Learning is a Cheat Code: Rather than throwing massive compute at inference time, training an 8B model sequentially from basic to medium difficulty tiers allows it to outperform much larger models.
- Verbosity is a Warning Sign: An error analysis revealed a strong inverse correlation between output length and success for distilled reasoning models. When a model outputs excessively long reasoning traces, it is usually a sign of derailment rather than deeper thinking.
Key Results
The authors conducted a large-scale evaluation of state-of-the-art models on SLR-BENCH:
- Curriculum Fine-Tuning Breakthrough: Fine-tuning a Llama-3-8B model using SLR's curriculum doubled its Logical Reasoning Level (LRL) score from 3.8 to 8.7, achieving parity with Gemini-2.0-Flash-Thinking while using significantly fewer tokens.
- Downstream Generalization: The capabilities gained from SLR training successfully transferred to external datasets. The SLR-tuned Llama model saw an average accuracy boost of +6.6 percentage points on CLUTRR and +17 percentage points on MMLU High School Statistics.
- Hard Tier Defeat: Even the strongest frontier models struggled with the highest levels of logical complexity; OpenAI's o3 topped the leaderboard but only solved 45% of the tasks in the Hard tier.
Limitations and Future Work
While SLR provides a powerful framework for scaling logic evaluation, the authors note several limitations:
- Simplified Logic Domain: The current implementation is restricted to first-order, function-free Horn clauses (single-rule tasks). Real-world applications often require multi-rule reasoning or higher-order logic.
- Lack of Real-World Noise: Because the tasks are entirely synthetic, they lack the ambiguity, noise, and imperfect data distributions found in real-world settings.
- Discrete Scoring: The symbolic judge relies on deterministic, discrete scoring. Future iterations could benefit from recognizing partial solutions or interpreting natural language logic formulations directly.
Summary updated