SLR: Automated Synthesis for Scalable Logical Reasoning

Lukas Helff, Ahmad Omar, Felix Friedrich, Antonia Wüst, Hikaru Shindo, Rupert Mitchell, Tim Woydt, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting

TL;DR

The paper introduces SLR (Scalable Logical Reasoning), an open-source framework that automatically synthesizes inductive reasoning tasks complete with natural language prompts, latent ground-truth rules, and executable validation programs. By evaluating models on a new 19k-task curriculum benchmark (SLR-BENCH), the authors demonstrate that while current LLMs excel at syntax, they fail at complex logical induction. Crucially, they show that curriculum learning can double the accuracy of smaller models like Llama-3-8B, matching heavy-hitting reasoning models at a fraction of the computational cost.

What Problem Does This Paper Address?

Most existing LLM evaluation benchmarks focus heavily on deductive reasoning—asking models to draw specific conclusions from given premises (e.g., math word problems or logic puzzles). However, inductive reasoning—the ability to infer general rules or patterns from specific examples—remains highly underexplored.

Furthermore, current evaluation frameworks suffer from three major flaws:

  1. Subjectivity and Gaming: They often rely on multiple-choice formats or use other LLMs as judges, which can be easily gamed or misinterpret logical validity.
  2. Data Contamination: As training datasets grow, benchmarks increasingly overlap with pre-training data, turning true reasoning tasks into mere exercises in memorization.
  3. Lack of Granular Control: There is no easy way to precisely scale the logical complexity of a task to see exactly where an LLM's reasoning breaks down.

To bridge this gap, the authors developed SLR to automate the generation of unique, uncontaminated inductive logic tasks with verifiable, symbolic ground truths.

Methods

The SLR framework operates on the Learning-from-Entailment paradigm from Inductive Logic Programming (ILP). Given a user's high-level task specification, the pipeline automatically executes three main stages:

Figure 1: Overview of the SLR Framework, including task specification, automated task synthesis, training, and evaluation. Left (blue): Language defines vocabulary and grammar, Task Config specifies configuration parameters for the synthesis. Middle (green): The task synthesizer automatically generates ground-truth rules, validation programs, and instruction prompts. Right (purple): Training LLMs on logic tasks via SFT (cross-entropy) or RL (symbolic judge feedback). Right (orange): Evaluates LLMs using feedback provided by the symbolic judge. Arrows denote data and control flow through synthesis, prompting, evaluation, and downstream training loops.
  1. Task Specification: Defines the vocabulary (predicates and constants) and grammar constraints.
  2. Automated Synthesis: A task synthesizer generates a latent ground-truth rule ($R^*$), samples background knowledge facts ($B$), and uses a Prolog engine to deterministically label examples as positive ($E^+$) or negative ($E^-$).
  3. Verifiable Logic Rewards (The Symbolic Judge): Instead of relying on exact text matching or an LLM judge, SLR uses a symbolic judge. It executes the LLM's generated hypothesis against a validation program. If the hypothesis correctly entails all positive examples and refutes all negative ones, it passes—allowing for semantically correct but syntactically unique solutions.
Figure 2: Verifiable Logic Rewards: A candidate hy- pothesis is evaluated by executing it against the valida- tion program. It outputs three metrics: syntactic validity (binary), perfect task completion (binary), and a partial score for the fraction of correctly classified examples.

Datasets

The authors introduced two primary datasets using SLR:

  • SLR-BENCH: A static benchmark of 19,000 tasks organized into a 20-level curriculum across four difficulty tiers (basic, easy, medium, hard). It scales complexity by adjusting rule length, problem size, and introducing advanced logical structures like recursion and arithmetic. It is inspired by the classic V-LoL trains domain.
  • SLR-Homes: An out-of-distribution (OOD) dataset containing 500 house-classification tasks across 5 difficulty levels, used specifically to test cross-domain generalization.

Additionally, the authors evaluated their SLR-tuned models on several established downstream benchmarks, including MMLU, CLUTRR, LogiQA/LogiQA2, GPQA, ARC, and HellaSwag.

Key Insights and Ideas

  • Syntax $\neq$ Semantics: Standard LLMs are highly proficient at generating syntactically valid Prolog code, but they consistently fail to capture the correct underlying logic as complexity scales.
  • The Steep Cost of "Thinking": Scaling test-time compute (e.g., OpenAI's o1/o3 or DeepSeek-R1) significantly improves inductive reasoning performance. However, it hits a harsh plateau on "hard" tasks and incurs massive financial costs (exceeding $300 for just 1,000 prompts).
  • Curriculum Learning is a Cheat Code: Rather than throwing massive compute at inference time, training an 8B model sequentially from basic to medium difficulty tiers allows it to outperform much larger models.
  • Verbosity is a Warning Sign: An error analysis revealed a strong inverse correlation between output length and success for distilled reasoning models. When a model outputs excessively long reasoning traces, it is usually a sign of derailment rather than deeper thinking.

Key Results

The authors conducted a large-scale evaluation of state-of-the-art models on SLR-BENCH:

Figure 3: Overview of SLR-BENCH: The benchmark curriculum spans from basic to hard tasks with increas- ing logical and combinatorial complexity (bars, right y-axis). As logical complexity increases, Model perfor- mance (red, left y-axis) declines, highlighting current LLMs’ limitations on more challenging reasoning tasks.
  • Curriculum Fine-Tuning Breakthrough: Fine-tuning a Llama-3-8B model using SLR's curriculum doubled its Logical Reasoning Level (LRL) score from 3.8 to 8.7, achieving parity with Gemini-2.0-Flash-Thinking while using significantly fewer tokens.
  • Downstream Generalization: The capabilities gained from SLR training successfully transferred to external datasets. The SLR-tuned Llama model saw an average accuracy boost of +6.6 percentage points on CLUTRR and +17 percentage points on MMLU High School Statistics.
  • Hard Tier Defeat: Even the strongest frontier models struggled with the highest levels of logical complexity; OpenAI's o3 topped the leaderboard but only solved 45% of the tasks in the Hard tier.

Limitations and Future Work

While SLR provides a powerful framework for scaling logic evaluation, the authors note several limitations:

  • Simplified Logic Domain: The current implementation is restricted to first-order, function-free Horn clauses (single-rule tasks). Real-world applications often require multi-rule reasoning or higher-order logic.
  • Lack of Real-World Noise: Because the tasks are entirely synthetic, they lack the ambiguity, noise, and imperfect data distributions found in real-world settings.
  • Discrete Scoring: The symbolic judge relies on deterministic, discrete scoring. Future iterations could benefit from recognizing partial solutions or interpreting natural language logic formulations directly.

Summary updated

Abstract

We introduce SLR, an end-to-end framework for systematic evaluation and training of Large Language Models (LLMs) via Scalable Logical Reasoning. Given a user's task specification, SLR automatically synthesizes (i) an instruction prompt for an inductive reasoning task, (ii) a validation program, executable on model outputs to provide verifiable rewards, and (iii) the latent ground-truth rule. This process is fully automated, scalable, requires no human annotations, and offers precise control over task difficulty. Using SLR, we create SLR-Bench, a benchmark comprising 19k prompts organized into 20 curriculum levels that progressively increase in relational, arithmetic, and recursive complexity. Large-scale evaluation reveals that contemporary LLMs readily produce syntactically valid rules, yet often fail at correct logical inference. Recent reasoning LLMs demonstrate improved performance but incur very high test-time computation, with costs exceeding $300 for just 1,000 prompts. Finally, curriculum learning via SLR doubles Llama-3-8B accuracy on SLR-Bench, achieving parity with Gemini-Flash-Thinking at a fraction of computational cost. Moreover, these reasoning capabilities generalize to a wide range of established benchmarks, underscoring the effectiveness of SLR for downstream reasoning.
BibTeX
@inproceedings{helff2026slr,
  author = {Lukas Helff and Ahmad Omar and Felix Friedrich and Antonia Wüst and Hikaru Shindo and Rupert Mitchell and Tim Woydt and Patrick Schramowski and Wolfgang Stammer and Kristian Kersting},
  title = {{SLR: Automated Synthesis for Scalable Logical Reasoning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.16},
}
Figure