SLR: Automated Synthesis for Scalable Logical Reasoning

Lukas Helff, Ahmad Omar, Felix Friedrich, Antonia Wüst, Hikaru Shindo, Tim Woydt, Rupert Mitchell, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting

TL;DR

This paper introduces SLR, a novel framework for automatically generating and evaluating logical reasoning tasks for LLMs. They also present SLR-BENCH, a new benchmark with 19,000 tasks that progressively increase in complexity. Their work shows that while LLMs can generate syntactically correct logical rules, they struggle with true logical inference, and that curriculum learning with SLR significantly improves performance and generalization.

What Problem Does This Paper Address?

Modern Large Language Models (LLMs) are increasingly capable, but they still struggle with robust logical reasoning, especially inductive reasoning (inferring general rules from specific examples). Existing benchmarks often focus on deductive reasoning or rely on LLMs to judge outputs, making it hard to truly assess a model's logical understanding. Furthermore, as benchmarks become more common, there's a risk of models simply memorizing answers rather than learning to reason. This paper aims to address these challenges by providing a scalable, automated way to create and evaluate logical reasoning tasks, specifically focusing on inductive reasoning.

Methods

The core of this work is the SLR (Scalable Logical Reasoning) framework. Given a user-defined logic task specification (language and configuration), SLR automatically synthesizes:

  1. Instruction Prompts: Natural language or Prolog-style prompts for an inductive reasoning task.
  2. Validation Programs: Executable logic programs that deterministically verify a model's output.
  3. Latent Ground-Truth Rules: The correct logical rule for the task.

This process is fully automated, requires no human annotation, and allows for precise control over task difficulty. The framework supports both model evaluation and training (via supervised fine-tuning or reinforcement learning using rewards from a symbolic judge).

The paper also introduces SLR-BENCH, a benchmark built using SLR. It consists of 19,000 tasks organized into 20 curriculum levels, progressively increasing in relational, arithmetic, and recursive complexity. This curriculum design is key to systematically assessing and improving LLM reasoning.

Datasets

The primary dataset introduced is SLR-BENCH, a benchmark comprising 19,000 automatically generated inductive logic reasoning tasks. These tasks are organized into a 20-level curriculum with increasing complexity. The paper also mentions evaluating their trained models on several established downstream benchmarks, including:

  • MMLU (and its subsets like MMLU-Stats)
  • CLUTRR
  • LogiQA (and LogiQA2)
  • GPQA (and its variants)
  • ARC-Easy and ARC-Challenge
  • HellaSwag

Key Insights and Ideas

  • LLMs struggle with inductive reasoning: While LLMs can generate syntactically correct logical rules, they often fail at the actual logical inference, especially as task complexity increases.
  • Curriculum learning is crucial: Training LLMs on a structured curriculum of increasing logical complexity (like SLR-BENCH) significantly boosts their reasoning abilities, both on the benchmark itself and on downstream tasks.
  • Automated synthesis is powerful: SLR's ability to automatically generate diverse, verifiable logic tasks removes the bottleneck of human annotation and allows for fine-grained control over difficulty and scale.
  • Reasoning LLMs are promising but costly: Specialized reasoning LLMs show improved performance but come with prohibitively high computational costs. SLR-tuned models offer a more efficient path to enhanced reasoning.
  • Generalization beyond the training domain: Models trained with SLR demonstrate robust zero-shot transfer to previously unseen domains and tasks, suggesting SLR learns domain-agnostic reasoning capabilities.

Key Results

  • SLR-BENCH Performance: Large-scale evaluation shows that contemporary LLMs produce valid rules but struggle with inference. Reasoning LLMs perform better but incur very high costs (e.g., >$300 for 1,000 prompts).
  • Llama-3.1-8B-SLR: Curriculum learning via SLR doubles Llama-3.1-8B's accuracy on SLR-BENCH and achieves parity with Gemini-Flash-Thinking, but at a fraction of the computational cost.
  • Downstream Generalization: SLR-tuned models show significant gains on established benchmarks like MMLU (+17%), CLUTRR, and GPQA, demonstrating effective transfer of learned reasoning skills.
  • Cost-Efficiency: SLR-tuned models achieve competitive or superior performance with significantly fewer inference tokens and lower compute compared to other advanced models.
  • Ablation Studies: Confirm that SLR's synthesis parameters (rule sampling, problem size, rule length) precisely control task difficulty, and that ordered curriculum learning is superior to random or reverse order.

Limitations and Future Work

Limitations: * The current focus is on first-order, function-free Horn clauses, which limits the complexity of logic expressible. * The synthetic nature of the benchmark, while beneficial for control, might not fully capture real-world data diversity and ambiguity. * The symbolic judge provides discrete scoring, and could potentially be enhanced to handle partial solutions or natural language formulations more nuancedly.

Future Work: * Extending SLR to more complex, multi-rule reasoning scenarios and different domains. * Incorporating higher-order logic or probabilistic reasoning. * Developing more sophisticated symbolic judges that can handle partial solutions and natural language. * Exploring reinforcement learning with SLR for more advanced training paradigms. * Investigating controlled forms of real-world noise, ambiguity, and multi-rule interactions to study robustness.

Summary updated

Abstract

We introduce SLR, an end-to-end framework for systematic evaluation and training of Large Language Models (LLMs) via Scalable Logical Reasoning. Given a user's task specification, SLR automatically synthesizes (i) an instruction prompt for an inductive reasoning task, (ii) a validation program, executable on model outputs to provide verifiable rewards, and (iii) the latent ground-truth rule. This process is fully automated, scalable, requires no human annotations, and offers precise control over task difficulty. Using SLR, we create SLR-Bench, a benchmark comprising 19k prompts organized into 20 curriculum levels that progressively increase in relational, arithmetic, and recursive complexity. Large-scale evaluation reveals that contemporary LLMs readily produce syntactically valid rules, yet often fail at correct logical inference. Recent reasoning LLMs demonstrate improved performance but incur very high test-time computation, with costs exceeding $300 for just 1,000 prompts. Finally, curriculum learning via SLR doubles Llama-3-8B accuracy on SLR-Bench, achieving parity with Gemini-Flash-Thinking at a fraction of computational cost. Moreover, these reasoning capabilities generalize to a wide range of established benchmarks, underscoring the effectiveness of SLR for downstream reasoning.
BibTeX
@inproceedings{helff2026slr,
  author = {Lukas Helff and Ahmad Omar and Felix Friedrich and Antonia Wüst and Hikaru Shindo and Tim Woydt and Rupert Mitchell and Patrick Schramowski and Wolfgang Stammer and Kristian Kersting},
  title = {{SLR: Automated Synthesis for Scalable Logical Reasoning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.16},
}
Figure