RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling

Yang Liu, Jiaqi Li, Zilong Zheng

Abstract

Rule-based reasoning is a fundamental challenge in artificial intelligence, where deviations in rule formats and complexity often hinder model performance. While large reasoning models (LRMs) have shown promise through reinforcement learning (RL), it remains unclear if smaller models (SRMs) can achieve similar robustness and generalization. We introduce RuleReasoner, a framework that utilizes a curated collection of tasks and a novel Domain-aware Dynamic Sampling (DADS) approach. By updating sampling weights based on historical rewards, RuleReasoner facilitates automated curriculum learning, eliminating the need for manual training recipes. Our evaluations show that RuleReasoner-8B outperforms frontier models like OpenAI-o1 by 4.1% on in-distribution tasks and 10.4% on out-of-distribution benchmarks, while achieving significantly higher training efficiency.

Introduction

Large Reasoning Models (LRMs) have demonstrated remarkable logical capabilities through Chain-of-Thought (CoT) processing. However, these models often struggle with strict, domain-specific rules found in fields like law or medicine. In these contexts, models cannot rely solely on internal "common sense"; they must adhere strictly to provided logic and facts.

Current models face three primary hurdles: format sensitivity to out-of-distribution (OOD) rules, the "lost in the middle" effect where models fail to link rules to facts in long contexts, and inefficient training due to static data mixing. RuleReasoner addresses these by treating rule-based reasoning as a reinforced learning problem with a dynamic, self-adjusting data distribution.

Methodology: Domain-aware Dynamic Sampling (DADS)

The core innovation of RuleReasoner is Domain-aware Dynamic Sampling (DADS). Unlike traditional RL training that uses a fixed ratio of tasks, DADS acts as an online scheduler. It monitors the model's performance (historical rewards) across various logical domains, such as induction and deduction. If the model underperforms in a specific area, DADS automatically increases the sampling weight for that domain in subsequent training batches.

The RuleReasoner framework utilizing DADS for online sampling weight updates based on domain rewards.

This approach is paired with Reinforcement Learning with Verifiable Rewards (RLVR), specifically a variant of the Group Relative Policy Optimization (GRPO) algorithm. To encourage exploration and prevent the model from being overly anchored to its initial supervised state, the authors discarded standard KL divergence and entropy bonus terms. Furthermore, Rules Order Shuffling was implemented to ensure the model parses logic rather than relying on positional patterns.

Dataset: RuleCollection-32K

The framework is trained on RuleCollection-32K, a curated dataset of 32,000 samples across eight tasks, including ProofWriter, ProntoQA, and FOLIO. These tasks range from 0 to 7 logical steps (hops) and include both explicit and implicit rules to test the model's depth of reasoning.

Experimental Results

RuleReasoner-8B demonstrates state-of-the-art performance across several rigorous benchmarks. On the BigBench Hard (BBH) benchmark, the model achieved a near-perfect score of 99.6%. More impressively, it showed significant gains on OOD benchmarks, outperforming OpenAI-o1 by an average of 10.4% across three challenging datasets.

Performance comparison of RuleReasoner variants against frontier models on BigBench Hard, ProverQA, and BigBench Extra Hard.

Key findings include:

  • Efficiency: DADS achieved comparable performance to existing methods like DAPO while requiring 72 fewer training steps, a 1.4x acceleration in training speed.
  • Small Model Efficacy: The 4B parameter version of RuleReasoner outperformed frontier models like Claude-3.7-Sonnet on several OOD benchmarks, maintaining an average pass@1 of 78.3%.
  • Cross-Domain Generalization: Training on rule-based logic improved performance on unrelated tasks like AIME 2025 (Math) and GPQA (Science), suggesting that logical reasoning is a highly transferable meta-skill.

Conclusion and Future Work

RuleReasoner proves that smaller models can achieve and even surpass the reasoning capabilities of frontier LRMs when trained with a dynamic, reward-driven curriculum. While currently limited to 4B and 8B scales, the success of DADS suggests a scalable path forward for specialized reasoning models. Future research will focus on handling the messy, contradictory rules found in real-world legal and technical documentation, as well as scaling the framework to larger parameter counts.

Summary updated

Abstract

Rule-based reasoning has been acknowledged as one of the fundamental problems in reasoning, while deviations in rule formats, types, and complexity in real-world applications pose severe challenges. Recent studies have shown that large reasoning models (LRMs) have remarkable reasoning capabilities, and their performance is substantially enhanced by reinforcement learning (RL). However, it remains an open question whether small reasoning models (SRMs) can learn rule-based reasoning effectively with robust generalization across diverse tasks and domains. To address this, we introduce Reinforced Rule-based Reasoning, a.k.a. RuleReasoner, a simple yet effective method to conduct rule-based reasoning via a wide collection of curated tasks and a novel domain-aware dynamic sampling approach. Specifically, RuleReasoner resamples each training batch by updating the sampling weights of different domains based on historical rewards. This facilitates domain augmentation and flexible online learning schedules for RL, obviating the need for pre-hoc human-engineered mix-training recipes used in existing methods. Empirical evaluations on in-distribution (ID) and out-of-distribution (OOD) benchmarks reveal that RuleReasoner outperforms frontier LRMs by a significant margin ($Δ$4.1% average points on eight ID tasks and $Δ$10.4% average points on three OOD tasks over OpenAI-o1). Notably, our approach also exhibits higher computational efficiency compared to prior dynamic sampling methods for RL.
BibTeX
@inproceedings{liu2026rulereasoner,
  author = {Yang Liu and Jiaqi Li and Zilong Zheng},
  title = {{RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=MQV4TJyqnb},
}
Figure