Abstract
Rule-based reasoning is a fundamental challenge in artificial intelligence, where deviations in rule formats and complexity often hinder model performance. While large reasoning models (LRMs) have shown promise through reinforcement learning (RL), it remains unclear if smaller models (SRMs) can achieve similar robustness and generalization. We introduce RuleReasoner, a framework that utilizes a curated collection of tasks and a novel Domain-aware Dynamic Sampling (DADS) approach. By updating sampling weights based on historical rewards, RuleReasoner facilitates automated curriculum learning, eliminating the need for manual training recipes. Our evaluations show that RuleReasoner-8B outperforms frontier models like OpenAI-o1 by 4.1% on in-distribution tasks and 10.4% on out-of-distribution benchmarks, while achieving significantly higher training efficiency.
Introduction
Large Reasoning Models (LRMs) have demonstrated remarkable logical capabilities through Chain-of-Thought (CoT) processing. However, these models often struggle with strict, domain-specific rules found in fields like law or medicine. In these contexts, models cannot rely solely on internal "common sense"; they must adhere strictly to provided logic and facts.
Current models face three primary hurdles: format sensitivity to out-of-distribution (OOD) rules, the "lost in the middle" effect where models fail to link rules to facts in long contexts, and inefficient training due to static data mixing. RuleReasoner addresses these by treating rule-based reasoning as a reinforced learning problem with a dynamic, self-adjusting data distribution.
Methodology: Domain-aware Dynamic Sampling (DADS)
The core innovation of RuleReasoner is Domain-aware Dynamic Sampling (DADS). Unlike traditional RL training that uses a fixed ratio of tasks, DADS acts as an online scheduler. It monitors the model's performance (historical rewards) across various logical domains, such as induction and deduction. If the model underperforms in a specific area, DADS automatically increases the sampling weight for that domain in subsequent training batches.
This approach is paired with Reinforcement Learning with Verifiable Rewards (RLVR), specifically a variant of the Group Relative Policy Optimization (GRPO) algorithm. To encourage exploration and prevent the model from being overly anchored to its initial supervised state, the authors discarded standard KL divergence and entropy bonus terms. Furthermore, Rules Order Shuffling was implemented to ensure the model parses logic rather than relying on positional patterns.
Dataset: RuleCollection-32K
The framework is trained on RuleCollection-32K, a curated dataset of 32,000 samples across eight tasks, including ProofWriter, ProntoQA, and FOLIO. These tasks range from 0 to 7 logical steps (hops) and include both explicit and implicit rules to test the model's depth of reasoning.
Experimental Results
RuleReasoner-8B demonstrates state-of-the-art performance across several rigorous benchmarks. On the BigBench Hard (BBH) benchmark, the model achieved a near-perfect score of 99.6%. More impressively, it showed significant gains on OOD benchmarks, outperforming OpenAI-o1 by an average of 10.4% across three challenging datasets.
Key findings include:
- Efficiency: DADS achieved comparable performance to existing methods like DAPO while requiring 72 fewer training steps, a 1.4x acceleration in training speed.
- Small Model Efficacy: The 4B parameter version of RuleReasoner outperformed frontier models like Claude-3.7-Sonnet on several OOD benchmarks, maintaining an average pass@1 of 78.3%.
- Cross-Domain Generalization: Training on rule-based logic improved performance on unrelated tasks like AIME 2025 (Math) and GPQA (Science), suggesting that logical reasoning is a highly transferable meta-skill.
Conclusion and Future Work
RuleReasoner proves that smaller models can achieve and even surpass the reasoning capabilities of frontier LRMs when trained with a dynamic, reward-driven curriculum. While currently limited to 4B and 8B scales, the success of DADS suggests a scalable path forward for specialized reasoning models. Future research will focus on handling the messy, contradictory rules found in real-world legal and technical documentation, as well as scaling the framework to larger parameter counts.
Summary updated