AutoRule: Reasoning Chain-of-Thought Extracted Rule-based Rewards Improve Preference Learning

Tevin Wang, Chenyan XiongSubmission to ICLR 2026 · Rejected

TL;DR

AutoRule introduces an automated framework that mines reasoning models' chain-of-thought (CoT) traces over pairwise preference data to discover universal, natural-language alignment rules. Using these extracted rules as an auxiliary verification reward during reinforcement learning, AutoRule boosts instruction-following and multi-turn conversational quality while curbing length-based reward hacking.

What Problem Does This Paper Address?

Reinforcement Learning from Human Feedback (RLHF) typically relies on learned reward models. However, these models are notorious for reward hacking—they frequently overfit to superficial heuristics like verbosity rather than genuine response quality.

While rule-based verifiers (popularized in math and coding post-training, such as in DeepSeek-R1) offer objective, unhackable boundaries, applying them to general, subjective instruction-following has historically required either expensive manual rule curation or generating custom rubrics per prompt (e.g., Rubrics-as-Rewards or RLCF). Per-prompt methods require massive compute overhead, calling large teacher models for every single training instance. The authors ask: Can we automatically extract a compact, reusable, global set of behavioral rules directly from preference data to guide alignment?

Methods

AutoRule extracts and applies rules through a staged pipeline:

  1. Reasoning-Driven Rule Extraction:
  2. Reasoning Generation: A frontier reasoning model (DeepSeek-R1 or Qwen3-8B) analyzes preference pairs (chosen vs. rejected) and generates step-by-step reasoning explaining the preference.
  3. Rule Extraction: Explicit, objective behavioral rules are extracted directly from the model's internal reasoning chains rather than its summary justifications.
  4. Rule Merging: Candidate rules across examples are deduplicated and consolidated into a compact set (compressing thousands of candidate statements down to ~25 global rules).
  5. Rule Verification as Auxiliary Reward:
  6. An LLM-as-a-judge verifier (Llama-3-8B-Instruct) evaluates each candidate model response against the merged rule set with a binary decision: a rule is satisfied only if the output adheres to the criterion and remains concise.
  7. The mean rule satisfaction is scaled and combined with a standard learned reward model: $$r_{\text{total}}(x, y) = r_{\text{AutoRule}}(x, y) + r_\theta(x, y) - \beta_{\text{KL}}\text{KL}$$
  8. Policy Optimization:
  9. The combined reward is optimized using Group Relative Policy Optimization (GRPO).

Unlike per-prompt rubric frameworks, AutoRule constructs a single, global rule set from just a small sample of preference data, avoiding teacher inference during policy rollouts.

Datasets

  • Training & Extraction:
  • UltraFeedback-Binarized: A 33K filtered subset of diverse pairwise preference data. AutoRule extracts its global rule set using only 256 sampled examples.
  • MT-Bench Human Judgments: A 40-question split used to extract a multi-turn dialogue rule set.
  • Evaluation Benchmarks:
  • UltraFeedback Test Split: In-distribution pairwise win rate evaluated by GPT-4o.
  • AlpacaEval 2.0: Single-turn instruction-following evaluated via length-controlled (LC) win rates against GPT-4 Turbo.
  • MT-Bench: Multi-turn conversational evaluation across 8 categories judged by GPT-4.

Key Insights and Ideas

  • Reasoning chains beat summary justifications: Extracting rules from the reasoning model's intermediate chain-of-thought traces yields significantly better downstream alignment (21.6% LC win rate on AlpacaEval) compared to extracting from final summary justifications (19.7%). CoT traces contain concrete, granular causal distinctions, whereas summary justifications tend to be vague.
  • Global rules drastically cut teacher costs: AutoRule extracts its rules from 256 examples using under 1M teacher tokens, compared to concurrent per-prompt methods like Rubrics-as-Rewards (~88.7M tokens) and RLCF (~203.1M tokens), achieving a ~100–200× reduction in teacher inference.
  • Rules serve as anti-hacking regularizers: As standard GRPO training progresses, win rates on true evaluation benchmarks degrade after roughly 50 steps due to reward hacking (e.g., hallucinating extra list items or padding length). AutoRule's binary constraint rewards anchor the policy to core instructions, maintaining high win rates throughout training.
  • The rule-count sweet spot: Ablations show that 25 rules optimize performance. Using 5 rules lacks coverage (UF win rate drops from 77.2% to 69.3%), while scaling to 50 rules introduces noisy, rarely applicable constraints that dilute the optimization signal.

Key Results

AutoRule was evaluated across two base models (Llama-3-8B and OLMo-2-7B):

  • AlpacaEval 2.0: On Llama-3-8B, AutoRule achieved a 21.6% length-controlled win rate against GPT-4 Turbo (a 20.7% relative improvement over the strongest baseline at 17.9%). On OLMo-2-7B, it achieved 20.0% (vs. 15.7% for vanilla GRPO).
  • MT-Bench: On Llama-3-8B, AutoRule reached an average score of 7.85, demonstrating a 6.1% relative gain in Turn 2 performance (7.83 vs. 7.38 baseline), indicating stronger multi-turn coherence.
  • UltraFeedback: AutoRule achieved 77.2% win rate vs. SFT on Llama-3-8B and 79.4% on OLMo-2-7B, outperforming PPO, GRPO, explicit length penalties, and per-prompt rubric baselines.

Limitations and Future Work

  • Inference overhead during RL: While teacher extraction costs are minimal, querying an LLM verifier across ~25 rules for every policy generation during RL training adds substantial verification FLOPs.
  • Judge capacity and context windows: The verifier was constrained to smaller open models (Llama-3-8B) with an 8k context window, limiting the use of extended reasoning inside the verifier itself.
  • Model scale: Experiments were conducted primarily on 7B–8B parameter base models; validating the method on larger scales (e.g., 70B+) remains to be explored.

Summary updated

Abstract

Existing rule-based rewards in preference-based reinforcement learning rely on manual engineering, limiting scalability. We present AutoRule, a fully automated method for extracting rules from preference feedback and formulating them into rule-based rewards. AutoRule extraction operates in three stages: it leverages a reasoning model to interpret user preferences, identifies candidate rules from the reasoning chains of these interpretations, and synthesizes them into a unified rule set. Using the finalized rule set, we employ language-model verifiers to judge rule satisfaction, using this metric as an auxiliary reward alongside the learned reward model during policy optimization. Empirically, AutoRule yields gains for both Llama-3-8B and Olmo-2-7B in both in-distribution and out-of-distribution benchmarks. On Llama-3-8B, it achieves a 20.7% relative improvement in length-controlled win rate against GPT4 on AlpacaEval2.0, and a 6.1% relative gain in second-turn performance on a held-out MT-Bench subset, compared to baseline models. Further analysis shows that the extracted rules exhibit strong agreement with dataset preferences and are behaviorally consistent across multiple runs, extraction scales, and aggregated scores. Notably, these rules also contribute to mitigating reward hacking in reward models, likely because they serve as constraints that prevent the policy from exploiting spurious features. Extracted rules are provided; code and model checkpoints will be open-sourced.
BibTeX
@misc{wang2025autorule,
  author = {Tevin Wang and Chenyan Xiong},
  title = {{AutoRule: Reasoning Chain-of-Thought Extracted Rule-based Rewards Improve Preference Learning}},
  year = {2025},
  eprint = {2506.15651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.15651},
}

Loading related content...

Figure