AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning

Tevin Wang, Chenyan Xiong

Abstract

Rule-based rewards offer a promising strategy for improving reinforcement learning from human feedback (RLHF), but current approaches often rely on manual rule engineering. We present AutoRule, a fully automated method for extracting rules from preference feedback and formulating them into rule-based rewards. AutoRule extraction operates in three stages: it leverages a reasoning model to interpret user preferences, identifies candidate rules from the reasoning chain of these interpretations, and synthesizes them into a unified rule set. Leveraging the finalized rule set, we employ language-model verifiers to compute the fraction of rules satisfied by each output, using this metric as an auxiliary reward alongside the learned reward model during policy optimization. Training a Llama-3-8B model with AutoRule results in a 28.6% relative improvement in length-controlled win rate on AlpacaEval2.0, and a 6.1% relative gain in second-turn performance on a held-out MT-Bench subset, compared to a GRPO baseline trained with the same learned reward model but without the rule-based auxiliary reward. Our analysis confirms that the extracted rules exhibit good agreement with dataset preference. We find that AutoRule demonstrates reduced reward hacking compared to a learned reward model when run over two episodes. Finally, our case study suggests that the extracted rules capture unique qualities valued in different datasets.

Introduction

Aligning Large Language Models (LLMs) with human preferences typically relies on Reinforcement Learning from Human Feedback (RLHF). This process usually involves training a neural Reward Model (RM) to predict human preference scores. However, these "black-box" reward models are susceptible to reward hacking—a phenomenon where the policy model exploits loopholes in the RM (such as increasing response length or excessive politeness) to achieve higher scores without improving actual helpfulness.

While objective rule-based rewards (e.g., code execution or math verification) can mitigate this, creating rules for subjective tasks like creative writing or general assistance has historically required labor-intensive manual engineering. AutoRule addresses this bottleneck by automating the extraction of verifiable, objective criteria from subjective preference data.

The AutoRule Framework

AutoRule introduces a three-stage pipeline to transform raw preference pairs into a robust reward mechanism:

  1. Reasoning Generation: For every pair of responses (preferred vs. rejected), a reasoning model (e.g., DeepSeek-R1) generates a detailed "Chain-of-Thought" (CoT) explanation. This explanation identifies the specific nuances that made the winning response superior.
  2. Rule Extraction: The system extracts atomic, explicit rules from these reasoning chains. For example, a CoT noting a lack of jargon might yield the rule: "The assistant should avoid unnecessary technical complexity."
  3. Rule Merging: To handle the thousands of rules generated across a large dataset, a merging step synthesizes them into a compact, non-redundant set. This typically reduces the rule count to approximately 1–2% of the initial candidate pool.
The AutoRule pipeline: from reasoning generation to rule-based reward integration.

Reward Integration and Optimization

During the reinforcement learning phase, an LLM-as-a-judge (specifically Llama-3-8B-Instruct) serves as a binary verifier. It evaluates new model outputs against the extracted rule set, calculating a "Rule Reward" based on the fraction of rules satisfied. This is combined with a standard neural reward model to guide the policy. The training utilizes Group Relative Policy Optimization (GRPO), which allows for efficient advantage calculation without the overhead of a separate value model.

Experimental Results

We evaluated AutoRule using a Llama-3-8B base model across several benchmarks, comparing it against a standard GRPO baseline that used only a neural reward model.

  • AlpacaEval 2.0: AutoRule achieved a 28.6% relative improvement in length-controlled win rate. This metric is particularly significant as it demonstrates that the model's gains are due to quality improvements rather than simply increasing verbosity.
  • MT-Bench: The model showed a 6.1% relative gain in second-turn performance. This indicates that the extracted rules help the model maintain coherence and instruction adherence throughout multi-turn interactions.
  • Reward Hacking Resistance: In multi-episode stress tests, models trained with standard neural rewards eventually saw performance degradation as they began to "game" the system. In contrast, AutoRule models maintained high performance, suggesting that rule-based constraints provide a more stable and harder-to-exploit signal.

Analysis and Interpretability

A key advantage of AutoRule is its transparency. Unlike neural reward models, the criteria for success are human-readable. Analysis of the extracted rules shows high agreement with human preference datasets. Furthermore, the rules are adaptive; those extracted from UltraFeedback focus on conversational flow, while those from MT-Bench emphasize technical accuracy and instruction following.

Quantitative analysis of rule agreement and score deltas across UltraFeedback and MT-Bench datasets.

As shown in the analysis above, the extracted rules demonstrate a strong positive correlation with existing preference labels. The "Rule score delta" (the difference in rule satisfaction between preferred and rejected responses) peaks significantly above zero, confirming that the automated rules capture the underlying reasons for human preference.

Conclusion

AutoRule demonstrates that reasoning chains contain rich, structured information that can be distilled into objective rewards. By automating rule extraction, we bridge the gap between the flexibility of neural reward models and the robustness of rule-based systems. This approach not only improves model performance on standard benchmarks but also provides a scalable path toward more interpretable and reliable AI alignment.

Summary updated

Abstract

Rule-based rewards offer a promising strategy for improving reinforcement learning from human feedback (RLHF), but current approaches often rely on manual rule engineering. We present AutoRule, a fully automated method for extracting rules from preference feedback and formulating them into rule-based rewards. AutoRule extraction operates in three stages: it leverages a reasoning model to interpret user preferences, identifies candidate rules from the reasoning chain of these interpretations, and synthesizes them into a unified rule set. Leveraging the finalized rule set, we employ language-model verifiers to compute the fraction of rules satisfied by each output, using this metric as an auxiliary reward alongside the learned reward model during policy optimization. Training a Llama-3-8B model with AutoRule results in a 28.6\% relative improvement in length-controlled win rate on AlpacaEval2.0, and a 6.1\% relative gain in second-turn performance on a held-out MT-Bench subset, compared to a GRPO baseline trained with the same learned reward model but without the rule-based auxiliary reward. Our analysis confirms that the extracted rules exhibit good agreement with dataset preference. We find that AutoRule demonstrates reduced reward hacking compared to a learned reward model when run over two episodes. Finally, our case study suggests that the extracted rules capture unique qualities valued in different datasets. The extracted rules are provided in the appendix, and the code is open-sourced at https://github.com/cxcscmu/AutoRule.
BibTeX
@misc{wang2025autorule,
  author = {Tevin Wang and Chenyan Xiong},
  title = {{AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning}},
  year = {2025},
  eprint = {2506.15651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.15651},
}
Figure