Abstract
Xiaoyu Liu, Di Liang, Hongyu Shan, Peiyang Liu, Yonghao Liu, Muling Wu, Yuntao Li, Xianjie Wu, Li Miao, Jiangrong Shen, Minlong Peng. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track.
2025.
Introduction
In Reinforcement Learning from Human Feedback (RLHF), Reward Models (RMs) serve as critical evaluators of Large Language Model (LLM) outputs. Existing RMs typically fall into two categories: efficient but opaque Scalar RMs, and interpretable but computationally expensive Generative RMs (GRMs). Scalar RMs provide a single score without justification, complicating error diagnosis. Conversely, GRMs generate reasoning chains but suffer from high latency due to sequential token decoding, making them unsuitable for high-throughput industrial applications. To bridge this gap, we introduce the Structural Reward Model (SRM), a modular framework that achieves both high interpretability and inference efficiency.
The Structural Reward Model Framework
The SRM architecture decomposes the evaluation process into specialized modules called Side-Branch Models (SBMs). These models act as auxiliary feature generators that analyze specific dimensions of a response in parallel.
The framework utilizes five distinct SBMs:
- SB-Semantic: Captures deep intent and thematic structure.
- SB-Entity: Leverages external knowledge graphs to verify core entities.
- SB-FactCheck: Assesses factual consistency and identifies hallucinations.
- SB-Style: Evaluates tone, wording, and stylistic alignment.
- SB-Quality: Monitors for creativity and avoids repetitive content.
As illustrated in Figure 2, when a prompt and response pair is processed, the SBMs extract fine-grained signals which are then appended to the original input as a "Side branch supplement." This enriched context allows a standard Reward Model head to produce a final score based on the Bradley-Terry preference framework. This modularity provides a "diagnostic label" for every response, enabling engineers to pinpoint exactly why a model's output was penalized.
Experimental Evaluation
The SRM was trained on a massive industrial dataset comprising 1.8 million preference-labeled samples across domains such as mathematics, coding, and logic. Performance was evaluated across several benchmarks: RM-Bench for general reward modeling, JudgeBench for knowledge-intensive tasks, and a new IFBench (Instruction Following Benchmark) designed to test adherence to complex constraints.
Performance and Efficiency
The SRM demonstrates significant improvements over baseline models. As shown in the comparative analysis, the integration of side-branches consistently elevates the performance of base models across all benchmarks.
Notably, the Llama3-8B Instruct model's overall performance increased from 11.3% to 60.8% when augmented with the SRM framework. Beyond accuracy, the SRM is approximately 4x faster than traditional Generative RMs (22.8s vs. 92.5s per 1,000 samples) due to its parallel feature extraction architecture.
Key Insights
Ablation studies highlighted the critical role of factual verification; removing the Fact-Checking module resulted in performance drops of up to 14.6% on JudgeBench. Furthermore, the SB-Entity branch effectively mitigates common RM failures by incorporating external knowledge, ensuring the model has the necessary context to judge specialized topics. In industrial deployment, the SRM reduced hallucination rates by 6% and improved complex reasoning scores by over 6%.
Conclusion and Future Work
The Structural Reward Model offers a scalable solution for interpretable AI feedback. While the current implementation relies on predefined SBMs and increases input sequence length, future work will explore more efficient fusion mechanisms, such as cross-attention, to optimize processing. The SRM represents a significant step toward transparent and efficient alignment in large-scale AI systems.
Summary updated