Abstract
Large language models (LLMs) have been widely deployed across numerous fields. Reinforcement Learning from Human Feedback (RLHF) leverages reward models (RMs) as proxies for human preferences to align LLM behaviors with human values, making the accuracy, reliability, and interpretability of RMs critical for effective alignment. However, traditional RMs lack interpretability, offer limited insight into the reasoning behind reward assignments, and are inflexible toward user preference shifts. While recent multidimensional RMs aim for improved interpretability, they often fail to provide feature-level attribution and require costly annotations. To overcome these limitations, we introduce the Sparse Autoencoder-Enhanced Reward Model (SARM), a novel architecture that integrates a pretrained Sparse Autoencoder (SAE) into a reward model. SARM maps the hidden activations of LLM-based RM into an interpretable, sparse, and monosemantic feature space, from which a scalar head aggregates feature activations to produce transparent and conceptually meaningful reward scores. Empirical evaluations demonstrate that SARM facilitates direct feature-level attribution of reward assignments, allows dynamic adjustment to preference shifts, and achieves superior alignment performance compared to conventional reward models.
Introduction
In the Reinforcement Learning from Human Feedback (RLHF) pipeline, the Reward Model (RM) serves as the critical judge, guiding the LLM toward human-preferred responses. Despite their central role, traditional RMs are often "black boxes" suffering from two major drawbacks: opacity and rigidity. They typically output a single scalar score without explanation, leaving developers unsure if a high reward stems from helpfulness or superficial traits like verbosity. Furthermore, these models are rigid; adapting to shifting user preferences (e.g., stricter safety requirements) usually necessitates expensive retraining. While multidimensional RMs attempt to address this by rating specific traits, they incur significant annotation costs and still lack granular feature-level transparency.
Methodology: The SARM Architecture
The Sparse Autoencoder-Enhanced Reward Model (SARM) addresses these challenges by mapping complex neural activations to human-understandable concepts. The approach consists of two primary stages:
- Sequence-level SAE Pretraining: Unlike standard SAEs that analyze every token, SARM focuses on the final token activations of an intermediate LLM layer. The final token effectively summarizes the sequence's semantics. The SAE disentangles these activations into millions of sparse, monosemantic features.
- Reward Modeling with a Linear Head: The LLM is truncated at the midpoint, where the pretrained SAE is attached. A simple linear "value head" is added on top of the sparse features. Because the final reward is a weighted sum of these interpretable features, the model's reasoning becomes transparent.
To ensure human readability, features were labeled using GPT-4o, which analyzed text snippets that activated specific latents to generate descriptions such as "Ethical considerations" or "Profanity."
Experimental Setup
SARM was developed using Llama-3 (3B and 8B) backbones. SAE pretraining utilized 50 million sequences from OpenWebText2, while RM training was conducted on the Skywork-Reward-Preference-80K-v0.2 dataset. Performance was primarily evaluated using RewardBench 2, which tests RM accuracy across safety, mathematics, and general alignment. Additionally, RM-Bench served as an out-of-distribution dataset to validate feature activation across diverse prompts.
Results and Discussion
Performance and Interpretability
SARM demonstrates that interpretability does not have to come at the cost of performance. SARM-4B (derived from an 8B backbone) achieved an overall score of 73.6 on RewardBench 2, outperforming significantly larger models like Llama-3.1-Tulu-3-70B (72.2) and closed-source models like GPT-4.1 (72.3). This "interpretability bonus" suggests that forcing the model to rely on sparse, meaningful features can enhance its evaluative capabilities.
Causal Controllability
A key advantage of SARM's linear value head is the ability to manually adjust behavior. By increasing the weight of specific features, such as "Safety," developers can shift the reward distribution to favor safe responses without retraining the entire model.
Feature Discovery
The model successfully identified specific "Positive" features (e.g., Feature 60427: Ethical considerations) and "Negative" features (e.g., Feature 13950: Derogatory and offensive) that directly correlate with human preferences. This allows for fine-grained attribution of why a specific response received its score, transforming the reward assignment from a single number into a clear, weighted list of conceptual factors.
Conclusion and Future Work
SARM represents a significant step toward transparent and steerable alignment. By leveraging SAEs, it provides feature-level interpretability and dynamic controllability while maintaining state-of-the-art performance. Future research will address the challenge of "dead latents" to improve computational efficiency and explore multi-layer feature aggregation for more nuanced judgment.
Summary updated