Interpretable Reward Model via Sparse Autoencoder

Shufen Zhang, Wei Shi, Sihang Li, Jiayi Liao, Tian Liang, Hengxing Cai, Xiang Wang

Abstract

Large language models (LLMs) have been widely deployed across numerous fields. Reinforcement Learning from Human Feedback (RLHF) leverages reward models (RMs) as proxies for human preferences to align LLM behaviors with human values, making the accuracy, reliability, and interpretability of RMs critical for effective alignment. However, traditional RMs lack interpretability, offer limited insight into the reasoning behind reward assignments, and are inflexible toward user preference shifts. While recent multidimensional RMs aim for improved interpretability, they often fail to provide feature-level attribution and require costly annotations. To overcome these limitations, we introduce the Sparse Autoencoder-Enhanced Reward Model (SARM), a novel architecture that integrates a pretrained Sparse Autoencoder (SAE) into a reward model. SARM maps the hidden activations of LLM-based RM into an interpretable, sparse, and monosemantic feature space, from which a scalar head aggregates feature activations to produce transparent and conceptually meaningful reward scores. Empirical evaluations demonstrate that SARM facilitates direct feature-level attribution of reward assignments, allows dynamic adjustment to preference shifts, and achieves superior alignment performance compared to conventional reward models.

Introduction

In the Reinforcement Learning from Human Feedback (RLHF) pipeline, the Reward Model (RM) serves as the critical judge, guiding the LLM toward human-preferred responses. Despite their central role, traditional RMs are often "black boxes" suffering from two major drawbacks: opacity and rigidity. They typically output a single scalar score without explanation, leaving developers unsure if a high reward stems from helpfulness or superficial traits like verbosity. Furthermore, these models are rigid; adapting to shifting user preferences (e.g., stricter safety requirements) usually necessitates expensive retraining. While multidimensional RMs attempt to address this by rating specific traits, they incur significant annotation costs and still lack granular feature-level transparency.

Methodology: The SARM Architecture

The Sparse Autoencoder-Enhanced Reward Model (SARM) addresses these challenges by mapping complex neural activations to human-understandable concepts. The approach consists of two primary stages:

  1. Sequence-level SAE Pretraining: Unlike standard SAEs that analyze every token, SARM focuses on the final token activations of an intermediate LLM layer. The final token effectively summarizes the sequence's semantics. The SAE disentangles these activations into millions of sparse, monosemantic features.
  2. Reward Modeling with a Linear Head: The LLM is truncated at the midpoint, where the pretrained SAE is attached. A simple linear "value head" is added on top of the sparse features. Because the final reward is a weighted sum of these interpretable features, the model's reasoning becomes transparent.
The SARM architecture: Stage 1 involves sequence-level SAE pretraining; Stage 2 integrates the SAE into the reward model to produce monosemantic features for scoring.

To ensure human readability, features were labeled using GPT-4o, which analyzed text snippets that activated specific latents to generate descriptions such as "Ethical considerations" or "Profanity."

Experimental Setup

SARM was developed using Llama-3 (3B and 8B) backbones. SAE pretraining utilized 50 million sequences from OpenWebText2, while RM training was conducted on the Skywork-Reward-Preference-80K-v0.2 dataset. Performance was primarily evaluated using RewardBench 2, which tests RM accuracy across safety, mathematics, and general alignment. Additionally, RM-Bench served as an out-of-distribution dataset to validate feature activation across diverse prompts.

Results and Discussion

Performance and Interpretability

SARM demonstrates that interpretability does not have to come at the cost of performance. SARM-4B (derived from an 8B backbone) achieved an overall score of 73.6 on RewardBench 2, outperforming significantly larger models like Llama-3.1-Tulu-3-70B (72.2) and closed-source models like GPT-4.1 (72.3). This "interpretability bonus" suggests that forcing the model to rely on sparse, meaningful features can enhance its evaluative capabilities.

Causal Controllability

A key advantage of SARM's linear value head is the ability to manually adjust behavior. By increasing the weight of specific features, such as "Safety," developers can shift the reward distribution to favor safe responses without retraining the entire model.

Reward distribution shift after manipulating specific feature weights, demonstrating the model's causal controllability.

Feature Discovery

The model successfully identified specific "Positive" features (e.g., Feature 60427: Ethical considerations) and "Negative" features (e.g., Feature 13950: Derogatory and offensive) that directly correlate with human preferences. This allows for fine-grained attribution of why a specific response received its score, transforming the reward assignment from a single number into a clear, weighted list of conceptual factors.

Conclusion and Future Work

SARM represents a significant step toward transparent and steerable alignment. By leveraging SAEs, it provides feature-level interpretability and dynamic controllability while maintaining state-of-the-art performance. Future research will address the challenge of "dead latents" to improve computational efficiency and explore multi-layer feature aggregation for more nuanced judgment.

Summary updated

Abstract

Large language models (LLMs) have been widely deployed across numerous fields. Reinforcement Learning from Human Feedback (RLHF) leverages reward models (RMs) as proxies for human preferences to align LLM behaviors with human values, making the accuracy, reliability, and interpretability of RMs critical for effective alignment. However, traditional RMs lack interpretability, offer limited insight into the reasoning behind reward assignments, and are inflexible toward user preference shifts. While recent multidimensional RMs aim for improved interpretability, they often fail to provide feature-level attribution and require costly annotations. To overcome these limitations, we introduce the Sparse Autoencoder-Enhanced Reward Model (SARM), a novel architecture that integrates a pretrained Sparse Autoencoder (SAE) into a reward model. SARM maps the hidden activations of LLM-based RM into an interpretable, sparse, and monosemantic feature space, from which a scalar head aggregates feature activations to produce transparent and conceptually meaningful reward scores. Empirical evaluations demonstrate that SARM facilitates direct feature-level attribution of reward assignments, allows dynamic adjustment to preference shifts, and achieves superior alignment performance compared to conventional reward models.
BibTeX
@inproceedings{zhang2026interpretable,
  author = {Shufen Zhang and Wei Shi and Sihang Li and Jiayi Liao and Tian Liang and Hengxing Cai and Xiang Wang},
  title = {{Interpretable Reward Model via Sparse Autoencoder}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i41.40783},
}
Figure