Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling

Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera DembergarXiv 2026 · arXiv

TL;DR

Standard reward models in RLHF assume a single, universal set of human values, failing to capture individual preference diversity. This paper introduces a sparse Mixture-of-Experts (MoE) reward model that uses explicit interpretability regularizers to train specialized, human-understandable experts from standard binary preference data. This approach enables highly efficient, transparent test-time personalization with minimal data.

What Problem Does This Paper Address?

Aligning Large Language Models (LLMs) via Reinforcement Learning from Human Feedback (RLHF) typically relies on a single global reward model. However, human preferences are inherently pluralistic and subjective—what one person finds funny or clear, another might find annoying or verbose.

While some methods use fine-grained annotations or user-specific labels to handle this heterogeneity, such data is expensive to collect. Recent unsupervised data-driven methods attempt to learn latent preference components directly from binary data, but they suffer from "entanglement": the learned components are opaque, incoherent, and hard to interpret or audit. This paper tackles the challenge of learning disentangled, specialized, and human-interpretable preference components using only standard binary preference data.

Methods

The authors propose a sparse Mixture-of-Experts (MoE) reward model architecture. Instead of a monolithic reward function, the model consists of $K$ experts (implemented as linear heads over a frozen backbone model, GRM-Llama3.2-3B) and a router (a one-hidden-layer MLP).

The core innovation lies in the training objective, which adds three explicit interpretability regularization terms to the standard reward modeling loss:

  1. Local Sparsity ($L_{ls}$): Minimizes the entropy of the router's output distribution for each input, forcing the router to confidently select only a few (or one) experts per prompt.
  2. Global Balance ($L_{gb}$): Maximizes the entropy of the average routing weights across a batch to prevent "expert collapse" (where the model relies entirely on a single expert).
  3. Expert Diversity ($L_{div}$): Minimizes the pairwise Pearson correlation between the reward differences of different experts, pushing them to learn functionally distinct preference signals.

For test-time personalization, the experts are frozen, and the router's weights are efficiently updated using lightweight algorithms (like the Hedge algorithm) on a small user-specific dataset.

Figure 1: Illustration of the pipeline for training a sparse MoE reward model, interpreting its experts, adapting it to individual preferences and inspecting the adaptation. An MoE reward model is trained on standard binary preference data with interpretability regularization, and its experts can be interpreted by summarizing their top- activating examples. During test time, the router can be fine-tuned to fit personalized preferences, while changes in routing weights provide a qualitative view of how the model adapts to these preferences.

Datasets

  • Stanford Human Preferences (SHP): Sourced from Reddit, containing domain annotations used here to validate unsupervised category recovery.
  • Reasonable Preference Reversal (RPR): Contains pairs of responses preferred under different attributes (e.g., clarity vs. creativity), used for attribute steering and personalization.
  • Preference-700K (700K): A large-scale real-world binary preference dataset used to evaluate interpretability and personalization in practical settings.
  • PersonalLLM: A synthetic dataset used to evaluate user-level personalization performance.

Key Insights and Ideas

  • Sparsity Breeds Monosemanticity: Just as sparse autoencoders (SAEs) help find interpretable features in LLM activations, forcing an MoE router to be sparse forces the experts to specialize in distinct, clean semantic concepts rather than fuzzy mixtures.
  • Functional vs. Assignment Specialization: True interpretability requires that an expert not only receives specific types of inputs (assignment) but also performs better on them (functional specialization). The diversity loss ensures experts actually learn different reward functions.
  • Interventional Control: Because the experts are cleanly disentangled, you can steer the model's overall preference post-hoc simply by manually shifting the routing weights, proving that the learned interpretability is functional, not just correlational.

Key Results

  • Unsupervised Category Recovery: On the SHP dataset, the sparse MoE successfully recovered all 4 major latent categories (Academic, Professional, Cooking, Fiction) with an Expert Purity of 85.60%, whereas standard MoE baselines collapsed and only covered 2 categories.
  • Effective Attribute Steering: Manually boosting a specific expert's weight shifted the model's preference toward the target attribute, achieving a 43.83% average flip rate on RPR, significantly outperforming vanilla MoE and MiCRo baselines.
Figure 2: Attribute steering results on RPR. Sparse MoE achieves the strongest steering performance. The improvements are statistically significant (α < 0.05).
  • Massive Personalization Gains: With just 50 adaptation examples on the RPR dataset, the sparse MoE model achieved a 25.81-point improvement in overall personalization accuracy, vastly outperforming all baselines (which gained less than 10 points).
  • Human-Verified Interpretability: Human annotators rated the sparse MoE as having significantly higher pattern coherence (3.88/5) compared to baselines, confirming that the automated natural language descriptions generated for each expert were highly accurate.
Figure 3: Human evaluation of model interpretability. Human annotators find that sparse MoE has the most interpretable routing patterns. The improvements in pat- tern coherence are statistically significant (α < 0.05).

Limitations and Future Work

  • Hyperparameter Sensitivity: The model introduces three new regularization strengths ($\lambda_{ls}, \lambda_{gb}, \lambda_{div}$), requiring careful tuning to avoid model collapse or unstable training.
  • Data Dependency: Because the method is entirely data-driven, the emergence of specific specialized experts depends heavily on the latent structures present in the training data; you cannot pre-define what experts will form.
  • Performance Trade-off: The structural constraints of interpretability regularization cause a minor drop (~2%) in universal preference modeling accuracy compared to unconstrained MoE baselines.
  • Future Directions: The authors suggest exploring hierarchical expert interpretation and introducing additional constraints to reduce expert redundancy.

Summary updated

Abstract

Preference modeling plays a central role in reinforcement learning from human feedback (RLHF), enabling large language models (LLMs) to align with human values. However, most existing approaches assume a universal reward function, neglecting the diversity and heterogeneity of human preferences. To address this limitation without additional annotation costs, recent work has proposed learning multiple preference components from binary data and combining them to model individual preferences. Nevertheless, these components often fail to capture coherent and disentangled patterns, limiting their interpretability and effectiveness for personalization. In this work, we propose a sparse Mixture-of-Experts (MoE) reward model that encourages sparse routing and expert diversity during training on binary preference data. Across controlled and real-world experiments, sparse MoE learns interpretable routing patterns and specialized experts. It also improves test-time personalization, and post-adaptation shifts in expert weights provide a qualitative lens for analyzing how the model adapts to personalized preferences.
BibTeX
@misc{wang2026sparse,
  author = {Yifan Wang and Jinyi Mu and Mayank Jobanputra and Yu Wang and Ji-Ung Lee and Soyoung Oh and Isabel Valera and Vera Demberg},
  title = {{Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling}},
  year = {2026},
  eprint = {2606.04284},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.04284},
}

Loading related content...

Figure