Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast

Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Soyoung Oh, Isabel Valera, Vera DembergarXiv 2026 · arXiv

TL;DR

Current methods for interpreting Mixture-of-Experts (MoE) reward models rely heavily on routing weights, which only reveal which prompts an expert receives, not how it evaluates the responses. To fix this, the authors introduce CoCo (Contribution-Contrast), a response-level interpretation method that looks at the product of routing weights and expert score differences. CoCo faithfully captures both routing and preference behavior, yielding significantly more coherent, faithful, and specialized expert interpretations without sacrificing reward modeling accuracy.

What Problem Does This Paper Address?

Reward models (RMs) are the backbone of Reinforcement Learning from Human Feedback (RLHF), guiding large language models (LLMs) to align with human preferences. However, understanding exactly why a reward model prefers one response over another remains a notorious black box.

To tackle this, recent architectures use sparse Mixture-of-Experts (MoE) reward models, where a router sends prompts to specialized "expert" heads. Historically, researchers interpreted these experts by looking at the prompts with the highest routing weights. But there is a glaring gap here: routing weights only tell you the input domain an expert handles (e.g., "math problems"), not the preference dimension it uses to judge the output (e.g., "prefers step-by-step reasoning over concise answers").

As summarized in the paper's conceptual breakdown, existing methods either miss the response-level nuances or ignore the router's influence entirely:

Conceptual comparison of different interpretability signals used for interpreting reward models.

Methods

The authors propose Contribution-Contrast (CoCo), which shifts the interpretation signal from pure routing weights to the actual contribution contrast of an expert to the final reward decision.

For a given prompt $x$ and a pair of chosen ($y_w$) and rejected ($y_l$) responses, CoCo defines the signed contribution contrast for expert $k$ as: $$\Delta c_k(x, y_w, y_l) = \pi_{\phi,k}(x) [r_{\theta_k}(x, y_w) - r_{\theta_k}(x, y_l)]$$

Where:

  • $\pi_{\phi,k}(x)$ is the routing weight (how much the model trusts this expert for prompt $x$).
  • $[r_{\theta_k}(x, y_w) - r_{\theta_k}(x, y_l)]$ is the expert score difference (how strongly this expert prefers the chosen response over the rejected one).

By multiplying these two terms, CoCo isolates examples where an expert is both heavily routed to and plays a decisive role in distinguishing the responses.

The framework consists of two main parts:

  1. CoCo-Based Regularization: The authors adapt standard interpretability regularizers (local sparsity, global balance, and expert diversity) to operate directly on these contribution contrast profiles during training. This forces the experts to learn distinct, sparse, and well-distributed preference dimensions.
  2. Interpretation Extraction: Post-training, they select the top 10 validation examples with the largest contribution contrast magnitudes for each expert and use an LLM to summarize the shared contrastive pattern (e.g., "prefers blunt, sardonic humor rather than polite, measured replies").

Datasets

The method was evaluated on two standard binary preference datasets:

  • 700K: A large-scale dataset commonly used for standard RLHF workflow evaluations.
  • Reddit: A dataset capturing more diverse, casual, and heterogeneous human preferences, making it an ideal testbed for mixture-of-experts modeling.

Key Insights and Ideas

  • The "Where" vs. the "How": Purely router-based interpretation only captures the prompt-level context (the "where"). CoCo proves that to truly understand an expert, you must couple the context with its scoring behavior (the "how").
  • Joint Optimization Matters: Simply applying CoCo post-hoc to a standard MoE isn't enough. Training the model with regularizers tied directly to the contribution contrast signal forces the network to develop cleaner, more specialized expert behaviors.
  • Capturing Nuance over Topics: Instead of experts specializing broadly in "cooking questions," CoCo allows experts to specialize in "cooking questions answered with references to specific authoritative sources and links."

Key Results

CoCo was compared against several baselines, including Sparse MoE (SMoE) and Sparse Autoencoder (SAE)-based methods (SARM, WIMHF).

As shown in the main results table, CoCo consistently achieved the highest fidelity and decision faithfulness (measured by Expert-Model Agreement and Expert Removal Flip Rate) while maintaining highly competitive reward modeling accuracy (RM Acc.):

Task performance and interpretability results of different interpretable reward models on 700K and Reddit.

Crucially, in controlled comparisons on the same MoE model, CoCo significantly outperformed pure router-based or score-based signals in pinpointing the exact examples that dictate model decisions:

Reddit results of interpretations derived from different signals.

Human evaluations also strongly favored CoCo, rating its generated descriptions highest for both pattern coherence and overall description quality:

Figure 1: Human annotators rate CoCo highest in both pattern coherence and description quality.

Limitations and Future Work

The authors note a few key limitations:

  • Dependence on the Base Model: CoCo can only explain the structure that the underlying MoE reward model actually manages to learn. If the model fails to separate clean preference dimensions, CoCo's interpretations will reflect that messiness.
  • Data-Centric Constraints: If a preference pattern is weak or absent in the training data, CoCo cannot magically recover it.
  • Lack of Ground Truth: Because true latent human preference structures are unknown, interpretability must be evaluated via indirect proxies (like flip rates and human judgment) rather than absolute mathematical proof of perfect disentanglement.

Summary updated

Abstract

Reward models are central to learning from human preferences, yet identifying what drives their predictions remains challenging. Recent sparse Mixture-of-Experts (MoE) reward models seek to improve interpretability by routing prompts to specialized experts and characterizing experts through examples with high routing weights. However, routing weights only reveal which prompts an expert $\textit{receives}$, not how it $\textit{judges}$ responses, providing only a partial account of expert behavior. We therefore propose $\textbf{Co}$ntribution-$\textbf{Co}$ntrast ($\textbf{CoCo}$) response-level interpretation, which faithfully characterizes experts' roles using chosen-rejected response pairs with the largest contribution contrasts, jointly capturing routing and preference behavior. Across automatic and human evaluations, CoCo yields more coherent, faithful, and specialized interpretations than router-based, score-based, and sparse autoencoder-based alternatives while maintaining competitive reward modeling accuracy. To the best of our knowledge, this is the first systematic study of interpretation methods for MoE reward models.
BibTeX
@misc{wang2026beyond,
  author = {Yifan Wang and Jinyi Mu and Mayank Jobanputra and Yu Wang and Soyoung Oh and Isabel Valera and Vera Demberg},
  title = {{Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast}},
  year = {2026},
  eprint = {2608.06400},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.06400},
}
Figure