TL;DR
Current methods for interpreting Mixture-of-Experts (MoE) reward models rely heavily on routing weights, which only reveal which prompts an expert receives, not how it evaluates the responses. To fix this, the authors introduce CoCo (Contribution-Contrast), a response-level interpretation method that looks at the product of routing weights and expert score differences. CoCo faithfully captures both routing and preference behavior, yielding significantly more coherent, faithful, and specialized expert interpretations without sacrificing reward modeling accuracy.
What Problem Does This Paper Address?
Reward models (RMs) are the backbone of Reinforcement Learning from Human Feedback (RLHF), guiding large language models (LLMs) to align with human preferences. However, understanding exactly why a reward model prefers one response over another remains a notorious black box.
To tackle this, recent architectures use sparse Mixture-of-Experts (MoE) reward models, where a router sends prompts to specialized "expert" heads. Historically, researchers interpreted these experts by looking at the prompts with the highest routing weights. But there is a glaring gap here: routing weights only tell you the input domain an expert handles (e.g., "math problems"), not the preference dimension it uses to judge the output (e.g., "prefers step-by-step reasoning over concise answers").
As summarized in the paper's conceptual breakdown, existing methods either miss the response-level nuances or ignore the router's influence entirely:
Methods
The authors propose Contribution-Contrast (CoCo), which shifts the interpretation signal from pure routing weights to the actual contribution contrast of an expert to the final reward decision.
For a given prompt $x$ and a pair of chosen ($y_w$) and rejected ($y_l$) responses, CoCo defines the signed contribution contrast for expert $k$ as: $$\Delta c_k(x, y_w, y_l) = \pi_{\phi,k}(x) [r_{\theta_k}(x, y_w) - r_{\theta_k}(x, y_l)]$$
Where:
- $\pi_{\phi,k}(x)$ is the routing weight (how much the model trusts this expert for prompt $x$).
- $[r_{\theta_k}(x, y_w) - r_{\theta_k}(x, y_l)]$ is the expert score difference (how strongly this expert prefers the chosen response over the rejected one).
By multiplying these two terms, CoCo isolates examples where an expert is both heavily routed to and plays a decisive role in distinguishing the responses.
The framework consists of two main parts:
- CoCo-Based Regularization: The authors adapt standard interpretability regularizers (local sparsity, global balance, and expert diversity) to operate directly on these contribution contrast profiles during training. This forces the experts to learn distinct, sparse, and well-distributed preference dimensions.
- Interpretation Extraction: Post-training, they select the top 10 validation examples with the largest contribution contrast magnitudes for each expert and use an LLM to summarize the shared contrastive pattern (e.g., "prefers blunt, sardonic humor rather than polite, measured replies").
Datasets
The method was evaluated on two standard binary preference datasets:
- 700K: A large-scale dataset commonly used for standard RLHF workflow evaluations.
- Reddit: A dataset capturing more diverse, casual, and heterogeneous human preferences, making it an ideal testbed for mixture-of-experts modeling.
Key Insights and Ideas
- The "Where" vs. the "How": Purely router-based interpretation only captures the prompt-level context (the "where"). CoCo proves that to truly understand an expert, you must couple the context with its scoring behavior (the "how").
- Joint Optimization Matters: Simply applying CoCo post-hoc to a standard MoE isn't enough. Training the model with regularizers tied directly to the contribution contrast signal forces the network to develop cleaner, more specialized expert behaviors.
- Capturing Nuance over Topics: Instead of experts specializing broadly in "cooking questions," CoCo allows experts to specialize in "cooking questions answered with references to specific authoritative sources and links."
Key Results
CoCo was compared against several baselines, including Sparse MoE (SMoE) and Sparse Autoencoder (SAE)-based methods (SARM, WIMHF).
As shown in the main results table, CoCo consistently achieved the highest fidelity and decision faithfulness (measured by Expert-Model Agreement and Expert Removal Flip Rate) while maintaining highly competitive reward modeling accuracy (RM Acc.):
Crucially, in controlled comparisons on the same MoE model, CoCo significantly outperformed pure router-based or score-based signals in pinpointing the exact examples that dictate model decisions:
Human evaluations also strongly favored CoCo, rating its generated descriptions highest for both pattern coherence and overall description quality:
Limitations and Future Work
The authors note a few key limitations:
- Dependence on the Base Model: CoCo can only explain the structure that the underlying MoE reward model actually manages to learn. If the model fails to separate clean preference dimensions, CoCo's interpretations will reflect that messiness.
- Data-Centric Constraints: If a preference pattern is weak or absent in the training data, CoCo cannot magically recover it.
- Lack of Ground Truth: Because true latent human preference structures are unknown, interpretability must be evaluated via indirect proxies (like flip rates and human judgment) rather than absolute mathematical proof of perfect disentanglement.
Summary updated