Abstract
Preference mechanisms, including Reinforcement Learning from Human Feedback (RLHF), Reward Models (RMs), and LLM-as-a-Judge (LaaJ), are critical for aligning Large Language Models (LLMs) with human values. However, these mechanisms often function as "black boxes," offering little insight into the rationale behind specific preferences. This paper introduces a fully automated framework to explain preferences across diverse domains. By combining LLM-based concept discovery with a novel Hierarchical Multi-Domain Regression (HMDR) model, we decompose preferences into universal rules and domain-specific nuances. Our framework identifies over 600 unique concepts and achieves high predictive accuracy (~80% for LLM judges), demonstrating that automated explanations are both interpretable and actionable for improving model alignment.
Introduction
As LLMs become increasingly integrated into specialized fields, understanding the "why" behind their evaluations is paramount. Current explainability efforts are often limited by human-predefined attributes and single-dataset scopes, failing to account for the context-dependent nature of quality. For example, the criteria for a "good" response in a legal forum differ significantly from those in a safety-critical environment. We address this gap by proposing a scalable, multi-domain approach to preference explainability that uncovers the underlying drivers of both human and AI judgments.
Methodology
The proposed framework utilizes a four-stage pipeline to transform opaque preferences into transparent insights:
- Automated Concept Discovery: We leverage Gemini 1.5 Pro to analyze "chosen" vs. "rejected" response pairs, automatically identifying distinguishing concepts such as "Technical Depth," "Tone," or "Concealment."
- Concept-based Representations: Responses are mapped into "concept vectors" using comparative or scoring methods, allowing for a quantitative analysis of qualitative attributes.
- Hierarchical Multi-Domain Regression (HMDR): This white-box model serves as the analytical core. It partitions concept importance into shared weights ($b$), representing global preferences (e.g., "Clarity"), and domain-specific deviations ($s^{(d)}$), capturing local priorities (e.g., "Safety" in restricted domains).
- Explanation Generation: The model calculates "lifts"—visual metrics that illustrate how specific concepts influence the probability of a response being preferred within a given context.
Experimental Evaluation
We evaluated our framework across twelve preference mechanisms—including human preferences, two reward models (QRM, Skywork), and nine LLM judges (e.g., GPT-4o, Llama-3.1)—spanning eight diverse domains such as Reddit (Legal, Food, Travel), Stack Overflow, and the PKU-SafeRLHF dataset.
Key Results
- Predictive Accuracy: The HMDR model achieved an average accuracy of ~80% for explaining LLM judges and ~66% for human preferences, significantly outperforming black-box baselines.
- Contextual Nuance: The model successfully identified that concepts like "Concealment" are critical for safety-focused models but irrelevant for general advice, validating the multi-domain approach.
- Human vs. AI Discrepancies: Analysis revealed that while LLM judges prioritize "Accuracy," human evaluators often favor "Authority" and "Clarity," potentially rewarding responses that appear expert-like regardless of factual precision.
- Actionability: Through "Judge Hack" experiments, we demonstrated that responses optimized for the identified top concepts achieved win rates as high as 85.9% against standard outputs.
Discussion and Limitations
While the HMDR model provides high interpretability, its linear nature may miss complex, non-linear interactions between concepts. Additionally, while the "Judge Hack" provides strong evidence, the method primarily identifies correlations rather than strict causality. Future work will focus on developing causal graphs to better understand the decision-making hierarchies of preference models.
Conclusion
This research provides a scalable path toward transparent AI alignment. By automating the discovery and quantification of preference drivers across multiple domains, we enable developers to better understand, evaluate, and refine the mechanisms that govern LLM behavior.
Summary updated