Multi-Domain Explainability of Preferences

Nitay Calderon, Liat Ein-Dor, Roi Reichart

TL;DR

This paper introduces a novel, automated method for explaining why preference mechanisms (like human judgment, LLM-as-a-judge, and reward models) make specific choices. By discovering and representing concepts that drive these preferences across diverse domains, the work offers a more transparent and trustworthy way to understand and align large language models.

What Problem Does This Paper Address?

In the rapidly evolving world of Large Language Models (LLMs), ensuring they align with human values and preferences is paramount. This is often achieved through "preference mechanisms" – systems that learn what humans (or other LLMs) prefer. However, the underlying reasons why these mechanisms favor one response over another remain largely a black box. Existing methods often rely on pre-defined concepts or manual annotations, limiting scalability and potentially introducing bias. This paper tackles the challenge of automatically discovering and explaining these preference drivers across various domains, moving beyond single-task or single-domain analyses.

Methods

The authors propose a four-stage, fully automated pipeline:

  1. Concept Discovery: An LLM is used to identify concepts (e.g., "clarity," "factuality," "helpfulness") that differentiate between preferred and rejected responses. This stage is designed to discover both general and domain-specific concepts.
  2. Concept-Based Representations: Each user query-response triplet is transformed into a "concept vector," where each dimension represents the presence or strength of a discovered concept. Two representation methods are explored: "Comp-rep" (a binary judgment of which response better aligns with a concept) and "Score-rep" (a 1-7 scale score for each response on a concept).
  3. Hierarchical Multi-Domain Regression (HMDR): A novel white-box model is introduced to predict preferences based on these concept vectors. The HMDR model is designed to capture both domain-general (shared) and domain-specific concept effects, promoting interpretability and generalization.
  4. Explanation Analysis: The learned weights of the HMDR model are analyzed to quantify concept importance, providing local (per-example) and global (overall) explanations.

Datasets

The study utilizes a curated dataset spanning eight diverse domains, including:

  • Reddit-based domains: General, Travel, Food, Legal, Picks (book/movie recommendations). Preference labels are derived from community upvotes.
  • Stack Overflow: For software-related questions.
  • RLHF datasets: PKU (safety-focused) and UltraFeedback (general RLAIF).

Each domain contains 800 examples, with 400 used for concept discovery and 400 for training and testing.

Key Insights and Ideas

  • Automated Concept Discovery is Feasible: The paper demonstrates that LLMs can effectively discover human-interpretable concepts that explain preferences, reducing reliance on manual annotation.
  • Preferences are Multi-faceted and Domain-Dependent: Concepts influencing preferences vary significantly across domains. For instance, "concealment" is crucial in safety domains but less so in recipe domains. The HMDR model's ability to capture these domain-specific nuances is a key innovation.
  • White-Box Models Offer Explainability: By using a white-box HMDR model, the authors can derive direct explanations from model weights, offering a more transparent alternative to black-box approaches.
  • Explanations Have Practical Utility: The research shows that these discovered concepts can be used to actively improve LLM behavior, either by guiding response generation or by helping LLM-as-a-judge models resolve ambiguous cases.

Key Results

  • Strong Predictive Performance: The proposed explainable method achieves prediction accuracy comparable to, and sometimes better than, state-of-the-art black-box LLM-as-a-Judge and reward models. For human preferences, it achieved 65.5% accuracy (Comp-rep) and 66.0% (Score-rep) on average across domains.
  • Effective Multi-Domain Learning: The HMDR model demonstrates robust performance both in-domain and out-of-domain, outperforming simpler models.
  • Judge Hack Success: LLM judges consistently preferred responses generated with guidance from the discovered concepts over vanilla responses, showing that these concepts capture meaningful judge behavior. Win rates improved by up to 20.8 points over random concepts.
  • Tie Break Improvement: Using explanations to resolve ambiguous LLM-as-a-Judge predictions led to consistent gains in alignment with human preferences, up to 10%.
  • Human-Aligned Concepts Recovered: The analysis of shared concepts successfully recovered findings from prior studies on human preferences, validating the method's ability to capture known insights.

Limitations and Future Work

  • Linearity Assumption: The HMDR model uses a linear function to relate concepts to preferences. While this enhances interpretability, it might not fully capture complex, non-linear decision processes. Future work could explore non-linear extensions.
  • Causality: The explanations are correlational, not causal. Discovering the causal graph over concepts is a promising direction.
  • Computational Costs: The method relies on numerous LLM calls, particularly for concept discovery and representation. While sparsity in the HMDR model helps, it's still more computationally intensive than simple zero-shot prompting.
  • Evaluation Challenges: Quantifying explanation quality remains difficult. The authors propose application-driven evaluations (Judge Hack, Tie Break) as a proxy for usefulness and faithfulness, but standardized metrics are still needed.

The authors envision this work as a step towards a new paradigm for explainability in the LLM era, enabling more transparent, trustworthy, and controllable AI systems.

Summary updated

Abstract

Preference mechanisms, such as human preference, LLM-as-a-Judge (LaaJ), and reward models, are central to aligning and evaluating large language models (LLMs). Yet, the underlying concepts that drive these preferences remain poorly understood. In this work, we propose a fully automated method for generating local and global concept-based explanations of preferences across multiple domains. Our method utilizes an LLM to identify concepts that distinguish between chosen and rejected responses, and to represent them with concept-based vectors. To model the relationships between concepts and preferences, we propose a white-box Hierarchical Multi-Domain Regression model that captures both domain-general and domain-specific effects. To evaluate our method, we curate a dataset spanning eight challenging and diverse domains and explain twelve mechanisms. Our method achieves strong preference prediction performance, outperforming baselines while also being explainable. Additionally, we assess explanations in two application-driven settings. First, guiding LLM outputs with concepts from LaaJ explanations yields responses that those judges consistently prefer. Second, prompting LaaJs with concepts explaining humans improves their preference predictions. Together, our work establishes a new paradigm for explainability in the era of LLMs.
BibTeX
@inproceedings{calderon2025multi,
  author = {Nitay Calderon and Liat Ein-Dor and Roi Reichart},
  title = {{Multi-Domain Explainability of Preferences}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure