Uncovering Factor-Level Preference to Improve Human-Model Alignment

Juhyun Oh, Eunsu Kim, Jiseon Kim, Wenda Xu, Inha Cha, William Yang Wang, Alice OhEMNLP 2025

TL;DR

Large Language Models (LLMs) often generate text that humans don't prefer, but why is hard to pinpoint with current methods. This paper introduces PROFILE, a framework to dissect human and LLM preferences at a granular, "factor-level" (e.g., conciseness, helpfulness). They discover LLMs over-prioritize length during generation, a key misalignment that can be addressed by leveraging their stronger alignment in evaluation tasks.

What Problem Does This Paper Address?

As LLMs become more sophisticated, aligning their output with human preferences is crucial. However, current evaluation methods often provide a coarse-grained "overall preference" score, masking the underlying reasons for misalignment. For instance, an LLM might produce a summary that's technically "good" by some metric but still feels off to a human reader. This paper tackles the challenge of understanding why LLMs and humans disagree on text quality by breaking down preferences into specific factors like "fluency," "helpfulness," "conciseness," and "intent alignment." They aim to uncover if LLMs prioritize these factors consistently with humans, especially when generating text versus when evaluating it.

Methods

The core contribution is the PROFILE (Probing Factors of Influence for Explainability) framework. It works in three main steps:

  1. Measure Overall Response-Level Preference: This involves determining which response is preferred in a given pair, both by humans and by the LLM itself. For LLMs, they use a "score-based prompting" method where the model generates responses conditioned on achieving a target quality score (1-5), and this target score serves as the preference proxy.
  2. Quantify Factor Manifestation: For each response, they automatically measure how strongly specific factors (defined in a taxonomy) are present. This uses a mix of rule-based methods (for length, novel words), established metrics (like UniEval for fluency/coherence), and LLM-based extraction (using GPT-4o for more subjective factors like intent alignment).
  3. Quantify Factor Influence and Compare: PROFILE calculates a "factor score" for each factor, representing its influence on overall preference decisions. By comparing these factor scores between humans and LLMs, they can quantify factor-level alignment. This allows for systematic comparisons across different tasks and settings (generation vs. evaluation).

Datasets

The study utilizes publicly available human preference datasets commonly used in preference optimization research:

  • Reddit TL;DR: Used for summarization tasks, providing human ratings across multiple dimensions.
  • StanfordHumanPreference-2 (SHP-2): Focuses on human preferences for helpfulness in responses to questions and instructions.
  • OpenAI WebGPT: Compares model-generated answers based on factual accuracy and usefulness.

These datasets were used for both training and evaluating the models.

Key Insights and Ideas

  • The "Length Bias" is Real and Problematic: A major finding is that LLMs, across various tasks, consistently over-prioritize output length. Humans, however, have more nuanced preferences that vary by task (e.g., conciseness for summaries, detail for QA). This length bias is a significant driver of misalignment.
  • Generation vs. Evaluation: A Tale of Two LLMs: LLMs show a striking difference in factor-level alignment between generation and evaluation tasks. While they struggle to align with human factor preferences when generating text, they exhibit much stronger alignment when evaluating or discriminating between responses.
  • Leveraging Evaluation for Better Generation: The "gen-eval gap" presents a key opportunity. The paper demonstrates that by using LLMs' superior evaluation capabilities, either through self-refinement during training or explicit feedback-driven generation, their alignment in generation tasks can be significantly improved.
  • Factor-Level Analysis is Crucial: Aggregate scores hide important details. PROFILE's factor-level approach reveals specific areas of misalignment that would otherwise be missed, paving the way for more targeted improvements.

Key Results

  • Summarization Misalignment: In summarization, humans prioritize intent alignment, formality, and source coverage, while LLMs consistently rank "length" as the top factor.
  • Evaluation Alignment is Stronger: For example, GPT-40 achieves a Kendall's τ correlation of 0.16 in generation but jumps to 0.82 in evaluation for summarization tasks. Similar improvements are seen across other models and tasks.
  • Improved Generation via Feedback: Using GPT-40 as an evaluator, feedback-driven generation approaches consistently improved alignment with human preferences, achieving performance comparable to GPT-40's own generation.
  • Cost-Effective Extraction: LLM-based factor extraction using GPT-40 was found to be relatively inexpensive, costing around $0.018 per sample for summarization.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Representativeness: The preference datasets used might not fully capture the entire spectrum of human preferences.
  • Human Evaluation Scale: Human evaluations were conducted on a limited scale due to budget constraints.
  • Focus on Post-Hoc Correction: The study primarily focuses on post-hoc correction methods.

Future work should investigate:

  • Incorporating Factors into Training: How to embed identified preference factors as signals during the training stage.
  • Embedding Signals in Datasets: Exploring how to integrate these signals into datasets used for preference optimization.
  • Broader Task Application: Applying and extending PROFILE to a wider range of tasks and domains.

Summary updated

BibTeX
@inproceedings{oh2025uncovering,
  author = {Juhyun Oh and Eunsu Kim and Jiseon Kim and Wenda Xu and Inha Cha and William Yang Wang and Alice Oh},
  title = {{Uncovering Factor-Level Preference to Improve Human-Model Alignment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure