Sample-efficient Personalization of Large Language Model Alignment with Multidimensional User Preference Data

Zhen-Yu Zhang, Jerry Huang, Gang Niu, Takashi Ishida, Masashi SugiyamaarXiv 2026 · arXiv

Aligning Large Language Models (LLMs) with human values typically relies on Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). While "online" versions of these methods—where the model generates new responses for evaluation in real-time—are highly effective, they are notoriously expensive. Every query requires a human (or a high-end "LLM-as-a-judge") to label preferences, creating a massive bottleneck for personalization.

A new paper from researchers at RIKEN and the University of Tokyo introduces MPG-DPO (Multidimensional Preference-guided Online DPO), a method designed to make this process significantly more sample-efficient by focusing only on the comparisons that actually matter to a specific user.

TL;DR

MPG-DPO improves the sample efficiency of online LLM alignment by constructing a personalized "preference cone" from historical user data and low-cost automated evaluations. By filtering out responses that are clearly inferior based on a user's past trade-offs, it directs expensive feedback toward genuinely unresolved disagreements, outperforming standard alignment methods under tight query budgets.

What Problem Does This Paper Address?

Most active learning methods for LLM alignment select pairs for comparison based on model uncertainty. If the model isn't sure which of two responses is better, it asks the user.

However, the authors argue that model uncertainty does not equal user uncertainty. For example, in code generation, a user might have already demonstrated that they are willing to sacrifice a bit of "conciseness" for a significant gain in "accuracy." Even if the LLM is uncertain about a new pair involving these two traits, the user’s past behavior might already imply a clear winner. Querying the user in this scenario is a waste of a limited annotation budget.

Methods

The core innovation of MPG-DPO is the use of multidimensional scores to build a geometric representation of user preferences.

  1. Automated Verifiers: Instead of relying solely on expensive human labels, the system uses low-cost, task-specific verifiers (like the Prometheus-2 model) to score responses on multiple dimensions: correctness, completeness, clarity, conciseness, and reasoning quality.
  2. The Preference Cone: Each time a user chooses response A over response B, they reveal a "preferred direction" in this multidimensional score space. MPG-DPO aggregates these directions into a personalized preference Pareto cone.
  3. Filtering and Selection:
    • Filtering: Using ridge regression with confidence bounds, the algorithm identifies and discards candidate responses that are "confidently dominated" by others within the preference cone.
  4. Max-Min Selection: From the remaining candidates, it selects a pair that maximizes the "weaker of two opposing preference margins." This ensures the selected pair represents a genuine, unresolved trade-off for that specific user.
Figure 1: Summary of the proposed MPG-DPO approach. Given an input prompt, candidate responses are sampled and filtered using the preference cone, after which a pair is selected to maximize the weaker of two opposing preference margins. The selected responses are evaluated and used to update the cone and policy.

The algorithm then updates both the LLM policy (using standard DPO) and the preference cone based on the new label.

Datasets

The researchers evaluated MPG-DPO across several diverse benchmarks:

  • IMDB: Sentiment generation.
  • TL;DR: Summarization win rates against human-written summaries.
  • AlpacaEval 2 & Arena-Hard: General-purpose instruction following and complex reasoning.
  • Oracles: They used Kimi-K3 to simulate human preference feedback and Prometheus-2 as the sub-objective oracle for multidimensional scoring.

Key Insights and Ideas

  • User-Perspective Uncertainty: The "aha moment" here is shifting the focus from what the model doesn't know to what the user hasn't decided yet.
  • Geometry of Preference: Treating preferences as a cone in a vector space allows the model to use linear algebra to "deduce" preferences for new pairs without asking the user.
  • Vanishing Regret: The authors provide a theoretical analysis showing that as the model learns, the fraction of queries spent on "cone-dominated" (and thus wasteful) responses vanishes with high probability.

Key Results

MPG-DPO consistently outperformed strong baselines like XPO (Exploratory Preference Optimization) and APL (Active Preference Learning) across three different 7B-parameter model families: Qwen2.5, Mistral, and OLMo-2.

Table 1: Main results across three 7B policies under the same preference-query budget. Higher is better for every metric. Each entry reports mean ± standard deviation over three seeds.

As shown in the results table, MPG-DPO achieved the highest win rates across almost all metrics. For instance, on the challenging Arena-Hard benchmark, MPG-DPO pushed the Qwen2.5-7B model to a 56.04% win rate, compared to 54.68% for standard Online DPO, using the exact same query budget. Ablation studies further confirmed that both the "cone filtering" and the "trade-off selection" components are essential for these gains.

Limitations and Future Work

  • Verifier Quality: The method's success depends on the accuracy of the low-cost multidimensional verifiers. If the verifiers provide noisy or biased scores, the preference cone will be malformed.
  • Static Preferences: The current framework assumes a user's preference cone is relatively stable. Future work could explore "drifting" preferences where a user's priorities change over time.
  • Computational Complexity: While the authors used the "double-description method" for efficiency, maintaining and updating high-dimensional cones can become computationally intensive as the number of sub-objectives increases.

Summary updated

Abstract

Preference alignment has become a central tool to align large language models (LLMs) with human values and desired behavior, and its online variant can further advance their capabilities beyond what static data alone offer by repeatedly generating and labeling new responses with human feedback. But costly, time-consuming annotations limit its scalability, making sample efficiency essential, and existing methods typically seek to improve it by expanding coverage of the response space or selecting pairs with the highest uncertainty for comparison. Although they are effective, these methods quantify uncertainty from the perspective of the LLM, which may differ from the uncertainty perceived by the user: a pair can appear highly uncertain to the LLM even when past comparisons from the user already imply a clear ordering. In this paper, we propose multidimensional preference-guided online direct preference optimization (MPG-DPO), which leverages low-cost multidimensional evaluations from automated verifiers on the user historical comparisons to construct a personalized preference cone guiding response selection for online preference alignment. Specifically, it first discards responses that are confidently dominated under the estimated user preference, then selects two responses that are competitive under the retained preference cone yet exhibit the strongest unresolved disagreement. Theoretical analysis shows that, with high probability, the fraction of queries spent on cone-dominated responses vanishes as learning progresses. Experiments across multiple LLM families show that MPG-DPO outperforms strong contenders under matched query budgets.
BibTeX
@misc{zhang2026sample,
  author = {Zhen-Yu Zhang and Jerry Huang and Gang Niu and Takashi Ishida and Masashi Sugiyama},
  title = {{Sample-efficient Personalization of Large Language Model Alignment with Multidimensional User Preference Data}},
  year = {2026},
  eprint = {2611.22210},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22210},
}
Figure