ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data

Haoran Gu, Handing Wang, Yi Mei, Mengjie Zhang, Yaochu Jin

TL;DR

ParetoHqD is a fast, offline multiobjective alignment algorithm that trains Large Language Models (LLMs) to balance conflicting human values (like helpfulness vs. harmlessness) using a fraction of the data. By representing user preferences as geometric directions and filtering for data near the Pareto front, it achieves superior alignment diversity and drastically reduces language output collapse while cutting training costs by over half compared to state-of-the-art baselines.

What Problem Does This Paper Address?

Aligning LLMs with human values is rarely a single-objective problem. Users often have multi-dimensional, conflicting expectations—for instance, wanting a model to be highly helpful but also strictly harmless. Current multiobjective alignment methods typically rely on linear scalarization, which blends multiple reward scores into a single value using weighted averages.

However, the authors identify three critical flaws in this standard approach:

  1. Ambiguous Preference Signals: Linear scalarization can give identical scores to completely different trade-offs, confusing the model and causing "language output collapse" (e.g., severe text repetition or truncated responses).
  2. Failure on Non-Convex Fronts: It geometrically fails to capture optimal trade-offs if the true Pareto front is concave.
  3. Data Imbalance & Inefficiency: Standard offline datasets are heavily skewed toward mediocre, average-scoring responses. Training on the entire dataset is computationally expensive and forces the model to overlearn low-quality data distributions.
Figure 2: Schematic diagram of issues in offline methods for representing preferences.

Methods

To solve these issues, the authors introduce ParetoHqD, an offline method that treats alignment as a true multiobjective optimization problem.

Instead of scalarizing rewards, ParetoHqD represents human preferences as preference directions (rays pointing from the ideal reward vector toward a specific compromise point in the objective space). It then uses a two-stage Supervised Fine-Tuning (SFT) pipeline focused entirely on a tiny subset of "high-quality" Pareto-optimal data:

  • Stage 1 (Pareto Filtering & SFT): The algorithm identifies the top layers of the Pareto front within the offline dataset. For each target preference direction, it extracts only the closest $k$ data points (e.g., $k=100$) and performs a lightweight SFT.
  • Stage 2 (Data Augmentation & Refinement): To prevent overfitting from the small dataset size, the Stage 1 models generate new responses to random prompts. The algorithm finds the new Pareto front from these generations, extracts the best matching data, and performs a final round of fine-tuning.
Figure 1: Comparison with other baselines, where SFT refers to supervised fine-tuning on LLMs.

Datasets

The method was evaluated on two distinct text generation benchmarks:

  • Helpful Assistant Task: Utilizing the HH-RLHF dataset (160k prompts), evaluated using three open-source reward models: harmless, helpful, and humor.
  • Reddit Summary Task: Utilizing the summarize_from_feedback dataset (14.9k posts), evaluated using two reward models: summary quality and faithfulness.

Key Insights and Ideas

  • Less is More for Alignment: A model's core knowledge is absorbed during pre-training; alignment simply teaches it style and formatting. ParetoHqD proves you only need a tiny fraction of data points (as few as 100 per preference) to successfully align a model, provided those points are strictly Pareto-optimal.
  • Direction Over Weights: Mapping preferences as geometric rays rather than scalar weights ensures the model receives distinct, unambiguous optimization signals. This prevents the model from learning conflicting patterns that lead to text degeneration.
  • Stable Scaling for Preferences: Because of a clever data augmentation strategy, increasing the number of target human preferences ($N$) doesn't scale the computational cost linearly. The most expensive step (data generation) remains constant regardless of how many distinct user profiles you want to align.

Key Results

ParetoHqD was benchmarked against five prominent baselines, including MORLHF, Rewarded Soups, and the state-of-the-art offline method, RiC.

  • Superior Pareto Fronts: As shown in the evaluation plots, ParetoHqD consistently achieved a wider, higher-performing, and more diverse Pareto front across both two-objective and three-objective tasks.
  • Drastic Reduction in Text Collapse: While linear scalarization baselines suffered from language collapse in 30% to 49% of their outputs, ParetoHqD maintained a remarkably low collapse rate of just 7.03%.
  • High Efficiency: ParetoHqD required only 55.87 GPU hours for training, representing a ~58% reduction in computational cost compared to the RiC baseline (~132 GPU hours).
Figure 5: Results of two tasks with normalized rewards, where each point represents the average rewards evaluated on the test set corresponding to a user preference. For the Helpful Assistant task (a), (d) and the Reddit Summary task (b), we set the human preferences ω to [0.0,1.0], [0.1,0.9], [0.2,0.8], [0.3,0.7], [0.4,0.6], [0.5,0.5], [0.6,0.4], [0.7,0.3], [0.8,0.2], [0.9,0.1] and [1.0,0.0], respectively. For the Helpful Assistant task with three objectives (c), we set the human preferences ω to [0.0,0.0,1.0], [0.0,1.0,0.0], [0.1,0.1,0.8], [0.1,0.8,0.1], [0.2,0.2,0.6], [0.2,0.6,0.2], [0.4,0.4,0.2], [0.6,0.2,0.2], [0.8,0.1,0.1], [0.33,0.33,0.33] and [1.0,0.0,0.0]. The 5 preferences for MORLHF and MODPO are highlighted in italics.

Limitations and Future Work

While ParetoHqD scales efficiently when adding more user preferences, scaling to a higher number of alignment objectives (e.g., balancing 5+ reward models simultaneously) significantly increases the computational overhead of locating the Pareto-optimal data layers. The authors note that developing lightweight adaptation strategies to efficiently incorporate a larger number of objectives is a key direction for future work.

Summary updated

Abstract

Aligning large language models with multiple human expectations and values is crucial for ensuring that they adequately serve a variety of user needs. To this end, offline multiobjective alignment algorithms such as the Rewards-in-Context algorithm have shown strong performance and efficiency. However, inappropriate preference representations and training with imbalanced reward scores limit the performance of such algorithms. In this work, we introduce ParetoHqD that addresses the above issues by representing human preferences as preference directions in the objective space and regarding data near the Pareto front as "high-quality" data. For each preference, ParetoHqD follows a two-stage supervised fine-tuning process, where each stage uses an individual Pareto high-quality training set that best matches its preference direction. The experimental results have demonstrated the superiority of ParetoHqD over five baselines on two multiobjective alignment tasks.
BibTeX
@inproceedings{gu2026paretohqd,
  author = {Haoran Gu and Handing Wang and Yi Mei and Mengjie Zhang and Yaochu Jin},
  title = {{ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models using Pareto High-quality Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i21.38799},
}
Figure