PMGDA: A Preference-Based Multiple Gradient Descent Algorithm

Xiaoyuan Zhang, Xi Lin, Qingfu Zhang

TL;DR

This paper introduces PMGDA, a novel gradient-based algorithm for multi-objective optimization that can find Pareto solutions tailored to a user's specific preferences. It tackles the challenge of large-scale problems, like those in deep learning, by efficiently optimizing multiple objectives and a user-defined preference constraint simultaneously.

What Problem Does This Paper Address?

Many real-world machine learning problems involve optimizing multiple, often conflicting, objectives simultaneously. Think of a recommendation system trying to maximize user engagement while minimizing cost, or a face recognition system balancing accuracy across different facial attributes. In these scenarios, there isn't a single "best" solution, but rather a set of "Pareto optimal" solutions, each offering a different trade-off. The challenge is that users often have specific preferences for which trade-off they'd like to achieve. Existing methods struggle to find these preference-aligned Pareto solutions, especially in large-scale deep learning models where the number of parameters can be in the thousands or millions. This paper aims to bridge this gap by developing an algorithm that can efficiently find a Pareto solution that precisely matches a decision-maker's desired outcome.

Methods

The core of PMGDA is a "predict-and-correct" framework.

  • Preference Function: The algorithm first defines a preference function, h(θ), which mathematically captures the user's desired outcome. This can be for an "exact" Pareto solution (where objectives align with a specific preference vector) or for a "region of interest" (ROI) on the Pareto front.
  • Predict-and-Correct: At each step, the algorithm first "predicts" a direction that simultaneously reduces all objectives while minimally impacting the preference constraint. This is achieved by solving a constrained optimization problem. Then, it "corrects" this direction to ensure the preference constraint is met, even if it means slightly sacrificing some objective improvements. This correction step is crucial for steering the solution towards the desired preference.
  • Efficient Gradient Reuse: To handle large-scale problems, PMGDA cleverly reuses gradients. It leverages the fact that the preference function often depends on the objective functions, allowing it to estimate the gradient of the preference function from the gradients of the objectives, thus avoiding extra computation.
  • Dual Formulation: For the prediction step, PMGDA uses a dual formulation of the optimization problem, which significantly reduces the number of variables to optimize from the model's parameter count (n) to the number of objectives (m), making it much more efficient.

Datasets

The paper evaluates PMGDA on several benchmarks:

  • Standard Multiobjective Benchmarks: ZDT1, MAF1, and DTLZ2 are used to test the algorithm's ability to find exact Pareto solutions.
  • Fairness Classification Problems: Adult, Compas, and Credit datasets are used to evaluate PMGDA on multi-objective classification tasks, where objectives include cross-entropy loss and fairness metrics (like the Difference of Equality of Opportunity loss).
  • Multi-Objective Reinforcement Learning (MORL) Tasks: Swimmer, HalfCheetah, and Ant tasks are used to demonstrate PMGDA's effectiveness in finding Pareto solutions that satisfy Region of Interest (ROI) constraints in complex RL scenarios.

Key Insights and Ideas

  • Flexible Preference Handling: PMGDA's ability to handle both "exact" Pareto solutions and "regions of interest" (ROIs) makes it highly versatile for various user needs.
  • Efficiency in Large-Scale Problems: By reformulating the optimization problem in its dual form and reusing gradients, PMGDA achieves significant efficiency gains, making it practical for deep learning models with millions of parameters.
  • First to Address ROI in Gradient-Based MOO: PMGDA is highlighted as the first gradient-based method capable of successfully identifying Pareto solutions that meet specific ROI constraints, a challenging task in multi-task learning due to noisy objective estimations.
  • Predict-and-Correct is Key: The iterative predict-and-correct mechanism is crucial for balancing the simultaneous optimization of multiple objectives and the user's preference, leading to more precise and stable convergence.

Key Results

  • Superior Precision and Convergence: PMGDA consistently outperforms existing methods like EPO and COSMOS in finding precise Pareto solutions that align with user preferences, especially on complex problems. It also demonstrates faster convergence than methods like mTche.
  • Successful ROI Identification: PMGDA successfully finds Pareto solutions satisfying ROI constraints in fairness classification and MORL tasks, a capability not demonstrated by previous methods.
  • Quantitative Improvements: In synthetic problems, PMGDA achieves better cross-angle and PBI indicator values compared to baselines. For fairness classification, it shows competitive or superior performance in terms of cross-angle and PBI. In MORL, it shows faster convergence and better final performance.

Limitations and Future Work

  • Gradient-Based Limitation: As a gradient-based method, PMGDA cannot guarantee finding globally optimal solutions and may get stuck in local optima.
  • Future Work: The authors suggest exploring the integration of gradient-based methods with evolutionary approaches to overcome local optima and enhance solution quality. They also aim to further investigate how to adapt PMGDA for more complex scenarios and potentially improve its robustness.

Summary updated

Abstract

It is desirable in many multi-objective machine learning applications, such as multi-task learning with conflicting objectives, multi-objective reinforcement learning, to find a Pareto solution that can match a given preference of a decision maker. These problems are often large-scale with available gradient information but cannot be handled very well by the existing algorithms. To tackle this issue, this paper proposes a novel predict-and-correct framework for locating a Pareto solution that fits the preference of a decision maker. In the proposed framework, a constraint function is introduced in the search progress to align the solution with a user-specific preference, which can be optimized simultaneously with multiple objective functions. Experimental results show that our proposed method can efficiently find a particular Pareto solution under the demand of a decision maker for standard multiobjective benchmark, multi-task learning, multi-objective reinforcement learning problems with more than thousands of decision variables.
BibTeX
@article{zhang2026pmgda,
  author = {Xiaoyuan Zhang and Xi Lin and Qingfu Zhang},
  title = {{PMGDA: A Preference-Based Multiple Gradient Descent Algorithm}},
  journal = {IEEE Trans. Emerg. Top. Comput. Intell.},
  year = {2026},
  doi = {10.1109/tetci.2025.3526459},
}
Figure