Multi-Task Learning with User Preferences: Gradient Descent with Controlled Ascent in Pareto Optimization

Debabrata Mahapatra, Vaibhav RajanICML 2020 · OpenReview

Introduction

Multi-Task Learning (MTL) is a powerful paradigm where a single model is trained to perform multiple tasks simultaneously—such as an autonomous driving system that must identify pedestrians while also localizing itself. However, these tasks often conflict, meaning that improving performance on one task might degrade another.

This paper introduces EPO (Exact Pareto Optimal) Search, the first gradient-based multi-task learning algorithm that allows users to specify exact preferences for task trade-offs. By combining standard gradient descent with a novel "controlled ascent" mechanism, EPO Search navigates the Pareto front to find the specific solution that matches a user's desired loss ratios, even in complex, non-convex optimization landscapes.

The Problem with Existing Approaches

The standard method for handling conflicting tasks is linear scalarization, which minimizes a weighted sum of losses (e.g., $w_1L_1 + w_2L_2$). This approach suffers from two major flaws:

  1. The Convexity Gap: It cannot find solutions in concave regions of the Pareto front (the set of optimal trade-offs).
  2. Lack of Control: Setting a weight $w_1=0.5$ does not guarantee that the final losses will be equal; it merely defines a search direction.
Comparison of optimization methods on a concave Pareto front. EPO Search (d) successfully finds exact preference targets, unlike Linear Scalarization, MGDA, or Pareto MTL.

Existing Multi-Objective Optimization (MOO) methods like MGDA find a Pareto optimal solution, but they are often "preference-blind"—yielding whatever solution the initialization leads to. Other methods like Pareto MTL can target sub-regions but lack exact precision. The core challenge is finding a preference-specific solution: if a designer wants Task 1 to be twice as important as Task 2, how do we guarantee the model converges to that exact ratio?

Methods: Descent and Controlled Ascent

The authors propose EPO Search, a gradient-based algorithm that iteratively updates model parameters to reach a precise user-defined trade-off. Its unique mechanism relies on balancing two forces:

  1. Descent towards the Pareto Front: Similar to standard MOO, the algorithm seeks directions that decrease all task losses simultaneously to improve overall model efficiency.
  2. Controlled Ascent for Balancing: If the current model favors one task too much relative to the user's preference, the algorithm identifies a "balancing direction." It allows a slight, controlled increase (ascent) in the loss of the over-performing task to pull the model back toward the desired preference ratio.
Visualization of a preference ray in a 3-task loss space, guiding the optimization toward a specific user-defined trade-off.

To achieve this, EPO Search uses KL Divergence as a non-uniformity metric to measure how far the current loss ratios deviate from the user's desired preferences. In each iteration, the algorithm solves a small $m$-dimensional Linear Program (LP)—where $m$ is the number of tasks—to find a descent direction that maximizes alignment with the preference vector. Because the LP size depends only on the number of tasks rather than the millions of model parameters, it remains highly scalable for deep learning applications.

Key Results and Insights

The authors evaluated EPO Search across synthetic non-convex data, Multi-MNIST, Multi-Fashion, and an 8-objective River Flow regression dataset.

  • Unprecedented Precision: On synthetic data, EPO Search reached the exact preference targets (the "Exact Ray"), whereas previous state-of-the-art methods only reached broad sub-regions near the target.
  • Superior Performance: On Multi-MNIST and Multi-Fashion, EPO Search achieved higher per-task accuracy than both linear scalarization and Pareto MTL.
  • Robustness and Scalability: Unlike previous methods that are highly sensitive to initialization, EPO Search can reliably traverse the Pareto front. Furthermore, in the 8-task River Flow experiment, EPO Search significantly outperformed baselines. As the number of tasks increases, competing methods require an exponentially growing number of reference vectors, giving EPO Search a distinct scalability advantage.

Limitations and Future Work

While highly effective, EPO Search requires task-specific loss functions to be differentiable; it cannot be directly applied to non-differentiable metrics (like raw accuracy or BLEU scores) without using proxies. Additionally, like most gradient-based methods, its performance is sensitive to the choice of learning rate.

Currently, the algorithm assumes preferences are fixed at the start of training. A promising future direction is Interactive MOO, where a user could adjust preferences "on the fly" during the training process to dynamically explore different trade-offs.

Summary updated

Abstract

Multi-Task Learning (MTL) is a well established paradigm for jointly learning models for multiple correlated tasks. Often the tasks conflict, requiring trade-offs between them during optimization. ...
BibTeX
@inproceedings{mahapatra2020multi,
  author = {Debabrata Mahapatra and Vaibhav Rajan},
  title = {{Multi-Task Learning with User Preferences: Gradient Descent with Controlled Ascent in Pareto Optimization}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=b16Dz2N4VrT},
}
Figure