Efficient Pareto Manifold Learning with Low-Rank Structure

Weiyu Chen, James T. Kwok

Abstract

Multi-Task Learning (MTL) often involves navigating trade-offs between conflicting objectives. While learning the entire Pareto manifold provides maximum flexibility for downstream preference selection, existing methods suffer from parameter explosion or training instability as the number of tasks increases. We present LORPMAN (LOw-Rank Pareto MANifold learning), a scalable approach that models the Pareto manifold using a shared base network and task-specific low-rank updates. By leveraging the inherent redundancy in task-specific parameters and employing orthogonal regularization, LORPMAN achieves state-of-the-art performance with up to 90% fewer parameters than previous manifold-learning techniques, demonstrating robust scalability on benchmarks with up to 40 tasks.

Introduction

In Multi-Task Learning, the goal is to optimize a model across several distinct objectives. However, these objectives are often in conflict: improving performance on one task frequently leads to degradation in another. To address this, researchers seek to identify the Pareto Front (PF)—the set of solutions where no task can be improved without harming at least one other.

Existing approaches to finding the Pareto Front generally fall into three categories, each with significant drawbacks:

  1. Discrete Methods: These identify specific points on the front but lack the flexibility to adapt to shifting user preferences post-training.
  2. Hypernetworks: These use a secondary "meta-model" to generate weights for a primary model. While flexible, hypernetworks are difficult to train and often exceed the size of the base model itself.
  3. Manifold Learning (e.g., PaMaL): These methods learn a linear combination of multiple "base networks." However, their memory requirements scale linearly with the number of tasks. For a 40-task problem, one would need 40 full-sized base networks, leading to a parameter explosion that makes high-dimensional MTL intractable.

LORPMAN addresses these limitations by observing that as training progresses, task-specific base networks often become highly redundant. This suggests that the entire Pareto manifold can be represented more efficiently through a low-rank structure.

Methodology

LORPMAN structures the model parameters $\theta$ as a continuous function of a preference vector $\alpha$:

$$\theta(\alpha) = \theta_0 + s \sum_{i=1}^{m} \alpha_i B_i A_i$$

Architectural Components

  • Main Network ($\theta_0$): A shared foundation that captures the common features and underlying representations relevant to all tasks.
  • Low-Rank Matrices ($B_i A_i$): Inspired by Low-Rank Adaptation (LoRA), LORPMAN approximates task-specific deviations from the main network using the product of two small matrices. This significantly reduces the number of trainable parameters compared to maintaining full-sized task networks.
  • Orthogonal Regularization: To ensure that task-specific updates remain distinct and do not become redundant, the authors implement a penalty that encourages orthogonality between the low-rank components. This "disentangles" the tasks, ensuring each $B_i A_i$ pair captures unique features.

Training Strategy

The model utilizes a Two-Phase Training approach. Initially, all components are trained jointly to establish a strong shared representation. Following a designated "freeze epoch," the main network ($\theta_0$) is locked, and optimization focuses solely on fine-tuning the low-rank components. This specialization phase is critical for accurately capturing the nuances of the Pareto manifold.

Experimental Results

The researchers evaluated LORPMAN across several benchmarks, ranging from standard 2-task problems to high-dimensional 40-task challenges.

Parameter Efficiency and Performance

On the CIFAR-100 benchmark (configured with 20 tasks using superclasses), LORPMAN demonstrated a massive reduction in overhead. While the previous state-of-the-art manifold method, PaMaL, required 296.5M parameters, LORPMAN achieved superior results with only 26.4M parameters—a reduction of over 90%.

Furthermore, LORPMAN significantly outperformed baselines in Hypervolume (HV), a metric measuring both the convergence and diversity of the learned Pareto front. On CIFAR-100, LORPMAN achieved an HV of 0.887, compared to PaMaL’s 0.058.

Scalability to High-Dimensional Task Spaces

In the CelebA dataset, which involves 40 distinct facial attribute tasks, LORPMAN successfully learned a diverse manifold where other algorithms failed or required complex "chunking" strategies. Additionally, on the CityScapes scene understanding dataset, a single point sampled from the LORPMAN manifold outperformed specialized state-of-the-art algorithms designed to find only a single optimal solution.

Discussion and Future Work

The success of LORPMAN highlights that redundancy in multi-task ensembles is not just a memory burden but an opportunity for regularization. By forcing task-specific updates into a low-rank space, the model avoids overfitting and shares features more effectively through the central $\theta_0$ network.

Despite its success, several avenues for improvement remain:

  • Rank Selection: Currently, a uniform rank ($r$) is applied across all layers. Future iterations could explore variable ranks, as different layers (e.g., early convolutional vs. late fully connected) likely require different levels of task-specific adaptation.
  • Hyperparameter Automation: The "freeze epoch" and rank $r$ currently require manual tuning. Automated search strategies could make the method more accessible for general use.
  • Optimization Objectives: While linear scalarization was used for optimization, integrating other scalarization methods could potentially improve the density and coverage of the learned manifold.

Conclusion

LORPMAN represents a significant step forward in making manifold learning practical for real-world Multi-Task Learning. By combining the efficiency of low-rank updates with the flexibility of manifold learning, it provides a scalable solution for complex systems requiring dynamic trade-off management.

Summary updated

BibTeX
@inproceedings{chen2024efficient,
  author = {Weiyu Chen and James T. Kwok},
  title = {{Efficient Pareto Manifold Learning with Low-Rank Structure}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=a2uFstsHPb},
}
Figure