Enhancing Meta Learning via Multi-Objective Soft Improvement Functions

Runsheng Yu, Weiyu Chen, Xinrun Wang, James KwokICLR 2023 · OpenReview

TL;DR

This paper introduces SIMOL (Soft Improvement Multi-Objective Meta-Learning), a scalable framework that treats meta-learning as a multi-objective optimization problem to ensure no single task is left behind. By using a novel "soft improvement function" and a neural re-weighting network, the authors enable multi-objective solvers to scale to millions of tasks while providing theoretical guarantees for reaching Pareto-optimal solutions.

The Problem: The Compromising Phenomenon in Meta-Learning

Meta-learning, or "learning to learn," typically aims to find a shared model initialization that can quickly adapt to new tasks. Standard approaches, such as Model-Agnostic Meta-Learning (MAML), achieve this by minimizing the average loss across a large set of training tasks.

However, this "average-case" focus leads to the compromising phenomenon: the optimization process might ignore "hard" tasks or tasks with conflicting gradients to favor a majority subset. This results in a model that performs excellently on some tasks but poorly on others.

While Multi-Objective Optimization (MOO)—which treats every task as an individual objective—could theoretically solve this by finding a "Pareto optimal" solution (where you cannot improve one task without hurting another), existing MOO solvers are computationally expensive. They usually require calculating gradients for every task in every iteration. In modern meta-learning, where there can be millions of possible task combinations (e.g., in 5-way classification on miniImageNet), standard MOO simply does not scale.

The SIMOL Framework

The authors propose SIMOL to bridge the gap between the theoretical rigor of MOO and the practical scalability needed for meta-learning. The key technical innovations include:

  1. The Improvement Function: Instead of just averaging losses, SIMOL uses an "improvement function" that measures how much a model update improves the worst-performing task relative to a reference point.
  2. Soft Weighting via Neural Networks: To make the "max" operation in the improvement function differentiable and scalable, they reformulate it using a re-weighting network ($r_\theta$). This network learns to assign weights to tasks dynamically in a mini-batch, focusing the meta-learner's attention on tasks that are currently performing poorly or have conflicting gradients.
  3. Mini-batch Scalability: Unlike previous MOO solvers that fail when mini-batched (losing their Pareto guarantees), the authors provide a mathematical proof showing that their specific formulation remains "Pareto stationary" even when using small subsets of tasks.
Optimization trajectories comparing full-batch MGDA, mini-batch MGDA, and the proposed SIMOL.

As seen in the optimization trajectories above, standard mini-batch MOO solvers (like mini-batch MGDA) fail to converge to the true Pareto optimal solution. In contrast, SIMOL successfully navigates the optimization landscape to reach the converged solution, mirroring the behavior of computationally prohibitive full-batch methods.

Key Results and Empirical Validation

The researchers evaluated SIMOL across three distinct domains: Few-shot Regression, Few-shot Image Classification (miniImageNet and tieredImageNet), and Reinforcement Learning (MuJoCo environments).

Better "Worst-Case" Performance

The core "aha moment" of SIMOL is that the compromising phenomenon is a fundamental failure of the average-loss objective. By shifting to a multi-objective view, the model is forced to find solutions that are robust across the entire task distribution.

Table comparing overall and worst-10% MSE across different methods, highlighting SIMOL's superior worst-case performance.

In few-shot regression, SIMOL significantly reduced the Mean Squared Error (MSE) for the worst-10% of tasks. As shown in the results table, SIMOL achieved an MSE of 1.69 on the worst-10% of 5-shot tasks, compared to 2.13 for standard MAML. This proves it successfully alleviates the compromising phenomenon without sacrificing overall performance.

State-of-the-Art Accuracy and Scalability

  • Classification Accuracy: On miniImageNet (5-way, 1-shot), SIMOL achieved 50.62% accuracy, outperforming standard MAML (49.24%) and several other specialized variants like Reptile and MTL.
  • Efficiency: The authors show that while full-batch MOO solvers are theoretically ideal, they are practically impossible. In a direct comparison, SIMOL's per-epoch training time was roughly 2.3 seconds, while a full-batch MOO solver (MGDA) was estimated to take 6.9 days for the same epoch.
  • Smoother RL Training: In reinforcement learning tasks, SIMOL showed more stable and higher accumulated rewards compared to the standard MAML-VPG baseline.

Limitations and Future Work

While highly effective, SIMOL does have limitations. The theoretical convergence rate for the outer loop of SIMOL is $O(1/\epsilon^8)$, which is mathematically slower than standard MAML's $O(1/\epsilon^2)$. While it performs well in practice, there is a gap between the current theory and empirical speed. Additionally, SIMOL introduces new hyperparameters (like regularization terms) that require careful tuning for the re-weighting network.

Future directions include exploring more complex architectures for the re-weighting network and applying this multi-objective approach to other areas of machine learning that suffer from task interference, such as Continual Learning.

Summary updated

Abstract

Meta-learning tries to leverage information from similar learning tasks. In the commonly-used bilevel optimization formulation, the shared parameter is learned in the outer loop by minimizing the average loss over all tasks. However, the converged solution may be comprised in that it only focuses on optimizing on a small subset of tasks. To alleviate this problem, we consider meta-learning as a multi-objective optimization (MOO) problem, in which each task is an objective. However, existing MOO solvers need to access all the objectives’ gradients in each iteration, and cannot scale to the huge number of tasks in typical meta-learning settings. To alleviate this problem, we propose a scalable gradient-based solver with the use of mini-batch. We provide theoretical guarantees on the Pareto optimality or Pareto stationarity of the converged solution. Empirical studies on various machine learning settings demonstrate that the proposed method is efficient, and achieves better performance than the baselines, particularly on improving the performance of the poorly-performing tasks and thus alleviating the compromising phenomenon.
BibTeX
@inproceedings{yu2023enhancing,
  author = {Runsheng Yu and Weiyu Chen and Xinrun Wang and James Kwok},
  title = {{Enhancing Meta Learning via Multi-Objective Soft Improvement Functions}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=hCmjBJeGXcu},
}
Figure