Robust MAML: Prioritization task buffer with adaptive learning process for model-agnostic meta-learning

Thanh Nguyen, Tung Minh Luu, Trung X. Pham, Sanzhar Rakhimkul, Chang D. YooICASSP 2021 · DOI · OpenReview

TL;DR

This paper introduces Robust MAML (RMAML), a novel meta-learning approach that tackles MAML's sensitivity to hyperparameter tuning and task distribution mismatches. RMAML uses an adaptive learning rate and a prioritization task buffer to improve scalability and generalization, particularly in challenging reinforcement learning scenarios.

What Problem Does This Paper Address?

Meta-learning aims to train models that can quickly adapt to new tasks with limited data. Model-Agnostic Meta-Learning (MAML) is a prominent algorithm that achieves this by finding a good initial set of model weights. However, MAML faces two significant challenges:

  1. Costly Hyperparameter Tuning: MAML's performance is highly sensitive to the learner's learning rate. Finding the optimal learning rate requires extensive tuning, making the process computationally expensive and hindering scalability.
  2. Task Distribution Mismatch: MAML assumes that training tasks are representative of future tasks. If the distribution of training tasks differs from the distribution of testing tasks (e.g., due to noisy or outlier tasks), MAML's generalization performance can suffer significantly.

This paper addresses these limitations by proposing RMAML, which aims to make MAML more robust, scalable, and less sensitive to hyperparameter choices and distribution shifts.

Methods

RMAML introduces two key innovations:

  1. Adaptive Learning Rate Scheme: Instead of using a fixed learning rate for the inner-loop adaptation (fine-tuning on a new task), RMAML employs a gradient-based approach to automatically optimize the learner's learning rate. This adaptive scheme can even vary the learning rate across different weights or layers, offering more flexibility and reducing the need for manual tuning.
  2. Prioritization Task Buffer (PTB): To combat task distribution mismatch, RMAML uses a PTB. During training, tasks are not uniformly sampled from the task distribution. Instead, the PTB stores tasks along with their validation performance. In subsequent training iterations, a portion of tasks are sampled from this buffer, prioritizing those that are most beneficial for learning. The paper proposes a "MEDIUM STRATEGY" where tasks with medium validation loss are prioritized, drawing inspiration from curriculum learning. This strategy gradually shifts the training task distribution to better align with the testing task distribution.

Datasets

The paper evaluates RMAML on meta reinforcement learning tasks, specifically:

  • MuJoCo Locomotion Tasks:
    • Half Cheetah Velocity (VEL): The robot learns to run at a specific velocity.
  • Half Cheetah Direction (DIR): The robot learns to run in a specific direction.
  • 2D REACHING Environment: A custom environment with 300 tasks designed to test different sampling strategies. This environment features a dominant "correct" task distribution and several "noise" task distributions.

Key Insights and Ideas

  • Adaptive Learning Rates are Crucial: The paper highlights that the learner's learning rate is a critical hyperparameter for MAML, and automatically optimizing it significantly improves robustness and reduces tuning effort.
  • Prioritizing "Useful" Tasks Matters: Simply sampling tasks uniformly can be inefficient, especially when the training and testing distributions differ. RMAML's PTB, by intelligently selecting tasks, helps to correct for distribution mismatches and guide the meta-learning process more effectively.
  • Curriculum Learning Principle for Task Selection: The "MEDIUM STRATEGY" for the PTB, which prioritizes tasks of intermediate difficulty, is inspired by curriculum learning. This suggests that a structured approach to task exposure can be more beneficial than random sampling for meta-learning.
  • Reinforcement Learning as a Challenging Testbed: The choice of meta-reinforcement learning environments underscores the robustness of RMAML, as these tasks are known to be particularly sensitive to hyperparameter choices and distribution shifts.

Key Results

  • Substantial Performance Gains: RMAML consistently outperforms standard MAML and its distributed implementation (MAML+) on the tested meta-reinforcement learning tasks. For instance, in the Half Cheetah Velocity task, RMAML achieves a significantly higher average test return compared to MAML+.
  • Robustness to Distribution Mismatch: RMAML demonstrates strong performance even when faced with noisy tasks (e.g., "Noise Cheetah Velocity" experiment), maintaining performance close to that achieved without noise.
  • Reduced Sensitivity to Hyperparameters: The adaptive learning rate scheme makes RMAML less sensitive to hyperparameter choices, simplifying the tuning process.
  • Improved Scalability: By addressing the costly hyperparameter tuning and distribution mismatch issues, RMAML is better positioned for scaling to a wider range of problems.

Limitations and Future Work

  • Limitations: The paper acknowledges that while RMAML is robust to distribution mismatch, the effectiveness of the PTB relies on the quality of the validation return for task prioritization. If validation returns are not discriminative, the PTB might not be as effective.
  • Future Work: The authors suggest that RMAML's success in meta-reinforcement learning indicates its potential for broader applications. Further exploration of different PTB strategies and their impact on various domains would be a valuable next step. The paper also implies that RMAML's robustness makes it suitable for more complex and diverse meta-learning scenarios.

Summary updated

Abstract

Model agnostic meta-learning (MAML) is a popular state-of-the-art meta-learning algorithm that provides good weight initialization of a model given a variety of learning tasks. The model initialized by provided weight can be fine-tuned to an unseen task despite only using a small amount of samples and within a few adaptation steps. MAML is simple and versatile but requires costly learning rate tuning and careful design of the task distribution which affects its scalability and generalization. This paper proposes a more robust MAML based on an adaptive learning scheme and a prioritization task buffer(PTB) referred to as Robust MAML (RMAML) for improving scalability of training process and alleviating the problem of distribution mismatch. RMAML uses gradient-based hyper-parameter optimization to automatically find the optimal learning rate and uses the PTB to gradually adjust train-ing task distribution toward testing task distribution over the course of training. Experimental results on meta reinforcement learning environments demonstrate a substantial performance gain as well as being less sensitive to hyper-parameter choice and robust to distribution mismatch.
BibTeX
@inproceedings{nguyen2021robust,
  author = {Thanh Nguyen and Tung Minh Luu and Trung X. Pham and Sanzhar Rakhimkul and Chang D. Yoo},
  title = {{Robust MAML: Prioritization task buffer with adaptive learning process for model-agnostic meta-learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413446},
}
Figure