Meta-learning the Learning Trends Shared Across Tasks

Jathushan Rajasegaran, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, Mubarak ShahOpenReview 2020 · DOI · OpenReview

TL;DR

This paper introduces PAMELA, a novel meta-learning approach that goes beyond learning a good initial model by also learning how to adapt that model over time. By incorporating learnable update directions, learning rates, and a mechanism to leverage past learning experiences, PAMELA achieves faster convergence and improved performance on few-shot learning tasks.

What Problem Does This Paper Address?

Meta-learning aims to enable models to "learn to learn," allowing them to quickly adapt to new tasks with limited data. Gradient-based meta-learning methods, like MAML, achieve this by learning a good initial set of parameters that can be fine-tuned. However, existing methods primarily focus on this initial parameter set and largely ignore how the adaptation process itself (the "learning trend") evolves across different tasks. This means they don't effectively share knowledge about how to adapt, relying only on the current task's information. The paper addresses this gap by proposing a method that learns these dynamic adaptation trends, making the meta-learning process more robust and efficient.

Methods

The core contribution is the Path-Aware Meta-Learning (PAMELA) framework. Unlike MAML, which uses standard gradient descent for inner-loop adaptation, PAMELA introduces two key innovations:

  • Learnable Gradient Preconditioning per Step (Q): PAMELA learns a separate preconditioning matrix ($Q_j$) for each inner-loop update step. This allows the model to dynamically adjust the direction and magnitude of gradient updates based on the current stage of learning within a task. This is a departure from methods like Meta-SGD, which use a single preconditioning matrix for the entire inner-loop.
  • Gradient-Skip Connections (P): PAMELA incorporates gradient-skip connections that allow information from previous update steps to be directly integrated into the current update. This is controlled by a meta-parameter ($P^w$) that learns how to combine past gradients with current ones, providing richer context and mitigating issues like gradient vanishing and overfitting.

These components allow PAMELA to model the "learning trend" – how update directions and learning rates evolve over time and how past experiences inform current adaptation.

Datasets

The paper evaluates PAMELA on several standard few-shot learning benchmarks:

  • miniImageNet: A widely used dataset for few-shot image classification.
  • CIFAR-FS: Another benchmark for few-shot image classification.
  • tieredImageNet: A hierarchical version of ImageNet, also used for few-shot classification.
  • Sine Wave Regression: A synthetic dataset for evaluating regression tasks, where tasks involve fitting sine waves with varying parameters.

Key Insights and Ideas

  • Learning the "How" of Learning: The most significant insight is that meta-learning can be substantially improved by explicitly learning the process of adaptation, not just the initial state. PAMELA demonstrates that understanding how gradients should evolve and how to leverage past learning is crucial for efficient few-shot learning.
  • Dynamic Adaptation is Key: The success of PAMELA highlights that a fixed adaptation strategy is suboptimal. By learning step-specific preconditioning ($Q_j$), the model can adapt its learning behavior as it approaches a task's solution, mirroring how humans refine their learning strategies.
  • Leveraging Past Experiences: Gradient-skip connections prove effective in providing context from previous learning steps. This not only helps in adapting to the current task but also acts as a regularizer, preventing overfitting by drawing on a broader learning history.
  • Faster Convergence and Better Generalization: The combination of learned adaptation trends and context aggregation leads to faster convergence during meta-training and improved generalization to unseen tasks.

Key Results

  • Few-Shot Classification: PAMELA achieves state-of-the-art results on miniImageNet (70.51% for 5-way 5-shot, 53.50% for 5-way 1-shot), CIFAR-FS (79.1% for 5-way 5-shot, 63.5% for 5-way 1-shot), and tieredImageNet (54.81% for 5-way 5-shot, 74.39% for 5-way 1-shot), outperforming MAML and Meta-SGD.
  • Regression: PAMELA achieves lower Mean Squared Error (MSE) on sine wave regression tasks compared to MAML and Meta-SGD across different numbers of training samples (K=5, 10, 20).
  • Efficiency: PAMELA shows competitive training and testing times, with only a modest increase in parameters and training time compared to MAML, while offering significant performance gains.

Limitations and Future Work

  • Computational Complexity: The paper acknowledges that PAMELA has higher memory and computational complexity than MAML due to the learnable per-step parameters ($Q_j$). However, they argue this overhead is minimized and justified by the performance gains.
  • Hyperparameter Sensitivity: While the length of gradient-skip connections ($w$) was found not to be highly significant, the paper suggests that a depth of 5-10 updates might not be sufficient to definitively prove this. Further exploration of hyperparameter tuning and its impact on performance could be a future direction.
  • Ablation Studies: The paper includes ablation studies that break down PAMELA's performance contributions from $Q$ and $P^w$. These studies confirm that the combination of both components yields the best results, and that learning step-wise trends ($Q$) and leveraging past context ($P^w$) are both critical.

The authors suggest that their approach provides a more principled and flexible way to encode prior knowledge, opening avenues for further research into dynamic meta-learning strategies.

Summary updated

Abstract

Meta-learning stands for 'learning to learn' such that generalization to new tasks is achieved. Among these methods, Gradient-based meta-learning algorithms are a specific sub-class that excel at quick adaptation to new tasks with limited data. This demonstrates their ability to acquire transferable knowledge, a capability that is central to human learning. However, the existing meta-learning approaches only depend on the current task information during the adaptation, and do not share the meta-knowledge of how a similar task has been adapted before. To address this gap, we propose a 'Path-aware' model-agnostic meta-learning approach. Specifically, our approach not only learns a good initialization for adaptation, it also learns an optimal way to adapt these parameters to a set of task-specific parameters, with learnable update directions, learning rates and, most importantly, the way updates evolve over different time-steps. Compared to the existing meta-learning methods, our approach offers: (a) The ability to learn gradient-preconditioning at different time-steps of the inner-loop, thereby modeling the dynamic learning behavior shared across tasks, and (b) The capability of aggregating the learning context through the provision of direct gradient-skip connections from the old time-steps, thus avoiding overfitting and improving generalization. In essence, our approach not only learns a transferable initialization, but also models the optimal update directions, learning rates, and task-specific learning trends. Specifically, in terms of learning trends, our approach determines the way update directions shape up as the task-specific learning progresses and how the previous update history helps in the current update. Our approach is simple to implement and demonstrates faster convergence. We report significant performance improvements on a number of FSL datasets.
BibTeX
@misc{rajasegaran2020meta,
  author = {Jathushan Rajasegaran and Salman H. Khan and Munawar Hayat and Fahad Shahbaz Khan and Mubarak Shah},
  title = {{Meta-learning the Learning Trends Shared Across Tasks}},
  year = {2020},
  eprint = {2010.09291},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.09291},
  howpublished = {OpenReview},
}
Figure