Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning

Peizhong Ju, Yingbin Liang, Ness B. Shroff

Introduction

In modern deep learning, models with millions of parameters often perfectly interpolate training data—a phenomenon traditionally labeled as "overfitting." Despite this, these models frequently demonstrate remarkable generalization capabilities on unseen data. While recent theoretical advancements have begun to explain this "benign overfitting" in standard supervised learning, the theoretical underpinnings of the meta-learning paradigm have remained largely unexplored.

This paper, authored by Peizhong Ju, Yingbin Liang, and Ness B. Shroff from The Ohio State University, provides a rigorous characterization of generalization in overfitted meta-learning. The authors demonstrate that in the context of Model-Agnostic Meta-Learning (MAML), overfitting is not merely a byproduct of high capacity but can be a functional feature that improves performance in noisy or diverse task environments.

Problem Statement and Motivation

Traditional statistical learning theory relies on the bias-variance tradeoff: as model complexity increases, training error decreases (lower bias), but sensitivity to noise increases (higher variance), typically leading to a U-shaped test error curve. However, modern Deep Neural Networks (DNNs) often operate in the "overparameterized" regime, where the number of parameters $p$ exceeds the number of samples $n$. In this regime, researchers have observed a "double descent" curve where test error decreases, increases, and then decreases again as $p$ continues to grow.

While double descent is well-documented for single-task linear regression, meta-learning introduces additional complexity. Meta-learning aims to learn a global initialization that can quickly adapt to new tasks with minimal data. This study addresses a critical gap: Does the double descent phenomenon persist in meta-learning, and how do task-specific adaptation and task diversity influence this behavior?

Methodology

The authors analyze the Model-Agnostic Meta-Learning (MAML) algorithm within a linear regression framework using Gaussian features. Their approach focuses on several key components:

  1. Model Specification: They assume $m$ training tasks where each task's ground truth is a linear model. The study decouples the number of true features ($s$) from the model parameters ($p$), specifically investigating the regime where $p \gg s$.
  2. Error Decoupling: The total test error is decomposed into two constituent parts:
  3. Adaptation Error: The error resulting from the one-step gradient descent used to adapt the meta-initialization to a specific test task.
  4. Model Error: The error inherent in the meta-initialization itself, representing the distance between the learned global starting point and the true mean of the task distribution.
  5. Min $\ell_2$-norm Solution: The analysis focuses on the "interpolator" solution—the specific solution that achieves zero training loss with the minimum possible weights. This is considered the natural limit of gradient descent in overparameterized models.
  6. Non-Asymptotic Bounds: Unlike previous studies that rely on infinite limits (asymptotic analysis), the authors derive high-probability upper bounds on generalization error for finite $p$ and $n$.

Key Insights: Task Diversity and Noise Filtering

The research yields several counter-intuitive insights into how meta-learning models behave in high-dimensional spaces:

Task Diversity as "Pseudo-Noise"

In single-task learning, double descent typically requires the presence of label noise. In meta-learning, the authors discover that task diversity—the variance between different training tasks—acts as a form of "pseudo-noise." Consequently, double descent can occur in meta-learning even if the underlying data is noise-free, provided the tasks themselves are sufficiently diverse.

Overfitting as a Noise Filter

The analysis suggests that as the number of parameters $p$ increases, the model error for the ideal interpolator actually trends toward zero. In this context, overparameterization provides the model with sufficient "room" to mitigate the negative impacts of both data noise and task fluctuation, effectively acting as a filter.

The Descent Floor

While error in single-task learning can sometimes vanish as $p \to \infty$, meta-learning models often encounter a "descent floor." This floor is determined by the inherent diversity of the tasks, representing a fundamental limit to how much the meta-initialization can generalize across a distribution of tasks.

Experimental Validation

The theoretical findings were validated through both synthetic simulations and empirical tests:

  • Synthetic Data: Simulations using Gaussian features confirmed the derived non-asymptotic bounds and the existence of the double descent curve.
  • MNIST Experiments: To test the theory's applicability to non-linear models, the authors utilized a 2-layer fully connected neural network on a modified MNIST dataset. By treating different digit sets as distinct tasks, they observed that test error clearly decreased as the hidden layer width increased beyond the interpolation threshold, mirroring the theoretical linear results.

Conclusion and Future Work

This work provides the first non-asymptotic theoretical characterization of benign overfitting and double descent in meta-learning. It demonstrates that overparameterized meta-learning models can outperform their underparameterized counterparts, particularly in high-noise or high-diversity settings.

While the current theory is grounded in linear models and Gaussian assumptions, the authors suggest that future research should extend these findings to the Neural Tangent Kernel (NTK) regime to bridge the gap with deep, non-linear networks. Additionally, relaxing Gaussian assumptions will be necessary to develop a more universal theory of generalization in meta-learning.

Summary updated

BibTeX
@inproceedings{ju2023theoretical,
  author = {Peizhong Ju and Yingbin Liang and Ness B. Shroff},
  title = {{Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=Jifob4dSh99},
}
Figure