PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees

Jonas Rothfuss, Vincent Fortuin, Andreas Krause

Introduction

Meta-learning, often described as "learning to learn," has emerged as a powerful paradigm for enabling machine learning models to adapt rapidly to new tasks using minimal data. By leveraging prior knowledge from a distribution of related tasks, meta-learners develop an inductive bias that facilitates efficient learning. However, a persistent challenge in this field is meta-overfitting. When the number of available training tasks is limited, meta-learners tend to memorize specific task characteristics rather than learning a generalizable prior.

Existing theoretical frameworks, such as PAC-Bayesian theory, offer a path toward understanding generalization. However, they have historically been limited by two factors: a reliance on bounded loss functions (which are impractical for standard regression) and computationally expensive "nested" optimization requirements. PACOH (PAC-Optimal Hyper-posterior) addresses these gaps by providing a theoretically rigorous and scalable framework for meta-learning with strong generalization guarantees.

The PACOH Framework

PACOH transforms meta-learning into a tractable stochastic optimization problem. The framework distinguishes between meta-training, where the model learns a hyper-posterior over task-specific priors, and target training/testing, where the learned prior is applied to new, unseen tasks.

Overview of the PACOH meta-learning framework, illustrating the flow from meta-training to target testing.

Theoretical Innovations

The paper introduces several key technical contributions that bridge the gap between theory and practice:

  1. Unbounded Loss Bounds: The authors derive the first PAC-Bayesian generalization bounds for meta-learners using unbounded loss functions, specifically targeting "sub-gamma" distributions. This allows the theory to be applied to standard regression tasks.
  2. The MLL Bridge: A critical insight of the paper is the derivation of a tractable objective. By assuming a Gibbs posterior as the base learner, the authors show that the PAC-Bayesian bound can be related directly to the marginal log-likelihood (MLL). This eliminates the need for the expensive nested optimization loops that plagued previous PAC-Bayesian meta-learning approaches.
  3. Principled Regularization: The derived objective naturally incorporates a meta-level KL-regularization term. This term penalizes overly complex priors, providing a mathematical safeguard against overfitting when the number of training tasks is small.

Scalable Implementation

To make the framework practical for complex models like Bayesian Neural Networks (BNNs) and Gaussian Processes (GPs), PACOH utilizes Stein Variational Gradient Descent (SVGD) for scalable inference. Unlike previous methods such as MLAP, which require storing task-specific posteriors and scale linearly with the number of tasks, PACOH employs task mini-batching. This ensures that memory and computational requirements remain constant regardless of the total task count, making it suitable for large-scale applications.

Empirical Validation

The authors evaluated PACOH across diverse environments, including synthetic regression (Sinusoids and Cauchy mixtures), real-world laser calibration (SwissFEL), and healthcare time-series (PhysioNet 2012).

Generalization performance (RMSE) on Sinusoids and Cauchy tasks, comparing PACOH-GP with standard MLL-GP.

Key Results

  • Superior Generalization: As shown in the performance plots, PACOH-GP maintains a negligible gap between meta-training and meta-testing performance. In contrast, standard MLL-based methods exhibit significant overfitting when the number of tasks is low.
  • Accuracy and Calibration: PACOH consistently outperformed baselines like MAML and Neural Processes in both Root Mean Squared Error (RMSE) and Expected Calibration Error (ECE), demonstrating better uncertainty quantification.
  • Real-world Impact: In a high-stakes vaccine design task involving MHC-I peptide sequences, PACOH-NN achieved near-optimal regret in fewer than 10 iterations, highlighting its utility for sequential decision-making in data-scarce environments.

Conclusion

PACOH represents a significant advancement in meta-learning by aligning theoretical PAC-Bayesian guarantees with practical, scalable optimization. By providing a principled way to regularize meta-learners, it enables robust performance even when training tasks are scarce. Future work may extend these guarantees to more complex reinforcement learning settings and explore non-Gaussian hyper-prior distributions to further enhance model expressivity.

Summary updated

Abstract

Meta-learning can successfully acquire useful inductive biases from data. Yet, its generalization properties to unseen learning tasks are poorly understood. Particularly if the number of meta-training tasks is small, this raises concerns about overfitting. We provide a theoretical analysis using the PAC-Bayesian framework and derive novel generalization bounds for meta-learning. Using these bounds, we develop a class of PAC-optimal meta-learning algorithms with performance guarantees and a principled meta-level regularization. Unlike previous PAC-Bayesian meta-learners, our method results in a standard stochastic optimization problem which can be solved efficiently and scales well. When instantiating our PAC-optimal hyper-posterior (PACOH) with Gaussian processes and Bayesian Neural Networks as base learners, the resulting methods yield state-of-the-art performance, both in terms of predictive accuracy and the quality of uncertainty estimates. Thanks to their principled treatment of uncertainty, our meta-learners can also be successfully employed for sequential decision problems.
BibTeX
@inproceedings{rothfuss2021pacoh,
  author = {Jonas Rothfuss and Vincent Fortuin and Andreas Krause},
  title = {{PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=Clz5G8qxBh},
}
Figure