Hypernetwork Approach to Bayesian MAML (Student Abstract)

Piotr Borycki, Piotr Kubacki, Marcin Przewiezlikowski, Tomasz Kusmierczyk, Jacek Tabor, Przemyslaw Spurek

TL;DR

This paper introduces BH-MAML, a novel framework for Bayesian Few-Shot Learning that uses hypernetworks to learn task-specific weight updates. This approach simplifies the Bayesian MAML process and allows for more complex and adaptive posterior distributions, leading to improved performance and uncertainty quantification.

What Problem Does This Paper Address?

Few-shot learning aims to train models that can quickly adapt to new tasks with very little data. A popular method for this is Model-Agnostic Meta-Learning (MAML), which learns a set of universal initial weights that can be fine-tuned for specific tasks. However, MAML has two main limitations: it can easily overfit with limited data, and it lacks good uncertainty quantification. Bayesian Neural Networks (BNNs) can address these issues by learning distributions over weights, but existing Bayesian MAML approaches often involve complex hierarchical structures and gradient-based updates that can be difficult to optimize. This paper tackles the challenge of creating a more efficient and effective Bayesian MAML framework that simplifies the optimization process while retaining the benefits of Bayesian methods for uncertainty and adaptation.

Methods

The core innovation of BH-MAML is the use of hypernetworks to generate task-specific weight updates. Instead of learning complex hierarchical Bayesian models, BH-MAML treats the universal weights as point estimates. The probabilistic structure is introduced only when adapting to a specific task. A hypernetwork takes the task's support set (labeled examples) and the universal weights as input and outputs parameters for a posterior distribution of task-specific weights.

The paper explores two variants for these posterior distributions:

  • BH-MAML-G (Gaussian): The hypernetwork outputs the mean and covariance matrix for a Gaussian distribution. Task-specific weights are then sampled from this Gaussian.
  • BH-MAML-CNF (Continuous Normalizing Flows): This more advanced variant uses the hypernetwork to condition a normalizing flow. This allows for learning more complex, non-Gaussian posterior distributions, offering greater flexibility in modeling uncertainty and adaptation.

This hypernetwork-based approach simplifies the optimization by avoiding direct gradient-based updates for posterior parameters and allows for more sophisticated adaptations due to the hypernetwork's capacity to model complex functions.

Datasets

The paper evaluates BH-MAML on cross-domain classification tasks using the following datasets:

  • Omniglot: Used for 1-shot and 5-shot learning.
  • EMNIST: Used as a target domain for classification.
  • mini-ImageNet: Used for classification.
  • CUB: Used as a target domain for classification.

The evaluation is performed in a cross-domain setting, meaning models are trained on one dataset (e.g., Omniglot) and tested on another (e.g., EMNIST), highlighting the model's generalization capabilities.

Key Insights and Ideas

  • Hypernetworks Simplify Bayesian MAML: By offloading the generation of task-specific weight updates to a hypernetwork, BH-MAML sidesteps the complex hierarchical optimization often found in Bayesian MAML.
  • Pointwise Universal Weights, Probabilistic Task Weights: This separation is key. Universal weights are learned as single values, while task-specific weights are treated probabilistically, offering a cleaner approach to Bayesian meta-learning.
  • Flexibility with Normalizing Flows: The CNF variant demonstrates the power of using more expressive probabilistic models to capture complex posterior distributions, potentially leading to better adaptation.
  • Amortized Learning Benefits: The hypernetwork's fixed parameter count, regardless of the number of tasks, leads to faster training and acts as a form of regularization.

Key Results

The paper reports classification accuracy on cross-domain tasks. While specific numbers are provided in Table 1, the general findings are:

  • BH-MAML-G achieved the best result in the 1-shot Omniglot→EMNIST classification task.
  • In 5-shot Omniglot→EMNIST, BH-MAML-G performed comparably to baseline methods.
  • On mini-ImageNet→CUB classification, BH-MAML (both Gaussian and CNF variants) performed comparably to established methods like MAML and ProtoNet.

The results suggest that BH-MAML offers competitive performance, particularly in scenarios where precise uncertainty estimation and adaptive updates are crucial.

Limitations and Future Work

  • Computational Cost of CNFs: While powerful, Continuous Normalizing Flows can be computationally more expensive than simpler Gaussian distributions, which might limit their application in very resource-constrained settings.
  • Further Exploration of Posterior Forms: The authors tested Gaussian and CNF-based posteriors. Future work could explore other types of normalizing flows or more general probabilistic models for even richer posterior representations.
  • Broader Benchmark Evaluation: While the chosen benchmarks are relevant, evaluating BH-MAML on a wider range of few-shot learning tasks and datasets would provide a more comprehensive understanding of its capabilities.
  • Investigating the "ada." Variants: The table includes "ada." variants of BH-MAML. While not explicitly detailed in the text, these likely represent adaptations or enhancements to the core BH-MAML approach, suggesting avenues for further improvement.

Summary updated

Abstract

The main goal of Few-Shot learning algorithms is to enable learning from small amounts of data. One of the most popular and elegant Few-Shot learning approaches is Model-Agnostic Meta-Learning (MAML). In this paper, we propose a novel framework for Bayesian MAML called BH-MAML, which employs Hypernetworks for weight updates. It learns the universal weights point-wise, but a probabilistic structure is added when adapted for specific tasks. In such a framework, we can use simple Gaussian distributions or more complicated posteriors induced by Continuous Normalizing Flows.
BibTeX
@inproceedings{borycki2025hypernetwork,
  author = {Piotr Borycki and Piotr Kubacki and Marcin Przewiezlikowski and Tomasz Kusmierczyk and Jacek Tabor and Przemyslaw Spurek},
  title = {{Hypernetwork Approach to Bayesian MAML (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35239},
}
Figure