Abstract
Standard meta-learning techniques like Neural Processes often struggle and overfit when training data is scarce. This paper introduces Amortised Pseudo-Observation Variational Inference (A-POVI-BNN), a method that meta-learns how to perform Bayesian inference on task-specific neural networks. By sharing the inference process rather than model weights, A-POVI-BNN provides well-calibrated predictions even in extremely small-scale data regimes where traditional methods fail.
Introduction
Meta-learning aims to "learn how to learn" by leveraging experience across many related tasks to solve new ones quickly. A popular family of models for this is Neural Processes (NPs). However, NPs rely on a large number of shared model parameters (the "decoder") common across all tasks. When the "meta-dataset" is small—meaning there are few individual datasets or each dataset has very few points—these shared parameters tend to overfit. This research argues that a better approach for small-scale probabilistic meta-learning is to move away from shared model parameters entirely, instead training a meta-model to approximate fully Bayesian inference over task-specific parameters.
Methodology: A-POVI-BNN
The authors build upon Global Inducing Point Variational Inference (POVI). The key innovation, A-POVI-BNN, involves several critical shifts:
- From Inducing Points to Real Data: Instead of using synthetic "inducing inputs," the authors use the actual training data to construct an approximate posterior.
- Amortization via Inference Networks: Rather than optimizing variational parameters from scratch for every new task, an inference network is employed. This meta-model takes data pairs $(x_n, y_n)$ and outputs parameters for an approximate likelihood.
- Structural Symmetry: By defining the variational distribution as a product of the prior and per-datapoint approximate likelihoods, the model's structure mirrors the mathematical form of a true Bayesian posterior.
This framework allows the model to share the inference process across tasks rather than sharing the model weights themselves. At test time, a single forward pass of the new dataset through the inference network yields an approximate posterior for a task-specific Bayesian Neural Network (BNN).
Experimental Evaluation
The method was evaluated using a synthetic meta-dataset generated from a Gaussian Process (GP) with a Squared Exponential (SE) kernel. Each task contained only 10 to 20 training points. The models were tested on meta-datasets of varying sizes: $|\Xi| = 1$ (extremely low data) and $|\Xi| = 100$.
Comparative Results
A-POVI-BNN was compared against Amortised Mean-Field Variational Inference (A-MFVI-BNN) and a Convolutional Conditional Neural Process (ConvCNP).
As illustrated in Figure 1 above:
- Robustness to Small Data: A-POVI-BNN was the only method to produce sensible predictive posteriors when trained on a meta-dataset of size 1 ($|\Xi| = 1$).
- Overfitting Resistance: ConvCNP performed poorly with small meta-datasets, likely due to its high number of shared parameters, which led to oscillatory behavior and poor data fitting in the $|\Xi| = 1$ regime.
- Posterior Quality: While A-MFVI-BNN improved as the meta-dataset size increased to 100, its predictive posterior remained inferior to A-POVI-BNN, which captured the underlying GP-like behavior more accurately.
Conclusion and Future Work
A-POVI-BNN demonstrates significant promise for meta-learning in data-constrained environments by learning an inference procedure rather than a specific mapping. However, current limitations include a scalability bottleneck—the entire task dataset must be processed at once, precluding standard mini-batching. Future work will explore more complex tasks like image completion and alternative training objectives.
Summary updated