Abstract: We develop a functional encoder-decoder approach to supervised meta-learning, where labeled data is encoded into an infinite-dimensional functional representation rather than a finite-dimensional one. Furthermore, rather than directly producing the representation, we learn a neural update rule resembling functional gradient descent which iteratively improves the representation. The final representation is used to condition the decoder to make predictions on unlabeled data. Our approach is the first to demonstrates the success of encoder-decoder style meta-learning methods like conditional neural processes on large-scale few-shot classification benchmarks such as miniImageNet and tieredImageNet, where it achieves state-of-the-art performance.
Introduction
In supervised meta-learning, a model must learn to solve new tasks quickly using only a small "context" set of labeled examples. Most existing models use an encoder-decoder architecture: the encoder squashes the context set into a task representation (usually a fixed-length vector), which the decoder then uses to make predictions.
However, this approach faces two major bottlenecks:
- The Representation Bottleneck: Using a fixed-size vector to represent a task often leads to underfitting. As the number of context points grows, a single vector cannot capture all the nuances of the data.
- Interaction Limits: Simple pooling methods (like sum-pooling) struggle to model complex interactions between different data points within a task.
MetaFun introduces a novel framework that represents tasks as infinite-dimensional functions. By using a learnable, iterative update rule in function space—akin to a neural version of functional gradient descent—it avoids the constraints of fixed-size vectors and specific parameter geometries.
Methods
The core of MetaFun is the Functional Iterative Update. Instead of producing a task representation in a single pass, the model starts with an initial functional representation and refines it over several steps.
Key components include:
- Functional Pooling: Unlike standard pooling that outputs a vector, functional pooling outputs a function. The authors use two flavors: Kernel-based (KFP) using Radial Basis Functions and Dot-product attention (DFP). This allows the "effective dimensionality" of the task representation to grow with the number of data points.
- Learnable Update Rule: The model learns a "local update function" $u$. At each iteration, $u$ compares the current functional prediction to the actual labels in the context set.
- Function Space Optimization: These local updates are aggregated via functional pooling into a global update. This process mimics Functional Gradient Descent, but with neural modules that can learn more sophisticated update logic than a standard gradient.
Key Insights and Ideas
- Infinite Dimensions for Finite Data: By representing a task as a function, the model avoids the "bottleneck" of fixed vectors. The representation is essentially as complex as the task requires.
- Learning the Optimizer: MetaFun can be viewed as learning an optimization algorithm that operates in function space. This is more flexible than MAML because it isn't "trapped" by the specific architecture's parameter landscape.
- The Power of Iteration: The paper shows that even two or three iterations significantly outperform a single-pass encoder. This "error-correcting" behavior allows the model to progressively hone in on the correct task solution.
Results
The authors evaluated MetaFun on both regression and large-scale classification tasks, including 1D Sinusoid Regression, Synthetic GP Data, miniImageNet, and tieredImageNet.
MetaFun achieved impressive quantitative gains, particularly on complex datasets:
- State-of-the-Art on tieredImageNet: MetaFun-DFP achieved 67.72% (1-shot) and 83.28% (5-shot) accuracy, outperforming established methods like LEO and MetaOptNet.
- Superior Regression: On sinusoid tasks, MetaFun achieved a Mean Squared Error (MSE) of 0.040, vastly outperforming the original MAML (0.390) while using fewer parameters.
- Uncertainty Quantification: The model's predictive variance closely matched the "Oracle" Gaussian Process, proving it can signal when it is unsure about a prediction.
Limitations and Future Work
While MetaFun sets a new standard for encoder-decoder meta-learners, areas for further exploration remain. The current version is deterministic; integrating a stochastic encoder (similar to Neural Processes) could better represent task-level ambiguity. Additionally, while efficient for small sets, the computational cost of functional pooling scales with the number of context and target points, suggesting a need for approximations in very large-scale applications.
Summary updated