Introduction
In supervised meta-learning, models are tasked with learning to solve new "target" tasks using only a small "context" set of labeled examples. Most existing approaches fall into two distinct camps: encoder-decoder models and optimization-based models.
Encoder-decoder models (like Neural Processes) summarize the context set into a fixed-dimensional vector. However, this single vector often acts as an information bottleneck, failing to capture complex interactions between data points and leading to underfitting. Conversely, optimization-based models (like MAML) adapt to new tasks by taking gradient steps in the model's parameter space. While powerful, these updates are constrained by the specific architecture's geometry and can be computationally expensive or unstable.
MetaFun introduces a novel meta-learning framework that bridges these gaps by representing tasks as infinite-dimensional functions rather than fixed-size vectors. By employing a learned, iterative update rule in function space—resembling a neural version of functional gradient descent—MetaFun overcomes the representation bottleneck and achieves state-of-the-art results on challenging few-shot classification benchmarks.
The MetaFun Architecture
The core of MetaFun is an iterative encoder-decoder pipeline that operates directly on functions rather than fixed-length embeddings.
- Functional Representation: Instead of a vector, the task is represented by a function $r(\cdot)$. This representation is "infinite-dimensional" because it can be evaluated at any point in the input domain, allowing its effective capacity to scale naturally with the size of the context set.
- Iterative Neural Updates: The model starts with an initial guess and refines it over $T$ steps. At each step, a local update function (parameterized by an MLP) compares the current functional representation to the actual context labels to compute "errors."
- Functional Pooling: These local errors are aggregated into a global functional update $\Delta r(\cdot)$ using one of two mechanisms:
- Kernel-based Functional Pooling (KFP): Uses deep kernels to measure similarity between points.
- Dot-product Functional Pooling (DFP): Uses a multi-head attention mechanism to aggregate information.
- The Update Rule: The representation is updated via $r^{(t+1)} = r^{(t)} - \alpha \Delta r^{(t)}$. This structure mimics functional gradient descent, but the update rule itself is learned directly from data.
- Decoder: A final module takes the refined functional representation and produces the actual predictions for the target data.
Key Results and Performance
The authors evaluated MetaFun on three main benchmarks: a synthetic 1D Sinusoid Regression task, miniImageNet, and tieredImageNet.
In the regression tasks, MetaFun significantly outperformed MAML. It achieved a 5-shot Mean Squared Error (MSE) of 0.040 compared to MAML's 0.208, despite MetaFun using fewer total parameters. Furthermore, the model produces high-quality predictive uncertainties that closely match those of a "Gold Standard" Gaussian Process.
For few-shot classification, MetaFun achieved top-tier performance, particularly on the more difficult tieredImageNet benchmark (a larger, 608-class dataset with a hierarchical structure ensuring distinct train/test tasks).
In the 5-way 5-shot setting on tieredImageNet, MetaFun reached 83.28% accuracy, outperforming established methods like LEO and MetaOptNet. Ablation studies further confirmed that using multiple iterations ($T > 1$) consistently improved accuracy over a single-pass approach, validating the iterative refinement process.
Insights and Limitations
MetaFun's success stems from breaking the representation bottleneck. Because the functional representation is evaluated at every data point, its effective dimensionality grows with the context set. Furthermore, the functional pooling mechanism naturally models interactions between context and target points—a significant upgrade over simple sum-pooling that treats data points in isolation.
Despite these advantages, the current version of MetaFun is deterministic. Incorporating stochasticity could be a promising direction for better probabilistic modeling. Additionally, while efficient for few-shot tasks, the computational cost of functional pooling (especially the attention-based DFP) scales quadratically with the number of points. This may limit its application in "many-shot" scenarios without further architectural optimizations.
Summary updated