TL;DR
This paper introduces Dense Gaussian Processes (DGPs) for few-shot learning, moving away from the traditional "sparse" approach to Gaussian Processes. By meta-learning a large set of "dense" inducing variables in a latent feature space, the model captures rich prior knowledge across tasks and adapts to new, unseen tasks using efficient gradient descent.
What Problem Does This Paper Address?
Few-shot learning—predicting new categories from just a handful of examples—is notoriously difficult because models must handle extreme uncertainty. Gaussian Processes (GPs) are mathematically elegant tools for this because they naturally quantify uncertainty and provide robust predictions.
However, standard GPs struggle to "share" knowledge across different tasks. Previous attempts, like Deep Kernel Transfer (DKT), assumed a single universal kernel could work for everything. In reality, every task is slightly different. The challenge is: How can we build a GP that inherits broad knowledge from many tasks but still adapts its internal logic to the specific nuances of a new, tiny dataset?
Methods
The authors propose a meta-learning framework that rethinks inducing variables.
- From Sparse to Dense: In traditional GP literature, "inducing variables" are a small set of points used to approximate a large dataset (to save on computation). This paper flips that logic. They use a dense set of inducing variables—meaning the set is much larger than the few examples provided in a few-shot task. These variables act as a "memory" or a "prior" representing the entire distribution of tasks.
- Latent Space Learning: Instead of placing these inducing points in high-dimensional raw data space (like pixels), they learn them in a low-dimensional deep feature space. This makes the math much faster and more stable.
- Task-Specific Adaptation (The Inner Loop): When the model sees a new task, it doesn't just use its global knowledge. It performs a few steps of gradient descent to "nudge" the inducing variables so they better fit the specific support set of that task.
- Meta-Update (The Outer Loop): The model is trained across thousands of episodes. The "outer loop" updates the neural network (the feature extractor) and the global starting positions of the inducing variables to ensure they provide the best possible starting point for any future task.
Key Insights and Ideas
- Inducing Variables as Meta-Knowledge: The most clever "aha moment" here is treating inducing variables not as a computational shortcut, but as a parameterization of a task-shared prior. They collect "experienced" knowledge from previous tasks.
- Efficiency through Features: By adapting in the feature space rather than the input space, the authors avoid the "prohibitive computation" usually associated with large-scale GPs.
- The "Dense" Advantage: Ablation studies show that performance scales with the number of inducing points ($m$). Performance grows monotonically and saturates around $m=256$, which is significantly larger than the 5 or 25 points typically found in a few-shot support set.
- Robustness to Domain Shift: Because the GP framework is inherently probabilistic, it handles "out-of-distribution" data better than purely deterministic models. This is evidenced by their state-of-the-art results in cross-domain experiments.
Key Results
The authors evaluated their method on standard few-shot classification benchmarks, including miniImageNet, tieredImageNet, CIFAR-FS, and CUB, as well as cross-domain tasks.
- State-of-the-Art Performance: On miniImageNet (5-way, 5-shot), the model achieved 72.64% accuracy, outperforming standard DKT (64.00%) and MAML (63.11%).
- Fine-Grained Excellence: On the CUB dataset, it reached 89.97% in the 5-shot setting, showing it excels at distinguishing very similar classes.
- Cross-Domain Superiority: In the difficult miniImageNet $\to$ CUB transfer task, DGPs achieved 60.48%, a significant jump over the previous best GP-based methods (~57%).
Limitations and Future Work
- Multi-class Strategy: The current implementation uses a "one-versus-rest" regression scheme to handle multi-class classification. While effective, a native multi-class GP formulation (e.g., using Categorical Likelihoods) might be more theoretically sound, though computationally harder.
- Computational Overhead: Even with feature-space optimization, maintaining and adapting 256+ inducing variables per class is more memory-intensive than simple metric-based methods like Prototypical Networks.
- Scaling to "Many-Shot": The paper focuses on the 1-shot and 5-shot regimes. It is unclear if the "dense" advantage holds as the support set grows large enough that the GP no longer needs an informative prior.
Summary updated