Learning to Learn Dense Gaussian Processes for Few-Shot Learning

Ze Wang, Zichen Miao, Xiantong Zhen, Qiang Qiu

TL;DR

This paper introduces Dense Gaussian Processes (DGPs) for few-shot learning, moving away from the traditional "sparse" approach to Gaussian Processes. By meta-learning a large set of "dense" inducing variables in a latent feature space, the model captures rich prior knowledge across tasks and adapts to new, unseen tasks using efficient gradient descent.

What Problem Does This Paper Address?

Few-shot learning—predicting new categories from just a handful of examples—is notoriously difficult because models must handle extreme uncertainty. Gaussian Processes (GPs) are mathematically elegant tools for this because they naturally quantify uncertainty and provide robust predictions.

However, standard GPs struggle to "share" knowledge across different tasks. Previous attempts, like Deep Kernel Transfer (DKT), assumed a single universal kernel could work for everything. In reality, every task is slightly different. The challenge is: How can we build a GP that inherits broad knowledge from many tasks but still adapts its internal logic to the specific nuances of a new, tiny dataset?

Methods

The authors propose a meta-learning framework that rethinks inducing variables.

Graphical model of the proposed Dense Gaussian Processes
  1. From Sparse to Dense: In traditional GP literature, "inducing variables" are a small set of points used to approximate a large dataset (to save on computation). This paper flips that logic. They use a dense set of inducing variables—meaning the set is much larger than the few examples provided in a few-shot task. These variables act as a "memory" or a "prior" representing the entire distribution of tasks.
  2. Latent Space Learning: Instead of placing these inducing points in high-dimensional raw data space (like pixels), they learn them in a low-dimensional deep feature space. This makes the math much faster and more stable.
  3. Task-Specific Adaptation (The Inner Loop): When the model sees a new task, it doesn't just use its global knowledge. It performs a few steps of gradient descent to "nudge" the inducing variables so they better fit the specific support set of that task.
  4. Meta-Update (The Outer Loop): The model is trained across thousands of episodes. The "outer loop" updates the neural network (the feature extractor) and the global starting positions of the inducing variables to ensure they provide the best possible starting point for any future task.

Key Insights and Ideas

  • Inducing Variables as Meta-Knowledge: The most clever "aha moment" here is treating inducing variables not as a computational shortcut, but as a parameterization of a task-shared prior. They collect "experienced" knowledge from previous tasks.
  • Efficiency through Features: By adapting in the feature space rather than the input space, the authors avoid the "prohibitive computation" usually associated with large-scale GPs.
  • The "Dense" Advantage: Ablation studies show that performance scales with the number of inducing points ($m$). Performance grows monotonically and saturates around $m=256$, which is significantly larger than the 5 or 25 points typically found in a few-shot support set.
  • Robustness to Domain Shift: Because the GP framework is inherently probabilistic, it handles "out-of-distribution" data better than purely deterministic models. This is evidenced by their state-of-the-art results in cross-domain experiments.

Key Results

The authors evaluated their method on standard few-shot classification benchmarks, including miniImageNet, tieredImageNet, CIFAR-FS, and CUB, as well as cross-domain tasks.

Few-shot classification results on miniImageNet, tieredImageNet, and CIFAR-FS
  • State-of-the-Art Performance: On miniImageNet (5-way, 5-shot), the model achieved 72.64% accuracy, outperforming standard DKT (64.00%) and MAML (63.11%).
  • Fine-Grained Excellence: On the CUB dataset, it reached 89.97% in the 5-shot setting, showing it excels at distinguishing very similar classes.
  • Cross-Domain Superiority: In the difficult miniImageNet $\to$ CUB transfer task, DGPs achieved 60.48%, a significant jump over the previous best GP-based methods (~57%).

Limitations and Future Work

  • Multi-class Strategy: The current implementation uses a "one-versus-rest" regression scheme to handle multi-class classification. While effective, a native multi-class GP formulation (e.g., using Categorical Likelihoods) might be more theoretically sound, though computationally harder.
  • Computational Overhead: Even with feature-space optimization, maintaining and adapting 256+ inducing variables per class is more memory-intensive than simple metric-based methods like Prototypical Networks.
  • Scaling to "Many-Shot": The paper focuses on the 1-shot and 5-shot regimes. It is unclear if the "dense" advantage holds as the support set grows large enough that the GP no longer needs an informative prior.

Summary updated

Abstract

Gaussian processes with deep neural networks demonstrate to be a strong learner for few-shot learning since they combine the strength of deep learning and kernels while being able to well capture uncertainty. However, it remains an open problem to leverage the shared knowledge provided by related tasks. In this paper, we propose to learn Gaussian processes with dense inducing variables by meta-learning for few-shot learning. In contrast to sparse Gaussian processes, we define a set of dense inducing variables to be of a much larger size than the support set in each task, which collects prior knowledge from experienced tasks. The dense inducing variables specify a shared Gaussian process prior over prediction functions of all tasks, which are learned in a variational inference framework and offer a strong inductive bias for learning new tasks. To achieve task-specific prediction functions, we propose to adapt the inducing variables to each task by efficient gradient descent. We conduct extensive experiments on common benchmark datasets for a variety of few-shot learning tasks. Our dense Gaussian processes present significant improvements over vanilla Gaussian processes and comparable or even better performance with state-of-the-art methods.
BibTeX
@inproceedings{wang2021learning,
  author = {Ze Wang and Zichen Miao and Xiantong Zhen and Qiang Qiu},
  title = {{Learning to Learn Dense Gaussian Processes for Few-Shot Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=hwf094GGNOv},
}
Figure