Effective Meta-Regularization by Kernelized Proximal Regularization

Weisen Jiang, James Kwok, Yu ZhangNeurIPS 2021 · OpenReview

Few-shot learning aims to teach models to recognize new concepts from just a handful of examples. Most popular approaches fall into two camps: Meta-Initialization (like MAML), which finds a highly adaptable starting point for fine-tuning, and Metric Learning, which learns a smart way to compare new samples to old ones in a latent space.

However, both approaches have notable vulnerabilities. MAML-style models often overfit during fine-tuning because they attempt to hit zero error on a tiny, potentially noisy dataset. Conversely, metric learning models often rely on fixed, "dumb" regularizers that fail to adapt to the meta-knowledge gained across different tasks.

This paper introduces MetaProx, an algorithm that bridges these gaps by learning a "proximal regularizer"—essentially a learned functional anchor—using the power of deep kernels.

The Overfitting and Regularization Dilemma

When a model is fine-tuned on very little data (the "support set"), it is at high risk of overfitting.

  1. The MAML Overfitting Problem: The authors theoretically demonstrate that in noisy scenarios, MAML's strategy of taking multiple gradient steps forces the model to converge to zero training error. While this sounds ideal, it actually causes the model to "memorize" the noise in those few samples, leading to poor generalization on unseen data.
  2. The Fixed Regularization Problem: Recent state-of-the-art methods like MetaOptNet use "deep kernels" to handle complex non-linearity. However, their base learners typically use Tikhonov regularization (regularizing weights toward zero). This ignores the valuable "meta-knowledge" that could be stored in a non-zero learned mean or prior.

The core challenge is learning a non-linear, task-specific model that is "anchored" to a learned global prior without making the mathematics computationally intractable.

The MetaProx Solution

The authors propose MetaProx, which utilizes Kernelized Proximal Regularization to stabilize learning.

  • The "Anchor" ($f_\theta$): Instead of regularizing the task-specific model toward zero, MetaProx regularizes it toward a learned function $f_\theta$. This $f_\theta$ represents the "common knowledge" shared across all tasks.
  • The Kernel Trick: To handle complex, non-linear data, this regularization is defined in a Reproducing Kernel Hilbert Space (RKHS). By leveraging the Representer Theorem, the authors transform a potentially infinite-dimensional functional optimization problem into a manageable dual problem.
  • Efficiency via Duality: Because the number of samples in few-shot learning is tiny (e.g., 1 or 5), solving the problem in the "dual space" is incredibly fast. The computational cost scales with the cube of the number of samples ($O(n_s^3)$) rather than the massive number of neural network parameters.
  • Implicit Differentiation: To update the meta-parameters (the anchor function and the neural network embedding), MetaProx uses implicit differentiation. This allows the computation of gradients through the optimization process without storing every intermediate step of the inner loop, saving significant memory.

Key Insights and Results

MetaProx fundamentally alters the bias-variance tradeoff in meta-learning. The paper provides a formal proof showing that proximal regularization handles label noise much better than gradient descent. While MAML overfits noise to reach zero error, MetaProx uses its learned "anchor" to stay closer to the true underlying distribution.

Training convergence across different noise and shot settings

The authors evaluated MetaProx across a diverse range of benchmarks, consistently outperforming baselines, particularly in "noisy" or "low-shot" settings:

  • Noisy Sine Regression: In a 2-shot setting with noise, MetaProx achieved a Mean Squared Error (MSE) of 0.43, while MAML lagged significantly behind at 1.91. Visually, MetaProx avoids the erratic curves that MAML produces when trying to fit noisy support points perfectly.
Sine wave regression fits comparing MetaProx to baselines
  • QMUL (Face Tilt Regression): On this real-world dataset predicting head tilt from images, MetaProx achieved the lowest MSE (0.012 in-range), significantly beating MetaOptNet (0.021).
  • mini-ImageNet Classification: On the gold-standard benchmark for few-shot image classification (100 classes, 5-way tasks) using a ResNet-12 backbone, MetaProx hit 63.82% accuracy on 1-shot tasks, outperforming MetaOptNet-SVM (62.31%) and ProtoNet (59.25%).

Limitations and Future Work

While highly effective, MetaProx does have limitations. The choice of the base kernel (e.g., linear vs. cosine) remains a hyperparameter that must be tuned. Additionally, while the $O(n_s^3)$ scaling is excellent for 1-shot or 5-shot learning, the matrix inversion required by the kernel method could become a computational bottleneck if the "few-shot" task grows to hundreds of samples.

Future directions suggested by the authors include extending this proximal framework to other domains of meta-learning, such as hyperparameter optimization or neural architecture search, where learned priors could similarly stabilize complex optimization landscapes.

Summary updated

Abstract

We study the problem of meta-learning, which has proved to be advantageous to accelerate learning new tasks with a few samples. The recent approaches based on deep kernels achieve the state-of-the-art performance. However, the regularizers in their base learners are not learnable. In this paper, we propose an algorithm called MetaProx to learn a proximal regularizer for the base learner. We theoretically establish the convergence of MetaProx. Experimental results confirm the advantage of the proposed algorithm.
BibTeX
@inproceedings{jiang2021effective,
  author = {Weisen Jiang and James Kwok and Yu Zhang},
  title = {{Effective Meta-Regularization by Kernelized Proximal Regularization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=mekyxmlLJNd},
}
Figure