Semi-Supervised Few-Shot Learning with MAML

Rinu Boney, Alexander IlinWorkshops @ ICLR 2018

This workshop paper from ICLR 2018 explores how to make one of the most popular meta-learning algorithms, Model-Agnostic Meta-Learning (MAML), work in scenarios where labeled data is scarce but unlabeled data is plentiful. By meta-learning an auxiliary cost function, the authors extend MAML to perform unsupervised and semi-supervised adaptation, significantly improving data efficiency.

The Challenge of Unlabeled Data in Meta-Learning

Standard few-shot learning assumes that for every new task, we have at least a few labeled examples (the "support set"). However, in many real-world applications—such as a material recognition system deployed across different factories—collecting unlabeled data is cheap, while labeling that data is labor-intensive and expensive.

The challenge is that MAML typically relies on a supervised loss function (like cross-entropy) to calculate the gradient updates needed to adapt to a new task. If you don't have labels, you don't have a loss, and the "inner loop" of MAML breaks down. This paper investigates how to bridge that gap by creating a mechanism for MAML to learn from unlabeled samples.

Proposed Methods

The authors propose two primary modifications to the standard MAML framework to handle unlabeled data:

1. Meta-Learned Unsupervised Cost: Instead of using a predefined unsupervised loss (like entropy minimization), they parametrize the cost function as a neural network. During the meta-training phase, the model learns the parameters of this cost function. The goal is for the cost function to "learn" what good feature distributions look like, so that in the inner loop, the model can update its feature extractor using the gradient of this learned cost on purely unlabeled data.

2. Semi-Supervised Pipeline: For tasks where some labels are available, they use a two-step adaptation. First, the feature extractor is updated using unlabeled data and the meta-learned cost function. Second, the classifier head is updated using the few available labeled examples.

3. Decoupled Learning Rates: A critical technical detail identified by the authors is the use of "layer-wise" or "parameter-wise" learning rates. Rather than using a single global learning rate for adaptation, the model learns a specific learning rate for different parts of the network. This helps balance the signals coming from the unsupervised cost and the supervised labels.

Experimental Setup: Synthetic Sine Wave

The paper evaluates the proposed methods on a synthetic 2D classification problem where the decision boundary is a sine wave. Each task features a different amplitude and phase for the sine wave, requiring the model to adapt its "understanding" of the boundary for every new task. They used 100 tasks for training and 1,000 tasks for testing to ensure statistical significance.

Visualizations of the synthetic sine wave dataset showing true labels, purely unsupervised adaptation, and semi-supervised adaptation.

Key Insights and Results

Learning the "How" of Adaptation: The most novel idea is that we don't need to hand-engineer an unsupervised loss. By meta-learning the cost function, the algorithm discovers an optimization objective that specifically helps it perform well on the final validation set. Visualizations of the decision boundaries show that even with only unlabeled data, the adapted MAML model correctly recovers the sinusoidal shape of the task's decision boundary.

Unsupervised Success and Baselines: In a purely unsupervised setting (zero labels for the new task), MAML proved remarkably efficient. As the number of unlabeled samples increased from 10 to 1,000, MAML's error rate dropped significantly, from roughly 10% to 4%.

Classification error of MAML vs. Prototypical Networks (PN) as the number of unlabeled samples increases.

When compared to Prototypical Networks (PN), MAML was significantly more effective at leveraging unlabeled data. While PN generally performed better in a purely supervised setting on this specific dataset, MAML's ability to adapt via the learned cost function gave it a clear edge when labels were missing.

The Necessity of Granular Learning Rates: The authors found that decoupling learning rates was crucial for success. Without this, the unsupervised updates could easily wash out useful features or fail to provide enough signal to the classifier. This aligns with other findings in the meta-learning community (like Meta-SGD) that global hyperparameters are often too restrictive.

Limitations and Future Work

The primary limitation of this work is that the evaluation is constrained to a 2D synthetic dataset. While this provides a strong proof of concept, it remains to be seen how well a meta-learned cost function scales to high-dimensional data like ImageNet or complex natural language tasks. Additionally, the authors used a relatively simple architecture for their unsupervised cost function; more complex architectures might be necessary for real-world tasks.

Future work could look into combining the optimization strengths of MAML with the metric-learning strengths of Prototypical Networks into a single, more robust adaptation scheme.

Summary updated

BibTeX
@inproceedings{boney2018semi,
  author = {Rinu Boney and Alexander Ilin},
  title = {{Semi-Supervised Few-Shot Learning with MAML}},
  booktitle = {Workshops @ ICLR},
  year = {2018},
}
Figure