HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning

Andrey Zhmoginov, Mark Sandler, Maksym Vladymyrov

TL;DR

The HyperTransformer is a novel meta-learning framework that uses a high-capacity Transformer to generate the weights of a convolutional neural network (CNN) directly from a few labeled examples. By using the Transformer as a "weight generator," the authors decouple the complexity of the broad task space from the efficiency of the individual task-specific model, achieving state-of-the-art results in few-shot and semi-supervised learning.

The Capacity Bottleneck in Few-Shot Learning

Few-shot learning (FSL) aims to train models that can generalize to new categories using only one or a few examples. Current approaches generally fall into two categories:

  1. Metric-based learning: These models learn a "universal" embedding space where similar objects are close together. However, a single fixed embedding often lacks the capacity to handle a vast and diverse set of tasks.
  2. Optimization-based learning (e.g., MAML): These models learn a set of initial parameters that can be quickly fine-tuned. The catch? The "knowledge" about how to solve tasks is stored in the model's own parameters. If you want a tiny, efficient model for inference, it simply won't have enough capacity to store the complex logic needed for meta-learning.

The HyperTransformer addresses this "capacity bottleneck" by separating the solver (a large Transformer) from the inference model (a small CNN).

The HyperTransformer Architecture

The core idea is to treat weight generation as a sequence-to-sequence problem.

  1. Input Encoding: Support samples (images) are passed through a feature extractor. These image embeddings are concatenated with their labels (or a special "unlabeled" token) and fed into a Transformer.
  2. Weight Generation: Alongside the sample tokens, the Transformer receives "weight placeholder" tokens. After processing the sequence, the Transformer outputs values for these placeholders, which are then reshaped into the weights of a target CNN.
  3. Layer-by-Layer Generation: The model can generate weights for specific layers or the entire network. For intermediate layers, the Transformer also receives "activation embeddings" to help it understand the specific data distribution at that stage of the network.
A diagram of the HyperTransformer architecture showing how Transformer-based generators produce CNN layer weights from support samples and labels.

This approach is end-to-end differentiable. During training, the system is optimized based on how well the generated CNN performs on a separate "query set" of images.

Evaluation Benchmarks

The authors evaluated their approach on three standard few-shot learning benchmarks:

  • Omniglot: A dataset of handwritten characters from 50 different alphabets.
  • miniImageNet: A subset of ImageNet frequently used to benchmark FSL algorithms.
  • tieredImageNet: A larger, more challenging subset of ImageNet with a more rigorous split between training and testing categories.

Key Insights

  • Decoupling Capacity: The "aha moment" is that the weight generator can be massive while the generated model remains tiny. This allows for high-performance inference on edge devices without sacrificing the complexity needed to understand the task manifold.
  • Self-Attention as an Optimizer: The authors demonstrate that a single self-attention layer can mathematically mimic a step of gradient descent or a nearest-neighbor algorithm. This provides a theoretical grounding for why Transformers are so effective at generating weights.
  • Emergent Semi-Supervised Learning: By simply adding unlabeled samples to the input sequence with a "no label" token, the Transformer learns to use that extra information to produce better weights. Interestingly, this only works well if the Transformer has at least two layers, allowing it to "propagate" labels from labeled to unlabeled samples via attention.
  • When to Generate All Layers: For very small CNNs (e.g., 4-8 channels), generating all convolutional layers significantly boosts accuracy. For larger models, generating just the final "logits" layer is often sufficient.

Performance and Results

The HyperTransformer consistently outperforms standard meta-learning baselines like MAML++ and RFS, particularly when the target CNN is small.

  • Small Model Superiority: On 1-shot Omniglot with an 8-channel CNN, HT achieved 87.2% accuracy compared to MAML++'s 81.4%.
  • Competitive at Scale: On tieredImageNet, the HyperTransformer matched or exceeded the performance of much more complex state-of-the-art methods.
Table comparing HT performance against MAML++ and RFS across different datasets and model sizes.
  • Semi-Supervised Gains: Adding unlabeled data provided a clear boost. On tieredImageNet (1-shot), accuracy rose from 56.0% to 61.5% just by including 9 unlabeled samples per class in the task description.

Limitations and Future Directions

  • Architecture Scaling: The current work focuses on 4-layer CNNs. The authors note that generating weights for much larger architectures like ResNets or WideResNets remains a challenge for future research.
  • Training Stability: The authors found that advanced optimizers were unstable for this task, requiring them to stick to standard gradient descent with a specific learning rate decay schedule.
  • Overfitting: On smaller datasets like miniImageNet, the high-capacity Transformer can sometimes overfit to the training categories, leading to poorer generalization on unseen test classes.

Overall, the HyperTransformer represents a powerful shift in few-shot learning, moving away from "learning to fine-tune" toward "learning to generate" specialized models on the fly.

Summary updated

Abstract

In this work we propose a HyperTransformer, a Transformer-based model for supervised and semi-supervised few-shot learning that generates weights of a convolutional neural network (CNN) directly from support samples. Since the dependence of a small generated CNN model on a specific task is encoded by a high-capacity Transformer model, we effectively decouple the complexity of the large task space from the complexity of individual tasks. Our method is particularly effective for small target CNN architectures where learning a fixed universal task-independent embedding is not optimal and better performance is attained when the information about the task can modulate all model parameters. For larger models we discover that generating the last layer alone allows us to produce competitive or better results than those obtained with state-of-the-art methods while being end-to-end differentiable.
BibTeX
@inproceedings{zhmoginov2022hypertransformer,
  author = {Andrey Zhmoginov and Mark Sandler and Maksym Vladymyrov},
  title = {{HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=lLbsQDjlj3},
}
Figure