HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning

Andrey Zhmoginov, Mark Sandler, Max Vladymyrov

Abstract

In this work we propose a HyperTransformer, a Transformer-based model for supervised and semi-supervised few-shot learning that generates weights of a convolutional neural network (CNN) directly from support samples. Since the dependence of a small generated CNN model on a specific task is encoded by a high-capacity Transformer model, we effectively decouple the complexity of the large task space from the complexity of individual tasks. Our method is particularly effective for small target CNN architectures where learning a fixed universal task-independent embedding is not optimal and better performance is attained when the information about the task can modulate all model parameters. For larger models we discover that generating the last layer alone allows us to produce competitive or better results than those obtained with state-of-the-art methods while being end-to-end differentiable.

Introduction

Few-shot learning (FSL) challenges models to recognize new categories from just one or a few examples. Existing approaches typically fall into two categories: metric-based learning, which uses a fixed universal embedding, and optimization-based learning (e.g., MAML), which fine-tunes models via gradient descent. However, metric-based methods are limited by fixed capacity, while optimization-based methods are computationally expensive at test time.

The HyperTransformer (HT) introduces a third paradigm: weight generation. By using a high-capacity Transformer to "write" a specialized tiny model for a specific task, it decouples the complexity of the overall task space from the simplicity of the inference model. This enables high performance in data-scarce regimes without the overhead of iterative optimization.

Architecture and Methodology

The HyperTransformer functions as a "weight generator" that is end-to-end differentiable. It processes a "support set" consisting of a few images and their corresponding labels. These inputs are converted into tokens comprising image embeddings, activation embeddings from intermediate layers, and label placeholders.

The HyperTransformer architecture generating weights for a target CNN.

A Transformer core processes these tokens using self-attention, which is naturally suited for unordered sets of support samples. The Transformer then outputs "weight slices" that are assembled into the parameters ($\theta$) of a target CNN. This generated CNN is subsequently used to classify new "query" images. A key innovation is the flexibility to generate either the entire CNN or just the final classification layer, depending on the target architecture's size.

Experimental Results

The authors evaluated the HyperTransformer on standard benchmarks: Omniglot, miniImageNet, and tieredImageNet. The results demonstrate that HT is particularly effective when the target model is small.

Performance comparison of HT against MAML++ and RFS across different model capacities.

As shown in the table above, HT significantly outperforms MAML++ and RFS in low-capacity regimes (e.g., 8 or 16 channels). For instance, on 1-shot Omniglot, HT achieved 87.2% accuracy compared to MAML++'s 81.4%. As the target model grows to 64 channels, HT remains competitive with state-of-the-art methods like RFS.

Key Insights

  • Self-Attention as Optimization: The authors provide a theoretical bridge showing that a single-layer self-attention mechanism can mathematically replicate a step of gradient descent.
  • Semi-Supervised Learning: HT handles unlabeled data by treating them as additional tokens with a special "unlabeled" tag. On tieredImageNet, adding 9 unlabeled samples per class improved 1-shot accuracy from 56.0% to 61.5%.
  • Decoupling Complexity: The system allows a massive Transformer to manage a complex task space while deploying a very efficient, tiny model for inference.

Conclusion

The HyperTransformer offers a powerful alternative for few-shot learning, especially for small-scale deployment. While scaling to massive architectures like ResNet-50 remains a challenge due to parameter count, the HT's ability to reason about task distributions and generate weights on the fly marks a significant step forward in meta-learning. Future work will focus on improving training stability and narrowing the gap with "oracle" models trained on full datasets.

Summary updated

Abstract

In this work we propose a HyperTransformer, a Transformer-based model for supervised and semi-supervised few-shot learning that generates weights of a convolutional neural network (CNN) directly from support samples. Since the dependence of a small generated CNN model on a specific task is encoded by a high-capacity Transformer model, we effectively decouple the complexity of the large task space from the complexity of individual tasks. Our method is particularly effective for small target CNN architectures where learning a fixed universal task-independent embedding is not optimal and better performance is attained when the information about the task can modulate all model parameters. For larger models we discover that generating the last layer alone allows us to produce competitive or better results than those obtained with state-of-the-art methods while being end-to-end differentiable.
BibTeX
@inproceedings{zhmoginov2022hypertransformer,
  author = {Andrey Zhmoginov and Mark Sandler and Max Vladymyrov},
  title = {{HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=lLbsQDjlj3},
}
Figure