For years, the dominant paradigm in few-shot learning was "learning to learn." Researchers designed complex episodic training loops—simulating low-data tasks during training—to teach models how to rapidly adapt their weights to new classes. However, a growing body of evidence suggests that the "meta" in meta-learning might be less important than we thought. Instead, the real driver of success is simply learning a high-quality, robust feature representation.
The Meta Learning Illusion
The core intuition behind episodic meta-learning was that a model must be trained in the same way it is tested. If you want a model to recognize a new bird species from five images, you should train it on thousands of "episodes" where it sees five images of different birds.
But as Yonglong Tian and colleagues famously argued in Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?, a "dead simple" baseline often outperforms these complex algorithms. By training a standard supervised model on all available base classes and then using a simple linear classifier on top of those frozen features, they achieved state-of-the-art results. This suggests that the primary benefit of meta-learning isn't the specialized adaptation logic, but rather the fact that it forces the model to learn a transferable feature space.
This sentiment is echoed in object detection. Xin Wang demonstrated in Frustratingly Simple Few-Shot Object Detection that a two-stage fine-tuning approach—freezing the backbone and only adapting the final layer—outperforms sophisticated meta-detectors by a wide margin.
The Geometry of Neural Collapse
Why does a simple linear classifier work so well on "frozen" features? The answer lies in the geometry of the feature space. Tomer Galanti and co-authors explore this in On the Role of Neural Collapse in Transfer Learning.
As a deep network trains, it undergoes "Neural Collapse": features of the same class cluster into tight, low-variance points, while the centers of different classes move as far apart as possible. Crucially, this geometric structure generalizes. When the model sees a new, unseen class, it doesn't need a complex update rule; it just needs to find where that new cluster sits in the existing, well-organized space. In a collapsed feature space, a linear boundary is theoretically optimal.
Better Backbones over Better Loops
If the representation is what matters, the research focus shifts from designing better "meta-optimizers" to designing better "backbones." Several researchers have shown that improving the inherent quality of the feature extractor is more effective than adding complexity to the adaptation phase:
- Flatness and Stability: Martin Trapp showed in Flatness Improves Backbone Generalisation in Few-shot Classification that using Sharpness-Aware Minimization (SAM) to find "flatter" minima in the loss landscape creates backbones that generalize much better to new domains.
- Self-Supervision and Invariance: Mamshad Nayeem Rizve found that forcing models to be both invariant and equivariant to geometric transformations (like rotations) creates richer features. In Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning, they showed these representations allow simple classifiers to beat complex meta-learning state-of-the-art.
- Multi-Task Equivalence: Bo Li proved in Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation that standard multi-task learning and gradient-based meta-learning optimize nearly identical objectives. The "meta" part is essentially a faster way to get a good multi-task representation.
Simple Adaptation at Test Time
Once you have a robust backbone, the "adaptation" to a new task becomes a matter of inference rather than heavy optimization.
In the graph domain, Zhen Tan showed that Transductive Linear Probing—simply training a linear layer on frozen embeddings—consistently beats meta-learning for node classification. Similarly, Malik Boudiaf introduced Mutual-Information Based Few-Shot Classification, which uses a standard feature extractor and adapts at test-time by maximizing the information between unlabeled samples and their predicted labels.
The consensus across these 53 papers is clear: the complexity of "learning to learn" is often a proxy for "learning a good representation." By focusing on the quality, flatness, and geometric structure of the feature space, we can solve few-shot tasks using the simplest of tools.