Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective

Haidong Kang

TL;DR

This paper introduces IBFS (Information Bottleneck for Few-shot Search), a framework that finds high-performing neural architectures for few-shot learning (FSL) without any training. By leveraging Information Bottleneck theory and the first-order loss landscape, IBFS achieves state-of-the-art results while reducing search costs from hundreds of GPU days to just a few minutes or seconds.

Performance comparison of IBFS against other NAS and meta-learning methods

What Problem Does This Paper Address?

Few-Shot Learning (FSL) aims to train models that can recognize new objects from just a handful of examples. While Neural Architecture Search (NAS) is great at finding optimal network structures, applying it to FSL is notoriously difficult.

Currently, researchers face a "pick your poison" scenario:

  1. Search from scratch: Extremely time-consuming, with some methods taking over 100 GPU days to find a suitable architecture.
  2. Borrow architectures: Using a model designed for a different task (like ImageNet) is often sub-optimal for the specific constraints and nuances of FSL.

The authors ask a bold question: Can we identify the best neural architectures for FSL without doing any training at all?

Methods: A "Zero-Cost" NAS Approach

The authors propose a zero-cost NAS approach called IBFS.

Comparison of Vanilla NAS, Meta-learning NAS, and the proposed IBFS framework

Its core components include:

  • First-Order Convergence Theory: The authors mathematically derive that the global convergence of Model-Agnostic Meta-Learning (MAML) can be guaranteed by looking only at the first-order loss landscape (the Jacobian). This simplifies the problem significantly, suggesting we don't need to simulate the entire training process to know if an architecture is "good."
  • Information Bottleneck (IB) Proxy: They use IB theory to measure an architecture's "expressivity." In simple terms, IB theory views learning as a process of squeezing input data through a bottleneck to retain only the most relevant information for the output.
  • Entropy-based Ranking: Instead of training, they pass a batch of data through an untrained network and calculate a score based on the entropy of the Jacobian matrix's eigenvalues. A higher entropy score indicates better expressivity and, consequently, better potential performance in FSL scenarios.
  • Zero-Training Search: Because the score is calculated at initialization, the search process requires zero weight updates, making it incredibly fast.

Key Insights and Ideas

  • Expressivity is Intrinsic: One of the most striking findings is that an architecture's expressivity (measured by entropy) remains remarkably stable from the moment of random initialization all the way through training. This justifies why "zero-cost" proxies work: the potential of a network is essentially "baked in" to its structure.
  • Stability Matters: The authors observed that existing zero-cost proxies (like NASWOT) suffer from high score variance, leading to unstable rankings. Their entropy-based proxy is much more stable, leading to more reliable architecture selection.
  • First-Order is Enough: By proving that MAML's convergence is dominated by the first-order landscape, they provide a theoretical "shortcut" that bypasses the need for expensive second-order (Hessian) calculations common in meta-learning.

Key Results

The framework was evaluated across several standard benchmarks, including NAS-Bench-201, mini-ImageNet, tiered-ImageNet, ImageNet-1k, and the AutoFormer Benchmark. The results highlight massive efficiency gains:

  • Massive Speedup: On the mini-ImageNet 5-way 5-shot task, IBFS found a superior architecture in just 0.1 hours, compared to 168 hours for MetaNAS—a staggering 1680x reduction in search cost.
  • Superior Accuracy: On ImageNet-1k, the IBFS-searched architecture achieved 76.7% Top-1 accuracy, outperforming a wide range of competitors including DARTS, PC-DARTS, and even manually designed models like ResNet-50.
  • Generalization: Architectures searched on small datasets (like CIFAR) generalized exceptionally well to large-scale datasets (ImageNet-1k), proving the robustness of the IB-driven selection criteria.
  • Transformer Design: When applied to Transformers, IBFS achieved the highest Top-1 accuracy (76.5%) on the AutoFormer benchmark with significantly lower search costs than evolution-based methods.

Limitations and Future Work

While the results are impressive, the authors note a few areas for future exploration:

  • Edge Deployment: The current work focuses on discovery; the next step is deploying these optimized, lightweight architectures onto resource-constrained edge devices like Raspberry Pi or IoT sensors.
  • Cross-Domain FSL: The authors plan to investigate how well these proxies hold up when the "few shots" come from a completely different domain than the base training set (e.g., training on natural images but testing on medical X-rays).
  • Search Space Constraints: Like most NAS methods, IBFS is still bound by the predefined search space (the "blocks" it is allowed to choose from). Expanding this to more diverse or irregular structures remains an ongoing challenge.

Summary updated

Abstract

Neural Architecture Search (NAS) has recently outperformed hand-designed networks in various artificial intelligence areas. However, previous works only target a pre-defined task. For a new task in few-shot learning (FSL) scenarios, the architecture is either searched from scratch, which is neither efficient nor flexible, or borrowed architecture from the ones obtained on other tasks, which may lead to sub-optimal. Can we select the best neural architectures without involving any training and eliminate a significant portion of the search cost for new tasks in FSL? In this work, we provide an affirmative answer by proposing a novel information bottleneck (IB) theory driven \textit{Few-shot Neural Architecture Search} (dubbed, IBFS) framework to address this issue. We first derive that the global convergence of Model-agnostic meta-learning (MAML) can be guaranteed by only considering the first-order loss landscape. Moreover, motivated by the observation that IB provides a unified view toward understanding machine learning models, we propose a novel Zero-Cost method tailored for FSL to rank and select architectures based on their \textit{expressivity} obtained by IB mechanisms. Extensive experiments show that IBFS achieves state-of-the-art performance in FSL without training, which demonstrates the effectiveness of our IBFS.
BibTeX
@inproceedings{kang2025revisiting,
  author = {Haidong Kang},
  title = {{Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=fNixzmprun},
}
Figure