Learning to Learn by Jointly Optimizing Neural Architecture and Weights

Yadong Ding, Yu Wu, Chengyue Huang, Siliang Tang, Yi Yang, Longhui Wei, Yueting Zhuang, Qi Tian

Abstract

Few-shot learning aims to adapt models to new tasks with minimal data. While Model-Agnostic Meta-Learning (MAML) focuses on weight initialization, Neural Architecture Search (NAS) explores optimal structures. However, traditional two-stage "search then train" approaches decouple architecture from weights, leading to sub-optimal results and high overhead. We present a joint optimization method featuring Progressive Connection Consolidation (PCC) and Connection-Adaptive Meta-learning (CAML). Our approach achieves state-of-the-art performance on Mini-Imagenet and FC100 while being 66% faster than existing NAS-based meta-learners.

Introduction: The Two-Stage Bottleneck

Meta-learning typically involves finding a set of meta-weights that can be quickly fine-tuned for specific tasks. When NAS is integrated, it often follows a rigid two-stage pipeline: searching for an architecture, then re-training weights from scratch. This separation ignores the Lottery Ticket Hypothesis, which suggests that specific architectures and their initialized weights are intrinsically linked.

Comparison of meta-learning paradigms: (a) MAML with fixed architecture, (b) traditional two-stage NAS-based meta-learning, and (c) the proposed joint optimization method.

Existing joint optimization attempts often suffer from biased updates because they treat architecture parameters and network weights as identical entities despite their distinct learning dynamics.

Methodology

To overcome these challenges, we introduce two core innovations:

1. Progressive Connection Consolidation (PCC)

Instead of pruning a massive "supernet" at the end of the search, PCC performs layer-by-layer consolidation. By calculating a "Layer Confidence" metric, the system identifies and fixes the most certain layers first. This allows remaining layers and meta-weights to adapt dynamically to the evolving structure.

Detail of the Progressive Connection Consolidation (PCC) process, showing the gradual solidification of network connections during training.

2. Connection-Adaptive Meta-learning (CAML)

Jointly optimizing architecture and weights creates a complex 4-level optimization problem. CAML addresses this by decoupling the backpropagation paths for architecture parameters ($\phi$) and weights ($\theta$). This ensures that meta-updates remain parallel to the true meta-gradient, preventing the optimization "drift" common in single-propagation methods.

Experimental Results

We evaluated our approach on standard benchmarks including Mini-Imagenet and FC100.

  • Performance: On Mini-Imagenet (5-way, 5-shot), our method reached 68.1% accuracy, surpassing the Auto-MAML baseline (64.1%).
  • Efficiency: The search process requires only 0.7 GPU days, a significant reduction from the 2 days required by Auto-MAML and 112 days for Auto-Meta.
  • Compactness: The resulting models are highly efficient, utilizing approximately 24.2K parameters, which is smaller than most competitive architectures.

Discussion and Conclusion

The primary insight of this work is the synergy between network structure and weights. By allowing them to "grow" together through PCC and ensuring unbiased updates via CAML, we achieve a more effective and efficient meta-learner. While our method currently utilizes the standard DARTS search space, future work could explore more diverse operations and task-specific architectures.

Summary updated

Abstract

Meta-learning enables models to adapt to new environments rapidly with a few training examples. Current gradient-based meta-learning methods concentrate on finding good model-agnostic initialization (meta-weights) for learners. In this paper, we aim to obtain better meta-learners by co-optimizing the architecture and meta-weights simultaneously. Existing NAS-based meta-learning methods apply a two-stage strategy, i.e., first searching architectures and then re-training meta-weights on the searched architecture. However, this two-stage strategy would break the mutual impact of the architecture and meta-weights since they are optimized separately. Differently, we propose progressive connection consolidation, fixing the architecture layer by layer, in which the layer with the largest weight value would be fixed first. In this way, we can jointly search architectures and train the meta-weights on fixed layers. Besides, to improve the generalization performance of the searched meta-learner on all tasks, we propose a more effective rule for co-optimization, namely Connection-Adaptive Meta-learning (CAML). By searching only once, we can obtain both adaptive architecture and meta-weights for meta-learning. Extensive experiments show that our method achieves state-of-the-art performance with 3x less computational cost, revealing our method's effectiveness and efficiency.
BibTeX
@inproceedings{ding2022learning,
  author = {Yadong Ding and Yu Wu and Chengyue Huang and Siliang Tang and Yi Yang and Longhui Wei and Yueting Zhuang and Qi Tian},
  title = {{Learning to Learn by Jointly Optimizing Neural Architecture and Weights}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00023},
}
Figure