Abstract
Few-shot learning aims to adapt models to new tasks with minimal data. While Model-Agnostic Meta-Learning (MAML) focuses on weight initialization, Neural Architecture Search (NAS) explores optimal structures. However, traditional two-stage "search then train" approaches decouple architecture from weights, leading to sub-optimal results and high overhead. We present a joint optimization method featuring Progressive Connection Consolidation (PCC) and Connection-Adaptive Meta-learning (CAML). Our approach achieves state-of-the-art performance on Mini-Imagenet and FC100 while being 66% faster than existing NAS-based meta-learners.
Introduction: The Two-Stage Bottleneck
Meta-learning typically involves finding a set of meta-weights that can be quickly fine-tuned for specific tasks. When NAS is integrated, it often follows a rigid two-stage pipeline: searching for an architecture, then re-training weights from scratch. This separation ignores the Lottery Ticket Hypothesis, which suggests that specific architectures and their initialized weights are intrinsically linked.
Existing joint optimization attempts often suffer from biased updates because they treat architecture parameters and network weights as identical entities despite their distinct learning dynamics.
Methodology
To overcome these challenges, we introduce two core innovations:
1. Progressive Connection Consolidation (PCC)
Instead of pruning a massive "supernet" at the end of the search, PCC performs layer-by-layer consolidation. By calculating a "Layer Confidence" metric, the system identifies and fixes the most certain layers first. This allows remaining layers and meta-weights to adapt dynamically to the evolving structure.
2. Connection-Adaptive Meta-learning (CAML)
Jointly optimizing architecture and weights creates a complex 4-level optimization problem. CAML addresses this by decoupling the backpropagation paths for architecture parameters ($\phi$) and weights ($\theta$). This ensures that meta-updates remain parallel to the true meta-gradient, preventing the optimization "drift" common in single-propagation methods.
Experimental Results
We evaluated our approach on standard benchmarks including Mini-Imagenet and FC100.
- Performance: On Mini-Imagenet (5-way, 5-shot), our method reached 68.1% accuracy, surpassing the Auto-MAML baseline (64.1%).
- Efficiency: The search process requires only 0.7 GPU days, a significant reduction from the 2 days required by Auto-MAML and 112 days for Auto-Meta.
- Compactness: The resulting models are highly efficient, utilizing approximately 24.2K parameters, which is smaller than most competitive architectures.
Discussion and Conclusion
The primary insight of this work is the synergy between network structure and weights. By allowing them to "grow" together through PCC and ensuring unbiased updates via CAML, we achieve a more effective and efficient meta-learner. While our method currently utilizes the standard DARTS search space, future work could explore more diverse operations and task-specific architectures.
Summary updated