TL;DR
This paper introduces MGAug, a novel meta-learning technique that tackles overfitting by first breaking rote memorization in the inner loop through network pruning and then augmenting meta-gradients to improve generalization in the outer loop. This data-independent approach significantly boosts performance on few-shot learning benchmarks by enhancing the adaptability of meta-learned models.
What Problem Does This Paper Address?
Meta-learning aims to train models that can quickly adapt to new tasks with limited data. However, current meta-learning methods often suffer from two types of overfitting: memorization overfitting (where the model relies on memorizing past tasks rather than learning to adapt) and learner overfitting (where the meta-learner itself overfits to the training tasks). These issues hinder the model's ability to generalize to truly unseen tasks. Existing solutions, like data manipulation, are often task-specific and lack flexibility. This paper addresses this by proposing a more general and effective regularization strategy.
Methods
The core of MGAug lies in its two-pronged approach:
- Inner-Loop Pruning for Memorization Breaking: Before the base learner fine-tunes on a new task's support set, MGAug prunes parts of the network. This forces the remaining network to re-learn and adapt, breaking the tendency to rely on rote memorization. Three pruning strategies are explored:
- Random Width Pruning (WP): Removes entire neurons.
- Random Parameter Pruning (PP): Randomly removes individual parameters.
- Catfish Pruning (CP): A novel method that prunes parameters based on a "Meta-Memorization Carrying Amount" (MMCA) score, identifying and removing parameters that contribute most to rote memorization. This is inspired by the "catfish effect" in fish transportation, where a predator (catfish) keeps prey (sardines) active.
- Meta-Gradient Augmentation: After pruning and fine-tuning, the gradients from these pruned sub-networks are used to augment the meta-gradients. This provides richer, more diverse task information to the outer loop, helping to alleviate learner overfitting and improve overall generalization.
The method is theoretically grounded with a PAC-Bayes generalization bound.
Datasets
The paper evaluates MGAug on two standard few-shot learning benchmarks:
- mini-ImageNet: A widely used dataset for few-shot image classification.
- CUB (Caltech-UCSD Birds-200-2011): A dataset focused on fine-grained bird image classification.
Experiments are conducted in both mutually exclusive (ME) and non-mutually-exclusive (NME) task settings.
Key Insights and Ideas
- Pruning as a "Memorization Breaker": The key insight is that pruning in the inner loop can effectively disrupt rote memorization, forcing the model to truly adapt rather than recall. This is a novel application of pruning in the meta-learning context.
- Catfish Pruning's Effectiveness: The proposed Catfish Pruning (CP) method, which quantifies a parameter's contribution to memorization, proves to be more effective than random pruning strategies in breaking rote memorization.
- Augmented Gradients for Generalization: Using gradients from pruned sub-networks as meta-gradient augmentation provides a powerful mechanism to combat learner overfitting and improve the meta-learner's ability to generalize.
- Data-Independent Regularization: MGAug's data-independent nature makes it broadly applicable across different meta-learning architectures and tasks, unlike many existing data-centric augmentation methods.
- Trade-off Between Training Loss and Accuracy: MGAug demonstrates a beneficial trade-off where lower training loss is achieved alongside higher validation accuracy, indicating better generalization.
Key Results
- MGAug significantly outperforms existing meta-learning baselines on both mini-ImageNet and CUB datasets across various settings (5-way 1-shot, 5-way 5-shot) and backbones (Conv-4, ResNet-10, ResNet-18).
- For example, MGAug improves accuracy by up to 6.96% on CUB and 6.67% on mini-ImageNet in the 5-way 1-shot + ResNet-10 scenario compared to other regularization methods.
- The Catfish Pruning (CP) strategy consistently yields better results than Random Width (WP) and Random Parameter (PP) pruning.
- Visualizations (Grad-CAM) show that MGAug can lead to more diverse attention patterns across sub-networks, suggesting improved adaptability.
- MGAug demonstrates a "plug-and-play" property, showing performance improvements even when applied only in the later stages of training.
Limitations and Future Work
- Computational Cost: While MGAug offers significant benefits, the pruning and multiple forward passes in the inner loop can increase computational overhead compared to standard meta-learning.
- Hyperparameter Tuning: The effectiveness of MGAug might depend on the choice of pruning rate and the hyperparameter
U(number of sub-networks). - Future Work: The authors suggest that MGAug's ideas can inspire further development of regularization strategies in conventional learning. Further exploration of more sophisticated pruning criteria and their integration with other meta-learning frameworks could be beneficial.
Summary updated