Improving Generalization in Meta-Learning via Meta-Gradient Augmentation

Ren Wang, Haoliang Sun, Yuxiu Lin, Xinxin Zhang, Yilong YinIJCAI 2025 · DOI

TL;DR

This paper introduces MGAug, a novel meta-learning technique that tackles overfitting by first breaking rote memorization in the inner loop through network pruning and then augmenting meta-gradients to improve generalization in the outer loop. This data-independent approach significantly boosts performance on few-shot learning benchmarks by enhancing the adaptability of meta-learned models.

What Problem Does This Paper Address?

Meta-learning aims to train models that can quickly adapt to new tasks with limited data. However, current meta-learning methods often suffer from two types of overfitting: memorization overfitting (where the model relies on memorizing past tasks rather than learning to adapt) and learner overfitting (where the meta-learner itself overfits to the training tasks). These issues hinder the model's ability to generalize to truly unseen tasks. Existing solutions, like data manipulation, are often task-specific and lack flexibility. This paper addresses this by proposing a more general and effective regularization strategy.

Methods

The core of MGAug lies in its two-pronged approach:

  • Inner-Loop Pruning for Memorization Breaking: Before the base learner fine-tunes on a new task's support set, MGAug prunes parts of the network. This forces the remaining network to re-learn and adapt, breaking the tendency to rely on rote memorization. Three pruning strategies are explored:
  • Random Width Pruning (WP): Removes entire neurons.
  • Random Parameter Pruning (PP): Randomly removes individual parameters.
  • Catfish Pruning (CP): A novel method that prunes parameters based on a "Meta-Memorization Carrying Amount" (MMCA) score, identifying and removing parameters that contribute most to rote memorization. This is inspired by the "catfish effect" in fish transportation, where a predator (catfish) keeps prey (sardines) active.
  • Meta-Gradient Augmentation: After pruning and fine-tuning, the gradients from these pruned sub-networks are used to augment the meta-gradients. This provides richer, more diverse task information to the outer loop, helping to alleviate learner overfitting and improve overall generalization.

The method is theoretically grounded with a PAC-Bayes generalization bound.

Datasets

The paper evaluates MGAug on two standard few-shot learning benchmarks:

  • mini-ImageNet: A widely used dataset for few-shot image classification.
  • CUB (Caltech-UCSD Birds-200-2011): A dataset focused on fine-grained bird image classification.

Experiments are conducted in both mutually exclusive (ME) and non-mutually-exclusive (NME) task settings.

Key Insights and Ideas

  • Pruning as a "Memorization Breaker": The key insight is that pruning in the inner loop can effectively disrupt rote memorization, forcing the model to truly adapt rather than recall. This is a novel application of pruning in the meta-learning context.
  • Catfish Pruning's Effectiveness: The proposed Catfish Pruning (CP) method, which quantifies a parameter's contribution to memorization, proves to be more effective than random pruning strategies in breaking rote memorization.
  • Augmented Gradients for Generalization: Using gradients from pruned sub-networks as meta-gradient augmentation provides a powerful mechanism to combat learner overfitting and improve the meta-learner's ability to generalize.
  • Data-Independent Regularization: MGAug's data-independent nature makes it broadly applicable across different meta-learning architectures and tasks, unlike many existing data-centric augmentation methods.
  • Trade-off Between Training Loss and Accuracy: MGAug demonstrates a beneficial trade-off where lower training loss is achieved alongside higher validation accuracy, indicating better generalization.

Key Results

  • MGAug significantly outperforms existing meta-learning baselines on both mini-ImageNet and CUB datasets across various settings (5-way 1-shot, 5-way 5-shot) and backbones (Conv-4, ResNet-10, ResNet-18).
  • For example, MGAug improves accuracy by up to 6.96% on CUB and 6.67% on mini-ImageNet in the 5-way 1-shot + ResNet-10 scenario compared to other regularization methods.
  • The Catfish Pruning (CP) strategy consistently yields better results than Random Width (WP) and Random Parameter (PP) pruning.
  • Visualizations (Grad-CAM) show that MGAug can lead to more diverse attention patterns across sub-networks, suggesting improved adaptability.
  • MGAug demonstrates a "plug-and-play" property, showing performance improvements even when applied only in the later stages of training.

Limitations and Future Work

  • Computational Cost: While MGAug offers significant benefits, the pruning and multiple forward passes in the inner loop can increase computational overhead compared to standard meta-learning.
  • Hyperparameter Tuning: The effectiveness of MGAug might depend on the choice of pruning rate and the hyperparameter U (number of sub-networks).
  • Future Work: The authors suggest that MGAug's ideas can inspire further development of regularization strategies in conventional learning. Further exploration of more sophisticated pruning criteria and their integration with other meta-learning frameworks could be beneficial.

Summary updated

BibTeX
@inproceedings{wang2025improving,
  author = {Ren Wang and Haoliang Sun and Yuxiu Lin and Xinxin Zhang and Yilong Yin},
  title = {{Improving Generalization in Meta-Learning via Meta-Gradient Augmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/711},
}
Figure