Catch-Up Mix: Catch-Up Class for Struggling Filters in CNN

Minsoo Kang, Minkoo Kang, Suhyun Kim

Abstract

Deep learning has achieved remarkable success in computer vision, yet models often suffer from complexity issues and overfitting. A significant concern is the "heavy reliance" problem, where a Convolutional Neural Network (CNN) depends on a limited subset of filters for predictions, compromising generalization and robustness. This phenomenon often occurs because "fast-learning" filters dominate the training process, depriving "slow-learning" filters of necessary gradient updates. We propose Catch-Up Mix, a novel feature-level augmentation strategy that identifies struggling filters and provides them with targeted learning opportunities. By mixing activation maps with lower norms, Catch-Up Mix promotes diverse representation and reduces reliance on dominant filters. Our experiments across various classification benchmarks demonstrate that Catch-Up Mix significantly enhances both accuracy and robustness against noise and adversarial attacks.

Introduction: The Problem of "Lazy Filters"

Despite their high accuracy, modern CNNs are surprisingly fragile. This fragility often stems from a "heavy reliance" on a small subset of filters. During the early stages of training, certain filters—termed "fast-learning" filters—latch onto easily identifiable dataset biases, such as specific background textures or colors, to minimize training loss quickly.

Once these dominant filters become sufficient to classify the training set, the global gradient drops significantly. This leads to "gradient starvation," where the remaining "slow-learning" filters are starved of the updates needed to develop complex, meaningful features. The resulting model may perform well on training data but lacks the feature diversity required to handle real-world variations like rotations, weather changes, or adversarial perturbations.

Methodology: The Catch-Up Mix Approach

Catch-Up Mix addresses this imbalance by operating in the feature space rather than the pixel space. Unlike traditional augmentation methods like Mixup or CutMix, which are often stochastic and model-agnostic, Catch-Up Mix is model-aware. It identifies the internal state of the network to determine where augmentation is most needed. The process follows four key steps:

  1. Filter Influence (FI) Calculation: For a specific layer, the method calculates the $\ell_2$ norm of the activation maps for every filter. This norm serves as a proxy for the filter's maturity and its contribution to the current prediction.
  2. Identifying Struggling Filters: The method compares the normalized influence of filters across a pair of images (source and target). It specifically identifies "struggling" filters—those with relatively low magnitudes that have not yet learned robust representations.
  3. Targeted Mixing: Instead of mixing entire feature maps, Catch-Up Mix selects these low-influence filters and mixes their activation maps between the two images.
  4. Forced Learning: By suppressing the dominant filters and presenting the network with "defective" feature maps that rely on the weaker ones, the model is forced to update the lagging filters. This provides a "catch-up" opportunity, ensuring a more balanced distribution of predictive responsibility across the network.

Experimental Evaluation

The researchers evaluated Catch-Up Mix across a wide array of standard benchmarks and architectures, including ResNet, WideResNet, and MobileNetV2.

Performance and Generalization

On the CIFAR-100 dataset, Catch-Up Mix achieved a 17.76% error rate, significantly outperforming the baseline (22.98%) and other popular methods like CutMix (19.95%) and ManifoldMixup (19.32%). Similar gains were observed in fine-grained recognition tasks; on the CUB-200-2011 (Birds) dataset, it outperformed the next best competitor by 0.62%, indicating that the method helps the model capture the subtle, diverse details necessary for distinguishing similar classes.

Robustness and Corruption

The method's impact on robustness was tested using CIFAR-100-C (19 types of corruptions) and ImageNet-O (out-of-distribution). Catch-Up Mix showed the highest robustness against common data corruptions (mCE) and reduced error rates in adversarial tests (FGSM) compared to other mixup variants.

Key Insights

  • Gradient Re-activation: Visualization of training dynamics shows that Catch-Up Mix effectively "re-activates" gradients for filters that would otherwise flatline early in training.
  • Flatter Loss Landscape: One of the most striking findings is that Catch-Up Mix results in a much flatter loss landscape. In deep learning theory, flatter minima are strongly correlated with better generalization to unseen data.
  • Improved Localization: The method improved Weakly Supervised Object Localization (WSOL), suggesting that the model focuses more on the actual object of interest rather than irrelevant background cues.

Limitations and Future Work

While highly effective, Catch-Up Mix does present some trade-offs. On massive datasets like ImageNet-1K, where model capacity may be fully utilized, forcing diversity can sometimes lead to a slight decrease in peak top-1 accuracy in exchange for significantly higher robustness. Additionally, the method introduces hyperparameters, such as the mixing ratio ($\alpha$) and the selection of layers for augmentation, which may require tuning for new architectures.

Future research may explore applying this "catch-up" philosophy to Vision Transformers (ViTs), where "attention collapse" presents a challenge similar to the lazy filter problem in CNNs.

Conclusion

Catch-Up Mix offers a sophisticated, model-aware alternative to traditional data augmentation. By identifying and supporting struggling filters, it prevents gradient starvation and fosters a more diverse and robust feature set. This leads to models that are not only more accurate but also significantly more resilient to the complexities of real-world data.

Summary updated

Abstract

Deep learning has made significant advances in computer vision, particularly in image classification tasks. Despite their high accuracy on training data, deep learning models often face challenges related to complexity and overfitting. One notable concern is that the model often relies heavily on a limited subset of filters for making predictions. This dependency can result in compromised generalization and an increased vulnerability to minor variations. While regularization techniques like weight decay, dropout, and data augmentation are commonly used to address this issue, they may not directly tackle the reliance on specific filters. Our observations reveal that the heavy reliance problem gets severe when slow-learning filters are deprived of learning opportunities due to fast-learning filters. Drawing inspiration from image augmentation research that combats over-reliance on specific image regions by removing and replacing parts of images, our idea is to mitigate the problem of over-reliance on strong filters by substituting highly activated features. To this end, we present a novel method called Catch-up Mix, which provides learning opportunities to a wide range of filters during training, focusing on filters that may lag behind. By mixing activation maps with relatively lower norms, Catch-up Mix promotes the development of more diverse representations and reduces reliance on a small subset of filters. Experimental results demonstrate the superiority of our method in various vision classification datasets, providing enhanced robustness.
BibTeX
@inproceedings{kang2024catch,
  author = {Minsoo Kang and Minkoo Kang and Suhyun Kim},
  title = {{Catch-Up Mix: Catch-Up Class for Struggling Filters in CNN}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=9BbQSakeHf},
}
Figure