Catch-Up Mix: Catch-Up Class for Struggling Filters in CNN

Minsoo Kang, Minkoo Kang, Suhyun Kim

Abstract

Modern Convolutional Neural Networks (CNNs) often suffer from a "lazy filter" problem, where the model over-relies on a small subset of fast-learning filters that latch onto easy dataset biases. This leads to vanishing gradients for the remaining "struggling" filters, resulting in brittle models with poor generalization. We present Catch-Up Mix, a novel feature-level augmentation strategy that identifies and mixes these struggling filters based on their activation norms. By suppressing dominant features and forcing the network to utilize underperforming channels, Catch-Up Mix encourages more diverse and robust feature representations. Our method significantly improves accuracy and resilience to noise, corruptions, and adversarial attacks across multiple benchmarks, including CIFAR-100, ImageNet, and fine-grained classification tasks.

Introduction: The Lazy Filter Problem

Modern CNNs are remarkably accurate but often suffer from a hidden structural inefficiency: they tend to rely heavily on a tiny fraction of their available filters to make predictions. "Fast-learning" filters often latch onto easy, dataset-specific biases (like background textures) early in training. Once the model achieves high training accuracy using these few filters, the gradients for the remaining "slow-learning" filters vanish. These "struggling filters" are effectively deprived of learning opportunities, leaving the model brittle. Standard regularization techniques like Dropout or Weight Decay mitigate overfitting but do not directly address this internal imbalance of filter utilization.

Methodology: Catch-Up Mix

Catch-Up Mix operates at the feature map level during training. Instead of mixing entire images or spatial regions, it mixes specific channels (filters) based on their current performance.

  1. Identify Struggling Filters: The method calculates "Filter Influence" ($FI$) by taking the $\ell_2$ norm of activation maps. Filters with low norms are considered "struggling."
  2. Relative Filter Influence (RFI): It compares the normalized influence of filters between a pair of images (source and target).
  3. Selective Mixing: It creates a binary mask that selects filters with relatively low $RFI$ and swaps/mixes their activation maps between the two images.
  4. The "Catch-Up" Mechanism: By training on these "defective" feature maps where dominant filters are suppressed, the model is forced to generate gradients for the struggling filters, giving them a "catch-up" opportunity to learn meaningful features.

Experimental Results

The authors validated Catch-Up Mix across a wide range of benchmarks, demonstrating superior performance in both accuracy and robustness.

  • General Classification: On CIFAR-100, Catch-Up Mix achieved a 17.76% Top-1 error, significantly lower than the Baseline (22.98%) and outperforming competitors like AlignMixup (19.20%).
  • Robustness: On Tiny-ImageNet, the model maintained 47.87% accuracy under extreme 40-degree rotations (vs. 42.99% baseline) and showed over 5% improvement in adversarial resilience against FGSM attacks.
  • Fine-Grained Classification: It outperformed the best competitors on CUB-200 by 0.62% and on Aircraft by 0.90%, capturing the fine details necessary for specialized tasks.

Discussion and Key Insights

The core of Catch-Up Mix's success lies in addressing gradient imbalance. In baseline models, gradient scales for many layers drop to near zero after just 60 epochs, even if many filters haven't learned useful features yet.

Gradient and activation norm distributions over training epochs for Baseline vs Catch-up Mix.

As visualized above, Catch-Up Mix maintains a healthier distribution of gradients across different activation norms throughout training. This forced diversity leads to flatter loss landscapes, which are strongly correlated with better generalization.

The impact of this distributed learning is further evidenced by feature-dropping experiments. When the most influential features are removed, models trained with Catch-Up Mix degrade much more gracefully than baselines or other Mixup variants.

Accuracy degradation when dropping features with highest vs lowest values.

This suggests that Catch-Up Mix forces the network to find alternative, more robust pathways for classification rather than relying on a few "easy" features.

Limitations and Future Work

While Catch-Up Mix significantly improves robustness and out-of-distribution detection, it may exhibit an accuracy-robustness trade-off on massive datasets like ImageNet-1K. Its raw Top-1 accuracy (78.71%) was slightly lower than RecursiveMix (79.14%). Future work will investigate adaptive balancing of filter utilization to ensure that forcing the use of every filter does not become counterproductive on less complex datasets.

Summary updated

Abstract

Deep learning has made significant advances in computer vision, particularly in image classification tasks. Despite their high accuracy on training data, deep learning models often face challenges related to complexity and overfitting. One notable concern is that the model often relies heavily on a limited subset of filters for making predictions. This dependency can result in compromised generalization and an increased vulnerability to minor variations. While regularization techniques like weight decay, dropout, and data augmentation are commonly used to address this issue, they may not directly tackle the reliance on specific filters. Our observations reveal that the heavy reliance problem gets severe when slow-learning filters are deprived of learning opportunities due to fast-learning filters. Drawing inspiration from image augmentation research that combats over-reliance on specific image regions by removing and replacing parts of images, our idea is to mitigate the problem of over-reliance on strong filters by substituting highly activated features. To this end, we present a novel method called Catch-up Mix, which provides learning opportunities to a wide range of filters during training, focusing on filters that may lag behind. By mixing activation maps with relatively lower norms, Catch-up Mix promotes the development of more diverse representations and reduces reliance on a small subset of filters. Experimental results demonstrate the superiority of our method in various vision classification datasets, providing enhanced robustness.
BibTeX
@inproceedings{kang2024catch,
  author = {Minsoo Kang and Minkoo Kang and Suhyun Kim},
  title = {{Catch-Up Mix: Catch-Up Class for Struggling Filters in CNN}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=9BbQSakeHf},
}
Figure