Abstract
Modern Convolutional Neural Networks (CNNs) often suffer from a "lazy filter" problem, where the model over-relies on a small subset of fast-learning filters that latch onto easy dataset biases. This leads to vanishing gradients for the remaining "struggling" filters, resulting in brittle models with poor generalization. We present Catch-Up Mix, a novel feature-level augmentation strategy that identifies and mixes these struggling filters based on their activation norms. By suppressing dominant features and forcing the network to utilize underperforming channels, Catch-Up Mix encourages more diverse and robust feature representations. Our method significantly improves accuracy and resilience to noise, corruptions, and adversarial attacks across multiple benchmarks, including CIFAR-100, ImageNet, and fine-grained classification tasks.
Introduction: The Lazy Filter Problem
Modern CNNs are remarkably accurate but often suffer from a hidden structural inefficiency: they tend to rely heavily on a tiny fraction of their available filters to make predictions. "Fast-learning" filters often latch onto easy, dataset-specific biases (like background textures) early in training. Once the model achieves high training accuracy using these few filters, the gradients for the remaining "slow-learning" filters vanish. These "struggling filters" are effectively deprived of learning opportunities, leaving the model brittle. Standard regularization techniques like Dropout or Weight Decay mitigate overfitting but do not directly address this internal imbalance of filter utilization.
Methodology: Catch-Up Mix
Catch-Up Mix operates at the feature map level during training. Instead of mixing entire images or spatial regions, it mixes specific channels (filters) based on their current performance.
- Identify Struggling Filters: The method calculates "Filter Influence" ($FI$) by taking the $\ell_2$ norm of activation maps. Filters with low norms are considered "struggling."
- Relative Filter Influence (RFI): It compares the normalized influence of filters between a pair of images (source and target).
- Selective Mixing: It creates a binary mask that selects filters with relatively low $RFI$ and swaps/mixes their activation maps between the two images.
- The "Catch-Up" Mechanism: By training on these "defective" feature maps where dominant filters are suppressed, the model is forced to generate gradients for the struggling filters, giving them a "catch-up" opportunity to learn meaningful features.
Experimental Results
The authors validated Catch-Up Mix across a wide range of benchmarks, demonstrating superior performance in both accuracy and robustness.
- General Classification: On CIFAR-100, Catch-Up Mix achieved a 17.76% Top-1 error, significantly lower than the Baseline (22.98%) and outperforming competitors like AlignMixup (19.20%).
- Robustness: On Tiny-ImageNet, the model maintained 47.87% accuracy under extreme 40-degree rotations (vs. 42.99% baseline) and showed over 5% improvement in adversarial resilience against FGSM attacks.
- Fine-Grained Classification: It outperformed the best competitors on CUB-200 by 0.62% and on Aircraft by 0.90%, capturing the fine details necessary for specialized tasks.
Discussion and Key Insights
The core of Catch-Up Mix's success lies in addressing gradient imbalance. In baseline models, gradient scales for many layers drop to near zero after just 60 epochs, even if many filters haven't learned useful features yet.
As visualized above, Catch-Up Mix maintains a healthier distribution of gradients across different activation norms throughout training. This forced diversity leads to flatter loss landscapes, which are strongly correlated with better generalization.
The impact of this distributed learning is further evidenced by feature-dropping experiments. When the most influential features are removed, models trained with Catch-Up Mix degrade much more gracefully than baselines or other Mixup variants.
This suggests that Catch-Up Mix forces the network to find alternative, more robust pathways for classification rather than relying on a few "easy" features.
Limitations and Future Work
While Catch-Up Mix significantly improves robustness and out-of-distribution detection, it may exhibit an accuracy-robustness trade-off on massive datasets like ImageNet-1K. Its raw Top-1 accuracy (78.71%) was slightly lower than RecursiveMix (79.14%). Future work will investigate adaptive balancing of filter utilization to ensure that forcing the use of every filter does not become counterproductive on less complex datasets.
Summary updated