Abstract
Convolutional Neural Networks (CNNs) often over-rely on a limited set of highly discriminative features, leading to brittleness under distribution shifts or adversarial attacks. This paper proposes TENET, a group-wise inhibition-based feature regularization method. By dynamically identifying and suppressing dominant features during training, TENET forces the network to explore diverse auxiliary features, enhancing both robustness and generalization without sacrificing standard accuracy.
Introduction
Standard CNNs are notoriously vulnerable to distribution shifts—small changes like motion blur, rain, or intentional adversarial perturbations. The authors argue this happens because CNNs are "lazy" learners: they focus on the most obvious discriminative patterns (e.g., the beak of a bird) and ignore subtler auxiliary features (e.g., the wings or tail). If an adversarial attack or noise obscures that primary feature, the model has no backup plan. Current solutions like data augmentation or adversarial training often involve a trade-off where increasing robustness hurts the model's standard accuracy.
Methodology: TENET Training
The authors propose TENET (Group-wise Inhibition based Feature Regularization), which employs a "two-pass" inference strategy during training to diversify the feature representations learned by the model.
The Two-Pass Strategy
- Channel-wise Feature Grouping (CFG): The model groups similar feature maps together using a clustering approach to identify which parts of the network are looking at the same types of patterns.
- Group-wise Map Weighting (GMW): The model calculates the "importance" of each group to the final prediction using gradients, identifying the dominant "star player" features.
- Rectified Reverse Function (RRF): Instead of simple deletion, which can destabilize training, a smooth function suppresses the most active regions.
- Group-wise Inhibition: The network is forced to make a prediction using only the remaining auxiliary features. By penalizing the model when it cannot classify correctly using these "backup" features, the training process forces the CNN to learn a much richer representation of the data.
Experimental Evaluation
The method was evaluated across a wide variety of tasks, including standard classification (PASCAL VOC 2012), robustness against corruption (CIFAR-10/100-C, ImageNet-C), and generalization in low-data regimes (CUB-200).
Key Results
- Robustness: On CIFAR-10, TENET improved the top-1 error rate of adversarial training from 36.37% down to 31.75%.
- Corruption Resistance: On ImageNet-C, TENET combined with AugMix achieved a Mean Corruption Error (mCE) of 69.6%, outperforming standard regularization techniques.
- Generalization: In low-data scenarios (CUB-200 with 10-30 samples per class), TENET consistently outperformed specialized few-shot methods, proving that feature diversity helps even when data is scarce.
Visual proof via Grad-CAM shows that while a baseline model might only look at a single discriminative part (like a bird's head), a TENET-trained model distributes its attention across the head, wings, and tail simultaneously.
Discussion and Insights
TENET represents a shift toward content-aware regularization. Unlike Dropout or Cutout, which remove neurons or patches randomly, TENET identifies what the model is actually looking at and suppresses it dynamically. This effectively "rotates" the model's attention to discover new dimensions of data that were previously ignored.
Furthermore, TENET is complementary to existing methods like Adversarial Training (AT). When combined, they achieve significantly better results than either method alone, suggesting that learning diverse features is a fundamental requirement for true robustness.
Limitations
Future work may address the computational overhead of the "two-pass" inference and the sensitivity of new hyperparameters (such as the number of groups $N_G$). Additionally, exploring how these dynamics scale to Vision Transformers (ViTs) remains an open area of research.
Summary updated