Abstract
Deep learning models are often criticized for being "right for the wrong reasons," frequently relying on spurious background correlations rather than the actual object of interest. We introduce Top-GAP, a novel regularization technique that forces Convolutional Neural Networks (CNNs) to focus on a limited number of salient pixels. By integrating a "size prior" during training via top-k pooling and sparsity regularization, Top-GAP achieves a 25% boost in object localization accuracy and significant robustness to adversarial attacks, all while maintaining standard classification performance.
Introduction
A common failure mode in computer vision is the model's reliance on background context—for instance, identifying a "boat" based on water texture rather than the vessel itself. This leads to fragile models that fail under distribution shifts (e.g., a boat on grass). While post-hoc explanation tools like Grad-CAM visualize these biases, they do not correct them during training. Top-GAP addresses this by baking "focus" into the architecture, requiring only a rough estimate of object size rather than expensive pixel-level segmentation masks.
Methods
The Top-GAP framework modifies the standard CNN pipeline through three key innovations:
- The Top-GAP Layer: Standard Global Average Pooling (GAP) averages all pixels in a feature map. Top-GAP replaces this with top-k pooling, which only averages the $k$ highest-value pixels. This constraint forces the network to concentrate its "attention" on a specific number of locations.
- Sparsity Regularization: An $\ell_1$ regularization term is added to the loss function, incentivizing the model to push background pixel values toward zero and clearly distinguish important features.
- High-Resolution Feature Maps: To provide fine-grained control, the authors utilize a simplified Feature Pyramid Network (FPN) to generate higher-resolution final feature maps (e.g., 56x56), ensuring the $k$-pixel constraint is spatially meaningful.
Key Insights
Shrinking the Effective Receptive Field (ERF)
Top-GAP physically alters the model's vision. Gradient analysis shows that Top-GAP models exhibit high sensitivity to the image center (where objects typically reside) and near-zero sensitivity to corners and backgrounds compared to standard architectures.
Robustness and Interpretability
By ignoring the background, the "attack surface" for adversarial perturbations is drastically reduced. Furthermore, the model becomes inherently interpretable; its Class Activation Maps (CAMs) are naturally sparser and more tightly bound to object boundaries.
Results
The effectiveness of Top-GAP was validated across several benchmarks, including ImageNet, COCO, and specialized microscopy datasets.
- Localization: Top-GAP improved Intersection over Union (IoU) scores by up to 25% over Grad-CAM, indicating superior alignment with ground-truth object boundaries.
- Adversarial Defense: On CIFAR-10, Top-GAP maintained 51.7% accuracy under PGD-20 attacks, whereas standard models dropped to 0%.
- Bias Mitigation: On the Waterbirds dataset, Top-GAP improved accuracy by 4-5% by successfully ignoring misleading background cues.
Conclusion and Future Work
Top-GAP provides a simple yet powerful mechanism for improving CNN reliability. While the current implementation requires manual selection of the $k$ hyperparameter, future work aims to automate this process, allowing the model to dynamically adapt to varying object sizes within complex scenes.
Summary updated