Abstract
This paper investigates the visual and quantitative characteristics of overfitting in image classifiers by transforming them into weakly supervised segmentation models using the SCrIBE (Implicit Background Estimation) technique. We introduce a label-free "Sparsity" metric derived from spatial attention maps to detect overfitting on unlabeled test data. Our findings reveal that overfit models exhibit dilated, non-specific attention areas, whereas well-generalized models maintain precise, localized focus. We further analyze how various data augmentations mitigate these effects and improve model calibration under diverse image corruptions.
Introduction
In safety-critical computer vision applications, such as autonomous driving, models must remain robust against corrupted inputs like weather, lighting changes, or motion blur. However, the lack of large, labeled datasets often leads to overfitting, which is traditionally difficult to detect without a labeled validation set. Standard performance metrics like accuracy often fail to explain why a model fails or what features it prioritizes. This research addresses these challenges by providing a visual framework for assessing model reliability and a quantitative, label-free metric for detecting overfitting.
Methodology
SCrIBE and Weak Segmentation
The authors propose modifying standard image classifiers using SCrIBE. By replacing the final Softmax layer with a Sigmoid function and employing Log-SumExp (LSE) pooling, the model treats the image background as the mathematical complement of the foreground. This architectural adjustment allows models trained only on image-level labels to produce Class Activation Maps (CAMs) or coarse segmentation masks, revealing the pixels that drive specific decisions.
Label-Free Overfit Detection
A key contribution is the Sparsity metric, defined as the percentage of zeros in the attention map. The authors observed that overfit models latch onto irrelevant, sample-specific noise, resulting in "smeared" or dilated attention areas (low sparsity). Conversely, well-generalized models produce tight, precise attention maps (high sparsity). This allows for the detection of overfitting purely by analyzing the shape of internal attention maps on unlabeled data.
Visualizing Overfitting and Generalization
Experiments on CIFAR-10 demonstrate that as the training subset size increases from 10% to 100% (reducing overfitting), the attention maps become significantly more localized. As shown in the visualization above, models trained on small subsets exhibit messy, dilated footprints. Data augmentations like RandomCrop and ColorJitter intentionally "smear" attention slightly to prevent hyper-focusing on specific pixels, which paradoxically leads to better generalization and calibration.
Quantitative Performance and Calibration
SCrIBE models achieve performance parity with standard ResNet-18 architectures while offering superior interpretability. Quantitative analysis shows that combining all data augmentations significantly reduces the Expected Calibration Error (ECE), dropping it to 2.2% compared to 7.2% without augmentation. While accuracy increases with more training data, the calibration benefits of augmentation remain consistent across different data regimes.
Robustness to Corruptions
Evaluating models against ImageNet-C corruptions revealed that specific augmentations act as targeted defenses. For instance, ColorJitter effectively mitigates brightness and fog corruptions, while RandomRotation provides the best defense against noise-based corruptions. The segmentation masks clarify these failure modes: noise-based corruptions often trick models into detecting textures associated with specific classes (e.g., "frogs"), while blur-based corruptions force an over-reliance on broad shapes.
Conclusion
This work demonstrates that overfitting has a distinct visual footprint that can be quantified without labeled data. By using implicit background models, researchers can gain intuitive insights into model decision-making and robustness. Future work may involve scaling these methods to higher-resolution datasets and integrating sparsity-based monitoring into real-time safety systems.
Summary updated