Occlusions for Effective Data Augmentation in Image Classification

Ruth Fong, Andrea VedaldiWorkshops @ ICCV 2019 · DOI · OpenReview

Abstract

While occluding parts of an image (e.g., "Cutout") helps models generalize on small datasets, it historically struggled to improve performance on large-scale benchmarks like ImageNet. This paper identifies joint training (batch augmentation) as the critical missing component. By training on both original and occluded versions of an image simultaneously, high-capacity models like ResNet50 achieve significant accuracy gains.

Introduction: The Problem of "Lazy" Networks

Deep neural networks often suffer from "photographer bias," where the subject of a photo is usually centered and clearly visible. Consequently, models tend to over-rely on a few "easy-to-recognize" features—such as a cat’s face or a specific texture—rather than understanding the object holistically. This makes them fragile; if those specific primary features are hidden, the model fails.

Researchers have attempted to address this using occlusions (blocking out parts of the image) to force networks to attend to secondary cues (like the cat's body). However, while this improved object localization, it often failed to boost, or even degraded, classification accuracy on large-scale datasets like ImageNet. This paper investigates this discrepancy and proposes a robust training strategy to overcome it.

Methods of Occlusion

The authors explore two primary categories of occlusion generation:

1. Stochastic Occlusions

  • Hide-and-Seek (H&S): This method divides the image into a grid and randomly hides patches. It forces the model to find the object even when significant portions are missing.
  • Cutout (CO): This simpler approach involves placing one or more random square "holes" in the image.
The Hide-and-Seek (H&S) method divides an image into a grid and randomly masks patches to encourage global feature learning.

2. Saliency-based Occlusions

Instead of random masking, the authors utilize gradient-based saliency (specifically "NormGrad") to identify the most discriminative pixels for the model. By intentionally blocking these "important" areas, the model is forced to learn from less obvious but still relevant features. This targets the model's most discriminative features directly.

Saliency-based occlusion: The original image is processed to find a saliency map, and the most salient region is then occluded to force the model to look elsewhere.

The Key Innovation: Joint Training

The authors argue that previous failures on ImageNet were due to a "distribution shift"—training only on occluded images makes the training data too different from the clean test set. They propose batch augmentation: for every batch of images, the model sees both the original images and an occluded version of those same images. This joint training allows the model to maintain its baseline performance while learning the robust features encouraged by occlusion.

Key Insights and Results

  • Necessity of Joint Training: Seeing the unoccluded image in the same batch is essential for ImageNet performance. Without it, accuracy often drops.
  • Architecture Capacity: High-capacity models like ResNet50 benefit significantly from large occlusions, whereas lower-capacity models like AlexNet show negligible gains. This suggests occlusion can serve as a measure of an architecture's robustness potential.
  • Efficiency of Saliency: Saliency-based occlusion is highly efficient; hiding only 6% of an image via saliency can match the performance gains of hiding 50% randomly.
  • Performance Gains: On ImageNet, ResNet50 with Cutout and joint training improved Top-1 accuracy by +0.85% (reaching 77.25%). Hide-and-Seek with joint training provided a +0.62% boost.

Conclusion

Occlusion is a powerful tool for regularization, provided it is paired with joint training. By forcing high-capacity models to look beyond the most obvious features without losing sight of the original data distribution, researchers can significantly improve the robustness and accuracy of large-scale image classifiers.

Summary updated

Abstract

Deep networks for visual recognition are known to leverage "easy to recognise" portions of objects such as faces and distinctive texture patterns. The lack of a holistic understanding of objects may increase fragility and overfitting. In recent years, several papers have proposed to address this issue by means of occlusions as a form of data augmentation. However, successes have been limited to tasks such as weak localization and model interpretation, but no benefit was demonstrated on image classification on large-scale datasets. In this paper, we show that, by using a simple technique based on batch augmentation, occlusions as data augmentation can result in better performance on ImageNet for high-capacity models (e.g., ResNet50). We also show that varying amounts of occlusions used during training can be used to study the robustness of different neural network architectures.
BibTeX
@inproceedings{fong2019occlusions,
  author = {Ruth Fong and Andrea Vedaldi},
  title = {{Occlusions for Effective Data Augmentation in Image Classification}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  doi = {10.1109/iccvw.2019.00511},
}
Figure