Survey: Image mixing and deleting for data augmentation

Humza Naveed, Saeed Anwar, Munawar Hayat, Kashif Javed, Ajmal Mian

TL;DR

This paper surveys image mixing and deleting techniques, a powerful data augmentation strategy that enhances neural network generalization and robustness by creating diverse training samples. It categorizes these methods, discusses their strengths and weaknesses, and highlights their broad applicability beyond image classification.

What Problem Does This Paper Address?

Deep neural networks, while achieving remarkable success, are prone to overfitting and memorizing training data patterns. This leads to poor generalization on unseen data and reduced robustness to real-world variations like occlusions or noise. Traditional data augmentation methods (e.g., rotation, cropping) help, but image mixing and deleting offer a more sophisticated approach by creating complex, realistic corruptions that force models to learn more robust representations. This survey aims to provide a comprehensive overview of these advanced techniques, their underlying principles, and their impact.

Methods

The paper categorizes image mixing and deleting techniques into three main types:

  • Cut and Delete: This involves erasing regions of an image to simulate occlusions. Techniques include:
  • Cutout: Randomly removes square patches and replaces them with a constant value.
  • Random Erasing: Similar to Cutout but with more flexible region selection and probability-based application.
  • Hide and Seek: Divides the image into grids and randomly turns cells on or off, replacing deleted areas with dataset averages.
  • GridMask: Creates evenly spaced black regions to balance information retention and deletion, aiming to avoid completely hiding important features.
  • Cut and Mix: This method replaces a selected image region with a patch from another image, introducing samples with mixed class labels. Key techniques include:
  • CutMix: Replaces a patch with a patch from another image in the mini-batch, with labels mixed proportionally to the patch area.
  • Attentive CutMix: Identifies "attentive" regions in an image and pastes them onto another, focusing on semantically important areas.
  • RICAP: Crops four random regions from four images and combines them into a new image with mixed labels.
  • SaliencyMix: Extracts salient regions from one image and pastes them onto another.
  • Mix and Up: This category mixes pixel values of two or more images, often through linear interpolation, to create new training samples. Examples include:
  • MixUp: Linearly interpolates pairs of images and their labels.
  • AugMix: Applies multiple random transformations to an image in parallel branches and then mixes the results.
  • SmoothMix: Uses soft masks with gradually decreasing intensity to blend images, creating smoother transitions.

The paper also discusses techniques tailored for specific tasks like object detection and fine-grained recognition, as well as those operating in feature spaces (Manifold MixUp) or employing more complex architectures.

Datasets

The paper references numerous standard datasets commonly used for evaluating image augmentation techniques, including:

  • CIFAR-10 and CIFAR-100: Widely used for image classification benchmarks.
  • ImageNet: A large-scale dataset for image classification, object detection, and other vision tasks.
  • VOC-07+12 and COCO: Standard datasets for object detection.
  • CUB, Cars, and Aircraft: Datasets for fine-grained image classification.

No new datasets were introduced in this survey paper.

Key Insights and Ideas

  • Beyond Simple Transformations: Image mixing and deleting go beyond traditional geometric and color transformations by creating more complex, realistic corruptions that better mimic real-world scenarios like occlusions and domain shifts.
  • Robustness to Occlusions: Techniques like Cutout and Random Erasing are particularly effective at improving robustness to occlusions by forcing the model to learn from partially visible objects.
  • Label Smoothing as a Byproduct: Methods like CutMix inherently introduce a form of label smoothing, which can improve classifier calibration and reduce overconfidence.
  • Focus on Salient Information: Many advanced techniques (e.g., Attentive CutMix, SaliencyMix, KeepAugment) leverage saliency maps to ensure that important image regions are either retained or strategically mixed, leading to more meaningful augmentations.
  • Generalization Across Tasks: The benefits of these augmentation strategies are not limited to image classification; they have shown significant improvements in object detection, fine-grained recognition, and even in areas like self-supervised learning and domain adaptation.
  • Computational Cost vs. Benefit: While some methods involve training multiple networks or complex saliency extraction, many offer significant performance gains with minimal additional computational overhead compared to training baselines.

Key Results

The paper presents comprehensive performance comparisons in Tables 4, 5, 6, and 7, showcasing the effectiveness of various mixing and deleting techniques across different tasks and datasets.

  • Image Classification: KeepAugment, Saliency Grafting, and SuperMix are highlighted as top performers on CIFAR-10, CIFAR-100, and ImageNet, respectively. The performance gap between salient and random mixing techniques is often small, suggesting that computational resources can guide the choice.
  • Object Detection: SaliencyMix outperforms on VOC-07, while KeepAugment leads on COCO-17. These methods, when used to train backbones, show notable improvements over baselines.
  • Fine-Grained Recognition: AttributeMix and SnapMix show strong results on datasets like CUB, Cars, and Aircraft, demonstrating their ability to capture subtle differences.
  • Vision Transformers: Techniques like TransMix and TokenMixup show gains when applied to vision transformers, indicating their adaptability to newer architectures.

Qualitatively, the paper shows examples of augmented images that are visually diverse and often retain semantic coherence, demonstrating the effectiveness of these methods in creating challenging yet informative training samples.

Limitations and Future Work

  • Computational Cost: Some salient-based augmentation methods require training separate networks for saliency extraction, increasing computational demands. Future work could focus on end-to-end training frameworks for augmentation and classification.
  • Irregular Structures: Generated images can sometimes have irregular structures or broken edges, contrasting with the consistent structure of traditional augmentations. Future research could focus on generating smoother, more structurally consistent augmented images.
  • Distribution Shift: There's a possibility that augmented image distributions might diverge significantly from original training data or target deployment data. Incorporating techniques that analyze sample similarity (like PointMixup) could help.
  • Feature Space Augmentation: While some work exists (e.g., Manifold MixUp), more research is needed on augmenting images in feature space, especially using saliency information.
  • Theoretical Understanding: While MixUp's regularization effects are well-understood, the theoretical underpinnings of Cutout and CutMix require further exploration beyond CAM visualizations.
  • Targeted Augmentation: Future work could explore corrupting foreground and background separately, or using StyleMix to alter style, texture, and content in a more controlled manner.

Summary updated

Abstract

Neural networks are prone to overfitting and memorizing data patterns. To avoid over-fitting and enhance their generalization and performance, various methods have been suggested in the literature, including dropout, regularization, label smoothing, etc. One such method is augmentation which introduces different types of corruption in the data to prevent the model from overfitting and to memorize patterns present in the data. A sub-area of data augmentation is image mixing and deleting. This specific type of augmentation either deletes image regions or mixes two images to hide or make particular characteristics of images confusing for the network, forcing it to emphasize the overall structure of the object in an image. Models trained with this approach have proven to perform and generalize well compared to those trained without image mixing or deleting. An added benefit that comes with this method of training is robustness against image corruption. Due to its low computational cost and recent success, researchers have proposed many image mixing and deleting techniques. We furnish an in-depth survey of image mixing and deleting techniques and provide categorization via their most distinguishing features. We initiate our discussion with some fundamental relevant concepts. Next, we present essentials, such as each category’s strengths and limitations, describing their working mechanism, basic formulations, and applications. We also discuss the general challenges and recommend possible future research directions for image mixing and deleting data augmentation techniques. Datasets and codes for evaluation are publicly available here.
BibTeX
@article{naveed2024survey,
  author = {Humza Naveed and Saeed Anwar and Munawar Hayat and Kashif Javed and Ajmal Mian},
  title = {{Survey: Image mixing and deleting for data augmentation}},
  journal = {Engineering Applications of AI},
  year = {2024},
  doi = {10.1016/j.engappai.2023.107791},
}
Figure