GuidedMixup: An Efficient Mixup Strategy Guided by Saliency Maps

Minsoo Kang, Suhyun Kim

Abstract

Data augmentation is now an essential part of the image training process, as it effectively prevents overfitting and makes the model more robust against noisy datasets. Recent mixing augmentation strategies have advanced to generate the mixup mask that can enrich the saliency information, which is a supervisory signal. However, these methods incur a significant computational burden to optimize the mixup mask. From this motivation, we propose a novel saliency-aware mixup method, GuidedMixup, which aims to retain the salient regions in mixup images with low computational overhead. We develop an efficient pairing algorithm that pursues to minimize the conflict of salient regions of paired images and achieve rich saliency in mixup images. Moreover, GuidedMixup controls the mixup ratio for each pixel to better preserve the salient region by interpolating two paired images smoothly. The experiments on several datasets demonstrate that GuidedMixup provides a good trade-off between augmentation overhead and generalization performance on classification datasets. In addition, our method shows good performance in experiments with corrupted or reduced datasets.

Introduction

Deep neural networks are notorious for "memorizing" training data, leading to poor generalization on new, unseen images. Data augmentation—specifically "mixup" techniques that blend two images together—is a popular fix. However, standard methods like Mixup (linear interpolation) or CutMix (pasting a random box) often fail because they might obscure the most important part of an image (the "saliency") or create nonsensical training samples where the primary object is missing.

CutMix uses a hard binary mask which can often obscure salient regions of the background image.

While newer "saliency-aware" methods like PuzzleMix and Co-Mixup solve this by identifying and preserving important regions, they are computationally heavy. They often require solving complex optimization problems for every batch, which slows down training significantly. The authors of GuidedMixup set out to capture the benefits of saliency-aware blending without the massive "computational tax."

The GuidedMixup Strategy

GuidedMixup breaks the augmentation process into two distinct, efficient steps:

1. Greedy Pairing Algorithm

Instead of pairing images randomly, the algorithm looks at the saliency maps of all images in a mini-batch. It calculates the distance between these maps and uses a Greedy Algorithm ($O(M^2)$ complexity) to pair images whose salient regions overlap the least. This ensures that when two images are combined, their primary objects aren't fighting for the same space. This "conflict minimization" ensures that the model sees "richer" augmented samples where both labels are clearly represented.

2. Pixel-wise Masking

Rather than using a single global blending ratio ($\lambda$) or a hard binary mask, GuidedMixup calculates a unique mixing ratio for every single pixel. If Image A has high saliency at a specific pixel and Image B does not, the resulting pixel will look more like Image A. This creates a smooth, "guided" interpolation that retains the fine details of both objects. This smoothness provides a more natural "decision boundary" for the model compared to the jarring edges of box-based cropping methods.

PuzzleMix achieves saliency preservation through complex optimization, which GuidedMixup aims to simplify.

The authors tested two versions: Guided-SR (using a lightning-fast Spectral Residual saliency detector) and Guided-AP (using a more precise, but slightly slower, gradient-based approach).

Experimental Evaluation

The researchers evaluated GuidedMixup across a wide range of benchmarks, including general classification (CIFAR-100, ImageNet), Fine-Grained Visual Classification (CUB-200-2011, Stanford Cars), and robustness tests (CIFAR-100-C).

Key Results

  • Superior Accuracy: On CIFAR-100, Guided-AP achieved a 18.80% Top-1 error, outperforming PuzzleMix (19.62%) and Co-Mixup (19.85%).
  • Massive Speedup: In terms of "augmentation overhead" (extra training time), Guided-SR added only ~7.7% overhead at batch size 16. In comparison, SaliencyMix added 353% and Co-Mixup added 292%.
  • Data Scarcity & Robustness: When training on only 10% of the CIFAR-100 data, Guided-AP outperformed the baseline by over 14%. It also showed the lowest error rates on corrupted datasets (like those with Gaussian noise), proving the model learns more robust features.

Conclusion

GuidedMixup represents a significant step forward in making saliency-aware data augmentation practical for large-scale training. By decoupling the pairing logic from mask generation and utilizing an efficient greedy algorithm, it avoids the high complexity of previous optimization-based methods. While it still relies on the quality of the underlying saliency detector, its efficiency and performance across diverse datasets make it a compelling choice for modern deep learning pipelines.

Summary updated

Abstract

Data augmentation is now an essential part of the image training process, as it effectively prevents overfitting and makes the model more robust against noisy datasets. Recent mixing augmentation strategies have advanced to generate the mixup mask that can enrich the saliency information, which is a supervisory signal. However, these methods incur a significant computational burden to optimize the mixup mask. From this motivation, we propose a novel saliency-aware mixup method, GuidedMixup, which aims to retain the salient regions in mixup images with low computational overhead. We develop an efficient pairing algorithm that pursues to minimize the conflict of salient regions of paired images and achieve rich saliency in mixup images. Moreover, GuidedMixup controls the mixup ratio for each pixel to better preserve the salient region by interpolating two paired images smoothly. The experiments on several datasets demonstrate that GuidedMixup provides a good trade-off between augmentation overhead and generalization performance on classification datasets. In addition, our method shows good performance in experiments with corrupted or reduced datasets.
BibTeX
@inproceedings{kang2023guidedmixup,
  author = {Minsoo Kang and Suhyun Kim},
  title = {{GuidedMixup: An Efficient Mixup Strategy Guided by Saliency Maps}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=TR-HjFExFzT},
}
Figure