SuperMix: Supervising the Mixing Data Augmentation

Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, Nasser M. NasrabadiCVPR 2021 · DOI

Abstract

SuperMix is a supervised data augmentation technique that intelligently combines multiple images by prioritizing their most salient regions. By leveraging a teacher model to guide the mixing process and employing a high-speed Newton iterative optimization algorithm, SuperMix generates synthetic training data that is semantically more coherent than "blind" mixing methods like MixUp or CutMix. This approach leads to significant performance improvements in image classification and knowledge distillation tasks.

Introduction: The Problem with Blind Mixing

Deep Neural Networks (DNNs) require vast amounts of data to generalize effectively. Data augmentation techniques like MixUp (linear interpolation of images) and CutMix (patch-based replacement) have become standard tools to prevent overfitting. However, these methods are essentially "blind" to image content. A random crop in CutMix might replace a critical object feature with background noise, while MixUp can create "ghosting" effects where no distinct object is recognizable. Furthermore, the labels for these mixed images are typically simple linear interpolations that may not accurately reflect the visual prominence of the constituent classes.

Comparison of CutMix, MixUp, and SuperMix mixing strategies and their corresponding masks.

SuperMix addresses these limitations by asking: What if we used a model's own knowledge to ensure we only mix the most important parts of each image?

Methods: Supervised Mixing via Optimization

The core of SuperMix is a supervised mixing function. Instead of relying on random processes, it optimizes a set of mixing masks ($M$) to determine pixel-wise contributions from input images ($x_i$) to the final output ($\hat{x}$).

The SuperMix optimization pipeline using a teacher model and Newton's method.

1. Teacher Supervision

The algorithm utilizes a pre-trained "teacher" model to identify salient regions. The masks are optimized such that the teacher's prediction on the mixed image aligns with a target "soft label"—a weighted combination of the original class labels.

2. Constraints for Realism

To maintain image quality and prevent chaotic collages, two key constraints are applied:

  • Smoothness: Masks are constrained to be spatially smooth, ensuring the resulting image resembles a natural scene.
  • Sparsity: The optimization encourages masks to favor one image over others at any given pixel, reducing "ghosting" or blurring.

3. Efficient Optimization

Standard Gradient Descent is too slow for real-time augmentation during training. The authors developed a modified Newton iterative method with a "smooth projection" that is 65x faster than SGD. This efficiency makes SuperMix feasible for large-scale datasets like ImageNet.

Key Insights

  • Saliency Preservation: By protecting salient regions, SuperMix creates "harder" yet more informative training examples.
  • Optimization as Augmentation: This method shifts data augmentation from a stochastic process to a targeted, data-driven optimization problem.
  • Knowledge Distillation (KD) Synergy: SuperMix is particularly effective for KD. Mixed images containing features from multiple classes force the "student" model to learn complex inter-class relationships captured by the "teacher."
  • Accurate Labeling: Unlike MixUp, SuperMix labels are more representative because the optimization ensures the visual content actually matches the target distribution.

Key Results

The authors validated SuperMix on CIFAR-100 and ImageNet, showing state-of-the-art performance:

  • ImageNet Classification: Combining SuperMix with RandAugment achieved 78.2% top-1 accuracy on ResNet50, outperforming MixUp and CutMix.
  • Superior Distillation: In KD tasks, SuperMix improved student performance by an average of 3.4% on CIFAR-100 and 3.1% on ImageNet. In several instances, the student model even surpassed the teacher's performance.
  • Computational Efficiency: The Newton-based optimizer is at least 19x faster than other saliency-based methods, enabling scalability.

Limitations and Future Work

While powerful, SuperMix has dependencies:

  • Teacher Model: It requires a pre-trained teacher. While self-training is possible, a strong static teacher yields the best results.
  • Overhead: Despite the speedup, it is more computationally intensive than simple random cropping.
  • Hyperparameters: New parameters like the smoothing factor ($\sigma$) and sparsity weight ($\lambda_s$) require tuning for optimal results across different architectures.

The authors have made their code available at: https://github.com/alldbi/SuperMix

Summary updated

BibTeX
@inproceedings{dabouei2021supermix,
  author = {Ali Dabouei and Sobhan Soleymani and Fariborz Taherkhani and Nasser M. Nasrabadi},
  title = {{SuperMix: Supervising the Mixing Data Augmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01358},
}
Figure