Decoupled Mixup for Generalized Visual Recognition

Haozhe Liu, Wentian Zhang, Jinheng Xie, Haoqian Wu, Bing Li, Ziqi Zhang, Yuexiang Li, Yawen Huang, Bernard Ghanem, Yefeng ZhengarXiv 2022 · DOI · arXiv

Abstract

Deep learning models often struggle with Out-of-Distribution (OOD) generalization, frequently over-fitting to environmental contexts rather than core object features. This paper presents Decoupled Mixup, a novel data augmentation strategy designed to enhance OOD robustness. By decoupling images into discriminative foregrounds and noise-prone backgrounds/textures, the method applies heterogeneous blending to prioritize informative signals while suppressing spurious correlations. Evaluated on the NICO Challenge, Decoupled Mixup achieved top-tier performance, demonstrating its effectiveness in recognizing objects across unseen contexts.

Introduction

Convolutional Neural Networks (CNNs) exhibit remarkable performance on in-distribution data but remain notoriously brittle when encountering domain shifts. A common failure mode is the reliance on "spurious correlations"—for instance, a model trained to recognize dogs on grass may fail to identify a dog on a beach because it has associated the "grass" context with the "dog" label.

While regularization techniques like Mixup have been widely adopted to improve generalization, they typically employ a "homogeneous" mixing strategy, blending all pixels of two images using a single interpolation weight. This approach inadvertently mixes environmental noise alongside the discriminative signal, potentially hindering the model's ability to focus on invariant object features. To address this, we propose Decoupled Mixup, which shifts from global blending to a targeted "decouple, mix, and suppress" framework.

Methodology: The Decoupled Mixup Framework

The core philosophy of Decoupled Mixup is to treat different image regions heterogeneously. The pipeline consists of two main variants: Context-aware and Frequency-aware decoupling.

Overview of the Decoupled Mixup pipeline, illustrating both Context-aware (CD-Mixup) and Frequency-aware (FD-Mixup) branches.

Context-aware Decoupled-Mixup (CD-Mixup)

CD-Mixup focuses on spatial disentanglement. It utilizes an unsupervised segmentation method, specifically Context-aware Contrastive Attention Maps (C2AM), to generate masks that separate the discriminative foreground (the object) from the noise-prone background.

During the mixing process, the foreground regions of two images are blended according to the target label ratio. In contrast, the background regions are either suppressed or mixed randomly. This forces the network to rely on the foreground features for classification, effectively breaking the reliance on specific environmental contexts.

Visual examples of Context-aware Decoupled-Mixup across various mixing ratios, showing how foreground objects are preserved while backgrounds are blended or suppressed.

Frequency-aware Decoupled-Mixup (FD-Mixup)

Recognizing that domain-specific "noise" often manifests as fine textures, FD-Mixup operates in the frequency domain using Fourier Transforms. It treats low-frequency components as "common patterns" representing shape and structure, while high-frequency components are viewed as "noise-prone" textures that vary across domains. By mixing low-frequency signals and suppressing high-frequency ones before reconstructing the image, the model is encouraged to learn more robust, structural representations.

Training Strategies

To maximize the efficacy of these augmentations, the authors implemented several advanced training techniques:

  • Curriculum Learning: Training progresses through increasing image resolutions (224x224 to 512x512), introducing Decoupled-Mixup at larger scales to refine feature learning.
  • Student-Teacher Regularization: A consistency loss ensures the model produces similar predictions for original images and their Fourier-augmented counterparts.
  • Model Ensembling: The final system leverages an ensemble of ResNet-34, WideResNet-50, and DenseNet-121 architectures to ensure robust predictions.

Experimental Results

The proposed method was rigorously evaluated on the NICO Challenge (Non-I.I.D. Image Classification with Out-of-distribution Objects). This dataset specifically tests a model's ability to generalize to objects in entirely unseen contexts.

The Decoupled Mixup approach demonstrated significant performance gains:

  • NICO Challenge Ranking: The team achieved 85.76% top-1 accuracy in Track-1 and 79.92% in Track-2, securing 4th place out of 39 competing teams.
  • Comparison with Baselines: Using a WideResNet-50 backbone, Decoupled-Mixup reached 89.83% accuracy on Track-1, substantially outperforming standard Mixup (86.76%) and Mixstyle (58.71%).
  • Architecture Agnostic: The improvements were consistent across various backbones, confirming the method's general applicability.

Conclusion

Decoupled Mixup addresses a fundamental limitation of standard data augmentation by acknowledging that not all pixels contribute equally to robust recognition. By explicitly separating and heterogeneously mixing foreground signals and background noise, the method forces models to learn domain-invariant features. While the approach currently relies on unsupervised saliency and frequency-based assumptions, it sets a strong precedent for more nuanced, content-aware augmentation strategies in the quest for truly generalized visual recognition.

Summary updated

Abstract

Convolutional neural networks (CNN) have demonstrated remarkable performance when the training and testing data are from the same distribution. However, such trained CNN models often largely degrade on testing data which is unseen and Out-Of-the-Distribution (OOD). To address this issue, we propose a novel "Decoupled-Mixup" method to train CNN models for OOD visual recognition. Different from previous work combining pairs of images homogeneously, our method decouples each image into discriminative and noise-prone regions, and then heterogeneously combines these regions of image pairs to train CNN models. Since the observation is that noise-prone regions such as textural and clutter backgrounds are adverse to the generalization ability of CNN models during training, we enhance features from discriminative regions and suppress noise-prone ones when combining an image pair. To further improve the generalization ability of trained models, we propose to disentangle discriminative and noise-prone regions in frequency-based and context-based fashions. Experiment results show the high generalization performance of our method on testing data that are composed of unseen contexts, where our method achieves 85.76\% top-1 accuracy in Track-1 and 79.92\% in Track-2 in the NICO Challenge. The source code is available at https://github.com/HaozheLiu-ST/NICOChallenge-OOD-Classification.
BibTeX
@misc{liu2022decoupled,
  author = {Haozhe Liu and Wentian Zhang and Jinheng Xie and Haoqian Wu and Bing Li and Ziqi Zhang and Yuexiang Li and Yawen Huang and Bernard Ghanem and Yefeng Zheng},
  title = {{Decoupled Mixup for Generalized Visual Recognition}},
  year = {2022},
  eprint = {2210.14783},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.14783},
}
Figure