SpliceMix: A Cross-Scale and Semantic Blending Augmentation Strategy for Multi-Label Image Classification

Lei Wang, Yibing Zhan, Leilei Ma, Dapeng Tao, Liang Ding, Chen Gong

TL;DR

SpliceMix is a novel data augmentation strategy specifically designed for Multi-label Image Classification (MLIC). By splicing downsampled images into a grid and training them alongside original images, it enables models to better handle varying object scales and reduce "contextual bias"—the tendency of models to only recognize objects when they appear in their usual surroundings.

What Problem Does This Paper Address?

Multi-label Image Classification (MLIC) is significantly more complex than standard single-label classification. In a typical MLIC scene, multiple objects of varying sizes co-occur (e.g., a "cup" on a "dining table").

Current state-of-the-art data augmentation techniques like Mixup (linearly blending two images) or CutMix (pasting a patch of one image onto another) were primarily designed for single-label tasks. When applied to MLIC, these methods often fail because they can inadvertently "cut" through small objects or create labels that don't accurately reflect the resulting messy image. Furthermore, MLIC models often suffer from contextual bias: if a model always sees a "baseball glove" with a "person," it might fail to recognize the glove when it appears alone.

The authors aim to create a simple, non-parametric augmentation that preserves object integrity, handles scale discrepancies, and breaks misleading contextual correlations.

Methods

The authors introduce two main components:

  1. SpliceMix: Instead of cropping or blending, SpliceMix takes several images from a mini-batch, downsamples them, and arranges them into a grid (e.g., a 2x2 grid of four different images). The label for this new "spliced" image is simply the union of all labels from the constituent images. This preserves the entirety of every object, albeit at a lower resolution.
  2. SpliceMixed Mini-batch: Rather than replacing the original training images, the authors "splice" the new mixed images into the original mini-batch. This creates a training step where the model sees the same categories at two different scales simultaneously: the original high-resolution version and the downsampled version in the grid.
  3. SpliceMix-CL (Consistency Learning): This is an optional extension. It uses the model's own predictions on the high-resolution "regular" images to supervise the predictions of the corresponding downsampled sub-images within the grid. This forces the model to learn consistent representations across different scales.

The researchers also experimented with different grid configurations, including asymmetric grids (e.g., 2x3) and "grid dropout," where some cells in the grid are randomly masked to prevent the scenes from becoming overly complex.

Datasets

The method was evaluated on three major benchmarks:

  • MS-COCO 2014: The gold standard for multi-label classification, featuring 80 object categories.
  • Pascal VOC 2007: A widely used dataset with 20 categories.
  • COCO-Stuff: Used specifically to evaluate contextual bias, as it provides pixel-level annotations for "stuff" (like sky, grass) and "things" (like cars, people).

Key Insights and Ideas

  • Preserving the "Whole": By downsampling rather than cropping (as in CutMix), SpliceMix ensures that small objects are never accidentally deleted from the training sample. This is crucial for MLIC, where small objects are common.
  • Cross-Scale Synergy: Including both original and downsampled images in the same batch allows for "cross-scale learning." The model learns that a "car" is a "car" whether it takes up 500 pixels or 50 pixels.
  • Breaking Contextual Chains: By placing a "refrigerator" in a grid next to a "sheep," the model is forced to recognize the refrigerator based on its own features rather than relying on the presence of a "kitchen" background.
  • Plug-and-Play: Unlike many MLIC solutions that require complex graph neural networks or attention mechanisms, SpliceMix is a pure data-level augmentation that can be added to any backbone (ResNet, ViT, Swin Transformer) with just a few lines of code.

Key Results

  • Superior Performance: On MS-COCO, SpliceMix-CL achieved 84.9% mAP, outperforming both standard baselines (83.0%) and other popular augmentations like Mixup and CutMix.
  • Robustness to Resolution: One of the most striking results was the model's performance on low-resolution images. When tested on 64x64 images, SpliceMix outperformed other methods by over 10% mAP, proving its effectiveness in learning scale-invariant features.
  • Small Object Mastery: The authors found that SpliceMix significantly boosted the Average Precision (AP) for the smallest object categories in the dataset, nearly doubling the improvement seen for large objects.
  • Faster Convergence: Models trained with SpliceMix reached higher accuracy levels faster than those using standard training or Batch Augmentation (BA).

Limitations and Future Work

While SpliceMix is highly effective, the authors note a few areas for further exploration:

  • Grid Complexity: Very large grids (e.g., 3x3) can sometimes make the scene too complicated, potentially harming the model's ability to learn. The "dropout" mechanism helps, but finding the optimal grid density remains an area for tuning.
  • Advanced Consistency: The consistency learning (CL) used here is non-parametric and simple. The authors suggest that "marrying" SpliceMix with more elaborate knowledge distillation or teacher-student frameworks could yield even better results.
  • Hyperparameter Sensitivity: While generally robust, the performance does vary slightly based on the number of mixed images added to each batch.

Summary updated

Abstract

Recently, Mix-style data augmentation methods (e.g., Mixup and CutMix) have shown promising performance in various visual tasks. However, these methods are primarily designed for single-label images, ignoring the considerable discrepancies between single- and multi-label images, i.e., a multi-label image involves multiple co-occurred categories and fickle object scales. On the other hand, previous multi-label image classification (MLIC) methods tend to design elaborate models, bringing expensive computation. In this article, we introduce a simple but effective augmentation strategy for multi-label image classification, namely SpliceMix. The “splice” in our method is two-fold: 1) Each mixed image is a splice of several downsampled images in the form of a grid, where the semantics of images attending to mixing are blended without object deficiencies for alleviating co-occurred bias; 2) We splice mixed images and the original mini-batch to form a new SpliceMixed mini-batch, which allows an image with different scales to contribute to training together. Furthermore, such splice in our SpliceMixed mini-batch enables interactions between mixed images and original regular images. We also provide a simple and non-parametric extension based on consistency learning (SpliceMix-CL) to show the potential of extending our SpliceMix. Extensive experiments on various tasks demonstrate that only using SpliceMix with a baseline model (e.g., ResNet) achieves better performance than state-of-the-art methods. Moreover, the generalizability of our SpliceMix is further validated by the improvements in current MLIC methods when married with our SpliceMix.
BibTeX
@article{wang2025splicemix,
  author = {Lei Wang and Yibing Zhan and Leilei Ma and Dapeng Tao and Liang Ding and Chen Gong},
  title = {{SpliceMix: A Cross-Scale and Semantic Blending Augmentation Strategy for Multi-Label Image Classification}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=5mlhgR9Dj6},
}
Figure