Abstract
Real-world data often follows a long-tailed distribution, where a few head classes occupy most of the data and a large number of tail classes only contain very limited samples. In practice, deep models often show poor generalization performance on tail classes due to the imbalanced distribution. To tackle this, data augmentation has become an effective way by synthesizing new samples for tail classes. Among them, one popular way is to use CutMix that explicitly mixups the images of tail classes and the others, while constructing the labels according to the ratio of areas cropped from two images. However, the area-based labels entirely ignore the inherent semantic information of the augmented samples, often leading to misleading training signals. To address this issue, we propose a Contrastive CutMix (ConCutMix) that constructs augmented samples with semantically consistent labels to boost the performance of long-tailed recognition. Specifically, we compute the similarities between samples in the semantic space learned by contrastive learning, and use them to rectify the area-based labels. Experiments show that our ConCutMix significantly improves the accuracy on tail classes as well as the overall performance. For example, based on ResNeXt-50, we improve the overall accuracy on ImageNet-LT by 3.0% thanks to the significant improvement of 3.3% on tail classes. We highlight that the improvement also generalizes well to other benchmarks and models.
Introduction
In real-world computer vision applications, datasets rarely exhibit the uniform class distributions found in curated benchmarks like CIFAR-10 or ImageNet. Instead, they typically follow a "long-tailed" distribution: a small number of "head" classes (e.g., common objects like dogs or cars) are represented by thousands of samples, while a vast majority of "tail" classes (e.g., rare species or specific medical conditions) have very few examples. Standard deep learning models trained on such data tend to overfit the head classes and generalize poorly to the tail.
Data augmentation, particularly regional mixing strategies like CutMix, has emerged as a powerful tool to mitigate this imbalance. CutMix generates synthetic training samples by pasting a patch from one image onto another. However, its reliance on area-based labeling—where the ground truth label is a weighted average based strictly on the number of pixels from each source image—presents a significant bottleneck in long-tailed scenarios.
The Semantic Mismatch Problem
The fundamental flaw in traditional CutMix is the assumption that pixel area correlates perfectly with semantic importance. For instance, if a small patch containing a cat's face is pasted onto a large image of a park, the area-based label might assign the image a "90% Park, 10% Cat" label. However, semantically, the cat is the most salient feature. This creates "noisy" training signals that confuse the model, especially for tail classes where every sample is critical.
Furthermore, mixing two distinct classes can occasionally result in a sample that semantically resembles a third, novel category. Traditional CutMix is incapable of recognizing this emergence, as it only considers the two parent classes. To address these issues, we introduce Contrastive CutMix (ConCutMix), a framework that rectifies labels using a learned semantic feature space.
Methodology: ConCutMix
ConCutMix moves beyond pixel-counting by leveraging semantic consistency. The method operates in three distinct phases:
1. Learning a Semantic Space
We utilize supervised contrastive learning (specifically the Balanced Contrastive Learning or BCL framework) to map images into a high-dimensional feature space. In this space, images of the same class are pulled together while different classes are pushed apart. This allows the model to establish "prototypes" or centers for each class that represent their core semantic features.
2. Semantic Label Construction
When a new synthetic image is generated via CutMix, it is projected into this learned semantic space. We then calculate the similarity between the augmented sample and all existing class prototypes. Rather than limiting the label to the two source images, we identify the TopK most similar classes. This allows the model to capture instances where the augmentation has created a sample that semantically aligns with a third category.
3. Label Rectification
The final training label is a weighted combination of the traditional area-based label and the new semantically-derived label. We employ a confidence function, $\gamma$, to determine the trust level of the semantic label. Because head classes have more data and more stable prototypes, the system is designed to trust semantic labels more heavily when head classes are involved, ensuring that the rectification is grounded in reliable features.
Experimental Evaluation
We evaluated ConCutMix across several rigorous long-tailed recognition benchmarks:
- ImageNet-LT: A subset of ImageNet designed for long-tailed evaluation.
- iNaturalist 2018: A naturally imbalanced dataset featuring over 8,000 species.
- CIFAR-10/100-LT: Artificially imbalanced versions of the CIFAR datasets.
- Places-LT: A long-tailed version of the Places scene recognition dataset.
Key Results
ConCutMix demonstrated consistent improvements over state-of-the-art baselines:
- ImageNet-LT: Using a ResNeXt-50 backbone, ConCutMix achieved a 3.0% overall accuracy increase over the BCL baseline. Notably, accuracy on rare tail classes improved by 3.3%.
- CIFAR-100-LT: The method showed a 1.76% improvement under high imbalance factors.
- iNaturalist 2018: We observed a 1.0% gain, a significant margin for this large-scale, complex dataset.
Discussion and Insights
Beyond raw accuracy, ConCutMix offers several qualitative advantages. Using Class Activation Mapping (CAM), we observed that models trained with ConCutMix focus more accurately on foreground objects. Standard CutMix often causes models to attend to irrelevant background patches (like grass or sky) because the area-based labels do not distinguish between the object and its surroundings.
Additionally, ConCutMix significantly improves model calibration. In long-tailed learning, models are often overconfident in their predictions for head classes. By providing more accurate, semantically-grounded labels during training, ConCutMix ensures that the model's confidence scores more closely reflect its actual predictive accuracy.
Conclusion
ConCutMix addresses a critical oversight in standard data augmentation: the disconnect between pixel area and semantic meaning. By rectifying labels through a contrastive semantic space, we provide clearer training signals that significantly benefit tail-class recognition. Future work will explore extending this semantic rectification to other augmentation techniques like Mixup and applying the framework to different modalities beyond image classification.
Summary updated