SMCL: Saliency Masked Contrastive Learning for Long-tailed Recognition

Sanglee Park, Seung-won Hwang, Jungmin So

Abstract

Long-tailed recognition remains a significant challenge in computer vision, where a few majority classes dominate the training data, causing models to over-rely on background correlations. This paper introduces Saliency Masked Contrastive Learning (SMCL), a framework designed to decouple background features from majority classes. By masking salient foreground objects and using supervised contrastive learning to associate the remaining background context with underrepresented minority classes, SMCL forces the model to learn more robust, class-agnostic background representations. Experimental results across CIFAR-10/100-LT and ImageNet-LT demonstrate that SMCL significantly improves classification accuracy, particularly for few-shot categories.

Introduction

In real-world scenarios, image datasets often follow a long-tailed distribution where a small number of "head" classes contain the vast majority of samples, while a "tail" of many classes contains very few. A primary obstacle in training deep neural networks on such data is the biased background problem.

When a specific background (e.g., grass) frequently co-occurs with a majority class (e.g., birds) but rarely with a minority class (e.g., horses), the model develops a shortcut: it learns to identify the background rather than the object itself. In balanced datasets, the variety of backgrounds across all classes eventually forces the model to ignore these non-discriminative features. However, in long-tailed scenarios, the lack of diversity in minority classes prevents this "unlearning," leading to poor generalization when a minority object appears in an unfamiliar context.

The SMCL Framework

To address this bias, the authors propose Saliency Masked Contrastive Learning (SMCL), a three-step pipeline designed to redistribute background associations across the feature space.

1. Saliency Masking

The process begins by identifying the most "salient" regions of an image—typically the primary foreground object—using a saliency detection mechanism. This region is then masked out, leaving an image that consists almost entirely of background context. This ensures that the model's subsequent learning is focused strictly on the environment rather than the object's unique features.

2. Weighted Sampling

Once the background is isolated, the system must determine which class this "orphaned" background should be associated with. SMCL employs a weighted sampling strategy based on the "effective number" of samples per class. This strategy prioritizes minority (tail) classes, ensuring they receive more background context during training to compensate for their lack of natural data diversity.

3. Contrastive Learning

The core innovation lies in how these masked backgrounds are integrated into the feature space. Rather than using standard Cross-Entropy loss, which can be noisy when labels are reassigned, the authors utilize Supervised Contrastive Learning (SCL). The model is tasked with "pulling" the feature representation of the masked background closer to the feature representations of the targeted minority class in the embedding space. This process teaches the model that background features like sky, grass, or water are shared resources across many categories and should not be used as primary discriminators for majority classes.

Experimental Results

The researchers evaluated SMCL on three standard benchmarks for long-tailed recognition: CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT.

Performance Benchmarks

  • CIFAR-100-LT: At a high imbalance ratio of 100, SMCL (combined with Deferred Re-weighting) achieved 50.1% accuracy. This represents a substantial improvement over the Empirical Risk Minimization (ERM) baseline of 38.3% and standard SCL at 39.4%.
  • ImageNet-LT: On this large-scale dataset, the model showed significant gains in the "Few-shot" category (classes with fewer than 20 samples), reaching 32.8% accuracy. This suggests that the background redistribution effectively helps the model recognize rare objects in diverse settings.

Key Insights

  • Background as a Shared Resource: The success of SMCL stems from treating the background as a feature to be reassigned rather than noise to be discarded.
  • Superiority of Contrastive Loss: Ablation studies indicated that contrastive loss provides a stronger, more stable signal for moving features in the embedding space compared to simple label-mixing techniques like Mixup.
  • Precision in Masking: Saliency-based masking consistently outperformed random or center-based masking, confirming that accurately isolating the foreground is critical to the method's efficacy.

Conclusion and Future Work

SMCL provides a robust framework for mitigating background bias in long-tailed datasets by effectively "transferring" context from majority to minority classes. While the method shows state-of-the-art potential, its performance is currently tied to the accuracy of the saliency detection and involves higher computational overhead due to the contrastive learning queue. Future research may explore the integration of SMCL with Vision Transformers (ViTs) or more advanced data augmentation techniques to further refine the decoupling of foreground and background features.

Summary updated

BibTeX
@inproceedings{park2023smcl,
  author = {Sanglee Park and Seung-won Hwang and Jungmin So},
  title = {{SMCL: Saliency Masked Contrastive Learning for Long-tailed Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10097143},
}
Figure