Improving Contrastive Learning on Imbalanced Data via Open-World Sampling

Ziyu Jiang, Tianlong Chen, Ting Chen, Zhangyang Wang

Introduction

Contrastive learning frameworks like SimCLR and MoCo have revolutionized unsupervised representation learning, but their performance often hinges on the availability of curated, balanced datasets. In real-world scenarios, data is typically long-tailed—where a few "head" classes are over-represented while many "tail" classes have very few examples—and is frequently contaminated with out-of-distribution (OoD) noise. Simply scraping more unlabeled data from the web often fails because random sampling tends to favor head classes and introduces distracting, irrelevant information.

The MAK Framework

To overcome these challenges, researchers developed the Model-Aware K-center (MAK) framework. MAK strategically samples unlabeled data from an "open-world" pool to improve model performance on rare classes while filtering out noise.

The MAK framework overview: re-balancing a long-tailed seed dataset by strategically sampling from an open-world pool based on tailness, diversity, and proximity.

The framework relies on three core selection principles:

  1. Tailness (via ECLE): Since labels are unavailable in an unsupervised setting, MAK identifies potential tail-class samples by looking for "hard" examples with high contrastive loss. To stabilize this noisy metric, the authors introduced Empirical Contrastive Loss Expectation (ECLE), which averages the loss of a single image over multiple random augmentations.
  2. Proximity: To ensure the model isn't distracted by irrelevant OoD data, MAK uses a feature distance regularizer. This ensures that sampled images are semantically related to the "seed" dataset, rejecting outliers.
  3. Diversity: To avoid the trap of sampling redundant, near-identical images, MAK incorporates a K-center greedy algorithm. This ensures the selected subset provides broad coverage of the feature space.

Experimental Setup and Results

The authors tested MAK using a seed dataset of ImageNet-100-LT and sampling pools including ImageNet-900 and a noisy ImageNet-Places-Mix.

Key Findings:

  • Enhanced Tail Performance: MAK improved the accuracy of "Few" (tail) classes by at least 2.6% compared to standard random sampling or K-center methods.
  • Overall Accuracy Gains: When sampling from ImageNet-900, MAK achieved an overall accuracy of 72.7%, a significant jump from the 67.3% seed-only baseline.
  • Robustness to Noise: In the messy ImageNet-Places-Mix environment, MAK successfully rejected irrelevant scene data, proving its effectiveness in "wild" data scenarios.
  • Training Efficiency: MAK outperformed random sampling baselines that were trained for significantly longer (700 additional epochs), demonstrating that data quality is more impactful than sheer training volume.

Conclusion

The MAK framework provides a surgical approach to data augmentation in contrastive learning. By leveraging the model's own internal state to identify under-represented and relevant data, MAK offers a path toward building more robust and fair AI models from the messy, imbalanced data found in the real world.

Summary updated

Abstract

Contrastive learning approaches have achieved great success in learning visual representations with few labels of the target classes. That implies a tantalizing possibility of scaling them up beyond a curated “seed" benchmark, to incorporating more unlabeled images from the internet-scale external sources to enhance its performance. However, in practice, larger amount of unlabeled data will require more computing resources due to the bigger model size and longer training needed. Moreover, open-world unlabeled data usually follows an implicit long-tail class or attribute distribution, many of which also do not belong to the target classes. Blindly leveraging all unlabeled data hence can lead to the data imbalance as well as distraction issues. This motivates us to seek a principled approach to strategically select unlabeled data from an external source, in order to learn generalizable, balanced and diverse representations for relevant classes. In this work, we present an open-world unlabeled data sampling framework called Model-Aware K-center (MAK), which follows three simple principles: (1) tailness, which encourages sampling of examples from tail classes, by sorting the empirical contrastive loss expectation (ECLE) of samples over random data augmentations; (2) proximity, which rejects the out-of-distribution outliers that may distract training; and (3) diversity, which ensures diversity in the set of sampled examples. Empirically, using ImageNet-100-LT (without labels) as the seed dataset and two “noisy” external data sources, we demonstrate that MAK can consistently improve both the overall representation quality and the class balancedness of the learned features, as evaluated via linear classifier evaluation on full-shot and few-shot settings. Thecode is available at: https://github.com/VITA-Group/MAK.
BibTeX
@inproceedings{jiang2021improving,
  author = {Ziyu Jiang and Tianlong Chen and Ting Chen and Zhangyang Wang},
  title = {{Improving Contrastive Learning on Imbalanced Data via Open-World Sampling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure