When training deep learning models on real-world datasets, we routinely encounter long-tailed or highly imbalanced class distributions. Standard supervised learning via cross-entropy loss struggles in these scenarios because it optimizes for absolute decision boundaries, allowing high-frequency "head" classes to dominate the loss and push the boundary at the expense of rare "tail" classes.
To bypass this, researchers frequently turn to contrastive learning. Instead of mapping inputs directly to class labels, contrastive learning constructs a continuous metric space by pulling similar samples together (attraction) and pushing dissimilar samples apart (repulsion).
However, standard contrastive learning frameworks—such as InfoNCE or Supervised Contrastive Learning (SupCon)—suffer from their own geometric distortions when exposed to class imbalance. Understanding why these distortions happen, and how recent architectures fix them, reveals a powerful toolkit for engineering imbalance-resistant representation spaces.
The Gravitational Pull of the Majority
To understand why standard contrastive learning fails on long-tailed data, we must look at how gradients are calculated across a batch. In an instance-based contrastive loss, the representation of a query sample is evaluated against a pool of positive and negative anchors.
When the underlying data distribution is heavily skewed, a batch will inevitably contain an overwhelming number of samples from the majority classes. This creates an asymmetric training signal in two distinct ways:
- Denominator Dominance: In the contrastive loss formulation, the denominator sums the exponential similarities of all negative pairs. If the batch is dominated by head-class instances, the negative push is overwhelmingly driven by these frequent categories. The learning signal for rare tail categories gets drowned out by the sheer volume of majority-class negatives.
- Cluster Crowding and Collapse: Because head classes possess vastly more positive pairs, they form massive, highly compact clusters that occupy a disproportionate volume of the representation space. As shown by Mildenberger et al. in A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasets, this asymmetry causes "representation collapse" for the minority class, where rare categories are either crowded out into poorly separated sub-spaces or completely subsumed by the dominant clusters.
Traditional contrastive methods often assume that representations should be uniformly distributed across a hypersphere. But as Zheng et al. demonstrate in Contrastive Attraction and Contrastive Repulsion for Representation Learning, relaxing this uniformity constraint is essential for real-world, skewed distributions, allowing the embedding space to naturally accommodate clusters of varying densities.
From Instances to Prototypes: Restoring Geometric Symmetry
The most direct way to counteract the numerical dominance of head classes is to shift the contrastive target from individual instances to class prototypes.
If a model contrasts an image against a static or learnable vector representing the entire class rather than every individual sample in the batch, the pair-counting imbalance disappears. In this paradigm, every class—regardless of whether it contains ten samples or ten thousand—occupies exactly one slot as an anchor in the loss function.
Instance-to-Instance (Imbalanced):
Query (Tail) ───► [ Many Head Negatives (Dominant Push) ]
───► [ Few Tail Positives (Weak Pull) ]
Instance-to-Prototype (Balanced):
Query (Tail) ───► [ Prototype Class A (Equal Weight) ]
───► [ Prototype Class B (Equal Weight) ]
In Balanced Contrastive Learning for Long-Tailed Visual Recognition, Zhu et al. implement this by introducing class-averaging and class-prototypes directly into the contrastive objective. This architectural adjustment forces the latent space to organize into a perfectly symmetrical "regular simplex" geometry, ensuring that tail classes are granted equal volume and distinct boundaries.
Similarly, Cui et al. propose Generalized Parametric Contrastive Learning, which integrates learnable class-wise centers into the framework. By pulling samples toward these representative vectors, the optimization process equalizes the learning signal across all categories.
This prototype-driven stabilization is also highly effective in semi-supervised and hybrid settings. For instance, Lin et al. introduce BaCon in BaCon: Boosting Imbalanced Semi-supervised Learning via Balanced Feature-Level Contrastive Learning, utilizing class-wise feature centers derived from a memory bank to serve as stable positive anchors. This prevents the generation of biased pseudo-labels for unlabeled data. In a parallel vein, the single-author work by Wang, Contrastive Learning Based Hybrid Networks for Long-Tailed Image Classification, demonstrates that prototypical contrastive objectives maintain superior generalization over standard cross-entropy when tail classes are over-sampled, avoiding the typical pitfall of overfitting to sparse data.
Asymmetric Supervision and Selective Clustering
Another highly effective strategy involves treating majority and minority classes with completely different optimization objectives. If the majority class already possesses enough data to map out a diverse, robust feature space, forcing its instances into tight, supervised clusters can actually backfire, reducing feature variance and causing representation collapse in the minority classes.
To solve this, Mildenberger et al. introduce a selective supervision strategy. They apply the full supervised contrastive loss exclusively to the minority class, while allowing the majority class to optimize via an unsupervised instance-discrimination objective.
$$\mathcal{L}_{\text{selective}} = \mathcal{L}_{\text{SupCon}}(\text{Minority}) + \mathcal{L}_{\text{Unsupervised}}(\text{Majority})$$
This asymmetry ensures that the majority class maintains its feature diversity and structural richness, while the minority class receives the explicit, targeted clustering signal it needs to remain distinct and identifiable.
Fine-Grained Resolution via Temperature and Focal Scaling
In contrastive learning, the temperature parameter $\tau$ acts as a scale factor that controls how severely the loss penalizes hard negative samples. A lower temperature forces the model to focus intensely on the closest, most confusing negatives, effectively increasing the "resolution" of the local feature space.
Kukleva et al. leverage this mechanism in Temperature Schedules for self-supervised contrastive methods on long-tail data. They demonstrate that introducing low-temperature phases during training prevents rare classes from being swallowed up by frequent clusters. The heightened resolution ensures that samples from sparse categories remain distinct and separable from nearby majority-class clusters.
High Temperature (Low Resolution):
[ Majority Cluster ] ─── (Swallows) ───► [Minority Samples]
Low Temperature (High Resolution):
[ Majority Cluster ] ◄─── (Pushed Apart) ───► [Minority Cluster]
When static temperature adjustments are insufficient, dynamic or "focal" weighting mechanisms can be introduced to prioritize underrepresented samples on the fly. In DyCON: Dynamic Uncertainty-aware Consistency and Contrastive Learning for Semi-supervised Medical Image Segmentation, Assefa et al. implement a dual focal mechanism that explicitly prioritizes hard positive pairs (which appear dissimilar) and hard negative pairs (which appear similar). This dynamic sensitivity adjustment prevents dominant, easily classified majority features from flattening the gradient updates, keeping the optimization focused on the fragile boundaries of minority classes.