Mitigating Class Imbalance through Loss and Representation Adaptation

749 concepts681 papers19 proposals

When training deep learning models, we often operate under the ideal assumption that our datasets are uniformly distributed. In reality, real-world data is stubbornly lopsided. Whether classifying rare cardiac responses in electrocardiograms (Ahran Arnold et al. in Discriminating electrocardiographic responses to His-bundle pacing using machine learning) or separating a handful of personal messages from a massive corpus of corporate communications (Sanaz Jabbari in Towards the Orwellian Nightmare: Separation of Business and Personal Emails), majority classes heavily outnumber minority ones.

Standard machine learning algorithms struggle in these environments because they are designed around Empirical Risk Minimization (ERM)—they minimize the average loss across all training samples. If 95% of the dataset belongs to a single majority class, a model can achieve a deceptive 95% accuracy by simply ignoring the minority classes entirely. To build robust models, we must understand why class imbalance paralyzes standard optimization and how we can adapt loss functions and representations to fix it.

The Gradient Imbalance Problem

To understand why class imbalance degrades performance, we have to look at how neural networks learn via gradient descent. Every time a model processes a batch of data, it computes a loss and backpropagates the resulting gradients to update its weights.

When a dataset is highly imbalanced, the mathematical contribution of the majority class dominates this gradient step. The network receives a flood of signals reinforcing how to identify the common class, while the rare class signals are reduced to a trickle of background noise. Consequently, the decision boundary gets pushed deep into the territory of the minority class, shrinking its margin and causing the model to misclassify rare instances during deployment.

Traditional fixes attempt to solve this at the data level. As noted by Naeemul Hassan et al. in Towards Automated Sexual Violence Report Tracking, practitioners frequently turn to resampling techniques: either undersampling the majority class or oversampling the minority class. However, these methods have fundamental limits. Undersampling discards valuable data that could help the model learn robust features, while naive oversampling simply repeats minority samples, leading the model to overfit to the specific nuances of those few examples rather than generalizing the underlying concept.

Dynamic Loss Adaptation

If changing the data distribution introduces unwanted side effects, a more elegant alternative is to change how the model values each error. Instead of treating every misclassification equally, we can modify the objective function to dynamically penalize mistakes on underrepresented classes.

In Fine-Grained Facial Expression Recognition in Multiple Smiles, Zhijia Jin et al. address this by introducing a dynamic weight loss function. The core intuition is simple: as training progresses, the loss function tracks how well the model is performing on different categories. If the model consistently misclassifies a rare, fine-grained expression because it is overwhelmed by a dominant class, the loss weight for that specific minority class automatically scales up.

Mathematically, this amplifies the magnitude of the minority class gradients. Even if a batch contains only a few rare samples, the heightened penalty forces a significant weight update, preventing the majority class from monopolizing the optimization landscape.

Exploiting Interclass Correlations

While loss weighting helps balance the gradients, it does not solve the fundamental information scarcity of minority classes; a rare class still provides very few examples from which to learn complex visual or textual features. However, classes in a dataset are rarely completely independent islands. They often share underlying structural or semantic features.

Kai Ma leverages this insight in Learning and Exploiting Interclass Visual Correlations for Medical Image Classification to combat severe imbalance in medical diagnostics. Standard classification models use "hard" one-hot encoded labels (e.g., representing a disease as a strict [1, 0, 0] vector). This binary framing forces the model to learn each class from scratch, ignoring visual similarities between different pathologies.

Standard Hard Labels:
Rare Disease A  --> [1, 0, 0]  (No relationship to common diseases)
Common Disease B --> [0, 1, 0]

Soft Correlation Labels (CCL-Net):
Rare Disease A  --> [0.7, 0.2, 0.1] (Shares 20% visual features with Common Disease B)

The proposed framework, CCL-Net, abandons hard labels in favor of learning a dictionary of class-specific embeddings that capture visual correlations. If a rare skin lesion shares 20% of its visual characteristics with a highly common melanoma variant, the model generates a "soft label" reflecting that relationship.

By mapping these interclass correlations, the model can transfer representation knowledge from data-rich majority classes to data-poor minority classes. The network no longer has to learn what the rare disease looks like in isolation; instead, it bootstraps its understanding from the well-represented features of the majority class, drastically lowering the sample size required to achieve robust generalization.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dataset_class_imbalance_class_imbalance_d38ffdac,
  title = {dataset class imbalance (class imbalance)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dataset-class-imbalance-class-imbalance/rdr_d38ffdac75d94ccd8ef433d940c90ff4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dataset_class_imbalance_class_imbalance_d38ffdac_v1,
  title = {dataset class imbalance (class imbalance)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dataset-class-imbalance-class-imbalance/rdr_d38ffdac75d94ccd8ef433d940c90ff4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake