Standard machine learning algorithms are inherently greedy: they aim to maximize overall accuracy or minimize global empirical risk. When trained on imbalanced datasets—where one class vastly outnumbers another—this objective function backfires. A model can achieve 99% accuracy simply by predicting the majority class every single time.
Because the loss function is dominated by the frequent patterns of the majority class, the optimization gradient pushes the decision boundary deep into the minority class's territory. This creates a severe majority class bias, rendering models blind to rare but critical events like credit card fraud, medical anomalies, or structural failures.
To solve this, researchers have moved beyond simple data duplication. Modern approaches combat majority class bias by reshaping the geometry of the feature space, adjusting algorithmic penalties, or calibrating the model's learned priors post-hoc.
Reshaping the Geometry of the Feature Space
The most intuitive way to eliminate majority class bias is to balance the dataset before or during training. However, naive oversampling (simply duplicating minority instances) or standard SMOTE (interpolating randomly between neighbors) often introduces noise or blurs the decision boundary.
Not all minority examples are created equal. In Influence of minority class instance types on SMOTE imbalanced data oversampling, Bartosz Krawczyk et al. demonstrate that minority instances fall into distinct structural categories: Safe, Borderline, Rare, or Outliers. Uniformly oversampling all types degrades performance because replicating outliers or highly isolated rare instances forces the model to overfit to noise. By selectively oversampling only specific structural combinations—such as focusing on the borderline regions where the majority and minority classes collide—the authors show that classifiers can map much cleaner, more accurate decision boundaries.
To generate higher-quality boundary examples, Sankha Subhra Mullick et al. introduce an adversarial approach in Generative Adversarial Minority Oversampling. Their framework, GAMO, sets up a three-player game between a generator, a discriminator, and the classifier itself. Instead of randomly interpolating between points, the generator is incentivized to synthesize minority points precisely in the "hard" regions where the classifier is most prone to failure. This forces the model to resolve ambiguity right at the competitive boundary, neutralizing majority dominance where it matters most.
Standard SMOTE: [Minority] ---- (Synthetic Point) ---- [Minority] (Fails to consider boundary)
GAMO (Adversarial): [Majority] <--- (Synthetic Point) ---> [Minority] (Placed precisely at the boundary)
Alternatively, instead of inflating the minority class, one can structurally decompose the majority class. In Modeling Skewed Class Distributions by Reshaping the Concept Space, Kyle D. Feuz et al. propose Intra-Class Clustering (ICC). The majority class is rarely a monolith; it usually contains multiple distinct sub-concepts. ICC breaks the massive majority class down into smaller, highly coherent sub-classes. This balances the relative frequencies of the categories presented to the classifier, preventing a single massive majority gradient from overwhelming the network while preserving all original data.
Algorithmic Compensation and Cost Sensitivity
When altering the data distribution is undesirable, researchers modify the learning process itself so that the model is forced to care equally about rare events.
Cost-sensitive learning scales the loss function so that misclassifying a minority instance incurs a significantly higher penalty than misclassifying a majority instance. Chong Zhang et al. apply this directly to deep architectures in A Cost-Sensitive Deep Belief Network for Imbalanced Classification. By embedding misclassification costs directly into the network's energy function and backpropagation phases, the model maintains strong gradients for rare classes, preventing the weights from drifting into a majority-favoring state.
However, manually setting these cost matrices requires tedious trial and error. To bypass this, Yuxin Peng introduces an automated optimization framework in Adaptive Sampling with Optimal Cost for Class-Imbalance Learning. The proposed method dynamically pairs adaptive data sampling with an automated tuning mechanism that calculates the mathematically optimal cost-sensitivity for the classifier. This dual adjustment ensures that the model's internal attention shifts dynamically as the class proportions change during training.
Post-Hoc Calibration and the Effective Prior
A common shortcut to fix majority bias is to train a model normally on the imbalanced data and then adjust its final prediction probabilities using Bayes' theorem and the raw training class counts. But does a model's internal bias perfectly mirror the raw data counts?
The answer is no. In Prior2Posterior: Model Prior Correction for Long-Tailed Learning, S Divakar Bhat et al. reveal that due to complex optimization dynamics, non-linear activations, and overfitting, a model's effective prior—the actual mathematical bias it internalizes—frequently diverges from the empirical training frequencies.
Raw Training Counts: [Majority: 95%] [Minority: 5%]
Model's Learned Bias: [Majority: 88%] [Minority: 12%] <-- Discrepancy due to optimization dynamics
Correcting outputs based purely on raw data counts leads to sub-optimal decision boundaries. The authors introduce Prior2Posterior (P2P), a post-hoc method that estimates this internalized effective prior directly from the model's outputs. By correcting for the learned bias rather than the observed data bias, P2P achieves precise boundary calibration without requiring expensive model retraining.
Structural and Domain-Level Imbalance
Majority class bias also mutates when moving from flat tabular data to more complex structures like graphs or multi-domain environments.
In graph neural networks (GNNs), minority nodes suffer from a double disadvantage: they are fewer in number, and they lack sufficient topological connections to propagate their features. To bridge this gap, Tianxiang Zhao et al. develop GraphSMOTE: Imbalanced Node Classification on Graphs with Graph Neural Networks. Instead of just interpolating node features in isolation, GraphSMOTE generates synthetic minority nodes within the latent embedding space and simultaneously uses an optimization-driven edge generator to predict where these new nodes should link within the existing network topology. This ensures the synthetic instances are fully integrated into the graph's message-passing structure.
Similarly, in multi-domain learning, models often exhibit a "minority domain bias," where the network prioritizes the statistical distributions of the most heavily represented source domains at the expense of smaller, unique datasets. In Virtual Classification: Modulating Domain-Specific Knowledge for Multidomain Crowd Counting, Mingyue Guo 0001 et al. mitigate this by modulating domain-specific knowledge through virtual classification. This mechanism prevents the shared network weights from being completely dominated by the majority domain's distribution, ensuring equitable performance across diverse data sources.