Deep Imbalanced Regression

73 concepts50 papers1 proposal

Imagine training a neural network to predict continuous values where the vast majority of your data points cluster tightly around a specific average, but the most critical predictions lie at the extreme ends of the spectrum. Whether you are forecasting heavy rainfall during a rare storm, estimating the age of a centenarian, or predicting the potency of a rare drug candidate, you face a fundamental challenge: the data distribution is highly skewed.

When deep learning models are trained on such data using standard loss functions like Mean Squared Error (MSE), they perform poorly on rare, tail-end values. This phenomenon is known as imbalanced regression. To build better intuition for why this happens and how to fix it, we must look at how standard models process continuous spaces and why traditional classification-based fixes fall short.

The Tyranny of the Majority in Continuous Spaces

Why does a standard regression model fail when data is non-uniformly distributed? The answer lies in how loss functions like MSE optimize globally.

MSE penalizes the square of the difference between the predicted and true values. In a dataset where 99% of the samples have a target value near $10$ and only 1% have a target value near $100$, a model can achieve an exceptionally low average loss simply by predicting values close to $10$ for everything. Making a small error on thousands of majority samples incurs a massive total loss penalty, whereas making a catastrophic error on a handful of rare samples barely registers in the global average. Consequently, the network biases its weights to satisfy the high-density regions, effectively "swallowing" minority samples.

In machine learning, data imbalance has historically been studied through the lens of discrete classification. In an imbalanced classification problem, you can easily count the instances of "Class A" versus "Class B" and re-weight the loss function or oversample the minority class.

However, as noted by Yuzhe Yang et al., continuous target spaces lack hard boundaries. If you attempt to calculate the frequency of a specific continuous label (such as an exact age of $82.34$ years), its empirical frequency in the dataset is likely to be exactly one or zero. More importantly, nearby continuous values share significant mutual information. A sample with a target of $82$ tells us a great deal about the expected behavior of a sample at $83$. Treating continuous targets as isolated categorical bins discards this vital topology, making traditional classification re-weighting strategies ineffective.

Leveraging Continuity: Label Distribution Smoothing

To address the lack of discrete boundaries, Yuzhe Yang et al. introduced a framework called Label Distribution Smoothing (LDS). Instead of calculating raw empirical frequencies, LDS convolves the expected label distribution with a symmetric kernel (such as a Gaussian kernel).

Empirical Distribution:  [... 0, 1, 0, 0, 0, 0, 1, ... ]  <- Hard to balance
                                 \   /         \   /
Kernel Convolution:              \ /           \ /
Smoothed Distribution:   [... 0.1, 0.4, 0.2, 0.1, 0.3, 0.5, ... ] <- Reflects regional scarcity

This kernel smoothing acknowledges that data scarcity is a regional property rather than an isolated one. By calculating a continuous effective density, the model can scale the loss function inversely to how densely populated a target region is. If a target value sits in a sparse neighborhood, its error is penalized more heavily, forcing the network to pay attention to the tail regions without ignoring the shared information of neighboring points.

Preventing Feature Collapse

The bias of imbalanced regression does not just impact the final prediction layer; it fundamentally alters the internal representations—the features—that the network learns. When majority samples dominate the gradient updates, the network maps vast swaths of the input space into a compressed, biased feature cluster that favors the majority. This is known as feature collapse. Even if you re-weight the final layer, the underlying features lack the resolution to distinguish between different rare values.

To combat this, researchers introduce geometric and structural constraints directly into the feature space:

  • Preserving Metric Topology (RankSim): Yu Gong et al. proposed RankSim, a regularization technique that forces the network's feature space to mirror the organization of the label space. If Sample A is closer to Sample B than to Sample C in their true continuous labels, RankSim ensures that their learned feature representations follow that exact same relative ranking. This prevents rare samples from being dragged into majority feature clusters, preserving the continuous geometry of the data.
  • Contrastive Pushing Forces (ConR): Similarly, Mahsa Keramati et al. developed ConR (Contrastive Regularizer). ConR dynamically identifies samples that the model mispredicts and applies a targeted pushing force in the feature space. This prevents minority representations from being overshadowed or crowded out by dominant majority representations.
  • Uniform Synthetic Anchors (PRIME): Taking a different geometric route, Jongin Lim et al. introduced PRIME. This framework maps continuous targets to a well-ordered, uniform map of learnable synthetic anchors called proxies. By aligning data samples to these uniformly distributed proxies, the model can leverage powerful classification-style balancing techniques without losing the continuous nature of the regression task.

Rethinking the Loss Function

Can we fix imbalanced regression by fundamentally altering how the loss function handles density, rather than just patching MSE with weights?

The authors of Balanced MSE for Imbalanced Visual Regression demonstrated that standard MSE inherently assumes a completely uniform label distribution during optimization. When the training distribution is skewed, this assumption introduces a mathematical bias. By formalizing imbalanced regression from a statistical standpoint, the authors developed Balanced MSE (BMSE). BMSE integrates the training label density directly into the objective function, mathematically enforcing a fair error distribution across the entire continuous spectrum without requiring manual weight tuning.

Interestingly, this structural mismatch between skewed data and continuous functions explains a common deep learning "trick": adding an auxiliary classification loss to a regression network to boost its performance. Silvia-Laura Pintea et al. analyzed why this works, revealing that classification loss acts as a powerful regularizer precisely when training data is imbalanced. It forces the network to partition its feature space cleanly, bridging the gap between imbalanced empirical distributions and the balanced continuous functions we want to learn.

Key Foundational Work

The problem of bias in training data in regression problems in medical decision support (2002) — contributed 2 concept elements to this direction.
Combined regression and ranking (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_imbalanced_regression_0d360616,
  title = {imbalanced regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/imbalanced-regression/rdr_0d36061678a648adadecb96c38e5cad6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_imbalanced_regression_0d360616_v1,
  title = {imbalanced regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/imbalanced-regression/rdr_0d36061678a648adadecb96c38e5cad6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake