Delving into Deep Imbalanced Regression

Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, Dina Katabi

Introduction

Most research on imbalanced data focuses on classification, where labels are discrete categories. However, many real-world tasks involve continuous targets, such as predicting age from a photo or estimating depth in a scene. This paper introduces Deep Imbalanced Regression (DIR), a framework designed to handle continuous target values that follow a skewed, long-tailed distribution.

The Problem: Imbalance in the Continuous Space

In regression, traditional imbalanced learning techniques like re-sampling or re-weighting based on class counts often fail. Unlike classification, continuous labels lack hard boundaries—age 29 is much more similar to age 30 than to age 70. Furthermore, raw counts can be misleading; a model might have few samples for a specific value but can still learn from its "neighborly" information in the continuous space. Finally, continuous ranges often suffer from "missing data" holes where no training samples exist, requiring the model to interpolate or extrapolate.

Deep Imbalanced Regression (DIR) involves learning from skewed continuous distributions, often with missing data regions.

Proposed Methods

The authors propose two novel techniques that leverage the inherent continuity of the label space: Label Distribution Smoothing (LDS) and Feature Distribution Smoothing (FDS).

1. Label Distribution Smoothing (LDS)

In classification, error rates usually correlate with sample counts per class. In regression, however, the authors discovered that the error is "smoother" than raw label counts. LDS addresses this by using Kernel Density Estimation to smooth the empirical label distribution. By convolving label counts with a symmetric kernel (e.g., Gaussian), they calculate an "effective" label density. This density is then used to re-weight the loss function, providing a more accurate signal of which regions are truly difficult to learn.

Label Distribution Smoothing (LDS) uses kernel density estimation to better correlate label density with test error.

2. Feature Distribution Smoothing (FDS)

FDS focuses on internal network representations. The intuition is that if target values are continuous, the learned features should be as well. In low-shot regions, feature statistics (mean and covariance) are often biased toward high-shot regions. FDS calibrates these statistics by smoothing them across nearby target bins during training. This allows the model to "transfer" knowledge from data-rich neighbors to data-poor ones, correcting feature bias.

Datasets and Evaluation

To standardize evaluation, the authors curated five large-scale DIR benchmarks across vision, NLP, and healthcare:

  • IMDB-WIKI-DIR & AgeDB-DIR: Age estimation from faces.
  • STS-B-DIR: Semantic textual similarity scores.
  • NYUD2-DIR: Single-image depth estimation.
  • SHHS-DIR: Health condition scores from physiological signals.

Each dataset features a skewed training set but a balanced test set to ensure fair evaluation across the entire target range.

Key Results and Insights

The proposed LDS and FDS techniques consistently outperformed existing methods across all benchmarks. Key findings include:

  • Significant Gains in Rare Regions: On IMDB-WIKI-DIR, the approach improved Mean Absolute Error (MAE) by 4.14 in few-shot regions compared to a vanilla model.
  • Zero-Shot Generalization: These smoothing techniques allow models to perform well even in regions completely missing from the training data by interpolating from the surrounding range.
  • Fairness: The Error Geometric Mean (GM) improved significantly, indicating more consistent performance across the whole target spectrum.

Conclusion

By recognizing that "continuity is a feature," DIR provides a robust framework for real-world regression tasks. While currently focused on 1D targets, the success of LDS and FDS suggests that accounting for label and feature continuity is essential for training deep networks on imbalanced data.

Summary updated

Abstract

Real-world data often exhibit imbalanced distributions, where certain target values have significantly fewer observations. Existing techniques for dealing with imbalanced data focus on targets with...
BibTeX
@inproceedings{yang2021delving,
  author = {Yuzhe Yang and Kaiwen Zha and Ying-Cong Chen and Hao Wang and Dina Katabi},
  title = {{Delving into Deep Imbalanced Regression}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=8H_Scl4crz-},
}
Figure