Data Augmentation for Imbalanced Regression

Stocksieker Sam, Samuel Stocksieker, Denys Pommeret, Arthur Charpentier

TL;DR

This paper introduces a novel data augmentation technique, DA-WR, to address the challenge of imbalanced covariates in regression problems. By combining weighted resampling with various data augmentation methods, DA-WR effectively rebalances the covariate distribution, leading to improved predictive performance in regression models.

What Problem Does This Paper Address?

Many real-world regression problems suffer from imbalanced covariates, where certain ranges of predictor variables are underrepresented in the training data. This imbalance can lead to biased model estimates and poor predictive performance, especially when the underrepresented regions are crucial for accurate predictions. While imbalanced data in classification has been extensively studied, imbalanced covariates in regression, particularly with continuous variables, remain a less explored area. Existing methods often focus on the target variable or categorical covariates, leaving a gap for handling imbalanced continuous predictors.

Methods

The paper proposes a two-step approach:

  1. Weighted Resampling (WR): This method aims to adjust the distribution of covariates to match a target distribution. It calculates weights for each observation based on the ratio of the target density to the empirical density of the covariates. These weights are then used to resample the data, effectively oversampling underrepresented regions.
  2. Data Augmentation - Weighted Resampling (DA-WR): This is the core contribution. It first applies a data augmentation (DA) procedure to generate synthetic data, expanding the support of the covariate distribution. Subsequently, the WR method is applied to this augmented dataset to further refine the covariate distribution towards the target. The paper explores six different DA generators, including perturbation-based methods (Gaussian Noise, ROSE, KDE), interpolation-based methods (SMOTE), latent structure models (GMM, FA), copula models, and deep learning approaches (GAN).

Datasets

The paper uses two main datasets for illustration and application:

  • Synthetic Dataset: A bi-dimensional dataset with a Beta distribution for the covariate X and a Gaussian distribution for Y. This dataset is designed with two inflection points to create a challenging scenario for imbalanced regression.
  • Automobile Insurance Dataset: A real-world dataset of driver telematics containing 100,000 observations. The target variable is the claim frequency, and the covariate of interest for imbalance is the total miles driven. This dataset is used to replicate a real-world sampling bias scenario.

Key Insights and Ideas

  • Imbalanced Covariates Matter: The paper strongly demonstrates that imbalanced covariates, not just imbalanced target variables, can significantly degrade regression model performance, particularly on the tails of the covariate distribution.
  • DA-WR Extends Support: The DA step in DA-WR is crucial for expanding the range of observed covariate values, addressing the limitations of WR which, by itself, might overfit to rare observations.
  • Generator Choice is Key: The effectiveness of DA-WR is highly dependent on the chosen data augmentation generator. Some generators (e.g., GAN, ROSE) performed poorly, while others (e.g., GN-GMM, ROSE-GMM) showed significant improvements.
  • Clustering Can Help: Applying DA generators within clusters (e.g., GN-GMM) often improved results compared to applying them to the entire imbalanced dataset, suggesting that capturing local data structures is beneficial.

Key Results

  • Improved Predictive Performance: Both WR and DA-WR methods generally improved predictive performance (measured by RMSE) compared to using the imbalanced dataset directly, especially for GAM and MARS models.
  • DA-WR Outperforms WR: In many cases, the DA-WR approach yielded better results than WR alone, particularly in extending the covariate distribution and reducing RMSE.
  • Varied Generator Performance: The performance varied significantly across generators. For instance, in the synthetic data illustration, DA-WR with GN-GMM, ROSE-GMM, and KDE-GMM showed notable improvements. In the insurance application, WR, GN-GMM, ROSE-GMM, KDE-GMM, and Smote-GMM were among the best performers.
  • Reduced Confidence Intervals: DA-WR also contributed to reducing the width of confidence intervals in predictions, indicating more stable and reliable estimates.

Limitations and Future Work

  • Generator Sensitivity: The performance of DA-WR is sensitive to the choice of the data augmentation generator. Some generators did not provide the expected results, possibly due to generating X and Y simultaneously.
  • Extrapolation Risk: The paper notes that some generators might still lead to extrapolation of atypical observations, suggesting a need for further treatment for extreme values.
  • Multivariate and Conditional Extension: The authors suggest extending DA-WR to handle multiple imbalanced covariates simultaneously and to apply it within different subpopulations (conditional approach).
  • Mixed Data: Extending the method to mixed data types (categorical and continuous covariates) is also proposed as a future direction.
  • Local Approach: A local approach for data augmentation, focusing on specific regions of the covariate support, could potentially improve results.

Summary updated

Abstract

In this work, we consider the problem of imbalanced data in a regression framework when the imbalanced phenomenon concerns continuous or discrete covariates. Such a situation can lead to biases in the estimates. In this case, we propose a data augmentation algorithm that combines a weighted resampling (WR) and a data augmentation (DA) procedure. In a first step, the DA procedure permits exploring a wider support than the initial one. In a second step, the WR method drives the exogenous distribution to a target one. We discuss the choice of the DA procedure through a numerical study that illustrates the advantages of this approach. Finally, an actuarial application is studied.
BibTeX
@misc{sam2023data,
  author = {STOCKSIEKER Sam and Samuel Stocksieker and Denys Pommeret and Arthur Charpentier},
  title = {{Data Augmentation for Imbalanced Regression}},
  year = {2023},
  eprint = {2302.09288},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.09288},
}
Figure