Abstract
In this work, we consider the problem of imbalanced data in a regression framework when the imbalanced phenomenon concerns continuous or discrete covariates. Such a situation can lead to biases in the estimates. In this case, we propose a data augmentation algorithm that combines a weighted resampling (WR) and a data augmentation (DA) procedure. In a first step, the DA procedure permits exploring a wider support than the initial one. In a second step, the WR method drives the exogenous distribution to a target one. We discuss the choice of the DA procedure through a numerical study that illustrates the advantages of this approach. Finally, an actuarial application is studied.
Introduction
In predictive modeling, a common assumption is that the training dataset is a representative sample of the population to which the model will be applied. However, real-world scenarios often involve selection bias, leading to imbalanced covariates. This occurs when the distribution of predictors ($X$) in the training set differs significantly from the target distribution ($F_0$). For instance, in insurance modeling, high-mileage drivers might be rare in the training data despite being a critical group for risk assessment.
When specific ranges of a covariate are sparse or entirely missing, standard regression models struggle to accurately learn the relationship between the features and the outcome ($Y$). This leads to high Root Mean Square Error (RMSE) and biased predictions in underrepresented regions. While weighted resampling (WR) is a standard technique to adjust for distribution shifts, it is ineffective when there are "holes" in the feature space—regions with zero observations cannot be reweighted.
The DA-WR Framework
To address the limitations of simple reweighting, we introduce a two-stage algorithm: DA-WR (Data Augmentation - Weighted Resampling).
1. Data Augmentation (DA)
The first stage focuses on expanding the "support" of the data. By generating synthetic ($X, Y$) pairs in sparse regions, we provide the model with the necessary "raw material" to learn local relationships. Several generative techniques were evaluated, including perturbation methods like Gaussian Noise and ROSE (Smoothed Bootstrap), interpolation methods like SMOTE, and deep learning approaches such as Conditional Generative Adversarial Networks (GANs).
A key innovation involves applying these DA methods within clusters identified by a Gaussian Mixture Model (GMM). This ensures that synthetic points respect the local covariance and structure of the data, preventing the creation of "impossible" observations that could mislead the learner.
2. Weighted Resampling (WR)
After expanding the support, the second stage aligns the augmented dataset with the target population. Drawing weights are calculated based on the ratio between the target density and the empirical density of the augmented set. This step ensures that the final training sample matches the desired statistical properties of the population of interest.
Results and Discussion
The DA-WR approach demonstrated substantial performance gains across various regression models, including Generalized Additive Models (GAM), Random Forests (RF), and Multivariate Adaptive Regression Splines (MARS).
As shown in the results table, incorporating GMM-based clustering into the DA process significantly improved outcomes. For example, in synthetic simulations, GMM-based perturbation reduced RMSE by up to 19% compared to the imbalanced baseline. In the actuarial application, the improvements were even more pronounced, with RMSE reductions ranging from 40% to 77% depending on the model used.
Interestingly, simpler perturbation methods (like ROSE or Gaussian Noise) combined with GMM often outperformed complex generators like GANs. This suggests that for regression tasks, capturing the precise local conditional relationship between $X$ and $Y$ is more critical than the complexity of the generative model. Furthermore, the method significantly narrowed confidence intervals in the distribution tails, indicating increased model certainty for rare observations.
Conclusion
The DA-WR framework provides a robust solution for regression tasks plagued by covariate imbalance. By combining synthetic data generation with targeted resampling, it allows models to generalize effectively to underrepresented regions of the feature space. Future research will focus on scaling this approach to high-dimensional spaces and handling mixed categorical-continuous imbalanced covariates.
Summary updated