Bias-Corrected Data Synthesis for Imbalanced Learning

Pengfei Lyu, Zhengchi Ma, Linjun Zhang, Anru R. Zhang

Abstract

Imbalanced data, where the positive samples represent only a small proportion compared to the negative samples, makes it challenging for classification problems to balance the false positive and false negative rates. A common approach to addressing the challenge involves generating synthetic data for the minority group and then training classification models with both observed and synthetic data. However, since the synthetic data depends on the observed data and fails to replicate the original data distribution accurately, prediction accuracy is reduced when the synthetic data is naïvely treated as the true data. In this paper, we address the bias introduced by synthetic data and provide consistent estimators for this bias by borrowing information from the majority group. We propose a bias correction procedure to mitigate the adverse effects of synthetic data, enhancing prediction accuracy while avoiding overfitting. This procedure is extended to broader scenarios with imbalanced data, such as imbalanced multi-task learning and causal inference. Theoretical properties, including bounds on bias estimation errors and improvements in prediction accuracy, are provided. Simulation results and data analysis on handwritten digit datasets demonstrate the effectiveness of our method.

1. Introduction

In machine learning, imbalanced data is a persistent challenge. Whether detecting credit card fraud, diagnosing rare diseases, or identifying network intrusions, the class of interest is often the one with the least available data. While generating synthetic data (e.g., via SMOTE) is a popular remedy, it introduces a hidden cost: systematic bias. The paper Bias-Corrected Data Synthesis for Imbalanced Learning proposes a framework to identify and remove this bias, ensuring synthetic data fulfills its potential in improving model performance.

2. The Problem of Synthetic Bias

When a majority class vastly outnumbers a minority class, standard algorithms often ignore the minority to maximize overall accuracy. Data augmentation—creating synthetic samples for the minority group—aims to fix this. However, synthetic data is typically a simplified approximation of reality. Treating these samples as "true" data introduces a distributional discrepancy, leading to poor probability calibration and suboptimal decision boundaries. The authors demonstrate that for methods like SMOTE, this bias is structural and does not vanish even as more data is added, scaling with the dimensionality of the feature space.

3. Proposed Method: Bias-Corrected Synthesis

The core innovation of this work is a Bias-Correction (BC) procedure formulated at the loss-function level. Instead of minimizing loss on a simple mixture of real and synthetic data, the authors minimize a "bias-corrected loss" ($L^{bc}$). This is achieved through a "Bias-Transfer" mechanism:

  1. Majority Splitting: The abundant majority-class data is split into groups.
  2. Error Simulation: A synthetic generator is trained specifically for the majority class using one of these groups.
  3. Bias Measurement: The loss difference between synthetic majority samples and real majority samples is calculated to represent the systematic bias ($\Delta_0$) introduced by the generator.
  4. Minority Correction: Assuming the generator's error patterns are consistent across classes, this measured bias is applied as a correction term for the synthetic minority samples.
Comparison of decision boundaries under raw data, standard synthetic augmentation, and the proposed bias-corrected synthesis.

This approach is architecture-agnostic, working effectively with logistic regression, neural networks, and support vector machines. It also extends to Multi-Task Learning (MTL) and Causal Inference, particularly for Average Treatment Effect (ATE) estimation.

4. Experimental Evaluation

The researchers validated the BC method using both synthetic simulations and real-world data:

  • MNIST Dataset: In detecting specific digits as minority classes, the BC method outperformed raw imbalanced training and standard SMOTE across Recall, F1-score, and the Matthews Correlation Coefficient (MCC).
  • Simulated Environments: Testing across various noise distributions (Gaussian, Student-t, Logistic) and non-linear geometric settings, the BC method consistently achieved the lowest parameter estimation errors.
  • Causal Inference: The approach significantly reduced the Root Mean Square Error (RMSE) in ATE estimation compared to standard estimators, proving that correcting propensity score bias is crucial.

5. Conclusion and Future Work

The Bias-Corrected Data Synthesis framework provides a principled way to mitigate the adverse effects of synthetic data. By "borrowing information" from the majority class, it prevents models from being led astray by suboptimal generators. While highly effective, future research could address the complexities of extremely high-dimensional settings where scalar loss might not capture all distributional mismatches, and extend the framework to multi-class imbalanced problems.

Reference: Lyu, P., Ma, Z., Zhang, L., & Zhang, A. R. (2026). Bias-Corrected Data Synthesis for Imbalanced Learning. arXiv:2510.26046v2 [stat.ML].

Summary updated

Abstract

Imbalanced data, where the positive samples represent only a small proportion compared to the negative samples, makes it challenging for classification problems to balance the false positive and false negative rates. A common approach to addressing the challenge involves generating synthetic data for the minority group and then training classification models with both observed and synthetic data. However, since the synthetic data depends on the observed data and fails to replicate the original data distribution accurately, prediction accuracy is reduced when the synthetic data is naïvely treated as the true data. In this paper, we address the bias introduced by synthetic data and provide consistent estimators for this bias by borrowing information from the majority group. We propose a bias correction procedure to mitigate the adverse effects of synthetic data, enhancing prediction accuracy while avoiding overfitting. This procedure is extended to broader scenarios with imbalanced data, such as imbalanced multi-task learning and causal inference. Theoretical properties, including bounds on bias estimation errors and improvements in prediction accuracy, are provided. Simulation results and data analysis on handwritten digit datasets demonstrate the effectiveness of our method.
BibTeX
@misc{lyu2025bias,
  author = {Pengfei Lyu and Zhengchi Ma and Linjun Zhang and Anru R. Zhang},
  title = {{Bias-Corrected Data Synthesis for Imbalanced Learning}},
  year = {2025},
  eprint = {2510.26046},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.26046},
}

Loading related content...

Figure