Abstract
The potential of realistic and useful synthetic data is significant. However, current evaluation methods for synthetic tabular data generation predominantly focus on downstream task usefulness, often neglecting the importance of statistical properties. This oversight becomes particularly prominent in low sample scenarios, accompanied by a swift deterioration of these statistical measures. In this paper, we address this issue by conducting an evaluation of three state-of-the-art synthetic tabular data generators based on their marginal distribution, column-pair correlation, joint distribution and downstream task utility performance across high to low sample regimes. The popular CTGAN model shows strong utility, but underperforms in low sample settings in terms of utility. To overcome this limitation, we propose MargCTGAN that adds feature matching of de-correlated marginals, which results in a consistent improvement in downstream utility as well as statistical properties of the synthetic data.
Introduction
Generating high-quality synthetic tabular data is a cornerstone for privacy-preserving data sharing and mitigating class imbalance. However, most state-of-the-art (SOTA) generators, such as CTGAN, are notoriously "data hungry." When the available training set is small—the "low-sample regime"—the statistical properties and utility of the generated data deteriorate rapidly.
Existing evaluation methods often focus solely on "downstream utility" (how well a model performs when trained on synthetic data). However, a robust generator must also maintain fidelity (matching the real data's distribution) and diversity without simply memorizing the training set. This paper introduces MargCTGAN, an enhancement to the CTGAN framework designed to maintain both statistical accuracy and utility when data is limited.
Methodology
MargCTGAN builds upon the CTGAN framework, which utilizes a Conditional GAN architecture with mode-specific normalization to handle heterogeneous tabular data. The primary innovation is the introduction of a moment-matching loss on decorrelated features:
- Decorrelation via PCA: Before calculating statistics, the data is transformed using Principal Component Analysis (PCA). The model retains the original dimensionality but uses PCA to decorrelate the features.
- Moment Matching Loss ($L_{marg}$): The generator is forced to match the first (mean) and second (standard deviation) moments of the real data within this decorrelated PCA space.
- Mechanism of Action: By decorrelating the marginals first, the moment-matching objective becomes more effective. The model can learn the basic "shape" of individual columns without being immediately overwhelmed by complex feature interdependencies.
Experimental Setup
The researchers evaluated MargCTGAN across four diverse benchmark tabular datasets: Adult, Census, News, and Texas. To test the "low-sample regime," models were trained on subsets ranging from as few as 40 samples up to the full dataset size. Performance was compared against CTGAN, TVAE (a Variational Autoencoder approach), and TableGAN.
Key Insights and Results
Performance across Sample Regimes
Standard CTGAN shows a sharp decline in performance once the training set drops below approximately 640 samples. MargCTGAN effectively "smooths" this decline, making it significantly more robust for small-scale applications. In sparse settings, CTGAN exhibited relative errors in machine learning efficacy of up to 57%, which MargCTGAN significantly reduced.
Categorical vs. Numerical Strengths
A key finding was the trade-off between data types. While TVAE excels at numerical data, it often struggles with categorical columns. Conversely, CTGAN is strong with categorical data but weaker with numerical. MargCTGAN balances these strengths, matching CTGAN’s categorical performance while significantly boosting numerical accuracy.
The Fidelity-Privacy Trade-off
The evaluation revealed a "Goldilocks" zone of fidelity. TVAE often achieves the highest utility but risks privacy leakage because its synthetic records are too close to the real ones. TableGAN stays far from the real data (favoring privacy) but suffers in utility. MargCTGAN finds a middle ground, offering better utility than CTGAN without the extreme overfitting risks associated with TVAE.
Limitations and Future Work
The authors identified a mathematical bottleneck termed Rank Deficiency. The PCA-based moment matching relies on the relationship between the number of samples ($N$) and the number of features ($D$). When $N < D$, the PCA transformation becomes less competitive as it cannot accurately capture all feature characteristics. Future work may explore alternative decorrelation methods or adaptive loss weights to address extremely high-dimensional, low-sample datasets.
Summary updated