Synthetic tabular data is increasingly used in place of real records that are locked behind privacy rules. Teams train models on generated rows, test them on generated rows, and report that fairness is preserved. Almost every published check of this kind first chops a continuous sensitive attribute such as age into two groups (say, under and over 40) and compares one number across the two sides. Fairness research has already shown that this chopping hides discrimination even in real data: binarized group gaps systematically understate disparities, and a model can look fair across two bins while treating people just either side of the cut very differently. This proposal asks whether the same hiding happens when we judge synthetic data. Does the generator quietly reshape how outcomes vary with age, income, or a neighborhood's racial composition, while the standard two-group audit reports nothing?
The core idea is to audit a synthetic dataset by comparing whole curves rather than single gaps. For a downstream model, we estimate its average prediction as a smooth function of the continuous sensitive attribute, once on real data and once on synthetic data, and measure how far apart the two curves are. We then test whether binarized audits miss distortions that the curve audit catches. We also test whether those distortions sit where the generators themselves discretize the attribute: at the Gaussian-mixture mode boundaries CTGAN fits to each continuous column, and at the histogram cells that marginal synthesizers such as MST and AIM build. If so, the generators' own preprocessing produces the same binning artifacts that continuous-fairness work warns about. Current synthetic-data fairness benchmarks cannot see these artifacts, because they also bin.
Core Claim
We claim three things, each testable. First, binarized fairness audits of synthetic tabular data underestimate how much synthesis distorts the outcome–attribute relationship along continuous attributes. Second, a large share of the missed distortion is localized, either in narrow bands of the attribute or at generator-internal discretization boundaries. Third, judging fairness on synthetic test data, rather than real test data, hides these localized distortions further. Together these would mean that a common conclusion of current benchmarks is partly an artifact of how they measure fairness. That conclusion is that marginal-based synthesizers preserve fairness well.
Where This Sits in Prior Work
The motivating evidence comes from the continuous-fairness literature. Do Existing Fairness Measures Suffice? shows on real data that binarizing multi-valued attributes yields substantially smaller detected disparities. Fairness-Aware Learning for Continuous Attributes and Treatments shows that bin-level parity can coexist with sharp local discrimination across a threshold. It also shows that linear correlation misses non-linear dependence that its kernel-density estimate of the Hirschfeld–Gebelein–Rényi (HGR) maximal correlation catches. Exceptional in so Many Ways shows that arbitrary boundaries fragment coherent subgroups so that they are never flagged. None of these papers involves a generator. We carry their concern from "the model we trained" to "the data we generated."
Our measurement tools come from methods built to train fair models rather than to audit data. Generalized Demographic Parity for Group Fairness defines generalized demographic parity (GDP): the average distance between the local mean prediction at each attribute value and the global mean. It gives histogram and kernel estimators with convergence rates of $O(N^{-2/3})$ and $O(N^{-4/5})$. A Joint-Distribution Route to Fair Representations uses the Hilbert–Schmidt Independence Criterion (HSIC), a kernel measure of statistical dependence with parametric error decay. Fair Representation Learning … Expectation of Integral Probability Metrics averages a distribution distance over attribute values. We use these as auditing estimators applied to real-versus-synthetic comparisons, not as training penalties.
The constraining and closest work is the synthetic-data fairness literature, and every paper in it bins. Downstream Fairness Caveats with Synthetic Healthcare Data found a "bias shift" under HealthGAN, including a drop of over 34% in Black-subgroup prevalence on MIMIC-3, with binary race and gender only. Evaluating the Fairness Impact of Differentially Private Synthetic Data compared MWEM, DP-CTGAN, PATE-CTGAN, and MST on Adult, ACS Income, and COMPAS, and found MST best, again using binary groups. Assessment of differentially private synthetic data … tabular data introduced the train-on-synthetic/test-on-real (TSTR) versus train-on-synthetic/test-on-synthetic (TSTS) contrast and concluded that marginal-based AIM and MWEM-PGM beat GANs on group fairness, with age either unprotected or binarized. Fair generators (DECAF, Imposing Fairness Constraints in Synthetic Data Generation) intervene only on discrete sensitive nodes.
Our work differs from all of these in what is measured, not in what is generated. We propose no new generator. Instead, we test whether these papers' conclusions survive a continuous audit. The mechanism we target is documented in Modeling Tabular data using Conditional GAN: CTGAN and TVAE encode each continuous value as a one-hot mixture mode plus an offset within that mode. No prior work checks whether those mode boundaries leave marks on downstream fairness.
Research Program
For a downstream classifier $f$ evaluated on a dataset $D$ with continuous sensitive attribute $S$, let $m_D(s) = \mathbb{E}_D[f(X) \mid S = s]$ be its mean prediction at attribute value $s$. We estimate it with a Gaussian kernel in $s$, following the GDP estimator. The curve audit compares a real test set $R$ against a synthetic set $G$:
$$\Delta_{\text{curve}} = \int \big| m_R(s) - m_G(s) \big| \, dP_R(s),$$
where $P_R$ is the real distribution of $S$. We report this together with the change in GDP itself, $\mathrm{GDP}(G) - \mathrm{GDP}(R)$. The binarized audit at threshold $t$ instead compares the two-group gap computed on each dataset, $\Delta_{\text{bin}}(t) = |\text{gap}_R(t) - \text{gap}_G(t)|$.
Every distance is calibrated against a real-versus-real null. We split the real data in two, compute the same statistic between the halves, and bootstrap it. A synthetic distortion counts as detected only if it exceeds the 95th percentile of that null. This puts both audits on the same footing, so "missed" means the binarized statistic stays within real-data noise while the curve statistic does not.
The program has three parts. The first is a broad audit. On the benchmark datasets, we generate synthetic data with each generator and train downstream models in the TSTR regime. We record how often, and by how much, curve-level distortions exceed the null while binarized ones do not. The second is a planted-signal study that gives ground truth. We take real Adult and ACS Income data and inject known, localized changes into the label–age relationship. These include a narrow dip in positive rate for ages 45–52, a non-monotone bump, and a slope change that stays inside one conventional bin. We then measure two things: whether each generator reproduces or erases the planted pattern, and which audits notice.
The third is a mechanism study. We extract CTGAN/TVAE mixture-mode boundaries on the sensitive column, and MST/AIM cell edges, and test whether the curve differences $m_R(s) - m_G(s)$ concentrate near them. The test compares derivative magnitude at boundaries against randomly placed pseudo-boundaries. Finally, we repeat the audits in the TSTS regime and measure how much of the TSTR curve distortion TSTS fails to report.
Baselines
The naive reference point is the real-versus-real null described above. It says how large a curve or gap difference arises from resampling alone, and any claimed distortion must clear it. Without it, a kernel audit could "find" differences that are only estimation noise.
The strongest existing method is the binarized group audit exactly as run in Bullwinkel et al. and the DP tabular assessment. That means demographic parity and equalized-odds gaps on two groups, with the conventional age threshold used in each dataset's fairness literature. It is the strongest competitor because it is what the field actually reports and what supports the "marginal models preserve fairness" conclusion. We run it on the identical synthetic samples, downstream models, and bootstrap null as our audit, so the only difference is the statistic. For reference on the same benchmarks, the file reports binarized results for fair generators on Adult. Imposing Fairness Constraints reports AUROC about 0.827 with demographic parity 0.025. DECAF reports a demographic-parity violation reduced from 0.180 to 0.001 at AUROC 0.672, against 0.807 on original data. For continuous-attribute fairness audits of synthetic tabular data, no reported result was found on Lacuna, so there is no published number to beat. Beating this baseline therefore means two things. On planted-signal data, the curve audit must detect planted distortions that the binarized audit misses at a materially higher rate. On real benchmarks, it must change the ranking of generators on fairness preservation.
Two further baselines are serious and are described under the alternatives below: a fine-binned audit using 10–20 quantile bins, and a threshold sweep that applies the binarized audit at every percentile. We also run Synthcity's general fidelity and detection scores (Synthcity; for orientation, it reports adversarial random forests at data quality 0.5475 and detection 0.6721 on its OpenML suite). This checks whether generic fidelity already predicts curve distortion.
The ablation that isolates the mechanism retrains CTGAN and TVAE with mode-specific normalization replaced by plain min-max scaling on the sensitive column only. It also reruns MST and AIM with the sensitive column discretized at several bin widths. If the boundary-localized distortion vanishes or moves with the boundaries, the generators' internal discretization is the cause.
Why not just…?
Why not just audit with many fine quantile bins? This is the most serious threat. The "over-partitioning hurts" result in General Fair Empirical Risk Minimization concerns training under fairness constraints, not auditing, so it does not rule this alternative out. A fine-binned audit is essentially the histogram GDP estimator. Its slower rate ($O(N^{-2/3})$ versus $O(N^{-4/5})$, from GDP) matters mainly on small data such as Communities & Crime (about 2,000 rows). It also matters when bin edges happen to straddle a planted band. We run it head-to-head. If it recovers the same distortions as the kernel audit on the larger datasets, we will say so: the substantive finding is then "stop collapsing to two groups," and the kernel estimator is a convenience rather than the contribution.
Why not just sweep the binary threshold across all ages? A sweep computes cumulative gaps, so a narrow dip is diluted into neighboring cut points and can fall below the noise level. The planted narrow-band experiment directly measures how much signal the sweep loses relative to the local curve. Because the sweep is cheap and familiar, we report its detection rate rather than assuming it fails.
Why not just trust the generator's overall fidelity scores? Scores such as pairwise correlation fidelity or real-versus-synthetic detection average over all columns. They are also largely linear or global, and the HGR paper shows linear dependence measures can miss non-linear patterns entirely. We test this empirically. Across all generator, dataset, and seed combinations, we compute the rank correlation between Synthcity fidelity and $\Delta_{\text{curve}}$. A strong correlation would make our audit redundant, and we would report that.
Why not just fix the downstream model with post-processing? Post-processing needs a correct target, and the issue here is that the target itself is mismeasured. Bhanot et al. also found that standard mitigations behave unreliably when trained on synthetic data. This alternative is therefore downstream of our question rather than a substitute for it.
Proposed Experiments
The datasets are those in the file that have a continuous sensitive attribute: UCI Adult and ACS Income (age), COMPAS (age), MEPS (age), and Communities & Crime (percentage Black, a regression task where we use the mean predicted rate in place of mean prediction). The generators are CTGAN, TVAE, TabDDPM, and adversarial random forests through Synthcity, plus MST and AIM, each with and without differential privacy at the budgets used in the DP tabular assessment. Downstream models are logistic regression, gradient boosting, and a small multilayer perceptron, all trained with an identical tuning budget on real and synthetic data. We use five generator seeds per configuration and 500 bootstrap draws for the null.
The first decisive check is planted-signal detection. For each planted pattern, we report the fraction of generator configurations in which the pattern is distorted beyond the null, and the fraction of those cases that each audit flags: binarized, threshold sweep, fine bins, kernel curve, and HSIC. The claim fails if binarized audits flag nearly as many as the curve audit.
The second is generator ranking. We check whether ordering generators by $\Delta_{\text{curve}}$ reverses the reported superiority of MST and AIM over GANs on fairness. Because those synthesizers discretize continuous columns into cells, we specifically expect them to flatten within-cell variation in $m_G(s)$. The third is boundary localization with the normalization ablation, as described above.
The fourth is the TSTS gap. This is the fraction of TSTR curve distortion that is invisible when the same audit is run entirely on synthetic test data, since that is the regime practitioners face when real data is inaccessible. Each check has a clear negative outcome, and a negative result would also be informative: it would tell practitioners that current binarized benchmarks are adequate for continuous attributes.
References
- Do Existing Fairness Measures Suffice? Assessing Discrimination in Algorithmic Decision-Making
- Fairness-Aware Learning for Continuous Attributes and Treatments
- Exceptional in so Many Ways - Discovering Descriptors That Display Exceptional Behavior on Contrasting Scenarios
- Generalized Demographic Parity for Group Fairness
- A Joint-Distribution Route to Fair Representations with Continuous Sensitive Attributes
- Fair Representation Learning for Continuous Sensitive Attributes Using Expectation of Integral Probability Metrics
- Downstream Fairness Caveats with Synthetic Healthcare Data
- Evaluating the Fairness Impact of Differentially Private Synthetic Data
- Assessment of differentially private synthetic data for utility and fairness in end-to-end machine learning pipelines for tabular data
- DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks
- Imposing Fairness Constraints in Synthetic Data Generation
- Modeling Tabular data using Conditional GAN
- Synthcity: a benchmark framework for diverse use cases of tabular synthetic data
- General Fair Empirical Risk Minimization