When a hospital releases synthetic patient records under a formal privacy guarantee, a model trained on those records usually predicts worse on real patients than a model trained on the real data. Everyone attributes this gap to the privacy noise. We think a large and unmeasured share of it comes from something else: how the variables were defined and encoded before any noise was added. A data release has to fix value ranges, bins, measurement grids and code dictionaries in advance, and a private pipeline is not allowed to read these off the raw data. This proposal builds an accounting procedure that splits the gap between real and synthetic data into three parts. The first is the cost of switching to definitions that do not depend on the data. The second is generator error. The third is the cost of the privacy noise itself. We then test whether the first part can be recovered at zero privacy cost by harmonizing to published clinical definitions.
The template comes from carbon accounting. Global land-use models and national inventories disagreed by 6.7 GtCO2/yr. That disagreement turned out to be almost entirely a mismatch in what counts as "managed land," not measurement error. Translating one side's definitions into the other's shrank it to 0.3 GtCO2/yr (Harmonising the land-use flux estimates of global models and national inventories for 2000–2020), and a five-method synthesis traced much of the remaining spread to process and definitional choices (Differences and uncertainties in land-use CO2 flux estimates). Our bet is that the private synthetic data literature reports a single headline gap where it should report a decomposed budget. That habit leads people to blame differential privacy for losses that are really definitional.
Core Claim
Differential privacy (DP) is a guarantee that adding or removing any one patient barely changes what the release reveals. It covers everything the pipeline reads from the raw data, including metadata. Synthetic Data - A Privacy Mirage showed that many DP generators quietly extract bounds and category lists from the raw data, which they called the "metadata trap." Once that leak was closed, accuracy on the Texas hospital discharge data fell from 72% on real data to 68% with a Bayesian-network generator, and collapsed at ε = 0.1. That paper motivates our work: the definitional step is real, it is costly, and it is entangled with privacy.
Our claim is that this cost has never been separated from the noise cost. For irregular ICU time series, which depend heavily on normalization bounds, quantization tables and missingness masks, we expect a substantial share of the reported real-to-synthetic gap to be definitional. We also expect that share to be largely recoverable by encoding against public definitions:
- physiological plausibility limits for value ranges,
- device and charting resolution for measurement grids,
- standard code systems for categorical dictionaries.
None of these touch the private data. If this is right, two things follow. Utility–privacy curves in this area understate what DP generators can do. And the right engineering target in many cases is the schema, not the noise mechanism.
Research Program
Let $U(D)$ be the test performance on held-out real data of a fixed downstream model trained on dataset $D$. We use train-on-synthetic, test-on-real (TSTR) AUROC. We construct a chain of four datasets:
- $D_0$: the real training data under data-derived definitions.
- $D_1$: the same real data re-encoded under public definitions. This is the harmonization step.
- $D_2$: a non-private generator's output trained on $D_1$.
- $D_3$: a DP generator's output trained on $D_1$ at budget ε.
The total gap then telescopes:
$$U(D_0)-U(D_3) = \underbrace{[U(D_0)-U(D_1)]}_{\text{definitional}} + \underbrace{[U(D_1)-U(D_2)]}_{\text{generator}} + \underbrace{[U(D_2)-U(D_3)]}_{\text{privacy noise}}.$$
The split depends on the order in which the steps are applied. To handle this, we also build the counterfactual chain: generators trained under data-derived definitions, which a real release could not legally use but which we can measure in the lab. We then report the Shapley-averaged share of each factor over both orderings, together with the interaction term. This mirrors the carbon work, which reports a residual (the Brazil, China and India offsets) rather than declaring the gap closed. We will likewise report residuals per patient subgroup and not just in aggregate.
The harmonization layer is the one artifact we build. It is a public, versioned schema for the MIMIC-IV and eICU variables used in GHOSTS-style tasks. It replaces every data-derived constant in the GHOSTS and EHR-Safe preprocessing:
- min/max bounds,
- quantization levels,
- stochastic-normalization anchors,
- category vocabularies.
Each replacement is a constant fixed from clinical reference sources without looking at the data. On the analysis side, we measure three things beyond TSTR:
- Ranking stability. The Synthetic Ranking Agreement of PATE-GAN, which asks whether model rankings on synthetic data match rankings on real data.
- Calibration. Brier score, following Synthetic data for privacy-preserving clinical risk prediction.
- Outlier privacy. Membership- and attribute-inference success restricted to outlier patients, because data-derived min/max values directly expose those patients, as Stadler et al. found for CTGAN (Privacy Gain < 0.005).
The DP generator is PATE-GAN, adapted to the latent space of an EHR-Safe-style encoder-decoder. We chose it because label-aggregation noise has held up better than gradient noise. The encoder is pretrained only on public-schema-encoded data, and the training run is charged its own share of the budget.
Baselines
The naive reference point is the real-data oracle, $U(D_0)$. The headline gap is measured against it. GHOSTS reports real-data SOFA AUROC of about 0.79, with GHOSTS at about 0.80 and DoppelGANger at about 0.74 on MIMIC-IV. EHR-Safe reports staying within 3% AUC of real data, with gaps as small as 0.009, on MIMIC-III and eICU, at membership-inference advantage near 0.5. Neither number comes with a formal ε, and we found no reported result on Lacuna for a formally DP generator on the GHOSTS-Bench SOFA task. Establishing that number is itself a contribution.
The strongest existing private method is PATE-GAN, run with its usual data-derived preprocessing. This is the configuration Stadler et al. warn about, so it is the inflated-utility reference. The strongest honest competitor is PATE-GAN with metadata estimated under DP: a small slice of ε spent on private quantiles and private category counts, with the remainder spent on training. This competitor has the same total ε, the same generator, the same encoder, the same downstream model and the same hyperparameter search budget as our harmonized variant. For context only, PATE-GAN's reported numbers are AUROC 0.8737 against DPGAN's 0.8578 on Kaggle Credit at ε = 1, and SRA 0.6909 against 0.5273 at ε = 0.01. DPGAN is included as a weak baseline because Walshe et al. documented its mode collapse on clinical distributions.
The key ablation applies harmonization only to bounds, versus also to quantization grids and missingness dictionaries. This tells us which definitional element carries the effect. Harmonization beats the DP-metadata baseline only if, at matched ε in {0.1, 1, 10}, it gives higher TSTR AUROC with confidence intervals from repeated seeds and bootstrap that exclude zero difference, while also giving equal or better outlier attack resistance.
Why not just…?
Why not just spend a little privacy budget estimating the metadata privately? This is the real threat, which is why it is our main baseline. Private quantiles cost budget, and that cost grows with the number of variables. ICU time series have many variables, each with grids and masks, so the slice taken from training may be large. If the DP-metadata baseline matches harmonization, the decomposition still stands. The conclusion would then become "the definitional share is cheap to recover either way," and we would report that.
Why not just run ordinary ablations of the preprocessing? Ablating one step at a time measures that step's effect in one particular order. Definitional choices and DP noise plausibly interact: coarse public bins may make the noise cost smaller. The two-ordering Shapley accounting and the explicit interaction term exist to expose this. The Texas replication described below tests whether the interaction is non-negligible.
Why not just use a better DP generator? A better generator moves the generator and noise terms. It cannot move the definitional term, because that term is measured on real data, $U(D_0)-U(D_1)$, before any generator is involved. If the definitional share turns out to be small, that result itself tells the field to focus on generators. A null result is informative.
Why not just use federated learning instead? As Walshe et al. argue, federated training does not support the exploratory analysis, debugging and feature-selection workflow that synthetic releases enable. That workflow is the use case here.
Proposed Experiments
The first decisive check is a replication on the Texas discharge data. Can our accounting reproduce Stadler et al.'s 72%-to-68% drop, and how much of it does it attribute to the definitional term? The second is the main study on GHOSTS-Bench SOFA prediction on MIMIC-IV. There we report the full three-part decomposition across ε, along with Brier score and SRA. The third tests whether the definitional term shrinks cross-site gaps, mirroring the carbon finding that translation, not remeasurement, closed most of the gap. Synthetic data is trained on MIMIC-IV and tested on eICU, comparing site-specific definitions against a shared public schema.
The fourth reports subgroup residuals and outlier membership-inference advantage. The concern comes from small strata: Beyond Statistical Fidelity shows cohorts as small as n = 7, where public definitions may clip valid rare values in the same way ontology gaps penalize rare phenotypes.
The hypothesis fails if the definitional share is under roughly a tenth of the total gap at every ε, or if harmonization loses to DP metadata at matched budget. In either case the honest result is that the privacy-noise story was right.
References
- Harmonising the land-use flux estimates of global models and national inventories for 2000–2020
- Differences and uncertainties in land-use CO2 flux estimates
- Synthetic Data - Anonymisation Groundhog Day
- GHOSTS: Generation of synthetic hospital time series for clinical machine learning research
- EHR-Safe: generating high-fidelity and privacy-preserving synthetic electronic health records
- PATE-GAN: Generating Synthetic Data with Differential Privacy Guarantees
- Synthetic data for privacy-preserving clinical risk prediction
- Beyond Statistical Fidelity: Causal-Valid Synthetic EHR Generation for Low-Resource Clinical AI