Auditing Temporal Drift in Private Datasets from Released Sufficient Statistics

Hospitals and companies cannot publish their data, so outsiders cannot check the claim that matters most once a model is deployed: how quickly its performance decays as the population, the hardware, and the guidelines change. Backtesting tools that replay a model's deployment year by year expose this decay, but they need the raw records. We propose that the data holder publish something much smaller instead. For each time period (say, each year), it releases a noise-protected summary of the data. The summary is the set of totals that fully determines a simple probabilistic model: per-class counts, feature sums, and sums of feature products. Anyone can then refit models on any past window, estimate how discrimination and calibration change over time, and choose how much history to retrain on, all without touching a patient record.

The key fact comes from exponential families, the class of distributions (Gaussian, Bernoulli, Poisson, and so on) whose data enter only through fixed-size running totals called sufficient statistics. Those totals add up across time, and they can be privatized once with calibrated noise. After that, any analysis run on the noisy totals costs no further privacy. The question is whether this compressed, noisy view keeps enough temporal signal to reproduce what a full-data backtest would conclude.

Core Claim

A one-time, differentially private release of per-period exponential-family sufficient statistics lets an external auditor reproduce three things a raw-data temporal backtest would show on the same private dataset. The first is the year-by-year trajectory of discrimination (AUROC) and calibration for a fixed family of linear models. The second is the data-driven choice of retraining window. The third is the ranking of retraining policies (all-history, fixed window, adaptive window). We also make a testable prediction: privacy noise adds a variance term that grows with the window, so the best window should lengthen predictably as the privacy budget tightens.

Concretely, the holder releases one privatized summary per period:

$$\tilde S_t = \sum_{i \in \text{period } t} T(x_i, y_i) + \xi_t .$$

Here $T(x,y)$ stacks, for each class $y$, the count, the clipped feature vector $x$, and its outer product $xx^\top$. These are the sufficient statistics of a Gaussian class-conditional model and also of least-squares regression. The noise $\xi_t$ is Gaussian or Laplace, scaled to the clipping bound and a per-period privacy budget $\epsilon$. Summing $\tilde S_t$ over the last $k$ periods gives window estimates whose sampling variance shrinks like $1/(kn)$ for $n$ records per period. The accumulated privacy noise contributes $k\sigma^2$ to a total of size $kn$, so its relative effect shrinks only like $1/\sqrt{k}$. That asymmetry is the source of the predicted window lengthening.

Discrimination can be computed in closed form from the summary. For a linear score $w^\top x$ and Gaussian class-conditionals with means $\mu_1, \mu_0$ and covariances $\Sigma_1, \Sigma_0$, the AUROC is

$$\mathrm{AUROC}(w) = \Phi\!\left(\frac{w^\top(\mu_1-\mu_0)}{\sqrt{w^\top(\Sigma_1+\Sigma_0)w}}\right),$$

where $\Phi$ is the standard normal distribution function. The auditor can therefore evaluate any linear model trained on any window against any later period's summary.

Position Relative to Prior Work

Four papers supply the method. On the Theory and Practice of Privacy-Preserving Bayesian Data Analysis shows that adding Laplace noise to exponential-family sufficient statistics gives private Bayesian inference with asymptotically optimal efficiency. It treats the data as one static batch, and we reuse the mechanism per period to study drift. Model Assessment and Selection under Temporal Distribution Shift chooses a rolling window with the Goldenshluger–Lepski rule, which compares estimates across window sizes and stops before bias dominates. Adaptive Estimation and Learning under Temporal Distribution Shift denoises shifting loss sequences with wavelet soft-thresholding. Both of those operate on scalar losses computed from raw data. We carry their window-selection logic over to a vector of noisy statistics and add the privacy noise explicitly to the variance term.

Four papers motivate the work. Evaluating Model Performance in Medical Datasets Over Time shows that random train–test splits hide deployment decay and that year-by-year backtests are needed. Its tool requires raw longitudinal records, which is exactly what private silos cannot share. Wild-Time documents an average 20% drop from past to future data. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine shows that training for invariance across time does not beat plain training on pooled history. That result pushes us toward reactive monitoring and retraining rather than robust training, which is what our release supports.

Three results constrain the design. First, the systematic review of temporal shift in clinical medicine finds that recalibration fixes calibration but not discrimination, so the audit must estimate both separately. Second, Temporal Generalization: A Reality Check finds that extrapolating neural weights forward fails against simply using the most recent model. For that reason we make no forecasting claim in the main line. Finally, the exponential family limits expressiveness (The Lie-Group Bayesian Learning Rule), so the audit covers linear and Gaussian-generative models, not arbitrary deep networks. EMDOT partly softens this limit: it reports that logistic regression matched or beat gradient-boosted trees and multilayer perceptrons as data aged.

Research Program

The first component is the release mechanism. We will clip features to a known bound, then count each patient at most once per period (or cap their contributions), so that the sensitivity of $T$ is bounded. Per-period budgets $\epsilon$ are composed across periods using standard accounting. Where noisy second-moment matrices are not positive definite, we project them to the nearest positive-definite matrix before use. This is post-processing, so it costs no additional privacy.

The second component is the auditor's toolkit. It fits discriminant-analysis and ridge-regression models on any window, computes AUROC with the closed form above, and computes calibration by numerical integration under the fitted Gaussians. On top of this sits a Lepski-style window selector. The selector compares window-$k$ estimates of the next-period statistics against shorter windows and accepts a longer window only while the differences stay within a threshold. That threshold includes both the sampling variance and the known $k\sigma^2$ noise variance. We will also build a wavelet variant that soft-thresholds each coordinate of the noisy statistic sequence.

The third component is analysis. We will derive the bias–variance trade-off for the window under a drift-rate assumption, extending Han et al.'s scalar analysis. This yields a predicted optimal $k$ as a function of $\epsilon$, which we will compare against empirical optima.

Baselines

The naive reference point is pooled all-history training on raw data, evaluated with a raw backtest. Guo et al. found this configuration hard to beat. The strongest reported degradation figure for it comes from Wild-Time, where MIMIC-IV mortality AUC dropped from 0.90 to 0.72 from training years to future years. Guo et al. separately report a maximum AUROC drop of 0.090 for sepsis. Our reconstruction must reproduce degradation of that magnitude from summaries alone; it is not trying to beat those numbers.

The strongest existing method for choosing a window is Han et al.'s adaptive rolling window applied to raw per-period losses of the same linear models. It is the strongest competitor because it is the direct non-private analogue of what our auditor does. We will run it on identical period splits, features, and model classes, and tune its thresholds on the same early years. No reported result was found on Lacuna for adaptive rolling windows on MIMIC-IV. Han et al. report no numerical errors on the Dubai housing data either, only that wavelet methods outperformed adaptive windows there. We will therefore produce the raw-data numbers ourselves. The wavelet estimator of Adaptive Estimation and Learning is run the same way as a second raw-data reference. The "Recent" policy from Madaan et al. and fixed one- and three-year windows complete the retraining comparison.

Two ablations isolate what the proposal contributes. The first releases only privatized per-period scalar losses of a pre-specified model and feeds them to Han et al.'s selector. This tests whether releasing the full statistic vector buys anything beyond monitoring one fixed model. The second runs our selector with the privacy-noise term removed from its threshold. This tests whether noise-aware selection, rather than just using statistics, is what makes the chosen windows sensible.

Beating the baselines means the following. At moderate $\epsilon$, the window our auditor selects achieves future AUROC within a small margin of the window selected by raw-data adaptive rolling windows. It also ranks retraining policies in the same order as the raw backtest. Finally, it strictly outperforms the scalar-loss ablation whenever the auditor evaluates a model the data holder did not pre-specify.

Why not just…?

Why not just release differentially private synthetic data and run EMDOT on it? This is a genuine threat. Sampling from our fitted Gaussians is itself a restricted form of synthetic data. The difference is that our release has a closed-form noise model, so the auditor knows exactly how much variance privacy added and can account for it when choosing windows. General-purpose generators give no such accounting, and they spend privacy budget during training. We will include a private synthetic-data generator at matched $\epsilon$, backtested with EMDOT, as an additional comparison. If it matches our fidelity, the contribution narrows to noise-aware window selection. No reported result was found on Lacuna for private synthetic data under temporal backtesting.

Why not just have the hospital publish private yearly performance numbers of its deployed model? This is our scalar-loss ablation, and for monitoring one fixed model it may be enough. It cannot answer a new question an auditor wants to ask, such as whether a model trained on 2012–2014 would have held up in 2018, or what the best window would have been. It also gives no way to check a claimed result for a model the auditor proposes. The ablation measures exactly how much this matters.

Why not just run the audit code inside the institution? Federated or enclave-based evaluation avoids the Gaussian restriction. However, each new query requires the institution to cooperate, and the results remain unreproducible by third parties. That is the reproducibility gap documented for private clinical datasets such as the pneumoconiosis study. A published summary can be re-analyzed by anyone indefinitely, which is a different kind of guarantee rather than a better model.

Why not just recalibrate? The systematic review shows that recalibration restores calibration but not discrimination. Our audit reports both quantities precisely so that it can tell the auditor when retraining, not recalibration, is needed.

Proposed Experiments

We will treat MIMIC-IV (2008–2019, with the mortality and sepsis tasks used by Guo et al. and Wild-Time), SEER breast, colon, and lung, OPTN liver, and the CDC and SWPA COVID-19 data as simulated silos. For regression we add the Dubai housing series. For each dataset we run a full raw-data EMDOT backtest as ground truth. We then run the statistics-only audit across a grid of $\epsilon$ values.

The primary metrics are the absolute error of the reconstructed per-year AUROC and calibration error, the rank correlation between retraining policies under the audit and under the raw backtest, and the future-AUROC regret of the selected window relative to the best window chosen in hindsight.

Three decisive checks follow. First, the reconstruction must recover the raw backtest's degradation curve on MIMIC-IV mortality, including a drop of the size Wild-Time reports. It must also recover the COVID result in EMDOT that sliding windows beat all-history training. If the Gaussian closed form badly misestimates AUROC on skewed clinical features, the approach fails and we will report that. Second, the empirically optimal window should lengthen as $\epsilon$ decreases, at roughly the rate the noise-aware bias–variance analysis predicts. Third, as a secondary test of the gap Madaan et al. leave open, we will check whether linear extrapolation of the per-period mean parameters beats the most recent window. Unlike neural weights, these parameters identify the model uniquely, so this is a fair test of whether identifiability is what doomed weight extrapolation. We treat a negative outcome here as informative, not as a failure of the main claim.