Cross-Site Label Borrowing for Accuracy Estimation Under Label Latency

When a credit model or a clinical risk model is deployed at many institutions, nobody knows how accurate it is today. The true outcomes, such as a loan default or a readmission, arrive months after the prediction. The data cannot leave each institution, so no one can pool it to check. Existing tools each handle part of this. Federated evaluation platforms measure accuracy without moving data, but only after labels exist. Label-efficient monitors combine the model's own guesses with a trickle of delayed true labels, but they assume those late labels come from the same distribution as today's predictions. When the relationship between inputs and outcomes is changing, that assumption fails, because the labels arriving today describe the world as it was months ago.

Our idea is that in a multi-site deployment, the missing labels for one site's present often already exist at another site. Drift rarely hits every institution at once, and institutions verify outcomes at different speeds. We propose to estimate each site's current accuracy using a correction term borrowed from other sites. At each site, the model's confidence gives a label-free first guess of accuracy. The correction term is how much that guess was off, measured wherever true labels have already arrived. We borrow it from sites whose already-labeled past looks, in label-free statistics, like the target site's unlabeled present. Sites exchange only binned, differentially private summaries of these errors, never raw records. The question is whether this beats the obvious federated version of existing monitors, which sums each site's own stale-label estimate.

Core Claim

Label-efficient monitoring rests on an exchangeability assumption that label latency breaks, and cross-site structure can repair it. Prediction-Powered Risk Monitoring (PPRM) pairs synthetic labels on a large unlabeled batch with a small stream of true labels. The true labels estimate the bias of the synthetic ones, and anytime-valid bounds follow. Its unbiasedness requires the labeled and unlabeled samples to come from the same distribution. Under a verification delay $L$, the labeled samples come from time $t-L$ and the unlabeled ones from time $t$. If $P(Y\mid X)$ moved in between, the correction is stale. Label-free estimators fail in the same situation for a different reason. Average confidence, shown by Confidence-based Estimators for Predictive Performance in Model Monitoring to be unbiased under its conditions, and PAPE both assume that $P(Y\mid X)$ is fixed. When it shifts, they fail silently.

Our claim is conditional. When drift is staggered across sites, as Federated Learning under Distributed Concept Drift documents, or when verification delays differ across sites, a matched cross-site correction reduces accuracy-estimation error during the latency window compared with each site using its own stale correction. The gain should grow with the ratio of latency to drift timescale. It should vanish when drift is simultaneous and latency is uniform. We will test both predictions.

Method

Each site $k$ runs the same deployed model. For a prediction with score $s$, let $c=\mathbb{1}[\text{prediction correct}]$, observed only after the site's delay $L_k$. Let $g(s)$ be the model's confidence. For each resolved time window $u$ and score bin $b$, the site computes the mean residual $\Delta_k(u,b)=\overline{c-g(s)}$ and the bin count. It releases these with Gaussian noise, in the manner of the perturbed sufficient statistics in distributed ROC-GLM. Because residuals are bounded in $[-1,1]$, the sensitivity is small. This avoids the large noise that model-sensitivity calibration forces in that paper.

Each site also releases, immediately and under the same privacy accounting, a label-free signature of its current window. The primary signature is a histogram of prediction scores. Our secondary variant uses coarse summaries of feature attributions, following Explanation Shift, which showed that attribution distributions reveal posterior shifts that raw-feature tests miss.

The server matches site $k$'s current signature against all resolved (site, window) pairs, including $k$'s own past, and forms the estimate

$$\hat A_k(t)=\frac{1}{n_k}\sum_{i}g(s_i)+\sum_{b}\hat\pi_{k,t}(b)\sum_{(j,u)}w_{k,t}(j,u)\,\Delta_j(u,b).$$

Here $n_k$ is the number of current predictions and $\hat\pi_{k,t}(b)$ is the fraction of them in bin $b$. The weights $w_{k,t}(j,u)$ are nonnegative, sum to one, and decrease with signature distance. They are fit by backtesting: we pretend already-resolved windows are unresolved and minimize the resulting error. Uncertainty intervals come from the same backtest residuals, using conformal-style calibration on resolved periods.

We are explicit about the guarantee. This is not PPRM's assumption-free bound. It is empirical coverage that holds if the relation between signatures and residuals is stable over time. We report coverage directly so readers can judge.

The privacy layer is deliberately light. The quantities we need are sums of bounded values, so secure aggregation plus differential privacy suffices. Homomorphic encryption and two-party protocols such as ML Confidential or the HE+2PC scheme in Proof of Federated Learning address a different threat: verifying accuracy on labels that already exist. They cannot supply labels that do not yet exist, which is our bottleneck.

Research Program

The work has three stages. First, we build a latency-and-drift simulator over the benchmarks below. It assigns each site a verification delay, possibly heterogeneous, and a drift onset, possibly staggered. It then replays the stream so that labels become visible only after their delay. This extends the delay-aware evaluation of Simonetto et al., who showed that ignoring labeling and deployment delays overstates how well retraining mitigates drift "by up to 39.86%" on BGL BNP Paribas. We move that evaluation to accuracy estimation across sites.

Second, we implement the estimator and its privacy accounting. The result is a streaming add-on to static federated evaluation platforms such as MedPerf and MEDS-DEV. Both platforms assume curated, fully labeled local test sets.

Third, we map where borrowing helps. We sweep the ratio of latency to drift timescale, the degree of stagger, the type of drift (covariate only versus posterior), the number of sites, and the privacy budget $\epsilon$.

Baselines

The naive reference is the last-resolved accuracy. Each site reports the accuracy of its most recent fully labeled window, aggregated by secure summation. This is what a MedPerf-style periodic re-benchmark would produce. Any method that cannot beat it during the latency window is useless.

The purely label-free reference is federated average confidence. Each site sums its predicted confidences and the sums are securely added. This is additive, needs no source data, and has the theoretical backing of Confidence-based Estimators. We also run a site-local PAPE variant, fitting its domain classifier on developer-held source data against each site's production features. That requires the developer to ship source data or the site to ship features, a privacy cost we report. On Folktables, PAPE reports an accuracy NMAE of 0.97 against 1.04 for importance weighting, and an F1 NMAE of 0.90 against 1.03 for confidence-based estimation. We reproduce that centralized setting first as a sanity check.

The strongest competitor is per-site PPRM with secure summation. Each site forms its own prediction-powered estimate, using its own delayed labels as the correction, and the estimates are summed across sites. This is the natural federated extension. Its only weakness on our question is the staleness we target. Its reported performance is on centralized streams: on CIFAR-10C it alarmed "around time step 760" against 800 for supervised monitoring, and on LLM QA "roughly 30 time steps earlier than supervised baselines" (PPRM). We found no reported result on Lacuna for PPRM, or any other method, estimating accuracy under multi-site label latency, so we establish that number ourselves.

For fairness, every method sees the same streams, the same delays, the same label-free proxy $g$, and the same privacy budget. Tuning uses only resolved historical windows, with an identical search budget per method. Beating per-site PPRM means lower mean absolute error on current-window accuracy during the latency window, pooled over sites, together with interval coverage within two points of the nominal level. This must hold in the staggered or heterogeneous-latency regimes, with no significant loss in the simultaneous-drift regime.

Three ablations isolate the key ingredient. Own-site only restricts borrowing to $k$'s own past and tests whether cross-site information matters. Random matching keeps cross-site borrowing but uses random weights and tests whether the signature does real work. Covariate-reweighted own correction reweights $k$'s stale residuals toward its current score distribution and tests whether simple covariate correction suffices.

Why not just…?

Why not just sum per-site PPRM estimates under secure aggregation? This is the real threat. When latency is short relative to drift, the stale correction is nearly current, and this baseline should match ours at lower complexity. We expect it to fail specifically when $L_k$ exceeds the drift timescale, because the correction then measures the old concept. The latency-to-drift sweep will locate the crossover. If our method never beats it at realistic delays, such as the roughly 38 days of total industrial delay reported for BGL BNP Paribas by Simonetto et al., the proposal fails, and we will say so.

Why not just use a label-free monitor like D3M or average confidence? D3M detects deterioration well: 99% true-positive rate on Camelyon17 at batch size 50, and near-zero false alarms on GEMINI during COVID-19. But it outputs an alarm, not an accuracy number. Confidence and density-ratio estimators assume $P(Y\mid X)$ is fixed. Our posterior-drift conditions, meaning the label-swap drifts of the FedDrift and FLASH suites, directly test whether they fail silently where delayed, borrowed labels do not.

Why not just reweight the stale labeled data to match today's inputs? Importance weighting corrects changes in $P(X)$ but cannot see a change in $P(Y\mid X)$ that has not yet appeared in any label. It is our third ablation. We expect it to match our method under pure covariate drift and to lose under posterior drift. If it matches everywhere, cross-site borrowing adds nothing.

Why not just use cryptographic exact evaluation? Exact encrypted evaluation, as in ML Confidential, still needs labels that do not exist during the latency window, and it costs orders of magnitude more computation. It solves a different problem.

Proposed Experiments

The primary testbed is the FedDrift synthetic suite (SINE, CIRCLE, SEA, MNIST). It already has staggered four-concept drift across clients, and we add per-client delays. The main real-data testbed is FMoW, which spans 16 years of satellite images. Regions are sites and drift is naturally staggered; FedDrift reports 64.8% learned accuracy against 58.8%, a result about model training that we use only to confirm the drift is real. The financial testbed is Lending Club (LCLD), with sites defined by geography and default outcomes providing natural months-long delays. The proprietary BGL data is not accessible to us. Folktables adds states as sites over four years. eICU, a multi-hospital database available through MEDS, adds a clinical sanity check. Its short time span limits temporal drift, so we treat it as secondary.

We report accuracy (or AUC) estimation error during the latency window, interval coverage, and detection delay for crossing a preset accuracy floor with a matched false-alarm rate. We also report the privacy budget $\epsilon$. As a reference point, distributed ROC-GLM stays within 0.01 AUC of the pooled estimate on static data (Schalk et al.), so we check how close our estimator comes to non-private pooled estimation at each $\epsilon$.

Three checks are decisive. First, in the heterogeneous-latency and staggered-drift regimes, our estimator must beat per-site PPRM and the covariate-reweighting ablation on estimation error during the latency window. Second, random matching must erase most of that gain, showing the label-free signature carries real information. Third, in the simultaneous-drift, uniform-latency regime, our method must not be meaningfully worse than per-site PPRM. That would show borrowing does no harm where there is nothing to borrow.