When a machine-learning platform hides its test labels and returns only a single average score — the log-loss or accuracy of your predictions — that one number is not as safe as it looks. A line of work on aggregate metric leakage has shown that an attacker who can choose what predictions to submit can encode the entire hidden answer key into the score: by assigning each test item a carefully sized contribution, the returned average becomes a arithmetic code for every secret label at once. The obvious defense is differential privacy (DP) — a mathematical guarantee that the released number is almost unchanged whether or not any one person's label is in the data, typically achieved by adding random noise. But DP proofs assume arithmetic over idealized real numbers, and real servers add that noise using IEEE 754 floating-point arithmetic, where numbers have uneven spacing, addition is order-dependent, and rounding follows fixed thresholds. A separate literature on floating-point privacy vulnerabilities has shown that this implementation gap routinely destroys DP guarantees outright.
This proposal argues that the two failures compound on evaluation oracles, and that nobody has checked. The attack literature treats floating-point precision purely as a constraint on the attacker — a limit on how many labels fit in one query — while the floating-point-vulnerability literature studies only raw database sums and training-time noise samplers, never loss aggregation over attacker-influenced terms. My core claim is that on a real evaluation server, the two standard defenses (rounding the reported score, and adding continuous DP noise with standard libraries) both fail for reasons specific to floating-point arithmetic: the noise itself leaks through its bit representation, and the loss summation leaks because the attacker controls half of every summed term. I then propose the matching defense — an evaluation oracle that quantizes each per-example loss to an integer grid, sums exactly in integer arithmetic, and adds discrete Gaussian noise — and prove/measure that it closes both channels at a stated utility cost.
Prior Work and the Gap
Three attacks established that aggregate metrics are label oracles. Label Inference Attacks from Log-loss Scores shows that log-loss is an exponentiated product, so assigning logarithms of distinct primes to each test item makes the score factor back into the labels; on IMDB (25,000 test items) the bounded-precision version of this attack recovered 100% of labels in 53.41 milliseconds using $N/5$ queries. Reconstructing Test Labels from Noisy Loss Functions generalizes this to any Bregman-divergence loss via power-of-two and Sidon-set constructions, shows the attack survives additive noise up to 100% of the raw score, and introduces a "Mutator Network" that makes the adversarial prediction vectors look like benign model outputs. Climbing the Kaggle Leaderboard by Exploiting the Log-Loss Oracle demonstrates the attack end-to-end on a live competition: Whitehill recovered all 512 test labels of the Intel & MobileODT cervical cancer screening contest in fewer than 100 queries, reaching a displayed log-loss of 0.00000 and rank #4 of 848, even though the server rounded scores to five decimal places. All three treat finite precision as an attacker-side bottleneck — the reason you cannot pack 25,000 primes into one 64-bit float — and model any curator noise as an idealized continuous variable.
On the other side, two papers showed that floating-point arithmetic breaks DP in practice, but only in settings where the adversary does not choose the summed quantities. Widespread Underestimation of Sensitivity in Differentially Private Libraries and How to Fix It proves that float summation (iterated, pairwise, or Kahan) in Google DP, IBM diffprivlib, and OpenDP can make the implemented sensitivity — how much the computed sum actually swings when one record changes — exceed the theoretical sensitivity the noise was calibrated to by up to $2^{24}$, yielding 100% membership inference against a system configured at $\epsilon = 0.5$. Are We There Yet? Timing and Floating-Point Attacks on Differential Privacy Systems extends Mironov's representation attack to Gaussian mechanisms in NumPy, PyTorch, and Go: the low-order bits of a noised output are often impossible under one of two neighboring datasets, giving near-certain distinguishing even at tiny $\epsilon$, plus a 92.56% canary-detection rate in DP-SGD on MNIST; the same paper's "Fix Paradox" shows that switching to discrete samplers opens a timing channel that recovers noise magnitudes with 99.65% accuracy. Supporting evidence that these are systemic, not exotic: Sound Randomized Smoothing in Floating-Point Arithmetic shows the same rounding-footprint phenomenon silently invalidates robustness certificates.
The gap is exactly the composition of the two. In a sum query over a curator's raw records, sensitivity blow-up is a worst-case pathology; in a loss oracle, the attacker submits $\hat{y}_i$ and thereby co-authors every summed term $\ell(y_i, \hat{y}_i)$. Nobody has asked whether an attacker can steer the accumulator into rounding thresholds on purpose, or whether float-sampled DP noise on a score can be inverted to restore an exact label oracle. Does Label Differential Privacy Prevent Label Inference Attacks? is the nearest defense-side work, but its threat model is inferring training labels from a model's predictions, under idealized arithmetic; it does not touch score oracles. On the defense side, The Discrete Gaussian for Differential Privacy supplies a float-free noise distribution defined on integers, and Avoiding Floating-Point Side Channels in the Report Noisy Max with Gap Mechanism shows how exact rational arithmetic with dynamic precision refinement closes float leaks in a selection mechanism — but neither has ever been applied to a machine-learning evaluation metric, where the query is a sum of continuous, attacker-influenced loss terms rather than counts.
Research Program
The attack half resolves a standing contradiction: is finite precision a barrier to the attacker (Aggarwal et al.) or an amplifier of leakage (Casacuberta et al.)? The answer, I claim, is both, in the defender's worst order. The server computes, in floating point,
$$\hat{S} \;=\; \mathrm{fl}\Big(\textstyle\sum_i \mathrm{fl}\big(\ell(y_i, \hat{y}_i)\big)\Big),$$
where $\mathrm{fl}(\cdot)$ denotes rounding to the nearest representable float and the "unit in the last place" (the spacing between adjacent floats) grows with the accumulator's magnitude. Two attacker primitives follow. First, staircase probing: because the attacker chooses $\hat{y}$, they can make early per-example losses large enough that the accumulator sits exactly on a Banker's-Rounding threshold, so whether a later term rounds away or jumps the accumulator by a whole unit depends on the secret $y_i$ — a label-dependent deviation that does not exist in the idealized real-arithmetic model and that requires only small, benign-looking prediction values rather than conspicuous primes. Second, noise inversion: if the curator defends by adding Gaussian or Laplace noise sampled with Box-Muller, Polar, or Ziggurat in floats, the set of floats reachable from a candidate loss value forms a candidate-specific lattice; an observed returned value is consistent with only some candidates, so each query acts as a consistency filter that prunes the label hypothesis space even at noise levels that would defeat the Sidon-subset-sum attack in ideal arithmetic. This extends Jin et al.'s two-dataset distinguishing attack to selection among exponentially many labelings, and its pruning power per query is precisely what the proposed experiments will measure rather than assume. Both primitives compound with active sensitivity blow-up: the attacker can inflate the implemented sensitivity of the sum far beyond the theoretical $\Delta = \ell_{\max}/n$ that the library's noise was calibrated to, in the spirit of Casacuberta et al. but now under attacker control.
The defense half builds what I call a grid evaluation oracle. The server quantizes each per-example loss onto a public integer grid of step $g$ and range $[0, Bg]$, sums the integers exactly (no rounding, no order dependence, implemented sensitivity exactly $B$ grid units under a one-label change), and releases the integer sum plus discrete Gaussian noise $\mathcal{N}_{\mathbb{Z}}(0, \sigma^2)$ of Canonne, Kamath, and Steinke:
$$\text{release} \;=\; \sum_i \Big\lfloor \ell(y_i, \hat{y}_i)/g \Big\rceil \;+\; \mathcal{N}_{\mathbb{Z}}(0, \sigma^2).$$
The research deliverables are: a proof that, on the integer grid, the codomain-separation gap between distinct labelings is at most $B$ per differing position, so a discrete-Gaussian scale $\sigma$ chosen from this bound (not from a continuous-DP formula alone) yields an explicit upper bound on per-query distinguishing advantage and, by composition over $q$ queries, an information-theoretic upper bound on label-recovery accuracy; a utility analysis showing randomized quantization makes the released score an unbiased estimate of the true loss with variance from $g$ and $\sigma$, so model ranking is preserved; and a timing audit, following the Fix Paradox finding and the mitigation guidance in Guidelines for Implementing and Auditing Differentially Private Systems, to verify that the integer pipeline and the discrete sampler's variable loop count do not open a timing channel that leaks the noise or the labels. A secondary question, deferred to a stretch goal, is whether the same construction extends to rank metrics, where How Does Knowledge of the AUC Constrain the Set of Possible Ground-Truth Labelings? and Exploiting an Oracle That Reports AUC Scores in Machine Learning Contests show the leakage is combinatorial rather than additive.
Baselines
The naive reference point is the unmodified oracle: a float64 log-loss computed by iterated summation and reported with no noise and no rounding. Its expected outcome is already known from the literature — 100% label recovery — so its role is to confirm our attack harness reproduces the published numbers before we test anything new; specifically, on IMDB ($N = 25{,}000$), the strongest published attack reports 100% accuracy in 53.41 milliseconds using $N/5$ queries (Label Inference Attacks from Log-loss Scores), and that is the number our replication must match.
The strongest practical baseline is decimal truncation of the reported score, exactly as deployed on Kaggle: the server rounds the log-loss to five decimal places. This is the strongest competitor on the "cheap and already fielded" axis, and it has already survived contact with a real attack — Whitehill recovered all 512 labels of the MobileODT contest in fewer than 100 queries against a five-decimal oracle, finishing at displayed log-loss 0.00000 and rank #4 of 848 (Climbing the Kaggle Leaderboard by Exploiting the Log-Loss Oracle). We will rerun it on our MobileODT replica with the same query budget and the same probe-optimization tuning time granted to every attacker, and record queries-to-full-recovery rather than only final accuracy.
The strongest formal baseline is the textbook defense: continuous Gaussian noise added to the float score in NumPy or PyTorch, calibrated to a target $\epsilon$ using the theoretical sensitivity $\Delta = \ell_{\max}/n$. It is the strongest competitor because it is what a privacy-conscious platform would actually deploy today, and because in ideal arithmetic it provably bounds leakage. We run it at matched noise magnitude (same variance as the grid oracle's discrete noise, so utility is equalized), matched query budget, and matched attacker tuning. The published evidence already counts against it — near-100% success against float Gaussian mechanisms in standard libraries (Are We There Yet? Timing and Floating-Point Attacks on Differential Privacy Systems), 100% membership inference at theoretical $\epsilon = 0.5$ via sensitivity blow-up (Widespread Underestimation of Sensitivity in Differentially Private Libraries and How to Fix It), and label recovery surviving injected noise as high as 100% of the raw log-loss score (Label Inference Attacks from Log-loss Scores) — but none of those numbers is for a noisy evaluation oracle, which is exactly the hole our experiments fill. Beating this baseline means: at matched utility, every attack in our suite (prime/radix, Sidon subset-sum, batch probing, staircase probing, float-consistency filtering, mutator-network delivery) recovers labels at or below the accuracy implied by the grid oracle's compositional bound.
Two ablations isolate the defense's ingredients. The first replaces exact integer accumulation with float accumulation but keeps the discrete noise, isolating whether per-example quantization before summation is what kills the blow-up channel; the second keeps exact integer accumulation but omits the noise, isolating whether the discrete noise is what kills codomain separability (it should not be — the integer grid alone still has separation, so this ablation should fail, confirming the noise is necessary). A third ablation applies a snapping-style coarse grid rounding to the final float sum instead of per-example, testing whether the cheaper Mironov-style fix suffices; my prediction is it does not, because the blow-up occurs during accumulation, before any post-hoc rounding.
Why not just…?
Why not just round the reported score to a few decimal places? This is the first thing every platform tries, and the file's evidence says it fails at realistic precisions: Whitehill's Kaggle attack ran against a server already rounding to five decimals and still recovered all 512 labels in fewer than 100 queries, and his simulated accuracy-oracle study reached 93.6% label accuracy at five digits of precision on $n = 2048$. Aggarwal et al. show the attacker simply shrinks the batch size when precision is tight — the IMDB attack used batches of 5. Pushing to two decimals raises the query count, but each query still leaks, and the utility cost for distinguishing close models becomes real. We will run truncation at 2, 3, and 5 decimals as baselines and report queries-to-recovery, so if coarse rounding turns out to be quantitatively adequate at acceptable utility, the experiments will say so plainly.
Why not just add standard DP noise? In ideal arithmetic, yes — that is the whole point of DP. The problem is the implementation, and this is not a hypothetical: Jin et al. demonstrate near-100% attacks against continuous Gaussian mechanisms exactly as implemented in NumPy and PyTorch, and Casacuberta et al. demonstrate 100% membership inference against real libraries at theoretical $\epsilon = 0.5$. The honest caveat is that a correctly implemented continuous mechanism (exact sampling, exact addition) would be sound, and my noise-inversion attack may lose pruning power against such an implementation; but no mainstream library works that way, and experiment E2 measures precisely how much the float implementation gives away relative to the ideal. The snapping variant — clamping and rounding the final sum to a coarse grid before adding discrete noise — is a genuine partial threat, which is why it appears as an ablation rather than being argued away: my claim is that it leaves the accumulation-phase blow-up untouched, and the ablation will test that claim directly.
Why not just evaluate each submission on a random hidden subset of the test set? Subsampling adds uncertainty about which labels contributed, and it is cheap. But Whitehill's subset-inference analysis reached 93.6% label recovery on a simulated $n = 2048$ oracle with hidden subsets, and CapBencher shows that injecting stochasticity into evaluation inflates estimator variance enough that substantially larger test sets are needed to keep model rankings reliable (they report ranking preservation at Kendall's $\tau \approx 0.92$ as the standard their own randomness must meet). Moreover, each per-query score is still a float sum of attacker-influenced terms, so the representation and staircase channels persist under subsampling. We will nevertheless layer subsampling on top of both the continuous-noise baseline and the grid oracle, because if subsampling plus cheap rounding suffices in practice, that is a cheaper deployment than exact integer arithmetic and the experiments should reveal it.
Proposed Experiments and Decisive Checks
Experiment E1 (staircase attack vs. truncation) uses the MobileODT replica ($n = 512$, $c = 3$, five-decimal oracle, per-query budget matched to Whitehill's fewer-than-100 queries) and measures labels recovered, queries used, and stealthiness — the maximum deviation of submitted predictions from a benign baseline, testing whether threshold-jump leakage works with inconspicuous values that prime-based attacks cannot use and that anomaly detection would not flag. Experiment E2 (noise inversion vs. continuous DP) uses IMDB ($N = 25{,}000$, binary log-loss) with Gaussian noise from NumPy and PyTorch at $\epsilon \in \{0.1, 1, 8\}$, comparing the Sidon subset-sum attacker, the float-consistency-filter attacker, and their combination; the decisive readout is recovery accuracy as a function of $\epsilon$ and query count, with the clean-oracle reference of 100% in 53.41 ms at $N/5$ queries from the published attack. Experiment E3 (active sensitivity blow-up) builds a diffprivlib-style float accumulation of per-example losses and measures the implemented-to-theoretical sensitivity ratio under attacker-chosen predictions, against Casacuberta et al.'s passive benchmark of up to $2^{24}$ and their 100% attack at $\epsilon = 0.5$; no reported result found on Lacuna for sensitivity blow-up specifically on loss-function summation, so this experiment is the first measurement. Experiment E4 (defense) runs the grid oracle against the full attack suite on IMDB, MobileODT, MNIST, and CIFAR-10 (the latter two chosen because the clean-oracle attack already reports 100% accuracy there in unnoised settings), with utility measured as Kendall's $\tau$ between model rankings under the noisy grid scores and the true scores, using CapBencher's reported $\tau \approx 0.92$ as the reference point for acceptable ranking distortion, plus wall-clock overhead of exact integer evaluation relative to the float pipeline. Experiment E5 (timing audit) measures whether the integer pipeline and discrete-Gaussian sampler introduce execution-time variation from which an adversary could reconstruct the noise or labels, with Jin et al.'s 99.65% timing recovery of geometric-sampler noise as the known-worst-case comparator and the padding mitigations recommended in the DP implementation guidelines as the fix to evaluate.
The decisive checks are, first, that at matched utility the continuous-noise baseline falls to some attack in the suite while the grid oracle holds every attack at or below its compositional bound — the single number that would establish the thesis — and, second, that the two ablations fail in the predicted directions, confirming that per-example integer quantization and calibrated discrete noise are each necessary and neither alone suffices. If instead the snapping-style final-sum rounding proves equally secure at lower cost, or subsampling plus coarse truncation proves adequate at benchmark scale, the proposal's practical contribution collapses to a negative result about float-implemented DP noise — still worth publishing, but a different paper, and the experimental design above is structured so that either outcome is unambiguous.
References
- Label Inference Attacks from Log-loss Scores
- Reconstructing Test Labels from Noisy Loss Functions
- Climbing the Kaggle Leaderboard by Exploiting the Log-Loss Oracle
- Widespread Underestimation of Sensitivity in Differentially Private Libraries and How to Fix It
- Are We There Yet? Timing and Floating-Point Attacks on Differential Privacy Systems
- Sound Randomized Smoothing in Floating-Point Arithmetic
- Does Label Differential Privacy Prevent Label Inference Attacks?
- The Discrete Gaussian for Differential Privacy
- Avoiding Floating-Point Side Channels in the Report Noisy Max with Gap Mechanism
- Guidelines for Implementing and Auditing Differentially Private Systems
- How Does Knowledge of the AUC Constrain the Set of Possible Ground-Truth Labelings?
- Exploiting an Oracle That Reports AUC Scores in Machine Learning Contests
- CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting