Reconstructing Ground Truth Labels from Aggregate Metrics

23 concepts9 papers1 proposal

When evaluating machine learning models on a hidden test set, we typically assume that aggregate performance metrics—such as a single average log-loss score or an Area Under the ROC Curve (AUC) value—are safe to share. Common intuition suggests that compressing thousands of individual model predictions into a single scalar summary statistic inherently destroys enough information to protect the underlying ground-truth labels.

However, a growing body of research turns this assumption on its head. Rather than acting as secure summaries, aggregate evaluation metrics frequently function as highly sensitive cryptographic side-channels. By treating an evaluation server as an information oracle, an adversary can submit carefully crafted prediction vectors to reverse-engineer and reconstruct the exact ground-truth labels of a private dataset without ever seeing the data features or training a model.

The Core Mechanism: Exploiting Decomposable Losses

To understand why aggregate scores leak so much information, we must look at the mathematical structure of standard loss functions. Most common metrics, such as Mean Squared Error (MSE) or cross-entropy, are linearly decomposable. This means the total reported loss is simply the average or sum of the losses incurred at each individual data point.

If an attacker can submit arbitrary prediction vectors to an evaluation oracle, they can exploit this linearity to encode the label of each data point into a specific magnitude or frequency within the aggregate sum. In Reconstructing Test Labels from Noisy Loss Functions, [Abhinav Aggarwal] et al. formalize this vulnerability through a property called codomain separability.

Consider a binary classification test set of size $n$ with unknown labels $y \in \{0, 1\}^n$. If an attacker submits a crafted prediction vector $\hat{y}$, the server returns an aggregate loss:

$$L(y, \hat{y}) = \frac{1}{n} \sum_{i=1}^n \ell(y_i, \hat{y}_i)$$

Because the loss function is additive, the attacker can choose the values of $\hat{y}_i$ such that the contribution of each individual label $y_i$ to the final scalar $L$ is uniquely identifiable. For instance, by scaling the predictions exponentially or using number-theoretic tools (such as choosing values based on distinct prime numbers or logarithms), the aggregate scalar score effectively becomes a unique identifier—like a mixed-radix or binary representation—of the entire label vector.

As demonstrated by [Abhinav Aggarwal] et al. in Label Inference Attacks from Log-loss Scores, because log-loss is additive in the log-domain, a small number of strategic queries allows an adversary to decode the entire ground-truth label configuration with perfect precision. This attack bypasses the model entirely; it requires no knowledge of the model parameters or data features, relying solely on the mathematical properties of the loss function itself.

Attacker Probes ──( Crafted Predictions ŷ )──> Evaluation Server (Oracle)
                                                        │
Attacker Decodes <──( Scalar Loss Score L )─────────────┘
(Using number-theoretic reconstruction)

Combinatorial Constraints from Rank-Based Metrics

The vulnerability is not limited to linearly decomposable losses. Non-linear, rank-based metrics like the Area Under the ROC Curve (AUC) are also highly susceptible, though through a different mathematical mechanism: combinatorial restriction.

AUC measures how well a model ranks positive examples above negative ones. In How Does Knowledge of the AUC Constrain the Set of Possible Ground-Truth Labelings?, Jacob Whitehill demonstrates that an exact, high-precision AUC score acts as a rigid mathematical constraint on the permutation space of the ground-truth vector.

When a participant submits a prediction vector to a leaderboard, the resulting AUC score drastically narrows down the search space of valid labelings. By treating the leaderboard as an oracle, an attacker can use Bayesian inference or combinatorial search to determine exactly which label configurations are mathematically compatible with the reported score. As shown in earlier work by Jacob Whitehill in Exploiting an Oracle That Reports AUC Scores in Machine Learning Contests, this structural leakage allows participants to systematically refine their predictions and reverse-engineer hidden test labels without traditional machine learning training.

Real-World Exploitation and the Failure of Naive Defenses

This theoretical vulnerability has severe practical consequences for machine learning infrastructure, such as public competition leaderboards and evaluation APIs. For example, Jacob Whitehill demonstrated in Climbing the Kaggle Leaderboard by Exploiting the Log-Loss Oracle that access to a standard log-loss oracle could be directly exploited to achieve near-perfect scores on a live competition dataset by systematically extracting the test labels.

When designing defenses against these attacks, intuitive or naive obfuscation techniques often fail:

  1. Restricting Outputs to Hard Labels: One might assume that hiding probability distributions and only returning predicted hard labels (e.g., via an argmax operation) prevents leakage. However, in the context of complex tasks like semantic image segmentation, [He Yang] et al. in Segmentations-Leak: Membership Inference Attacks and Defenses in Semantic Image Segmentation showed that the inherent spatial and structural relationships present in predicted label maps still retain enough information for attackers to successfully execute inference attacks.
  2. Adding Noise to Scores: A common defense is to inject noise into the reported metrics. Yet, [Abhinav Aggarwal] et al. proved in Reconstructing Test Labels from Noisy Loss Scores (Extended Abstract) that reconstruction remains highly effective even under noisy conditions, sometimes requiring only a single noisy query augmented by a simple neural network decoder.
  3. Randomizing Benchmarks: To prevent test-set contamination in large language models, protocols like CapBencher ([Takashi Ishida] et al. in CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting) inject controlled randomness into answers to establish a performance ceiling (a Bayes accuracy limit). However, the authors note that the specific realized values of these randomized benchmarks can still leak subtle clues that advanced, high-reasoning models might exploit to reverse-engineer the underlying ground truth.

Achieving True Security via Differential Privacy

If structural restrictions and noise injection are insufficient, how can evaluation platforms be secured? The breakdown of traditional security models stems from a fundamental misunderstanding of what aggregate metrics hide. Security frameworks often focus on limiting the number of queries, assuming a single score leaks negligible information. But as the literature demonstrates, a single un-degraded query against a linearly decomposable loss function can compromise an entire dataset.

To isolate and quantify this leakage, [Ruihan Wu] et al. in Does Label Differential Privacy Prevent Label Inference Attacks? introduce the concept of attack advantage. By subtracting the performance of a Bayes optimal classifier from an attacker's success rate, they isolate the specific information leaked by memorized or exposed labels versus legitimate statistical generalization.

Ultimately, the math dictates that the only robust defense against label reconstruction attacks is the rigorous application of Differential Privacy (DP). As argued by [Abhinav Aggarwal] et al. in Reconstructing Test Labels from Noisy Loss Functions, because standard loss functions are inherently leaky due to their decomposable nature, evaluation systems must apply mathematically calibrated noise—such as via Laplace or Gaussian mechanisms—specifically designed to provide formal guarantees of deniability. Only by bounding the maximum impact that any single ground-truth label can have on the aggregate reported score can we ensure that summary statistics remain truly opaque.

Key Foundational Work

Exploiting an Oracle That Reports AUC Scores in Machine Learning Contests (2016) — contributed 2 concept elements to this direction.
Climbing the Kaggle Leaderboard by Exploiting the Log-Loss Oracle (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_aggregate_metric_leakage_305e2bfb,
  title = {aggregate metric leakage},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/aggregate-metric-leakage/rdr_305e2bfbc8a841b7ab44f5edd95e8006},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_aggregate_metric_leakage_305e2bfb_v1,
  title = {aggregate metric leakage},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/aggregate-metric-leakage/rdr_305e2bfbc8a841b7ab44f5edd95e8006/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake