A Note On Interpreting Canary Exposure

Matthew JagielskiarXiv 2023 · DOI · arXiv

Abstract

As large language models (LLMs) are increasingly trained on massive, often sensitive datasets, the need for robust privacy auditing has become paramount. One popular empirical method for measuring data memorization is the "Secret Sharer" framework, which utilizes a metric known as canary exposure. However, interpreting these exposure scores has historically been difficult, lacking a direct connection to formal privacy guarantees. This note provides a mathematical "Rosetta Stone" for canary exposure, linking empirical observations to Membership Inference Attacks (MIA) and Differential Privacy (DP). By shifting the focus to median exposure, the author provides a rigorous method for calculating formal $\epsilon$ lower bounds from experimental results.

Introduction

In the context of machine learning privacy, "memorization" refers to a model's tendency to output verbatim training data, such as credit card numbers or private addresses. To audit this risk, researchers often use the Secret Sharer framework: they insert unique, random data points (canaries) into the training set and later measure how easily the model can distinguish these canaries from unseen data (references) based on their loss values.

The standard metric for this audit is exposure, which is calculated based on the rank of a canary's loss compared to a set of reference points. While exposure is widely reported in the literature, its interpretation remains non-intuitive. For instance, it has been unclear how a specific exposure value relates to the formal guarantees provided by Differential Privacy (DP). This note bridges that gap, providing a framework to translate empirical leakage into formal privacy bounds.

The Exposure Metric and the Case for the Median

The exposure of a canary $c$ is defined relative to $n$ reference points. If a canary has a lower loss than all $n$ references, it is considered highly "exposed." Traditionally, researchers have reported the mean exposure across multiple canaries. However, this note argues that median exposure is a superior metric for two primary reasons:

  1. Intuitive Baselines: In a scenario with no memorization (random guessing), the average exposure is approximately $1.44$ as $n$ grows. In contrast, the median exposure for random guessing is exactly $1.0$, providing a much cleaner baseline for researchers.
  2. Mathematical Robustness: The median is less sensitive to outliers and simplifies the mathematical connection to other privacy frameworks, specifically Membership Inference Attacks.

Reframing Exposure as a Membership Inference Attack

A central insight of this work is that the exposure calculation is essentially a specific type of Membership Inference Attack (MIA). If an auditor can distinguish a canary from a reference point based on its loss, they are performing an attack to determine if that canary was present in the training set.

By treating the median exposure as the point where the "True Positive Rate" of an attack is fixed at 50%, the author establishes a formal link to Differential Privacy. This allows for the derivation of a lower bound for $\epsilon$ (epsilon), the central parameter in DP that quantifies privacy loss.

Formalizing the Leakage: The DP Lower Bound

The most significant contribution of this note is the formula to convert median exposure into a lower bound for $\epsilon$. The author demonstrates that:

$$\epsilon \geq \ln(2) \cdot (\text{Exposure}(c_{\text{Median}}) - 1)$$

This formula allows auditors to provide concrete evidence of privacy violations. For example, if a researcher observes a median exposure of 10, they can state with mathematical certainty that the model does not satisfy $\epsilon$-DP for any $\epsilon$ smaller than approximately $6.23$ (calculated as $\ln(2) \cdot 9$).

Handling Duplicate Canaries

The note also clarifies the implications of "group privacy." If a canary is inserted multiple times (e.g., 10 times) to make it easier for the model to learn, the resulting $\epsilon$ lower bound must be adjusted. In this case, the calculated $\epsilon$ should be divided by the number of insertions to reflect the privacy risk associated with a single instance of that data point.

Practical Considerations and Limitations

While this framework provides a powerful tool for interpretation, the author highlights several practical considerations for researchers:

  • Sampling Error: Because these bounds are estimated from finite samples of canaries and references, they are subject to statistical noise. For high-stakes auditing, the author recommends using statistical confidence intervals (such as Clopper-Pearson intervals) to ensure the reported lower bound is statistically significant.
  • Independence: The derivation assumes that the losses of different canaries and references are independent. While this is a standard heuristic in the field, it may be challenged in complex training environments where data points are highly correlated.
  • Calibration: To obtain the most accurate picture of privacy risk, the author suggests using "calibration" techniques—adjusting scores based on the inherent "difficulty" of a data point. Calibration can significantly increase the measured exposure, leading to tighter and more meaningful privacy lower bounds.

Conclusion

By providing a mathematical link between empirical exposure and formal privacy theory, this note transforms canary exposure from a heuristic observation into a rigorous auditing tool. Researchers can now move beyond reporting abstract scores and begin providing formal lower bounds on the privacy guarantees of their models, facilitating more transparent and comparable privacy research in the age of LLMs.

Summary updated

Abstract

Canary exposure, introduced in Carlini et al. is frequently used to empirically evaluate, or audit, the privacy of machine learning model training. The goal of this note is to provide some intuition on how to interpret canary exposure, including by relating it to membership inference attacks and differential privacy.
BibTeX
@misc{jagielski2023note,
  author = {Matthew Jagielski},
  title = {{A Note On Interpreting Canary Exposure}},
  year = {2023},
  eprint = {2306.00133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.00133},
}
Figure