Quantifying Information Disclosure During Gradient Descent Using Gradient Uniqueness

Sleem Mahmoud Abdelghafar, Maryam Aliakbarpour, Chris JermainearXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) pose significant privacy risks by potentially leaking sensitive training data. Traditional auditing methods are often computationally prohibitive or limited to specific attack vectors. We propose Gradient Uniqueness (GNQ), a principled, attack-agnostic metric that upper-bounds information disclosure by quantifying the distinctiveness of a data point's gradient. To facilitate real-time auditing, we develop BS-Ghost GNQ, an efficient algorithm that reduces computational complexity from parameter-space to batch-space. Our evaluation on various datasets demonstrates that GNQ accurately identifies high-risk "outlier" data points and predicts vulnerability to extraction attacks with minimal training overhead.

Introduction: The Privacy Challenge in LLMs

As Large Language Models (LLMs) scale, they increasingly risk verbatim disclosure of sensitive training data, ranging from personally identifiable information (PII) to copyrighted material. Auditing these risks is traditionally difficult. Existing approaches typically fall into three categories: attack-based methods (e.g., Membership Inference), which are only as good as the attack itself; canary-based methods, which require modifying training data; and counterfactual analysis, which is mathematically rigorous but computationally infeasible for LLMs as it requires training multiple model variants.

To address these limitations, we introduce Gradient Uniqueness (GNQ), a metric that is attack-agnostic, computationally efficient, and grounded in information theory.

Defining Gradient Uniqueness (GNQ)

The fundamental intuition behind GNQ is that a model discloses information about a data point if that point exerts a unique and significant influence on the model's parameters during optimization. Mathematically, GNQ measures the degree to which a specific data point's gradient is an outlier relative to the rest of the training set. If a point's gradient is highly unique, the model must adjust its parameters specifically to accommodate it, thereby encoding more information about that specific instance.

Figure 1: Relationship between data space outliers and gradient space uniqueness.

As shown in Figure 1, points that are outliers in the data space (like point 7) produce distinct vectors in the gradient space. GNQ quantifies this "uniqueness" by comparing the individual gradient to the distribution of gradients in the training batch.

Efficient Auditing via BS-Ghost GNQ

Computing GNQ naively for LLMs is impossible because it involves inverting a matrix of size $P \times P$, where $P$ is the number of parameters (often in the billions). We overcome this through BS-Ghost GNQ, which utilizes two key technical innovations:

  1. Batch-Space Transformation: By applying the Push-through Identity and Sherman-Morrison formula, we shift the computation from parameter-space to batch-space. This reduces the matrix inversion requirement to the size of the training batch (e.g., $1024 \times 1024$), which is computationally trivial.
  2. Ghost Kernels: We compute the necessary Gram matrix (representing gradient correlations) without explicitly materializing the massive gradient vectors, significantly reducing memory consumption.

These optimizations allow GNQ to be computed during training with only a 12% computational overhead (1.12x training time), making real-time privacy auditing feasible for production-scale models.

Experimental Results and Validation

We validated GNQ across several datasets, including WikiText-2, MNIST, and AG-News. A critical test involved a custom set of 400 sentences, split between "common knowledge" facts and "surprising" falsehoods. GNQ successfully identified "surprising" information as having higher disclosure risk, solving a common problem where models are penalized for "leaking" general knowledge.

Figure 6: GNQ-based auditing accuracy and thresholding in gradient space.

As illustrated in Figure 6, GNQ serves as an effective proxy for counterfactual influence. In Figure 6(a), the removal of a high-GNQ point (point 7) significantly shifts the regression line, demonstrating its high influence. Figure 6(b) shows how thresholds ($\tau$) in the gradient space can be used to audit and flag these high-risk points.

Key findings include:

  • Predictive Power: High GNQ scores during training are strong predictors of future extraction success. For the top 5% of sentences ranked by GNQ, a simple prefix-completion attack achieved 100% extraction success.
  • Heterogeneous Disclosure: Privacy risk is not uniform; it concentrates heavily on a small subset of outlier examples.

Conclusion

Gradient Uniqueness provides a scalable, principled framework for auditing privacy in LLMs. By moving from parameter-space to batch-space with BS-Ghost GNQ, we enable developers to monitor information disclosure in real-time during the training process. Future work will explore the impact of batch size scaling and the extension of GNQ to non-SGD optimization techniques and parameter-efficient fine-tuning methods like LoRA.

Summary updated

Abstract

Disclosing information via the publication of a machine learning model poses significant privacy risks. However, auditing this disclosure across every datapoint during the training of Large Language Models (LLMs) is computationally prohibitive. In this paper, we present Gradient Uniqueness (GNQ), a principled, attack-agnostic metric derived from an information-theoretic upper bound on the amount of information embedded in a model about individual training points via gradient descent. While naively computing GNQ requires forming and inverting an $P \times P$ matrix for every datapoint (for a model with $P$ parameters), we introduce Batch-Space Ghost GNQ (BS-Ghost GNQ). This efficient algorithm performs all computations in a much smaller batch-space and leverages ghost kernels to compute GNQ ``in-run'' with minimal computational overhead. We empirically validate that GNQ successfully accounts for prior/common knowledge. Our evaluation demonstrates that GNQ strongly predicts sequence extractability in targeted attacks and reveals how disclosure risk concentrates heterogeneously on specific examples over the course of LLM training.
BibTeX
@misc{abdelghafar2025quantifying,
  author = {Sleem Mahmoud Abdelghafar and Maryam Aliakbarpour and Chris Jermaine},
  title = {{Quantifying Information Disclosure During Gradient Descent Using Gradient Uniqueness}},
  year = {2025},
  eprint = {2510.10902},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10902},
}
Figure