In the standard framework of Differential Privacy (DP), we typically assume the worst-case scenario: an adversary sees every single update, gradient, and intermediate checkpoint produced during training. Under this "white-box" lens, the privacy budget ($\epsilon$) depletes linearly with every iteration. As Jiayuan Ye and Reza Shokri note in Differentially Private Learning Needs Hidden State (Or Much Faster Convergence), this leads to overly pessimistic guarantees that often force a steep "privacy tax" on model utility.
However, a growing body of research suggests that if we only release the final model weights—treating the intermediate training steps as hidden states—the privacy loss does not grow indefinitely. Instead, it can saturate or even improve over time.
The Intuition of Forgetting
The core mechanism behind this phenomenon is Privacy Amplification by Iteration (PABI). To understand why it works, imagine a sequence of noisy updates. In standard DP-SGD, we account for the information leaked by the gradient at step $t$. But if an observer only sees the model at step $t+100$, the specific influence of the data used at step $t$ has been "washed out" by 100 subsequent rounds of added noise and mathematical transformations.
As explored by Ding Chen and Chen Liu in Differentially Private Neural Network Training under Hidden State Assumption, the hidden state assumption allows the privacy loss from early training steps to decay exponentially. The iterative process acts as a natural privacy-preserving mechanism where the "entropy" of the training trajectory masks individual data points.
Convergence to a Steady State
The most striking result of this research direction is that privacy loss can reach a convergent state. In Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness, Eli Chien and Pan Li prove that for certain noisy optimization processes, the total privacy cost $\epsilon$ hits a ceiling.
This happens because of the contractive nature of many optimization operators. If an update rule brings different starting points closer together (a property of many convex and even some non-convex functions), it effectively "forgets" the initial state. Zhou Li applies this to Graph Neural Networks in Convergent Privacy Framework with Contractive GNN Layers for Multi-hop Aggregations, showing that deep models can capture long-range interactions without the privacy budget exploding, provided the layers are contractive.
The Last Iterate Advantage
Empirical evidence supports this theoretical "forgetting." In The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD, Thomas Steinke and Milad Nasr demonstrate a massive gap between the theoretical upper bounds of standard DP and the actual empirical leakage of the final model.
By using "canaries" (randomly inserted data points) to audit privacy, Galen Andrew and Peter Kairouz found in One-shot Empirical Privacy Estimation for Federated Learning that the final model state is significantly more private than the sequence of updates. This suggests that practitioners who only release a final artifact are providing much stronger protection than their formal $\epsilon$ values might suggest.
Challenges to the Hidden State Model
While promising, the hidden state assumption is not a silver bullet. Its validity depends entirely on the secrecy of intermediate iterates. Wang-Chien Chen and Wei-Ning Chen emphasize in Privacy Amplification in Differentially Private Zeroth-Order Optimization with Hidden States that if a researcher releases checkpoints or even averages weights across epochs, the mathematical guarantees for privacy saturation may vanish.
Furthermore, the "safety" of the hidden state can be circumvented by sophisticated adversaries. In Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model, Aurélien Bellet and Nicolas Papernot introduce "gradient-crafting" attacks. They show that if a specific data point is included in every training step, its signal can persist into the final model weights, potentially reaching the same worst-case leakage as the standard white-box model.
Expanding the Scope: Correlated Data and Diffusion
The principles of PABI are also being extended to complex data types. Clément Pierquin and Aurélien Bellet have adapted these ideas to Pufferfish Privacy, which handles correlated data like social networks. In Rényi Pufferfish Privacy: General Additive Noise Mechanisms and Privacy Amplification by Iteration via Shift Reduction Lemmas, they show that as iterations progress, the correlation between initial sensitive attributes and the current model state decays, allowing for long-term training on sensitive, interconnected datasets.
Similarly, in the realm of generative AI, Rongzhe Wei and Eli Chien demonstrate in Differentially Private Graph Diffusion with Applications in Personalized PageRanks that injecting noise directly into iterative diffusion steps allows the system's inherent convergence properties to "absorb" privacy loss, leading to a much better utility-privacy tradeoff than simply noising the final output.