Abstract
Dataset Inference (DI) has emerged as a state-of-the-art "fingerprinting" technique to protect the intellectual property of machine learning models. By analyzing prediction margins, DI aims to identify if a suspect model was trained on a specific private dataset. This paper critically evaluates the robustness of DI, revealing significant vulnerabilities. We demonstrate that DI suffers from high False Positive rates, often failing to distinguish between models trained on the same data distribution versus the same specific samples. Furthermore, we show that adversarial training can effectively act as a cloaking mechanism, inducing False Negatives. Our findings suggest that current DI methods lack the uniqueness and robustness required for rigorous model ownership proof.
Introduction
The high cost of training high-quality machine learning models makes them valuable intellectual property, vulnerable to "model stealing" attacks. Dataset Inference (DI) was proposed to combat this by using "prediction margins"—the distance from data points to a model's decision boundary—as a unique fingerprint. The core assumption is that a model will exhibit distinct margin patterns for the specific samples it was trained on. This paper investigates whether these patterns are truly unique to specific samples or merely indicative of the underlying data distribution.
The Distribution Trap and the Privacy Paradox
A fundamental challenge identified in this work is the "Distribution Trap." Theoretical analysis using a PAC-Bayesian framework suggests that models trained on the same distribution will naturally develop similar decision boundaries, even if their training sets are disjoint. Consequently, DI often confuses distribution-level signatures with sample-level signatures.
This leads to a "Privacy Paradox." To mitigate False Positives and improve the statistical significance of the inference, the victim must provide a large number of private training samples for verification.
As shown in Figure 1, achieving a low probability of a False Positive requires testing a large number of samples ($k$), which necessitates leaking the very data the owner intends to protect.
Empirical Evaluation of Robustness
We evaluated the "Blind Walk" black-box DI method on CIFAR-10 and CIFAR-100 datasets. By using properly disjoint data splits, we tested whether DI could distinguish between truly independent models trained on the same distribution.
High False Positive Rates
Our experiments revealed that DI frequently flags independent models as stolen with extremely high confidence. In experiments on CIFAR-10, DI flagged independent models with $p$-values as low as $10^{-8}$.
In Figure 3, we observe that for both the stolen model ($f_v$) and an independent model trained on the same distribution ($f_I$), the $p$-values remain near zero as the number of samples $m$ increases. This indicates that the verifier becomes increasingly "confident" in its false accusation of the independent model.
Adversarial Evasion and False Negatives
We also explored the susceptibility of DI to False Negatives. By employing adversarial training (specifically PGD), an adversary can smooth the model's decision boundaries. This regularization effectively erases the margin-based fingerprint, allowing a stolen model to evade detection. In our CIFAR-100 tests, the margin difference ($\Delta\mu$) dropped from a clear "stolen" signal (1.58) to a level indistinguishable from an independent model (0.11), albeit with a minor trade-off in model accuracy.
Conclusion
While Dataset Inference represents a creative approach to model fingerprinting, our analysis shows it is currently not robust enough for reliable ownership verification. The confusion between sample-specific and distribution-wide features leads to high False Positive rates, while adversarial training provides a simple path for evasion. Future research must focus on identifying features that are uniquely tied to specific training samples and developing methods that do not require the disclosure of large private datasets.
Summary updated