A Statistical Hypothesis Testing Framework for Data Misappropriation Detection in Large Language Models

Yinpeng Cai, Lexin Li, Linjun ZhangarXiv 2025 · DOI · arXiv

Introduction

The rapid advancement of Large Language Models (LLMs) has transformed the digital landscape, but it has also created a new frontier for legal and ethical disputes regarding data ownership. A primary concern for model developers is "data misappropriation"—the unauthorized use of one model's high-quality outputs to train or fine-tune a competitor's model, often referred to as model distillation. While companies like OpenAI or Anthropic provide API access to their models, their terms of service typically prohibit using those outputs to develop competing LLMs.

Detecting such misappropriation is a significant technical challenge. Unlike traditional software piracy, where code might be copied verbatim, LLMs are probabilistic engines. A "thief" model trained on misappropriated data may generate content that mimics the style, reasoning, and logic of the source model without reproducing identical text strings. This paper introduces a rigorous statistical framework to address this problem, formulating data theft detection as a minimax hypothesis testing problem.

The Statistical Framework

The authors propose a multi-step detection pipeline that relies on the proactive embedding of signals into the "victim" model's output.

1. Watermarking

To prove a model was trained on specific data, that data must contain a unique, verifiable signature. The framework utilizes two primary watermarking techniques:

  • Gumbel-max Watermarking: An unbiased method that embeds signals without altering the original output distribution.
  • Red-green List Watermarking: A biased method that subtly shifts token probabilities based on a secret key, making certain tokens ("green") more likely than others ("red").

2. Hypothesis Formulation

The detection process is framed as a statistical test between two competing hypotheses:

  • Null Hypothesis ($H_0$): The suspect model was developed independently. The tokens generated by the suspect model are independent of the secret keys used in the victim model's watermarking process.
  • Alternative Hypothesis ($H_1$): The suspect model was trained on the victim's watermarked data. Consequently, the suspect model "inherited" the statistical patterns of the watermark.

3. Inheritance Models and Minimax Optimality

A critical contribution of this work is the distinction between Complete Inheritance and Partial Inheritance. In complete inheritance, the suspect model perfectly replicates the watermark rule. However, real-world training is "lossy." The authors focus on partial inheritance, where the suspect model learns only a noisy approximation of the watermark.

Using Large Deviation Theory, the authors derive optimal rejection thresholds. They frame the detection as a minimax optimization problem, aiming to minimize the worst-case error rates—specifically Type I (False Positives) and Type II (False Negatives)—across a broad class of potential token distributions. This ensures the test remains robust even if the exact nature of the data theft is unknown.

Theoretical and Empirical Validation

The Impossibility Result

The paper provides a striking theoretical proof that without watermarking, it is mathematically impossible to distinguish between two models trained on the same underlying data distribution. This reinforces the necessity of watermarking as a foundational requirement for any legal or technical proof of data theft.

Experimental Setup

The framework was validated through both controlled simulations and real-world scenarios:

  • Simulations: Using a vocabulary of 1,000 tokens and sequences up to 3,000 tokens, the researchers verified the asymptotic properties of their statistical tests.
  • Real-World LLM Study: The authors used OPT-1.3B to generate 1,500 watermarked instruction-response pairs. These were then used to fine-tune a GPT-2 model. This setup successfully demonstrated that a smaller model "stealing" data from a larger one could be detected even across different architectures and tokenizers.

Key Results

The proposed optimal score functions consistently outperformed standard baseline methods, such as simple log-likelihood ratios, in both Gumbel-max and Red-green list scenarios. Notably, the framework proved effective even when the detector had to rely on approximate estimates for the suspect model's entropy and the degree of inheritance. This suggests that the method is highly practical for "in the wild" detection where internal parameters of the suspect model are inaccessible.

Limitations and Future Directions

While the framework is robust, it currently operates under the assumption that next-token predictions are Independent and Identically Distributed (i.i.d.). Future research is needed to account for the complex, long-range dependencies inherent in natural language, which may impact the tightness of the statistical error bounds. Additionally, as detection methods improve, "thieves" may develop evasion tactics, such as adversarial training designed to scrub watermarks. Adapting the minimax framework to counter these evolving defenses remains a vital area for future work.

Summary updated

Abstract

Large Language Models (LLMs) are rapidly gaining enormous popularity in recent years. However, the training of LLMs has raised significant privacy and legal concerns, particularly regarding the distillation and inclusion of copyrighted materials in their training data without proper attribution or licensing, an issue that falls under the broader concern of data misappropriation. In this article, we focus on a specific problem of data misappropriation detection, namely, to determine whether a given LLM has incorporated the data generated by another LLM. We propose embedding watermarks into the copyrighted training data and formulating the detection of data misappropriation as a hypothesis testing problem. We develop a general statistical testing framework, construct test statistics, determine optimal rejection thresholds, and explicitly control type I and type II errors. Furthermore, we establish the asymptotic optimality properties of the proposed tests, and demonstrate the empirical effectiveness through intensive numerical experiments.
BibTeX
@misc{cai2025statistical,
  author = {Yinpeng Cai and Lexin Li and Linjun Zhang},
  title = {{A Statistical Hypothesis Testing Framework for Data Misappropriation Detection in Large Language Models}},
  year = {2025},
  eprint = {2501.02441},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.02441},
}
Figure