EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint

Zhenhua Xu, Meng Han, Wenpeng XingEMNLP 2025

TL;DR

This paper introduces EverTracer, a novel framework for verifying ownership of Large Language Models (LLMs) without requiring white-box access or introducing detectable anomalies. By repurposing Membership Inference Attacks (MIAs) and leveraging probabilistic variations, EverTracer offers a stealthy and robust solution against sophisticated adversaries aiming to steal or misuse LLMs.

What Problem Does This Paper Address?

The rapid proliferation of LLMs has led to increased concerns about model theft and license violations. Verifying ownership of these complex models is crucial, but existing methods face significant challenges. White-box access, which requires full knowledge of the model's architecture and parameters, is often impractical in real-world scenarios where models are accessed via APIs. On the other hand, methods that introduce detectable statistical anomalies or rely on specific trigger-response patterns can be easily bypassed by adversaries. This creates a need for a verification method that is both stealthy (undetectable) and robust (resilient to modifications and attacks).

Methods

EverTracer introduces a novel "gray-box" fingerprinting framework that operates without requiring white-box access. Its core innovation lies in repurposing Membership Inference Attacks (MIAs) for defensive purposes, focusing on detecting latent memorization rather than relying on artificial trigger-output overfitting. The framework has two main phases:

  • Fingerprint Injection: The model owner fine-tunes the LLM on a natural language dataset (Dtr) to embed ownership signals. Crucially, this is done without introducing detectable artifacts. Simultaneously, a reference model is trained on a similar dataset (Dref) to capture baseline probability distributions. Both models are fine-tuned using Low-Rank Adaptation (LoRA) for efficiency.
  • Probabilistic Variation-Based Verification: This phase assesses whether a suspect model retains latent memorization of the injected fingerprint data. It quantifies how strongly a fingerprint sample deviates from its semantically perturbed variants in the model's likelihood space. This "probabilistic variation signal" acts as a proxy for memorization depth. A calibrated signal is computed by comparing the suspect model's output probabilities against those of the reference model. This approach is designed to be robust against adaptive adversaries, including input-level modifications and model-level transformations.

Datasets

The paper uses the following datasets for training and evaluation:

  • Fingerprint Datasets (Dtr): AG News (AG) and XSum. These are standard benchmarks chosen to demonstrate general applicability.
  • Reference Dataset (Dref): Sampled from the same distribution as Dtr.
  • Unseen Dataset (Dunseen): Sampled from the same distribution as Dtr for verification.
  • Evaluation Tasks: A diverse set of 16 benchmark tasks are used for harmlessness evaluation, including ANLI R1-3, ARC, Open-BookQA, Winogrande, LogiQA, SciQ, BoolQ, CB, RTE, WiC, WSC, CoPA, and MultiRC.

Key Insights and Ideas

  • MIAs for Defense: The paper demonstrates that MIAs, typically used for adversarial attacks, can be effectively repurposed for defensive fingerprinting by focusing on latent memorization.
  • Stealthy Memorization: EverTracer achieves stealth by embedding ownership signals through memorization of natural language data, avoiding artificial triggers that could be detected.
  • Probabilistic Variation as a Signal: The core idea is that memorized data points create localized "peaks" in a model's likelihood landscape. Measuring the probabilistic variation around these points provides a robust signal of ownership.
  • Reference Model for Calibration: Using a reference model trained on similar data is crucial for mitigating false positives and ensuring reliable calibration of the probabilistic variation signal.
  • Robustness Beyond Surface Artifacts: By relying on deeper distributional behavior (memorization) rather than surface-level artifacts (triggers), EverTracer achieves greater resilience against adversarial attacks.

Key Results

  • High Effectiveness: EverTracer achieves consistently high Fingerprint Success Rate (FSR) values, often exceeding 90%, comparable to or better than existing methods, even without explicit overfitting.
  • Strong Robustness: The framework demonstrates significant resilience against various adversarial attacks, including input perturbations (e.g., Remove-Perturbation) and model-level attacks like pruning and model merging. For instance, it shows superior resilience against model merging compared to other methods.
  • Stealthiness: The method avoids detectable statistical anomalies, making it difficult for adversaries to identify the fingerprint.
  • Efficiency: Fingerprint injection using LoRA can be completed in under 30 minutes on consumer-grade GPUs. Verification is also efficient, with PV signals computed in seconds.
  • Harmlessness: The impact on model performance is generally minimal, with some variations depending on the chosen fingerprint dataset. Ablation studies suggest that controlling dataset size and training epochs can optimize harmlessness.

Limitations and Future Work

  • LoRA Implementation: The current implementation uses LoRA for efficiency. Further research could explore alternative fine-tuning strategies like full-parameter tuning or other PEFT variants.
  • Gray-Box vs. Black-Box: EverTracer currently operates in a gray-box setting, requiring access to token-level log-probabilities. Extending the method to stricter black-box scenarios is a future direction.
  • Limited Adaptive Attacks: While robust against common attacks, the paper acknowledges that explicit adaptive attacks specifically designed to erase memorized content are not yet systematically studied.
  • Fingerprint Transferability: Investigating the transferability of EverTracer-injected fingerprints across models with common pretraining origins is a promising avenue for future work.
  • Memory Erasure Techniques: The resilience of EverTracer against advanced memory erasure techniques (e.g., MEraser) needs further investigation.

Summary updated

BibTeX
@inproceedings{xu2025evertracer,
  author = {Zhenhua Xu and Meng Han and Wenpeng Xing},
  title = {{EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure