Introduction
As Large Language Models (LLMs) become increasingly valuable intellectual property, the risk of unauthorized redistribution and "model laundering"—where a stolen model is fine-tuned and rebranded—has escalated. Traditional fingerprinting methods, which embed specific input-output behaviors to identify a model's origin, often struggle with a "trilemma" of trade-offs: performance degradation, the need for "white-box" weight access for verification, and high computational costs.
Researchers have introduced UTF (Under-trained Tokens as Fingerprints) to address these challenges. UTF leverages "under-trained" tokens—rarely used characters or placeholders—to embed a robust, efficient, and black-box verifiable "secret handshake" into LLMs.
Methodology
The core innovation of UTF lies in utilizing the "blank spaces" within a model's vocabulary.
1. Identifying Under-trained Tokens
Every LLM contains tokens it has barely encountered during pre-training, such as obscure Unicode characters or unused placeholders (e.g., <unused_token123>). These are identified by analyzing the model's unembedding matrix ($U$). In well-trained models, unused tokens typically have logits converging toward negative infinity. By calculating the cosine distance between token embeddings and the mean "unused" embedding, researchers can isolate the top 2% of tokens that the model has effectively ignored.
2. Fingerprint Embedding via SFT
Once identified, these tokens are used to create a unique fingerprint pair $(x, y)$:
- Input ($x$): A sequence of 11–15 randomly selected under-trained tokens.
- Output ($y$): A sequence of 5 under-trained tokens.
The model undergoes Supervised Fine-Tuning (SFT) on this single pair for 30 epochs. Because the model has no established representations for these tokens, it learns the association rapidly without interfering with its existing knowledge base—a strategy akin to writing a secret code in the margins of a notebook.
3. Black-Box Verification
Verification requires only API (black-box) access. By feeding the rare token string ($x$) to a suspicious model, ownership is confirmed if the model returns the specific output string ($y$).
Evaluation and Results
The UTF method was evaluated across several popular open-source models, including Llama2-7B-Chat, Vicuna7B-v1.5, LLM360 Amber-7B, and Gemma-7B-Instruct. Performance was measured across five dimensions: Effectiveness, Reliability, Efficiency, Persistence, and Harmlessness.
Key Findings
- Effectiveness and Reliability: UTF achieved a 100% Fingerprint Success Rate (FSR) across all models. Furthermore, the chance of a user accidentally triggering the fingerprint (false positive) is nearly zero, as the trigger consists of tokens that almost never appear in natural language.
- Extreme Efficiency: The model learns the fingerprint mapping significantly faster than previous methods. Embedding takes only 6 to 26 minutes, representing a 76% reduction in compute time compared to prior state-of-the-art techniques.
- Persistence: The fingerprint proved remarkably "sticky." Even after extensive fine-tuning on new datasets like GSM8k (math) or ShareGPT (dialogue), the models retained the fingerprint 100% of the time.
- Harmlessness: Testing on benchmarks like LAMBADA and SciQ confirmed that embedding the UTF fingerprint does not degrade the model's general language modeling or reasoning capabilities.
Conclusion
UTF offers a practical solution to the LLM fingerprinting trilemma. By repurposing under-trained tokens, it provides a method that is highly efficient to implement, robust against further fine-tuning, and verifiable through simple black-box queries. Future work may explore the scalability of this approach to larger models (e.g., 70B+ parameters) and its resilience against more aggressive adversarial model modifications.
Summary updated