TL;DR
Open-source large language models (LLMs) pose a fundamental challenge for text watermarking: users can easily bypass decoding-time watermarks, while retraining or distilling models with watermarked data ruins their specialized, fine-tuned capabilities. The authors introduce WAPITI (WAtermark Parameter InTegratIon), a training-free parameter editing technique that isolates a "watermark vector" from a distilled base model and adds it directly into fine-tuned models—preserving downstream task performance while embedding a detectable statistical watermark.
What Problem Does This Paper Address?
Watermarking LLM outputs embeds a hidden, statistically verifiable signal into generated text to track provenance, detect synthetic spam, and prevent misuse. In closed-source models, providers can apply watermarking directly at decoding time (e.g., perturbing logit distributions via green/red token lists).
However, open-source models expose all model weights. Users can customize decoding pipelines and bypass decoding-based watermarks entirely. Prior work attempted weight-based watermarking via distillation—training an LLM to internalize watermark token biases directly into its weights. But applying distillation to fine-tuned models breaks down completely:
- Severe capability loss: Distilling specialized models (e.g., on math or question-answering) triggers catastrophic forgetting, dropping task accuracy to near zero.
- Data scarcity & degradation: Task-specific datasets (like GSM8K) are far too small for effective distillation, and model-generated watermarked answers degrade in quality.
- Vulnerability to fine-tuning attacks: Prior distilled watermarks could be scrubbed in as few as 500 optimization steps on clean data.
Methods
WAPITI eliminates the need to train or distill each fine-tuned model individually:
- Watermark Vector Extraction: First, distill a base model ($\theta_{\text{Base}}$) on watermarked data using a standard base corpus (e.g., OpenWebText) to obtain a watermarked base model ($\theta^\dagger_{\text{Base}}$). The watermark parameter vector is computed simply as: $$\Delta\theta_{\text{Base}} = \theta^\dagger_{\text{Base}} - \theta_{\text{Base}}$$
- Watermark Parameter Integration: For any downstream fine-tuned model ($\theta_{\text{FT}}$) derived from that base architecture, apply the watermark vector directly via parameter arithmetic: $$\theta^\dagger_{\text{FT}} = \theta_{\text{FT}} + \lambda_{\text{FT}} \cdot \Delta\theta_{\text{Base}}$$ where $\lambda_{\text{FT}}$ is a tunable scaling factor (typically set to $1.0$).
The authors provide a first-order Taylor expansion derivation demonstrating that when the Hessian operator behaves similarly across parameter regions and the task parameter change is approximately orthogonal to the watermark vector, the model's next-token output probability distribution shifts by the intended watermark perturbation without altering task functionality.
Datasets
The authors evaluate WAPITI across base and fine-tuned models from the Llama-2-7B and Pythia-1.4B families using:
- OpenWebText: Used to perform the initial distillation of the base models and to simulate fine-tuning attacks.
- C4 (RealNewsLike subset): 5,000 samples used to evaluate general generation quality, perplexity, sequence repetition, and watermark detectability.
- GSM8K: Grade-school math word problems evaluated on the standard 1,319-problem test split.
- MMLU: Massive Multitask Language Understanding benchmark (~14,000 questions across 57 subjects) used to evaluate question-answering performance.
No new benchmarks were created; all evaluations rely on established open-source datasets.
Key Insights and Ideas
- Orthogonal Parameter Subspaces: Cosine similarity analysis shows that fine-tuning task vectors ($\theta_{\text{FT}} - \theta_{\text{Base}}$) and watermark vectors ($\Delta\theta_{\text{Base}}$) have near-zero overlap ($\approx 0.1\% - 0.5\%$). Because the parameter edits live in nearly orthogonal subspaces, watermarks can be added algebraically without trampling fine-tuned skills.
- Watermarks as $n$-gram Frequency Biases: Distillation does not teach the model complex cryptographic hash functions; instead, it memorizes high-frequency watermarked $n$-gram transitions. Both base and fine-tuned models undergo aligned, universal distribution shifts when watermarked.
- Defense Against Fine-Tuning Attacks: Parameter-level integration binds watermarking to model weights. If an attacker fine-tunes the model on unwatermarked data to wash out the watermark, the model's specialized downstream capabilities (e.g., math reasoning) collapse simultaneously, rendering the attack self-defeating.
- Amortized Efficiency: Instead of spending hours distilling every custom checkpoint on multi-GPU clusters, providers only need to run base distillation once. Watermarking downstream variants is instantaneous and training-free.
Key Results
- High Detectability with Zero Degradation: On Llama-2-7B models (QA, Chat, and Math), WAPITI achieves an AUROC of 0.90–0.92 and median detection $p$-values as low as $10^{-7}$ to $10^{-12}$ under KGW watermarking.
- Intact Task Performance: Unlike direct distillation (which reduced GSM8K accuracy from ~40% to near 0%), WAPITI preserved original accuracy on both MMLU and GSM8K benchmarks with virtually no drop in generation quality (perplexity remained ~3.1–4.1).
- Robustness: Watermark detectability remained robust to random token replacement edits up to 20% token corruption and held steady across varying sampling temperatures ($t \in [0, 0.75]$).
- KGW vs. AAR: WAPITI transfers logit-based watermarks (KGW) significantly better than sampling-based schemes (AAR), as KGW decomposes more cleanly into predictable $n$-gram shifts.
Limitations and Future Work
- Model Capacity Constraints: Watermark detectability in smaller models (Pythia-1.4B) was notably lower (AUROC ~0.61–0.78) than in Llama-2-7B, indicating smaller models struggle to retain parameter-transferred watermarking signals.
- Scheme Dependence: Sampling-based schemes like AAR showed non-linear, chaotic perplexity-to-detectability trade-offs compared to logit-based methods like KGW.
- Secret Parameter Assumption: The watermark integration vector must be kept strictly confidential. If an adversary gains access to $\Delta\theta_{\text{Base}}$, removing the watermark reduces to a trivial vector subtraction.
- Theoretical Approximations: The formal guarantees rely on empirical approximations (e.g., identity-like Hessian channels and strict orthogonality) that may vary across newer architectures or extreme fine-tuning domains.
Summary updated