TL;DR
This paper presents the first systematic cross-lingual fairness audit of large language model (LLM) watermarking schemes, revealing that standard English-centric evaluations mask massive disparities. The authors introduce a robust evaluation framework and demonstrate that current watermarking techniques exhibit severe, structural performance gaps rooted in language typology rather than random language quirks.
What Problem Does This Paper Address?
As AI-generated text becomes indistinguishable from human writing, text watermarking has emerged as a crucial tool to detect machine-generated content and combat misinformation. Watermarking works by subtly biasing an LLM's token selection during generation to leave a recoverable statistical signature.
However, existing watermarking schemes are evaluated almost exclusively on English text. When these systems are deployed multilingually, hidden assumptions about token distributions, vocabulary sizes, and generation entropy fall apart. This creates a massive fairness issue: non-English speakers might suffer from significantly degraded text quality or find themselves unprotected by reliable detection. This paper addresses this gap by auditing how watermarking choices impact cross-lingual fairness across different language families and scripts.
Methods
The authors propose a comprehensive four-component evaluation framework to audit cross-lingual watermarking:
- Empirical FPR Threshold Calibration: Instead of relying on a scheme's theoretical, hardcoded detection thresholds (which often fail on non-English text), they empirically calibrate thresholds per language to maintain a strict False Positive Rate (FPR).
- Threshold-Independent Companion Measurement: They use Area Under the ROC Curve (AUC) to distinguish between a calibration failure (where the watermark signal exists but the threshold is misplaced) and a true detection failure (where the signal is completely lost).
- Three Disjoint Quality Paradigms: To holistically evaluate text degradation, they measure quality across three distinct dimensions: distributional similarity (MAUVE), paired-semantic preservation (BERTScore), and local fluency/likelihood (Reference Perplexity).
- Generalized-Entropy Decomposition ($GE_2$): They apply inequality metrics from economics to decompose cross-language performance variance into within-family and between-family components based on linguistic typology.
The audit evaluates six prominent watermarking schemes (including KGW, Unigram, SynthID, and EXPEdit) across three open-weight generators (Mistral-NeMo, Gemma-3, Qwen2.5).
Datasets
The framework evaluates 11 languages spanning 4 scripts and 8 typological families using two distinct generation regimes:
- FLORES+ devtest: Used for the base regime. Because FLORES+ provides parallel translations of the same source sentences, it allows the authors to hold prompt content perfectly constant across all 11 languages, isolating purely linguistic and tokenization effects.
- AYA Dataset: Used for the instruct regime. The authors use the original-annotations subset (no machine translations) to provide linguistically natural, native-speaker prompt distributions for instruction-tuned models.
- GlotLID-v3: A language identification tool used to filter and verify that the generated text actually matches the target language.
Key Insights and Ideas
- The "Distortion-Free" Paradox: Schemes like SynthID and EXPEdit are mathematically designed to be "distortion-free" in expectation. However, the audit reveals that in finite multilingual samples, they are actually the most distorting schemes, severely degrading text quality compared to logit-bias methods.
- Disparities are Structural, Not Idiosyncratic: Cross-lingual fairness gaps are not random anomalies unique to specific languages. Instead, the variance is predominantly between-family (e.g., Germanic vs. Romance vs. Indic). This means the unfairness is baked into structural properties like tokenizer fertility (how many tokens a language requires per sentence) and language morphology.
- Instruction-Tuning Collapses Detection: Moving from base models to instruction-tuned models causes a massive drop in detection rates across almost all watermarking schemes. This is primarily a calibrational failure driven by the higher entropy of instruction-following generations.
Key Results
- Rescuing "Broken" Watermarks: Under global default thresholds, the Unigram scheme on Gemma models appeared to be an outright detection failure (0.0 True Positive Rate across all 11 languages). However, the threshold-independent AUC companion metric remained incredibly high (~0.986). Applying per-language empirical recalibration successfully recovered the detection rate to 0.799, proving the failure was purely calibrational.
- Quantifying Typological Disparity: Using the generalized-entropy index ($GE_2$), the authors found that for cells with meaningful performance dispersion, the between-family share of variance consistently sat between 61.3% and 99.9%, with a median above 90% across both regimes.
As shown in the quality heatmaps (Figure 1), distortion-free schemes like SynthID and EXPEdit exhibit severe quality degradation (darker blue/purple blocks) across non-English languages in the base regime, which compresses upward but remains distinct in the instruct regime.
Figure 2 illustrates that the absolute disparity ($GE_2$) is overwhelmingly dominated by the dark blue "between-family" component, reinforcing that watermarking unfairness operates at the language-family level.
Limitations and Future Work
While the study covers a diverse grid, 11 languages represent only a fraction of global linguistic diversity. Additionally, the rapid pace of watermarking research means new schemes are constantly emerging that were not captured in this audit.
The authors note that because these fairness gaps operate at the typological family level, they cannot be fixed simply by throwing more training data at individual languages. Instead, future work must target core architectural choices, such as designing tokenizer vocabularies and watermarking partition schemes that are inherently aware of cross-lingual structural differences.
Summary updated