Auditing Cross-Lingual Fairness in Language Model Watermarking

Alexander Nemecek, Osama Zafar, Debargha Ganguly, Vikash Singh, Vipin Chaudhary, Erman AydayarXiv 2026 · arXiv

TL;DR

This paper presents the first systematic cross-lingual fairness audit of large language model (LLM) watermarking schemes, revealing that standard English-centric evaluations mask massive disparities. The authors introduce a robust evaluation framework and demonstrate that current watermarking techniques exhibit severe, structural performance gaps rooted in language typology rather than random language quirks.

What Problem Does This Paper Address?

As AI-generated text becomes indistinguishable from human writing, text watermarking has emerged as a crucial tool to detect machine-generated content and combat misinformation. Watermarking works by subtly biasing an LLM's token selection during generation to leave a recoverable statistical signature.

However, existing watermarking schemes are evaluated almost exclusively on English text. When these systems are deployed multilingually, hidden assumptions about token distributions, vocabulary sizes, and generation entropy fall apart. This creates a massive fairness issue: non-English speakers might suffer from significantly degraded text quality or find themselves unprotected by reliable detection. This paper addresses this gap by auditing how watermarking choices impact cross-lingual fairness across different language families and scripts.

Methods

The authors propose a comprehensive four-component evaluation framework to audit cross-lingual watermarking:

  1. Empirical FPR Threshold Calibration: Instead of relying on a scheme's theoretical, hardcoded detection thresholds (which often fail on non-English text), they empirically calibrate thresholds per language to maintain a strict False Positive Rate (FPR).
  2. Threshold-Independent Companion Measurement: They use Area Under the ROC Curve (AUC) to distinguish between a calibration failure (where the watermark signal exists but the threshold is misplaced) and a true detection failure (where the signal is completely lost).
  3. Three Disjoint Quality Paradigms: To holistically evaluate text degradation, they measure quality across three distinct dimensions: distributional similarity (MAUVE), paired-semantic preservation (BERTScore), and local fluency/likelihood (Reference Perplexity).
  4. Generalized-Entropy Decomposition ($GE_2$): They apply inequality metrics from economics to decompose cross-language performance variance into within-family and between-family components based on linguistic typology.

The audit evaluates six prominent watermarking schemes (including KGW, Unigram, SynthID, and EXPEdit) across three open-weight generators (Mistral-NeMo, Gemma-3, Qwen2.5).

Datasets

The framework evaluates 11 languages spanning 4 scripts and 8 typological families using two distinct generation regimes:

  • FLORES+ devtest: Used for the base regime. Because FLORES+ provides parallel translations of the same source sentences, it allows the authors to hold prompt content perfectly constant across all 11 languages, isolating purely linguistic and tokenization effects.
  • AYA Dataset: Used for the instruct regime. The authors use the original-annotations subset (no machine translations) to provide linguistically natural, native-speaker prompt distributions for instruction-tuned models.
  • GlotLID-v3: A language identification tool used to filter and verify that the generated text actually matches the target language.

Key Insights and Ideas

  • The "Distortion-Free" Paradox: Schemes like SynthID and EXPEdit are mathematically designed to be "distortion-free" in expectation. However, the audit reveals that in finite multilingual samples, they are actually the most distorting schemes, severely degrading text quality compared to logit-bias methods.
  • Disparities are Structural, Not Idiosyncratic: Cross-lingual fairness gaps are not random anomalies unique to specific languages. Instead, the variance is predominantly between-family (e.g., Germanic vs. Romance vs. Indic). This means the unfairness is baked into structural properties like tokenizer fertility (how many tokens a language requires per sentence) and language morphology.
  • Instruction-Tuning Collapses Detection: Moving from base models to instruction-tuned models causes a massive drop in detection rates across almost all watermarking schemes. This is primarily a calibrational failure driven by the higher entropy of instruction-following generations.

Key Results

  • Rescuing "Broken" Watermarks: Under global default thresholds, the Unigram scheme on Gemma models appeared to be an outright detection failure (0.0 True Positive Rate across all 11 languages). However, the threshold-independent AUC companion metric remained incredibly high (~0.986). Applying per-language empirical recalibration successfully recovered the detection rate to 0.799, proving the failure was purely calibrational.
  • Quantifying Typological Disparity: Using the generalized-entropy index ($GE_2$), the authors found that for cells with meaningful performance dispersion, the between-family share of variance consistently sat between 61.3% and 99.9%, with a median above 90% across both regimes.
Figure 1: Per-language quality preservation under three paradigms, base-FLORES (top) and instruct- AYA (bottom); n = 500 matched pairs per cell. Left: MAUVE on mean-pooled XLM-R-large embeddings (1.0 identical, 0.0 fully separable). Center: BERTScore F1, rescaled against per-language FLORES non-pair baselines (negative: paired generations less similar than random non-pairs). Right: PPL preservation under XGLM-7.5B, exp(−| log PPLR(X +) −log PPLR(X −)|) ∈(0, 1]. Rows: (scheme, generator) cells; columns: eleven languages grouped by typological family.

As shown in the quality heatmaps (Figure 1), distortion-free schemes like SynthID and EXPEdit exhibit severe quality degradation (darker blue/purple blocks) across non-English languages in the base regime, which compresses upward but remains distinct in the instruct regime.

Per-scheme disparity decomposition under three quality paradigms

Figure 2 illustrates that the absolute disparity ($GE_2$) is overwhelmingly dominated by the dark blue "between-family" component, reinforcing that watermarking unfairness operates at the language-family level.

Limitations and Future Work

While the study covers a diverse grid, 11 languages represent only a fraction of global linguistic diversity. Additionally, the rapid pace of watermarking research means new schemes are constantly emerging that were not captured in this audit.

The authors note that because these fairness gaps operate at the typological family level, they cannot be fixed simply by throwing more training data at individual languages. Instead, future work must target core architectural choices, such as designing tokenizer vocabularies and watermarking partition schemes that are inherently aware of cross-lingual structural differences.

Summary updated

Abstract

Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements. Multilingual deployment exposes evaluation-design choices that are inconsequential on English but determine conclusions cross-lingually. We propose an evaluation framework with four components: detection thresholds calibrated empirically per deployment context, a threshold-independent companion measurement that distinguishes calibration failures from detection failures, three disjoint quality measurement paradigms (distributional, paired-semantic, and reference-perplexity), and a generalized-entropy decomposition of cross-language disparity over a typological family partition. Applied to six watermarking schemes, three open-weight generators, eleven languages spanning four scripts and eight typological families, and both base and instruction-tuned regimes, the framework reveals failure modes that single-language single-paradigm evaluation cannot surface. Across detection and quality, observed disparity is predominantly between-family on the typological partition, indicating that cross-lingual fairness gaps in watermarking are structural to language properties rather than idiosyncratic to particular languages.
BibTeX
@misc{nemecek2026auditing,
  author = {Alexander Nemecek and Osama Zafar and Debargha Ganguly and Vikash Singh and Vipin Chaudhary and Erman Ayday},
  title = {{Auditing Cross-Lingual Fairness in Language Model Watermarking}},
  year = {2026},
  eprint = {2608.20047},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.20047},
}
Figure