A machine-generated text detector is a classifier that decides whether a passage was written by a person or by a language model. When such a detector is trained on many languages at once, the share of machine text almost never matches across languages. Some languages come mostly from generator outputs and have few human samples; others are the reverse. A classifier trained on this pooled data can learn a cheap rule, "text in language X is usually machine-written," and fold it into every score it produces. The detector then looks accurate on a test set with the same mix and fails when the mix changes. It falsely flags human writers in languages that were machine-heavy in training, and it misses machine text in languages that were human-heavy. Deployment changes the mix almost by definition, because the prevalence of machine text in a Telegram feed in Ukrainian has no reason to match that of a curated news benchmark.
We propose to treat the detector's output as two separate quantities. The first is evidence, meaning how machine-like this passage is compared with human text in the same language. The second is prevalence, meaning how common machine text is in this language in this data stream. We train the shared multilingual model so that its score carries only evidence. Prevalence is then added back per language at deployment, either from a known rate or estimated from unlabeled target text. The claim is that this split beats both naive joint training and existing per-language reweighting under realistic distribution shift. It also gives the only sensible handling of languages that had no labels in training, because a prevalence can be estimated from unlabeled text where a per-language weight or threshold cannot be computed.
Core Claim
In our notation, $s(x)$ is the detector's logit for passage $x$, $L$ is its language, and $\pi_L$ is the fraction of machine text in language $L$ in the stream being scored. The goal is that $s(x)$ approximates the within-language log-likelihood ratio $\log p(x \mid \text{machine}, L) - \log p(x \mid \text{human}, L)$, so that the posterior probability of machine authorship is
$$ \operatorname{logit} P(\text{machine} \mid x, L) = s(x) + \log \frac{\pi_L}{1 - \pi_L}. $$
This is Bayes' rule written in log-odds. The first term should transfer across datasets and languages. The second term should be supplied per language and per stream, never learned from training frequencies.
Three testable consequences follow. First, a detector trained under skewed language-by-label frequencies will encode language as a label shortcut, and this will show up both as higher language identifiability in its representations and as prevalence-dependent false-positive rates on human text. Second, conditionally balanced training followed by the prior correction above will make per-language accuracy nearly invariant to test-time prevalence shifts. Third, for held-out languages, estimating $\pi_L$ from an unlabeled pool with the evidence-only detector will recover most of the gap to an oracle that knows $\pi_L$.
What Prior Work Establishes and What It Leaves Open
The closest work is Understanding the effects of language-specific class imbalance in multilingual fine-tuning. It induces skewed label distributions within languages for mBERT on XNLI and Amazon Reviews. It shows that the model starts using language identity as a shortcut: language identifiability on the classification token rises from 0.614 to 0.899 on XNLI. It then shows that weighting the loss per language-label pair partly undoes this, bringing identifiability to 0.679 and accuracy from 0.556 to 0.569. That paper motivates our first consequence and supplies our main training ingredient.
It stops at training-time repair, and it evaluates on test sets that follow the training mix. It does not ask what happens when prevalence changes at deployment, which is the defining condition of detection. It does not address languages absent from training. It does not study machine-text detection, where the imbalance arises naturally rather than being induced. Our contribution is the other half of the problem: prevalence as an explicit per-language deployment variable, estimated from unlabeled text when needed. We also provide the stress tests that separate the two failure modes.
The detection benchmarks both motivate and constrain the design. MULTITuDE shows that joint multilingual fine-tuning of mDeBERTa-v3 beats English-only training. It reports that an English-only detector falls from 0.9292 F1 on English to 0.6903 on other languages, and that a multilingual mix reaches 0.8480 macro F1. Its corpus is itself skewed, with roughly 8,000 human articles against 66,000 machine texts according to the file. That suggests joint training on this benchmark is already exposed to the shortcut we describe, although no one has measured it. MultiSocial reports fine-tuned AUC ROC of 0.97–0.98 across 22 languages on short social-media posts. M4GT-Bench shows that supervised detectors collapse on unseen generators and languages.
All three score detectors on test splits whose class mix matches training, or on threshold-free metrics that hide the problem. Our protocol deliberately breaks that match. Nota AI's shared-task system handles unseen languages by routing them to a separate feature pipeline built on small-LLM token statistics. Its approach is architectural. Ours keeps a single jointly trained model and makes the unseen-language question one of estimating a scalar per language.
Two further lines constrain the design. AIDER strips topic information from an English detector with a gradient reversal layer, which trains the encoder to defeat an auxiliary classifier. We reuse that tool as an ablation that removes language information adversarially, to test whether representation-level invariance adds anything beyond reweighting. Role of Language Relatedness in Multilingual Fine-tuning finds that pooling more languages eventually hurts, and attributes this to capacity. In detection, adding a language with an extreme machine share also adds a new shortcut. We test whether part of the "interference" seen when pooling languages for detection is actually prevalence skew rather than capacity.
Research Program
The first component is a controlled-skew protocol built on MULTITuDE and MultiSocial. For each training language we subsample so that the machine share takes a chosen value. Some languages are set near 5%, some near 50%, and some near 95%, with the total per-language sample size held fixed so that skew is not confounded with data volume. We then build three test conditions for every language. The first matches the training skew. The second is balanced. The third reverses the skew, so that languages that were machine-heavy in training become human-heavy. A small set of languages, including at least one Slavic and one non-Indo-European language, is withheld entirely and appears only as unlabeled pools at test time with a hidden prevalence. The same protocol is also run on the benchmark's natural, unmanipulated ratios, so that the results do not depend on an artificial construction.
The second component is the method. The shared encoders are mDeBERTa-v3-base, XLM-RoBERTa, and multilingual E5-large, plus one LoRA-tuned decoder in the spirit of MultiSocial's Llama-3-8B detector if compute allows. Each is trained with loss weights per language-label pair, so that within each language human and machine contribute equally. This is the device from the EACL work, and it is what makes $s(x)$ an estimate of the likelihood ratio rather than a posterior. We then add a light post-hoc temperature fit on held-out balanced data, so that $s(x)$ is calibrated as a ratio.
At deployment, $\pi_L$ is used directly when a rate is known. When it is not, $\pi_L$ is estimated per language from unlabeled target text by expectation-maximization. The procedure alternates between computing posteriors with the current prior and resetting the prior to the mean posterior. This estimate is valid only if $s(x)$ is a stable likelihood ratio, which is exactly what the training step aims for. The procedure adds no parameters and runs on the same model the baselines use.
The third component is analysis. We probe language identifiability in the final representation, as in the EACL work, and run a language-swap diagnostic. The diagnostic takes parallel human texts across languages, where MULTITuDE's news sources allow, and measures how much the raw score moves with language alone. We also rerun a relatedness-style pooling experiment: we add languages one at a time and track per-language balanced accuracy under naive pooling and under conditional balancing. This tests how much of the degradation seen when adding languages disappears once skew is controlled.
Baselines
The naive reference point is joint multilingual fine-tuning with plain cross-entropy on the pooled skewed data and a single global threshold. This is the MULTITuDE recipe, and the file reports 0.8480 macro F1 for mDeBERTa-v3-base on its multilingual mix (MULTITuDE). We rerun it under our protocol rather than quoting that number, since our splits differ. We also report a translate-to-English pipeline: NLLB-style translation followed by an English detector trained on the same budget. Finally, we report a zero-shot statistical detector built from token log-probabilities and entropy under a small multilingual LLM. MultiSocial reports AUC ROC of 0.72–0.76 for this family on social text (MultiSocial).
The strongest existing competitor is per-language-label weighting from the EACL 2024 study, deployed as that paper does, without a prior correction. It is the strongest because it already removes much of the representation shortcut, and on matched test sets it may be indistinguishable from our method. The file reports no machine-text detection result for it. Its reported XNLI numbers are 0.569 accuracy and 0.679 language identifiability.
A second strong competitor is per-language threshold tuning on labeled validation data. It can absorb a prior shift for seen languages whenever labeled target data exist. For the shared-task setting we compare against the file's reported Nota AI result of 0.7532 macro F1 on GenAI Detection Task 1 (Nota AI). This is also where their routing system is the natural rival for unseen languages.
All systems share the same encoder, training data, number of steps, and a hyperparameter search of equal size, tuned on a balanced validation split that is never a reversed-skew split. Two ablations isolate the key ingredient. The first applies conditional balancing without the prior correction, which is the EACL baseline. The second applies the prior correction on top of naive training. A third variant adds the language-adversarial head from AIDER. We also report an oracle that knows the true $\pi_L$, to bound what estimation can deliver.
The success criterion has four parts. Under reversed skew, our method must beat EACL-style weighting on worst-language F1 and on human false-positive rate by a margin larger than the seed variance. Under matched skew, it must stay within noise of that baseline. On held-out languages, estimated-prior results must close most of the gap between no correction and the oracle.
Why not just…?
Why not just use global class weights or focal loss? Global weighting fixes the marginal ratio but leaves language and label correlated. The EACL study shows directly that language-specific skew survives global treatment and that the shortcut persists (Understanding the effects of language-specific class imbalance). We include global weighting as a baseline, and the reversed-skew test will show whether its human false-positive rate follows training prevalence.
Why not just use per-language class weighting and stop? This is the real threat, and we treat it as the main competitor rather than a strawman. Reweighting corrects what the model learns, but by itself it assumes a balanced prior at test time, and it has nothing to offer for a language with no labels. If reweighting alone matches our method under reversed skew and on held-out languages, then the prior-correction half of our claim is false. The ablation that removes the correction settles this directly.
Why not just tune a threshold per language on validation data? For seen languages with labeled target data, this is equivalent to supplying $\pi_L$ and may match us. We will report it as such. It fails exactly where detection is hardest: new languages, new platforms, and streams whose prevalence drifts without labels. Our held-out-language experiments compare our unlabeled prior estimate against thresholds that do not exist there, and against thresholds borrowed from a related language, following the intra-family transfer MULTITuDE observes.
Why not just translate to English, or use a zero-shot statistical detector? Authorship Obfuscation in Multilingual Machine-Generated Text Detection shows that back-translation is itself an evasion attack, because it wipes out generator statistics. MultiSocial reports zero-shot AUC of only 0.72–0.76 on short text. Both are included as reference points, but neither is expected to be competitive on the evidence term.
Decisive Checks
We see four decisive results. The first is whether naive joint training shows a human false-positive rate per language that tracks training prevalence under reversed test skew. If it does not, the problem we target is small, and the proposal reduces to a negative finding worth reporting. The second is whether conditional balancing plus prior correction flattens that dependence, compared with conditional balancing alone. The third is whether EM-estimated prevalence on held-out languages lands close to the truth. This will fail if calibration of $s(x)$ breaks on unseen languages. We will measure that failure directly by comparing estimated with true prevalence and reliability curves per language, and it would indicate that the evidence term itself does not transfer. The fourth is whether controlling skew removes a measurable part of the degradation observed when pooling more languages, which would reframe part of the "negative interference" story for detection.
References
- Understanding the effects of language-specific class imbalance in multilingual fine-tuning
- MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark
- MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts
- M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection
- Nota AI at GenAI Detection Task 1: Unseen Language-Aware Detection System for Multilingual Machine-Generated Text
- AIDER: a Robust and Topic-Independent Framework for Detecting AI-Generated Text
- Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages
- Authorship Obfuscation in Multilingual Machine-Generated Text Detection