Is the Short-Text AI Detector Just Detecting Translation?

Tools that decide whether a short piece of text was written by a person or generated by a language model are increasingly built from large language models themselves. The model is prompted, or fine-tuned, to act as a judge that reads a snippet and gives a verdict. These tools are known to get worse as texts get shorter. They are also used on multilingual content, which is full of translated writing. Translated text carries a recognizable fingerprint called translationese: simpler vocabulary, over-explained references, and grammar leaking in from the source language. Several of these cues are the same ones detectors and judges read as signs of machine authorship: a formal register, explicit connectives, low lexical variety.

This note proposes a controlled test of a specific worry. On short multilingual texts, part of what detectors call "machine-generated" may really be "translated". Part of what they call "human" may be machine text that went through translation. We also predict that the two main detector families fail in opposite directions. LLM judges that reason about style should falsely flag human translations. Likelihood-based detectors should miss LLM text that has been translated, because translationese raises perplexity. We would build a benchmark that crosses authorship, translation status and length, measure the confound as a function of length, and test whether a cheap correction closes the gap. The correction scores each text with a graded translationese measure and conditions the detector's threshold on that score. We compare it against the obvious alternative of retraining detectors on translated data.

Core Claim

The claim is empirical and falsifiable. We define the translationese gap of a detector as its false positive rate on human-translated human writing minus its false positive rate on native human writing, at a threshold fixed on native text. We predict this gap grows as texts shrink from 100 words to 10 words. There is less content to reason about in short text, so surface style dominates the decision.

We further predict a dissociation between detector families. For a prompted LLM judge, the gap should be positive and large, because its rationales will cite translationese markers as evidence of AI authorship. Zero-shot likelihood detectors should show the mirror failure: a drop in true positive rate on LLM text that was generated in one language and translated into another.

The dissociation is motivated by a tension that has not been resolved in the literature. Detection work treats low perplexity as the machine signature. Lost in Literalism, which annotated unnatural spans in LLM translations, finds that such text has higher perplexity under native language models. The two forces push the same statistic in opposite directions. Nobody has checked which one wins at 20 words.

If the gap is near zero for all detectors at all lengths, the worry is unfounded and the paper reports a clean negative result. That outcome would still be useful, because deployers currently have no evidence either way.

Why This Is Not Already Done

The closest work is DIBJudge. It shows that multilingual LLM judges prefer machine-translated text over native human text when scoring quality, and it removes this bias with a two-branch information-bottleneck architecture. It evaluates reading-comprehension passages, summaries and reward benchmarks. It never asks judges who wrote a text, and it never varies length.

MultiSocial is the main benchmark for short multilingual detection. It covers 22 languages of social-media posts, with fine-tuned detectors near 0.97–0.98 AUC ROC and zero-shot statistical detectors at 0.72–0.76. However, it neither controls for nor labels translation status, so a translationese shortcut would be invisible in it. Authorship Obfuscation in Multilingual Machine-Generated Text Detection shows that backtranslation helps machine text evade detectors, but only on full news articles and only for encoder and entropy detectors.

An Investigation of Translationese in the Generations of Multilingual LLMs is the motivating evidence. It finds that SVMs trained on human translationese flag 60.2% of Gemini's native German and 37.6% of Llama's, against 0% of native human German. That study uses long Europarl speeches, and it asks whether LLMs write like translators, not whether detectors mistake translators for LLMs.

On the detection side, Bhushan et al. fine-tune GPT-4o as a short-answer judge and reach 80% accuracy. Their data is English only, and their labeling rubric uses cues such as formal tone and listicle structure, which overlap with explicitation.

Our contribution is none of these. It is a factorial measurement of whether translation status confounds short-text authorship verdicts, together with a test of a lightweight correction.

Research Program

Benchmark. We build a benchmark for German and Spanish, which prior SVM work identifies as the most translationese-prone languages, plus Chinese, where the translationese measure we reuse was developed. Every text belongs to one of five cells. The first is native human writing. The second is human-translated human writing. The third is machine-translated human writing, reported separately and never scored as either class, since its ground-truth label is contested. The fourth is LLM text generated natively in the target language. The fifth is LLM text generated in English and then machine-translated.

For the parliamentary domain, native and human-translated German and Spanish come from Europarl-UdS, whose translation direction is known. Machine translations follow the many-to-one construction of GTNC. LLM cells are produced by prompting several open and closed models with the same speech topics.

For the social domain, we take MultiSocial's pre-2022 human posts and add the translated cells by translating human posts from other languages in the benchmark. Human translation of a stratified subset of roughly a few thousand posts will be commissioned, because human-translated social text does not otherwise exist.

Every text is cut into windows of 10, 20, 35, 50 and 100 words. The 35-word point matches the hardest condition in the PAN 2024 Voight-Kampff task. We match topic and length across cells, so that length and content cannot explain differences between them.

Correction. We use the likelihood-ratio translationese index from Translationese-index. That work trains two 0.5B models, one on literal and one on idiomatic translations of the same sources. The difference in their log-probabilities cancels meaning and leaves naturalness. On single sentences it reaches 95.8% in-domain accuracy and 84.6% agreement with unanimous human pairwise judgments. We retrain the pair for German and Spanish.

The correction is deliberately simple. We fit a detector's decision threshold as a monotone function of the index on a small calibration set of native and translated human text. The detector is left untouched. A text that reads as heavily translated must clear a higher bar to be called machine-generated.

Mechanism. For every LLM-judge verdict we collect the judge's written rationale. We code each cited cue against the established translationese taxonomy: simplification, explicitation such as cohesive adverbs and full noun phrases where a pronoun would do (Koppel et al.), and interference such as retained optional pronouns (Kunilovskaya et al.). We then ask whether false positives on translated human text are driven by these cues specifically. This analysis tells us whether judges genuinely conflate translation with generation, or whether the effect comes from something else, such as topic.

Baselines

The naive reference point has two parts. The first is a detector that ignores translation status. Every detector is run uncorrected, which is exactly the current deployment practice. The second is a length-only logistic regression, which checks that our cells are not separable by trivial features.

The zero-shot likelihood family consists of Binoculars and Fast-DetectGPT, as benchmarked in MultiSocial (reported 0.72–0.76 AUC ROC on short social posts), and Telescope. Telescope scores how unlikely a model finds the token it has just read, and it reports 0.992 AUROC against 0.893 for plain perplexity on the Detect LLM Text dataset. These are the natural test of the "translation raises perplexity" prediction. All of them use the same scoring backbone per language, so that differences come from the scoring rule rather than from the model.

The rewriting family is represented by Raidar, which measures how much an LLM edits a text when asked to rewrite it and claims effectiveness down to 10 words. It is the strongest zero-shot competitor at short lengths, and it has an unstudied interaction with translationese: a rewriter may heavily "fix" human translations, making them look human, or leave machine translations untouched. We run it with the same rewriter model for all cells and the same number of rewrite calls.

The LLM-judge family consists of one open multilingual judge, M-Prometheus-14B (M-Prometheus), and one frontier model. Each is prompted zero-shot, with chain-of-thought, and with an authorship checklist modeled on the rubric in Bhushan et al. We also run a debiased DIBJudge model prompted for authorship, if its weights are available. This is the fairest test of whether preference-time debiasing transfers to detection.

The strongest supervised baseline is a multilingual detector fine-tuned in the MultiSocial recipe, with Llama-3-8B, whose reported AUC ROC is 0.97–0.98 on MultiSocial. We train two versions with identical compute and hyperparameter search: one on MultiSocial as-is, and one augmented with translated human and translated LLM text.

For the headline confound number itself, no reported result was found on Lacuna for false positive rates on human-translated short text. There is also no per-length number for the 35-word PAN slice. The PAN overview reports only that it was the hardest condition, with the winning system at 0.961 ROC-AUC averaged over all variants.

Beating a baseline means shrinking the translationese gap to within the calibration noise at every length, while losing no more than one point of AUROC and no more than one point of true positive rate at 1% false positive rate on native text.

The key ablation replaces the translationese index with two alternatives for conditioning the threshold. One is raw perplexity. The other is the probability output of a classical translationese SVM, trained as in the investigation paper. If raw perplexity works equally well, the specific likelihood-ratio construction is not what matters, and we will say so.

Why not just…?

Why not just fine-tune the detector on translated data? This is the real threat. MultiSocial shows that fine-tuning on short multilingual text is very effective, and the obfuscation study shows that adversarial retraining on backtranslation transfers to other attacks. Augmentation will likely close the gap in-distribution. The question is whether it does so out of distribution. We test the augmented detector on held-out source languages, a held-out translation system, and held-out human translators. We also run it with post-2022 LLMs not used in training.

If augmentation closes the gap out of distribution as well, the correction adds nothing, and the contribution reduces to the measurement and the prescription "augment with translations". We would report exactly that. The threshold correction is only worth keeping if it helps where augmentation data is missing, and for the zero-shot and LLM-judge detectors that cannot be retrained cheaply.

Why not just use stylometric or translationese features as the detector itself? Bhushan et al. show that 13 stylometric features reach 77% on short English answers, only 3 points below fine-tuned GPT-4o. Translationese SVMs are also strong. However, Collins-Thompson and Callan document that such statistics become unstable on short fragments. More fundamentally, a translationese classifier answers a different question from "who wrote this", which is the confound we are measuring. We include the SVM as a standalone detector and as a conditioning signal, so the benchmark directly shows whether it is enough.

Why not just make texts longer by pooling several posts or windows before deciding? The easy-to-hard supervision work proves that concatenation amplifies the human–machine difference. It raises TPR at 1% FPR on DetectRL from 27.98% to 31.65%. Pooling is not available when a single short answer or post must be judged on its own. Pooling also amplifies translationese along with machine signal, so it may widen rather than close the gap. Our length sweep measures directly whether the gap shrinks or grows with length.

Why not just use DIBJudge? DIBJudge removes translationese from a judge's quality preferences on long passages. Whether that removal survives a change of task, to authorship, and a change of length, down to 10 words, is precisely one of our measurements. If it does, that is a strong positive result for DIBJudge, and it becomes the recommended judge.

Decisive Checks

Three results would decide the proposal.

The first is the length curve of the translationese gap for each detector family. A positive, growing gap for LLM judges confirms the core claim. A flat, near-zero gap refutes it.

The second is the sign dissociation. The claim is that judges show false positives on human translations while likelihood detectors show false negatives on translated LLM text. If both families fail in the same direction, the perplexity tension is resolved in one direction, and the mechanism story must change.

The third is the out-of-distribution comparison between the index-conditioned threshold and translation-augmented fine-tuning at 10, 20 and 35 words. It decides whether the correction is a method worth keeping or only a diagnostic.

The rationale coding supplies the causal check behind all three. If false positives on translated human text do not disproportionately cite explicitation and interference cues, then translationese is not the mechanism, even if the gap exists.