AI-text detectors flag human essays by non-native English writers as machine-written far more often than essays by native writers. The standard explanation is linguistic. Second-language writers use safer words and simpler syntax, so a language model finds their text predictable, measured as low "perplexity" (how surprised the model is by each next word), and low perplexity is what detectors read as machine output. Every number behind this story, however, comes from public human corpora such as the TOEFL essays in GPT detectors are biased against non-native English writers, OpenReview reviews, and open-access papers. Those same corpora can sit inside the pretraining data of the models detectors use for scoring, and they have since become the fairness yardsticks newer detectors tune and report against.
This proposal separates three causes that are currently tangled together in the measured non-native gap. The first is genuine linguistic predictability. The second is memorization: the scoring model may have seen these exact essays during training, which lowers their perplexity. The third is fairness-benchmark overfitting: detectors may have been calibrated or selected on these exact essays. The method follows GSM1k, which exposed contamination in math benchmarks by writing a fresh test set that matched the original's distribution. Here it is applied to detector fairness, together with controlled "planted contamination" experiments on open-data scoring models, where the true training membership of each text is known rather than guessed.
Core Claim
Published non-native false-positive rates are measurements on possibly contaminated, possibly overfit test sets, so they may misstate the true bias in either direction. Memorization pushes all public human text toward the "machine" side. This matters most for non-native text, because formulaic phrasing is plausibly easier to memorize. Tuning on the public TOEFL set pushes the measured bias down for detectors that report on it. The test is direct. Build a fresh, post-cutoff, proficiency-matched corpus of native and non-native writing. Compare each detector's false-positive gap on it against the gap on public corpora. Then use planted contamination to show how much of any difference memorization can produce.
This differs from its closest relatives in two ways. GPT detectors are biased against non-native English writers established the gap and attributed it entirely to linguistic constraint, but it never asked whether its scoring model (GPT-2 XL) had seen the essays. A Careful Examination of Large Language Model Performance on Grade School Arithmetic supplies the fresh-mirror methodology, but it measures capability inflation in math, not fairness metrics on human text.
Research Program
Motivating evidence. Two observations motivate the work.
- Spotting LLMs With Binoculars defines its score as a ratio of perplexity under one model to "cross-perplexity," which compares the predictions of two closely related models. Its authors concede a "memorization trap": text the models have memorized scores as machine-written.
- Rethinking Pretraining Data Detection for LLMs: From Local to Global shows that simple membership tests, which ask whether a text was in the training data, confuse membership with lexical commonness. That confusion is exactly the one at stake for non-native text.
Constraining evidence. SciContrib-Bench shows that detectors with near-perfect in-domain scores collapse under mild distribution shift. Any fairness estimate drawn from one public split should therefore be treated as fragile.
Component 1: a fresh fairness set. We will commission timed, supervised essays written after the training cutoffs of every scoring model used. The writers will be non-native authors across several first languages and graded proficiency bands, plus native writers. Prompts will be matched to the TOEFL prompt style. Proficiency is rated independently so the two groups can be compared at equal skill. To keep the set uncontaminated, it is held privately and scored through a gated service. A zero-knowledge-proof scheme such as P⁴Bench is one option for letting outsiders verify results without seeing the essays.
Component 2: planted contamination. We use open models whose pretraining corpus is public, so membership is known; the Pile-based models behind the MIMIR membership benchmark are one example. We continue pretraining these scoring models on a random half of the fresh essays from each group, at exposure counts of 0, 1, 4, and 16. We then measure how perplexity, the Binoculars score, and the false-positive rate at a fixed threshold shift for members versus non-members, separately for native and non-native writers.
This gives a dose-response curve: how many exposures it takes to move a human essay across the detection threshold, and whether non-native text moves faster. The same planted ground truth lets us test whether membership detectors can tell memorized non-native essays from unmemorized ones, or whether they simply flag non-native style as "member." The candidates are Min-K% probability and the AECA method from Rethinking Pretraining Data Detection.
Component 3: fairness overfitting. Each detector's tunable parts are calibrated on public TOEFL and, separately, on a disjoint slice of the fresh set. Tunable parts include the threshold, and the labelled validation set that Telescope needs. We then measure false-positive rates on the held-out fresh essays. The gap between the two calibrations is the overfitting penalty.
Following the leaderboard-sensitivity analysis of When Benchmarks are Targets, we also check whether detector rankings by non-native false-positive rate survive the move from public to fresh data.
Baselines
Naive reference. A raw perplexity threshold under GPT-2 XL, set to a fixed false-positive rate on native essays. This reproduces the mechanism GPT detectors are biased against non-native English writers analyzed. That paper reports an average false-positive rate of 61.22% on TOEFL essays across seven detectors, with individual detectors reaching 76%. The naive baseline tells us whether any measured gap is specific to modern detectors or is already present in plain perplexity.
Strongest existing detectors. These are the systems claimed to fix the bias.
- Binoculars, run with its published Falcon-7B observer and performer pair. It is the main detector reported to keep "high accuracy on ESL essays without an increase in false positives." No ESL-specific numerical result for Binoculars was found on Lacuna; its general headline is 93–99% true-positive rate at 0.01% false-positive rate on ChatGPT text.
- Telescope, which reports 0.99983 AUC on ESL rewriting tasks. Note that this is a ranking score on a rewriting task, not a false-positive rate on untouched human essays.
- Lastde++, run from the released Lastde_Detector code, as a representative of probability-sequence methods.
All detectors receive identical text, identical length bins, and the same threshold-selection rule: the threshold that gives 1% false positives on native essays, chosen on the same calibration split. All are scored with the same open models wherever the method permits.
Contamination-audit baselines. Min-K% probability and AECA, applied to public TOEFL, to fresh essays, and to planted members.
Ablation. The key ingredient is known membership. The comparison of dose 0 versus dose k on the same model and the same essays isolates memorization's effect from everything else.
What counts as success. The claim holds if, for at least one of the strongest detectors, the non-native false-positive gap on fresh data differs from its public-data gap by more than the bootstrap 95% interval. It also holds if planted exposure moves non-native false-positive rates measurably more than native ones. A clean null, where fresh and public gaps match and dose-response is flat at realistic exposure counts, would also be a publishable result. It would confirm the purely linguistic account.
Why not just…?
Why not just collect fresh essays and rerun the detectors, GSM1k-style? This is the most serious alternative, and Component 1 is exactly that. It would detect a discrepancy, but it cannot say why the discrepancy arises. Fresh and public essays differ in era, prompt pool, and test-taker population as well as membership. Only the planted-contamination arm holds the text fixed and varies membership alone. If the fresh-versus-public gap is large but the dose-response is flat, we will report that population shift, not memorization, explains it, and drop the contamination story.
Why not just run a membership detector on the TOEFL corpus? Membership scores are themselves built from token probabilities. Rethinking Pretraining Data Detection shows that local scores of this kind confuse membership with commonness, and AECA was validated only on Wikipedia and Pile subsets. On non-native text, a high "member" score is the very ambiguity we are trying to resolve. The planted arm supplies the ground truth needed to check whether these tools work on this population at all.
Why not just calibrate thresholds separately per group? This requires labelling each writer as native or non-native at deployment, which is itself a fairness hazard. Liang et al. also show that light prompting ("use advanced technical language") pushes AI text into the high-perplexity region, so loosening thresholds invites evasion. More basically, a per-group threshold tuned on a contaminated public set inherits that set's distortion. Component 3 measures exactly this inherited distortion.
Proposed Experiments and Decisive Checks
Corpora. The primary corpora are public TOEFL essays and the US 8th-grade native essays, against the fresh matched set. As an observational temporal check, we compare human-labelled ICLR/NeurIPS reviews in the AI-Peer-Review-Detection-Benchmark from Is Your Paper Being Reviewed by an LLM? from before and after each scoring model's cutoff. This check comes with a caveat: post-2022 "human" reviews may be AI-edited, so we restrict to pre-2023 venues for the after-cutoff contrast wherever cutoffs allow. That paper reports the Anchor method at 63.5% true positives at 0.1% false positives versus 17.1% for Binoculars. We will test whether Anchor's false-positive rate on human reviews also tracks scorer membership, since Anchor compares a review against a model-generated one for the same paper.
Metrics. We report false-positive rate at a fixed native false-positive rate, the gap between non-native and native false-positive rates within each proficiency band, AUROC, and the rank correlation of detector orderings between public and fresh data.
Decisive checks. There are three.
- The public-minus-fresh difference in the non-native gap, at matched proficiency.
- The slope of non-native minus native false-positive rate against planted exposure count.
- The fresh-set false-positive penalty for calibrating on public TOEFL.
If all three are near zero, the linguistic account stands and fairness benchmarks can be trusted as is. If any one is large, published detector fairness claims need re-measurement on held-out, post-cutoff text, and the fresh set becomes the standard for that re-measurement.
References
- GPT detectors are biased against non-native English writers
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic
- Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
- Rethinking Pretraining Data Detection for LLMs: From Local to Global
- SciContrib-Bench: Mapping the Autonomy Landscape of AI Scientists Through Stage-Dependent Detectability
- $\text{P}^4$Bench: Contamination-Proof, Publicly Verifiable, and Privacy-Preserving LLM Evaluation via Zero-Knowledge Proofs
- Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
- Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review