Conferences now receive more submissions than their reviewer pools can absorb, and a common proposal is to let a reasoning language model do a first pass: read each manuscript and flag critical problems before humans spend time on it. These models also carry social biases. They favor rich countries and elite institutions, and their bias grows in languages with little training data. This proposal asks a narrow, testable question at the meeting point. If the scientific content is held fixed and only the surface of the text changes, does a model acting as a flaw checker find real errors less reliably, or invent errors more often? The surface changes are English written by a speaker of a low-resource language, English machine-translated from a draft in that language, or a paper whose object of study is a low-resource language. We measure bias as a gap in error rates on known ground-truth flaws, not as a gap in how much the model "likes" a paper.
The second claim is methodological. The cheap way to run such an audit is to have an LLM rewrite papers "as a non-native speaker would." We expect this synthetic proxy to misestimate the real disparity, just as down-sampled English misrepresents genuine low-resource data. We therefore build a small set of authentic variants with bilingual researchers and compare the two directly. If the proxy tracks authentic text closely, that is a useful negative result: future audits can safely stay cheap.
Core Claim
The claim has three parts. First, reasoning-model flaw checkers show a measurable drop in flaw-detection hit rate, and a rise in false flags on sound papers, on authentic low-resource-origin English at matched content. Second, synthetic "accented" rewrites misestimate this gap, most likely underestimating it, because they lack the real lexical and structural signatures of translated or L2 (second-language) writing. Third, normalizing the prose before screening, by first polishing it into fluent English, should close most of the surface-form gap. It cannot close a gap tied to what the paper studies. The part of the bias that survives normalization is therefore the part that matters for deployment.
Grounding in Prior Work
Three papers motivate the question. Can We Automate Scientific Reviewing? trained a BART review generator on ICLR and NeurIPS reviews. It found that the model removed the human "clarity" penalty against non-native authors but rated those same papers harsher on originality. That result concerns a fine-tuned generator's preference scores on English ML papers. We instead study modern reasoning models as flaw checkers, and we score them against ground-truth errors.
Mind the Language Gap translated bias templates into Catalan and Luxembourgish and found bias rising and becoming more erratic as data availability falls. Investigating Subtler Biases in LLMs documented a halo effect: models attach competence and creativity to rich nations and elite universities. Both work with short prompts to chatbots. Neither studies long technical documents or a model making screening decisions.
Uncovering Stereotypes in Large Language Models reports that refusals of biased choices fall as tasks become more complex, from 32.8% to 29.4% for ChatGPT. Reading a full paper is far more complex than any of its task levels, so we test whether disparity grows from abstract-only screening to full-paper screening.
Two papers constrain the design. High-Resource Methodological Bias in Low-Resource Investigations showed that simulating scarcity by down-sampling high-resource data produces misleading evaluations. That is the reason we refuse to rely on synthetic rewrites alone. NLPeer found that its skimming model exploited paragraph length, so every variant must be length-matched to the original.
Two papers supply methods. Reviewing Scientific Papers for Critical Problems With Reasoning LLMs provides WITHDRARXIV-CHECK, a set of arXiv papers withdrawn for errors that invalidate their conclusions, along with its LLM-as-judge scoring of whether a flagged problem matches the true flaw. We reuse both. From Bias to Balance supplies a post-hoc fairness re-ranker. It works only on tabular scores and author metadata and never reads text. It serves as the downstream-correction comparison.
Research Program
We start from the 245-case WITHDRARXIV-CHECK test set and select roughly 60 cases in which the invalidating flaw sits in prose, not only in a figure or proof. To these we add about 60 sound papers drawn from accepted work in Research-14k, the corpus of accepted papers from CycleResearcher. The sound papers let us measure false flags. For each paper we rewrite the sections that carry the claims: abstract, introduction, and the method and experiment passages containing the flaw. Every other part stays unchanged.
We create five surface-form conditions for each paper:
- Original. The text as submitted.
- Authentic L2. Bilingual researchers whose first language is Catalan, Luxembourgish, isiZulu or isiXhosa rewrite the English in their own natural English. These languages are chosen to connect to the MLA-BiTe and NGLUEni communities.
- Translated-from-L1. The same researchers render the sections in their first language, and standard machine translation brings them back to English. This mimics authors who draft in their own language.
- Synthetic L2. An LLM is prompted to write as a non-native speaker of the same language.
- Normalized. Each variant is polished back into fluent English by an LLM.
Annotators confirm that each rewrite preserves the flaw's claim and its location. An automatic check confirms that the numbers and equations are identical. Length is held within 5% of the original.
We add two metadata and content conditions. The first is an institution and country line that varies between high-GDP and low-GDP affiliations, which tests the halo effect under screening. The second is a topic swap, restricted to NLP papers whose evaluation language can be changed without touching the method or the reported numbers. For example, "German" is replaced with "isiXhosa" and the dataset names are adjusted to match. This is the condition we expect normalization cannot fix.
Checkers are run with the source paper's protocol: top-five flagged problems, judged against the true flaw. Screening is run at two input lengths, abstract-only and full text, to test the complexity effect.
The primary quantity is the paired disparity in hit rate. For the same flaw, it is the difference between the fraction detected in the original and the fraction detected in a variant. The false-flag rate on sound papers is reported the same way. Paired bootstrap confidence intervals are computed over papers. The proxy question is answered by comparing the synthetic-L2 disparity with the authentic-L2 disparity for each language.
Baselines
Naive reference. The first reference point is the checker on unmodified text. It defines zero disparity by construction and gives the absolute hit rate against which gaps are read. We also run a length-and-formatting heuristic screener. If that heuristic shows the "same" disparity, our variants have leaked surface artifacts.
Strongest checker. The strongest checker is OpenAI o3 at medium reasoning effort, run with the protocol of Reviewing Scientific Papers for Critical Problems With Reasoning LLMs. That paper reports o3 as best on HR@5, the fraction of papers whose true flaw appears among the top five flagged problems, and reports that it found 350 true-positive problems on the 245-case test set. Gemini Pro found 293 and Claude 3.7 Sonnet found 149. The file gives no numeric HR@5 value. We re-run o3, Gemini Pro and Claude on our subset with identical inputs in both PDF and LaTeX, the same judge, and the same prompt. "Beating" is not the goal for this baseline. The finding would be a disparity whose paired confidence interval excludes zero while the original-text hit rate reproduces the published model ranking.
Mitigation baselines. Each mitigation is given the same compute and a comparable prompt-tuning budget, tuned on a held-out split. A fairness system prompt instructs the model to ignore fluency, nationality and prestige. Separately, Fair-PaperRec-style post-hoc score recalibration is applied by region. It is tuned across its fairness weight, where the paper reports that a weight around 3 was needed in high-bias settings.
Key ablation. The key ablation is normalize-then-check. A mitigation counts as winning if it closes the disparity to within the confidence interval of zero without lowering the original-text hit rate by more than two points.
Audit-method baseline. The synthetic-L2 audit is itself a baseline. Our authentic audit "beats" it if the two disparity estimates differ beyond their joint confidence interval in at least two of the four languages.
Why not just…?
Why not just polish every submission into fluent English before screening? This is the most serious threat, and we treat it as the main ablation, not as a strawman. If normalization removes the surface-form gap, that is a practical, deployable result, and we will report it plainly. It has two limits. It cannot touch the topic-swap condition, because the language being studied is content. And it may paraphrase away the very sentence that carries a flaw, lowering the hit rate for everyone. The experiment measures both effects. A finding that normalization fixes form but leaves topic disparity intact would sharpen the proposal's claim, not weaken it.
Why not just tell the model to be fair? The complexity study in Uncovering Stereotypes in Large Language Models suggests that guardrails weaken as tasks grow harder. FairBelief argues that biased associations sit at the model's most likely outputs, not in its rare ones. Neither result was obtained on document-length scientific reasoning, though. Our abstract-versus-full-text contrast, run with and without the fairness prompt, directly tests whether the prompt helps and whether its effect shrinks with length.
Why not just correct downstream with a fairness re-ranker? From Bias to Balance raised underrepresented participation by up to 42.03%, while its h-index-weighted utility changed by at most 3.16%. However, its individual-level gains were unstable, and it adjusts scores without looking at why they are wrong. A re-ranker can raise the average score of a region. It cannot recover a missed flaw in a specific paper, and it may even promote flawed papers from that region. Measuring hit-rate and false-flag disparities after recalibration shows whether score parity hides error-rate disparity.
Why not just audit with synthetic rewrites, which cost nothing? If synthetic and authentic disparities agree, we will say so, and the cheap audit becomes the recommended method. The down-sampling result in High-Resource Methodological Bias in Low-Resource Investigations gives a concrete reason to doubt this: generated data tends to be cleaner and lexically richer than authentic text. Our per-language comparison settles the question.
Decisive Checks
The proposal succeeds or fails on four measurements:
- Disparity exists. Does o3's paired hit-rate disparity on authentic L2 or translated-from-L1 text exclude zero?
- Proxy fidelity. Does the synthetic-L2 estimate differ from the authentic estimate?
- What normalization fixes. Does normalization close the surface-form gap while the topic-swap gap remains?
- Complexity effect. Is disparity larger in full-text screening than in abstract-only screening?
A null on the first check with a working protocol is still informative, since it would mean current flaw checkers are robust to language origin at matched content. For benchmark context outside peer review, no reported result was found on Lacuna for any flaw-checker evaluated on non-native or translated scientific text.
References
- Can We Automate Scientific Reviewing?
- Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages
- Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models
- Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach
- High-Resource Methodological Bias in Low-Resource Investigations
- NLPeer: A Unified Resource for the Computational Study of Peer Review
- Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluation
- From Bias to Balance: Fairness-Aware Paper Recommendation for Equitable Peer Review
- CycleResearcher: Improving Automated Research via Automated Review
- NGLUEni: Benchmarking and Adapting Pretrained Language Models for Nguni Languages
- FairBelief - Assessing Harmful Beliefs in Language Models