Systematic reviews decide which studies count as evidence, and they have long been skewed toward English-language, Western-indexed literature. Bibliometric and clinical reviews keep reporting this as a limitation. Dong et al., for example, analyzed artemisinin research using only English Web of Science records and acknowledged missing work from malaria-endemic regions. Screening is now being handed to fine-tuned language models, and those models learn from the inclusion decisions of past, mostly English, reviews. This proposal asks whether these models inherit the bias as a learned shortcut: do they treat "this record comes from a Chinese, Japanese or European registry, or is written in that registry's style or language" as evidence for exclusion? If they do, automation will make language bias cheaper to repeat and harder to see.
The proposal has two parts, one a measurement and one a fix. The measurement is a cross-registry audit. We train a screener on English-dominant sources and test it on trial records from other registries. The key test set is trials registered in two places, so the same study appears once in an English record and once in a non-English-registry record. A fair screener should make the same decision on both copies. The fix borrows a multi-task trick from computer vision. We add a second, auxiliary output that predicts which registry and language a record came from, and we force it to be geometrically orthogonal to the screening output, so that the screening decision cannot read source information. The source becomes a nuisance variable the model is forced to separate out, instead of a cue it can quietly use.
Core Claim
We claim that screening models fine-tuned on review-derived labels carry a measurable language exclusion gap: lower recall on eligible studies whose record of origin is a non-English registry, even when the trial is the same. Standard same-distribution test splits cannot reveal this gap. Only leave-one-registry-out validation exposes it. We further claim that a Compositional Feature Alignment-style auxiliary source head closes most of the gap without lowering English screening accuracy. Compositional Feature Alignment is a two-stage method that first learns a class head and a domain head constrained to be orthogonal, then fine-tunes the encoder to place features into those separate subspaces.
The novelty is narrow. The orthogonality method comes from Enhancing Compositional Generalization via Compositional Feature Alignment, which separated object class from visual domain on image benchmarks and noted that text lacks clean domain labels. Registry of origin and record language are exactly the clean discrete labels that limitation called for. The new parts are the claim that screening models use source as a shortcut, the paired-record test that measures it, and the use of a multi-task auxiliary head as a control on bias rather than as a source of extra supervision.
Relation to Prior Work
Two papers motivate the gap. Bio-SIEVE instruction-tunes 7B models on 7,330 Cochrane reviews for include/exclude screening and finds a "multi-task penalty": adding generative PICO extraction (Population, Intervention, Comparison, Outcome) and rationale generation hurt screening, which the authors trace to hallucinated extractions. It works only on English Cochrane abstracts and tests on its own distribution. TrialPanorama aggregates 1.66M trial records from 15 sources, including CHICTR, EUCTR and JAPIC, and trains multi-task 8B models across eight tasks including study screening. However, it evaluates through English queries and never trains on one registry and tests on another. Our work is the cross-registry evaluation that TrialPanorama's data makes possible but does not run.
Two papers constrain the design. Bio-SIEVE's penalty suggests that auxiliary tasks should be discriminative heads on shared features, not generated text, and our design follows that. Multilingual Clinical NER found that, for token-level clinical entity tagging, zero-shot transfer with a multilingual encoder beat both translate-train and translate-test. That result supports the expectation that a multilingual backbone can transfer. It also warns that translation pipelines compound errors. It does not tell us whether the same holds for document-level eligibility decisions, so translate-test stays a live baseline.
Two papers supply methods. Compositional Feature Alignment supplies the orthogonal-heads procedure. RotoGrad reduces conflict between task gradients by reweighting gradient magnitudes and learning per-task rotations of the shared features, and validated this only on vision. We use it as a competing way to combine screening with a PICO auxiliary task. Finally, Square One Bias in NLP found that about 70% of 461 ACL 2021 oral papers evaluated only on English. Sources of bias in AI showed that clinical AI datasets are concentrated in the US (40.8%) and China (13.7%). Both frame why an English-only screening leaderboard is the wrong yardstick.
Research Program
Data construction. We build the evaluation from TrialPanorama records. The first step is to verify whether registry entries keep their native-language fields. If the harmonized schema stores only English or translated fields, the nuisance becomes registry formatting and translation artifacts. That is still worth measuring, but it is weaker, and in that case we would re-attach native text from the source registries.
Paired test set. Many trials are registered in two places, typically ClinicalTrials.gov plus CHICTR, EUCTR or JAPIC. We link these copies using secondary identifiers and sponsor protocol numbers. This yields a paired test set in which the study is held fixed and only the record source varies.
Labels. Screening labels come from TrialPanorama's screening task, which uses inclusion in published reviews. These labels carry the very bias under study, because a non-English trial that a past review never searched for appears as a negative. We therefore have bilingual annotators re-label a few hundred non-English-registry records against review criteria. This is analogous to Bio-SIEVE's 108-sample "Safety-First" set, and it serves as clean ground truth.
Reviews that legitimately depend on geography. Some reviews restrict study population by country. In those reviews, geography is a valid criterion, and the model should not become blind to population country. It should only be blind to record source. We tag such reviews and report results on them separately.
Model. The backbone is an 8B open model fine-tuned with low-rank adapters, which matches the scale of both Bio-SIEVE and TrialPanorama. Screening uses a linear head on the final hidden state, which gives Compositional Feature Alignment a feature space to act on. Training follows two stages:
- With the backbone frozen, we train a screening head $W_1$ and a source head $W_2$, where $W_2$ predicts registry and language. We impose the constraint $W_1 W_2^\top = 0$, meaning no direction the screening head reads is shared with the source head.
- We freeze both heads and fine-tune the adapters so that features fit both heads.
Training uses every registry except one held-out registry, rotated across folds.
Diagnostic probe. To check that source information has actually left the screening decision, we fit a linear probe that predicts source from the features after projecting them onto the row space of $W_1$. The probe should fall to chance accuracy.
Baselines
Naive reference. The do-nothing reference is a single-task screener: the same backbone, adapters and data, with only the include/exclude head. This is the analogue of Bio-SIEVE's best configuration. Its single-task Guanaco-7B reached 0.82 accuracy on the Safety-First set, against 0.60 for zero-shot ChatGPT (Bio-SIEVE). We also evaluate our models on Safety-First as an English cross-dataset check, because it is Cochrane-derived and not TrialPanorama-derived. Matching 0.82 there shows the method costs nothing in English.
Strongest existing method. The strongest existing system is TrialPanorama's multi-task screener (Llama3-8B-TP or Qwen3-8B-TP), which is trained with supervised fine-tuning and then reinforcement learning with verifiable rewards. It is the strongest competitor because it was trained on these exact registries across eight related tasks. The file reports only relative gains over Llama-3-70B (67.6% in study screening). No reported absolute screening number was found on Lacuna for TrialPanorama, so we will measure it ourselves under our held-out-registry folds. If released weights include the held-out registry in training, we retrain its supervised stage on our folds with the same data and the same number of tuning trials. Beating it means a smaller language exclusion gap and better paired consistency, at equal or better English recall.
Ablations. Three ablations isolate the key ingredient:
- A source head trained jointly without the orthogonality constraint. Plain multi-task source prediction could increase source reliance, so this tests whether orthogonality is what matters.
- A source head attached through gradient reversal, which is the classic adversarial way to remove domain information.
- Screening plus a discriminative PICO head combined with RotoGrad. This tests whether generic gradient-conflict resolution helps at all.
Compute and tuning parity. All variants share the backbone, adapter rank, token budget and a fixed hyperparameter search of equal size.
Why not just…?
Why not just translate everything into English and screen with an English model? This is a real threat, because it removes language from the input entirely. Multilingual Clinical NER found translate-test weakest for large encoders because errors compound, but that finding was for token tagging. Translation does not remove registry-format cues, and it may introduce its own "translationese" signal. We run translate-test as a full baseline on the paired set. If it closes the gap as well as the orthogonal head, we will report that the cheap fix suffices.
Why not just oversample non-English registries in training? If the gap comes from too few positive examples rather than a learned cue, reweighting should fix it, and our method would be unnecessary. The paired set separates the two explanations. Reweighting changes how often source appears, but on identical studies only the removal of the cue should restore agreement between the two copies. We include an oversampled single-task baseline, and we will say plainly if it matches.
Why not just use a large general model zero-shot? In Bio-SIEVE, zero-shot ChatGPT was too lenient (0.60 accuracy), and TrialPanorama's 8B specialists beat a 70B general model. A zero-shot frontier model is still worth measuring for the gap, since it was not trained on biased review labels. If it shows no gap, that suggests fine-tuning on review labels is what induces the gap, which is itself a finding.
Why not adversarial domain removal instead of orthogonal heads? Gradient reversal is the standard tool, and it is in our ablations. It is known to be unstable, and the file notes that enforcing alignment can discard predictive signal specific to sub-populations. The comparison that settles the choice is English recall and accuracy on the geography-restricted reviews, where over-aggressive invariance should visibly hurt.
Proposed Experiments and Decisive Checks
The first experiment is the audit, and it can falsify the whole premise. We measure the single-task screener's recall on eligible studies by registry of origin, at a threshold set to 95% recall on English validation data. We also measure paired consistency, defined as the fraction of dual-registered pairs that receive the same decision. If the gap is within noise and consistency is near perfect, there is no learned exclusion cue to remove. We would then publish the audit and the paired benchmark as the contribution and drop the method.
If a gap exists, the decisive comparison is on held-out registries. We compare paired consistency, recall on the bilingually annotated non-English set, and work saved at 95% recall (the share of records reviewers can skip while still catching 95% of eligible studies). This comparison covers the single-task baseline, TrialPanorama's model, translate-test, oversampling, gradient reversal, unconstrained multi-task training, and the orthogonal head, alongside English accuracy on Safety-First.
The method succeeds only if all of the following hold:
- It raises paired consistency and non-English recall beyond every simpler fix.
- It does not drop below 0.82 on Safety-First.
- The source probe on screening-head features falls to chance.
- It does not lose accuracy on geography-restricted reviews.
If translate-test or oversampling matches it, the honest conclusion is that the bias is real but cheaply fixable. That result would still change how screening tools should be validated.
References
- Worldwide Research Trends on Artemisinin: A Bibliometric Analysis From 2000 to 2021
- Enhancing Compositional Generalization via Compositional Feature Alignment
- Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation
- Developing Large Language Models for Clinical Research Using One Million Clinical Trials
- Multilingual Clinical NER: Translation or Cross-lingual Transfer?
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold
- Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review