Automated evidence synthesis uses language models to find, screen and extract clinical trials for systematic reviews and meta-analyses. In practice these models will be updated over time, as new trial records arrive. They will not arrive evenly. English-language records from PubMed and ClinicalTrials.gov come in large batches. Records from regional registries such as China's CHICTR, Europe's EUCTR and Japan's JAPIC come later, in smaller batches, and in different formats.
Continual learning research has shown that the order in which a model sees tasks can decide what it remembers and what it forgets. Research on linguistic publication bias has shown that leaving out non-English studies shifts the pooled estimates that clinical guidelines rely on. This note proposes to test a specific link between the two. We ask whether sequential updating of a screening model forgets or under-learns regional registries unevenly, depending on arrival order. If it does, the model is quietly recreating publication bias. This would happen even when reviewers deliberately include non-English sources.
The question is not settled by existing evidence. The closest study, Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning, found that the order in which languages were presented barely mattered on short-sentence tasks. Evidence screening differs from those tasks in two ways. The data sources are very unequal in size and structure, and a small per-registry error propagates into a biased pooled estimate. We therefore expect order effects to reappear. We also propose to measure them in units that matter to a meta-analyst, not in average accuracy.
Core Claim
The claim is that, for continually updated trial-screening models, order sensitivity shows up as registry-differential recall. Recall here means the fraction of relevant trials the model correctly keeps. This recall gap is large relative to seed noise. It depends mainly on whether small regional blocks arrive before or after large English blocks. It is not repaired by the standard fix of replaying a random sample of old data.
The effect is visible only if the evaluation is disaggregated. Performance must be reported per source, and the downstream consequence must be traced through to a pooled estimate. Averaged accuracy, which is how both continual learning benchmarks and evidence-synthesis benchmarks currently report results, would hide it.
We will test one targeted remedy. The remedy is registry-stratified replay with a per-registry recall floor: the replay budget is shifted toward whichever source's recall on a held-out probe set has dropped most. If the audit finds no effect beyond seed variance, that null result is itself useful. It would tell practitioners that continual deployment of screening models is safe with respect to this bias.
Research Program
The substrate is TrialPanorama. It aggregates 1,657,476 trial records from 15 sources: 48.7% from PubMed, 31.6% from ClinicalTrials.gov, and 19.6% from 13 other registries including CHICTR, EUCTR and JAPIC. It defines study search and study screening tasks, and its 8B domain-adapted models are the natural backbones. TrialPanorama trains these models once, offline, and reports aggregate results. We reorganise the same data into a stream of source blocks and train sequentially.
We will build a stream in which each block contains screening instances from one source family. The families are PubMed, ClinicalTrials.gov, CHICTR, EUCTR, JAPIC, and the remaining registries pooled together. We will compare three structured orders:
- Historical order: English sources first, regional sources last. This mirrors how indexing actually expanded.
- Reverse order: regional sources first, English sources last. This mirrors a system bootstrapped on local data and then flooded by English updates.
- Interleaved order: sources mixed over time.
We will also run a sample of random permutations to estimate order variance directly.
The mechanism we expect comes from the continual-learning literature. Cai et al. show that complex tasks arriving late disrupt earlier ones asymmetrically. Jin et al. show that early tasks anchor which representations the model builds. Together these predict that the harm will be asymmetric. A small regional block followed by a large English block should be overwritten more than the reverse.
One caveat shapes the design. TrialPanorama normalises records to shared ontologies, so some regional records may already be partly in English. Where original-language fields exist, we will keep them. Wherever we cannot separate "language" from "source format," we will report the effect as source bias rather than purely linguistic bias. The meta-analytic consequence is the same either way: some populations' trials drop out of the evidence.
To connect recall to effect sizes, we will run semi-synthetic meta-analyses. We take review topics where relevant trials span several registries. We assign each registry's trials effect sizes drawn with a registry-level offset, which is the kind of regional heterogeneity Qian et al. document: pooled eating-disorder prevalence was 1.89% in Western countries versus 0.22% in Asian ones. We then compute how far the pooled estimate built from each model's inclusions drifts from the estimate built from all relevant trials. This step makes explicit the assumption that regional offsets exist. We will vary the offset size, including zero, rather than claim a particular clinical magnitude.
Baselines
Joint training. The naive reference point is a model trained jointly on all sources at once, which is the TrialPanorama recipe. It has no order by construction and sets the recall each source should reach. For the strongest static systems, the file reports only relative figures: the 8B TP models beat Llama-3-70B by up to 73.7% in study search and 67.6% in study screening (TrialPanorama). No reported result was found on Lacuna for per-registry screening recall, or for any continual training on TrialPanorama. We will therefore produce these reference numbers ourselves.
Naive sequential fine-tuning. This is the lower bound. It shows the raw size of the order effect.
Uniform replay combined with learning-rate decay. This is the strongest multilingual continual learner in the file. Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning combines A-GEM, a method that stores past examples and blocks updates that would increase loss on them, with LR ADJUST, an exponential decay of the learning rate across tasks. With this combination it cut forgetting on MASSIVE slot filling from a backward-transfer score of −7.92 to −2.48, and on WikiAnn from −16.90 to −3.59. It is the most direct competitor because it was designed for exactly this multi-language stream and needs no architectural change.
Typology-scaled learning rates. LAFT-URIEL uses language-specific adapters and lowers the learning rate of the shared backbone when a new language is syntactically distant from those already seen.
Per-task adapters on a frozen backbone. Following Khan et al., this baseline is order-insensitive by construction. It tests whether isolating parameters simply dissolves the problem.
All methods share the same backbone, data, number of gradient steps, and replay budget measured in stored examples. Each receives the same small hyperparameter sweep, tuned on a validation split drawn from the first block only. This prevents methods from indirectly seeing future sources.
Ablation. The key ingredient is stratification guided by per-registry probes. The ablation that isolates it is our method with the same buffer size but uniform sampling. Beating the baselines means two things at matched budget. First, the method should shrink the worst-registry recall gap relative to joint training, and reduce order variance to within seed variance. Second, it should do this without lowering English recall by more than seed noise. As a concrete anchor for what "small order disparity" looks like, APD-Net reports an order-induced performance disparity of 1.79% versus 7.06% for EWC on Split CIFAR-100.
Why not just…?
Why not just retrain jointly from scratch whenever new trials arrive? For an 8B model this is feasible at modest scale. If organisations always did it, order would not matter, so this is a real threat to the relevance of the problem, though not to its existence. We treat joint training as the reference, and we report the compute ratio between retraining and continual updates as the stream grows. Our claim is conditional: when deployments update incrementally, which living reviews push toward, they need per-source auditing. If the recall gaps from continual updating are within seed noise of joint training, we will say the problem does not arise.
Why not just translate everything to English and use an English-only screener? Translation removes the language difference but not the differences in registry format, terminology and reporting norms. Zhichao Jin et al. document that Chinese journals follow distinct reporting conventions; for example, over 63% of RCTs in 2008 did not report sample-size estimation. Such differences would survive translation. We will run a translate-then-screen arm. If it closes the per-registry gap, the effect was linguistic and translation is the cheaper fix. If a gap remains, the effect lies in source structure.
Why not just use uniform replay? Uniform replay samples sources in proportion to their size. The largest sources are English, so the small regional sources that need rehearsal most receive the smallest share of the buffer. This argument is plausible but untested, and the uniform-replay ablation at matched buffer size settles it directly.
Why not just order by language family, or pick the order actively? The multilingual study found that family-based ordering was no better than random. Active selection in the style of Ruvolo et al. needs to preview labelled data from every candidate task, and in a real registry stream arrival order is not under the system's control. We include family-grouped order as one of our fixed orders, but we do not rely on controlling order.
Proposed Experiments and Decisive Checks
The first check is whether an effect exists at all. Across random permutations, is the variance in per-registry screening recall larger than the variance across three seeds with the order held fixed? If it is not, the headline is a null result that extends the multilingual study's finding to evidence screening.
The second check is asymmetry. Does reverse order, with regional sources first, lose more regional recall than historical order does? The complementary question is whether historical order leaves regional recall intact but English recall degraded.
The third check traces the downstream consequence. In the semi-synthetic meta-analyses, does the drift of the pooled estimate from its full-inclusion value grow with the regional recall gap? When the registry offset is set to zero, the drift should vanish. This serves as a built-in sanity check.
The fourth check is repair. Does stratified replay reduce worst-registry recall loss relative to uniform replay at equal buffer size? Does it bring the effect-size drift near the joint-training level? Does it beat both A-GEM with learning-rate decay and typology-scaled learning rates?
All metrics are reported per source, never only as an average, because the averaged number is exactly what hides the bias.
References
- Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning
- Developing Large Language Models for Clinical Research Using One Million Clinical Trials
- Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation
- Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning
- An update on the prevalence of eating disorders in the general population: a systematic review and meta-analysis
- Parameter-Efficient Finetuning for Robust Continual Multilingual Learning
- Lifelong Language Learning with Adapter based Transformers
- Scalable and Order-robust Continual Learning with Additive Parameter Decomposition
- A Retrospective Survey of Research Design and Statistical Analyses in Selected Chinese Medical Journals in 1998 and 2008
- Active Task Selection for Lifelong Machine Learning