A living systematic review is a meta-analysis that is re-run automatically each time a new clinical trial appears. Pooling many trials averages out random noise. It does not remove systematic flaws such as broken randomization or unblinded outcome assessment. Each living update can therefore make a biased pooled effect look more precise without making it more correct. Automated tools can now judge a trial's "risk of bias," meaning whether its design and reporting are trustworthy. These tools are scored on per-item classification accuracy. Nobody checks whether their mistakes actually change what the living review concludes, and that is what matters. We propose to build and test a pipeline that carries an automated bias assessment's uncertainty into the pooled estimate. The same pipeline sends the scarce human reviewer only to the trials whose bias label could flip the review's conclusion.
The two-sentence version: errors in automated bias assessment matter only when they change a review's verdict, and most of them do not. We should therefore measure and spend human checking effort at the level of the conclusion, by replaying real Cochrane reviews as if they had been living all along and asking how many human checks are needed to keep the automated verdict matching the expert one.
Core Claim
We claim that a living meta-analysis can reach conclusion-level agreement with expert-curated reviews at a fraction of the human verification cost. This requires two changes to current practice. First, each trial's probability of high risk of bias enters the pooled estimate directly, rather than being thresholded into a hard label. Second, human checks go to trials ranked by how much verifying them would change the decision, not by how unsure the classifier is. A secondary, narrower claim is that domain-adapted models help mainly in finding the methods-section evidence for a bias judgment in a long full text. They help much less with the judgment itself, and only that finding step is worth specializing.
The motivating evidence comes from the quality-inheritance literature. Rahim Fazal et al. show that trials omitting blinding or proper randomization inflate treatment effects. Negin Hesam‐Shariati et al. show that pooling such trials yields narrow confidence intervals around a biased effect. The authors of PALM concede that even large multivariate network meta-analysis inherits whatever bias sits in its input trials. Together these establish that the bias label of each trial is a load-bearing input, not decoration.
What Exists and What Is Missing
Three systems are close to this proposal, and each does a different part.
The first two are automated bias assessors. RobotReviewer predicts risk-of-bias labels from full-text PDFs and extracts supporting sentences, trained by distant supervision from Cochrane reviews. It runs 5–10 points below human experts and does not feed any pooled estimate. RoBGuard improves LLM risk-of-bias judgments with question reformulation, table and flow-diagram parsing, and multiple reviewer personas. It too treats each trial in isolation.
The third group is living pipelines that pool or ingest without appraising. The living meta-review interface over the Cooperation Databank re-runs metafor pooling on demand, but assumes every study in its knowledge graph was already vetted by hand. Trialstreamer ingests about 142 new trials per day with 94–97% recall for trial identification. It works mainly from abstracts, where most bias-relevant details do not appear.
The RoBBR benchmark supplies the key diagnostic. Frontier models reason well once handed the supporting sentences, but they struggle to retrieve those sentences from long full texts. Fine-tuning Llama-3-8B improves that retrieval.
No paper in this set evaluates automated bias assessment by its effect on a pooled verdict. None allocates human checking effort by that effect.
The proposal also respects a constraint from NLI4CT. There, biomedical pre-training (BioBERT, BioMegatron) peaked at F1 0.644 on cross-trial inference, barely above a 0.626 baseline that read only the hypothesis statements. This is strong evidence against building the proposal around a specialized model doing the reasoning. We therefore confine domain adaptation to the retrieval step, where RoBBR suggests it pays.
Research Program
Retrospective living replay. We will take Cochrane reviews with study-level outcome data and expert risk-of-bias judgments, the same source used by RoBBR and RoB-Domain. We order each review's included trials by publication date and feed them one at a time to the pipeline. This simulates a living review that was started when the first trial appeared.
At every step we compute two verdicts. The reference verdict uses the Cochrane experts' bias labels. The pipeline verdict uses automated assessments plus whatever human checks have been bought so far.
A "verdict" here means three things together: the sign of the pooled effect, its statistical significance, and whether the result survives the standard sensitivity analysis restricted to low-risk trials. These are the quantities a guideline panel acts on.
To isolate bias assessment from data extraction, the main condition uses expert-extracted effect sizes. A second condition swaps in automated extraction in the style of the dual-LLM extraction workflow. This shows how the two error sources combine.
Probabilistic bias weighting. The assessor outputs, for each trial $i$ and bias domain, a calibrated probability $p_i$ that the trial is at high risk. Instead of including or excluding trials by a threshold, we compute the pooled estimate as an expectation over label configurations. We do this by Monte Carlo sampling of labels from the $p_i$, rerunning the low-risk sensitivity analysis on each sample, and reporting both the expected verdict and the probability that it differs from the hard-label verdict. Calibration is fitted on held-out reviews.
Decision-aware routing of human checks. Given a budget of human checks, we rank the unverified trials by
$$ V_i = p_i (1 - p_i)\, \lvert D(\text{with } i) - D(\text{without } i) \rvert , $$
where $D$ is the decision statistic of the current analysis (the z-score of the low-risk-only pooled effect against the significance threshold). The factor $p_i(1-p_i)$ is the label's variance under the model. The absolute difference is how much the decision moves if trial $i$ enters or leaves the low-risk set. A trial the model is unsure about but which barely affects the pooled estimate scores low, as does an influential trial whose label is near-certain. In the replay, a human check reveals the Cochrane label.
Retrieval specialization. We fine-tune a small biomedical encoder and a Llama-3-8B model on RoBBR's support-sentence retrieval task. Their outputs serve as the evidence passed to a general reasoning model. We compare this against having the general model read the full text itself. This is the only place domain-specific modeling enters, and it is tested as a component, not assumed.
Baselines
The naive reference point is pooling every ingested trial with no bias assessment, which is effectively what an abstract-only living feed provides. It shows how often quality inheritance actually flips verdicts in the replay. If that rate is near zero for most reviews, the problem is smaller than the literature implies, and we will say so.
The strongest existing assessor is RoBGuard run on Llama-3-8B. It reports item 5.2 accuracy of 0.91 against 0.34 for base Llama, and domain 3 (missing outcome data) F1 of 0.93 against 0.86, on the RoB-Item and RoB-Domain sets of RoBGuard. We will re-implement it on the same full texts and the same backbone. Its hard labels will feed the identical replay and sensitivity analysis.
The strongest zero-shot alternative is GPT-4o or Sonnet-3.5 prompted per bias domain. On RoBBR they achieve comparable macro-F1, but no reported numerical F1 value was found on Lacuna for either model. We will therefore measure it ourselves under the same prompts and token budget.
A RobotReviewer-style classifier, reported 5–10 points below experts, gives a cheaper non-LLM reference.
For routing, we compare decision-aware routing against random checks and against confidence-based checks (largest $p_i(1-p_i)$ alone, the usual active-learning default). All three get identical budgets.
Two ablations isolate the key ingredients. The first replaces probabilistic weighting with thresholded labels while keeping routing fixed. The second runs routing with oracle probabilities, which separates the value of the ranking rule from the quality of the calibration.
Beating the baselines means one of two things. Either the pipeline reaches a fixed conclusion-agreement level, such as matching the expert verdict at 95% of replay steps, with significantly fewer human checks than confidence-based routing on top of RoBGuard labels. Or, at equal budget, it reaches higher agreement, with confidence intervals computed over reviews rather than over trials.
Why not just…?
Why not just run a frontier model zero-shot and apply the standard low-risk sensitivity analysis? This is the real threat. If modern models are accurate enough, the hard-label verdict will already match the expert one almost everywhere, and routing adds nothing. The RoBBR finding that frontier models fail at locating support sentences in full text suggests systematic errors on exactly the trials where methods details are buried. Still, a suggestion is not a result. The replay settles it directly. If zero-shot hard labels already reach the agreement target with zero human checks on most reviews, we will report that the contribution reduces to an evaluation protocol showing the cheap approach suffices.
Why not just send the human reviewer the trials the model is least sure about? Confidence-based review ignores influence. A small trial with an ambiguous blinding description may be uncertain yet unable to move the verdict. A large trial with a near-certain label may move it a lot, and both cases are common in practice. This baseline is run head-to-head, and the oracle-probability ablation shows whether any advantage comes from the ranking rule itself.
Why not just pool everything and trust that more trials wash out the problems? The work of Rahim Fazal et al. and Negin Hesam‐Shariati et al. shows that systematic bias does not shrink with the number of trials. Bin Liu shows that after-the-fact corrigenda can fix arithmetic but not design flaws in the underlying trials. The naive baseline quantifies how often this matters in practice, rather than assuming it does.
Why not just pre-train a biomedical model end to end and let it do the whole assessment? NLI4CT shows biomedical pre-training barely beating a model that reads only the hypothesis statements on structured clinical inference. That argues against expecting domain pre-training to supply judgment. We test domain adaptation only on retrieval, and we report the full pipeline with and without it.
Why not just filter by journal prestige? AutoConfidence predicts Q1-journal acceptance with AUROC 0.747. Venue is a proxy for internal validity, and published trials routinely carry high risk of bias. We include a venue filter as a cheap reference in the replay.
Proposed Experiments
The decisive experiment is the replay across a set of Cochrane reviews. We plot conclusion agreement against the human-check budget for each assessor-and-routing combination.
The headline test asks whether decision-aware routing with probabilistic weighting dominates confidence-based routing on RoBGuard labels across budgets. A second test asks whether verdict flips under the naive baseline concentrate in a minority of reviews, which would show where living reviews most need appraisal.
A secondary experiment uses RoBBR's support-sentence retrieval task. It checks whether the fine-tuned biomedical retriever improves end-to-end bias accuracy and replay agreement when paired with a general reasoner. A third experiment swaps in automated effect extraction. It measures whether extraction or bias assessment errors dominate conclusion drift, which tells future living systems where to spend effort.
We will also report the time lag between a trial's arrival and the pipeline verdict converging to the expert verdict. This makes concrete what "living" buys, compared with the 2.5–6.5 year lag cited by the Human Behaviour-Change Project.
References
- Epidermal Growth Factor Outperforms Placebo in the Treatment of Diabetic Foot Ulcer: A Meta-Analysis
- The analgesic effect of electroencephalographic neurofeedback for people with chronic pain: A systematic review and meta‐analysis
- PALM: Patient-centered treatment ranking via large-scale multivariate network meta-analysis
- Automating Biomedical Evidence Synthesis: RobotReviewer
- RoBGuard: Enhancing LLMs to Assess Risk of Bias in Clinical Trial Documents
- Living Meta-Review Generation for Social Scientists: An Interface and A Case Study on Human Cooperation
- Trialstreamer: A living, automatically updated database of clinical trial reports
- Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark
- NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports
- Collaborative Large Language Models for Automated Data Extraction in Living Systematic Reviews
- Corrigendum: Improvement of Sepsis Prognosis by Ulinastatin: A Systematic Review and Meta-Analysis of Randomized Controlled Trials
- Enabling inclusive systematic reviews: incorporating preprint articles with large language model-driven evaluations
- The Human Behaviour-Change Project: harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation