Thousands of papers each year report that some gene, or a small set of genes, predicts cancer survival. Most of these signatures were found on one measurement technology, usually RNA sequencing of a public cohort. Many of them fail when someone measures the same genes on a different technology, such as an older microarray or a protein assay. Literature-synthesis systems read these papers and file each claim as a fact. Their notion of "faithful" means only that a claim matches its source paper. This proposal adds a second check that is external to the literature. Public repositories hold cohorts measured on several platforms, so for a large fraction of published biomarker claims we can automatically recompute whether the claim holds on a technology other than the one it was discovered on. That recomputed outcome becomes a training label. A biomedical language model trained on these labels can then estimate, from text alone, how likely a new claim is to survive a change of platform. This matters most for the many claims that no public cohort can yet check.
In two sentences: we turn cross-platform replication in public omics data into automatic supervision for a scientific language model. The model learns which reported biomarker findings are likely to be platform artifacts, and synthesis systems can use its score to down-weight them. This treats cross-platform validation as the source of ground truth, domain-specific pre-training as the modeling substrate, and literature overload as the application that needs a triage signal.
Core Claim
Current literature-synthesis pipelines measure truth only against text. Affinage uses a two-stage LLM reader and synthesizer to annotate all 19,293 human protein-coding genes. It reports that 97.95% of its claims are supported by the literature, which is a faithfulness measure and not a replication measure. Scaling Scientific Knowledge Discovery with Neuro-Symbolic AI grounds GPT-4 in a knowledge graph built from papers, so it too ends at the text. SciFact-Open shows why text-only truth is fragile: about 20% of well-studied claims have conflicting evidence in an open corpus, and 44% of evidence pairs mismatch in specificity. These papers motivate the gap. On the data side, cross-platform biomarker studies such as the lung adenocarcinoma four-gene model in Predicting the survival of patients with lung adenocarcinoma train on RNA sequencing from The Cancer Genome Atlas (TCGA) and validate on a microarray cohort from the Gene Expression Omnibus (GEO). They do this one signature at a time, by hand, with no connection to automated reading. The network-based organoid work in Network-based machine learning in colorectal and bladder organoid models validates across preclinical and patient settings, but it draws its biology from curated databases rather than from text.
Our claim is specific and falsifiable. We first construct replication labels automatically at scale. We then train a domain-pre-trained biomedical model on those labels. The resulting model predicts held-out cross-platform replication better than three alternatives: study-design metadata, a frontier LLM asked to judge replicability, and the same model fine-tuned without replication-specific training. It should also do so on cancer types and platform pairs it never saw. We are not claiming that text can replace reanalysis. Where reanalysis is possible, it is the oracle. The model is meant for the claims that reanalysis cannot yet reach.
Research Program
The first component is a claim corpus. We extract structured biomarker claims from PubMed abstracts. Each claim records the gene or signature, cancer type, outcome (overall or progression-free survival), direction of effect, discovery platform, discovery cohort, and reported statistics. We reuse Affinage's design of a skeptical reader kept separate from any synthesis, plus a deterministic audit layer that rejects claims whose gene symbols or cohort identifiers do not appear in the source. Affinage reports that naive LLM queries systematically overclaim, and this design is its fix. We restrict the corpus to abstracts. This follows Affinage's finding that full text added 16% more findings at 15 times the cost without better summaries. Abstracts also match the realistic triage setting of a reader facing thousands of papers.
The second component is automatic labeling. For each claim, we choose validation cohorts on a platform different from the discovery platform. If a claim came from TCGA RNA sequencing, it is validated on GEO microarray cohorts for the same cancer, and the reverse for microarray discoveries. We explicitly exclude any cohort the paper itself reports using, such as GSE26939 for the lung adenocarcinoma model. Replication means that the claimed direction holds and that a prespecified survival association test passes in the orthogonal cohort. For multi-gene signatures, we refit the published weights where the abstract gives them and otherwise use an unweighted score. Claims with no orthogonal cohort are labeled untestable and become the deployment set. Feature alignment across platforms is a known weak point: probe annotations, gene aliases, and isoforms all interfere, and the research file lists this as an open problem. We therefore use a conservative gene-to-probe mapping, drop ambiguous genes, and report every result under two alternative mappings so the labels' sensitivity is visible.
The third component is the model. We start from BioBART, which continued pre-training of BART on 41GB of PubMed abstracts using text infilling (filling in masked spans). BioBART also found that shuffling sentence order hurts biomedical performance. Following that constraint, we add a short stage of continued pre-training that masks only the results and conclusion spans of biomarker abstracts while keeping sentence order intact. The model must reconstruct the direction and strength of findings from the design and methods text. After this stage, we train a classification head on the automatic replication labels, conditioned on the tagged discovery and target platforms.
The research question is which training signal does the work. The domain pre-training is a supporting ingredient. Our design choices rest on the evidence that base domain pre-training dominates adaptation method in A Comparative Study of PEFT Methods and on the drowning effect reported in MonoCoder. Neither result directly tests replication prediction, so both remain hypotheses here.
Baselines
The naive reference is the base rate: predict that every claim replicates, or predict the majority label. We report the base rate itself so that the other gains can be read against it. We are not aware of any published replication rate for literature-derived survival biomarkers across platforms, and there is no reported result found on Lacuna for this prediction task. The benchmark is new, so every number in it will come from our own runs.
The strongest non-neural competitor is a logistic regression on study-design metadata. Its inputs are discovery cohort size, number of genes, reported p-value or hazard ratio, whether multi-step regularization was used (univariate Cox, then LASSO, then multivariate Cox, which Identification of RNA binding protein genes associated with colorectal cancer reports as needed for cross-platform survival), cancer type, and platform pair. It is the strongest competitor because much of replicability may simply be statistical power. The audited extractor fills all of these features, so this baseline sees exactly the information our model sees. Beating it requires a statistically significant gain in held-out AUROC and in calibration error across five seeds and all held-out splits.
The strongest LLM competitor is an Affinage-style frontier model. It receives the same abstract and a structured prompt asking it to estimate cross-platform replication, with the platform pair stated. We also test a few-shot variant with labeled examples drawn only from training folds. We tune its prompts with the same number of validation evaluations used for our hyperparameter search.
Two ablations isolate the key ingredient. The first trains BioBART's classification head on the replication labels without results-infilling continued pre-training. The second replaces the platform-derived labels with text-derived pseudo-labels: whether later citing abstracts agree with the claim, extracted by the same pipeline. The second ablation tests whether supervision from outside the literature adds anything beyond the literature's own consensus. General-domain BART under the identical recipe measures the value of domain pre-training. All neural variants share data, compute budget, and tuning budget.
Why not just…?
Why not just reanalyze every gene in TCGA and GEO directly and skip the literature? For claims that public cohorts can test, we do exactly that, and the result is the label. The threat is real if nearly every claim turns out to be testable. The proposal's value rests on the untestable set: claims about rarer cancers, subtypes, stages, or platforms such as targeted proteomics, where no orthogonal public cohort exists. We will report what fraction of extracted claims are untestable. If that fraction is small, the triage model matters little, and we will say so and reframe the contribution as the labeled benchmark alone.
Why not just prompt a frontier LLM, as Affinage does? Affinage shows such models extract claims faithfully, but faithfulness to text is the wrong target. A model that reads "validated in an independent cohort, P = 0.021" has no access to whether the finding holds on a different technology. The trial summarizer in Automatically Summarizing Evidence from Clinical Trials reached only about 52% F1 on whether a treatment worked, which suggests fluent models misjudge effect direction even inside the text. The frontier-LLM baseline settles this empirically. If it matches our model, the replication-trained model is unnecessary.
Why not just use study-design metadata? This is the most serious threat. The lung adenocarcinoma model, for example, worked in early stages and was "less effective" in stages III and IV. That kind of scope limitation may be visible from simple features, or it may be buried in wording. If the metadata regression matches the language model, the honest conclusion is that replicability is mostly power and design, and that structured extraction plus a small regression is the right tool. We would report that as the result.
Why not just fine-tune PubMedBERT or BioBART on the labels without the extra pre-training? This is the first ablation. If it matches the full model, results-infilling pre-training is dropped from the recipe.
Proposed Experiments
The main experiment uses four held-out splits. The first holds out cancer types. The second holds out platform pairs: train on RNA sequencing to microarray, test on microarray to RNA sequencing. The third is a temporal split in which the model trains on older claims and is tested on newer ones. The fourth holds out entire signatures, so no gene set appears in both training and test. Metrics are AUROC, expected calibration error, and precision among the top-scored claims. The last one matters because a triage tool is judged by the claims it flags first.
A small expert audit of 200 labels will estimate how much label noise comes from the automatic pipeline itself, particularly from feature alignment.
A secondary downstream check asks whether replication-aware scores help synthesis. On the SciFact-Open conflicting-evidence claims, we re-rank evidence by replication score where the claim is biomarker-related. MultiVerS drops from 72.5 to 52.4 F1 in the open setting, and we will measure whether this re-ranking recovers part of that drop. On PubMedQA-labeled questions about prognostic markers, we compare against BioBERT's reported 68.1% accuracy, the 55.2% majority baseline, and the 78.0% human ceiling (PubMedQA). We treat these transfers as exploratory, since neither benchmark was built for replication.
The decisive result is a model that clears the metadata regression and the frontier-LLM judge on held-out cancer types and held-out platform pairs, while the citation-consensus ablation does not. If only the first half holds, the replication labels still yield a useful public benchmark, but the claim that platform-derived supervision teaches something beyond the literature fails.
References
- Affinage: Genome-Scale Mechanistic Gene Annotation from the Published Literature
- Scaling Scientific Knowledge Discovery with Neuro-Symbolic AI and Large Language Models
- SciFact-Open: Towards open-domain scientific claim verification
- Predicting the survival of patients with lung adenocarcinoma using a four‑gene prognosis risk model
- Network-based machine learning in colorectal and bladder organoid models predicts anti-cancer drug efficacy in patients
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- A Comparative Study of PEFT Methods for Python Code Generation
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- Identification of RNA binding protein genes associated with colorectal cancer by bioinformatics analysis
- Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges
- PubMedQA: A Dataset for Biomedical Research Question Answering