Pair-Blind Literature Pre-Training for Cold-Start Synthetic Lethality

Two genes are synthetically lethal when a cancer cell survives losing either one alone but dies when it loses both. The textbook case is PARP inhibitors, which kill BRCA-deficient tumors. There are about 200 million human gene pairs, and lab screens cannot test more than a sliver of them. Computational prediction is therefore the main way to choose which pairs to test. It fails where it matters most: predicting pairs in which neither gene has any known synthetic-lethal partner. Language models pre-trained on biomedical papers look like the obvious fix, because they have "read" about every gene. That appeal hides a problem. A model trained on PubMed may already have seen the sentence reporting that two genes are synthetically lethal. If so, its apparent skill on "unseen" genes is recall rather than discovery.

This note proposes to pre-train a gene-aware language model on a pair-blind, time-sliced literature corpus. Pair-blind means every passage that co-mentions a known synthetic-lethal pair is removed. Time-sliced means only papers published before a cutoff year are used. The model is then tested on pairs where neither gene appears in the training labels and whose lethality was first reported after the cutoff. The claim is twofold. Literature-derived gene representations carry transferable biological signal for these cold-start pairs beyond what gene-dependency screens already provide. And only the portion of that signal that survives leakage control should count as evidence of discovery. We build the corpus, the model, and the evaluation protocol, and we measure how much of the text advantage is real.

Why this gap, and what is already done

Cold start is the central failure. Benchmarking machine learning methods for synthetic lethality prediction in cancer compares 11 methods under three splits: seen genes (CV1), one unseen gene (CV2), and both genes unseen (CV3). Many graph models, GCATSL among them, collapse in CV3 because they memorize the local graph neighborhood of each gene. The multi-view graph autoencoder SLMGAE holds up best. This paper motivates the proposal and supplies its evaluation protocol.

Three lines of work already bring language or attention models to the problem, and each stops short of the cold-start question. KR4SL reasons over knowledge-graph paths and adds frozen text embeddings of gene and Gene Ontology descriptions from CODER, a biomedical BERT. Its ablations show that text helps ranking. It never asks whether that text already encodes the answer, and it does not report a both-genes-unseen split. NexLeth fine-tunes BioGPT and BioMedLM to write explanations for 707 already-known pairs, so it does not predict new ones. MSGT-SL applies self-attention to omics features from BioGRID, CCLE and DepMap for specific cell lines, trained from scratch with no text at all.

Our proposal differs from all three in the same respect. We treat the literature prior as the object under test, and we build it so that it cannot contain the labels.

Two papers constrain the design. Zero-shot evaluation reveals limitations of single-cell foundation models shows that masked-token pre-training on 33 million cells lost to simple highly-variable-gene selection. That result warns that generic language-model objectives do not automatically yield biology, so our objective has to target gene function directly, and our baselines have to include cheap non-neural features. BioBART supplies two method lessons: continued pre-training on PubMed pays off, and inherited general-purpose tokenizers break up technical names badly. We therefore give each gene symbol its own token.

Core Claim

A gene representation learned only from pre-cutoff literature with synthetic-lethal co-mentions scrubbed out will improve prediction for both-genes-unseen pairs over the best graph model and over dependency-screen features. It will also add signal on top of those features. A model trained on the same corpus without scrubbing will show an extra advantage concentrated on pairs published before the cutoff. That extra advantage is the signature of leakage, and it currently inflates any text-based evaluation.

Research Program

Corpus. We take PubMed abstracts published before a cutoff year T, with T chosen so that a substantial share of SynLethDB 2.0 human pairs were first reported later. SynLethDB 2.0 holds 50,868 pairs, including more than 16,000 recent additions from CRISPR screens; we use its curated references to recover publication dates wherever they are available.

Gene mentions are normalized to official symbols. We then delete every sentence that co-mentions both genes of any SynLethDB pair, whether that pair falls in training or test. This removes direct textual statements of lethality while keeping each gene's individual functional context: pathways, complexes, phenotypes. As a filtering control, we build a second corpus that deletes an equal number of sentences co-mentioning random non-lethal gene pairs, and a third corpus that is left unfiltered.

Model. We train a roughly 100M-parameter encoder from scratch on each corpus. Training from scratch avoids the residual leakage that comes with starting from web-trained weights. Each gene symbol gets a dedicated token. The objective is standard masked-token prediction plus a heavily up-weighted gene-masking task, in which the model must recover a masked gene symbol from its surrounding sentence. This forces the gene's embedding to summarize the contexts in which the gene appears. A gene's representation is its token embedding concatenated with the average of its contextual embeddings across the corpus.

Supervised stage. A small symmetric pair classifier takes the two gene vectors, optionally concatenated with DepMap co-essentiality and CCLE expression features. It is trained on SynLethDB labels.

Baselines

The naive reference point scores a pair by the correlation between the two genes' DepMap CRISPR knockout profiles. It needs no training, and it is available for every gene, including cold-start genes. Any method that cannot beat it on CV3 has learned nothing beyond co-essentiality.

The strongest existing cold-start method is SLMGAE. Benchmarking machine learning methods for synthetic lethality prediction in cancer reports an F1 of about 0.73 for it in CV3. The file does not state the negative ratio for that figure, so we will rerun SLMGAE under our exact splits and ratios rather than compare against the published number. That paper's full AUROC/AUPRC grid is not reported in the file. To count as beating SLMGAE, our model must achieve higher area under the precision-recall curve (AUPRC) at 1:50 negatives on post-cutoff CV3 pairs, with confidence intervals from five gene-disjoint resamplings that do not overlap.

A KR4SL-style baseline replaces our encoder with frozen CODER embeddings of gene descriptions feeding the same pair classifier. KR4SL reports a 78.1% NDCG@50 gain over NSF4SL in its transductive setting. No cold-start figure for it was found on Lacuna. This baseline represents "text helps" as practiced today, with no leakage control.

We also include GRSMF matrix factorization as a sanity check. The benchmark found it stable under class imbalance, but it structurally cannot score unseen genes, so it should fail CV3. A Random Forest on SynLethKG features is included too; SynLethDB 2.0 reports an AUROC of 0.938 for it on the full graph, but not in cold start. Finally, a zero-shot BioMedLM prompt tests whether an off-the-shelf generative model can rank pairs directly.

The key ablation trains the identical architecture on the three corpora (pair-blind, random-filtered, unfiltered). All methods share the same splits, the same negative sampling (both random negatives and the expression-informed negatives from the benchmark), and the same hyperparameter search budget of 50 trials on a gene-disjoint validation fold.

Why not just…?

Why not just use frozen CODER or another biomedical encoder, as KR4SL does? This is a real threat. On a leaky benchmark it may match or beat our model, precisely because it has read the answers. The experiment settles the question by splitting results into pre-cutoff and post-cutoff test pairs. If frozen CODER's advantage over our pair-blind model disappears on post-cutoff pairs, its apparent strength was recall. If it persists, frozen encoders are good enough and our custom pre-training is not needed. We would report that as a negative result, and the evaluation protocol would still stand.

Why not just use DepMap co-essentiality and expression features with a gradient-boosted classifier? These features exist for every gene, so they sidestep cold start entirely, and MSGT-SL shows that omics-driven attention is competitive. The single-cell result in Zero-shot evaluation reveals limitations of single-cell foundation models makes it plausible that such simple features beat learned text representations. We test this directly. The deciding comparison is omics-only against omics plus pair-blind text. If adding text does not raise AUPRC on post-cutoff CV3 pairs, the literature prior carries no complementary signal.

Why not just prompt or LoRA-tune BioMedLM? NexLeth found that zero-shot prompts produced poor explanations even for known pairs, and that knowledge-graph prompts were needed to reach FMR 0.14. More fundamentally, BioMedLM was trained on the full, unfiltered PubMed, so any success it shows is uninterpretable for discovery. We run it anyway as a leakage-exposed upper reference and report it separately.

Proposed Experiments

We use CV3 splits from the benchmarking protocol at 1:1, 1:10 and 1:50 negative ratios. Each CV3 test set is further partitioned by first-report date relative to T. Metrics are AUPRC as the primary measure, plus NDCG@50 and F1 for comparability with the literature. A secondary evaluation uses the K562 and Jurkat cell-line splits from MSGT-SL, for which only relative gains are reported (4.5% transductive, 3.8% inductive). We rerun its code under our splits.

We also run a direct memorization probe. We present each model with templated sentences of the form "loss of X is synthetically lethal with loss of [gene]" and measure how often the true partner is ranked first. The pair-blind model should score near chance on this probe. The unfiltered model's score measures how much the literature has leaked.

The decisive checks are three. First, the pair-blind model plus DepMap features must beat both SLMGAE and DepMap-only on post-cutoff CV3 AUPRC at 1:50. Second, the gap between the unfiltered and pair-blind models should be large on pre-cutoff pairs and near zero on post-cutoff pairs. Third, the random-filter control should perform like the unfiltered model, which shows that scrubbing hurts because it removes answers and not because it removes data.

If the first check fails while the second succeeds, the conclusion is that current text-assisted synthetic-lethality results largely measure recall. That finding would itself redirect the field toward omics priors.