Biological pathway databases such as KEGG, Reactome and WikiPathways describe the same processes (apoptosis, glycolysis, insulin signalling) but were built independently. Linking their entries, "this KEGG pathway is the same thing as that Reactome pathway," is a daily curation chore. Automatic matchers are always tested on the same database pair they were tuned on. This proposal asks a deployment question that has not been measured. If you build or choose a matcher using one pair of databases, does it still work, and does it still beat the alternatives, when pointed at a pair it has never seen?
We will build a leave-one-database-out benchmark for pathway alignment and run a deliberately diverse set of matchers through it. The matchers range from plain string matching to biomedical language models trained in different ways. Our central hypothesis is that two cheap signals transfer better across databases than a fine-tuned pair classifier. The first is name embeddings from a model pre-trained to cluster synonyms. The second is a gene-overlap score that tolerates differences in where each database draws a pathway's boundary. If this is right, the matcher you should deploy on a new database is not the one that wins in-domain, and current single-pair benchmarks rank methods misleadingly.
Core Claim
Three findings in the literature point the same way but have never been put in one experiment. Consistency, comprehensiveness, and compatibility of pathway databases compared longest-common-substring name matching with gene-overlap and gene-pair-overlap matching across KEGG, Ingenuity and WikiPathways. Name matching won because the databases disagree badly on membership: gene agreement ranged from 0% to 88%, and for apoptosis only 30–46%. Predicting instances of pathway ontology classes for pathway integration mapped Reactome entries into the Pathway Ontology with static word embeddings and a recurrent network. It names the neglect of pathway contents as its main limitation, and its transfer to HumanCyc and WikiPathways was never scored. The third finding is a tension between two language-model approaches to alignment. Biomedical ontology alignment with BERT reports that off-the-shelf BERT embeddings compared by cosine similarity align poorly and must be fine-tuned as a pair classifier. SAPBERT instead pre-trains PubMedBERT with a metric-learning loss over UMLS synonyms, so that plain cosine similarity links entities zero-shot across six independent datasets.
Our claim has two parts. The first is that the Soh et al. failure of content matching comes from using a symmetric overlap. When one database splits a process into sub-pathways that another lumps together, symmetric overlap is small even though one gene set nearly contains the other. An asymmetric containment score should recover that signal. For gene sets $A$ and $B$ of two candidate pathways, we use
$$C(A,B) = \frac{|A \cap B|}{\min(|A|,|B|)},$$
which is near 1 when the smaller pathway sits inside the larger, regardless of how much extra the larger one carries. The second part is that self-aligned name embeddings will lose less accuracy when moved to an unseen database pair than a fine-tuned cross-encoder. We expect this because the self-aligned model learned synonymy from UMLS as a whole, not from one pair's labelling quirks. Combining the two signals in a small learned scorer should give the best cross-pair accuracy. It should also make algorithm rankings measured on a source pair predict the rankings on a target pair, which they currently may not.
Research Program
The first component is the benchmark itself. We take KEGG, Reactome and WikiPathways, plus the Pathway Ontology as a hub. All gene identifiers are normalised to NCBI Gene IDs, following the convention of PathwayAPI from Consistency, comprehensiveness, and compatibility of pathway databases. Where KEGG content must be read in its own format, we use the KGML converters of Java DOM Parsers to Convert KGML into SBML and BioPAX Common Exchange Formats. We only need gene membership and names, which avoids the semantic losses those authors document for reaction-level translation.
Gold mappings come from two sources. One is the curator-reviewed Reactome–Pathway Ontology mappings used by Predicting instances of pathway ontology classes for pathway integration. The other is new double-annotated samples for the other pairs, following that paper's protocol of reviewing a random subset of about a hundred pathways and measuring inter-rater agreement. Mappings are typed as equivalent, broader or narrower. This typing matters because boundary disagreement is the phenomenon under study. The protocol trains or tunes on one database pair and evaluates on every other pair. Any database involved in the evaluation pair is excluded from all tuning data, including any additional pre-training data.
The second component is the set of competing matchers, chosen to span different inductive biases rather than to pad a leaderboard. They are: longest-common-substring name matching; bag-of-words similarity; symmetric gene Jaccard; asymmetric containment; zero-shot SAPBERT cosine similarity on names and definitions; and BERTMap, a fine-tuned Bio-Clinical BERT pair classifier with graph extension and logic repair. We also include a combiner. It is a logistic regression and, separately, a gradient-boosted tree over the string, embedding and containment features, trained on the source pair only. Running both a linear and a tree combiner applies the multi-algorithm practice of the clinical benchmarking literature, e.g. Can some algorithms of machine learning identify osteoporosis patients after training and testing some clinical information about patients?. We stress-test that practice under database shift, which those studies never do.
The third component is a modest pre-training probe aimed at the gap between prose and database strings. BioBART documents a performance drop when moving from PubMed text to a different register. Pathway names are a register of their own: terse, controlled and full of abbreviations. We continue SAPBERT's synonym-clustering training on Pathway Ontology synonym sets and cross-references, restricted to source databases only. We then ask whether this database-native training reduces the cross-pair drop beyond what generic UMLS self-alignment achieves.
The analysis output for each method is its in-pair score, its cross-pair score and the gap between them. Across methods, we also report how well the method ranking on the source pair predicts the ranking on the target pair.
Baselines
The naive reference point is longest-common-substring matching on pathway names. It needs no training, so it has no source pair to overfit, and Soh et al. found it more accurate and complete than gene-overlap or gene-pair-overlap matching. No reported numeric precision or recall was found on Lacuna for it; the file states the comparison only qualitatively. We run it identically on every pair, with its threshold tuned on the source pair. Any learned method that does not beat it on unseen pairs has no practical case.
The prior pathway-specific method is the name-and-definition neural model with bag-of-words comparison from Predicting instances of pathway ontology classes for pathway integration. On its Reactome subset it reports precision 0.39 and recall 0.78, against 0.49 and 0.42 for bag-of-words, with mapping yield of 80% versus 50%. We reimplement both models on the same splits. On the Reactome–Pathway Ontology pair, our combiner must match its recall without falling below the bag-of-words precision. It must also beat it on every unseen pair.
The strongest existing language-model aligner is BERTMap, run from the released code (KRR-Oxford BERTMap) and described in BERTMap: A BERT-Based Ontology Alignment System. On FMA–SNOMED it reports unsupervised F1 1.4% above AML and 4.2% above LogMap. The workshop version, Biomedical ontology alignment with BERT, reports margins of 2.0% and 4.6%, rising to 3.7% and 6.1% with 20% of gold mappings. The file gives these margins but no absolute F1. BERTMap is the strongest competitor because it is supervised, uses domain pre-training and already adds structural repair.
For fairness, BERTMap gets the same source-pair labels as our combiner, the same hyperparameter search budget measured in GPU-hours, and the same candidate pool. We also run it once on the Anatomy track, to test the realistic case of adopting an aligner tuned on anatomy for pathways. Beating it means higher cross-pair F1 on at least two of three held-out pairs, with bootstrap confidence intervals that exclude zero, while costing far less compute per new database.
Three ablations isolate the key ingredients. The first swaps containment for symmetric Jaccard, which tests the boundary-tolerance claim directly. The second swaps SAPBERT for base PubMedBERT, which tests whether self-alignment rather than domain text alone carries the transfer. The third adds or removes the database-native continued training.
Why not just…?
Why not just use name string matching, as Soh et al. concluded? Pure string matching misses equivalences that share few characters. BERTMap's motivating examples, "third cervical" versus "C3" and "lanugo" versus "primary hair", show such equivalences are common in biomedical vocabularies. In the pathway setting, bag-of-words recall was 0.42 against 0.78 for an embedding model. String matching remains our floor. If the combiner fails to clear it on unseen pairs, we will report that learned matchers are not worth deploying across databases.
Why not just fine-tune BERTMap on a few labels from each new database pair? This is a real threat. BERTMap's lead grew when given 20% of gold mappings, and labelling a few hundred pathway pairs is not prohibitive. We settle it with a label-efficiency curve on each target pair, fine-tuning BERTMap and our combiner with 0, 25, 50, 100 and 200 target labels. If BERTMap overtakes the zero-shot combiner within roughly a hundred labels, the honest conclusion is that cheap target labelling beats transfer. The benchmark would then report the crossover point instead of a winner.
Why not just add plain gene overlap to a name matcher? Soh et al. showed that symmetric content overlap is weak across databases, with apoptosis gene-pair agreement at only 11–16%. A naive feature may therefore add noise. The Jaccard-versus-containment ablation tests exactly this. If containment gives no gain over Jaccard, our boundary explanation is wrong, and we will say so.
Why not just prompt a large general-purpose model to judge whether two pathways match? The file contains no reported result on pathway alignment for such models. Evidence from other domains, including MonoCoder and BERTHop, favours targeted domain training over scale, but that evidence comes from neighbouring tasks, not this one. We include a prompted 8-billion-parameter open model as an additional zero-shot baseline. It sees the same names, definitions and gene lists and runs at matched inference budget, so the question is answered empirically rather than by citation.
Proposed Experiments and Decisive Checks
The main experiment trains on each of the database pairs in turn and evaluates on the others. It reports precision, recall and F1 for equivalence mappings, and separately for broader/narrower mappings. Rankings are compared with Kendall's tau between source-pair and target-pair method orderings.
Two results would decide the core claims. For the first, define the cross-pair gap as in-pair F1 minus mean cross-pair F1. If SAPBERT-based matchers show a smaller gap than BERTMap with overlapping or better in-pair accuracy, the metric-learning side of the tension between SAPBERT and BERTMap is resolved for this setting. For the second, if containment raises recall on broader/narrower mappings without reducing equivalence precision, the Soh et al. content-matching failure is shown to be partly a measurement artefact.
A third check is whether the source-pair ranking predicts the target-pair ranking (tau above 0.5). If it does not, single-pair benchmarking of pathway matchers is unreliable as a selection procedure. That would be the most useful negative result this program could produce.
A final sanity transfer runs every matcher on the FMA–SNOMED and FMA–NCI tasks of the Anatomy (LargeBio) track. This shows whether conclusions drawn on pathways also hold where BERTMap was originally strongest.
References
- Consistency, comprehensiveness, and compatibility of pathway databases
- Predicting instances of pathway ontology classes for pathway integration
- Biomedical ontology alignment with BERT
- Self-Alignment Pretraining for Biomedical Entity Representations
- Java DOM Parsers to Convert KGML into SBML and BioPAX Common Exchange Formats
- Can some algorithms of machine learning identify osteoporosis patients after training and testing some clinical information about patients?
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- BERTMap: A BERT-Based Ontology Alignment System
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis