Do Expert-Chosen Pathways Make Gene-Expression Models Travel Across Hospitals?

Models that predict clinical outcomes from gene expression are usually trained on samples from a limited set of hospitals or labs. They then lose accuracy at new sites, because each site's sample handling, sequencing protocol and patient mix leave fingerprints in thousands of individual genes. Biologists have a standard way to summarize noisy gene lists: functional enrichment analysis, which scores curated groups of genes that act together, such as Gene Ontology processes or KEGG pathways, rather than individual genes. We propose to test whether these pathway-level summaries, when a small panel of domain experts chooses which pathways matter for each prediction task, make classifiers hold up better on hospitals they never saw. We also test whether the experts add anything beyond what an automatic pathway selector or plain dimensionality reduction already provides.

The question matters because the obvious fixes have stalled. Algorithms designed for domain generalization, meaning training that aims to work on unseen environments, rarely beat plain pooled training on real clinical shifts. When the evaluation selects models honestly, their gains disappear. The authors who documented this failure call for clinicians to specify which features should stay invariant across sites. Nobody has tried that on molecular data, where curated pathway knowledge offers a natural vocabulary for such specification.

Core Claim

We claim that grouping transcriptomic features into a task-specific set of pathways, with the set elicited from experts through a structured participatory protocol, shrinks the drop in accuracy between training sites and held-out sites. The shrinkage should be larger than what comes from reducing the number of input features alone. The claim is falsifiable in three ways. Pathway features might help no more than random gene groups of the same sizes. Expert-chosen pathways might help no more than pathways ranked automatically on training-site data. Or none of it might beat a well-regularized pooled model on raw genes. We regard all three outcomes as plausible, and the design is built to tell them apart.

The closest prior pieces each cover only part of this. The TCGA Meta-Dataset Clinical Benchmark defines 174 prediction tasks from The Cancer Genome Atlas (TCGA) RNA-seq data covering 20,530 genes, but it feeds raw genes to classifiers and splits samples randomly, so site shift is never measured. Looking for Out-of-Distribution Environments in Critical Care and An Empirical Framework for Domain Generalization in Clinical Settings measure hospital shift carefully, but only on intensive-care time series and chest X-rays. Both find that simpler models and pooled training win, and both findings constrain our design. The second paper is also what motivates the expert-elicitation step: it asks for clinician-built invariance structure. Improving Ontology Requirements Engineering with OntoChat and Participatory Prompting supplies the method we reuse for eliciting structured knowledge from experts: co-designed prompt templates with Persona, Goal, Scenario, Example Data and Data Resources modules. It was tested only on cultural-heritage ontologies and never connected to data or to generalization.

Research Program

The first component is a benchmark in which the held-out unit is the collection site. TCGA sample identifiers record the tissue source site. We will re-split a subset of the meta-dataset's tasks by leave-site-out, which means training on some sites and testing on a site never seen during training. We will drop tasks whose label is essentially the site itself (for example, tumor tissue site) or is nearly constant within sites, because those tasks cannot measure transfer. We will keep only tasks with at least four sites that each contribute enough labeled samples.

A second, smaller testbed uses the four Gene Expression Omnibus cohorts for chronic obstructive pulmonary disease (COPD) analyzed in m6A RNA methylation regulators could contribute to the occurrence of chronic obstructive pulmonary disease. These are GSE5058, GSE8545, GSE11906 and GSE20257, which range from 39 to 135 samples. We will hold out one cohort at a time to predict case versus control. These are institution-level and lab-level shifts rather than shifts between countries. We state plainly that geography enters here as a proxy bundle of site protocols and populations, which is how the geographic-scope literature frames the confounder.

The second component is the representation. Each sample's expression profile is mapped to one score per pathway. We use rank-based single-sample scores because they are insensitive to per-site scaling, and we choose them over cutoff-based overrepresentation counts because cutoffs discard signal. A linear or shallow model is then trained on these scores. Whether a pathway enters is decided in one of three ways: all GO Biological Process and KEGG sets; an automatic top-$m$ ranking computed only on training sites; or an expert shortlist of the same size $m$.

The third component is the elicitation. Following the median clinician sample size reported in A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems, we plan roughly 12–16 oncologists, pathologists and pulmonary researchers. We adapt participatory prompting as follows. Experts co-write, for each task, a story describing the clinical question, the known biology, and the site artifacts they expect. A language model drafts a candidate pathway list from the story. Experts then revise the list in think-aloud sessions and mark each pathway as "should be predictive everywhere" or "likely site-sensitive."

This design responds to the documented risks. The survey's "experience paradox," where senior clinicians find generic explanations more burdensome, and OntoChat's reported problems of overthinking and dependence on the facilitator are the usability risks. We record session time and a standard workload questionnaire so that the cost of elicitation is reported alongside any gain. Crucially, experts never see held-out-site data, and every shortlist is frozen before evaluation.

Baselines

The naive reference point is the majority classifier. On the original random splits of the TCGA meta-dataset, it averages 66.18% accuracy across 174 tasks, against 68.78% for logistic regression and 71.19% for a multilayer perceptron, per The TCGA Meta-Dataset Clinical Benchmark. No reported result was found on Lacuna for leave-site-out TCGA or for leave-cohort-out on the COPD cohorts. We will therefore re-run all three on our splits and report both the random-split and the site-split numbers, so that the size of the site gap is itself a finding.

The strongest competitor is pooled empirical risk minimization, meaning ordinary training on all source sites combined. We run it as L1- or elastic-net-regularized logistic regression on raw log-expression. This is the method that the clinical domain-generalization studies found hard to beat: An Empirical Framework for Domain Generalization in Clinical Settings reports chest X-ray AUROC (area under the ROC curve) of 0.862 for pooled training versus 0.860 for Group DRO, the best of eight generalization methods. We also run a small multilayer perceptron and Group DRO on the same inputs, so the proposal is tested against the actual state of the art on this question.

For fairness, every method gets the same training sites, the same number of hyperparameter configurations, and model selection only on held-out source-site validation (leave-one-source-site-out within training). This is the protocol whose absence inflated earlier gains.

The key ablations isolate each ingredient. The first uses size-matched random gene sets, drawn to have the same set-size distribution as the chosen pathways, and tests whether curated biology matters beyond grouping. The second uses principal components of matching dimension and tests whether low dimension alone explains any gain. The third compares the automatic top-$m$ selector with the expert shortlist and tests whether the humans matter.

The proposal beats pooled training if two things hold. First, held-out-site AUROC must improve and the site gap must shrink. Second, the improvement must be consistent across held-out sites under a paired test across site-task pairs, and it must exceed the random-set and principal-component ablations by more than their seed-to-seed spread.

Why not just…?

Why not just pool all sites and regularize hard? This is the most serious threat. Pooled training won in both clinical domain-generalization studies, and Looking for Out-of-Distribution Environments in Critical Care found that simpler models held up better under hospital shift. Against that, WILDS shows pooled training still dropping from about 93% to about 70% accuracy on unseen Camelyon17 hospitals, so pooling alone does not close the gap. Regularized pooled logistic regression is our headline baseline for exactly this reason. If it matches pathway models on held-out sites, we will report that pathways give no transfer benefit for these tasks.

Why not just reduce dimension with principal components or random gene groups? Pathway scores are low-dimensional and averaged, and averaging alone may explain any robustness, since averaging many genes dilutes the site noise in any one of them. Nothing in the file separates curated from random grouping for transfer. The size-matched random-set and principal-component ablations are designed to settle this. If they match the pathway models, the contribution collapses to "use fewer, smoother features," and we would say so.

Why not skip the experts and let training-site statistics or a language model choose pathways? Automatic selection is cheap and does not require a panel whose small size, as the clinical explainability survey notes, limits statistical power. Our reason to expect experts to add value is that source-site statistics cannot distinguish biology from artifacts shared by the training sites. This is the core argument of the clinical domain-generalization paper. The head-to-head test at fixed $m$ decides the question, along with an arm that uses the language-model draft before expert revision. If automatic selection ties, the participatory step is not justified for this purpose, though it might still matter for interpretability, which we would then report separately.

Why not use a domain-generalization algorithm on raw genes? WILDS reports that robust training methods gave no meaningful improvement over pooled training in 8 of 10 datasets. We include Group DRO as a baseline, but we do not expect it to be the answer.

Proposed Experiments

The main experiment crosses representation (raw genes, principal components, random sets, all pathways, automatic top-$m$, expert top-$m$) with model (regularized logistic, small multilayer perceptron, Group DRO) on the leave-site-out TCGA tasks and the leave-cohort-out COPD splits. We report held-out-site AUROC and accuracy, the in-site minus held-out-site gap, and worst-site AUROC. We also report a site-level concentration score for errors, borrowing the relative-entropy idea of GeoBS: the divergence of the distribution of errors across sites from the distribution of samples across sites.

The decisive check is a single paired comparison: expert pathways against size-matched random sets and against automatic pathways, at matched $m$, across all held-out site-task pairs. A secondary check looks at the pathways experts labeled as site-sensitive. If the experts are right, dropping those pathways should reduce the gap more than dropping an equal number of random pathways.

We will also examine annotation bias. The research file warns that GO and KEGG coverage is thin outside well-studied disease, so we will report results separately for cancer types with sparse versus rich pathway coverage. The pathway advantage may exist only where the curated knowledge does.