Labs that want saltier soils to stop killing crops usually pick a gene, make the plant produce much more of it ("overexpression"), and check whether the transgenic plant keeps growing under salt. There are tens of thousands of candidate genes and each transgenic test costs months, so computational rankers increasingly decide what gets tested. These rankers are trained and scored on gene-function databases that are distilled from the published literature. Much of that literature comes from one indexing source, mostly PubMed or Web of Science, and it concentrates on the same famous genes. A ranker can therefore look excellent simply by recommending genes that are already well studied. This note proposes measuring how much of the reported accuracy in rice salt-tolerance gene prioritization is explained by gene popularity and by the coverage of a single literature source. It then asks which simple, literature-independent signals still find salt-tolerance genes that nobody had yet written about. The final step tests a small, pre-registered set of those understudied predictions in the lab.
The opportunity is concrete. On the rice knowledge graph OsGenePrio, a graph-embedding ranker reaches Hits@10 of 0.914 (Candidate Gene Prioritization using Graph Embedding). Every edge in that graph comes from literature-curated sources such as Oryzabase, Funricegene, Gramene and UniProt. In human genetics, GhostBuster showed the "performance paradox": models trained on literature-derived inputs score about 0.88 ROC-AUC but mostly memorize publication patterns. Models trained on experimental expression data are 2–3× better at predicting annotations discovered later. Nobody has checked whether the rice number hides the same effect, and nobody has followed through to the bench.
Core Claim
We claim that standard evaluations of crop stress-gene rankers make two mistakes. They split edges at random, and they draw positive labels from a single literature-derived source. Under that protocol, a trivial "rank by how much has been published" baseline will match learned knowledge-graph models. The protocol we propose differs in three ways. It splits by time, so positives are genes first reported as salt-relevant after a cutoff. It stratifies results by how well studied each gene was at the cutoff. It holds out label sources, keeping some positives that never appear in the PubMed-derived vocabulary. Under this protocol the ordering of methods changes, and a ranker built mainly on salt-stress expression and sequence features recovers more future salt-tolerance genes among understudied genes.
The deliverable is the evaluation protocol plus an audit, not a new architecture. We also expect, and will report honestly, that a plain differential-expression ranking may be very hard to beat. If it is, that is itself a useful result for the plant community.
Research Program
The work has three linked parts.
The first part is a source audit of the training graph. OsGenePrio merges eight databases. We rebuild it with each source tagged on every edge, then measure per-gene degree, per-source coverage, and how strongly degree correlates with a gene's PubMed publication count at each yearly snapshot. The OsGenePrio paper merged the sources without this audit. Its authors acknowledged noise and unbalanced connectivity but did not ask where the imbalance comes from. This part treats single-source bias as a property of the training data rather than only of bibliometric maps. Bibliometric critiques such as Panaroma of microglia in TBI and Green economy studies raise single-database exclusion of regional and non-English work, but they stop at descriptive maps.
The second part is a time-split, source-held-out benchmark of salt-tolerance labels. Positive labels will be rice genes with published functional evidence of altered salt tolerance, such as overexpression, knockout or complementation. Each label records three things: its first-report date, the direction of the effect, and the source it was found in.
Direction matters because overexpression can lower tolerance. OsHAD3 decreased drought tolerance in rice, so a gene labelled "stress-related" is not necessarily a useful engineering target.
We build three label sets:
- A PubMed-derived set, assembled the way OryzaGP assembled its corpus. OryzaGP harvested 10,400 PubMed abstracts against the Oryzabase vocabulary.
- A set taken from Oryzabase trait-gene curation.
- A hand-curated set from regional, largely Chinese-language agronomy journals that PubMed does not index.
The third set is small and expensive, and it is the part that tests single-source bias directly. Rice research is concentrated in Asia, and as in AtlasNLP, what a single archive omits can only be measured against an independently built reference. We will not simply add another Western index, since PubMed and Web of Science overlapped 93% in Profiling COVID-19 Genetic Research.
Inputs will be frozen at a cutoff year $T$. Test positives are genes first reported after $T$.
The headline metric is popularity-stratified recall. We bin genes by publication count at $T$ and report Recall@100 of future positives within the least-studied half. We also report a lift over the popularity ranker:
$$L = \frac{R_{\text{low}}(m)}{R_{\text{low}}(\text{pop})}$$
Here $R_{\text{low}}(m)$ is method $m$'s Recall@100 restricted to future positives in the least-studied half, and $\text{pop}$ ranks genes by publication count at $T$. A method with $L \le 1$ adds nothing over fame where it matters most.
The third part is a small, pre-registered wet-lab check. We take the top understudied candidates from the best literature-independent ranker and from the knowledge-graph ranker. Each is triaged with the established workflow.
Predicted sodium transporters go to yeast complementation in $\Delta ena1\text{-}4$ and $\Delta ena1\text{-}4\,\Delta nha1$ strains, following MdCCX1. All candidates go to Arabidopsis overexpression with root-length and tissue Na⁺ assays at 100 and 150 mM NaCl, following the sunflower HHA4/HHA11 study.
We accept the limitation that study names: Arabidopsis results do not guarantee crop function. The Arabidopsis assay serves as a cheap filter. Rice transformation of survivors is follow-up work.
Homology-based annotation will be treated with caution. MdCCX1 sits on the plasma membrane and exports Na⁺, while its Arabidopsis ortholog AtCCX1 sits on internal membranes. Annotation copied from a model-species gene can therefore be wrong.
Baselines
The naive reference is the popularity ranker, which orders rice genes by PubMed publication count at the cutoff. It costs nothing and directly embodies the bandwagon effect, so any learned method must beat it on understudied genes to justify itself.
The strongest existing method on this exact graph is TransE-initialized ConvKB from Candidate Gene Prioritization using Graph Embedding. It reports filtered MRR 0.7 and Hits@10 0.914 on OsGenePrio link prediction under random edge splits. No reported result was found on Lacuna for any rice salt-tolerance ranker under a time split or popularity stratification. We will reproduce the reported figure first to confirm our setup. We will then retrain ConvKB on the graph frozen at $T$, using the same hyperparameter search budget as every other learned method. Beating it means a higher $R_{\text{low}}$ with confidence intervals that do not overlap, estimated by bootstrap over genes.
The literature-independent competitor is differential-expression ranking: genes ordered by salt-induced fold change across public rice salt-stress RNA-seq experiments released before $T$. This mirrors what wet-lab groups do now with qRT-PCR, as in the HHA and MdCCX1 studies. It plays the role that TCGA played for GhostBuster.
The learned literature-independent model is a GhostBuster-style encoder adapted to rice. It uses an MLP embedding with a positive-unlabeled margin classifier, and its inputs are expression and sequence features only.
The key ablation feeds this same model the knowledge-graph edges in three forms: all edges, edges from literature-curated sources only, and predicted or experimental edges only (PRIN interactions and SNP-Seek variation). This isolates whether literature-derived structure helps or hurts on understudied genes.
A second ablation retrains ConvKB with inverse-degree negative sampling. This tests whether a simple reweighting removes the popularity advantage without changing the inputs.
Finally, GPT-4o asked zero-shot for rice salt-tolerance candidates as of $T$ serves as the language-model reference. All methods see identical gene universes and cutoffs.
Why not just…?
Why not just merge more literature databases? Merging adds more text about the same famous genes. PubMed and Web of Science overlapped 93% in the COVID corpus (Profiling COVID-19 Genetic Research), and OsGenePrio already merges eight sources. Our source ablation tests this directly. If the union graph raises $R_{\text{low}}$ over single-source graphs, merging is enough and we will say so. We expect it raises recall on popular genes only.
Why not just rank by salt-induced expression? This is the real threat, and the proposal is built around it. If differential expression alone matches every learned model on understudied future positives, the result tells breeders to skip knowledge graphs for this task. That is a publishable finding about what the reported 0.914 actually measures. The learned model claims value only if it beats expression ranking on $R_{\text{low}}$ and on wet-lab hit rate.
Why not just ask a large language model? GhostBuster reports that GPT-4o mini reproduces existing publication frequencies. MMClima shows frontier models scoring above 90 on free-form explanations but below 50 on precise cloze questions. We include GPT-4o as a baseline anyway. Its $L$ relative to the popularity ranker settles the question empirically.
Why not just debias the graph model's training loss? Inverse-degree sampling is cheap and might work. It is the second ablation. If it closes the gap, the conclusion is that the protocol matters more than the inputs.
Decisive Checks
Three results would decide the project.
The first is whether the popularity ranker reaches at least most of ConvKB's time-split Recall@100 overall. That would confirm the inflation hypothesis. Failing to see this would refute it.
The second is whether method rankings change between the PubMed-derived and regional-journal label sets. If they do, that directly shows single-source label bias in crop gene benchmarks.
The third is the pre-registered bench comparison. We will test roughly ten understudied candidates per ranker in Arabidopsis overexpression, plus yeast complementation for transporters. Success means significantly reduced shoot Na⁺ and longer roots at 150 mM NaCl relative to empty-vector lines. If literature-independent picks do not achieve a higher hit rate than ConvKB picks, the computational advantage does not survive contact with biology, and we will report that.
References
- Candidate gene prioritization using graph embedding
- GhostBuster: A Deep-Learning-based, Literature-Unbiased Gene Prioritization Tool for Gene Annotation Prediction
- Panaroma of microglia in traumatic brain injury: a bibliometric analysis and visualization study during 2000–2023
- Green economy studies amongst the global climate change challenge between 2016 and 2022: a bibliometric review
- Overexpression of OsHAD3, a Member of HAD Superfamily, Decreases Drought Tolerance of Rice
- OryzaGP: rice gene and protein dataset for named-entity recognition
- AtlasNLP: A Country-Aware Atlas of Dataset Representation in NLP
- Profiling COVID-19 Genetic Research: A Data-Driven Study Utilizing Intelligent Bibliometrics
- Cation/Ca2+ Exchanger 1 (MdCCX1), a Plasma Membrane-Localized Na+ Transporter, Enhances Plant Salt Tolerance by Inhibiting Excessive Accumulation of Na+ and Reactive Oxygen Species
- Genome-Wide Identification and Analysis of P-Type Plasma Membrane H+-ATPase Sub-Gene Family in Sunflower and the Role of HHA4 and HHA11 in the Development of Salt Stress Resistance
- MMClima: A Framework for Multimodal Climate Science Data and Evaluation