Whose Literature Do Gene Ontology Annotations Encode? A Provenance Audit of Enrichment Analysis

When a biologist finds a list of a few hundred genes that change in a disease, they almost always ask a database which biological processes those genes belong to. This step is called functional enrichment analysis: a statistical test of whether, say, "DNA repair" genes show up in the list more often than chance would predict. The test can only find processes the database knows about. The Gene Ontology and similar resources are built by curators who read papers and record what each gene does. Separately, a large literature on linguistic publication bias shows that the papers which get indexed, read and cited are skewed toward English-language, Western sources. These two facts have never been joined. This proposal asks two questions. Does the literature behind manually curated gene annotations carry that same language and geography skew? And if it does, does the skew change the biological conclusions that enrichment analyses reach, by more than random gaps of the same size would?

The test is concrete. Most manual Gene Ontology annotations carry the identifier of the paper they came from. That lets us tag every annotation with the language and author country of its source. We then measure how often papers of each kind turn into annotations, and rerun standard enrichment analyses after deleting annotations by provenance. The comparison is against deletions of the same size made at random, and against deletions matched on how well-studied each gene already is. If enrichment results are no more fragile to language- and geography-structured gaps than to random ones, that is a useful negative result: robustness claims made under random missingness would then cover the real failure mode. If they are more fragile, every robustness study that deletes edges or annotations at random is measuring the wrong thing.

Core Claim

Existing robustness evidence for enrichment uses random missingness as its null. The clearest example is Network-based pathway enrichment analysis with incomplete network information, which estimates disease-specific gene networks guided by incomplete external databases. It reports that enrichment results "remained comparable" with up to 80% misspecified external information. Bibliometric work suggests that real omissions are not random. Geographic Citation Gaps in NLP Research shows that citation deficits by region persist even within the same venue and grow over time. A bibliometric analysis of global research output on network meta-analysis argues that English-dominant indexes exclude regional output.

Our claim is that annotation gaps cluster by source language and author geography. Because they cluster in particular genes and processes, they shift enrichment conclusions by more than random gaps of equal size. This is the excess we propose to measure. Excess fragility of this kind would not show up in any random-deletion robustness check.

The newness is not "bias exists." It is a measurable link from the provenance of each paper to the provenance of each annotation, and from there to the stability of each enrichment result. No paper in our search makes that link. UniEntrezDB unifies Gene Ontology annotations from 21 databases and notes the taxonomic gap, but it never asks where the supporting papers came from. Africa Health Check measures a Western default in medical language models, but only at the level of treatment recommendations, with no molecular layer.

Research Program

Stage 1: provenance tagging. We build a provenance table for every manually reviewed Gene Ontology annotation that cites a PubMed record. For each source paper we record:

  • the publication-language field from PubMed;
  • the first- and last-author country, taken from affiliation strings.

The affiliation tagging follows the approach of Geographic Citation Gaps in NLP Research, which reports 84.5% accuracy for automated country tagging. We will check our own tagging accuracy on a hand-labeled sample.

PubMed itself under-indexes regional journals. Every Stage 1 estimate is therefore a lower bound on the true skew. We say so explicitly and do not present it as the full picture.

Stage 2: curation uptake. We estimate the curation uptake rate: the probability that a PubMed paper reporting a function for a given gene is ever cited by an annotation for that gene. We compare uptake across language and country strata, holding fixed:

  • the gene;
  • the publication year;
  • the journal.

Holding these fixed is essential because of a known confound. Non-English and Global-South papers may study less-studied genes or appear in lower-visibility journals. That would produce lower uptake through the well-known bias toward already-studied genes, not through language. We want the language effect that remains after this is accounted for.

Stage 3: counterfactual enrichment. We delete the annotations supported by a given provenance stratum, for example all non-English sources or all sources with Global-South authors. We then rerun three families of tests:

  • overrepresentation analysis, a hypergeometric test on a thresholded gene list;
  • rank-based gene set enrichment analysis;
  • the network-guided method above.

Each deletion is compared to two matched controls with the same number of deleted annotations:

  • a uniformly random deletion;
  • a stratified random deletion matched on each gene's annotation count and each term's size.

The quantity of interest is the excess instability:

$$\Delta = I(\text{provenance deletion}) - I(\text{matched random deletion})$$

Here $I$ is one minus the Jaccard overlap between the sets of significant terms at a false discovery rate (FDR, the expected share of false positives among reported hits) of 0.05, before and after deletion. $\Delta$ is averaged over datasets, with confidence intervals from repeated random draws.

Stage 4: recovery from non-English literature. In the forward direction, we take non-English PubMed records and a sample from one regional-language biomedical journal corpus. We extract gene-function claims and map the extracted function phrases onto Gene Ontology terms using BERTMap, a BERT-based tool that matches classes across ontologies. Curators verify a random sample of the extracted claims. The verified annotations are added back into the database, and enrichment is rerun.

The aim is to show what is recoverable, not to deploy a product.

Baselines

Do nothing. The naive reference is enrichment on the current English-dominated annotations, treated as ground truth. Every instability we report is measured against it.

Random-deletion robustness check. This is the protocol implied by Network-based pathway enrichment analysis with incomplete network information. We run it by drawing deletions of exactly the size of each provenance stratum, using the same datasets, tests and FDR thresholds. The provenance hypothesis beats this baseline if $\Delta$ is reliably above zero.

Strongest existing method. The strongest robust-enrichment method is that same network-guided approach. On TCGA breast cancer data it "identified 25 significantly enriched pathways at a false discovery rate (FDR) of 0.01, while GSA methods identified none or very few," and it was robust to 80% misspecification. We run it with its published settings on the same datasets: the breast cancer and lung microarray sets, plus a subset of the GenoTEX tasks from GenoMAS. Evidence against its robustness claim for real-world gaps would be this: a provenance deletion far smaller than 80% shifts its significant-pathway set more than a random deletion of the same size.

Matched-random ablation. The ablation that isolates our key ingredient is the deletion matched on gene annotation count and term size. If $\Delta$ disappears under this control, the effect is the familiar "well-studied genes get studied more" bias rather than anything about language. We will report that plainly.

Recovery baselines. For Stage 4, the baseline is translate-then-extract: a general machine translation system followed by the same English extraction step and the same ontology mapper. Both arms share:

  • the same documents;
  • the same language model budget;
  • the same verification sample size.

Success means higher verified precision at equal yield. We found no reported result on Lacuna for multilingual gene-function extraction into Gene Ontology terms, so both arms will be measured from scratch. The closest reported number is from GenoMAS, where language-model agents scored 91% on structured gene data but "dropping to 32%" on free-text clinical phenotypes. That gap warns that free-text extraction is the weak link.

Why not just…?

Why not just trust the published robustness to 80% misspecification? That figure was obtained with misspecified or permuted edges, not with gaps concentrated in particular genes and processes. When a whole function is missing, a test cannot detect it at all, however robust it is to noise elsewhere. Stage 3 settles this directly. If $\Delta$ is near zero across tests and datasets, the reviewer is right, and we will say that random-deletion checks are adequate.

Why not just machine-translate the non-English literature and feed it to existing curation tools? This is a real threat to Stage 4, and it is why translate-then-extract is a baseline rather than a straw man. If it matches our pipeline, recovery turns out to be an engineering problem, and that is a fine outcome. It does not touch Stages 1 to 3, which ask whether the gap matters at all. Translation also does not explain why a paper was never curated in the first place. The uptake analysis does.

Why not just let semantic post-processing absorb the gaps? Contrastive Learning for Gene Set Enrichment Analysis Post-Processing (gtCLIP) groups redundant pathways by aligning gene sets with PubMedBERT text. It reports a Recall@5 of 51.3% for retrieving pathways from text. But it reorganizes terms that already exist; it cannot add an annotation that was never made. Its text encoder is also trained on English PubMed. We will run gtCLIP clustering on the outputs before and after deletion. If cluster-level conclusions stay stable while term-level ones change, that is worth reporting as partial protection.

Why not just call this the known well-studied-gene bias? It may be exactly that. The matched-random ablation exists to tell the two apart, and Stage 2 controls for gene and journal for the same reason.

Proposed Experiments

Datasets. Stage 3 uses the TCGA breast cancer RNA-seq and lung cancer microarray sets from the network enrichment paper, plus a stratified sample of GenoTEX gene–trait tasks.

A second arm targets non-model organisms analyzed by projecting onto genes of related model species, where regional literature should matter most. It uses:

Decisive checks. There are three. First, Stage 2 uptake ratios across language and country strata, with confidence intervals after controlling for gene, year and journal. Second, $\Delta$ for each test family, with the matched-random ablation reported alongside. Third, the change in significant terms when verified recovered annotations are added back to the non-model-organism datasets.

What would falsify the hypothesis. The hypothesis fails if both of the following hold:

  • uptake is equal across strata after controls;
  • $\Delta$ is indistinguishable from zero under the matched control.

Expected limitation. Because the audit works through PubMed, the measured skew is a floor on the true skew. The regional-corpus sample in Stage 4 gives one estimate of how far that floor sits below the real gap.