Auditing Protein Localization Labels Against the Non-English Experimental Record

The labels that tell machine learning models where each human protein lives in the cell (nucleus, membrane, mitochondria and so on) come almost entirely from two curated repositories, UniProt/SwissProt and the Human Protein Atlas (HPA). Benchmarks such as DeepLoc 2.0 and CAPSUL treat these labels as ground truth. Meanwhile, many thousands of cell-biology papers each year establish where a protein goes using two independent lab techniques. The first is immunofluorescence: staining cells with glowing antibodies and imaging them. The second is fractionation Western blotting: splitting cells into nuclear and cytoplasmic portions and measuring how much of the protein ends up in each. A large share of these papers appears in Chinese, Japanese, Portuguese and Spanish regional journals that the English-centric indexes feeding curators rarely cover.

We propose to measure what this single-source, English-filtered labeling misses. We will mine a multilingual corpus for protein–compartment claims that are backed by both techniques, screen the blot evidence for manipulation, and check the survivors against what curators later add. The two-sentence version is this: the "wrong" predictions of localization models may partly be correct localizations the reference databases never ingested, and we can find them by requiring two independent assays in the literature, including literature curators do not read. If that holds, benchmark scores for protein localization are partly measuring agreement with one curation pipeline's coverage rather than with biology.

Core Claim

We claim that a filter requiring two assays turns the noisy global literature into a usable second source of localization labels. A protein–compartment pair counts as supported only when the same paper reports both an imaging result and a fractionation blot result. This source adds pairs that are absent from the union of UniProt and HPA. These additions are concentrated in non-English regional literature and in localizations that depend on experimental condition, such as the stress-driven shuttling of YAP1 described in Calponin 3 regulates myoblast differentiation through YAP1 signaling or the loss of ZO-1 and Occludin from cell junctions under inflammatory insult shown in Kuwanon G preserves LPS-induced barrier disruption.

The idea is new for two reasons. The unit of evidence is a multi-assay claim rather than a text mention. The comparison target is the labeling behind machine learning benchmarks, not citation counts.

Relation to Prior Work

Four lines of work motivate the proposal.

Bibliometric studies such as the network meta-analysis bibliometric study and the microglia-in-brain-injury analysis acknowledge single-database, English-only restriction as a limitation, but they never quantify it. We quantify it in a setting where it changes a downstream benchmark.

Work on structural databases, such as Training data composition affects protein structure algorithms, shows that the Protein Data Bank's ascertainment bias propagates into models. We ask the same question of localization labels, which that work does not touch.

AtlasNLP and Geographic Citation Gaps in NLP Research audit geographic skew, but only within the ACL Anthology. We move the audit to experimental biology. We also borrow AtlasNLP's practice of measuring extraction precision against a human gold set; its country assignment reached 94.7% precision.

The image-based localization models, the HPA multilabel oversampling work and DeePSLoc, fight class imbalance inside HPA. We ask whether some apparent minority-class scarcity is actually coverage scarcity.

Two lines of work supply method. ResearchDB provides the PDF-to-structured-record pipeline: minerU conversion followed by extraction with DeepSeek-R1-32B. Its authors note it was restricted to English AI venues, and we extend it to multilingual biomedical text. Forensic analysis of synthetic Western blots supplies a one-class detector trained only on real blots, which we use to discard suspicious evidence.

Two papers constrain the design. SciFIBench found that multimodal models reach only about 70–75% on figure interpretation, against roughly 86% for humans, and that they follow misleading text. VeriSciQA found that only 3.0% of machine-generated figure QA candidates survived verification. Together these rule out trusting a vision-language model to read blots and micrographs unaided. For that reason our pipeline treats text as primary and images as a verification layer.

CAPSUL's finding that 3D structure is decisive for localization might seem to contradict our emphasis on condition-dependent evidence. We think the two are compatible. Structure fixes the set of places a protein can go, while condition decides which of those places it occupies. We therefore record the experimental condition with each pair rather than folding it into a static label.

Research Program

We restrict scope to human proteins so that results align with CAPSUL's 20,181 proteins and its 20 compartments, and with DeepLoc 2.0's ten.

The corpus combines PubMed Central open-access papers (English) with regional full text from CNKI (Chinese), J-STAGE (Japanese) and SciELO (Portuguese/Spanish). Each paper carries a language tag and a flag for whether it is indexed in PubMed.

Each paper passes through minerU and a structured extractor. The extractor emits records containing the protein, the compartment, the direction of the finding (present, enriched or lost), the experimental condition, and assay type. Assay type is imaging, fractionation blot, or both. A fractionation blot counts only if the paper names loading-control markers for the fractions, such as a nuclear marker and a cytosolic marker; without them the fractions cannot be trusted.

A pair enters the "corroborated" set only when imaging and fractionation-blot records in the same paper agree. For papers with extractable blot panels, the blot image must also pass the one-class forensic screen.

We then compare the corroborated set with the union of UniProt, HPA and Gene Ontology annotations. The comparison is stratified by language, compartment and whether a condition is attached.

Baselines

The naive reference point is the current labels: the union of UniProt, HPA and GO annotations at a fixed snapshot date. Every other source must add pairs beyond this union at acceptable precision to matter. It is the right "do nothing" comparison because it already pools the three major English curation streams. A reviewer could reasonably argue that this pooling is enough.

The strongest existing competitor for supplying missing localizations is a predictive model rather than a mining pipeline. Here that is DeepLoc 2.0 run at its published thresholds on all human proteins. Its high-confidence predictions that fall outside the database union form an alternative "discovered pairs" set. For DeepLoc 2.0 the file reports only that it "outperformed other tools on several metrics, including accuracy and MCC for most locations" on the independent HPA benchmark. No reported numeric result was found on Lacuna for DeepLoc 2.0 on HPA.

For image-based models, the strongest numbers in the file come from two papers. The oversampling work reports macro F1 of 0.74 on its custom HPA test set TS1. DeePSLoc reports 61.2% subset accuracy on HPA-18.

To keep the comparison fair, the mining pipeline and DeepLoc will each produce the same number of candidate pairs, matched by count. Both sets are then scored by the same blinded expert panel and the same temporal holdout described below. The mining pipeline beats DeepLoc if, at matched count, its candidates have higher expert-judged precision and are added by curators at a higher rate.

The key ablation drops the two-assay requirement and accepts any single-assay text claim. This isolates whether corroboration, rather than sheer volume, is what makes literature labels usable. A second ablation removes the forensic screen. A third removes the non-English sources, which shows how much the linguistic dimension contributes beyond PubMed alone.

Why not just…?

Why not just translate regional abstracts and text-mine localization claims? This is the most serious threat. It is exactly our single-assay ablation, run on translated abstracts. If abstract-level claims reach the same expert precision as corroborated full-text claims, the heavier pipeline is unnecessary, and we will report that. We expect it to fall short for two reasons. Abstracts routinely state localization conclusions without saying how they were measured. And the bibliometric evidence in the network meta-analysis study is about indexing boundaries: many regional journals never reach PubMed at all, so translation cannot recover papers that were never indexed.

Why not just rely on the curated databases, which already read the literature? The temporal holdout answers this directly. If curators already see this evidence, pairs we extract before a snapshot date should appear in later releases at similar rates regardless of source language. A large gap between English and non-English papers in that later-addition rate would show that curators systematically do not see the non-English evidence.

Why not just prompt a multimodal model on the figures? SciFIBench shows GPT-4o at roughly 70–75% on figure-caption matching and shows susceptibility to misleading text. VeriSciQA found that 97% of machine-generated candidates failed verification. We still include a GPT-4o figure-reading arm, judged on the same gold set, so the claim is tested rather than assumed.

Why not just trust sequence or structure models to fill the gaps? This is the DeepLoc baseline itself. The rescoring experiment below also turns it into a positive result: if many of DeepLoc's "false positives" are literature-corroborated, the model is being penalized for being right.

Proposed Experiments

The first decisive check is expert precision. Bilingual cell biologists, blinded to source, will double-annotate a stratified sample of about 1,000 candidate pairs drawn from each arm. They will judge whether the paper's figures actually support the claimed localization.

The second is a temporal holdout. We freeze databases and literature at an earlier snapshot, then measure what fraction of each arm's pairs appear in current UniProt, HPA or GO releases, compared against a random-pair base rate. A pipeline that surfaces real but uncurated biology should show enrichment well above that base rate. Pairs from the non-English literature should show a lower later-addition rate than English pairs of equal expert precision; that combination is what curation bias would predict.

The third is benchmark rescoring. We take DeepLoc 2.0's predictions, and CAPSUL-benchmark structure models where available, and recompute their precision after relabeling database "negatives" that our corroborated set supports. Rescored precision is reported by compartment. Two outcomes would matter most: minority compartments gaining the most would suggest that part of the imbalance these papers fight is a coverage artifact, and condition-dependent pairs being mis-scored would support recording condition as part of the label rather than folding it into a static one.

The negative outcome would also be publishable. If the corroborated non-English set adds few pairs beyond the union, or adds them at low precision, that is direct evidence that single-source localization labels are safer than the bibliometric literature implies.