Most protein function predictors work the same way. They squeeze a protein into a single vector and then pick labels from a fixed list. That list is the Gene Ontology (GO), a curated vocabulary of tens of thousands of function terms, each with a written definition and arranged in a parent–child hierarchy. Recent models swap the fixed label list for embeddings of each term's written definition, so in principle they can score a term they never saw during training. They still match that text against one pooled protein vector, though, and the pooled vector comes from sequence alone. We propose to reverse the direction of the comparison. The text of a GO definition becomes a query that attends over the individual residues of a protein's 3D contact graph, where "residue" means one amino-acid position. The model then decides whether the protein has the function by looking at the residues the definition points to, instead of at a global average.
The motivation is a bottleneck that existing work has measured but not acted on. Protein-level embeddings from current foundation models, built by mean pooling, collapse to an intrinsic dimension of about 10, while residue-level embeddings keep 15 to 30 (Probing the Embedding Space of Protein Foundation Models through Intrinsic Dimension Analysis). Matching about 10 effective degrees of freedom against thousands of distinct GO definitions invites collisions, and the risk is worst for fine-grained catalytic and binding terms. Those terms are decided by a small pocket of residues that may lie far apart in the sequence but touch in 3D. Our core claim is that, for unseen GO terms, a definition-conditioned readout over residues beats definition matching against a pooled vector. We further claim that the gain from adding explicit 3D structure is concentrated in terms whose functional residues are scattered along the sequence.
Where the field stands
Two lines of work almost meet. The first supplies the structural side. DeepFRI runs graph convolutions over 10Å residue contact maps on top of language-model features (Structure-based protein function prediction using graph convolutional networks). It reaches $F_{max}$ = 0.657 on PDB700, against 0.525 for a sequence CNN, and its saliency maps locate catalytic residues with AUROC 0.81. Its outputs, however, are a fixed set of independent output units, one per term, so it cannot score a term outside that set. ProRefiner shows that graphs built only from each residue's nearest neighbours miss long-range contacts such as salt bridges and disulfide bonds (ProRefiner: an entropy-based refining strategy for inverse protein folding with global graph attention). It fixes this by adding a learned "pseudo-edge" that summarises all absent edges, which gives global context at $O(N \times k)$ memory. ProRefiner addresses sequence design, not function prediction. We reuse its layer as a method.
The second line supplies the label side, and every member uses sequence alone. OntoProtein encodes GO definitions with PubMedBERT and aligns them to a ProtBert sequence encoder through knowledge-graph triples (OntoProtein: Protein Pretraining With Gene Ontology Embedding). ProtGO aligns definition text to ESM-2 representations with attention and adds a graph network over the GO hierarchy (ProtGO: universal protein function prediction utilizing multi-modal gene ontology knowledge). It reports 8 to 27% $F_{max}$ gains over plain language models on CAFA5. It projects its attention onto structures only after prediction, and it prunes rare terms. TALE embeds GO terms and sequences in a shared space, with a loss that forbids a child term from outscoring its parent (TALE: Transformer-based protein function Annotation with joint sequence–Label Embedding). Barot et al. generate free-text function descriptions for novel classes (Automated Protein Function Description for Novel Class Discovery). These papers motivate the label half of our design. None of them lets the label text look at 3D residues inside the forward pass. That interface is what we build.
Core Claim
Let $h_1,\dots,h_N$ be residue states from a structure encoder for a protein with $N$ residues. Let $q_g$ be a query vector derived from the PubMedBERT encoding of GO term $g$'s definition. The model scores the term as
$$ s(x,g) = \sigma\!\Big( u^\top \sum_{i=1}^{N} \alpha_i(g)\, h_i \Big), \qquad \alpha_i(g) = \mathrm{softmax}_i\big(q_g^\top W h_i\big), $$
where $\sigma$ is the logistic function, $W$ is a learned bilinear map, $u$ is a learned scoring vector, and $\alpha_i(g)$ is the attention the definition places on residue $i$. Each term therefore receives its own protein summary, built from the residues its definition selects, instead of sharing one pooled vector. The attention weights $\alpha(g)$ double as a residue-level explanation tied to the definition text.
We make three falsifiable predictions. First, on GO terms held out entirely from training, this readout beats a pooled dual encoder that uses the same backbone and the same text encoder. Second, replacing the structure encoder with sequence-only ESM-2 residue states loses accuracy mainly on terms whose annotated functional residues are spread far apart in sequence, and loses little on the rest. Third, for unseen catalytic terms, $\alpha(g)$ concentrates on Catalytic Site Atlas residues without any site supervision.
Research Program
The structure encoder takes ESM-2 residue features as node inputs on DeepFRI's 10Å contact graph. Graph convolutions are interleaved with ProRefiner-style pseudo-edge global attention, so pocket residues that are distant in sequence and only indirectly connected in the graph can still exchange information. Following DeepFRI's practice, we use experimental structures where available and homology models elsewhere. The label encoder is a frozen PubMedBERT applied to each GO name plus definition. A small learned projection maps it to $q_g$. In a second variant, $q_g$ is replaced by a few query vectors per term, which lets one definition attend to several sites.
Training uses binary cross-entropy over sampled positive terms and hard negatives drawn from the same GO branch, the sampling scheme OntoProtein used. We add TALE's parent–child consistency penalty. Scoring every protein against all ~30k terms with cross-attention is expensive. Inference therefore runs in two stages. A cheap pooled dual encoder shortlists the top few hundred terms, and the cross-attention readout reranks them. We report recall of the shortlist so that losses from the first stage are visible.
The analysis component tests the bottleneck explanation directly. We measure the TwoNN intrinsic dimension of the set of term-conditioned readouts $\sum_i \alpha_i(g) h_i$ across terms for a fixed protein, and compare it with the pooled embedding. If the readouts span no more dimensions than the pooled vector, the bottleneck story is wrong, even if accuracy improves for some other reason.
Baselines
The naive reference points are a term-frequency predictor, which ranks GO terms by their training prior, and DIAMOND homology transfer. The frequency predictor matters because CAFA showed generic terms such as "protein binding" make up about 38% of experimental annotations and inflate scores (A large-scale evaluation of computational protein function prediction). Homology transfer cannot predict unseen terms at all, but it bounds performance on terms seen in training. Our model must clearly beat both on fine-grained Molecular Function terms. We also report results with "protein binding" removed.
On terms seen during training, the strongest structure-aware baseline is DeepFRI with its flat output units, retrained on our exact protein split. The reported reference is $F_{max}$ = 0.657 on PDB700 (Structure-based protein function prediction using graph convolutional networks). On seen terms we aim only to match it within noise. Text-defined labels are not expected to help much where labelled examples are plentiful, and losing there would mean the readout costs too much.
The strongest competitor on the actual question is a pooled dual encoder. It uses ESM-2 with mean pooling, the same frozen PubMedBERT definition embeddings, and inner-product scoring. This is the shared core of OntoProtein and ProtGO, and we optionally add ProtGO's GO-hierarchy graph module. ProtGO reports 8 to 27% relative $F_{max}$ gains on CAFA5 but gives no absolute value. For zero-shot GO terms on structure benchmarks, we found no reported result on Lacuna for any model. We therefore train this baseline ourselves on the same proteins, the same held-out terms, the same negative sampling, and the same hyperparameter search budget, measured in GPU-hours. A win means a statistically significant gain in term-centric AUPR and protein-centric $F_{max}$ on unseen Molecular Function terms, using bootstrap confidence intervals over proteins.
Three ablations isolate the ingredients. The first keeps the cross-attention readout but feeds it sequence-only ESM-2 residue states. This tests whether structure matters beyond the readout itself. The second keeps the contact graph but removes pseudo-edge global attention. This tests whether global context matters. The third replaces PubMedBERT definitions with random per-term vectors. This leaves zero-shot transfer only through the hierarchy and tests whether the definition text carries the signal.
Why not just…?
Why not just use a pooled dual encoder with a bigger language model? ProtGO's finding that a GO-enhanced ESM-150M beats a plain ESM-15B suggests scale alone is not the lever. However, that comparison does not test pooled versus residue-level matching. This is the main threat to the proposal, and it is settled by the head-to-head comparison on held-out terms. If pooled matching ties, the bottleneck story fails, and we will report that.
Why not just do the cross-attention over sequence-only residue states? This alternative may well work about as well. Sequence models already show long-range correlations (Probing the Embedding Space of Protein Foundation Models through Intrinsic Dimension Analysis). For that reason the proposal does not stake its claim on structure everywhere. We predict a structure advantage only where functional residues are scattered along the sequence, in line with weak sequence signal for distant paired strands (Prediction of Beta Sheets in Proteins) and with local-bias failures on discontinuous pockets (Zero-Shot Learning with Subsequence Reordering Pretraining for Compound-Protein Interaction). If the gap does not track sequence spread, the contribution reduces to the label-queried readout itself, and we will report it that way.
Why not just train DeepFRI and map unseen terms to the nearest seen term by definition similarity? This approach is cheap and reuses a strong model. It can only express an unseen term as a mix of seen terms. It cannot represent a definition that names a new chemistry. We include it as an extra zero-shot baseline under the same split. Beating it specifically on held-out leaf terms with no close text neighbour is the evidence we need.
Proposed Experiments
The primary benchmark is PDB700 together with DeepFRI's training structures. We build a zero-shot split by holding out about 20% of Molecular Function and Biological Process leaf terms, together with every training protein annotated with them, so that held-out terms never appear as positives. Secondary evaluation uses the ProtGO CAFA5 split, with homology-model structures supplied, and TALE's Swiss-Prot time split. We report TALE's CCO result only as a relative claim: TALE+ improved $F_{max}$ by 8% over NetGO. We also break results down by homology, including the below-30%-identity setting used by TALE and Barot et al. Metrics are $F_{max}$, $S_{min}$, and term-centric AUPR. Residue-level grounding is scored as AUROC of $\alpha(g)$ against Catalytic Site Atlas residues on unseen catalytic terms, using DeepFRI's supervised-label Grad-CAM AUROC of 0.81 as an upper reference rather than a direct competitor.
The decisive check is a single regression. For each held-out term, we regress the per-term AUPR gain of the structure model over the sequence-only readout on the median sequence separation of that term's annotated site residues. We predict a positive slope. The project fails cleanly if two things both hold: the pooled dual encoder matches the readout on unseen terms, and the structure gain shows no dependence on sequence spread. In that case pooled text matching is good enough, and structure adds nothing specific to label grounding.
References
- Probing the Embedding Space of Protein Foundation Models through Intrinsic Dimension Analysis
- Structure-based protein function prediction using graph convolutional networks
- ProRefiner: an entropy-based refining strategy for inverse protein folding with global graph attention
- OntoProtein: Protein Pretraining With Gene Ontology Embedding
- ProtGO: universal protein function prediction utilizing multi-modal gene ontology knowledge
- TALE: Transformer-based protein function Annotation with joint sequence-Label Embedding
- Automated Protein Function Description for Novel Class Discovery
- A large-scale evaluation of computational protein function prediction
- Prediction of Beta Sheets in Proteins
- Zero-shot learning with subsequence reordering pretraining for compound-protein interaction