Predicting what a protein does means assigning it a set of Gene Ontology (GO) terms, a vocabulary of thousands of function labels arranged as a directed acyclic graph in which specific terms imply their broader ancestors. Fewer than 0.5% of known sequences carry experimentally curated labels. Most annotation is therefore done by models that read the sequence and score every GO term independently and in parallel, and then repair the output so that it respects the hierarchy. A long line of multi-label work says this leaves accuracy unused. Labels inform one another, and a "classifier chain", which predicts labels one at a time and feeds earlier predictions forward as inputs, can exploit those dependencies. Chains have a known weakness, though. Their accuracy depends on the order in which labels are predicted, and an early mistake snowballs down the chain. The few chained protein-function models fix the order top-down, from broad terms to specific ones. That is arguably backwards for proteins, because a specific catalytic motif is often the most reliable signal in the sequence while broad biological-process terms are diffuse and error-prone.
This note proposes training one model that can condition on any subset of already-decided GO terms. At inference it orders the chain separately for each protein: it commits first to the terms it is most confident about, wherever they sit in the hierarchy, and conditions everything else on them. The same model also covers the regime the annotation bottleneck produces in practice, where a protein already has a few known terms (a localization, an inferred family) and the task is to complete the rest. The main question is empirical and could come out against us. Once inputs are strong protein-language-model embeddings, does label order still matter? If it does, does a confidence-first order beat top-down chains, order-free stacking, and flat prediction with hierarchy repair?
Core Claim
The claim is that per-protein, confidence-ordered conditioning gives higher Fmax than both flat prediction and fixed top-down chaining. Fmax is the best F1 over decision thresholds, the standard CAFA metric. We expect the gains to concentrate on deep, rare GO terms and on low-homology proteins. We also expect the benefit to come specifically from conditioning on high-confidence terms first. Conditioning on all predictions at once, or in a random order, should not produce it. The theoretical motivation comes from On the Optimality of Classifier Chain for Multi-label Classification, which shows for linear chains that the generalization bound grows with the sum of inverse squared margins of the classifiers along the chain. Placing large-margin decisions early is a direct way to keep that sum small. Their CC-DP search costs $O(q^3nd)$ for $q$ labels and fixes one global order. That is infeasible for GO, and it ignores the fact that the confident labels differ from protein to protein. We replace the global search with a cheap per-instance ordering rule. The bound is used as motivation only, and we make no claim that it holds for deep models.
How This Differs From Existing Work
The work this proposal builds on falls into three groups. The first group already chains protein-function predictions, but only top-down. Reduction strategies for hierarchical multi-label classification in protein function prediction (HMC-LMLP) trains one network per FunCat level and appends each level's predicted probabilities to the next level's input. Its authors leave multi-parent DAGs like GO unsolved. Hierarchical Multi-Label Classification Networks maps levels onto LSTM steps in HMCN-R, which needs aggressive gradient clipping, while its feed-forward variant HMCN-F with a hierarchy-violation penalty is the stronger model. Both papers motivate the proposal, since they show that conditioning on earlier labels helps. Neither tests any order other than root-to-leaf, and that untested choice is the one we vary.
The second group treats ordering as the central problem but works on small flat label sets. BNCC learns a dependency DAG from conditional entropy and topologically sorts it. Polytree-Augmented Classifier Chains restricts dependencies to a polytree and decodes exactly. Both are evaluated on Yeast and Genbase with hand-built features, and the file notes that neither scales to label spaces of more than a few hundred. They limit our design. Learning a global label graph from co-occurrence is off the table at GO scale, and polytrees cannot represent GO's diamond-shaped structures. That is why we use GO's own graph only for consistency and let the order come from the model's confidence. Leveraging Text-to-Text Transformers as Classifier Chain for Few-Shot Multi-Label Classification (T5 Chain) is the closest methodological source. It shows that order-agnostic training through label permutation is needed before a chain works, and it then freezes one order chosen on validation data. We reuse the order-agnostic training idea. We differ in three ways: our order is chosen per protein, our labels are committed in blocks rather than generated token by token (which would be too slow over thousands of terms), and our inputs are protein embeddings rather than text.
The third group models GO label dependencies without any ordering. GoBERT learns from the GO graph and from masked annotation sets to infer new terms from known ones, but it never reads the sequence. TALE embeds sequences and GO terms jointly with a hierarchy penalty but predicts all terms in parallel. NetGO 3.0 shows that logistic regression on frozen ESM-1b embeddings (LR-ESM) already beats homology search on difficult proteins. GenePheno inserts a GO bottleneck before phenotype heads, and errors in that bottleneck pass through uncorrected. We take two things from these papers: GO term embeddings and frozen ESM-1b features. Our model is roughly what you get by giving GoBERT's annotation-conditioning ability a sequence input and a decoding order.
Research Program
The model has three parts. The first is a frozen ESM-1b embedding $h(x)$ of protein $x$, inherited along with NetGO 3.0's 1024-residue truncation, which keeps the comparison to LR-ESM clean. The second is a set of "revealed" GO terms $R$. Each revealed term is represented by a term embedding initialized from its text definition, as GoBERT and TALE do. The third is a small transformer that attends from the protein embedding to the revealed terms and outputs a probability for every unrevealed term. Training is order-agnostic. For each protein we sample a random fraction of its true positive terms, close the sample under the ancestor relation, reveal it, and train the model to predict the remaining terms:
$$\mathcal{L} = \mathbb{E}_{R \sim \pi(Y)} \sum_{j \notin R} \mathrm{BCE}\big(y_j,\, f_j(h(x), R)\big) + \lambda \sum_{(i,j)\in E_{GO}} \max(0, f_j - f_i),$$
where $Y$ is the protein's annotated term set, $\pi$ is the random revealing distribution, $f_j$ is the predicted probability of term $j$, $E_{GO}$ is the set of parent $i$ to child $j$ edges, and the second term is the TALE/HMCN hierarchy penalty with weight $\lambda$.
Two design choices respond directly to the weaknesses identified in the research file. First, the model only ever reveals positives. Because GO annotations are incomplete, a missing label is not evidence of absence, so conditioning on "negatives" would push false negatives through the dependency structure, which is the failure mode the file flags for correlation-based models under incomplete supervision. Second, to blunt the snowball effect, a fraction of training reveals are drawn from the model's own high-confidence predictions instead of the ground truth. The model therefore learns to recover from wrong conditioning.
Inference runs for $T$ rounds, with $T$ between 4 and 8. Each round commits the $k$ unrevealed terms with the largest margin $|f_j - 0.5|$, adds their ancestors to $R$ to keep the hierarchy consistent, and rescores the rest. Under this rule the order is decided per protein. A strong motif-driven molecular-function term can be committed first and then condition the broad process terms. A clear compartment term can lead for another protein. The cost is $T$ forward passes of a small head over a fixed embedding, which stays cheap at GO scale. The same procedure runs unchanged in annotation-completion mode, where $R$ simply starts out containing the protein's known terms.
The analysis has three parts. We measure how sensitive the trained model is to order by decoding the same proteins under many random orders. We record which ontology and depth the first-committed terms come from. We break down where any gains fall, by term depth, term frequency, and sequence-identity bin.
Baselines
The naive reference point is LR-ESM: one independent logistic regression per GO term on the same frozen ESM-1b embeddings, followed by true-path repair, where each child's score is capped at its parent's. This reproduces NetGO 3.0's protein-language-model component. NetGO 3.0 reports that LR-ESM alone improves molecular-function results by 21.6% over Seq-RNN. The file gives no absolute Fmax for LR-ESM, so we will measure it ourselves on our split. The strongest published system on this benchmark family is NetGO 3.0 itself, which reports Fmax 0.378 in BP overall and 0.369 in BP on difficult proteins on its Swiss-Prot time split. Because NetGO 3.0 ensembles homology and other components, we will quote its number as an external reference rather than as an equal-compute comparison. Our fair comparison will add DIAMOND to every system in the same way, mirroring TALE+, so that any gain cannot be credited to homology. TALE+ reports an 8% Fmax gain over NetGO in CCO; the file has no absolute value for it.
The strongest competitor on the specific question of ordering is a fixed top-down chain in the style of HMCN-R and HMC-LMLP. We implement it inside our own model: the identical trained network decodes GO depth level by level from the root. Because the network is shared, this comparison isolates the order alone. We also include the T5 Chain recipe of a single global order chosen on validation data from 20 candidates, again with the same network. HMCN-F, the feed-forward model with a violation penalty that ranked 1.07 on average across 21 hierarchical benchmarks, is trained on the same ESM features as a strong parallel baseline that respects the hierarchy. On the small flat benchmarks Yeast and Genbase, we compare against CC-DP, BNCC and PACC, which report average ranks of 1.50 on Example-F1 (CC-DP), parity with a 10-chain ensemble (BNCC), and 2.4 on global accuracy (PACC). These comparisons show whether confidence-first ordering competes with explicit order search where that search is still affordable.
Two ablations isolate the key ingredient. The first is stacking with the same network: all scores from round one are revealed at once, as soft inputs and without any order. The second uses random-order decoding, applies the same confidence-first rule to a model trained without self-prediction reveals, and orders by reverse confidence. All systems share the embeddings, the time split, and a hyperparameter budget of equal GPU hours tuned on the 2020 validation year. To count as beating a baseline, our method must improve BP and MF Fmax on the 2021 test year by more than the seed-to-seed standard deviation over five seeds, and the stacking ablation must not close that gap.
Why not just…?
Why not just stack: predict everything flat, then feed all predictions back in once? Stacking captures label dependencies with no ordering at all, and it is the most serious threat to the proposal. If a single parallel refinement pass matches confidence-first decoding, then ordering does not matter at GO scale, and the right conclusion is a cheaper GoBERT-plus-sequence model. The stacking ablation shares our exact network, so the experiment settles the question directly. The specific failure we expect from stacking is that it conditions on many mid-confidence wrong scores at once, which should show up as lower precision on deep terms.
Why not just average an ensemble of randomly ordered chains? Liu and Tsang report that one well-ordered chain (CC-DP, average rank 1.50) beats such ensembles, and BNCC matches a 10-chain ensemble at about half the training time on Bibtex. Both results come from small flat label sets with linear models, however. We will run random-order decoding with 10 orders using our network, at matched inference compute. This is cheap because the model is order-agnostic. If averaging matches confidence-first decoding, the per-instance order adds nothing.
Why not just train a flat classifier and repair the hierarchy afterwards? HMCN shows that enforcing the hierarchy during training beats post-hoc filtering, and HMC-LMLP's feature augmentation significantly beat flat local strategies on FunCat. Neither result used protein-language-model embeddings, though, and strong embeddings may already encode most label co-occurrence. LR-ESM plus repair is our naive baseline for exactly this reason. A small or nonexistent gap over it would be an informative negative result for the whole chaining line of work.
Why not just use a top-down chain, which already exists? Top-down decoding commits first to broad process terms, which are the least sequence-specific, so early errors there contaminate everything below. Testing top-down against confidence-first with the same network is the cleanest test of whether the direction of conditioning matters.
Proposed Experiments
The main benchmark is the CAFA-style UniProtKB/Swiss-Prot time split used by NetGO 3.0: training data before January 2020, validation during 2020, and testing during 2021. We report Fmax, Smin and AUPR for BP, MF and CC, broken down by "difficult" proteins and by sequence-identity bins below and above 30%, following TALE. Secondary benchmarks are the FunCat yeast datasets, for comparison with HMC-LMLP's top-down chaining on trees, and the GO datasets in the HMCN suite, scored with AU(PRC). Yeast and Genbase serve for direct comparison with CC-DP, BNCC and PACC.
There are four decisive checks. The first measures order sensitivity: for 1,000 test proteins we decode under 50 random orders and report the spread in per-protein F1. If the spread is negligible, the premise fails and we report that. The second is the head-to-head among confidence-first, top-down, bottom-up, single best global order, random-order averaging and stacking, all on one trained network. The claim requires confidence-first to win on deep and rare terms and on low-identity proteins. The third is annotation completion: we reveal 10%, 30% or 50% of each test protein's true terms and compare against LR-ESM, which cannot use them, and against a sequence-free GoBERT-style variant of our model, which separates what the sequence adds beyond label co-occurrence. The fourth tests robustness to wrong conditioning: we corrupt revealed terms with plausible but false sibling terms and measure the drop in Fmax with and without self-prediction training. Recovering from wrong conditioning is the property that would make chained conditioning safe to place inside bottlenecks of the kind GenePheno uses.
References
- On the Optimality of Classifier Chain for Multi-label Classification
- Reduction strategies for hierarchical multi-label classification in protein function prediction
- Hierarchical Multi-Label Classification Networks
- Bayesian network based label correlation analysis for multi-label classifier chain
- Polytree-Augmented Classifier Chains for Multi-Label Classification
- Leveraging Text-to-Text Transformers as Classifier Chain for Few-Shot Multi-Label Classification
- GoBERT: Gene Ontology Graph Informed BERT for Universal Gene Function Prediction
- TALE: Transformer-based protein function Annotation with joint sequence-Label Embedding
- NetGO 3.0: Protein Language Model Improves Large-scale Functional Annotations
- GenePheno: Interpretable Gene Knockout-Induced Phenotype Abnormality Prediction from Gene Sequences