Language models used as bioinformatics assistants increasingly answer questions such as "what does this protein do?" by writing out an identifier like the Enzyme Commission (EC) number 3.4.21.62. An EC number is a four-level address in a function tree: class, subclass, sub-subclass, and a serial number within that group. A subword tokenizer splits text into pieces based on how often character strings co-occur in its training corpus, and it knows nothing about that tree. The model therefore never sees "3.4.21" as one shared parent. It sees whatever fragments the tokenizer produced, which may cut across levels. Two enzymes that share a parent may share few output tokens, so the model cannot pool evidence across the rare leaves of the tree. We propose to retokenize the output side so that each generated token is exactly one level of the hierarchy, conditioned on the levels already emitted. We then test whether this one change gives a small generative annotator the benefits that factorized classification heads have shown in other domains: borrowed statistical strength for rare labels, no invalid labels, and the ability to stop at a confident partial label such as "3.4.21.-".
This matters for the accessibility direction for a concrete reason. Natural-language analysis agents such as CellAgent currently depend on large proprietary models, which brings API cost and the risk of hallucinated biological conclusions. If small open models are going to replace them on ordinary lab hardware, they have to emit structured biological labels reliably and cheaply. Our claim is that the bottleneck is partly a tokenization problem on the output side, and that it is fixable without enlarging the model.
Core Claim
When a generative model predicts a label from a tree, hierarchy-aligned output tokenization should help in three ways compared with generic subword tokenization of the same label strings. The setup is one token per tree level, drawn from a vocabulary indexed by (level, value), and decoded under a constraint that only allows children that exist in the tree. We expect it to:
- improve accuracy on rare leaves;
- eliminate invalid outputs;
- give calibrated early stopping at intermediate depth.
It should do all three at a fraction of the output vocabulary of an atomic one-token-per-label scheme. The gain must come from aligning tokens with the semantic tree. A control that aligns tokens to a randomly permuted tree of identical shape should lose most of the benefit.
This is not the same as factorizing the input vocabulary, and not the same as attaching parallel classification heads. It treats the label string itself as a tokenization target. That way the generative interface used by accessible tools inherits the structure that classification-head methods exploit.
Where This Sits in Prior Work
Evidence that output factorization helps. Several papers show that splitting a composite label into structured parts helps rare and unseen combinations:
- Bengali grapheme recognition (Alam et al.) predicts root, vowel diacritic, and consonant diacritic with separate heads. It generalizes to out-of-dictionary graphemes with only a 1.31% higher error rate.
- FLOAT splits object-part labels across parallel decoders.
- Vilares et al. decompose constituent-tree labels into sub-tasks to reduce label sparsity.
- In enzymes specifically, EPP-HMCNF shares parameters vertically and horizontally across the EC tree and beats flat baselines.
All of these use classification heads. None asks what happens when the same label is produced as text by a generative model, which is exactly the setting where tokenization intervenes.
Evidence that the label string's tokenization should matter. These papers describe the failure we expect to see:
- Specialised or Generic? shows that generic tokenizers fragment domain terms. This roughly doubles sequence length and lowers summarization quality.
- Gambardella et al. show that awkward tokenizations can make models prefer outputs that tokenize cleanly over outputs that are correct.
- Truong et al. argue that models spend capacity reassembling meaning that the tokenizer has broken apart.
All three concern natural-language input or output text. We carry the argument over to structured identifiers.
The closest technical relatives work on the input side. FACTORIZER factorizes each subword into three small codebook indices and cuts embedding parameters by over 97%. KinyaBERT encodes words through their morphemes. Adaptive Protein Tokenization learns coarse-to-fine protein structure tokens whose early prefixes describe the whole protein, and supports entropy-based stopping.
We reuse two ideas from these papers, applied to the output rather than the input:
- Factored codes, borrowed from FACTORIZER.
- Prefix-meaningful sequences with confidence-based stopping, borrowed from APT. APT stops once its structure tokens are confident enough. We stop at the deepest EC level we are confident about.
The enzyme classifiers we must compete with.
- PoinnCARE embeds the EC tree in hyperbolic space and fuses structure graphs.
- CLEAN-Contact adds contact maps to contrastive sequence embeddings.
- ProtGO brings the Gene Ontology graph into function prediction. Its finding that an ESM-150M model with structured label knowledge beats vanilla ESM-15B is the strongest evidence in the file that label structure can substitute for scale.
ProtGO also prunes rare GO terms, which is the long tail we target.
One honest constraint. The fourth EC digit is a serial number. Truly unseen leaves therefore cannot be predicted compositionally the way unseen Bengali graphemes can. Our zero-shot claim is limited to emitting the correct partial label for proteins whose leaf is absent from training.
Research Program
The model. A frozen protein language model encoder (ESM-2 at roughly 150M parameters, the size ProtGO shows benefits from label structure) feeds a small autoregressive decoder of a few million parameters. The decoder emits a set of EC labels separated by a delimiter, so promiscuous enzymes with several functions are handled naturally. The only variable we change is the output vocabulary and decoding rule:
- Generic subword. A standard BPE tokenizer trained on general text, applied to the EC strings.
- Domain BPE. BPE retrained on EC strings, analogous to the domain tokenizer in the radiology study.
- Atomic. One token per full EC number.
- Character. One token per digit and dot.
- Hierarchy-aligned. One token per level, indexed by (level, value).
- Hierarchy-aligned with trie constraint. The same, with decoding restricted to children that exist under the emitted prefix.
Diagnostic before training. We first measure how often each tokenizer's boundaries coincide with tree-level boundaries across all EC numbers in the training split. If generic BPE turns out to align almost perfectly with levels, the premise weakens, and we report that before going further.
Partial-label stopping. The decoder may emit a stop token after any level. We train with a nested-truncation augmentation in the spirit of APT's nested dropout: we randomly truncate target labels to shallower prefixes, weighting each truncation by how ambiguous the training labels are at that depth. At test time, a confidence threshold on the next-level token trades depth against precision.
Accessibility component. We report peak memory, latency per protein, and output-layer parameter count on a single consumer GPU. We release the annotator as a callable tool whose outputs are guaranteed to be valid, so that an open-model agent could call it in place of free-text generation.
Extension to the Gene Ontology. As a second-stage test, we extend hierarchy-aligned tokens to Gene Ontology terms by linearizing each term as its shortest path from the root. Because GO is a directed acyclic graph where a term can have several parents, this is a harder and explicitly exploratory test on CAFA5.
Baselines
Naive reference point. The decoder with generic subword tokenization of EC strings, decoded freely. This represents what a general-purpose language model does today when asked for an EC number. Beating it is necessary but not sufficient.
Atomic-token decoder. One token per full EC label. This is the generative equivalent of a flat softmax. It is the fair test of whether factorization helps at all once tokenization artifacts are removed. The flat-softmax literature cited in the research file (Bhatia et al.) predicts that it collapses on rare leaves.
Constrained generic BPE. Generic BPE decoding restricted by a trie of valid EC strings. This is the ablation that isolates our key ingredient. It removes invalid outputs exactly as our method does, but leaves tokens misaligned with the tree. Any gain we retain over it can be attributed to alignment rather than validity.
Permuted-tree control. Hierarchy-aligned tokens built from a random relabeling of the EC tree that preserves its branching shape but scrambles which leaves share parents. This separates semantic sharing from mere sequence length and vocabulary size.
Strongest discriminative competitor. A hierarchical multi-label classification network in the style of EPP-HMCNF, trained on the same frozen ESM-2 embeddings. It exploits exactly the tree sharing we hope to recover, without a generative interface. We also run nearest-neighbor retrieval over contrastive embeddings in the style of CLEAN, since that is the standard sequence-only enzyme annotator.
Fairness rules. All in-house systems use the same embeddings and the same training splits. The decoder and classifier head have matched parameter counts, and every system gets the same hyperparameter search budget, with each configuration's seeds trained to the same compute.
External references. We cite PoinnCARE and CLEAN-Contact as references, noting that they use 3D structure or contact maps and so receive more input information. The research file reports their results only relative to CLEAN:
- PoinnCARE: a 10.4% Level-4 accuracy improvement over CLEAN on the CARE split with under 30% sequence identity.
- CLEAN-Contact on New-392: a 16.22% precision gain and a 12.30% F1 gain over CLEAN, plus a 30.4% precision gain on enzymes with fewer than 11 training instances.
No absolute value for any method on these benchmarks was found on Lacuna, so we will rerun CLEAN ourselves to anchor the scale.
What counts as success. We beat the naive and constrained-BPE baselines on rare-leaf F1 by a margin larger than the seed variance. We match the hierarchical classification head within a small gap at Level 4. We clearly beat that head on calibrated partial-label precision.
Why not just…?
Why not just use a hierarchical classification head and skip generation? This is the most serious threat, and the proposal is built to measure it rather than dodge it. On fixed embeddings, a classifier like HMCN-F may well match or beat any decoder at Level 4. If it does, our contribution narrows to a different claim: when a generative interface is required, as in agent tools that read and write text, hierarchy-aligned tokens close most of the gap that generic tokenization opens. The matched-parameter comparison against the hierarchical head reports that gap directly. Separately, a set-valued decoder handles promiscuous enzymes and variable-depth answers without per-level thresholds, and the CARE promiscuous split tests whether that matters.
Why not just constrain decoding to valid EC strings and keep the existing tokenizer? A trie constraint fixes invalid outputs but does not change which output embeddings rare leaves share with frequent siblings. The constrained-BPE ablation is the decisive comparison here. If it matches hierarchy-aligned tokens on rare-leaf F1, alignment adds nothing beyond validity, and we will say so.
Why not just add one special token per EC number? This is the atomic baseline. The evidence on flat label spaces, including ProtGO's need to prune rare GO terms and EPP-HMCNF's gains from vertical sharing, predicts that rare leaves suffer when nothing is shared. Atomic tokens also cannot express "3.4.21.-" without separate tokens for every prefix. We test this directly in the frequency-binned evaluation.
Why not just use a bigger model? ProtGO reports that ESM-150M with label structure beats vanilla ESM-15B on CAFA5. Scale also works against the accessibility goal. We nonetheless include one larger-encoder run with generic tokenization, to check whether scale alone erases the rare-leaf gap.
Proposed Experiments and Decisive Checks
Datasets. The main benchmark is CARE, using its under-30% identity, 30–50% identity, Price, and promiscuous splits. New-392 and Price-149 serve as external test sets. CAFA5 is used for the Gene Ontology extension.
Metrics.
- Per-level accuracy and Level-4 F1.
- F1 binned by training-set frequency of the leaf, including the under-11 bin used by CLEAN-Contact.
- Invalid-label rate and hierarchical consistency.
- Risk–coverage curves for partial-label stopping, with precision at each emitted depth.
- For held-out proteins whose leaf is missing from training, the rate of emitting the correct deepest available prefix.
- Output parameters, peak memory, and latency on a consumer GPU.
Three outcomes decide the idea.
- If hierarchy-aligned tokens do not beat constrained generic BPE on rare-leaf F1, the core claim fails.
- If the permuted-tree control performs as well as the true tree, the effect comes from token count, not semantics, and the claim fails.
- If partial-label stopping does not produce higher precision at intermediate depth than thresholding the hierarchical classifier's per-level probabilities, the accessibility argument for a stoppable generative annotator weakens to a matter of convenience.
A positive result on all three would show that a misalignment between the tokenizer and the label tree is a measurable, cheap-to-fix source of error in generative biological annotation.
References
- CellAgent: LLM-Driven Multi-Agent Framework for Natural Language-Based Single-Cell Analysis
- A Large Multi-target Dataset of Common Bengali Handwritten Graphemes
- FLOAT: Factorized Learning of Object Attributes for Improved Multi-object Multi-part Scene Parsing
- Better, Faster, Stronger Sequence Tagging Constituent Parsers
- Enzyme promiscuity prediction using hierarchy-informed multi-label classification
- Specialised or Generic? Tokenization Choices for Radiology Language Models
- Inconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar
- Revisiting subword tokenization: A case study on affixal negation in large language models
- Tokenization with Factorized Subword Encoding
- KinyaBERT: a Morphology-aware Kinyarwanda Language Model
- Adaptive Protein Tokenization
- PoinnCARE: Hyperbolic Multi-Modal Learning for Enzyme Classification
- Improved enzyme functional annotation prediction using contrastive learning with structural inference
- ProtGO: universal protein function prediction utilizing multi-modal gene ontology knowledge
- Automatic Classification of Communicative Functions of Definiteness