Shortcut Collapse in Linearized Protein Fold Taxonomies

Protein structure databases are not a fair sample of nature. The Protein Data Bank and the AlphaFold Database overrepresent medically important, easy-to-crystallize, rigid proteins, so a handful of folds — TIM barrels, Rossmann folds — dominate while a quarter of clustered structural space has no match to any known family. Machine learning models trained on these databases quietly inherit that skew. At the same time, a separate line of work has shown how to feed hierarchical, tree-shaped data — a taxonomy, a parse tree, a molecule — to an ordinary sequence model by flattening the tree into a bracketed token string, and that such models often succeed by exploiting positional shortcuts rather than actually reasoning about the hierarchy. This proposal asks what happens when you put those two facts together: if we train a sequence model to generate protein classification trees (the Class → Architecture → Topology → Superfamily hierarchy of CATH, or ECOD's domain hierarchy) linearized into token sequences, does the fold-frequency bias of the underlying database cause the model to memorize which paths are popular and collapse on rare folds and unclassified structural "dark matter"? And can bias-aware sampling of the training trees fix that collapse?

The question matters because hierarchical classification is becoming the interface through which models navigate the hundreds of millions of predicted structures now available. ECOD domain classification of 48 whole proteomes from AlphaFold Structure Database using DPAM2 classified 746,349 domains across 536,808 proteins and found that only 57.6% of eukaryotic residues land in well-assigned domains versus 81.1% for bacteria, while Unsupervised domain classification of AlphaFold2-predicted protein structures showed that 24% of structural metaclusters from 15 million AlphaFold structures have no homology to CATH, SCOP, or Pfam. Both works expose the hierarchy and the bias but treat assignment as a search or clustering problem, not a generative sequence modeling problem whose learned distribution we can inspect and correct.

Core Claim

The claim is two-part, one diagnostic and one corrective. First, Transformers trained on tree-linearized fold taxonomies sampled from PDB/AFDB will score well on average path accuracy while failing specifically on the tail: topologies and deep hierarchy branches that are rare in the database. Following the methodology of Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion, which showed that Transformers trained on linearized trees pass standard tests yet fail 100% of the time on under-represented edge cases by relying on position-based heuristics, I claim the failure on rare folds is a shortcut driven by fold frequency, not a capacity limit — so it should follow a measurable frequency-accuracy law, roughly $\text{acc}(\ell) \approx a + b\log f(\ell)$ where $f(\ell)$ is the training-set frequency of leaf $\ell$, and it should degrade under sibling-reordering probes that a structure-driven model would survive. Second, the collapse is correctable by changing what we sample, not just how we weight: a hierarchy-aware, tempered sampling distribution over training trees should substantially raise tail accuracy at a small, bounded cost on head folds. If a plain inverse-frequency loss reweighting does just as well, the corrective claim is wrong, and the experiments are designed to find that out.

Research Program

Three components. The first is dataset construction. I will build training corpora of (protein, taxonomy-path) pairs from two sources: the CATH v4.2/v4.3 domain hierarchy, and the ECOD assignments produced by DPAM2 over the 48 predicted proteomes from ECOD domain classification of 48 whole proteomes from AlphaFold Structure Database using DPAM2. Each domain's input will be encoded either as an amino-acid sequence or as discrete structure tokens from the vector-quantized autoencoder of Learning the Language of Protein Structure, released as instadeepai/protein-structure-tokenizer with codebooks of 4,096 to 64,000 tokens and backbone reconstruction error of roughly 1–5 Å. The target is the classification path flattened by depth-first traversal with explicit backtracking POP tokens, exactly the Seq2Tree scheme of Constrained Sequence-to-Tree Generation for Hierarchical Text Classification, which that paper used for text taxonomies and which has never been applied to a structural taxonomy. A held-out dark-matter set comes from the 24% of DPCstruct metaclusters with no known homology in Unsupervised domain classification of AlphaFold2-predicted protein structures; for these, the correct output is a calibrated refusal into an explicit novelty branch, not a forced assignment to a familiar fold.

The second component is the diagnostic suite. I will fit the frequency-accuracy law above and test whether its slope $b$ flattens under interventions that break positional shortcuts: permuting the order of sibling nodes at test time, evaluating on branches deeper than any seen in training (the regime where the recursion paper found total failure), and counterfactual probes that pair a tail-fold structure encoding with a head-fold path prefix to see which signal wins. I will also run modality splits in the spirit of Training data composition affects performance of protein structure analysis algorithms, which showed that models trained only on X-ray structures degrade badly on NMR and cryo-EM test sets, to ask whether the taxonomy model's errors concentrate on the flexible, solution-state structures that crystallography undersamples.

The third component is the mitigation. Training examples will be drawn not from the natural database distribution but from a tempered one: a domain is sampled with probability

$$p_\tau(\ell) \;\propto\; f(\ell)^{1/\tau},$$

where $f(\ell)$ is the leaf's empirical frequency in the database and the temperature $\tau \geq 1$ interpolates between the raw biased distribution ($\tau = 1$) and a uniform one (large $\tau$). Crucially, tempering is applied at every level of the tree, not just at leaves, so that deep branches and rare architectures are covered at each branching point; this depth-stratified design is what distinguishes the proposal from generic class balancing. Decoding uses the constrained vocabulary scheme of Constrained Sequence-to-Tree Generation for Hierarchical Text Classification so that every generated path is structurally valid, which also removes the post-hoc repair problem that paper documented. The design choice is directly motivated by Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion, which found that physically oversampling rare edge cases in the training data was substantially more effective than increasing their loss weights.

This proposal is deliberately distinct from the nearest neighbors. ProtInvTree: Deliberate Protein Inverse Folding with Reward-guided Tree Search builds an inference-time search tree over residue assignments (reporting a self-consistency TM-score of 0.881 versus 0.816 for ESM-3 on CATH v4.2) but never linearizes a structural hierarchy and treats its CATH splits as unbiased ground truth. Protein Autoregressive Modeling via Multiscale Structure Generation generates backbones coarse-to-fine but through continuous flow-matching decoders, not a discrete tree grammar, and it does not analyze how the PDB/AFDB composition of its training data shapes the generated distribution. LAUGHS: An LLM-compatible Molecular String Representation is the closest methodological ancestor — it linearizes molecular graphs as functional-group trees and beats flat strings decisively (91.4% exact match on localized editing versus 69.9% for SMILES) — but it operates on 2D small-molecule chemistry, ignores 3D conformation and database sampling entirely, and has never been tested on proteins. My proposal takes LAUGHS's representation argument, Seq2Tree's decoding machinery, and the recursion paper's diagnostic lens, and points all three at the sampling-bias problem that the structural-database literature has documented but never connected to sequence-to-tree learning.

Proposed Experiments

The main experiment trains one Transformer architecture (held fixed across conditions) on CATH and ECOD linearized trees under four sampling regimes: natural biased sampling, leaf-level inverse-frequency reweighting, tempered leaf sampling, and the full depth-stratified tempered sampling. Evaluation is on frequency-stratified test slices — head, mid, and tail deciles by topology frequency — plus the DPCstruct dark-matter clusters, plus NMR/cryo-EM modality slices. Primary metrics are path-level exact accuracy, Macro F1 over leaves (so that rare folds count equally), and a path-consistent C-MacroF1 following Constrained Sequence-to-Tree Generation for Hierarchical Text Classification. Success on the corrective claim means tail-decile path accuracy improves materially over natural sampling while head-decile Macro F1 drops by no more than one point. No reported result found on Lacuna for hierarchical classification on CATH or ECOD taxonomies, so all protein-taxonomy numbers will be established by this project's own runs; the only directly comparable published figure is Seq2Tree's text-taxonomy result of 70.01 Macro F1 on RCV1-V2 versus 68.66 for HiMatch-BERT, which calibrates the method's strength but not its protein behavior.

The second experiment is the shortcut diagnosis: fit the frequency-accuracy law per sampling regime, run sibling-permutation and depth-extrapolation probes, and test whether depth-stratified sampling specifically repairs deep-branch performance where leaf-level reweighting does not. The third experiment is dark-matter routing: compare all models on their ability to output calibrated novelty decisions for the 24% of metaclusters with no known homology, measured by an accuracy-rejection curve. A secondary study replicates the modality analysis of Training data composition affects performance of protein structure analysis algorithms at the taxonomy level, asking whether mixing NMR and cryo-EM derived assignments into training — which that paper showed reduced sequence-design perplexity on NMR structures with $p < 10^{-90}$ — also repairs taxonomy errors on flexible proteins. As an anchor for whether taxonomy-level corrections propagate downstream, I will track whether better-calibrated hierarchy predictions improve zero-shot stability scoring in the setup of Zero-shot protein stability prediction by inverse folding models: a free energy interpretation, where explicit corrections for database bias raised Pearson correlation on Protein G from 0.66 to 0.72; this is exploratory, not a headline claim.

Baselines

The do-nothing reference point is the per-level frequency prior: at each hierarchy level, predict the most common child of the current node, ignoring the input protein entirely. On a database where a few folds dominate, this is deceptively strong on average accuracy and defines the floor that any input-driven model must beat; a model that fails to beat it on the head decile is not using structure at all, which is itself a finding. The strongest standard machine-learning competitor is a flat multi-label classifier — one independent sigmoid per taxonomy node over the same frozen input encodings, with predicted node sets projected onto the nearest valid path. This is the comparison Constrained Sequence-to-Tree Generation for Hierarchical Text Classification explicitly ran, and it matters because that paper found post-hoc repair of flat predictions actually decreased performance by injecting noise, while its tree-linearized generator reached 70.01 Macro F1 against 68.66 for HiMatch-BERT on RCV1-V2. The strongest existing tree-linearization method, and simultaneously the key ablation for this proposal, is Seq2Tree with constrained decoding trained on the natural biased database distribution: same architecture, same tokenizer, same compute and tuning budget as every other condition, differing only in the training-sample distribution. Beating it means beating it where the bias lives — tail-decile and dark-matter slices — not on head accuracy, which biased training will win by construction. Finally, for the assignment task on known folds, the non-generative competitor is alignment-based assignment in the style of DPAM2 from ECOD domain classification of 48 whole proteomes from AlphaFold Structure Database using DPAM2; on familiar folds it may well win, and the honest target is the dark-matter and deep-branch slices where homology-based methods have nothing to align to. All generative baselines share the identical data, tokenizer, parameter count, and tuning budget; only the sampling distribution and decoding constraint vary.

Why not just…?

Why not just use flat structural token sequences and a plain classifier? The evidence against flat representations is consistent across two domains: LAUGHS: An LLM-compatible Molecular String Representation beat flat SMILES by 21.5 points on localized editing (91.4% versus 69.9% exact match), and Code Representation Learning Using Prüfer Sequences (Student Abstract) improved code summarization by roughly 5% on one dataset and up to 15.11% on another over flat-traversal baselines. Still, this is a real threat for average accuracy, which is why the flat multi-label classifier is run as the strongest standard baseline rather than argued away. The decisive test is not head-decile accuracy but tail-decile and dark-matter performance, where the flat model has no mechanism enforcing valid deep paths and its post-hoc repair is documented to hurt.

Why not just reweight the loss by inverse fold frequency? This is the cheapest fix and must be taken seriously. Three papers in the file report that reweighting underperforms data-level intervention: Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion found loss weighting failed where physical oversampling of rare cases succeeded; Training data composition affects performance of protein structure analysis algorithms found loss tuning on X-ray data could not substitute for actually mixing NMR and cryo-EM data into training; and Zero-shot protein stability prediction by inverse folding models: a free energy interpretation found bias correction required explicit reference-state modeling, not reweighting. My hypothesis is sharper than "rewriting is bad": leaf-level reweighting fixes the leaf marginal but leaves deep branches and mid-level rare architectures undersampled, so it should fail specifically on the depth-extrapolation and deep-branch probes. The main experiment includes inverse-frequency reweighting as a first-class condition precisely so this can be settled empirically; if it matches depth-stratified sampling everywhere, the corrective half of the proposal is falsified and I will say so.

Why not just use structural alignment (DPAM2/DPCstruct-style) and skip learning altogether? For known folds this is genuinely the strongest option and I expect it to win the head deciles — it requires no training and no bias analysis. But it cannot solve the problem this proposal targets: by construction, the 24% of DPCstruct metaclusters with no homology to CATH, SCOP, or Pfam have nothing to align to, and DPAM2 itself leaves 42.4% of eukaryotic residues outside well-assigned domains. Alignment gives a binary "no match" with no calibrated sense of how novel a structure is, whereas a generative model over the tree can place a structure at the right depth of the hierarchy (correct architecture, unknown topology) and quantify its uncertainty. The dark-matter routing experiment, scored by accuracy-rejection curves, is designed to be the head-to-head that settles this; if alignment-plus-threshold matches the generative model's calibration there too, the practical case for the learned approach weakens considerably.

Decisive Checks

Three outcomes would kill specific parts of the proposal, and the experiments are built to detect them early. If the frequency-accuracy law is flat — if tail-fold accuracy does not degrade with training frequency — then the diagnostic claim of shortcut collapse is wrong and the project becomes a negative result about the robustness of sequence-to-tree models to sampling bias, still worth reporting. If leaf-level inverse-frequency reweighting matches depth-stratified tempered sampling on every slice including deep branches, the corrective claim reduces to standard class balancing and the hierarchy-aware machinery is unjustified. And if the flat classifier matches the tree-linearized model on tail and dark-matter slices, then the representation argument inherited from LAUGHS and the Prüfer-sequence work does not transfer from chemistry and code to protein taxonomy. Each check uses the same trained models and splits as the main experiment, so none requires new infrastructure, and all three can be evaluated within the first two experimental phases.