Protein structure databases now hold millions of predicted structures, and a growing share of them do not match any known fold. The useful question for such a protein is often not "which of the known folds is it?" but "how far up the family tree of shapes do I have to climb before it fits?" It might be a new topology within a familiar architecture, or something stranger. Current deep fold classifiers cannot answer this. They place every fold class at an equal distance from every other, so the distance from a new protein to the nearest class says nothing about where in the hierarchy it belongs. We propose to build the hierarchy directly into the geometry of the classifier. Each fold is represented by a fixed "prototype" direction on a unit sphere, and these directions are arranged so that folds sharing an architecture or class lie close together. A protein's embedding is trained to sit near its own prototype, with a safety gap whose size scales with how different the competing fold is. The gap is set so that it can always be met given the prototype layout. At test time a protein that fits no known fold should still land inside the right architecture's region. Coarse-level placement and fine-level rejection then become one geometric test.
The input side uses the pairwise distance map: an $N\times N$ image of distances between all residue pairs, computed from a predicted or experimental structure. It is paired with a protein language model embedding, following the dual-stream design of CLEAN-Contact. The contribution is not the input. It is the claim that a taxonomy-shaped prototype layout, combined with margins tied to that layout, improves error severity and unseen-fold placement over the flat, equidistant prototypes used in prior protein work. We expect it to do this while costing little or nothing in closed-set accuracy.
What exists and what is missing
The closest work is FoldHSphere. It fixes 1,154 SCOPe fold prototypes on a hypersphere by solving the Thomson problem, which spreads points as far apart as possible, like mutually repelling electrons. It then trains a 1D sequence network on evolutionary profiles to hit those prototypes with a large-margin cosine loss. Its ensemble reaches 81.3% fold-level accuracy on LINDAHL. Villegas-Morcillo et al. apply the same cosine-margin loss to protein language model embeddings, with ordinary learned class weights instead of fixed prototypes, and report 82.6% top-1 fold accuracy on LINDAHL for ProtT5 with a light-attention head. Both papers supply our method's skeleton: a hyperspherical embedding, a margin loss, and fixed prototypes. Both also motivate the gap, because their layouts are flat. In Thomson prototypes, two topologies in the same CATH architecture are as far apart as an all-alpha fold and an all-beta fold.
Evidence that flatness matters comes from outside biology and from inside it. In vision, HiR² found that pure spherical dispersion erodes parent–child structure. It needed a separate hyperbolic entailment term to recover hierarchical consistency on iNaturalist; hierarchical consistent accuracy rose from 14.06 to 18.07 on iNat-Animal. Inside biology, the TED candidate-novel-fold analysis found that many "novel" domains are insertions in, or diverged relatives of, existing ECOD domains. About 34% of well-assigned domains had the candidate novel fold embedded inside a known domain. Novelty in fold space is therefore mostly partial, and a classifier should say how novel a domain is.
On the input side, 2D distance maps have been classified with convolutional networks under cross-entropy, by Zhu et al. for twilight-zone folds and by DPAM-AI for domain parsing. CLEAN-Contact fused ESM-2 embeddings with contact maps under a Euclidean triplet loss and gained 16.22% precision over sequence-only CLEAN on New-392. None of these uses angular margins or prototypes. Conversely, angular-margin prototypical networks on endoscopy images show that adding an additive angular margin to prototype distances helps few-shot classification. They report roughly 2% over plain cosine and more than 6% over triplet Siamese networks, but with flat classes and no structure. The obvious combination, swapping CosFace for ArcFace on a distance-map backbone, would amount to a loss-function swap with an expected effect of a point or two. We treat that swap as an ablation, not as the claim.
Core Claim
Arranging fixed prototypes according to the CATH hierarchy, and making each pairwise angular margin a fixed fraction of the angle between the two prototypes, yields three benefits over equidistant prototypes. First, errors are milder: mistakes stay within the correct architecture. Second, proteins from topologies never seen in training can be placed at the correct architecture and class and flagged as new at the topology level. Third, a new topology can be added from a handful of examples by averaging their embeddings, and this works better because the average lands in a region the hierarchy has already reserved.
Research Program
Prototype construction. We give every CATH node a unit vector $u$, laid out so that siblings are spread apart as in a Thomson problem restricted to the siblings of one parent. The prototype of topology $t$ with architecture $a$ and class $c$ is $$p_t = \frac{\alpha_1 u_c + \alpha_2 u_a + \alpha_3 u_t}{\Vert \alpha_1 u_c + \alpha_2 u_a + \alpha_3 u_t \Vert},$$ where the weights $\alpha_1 > \alpha_2 > \alpha_3$ set how strongly each level pulls on the prototype. The angle between two prototypes then grows with the depth of their lowest common ancestor. The weights are chosen on a validation split. Setting them all equal and removing the tree recovers a FoldHSphere-style layout, which gives us a clean control.
Margins tied to the geometry. A uniform angular margin is infeasible when two sibling prototypes are closer than twice the margin, which is one concrete source of the tuning sensitivity reported for margin losses. We instead set a margin for each pair, equal to a fixed fraction $\lambda \in (0,1)$ of half the angle between the two prototypes, and use the pairwise form of the additive angular margin loss: $$\mathcal{L} = \log\Big(1 + \sum_{j \neq y} \exp\big(s\,[\cos\theta_j - \cos(\theta_y + m_{yj})]\big)\Big), \qquad m_{yj} = \tfrac{\lambda}{2}\,\angle(p_y, p_j).$$ Here $\theta_j$ is the angle between the normalized embedding and prototype $p_j$, $y$ is the true topology, and $s$ is a scale factor. Because the margin never exceeds half the prototype separation, the target region is always non-empty. The margin is also largest against distant folds, where confusion would be most damaging.
Encoder. The encoder has two streams. One is a ResNet over the domain-cropped distance map, with residue pairs down-weighted by AlphaFold predicted aligned error. This addresses the finding in Cong that missed regions track low model confidence. The other stream is a pooled ESM-2 embedding. The two are fused by concatenation and projected onto the sphere.
Placement rule. At inference, the model descends the tree. At each level it accepts the best-matching node if the embedding's angle to that node's direction is below a per-level threshold calibrated on validation data. It stops and reports "new at this level" otherwise.
Adding a new topology from a few examples. For few-shot addition, a new topology's prototype is the normalized mean of its support embeddings, following Prototypical Networks. We compare this with a version that blends in its predicted parent direction.
Data. The main data is TEDBench: about 462k AlphaFold-derived structures with CATH labels, plus a 27,638-structure experimental CATH v4.4 test set. Because TEDBench labels a whole protein by its largest domain, we crop distance maps to the TED boundary of that labeled domain. We report that restriction honestly rather than claiming multi-domain handling. We build a held-out-topology split from TEDBench in which entire topologies, about 10%, are removed from training, stratified by architecture. LINDAHL provides comparability with the 1D angular-margin literature. The TED candidate-novel-fold set provides an unlabeled stress test.
Baselines
The naive reference point is nearest-neighbour retrieval on mean-pooled ESM-2 embeddings with no training. It shows how much any learned geometry adds. On the closed-set task we also report a linear softmax head on the same fused encoder, trained with the same budget.
The strongest existing closed-set methods on TEDBench are the structural encoders reported in the TEDBench paper. MiAE-B reaches 74.56 macro-F1, and SaProt-650M reaches 73.48 macro-F1, against 66.19 for ESM2-650M. These are the numbers to meet. We do not claim to beat them on closed-set macro-F1. Beating them means matching within about one macro-F1 point while clearly improving the metrics below. If released MiAE features are usable, we also train our prototype head on top of them to separate the geometry's effect from the encoder's. On LINDAHL, the reference numbers are 82.6% fold-level top-1 for ProtT5 with light attention (Villegas-Morcillo et al.) and 81.3% for FoldHSpherePro (FoldHSphere). We treat these as sanity checks, since that benchmark does not test hierarchy.
The competitor that isolates our claim is a FoldHSphere-style model re-implemented on our exact encoder, data and tuning budget. It uses Thomson-equidistant prototypes over CATH topologies with a uniform cosine margin. We run it in two variants: the original cosine margin, and a uniform additive angular margin. The key ablations form a two-by-two design: flat versus tree-shaped layout, crossed with uniform versus geometry-tied margin. A further ablation turns off the distance-map stream. Every variant gets the same number of hyperparameter trials, 30 random-search configurations over scale, margin and learning rate, three seeds, and identical epochs. The claim is supported if the tree-shaped, geometry-tied model beats the matched flat model on three measures: unseen-topology architecture placement, fold-level rejection, and the share of errors that stay inside the correct architecture. Each gain must hold across seeds and exceed seed variance.
Why not just…?
Why not just use a protein language model with a cosine-margin head, as Villegas-Morcillo et al. did? It is strong: 82.6% on LINDAHL. It is also cheap. But learned class weights under a flat margin carry no guaranteed meaning in the angles between classes, so the model has no principled way to place an unseen topology. The same model is our "flat layout, learned weights" arm. If it matches the tree layout on unseen-topology placement, our claim fails, and we will report that.
Why not just train separate softmax heads for class, architecture and topology? This is the most serious threat. It encodes the hierarchy with no geometric machinery, and it can predict an architecture for a protein whose topology is unseen. We run it on the same encoder. What it lacks is a single embedding in which the distance to a prototype is calibrated across levels. Its "new topology" decision must come from low softmax confidence, which is known to behave poorly under distribution shift. The held-out-topology experiment decides between the two. If multi-head softmax matches placement accuracy and rejection AUROC, the geometric construction is not justified and the paper becomes a negative result.
Why not just align the new structure against the database with Dali? Structural alignment is what DPAM-AI and the TED analysis actually use to assign folds, and it may be the most accurate placement method available. We found no reported result on Lacuna for Dali-based nearest-neighbour placement on TEDBench or on a held-out-topology split, so we run it ourselves on a subsample. We report accuracy and wall-clock cost per query. Our method competes on the second axis, and on producing a calibrated level of novelty rather than a ranked hit list.
Why not just encode 3D coordinates directly instead of distance maps? MiAE shows that invariant 3D encoders are strong on TEDBench. The prototype geometry does not depend on the backbone, so we test it on MiAE features where possible. We keep distance maps as the primary input for two reasons: they are invariant to rotation for free, and they let us down-weight low-confidence regions pair by pair using predicted aligned error.
Proposed Experiments
On the held-out-topology split, the decisive measurements are three. The first is architecture-level placement accuracy for proteins from unseen topologies. The second is the area under the ROC curve for flagging them as new at the topology level. The third is, for misclassified seen-topology proteins, the fraction whose predicted fold shares their true architecture. We also measure the Spearman correlation between embedding angle and CATH tree distance on the CATH v4.4 external set, which tests whether the geometry generalizes from predicted to experimental structures. For few-shot addition, we reintroduce held-out topologies with 1, 5 and 10 support examples and measure topology accuracy, comparing plain support means with parent-blended means and flat against tree-shaped layouts. A closed-set check on TEDBench macro-F1 and LINDAHL fold accuracy confirms that the hierarchy costs nothing. Finally, on the TED candidate-novel-fold domains, we check whether domains that the TED analysis found embedded within known ECOD domains are placed at shallower novelty levels than the remaining candidates. This gives an independent, label-free test of whether the "how novel" readout means anything. If the tree-shaped layout wins only on error severity but not on unseen-topology placement, we will present it as a modest calibration improvement rather than a discovery tool.
References
- FoldHSphere: deep hyperspherical embeddings for protein fold recognition
- An analysis of protein language model embeddings for fold prediction
- Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models
- Using evolutionary context to classify difficult protein folds
- Improving protein fold recognition by extracting fold-specific features from predicted residue-residue contacts
- DPAM-AI: a domain parser for AlphaFold models powered by artificial intelligence
- Improved enzyme functional annotation prediction using contrastive learning with structural inference
- Additive Angular Margin for Few Shot Learning to Classify Clinical Endoscopy Images
- Prototypical Networks for Few-shot Learning
- Protein Fold Classification at Scale: Benchmarking and Pretraining