Finite-Sites Attribute Priors: Repairing Unseen-Class Descriptions with Loss-Aware Trait Evolution

Zero-shot recognition systems identify categories they have never seen by matching images against a description of each class. The description is usually a vector of attributes such as "has webbed feet" or "striped." These systems are only as good as that vector, and in practice it is often partial, noisy, or written by a language model that makes things up. When methods have tried to fill those gaps using the class taxonomy, they have assumed something cancer biologists already had to drop: that a trait, once gained in a lineage, is kept by every descendant. Dolphins inherit no legs from their mammal ancestors. Penguins do not fly. In tumor genomics this assumption is called the infinite sites assumption. Relaxing it, by explicitly modeling losses, repeated independent gains, and technical dropouts, changed which mutations could be detected at all.

We propose to bring that relaxed model into attribute-based generalization. Each attribute is treated as a character that evolves along the class tree by gain and loss, under a continuous-time Markov model of the kind used in finite-sites phylogenetics. This model serves as a prior that infers missing or corrupted attribute values for unseen classes. A separate observation layer distinguishes an attribute that is truly absent in a class from one that is merely invisible in a particular image. The core idea fits in two sentences: treat attributes as evolving traits that can be lost or re-gained along the taxonomy, and use that evolutionary model to fill in and clean the descriptions zero-shot classifiers depend on. The test is whether loss-aware inference beats both the monotone hierarchical transfer used today and plain smoothing over taxonomic neighbors.

Core Claim

Hierarchical attribute transfer in zero-shot learning embeds a monotone inheritance assumption. HAT propagates attributes bottom-up through WordNet and combines ancestral and child-versus-parent classifiers. It has no way to represent a clade that loses an ancestral trait, and it can only explain a trait shared by distant clades by placing it at a general ancestor, which dilutes it. The genomics literature shows what that assumption costs. When single-cell tumor callers ignore deletions, DelSIEVE reports that its predecessor SIEVE reaches a median F1 of 0.65 on double mutants in simulated data with deletions, against DelSIEVE's above 0.93. On the TNBC16 dataset, SIEVE finds 454 mutation events where DelSIEVE finds 689, roughly 39% of which involve deletions. Tomlinson et al. and Zhang et al. show that enforcing the assumption on violating data can leave no valid tree at all, or can inflate the number of recombination events needed to explain the data. Those papers motivate the work. DelSIEVE and COMPASS supply the method, through their central inferential device: an absence shared by a whole clade is evidence of a real loss, whereas an absence in one isolated leaf is more likely technical dropout.

Our claim is specific. When attribute vectors for unseen classes are partially missing or noisy, a gain/loss Markov model on the class tree recovers them more accurately than monotone propagation or tree-agnostic completion. The recovered vectors raise zero-shot harmonic-mean accuracy. We also predict where the gains will appear: on attributes the model assigns high loss or re-gain rates, and on classes sitting below inferred loss branches. If the gains are spread uniformly instead, the evolutionary structure is not what is helping.

Research Program

The model has two layers. The first is the trait-evolution layer. For each attribute $m$, let its presence at every node of the class tree $T$ follow a two-state continuous-time Markov chain with gain rate $\alpha_m$ and loss rate $\beta_m$:

$$Q_m = \begin{pmatrix} -\alpha_m & \alpha_m \\ \beta_m & -\beta_m \end{pmatrix}, \qquad P_m(t) = e^{Q_m t},$$

Here $P_m(t)$ gives the probability of each state after a branch of length $t$. Setting $\beta_m = 0$ and allowing only a single gain recovers the monotone, infinite-sites-like regime that HAT implicitly uses. Letting both rates be free allows loss and repeated independent gain, which is the finite-sites regime. We start with this two-state model rather than DelSIEVE's seven-genotype space because attributes lack the allelic structure that motivates the larger space. Graded attribute strengths would extend it to a few ordered states. Rates are shared through a small hierarchical prior across attributes, so that rare attributes borrow strength. They are fitted by maximizing the Felsenstein pruning likelihood (the standard recursive way of summing over unobserved ancestral states) on the seen classes' attribute matrix. Missing entries for unseen classes are then imputed from their posterior given everything else on the tree.

The tree comes from WordNet as in HAT. For AwA and CUB we also build a variant from an external species-level taxonomy, to check whether conclusions depend on the tree source. Branch lengths are either set uniformly per taxonomic level or fitted jointly. Both are reported.

The second layer handles observation. For per-image attribute evidence, such as a backbone's attribute scores or CUB's image-level annotations, we write the probability of observing attribute $m$ in image $x$ of class $c$ as

$$p(o_{xm}=1 \mid z_{cm}) = z_{cm}(1-\delta_{xm}) + (1-z_{cm})\,\epsilon_m,$$

where $z_{cm}$ is the class-level state from the evolutionary layer, $\delta_{xm}$ is an image-specific dropout probability (occlusion or viewpoint), and $\epsilon_m$ is a false-positive rate. This is the visual counterpart of allelic dropout in single-cell sequencing. A trait that is missing across all images of a whole clade pushes the evolutionary layer toward a loss. A trait missing only in occluded images is absorbed by $\delta$. This replaces the heuristic down-weighting of absent descriptions in TAP and the prototype matching in CSAP-ZSL with an explicit likelihood. We use it for two purposes: to clean class vectors, and to produce attribute evidence for the classifier that accounts for visibility.

The repaired class vectors are passed to an unchanged downstream zero-shot classifier. This makes the contribution easy to isolate: we change the class descriptions, not the recognizer. Pairwise dependencies between attributes, analogous to the promote/inhibit structure of TreeMHN, are deliberately out of scope. TreeMHN is a cautionary constraint here, because its authors note it is itself restricted to monotone accumulation. Adding dependencies before establishing that loss-awareness helps would confound the result.

Baselines

The naive reference point leaves corrupted entries at the class mean, or at zero, and feeds the damaged vectors to the classifier. This measures how much harm corruption does.

The strongest existing recognizer is CSAP-ZSL, which reports a generalized zero-shot harmonic mean of 74.6% on AWA2, 70.6% on CUB and 52.2% on SUN using clean attribute vectors. We use it as a fixed downstream classifier, retrained once on seen classes with the published setting. Every imputation method then hands it vectors for unseen classes. The clean-vector numbers are the ceiling, and beating a competitor means recovering more of the gap between the corrupted-vector score and that ceiling. For the noisy- or partial-attribute regime itself we found no reported result on Lacuna for CSAP-ZSL or any other method, so all corrupted-setting numbers will be produced by us under identical seeds and splits.

The strongest existing structured transfer method is HAT. It reports multi-class accuracy of 74.9% on AwA, 51.8% on CUB and 45.4% on aPaY with GoogLeNet features, and a 31% improvement over direct attribute prediction on CUB. We reimplement its bottom-up propagation as an imputation rule on the same tree and features.

The key ablation is our own model with the loss rate fixed at zero and a single gain allowed per attribute. This is the infinite-sites version of the proposal, with identical trees, likelihood, tuning budget and code. Any gap between it and the full model is attributable solely to permitting loss and re-gain.

Two tree-agnostic or tree-shallow competitors complete the set. One is low-rank matrix completion of the class-attribute matrix. The other is sibling averaging, which fills an unseen class's missing entries with the mean over its nearest taxonomic relatives. All methods get the same hyperparameter search budget on a validation split of seen classes, with a subset held out as pseudo-unseen.

Why not just…?

Why not just use the full class-attribute matrix as modern zero-shot methods do, with no taxonomy at all? A flat per-class vector already says penguins do not fly, so in the clean setting the infinite-sites problem never arises. We concede this. Our proposal does not claim to help when vectors are complete and correct. Its regime is unseen classes whose descriptions are partial, crowd-sourced, or generated by a language model. Our clean-vector control should show near-zero gain, and that is the expected result rather than a failure.

Why not just average the attributes of taxonomic neighbors? This is the most serious threat. Sibling averaging captures much of what any tree prior offers, and when siblings share a loss it implicitly respects that loss. Where we expect it to fail is when an unseen class's closest relatives differ from it because of a loss on its own short branch. Averaging over the clade then reintroduces the ancestral trait. It should also fail when a convergent trait appears in an unseen class whose relatives lack it. The full model can raise the posterior probability of such a trait from partial evidence, because it has learned a non-zero re-gain rate for that attribute. If sibling averaging matches the full model on the loss-heavy class subsets defined below, we will conclude that the explicit evolutionary model adds nothing beyond neighborhood smoothing, and report that.

Why not just have a language model fill in the missing attributes, or rely on TAP's vision-conditional pooling to ignore wrong ones? TAP itself documents that language-model descriptions include features absent from real instances. Its pooling suppresses such features per image but never corrects the class description, and it builds a separate tree for each class with no structure shared across classes. We include language-model filling as a corruption source and as a baseline. The evolutionary model can then be tested as a corrector of language-model output, not only as a replacement for it.

Why not just treat every disagreement with the tree as annotation noise and filter it out? In genomics, this "keep the assumption, clean the data" strategy is exactly what the DelSIEVE comparison above shows losing badly once real losses are common. The ablation with the loss rate fixed at zero is this alternative in its cleanest form, so the experiment settles it directly.

Proposed Experiments

The main benchmark is generalized zero-shot learning on AWA2, CUB and SUN with standard splits, plus AwA and aPaY in HAT's setting for comparability. We corrupt unseen-class attribute vectors in three ways: masking a fraction of entries uniformly at random, from 10% to 70%; flipping a fraction of entries; and replacing vectors with attributes elicited from a language model and then binarized. Each repaired matrix is evaluated in two ways: imputation accuracy against the held-out ground truth, measured by AUC per attribute, and downstream harmonic mean with the fixed CSAP-ZSL classifier.

The decisive check is stratified. Before seeing any zero-shot result, we fit the full model on seen classes only and use the inferred rates and ancestral states to mark which (class, attribute) pairs lie below an inferred loss branch or involve an inferred independent re-gain. On AwA these are expected to include aquatic mammals and limbs. On CUB they should include crest and coloration traits. The prediction is that the full model's margin over both the zero-loss ablation and sibling averaging is concentrated in these marked pairs, and is near zero on the unmarked ones. A uniform margin, or none, falsifies the mechanism even if overall accuracy rises.

A second check targets the observation layer. On CUB we compare class vectors cleaned from per-image attribute evidence with and without the dropout term $\delta$. We expect the dropout term to matter most for attributes commonly hidden by pose, and to have little effect on attributes that are always visible. The clade-versus-isolated-absence logic from DelSIEVE predicts this difference. Finally, the WordNet versus species-taxonomy comparison tests robustness. If conclusions flip with the tree source, the method depends on tree quality more than on the finite-sites idea, and we will say so.