Vision models pretrained on large image collections are increasingly used to read plant functional traits, such as leaf area or specific leaf area, from photographs. The usual recipe freezes the model, extracts features from its final layer, and fits a simple regressor on top. Here is the catch. Leaf traits are strongly phylogenetically conserved, meaning close relatives tend to share trait values because they inherited them. Ancestry explains up to 86.6% of the variance in leaf vein density across 93 woody species, while climate explains under 10% (Wang et al.). A model that only recognizes which lineage a plant belongs to will therefore look as if it predicts traits. It will then fail on lineages it has not seen, and on the variation within a species that ecologists most want to measure.
This note proposes a diagnostic that separates the two possibilities. We fit linear probes, single linear regressors trained on frozen features, at every layer and sub-module of several networks. We train them on leaf traits measured per specimen, then evaluate them in two ways: on plant clades (families and orders) withheld from training, and on deviations from each genus's mean. The aim is to find where in the network lineage identity lives and where transferable leaf morphology lives. We also ask whether that division looks the same in a flat Vision Transformer and in a hierarchical convolutional network.
Core Claim
We hypothesize that, in a frozen vision encoder, lineage identity and leaf morphology peak at different depths. The final layers encode lineage most strongly, especially after training on species labels. Intermediate layers carry morphological information that transfers across clades.
This predicts a specific pattern. On ordinary random splits, where every species appears in both training and test data, final-layer probes win and perform about as well as a taxonomic lookup table. On clade-held-out splits, the familiar "hump" appears: intermediate layers beat the final layer. Within-genus residual skill peaks at the same intermediate depths.
The claim fails if lineage and morphology peak at the same depth, or if a probe never beats nearest-relative imputation on withheld clades. In that case, published image-to-trait skill would be largely recovered taxonomy.
The closest work stops short of this question in three ways. Layer by layer, module by module found that intermediate ViT layers, and especially the expanded feedforward activation (Act), beat the final block under distribution shift. That study used only one architecture, categorical labels, and synthetic or stylistic shifts, and it had no notion of lineage as a confounder. PlantTraitNet predicts traits from final-layer DINOv2 features using species-median labels from the TRY trait database. Because those labels are constant within a species, any skill is lineage-confounded by construction, and the paper never controls for phylogeny. The Herbarium 2021 challenge shows that more than half of top-model errors fall within the correct genus, which motivates the idea that taxonomy is geometrically encoded. It never probes frozen layers or traits.
Research Program
Specimen-level labels. The first step is a testbed where within-species variation is actually labeled. On a stratified subset of the Herbarium 2021 sheets, which span 64,500 taxa with standardized family and order labels, we run the segmentation pipeline of Automated Extraction of Phenotypic Leaf Traits. That pipeline uses DeepLabv3+ followed by a filter that keeps only intact leaves, and it reports 93.71% test mean intersection-over-union on its UBDH herbarium dataset. We use it purely as a measurement tool. From it we obtain per-specimen leaf area, length, width, and shape ratios.
Two assumptions need checking. First, we convert pixels to physical units using the standard sheet dimensions, and we validate this calibration on sheets that include a scale bar. Second, the segmentation must be accurate on our subset; we check this against hand-labeled masks on a held-out sample.
We deliberately leave out vein density. Its microscopic structure is not resolvable in standard sheet scans, and we state this as a scope limit.
Measuring lineage dependence. Because these traits are measured from the same image the probe sees, they serve as a controlled ground truth for what the features encode. They are not a competitor to segmentation, which would of course measure leaf area better.
We measure lineage dependence of each layer's probe in three ways. The first is R² on test sets from withheld orders and families. The second is R² on genus-centered residuals, meaning the trait minus its genus mean, predicted from features. The third is Pagel's λ, the standard statistic for how strongly a quantity follows the phylogeny, computed on the probe's errors over a taxonomy-derived tree. A probe whose errors carry high phylogenetic signal is borrowing from lineage.
The ablation. We fit a linear map from features to family identity and remove that subspace from the features before fitting the trait probe. If cross-clade trait skill survives this removal at intermediate layers but collapses at final layers, those layers hold morphology beyond lineage.
Models. We use four backbones under one protocol, with probes at all eight sub-module taps of Layer by layer, module by module for transformers and at every stage output for CNNs:
- an ImageNet-21k ViT-B/16, matching that study;
- DINOv2, matching PlantTraitNet;
- a ResNet-50, the Herbarium 2021 baseline;
- a ViT fine-tuned on Herbarium species classification.
The fine-tuned ViT is the case where training should actively push lineage into the top layers. Comparing the flat transformers with the staged CNN addresses whether layer-wise conclusions transfer back to hierarchical backbones, a direction that is rarely tested.
Transfer to an applied pipeline. Finally, we carry the protocol to the PlantTraitNet citizen-science setting (iNaturalist and Pl@ntNet images with TRY labels, evaluated against the sPlotOpen vegetation-plot survey). There we ask how much of its reported skill survives clade-held-out evaluation.
Baselines
Taxonomic lookup. The naive reference point assigns each test specimen the median trait of its nearest training relative: its genus if present, otherwise its family, otherwise its order. This is the do-nothing baseline that ecologists would actually use, and it is the strongest competitor on our question because phylogenetic conservatism makes it strong by design. It uses the same training specimens as the probes and has no tuning. A probe "beats" it only if it shows higher R² on withheld clades, or nonzero R² on within-genus residuals, where lookup scores zero by construction.
Final-layer probe. This is the default in applied pipelines such as PlantTraitNet, which reports R² of 0.34 for leaf area and 0.27 for specific leaf area against sPlotOpen at one-degree resolution. We reproduce this setting with identical features, splits, and ridge-regularization search for every layer.
Layer Mean. The strongest existing layer-aggregation method is Layer Mean from Choose Your Transformer, which averages hidden states across layers. It raised transferability-ranking correlation from 0.58 to 0.86 for LogME (a fast estimator of how well frozen features will transfer) on language models. We run it with the same probe and the same tuning budget as single-tap probes.
Single best tap. We also probe the recommended Act and LN2 taps from Layer by layer, module by module. That paper prints no per-benchmark numbers in the material available here, so there is no reported result found on Lacuna for its numeric accuracies. A chosen single tap counts as better than Layer Mean if it gives higher withheld-clade R² with the layer selected on a separate validation clade.
Lineage-subspace removal. This is the ablation that isolates the key ingredient. Comparing probes with and without the family subspace, at matched feature dimension, shows whether any gain comes from morphology or from lineage.
Taxonomy classification ceiling. For context on how strongly taxonomy is encoded, the reported Herbarium 2021 numbers are macro F1 of 0.442 for a baseline ResNet-50, 0.494 with a loss that uses family and order labels, and 0.757 for the winning entry (de Lutio et al.).
Why not just…?
Why not just use a species or genus lookup table? On random splits, this baseline may genuinely match the probes, and we expect it to. That outcome is part of the argument, not a threat to it. A lookup table cannot represent variation within a lineage. Wang et al. document opposite altitudinal trait shifts in trees and shrubs, and PlantTraitNet had to filter species-median labels because seedlings and mature plants differ. A lookup table also cannot extrapolate to unsampled clades. Our residual and withheld-clade tests are exactly where lookup scores zero or degrades. If probes also score near zero there, we report that images add nothing beyond taxonomy for these traits.
Why not just fine-tune the whole network on traits? Fine-tuning may give the best raw accuracy, and we include it as a reference row. But it cannot answer where information already lives, because it rewrites the features. It also risks overfitting in the long tail: Herbarium accuracy is only 56% for taxa with two training images. More fundamentally, a fine-tuned model trained on lineage-structured labels can learn lineage shortcuts just as easily. We therefore evaluate the fine-tuned model under the same withheld-clade and residual tests. If it beats the best frozen tap there, we report that as a real limit on the frozen-probe approach.
Why not just average all layers? Layer Mean is cheap and was strong in language models, so it is a real threat. If it matches the best single tap on withheld clades, the practical recommendation becomes "average layers" and the localization result is scientific only. The lineage-removal ablation still discriminates the two hypotheses: averaging mixes lineage-heavy top layers with morphology-heavy middle layers. It should therefore lose more skill than a single middle tap when the family subspace is removed.
Proposed Experiments and Decisive Checks
The central experiment is a split comparison over about 2,000 genera from Herbarium 2021, run at every layer and module of all four backbones. We compare three regimes: random specimen splits, genus-held-out splits, and order-held-out splits. We report R², within-genus residual R², and Pagel's λ of the errors as functions of depth.
The hypothesis is supported if three things hold together:
- the best depth moves earlier as the held-out taxonomic level gets coarser;
- residual R² peaks at intermediate depth;
- the taxonomy-fine-tuned ViT shows the largest final-layer collapse on withheld clades.
The hypothesis is refuted in any of these cases:
- lineage-decoding accuracy and trait-residual skill peak at the same tap;
- the hump appears in ViTs but the ResNet shows monotone decline;
- no tap beats nearest-relative imputation on withheld orders.
In the second case, the claim narrows to an architecture-specific one.
The transfer check reruns PlantTraitNet-style DINOv2 probing on its citizen-science data under family-held-out splits. We test whether its leaf-area skill (reported R² of 0.34) survives, and whether an intermediate tap recovers it. As a sanity anchor, the same protocol on its leaf-nitrogen target (reported R² of −0.12) should show no tap beating lookup. Any apparent nitrogen skill would indicate leakage in our pipeline.
References
- Plant Phylogeny and Growth Form as Drivers of the Altitudinal Variation in Woody Leaf Vein Traits
- Layer by layer, module by module: Choose both for optimal OOD probing of ViT
- PlantTraitNet: An Uncertainty-Aware Multimodal Framework for Global-Scale Plant Trait Inference from Citizen Science Data
- The Herbarium 2021 Half–Earth Challenge Dataset and Machine Learning Competition
- The Herbarium 2021 Half-Earth Challenge Dataset
- Automated Extraction of Phenotypic Leaf Traits of Individual Intact Herbarium Leaves from Herbarium Specimen Images Using Deep Learning Based Semantic Segmentation
- Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks