Evidence-Weighted Climbing: Correcting Class-Frequency Bias Where Hierarchical Classifiers Retreat

A hierarchical classifier that is unsure of the exact species can still say "this is a beetle," and that coarse answer is often what a user needs. The best current method for this, Hierarchical Selective Classification (HSC), starts at the single most probable leaf and walks up the tree until it reaches an ancestor the model is confident about. On long-tailed data, that starting leaf is chosen by probabilities that are skewed toward common classes. A rare insect can be sent into the subtree of a common relative, and from there climbing can only move up the wrong branch. One frequency error at the bottom of the tree becomes a confident wrong answer higher up.

We propose to fix the score that decides where climbing starts and where it stops. Each node's probability gets a frequency correction whose strength depends on how much training evidence that node has. The idea comes from decades of work on word-association statistics. Raw counts favor popular items. The usual fix, pointwise mutual information, divides by the item's base rate and swings too far, inflating rare items that rest on almost no evidence. Subtracting a class's log prior from a classifier's log posterior is exactly pointwise mutual information. So the common long-tail fix inherits the same rare-event inflation, and that inflation is worst at the sparse leaves where climbing begins. In a tree, internal nodes pool the counts of their descendants, so the right amount of correction differs from node to node. We build that per-node correction, keep the conformal risk guarantee of HSC, and test whether it reduces wrong-branch errors on the long tail.

Core Claim

Classifier errors in hierarchical retreat run in two opposite directions, mirroring the two failures documented for word association. With no correction (the "raw counts" regime), frequent leaves capture ambiguous tail inputs, as Hindle (1990), Yang et al. (2005) and Bansal et al. (2012) showed for co-occurrence counts dominated by popular words. With full prior correction, rare leaves backed by a handful of training examples are inflated, as Damani et al. (2013) showed for standard PMI. Damani et al. also showed that the fix is to weight the correction by the right amount of evidence, and that getting the evidence count wrong is worse than no fix: their min-frequency variant scored 0.56 on TOEFL versus 0.59 for plain PMI and 0.61 for their max-frequency version.

Our claim is that hierarchical retreat should score every node $n$ given input $x$ as

$$s(n \mid x) = \log p(n \mid x) - \lambda(N_n)\,\log \pi(n),$$

where $p(n \mid x)$ is the model's probability summed over the leaves below $n$, $\pi(n)$ is the training frequency of $n$, $N_n$ is the number of training examples under $n$, and $\lambda(\cdot)$ is a correction strength between 0 and 1. The correction strength grows with $N_n$, so well-supported nodes receive near-full prior correction and sparse leaves receive only partial correction.

Climbing then starts from the leaf with the highest $s$, not the highest raw probability. The stopping threshold is calibrated by the same split-conformal procedure HSC uses. Because that guarantee holds for any fixed scoring function, fitting $\lambda$ on a separate data split leaves it intact.

What is new is not prior correction itself, which is standard for long-tailed classification. The new parts are three. We treat the correction's strength as a per-node evidence question across tree levels. We apply it inside a calibrated retreat rule. And we measure its effect on cascading wrong-branch errors, not on flat top-1 accuracy.

Research Program

The first step is diagnostic. On a long-tailed hierarchy, we will count "wrong-branch errors": inputs whose entire HSC climbing path never contains the true label. We will stratify these by the training frequency of the true leaf. If wrong-branch errors are not concentrated in the tail, or are rare overall, the motivating mechanism is weak and the project narrows to a calibration study.

The second step builds the scoring rule. The parametric family for $\lambda$ is $\lambda(N) = N/(N+\kappa)$, a standard shrinkage form that says "trust the base-rate correction in proportion to the evidence behind it." The single parameter $\kappa$ is fit on a held-out split, separate from the conformal calibration split. We will also test a Damani-style variant that weights the correction by the square root of the larger of the node's count and its parent's count. This carries over their argument that the more frequent party sets the number of chances to co-occur.

The third step tests a path-unconfined rule. Here climbing is replaced by choosing the deepest node, anywhere in the tree, whose corrected aggregate score passes the calibrated threshold. This separates two questions: whether gains come from better scoring or from escaping path confinement.

The fourth step is analysis. We will relate the gains to how unevenly frequency is spread across levels. We will also check whether the correction matters more for models trained without hierarchy-aware objectives than for hierarchy-regularized models such as CLiBD-HiR from Ahmed et al. (2026). That paper anchors fine-grained embeddings to their coarse taxonomic neighborhoods for robustness to noise. However, it evaluated only a closed-set split and applied no tail-frequency correction. This project tests whether that anchoring absorbs the frequency bias or leaves it in place.

Baselines

The naive reference is flat selective classification: predict the argmax leaf or abstain, using a threshold on maximum softmax probability. HSC reports that its Climbing rule improves on this by a 14.94% hierarchical gain on ImageNet-1k across 1,115 models (Hierarchical Selective Classification). That paper reports no number for iNaturalist 2021, and we found no reported HSC result on Lacuna for BIOSCAN-1M.

The strongest existing method is HSC itself, with $\lambda = 0$. It is the most direct competitor because it already provides calibrated, risk-bounded retreat. Any gain we claim must come from the scoring change alone.

Every variant shares the same frozen backbone, the same training checkpoints, the same conformal calibration split and the same risk target. The only extra fitted quantity in our method is $\kappa$. It is fit on a separate split of the same size that every baseline is also allowed to use for its own tuning. Beating HSC means a lower area under the hierarchical risk–coverage curve on long-tailed data, a lower wrong-branch error rate in the bottom frequency third, and a valid risk guarantee. We also require no loss on balanced ImageNet-1k, where the correction should do nearly nothing.

The second strong baseline is full prior correction, $\lambda = 1$. This is the PMI regime, and it is what most practitioners would try first.

The ablation that isolates our key ingredient is a single constant $\lambda$ tuned on the same split. If a tuned constant matches evidence-weighted $\lambda(N_n)$, the per-node significance idea adds nothing. We will report that result rather than argue around it.

For the conformal-set setting, the reference is CRSVP-r from Mortier et al. (2026). It builds tree-structured prediction sets with guaranteed coverage, reporting an average set size of about 123 for $r=1$ and about 44 for $r=3$ on PlantCLEF 2015 at roughly 90% coverage. We will feed it our corrected scores as nonconformity inputs. Success means smaller sets at fixed $r$.

Why not just…?

Why not just retrain with a balanced or logit-adjusted loss and then run HSC? This is the most serious alternative. A model trained with a class-balanced objective has a prior correction built in, which is roughly the $\lambda=1$ regime applied at training time. We expect it to reduce popularity errors but inflate sparse leaves, the PMI trap of Damani et al. (2013) and Bron et al. (2009). It also cannot vary the correction across tree levels, because the loss acts only on leaves. We will include a balanced-loss retrained model under HSC with matched compute. If it matches evidence-weighted climbing on tail wrong-branch rate, then the contribution reduces to a post-hoc shortcut, and we will say so.

Why not just merge rare leaves into their parents, a frequency cutoff? This removes rare-event inflation by removing the rare classes, which on BIOSCAN-1M means giving up most species. That is the destruction-of-the-tail problem that work on Zipfian data warns against. Climbing already lets the model retreat to the parent when it is unsure. A cutoff forces that retreat for every input, including tail inputs the model could have named. We will measure this directly as the average depth of correct predictions for tail classes under each method.

Why not just temperature-scale the softmax? Temperature scaling changes how sharp the probabilities are, but it preserves the ranking of leaves. So it cannot change which leaf climbing starts from, and it cannot fix wrong-branch errors. It is still a useful control for the "where to stop" half of the problem, and we will report HSC plus temperature scaling alongside our method.

Proposed Experiments

The main testbed is BIOSCAN-1M, with 903,536 training and 224,777 test samples over Order, Family, Genus and Species. Its long tail is noted by Ahmed et al. (2026). We will use backbones with and without CLiBD-HiR regularization, under both clean and noisy inputs. Noise makes the input ambiguous, which is when prior bias should matter most.

Secondary testbeds are iNaturalist 2021 with its 7-level taxonomy, and PlantCLEF 2015 for the conformal-set comparison against Mortier et al. (2026). ImageNet-1k with WordNet serves as a near-balanced control. We will also build a Zipf-subsampled version of ImageNet-1k, in which we vary the tail exponent so that frequency skew is the only thing changing.

Metrics are hierarchical risk–coverage curves and their area, hierarchical gain relative to flat selective classification, and empirical risk against the target. We will also report the wrong-branch error rate and the average returned depth, both stratified by training-frequency terciles.

The decisive checks are three. First, on the Zipf-subsampled ImageNet, the gap between evidence-weighted climbing and HSC should grow with tail skew and vanish on balanced data. If it does not, the frequency mechanism is not what drives the result. Second, evidence-weighted $\lambda$ must beat both $\lambda=1$ and the tuned constant $\lambda$ in the tail tercile. If it does not, the significance-weighting idea fails. Third, empirical hierarchical risk must stay at or below the target on every dataset. If it does not, the separation between the fitting split and the calibration split has failed.