Sibling Competition in Tree-Structured Label Prediction

Many classification problems come with a label tree. A news story tagged "Markets" might also be tagged "Bond Markets," "Equity Markets," or both. The tree adds a hard rule: a story tagged with a child label must also carry its parent. Today's strongest encoder models for these problems usually ignore a second, softer rule that the tree implies. When the evidence says "this is about Markets" and one child already accounts for that evidence, the other children should become less likely. This is explaining away: two independent causes of a shared effect start to compete once the effect is observed. Classifiers that attach independent yes/no heads to each label cannot express this competition. They either hedge by switching on several siblings, or they need hand-tuned thresholds to stop doing so.

We propose a small output layer that sits on a standard text encoder and adds this competition explicitly. For each parent, it adds one factor over its children. The factor says that the parent is "on" only if some child explains it, and it scores how many children are typically on for this particular input. The tree rule is built in as a constraint. Because the label structure is a tree, the whole joint distribution over label sets can still be normalized and decoded exactly in time linear in the tree size, using a counting dynamic program inside each family of siblings. The claim is testable in one direct way. On benchmarks where documents often carry several children of the same parent (RCV1-v2, NYT), the layer should reduce false-positive siblings and raise macro-F1 over independent heads and over a tree model with only parent–child links. On a benchmark where documents follow a single path (Web-of-Science), a plain softmax already enforces exclusivity, so the gain should vanish.

Core Claim

The claim is that the missing ingredient in current hierarchical classifiers is neither more graph machinery nor autoregressive decoding. It is a higher-order, input-dependent factor coupling siblings under a shared parent, and a tree keeps this factor exactly tractable.

Existing methods cover pieces of this. Independent level heads (HYDRA) have no coupling at all. Parent–child path constraints, as in Making Better Mistakes's hierarchical cross-entropy, apply only to single-label trees. Autoregressive path generation (PAAM-HiA-T5) captures dependencies only implicitly and pays roughly a 3× decoding cost. In a pairwise tree model with parent–child potentials only, siblings are conditionally independent given their parent, so such a model cannot produce explaining away by construction. That is the ablation the proposal turns on.

The Model

Let $y_v \in \{0,1\}$ indicate whether label $v$ is assigned to input $x$. Let $s_v(x)$ be a per-label score from a shared encoder. Let $k_v = \sum_{c \in C(v)} y_c$ count the active children of node $v$. The model is

$$p(y \mid x) \propto \mathbb{1}[y_c \le y_v \ \forall c \in C(v)] \exp\Big(\sum_v s_v(x)\, y_v + \sum_{v \text{ internal}} y_v\, \psi_v(k_v; x)\Big),$$

where the indicator enforces the tree rule and $\psi_v(\cdot\,; x)$ is a small learned table over child counts. The table is produced by a head on the encoder and truncated at a maximum count $K$.

Setting $\psi_v(0; x)$ very negative says that a parent needs a child to explain it. A finite value for $\psi_v(0; x)$ acts as a "leak" for documents tagged only at an internal node, which the RCV1 labeling policy permits (RCV1). A table that falls sharply after $k=1$ says that one child usually suffices. This is the negative product synergy that Henrion identified as the condition under which confirming one cause lowers another. A flat table recovers the pairwise tree model.

Normalization is a bottom-up pass. Each child sends a two-state message. The parent combines its children's messages by convolving them over the running count, which costs $O(|C(v)|K)$ per node. The log-partition function is therefore exact and differentiable, and training maximizes the exact log-likelihood of observed label sets. Max-product decoding returns a label set that is consistent with the tree by construction.

This is the discrete, label-space counterpart of the argument in Structured Recognition for Generative Models with Explaining Away. That paper showed that factorized recognition models destroy explaining away between continuous latents and restored the coupling with joint Gaussian potentials. We restore it among discrete labels, where the tree lets the coupling be exact rather than variational. The contrast with Deep Exponential Families is similar. There, latents within a layer compete to explain the data, but inference is mean-field and the hierarchy is emergent rather than a given taxonomy. The contrast with GraphLDA is that it restricts each document to a single path, so alternative lineages never compete. Our model allows multiple active paths and makes them compete.

Research Program

The work has three parts, and the claim stands or falls on the second.

The first part is a controlled synthetic study. Label sets are sampled from a known tree with planted sibling-count distributions, and document features come from a noisy-OR emission. We check that the learned $\psi_v$ recovers the planted tables, and that independent heads show the predicted hedging when evidence for the parent is strong but ambiguous across children.

The second part attaches the layer to a RoBERTa encoder and compares it on RCV1-v2, NYT, and WOS against the baselines below. Encoder, data splits, and tuning budget are identical across all methods.

The third part is a diagnostic study of explaining away itself. For each parent, we condition on one child being on and measure how much the posterior probability of each sibling drops. This model-implied effect is compared with the empirical conditional from held-out label co-occurrence. A model that truly captures explaining away should match the empirical effect's sign and magnitude parent by parent. Independent heads are fixed at zero by construction.

A secondary study targets the distant-supervision setting that motivated Path-Based Attention. We inject synthetic sibling-and-ancestor label noise into RCV1 training labels and test whether the count factor absorbs spurious siblings. TRAM needs annotator metadata to explain away noise; here the tree structure alone would do the work. We treat this as exploratory, since the file reports no prior result in this exact setting.

We deliberately exclude multi-parent hierarchies. Exact tractability depends on each node having one parent. Shahbaba and Neal already noted that path-summing priors break on DAGs, and extending our layer to DAGs would reintroduce loops. We state this as a scope limit rather than a promise.

Baselines

The naive reference point is a flat multi-label classifier: RoBERTa with an independent sigmoid per label, trained with binary cross-entropy, with a single global threshold tuned on validation. It tells us how much any structure buys. HYDRA reports its own gain relative to exactly this kind of flat RoBERTa model: +2.52 macro-F1 on RCV1-v2. The absolute value is not given in the file.

The strongest encoder-only competitor is HYDRA itself: level-specific heads plus a global head on a shared RoBERTa. The file reports micro-F1 81.96 and macro-F1 72.74 on NYT. It is the strongest because it matched or beat graph encoders at minimal cost, and it directly challenges the idea that coupling is needed. We rerun it with the same backbone checkpoint, sequence length, epochs, and number of hyperparameter trials as our model. Beating it means higher macro-F1 on RCV1-v2 and NYT with overlapping seeds excluded, plus a clear drop in false-positive siblings, with micro-F1 no worse.

The strongest structured competitor is PAAM-HiA-T5, which reports macro-F1 70.02 and a label inconsistency rate of 0.31% on RCV1-v2. We compare against the reported numbers. Where compute allows, we also rerun it under our splits and report wall-clock decoding time. Beating it means matching or exceeding its macro-F1 with 0% inconsistency by construction and encoder-only decoding speed.

The key ablation is the same layer with $\psi_v$ fixed flat. This is a pairwise tree model with the hierarchy constraint but no sibling competition, and every other component is identical. If the full model does not beat this ablation, the explaining-away ingredient has not earned its place, whatever it does against the other baselines. A second ablation uses an input-independent $\psi_v$. It separates "siblings compete" from "how strongly they compete depends on the document." On WOS, no result is reported on Lacuna for either HYDRA or PAAM, so we produce all WOS numbers ourselves.

Why not just…?

Why not just use HYDRA and repair inconsistent outputs afterward? Post-hoc repair fixes tree violations but not hedging. If independent heads assign 0.6 to three siblings, repair keeps all three. The flat-$\psi$ ablation roughly represents "consistency without competition." The sibling false-positive rate on parents with a single active child will show whether repair is enough. This is a genuine threat on micro-F1, where gains may be small. The prediction is concentrated in macro-F1 and sibling precision.

Why not just tune per-label thresholds? Thresholds are input-independent. Explaining away is a property of each particular document: whether a sibling should be suppressed depends on how strongly another sibling is supported in that document. We test this by giving the independent-heads baseline per-label thresholds tuned on validation. We then check whether our input-dependent $\psi$ beats the input-independent $\psi$ ablation. If it does not, tuned thresholds are a sufficient explanation and we will say so.

Why not just generate label paths autoregressively? A decoder can in principle learn sibling competition, and PAAM-HiA-T5's low inconsistency shows it learns the tree rule. The costs are slower decoding and no explicit, inspectable posterior. The explaining-away diagnostic is the discriminating experiment. We can compute conditional sibling drops from the decoder by forcing a prefix and compare their fidelity to the empirical conditionals against ours. If the decoder matches both accuracy and fidelity, our advantage reduces to speed and exactness.

Why not just add learned pairwise potentials between siblings? Pairwise sibling links create loops within each family, which forces approximate inference. They can express "a and b rarely co-occur" but not "one is enough, whichever it is." The count factor expresses the latter directly and stays exact. We include a mean-field pairwise-sibling variant for completeness.

Decisive Checks

The central prediction is a pattern across datasets, not a single leaderboard number. Relative to the flat-$\psi$ ablation, the full model should improve macro-F1 and sibling precision on RCV1-v2 and NYT, where many parents have several children active at once. On WOS, where each document follows one path, the difference should be close to zero. If gains appear equally on WOS, they come from something other than sibling competition. If gains appear nowhere, HYDRA's implication stands: a feed-forward encoder already captures what is needed, which resolves the tension raised by Yuille and Kersten in favor of feed-forward models.

We will also stratify results by parent, plotting improvement against the empirical strength of sibling anticorrelation. We expect a positive slope. We report inconsistency rate, which should be 0% by construction, decoding throughput against PAAM-HiA-T5, and five-seed confidence intervals for every comparison.