Many prediction problems are solved in levels. A model first finds a coarse answer, such as "this span is an entity" or "this region is a horse," and later stages refine it into "this is a protein inside a cell-line name" or "this is the horse's leg." When an early level is wrong, later levels usually inherit the error and cannot repair it. This failure is well documented. A missed inner entity blocks recognition of the outer entity that contains it (A Neural Layered Model for Nested Named Entity Recognition), and a misrouted medical scan sends the fine-grained search into the wrong subspace (Content-Based Retrieval from Medical Image Databases). The classical fix is stacking: train each finer level on predictions its parent made on held-out data, so the finer level sees realistic parent mistakes and learns to discount them (Stacked Hierarchical Labeling). Stacking breaks down when labeled data is scarce. Splitting fifty or a hundred examples into folds starves every model in the cascade.
We propose to move that error-correction training out of the small target task and into a multi-task warm-start phase. In multi-task initialization, one model is first trained on many related source tasks and then adapted to the target. Source data is abundant, so the expensive cross-fitting can be done there. Across many hierarchical source tasks, a shared backbone and a task-agnostic "parent-trust gate" learn how to recognize and override realistic parent errors. The target task then only fine-tunes the label heads, and the learned skepticism toward parents carries over. Put in two sentences: stacking teaches child classifiers to correct parent errors but needs data the small-sample regime does not have. We pay that cost once, on many source tasks during multi-task pre-training, and test whether error recovery transfers.
Core Claim
The ability to recover from upstream mistakes in a hierarchical predictor is a transferable skill. It can be learned during multi-task initialization by exposing finer levels to cross-fitted parent predictions on source tasks. In small-sample target regimes, this should close a measurable part of the gap between running the cascade on gold parent labels and running it on predicted ones. Neither plain multi-task pre-training nor target-side noise injection should close that gap to the same degree.
The quantity we care about is the cascade gap. For a target task at sample size $n$, let $F^{\text{gold}}_n$ be the fine-level F1 when the child receives gold parent labels at test time, and $F^{\text{pred}}_n$ the F1 when it receives the model's own parent predictions:
$$G_n = F^{\text{gold}}_n - F^{\text{pred}}_n .$$
We also measure the recovery rate $R_n$: among fine-level gold items whose parent prediction was wrong, the fraction the child still gets right. The claim predicts three things. Our initialization lowers $G_n$ and raises $R_n$ relative to every baseline. The advantage is largest at small $n$, and it shrinks as $n$ grows, because target-side stacking becomes affordable with more data.
Relation to Prior Work
Four lines of work bear on this proposal, each in a different role.
The closest existing work is Hierarchical Prompt Learning for Multi-Task Learning (HiPro). HiPro clusters tasks into a tree using gradient affinity, attaches soft prompts to each node, and averages classifier weights along the root-to-leaf path. It sidesteps cascades by never conditioning a child on a parent prediction. Its hierarchy is a hierarchy over tasks, and its fusion is a fixed linear average. Our hierarchy is over the levels of a single prediction, and the child consumes the parent's actual output through a learned gate. HiPro motivates the proposal, since it shows hierarchy helps in few-shot multi-task settings, but it does not address error propagation inside a prediction.
Three works supply the method:
- Munoz et al. (Stacked Hierarchical Labeling) trains children on out-of-fold parent predictions. We reuse that mechanism, but apply it across source tasks rather than within one task.
- MAS (MAS: Towards Resource-Efficient Federated Multiple-Task Learning) shows that weights consolidated in a merged multi-task phase initialize split models well. We use the same merge-then-adapt structure.
- Order Matters (Order Matters in the Presence of Dataset Imbalance for Multilingual Learning) finds that pre-training on large tasks and then resetting the optimizer before joint training with small tasks protects the small ones. We adopt that reset when moving to the target.
Neither MAS nor Order Matters models hierarchical dependencies between output levels. Both treat tasks as flat sets.
Other work constrains the design. Joint bidirectional inference with conditional random fields lets coarse and fine predictions correct each other (Joint Object and Part Segmentation Using Deep Learned Potentials), but it relies on dense large-scale supervision. Similar smoothing washed out small categories such as sunglasses in A Part-Based Deep Neural Network Cascade Model for Human Parsing. Munoz et al. also argue that full joint optimization is slower and less robust than feed-forward stacking. We therefore keep the model feed-forward and put the correction into training rather than inference, which also avoids the latency cost. Zhou and Mok report about 4.98 s versus 0.29 s per image for their cascade against a plain fully convolutional network.
Finally, BigBio supplies the source pool. It shows that flat multi-task training over 107 biomedical tasks, followed by fine-tuning, lands within 3.2 F1 of specialized models. Flat BigBio pre-training is therefore the natural strong initialization to beat. It is also exactly the "no hierarchy-aware pre-training" condition.
Research Program
Model. The model is a shared text encoder with an ordered set of level heads. Each level head reads encoder features, plus the previous level's soft output passed through a parent-trust gate. The gate is a small network that outputs a per-token weight on the parent message. It is deliberately label-agnostic: it reads the parent's confidence profile (entropy and margin of the parent distribution) together with local encoder features, not the parent's label identities. That design is what lets it transfer across tasks whose label sets differ.
Source hierarchies. We build hierarchical source tasks from the BigBio pool in two ways:
- nested or layered entity recognition, where an inner layer feeds an outer one;
- coarse-to-fine typing, where span detection or a coarse semantic group feeds fine entity types.
Error-exposure pre-training. For each source task, we split its abundant data into $K$ shards. A parent head trained on the other shards produces the parent predictions that the child receives on the held-out shard. Children therefore always see parent outputs with realistic, structured errors, such as confusions between sibling types, missed short spans, and boundary slips. They never see gold parents. An auxiliary loss asks the gate to predict whether the parent was wrong, using cross-fitted correctness labels that come for free on source data. All source tasks are trained jointly with balanced sampling, as Yang et al. describe for avoiding premature overfitting.
Target adaptation. We reset the optimizer, following Order Matters. New label heads are attached for the target hierarchy. The encoder and heads are fine-tuned on the small target set, and the gate is either frozen or tuned at a low learning rate. No cross-fitting is done on the target.
Analysis. Beyond the main results, we want to know which properties of the source tasks drive transfer: the number of hierarchical source tasks, which by analogy to the shared-representation bound in Sharing Knowledge in Multi-Task Deep Reinforcement Learning we expect to help sublinearly; the depth of the source hierarchies; and how closely their error patterns resemble the target's. We characterize "error-pattern similarity" by comparing parent confusion matrices, collapsed to confidence bins, between source and target.
Baselines
Naive reference. The do-nothing reference is the layered nested-entity model of Ju et al., trained from the generic pre-trained encoder on the target subset only, with teacher-forced gold parents during training. This is the standard cascade, and it shows how large $G_n$ is with no intervention. Ju et al. report 74.7% F1 on GENIA and 72.2% on ACE2005 with full training data. We reproduce the full-data numbers as a sanity check. We found no reported result on Lacuna for this model at 50 to 500 training sentences.
Strongest existing initialization. The strongest competitor is flat multi-task pre-training on the same BigBio source pool, followed by fine-tuning the same layered architecture on the target. This is the BigBio recipe, which reports fine-tuned multi-task models within 3.2 F1 of specialized models such as LinkBERT. It is the strongest competitor because it gets every benefit of our source data and compute except the error exposure. It uses identical source tasks, identical sampling, an identical number of source update steps, and an identical target tuning budget: the same learning-rate grid, the same number of seeds, and the same early-stopping rule on a small target development split. To beat it, we must lower $G_n$ and raise outer-level F1 by more than the seed-to-seed standard deviation at $n \le 200$, across at least two target datasets.
Hierarchical multi-task alternative. We adapt HiPro's root-to-leaf weight averaging, with a task tree built from source gradient affinity, to the same encoder. HiPro reports roughly 6.7% accuracy gain over CoOp on DomainNet in its lowest-data settings. This tests whether task-level hierarchy alone yields the benefit.
Target-side stacking. We run Munoz et al.'s stacking with $K$-fold cross-fitting on the target set itself, starting from the flat BigBio initialization. This is the method we claim to amortize, so if it matches us at small $n$, our claim fails.
Ablations that isolate the key ingredient. Two ablations reuse our full pipeline but change only what the child sees during source pre-training. In the first, children receive gold parent labels. This removes error exposure while keeping the gate and every other component. In the second, children receive gold parents corrupted at random, at the same overall error rate as the cross-fitted parents. This keeps exposure but removes the structure of the errors. A third ablation freezes versus fine-tunes the gate at the target, to test whether the transferred component is really the gate or just better encoder features.
Why not just…?
Why not just pre-train flat on BigBio and fine-tune the hierarchy end to end? This is the most serious threat. A well-initialized encoder may already make parent errors rare enough that the cascade gap is negligible. Prior work does not settle this. BigBio reports aggregate F1 and nothing about cascades, and Ju et al.'s analysis suggests inner-entity misses still block outer ones even with learned features. The flat-initialization baseline above answers it directly. If $G_n$ under flat initialization is already small at $n = 50$, the problem we target does not exist in this setting, and we would report that as a negative result rather than claim a gain.
Why not just inject random noise into parent labels during target training? Scheduled-sampling-style corruption is cheap and needs no source cross-fitting. Our hypothesis is that real parent errors are structured. They cluster on short spans, sibling types, and low-confidence regions, and the gate's value comes from learning that structure, which random noise does not reproduce. This is a real threat. The random-corruption ablation, run both at source and at target, settles it. If random corruption matches cross-fitted exposure, the contribution reduces to noise injection, and we would reframe the work around that simpler finding.
Why not just stack on the target data as Munoz et al. do? Stacking needs $K$ held-out folds at each level. With 50 target sentences, each parent sees only about 40 sentences, and its errors are then larger than at test time. That over-trains the child to distrust parents. The target-side stacking baseline, run across sample sizes, should show this crossover. We expect stacking to be weaker at small $n$ and to catch up at larger $n$, and a crossover point is itself a useful finding.
Why not just add post-hoc conformal prediction sets to the cascade? Conformal sets over hierarchies (Conformal Prediction in Hierarchical Classification with Constrained Representation Complexity) guarantee coverage, but they leave the underlying predictor unchanged. Weak features yield large sets: CRSVP needed sets of about 123 on PlantCLEF 2015, against about 44 for CRSVP-3. We treat conformal prediction as complementary and use it as a diagnostic. If our initialization truly improves the hierarchy, CRSVP-$r$ sets built on top of it should shrink at the same nominal coverage.
Proposed Experiments
Primary setting: biomedical nested entity recognition. Sources are BigBio entity-recognition datasets restructured into layered or coarse-to-fine hierarchies. GENIA is held out from the source pool entirely. The target is GENIA subsampled to 50, 100, 200, 500 and the full set of training sentences, with five random subsets per size. A cross-domain probe uses ACE2005, which is outside the biomedical source pool, to test whether the gate transfers when the domain shifts. JNLPBA, a flat dataset, serves as a control: there should be no gain there, because it has no hierarchy.
Metrics. We report outer-level and overall F1, cascade gap $G_n$, recovery rate $R_n$, and gate calibration, meaning how well the gate's distrust predicts actual parent errors on the target.
Secondary setting: hierarchical classification. A hierarchical classification probe uses DBpedia and PlantCLEF 2015, both of which have label trees. Coarse-to-fine heads are trained on few-shot target subsets. We report fine accuracy alongside CRSVP-$r$ set size at 90% nominal coverage, against the PlantCLEF 2015 figures reported by the conformal paper.
Decisive checks. Four results would decide the claim:
- Error-exposure pre-training must lower $G_n$ relative to the matched flat BigBio baseline at $n \le 200$. Otherwise the claim is false.
- Cross-fitted exposure must beat random-corruption exposure at matched error rate. Otherwise the mechanism is noise robustness, not learned error structure.
- Freezing the gate at the target must retain most of the gain. Otherwise what transfers is encoder quality rather than correction behavior.
- The advantage over target-side stacking must shrink as $n$ grows. That is the signature of amortizing a data cost, rather than of a generically better model.
If all four hold on GENIA and at least one of ACE2005 or the classification probes, error recovery is a transferable product of multi-task initialization. If check 2 fails, the simpler story wins and we will say so.
References
- A Neural Layered Model for Nested Named Entity Recognition
- Content-Based Retrieval from Medical Image Databases: A Synergy of Human Interaction, Machine Learning and Computer Vision
- Stacked Hierarchical Labeling
- Hierarchical Prompt Learning for Multi-Task Learning
- MAS: Towards Resource-Efficient Federated Multiple-Task Learning
- Order Matters in the Presence of Dataset Imbalance for Multilingual Learning
- Joint Object and Part Segmentation Using Deep Learned Potentials
- A Part-Based Deep Neural Network Cascade Model for Human Parsing
- BigBio: A Framework for Data-Centric Biomedical Natural Language Processing
- Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
- Sharing Knowledge in Multi-Task Deep Reinforcement Learning
- Conformal Prediction in Hierarchical Classification with Constrained Representation Complexity