Large language models can sort a document into a deep category tree, such as Science → Biology → Enzymes → Hydrolases, without any training. You show the model a few labeled examples in the prompt, an approach called in-context learning, and ask it to write out the full path for a new document. This one-pass approach is attractive because the alternatives are worse in practice. Fine-tuned path generators need large labeled sets, and the best existing prompt-based method asks the model once per tree level, which is slow. In real deployments, though, the example pool is noisy. Product catalogs, patent offices and multilingual annotation teams produce mislabeled examples. A mislabeled path is not one wrong answer. It is a wrong parent-to-child step at a specific depth, and the model copies that step. Prior work shows that noisy examples hurt generation far more than classification, and that adding more examples makes it worse. Nobody has looked at how this plays out when the output is a tree path, or when the examples come from many languages.
We propose to treat demonstration noise in hierarchical prompts as a problem of the individual edge in the tree, not of the whole example. We will build a level-wise noise detector. For each step of each example path, it compares how surprising the model finds that step with how surprising the model finds the same step in semantically similar examples. Those comparison examples are retrieved across languages, so a scarce Bulgarian or Chinese pool can borrow evidence from richer languages. Flagged steps are repaired in place, not discarded. The claim to test is this: under realistic noise, repairing at the level of individual tree steps keeps most of the accuracy that whole-example filtering loses, especially on rare leaf categories and low-resource languages. Constraining the output to valid paths does not remove the damage, because most noise yields paths that are valid but wrong.
Core Claim
Hierarchical noise has structure. Write a demonstration as a document $x_j$ with path $y_j = (l_1, \dots, l_n)$. Corrupting the label at depth $i$ comes in two kinds. Ancestor corruption picks a wrong label at a shallow level, which drags the whole subtree with it. Sibling transposition keeps the parent right but picks the wrong child. Constrained decoding, which masks any token that would leave the tree, guarantees a valid path (A Study on Hierarchical Text Classification as a Seq2seq Task; Constrained Sequence-to-Tree Generation for Hierarchical Text Classification). It does nothing about a demonstration that teaches a valid but wrong edge. We predict three things. Ancestor corruption hurts more than sibling corruption at equal noise rate. Constrained decoding recovers almost none of the gap. A detector that scores each depth separately finds the corrupted edge more reliably than one that scores the whole path string. Whole-string scoring mixes the hard-but-correct deep leaf with the actual error.
Concretely, let $s_{j,i} = -\log P(l_i \mid l_{<i}, x_j)$ be the frozen model's surprise at step $i$ of example $j$. Let $N_i(j)$ be the $k$ nearest neighbors of $x_j$ under a multilingual sentence encoder whose paths share the prefix $l_{<i}$. The level-wise noise score is
$$z_{j,i} = \frac{s_{j,i} - \operatorname{mean}_{m \in N_i(j)} s_{m,i}}{\operatorname{std}_{m \in N_i(j)} s_{m,i}}$$
This comparison separates "this leaf is inherently hard" from "this edge disagrees with its neighbors." It is the same separation that Local Perplexity Ranking makes for flat outputs (On the Noise Robustness of In-Context Learning for Text Generation). Here it is conditioned on the shared ancestor prefix, so the comparison happens within one subtree.
Positioning
Three groups of prior work shape the design. The first motivates the problem. On the Noise Robustness of In-Context Learning for Text Generation shows that noisy demonstrations damage prompt-based generation and that more demonstrations amplify the damage. Its repair method, LPR, detects noisy examples by neighborhood perplexity and replaces them wholesale, and it was tested only on flat QA, code and classification. We reuse its core idea of comparing an example's perplexity with its neighbors'. We change the unit to a single tree edge and the action to an in-place relabel. Wholesale replacement is costly in taxonomies, where the replaced example may be the only one for a rare leaf. HiGen's gains on long-tailed classes show how much the tail matters (HiGen: Hierarchy-Aware Sequence Generation for Hierarchical Text Classification).
The second group supplies the generation method. Fine-tuned sequence-to-sequence classifiers write the root-to-leaf path token by token, and they establish that depth-first ordering with constrained decoding gives fully consistent paths and the best accuracy (A Study on Hierarchical Text Classification as a Seq2seq Task; Constrained Sequence-to-Tree Generation for Hierarchical Text Classification). These systems are all supervised and monolingual, and they assume clean training labels. We keep their output format and decoding constraint but run them on a frozen model driven by prompts. The closest prompt-based competitor, Retrieval-style In-Context Learning for Few-shot Hierarchical Text Classification, deliberately avoids single-pass path generation. It queries the model once per level with a pruned candidate set and a trained retriever. It treats English and Chinese as separate benchmarks and never tests corrupted retrieved examples. Our system makes one call per document, pools examples across languages, and is built for a noisy pool.
The third group constrains the multilingual design, and two findings pull in opposite directions. Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning finds that mixing high-resource-language demonstrations beats English-only prompts for low-resource targets. In the same paper, prepending irrelevant non-English sentences helps in about 71% of cases. The distraction literature finds the opposite: irrelevant context spreads attention thin and degrades precise answers (Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation; Neil Dhruva et al.; Sukmin Cho et al.). Taxonomy paths demand exact constraint adherence, so it is unclear which effect wins. We treat the multilingual pool as the regime that tests this, not as a free gain. Fralak at SemEval-2024 Task 4 shows that English-trained path generators can transfer zero-shot to Bulgarian, Macedonian and Arabic. It also reports frequent early stopping, where the generator emits its end-of-sequence token too soon and loses recall. That warns us against any repair that shortens demonstration paths.
Research Program
The system has three parts. The first is a multilingual example pool with a retriever. We train it jointly across all languages on the pooled few-shot data, with a contrastive objective that separates sibling categories, following the indexer design of the retrieval-style ICL paper. Joint training across languages is the "multilingual joint training" ingredient. Its role is to make neighbors in a high-resource language useful evidence for a low-resource example.
The second part is the level-wise detector above. It scores every step of every pool example once, offline, so it adds no cost at query time.
The third part is the repair step. It replaces a flagged label $l_i$ with the most common child of $l_{i-1}$ among the example's neighbors, and it repairs the label's descendants the same way. We will compare this against two alternatives. One truncates the path at depth $i-1$, which we expect to teach early stopping. The other drops the example, which is LPR's choice.
At query time, the prompt holds the retrieved, repaired examples, and the frozen model writes the depth-first path under trie-constrained decoding. Labels are verbalized as natural-language names, since replacing names with codes costs 4–5 Macro-F1 on RCV1-V2 (A Study on Hierarchical Text Classification as a Seq2seq Task). We will test labels in English and in the document's language as a separate factor. This addresses the open question of whether translated taxonomies add translation artifacts (Translation Artifacts in Cross-lingual Transfer Learning).
The analysis has two parts. The first is a controlled noise study on clean pools. We inject ancestor corruption, sibling transposition, and irrelevant padding text at rates from 0 to 60%, with demonstration counts from 1 to 16. This measures how damage depends on depth and on the number of examples. The second tests the multilingual tension directly. We compare same-language, English-only and mixed high-resource demonstrations, each with and without irrelevant non-English sentences, under constrained and unconstrained decoding. The question is whether the priming benefit survives when the output must follow the tree exactly.
Baselines
The naive reference is randomly sampled demonstrations with constrained decoding and no noise handling. Next to it sits a zero-shot prompt that lists the label names. Every method uses the same frozen backbones, the same pools and the same maximum prompt length. These two show how much any demonstration selection matters.
The strongest existing prompt-based method is the retrieval-style iterative policy of Retrieval-style In-Context Learning for Few-shot Hierarchical Text Classification. That paper reports 68.91 Micro-F1 on 1-shot WOS, against 58.95 for HierVerb. We will re-run it with its trained indexer and label descriptions on our backbones and our noisy pools, and give it the same retrieval budget. We will report Micro-F1 and the number of model calls per document. Beating it means matching or exceeding its Micro-F1 on clean 1-shot WOS with one call instead of one per level, and exceeding it at noise rates of 20% and above. We expect the advantage under noise because it filters nothing.
The direct competitor for noise handling is LPR from On the Noise Robustness of In-Context Learning for Text Generation, applied to the whole path string. It uses the same neighbor count and encoder as our detector. On flat tasks it reports raising NQ exact match from 28.21 to 42.00 at 60% noise. On hierarchical paths no reported result was found on Lacuna. Beating it means higher path-constrained Micro- and Macro-F1 at matched noise, with most of the margin on Macro-F1 and deep levels, where whole-example replacement removes rare leaves.
Three ablations isolate the key ingredients. The first keeps level-wise detection but swaps in-place repair for removal. The second keeps repair but restricts neighbors to the same language, which isolates the jointly trained cross-lingual retriever. The third runs constrained decoding alone, with no detection, to show how much of the noise damage the constraint absorbs. As supervised reference points we cite the fine-tuned DFS-C result of 86.42 Micro-F1 on full-data WOS (A Study on Hierarchical Text Classification as a Seq2seq Task). We also cite Fralak's hierarchical F1 on SemEval-2024 Task 4: 0.56 English, 0.46 Bulgarian, 0.46 North Macedonian and 0.43 Arabic (Fralak). On that task our pooled multilingual pool should beat its zero-shot numbers.
Why not just…?
Why not just rely on constrained decoding, since it already guarantees valid paths? Constrained decoding blocks paths that leave the tree. A sibling-transposed demonstration still teaches a legal edge, and an ancestor-corrupted one teaches a legal but wrong subtree. Neither is blocked. This is an empirical claim, and the constrained-only ablation settles it. If constrained decoding alone recovers most of the clean-pool accuracy at 40% noise, the detector is unnecessary and we will say so.
Why not just run LPR on the path string? This is the real threat, because LPR already does neighbor-relative perplexity. Our bet rests on two differences. Whole-path perplexity is dominated by hard deep tokens, so it confuses rare correct leaves with errors. Replacement also deletes the rare-leaf coverage that hierarchies depend on. If whole-string LPR matches level-wise repair on Macro-F1 and deep-level accuracy, our contribution reduces to a benchmark study. The head-to-head at matched noise will decide this.
Why not just use the iterative per-level policy, which already beats HierVerb? It does work well on clean data. Its cost grows with tree depth, and it never checks its retrieved examples, so a corrupted ancestor example misleads the first call and every later call inherits the error. We report both accuracy and calls per document. On the 4-level Chinese Patent hierarchy, it would have to beat us under noise by enough to justify roughly four times the calls.
Why not just fine-tune a small seq2seq model or a flat multilingual encoder on the few-shot pool? In the 1-shot regime, fine-tuned and flat contrastive baselines fall to near 0% path-constrained Micro-F1 on WOS, while HierVerb reaches 39.77% (Hierarchical Verbalizer for Few-Shot Hierarchical Text Classification). Fine-tuning also trains directly on the noisy labels. We still include a fine-tuned Seq2Tree at 16 shots per class, to find the shot count where it overtakes prompting.
Why not just translate everything into English? Translation adds artifacts that classifiers trained on original text learn to exploit (Translation Artifacts in Cross-lingual Transfer Learning). Mixed high-resource prompts beat English-only prompts in 23 of 30 setups (Blessing of Multilinguality). Neither finding was measured on taxonomy paths, however, and machine-translated English patents doubled French subgroup precision (Exploring Data-Centric Strategies for French Patent Classification). We therefore run a translate-test arm, with inputs translated to English and English labels, as a full baseline.
Proposed Experiments
The English benchmarks are WOS (2 levels, 141 classes) and DBpedia (3 levels, 298 classes) in 1-, 4- and 16-shot settings. For multilingual coverage we use the Chinese Patent set (4 levels, 437 classes), SemEval-2024 Task 4 (English training, Bulgarian, Macedonian and Arabic tests) and the French IPC patents. On French IPC we stop at group level L6, where the published ensemble reaches 63.97% P@1, as a large-label stress test of trie-constrained prompting. RCV1-V2 and BGC test multi-path outputs.
The metrics are Micro- and Macro-F1, path-constrained Micro- and Macro-F1, per-depth accuracy, the rate of early end-of-sequence emission, hierarchical consistency for unconstrained runs, and model calls per document. We will also report detector precision and recall at each depth against the injected ground truth, broken down by noise type and language.
Three decisive checks follow. First, at 40% injected noise on WOS and Patent, level-wise repair should beat whole-string LPR on Macro-F1. Most of the gain should sit at the deepest level, and constrained decoding alone should recover less than half the clean-to-noisy gap. If not, the edge-level framing is wrong. Second, for low-resource test languages, cross-lingual neighbors should give higher detection recall than same-language neighbors. Third, in the multilingual tension study, the irrelevant-sentence priming effect will be measured under constrained path generation. Whether it helps or hurts, the result tells practitioners whether priming tricks from multilingual reasoning carry over to tasks where every output token must stay in the tree.
References
- A Study on Hierarchical Text Classification as a Seq2seq Task
- Constrained Sequence-to-Tree Generation for Hierarchical Text Classification
- On the Noise Robustness of In-Context Learning for Text Generation
- HiGen: Hierarchy-Aware Sequence Generation for Hierarchical Text Classification
- Retrieval-style In-context Learning for Few-shot Hierarchical Text Classification
- Blessing of Multilinguality: A Systematic Analysis of Multilingual In-Context Learning
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- Solving Open-Domain Multiple Choice Questions with Textual Entailment and Text Similarity Measures
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Fralak at SemEval-2024 Task 4: combining RNN-generated hierarchy paths with simple neural nets for hierarchical multilabel text classification in a multilingual zero-shot setting
- Translation Artifacts in Cross-lingual Transfer Learning
- Hierarchical Verbalizer for Few-Shot Hierarchical Text Classification
- Exploring Data-Centric Strategies for French Patent Classification: A Baseline and Comparisons