Evidence-Anchored Taxonomy Placement for Concepts the Model Has Never Seen

Large taxonomies, the "is-a" trees behind search, product catalogs, and biomedical indexing, can no longer be curated by hand. The newest systems insert each new concept by asking a language model to pick its parent. The concepts that most need automation, though, are the ones that appeared after the model was trained: a new method name, a newly approved drug class, a new product type. For those concepts the model has no stored knowledge. It still answers fluently, filling in a plausible definition from the surface form of the name and placing the concept under the wrong branch. This is hallucination in structural form, and nothing in the output marks it as a guess.

We propose to attach retrieval to each decision in the placement process. The model walks down the tree from the root. At every point where the remaining candidate parents split into competing sibling branches, it retrieves passages about the new concept together with those specific siblings, and it must quote a passage that supports taking one branch. If no quote supports a branch, it stops at the deepest supported ancestor and flags the concept rather than guessing. The claim is narrow and testable. For concepts that fall after the model's training cutoff, most placement errors come from choosing between close siblings on the basis of a made-up understanding of the concept. Evidence retrieved for that specific choice should fix more of these errors than evidence retrieved once for the concept in general.

Core Claim

Existing LLM-based expansion systems draw their knowledge of the new concept only from the model's parameters. LORex ranks candidate parents with an Euler-tour linearizer, TEMPORA, that writes each root-to-node path as text. It then prunes and verifies candidates through an LLM loop, scoring whole paths by average token log-probability instead of accepting free text. LORex is our method source and our strongest competitor. It explicitly leaves out external document retrieval, and candidates missed by its ranker cannot be recovered. TaxoAdapt grows multi-dimensional taxonomies from paper density, but it suffers from what the file calls parametric knowledge decay: the model labels emerging concepts with outdated terminology. GenTaxo generates missing concept names token by token from graph and sequence encoders, again with no retrieval to anchor them. These three papers motivate the gap. None of them tests on concepts the model provably cannot know, and none gives the model evidence at the moment it must choose between siblings.

The retrieval side has built hierarchical structure too, but in the reverse direction. HiRAG clusters entities into a layered knowledge graph to improve question answering. HiReview induces a topic tree from citation graphs to constrain review writing. Both use a hierarchy to help retrieval. Neither uses retrieval to build or verify an "is-a" hierarchy, and neither evaluates hypernym correctness. Our proposal sits in that unoccupied direction.

Research Program

The system keeps LORex's front end so that any gain can be attributed to retrieval. TEMPORA proposes the top-k candidate parents. We then find the divergence points: the nodes where the lineages of the remaining candidates split. Placement becomes a short sequence of isolated sibling choices, one per divergence point. This design follows the analysis of divide-and-conquer prompting, which shows that splitting a task into isolated sub-decisions limits how far errors spread. It also follows the finding in Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? that small distractors break otherwise capable models. Close siblings are exactly such distractors.

At each divergence point the system builds a retrieval query from three parts: the new concept's name, its own short context, and the labels of the competing siblings. It runs both keyword retrieval and dense retrieval over a time-stamped external corpus, using PubMed abstracts for biomedical taxonomies and paper text for computer science. The keyword retrieval can use LLM-extracted n-grams in the style of Boros et al.. The model must output a branch choice together with a verbatim quote. The quote is accepted only if it overlaps sufficiently with a retrieved passage. We borrow this quote-faithfulness check from RLSeek, which trains a hallucination detector to quote its sources, and apply it at inference time without any reinforcement learning.

If no accepted quote supports any branch, descent stops. The concept attaches to the last supported ancestor and is flagged for review. This stopping rule creates a real risk: the system may deliberately label concepts too generally, which is exactly the "lazy" labeling behavior Hui Shen et al. document in human annotators. We therefore report over-general placements as a separate error type instead of hiding them inside accuracy.

The final path is scored with LORex's log-probability verifier, now conditioned on the accepted quotes. We also record the evidence shift: the change in the verifier's path score when the quotes are added versus withheld. A large shift suggests that the model's parametric judgment disagreed with the evidence. We test whether this shift predicts errors in the no-retrieval system, which would make it a cheap signal for deciding when to retrieve at all, in the spirit of Adaption-of-Thought. Finally, retrieved passages that name a hypernym outside TEMPORA's top-k are added back as candidates. This directly targets LORex's inability to recover from ranker omissions.

The essential new artifact is a temporal-split evaluation: concepts added to a taxonomy after the language model's training cutoff. For MeSH, we take descriptors introduced in releases after the cutoff and use the pre-cutoff tree as the seed. For computer science, we use the gap between TaxoAdapt's EMNLP 2022 and EMNLP 2024 corpora. The file does not state that TaxoAdapt releases gold placements for the 2024 concepts, so if they are unavailable, gold placements for new concepts are obtained by expert annotation. We exclude MeSH scope notes and any page derived from the taxonomy itself from the retrieval corpus, so that retrieval cannot simply look up the answer.

Baselines

The naive reference point is the LORex pipeline run exactly as published: same ranker, same LLM, and no retrieval. This measures how far parametric knowledge alone goes. On the temporal split we expect it to degrade, and the size of that degradation is the opportunity. The file reports only relative figures for it. LORex improves accuracy by 12% and Wu & Palmer similarity by 5% over FLAME and CodeTaxo, and TEMPORA improves Hit@k by 21.3% over TEMP, all from LORex. No reported absolute result was found on Lacuna for LORex accuracy on SemEval-2016, WordNet, or MeSH, so we will rerun it ourselves. Beating it means higher accuracy and Wu & Palmer similarity on the post-cutoff split without a drop on SemEval-2016 and WordNet.

The strongest non-LLM competitor is TaxoBell. TaxoBell represents each concept as a Gaussian and uses an asymmetric KL divergence to test whether a child sits inside its parent. It reports a 43% Mean Rank improvement on MeSH and 19% MRR and roughly 25% Recall@k gains across its datasets. We include it because it does not depend on the LLM's training cutoff, so it may degrade less on new concepts than LLM methods do. It will be trained on the same pre-cutoff seed taxonomy.

The key ablation is concept-level RAG. It retrieves passages once, using the concept name alone, prepends them to the unchanged LORex loop, and keeps the same retriever, corpus, and token budget. This isolates our key ingredient, retrieval conditioned on competing siblings at each divergence point, from the simple fact of having any evidence at all. A second ablation keeps sibling-conditioned retrieval but drops the quote requirement and the stopping rule, which isolates the anchoring step.

All systems use the same LLM, the same top-k from TEMPORA, and the same retrieval corpus where retrieval applies. We match or report retrieval calls and generated tokens per query. LORex's semantic filter reduced LLM calls from 3.1 to 1.7 per query, and we report our call count on the same footing.

Why not just…?

Why not just prepend a retrieved definition of the new concept? This is the most serious threat, and it may well capture much of the gain. A definition often states the answer outright, as in "X is a type of Y." The concept-level RAG ablation settles the question directly. We commit to a threshold in advance: if concept-level retrieval recovers at least 80% of our method's accuracy gain over LORex on the post-cutoff split, we will conclude that per-step conditioning is unnecessary and report that result. We expect a gap mainly on sibling-ambiguous cases, where a generic definition does not separate two close children. We will stratify results by sibling similarity to test this.

Why not just mine "X is a Y" patterns from retrieved text, as classic extraction does? Pattern mining is cheap and does not hallucinate. We will run it as a baseline over the same corpus. GenTaxo shows that extraction methods fail on concepts that occur fewer than 100 times in 30 million abstracts, and newly introduced concepts are rare by construction. We therefore expect low recall, with extraction winning only on the minority of concepts that do have explicit pattern sentences. Reporting coverage and precision separately will show where each approach wins.

Why not just use a newer frontier model with the whole candidate set in its prompt? A newer model moves the cutoff forward but does not remove it, and a 50,000-node tree does not fit in a single prompt, as LORex notes. Prompted large models also violated single-parent rules that fine-tuned small models kept, according to Boqi Chen et al.. We will plot accuracy against concept introduction date relative to each model's cutoff. If a sufficiently fresh model erases the gap, the method's value reduces to keeping systems current between model releases, and we will state that plainly.

Why not just ask for step-by-step reasoning without quotes? Chain-of-thought is the no-quote ablation. RLSeek found that wrong reasoning steps cluster where no source text is quoted. Evaluating Intermediate Reasoning of Code-Assisted LLMs shows that correct answers often come from unsound steps. Beyond accuracy, we will measure how often a correct placement rests on an invalid or fabricated justification.

Proposed Experiments

Evaluation covers SemEval-2016 Task 13 (Environment, Science, and Food), the 114 WordNet sub-taxonomies, MeSH, and the two temporal splits. The retrieval corpora are PubMed abstracts for biomedical concepts and the TaxoAdapt paper sets for computer science. The standard metrics are accuracy, MRR, Hit@k, and Wu & Palmer similarity. Three structural-hallucination metrics accompany them. The confident-wrong rate is the share of wrong placements that the verifier scores in its top confidence decile. The over-general rate is the share of placements attached to a true ancestor rather than the true parent. Quote validity is judged by human raters on a sample of 300 traces: does the quoted passage actually support the chosen branch?

The decisive checks are these. First, on SemEval-2016 and WordNet, whose concepts all predate the cutoff, our method should roughly match LORex. A large gain there would suggest leakage, not grounding. Second, on the post-cutoff splits it should beat both LORex and concept-level RAG, with the margin concentrated in cases where siblings are close. Third, the confident-wrong rate should fall without the over-general rate rising enough to erase the net accuracy gain, which we check with selective-accuracy curves. Fourth, the evidence shift should predict no-retrieval errors well enough to permit adaptive retrieval at lower cost. Finally, a leakage audit will rerun the temporal split with a retrieval corpus cut off at each concept's introduction date. The method should degrade toward the LORex baseline, confirming that its gains come from evidence and not from the corpus leaking the answer.