Sibling Distractors in Retrieval-Based Medical Concept Normalization

Patients describe symptoms in their own words, such as "room is spinning" or "feeling completely drained." Clinical systems index formal terms like vertigo or fatigue. The task of mapping the first onto the second, called medical concept normalization, is now often done with retrieval. An embedding model pulls the ten or so ontology terms closest to the patient's phrase, and a large language model (LLM) picks one. The trouble is that the closest terms in a medical ontology are usually siblings: several distinct conditions filed under the same parent, such as different kinds of shortness of breath. Research on retrieval-augmented generation (RAG), where a model answers from retrieved text placed in its prompt, shows that passages resembling the right answer without being it do much more damage than unrelated ones. They can even do more damage than random text, which sometimes helps. A normalization pipeline that ranks candidates by similarity therefore builds its candidate list almost entirely out of the most harmful kind of distractor.

This note proposes to measure that effect directly and then remove it by changing how the candidate list is shaped, without changing the retriever or the LLM. The central claim is that errors in retrieval-based normalization come mainly from the ontological composition of the candidate list, not its length. A two-step selection fixes much of this. The model first picks a branch of the ontology from a deliberately diverse list, which is the regime where extra candidates are cheap. It then picks within that branch from siblings that are shown alongside what distinguishes them from each other. If the claim holds, it explains why fixed top-10 lists work as well as they do and fail where they fail, and it gives a cheap, tuning-free alternative to multi-agent ranking.

Where the directions meet

The evidence for distractor toxicity comes from open-domain QA. Rethinking Relevance separated semantically similar non-answer passages from random Wikipedia passages for Llama 2 7B. Adding 18 similar distractors to the gold passage dropped accuracy from 56.42% to 37.81%. Adding 15 random passages to 4 retrieved ones raised accuracy from 18.66% to 25.30%. Beyond RAG vs. Long-Context found an inverted-U relationship between context size and accuracy. It proposed LDAR, which reads the distribution of retriever similarity scores to choose how many passages to include. Aligning Extraction and Generation trains the generator to extract evidence sentences before answering, which makes it robust to 25 noisy chunks. All three motivate this proposal, and none of them involves a vocabulary gap. In Ext2Gen the answer can be copied out of the context. In normalization the model must translate between registers and choose a label.

Two recent systems already combine retrieval with normalization, and both use flat top-k candidate lists. TLab at #SMM4H 2024 embeds the extracted span, retrieves the 10 nearest MedDRA Preferred Terms, and asks GPT-4 to choose. The authors say the choice of 10 was arbitrary. RankNorm first expands the mention into a "Knowledge Card" (an LLM-written description of its clinical meaning) to improve retrieval. It then narrows 200 candidates to 10 and has three doctor-persona agents vote. RankNorm supplies our strongest baseline, and its Knowledge Card is also an input we reuse. It improved HR@1 on SMM4H-17 from 47.36% to 57.68%. However, the expansion makes the query read more like formal clinical text, which should pull in even more formal-sounding siblings. Neither paper measures what the siblings in the list do to the final choice.

Two papers show that siblings specifically are the failure point. Evaluating the Robustness of Biomedical Concept Normalization concatenated a mention with a sibling concept from the same ontology parent. PubMedBERT F1 on NCBI Disease fell from 0.853 to 0.580. Am I eligible? shows that lay phrasing often lacks the detail needed to separate closely related clinical categories. This finding constrains the proposal: some sibling confusions are irreducible, and the evaluation has to allow for them. Rationale-Guided RAG prunes distractor snippets by how much they lower the generator's perplexity on the answer. It scores each snippet independently and was only tested on multiple-choice QA. Our method differs because it uses the ontology's own hierarchy to decide how candidates should be grouped, rather than scoring each candidate alone.

Core Claim

We make two claims. The first is diagnostic. Hold the gold concept's presence and position fixed in the candidate list. Accuracy then falls with the number of gold-concept siblings in the list, and is roughly flat or slightly rising with the number of candidates from distant branches. This is the similar-versus-random asymmetry, carried over to ontology space.

The second claim is constructive. A coarse-to-fine selection recovers most of the sibling-induced errors while using the same candidate pool, the same retriever, and the same LLM. In the coarse step, the LLM sees one representative per parent node, which is a diverse list. In the fine step, it sees only the siblings in the chosen branch, each annotated with a short contrastive note. We generate each note once per concept, offline, from ontology definitions: "unlike X, this term implies Y." That makes the notes a fixed cost, not a per-query one. The new element is the separation. Diversity is used where it helps, and unavoidable similarity is made explicit where it cannot be avoided.

Research Program

We use the MedDRA hierarchy for the SMM4H data and SNOMED-CT for CADEC and AskAPatient. MedDRA's levels are Lowest Level Term, then Preferred Term (PT), then higher-level grouping terms, then System Organ Class. Normalization targets PTs, so the "branch" is the higher-level term that groups them.

The first phase is a controlled injection study. For each test mention, we build candidate lists of fixed length that always contain the gold PT at a randomized position. We vary the rest of the list along a single axis: the share of slots filled by siblings, by cousins (same grouping term two levels up), by distant PTs from other organ classes, or by nothing. We run this with two open 8B-class instruction models and one frontier model. That lets us test whether the effect depends on model capacity, as LDAR reported for noise tolerance. We also label every error in the naive pipeline by its ontological distance from the gold concept. This gives a sibling error rate: the fraction of wrong predictions that share a parent with the gold PT.

The second phase builds the coarse-to-fine selector on top of a retrieval stage that is the same across all conditions, so retrieval recall is held constant. That stage is RankNorm-style Knowledge Card retrieval with an open embedding model. From the top 200 candidates, we group candidates by parent. The coarse step shows the highest-scoring member of each of the top branches and asks for the best branch, with the option of naming two. The fine step shows every retrieved sibling within the chosen branch or branches, together with its contrastive note. Coarse-step errors cannot be recovered, so we report branch-level recall on its own and treat it as the main risk to the method.

Baselines

The naive reference is the TLab pipeline as published: embed the span, take the 10 nearest PTs, and have the LLM choose. On SMM4H 2024 it reports normalization F1 0.359 and unseen-concept F1 0.363 (TLab). We rerun it with our backbone LLM so that all comparisons share one model.

The strongest existing method is RankNorm, which reports 90.84% accuracy on SMM4H-17 against 89.40% for supervised BioBERT (RankNorm). It is the strongest competitor because it is tuning-free and already attacks the vocabulary gap with query expansion. We run it in two ways. The first reproduces its full multi-agent configuration with its reported backbone, to check our numbers against the published ones. The second puts it on the same backbone as our method. That setting allows about the same number of LLM tokens per mention, and the multi-agent version gets a stated multiple of that budget. Our method counts as beating RankNorm if it matches RankNorm's accuracy within noise at a fraction of the tokens. It counts as a stronger result if it beats RankNorm at an equal token budget. Either way, the gain should be concentrated in the sibling-error slice.

The adaptive-size baseline is LDAR (Beyond RAG vs. Long-Context). It is trained on retrieval score distributions from the training split to choose each mention's candidate band. It gets the same pool and backbone. We found no reported result on Lacuna for LDAR on concept normalization.

For supervised reference points we use Ontology Pretraining plus fine-tuning (Portelli et al.), which reports 40% accuracy on unseen concepts in its SMM4H split. We also use GRU with attention and TF-IDF features (Tutubalina et al.), which reports 70.05% on CADEC's zero-overlap split.

The key ablation is a flat hierarchy-annotated list. It is the same candidates and the same contrastive notes, with each candidate tagged by its parent path, but shown in one step. This isolates whether the two-step procedure matters or whether exposing the hierarchy is enough.

Why not just…?

Why not just tune k, or use LDAR to choose it adaptively? This is a real threat. If sibling harm were simply harm from list length, a smaller or adaptive k would remove it. The injection study settles the question, because it holds length fixed and varies only composition. If composition has no effect at fixed length, our claim fails and LDAR-style sizing is the right fix. Even so, a shorter list keeps exactly the nearest siblings and drops the distant candidates, which is the opposite of what the paradox in Rethinking Relevance recommends.

Why not just add random distant candidates as regularizers? This intervention is cheap, and we include it as a condition in its own right. We pad the naive top-10 with distant PTs. If it recovers as much as coarse-to-fine does, the simpler method wins and we will report that. We expect it to help only partly, because it leaves the siblings in place and the siblings are where the errors are. The sibling-error-rate breakdown will show which errors each method removes.

Why not just fine-tune a cross-encoder or supervised normalizer? Supervised models embed the label set in their weights, and they fail on concepts absent from training: 0% for standard fine-tuning on the unseen split in Portelli et al.. Sibling concatenation also cuts BERT-based normalizers by about 30% F1 (Robustness of Biomedical Concept Normalization). We still report these models on the seen-concept split, where they may well win. Our claim covers only the tuning-free, ontology-updatable setting and unseen concepts.

Why not just use a bigger model with a longer list? The inverted-U finding suggests longer lists eventually hurt. Whether frontier models suffer from siblings at all is an empirical question, and the injection study tests it directly by including a frontier model. If the frontier model is insensitive to siblings, the contribution narrows to cost-constrained open models, and we will state that plainly.

Proposed Experiments

We evaluate on SMM4H-17, TwADR-L, and AskAPatient, which are RankNorm's datasets. We also use SMM4H 2024 Task 1 with its unseen-concept subset, and CADEC's zero-overlap split. The metrics are accuracy@1 (or the task's normalization F1), branch-level recall for the coarse step, sibling error rate, and LLM tokens per mention.

Three results decide the question. First, at fixed list length and gold position, accuracy must fall monotonically as the sibling share of the list rises, and distant fillers must not cause a comparable fall. Without this, the premise is wrong. Second, coarse-to-fine must beat the flat hierarchy-annotated ablation, with most of the gain coming from reduced sibling errors. Without this, a better prompt would be enough. Third, the gains must hold on unseen-concept splits and must not be explained by lost branch recall.

To handle genuinely underspecified mentions of the kind described in Am I eligible?, we also report accuracy at the parent level. That separates errors the method could fix from confusions where the patient's phrasing does not contain enough information to decide.