Prototype Leakage: When Editing a Retrieved Report Copies Someone Else's Findings

Retrieve-and-edit generation writes a new text by fetching a similar human-written example (a "prototype") and rewriting it to fit the current input. The idea is that editing is easier than writing from scratch. In clinical reporting the premise is appealing, because radiology reports are highly templated and models that write from scratch tend either to produce bland boilerplate or to invent details. Retrieval has a cost that nobody has measured, though. A retrieved report describes a different patient. When the editor fails to overwrite that patient's findings, the output states facts that were never in the input. We call this prototype leakage. It is a hallucination whose source can be traced, and a model that sounds fluent is exactly the kind of model likely to hide it.

This note proposes to measure prototype leakage directly and to train it away. We decompose every unsupported clinical statement in a generated report into two groups: statements copied from the prototype and statements invented from nothing. We then test two remedies. The first trains a biomedical pre-trained editor on deliberately corrupted prototypes so that it learns not to trust their content. The second reranks several edited candidates by how much prototype content leaked through. The central empirical question is whether prototypes still help once leakage is controlled. That means beating a strong domain-pre-trained model that writes from scratch and is given the same candidate-reranking budget. If they do not help, the field's "fluency anchor" story for clinical text is mostly an illusion. That result would also be worth reporting.

Core Claim

In domain-specific retrieve-and-edit generation, a measurable share of factual errors comes from content carried over from the prototype rather than from free invention. Two changes can remove most of that share while keeping the structural benefit of editing. The first is training the editor on counterfactual prototypes, meaning prototypes whose findings contradict the target. The second is selecting among multiple edits with a leakage-aware score. The claim is falsifiable at three points. Leakage may turn out to be a negligible fraction of errors. The anti-leakage training may not reduce it. Or a from-scratch model with equal reranking may match the final system on faithfulness.

Where This Sits

The prior work splits into three roles.

Some work motivates the proposal by showing that prototypes help in clinical text. Clinical Report Auto-completion (CLARA) retrieves human-written report sentences with Lucene and edits them with a recurrent sequence-to-sequence network, reporting CIDEr 0.393 on IU X-ray and a neurologist rating of 3.74/5 against 2.52/5 for baselines. Replace and Report anchors generation to a fixed 23-sentence normal template written by a radiologist and swaps in sentences describing abnormal findings. A radiologist judged 78% of its reports clinically accurate, and 99.95% of normal cases were accurate. Neither system measures whether prototype content survives into outputs where it does not belong. CLARA sidesteps the question by having clinicians supply the key findings as "anchor words," and its accuracy drops sharply when those words must be predicted automatically. Replace and Report sidesteps it by using a template that contains only normal findings. Our setting, with dynamic prototypes containing abnormal findings and no human in the loop, is the one where leakage should be worst.

Other work supplies the method. Simple and Effective Retrieve-Edit-Rerank Text Generation shows that editing N retrieved prototypes and choosing among the outputs afterward beats committing to one prototype early. It reports gains of 6.5 and 7.5 BLEU on translation. Its oracle analysis leaves another 3 to 5 BLEU on the table, which says the scoring step is the bottleneck. We reuse this edit-then-choose design but replace likelihood scoring with a score based on clinical facts. RadGraph turns a report into a graph of clinical entities with their assertion status (present, absent, or uncertain) and the relations between them. Its extractor, built on the biomedical encoder PubMedBERT, reaches entity micro-F1 of 0.94 and relation micro-F1 of 0.82 on MIMIC-CXR. That makes it accurate enough to attribute each output statement to the source findings, to the prototype, or to neither. Better Late Than Never (MEDAL) synthesizes non-factual clinical summaries by masking error-prone medical terms and having BioBART fill them with low-ranked tokens, then trains a correction model on those examples. We reuse this corruption procedure, but we apply it to the prototype the editor sees rather than to the output. The editor then learns to ignore misleading prototype content when writing, instead of relying on a separate step to repair it afterward.

Other work constrains the design. BioBART continued BART's pre-training on 41GB of PubMed abstracts. It found that sentence-permutation denoising hurts biomedical generation, which suggests discourse order matters in this domain. It also flags a "clinical gap" between published abstracts and hospital reports. Both findings shape how we treat the editor backbone. Pre-Training Language Models on Specialized Corpora argues that domain data mixed into web corpora gets drowned out, which is why we compare a domain-pre-trained editor against a general one rather than assuming either. MHB shows that frontier models are highly vulnerable to adversarial clinical traps, with hallucination rates of 29.1% for Claude-4-Sonnet and 59.0% for Llama-4-Scout. Its tasks are dialogue and case reports rather than report summarization, so we do not use it as a test set. Instead we build our own adversarial prototypes, which play the same role as a misleading cue planted in context.

Research Program

The main task is text-only: generating the impression section of a MIMIC-CXR report from its findings section. MEDAL also evaluated on this task. Because the output can be checked against a written source, faithfulness is well defined and does not depend on an image model. Prototypes are impressions from training-set reports whose findings are nearest to the input findings under BM25 and under a dense retriever, with the same patient excluded. MeQSum, where BioBART reports Rouge-L 53.15 against 51.05 for BART-Large, serves as a second domain with a different structure. IU X-ray impression generation serves as a small-data check, because prototypes should matter most when little paired data exists.

The first component is the leakage measurement. We run RadGraph on the source findings $x$, the prototype $p$, and the output $y$. Let $E(\cdot)$ denote the set of extracted entities, each paired with its assertion status. Leakage is defined as

$$L(y) = \frac{|E(y) \cap E(p) \setminus E(x)|}{|E(y)|}$$

that is, the fraction of output entities that appear in the prototype but not in the source. We report it alongside the de novo rate: the fraction of output entities that appear in neither the source nor the prototype. Their sum is the total unsupported-entity rate. Assertion status is part of the entity key. If the source says "no effusion," the prototype says "small effusion," and the output says "small effusion," that counts as leakage. This captures the overconfidence and negation-flip errors that the hallucination literature highlights. Entity matching uses normalized surface forms, and we validate it on a radiologist-annotated sample.

The second component is the editor itself. Its backbone is BioBART-Large. It receives the source findings and one prototype and is fine-tuned to output the target impression. The key training change is to use counterfactual prototypes. With a probability that we tune, the prototype given to the editor during training is corrupted by MEDAL-style infilling on its RadGraph entities: a finding the source does not contain is inserted, an absent finding is made present, or the laterality or severity is changed. The target impression stays the correct one. The editor is thus explicitly rewarded for overwriting prototype content that conflicts with the source, while it can still copy the prototype's phrasing and order. This also bears on the BioBART tension about discourse. We measure whether this editing-heavy fine-tuning damages the model's discourse behavior, using a held-out sentence-order discrimination probe applied before and after fine-tuning.

The third component is the selection step. For each input we retrieve N=8 prototypes and produce one edit per prototype. We then choose the candidate that maximizes

$$s(y) = \log P(y \mid x, p) - \lambda L(y) - \mu D(y)$$

where $\log P(y \mid x, p)$ is the editor's own log-likelihood, $D(y)$ is the de novo rate, and $\lambda$ and $\mu$ are tuned on the validation set. A reader may worry that reranking with RadGraph and then evaluating with RadGraph is circular. We address this in two ways. We report SummaC, which is not graph-based and was used by MEDAL, as an independent faithfulness metric. And the final comparison relies on blinded radiologist annotation of a fixed sample of 300 impressions per system.

We also run a secondary analysis of the backbone. We compare leakage for BART-Large, BioBART-Large, and BioBART-Large given a short additional round of pre-training on MIMIC-CXR report text, since the clinical gap suggests PubMed pre-training alone may not suit report language. One plausible hypothesis is that domain pre-training increases leakage. A model fluent in radiology may find a clinical-sounding prototype more plausible and copy it more readily. That result would be interesting in its own right.

Baselines

The naive reference points come first. The first is nearest-neighbor copying: output the retrieved prototype impression unchanged. This fixes the maximum possible leakage and shows how much faithfulness retrieval alone provides. The second is a from-scratch model: BioBART-Large fine-tuned on findings-to-impression with no prototype, decoded by beam search. This is the "do nothing new" model a reviewer would reach for first. BioBART's reported MeQSum Rouge-L of 53.15 is the published number we must reproduce on that dataset before any comparison is trusted. For MIMIC-CXR impression generation with a RadGraph-based metric, no reported result was found on Lacuna for a BioBART baseline.

The strongest competitor on faithfulness specifically is a two-stage pipeline that applies MEDAL post-hoc correction to the from-scratch model's outputs. It is the strongest because it is the published method in this file that targets hallucination in exactly this task family. The paper reports a 6% to 7% faithfulness improvement on Llama-2 outputs across SummaC and Faithfulness-adjusted Recall on MIMIC-CXR, MeQSum, IU, and ACI-BENCH. It does not report an absolute score for our exact backbone, so we retrain its Flan-T5 corrector on our splits with the same corruption budget we give our own editor.

The strongest competitor on the prototype side is standard retrieve-edit-rerank, following Simple and Effective Retrieve-Edit-Rerank Text Generation. It uses the same BioBART editor and the same eight prototypes but no counterfactual training, and it reranks by likelihood alone. Its reported gain over Re³Sum on Gigaword is 1.2 ROUGE. It has not been reported on clinical data.

The ablation that isolates our key ingredients is a from-scratch model with the same selection step. We sample eight candidates from the from-scratch BioBART and choose among them with the same score minus the leakage term, which is undefined without a prototype. This tests whether our gains come from prototypes or merely from having multiple candidates and a fact-aware scorer. We further ablate counterfactual training and the leakage term separately.

All systems share the same backbone size, training data, retrieval index, number of candidates, and a fixed budget of hyperparameter search trials on the validation set. We would count our system as winning if two conditions hold. First, it lowers the total unsupported-entity rate relative to both MEDAL-on-scratch and the from-scratch model with equal reranking, and the radiologist-judged error rate is significantly lower than both in a paired test. Second, it is not worse on RadGraph F1 against the reference impression.

Why not just…?

Why not just fine-tune BioBART to write impressions from scratch? This is the most serious threat. Impressions are short, and a well-trained domain model may already produce the template language that prototypes are supposed to provide. The evidence that prototypes help, from CLARA and Replace and Report, compares against weak generic baselines such as recurrent networks and general-domain T5, not against a domain-pre-trained transformer. We settle the question with the from-scratch-plus-reranking ablation. If it matches our system, we will report that prototypes add nothing once the editor is domain-pre-trained. We expect prototypes to earn their place mainly on IU X-ray, where paired data is scarce, and on rare abnormal findings, which we analyze as a separate slice.

Why not just correct errors afterward with MEDAL? Its own paper notes that masking and self-examination can turn correct text into errors when the self-check misfires, and that its ROUGE-L context windowing breaks down on long, multi-finding inputs. Post-hoc correction also cannot tell whether an error came from a prototype, so it cannot fix the cause. It remains a strong competitor, however. We run it both as a baseline and stacked on top of our system, because a finding that the two are complementary is a plausible and useful outcome.

Why not just put several retrieved reports in an LLM's prompt as examples? Retrieval-augmented prompting is the same leakage risk without the control. The model sees another patient's findings and may copy them, and there is no training signal telling it not to. We include a general instruction-tuned LLM prompted in context with the same eight retrieved impressions, and we measure its leakage with the same decomposition. MHB's high hallucination rates for such models under misleading context suggest it will leak more, but that is an empirical question this experiment answers directly.

Why not just filter prototypes so they share the source's findings? Filtering on RadGraph overlap before editing is cheap. Yet the report retrieved as most similar almost never matches every finding, and filtering hard trades leakage for weaker structural anchors. We include retrieval filtered by entity overlap as an extra variant of the standard retrieve-edit-rerank baseline.

Decisive Checks

The first check is whether leakage matters at all. If, under standard retrieve-edit-rerank, leakage accounts for less than roughly a fifth of unsupported entities on MIMIC-CXR, the problem is minor and the proposal reduces to a negative diagnostic. The second check is the adversarial prototype test. At test time we deliberately feed each editor one corrupted prototype containing a planted contradictory finding and measure how often that finding appears in the output. Counterfactual training should cut this rate substantially relative to the untrained editor. If it does not, the training signal is not doing what we claim. The third and final check is the head-to-head radiologist evaluation against MEDAL-on-scratch and the from-scratch model with equal reranking. Only a significant win there supports the claim that prototypes are worth retrieving once their leakage is controlled.