Graph-Keyed Error Notebooks for Transferring Radiology Reasoning Across Vision-Language Models

Small vision-language models that write chest X-ray reports usually see the right thing and then say the wrong thing. They notice an opacity but put it on the wrong side, drop a "no" that reverses the finding, or invent a device that is not in the image. A large model can often catch these mistakes, but running it on every study is expensive, and fine-tuning a small model on its corrections ties the fix to one set of weights. We propose that a frontier "teacher" model write a notebook of short corrective notes about the errors small models make. The notebook is indexed by the structured clinical graph of the student's own draft (which findings, where, negated or not), not by image or text embeddings. Because that graph is the same no matter which model wrote the draft, one notebook can be plugged into many student models at inference time with no weight updates. The same graph indexing also tells the system when the notebook has nothing relevant to say, so it can stay silent instead of injecting misleading near-matches.

This matters because radiology report generation is where three known failures meet. Fluent text hides reversed facts that ordinary overlap metrics miss. Models fail at reasoning even when perception is adequate. Retrieval systems return confident noise when the reference store does not cover a rare finding. The proposal treats cross-model memory transfer as the core mechanism and radiology as the application. A clinical knowledge-graph metric does three jobs: it is the index, it is the gate that admits notes, and it detects coverage gaps. The perception-reasoning gap is the failure being repaired.

Core Claim

A memory of teacher-written corrections transfers across student architectures better, and fails more safely on rare findings, when it is keyed and checked in a shared symbolic space than in any model's embedding space. Concretely, a notebook keyed on RadGraph patterns of student drafts should do three things. It should improve clinical-fidelity scores of students that never contributed errors to it. It should add gains on top of fact-aware report retrieval rather than duplicate them. And its graph-match coverage score should flag the cases where retrieved notes would hurt, more reliably than embedding distance does. A RadGraph pattern is the entities, locations, modifiers and negation status that the parser extracts.

Where This Sits

The mechanism comes from training-free memory transfer. Error Notebook-Guided, Training-Free Part Retrieval in 3D CAD Assemblies via Vision-Language Models has a model reflect on failed trajectories, writes corrected reasoning paths, filters them with a grammar verifier, and retrieves them as exemplars. It reports that GPT-4o rose from 41.7% to 65.1% accuracy and that Qwen2-VL-2B came within about 4 points of GPT-4o mini using a GPT-4o-authored notebook. MemRL and AdaMEM show that text experience banks built by one model family help another: Gemini-3-pro to Qwen3-235B, and Qwen to Gemma-27b. These papers motivate the transfer claim, but all three verify memory with binary success or string checks. They also retrieve by dense similarity, which is exactly what fails in radiology. We keep their "teacher writes, student reads" structure and replace both the verifier and the index.

The domain-metric work supplies our verifier and constrains our evaluation. RadGraph parses reports into clinical entities and relations. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation builds RadGraph F1 and the composite RadCliQ, reports a Kendall-tau b of 0.522 with radiologist error counts, and finds that generators beat a random retrieval baseline by only 5.5% on RadCliQ. That last result is a warning: in this domain, plain retrieval of a similar report is a hard baseline. Fact-Aware Multimodal Retrieval Augmentation (FactMM-RAG) uses RadGraph to mine clinically equivalent report pairs and trains a retriever on them, gaining up to 6.5% F1CheXbert and 2% F1RadGraph. It is the closest prior work. It retrieves reference reports, the content a correct report should contain. We retrieve corrections, the specific mistakes a draft with this graph pattern tends to contain. That is a different object, and the experiments test whether it adds anything. Style-Aware Radiology Report Generation shows that generating a serialized RadGraph first and verbalizing it with a frozen LLM improves RadGraph F1 from 0.186 to 0.228. This supports treating the graph as a model-agnostic interface.

The coverage literature tells us what to guard against. MOTOR documents dense retrievers pulling unrelated pathologies, such as edema for a lung-base opacity query, and fixes ranking with spatially grounded optimal transport. It still assumes the database contains a good match. From Conflict to Consensus (MA-RAG) detects coverage gaps from disagreement among sampled answers, reporting +6.8 points over Qwen3-8B on text-only medical QA. It notes that token-level confidence is a poor trigger and that shared biases can produce false consensus. Our coverage signal is deterministic and requires no extra sampling, which makes MA-RAG's disagreement signal the natural competitor. A real tension shapes the index design. DCoT found text-only demonstration retrieval beat image-plus-text retrieval, while MOTOR found spatial information indispensable. Graph keys keep laterality and location as discrete symbols, which is our bet on reconciling the two. The bet is tested by an ablation, not assumed.

Research Program

Building the notebook. Several "builder" students generate draft reports on a training split of MIMIC-CXR: Qwen2-VL-2B-Instruct and LLaVA-Med. RadGraph parses each draft and its reference. The graph difference lists what went wrong: missing findings, hallucinated findings, flipped negations, and wrong anatomical attachment. GPT-4o, acting as teacher, sees the image, the draft, the reference, and the difference. It writes one short checkable note, for example "when a draft reports a left effusion with no blunting described, verify the costophrenic angle on the stated side before asserting laterality." The note is stored under a key: the minimal sub-graph of the draft that triggered the error.

Admission rule. A note is admitted only if it transfers. Let $s$ be an admission student that did not produce the error, $x$ the study, $r$ the reference, and $F(\cdot, r)$ the RadGraph F1 against $r$. The note $n$ is kept if

$$F(s(x \mid n), r) - F(s(x), r) \ge \delta$$

on the triggering study and on a small set of other studies whose drafts share the key. Here $s(x \mid n)$ is the admission student's report when the note is in its prompt, and $\delta$ is a small threshold tuned on validation. Requiring improvement in a second architecture is what separates transferable lessons from fixes specific to one model. This extends the CAD paper's grammar verifier to a verifier that encodes clinical meaning.

Retrieval in graph space. At test time, a new student, never used to build or admit notes, writes a draft. The draft is parsed, and notes are retrieved whose key sub-graphs match it exactly or after relaxing one attribute. The student then revises its draft with up to $k$ notes. Notes are short, so they take up little context next to the image tokens.

Knowing when memory does not apply. Each draft gets a coverage score: the fraction of its findings and relations that have any matching key in the notebook. When coverage is low, or when a draft entity has no key, the system injects nothing about that entity rather than the nearest neighbor. This is the coverage-bottleneck lesson: fail toward silence, not toward noise.

Guarding against circularity. RadGraph is the key, the gate, and one of the evaluation metrics. A system optimized against it could game it. We therefore treat F1CheXbert, which uses an independent labeler, and a radiologist error audit as the primary checks of real improvement. RadGraph F1 and RadCliQ are reported as secondary.

Baselines

The naive reference point is each test student generating reports zero-shot with no memory. This is the "do nothing" floor. The teacher-only system sets the ceiling: GPT-4o writing reports directly on the same studies. Every method is scored on the gap it closes between the two.

The strongest existing competitor is FactMM-RAG-style retrieval. The test student receives the top reference reports, retrieved by a retriever trained on RadGraph-mined fact-equivalent pairs, and generates with them in context. This is the method most likely to make our notebook redundant. It already injects clinically matched content, and the 5.5% RadCliQ margin over random retrieval in Yu et al. shows how strong retrieval is here. The file reports only relative gains for FactMM-RAG (up to +6.5% F1CheXbert, +2% F1RadGraph), with no absolute score on Lacuna. We will therefore reproduce it on our split rather than quote a number. The only absolute RadGraph F1 in the file is 0.228 from the Style-Aware pipeline Style-Aware Radiology Report Generation. We will run that pipeline as a second strong reference, keeping in mind that its split may differ from ours.

Fairness rules are the same for every system. Every retrieval-based system uses the same MIMIC-CXR training split as its store. The teacher gets the same call budget, whether spent on writing notes or on writing reports to be retrieved. Context length and retrieved-item count $k$ are tuned on the same validation set with the same search budget. Beating the strongest competitor means our notebook, added on top of FactMM-RAG retrieval, raises F1CheXbert and lowers radiologist-counted clinically significant errors by more than seed-to-seed variation for students not used during construction. Our notebook alone matching FactMM-RAG would count as a partial result, not a win.

Two ablations isolate the key ingredients. The first keeps the identical notes but retrieves them by dense image-text embedding, as in the CAD Error Notebook. It also tests a MOTOR-style spatially grounded variant. This isolates whether graph keys matter, and directly confronts the DCoT-versus-MOTOR tension. The second removes the cross-architecture admission rule and admits notes verified only on the builder student. This tests whether that rule is what makes transfer work. For coverage gating, the comparison points are always-inject, embedding-distance thresholding, and MA-RAG-style disagreement among sampled drafts, with the sampling cost matched.

Why not just…?

Why not just fine-tune the small model on MIMIC-CXR or on teacher reports? Fine-tuning is the default way report generators are built, and on a single fixed student it may well beat any prompt-time memory. We take that threat seriously and will run it. The question here is different. One notebook should serve several students, including closed or newly released ones, without retraining. The CAD paper notes that frontier teachers are often API-only, and the memory papers document the cost and architecture lock-in of weight-based transfer. We also apply the notebook on top of a fine-tuned student. If fine-tuning absorbs all of the notebook's gain, that is a negative result worth reporting.

Why not just retrieve similar reference reports, as FactMM-RAG does? This is the most serious alternative. The experiment that settles it is the additive test, split by error type. If notes only help when retrieved reports are already informative, they are redundant. If they reduce negation and laterality errors on studies where retrieved reports are correct but the student still misreasons, they target the perception-reasoning gap that report retrieval does not. If the additive gain is null, we will drop the claim that corrections differ from content and report that.

Why not just prompt with a fixed checklist of common radiology errors? A static list of the most frequent graph-difference categories, written once by the teacher, costs nothing at retrieval time. It may capture much of the benefit if errors are concentrated in a few patterns. We include it as a baseline with the same token budget. Graph-keyed retrieval must beat it, especially on lower-frequency patterns, to justify the indexing machinery.

Why not just sample several drafts and retrieve when they disagree? MA-RAG shows disagreement is a better gap detector than confidence in text QA. It costs multiple generations, though, and can be fooled by false consensus. We compare it against graph coverage on the held-out-findings split described below, with matched compute. If disagreement flags harmful injections better, we adopt it as the gate.

Proposed Experiments

The main study uses MIMIC-CXR report generation. Builder students are Qwen2-VL-2B-Instruct and LLaVA-Med, and LLaVA-1.5-13B is held out as the transfer test student. All builder/test role assignments are rotated so every student serves as a held-out test student once. CheXpert evaluation tests transfer across datasets with the notebook frozen. Metrics are F1CheXbert, RadGraph F1, RadCliQ, and counts of clinically significant errors from radiologists on a stratified subset, following the protocol of Yu et al. Results are broken down by error category so it is visible whether the gains land on reasoning errors (negation, laterality, comparison) rather than on perception errors.

The decisive transfer check compares the held-out student's gain from the graph-keyed notebook with its gain from the embedding-keyed notebook, the no-admission-rule variant, and the static checklist. The core claim needs the graph-keyed version to win on the held-out student specifically, not just on builders.

The decisive coverage check removes all notes, and all training studies, involving a chosen set of rarer findings before building the notebook, then evaluates on test studies that contain those findings. Always-inject retrieval should hurt these studies relative to no memory, reproducing the structured-noise failure MOTOR describes. Graph-coverage gating should recover the no-memory score on them while keeping the gains elsewhere. We will compare how well each gating signal identifies the harmful injections, at matched compute.

A secondary check on Medical-Diff-VQA asks whether notes about comparison reasoning, such as "worsened" versus "unchanged," transfer to longitudinal question answering. MOTOR's reported 9.12% gain over base re-rankers there is the reference. The claim fails if the held-out student gains nothing beyond FactMM-RAG retrieval, or if improvements show up in RadGraph F1 but not in F1CheXbert or radiologist error counts.