Systems that write radiology reports from brain MRI scans make a specific and dangerous kind of mistake. They describe findings that are not there, or put a real finding on the wrong side of the brain. A "left temporal lesion" that is actually right-sided can send a surgeon or a deep brain stimulation plan to the wrong hemisphere. The brain offers a check that chest X-rays and natural images do not: it is roughly mirror-symmetric, and most disease starts on one side. Neuroradiologists exploit this all the time by comparing a suspicious structure with its partner on the other side. We propose to build that comparison into an automatic checker. The checker breaks a generated report into short lateralized claims (side, structure, finding). It then accepts or rejects each claim by asking whether the scan's left–right difference at that location falls outside the range seen in healthy brains.
The core claim is that a patient's own opposite hemisphere, calibrated against normal healthy asymmetry, catches two errors that current grounding methods miss: reporting a finding on the wrong side, and describing a finding that does not exist, including findings invented because an upstream segmentation network produced a false positive. It catches both without labeled hallucination data and without access to the generator's internals. If this holds, the model that writes the report and the evidence that verifies it become separate, so any generator can be wrapped, including closed API models.
Where this sits
Two brain MRI report systems are closest to this proposal, and both motivate it. AutoRG-Brain segments abnormal regions with a 3D U-Net and then conditions a text decoder on those masks. It reports a RadGraph score of 41.74% versus 16.93% for MedFlamingo on an in-house cohort. Because the decoder trusts the masks completely, any false-positive mask becomes a fluently described false finding. Nothing in the system compares the region with its opposite side. The second system, PIRTA, avoids generating directly from voxels. It retrieves the five most similar past scans by whole-brain embedding and lets an LLM synthesize a report from their expert reports, reaching 94.04% Acc@1 for stroke territory on internal data. Whole-brain similarity does not distinguish a left stroke from a right one, so a retrieved neighbor can import the wrong side. Our checker sits downstream of either system and does not replace them.
Three hallucination-mitigation methods supply techniques or constrain the design. MEDA edits a vision-language model's middle-layer activations using steering directions built from diagnostic principles, gaining 10.6% accuracy on IU-Xray. It needs white-box access and was tested only on 2D images. Cite-While-You-Generate attributes each generated sentence to source spans or image patches via pooled attention, reaching 76.33 Macro-F1 on text attribution. It attributes to single regions, whereas a claim like "left hippocampal atrophy" is only supported by a difference between two regions. MedCite shows that checking each atomic statement separately beats single-pass retrieval. We reuse its decompose-then-verify structure, but the evidence is image-derived rather than literature-derived.
On the evaluation side, Med-HallMark names "Attribute" errors, which include laterality, as a hallucination tier. Its benchmarks are 2D chest and abdominal images, so laterality failure rates on 3D brain MRI remain unmeasured. We fill that hole.
On the neuroanatomy side, Reuter et al. show that averaging left and right volumes halves the effect size of unilateral pathology. Zhu et al. mirror the right hippocampus onto the left and compare surface points along the surface normal. On OASIS3 (208 AD patients, 717 controls), whole-structure volume asymmetry does not separate AD from controls (p = 0.80), while point-wise shape asymmetry does (Hotelling p ≈ 0; 177 of 512 points). This result both supplies our local measurement method and warns that a volume-only checker will fail on subtle atrophy claims.
Research Program
The checker has three parts.
The first is a claim parser. An LLM extracts tuples of the form (side ∈ {left, right, bilateral, midline}, structure, finding type, rough severity) from report text. It is validated against hand-parsed reports from RadGenome-Brain MRI.
The second is a measurement step. Each scan is registered to its own mid-sagittal reflection, so every voxel is paired with its contralateral homologue. From that pairing we compute two kinds of asymmetry. For focal findings we use regional intensity asymmetry within atlas parcels. For atrophy claims we use shape asymmetry with point correspondences, in the ShapeWorks style of Zhu et al..
The third is a normative calibration. Healthy brains are not perfectly symmetric, and scanners and head coils add their own left–right bias. For a structure $s$ we therefore compute an asymmetry index $a_s = (m_{s,L} - m_{s,R}) / \tfrac{1}{2}(m_{s,L} + m_{s,R})$, where $m_{s,L}$ and $m_{s,R}$ are a measurement (volume, mean intensity, or point-wise normal displacement) on the left and mirrored right side. We convert it to a patient-specific score
$$z_s = \frac{a_s - \mu_s(\text{age}, \text{sex}, \text{site})}{\sigma_s(\text{age}, \text{sex}, \text{site})},$$
where $\mu_s$ and $\sigma_s$ are the mean and spread of $a_s$ in healthy UK Biobank subjects, modeled as smooth functions of the covariates. The covariates follow the age, sex and intracranial-volume corrections standard in Muñoz-Ruiz et al.. A claim "left $s$ abnormal" is supported when $z_s$ exceeds a threshold in the direction the finding type predicts. Bilateral claims require both sides to deviate from absolute norms with small $|z_s|$. Unilateral claims with opposite-signed $z_s$ are flagged as likely laterality inversions.
The thresholds come from a selective-prediction view in the spirit of Mohri et al.. For each claim the system can accept it, flip its side, or abstain and flag it for a radiologist, with error costs weighted by Med-HallMark severity.
Two cases fall outside what a contralateral reference can check: truly bilateral symmetric disease and midline structures. For these the checker falls back to absolute normative z-scores and says so in its output. We will measure how often this fallback is needed rather than hide it.
Baselines
The naive reference point is the unfiltered generator. We take report outputs from an AutoRG-Brain-style mask-conditioned generator and a PIRTA-style retrieval generator, both re-implemented and trained on RadGenome-Brain MRI, plus a general multimodal model given multi-slice input. Nothing is filtered. This gives raw rates of laterality inversion and nonexistent findings. No reported result was found on Lacuna for laterality error rates of any brain MRI report generator, so these numbers are themselves a contribution.
The strongest existing system-level reference is AutoRG-Brain at 41.74% RadGraph on SSPH. SSPH is in-house, and no reported result was found on Lacuna for any method on RadGenome-Brain MRI. We will therefore report RadGraph for our re-implementation on RadGenome-Brain and treat SSPH only as context.
The strongest learned detector is MediHallDetector, reported at 93.88% multi-round consistency, which measures self-agreement rather than accuracy. We fine-tune it on the same brain-MRI training split, using the same synthetic perturbations our checker is evaluated on. That gives it an advantage our label-free checker does not have. Beating it means higher claim-level detection AUROC on the held-out real-generator outputs, where synthetic training perturbations may not match real errors.
The strongest attribution baseline is Cite-While-You-Generate applied in its caption mode, which reports 76.33 text Macro-F1 and 34.87% multimodal joint exact match on its own tasks. Here a claim counts as supported if its attention evidence lands on the stated side. It gets the same backbone and the same tuning budget for its top-$k$ and threshold settings as our checker's threshold.
Two ablations isolate the key ingredient. The absolute-norm ablation drops the contralateral comparison and uses only per-side deviation from healthy norms, which is the brain-age-style approach of Peng et al. applied regionally. The uncalibrated ablation uses the raw asymmetry index $a_s$ without the healthy-asymmetry model. If the absolute-norm ablation matches the full checker, the contralateral idea adds nothing. If the uncalibrated ablation matches, the normative modeling is unnecessary.
Why not just…?
Why not just read the side off the segmentation mask? For a focal lesion, taking the centroid of the predicted mask and checking it against the stated side is cheap, and it will likely fix most laterality inversions. This is a real threat in the focal regime, and we run it as a baseline. It has one blind spot: it verifies where a finding is, not whether it exists. If the mask itself is a false positive, the case AutoRG-Brain itself flags, the centroid check confirms the hallucination. It also says nothing about atrophy claims, which have no mask. We expect to tie it on inversion detection and beat it on false-finding rejection. If we only tie it everywhere, the focal-lesion contribution reduces to the benchmark.
Why not just test flip consistency, mirroring the scan and checking that the report flips? This needs only black-box access to the generator, and it directly targets inversions caused by model bias toward one side. It tests whether the generator behaves the same under mirroring, not whether the report is true. A generator that hallucinates a lesion from a scanner artifact will usually move that lesion with the flip and pass. It also doubles generation cost. We run it with the same generators. We report its AUROC separately on inversion errors and on nonexistent-finding errors, and expect the split to show where it works and where it fails.
Why not just fine-tune a hallucination detector? A detector like MediHallDetector can learn whatever signal predicts errors, possibly including asymmetry. But it needs labeled brain-MRI hallucinations, which do not exist at scale. The file reports its consistency, not its accuracy on 3D laterality. Our comparison deliberately gives the fine-tuned detector the synthetic training labels. A gap on real generator outputs would show that the label-free contralateral signal generalizes where learned detection overfits to synthetic perturbations. No gap would mean the internal-control idea is unnecessary given labels.
Why not just steer the generator, as MEDA does? Steering changes the generator and requires access to its internals, while our checker works on any output. Steering with a symmetry principle is a natural follow-up, but it cannot be compared fairly on closed models. We therefore leave it out of the main claims.
Proposed Experiments
Experiment 1: focal lesions. We use RadGenome-Brain MRI (3,408 3D scans, six modalities, five disease types, voxel masks with human reports), split by patient. From the ground-truth reports we build a lateralization hallucination set with four perturbation types: sides swapped, unilateral findings rewritten as bilateral, findings inserted into healthy parcels, and true claims left untouched. We also include real errors from the three generators, labeled by two radiologists on a stratified subset of a few hundred reports. The primary metrics are claim-level detection AUROC and the trade-off between how many claims the checker keeps and how many errors remain, overall and by error type. We also report a severity-weighted error score in the style of the MediHall score. For stroke specifically, the ISLES 2022 data used by PIRTA gives an external check of territory-plus-side accuracy before and after checking.
Experiment 2: subtle atrophy. Here no mask exists. On OASIS3 and ADNI we generate templated atrophy claims ("left greater than right hippocampal atrophy," "symmetric atrophy," "no atrophy"). Ground truth comes from radiologist visual ratings on a held-out subset, not from our own shape pipeline, so the evaluation is not circular. Comparing volume-based and point-wise shape asymmetry in the checker tests whether the gap reported by Zhu et al. carries over to claim verification. We additionally check whether supported left-lateralized claims are enriched in APOE ε4 carriers, as the left-hemisphere effect in Li et al. predicts.
Decisive checks. Three results would decide the proposal. First, on healthy controls across scanners and sites, the checker's false-flag rate for unilateral claims must stay near its nominal level. Scanner-induced asymmetry is the most likely way for the method to fail, and if site calibration cannot control it the method fails. Second, on inserted-finding errors, the full checker must beat both the mask-centroid and flip-consistency baselines. That is the case where the internal control is supposed to matter. Third, the full checker must beat the absolute-norm ablation on real generator errors. Otherwise the opposite hemisphere is not doing the work we claim.
References
- AutoRG-Brain: Grounded Report Generation for Brain MRI
- Improving Factuality of 3D Brain MRI Report Generation with Paired Image-domain Retrieval and Text-domain Augmentation
- MEDA: Medical-Oriented Activation Editing for Hallucination Mitigation in Medical Large Vision-Language Model
- Cite-While-You-Generate: Training-Free Evidence Attribution for Multimodal Clinical Summarization
- MedCite: Can Language Models Generate Verifiable Text for Medicine?
- Detecting and Evaluating Medical Hallucinations in Large Vision Language Models
- Abtract: Shape Analysis in Human Brain MRI
- Quantifying Hippocampal Shape Asymmetry in Alzheimer's Disease Using Optimal Shape Correspondences
- Structural MRI in Frontotemporal Dementia: Comparisons between Hippocampal Volumetry, Tensor-Based Morphometry and Voxel-Based Morphometry
- Learning to Reject with a Fixed Predictor: Application to Decontextualization
- Accurate brain age prediction with lightweight deep neural networks
- Influence of APOE Genotype on Hippocampal Atrophy over Time - An N=1925 Surface-Based ADNI Study