Generative AI models like Generative Adversarial Networks (GANs) and Denoising Diffusion Probabilistic Models (DDPMs) have fundamentally changed image synthesis. They can generate highly detailed, visually stunning images from random noise or sparse inputs. However, when applied to medical imaging—such as MRI reconstruction, CT synthesis, or digital pathology—these models encounter a critical safety barrier: anatomical hallucination.
A standard generative model is optimized to match a data distribution, meaning it prioritizes visual plausibility over mathematical or biological truth. In a consumer application, a realistic-looking texture is a success; in a clinical setting, a realistic-looking texture that fabricates a lesion or erases a blood vessel is a diagnostic hazard.
To understand why generative models hallucinate anatomy and how researchers are fixing it, we must look at the tension between perception and distortion, and the mechanisms being developed to anchor AI to physical reality.
The Root Cause: Unconstrained Prior Sampling
Why do state-of-the-art models hallucinate biological structures? The problem is most acute in inverse problems, such as reconstructing a full image from highly accelerated, undersampled MRI data or ultra-sparse CT views.
When data is sparse, the mathematical mapping from the raw sensor measurements back to the true image is under-determined—there are infinitely many images that could fit the sparse data. As Huang et al. demonstrate, at extreme data undersampling rates, generative adversarial frameworks prioritize visual realism over anatomical accuracy. Because the objective function rewards images that look like they belong to the training set, the model fills in the missing information by sampling from its learned statistical prior.
[Sparse/Noisy Input] ───> [Generative Model Prior] ───> [Visually Sharp Output]
│
(Fails standard metrics)
▼
Anatomical Hallucinations
(Fabricated/Deleted Structures)
If the prior is unconstrained, the model drifts. It generates plausible-looking tissue details, tumors, or structural boundaries that do not exist in the patient's actual anatomy. As Konz et al. point out, standard perceptual benchmarks like the Fréchet Inception Distance (FID) actually reward this behavior because they measure stylistic realism rather than clinical validity. An image can score exceptionally well on standard AI metrics while being completely invalid for medical diagnosis.
Strategy 1: Explicit Segmentation and Mask Guidance
The most direct way to prevent a model from fabricating anatomy is to restrict its creative freedom using explicit spatial boundaries. Instead of allowing a diffusion model to generate pixels from an unconstrained latent space, researchers inject structural roadmaps.
Konz et al. address this by introducing segmentation-guided diffusion models. By conditioning the generation process on precise anatomical masks, the model is forced to synthesize textures only within biologically valid boundaries.
This structural grounding is equally vital when models are used to simulate changes in patient anatomy, such as imagining a healthy version of a diseased organ. De Sousa Ribeiro et al. demonstrate that the accuracy of downstream clinical tools depends entirely on preventing the generative model from altering underlying physiological structures when removing disease features. If the model modifies healthy tissue while attempting to erase a pathology, the resulting images become useless for quantitative clinical analysis.
Strategy 2: Topological and Geometric Invariants
While segmentation masks provide coarse spatial boundaries, they often fail to protect fine-grained, interconnected biological structures like vascular networks. Pixels that belong to a blood vessel are highly correlated not just locally, but topologically; a single break in a generated line completely changes the anatomical meaning of a scan.
Standard pixel-wise losses (like L1 or L2 distance) and adversarial losses do not understand connectivity. If a model shifts a thin vessel by two pixels, a pixel loss penalizes it heavily, while an adversarial loss might accept it as a "realistic texture," even though the connectivity is broken.
True Vessel: ───────●───────
Hallucinated: ─── ● ─── (Broken topology, visually plausible)
To solve this, Dong et al. developed Topology Preserving Optimal Transport (TPOT). Instead of relying solely on visual style matching, their framework integrates persistence diagrams from algebraic topology. By mapping the mathematical "connectedness" of the image during the enhancement process, the model ensures that the complex network of retinal blood vessels remains structurally accurate, preserving the diagnostic map of the eye.
Strategy 3: Warping Templates via Deformation Fields
Another elegant way to bypass anatomical hallucination entirely is to change what the model is generating. If a model generates pixels from scratch, the risk of hallucination is high. But what if the model never generates a single pixel?
The authors of Diff-Def apply this intuition to the creation of anatomical atlases. Instead of synthesizing a 3D brain image from scratch, their latent diffusion model generates a 3D deformation field. This field acts as a coordinate warping map that stretches and bends a standard, physically perfect population template to match specific patient conditions (like age or disease state).
Because the model only modifies the spatial coordinates of an already valid biological template, it preserves the underlying topology, making it physically impossible for the system to invent non-existent tissue structures or introduce random artifacts.
Strategy 4: Test-Time Physical Anchoring
A final class of solutions focuses on the sampling process itself, ensuring that the model remains anchored to the physical sensor data collected by the scanner.
During iterative diffusion sampling, a model can easily drift into hallucination if it relies purely on its internal prior. To counteract this, Roosta et al. introduce bi-level guided diffusion models that inject refinement gradients directly into the sampling steps. These gradients act as a real-time correction mechanism, checking the intermediate generated image against the raw physical measurements and suppressing any generative features that lack physical evidence.
Similarly, Ozturkler et al. leverage Stein’s Unbiased Risk Estimator (SURE) to calculate reconstruction error on the fly during testing. By monitoring this mathematical error metric in real-time without needing a ground-truth image, the system can automatically optimize its regularization and terminate the generation process before the model begins overfitting to noise and fabricating fine-grained anatomical details.