Introduction
In modern clinical practice, a comprehensive diagnostic picture often requires multiple imaging modalities: MRI for soft tissue contrast, CT for bone density and dose planning, and PET for metabolic activity. However, acquiring this full suite is frequently impractical due to high costs, time constraints, and the risks of ionizing radiation—particularly in pediatric care. Furthermore, contraindications like metal implants can preclude certain patients from receiving necessary scans.
Cross-modality synthesis offers a solution by "predicting" one modality from another (e.g., generating a pseudo-CT from an MRI). While traditional handcrafted methods struggled with the non-linear complexity of this "ill-posed" mapping, deep learning has revolutionized the field. This review tracks the rapid evolution of these technologies from 2018 to 2023.
The Evolution of Synthesis Architectures
The field has progressed through several distinct architectural "waves," moving from local pixel-matching to models that understand global anatomy.
- U-Nets: The foundational workhorse of medical imaging, prized for their symmetric encoder-decoder structure and skip connections that preserve spatial detail.
- Generative Adversarial Networks (GANs): The dominant paradigm for much of the last five years. This includes cGANs (for paired data) and CycleGANs (for unpaired data), which use a "discriminator" to force the "generator" to produce images that look indistinguishable from real scans.
- Vision Transformers (ViTs): A more recent shift aimed at overcoming the "limited receptive field" of CNNs. By using self-attention mechanisms, Transformers can capture global context—understanding how a structure in one part of the brain relates to another far away.
- Denoising Diffusion Probabilistic Models (DDPMs): The current "state-of-the-art" frontier. These models learn to reverse a gradual noise process to generate high-fidelity images, often producing more stable results than GANs, though at a higher computational cost.
Review Scope and Methodology
This analysis categorizes the literature not only by architecture but also by the specific synthesis task and the metrics used to evaluate success.
The review identifies a shift in Loss Functions, moving from simple pixel-wise errors (L1/L2) to complex "perceptual losses" and "structural similarity" (SSIM) metrics that better align with how human radiologists perceive image quality.
Key Insights and Clinical Performance
Synthesis performance is not uniform across the body. Brain synthesis is relatively mature because the skull provides a rigid frame. In contrast, pelvic and abdominal synthesis is much harder due to organ deformation, respiratory motion, and gas pockets.
- Pseudo-CT Accuracy: For MRI-to-CT synthesis, Mean Absolute Error (MAE) typically falls between 50 and 90 Hounsfield Units (HU). Transformers and Diffusion models are currently setting the high bar for accuracy in brain imaging.
- Synthetic PET: Deep learning has proven exceptionally good at "full-dose" PET synthesis from "low-dose" scans, effectively denoising the image while preserving metabolic "hotspots" crucial for cancer diagnosis.
- Downstream Utility: The real test is clinical performance—e.g., can a synthesized "pseudo-CT" be used for accurate radiation dose calculation in cancer treatment? High mathematical scores (PSNR) do not always guarantee clinical safety.
Challenges and Future Directions
Despite rapid progress, several hurdles remain for clinical adoption:
- The "Black Box" & Uncertainty: Models can occasionally "hallucinate" features. Future research must focus on Uncertainty Estimation—the model should be able to alert a clinician when it is unsure about a specific synthesized area.
- Computational Efficiency: Diffusion models are slow. Future work needs to focus on making these models efficient enough for real-time clinical use.
- Generalizability: A model trained on a Siemens scanner might fail on a GE scanner. Developing "foundation models" that are robust across different hardware and patient populations is the next big challenge.
- Clinical Validation: Most papers stop at mathematical metrics. The authors argue for more multicenter clinical trials to prove these synthetic images are safe for actual diagnostic use.
Reference: Dayarathna, S., et al. (2024). Deep learning based synthesis of MRI, CT and PET: Review and analysis. Medical Image Analysis, 92, 103046.
Summary updated