Abstract
MedNeRF is a novel deep learning framework that reconstructs 3D-aware CT projections from as little as a single 2D X-ray. By adapting Generative Radiance Fields (GRAF) with a specialized self-supervised discriminator, the model learns to disentangle anatomical shape from X-ray attenuation, offering a path toward high-quality 3D medical imaging with significantly reduced radiation exposure.
Introduction
Computed Tomography (CT) is essential for 3D medical diagnosis, but it comes at a cost: patients are exposed to high doses of ionizing radiation, and the scanning process can be time-consuming. While standard X-rays are safer and faster, they only provide 2D projections, losing the depth information critical for complex diagnoses.
The authors aim to bridge this gap using Neural Radiance Fields (NeRF). However, applying standard NeRF to medical imaging is difficult because X-rays represent the accumulation of attenuation through a volume, medical datasets are often small, and NeRF typically requires many views of a static scene, which isn't always feasible in a clinical setting.
Methodology
MedNeRF builds upon Generative Radiance Fields (GRAF), a variant of NeRF that operates within a Generative Adversarial Network (GAN).
Key technical innovations include:
- Disentangled Latent Representations: The model uses separate latent codes for shape ($z_s$) and appearance/attenuation ($z_a$). This allows the network to learn a continuous 3D representation of anatomy across different patients.
- Self-Supervised Discriminator: To handle small datasets, the authors replaced the standard discriminator with one that includes an auto-encoding pretext task. It uses two decoders to reconstruct feature maps, providing a much richer gradient signal to the generator than a simple "real vs. fake" binary.
- Data Augmentation Optimized for GAN (DAG): The model employs multiple discriminator heads with shared weights to apply differentiable augmentations, improving stability and generalization without misleading the generator.
- Single-View Fine-Tuning: For reconstruction, the model takes a single X-ray and "fits" the generator by slightly fine-tuning the latent codes and weights to match that specific patient's anatomy, allowing it to then render the volume from any other angle.
Experimental Results
The researchers evaluated the model using Digitally Reconstructed Radiographs (DRRs)—simulated X-rays generated from existing CT volumes of the chest and knee.
MedNeRF outperformed baselines like pixelNeRF and standard GRAF across multiple metrics. For the Chest dataset, it achieved a Frechet Inception Distance (FID) of 60.26, significantly lower than GRAF's 68.25 and pixelNeRF's 112.96. In single-view reconstruction tasks, the model achieved a PSNR of 30.17 dB for knees and 28.54 dB for chests. Qualitatively, the model successfully disentangles bone from soft tissue, allowing clinicians to apply contrast transformations to improve visibility of dense structures.
Discussion and Conclusion
The core innovation of MedNeRF lies in modeling attenuation as radiance, allowing the volume rendering equation to naturally represent how X-rays pass through bone and tissue. By forcing the discriminator to understand features via auto-encoding decoders, the generator receives the refined features needed to estimate accurate volumetric depth.
While the model performs well on small datasets, clinical validation requires larger samples. Current resolution is limited to 128x128, but MedNeRF represents a powerful tool for applications like osseous trauma evaluation, surgical planning, or reducing radiation in pediatric cases.
Summary updated