Neural Radiance Fields in Medical Imaging: Challenges and Next Steps

Xin Wang, Shu Hu, Heng Fan, Hongtu Zhu, Xin LiarXiv 2024 · DOI · arXiv

Introduction: The Translucency Problem

Traditional 3D medical reconstruction, such as CT or MRI, is essential for modern diagnostics but comes with significant drawbacks: high ionizing radiation doses, long scan times, and substantial equipment costs. While standard computer vision techniques for 3D reconstruction—specifically Neural Radiance Fields (NeRF)—have seen explosive growth, they typically focus on the surfaces of opaque objects.

Medical imaging presents a unique challenge. Unlike an RGB camera capturing light reflecting off a surface, an X-ray penetrates and attenuates through a volume. This fundamental difference requires a shift from surface-based rendering to volume-based attenuation modeling. This "translucency problem" means that medical NeRFs must account for how energy is absorbed throughout a volume rather than just how light hits a surface.

Comparison between standard RGB imaging and medical imaging systems, highlighting the shift from surface reflection to volumetric projection.

Methodological Adaptations for Clinical Use

To bridge the gap between computer vision and clinical utility, researchers are adapting NeRF architectures to handle the specific physical requirements of medical data. These adaptations focus on internal structural complexity, varying densities (representing blood flow or contrast agents), and the need for precise boundary definition between healthy and pathological tissues.

The general pipeline of medical NeRFs: mapping position and direction to volumetric color and density for multi-angle reconstruction.

Key methodological shifts include:

  • MedNeRF & UMedNeRF: These models utilize Generative Adversarial Networks (GANs) and uncertainty-aware learning to render 3D volumes from as little as a single 2D view.
  • Neural Attenuation Fields (NAF): This approach moves away from standard "color and density" outputs, instead modeling X-ray attenuation coefficients as a continuous function across 3D space.
  • Masked NeRF (mNeRF): By implementing 3D head masks, these models constrain extrapolations to prevent the "hallucination" of artifacts outside the actual bone structure.
  • Digitally Reconstructed Radiographs (DRR): A crucial preprocessing step where 3D CT data is used to simulate synthetic X-rays, allowing models to be trained without additional patient radiation exposure.

Datasets and Benchmarking

The transition to medical NeRFs relies on high-quality, specialized datasets to benchmark performance across different anatomical structures:

  • MedNeRF Dataset: Contains chest and knee CT scans specifically processed for single-view reconstruction.
  • NAF Dataset: A public collection of Cone Beam Computed Tomography (CBCT) scans covering the chest, jaw, foot, and abdomen.
  • PerX2CT: Based on the LIDC-IDRI collection, this uses DRR technology to generate synthetic X-ray/CT pairs.
  • DIF-Net: A large-scale knee CBCT collection (614 scans) designed for training intensity fields in sparse-view scenarios.

Key Insights: From Surface to Volume

The primary motivation for medical NeRFs is dose reduction. If a high-quality 3D model can be generated from 5-10 sparse X-rays instead of a full 360-degree CT scan, the biological risk to the patient is significantly lowered. However, several hurdles remain:

  • The "Inside-Out" Challenge: Medical NeRFs must account for energy absorption through a volume, necessitating a fundamental change in the rendering equation compared to standard RGB NeRFs.
  • Sparsity vs. Accuracy: Reconstructing thin structures like coronary arteries is difficult; current sampling methods often "miss" these vessels, leading to incomplete models.
  • The "Black Box" Hurdle: Clinical adoption requires interpretability. Unlike in entertainment, a "hallucinated" artifact in a lung scan could lead to a critical misdiagnosis. For NeRFs to be used in surgery, they must be reliable and explainable.

Results and Future Outlook

Current research shows promising results in single-view reconstruction and artifact reduction. Models like mNeRF have successfully eliminated "floaters" and geometric artifacts that previously plagued unconstrained skull reconstructions. Quantitatively, NeRF-based methods are beginning to rival traditional filtered back-projection (FBP) while using significantly less input data, as measured by Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM).

Looking forward, the field is moving toward solving imminent challenges:

  1. Real-Time Performance: Exploring 3D Gaussian Splatting to overcome slow training times, which is critical for emergency medicine.
  2. Boundary Precision: Integrating edge-detection or hybrid models to resolve "fuzzy" edges often produced by continuous radiance fields.
  3. Generalization: Developing Foundation Models that can work across different organs and modalities, as current models often fail when applied to anatomy outside their training set.
  4. Agentic Scanning: Using Reinforcement Learning to allow models to "decide" which views are most informative, potentially reducing scan times even further.

Summary updated

Abstract

Neural Radiance Fields (NeRF), as a pioneering technique in computer vision, offer great potential to revolutionize medical imaging by synthesizing three-dimensional representations from the projected two-dimensional image data. However, they face unique challenges when applied to medical applications. This paper presents a comprehensive examination of applications of NeRFs in medical imaging, highlighting four imminent challenges, including fundamental imaging principles, inner structure requirement, object boundary definition, and color density significance. We discuss current methods on different organs and discuss related limitations. We also review several datasets and evaluation metrics and propose several promising directions for future research.
BibTeX
@misc{wang2024neural,
  author = {Xin Wang and Shu Hu and Heng Fan and Hongtu Zhu and Xin Li},
  title = {{Neural Radiance Fields in Medical Imaging: Challenges and Next Steps}},
  year = {2024},
  eprint = {2402.17797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.17797},
}
Figure