UMedNeRF: Uncertainty-aware Single View Volumetric Rendering for Medical Neural Radiance Fields

Jing Hu, Qinrui Fan, Shu Hu, Siwei Lyu, Xi Wu, Xin Wang

Abstract

In the field of clinical medicine, computed tomography (CT) is an effective medical imaging modality for the diagnosis of various pathologies. Compared with X-ray images, CT images can provide more information, including multi-planar slices and three-dimensional structures for clinical diagnosis. However, CT imaging requires patients to be exposed to large doses of ionizing radiation for a long time, which may cause irreversible physical harm. In this paper, we propose an Uncertainty-aware MedNeRF (UMedNeRF) network based on generated radiation fields. The network can learn a continuous representation of CT projections from 2D X-ray images by obtaining the internal structure and depth information and using adaptive loss weights to ensure the quality of the generated images. Our model is trained on publicly available knee and chest datasets, and we show the results of CT projection rendering with a single X-ray and compare our method with other methods based on generated radiation fields.

Introduction

Computed Tomography (CT) scans are indispensable for modern medical diagnosis, providing critical 3D volumetric data that standard 2D X-rays lack. However, the high dose of ionizing radiation associated with CT scans presents significant health risks. A major goal in medical imaging research is the reconstruction of 3D volumetric information from minimal 2D data, effectively reducing radiation exposure.

While Neural Radiance Fields (NeRF) have seen success in 3D scene reconstruction, medical applications face unique challenges. Medical images represent complex internal densities rather than solid surfaces. Previous models, such as MedNeRF, often struggled with a "distortion-perception" trade-off: optimizing for pixel-wise accuracy often resulted in blurry images, while optimizing for sharpness via Generative Adversarial Networks (GANs) could compromise anatomical fidelity. Traditionally, balancing these competing objectives required manual tuning of loss weights, a process that is both inefficient and prone to error.

Methodology

UMedNeRF addresses these challenges by introducing an uncertainty-aware multitask learning framework. The architecture builds upon a conditional radiance field that maps 2D positions and viewing angles to volume density ($\sigma$) and pixel values ($c$).

Overview of the UMedNeRF architecture, illustrating the uncertainty-aware training pipeline and the single-view rendering process used during testing.

The core innovation lies in its dual-loss strategy and automated weighting mechanism:

  1. Perceptual Loss ($L_r$): Utilizes a pre-trained VGG16 network to ensure generated images maintain high visual sharpness and realism.
  2. Accuracy Loss ($L_{MSE}$): Employs Mean Squared Error to ensure the generated projections remain anatomically faithful to the ground truth.
  3. Uncertainty-Aware Weighting: Instead of static weights, the model treats loss weights as learnable parameters tied to task-dependent uncertainty. This allows the network to adaptively balance sharpness and accuracy during training, finding an optimal equilibrium that manual tuning rarely achieves.

Experimental Results

The model was evaluated using Digitally Reconstructed Radiographs (DRRs) from public chest and knee CT datasets. UMedNeRF consistently outperformed baseline models, including MedNeRF, GRAF, and pixelNeRF.

Qualitative comparison of 3D volumetric rendering from a single X-ray view. UMedNeRF (Ours) demonstrates superior anatomical detail and clarity in bone structures compared to the baseline MedNeRF.

Quantitative analysis on the Knee dataset showed that UMedNeRF achieved a PSNR of 31.62 and an SSIM of 0.723, surpassing the original MedNeRF's scores of 30.93 and 0.670. Furthermore, lower Frechet Inception Distance (FID) and Kernel Inception Distance (KID) scores across both datasets indicate that UMedNeRF generates projections that are more indistinguishable from real clinical images. Qualitatively, the model reveals finer details in bone structures that appear washed out in previous generative models.

Conclusion

UMedNeRF represents a significant step toward low-radiation 3D medical imaging by successfully reconstructing volumetric projections from a single view. By automating the balance between image clarity and anatomical precision, the framework overcomes the limitations of manual weight tuning and improves generalization. Future work will focus on expanding the training datasets to more diverse populations, conducting clinical validation with radiologists to ensure medical accuracy, and optimizing computational efficiency for real-time clinical applications.

Summary updated

Abstract

In the field of clinical medicine, computed tomography (CT) is an effective medical imaging modality for the diagnosis of various pathologies. Compared with X-ray images, CT images can provide more information, including multi-planar slices and three-dimensional structures for clinical diagnosis. However, CT imaging requires patients to be exposed to large doses of ionizing radiation for a long time, which may cause irreversible physical harm. In this paper, we propose an Uncertainty-aware MedNeRF (UMedNeRF) network based on generated radiation fields. The network can learn a continuous representation of CT projections from 2D X-ray images by obtaining the internal structure and depth information and using adaptive loss weights to ensure the quality of the generated images. Our model is trained on publicly available knee and chest datasets, and we show the results of CT projection rendering with a single X-ray and compare our method with other methods based on generated radiation fields.
BibTeX
@inproceedings{hu2024umednerf,
  author = {Jing Hu and Qinrui Fan and Shu Hu and Siwei Lyu and Xi Wu and Xin Wang},
  title = {{UMedNeRF: Uncertainty-aware Single View Volumetric Rendering for Medical Neural Radiance Fields}},
  booktitle = {ISBI},
  year = {2024},
  doi = {10.1109/isbi56570.2024.10635864},
}
Figure