Implicit Neural Representations for Joint Sparse-View CT Reconstruction

Jiayang Shi, Junyi Zhu, Daniel Pelt, Joost Batenburg, Matthew B. BlaschkoSubmission to ICLR 2024 · Rejected

TL;DR

Sparse-view Computed Tomography (CT) dramatically cuts radiation exposure but creates an ill-posed inverse problem that causes conventional implicit neural representations (INRs) to overfit to measurement noise. This paper presents INR-Bayes, a variational Expectation-Maximization framework that jointly reconstructs multiple CT scans by continuously coupling their neural representations through an adaptive, axis-aligned Gaussian prior. This shared prior prevents noise overfitting, captures inter-patient regularities, and consistently outperforms classical, meta-learning, and federated reconstruction baselines.

What Problem Does This Paper Address?

Acquiring fewer X-ray projection angles in CT imaging minimizes dangerous ionizing radiation, but undersampling makes 3D volume reconstruction mathematically ill-posed ($m \ll n$). While Coordinate-based Implicit Neural Representations (INRs)—such as MLPs mapping $(x, y, z)$ coordinates to tissue density—can represent continuous volume fields without massive external training datasets, fitting an INR to sparse, noisy projections leads to severe noise overfitting and artifacts.

Modern CT scanners routinely image similar anatomical structures across different patients or respiratory cycles. Existing multi-scan techniques either:

  1. Decompose scenes into static and transient components (e.g., NeRF-in-the-Wild), which assumes largely static geometry across scans.
  2. Use meta-learning (e.g., MAML or FedAvg) to find a shared weight initialization.

However, meta-initializations discard shared statistical regularities once local fine-tuning begins, causing models to rapidly overfit to individual measurement noise. The authors ask: How can we continuously share statistical regularities across multiple INR reconstructions to prevent overfitting and boost image quality?

Methods

The authors introduce INR-Bayes, framing joint CT reconstruction across $J$ objects as marginal likelihood maximization optimized via Variational Expectation-Maximization (V-EM):

  • Probabilistic Formulation: Each scan's INR weights $w_j$ are generated from a latent Gaussian prior $\mathcal{N}(\boldsymbol{\omega}, \boldsymbol{\sigma})$. Each network is approximated by a factorized Gaussian posterior $q(w_j) = \mathcal{N}(\boldsymbol{\mu}_j, \boldsymbol{\rho}_j)$ using the reparameterization trick with a softplus variance parameterization.
  • E-Step (Local Reconstruction): With latent prior parameters $\{\boldsymbol{\omega}, \boldsymbol{\sigma}\}$ fixed, each INR independently minimizes its CT projection loss (via Monte Carlo sampling) plus a Kullback–Leibler (KL) divergence penalty against the prior: $$\mathcal{L}(q(w_j)) \approx \|A_j F_{\widehat{w}_j}(C) - y_j\|_2^2 + \beta D_{\mathrm{KL}}(q(w_j) \,\|\, p(w_j \mid \boldsymbol{\omega}, \boldsymbol{\sigma}))$$
  • M-Step (Global Prior Aggregation): With local posteriors fixed, the latent parameters have an exact closed-form solution: $$\boldsymbol{\omega}^* = \frac{1}{J}\sum_{j=1}^J \boldsymbol{\mu}_j, \qquad \boldsymbol{\sigma}^* = \frac{1}{J}\sum_{j=1}^J \left(\boldsymbol{\rho}_j + (\boldsymbol{\mu}_j - \boldsymbol{\omega}^*)^2\right)$$ Here, $\boldsymbol{\omega}$ acts as the ensemble mean, while $\boldsymbol{\sigma}$ acts as an element-wise variance. Weight dimensions with high inter-subject variance receive weaker regularization, whereas shared anatomical weights are pulled strongly toward the consensus mean.

Datasets

The authors evaluate their approach on three medical benchmarks and one natural image dataset:

  • LungCT (Medical Segmentation Decathlon): 96 patient lung volumes (evaluated in intra- and inter-patient slice setups).
  • 4DCT: 10 temporal respiratory phases from a single patient's dynamic chest scan.
  • BrainCT (RSNA Hemorrhage Challenge): Axial head slices annotated for hemorrhage.
  • CelebA: 100 human face images (treated as 3-slice RGB volumes under parallel projection) to test cross-domain generalization.

No new datasets were introduced.

Key Insights and Ideas

  • Overfitting Immunity via Bayesian Priors: In standard iterative and INR reconstruction, training beyond 30,000 iterations degrades PSNR because the network memorizes projection noise. INR-Bayes holds a steady performance plateau through 60,000 iterations—the global Gaussian prior acts as an anchor against noise memorization.
  • Adaptive, Parameter-Wise Elasticity: Rather than enforcing rigid weight-sharing or simple centroid averaging (FedAvg), the learned variance $\boldsymbol{\sigma}$ automatically identifies which network layers must accommodate unique anatomy versus which represent invariant structural priors.
  • Failure of Static/Transient Splitting: Architectures adapted from NeRF-in-the-Wild (termed INRWild) collapse or produce blurry results in CT because anatomical variations between different slices/patients are structural shifts, not simple transient occlusions.
  • Zero-Shot Prior Transfer to Unseen Patients: An $\mathcal{N}(\boldsymbol{\omega}, \boldsymbol{\sigma})$ prior pre-trained on 10 patients guides the reconstruction of a completely unseen patient, outperforming single-scan INR and FedAvg without requiring ground-truth template registration (unlike Nerp).

Key Results

  • 2D CT Benchmarks: INR-Bayes achieves the highest PSNR and SSIM across all datasets:
  • Intra-patient LungCT (60 angles): 33.90 dB / 0.840 SSIM vs. 32.80 dB / 0.815 (SingleINR) and 33.26 dB / 0.825 (MAML).
  • Inter-patient LungCT (60 angles): 33.75 dB / 0.847 SSIM vs. 32.64 dB / 0.821 (SingleINR).
  • 4DCT (40 angles): 34.79 dB / 0.894 SSIM vs. 34.07 dB / 0.877 (SingleINR).
  • 3D Cone-Beam CT (CBCT): Reconstructing $128^3$ patient volumes from 40 angles yields 34.19 dB / 0.945 SSIM, outperforming FDK (19.40 dB), SIRT (24.91 dB), and MAML (33.67 dB).
  • Transfer to Unseen Patients: Reconstructing 5 new patients using the learned prior reaches 31.31 dB / 0.799 SSIM, whereas FedAvg drops to 29.89 dB / 0.754 (worse than naive SingleINR at 30.22 dB).

Limitations and Future Work

  • Computational Cost: Joint INR optimization remains compute-heavy. Reconstructing 10 nodes takes ~9.8 hours on a single RTX 3070 GPU (about a 10% overhead relative to SingleINR due to Gaussian weight sampling), far slower than analytical methods like FBP (seconds).
  • Clinical Utility Metrics: Performance is measured in mathematical fidelity metrics (PSNR, SSIM); formal clinical observer studies are necessary to evaluate whether diagnostic lesion detectability is preserved.
  • Architectural Scaling: SIREN backbones were utilized; integrating faster multiresolution hash grids (Instant-NGP) could drastically reduce training overhead while retaining the Bayesian formulation.

Summary updated

Abstract

Sparse-view Computed Tomography (CT) is favored over standard CT for its reduced ionizing radiation but poses challenges due to its inherently ill-posed nature arising from undersampled measurement data. Implicit Neural Representations (INRs) have emerged as a promising solution, demonstrating effectiveness in sparse-view CT reconstruction. Given that modern CT often scans similar subjects, we propose to improve reconstruction quality via joint reconstruction of multiple objects using INRs. This approach can potentially leverage both the strengths of INRs and the statistical regularities across multiple objects. While existing techniques of INR joint reconstruction focus on enhancing convergence rates through meta-initialization, they do not optimize for final reconstruction quality. To fill this gap, we introduce a novel INR-based Bayesian framework that incorporates latent variables to capture inter-object relationships. These latent variables act as a continuously updated reference during the optimization process, thereby enhancing the quality of individual reconstructions. We conduct extensive experiments to evaluate various aspects such as reconstruction quality, susceptibility to overfitting, and generalizability. Results demonstrate that our method sets a new standard in CT reconstruction performance. Our code will be released.
BibTeX
@article{shi2024implicit,
  author = {Jiayang Shi and Junyi Zhu and Daniel Pelt and Joost Batenburg and Matthew B. Blaschko},
  title = {{Implicit Neural Representations for Joint Sparse-View CT Reconstruction}},
  journal = {Accepted by TMLR},
  year = {2024},
  url = {https://openreview.net/forum?id=XCzuQI0oXR},
}
Figure