NeAT: Neural Adaptive Tomography

Darius Rückert, Yuanhao Wang, Rui Li, Ramzi Idoughi, Wolfgang Heidrich

Abstract

In this paper, we present Neural Adaptive Tomography (NeAT), the first adaptive, hierarchical neural rendering pipeline for multi-view inverse rendering. Through a combination of neural features with an adaptive explicit representation, we achieve reconstruction times far superior to existing neural inverse rendering methods. The adaptive explicit representation improves efficiency by facilitating empty space culling and concentrating samples in complex regions, while the neural features act as a neural regularizer for the 3D reconstruction. The NeAT framework is designed specifically for the tomographic setting, which consists only of semi-transparent volumetric scenes instead of opaque objects. In this setting, NeAT outperforms the quality of existing optimization-based tomography solvers while being substantially faster.

Introduction

Computed Tomography (CT) is a cornerstone of non-destructive internal imaging, yet it typically requires extensive X-ray projections from numerous angles to produce clear results. In many practical scenarios—such as medical imaging aimed at reducing radiation dose or industrial scanning with limited rotation—data is often "sparse-view" or "limited-angle."

Traditional mathematical methods like Filtered Backprojection produce heavy artifacts under these constraints. While Neural Radiance Field (NeRF) approaches offer improved results by learning scene representations, they are computationally expensive and primarily optimized for opaque surfaces. Neural Adaptive Tomography (NeAT) addresses these challenges by providing a fast, accurate reconstruction method for semi-transparent volumetric scenes, even when initial calibration is noisy or sensors are uncalibrated.

Overview of the NeAT pipeline, showing the transition from sparse X-ray projections to a high-fidelity 3D reconstruction.

Methodology

The core of NeAT is a hybrid explicit-implicit representation designed for both computational efficiency and structural regularization.

Hybrid Representation

Instead of relying on a dense grid or a single large neural network, NeAT utilizes a sparse, adaptive octree. Leaf nodes in this octree contain a small grid of neural features. To determine the density at any specific point, the system performs trilinear interpolation on these features and processes the result through a compact, 3-layer Multi-Layer Perceptron (MLP).

Differentiable Rendering and Optimization

NeAT employs a differentiable volume renderer that simulates X-ray attenuation using the Beer-Lambert law. This allows the system to use backpropagation to update the 3D model based on 2D X-ray images. Crucially, NeAT performs joint optimization of the 3D geometry alongside "geometric" (camera poses) and "radiometric" (exposure, sensor bias, and response) parameters.

The NeAT architecture: a hybrid approach combining an adaptive octree with neural features and a differentiable volume renderer.

Dynamic Refinement

During training, the octree automatically splits in regions of high complexity and culls empty space. This adaptive behavior concentrates computational resources where they are most needed, significantly accelerating the reconstruction process compared to dense methods.

Key Results and Insights

NeAT demonstrates significant improvements over existing benchmarks in both speed and accuracy:

  • Efficiency: NeAT reaches high-quality convergence in approximately 45 minutes on an NVIDIA A100 GPU. For comparison, a standard NeRF implementation takes roughly 24 hours to reach similar or inferior results.
  • Reconstruction Quality: On synthetic datasets, NeAT outperformed traditional iterative solvers (such as SART and PSART-TV) and other neural methods (Kilo-NeRF) by up to 2dB in PSNR.
  • Robustness to Calibration Errors: In tests with intentionally noisy camera poses, NeAT recovered the correct geometry and improved reconstruction quality from 17dB to 35.96dB, actually surpassing the original factory calibration of the industrial scanner.
  • The "Missing Wedge" Solution: In limited-angle CT, the neural decoder acts as a powerful regularizer, filling in data gaps with physically plausible structures more effectively than traditional mathematical constraints.

Conclusion

NeAT represents a significant step forward in tomographic reconstruction, combining the speed of explicit representations with the regularizing power of neural networks. While it is currently optimized for semi-transparent scenes, its adaptive hierarchical approach offers a promising foundation for future work in opaque scene reconstruction and 4D dynamic imaging.

Summary updated

Abstract

In this paper, we present Neural Adaptive Tomography (NeAT), the first adaptive, hierarchical neural rendering pipeline for multi-view inverse rendering. Through a combination of neural features with an adaptive explicit representation, we achieve reconstruction times far superior to existing neural inverse rendering methods. The adaptive explicit representation improves efficiency by facilitating empty space culling and concentrating samples in complex regions, while the neural features act as a neural regularizer for the 3D reconstruction. The NeAT framework is designed specifically for the tomographic setting, which consists only of semi-transparent volumetric scenes instead of opaque objects. In this setting, NeAT outperforms the quality of existing optimization-based tomography solvers while being substantially faster.
BibTeX
@article{ruckert2022neat,
  author = {Darius Rückert and Yuanhao Wang and Rui Li and Ramzi Idoughi and Wolfgang Heidrich},
  title = {{NeAT: Neural Adaptive Tomography}},
  journal = {ACM Trans. Graphics},
  year = {2022},
  doi = {10.1145/3528223.3530121},
}
Figure