TL;DR
NeAT (Neural Adaptive Tomography) is a hybrid explicit-implicit neural representation designed to solve the "ill-posed" problems of X-ray tomography, such as having too few views or limited angular coverage. By combining an adaptive octree structure with neural features and a differentiable renderer, it achieves state-of-the-art reconstruction quality in roughly 45 minutes—a massive leap over the 24 hours required by standard neural radiance fields (NeRF).
What Problem Does This Paper Address?
Computed Tomography (CT) is essential for seeing inside objects, but it typically requires a large number of X-ray projections from many angles. In many real-world scenarios—like medical imaging to reduce radiation dose or industrial scanning where an object can't be fully rotated—we end up with "sparse-view" or "limited-angle" data.
Traditional mathematical methods (like Filtered Backprojection) produce heavy artifacts under these conditions. Recent "Neural Radiance Field" (NeRF) approaches offer better results but are notoriously slow and optimized for opaque surfaces. NeAT tackles the challenge of reconstructing semi-transparent, volumetric scenes quickly and accurately, even when the initial camera calibration is noisy or the sensor is uncalibrated.
Methods
The core of NeAT is its hybrid representation:
- Adaptive Octree: Instead of a dense grid or a single massive neural network, NeAT uses a sparse octree. Leaf nodes contain a small grid of "neural features."
- Neural Decoding: To find the density at any point, the system performs trilinear interpolation on these features and passes the result through a tiny, 3-layer Multi-Layer Perceptron (MLP).
- Differentiable Volume Rendering: The system simulates X-rays passing through the volume using the Beer-Lambert law. Because the entire pipeline is differentiable, the system can use backpropagation to update the 3D model based on 2D X-ray images.
- Self-Calibration: NeAT doesn't just optimize the 3D object; it also optimizes the "geometric" (camera poses) and "radiometric" (exposure, sensor bias) parameters simultaneously.
- Dynamic Refinement: During training, the octree automatically splits in complex areas and culls empty space, focusing computational power where it's needed most.
Datasets
The authors evaluated NeAT on two types of data:
- Synthetic Data: Volumes of a "Flower" and an "Orange" were used to generate ground-truth projections for controlled testing.
- Real Data: High-resolution scans of a ceramic coral, a pepper, a pomegranate, and a wind-up teapot, captured using a Nikon industrial CT scanner. These provided a test for the system's ability to handle real-world noise and calibration errors.
Key Insights and Ideas
- Explicit Beats Implicit (Sometimes): The authors found that for tomography, storing features explicitly in an octree is actually superior to "hybrid" models like ACORN. Purely implicit coordinate networks tend to introduce a "blurring" bias that is particularly damaging in limited-angle tomography.
- The "Missing Wedge" Solution: In limited-angle CT, there is a literal gap in the data (the missing wedge). NeAT’s neural decoder acts as a powerful regularizer, "filling in" these gaps with physically plausible structures better than traditional mathematical constraints.
- Custom Kernels for Speed: By implementing a custom "indirect 3D grid sample kernel," the authors made their feature retrieval three times faster than standard deep learning operators, contributing to the overall speedup.
Key Results
- Speed: NeAT reaches high-quality convergence in about 45 minutes on an NVIDIA A100 GPU. For comparison, a standard NeRF implementation takes roughly 24 hours to reach similar or inferior results.
- Accuracy: On synthetic datasets, NeAT outperformed traditional iterative solvers (like SART and PSART-TV) and other neural methods (Kilo-NeRF) by up to 2dB in PSNR.
- Robustness: In a "geometric self-calibration" test, the system was given intentionally noisy camera poses that degraded the baseline PSNR to 17dB. NeAT was able to recover the correct geometry and boost the final reconstruction to 35.96dB, actually outperforming the scanner's original factory calibration.
Limitations and Future Work
- Physical Accuracy vs. Plausibility: While NeAT produces "physically plausible" completions in limited-angle scenarios, the authors note that these completions might not always match the true underlying geometry (e.g., a topology change in the teapot handle). This is a common hurdle for any "generative" approach in scientific imaging.
- Order Independence: The current model assumes the scene is semi-transparent and order-independent. It cannot currently handle opaque objects where "front-to-back" compositing is required (like standard NeRF).
- Future Directions: The authors suggest that this adaptive hierarchical approach could be extended to more complex scenes with opaque surfaces or used to reconstruct dynamic, moving objects in 4D.
Summary updated