TL;DR
NerfAcc is an open-source PyTorch toolbox that accelerates Neural Radiance Field (NeRF) training by up to 20× through highly optimized sampling. By unifying diverse acceleration techniques—like spatial skipping and proposal networks—into a single "transmittance estimator" framework, it allows researchers to add state-of-the-art speedups to almost any NeRF implementation with just a few lines of code.
What Problem Does This Paper Address?
The primary bottleneck in NeRFs is the sheer number of samples required to render a single pixel. Standard volume rendering marches along a ray, querying a neural network at hundreds of points to determine color and density. Most of these points are "wasted" because they fall in empty space or behind opaque objects where they don't contribute to the final image.
While landmark papers like Instant-NGP and Plenoxels introduced clever ways to skip empty space, their implementations were often written in highly customized, "hard-coded" CUDA. This made it incredibly difficult for researchers to port those speedups to new architectures or dynamic scenes. The community lacked a modular, "plug-and-play" library that could handle the heavy lifting of efficient sampling across different NeRF variants.
Methods
The authors introduce NerfAcc, a standalone Python library built on three core pillars:
- Unified Transmittance Estimator: The authors provide a mathematical proof that all common sampling strategies (uniform, spatial skipping, and PDF-based) can be viewed as different ways to estimate transmittance—the probability that light travels through a point without being absorbed. They show that the Cumulative Distribution Function (CDF) used for importance sampling is simply $1 - T(t)$, where $T$ is transmittance.
- Packed Tensors: To handle the fact that every ray might have a different number of "valid" samples after skipping empty space, NerfAcc uses a "packed" tensor format. This avoids the memory waste of padding all rays to a maximum length, which is a common bottleneck in standard PyTorch implementations.
- Sample as Interval: Instead of representing a sample as a single point $x$, NerfAcc treats it as an interval $[t_0, t_1]$. This design choice natively supports anti-aliasing techniques (like those in Mip-NeRF) and allows the sampling process to be detached from the differentiable computational graph, maximizing speed without sacrificing the ability to train the underlying radiance field.
Datasets
The authors evaluated NerfAcc across a wide range of standard benchmarks to prove its versatility:
- Static Scenes: NeRF-Synthetic, Tanks & Temples, and the unbounded 360 Dataset.
- Dynamic Scenes: D-NeRF and HyperNeRF.
- Camera Optimization: Evaluated using the BARF (Bundle-Adjusting NeRF) framework on synthetic data.
Key Insights and Ideas
- The "Aha" Moment on Transmittance: By realizing that transmittance is the only variable that truly matters for importance sampling, the authors collapsed several seemingly different algorithms into one framework. This allows for "hybrid" sampling: for example, using an occupancy grid to skip empty space and a proposal network to concentrate samples on the surface simultaneously.
- Python Simplicity, CUDA Speed: The library proves that you don't need to write thousands of lines of custom CUDA to match the performance of Instant-NGP. By providing optimized kernels for the sampling logic, the rest of the NeRF pipeline can stay in "native" PyTorch.
- No-Gradient Filtering: A subtle but powerful trick used in the library is "No Gradient Filtering." It evaluates transmittance with gradients disabled to filter out low-contribution samples before the expensive backward pass, leading to a ~10× speedup in the filtering stage itself.
Key Results
NerfAcc delivers significant speedups across the board with minimal code changes (often <100 lines):
- Vanilla NeRF: Achieved a 20× speedup on the Synthetic dataset, reducing training from over 1000 minutes to just 52 minutes, while actually improving PSNR (+0.5dB).
- Dynamic NeRFs: Accelerated T-NeRF by 20× and TiNeuVox by 2.8×.
- Camera Optimization: Sped up BARF by 4.5× and significantly improved camera pose reconstruction accuracy.
- Instant-NGP: The authors were able to replicate the performance of the official, pure-CUDA Instant-NGP implementation using pure Python and NerfAcc, even achieving a slight boost in image quality (+0.2dB).
Limitations and Future Work
While NerfAcc is highly flexible, the authors note a few areas for growth:
- Unbounded Scenes: Occupancy grids (voxel-based) are naturally limited by a bounding box. While NerfAcc supports mapping functions for unbounded scenes, voxel-based estimators can still suffer from aliasing or resolution limits in these environments.
- Hyperparameter Sensitivity: The performance of the occupancy grid vs. proposal networks can vary depending on the scene. For example, occupancy grids excel in synthetic scenes with lots of empty space, while proposal networks often perform better in dense, real-world unbounded scenes.
- Future Directions: The authors suggest that the unified framework could be extended to incorporate other priors, such as depth information from LiDAR or SfM, to further refine the transmittance estimator.
The code is open-sourced at https://www.nerfacc.com.
Summary updated