NerfAcc: Efficient Sampling Accelerates NeRFs

Ruilong Li, Hang Gao, Matthew Tancik, Angjoo Kanazawa

Abstract

Optimizing and rendering Neural Radiance Fields is computationally expensive due to the vast number of samples required by volume rendering. Recent works have included alternative sampling approaches to help accelerate their methods, however, they are often not the focus of the work. In this paper, we investigate and compare multiple sampling approaches and demonstrate that improved sampling is generally applicable across NeRF variants under an unified concept of transmittance estimator. To facilitate future experiments, we develop NerfAcc, a Python toolbox that provides flexible APIs for incorporating advanced sampling methods into NeRF related methods. We demonstrate its flexibility by showing that it can reduce the training time of several recent NeRF methods by 1.5x to 20x with minimal modifications to the existing codebase. Additionally, highly customized NeRFs, such as Instant-NGP, can be implemented in native PyTorch using NerfAcc.

Introduction

The primary bottleneck in Neural Radiance Fields (NeRFs) is the sheer number of samples required to render a single pixel. Standard volume rendering marches along a ray, querying a neural network at hundreds of points to determine color and density. Most of these points are "wasted" because they fall in empty space or behind opaque objects where they don't contribute to the final image.

While landmark papers like Instant-NGP and Plenoxels introduced clever ways to skip empty space, their implementations were often written in highly customized, "hard-coded" CUDA. This made it incredibly difficult for researchers to port those speedups to new architectures or dynamic scenes. The community lacked a modular, "plug-and-play" library that could handle the heavy lifting of efficient sampling across different NeRF variants. NerfAcc is an open-source PyTorch toolbox designed to bridge this gap, accelerating training by up to 20× through highly optimized sampling.

The NerfAcc Framework

NerfAcc is built on three core pillars that unify diverse acceleration techniques into a single framework:

  1. Unified Transmittance Estimator: The authors provide a mathematical proof that all common sampling strategies (uniform, spatial skipping, and PDF-based) can be viewed as different ways to estimate transmittance—the probability that light travels through a point without being absorbed. They show that the Cumulative Distribution Function (CDF) used for importance sampling is simply $1 - T(t)$, where $T$ is transmittance. This allows for "hybrid" sampling: for example, using an occupancy grid to skip empty space and a proposal network to concentrate samples on the surface simultaneously.
  2. Packed Tensors: To handle the fact that every ray might have a different number of "valid" samples after skipping empty space, NerfAcc uses a "packed" tensor format. This avoids the memory waste of padding all rays to a maximum length, which is a common bottleneck in standard PyTorch implementations.
  3. Sample as Interval: Instead of representing a sample as a single point $x$, NerfAcc treats it as an interval $[t_0, t_1]$. This design choice natively supports anti-aliasing techniques (like those in Mip-NeRF) and allows the sampling process to be detached from the differentiable computational graph, maximizing speed without sacrificing the ability to train the underlying radiance field.

Implementation and Efficiency

NerfAcc proves that researchers do not need to write thousands of lines of custom CUDA to match the performance of specialized implementations like Instant-NGP. By providing optimized kernels for the sampling logic, the rest of the NeRF pipeline can stay in native PyTorch.

A subtle but powerful trick used in the library is "No-Gradient Filtering." It evaluates transmittance with gradients disabled to filter out low-contribution samples before the expensive backward pass, leading to a ~10× speedup in the filtering stage itself.

Experimental Results

The authors evaluated NerfAcc across a wide range of standard benchmarks, including static scenes (NeRF-Synthetic, Tanks & Temples), dynamic scenes (D-NeRF, HyperNeRF), and camera optimization (BARF). The library delivers significant speedups across the board with minimal code changes (often <100 lines):

  • Vanilla NeRF: Achieved a 20× speedup on the Synthetic dataset, reducing training from over 1000 minutes to just 52 minutes, while actually improving PSNR (+0.5dB).
  • Dynamic NeRFs: Accelerated T-NeRF by 20× and TiNeuVox by 2.8×.
  • Camera Optimization: Sped up BARF by 4.5× and significantly improved camera pose reconstruction accuracy.
  • Instant-NGP: The authors replicated the performance of the official, pure-CUDA Instant-NGP implementation using pure Python and NerfAcc, even achieving a slight boost in image quality (+0.2dB).

Limitations and Future Work

While NerfAcc is highly flexible, voxel-based occupancy grids are naturally limited by a bounding box. While NerfAcc supports mapping functions for unbounded scenes, these estimators can still suffer from aliasing or resolution limits in such environments. Additionally, the performance of the occupancy grid versus proposal networks can vary depending on the scene; occupancy grids excel in synthetic scenes with lots of empty space, while proposal networks often perform better in dense, real-world unbounded scenes.

Future work may involve extending the unified framework to incorporate other priors, such as depth information from LiDAR or SfM, to further refine the transmittance estimator. The code is open-sourced at https://www.nerfacc.com.

Summary updated

Abstract

Optimizing and rendering Neural Radiance Fields is computationally expensive due to the vast number of samples required by volume rendering. Recent works have included alternative sampling approaches to help accelerate their methods, however, they are often not the focus of the work. In this paper, we investigate and compare multiple sampling approaches and demonstrate that improved sampling is generally applicable across NeRF variants under an unified concept of transmittance estimator. To facilitate future experiments, we develop NerfAcc, a Python toolbox that provides flexible APIs for incorporating advanced sampling methods into NeRF related methods. We demonstrate its flexibility by showing that it can reduce the training time of several recent NeRF methods by 1.5x to 20x with minimal modifications to the existing codebase. Additionally, highly customized NeRFs, such as Instant-NGP, can be implemented in native PyTorch using NerfAcc.
BibTeX
@inproceedings{li2023nerfacc,
  author = {Ruilong Li and Hang Gao and Matthew Tancik and Angjoo Kanazawa},
  title = {{NerfAcc: Efficient Sampling Accelerates NeRFs}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=qQq0s0KyRH},
}
Figure