SpikingNeRF: Making Bio-inspired Neural Networks See through the Real World

Xingting Yao, Qinghao Hu, Fei Zhou, Tielong Liu, Zitao Mo, Zeyu Zhu, Zhengyang Zhuge, Jian Cheng

TL;DR

SpikingNeRF bridges the gap between energy-efficient neuromorphic computing and high-quality 3D scene reconstruction by mapping neural radiance rays directly to the temporal dimension of Spiking Neural Networks (SNNs). By replacing power-hungry multiplications with addition-only spike operations and introducing a hardware-friendly data condensing strategy, it slashes rendering energy consumption by over 70% with minimal impact on image quality.

What Problem Does This Paper Address?

Neural Radiance Fields (NeRFs) have revolutionized 3D scene generation and view synthesis, but they come with a massive computational tax. Querying dense Artificial Neural Networks (ANNs) millions of times per frame consumes significant energy, making real-time deployment on edge devices or neuromorphic hardware highly impractical.

Spiking Neural Networks (SNNs) offer a bio-inspired, energy-efficient alternative by using multiplication-free, spike-driven addition operations. However, adapting SNNs to 3D reconstruction tasks has proven difficult. Standard SNN data encoding methods (like Poisson or direct duplication) do not naturally match the geometric structure of 3D rays. Furthermore, common NeRF efficiency tricks—like masking out empty voxels—create irregular data shapes that break parallel processing on standard GPUs and neuromorphic hardware.

Methods

The authors introduce SpikingNeRF, a framework that combines explicit voxel grids with an implicit spiking Multi-Layer Perceptron (sMLP) to achieve fast, low-power rendering.

Figure 3: Overview of the proposed SpikingNeRF. (a) The rendering process of SpikingNeRF. The whole 3D volumetric parameters are stored in the voxel grids. The irrelevant or unimportant samples are masked before the sMLP querying. The expected scenes are rendered with the volumetric information yielded by the sMLP. (b) Alignment between the temporal dimension and the ray. The sMLP queries each sampled point step-by-step to yield the volumetric information. (c) Proposed temporal padding (left) and temporal condensing-and-padding (right) methods. For simplification, the channel length of the volumetric parameters is set to 1.

Its core innovations include:

  • Time-Ray Alignment (TRA): Instead of artificially duplicating data to fill a temporal dimension, TRA elegantly maps the spatial points sampled along a camera ray to sequential time steps in the sMLP. The geometric continuity of the ray translates directly into the temporal continuity of the SNN.
  • Temporal Condensing-and-Padding (TCP): To optimize rendering, empty space is masked out, leaving rays with highly irregular numbers of valid points. A naive padding approach leaves sparse zeros that cause hardware inefficiencies. TCP discards these masked points entirely, shifts the remaining valid points tightly to the left, and pads the remaining tensor space with zeros. This creates a dense, regular data structure optimized for parallel processing on both GPUs and neuromorphic accelerators.

Datasets

The authors evaluated SpikingNeRF across four standard 3D reconstruction benchmarks:

  • Synthetic-NeRF: Standard synthetic objects with complex geometries and materials.
  • Synthetic-NSVF: Synthetic scenes used to test sparse voxel methods.
  • BlendedMVS: A large-scale dataset for generalized multi-view stereo networks.
  • Tanks&Temples: Real-world large-scale scenes used to benchmark reconstruction fidelity.

Key Insights and Ideas

  • Geometry as Time: The most elegant realization of the paper is that a ray's spatial progression is fundamentally sequential. By treating space (points along a ray) as time (steps in an SNN), the authors bypass the efficiency losses of conventional SNN encoding schemes.
  • The Sparsity Paradox: In hardware acceleration, mathematical sparsity doesn't always equal efficiency. Voxel masking saves computation but introduces structural irregularity that harms GPU parallelism. TCP resolves this by packing valid data tightly, proving that structural regularity is often more important for hardware efficiency than pure mathematical sparsity.

Key Results

  • Massive Energy Savings: SpikingNeRF-D (built on the DVGO framework) achieved an average 70.79% energy reduction compared to its ANN baseline.
  • High Fidelity: The massive drop in power consumption comes with a negligible quality trade-off—only a 0.53 dB average drop in PSNR across the board.
Figure 1: Comparisons of our SpikingNeRF with other NeRF-based works in synthesis quality and model rendering energy. Different colors represent different works, and our SpikingNeRF with two different frameworks are denoted in red and violet, respectively. A detailed notation explana- tion is specified in the Experiments section. Different testing datasets are denoted by different shapes.
  • Neuromorphic Hardware Validation: When simulated on the neuromorphic hardware accelerator SpikeSim, the proposed TCP strategy outperformed a naive temporal padding approach by an order of magnitude in both inference speed and energy cost.

Limitations and Future Work

While SpikingNeRF drastically reduces energy consumption, it inherits some qualitative limitations from its underlying ANN frameworks, such as occasional texture distortion in highly complex scenes. Additionally, the temporal condensing operation is currently performed off-chip. While its latency is successfully hidden by hardware pipelining in simulations, integrating this data-reshaping step directly onto neuromorphic hardware remains an important direction for future research.

Summary updated

Abstract

In this paper, we propose SpikingNeRF, which aligns the temporal dimension of spiking neural networks (SNNs) with the radiance rays, to seamlessly accommodate SNNs to the reconstruction of neural radiance fields (NeRF). Thus, the computation turns into a spike-based, multiplication-free manner, reducing energy consumption and making high-quality 3D rendering, for the first time, accessible to neuromorphic hardware. In SpikingNeRF, each sampled point on the ray is matched to a particular time step and represented in a hybrid manner where the voxel grids are maintained as well. Based on the voxel grids, sampled points are determined whether to be masked out for faster training and inference. However, this masking operation also incurs irregular temporal length, making it intractable for hardware processors, e.g., GPUs, to conduct parallel training. To address this problem, we develop the temporal padding strategy to tackle the masked samples to maintain regular temporal length, i.e., regular tensors, and further propose the temporal condensing strategy to form a denser data structure for hardware-friendly computation. Experiments on various datasets demonstrate that our method can reduce energy consumption by an average of 70.79\% and obtain comparable synthesis quality with the ANN baseline. Verification on the neuromorphic hardware accelerator also shows that SpikingNeRF can further benefit from neuromorphic computing over the ANN baselines on energy efficiency. Codes and the appendix are in \url{https://github.com/Ikarosy/SpikingNeRF-of-CASIA}.
BibTeX
@misc{yao2023spikingnerf,
  author = {Xingting Yao and Qinghao Hu and Fei Zhou and Tielong Liu and Zitao Mo and Zeyu Zhu and Zhengyang Zhuge and Jian Cheng},
  title = {{SpikingNeRF: Making Bio-inspired Neural Networks See through the Real World}},
  year = {2023},
  eprint = {2309.10987},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.10987},
}
Figure