Abstract
Spiking neural networks (SNNs) have recently demonstrated significant progress across various computational tasks, due to their potential for energy efficiency. Neural radiance fields (NeRFs) excel at rendering high-quality 3D scenes but require substantial energy consumption, with limited exploration of energy-saving solutions from a neuromorphic approach. In this paper, we present SpiNeRF, a novel method that integrates the sequential processing capabilities of SNNs with the ray-casting mechanism of NeRFs, aiming to enhance compatibility and unlock new prospects for energy-efficient 3D scene synthesis. Unlike conventional SNN encoding schemes, our method considers the spatial continuity inherent in NeRF, achieving superior rendering quality. To further improve training and inference efficiency, we adopt a hybrid volumetric representation that allows the predefinition and masking of invalid sampled points along pixel-rendering rays. However, this masking introduces irregular temporal lengths, making it intractable for hardware processors, such as graphics processing units (GPUs), to conduct effective parallel training. To address this issue, we present two methods: Temporal padding (TP) and temporal condensing-and-padding (TCP). Experiments on multiple datasets demonstrate that our method outperforms previous SNN encoding schemes and artificial neural network (ANN) quantization methods in both rendering quality and energy efficiency. Compared to the full-precision ANN baseline, our method reduces energy consumption by up to 72.95% while maintaining comparable synthesis quality.
The Energy Challenge of Neural Radiance Fields
Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction, allowing for the generation of photorealistic views from limited image sets. However, NeRFs are notoriously power-hungry, requiring thousands of queries to a heavy Multi-Layer Perceptron (MLP) for every single pixel rendered.
Researchers have looked toward Spiking Neural Networks (SNNs)—the "third generation" of neural networks—as a solution. SNNs mimic biological neurons by communicating via discrete, binary "spikes," making them incredibly energy-efficient on specialized neuromorphic hardware. Previous attempts to combine SNNs and NeRFs relied on converting pre-trained models, which required hundreds of time-steps to represent a single data point, erasing any energy gains. Furthermore, modern NeRFs use "masking" to skip empty space, creating irregular data structures that are difficult to process in parallel on standard GPUs when using SNNs.
Methodology: Bridging SNNs and NeRFs
SpiNeRF moves away from conversion-based methods in favor of direct training. The approach rests on three main pillars:
- Time-Ray Alignment (TRA): Instead of treating time as a way to encode a single number (like traditional SNNs), SpiNeRF maps the sequence of sampled points along a camera ray directly to the temporal sequence of the SNN. As the SNN processes a ray, each time-step corresponds to a different point in 3D space, preserving spatial continuity.
- Temporal Condensing-and-Padding (TCP): Because NeRFs mask out empty space, different rays have different numbers of valid points. TCP condenses valid points together and pads them to a uniform length, creating a dense data structure that is GPU-friendly for training while remaining SNN-friendly for sparse computation.
- Hybrid Volumetric Representation: SpiNeRF acts as a plug-in for existing voxel-grid frameworks, using explicit grids to store scene geometry and the SNN-based MLP to refine color and density.
The Importance of Ray Directionality
SNNs possess an inherent "memory" via membrane potential. A significant discovery in this research was that the direction of the ray (front-to-back vs. back-to-front) matters for rendering quality. Aligning the SNN's temporal flow with the camera's viewing direction yielded the best results. This "Temporal Flip" ensures that the direction of the temporal dimension remains consistent with the pixel-rendering ray.
Key Results and Efficiency Gains
SpiNeRF was evaluated across four benchmarks: Synthetic-NeRF, Synthetic-NSVF, BlendedMVS, and Tanks&Temples. The results highlight a major leap in efficiency:
- Massive Energy Savings: Compared to a full-precision ANN baseline, SpiNeRF reduced energy consumption by up to 72.95%.
- High Fidelity: Despite the binary nature of spikes, SpiNeRF maintained comparable visual quality, with a PSNR drop of only 0.33 dB on the Tanks&Temples dataset.
- Training Speed: The TCP strategy reduced training time by 72.68% and inference time by 63.93% on an NVIDIA A100 GPU compared to basic temporal padding.
- Superiority over Conversion: SpiNeRF outperformed previous SNN-NeRF conversion methods, which required 256 time-steps to reach similar accuracy, making SpiNeRF orders of magnitude more efficient.
Conclusion
SpiNeRF demonstrates that integrating the sequential processing of SNNs with the spatial rays of NeRF can unlock significant energy savings without sacrificing rendering quality. While a slight performance gap remains compared to full-precision ANNs in high-frequency textures, SpiNeRF represents a major step toward deploying high-quality 3D rendering on energy-constrained edge neuromorphic devices.
Summary updated