TL;DR
SpikeGS is the first framework to combine the real-time efficiency of 3D Gaussian Splatting (3DGS) with the ultra-high temporal resolution of neuromorphic spike cameras. By leveraging a continuous 40,000 Hz spike stream, it achieves high-quality 3D scene reconstruction from fast-moving cameras, training in minutes where previous NeRF-based methods took hours.
What Problem Does This Paper Address?
Novel View Synthesis (NVS)—the ability to generate new 2D perspectives of a 3D scene—has been revolutionized by Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). However, these methods hit a wall when the camera moves quickly.
Standard cameras use discrete exposure windows. If the camera moves fast during that window, you get motion blur. For 3DGS, this is catastrophic: blurry images lead to poor initial point clouds and incorrectly optimized Gaussian shapes. While NeRF-based solutions for spike cameras (like SpikeNeRF) exist, they are notoriously slow to train and render. The authors identify a need for a system that is both robust to high-speed motion and fast enough for real-time applications.
Methods
The researchers introduce SpikeGS, which replaces traditional RGB frames with data from a spike camera.
The Spike Camera Advantage
Unlike standard cameras, a spike camera pixel accumulates photons until a threshold is reached, at which point it fires a binary "spike" and resets. This happens at extremely high frequencies (up to 40,000 Hz), effectively providing a continuous, blur-free record of the scene.
The SpikeGS Pipeline
To bridge the gap between binary spikes and 3D Gaussians, the authors developed two key components:
- Spike-based Instant Mapping (SIM): A tiny, self-supervised network (only 30k parameters) that converts segments of spikes into sharp "instant" grayscale images. It uses a Blind Spot Network (BSN) architecture, which ensures the model learns to reconstruct the scene rather than just memorizing noisy spike patterns.
- Dual-Loss Training:
- Instantaneous Loss ($L_{instant}$): Supervises the 3DGS rendering using the sharp images produced by SIM.
- Exposing-like Loss ($L_{exposure}$): A novel constraint that relates the average of multiple 3DGS renders (simulating a long exposure) to the total spike count over that same time interval. This ensures the 3D model respects the physical light intensity recorded by the sensor.
Datasets
The authors evaluated SpikeGS on two fronts:
- Synthetic Dataset: They created a new benchmark by simulating spiral camera paths in Blender for classic scenes (Lego, Chair, etc.) and passing the high-frame-rate output through a physically-based spike simulator.
- Real-World Dataset: They used the existing SpikeNeRF dataset, which contains real-world captures from a 250x400 resolution spike camera moving at high speeds.
Key Insights and Ideas
- Continuous Supervision: Traditional NVS treats video as a collection of snapshots. SpikeGS treats the spike stream as a continuous flow of information, filling the "temporal voids" between frames that usually plague high-speed reconstruction.
- Efficiency by Design: By keeping the SIM network extremely lightweight (running at 1200+ FPS), the authors ensure that the spike-to-image conversion doesn't become a bottleneck for the 3DGS optimization.
- Self-Supervised Synergy: The SIM doesn't need ground-truth sharp images for training. It uses the noisy "Texture from Playback" (TFP) method as a target, relying on the spatial consistency of the BSN to filter out the noise.
Key Results
SpikeGS significantly outperforms existing spike-based 3D reconstruction methods in both quality and speed:
- Training Speed: On real-world data, SpikeGS trained in just 0.28 minutes, compared to 330 minutes for SpikeNeRF—a speedup of over 1000x.
- Image Quality: On synthetic benchmarks, SpikeGS achieved a PSNR of 32.67 dB on the Lego scene, whereas SpikeNeRF only reached 19.24 dB.
- Visual Fidelity: Qualitatively, SpikeGS recovers fine textures and sharp edges (like the treads on a Lego bulldozer) that are completely lost to blur or noise in baseline methods.
Limitations and Future Work
While SpikeGS is a major leap forward, the authors note a few areas for growth:
- Grayscale Limitation: Current spike cameras are primarily grayscale. While color spike sensors are emerging, this work focuses on intensity-based reconstruction.
- Pose Dependency: The system still relies on Structure-from-Motion (SfM) to get initial camera poses. Integrating spike-based odometry could make the system even more robust.
- Dynamic Scenes: The current model assumes a static scene with a moving camera. Extending this to dynamic objects in high-speed environments is a natural next step.
Summary updated