SpikeGS: 3D Gaussian Splatting from Spike Streams with High-Speed Camera Motion

Jiyuan Zhang, Kang Chen, Shiyan Chen, Yajing Zheng, Tiejun Huang, Zhaofei Yu

TL;DR

SpikeGS is the first framework to combine the real-time efficiency of 3D Gaussian Splatting (3DGS) with the ultra-high temporal resolution of neuromorphic spike cameras. By leveraging a continuous 40,000 Hz spike stream, it achieves high-quality 3D scene reconstruction from fast-moving cameras, training in minutes where previous NeRF-based methods took hours.

What Problem Does This Paper Address?

Novel View Synthesis (NVS)—the ability to generate new 2D perspectives of a 3D scene—has been revolutionized by Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). However, these methods hit a wall when the camera moves quickly.

Standard cameras use discrete exposure windows. If the camera moves fast during that window, you get motion blur. For 3DGS, this is catastrophic: blurry images lead to poor initial point clouds and incorrectly optimized Gaussian shapes. While NeRF-based solutions for spike cameras (like SpikeNeRF) exist, they are notoriously slow to train and render. The authors identify a need for a system that is both robust to high-speed motion and fast enough for real-time applications.

Methods

The researchers introduce SpikeGS, which replaces traditional RGB frames with data from a spike camera.

The Spike Camera Advantage

Unlike standard cameras, a spike camera pixel accumulates photons until a threshold is reached, at which point it fires a binary "spike" and resets. This happens at extremely high frequencies (up to 40,000 Hz), effectively providing a continuous, blur-free record of the scene.

Figure 2: Illustration of the principle of spike cameras.

The SpikeGS Pipeline

To bridge the gap between binary spikes and 3D Gaussians, the authors developed two key components:

  1. Spike-based Instant Mapping (SIM): A tiny, self-supervised network (only 30k parameters) that converts segments of spikes into sharp "instant" grayscale images. It uses a Blind Spot Network (BSN) architecture, which ensures the model learns to reconstruct the scene rather than just memorizing noisy spike patterns.
  2. Dual-Loss Training:
    • Instantaneous Loss ($L_{instant}$): Supervises the 3DGS rendering using the sharp images produced by SIM.
  3. Exposing-like Loss ($L_{exposure}$): A novel constraint that relates the average of multiple 3DGS renders (simulating a long exposure) to the total spike count over that same time interval. This ensures the 3D model respects the physical light intensity recorded by the sensor.
Comparison showing how spike cameras avoid the "blind areas" of traditional exposure-based cameras.

Datasets

The authors evaluated SpikeGS on two fronts:

  • Synthetic Dataset: They created a new benchmark by simulating spiral camera paths in Blender for classic scenes (Lego, Chair, etc.) and passing the high-frame-rate output through a physically-based spike simulator.
  • Real-World Dataset: They used the existing SpikeNeRF dataset, which contains real-world captures from a 250x400 resolution spike camera moving at high speeds.

Key Insights and Ideas

  • Continuous Supervision: Traditional NVS treats video as a collection of snapshots. SpikeGS treats the spike stream as a continuous flow of information, filling the "temporal voids" between frames that usually plague high-speed reconstruction.
  • Efficiency by Design: By keeping the SIM network extremely lightweight (running at 1200+ FPS), the authors ensure that the spike-to-image conversion doesn't become a bottleneck for the 3DGS optimization.
  • Self-Supervised Synergy: The SIM doesn't need ground-truth sharp images for training. It uses the noisy "Texture from Playback" (TFP) method as a target, relying on the spatial consistency of the BSN to filter out the noise.

Key Results

SpikeGS significantly outperforms existing spike-based 3D reconstruction methods in both quality and speed:

  • Training Speed: On real-world data, SpikeGS trained in just 0.28 minutes, compared to 330 minutes for SpikeNeRF—a speedup of over 1000x.
  • Image Quality: On synthetic benchmarks, SpikeGS achieved a PSNR of 32.67 dB on the Lego scene, whereas SpikeNeRF only reached 19.24 dB.
  • Visual Fidelity: Qualitatively, SpikeGS recovers fine textures and sharp edges (like the treads on a Lego bulldozer) that are completely lost to blur or noise in baseline methods.

Limitations and Future Work

While SpikeGS is a major leap forward, the authors note a few areas for growth:

  • Grayscale Limitation: Current spike cameras are primarily grayscale. While color spike sensors are emerging, this work focuses on intensity-based reconstruction.
  • Pose Dependency: The system still relies on Structure-from-Motion (SfM) to get initial camera poses. Integrating spike-based odometry could make the system even more robust.
  • Dynamic Scenes: The current model assumes a static scene with a moving camera. Extending this to dynamic objects in high-speed environments is a natural next step.

Summary updated

Abstract

Novel View Synthesis plays a crucial role by generating new 2D renderings from multi-view images of 3D scenes. However, capturing high-speed scenes with conventional cameras often leads to motion blur, hindering the effectiveness of 3D reconstruction. To address this challenge, high-frame-rate dense 3D reconstruction emerges as a vital technique, enabling detailed and accurate modeling of real-world objects or scenes in various fields, including Virtual Reality or embodied AI. Spike cameras, a novel type of neuromorphic sensor, continuously record scenes with an ultra-high temporal resolution, showing potential for accurate 3D reconstruction. Despite their promise, existing approaches, such as applying Neural Radiance Fields (NeRF) to spike cameras, encounter challenges due to the time-consuming rendering process. To address this issue, we make the first attempt to introduce the 3D Gaussian Splatting (3DGS) into spike cameras in high-speed capture, providing 3DGS as dense and continuous clues of views, then constructing SpikeGS. Specifically, to train SpikeGS, we establish computational equations between the rendering process of 3DGS and the processes of instantaneous imaging and exposing-like imaging of the continuous spike stream. Besides, we build a very lightweight but effective mapping process from spikes to instant images to support training. Furthermore, we introduced a new spike-based 3D rendering dataset for validation. Extensive experiments have demonstrated our method possesses the high quality of novel view rendering, proving the tremendous potential of spike cameras in modeling 3D scenes.
BibTeX
@inproceedings{zhang2024spikegs,
  author = {Jiyuan Zhang and Kang Chen and Shiyan Chen and Yajing Zheng and Tiejun Huang and Zhaofei Yu},
  title = {{SpikeGS: 3D Gaussian Splatting from Spike Streams with High-Speed Camera Motion}},
  booktitle = {ACM Multimedia},
  year = {2024},
}
Figure