TL;DR
SweepEvGS combines the speed of 3D Gaussian Splatting (3D-GS) with the high temporal resolution of event cameras to reconstruct detailed 3D scenes from a single, fast camera sweep. By using event streams to supervise the radiance field, the method eliminates motion blur and achieves rendering speeds 150x faster than NeRF-based alternatives while using significantly less memory.
What Problem Does This Paper Address?
Creating high-quality 3D reconstructions (Radiance Fields) usually requires a "dense" set of sharp, high-resolution images. In practice, this is a headache. If you move the camera quickly to capture the scene, you get motion blur. If you move it slowly to keep images sharp, the process becomes tedious and inefficient.
This problem is magnified in macro and micro imaging. At these scales, even microscopic vibrations or slight camera movements result in significant blur, making traditional 3D Gaussian Splatting (3D-GS) or Neural Radiance Fields (NeRF) difficult to apply. While event cameras—sensors that only record changes in brightness at microsecond resolution—are immune to motion blur, existing event-based reconstruction methods (like EventNeRF) are computationally expensive and often struggle with real-world noise.
Methods
The authors introduce SweepEvGS, a hardware-integrated pipeline that uses a single "sweep" of an event camera to build a 3D scene.
The Core Workflow:
- Initial Anchor: The system captures one static RGB frame at the start of the sweep to provide a clean baseline and assist with camera calibration.
- Event Supervision: As the camera sweeps across the scene, it records a dense stream of events. Instead of trying to reconstruct frames from these events first, SweepEvGS uses the events directly to supervise the 3D-GS model.
- Linlog Mapping: Because event cameras respond to logarithmic changes in brightness, the authors use a "linlog" mapping. This allows the model to accurately compare the predicted brightness changes from the 3D Gaussians against the ground-truth event stream.
- Noise Filtering: Real-world event data is noisy, especially under a microscope. The researchers implemented a $y$-noise filter to strip out interference signals before training, ensuring the model learns the scene structure rather than sensor artifacts.
Datasets
The paper validates SweepEvGS across three distinct environments:
- Synthetic Sequences: Standard 3D models (Chair, Ficus, Hotdog) with simulated event data.
- Real-World Macro: Objects like a toy house, a keyboard, and cloth captured using a DAVIS 346C event camera mounted on a high-precision translation stage or turntable.
- Real-World Micro: A newly introduced benchmark for microscopic novel view synthesis, featuring high-resolution event data of an onion skin and a fly, captured via a DAVIS 346C mounted on a widefield microscope.
Key Insights and Ideas
- Events as the Primary Signal: Most 3D-GS methods treat images as the source of truth. SweepEvGS proves that the differential signal (the events) is actually superior for reconstruction during fast motion because it preserves high-frequency details that would be lost to blur in a standard frame.
- Efficiency over NeRF: By moving from a NeRF-based architecture to 3D Gaussian Splatting, the authors bypassed the need for expensive volumetric rendering. This results in a massive jump in performance—rendering at ~60 FPS compared to the ~0.3 FPS of previous event-based NeRFs.
- Scale Invariance: One of the most "aha" moments is the system's robustness across scales. The same underlying loss function and noise-filtering logic work for both a Lego set (macro) and a fly's wing (micro), suggesting that event-based 3D-GS is a generalized solution for "sweep-based" scanning.
Key Results
- Visual Quality: SweepEvGS consistently outperformed traditional 3D-GS (which failed due to blur) and existing event-based methods (Ev-GS, EventNeRF) in both PSNR and SSIM metrics. It successfully reconstructed sharp text on keyboards and intricate textures on cloth that other methods rendered as gray smears.
- Speedup: The method achieves approximately 150x faster rendering speeds than EventNeRF.
- Resource Efficiency: It requires roughly 3x less GPU memory (5GB vs 15GB) during the rendering stage, making it much more accessible for consumer-grade hardware.
- Training Time: Training takes roughly 8–10 minutes per scene, compared to the 14 hours required by NeRF-based event models.
Limitations and Future Work
The primary limitation acknowledged by the authors is the reliance on precise camera poses. Currently, the system requires a known trajectory (provided by a translation stage or turntable) because there is no "event-based COLMAP" mature enough to generate accurate camera coordinates from raw events alone.
Future Work: The authors aim to integrate event-based visual odometry. This would allow a user to simply "sweep" a handheld event camera over an object to create a 3D model, removing the need for specialized industrial motion controllers and making the technology truly portable.
Summary updated