Introduction
Snapshot Compressive Imaging (SCI) is a powerful technique for capturing high-speed or multi-view data by compressing it into a single 2D frame using temporal or spatial masks. While efficient for storage and privacy, reconstructing the original 3D scene from this "smeared" measurement is a significant challenge. Traditional methods treat this as a 2D image restoration task, often failing to maintain 3D consistency or recover fine details. SCINeRF addresses these limitations by incorporating the physical principles of SCI directly into a Neural Radiance Field (NeRF) framework.
Methodology
Instead of attempting to "de-blur" the image first, SCINeRF optimizes a 3D scene representation such that its projection through the SCI imaging model matches the captured measurement. The system treats the SCI measurement not as an image to be fixed, but as a signal to be explained by a 3D model.
Joint Pose and Geometry Optimization
A major hurdle in 3D reconstruction from a single SCI snapshot is the lack of known camera poses. Standard tools like COLMAP cannot process the coded blur. SCINeRF solves this by performing joint optimization of both the scene geometry and the camera trajectory. Assuming a linear trajectory during the extremely short exposure time (often <20ms), the system learns the start and end poses simultaneously with the NeRF parameters. This test-time optimization avoids the generalization issues common in deep-learning decoders that require massive training datasets.
Experimental Results
The authors evaluated SCINeRF against state-of-the-art (SOTA) 2D decoders, such as EfficientSCI and PnP-FFDNet, using both synthetic and real-world datasets captured with custom SCI hardware.
Texture and Detail Recovery
One of the most striking advantages of SCINeRF is its ability to recover high-frequency textures that are lost in 2D-based methods. For example, in the "Baltimore Whiskey" scene, SCINeRF successfully reconstructs legible text on signage where competing methods produce unreadable artifacts. Quantitatively, on the "Vender" synthetic scene, SCINeRF achieved a PSNR of 36.40, significantly outperforming EfficientSCI’s 33.17.
Robustness to Compression
SCINeRF demonstrates remarkable robustness to high compression ratios (CR). While traditional methods degrade significantly at 32x compression, SCINeRF leverages the global 3D prior of the NeRF to maintain high image quality and multi-view consistency. Furthermore, because it learns a true 3D representation, SCINeRF can render high-quality images from novel viewpoints not present in the original measurement.
Conclusion and Future Work
SCINeRF represents a shift in SCI decoding from 2D restoration to 3D scene understanding. By treating the compressed measurement as a signal to be explained by a 3D model, it achieves superior detail recovery and enables novel view synthesis from a single snapshot. While the current implementation assumes linear camera motion and requires significant computational time for optimization, future work could integrate faster NeRF variants like Instant-NGP or extend the framework to handle non-rigid, dynamic scenes.
Summary updated