SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image

Yunhao Li, Xiaodong Wang, Ping Wang, Xin Yuan, Peidong Liu

Introduction

Snapshot Compressive Imaging (SCI) is a powerful technique for capturing high-speed or multi-view data by compressing it into a single 2D frame using temporal or spatial masks. While efficient for storage and privacy, reconstructing the original 3D scene from this "smeared" measurement is a significant challenge. Traditional methods treat this as a 2D image restoration task, often failing to maintain 3D consistency or recover fine details. SCINeRF addresses these limitations by incorporating the physical principles of SCI directly into a Neural Radiance Field (NeRF) framework.

Methodology

Instead of attempting to "de-blur" the image first, SCINeRF optimizes a 3D scene representation such that its projection through the SCI imaging model matches the captured measurement. The system treats the SCI measurement not as an image to be fixed, but as a signal to be explained by a 3D model.

The SCINeRF pipeline integrates the physical imaging process, including mask modulation and volumetric rendering, into the differentiable training loop to minimize the loss between captured and synthesized measurements.

Joint Pose and Geometry Optimization

A major hurdle in 3D reconstruction from a single SCI snapshot is the lack of known camera poses. Standard tools like COLMAP cannot process the coded blur. SCINeRF solves this by performing joint optimization of both the scene geometry and the camera trajectory. Assuming a linear trajectory during the extremely short exposure time (often <20ms), the system learns the start and end poses simultaneously with the NeRF parameters. This test-time optimization avoids the generalization issues common in deep-learning decoders that require massive training datasets.

Experimental Results

The authors evaluated SCINeRF against state-of-the-art (SOTA) 2D decoders, such as EfficientSCI and PnP-FFDNet, using both synthetic and real-world datasets captured with custom SCI hardware.

Texture and Detail Recovery

One of the most striking advantages of SCINeRF is its ability to recover high-frequency textures that are lost in 2D-based methods. For example, in the "Baltimore Whiskey" scene, SCINeRF successfully reconstructs legible text on signage where competing methods produce unreadable artifacts. Quantitatively, on the "Vender" synthetic scene, SCINeRF achieved a PSNR of 36.40, significantly outperforming EfficientSCI’s 33.17.

Qualitative comparison showing the progression from the blurred SCI measurement to the high-fidelity reconstruction of the "Baltimore Whiskey" sign.

Robustness to Compression

SCINeRF demonstrates remarkable robustness to high compression ratios (CR). While traditional methods degrade significantly at 32x compression, SCINeRF leverages the global 3D prior of the NeRF to maintain high image quality and multi-view consistency. Furthermore, because it learns a true 3D representation, SCINeRF can render high-quality images from novel viewpoints not present in the original measurement.

Conclusion and Future Work

SCINeRF represents a shift in SCI decoding from 2D restoration to 3D scene understanding. By treating the compressed measurement as a signal to be explained by a 3D model, it achieves superior detail recovery and enables novel view synthesis from a single snapshot. While the current implementation assumes linear camera motion and requires significant computational time for optimization, future work could integrate faster NeRF variants like Instant-NGP or extend the framework to handle non-rigid, dynamic scenes.

Summary updated

Abstract

In this paper, we explore the potential of Snapshot Compressive Imaging (SCI) technique for recovering the underlying 3D scene representation from a single temporal compressed image. SCI is a cost-effective method that enables the recording of high-dimensional data, such as hyperspectral or temporal information, into a single image using low-cost 2D imaging sensors. To achieve this, a series of specially designed 2D masks are usually employed, which not only reduces storage requirements but also offers potential privacy protection. Inspired by this, to take one step further, our approach builds upon the powerful 3D scene representation capabilities of neural radiance fields (NeRF). Specifically, we formulate the physical imaging process of SCI as part of the training of NeRF, allowing us to exploit its impressive performance in capturing complex scene structures. To assess the effectiveness of our method, we conduct extensive evaluations using both synthetic data and real data captured by our SCI system. Extensive experimental results demonstrate that our proposed approach surpasses the state-of-the-art methods in terms of image reconstruction and novel view image synthesis. Moreover, our method also exhibits the ability to restore high frame-rate multi-view consistent images by leveraging SCI and the rendering capabilities of NeRF. The code is available at https://github.com/WU-CVGL/SCINeRF.
BibTeX
@inproceedings{li2024scinerf,
  author = {Yunhao Li and Xiaodong Wang and Ping Wang and Xin Yuan and Peidong Liu},
  title = {{SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01003},
}
Figure