EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive Imaging

Lishun Wang, Miao Cao, Xin Yuan

Abstract

Video Snapshot Compressive Imaging (SCI) captures high-speed motion by compressing multiple temporal frames into a single 2D measurement. However, reconstructing high-quality video from these measurements remains computationally prohibitive for large-scale data. This paper introduces EfficientSCI, a novel end-to-end architecture that achieves state-of-the-art reconstruction quality with significantly reduced computational overhead. By employing hierarchical dense connections and space-time factorization, EfficientSCI enables the first high-fidelity reconstruction of UHD color video from a single snapshot.

Introduction

Video SCI systems offer a cost-effective alternative to high-speed cameras by using a standard sensor and a temporal modulation mask. The primary challenge is the reconstruction algorithm, which must solve an ill-posed inverse problem. While deep learning models have improved accuracy, they often suffer from high memory consumption and slow inference speeds, particularly when scaling to 4K/UHD resolutions. EfficientSCI addresses these limitations through a design focused on memory efficiency and linear computational complexity.

Proposed Method: EfficientSCI

The EfficientSCI architecture is built upon two core innovations: the ResDNet block and the CFormer module.

Hierarchical Dense Connections (ResDNet)

Traditional residual networks often lack sufficient feature reuse. EfficientSCI introduces the ResDNet Block, which incorporates dense connections within a single residual unit. This design aggregates features at a more granular level, boosting learning capacity while reducing overall model complexity compared to standard dense architectures.

Space-Time Factorization (CFormer)

To handle the high-dimensional nature of video data efficiently, the network utilizes CFormer blocks. These blocks factorize the processing into spatial and temporal domains:

  • Spatial Domain: 2D convolutions are used to exploit local spatial correlations, maintaining low memory usage and high speed.
  • Temporal Domain: A Temporal Self-Attention Branch (TSAB) captures long-range dependencies across frames. By restricting attention to the temporal axis, the model avoids the quadratic computational explosion typical of full 3D Transformers relative to spatial resolution.
The overall pipeline of EfficientSCI, including preprocessing, the proposed network architecture with ResDNet blocks, and the final reconstruction.

Experiments and Results

The model was evaluated on various benchmarks, ranging from 256×256 grayscale simulations to UHD (1644×3840) color videos.

Quantitative Performance

EfficientSCI-L (the "Large" variant) achieves a PSNR of 36.92 dB on standard grayscale benchmarks, setting a new state-of-the-art. More importantly, the "Tiny" variant (EfficientSCI-T) provides a significant speedup—nearly 8x faster than previous leading models like DUN-3DUnet—while maintaining competitive quality.

Efficiency and Scalability

A key highlight is the model's ability to reconstruct UHD video. On the "Football" UHD dataset, EfficientSCI-S achieved a 2.5 dB improvement over PnP-FastDVDnet while being approximately 200 times faster, reducing processing time from hours to minutes. The linear complexity of the temporal attention mechanism ensures that memory usage remains manageable even for extremely high resolutions.

Performance comparison of EfficientSCI variants against state-of-the-art models in terms of PSNR, testing time, and parameter count.

Conclusion

EfficientSCI represents a significant advancement in video snapshot compressive imaging. By combining the strengths of CNNs for spatial processing and Transformers for temporal modeling, it breaks the computational barriers that previously limited SCI to low-resolution applications. The architecture's efficiency and scalability open new possibilities for real-time high-speed imaging and large-scale video reconstruction tasks, including potential applications in spectral SCI.

Summary updated

Abstract

Video snapshot compressive imaging (SCI) uses a twodimensional detector to capture consecutive video frames during a single exposure time. Following this, an efficient reconstruction algorithm needs to be designed to reconstruct the desired video frames. Although recent deep learning-based state-of-the-art (SOTA) reconstruction algorithms have achieved good results in most tasks, they still face the following challenges due to excessive model complexity and GPU memory limitations: 1) these models need high computational cost, and 2) they are usually unable to reconstruct large-scale video frames at high compression ratios. To address these issues, we develop an efficient network for video SCI by using dense connections and space-time factorization mechanism within a single residual block, dubbed EfficientSCI. The EfficientSCI network can well establish spatial-temporal correlation by using convolution in the spatial domain and Transformer in the temporal domain, respectively. We are the first time to show that an UHD color video with high compression ratio can be reconstructed from a snapshot 2D measurement using a single end-to-end deep learning model with PSNR above 32 dB. Extensive results on both simulation and real data show that our method significantly outperforms all previous SOTA algorithms with better real-time performance. The code is at https://github.com/ucaswangls/EfficientSCI.git.
BibTeX
@inproceedings{wang2023efficientsci,
  author = {Lishun Wang and Miao Cao and Xin Yuan},
  title = {{EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive Imaging}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure