CSMCNet: Scalable Video Compressive Sensing Reconstruction with Interpretable Motion Estimation

Bowen Huang, Xiao Yan, Jinjia Zhou, Yibo FanarXiv 2021 · DOI · arXiv

Abstract

Most deep network methods for compressive sensing reconstruction suffer from the "black-box" characteristic of DNNs. In this paper, a deep neural network with interpretable motion estimation named CSMCNet is proposed. The network realizes high-quality reconstruction of video compressive sensing by unfolding the iterative steps of optimization-based algorithms. A DNN-based, multi-hypothesis motion estimation module is designed to improve reconstruction quality, and a residual module is employed to further narrow the gap between reconstruction results and the original signal. Additionally, an interpolation (ITP) module with a corresponding training strategy is proposed to realize scalable CS reconstruction, enabling a single model to decode various compression ratios. Experiments show that a PSNR of 29.34dB can be achieved at a 2% CS ratio (98% compression), outperforming other state-of-the-art methods. The interpolation module is proven effective, offering significant cost savings with acceptable performance losses.

Introduction

Compressive Sensing (CS) allows for signal acquisition at rates significantly lower than the Nyquist-Shannon sampling theorem suggests, by performing compression during the sensing phase. While traditional optimization-based reconstruction methods offer interpretability, their iterative nature makes them computationally expensive. Conversely, modern Deep Neural Network (DNN) approaches provide speed but often operate as "black boxes" and lack scalability across different compression ratios (CR). CSMCNet aims to bridge this gap by creating a fast, high-quality reconstruction network that is both interpretable and scalable.

The CSMCNet Framework

The authors introduce CSMCNet, a deep unfolding neural network designed for video CS reconstruction. By mapping iterative optimization steps onto network layers, CSMCNet maintains interpretability while leveraging the speed of deep learning.

The end-to-end pipeline of CSMCNet, from original frame sampling to final reconstruction.

The architecture is divided into three primary stages:

  1. Preliminary Reconstruction: Generates an initial estimate from sparse measurements.
  2. Multi-Hypothesis Motion Estimation (MHME): Explicitly predicts temporal changes using reference frames.
  3. Residual Reconstruction: Refines the prediction to match the original signal.

Interpretable Reconstruction and Motion Estimation

A key innovation is the MHME module, which replaces the opaque temporal modeling of standard DNNs with an interpretable process. It utilizes a buffer of previously reconstructed frames to perform motion compensation, effectively "borrowing" details from the past to reconstruct the present. This "unfolding" approach mimics traditional video codecs, making the network's behavior easier to analyze and tune.

Detailed architecture of the Reconstruction Model, highlighting the interframe reference improvement stage.

Scalability via the ITP Module

To address the "fixed-rate" limitation of most DNNs, CSMCNet incorporates an Interpolation (ITP) module. Using a 1D-deconvolution layer, the ITP module resizes measurements from various compression ratios into a standardized format. This allows a single trained model to handle multiple bitrates, reducing storage requirements by 5x and significantly cutting training time.

Experimental Results

Evaluated on the vimeo-90K and standard video datasets, CSMCNet demonstrates state-of-the-art performance. At a 2% CS ratio (98% compression), it achieves a PSNR of 29.34dB, outperforming competitors like CSVideoNet and ISTANet. While the scalable ITP module introduces a minor performance trade-off (approx. 10% PSNR loss), the gains in efficiency and storage make it a practical solution for real-world deployment.

Conclusion

CSMCNet successfully combines the interpretability of classical optimization with the efficiency of deep learning. Its scalable design and motion-aware reconstruction set a new benchmark for video compressive sensing, offering a robust path forward for high-speed, low-bandwidth visual data acquisition.

Summary updated

Abstract

Most deep network methods for compressive sensing reconstruction suffer from the black-box characteristic of DNN. In this paper, a deep neural network with interpretable motion estimation named CSMCNet is proposed. The network is able to realize high-quality reconstruction of video compressive sensing by unfolding the iterative steps of optimization based algorithms. A DNN based, multi-hypothesis motion estimation module is designed to improve the reconstruction quality, and a residual module is employed to further narrow down the gap between re-construction results and original signal in our proposed method. Besides, we propose an interpolation module with corresponding training strategy to realize scalable CS reconstruction, which is capable of using the same model to decode various compression ratios. Experiments show that a PSNR of 29.34dB can be achieved at 2% CS ratio (compressed by 98%), which is superior than other state-of-the-art methods. Moreover, the interpolation module is proved to be effective, with significant cost saving and acceptable performance losses.
BibTeX
@misc{huang2021csmcnet,
  author = {Bowen Huang and Xiao Yan and Jinjia Zhou and Yibo Fan},
  title = {{CSMCNet: Scalable Video Compressive Sensing Reconstruction with Interpretable Motion Estimation}},
  year = {2021},
  eprint = {2108.01522},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.01522},
}
Figure