Abstract
Most deep network methods for compressive sensing reconstruction suffer from the "black-box" characteristic of DNNs. In this paper, a deep neural network with interpretable motion estimation named CSMCNet is proposed. The network realizes high-quality reconstruction of video compressive sensing by unfolding the iterative steps of optimization-based algorithms. A DNN-based, multi-hypothesis motion estimation module is designed to improve reconstruction quality, and a residual module is employed to further narrow the gap between reconstruction results and the original signal. Additionally, an interpolation (ITP) module with a corresponding training strategy is proposed to realize scalable CS reconstruction, enabling a single model to decode various compression ratios. Experiments show that a PSNR of 29.34dB can be achieved at a 2% CS ratio (98% compression), outperforming other state-of-the-art methods. The interpolation module is proven effective, offering significant cost savings with acceptable performance losses.
Introduction
Compressive Sensing (CS) allows for signal acquisition at rates significantly lower than the Nyquist-Shannon sampling theorem suggests, by performing compression during the sensing phase. While traditional optimization-based reconstruction methods offer interpretability, their iterative nature makes them computationally expensive. Conversely, modern Deep Neural Network (DNN) approaches provide speed but often operate as "black boxes" and lack scalability across different compression ratios (CR). CSMCNet aims to bridge this gap by creating a fast, high-quality reconstruction network that is both interpretable and scalable.
The CSMCNet Framework
The authors introduce CSMCNet, a deep unfolding neural network designed for video CS reconstruction. By mapping iterative optimization steps onto network layers, CSMCNet maintains interpretability while leveraging the speed of deep learning.
The architecture is divided into three primary stages:
- Preliminary Reconstruction: Generates an initial estimate from sparse measurements.
- Multi-Hypothesis Motion Estimation (MHME): Explicitly predicts temporal changes using reference frames.
- Residual Reconstruction: Refines the prediction to match the original signal.
Interpretable Reconstruction and Motion Estimation
A key innovation is the MHME module, which replaces the opaque temporal modeling of standard DNNs with an interpretable process. It utilizes a buffer of previously reconstructed frames to perform motion compensation, effectively "borrowing" details from the past to reconstruct the present. This "unfolding" approach mimics traditional video codecs, making the network's behavior easier to analyze and tune.
Scalability via the ITP Module
To address the "fixed-rate" limitation of most DNNs, CSMCNet incorporates an Interpolation (ITP) module. Using a 1D-deconvolution layer, the ITP module resizes measurements from various compression ratios into a standardized format. This allows a single trained model to handle multiple bitrates, reducing storage requirements by 5x and significantly cutting training time.
Experimental Results
Evaluated on the vimeo-90K and standard video datasets, CSMCNet demonstrates state-of-the-art performance. At a 2% CS ratio (98% compression), it achieves a PSNR of 29.34dB, outperforming competitors like CSVideoNet and ISTANet. While the scalable ITP module introduces a minor performance trade-off (approx. 10% PSNR loss), the gains in efficiency and storage make it a practical solution for real-world deployment.
Conclusion
CSMCNet successfully combines the interpretability of classical optimization with the efficiency of deep learning. Its scalable design and motion-aware reconstruction set a new benchmark for video compressive sensing, offering a robust path forward for high-speed, low-bandwidth visual data acquisition.
Summary updated