Abstract
Sampling high-dimensional images is challenging due to limited availability of sensors; scanning is usually necessary in these cases. To mitigate this challenge, snapshot compressive imaging (SCI) was proposed to capture high-dimensional (usually 3D) images using a 2D sensor (detector). Via novel optical design, the measurement captured by the sensor is an encoded image of multiple frames of the 3D desired signal. Following this, reconstruction algorithms are employed to retrieve the high-dimensional data. Though various algorithms have been proposed, the total variation (TV) based method is still the most efficient one due to a good trade-off between computational time and performance. This paper aims to answer the question of which TV penalty (anisotropic TV, isotropic TV and vectorized TV) works best for video SCI reconstruction? Various TV denoising and projection algorithms are developed and tested for video SCI reconstruction on both simulation and real datasets.
Introduction
Capturing high-speed video traditionally requires expensive sensors with high data throughput. Snapshot Compressive Imaging (SCI) offers a sophisticated workaround by using a physical mask—a coded aperture—to modulate high-speed frames before they are integrated by a standard 2D sensor. This results in a single "snapshot" that contains compressed information from multiple time points, effectively "smearing" the temporal dimension into a single spatial measurement.
The primary challenge in SCI is the inverse problem: mathematically "un-smearing" the 2D measurement back into a high-quality 3D video. While deep learning approaches are emerging, Total Variation (TV) regularization remains a workhorse in the field due to its computational efficiency and lack of training data requirements. This study benchmarks various TV formulations and optimization frameworks to determine the most effective combination for video SCI.
Methodology
The research evaluates three primary TV formulations designed to exploit spatial and temporal redundancies:
- Anisotropic TV (ATV): Penalizes horizontal and vertical gradients independently.
- Isotropic TV 2D (ITV2D): Penalizes the combined magnitude of spatial gradients per frame.
- Isotropic TV 3D (ITV3D): Extends the penalty across the temporal dimension, leveraging the high correlation between neighboring frames in a video sequence.
These formulations were integrated into four major optimization frameworks: FISTA, TwIST, ADMM (Alternating Direction Method of Multipliers), and GAP (Generalized Alternating Projection). GAP is particularly noteworthy for SCI as it exploits the block-diagonal structure of the sensing matrix to perform element-wise calculations, significantly reducing memory overhead.
Experimental Results
The algorithms were tested on both simulated benchmark datasets (Kobe, Traffic, Runner, Drop) and real-world data captured from a custom DMD-based SCI camera.
Simulation Performance
In simulations involving 8-frame compression, GAP-ATV-Clip achieved the highest average PSNR of 27.56 dB. However, the performance gap between top-tier TV methods was narrow, typically less than 0.5 dB. A critical finding was the disparity in convergence speed: GAP and ADMM reached high-quality results in approximately 60 iterations, whereas TwIST required over 500 iterations to achieve comparable quality.
Real-World Reconstruction
Real-world tests involved reconstructing 10 frames (512×512 resolution) from a single snapshot of a moving hand. As shown in the visual comparison, GAP and ADMM frameworks successfully recovered sharp motion and clear finger boundaries, whereas FISTA produced noticeably blurrier results.
Discussion and Conclusion
The study concludes that while 3D Isotropic TV (ITV3D) offers superior theoretical modeling by accounting for temporal correlation, the GAP solver combined with Fast Gradient Projection (FGP) denoising provides the best practical balance of reconstruction speed and image quality.
While TV regularization can sometimes lead to "blocky" artifacts in complex textures, its efficiency makes it an ideal candidate for real-time applications or as a high-quality initialization for more computationally intensive methods like Rank-minimization (DeSCI) or Deep Learning-based Plug-and-Play (PnP) frameworks.
Summary updated