A Unified Framework for Compressive Video Recovery From Coded Exposure Techniques

Prasan Shedligeri, Anupama S, Kaushik MitraWACV 2021

TL;DR

This paper introduces a unified deep learning framework to compare different coded exposure techniques for high-speed video acquisition under low bandwidth. Their novel framework, utilizing a shift-variant convolutional layer and a U-Net architecture, achieves state-of-the-art results and demonstrates that the Coded-2-Bucket (C2B) sensor offers significant advantages for static scenes, while its benefit diminishes for highly dynamic scenes.

What Problem Does This Paper Address?

Capturing high-frame-rate videos typically requires high sensitivity cameras and substantial data bandwidth, making them expensive. Coded exposure techniques offer a solution by acquiring compressed measurements at low bandwidth, which are then computationally reconstructed into high-frame-rate videos. While various coded exposure methods exist, including those capturing a single measurement (e.g., Flutter Shutter, Pixel-wise coded exposure) and newer methods capturing two measurements per exposure (e.g., Coded-2-Bucket or C2B), a clear quantitative comparison of their performance and advantages has been lacking. This paper aims to bridge this gap by providing a unified framework to evaluate and compare these techniques, offering insights into when each method is most effective.

Methods

The authors propose a unified, fully convolutional deep learning framework for compressive video recovery. The architecture consists of two main stages:

  1. Exposure-Aware Feature Extraction: This stage employs a Shift-Variant Convolutional (SVC) layer. Unlike standard convolutional layers that share weights across the entire image, the SVC layer allows for learning different weights for local neighborhoods. This is crucial for techniques like pixel-wise coded exposure where each pixel might have a unique exposure sequence, enabling adaptive feature extraction.
  2. Refinement Stage: A U-Net-like deep neural network takes the extracted features and reconstructs the full-resolution video sequence in a single forward pass. This architecture is chosen over fully connected networks due to its efficiency and effectiveness in image processing tasks.

The framework is designed to be flexible, allowing it to be trained for different coded exposure inputs. The loss function used for training includes an L1 loss for reconstruction fidelity and a TV-smoothness loss to encourage spatially coherent video outputs.

Datasets

  • GoPro dataset: Used for training and evaluation, consisting of 22 video sequences at 240 fps and 720x1280 resolution. Patches were extracted and downsampled for training.
  • DNN set [39]: A test set of 14 videos with 256x256 resolution and 16 frames each, used for evaluation.

The paper does not introduce new datasets but utilizes existing ones for comprehensive evaluation.

Key Insights and Ideas

  • Unified Framework is Key: The core idea is to create a single, adaptable framework that can handle diverse coded exposure inputs. This allows for a fair and direct comparison of different sensing strategies.
  • SVC Layer for Adaptability: The shift-variant convolutional layer is a critical innovation. It addresses the challenge of pixel-wise varying exposure codes, allowing the network to learn spatially adaptive filters, which is more efficient and effective than global filters for such scenarios.
  • Fully Convolutional Networks for Efficiency: The choice of a fully convolutional network (FCN) over fully connected networks is motivated by computational efficiency and better performance in handling large spatial and temporal resolutions, especially for tasks where local connectivity is sufficient.
  • C2B Advantage in Static Scenes: The most significant insight is that the C2B sensor's ability to capture two measurements per exposure provides a substantial advantage in reconstructing videos with many static regions. This is because the additional measurement can better capture details that might be lost in a single exposure.
  • Diminishing Returns in Dynamic Scenes: Conversely, when most scene points are in motion, the advantage of C2B over single pixel-wise coded exposure is marginal. This suggests that the benefit of C2B is tied to the presence of static information.

Key Results

  • State-of-the-Art Performance: The proposed unified framework achieves state-of-the-art reconstruction results across all three tested coded exposure techniques: Flutter Shutter (FS), Pixel-wise coded exposure, and C2B.
  • Quantitative Comparison:
    • FS: Performs reasonably well for 8-frame reconstruction but struggles with 16-frame reconstruction.
  • Pixel-wise Coded Exposure: Achieves good fidelity for 16-frame reconstruction.
  • C2B: Generally provides the best reconstruction, closely followed by pixel-wise coded exposure. FS lags significantly behind.
  • Qualitative Comparison: Visual comparisons (Figure 4 and 5) show that C2B produces sharper and more artifact-free reconstructions, especially in scenes with static elements.
  • Efficiency: The proposed method is computationally efficient, with GPU runtimes significantly lower than other methods like GMM, DNN, and AAUN for reconstructing a single sequence.
  • Ablation Studies:
    • The SVC layer (specifically SVC(64)+U-Net) significantly improves performance over a standard U-Net.
  • Using a coded-blurred image pair as input for C2B yields slightly better results than a coded-complement pair.
  • Jointly learning the coded mask and the reconstruction algorithm (Table 4) leads to better performance, particularly in noisy conditions.

Limitations and Future Work

  • Limitations:
    • The paper focuses on comparing existing coded exposure techniques. While it shows C2B has an advantage in static scenes, the exact trade-offs and optimal use cases for different scene dynamics could be further explored.
  • The performance gain of C2B over pixel-wise coded exposure is noted as "marginal" for dynamic scenes, suggesting that for highly dynamic scenarios, the benefits might not justify the increased complexity.
  • Future Work:
    • The authors suggest that jointly learning the coded mask and the reconstruction algorithm can lead to improved results, indicating that further exploration in this area could be beneficial.
  • The framework's ability to handle different coded exposure inputs opens avenues for evaluating new or custom coded exposure designs.
  • Further investigation into the specific types of static regions where C2B offers the most significant advantage could be a direction for future research.

Summary updated

BibTeX
@inproceedings{shedligeri2021unified,
  author = {Prasan Shedligeri and Anupama S and Kaushik Mitra},
  title = {{A Unified Framework for Compressive Video Recovery From Coded Exposure Techniques}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure