Space-Time-Brightness Sampling Using an Adaptive Pixel-Wise Coded Exposure

Hajime Nagahara, Toshiki Sonoda, Dengyu Liu, Jinwei Gu

TL;DR

This paper introduces an adaptive pixel-wise coded exposure technique for video cameras that dynamically adjusts sampling patterns based on scene content. By intelligently allocating resources to moving regions for high temporal resolution and static regions for high dynamic range, it overcomes the traditional trade-offs in video capture, leading to more accurate reconstructions with reduced computational cost.

What Problem Does This Paper Address?

Modern digital video cameras face a fundamental challenge: balancing spatial resolution, temporal resolution, and dynamic range (brightness resolution) within limited data transmission bandwidth. Conventional cameras often make compromises, excelling in one area at the expense of others. While previous research explored non-uniform space-time sampling, like pixel-wise coded exposure, these methods used fixed, pre-defined patterns that weren't optimal for diverse scene content. This paper tackles the problem of achieving high-quality video capture across all these dimensions simultaneously, especially for dynamic scenes, by introducing an adaptive sampling strategy.

Methods

The core innovation is an adaptive space-time-brightness sampling method. This approach dynamically adjusts the pixel-wise coded exposure patterns based on scene analysis. Specifically:

  • Motion Segmentation: The system first generates a motion mask by comparing consecutive frames (f-1 and f-2). This mask identifies regions that are moving versus static.
  • Adaptive Pattern Generation:
    • Moving Regions: These are sampled using random exposure patterns to prioritize high temporal resolution and capture fast motion accurately. This leverages techniques from prior work on compressive video reconstruction.
  • Static Regions: These are sampled using a spatially varying exposure (HDR) pattern. This pattern uses a mosaic of different exposure densities across neighboring pixels to achieve high dynamic range.
  • Reconstruction: Moving and static regions are reconstructed separately using their respective sampling strategies and then integrated into a final high-resolution output image. The method aims to optimize hardware resource allocation by applying computationally intensive techniques only where needed.
  • Hardware Implementation: The authors built a prototype camera using a Liquid Crystal on Silicon (LCoS) display to dynamically update the coded exposure patterns in real-time, demonstrating the practical feasibility of their adaptive approach.

Datasets

The paper describes using simulated experiments with real video sequences captured by a high-speed camera (Point Grey GS3-U3-23S6C) with 16-bit brightness resolution as ground truth. They also conducted real experiments using their prototype camera. No new datasets were introduced.

Key Insights and Ideas

  • Scene-Aware Sampling is Key: The most significant insight is that a "one-size-fits-all" sampling strategy is suboptimal. Adapting the sampling based on whether a region is moving or static unlocks significant improvements in video quality and efficiency.
  • Intelligent Resource Allocation: By applying random coding to moving parts (prioritizing temporal resolution) and HDR coding to static parts (prioritizing brightness resolution), the system efficiently uses available bandwidth and computational resources.
  • Bridging the Gap: The adaptive approach effectively bridges the gap between methods that excel at high temporal resolution (often sacrificing spatial or brightness) and those that excel at high dynamic range (often sacrificing temporal).
  • Real-time Feasibility: The successful implementation of a prototype camera that performs adaptive coding in real-time is a crucial demonstration of the method's practical viability.

Key Results

  • Improved Reconstruction Quality: Simulations and real experiments show that the adaptive space-time-brightness sampling scheme achieves more accurate video reconstruction results and higher dynamic range compared to previous fixed-pattern methods.
  • Reduced Computational Cost: By applying compressive video reconstruction only to moving regions, the method reduces overall computational cost.
  • Prototype Demonstration: The paper presents a prototype camera that successfully captures and reconstructs video with adaptive coded exposure in real-time, showcasing the practical application of the proposed method.

Limitations and Future Work

  • Segmentation Accuracy: The effectiveness of the adaptive scheme is strongly dependent on the accuracy of the motion/static region segmentation.
  • Latency: The current implementation has a three-frame latency between motion detection and adaptive pattern application. This means sudden scene changes or motion might not be optimally captured in the initial frames.
  • Future Work: The authors suggest that improving the segmentation accuracy is a key area for future work. They also propose the ideal solution would be a specialized CMOS imager that can detect motion and apply adaptive exposure on-chip to eliminate latency.

Summary updated

BibTeX
@inproceedings{nagahara2018space,
  author = {Hajime Nagahara and Toshiki Sonoda and Dengyu Liu and Jinwei Gu},
  title = {{Space-Time-Brightness Sampling Using an Adaptive Pixel-Wise Coded Exposure}},
  booktitle = {Workshops @ CVPR},
  year = {2018},
  url = {https://openreview.net/forum?id=BJZo_C-_ZB},
}

Loading related content...

Figure