Towards Mobile HDR Video

Tássio Knop de Castro, Alexandre Chapiro, Marcelo Cicconet, Luiz VelhoEurographics (Areas Papers) 2011 · DOI

TL;DR

This paper presents a novel, computationally efficient method for creating High Dynamic Range (HDR) video on mobile devices. By leveraging histogram-based techniques for photometric calibration and image registration, the approach avoids expensive optical flow computations, enabling real-time HDR video reconstruction directly on resource-constrained smartphones.

What Problem Does This Paper Address?

Capturing High Dynamic Range (HDR) images and videos allows for scenes with extreme variations in brightness (e.g., a bright window with a dark interior) to be rendered with detail in both the highlights and shadows. While HDR still image processing has become relatively mature, HDR video presents a greater challenge. Traditional methods often rely on computationally intensive optical flow algorithms to align frames with different exposures, which can be too slow for real-time processing on mobile devices. Furthermore, existing HDR video solutions often require specialized, expensive hardware not available to the general consumer. This paper aims to bridge this gap by developing an HDR video reconstruction method that is both computationally efficient and suitable for implementation on readily available mobile platforms.

Methods

The proposed method breaks down HDR video reconstruction into three main steps, all based on histogram analysis:

  1. Photometric Calibration: This step estimates the camera's response function. Instead of relying on pixel-to-pixel correspondences, it uses the monotonic relationship between pixel values across differently exposed frames. The intuition is that the brightest pixels in one frame will correspond to the brightest pixels in another, even if their exact values differ due to exposure.
  2. Histogram-Based Registration: To avoid reducing the video frame rate, an HDR frame is generated for each input frame. This step addresses camera motion by performing a multi-resolution alignment of images using histogram-based techniques. Specifically, it constructs Median Threshold Bitmap (MTB) images and tests for offsets, making it less computationally demanding than optical flow.
  3. Radiance Map Reconstruction: This final step reconstructs the HDR frame by considering the variance of radiance values for corresponding pixels across the aligned, differently exposed images. Pixels with high variance (indicating scene movement or "ghosting") are weighted towards the optimally exposed frame, while low variance pixels are weighted towards the more stable exposures.

Datasets

Not discussed in the paper. The authors implemented their method on a Nokia N900 smartphone and evaluated it using captured video sequences.

Key Insights and Ideas

  • Histogram-based processing is key for mobile HDR: The core insight is that histogram analysis offers a computationally cheaper and more robust alternative to traditional optical flow for HDR video tasks like calibration and registration, making it ideal for mobile devices.
  • Maintaining frame rate is crucial: The method's design prioritizes generating an HDR frame for each input frame, thus preserving the original video frame rate, a significant advantage over methods that might reduce temporal resolution.
  • Ghost removal via variance analysis: Analyzing the variance of radiance values across aligned frames provides an effective way to mitigate ghosting artifacts caused by scene motion, a common problem in multi-exposure HDR reconstruction.
  • FCam API unlocks mobile HDR potential: The availability of low-level camera control through APIs like FCam is highlighted as a critical enabler for developing advanced computational photography applications on mobile platforms.

Key Results

The paper demonstrates a functional HDR video reconstruction pipeline implemented on a Nokia N900 smartphone. The method successfully generates HDR video sequences from captured frames with varying exposures. Qualitative results are presented in Figures 2 and 3, showing the captured frames and their corresponding tone-mapped HDR outputs for both outdoor and indoor scenes with camera and object motion. The authors emphasize that their method is suitable for devices with less powerful processors due to its histogram-based approach.

Limitations and Future Work

The current implementation requires transferring captured frames to a desktop computer for processing. A primary future work direction is to enable the entire HDR video reconstruction process to run directly on the mobile device, making it a fully self-contained solution. The authors also suggest exploring other pixel correspondence methods, such as optical flow, for potentially further enhancing result quality, and investigating other tone enhancement techniques. A significant challenge acknowledged is the limited memory on mobile devices, which restricts the length of captured videos. A potential solution proposed is using a low-priority thread to save images to disk while the program is running. The authors also believe that access to low-level hardware parameters opens doors for user-aided capture, intelligent capture processes, and retrieval of geometric properties.

Summary updated

BibTeX
@inproceedings{castro2011towards,
  author = {Tássio Knop de Castro and Alexandre Chapiro and Marcelo Cicconet and Luiz Velho},
  title = {{Towards Mobile HDR Video}},
  booktitle = {Eurographics (Areas Papers)},
  year = {2011},
  doi = {10.2312/eg2011/areas/075-076},
}
Figure