An Analysis and Implementation of the HDR+ Burst Denoising Method

Antoine Monod, Julie Delon, Thomas Veit

TL;DR

This paper dissects Google's HDR+ burst denoising algorithm, a key component in smartphone photography for producing high-quality images from a burst of raw frames. The authors provide a detailed analysis and an open-source Python implementation, offering valuable insights into its inner workings and practical considerations for achieving natural-looking results.

What Problem Does This Paper Address?

Modern smartphones, with their small sensors, struggle with image noise and limited dynamic range, especially in low-light conditions. Computational photography techniques, like Google's HDR+ pipeline, aim to overcome these limitations by processing multiple raw images (a burst) to create a single, superior image. While HDR+ is widely used and praised for its natural output, its internal mechanisms and parameter impacts have not been extensively detailed in academic literature. This paper aims to fill that gap by analyzing the core denoising algorithm of HDR+, explaining its architecture, and providing a reproducible implementation.

Methods

The core of the HDR+ pipeline analyzed here is a raw burst alignment and merging algorithm. This involves:

  • Bayer Tile-based Alignment: The reference image is divided into tiles. Motion estimation is performed between the reference tile and corresponding tiles in other burst images. This is done using a multi-scale pyramid approach (coarse-to-fine) to efficiently search for the best alignment. Subpixel alignment is then applied to refine the motion vectors.
  • Pairwise Wiener Temporal Denoising: Once aligned, tiles from different frames are merged. This is achieved by comparing each alternate tile to the reference tile and using a Wiener filter in the frequency domain (2D DFT) to combine them. This process is performed independently for each color channel.
  • Wiener Spatial Denoising: A secondary spatial denoising step is applied in the frequency domain to further reduce residual noise, using a shrinkage operator similar to the temporal denoising.
  • Overlapped Tiles and Raised Cosine Window: To ensure smooth transitions between merged tiles and mitigate DFT artifacts, tiles are overlapped by half their dimension, and a modified raised cosine window is used for blending.

The paper also discusses the Google's full pipeline which includes additional steps like black level subtraction, lens shading correction, white balance, demosaicing, color correction, tone mapping, dehazing, sharpening, and JPEG compression. The authors also present their simplified pipeline for comparison.

Datasets

  • HDR+ Burst Photography Dataset: This is the primary dataset used, comprising 3640 raw bursts (28461 images) captured by various smartphones. A curated subset of 153 bursts was used for detailed analysis.
  • GoPro HERO 8 Black and HERO 9 Black bursts: Used to test the implementation on a different camera platform.

Key Insights and Ideas

  • Naturalness over Metrics: HDR+ prioritizes producing visually pleasing, natural images over maximizing standard denoising metrics, a key design philosophy for smartphone cameras.
  • Robustness to Alignment Errors: The merging algorithm is designed to be somewhat robust to imperfect alignment, which is crucial given the computational constraints of mobile devices.
  • Frequency Domain Processing: The use of 2D DFT for both temporal and spatial denoising is a computationally efficient way to process image tiles, especially when already in frequency space.
  • Wiener Filter for Temporal Denoising: The pairwise Wiener filter effectively leverages temporal information to reduce noise by weighting contributions from different frames based on their difference and estimated noise levels.
  • Overlapping Tiles for Seamlessness: The overlapping tile strategy with a raised cosine window is critical for avoiding visible seams and artifacts at tile boundaries, ensuring a coherent final image.

Key Results

  • The authors provide a publicly available open-source Python implementation of the HDR+ burst denoising algorithm, along with an interactive demo.
  • Their implementation achieves a +7dB PSNR improvement on average compared to the reference image when compared to Google's 2017 results, suggesting effective alignment and temporal denoising.
  • Visual comparisons show that their implementation significantly reduces noise, though Google's full pipeline (with additional steps like spatial denoising and tone mapping) produces more refined results.
  • The paper demonstrates that the core alignment and merging algorithm can be applied to GoPro camera bursts, showing noise reduction capabilities on different hardware.

Limitations and Future Work

  • Simplified Finishing Pipeline: The authors acknowledge that their simplified finishing pipeline does not match the visual quality of Google's full pipeline, which involves many more complex steps. Fully implementing these would require significant additional development.
  • Tuning Factors: The exact tuning factors used in Google's proprietary implementation are unknown, making direct replication challenging. The authors had to tune their parameters to match Google's results as closely as possible.
  • Spatial Denoising Specifics: The paper notes that the specifics of Google's spatial denoising function and its tuning are not detailed, making direct comparison difficult.
  • Computational Constraints: While the HDR+ pipeline is designed for mobile, achieving Google-level quality might require further optimization for real-time performance on mobile ISPs.
  • Future Work: The authors suggest that fully implementing the additional finishing steps would be a logical next step for achieving comparable visual quality. Further exploration of parameter tuning and its impact on different camera types is also implied.

Summary updated

Abstract

HDR+ is an image processing pipeline presented by Google in 2016. At its core lies a denoising algorithm that uses a burst of raw images to produce a single higher quality image. Since it is designed as a versatile solution for smartphone cameras, it does not necessarily aim for the maximization of standard denoising metrics, but rather for the production of natural, visually pleasing images. In this article, we specifically discuss and analyze the HDR+ burst denoising algorithm architecture and the impact of its various parameters. With this publication, we provide an open source Python implementation of the algorithm, along with an interactive demo.
BibTeX
@inproceedings{monod2021analysis,
  author = {Antoine Monod and Julie Delon and Thomas Veit},
  title = {{An Analysis and Implementation of the HDR+ Burst Denoising Method}},
  booktitle = {Image Process. Line},
  year = {2021},
  url = {https://openreview.net/forum?id=TIotqRNHCp},
}
Figure