Towards Real-World HDR Video Reconstruction: A Large-Scale Benchmark Dataset and A Two-Stage Alignment Network

Yong Shu, Liquan Shen, Xiangyu Hu, Mengyao Li, Zihao Zhou

TL;DR

Researchers from Shanghai University have introduced Real-HDRV, the largest real-world dataset for HDR video reconstruction, alongside a novel two-stage alignment network. By training on diverse, real-world data and separating global camera motion from local object motion, their approach significantly reduces ghosting artifacts and recovers superior detail in over-exposed regions compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

High Dynamic Range (HDR) video offers a vastly superior visual experience, but most consumer cameras can only capture Low Dynamic Range (LDR) footage. A common workaround is to capture sequences with alternating exposures (e.g., one dark frame, one bright frame) and reconstruct HDR video computationally.

However, two major hurdles remain:

  1. Data Scarcity: Most existing models are trained on synthetic datasets. These models often fail in the real world because synthetic noise and saturation models don't accurately reflect real-world sensor behavior.
  2. Alignment Artifacts: Merging frames with different exposures is difficult when there is motion. Inaccurate alignment leads to "ghosting"—blurry, double-image artifacts that ruin video quality.

Methods

The authors propose a two-stage strategy to tackle the alignment problem, implemented in an end-to-end network.

  • Stage 1: Global Alignment Module (GAM): Instead of jumping straight to complex pixel-wise flow, the GAM first estimates global offsets (translation, rotation, scale, perspective) using 8-degree-of-freedom bases. This compensates for camera shake and movement, simplifying the task for the next stage.
  • Stage 2: Local Alignment Module (LAM): This module handles object motion. It uses a coarse-to-fine pyramid structure and Adaptive Separable Convolution (ASConv). ASConv allows the network to flexibly integrate information from neighboring frames to fill in missing details in saturated or noisy areas of the reference frame.
  • Fusion & Reconstruction: Finally, a fusion module uses learned masks to suppress harmful features (like noise in dark areas or clipped highlights) before a series of residual blocks reconstructs the final HDR frame.
Figure 3. The architecture of our proposed network.

Datasets

The core of this work is the Real-HDRV dataset.

  • Scale: 500 LDR-HDR video pairs, totaling ~28,000 LDR frames and 4,000 high-quality HDR labels.
  • Diversity: Covers indoor/outdoor and daytime/nighttime scenes with various motion patterns (global, local, and full motion).
  • Capture Method: To get ground-truth HDR for dynamic scenes, the authors used a high-speed camera (40 fps) to capture 7-exposure stacks for static scenes, then manually introduced controlled motion between these stacks. This ensures the HDR labels are of much higher quality than those generated from just two exposures.

The authors also evaluated their models on the Chen21 dataset to test generalization.

Key Insights and Ideas

  • Real Data is King: The paper demonstrates a massive performance gap between models trained on synthetic vs. real data. Models trained on Real-HDRV recover significantly more detail in over-exposed regions (like the sun in a sunset) where synthetic models produce flat, discolored blobs.
  • Divide and Conquer Alignment: By separating global camera motion from local object motion, the network avoids the common pitfalls of pure optical flow-based methods, which often struggle with the drastic appearance changes between dark and bright frames.
  • Flexible Feature Integration: The use of ASConv in the LAM stage acts as a smart "patching" mechanism, effectively borrowing valid pixels from neighboring frames to reconstruct areas that are completely lost to saturation in the current frame.
Figure 1. Row 1 shows a real-world sample from the Chen21 dataset[4]. Row 2-3 show the HDR frames reconstructed by mod- els trained on the synthetic dataset [4] and our Real-HDRV, re- spectively. Obviously, models trained on our dataset are able to recover more and better details of the over-exposed regions.

Key Results

  • Quantitative Superiority: The proposed method achieved the best average performance across all six evaluation metrics (including PSNR-µ, SSIM-µ, and HDR-VDP-2) on the Real-HDRV dataset.
  • Generalization: When evaluated on the external Chen21 dataset, models trained on Real-HDRV outperformed the same models trained on synthetic data by over 1 dB in PSNR-µ.
  • Efficiency: Despite its two-stage approach, the model maintains a competitive balance between parameter count (5.98M) and execution time (0.34s for a 1500x1000 frame), making it more practical than several heavier SOTA architectures.

Limitations and Future Work

While the dataset is a major step forward, the authors note that capturing real-world dynamic HDR ground truth remains "extremely challenging." Their method of manually creating motion between static stacks is a clever proxy, but future work might explore even more complex, naturally dynamic scenes. Additionally, while the network is designed for two alternating exposures, the authors mention it can be extended to three or more, which could be a promising direction for professional-grade video applications.

Summary updated

Abstract

As an important and practical way to obtain high dynamic range (HDR) video, HDR video reconstruction from sequences with alternating exposures is still less explored, mainly due to the lack of large-scale real-world datasets. Existing methods are mostly trained on synthetic datasets, which perform poorly in real scenes. In this work, to facilitate the development of real-world HDR video reconstruction, we present Real-HDRV, a large-scale real-world benchmark dataset for HDR video reconstruction, featuring various scenes, diverse motion patterns, and high-quality labels. Specifically, our dataset contains 500 LDRs-HDRs video pairs, comprising about 28,000 LDR frames and 4,000 HDR labels, covering daytime, nighttime, indoor, and outdoor scenes. To our best knowledge, our dataset is the largest real-world HDR video reconstruction dataset. Correspondingly, we propose an end-to-end network for HDR video reconstruction, where a novel two-stage strategy is designed to perform alignment sequentially. Specifically, the first stage performs global alignment with the adaptively estimated global offsets, reducing the difficulty of subsequent alignment. The second stage implicitly performs local alignment in a coarse-to-fine manner at the feature level using the adaptive separable convolution. Extensive experiments demonstrate that: (1) models trained on our dataset can achieve better performance on real scenes than those trained on synthetic datasets; (2) our method outperforms previous state-of-the-art methods. Our dataset is available at https://github.com/yungsyu99/Real-HDRV.
BibTeX
@inproceedings{shu2024towards,
  author = {Yong Shu and Liquan Shen and Xiangyu Hu and Mengyao Li and Zihao Zhou},
  title = {{Towards Real-World HDR Video Reconstruction: A Large-Scale Benchmark Dataset and A Two-Stage Alignment Network}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure