TL;DR
Researchers from Shanghai University have introduced Real-HDRV, the largest real-world dataset for HDR video reconstruction, alongside a novel two-stage alignment network. By training on diverse, real-world data and separating global camera motion from local object motion, their approach significantly reduces ghosting artifacts and recovers superior detail in over-exposed regions compared to existing state-of-the-art methods.
What Problem Does This Paper Address?
High Dynamic Range (HDR) video offers a vastly superior visual experience, but most consumer cameras can only capture Low Dynamic Range (LDR) footage. A common workaround is to capture sequences with alternating exposures (e.g., one dark frame, one bright frame) and reconstruct HDR video computationally.
However, two major hurdles remain:
- Data Scarcity: Most existing models are trained on synthetic datasets. These models often fail in the real world because synthetic noise and saturation models don't accurately reflect real-world sensor behavior.
- Alignment Artifacts: Merging frames with different exposures is difficult when there is motion. Inaccurate alignment leads to "ghosting"—blurry, double-image artifacts that ruin video quality.
Methods
The authors propose a two-stage strategy to tackle the alignment problem, implemented in an end-to-end network.
- Stage 1: Global Alignment Module (GAM): Instead of jumping straight to complex pixel-wise flow, the GAM first estimates global offsets (translation, rotation, scale, perspective) using 8-degree-of-freedom bases. This compensates for camera shake and movement, simplifying the task for the next stage.
- Stage 2: Local Alignment Module (LAM): This module handles object motion. It uses a coarse-to-fine pyramid structure and Adaptive Separable Convolution (ASConv). ASConv allows the network to flexibly integrate information from neighboring frames to fill in missing details in saturated or noisy areas of the reference frame.
- Fusion & Reconstruction: Finally, a fusion module uses learned masks to suppress harmful features (like noise in dark areas or clipped highlights) before a series of residual blocks reconstructs the final HDR frame.
Datasets
The core of this work is the Real-HDRV dataset.
- Scale: 500 LDR-HDR video pairs, totaling ~28,000 LDR frames and 4,000 high-quality HDR labels.
- Diversity: Covers indoor/outdoor and daytime/nighttime scenes with various motion patterns (global, local, and full motion).
- Capture Method: To get ground-truth HDR for dynamic scenes, the authors used a high-speed camera (40 fps) to capture 7-exposure stacks for static scenes, then manually introduced controlled motion between these stacks. This ensures the HDR labels are of much higher quality than those generated from just two exposures.
The authors also evaluated their models on the Chen21 dataset to test generalization.
Key Insights and Ideas
- Real Data is King: The paper demonstrates a massive performance gap between models trained on synthetic vs. real data. Models trained on Real-HDRV recover significantly more detail in over-exposed regions (like the sun in a sunset) where synthetic models produce flat, discolored blobs.
- Divide and Conquer Alignment: By separating global camera motion from local object motion, the network avoids the common pitfalls of pure optical flow-based methods, which often struggle with the drastic appearance changes between dark and bright frames.
- Flexible Feature Integration: The use of ASConv in the LAM stage acts as a smart "patching" mechanism, effectively borrowing valid pixels from neighboring frames to reconstruct areas that are completely lost to saturation in the current frame.
Key Results
- Quantitative Superiority: The proposed method achieved the best average performance across all six evaluation metrics (including PSNR-µ, SSIM-µ, and HDR-VDP-2) on the Real-HDRV dataset.
- Generalization: When evaluated on the external Chen21 dataset, models trained on Real-HDRV outperformed the same models trained on synthetic data by over 1 dB in PSNR-µ.
- Efficiency: Despite its two-stage approach, the model maintains a competitive balance between parameter count (5.98M) and execution time (0.34s for a 1500x1000 frame), making it more practical than several heavier SOTA architectures.
Limitations and Future Work
While the dataset is a major step forward, the authors note that capturing real-world dynamic HDR ground truth remains "extremely challenging." Their method of manually creating motion between static stacks is a clever proxy, but future work might explore even more complex, naturally dynamic scenes. Additionally, while the network is designed for two alternating exposures, the authors mention it can be extended to three or more, which could be a promising direction for professional-grade video applications.
Summary updated