Towards Real World HDRTV Reconstruction: A Data Synthesis-based Approach

Zhen Cheng, Tao Wang, Yong Li, Fenglong Song, Chang Chen, Zhiwei Xiong

TL;DR

This paper tackles the challenge of training HDRTV reconstruction models by proposing a novel data synthesis approach. Instead of relying on traditional, imperfect tone mapping operators (TMOs) to create training data, they introduce a learning-based method that better captures the nuances of real-world SDR-to-HDR transformations, leading to significantly improved reconstruction performance on real-world data.

What Problem Does This Paper Address?

High Dynamic Range Television (HDRTV) offers a richer visual experience with wider brightness and color ranges compared to standard dynamic range television (SDRTV). However, most available content is still in SDR. Reconstructing HDRTV from SDRTV is an "ill-posed" problem, meaning there isn't a unique solution. Deep learning models have shown promise, but they require large datasets of paired SDRTV-HDRTV images for training. Creating these pairs is difficult. Existing methods often synthesize these pairs by applying traditional Tone Mapping Operators (TMOs) to HDR content to simulate the SDR degradation. The problem is that these TMOs, while good at compressing dynamic range, often fail to accurately model real-world SDR degradation. They can over-preserve details from the HDR source that wouldn't appear in a real SDR image, introduce color biases, and create artifacts. This leads to reconstruction models that perform poorly when applied to actual real-world SDRTV content.

Methods

The core contribution is a learning-based data synthesis approach for generating realistic SDRTV-HDRTV pairs. This approach integrates "tone mapping priors" into both the network architecture and the loss functions.

  • Conditioned Two-Stream Network: The synthesis network has two parallel streams:
  • Global Mapping Stream: Mimics global TMOs for overall color and brightness adjustments.
  • Local Adjustment Stream: Employs "highlight-aware convolutions" to better handle extreme-light regions, a common challenge in SDR.
  • Conditioning: The network is conditioned by results from several prior TMOs. These prior TMO outputs are processed to create a "condition vector" that guides the synthesis process, ensuring it learns from the strengths of different TMOs without being limited by their individual weaknesses.
  • Hybrid Tone Mapping Prior (HTMP) Loss: A novel loss function is introduced to train the synthesis network. It's designed to ensure the synthesized SDRTVs have:
  • Globally compressed dynamic range.

    • Accurate color gamut.
    • Lost details at extreme-light regions (crucial for realism). This loss function divides the image into high-light, mid-light, and low-light regions, applying different loss objectives to each:
  • High-light loss: Encourages saturation.

  • Low-light loss: Uses a simple linear compression as a supervisor.
  • Mid-light loss: A more complex combination of global color mapping, a µ-law function (for perceptual compression), and a Youtube TMO (for vividness), aiming for a balance of compression and detail preservation.
  • Adversarial Loss: To further enhance realism, an adversarial loss is employed, using a discriminator to distinguish between synthesized and real-world SDRTVs.

Datasets

  • Training Data Synthesis:
    • Input HDRTVs: 3679 HDRTVs from public datasets (BT.2020 with PQ EOTF).
  • Conditioning TMOs: Various traditional and learning-based TMOs were used to generate guidance signals.
  • Real-world SDRTV dataset (for discriminator): 3603 4K SDRTVs from public datasets.
  • Evaluation Datasets:
    • RealHDRTV (Labeled): A newly collected dataset of 93 SDRTV-HDRTV pairs captured with a smartphone in controlled indoor/static scenes, with careful alignment to ensure high quality.
  • Stest (Unlabeled): 25 SDRTVs from the synthesized dataset, used for qualitative evaluation and user studies.

Key Insights and Ideas

  • TMOs are insufficient for realistic SDR degradation modeling: The paper strongly argues that traditional TMOs, by design, try to preserve too much information, leading to synthesized SDR data that doesn't accurately reflect real-world SDR content. This is a fundamental limitation for training robust HDRTV reconstruction models.
  • Learning from multiple TMO "priors" is key: Instead of picking one TMO, the approach leverages the strengths of multiple TMOs as guidance signals for a learning-based synthesis network. This allows the network to learn a more nuanced and comprehensive model of degradation.
  • Region-aware loss functions are crucial: Different image regions (high-light, mid-light, low-light) have distinct degradation characteristics. A specialized loss function that treats these regions differently is essential for generating realistic SDR content.
  • Data synthesis directly impacts reconstruction performance: The paper demonstrates a clear link between the quality of synthesized training data and the generalization ability of HDRTV reconstruction models on real-world data.

Key Results

  • Improved Generalization: HDRTV reconstruction networks trained on data synthesized by the proposed method significantly outperform those trained on data from traditional TMOs across various metrics (PSNR, SSIM, HDR-VDP-3, SR-SIM, ∆EITP) on both labeled (RealHDRTV) and unlabeled (Stest) real-world inference datasets.
  • Quantitative Gains: For example, when training the HDRTVNet-AGCM network, the proposed method achieved gains of 2.60dB in PSNR, 0.014 in SR-SIM, and 6.7 in ∆EITP compared to the best-performing TMO baseline (Hable).
  • Qualitative Improvements: Visual comparisons show that reconstructions from models trained with the proposed method exhibit fewer artifacts, better color accuracy, and more realistic detail preservation, especially in challenging high-light and low-light regions.
  • Synthesized Data Quality: The synthesized SDRTVs themselves were shown to be of higher quality (closer to ground truth SDRTVs) than those generated by state-of-the-art TMOs, with better fidelity and color accuracy.

Limitations and Future Work

  • Limitations of TMOs: The paper extensively details the limitations of existing TMOs for data synthesis, which is the primary motivation for their work.
  • Future Work: The authors suggest that their framework could be extended to other low-level vision tasks by integrating degradation priors into learning frameworks. They also mention a desire to explore their framework on the MindSpore computing framework.

Summary updated

Abstract

Existing deep learning based HDRTV reconstruction methods assume one kind of tone mapping operators (TMOs) as the degradation procedure to synthesize SDRTV-HDRTV pairs for supervised training. In this paper, we argue that, although traditional TMOs exploit efficient dynamic range compression priors, they have several drawbacks on modeling the realistic degradation: information over-preservation, color bias and possible artifacts, making the trained reconstruction networks hard to generalize well to real-world cases. To solve this problem, we propose a learning-based data synthesis approach to learn the properties of real-world SDRTVs by integrating several tone mapping priors into both network structures and loss functions. In specific, we design a conditioned two-stream network with prior tone mapping results as a guidance to synthesize SDRTVs by both global and local transformations. To train the data synthesis network, we form a novel self-supervised content loss to constraint different aspects of the synthesized SDRTVs at regions with different brightness distributions and an adversarial loss to emphasize the details to be more realistic. To validate the effectiveness of our approach, we synthesize SDRTV-HDRTV pairs with our method and use them to train several HDRTV reconstruction networks. Then we collect two inference datasets containing both labeled and unlabeled real-world SDRTVs, respectively. Experimental results demonstrate that, the networks trained with our synthesized data generalize significantly better to these two real-world datasets than existing solutions.
BibTeX
@inproceedings{cheng2022towards,
  author = {Zhen Cheng and Tao Wang and Yong Li and Fenglong Song and Chang Chen and Zhiwei Xiong},
  title = {{Towards Real World HDRTV Reconstruction: A Data Synthesis-based Approach}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-19800-7\_12},
}
Figure