Quality Assessment for Tone-Mapped HDR Images Using Multi-Scale and Multi-Layer Information

Qin He, Dingquan Li, Tingting Jiang, Ming Jiang

Abstract

Tone mapping operators (TMOs) and multi-exposure fusion methods are essential for displaying high dynamic range (HDR) content on standard dynamic range (SDR) devices. However, these processes often introduce visual distortions. We propose a novel no-reference method for tone-mapped image quality assessment (TMIQA) that utilizes multi-scale and multi-layer features extracted from a pre-trained deep convolutional neural network (DCNN). By aggregating features across different scales and network depths, our model captures both local textures and global semantic information. Experimental results on the ESPL-LIVE HDR database demonstrate that our approach significantly outperforms existing state-of-the-art no-reference methods.

Introduction

High Dynamic Range (HDR) imaging captures a broader range of luminance than traditional formats, but most consumer displays are limited to Standard Dynamic Range (SDR). Bridging this gap requires Tone Mapping Operators (TMOs) or multi-exposure fusion techniques. While these methods make HDR content viewable, they can cause loss of detail, unnatural color shifts, or halo artifacts.

Evaluating the quality of these tone-mapped images is critical. While full-reference metrics exist, they require the original HDR source, which is often unavailable in end-user applications. This necessitates No-Reference Image Quality Assessment (NR-IQA) models that can predict perceived quality based solely on the tone-mapped image.

Proposed Methodology

The complexity of distortions in tone-mapped images requires a multi-faceted approach. Our framework leverages a pre-trained ResNet-50 model to extract rich feature representations.

The proposed TMIQA framework illustrating multi-scale representation, feature extraction, and aggregation.

1. Multi-Scale Representation

DCNNs typically have a fixed receptive field. To capture global structural distortions alongside fine-grained local artifacts, we input both the original image and a down-sampled version into the network. This dual-scale approach ensures the model "sees" the image at different levels of granularity.

2. Multi-Layer Feature Extraction

Instead of relying only on the final layer, we extract features from multiple stages of the ResNet-50 architecture:

  • Low-level features: Capture fundamental textures and edges.
  • Mid-to-High-level features: Capture complex shapes and semantic scene information.

3. Feature Aggregation and Prediction

The extracted feature maps are aggregated using mean and standard deviation pooling to form a comprehensive feature vector. Given the high dimensionality (4,608) relative to the dataset size, we employ Partial Least Squares Regression (PLSR) to map these features to a final quality score. PLSR effectively handles high-dimensional data and prevents overfitting.

Experimental Setup and Datasets

We evaluated our model on the ESPL-LIVE HDR Database, the largest public dataset for TMIQA. It contains 1,811 images generated from 11 different processing pipelines.

Examples of tone-mapped images from the ESPL-LIVE HDR database with their corresponding Mean Opinion Scores (MOS).

The database provides Mean Opinion Scores (MOS) for each image, representing the collective judgment of over 5,000 human participants. This serves as the ground truth for training and testing our quality predictor.

Results and Discussion

The performance was measured using Spearman’s Rank Correlation Coefficient (SROCC) to determine how well the model's rankings align with human perception.

  • Superior Performance: Our method achieved an SROCC of 0.823, outperforming specialized NR-IQA methods like HIGRADE (0.730) and BIBQA (0.702).
  • Domain Specificity: General-purpose NR-IQA models like BRISQUE (0.418) performed poorly, highlighting that tone-mapping distortions are distinct from common noise or blur.
  • Ablation Study: Removing either the multi-scale or multi-layer components resulted in a performance drop, confirming that both are vital for accurate quality prediction.

Conclusion

By combining multi-scale inputs with multi-layer DCNN features, our proposed method provides a robust solution for assessing the quality of tone-mapped images without a reference. This approach effectively captures the diverse range of distortions introduced by TMOs, matching human visual preference more closely than previous techniques.

Summary updated

Abstract

Tone mapping operators and multi-exposure fusion methods allow us to enjoy the informative contents of high dynamic range (HDR) images with standard dynamic range devices, but also introduce distortions into HDR contents. Therefore methods are needed to evaluate tone-mapped image quality. Due to the complexity of possible distortions in a tone-mapped image, information from different scales and different levels should be considered when predicting tone-mapped image quality. So we propose a new no-reference method of tone-mapped image quality assessment based on multi-scale and multi-layer features that are extracted from a pre-trained deep convolutional neural network model. After being aggregated, the extracted features are mapped to quality predictions by regression. The proposed method is tested on the largest public database for TMIQA and compared to existing no-reference methods. The experimental results show that the proposed method achieves better performance.
BibTeX
@inproceedings{he2018quality,
  author = {Qin He and Dingquan Li and Tingting Jiang and Ming Jiang},
  title = {{Quality Assessment for Tone-Mapped HDR Images Using Multi-Scale and Multi-Layer Information}},
  booktitle = {Workshops @ ICME},
  year = {2018},
  url = {https://openreview.net/forum?id=253ONDgAVGr},
}
Figure