Assessing UHD Image Quality from Aesthetics, Distortions, and Saliency

Wei Sun, Weixia Zhang, Yuqin Cao, Linhan Cao, Jun Jia, Zijian Chen, Zicheng Zhang, Xiongkuo Min, Guangtao Zhai

Abstract

As Ultra-High-Definition (UHD) content becomes the standard for digital media, traditional Image Quality Assessment (IQA) methods face significant challenges related to computational complexity and information loss. This paper presents a multi-branch deep learning architecture that evaluates UHD images across three distinct dimensions: global aesthetics, local technical distortions, and salient content. By utilizing a Swin Transformer backbone and an innovative grid mini-patch sampling strategy, the proposed model achieves state-of-the-art performance while maintaining high efficiency. The approach secured first prize in the ECCV AIM 2024 UHD-IQA Challenge, demonstrating its effectiveness in balancing accuracy with practical computational requirements.

Introduction: The "Resolution Wall"

With the proliferation of 4K and 8K displays, the demand for high-quality UHD content has never been higher. However, assessing the quality of these images automatically presents a "resolution wall." Standard Deep Neural Networks (DNNs) are often ill-equipped to handle UHD content due to two primary constraints:

  1. Computational Complexity: Processing full-resolution 4K images requires immense memory and power, making real-time or large-scale assessment difficult.
  2. Information Loss: To manage complexity, many models either downsample images—which destroys high-frequency textures like sharpness and fine noise—or use random cropping, which loses the global artistic context and composition.

To address these issues, we propose a multi-perspective model that "sees" both the broad aesthetic intent and the fine-grained technical execution without requiring excessive hardware resources.

Methodology

The core of our approach is a three-branch architecture built on the Swin Transformer Tiny backbone. Each branch is designed to process a specific "view" of the image, optimized for a different dimension of quality.

1. Aesthetic Branch

This branch focuses on the "forest" rather than the "trees." The UHD image is aggressively downsampled to 480x480 pixels. While this process removes fine textures, it preserves the global features necessary to judge color style, lighting, and overall composition. This allows the model to assess the artistic value of the image efficiently.

2. Distortion Branch

To capture technical flaws such as compression artifacts, sensor noise, or motion blur, we utilize a grid mini-patch sampling strategy. The image is divided into a 15x15 grid, and a 32x32 patch is extracted from each cell. These patches are then stitched together into a 480x480 "fragment image." This technique allows the model to inspect local textures at their original resolution without the memory overhead of processing the entire 4K frame.

3. Salient Content Branch

Human perception is naturally biased toward the subject of an image. Leveraging the common "center bias" in photography, this branch crops a 480x480 patch from the center of the image. This ensures the model prioritizes the quality of the most important visual content, where viewers are most likely to notice defects.

Feature Fusion and Training

The features extracted from these three branches are concatenated and passed through a Multi-Layer Perceptron (MLP) to generate a final quality score. The model is trained using a combination of Mean Square Error (MSE) for absolute score accuracy and Fidelity Loss to improve the model's ability to rank the relative quality between image pairs.

Experimental Results

Performance and Accuracy

The model was evaluated on the UHD-IQA dataset, which contains 6,073 4K images. It achieved a Spearman Rank Correlation (SRCC) of 0.846, significantly outperforming existing state-of-the-art models such as QualiCLIP (0.770) and HyperIQA (0.553). This performance earned the model 1st place in the AIM 2024 UHD-IQA Challenge.

Computational Efficiency

Efficiency was a primary design goal. While many top-tier UHD-IQA models require over 200G MACs (Multiply-Accumulate operations), our model operates at just 43.5G MACs. On an NVIDIA RTX 3090 GPU, the model can score an image in approximately 0.068 seconds, making it 4 to 20 times more efficient than its competitors.

Discussion and Future Work

While the model demonstrates exceptional performance, it relies on the assumption of center-weighted saliency. Future iterations could incorporate dynamic saliency detection to better handle complex compositions, such as those following the rule of thirds. Additionally, while the Swin-T backbone was chosen for its efficiency, experiments suggest that scaling to a Swin-Base architecture could yield further accuracy gains for applications where computational budgets are less constrained.

In conclusion, by decomposing image quality into aesthetics, distortion, and saliency, we provide a robust and efficient framework for the next generation of UHD content assessment.

Summary updated

BibTeX
@inproceedings{sun2024assessing,
  author = {Wei Sun and Weixia Zhang and Yuqin Cao and Linhan Cao and Jun Jia and Zijian Chen and Zicheng Zhang and Xiongkuo Min and Guangtao Zhai},
  title = {{Assessing UHD Image Quality from Aesthetics, Distortions, and Saliency}},
  booktitle = {Workshops @ ECCV},
  year = {2024},
  url = {https://openreview.net/forum?id=Nv8Fh222j9},
}
Figure