A Style Transfer-Based Fast Image Quality Assessment Method for Image Sensors

Weizhi Xian, Bin Chen, Jielu Yan, Xuekai Wei, Kunyin Guo, Bin Fang, Mingliang ZhouSensors 2025 · DOI

TL;DR

This paper introduces SCIQA, a novel image quality assessment method that uses principles from style transfer to better capture both the content and stylistic elements of an image. By combining content and style distances derived from deep neural networks, SCIQA achieves state-of-the-art accuracy and interpretability while remaining computationally efficient, making it suitable for real-world applications.

What Problem Does This Paper Address?

Accurate image quality assessment (IQA) is crucial for many applications, from medical imaging to autonomous systems, as it directly impacts the reliability and performance of these technologies. However, existing IQA methods face several challenges:

  • Generalization: Many methods struggle to perform well across diverse image content and distortion types.
  • Real-time processing: The demand for fast IQA in applications like video streaming necessitates efficient algorithms.
  • Interpretability: The "black-box" nature of many deep learning models makes it hard to understand why a certain quality score is assigned, hindering system improvement.

Traditional methods often rely on handcrafted features that don't fully capture human perception, while deep learning methods, though powerful, can be computationally expensive and difficult to interpret. This paper aims to bridge this gap by developing a method that is accurate, interpretable, and efficient.

Methods

The core of SCIQA is a novel full-reference IQA framework inspired by style transfer. It involves three main steps:

  1. Hierarchical Feature Extraction: A pretrained VGG-16 network is used to extract deep features from both the reference and distorted images. This process captures a hierarchy of visual information, from low-level textures to high-level semantic content.
  2. Dual-Component Perceptual Dissimilarity Metric: SCIQA defines a new metric that quantifies the difference between images based on two components:
  3. Content Distance: This measures structural and semantic discrepancies by calculating the direct distance between feature maps extracted by the VGG-16 network.
  4. Style Distance: This captures textural and stylistic differences by computing the distance between Gram matrices of the feature maps. Gram matrices represent feature correlations, which are indicative of texture and style.
  5. Convex Optimization for Parameter Estimation: The model's parameters (weights for content and style distances, and a bias term) are learned by formulating a convex optimization problem. This approach ensures stable numerical solutions and computational efficiency, allowing the model to learn from subjective quality assessments.

Datasets

The paper evaluates SCIQA on five widely recognized subjective quality assessment databases:

  • LIVE: Contains 29 reference images and 779 distorted images with five types of distortions (JPEG2000, JPEG, white noise, Gaussian blur, fast fading).
  • CSIQ: Features 30 reference images and 866 distorted images with six distortion types.
  • TID2013: Includes 25 reference images and 3000 distorted images with 24 distortion categories.
  • KADID-10k: A large dataset with 81 reference images and over 10,000 distorted images covering 25 distortion types. This dataset was used for training.
  • PIPAL: A comprehensive benchmark with 250 reference images and over 25,000 distorted images, covering 40 distortion types, including GAN-generated artifacts. This dataset was used for evaluating generalization.

Key Insights and Ideas

  • Style Transfer as an IQA Framework: The paper ingeniously adapts the principles of style transfer, which separates and recombines content and style, to the problem of IQA. This provides a novel perspective on how to measure image quality by considering both what an image depicts and how it depicts it.
  • Dual-Component Metric is Crucial: The study demonstrates that both content and style are vital for accurate IQA. While content fidelity is fundamental, style consistency refines predictions and better aligns with human perception, especially for complex distortions.
  • Interpretability through Perceptual Distances: By explicitly defining content and style distances, SCIQA offers a degree of interpretability. The method can identify whether a perceived quality difference is due to structural changes or stylistic variations.
  • Pretrained Networks are Key: Using a pretrained VGG-16 network significantly boosts performance, highlighting the value of leveraging features learned from large, diverse datasets for IQA tasks, especially when IQA-specific datasets are smaller.

Key Results

  • State-of-the-Art Accuracy: SCIQA outperforms existing traditional and deep learning-based FR-IQA methods on several benchmark datasets. For instance, on LIVE, CSIQ, and TID2013, it achieves PLCC scores of 0.956, 0.941, and 0.895, respectively, surpassing methods like SSIM and DISTS.
  • Robust Generalization: On the large-scale PIPAL dataset, SCIQA achieves a high SROCC of 0.702, demonstrating its strong ability to generalize across diverse distortions and content.
  • Computational Efficiency: SCIQA achieves a superior inference speed (23.6 FPS) compared to methods like SSHMPQA (0.76 FPS) and DISTS (12.3 FPS), making it practical for real-time applications.
  • Interpretability Demonstrated: An interpretability experiment shows that SCIQA correctly identifies images with similar semantic content as being closer in quality, and also captures stylistic similarities.
  • Robustness to Transformations: Subjective consistency experiments show SCIQA is more robust to geometric transformations (like rotation) and better at penalizing compression artifacts compared to SSIM.

Limitations and Future Work

  • Reliance on Pretrained Network: SCIQA relies on a VGG-16 network pretrained on ImageNet. While effective, this might limit its adaptability to highly specialized domains or datasets with unique characteristics.
  • "Black-Box" Aspects: Despite efforts towards interpretability, the deep feature extraction process might still retain some "black-box" behavior, making it challenging to fully explain all model decisions.
  • Future Directions:
    • Video Quality Assessment: Extending SCIQA to video by incorporating temporal feature analysis.
  • Real-time Optimization: Further optimizing SCIQA for even faster performance through lightweight architectures or hardware acceleration.
  • Domain Adaptation: Fine-tuning SCIQA for specific domains like medical imaging or satellite imagery.
  • Visualization Techniques: Developing better visualization methods for deep features to improve interpretability.

Summary updated

Abstract

Accurate image quality evaluation is essential for optimizing sensor performance and enhancing the fidelity of visual data. The concept of "image style" encompasses the overall visual characteristics of an image, including elements such as colors, textures, shapes, lines, strokes, and other visual components. In this paper, we propose a novel full-reference image quality assessment (FR-IQA) method that leverages the principles of style transfer, which we call style- and content-based IQA (SCIQA). Our approach consists of three main steps. First, we employ a deep convolutional neural network (CNN) to decompose and represent images in the deep domain, capturing both low-level and high-level features. Second, we define a comprehensive deep perceptual distance metric between two images, taking into account both image content and style. This metric combines traditional content-based measures with style-based measures inspired by recent advances in neural style transfer. Finally, we formulate a perceptual optimization problem to determine the optimal parameters for the SCIQA model, which we solve via a convex optimization approach. Experimental results across multiple benchmark datasets (LIVE, CSIQ, TID2013, KADID-10k, and PIPAL) demonstrate that SCIQA outperforms state-of-the-art FR-IQA methods. Specifically, SCIQA achieves Pearson linear correlation coefficients (PLCC) of 0.956, 0.941, and 0.895 on the LIVE, CSIQ, and TID2013 datasets, respectively, outperforming traditional methods such as SSIM (PLCC: 0.847, 0.852, 0.665) and deep learning-based methods such as DISTS (PLCC: 0.924, 0.919, 0.855). The proposed method also demonstrates robust generalizability on the large-scale PIPAL dataset, achieving an SROCC of 0.702. Furthermore, SCIQA exhibits strong interpretability, exceptional prediction accuracy, and low computational complexity, making it a practical tool for real-world applications.
BibTeX
@inproceedings{xian2025style,
  author = {Weizhi Xian and Bin Chen and Jielu Yan and Xuekai Wei and Kunyin Guo and Bin Fang and Mingliang Zhou},
  title = {{A Style Transfer-Based Fast Image Quality Assessment Method for Image Sensors}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25165121},
}
Figure