TL;DR
This paper introduces RichIQA, a novel approach to image quality assessment (IQA) that goes beyond traditional mean opinion scores (MOS) to leverage richer subjective quality information like opinion score distributions (DOS) and standard deviations (SOS). By employing a three-stage network inspired by human memory and a multi-label training strategy, RichIQA achieves state-of-the-art performance on challenging "in the wild" image datasets.
What Problem Does This Paper Address?
Traditional image quality assessment (IQA) models primarily rely on the Mean Opinion Score (MOS) as the sole indicator of image quality. However, MOS represents only an average perception and misses the nuances present in the full spectrum of subjective ratings. This includes the variability in how different individuals perceive quality (captured by the Standard Deviation of Opinion Scores, SOS) and the overall distribution of these opinions (DOS). For images captured "in the wild" – meaning those encountered in real-world scenarios with complex and diverse distortions – this limited information can lead to less accurate and generalizable quality assessments. The paper aims to bridge this gap by developing an IQA model that can effectively utilize this richer subjective information to better predict image quality in uncontrolled environments.
Methods
The core of RichIQA lies in two novel designs:
- Three-Stage Image Quality Prediction Network: This network leverages the power of the Convolutional Vision Transformer (CvT) for multi-level feature extraction, combining the local feature capturing ability of CNNs with the global context understanding of transformers. It then incorporates a "Short-Term and Long-Term Memory" (SLM) module, inspired by human brain memory mechanisms. This SLM module processes image features through both direct and indirect pathways, mimicking how humans integrate immediate perceptions with past experiences to form a final judgment.
- Multi-Label Training Strategy: Instead of solely training on MOS, RichIQA utilizes a multi-label approach. It trains the network to predict not only the MOS but also the DOS of image quality. This is achieved by employing different loss functions (EMDLoss for DOS, L1Loss for MOS, and ESDLoss for SOS) that are combined to form a total loss. For datasets lacking complete subjective information (e.g., only MOS), the paper proposes methods to infer or approximate the missing DOS and SOS, enhancing the model's applicability.
Datasets
The paper evaluates RichIQA on five widely used "in the wild" IQA datasets:
- KonIQ-10K: Contains 10,073 images with MOS and DOS labels.
- BID: Focuses on blurred images, with 6,000 images and MOS and DOS labels.
- LIVE Challenge: A large-scale dataset with 1,162 images and MOS and SOS labels.
- SPAQ: Features 11,125 smartphone-captured images with MOS labels.
- FLIVE: A large dataset of 39,810 images with MOS labels.
The paper does not introduce a new benchmark dataset but rather demonstrates the effectiveness of its method on existing ones, particularly highlighting its ability to handle datasets with varying levels of provided subjective information.
Key Insights and Ideas
- Beyond MOS is Crucial: The paper strongly advocates that relying solely on MOS for "in the wild" IQA is insufficient. Richer subjective information like DOS and SOS contains valuable insights into perceptual quality that can significantly improve model performance.
- Human Memory as Inspiration: Mimicking human memory processes (short-term and long-term) through the SLM module proves effective in modeling complex image quality judgments, especially for diverse real-world images.
- Multi-Label Learning for Richer Representation: Training a model to predict multiple aspects of quality (MOS and DOS) simultaneously leads to a more comprehensive understanding of image quality and better generalization.
- Adaptability to Data Scarcity: The proposed methods for inferring missing subjective labels (DOS/SOS) demonstrate the model's robustness and applicability even when complete ground truth is not available.
Key Results
- State-of-the-Art MOS Prediction: RichIQA consistently outperforms existing state-of-the-art methods across all five tested "in the wild" IQA databases in MOS prediction, achieving significant gains in SRCC and PLCC metrics. For instance, on KonIQ-10K, it shows a 3.1% SRCC gain over the second-best method.
- Superior DOS Prediction: On datasets providing DOS labels (KonIQ-10K and BID), RichIQA demonstrates superior DOS prediction accuracy, with intersection gains of 2.6% and 1.2% over the next best method.
- Strong Generalizability: Cross-database validation experiments show that RichIQA exhibits significantly better generalizability than competing methods, achieving substantial SRCC gains even when trained on one database and tested on another.
- Balanced Computational Complexity: RichIQA achieves its superior performance with a favorable balance between computational complexity (FLOPS and parameters) and prediction accuracy, making it practical for real-world applications.
Limitations and Future Work
- Limitations: While RichIQA shows excellent performance, the paper acknowledges that the acquisition of rich subjective quality ratings is still a bottleneck. The methods for inferring missing labels are approximations.
- Future Work: The authors suggest that exploring richer subjective quality information beyond MOS, SOS, and DOS could further enhance IQA models. They also propose that the RichIQA methodology, which leverages human memory and multi-label learning, could be generalized to other perceptual quality assessment scenarios beyond image quality.
Summary updated