TL;DR
This paper introduces a novel unsupervised framework for fusing image quality assessment (IQA) scores from multiple models. By employing deep Maximum a Posteriori (MAP) estimation with fine-grained score-level uncertainty estimation, the method achieves superior performance and robustness, even capable of identifying and down-weighting underperforming IQA models without relying on ground truth human ratings.
What Problem Does This Paper Address?
Image Quality Assessment (IQA) models aim to predict how humans perceive image quality. While individual models have improved, they often excel at specific types of distortions or image content, leading to biases. A natural solution is to combine the strengths of multiple IQA models by fusing their scores. However, existing fusion methods often rely on handcrafted formulas, struggle with integrating new models, or require supervised learning with human-rated datasets (Mean Opinion Scores - MOS). This reliance on MOS can reintroduce biases and limit generalizability. The core challenge this paper addresses is how to effectively fuse IQA scores in an unsupervised manner, maximizing performance without human ground truth and accounting for the inherent uncertainties in individual model predictions.
Methods
The paper proposes a general framework for unsupervised IQA score fusion based on deep Maximum a Posteriori (MAP) estimation. The key innovation lies in its fine-grained uncertainty estimation at the score level.
Here's a breakdown of the approach:
- Encoder-Decoder Architecture: An encoder network takes the scores from multiple IQA models for a given image and produces a latent representation. This representation is then used by decoders to predict the final quality score. This amortized inference approach speeds up the process and simplifies inference.
- Score-Level Uncertainty Estimation: Instead of just estimating uncertainty at the model level, the framework models the uncertainty of each individual score generated by an IQA model. This is crucial because the reliability of a score can vary significantly depending on the image content and the specific IQA model.
- Probabilistic Modeling: The framework models the relationship between the true quality (MOS) and the predicted score using a probabilistic approach. Specifically, it uses a skew-normal distribution to capture the asymmetric nature of IQA score distributions, which is observed to be skewed towards higher scores for higher quality images. This allows for more accurate modeling of uncertainty.
- MAP Estimation: The entire system is trained end-to-end using MAP estimation, aiming to find the most likely true quality score given the observed IQA scores and their estimated uncertainties.
- Rank Fusion Integration: The framework is designed to be flexible and can seamlessly integrate rank fusion methods, offering an alternative way to handle score distributions.
Datasets
The authors evaluated their proposed model on ten diverse subjectively rated IQA datasets. These datasets include:
- Six datasets with single distortion types (LIVE R2, TID2013, CSIQ, VCL@FER, CIDIQ50, CIDIQ100).
- Four datasets with multiply distorted images (MDID, MDID2013, LIVE MD, MDIVL).
They also mention using the KADID-10K dataset for visualizing empirical distributions.
Key Insights and Ideas
- Unsupervised Fusion is Achievable and Superior: The paper demonstrates that effective IQA score fusion can be achieved without relying on human-rated MOS, a significant departure from many existing methods. This unsupervised approach leads to better generalizability.
- Score-Level Uncertainty is Crucial: Estimating uncertainty for each individual score, rather than just for each model, is key to improving fusion accuracy. This allows the model to intelligently weigh the contributions of different IQA models based on their reliability for a specific image.
- Skew-Normal Distribution Captures Real-World IQA Behavior: The use of a skew-normal distribution to model score uncertainties accurately reflects the observed behavior of IQA models, where distributions are often asymmetric and variance changes with quality.
- Robustness to "Bad" Models: The framework exhibits a remarkable ability to automatically identify and down-weight the influence of underperforming or "bad" IQA models during fusion, without prior knowledge of their quality. This is a significant advantage for practical applications where the set of available IQA models might vary.
- Rank Fusion Integration: The ability to seamlessly integrate rank fusion methods within the same framework highlights its versatility.
Key Results
- State-of-the-Art Performance: All three variants of the proposed model (SF-ms, RF-ms, SF-m) significantly outperform other fusion methods and individual IQA models in terms of average SRCC and PLCC across ten testing datasets.
- Score-Level Uncertainty Boosts Performance: The SF-ms and RF-ms models, which incorporate score-level uncertainty estimation, consistently rank among the top three performers on individual testing sets. Removing score-level uncertainty (SF-m) leads to a performance drop, underscoring its importance.
- Competitive with Rank Fusion: The RF-ms variant demonstrates that the proposed framework is superior to existing rank fusion methods like RRF and RRFW.
- Robustness Demonstrated: In an experiment where two "bad-performing" IQA metrics (random number generators) were included, the proposed method's performance remained competitive, while RRF and RRFW declined significantly. The uncertainty estimation module correctly identified these bad metrics by assigning them significantly higher uncertainty values.
Limitations and Future Work
- Parametric Assumptions: The framework relies on specific parametric assumptions for the encoder, decoder, and uncertainty estimation modules (e.g., Fully Connected Network for the encoder, exponential for $f^j$, quadratic for $g^j$). While these are chosen for simplicity and effectiveness, exploring more complex architectures could potentially yield further improvements.
- Generalizability of Priors: The paper uses a uniform prior for the predicted MOS ($p(z) = U(0,1)$) to promote generalizability. However, the authors acknowledge that the use of informative priors might compromise generalizability if the dataset's MOS distribution deviates significantly from uniform. Future work could explore adaptive prior selection or more sophisticated prior modeling.
- Computational Cost: While the amortized inference with the encoder significantly speeds up the process compared to iterative MAP estimation, the computational cost of training the deep network and the uncertainty estimation modules is still a factor.
The authors suggest that the framework is versatile and can be extended to incorporate new IQA models and potentially other related tasks. The demonstrated robustness in rejecting "bad" models opens avenues for more reliable and automated IQA systems.
Summary updated