TL;DR
This paper introduces NIMA, a novel deep learning approach for no-reference image quality assessment that predicts the distribution of human opinion scores rather than just the mean. This allows for a more nuanced understanding of image quality and enables applications like optimizing image enhancement algorithms.
What Problem Does This Paper Address?
Assessing image quality is crucial for various applications, from image processing pipelines to content sharing. However, human perception of image quality is inherently subjective. Traditional methods often rely on comparing a distorted image to a pristine "golden" reference (full-reference methods) or use hand-crafted features to predict a single quality score (no-reference methods). While these methods have improved, they often struggle to capture the full spectrum of human opinion, which can vary significantly even for the same image. Existing deep learning approaches, while powerful, typically predict a single mean opinion score, failing to account for the variability and distribution of human judgments. This paper aims to bridge this gap by developing a no-reference quality assessment system that predicts the entire distribution of human scores, leading to more robust and perceptually aligned quality evaluations.
Methods
The core of NIMA (Neural Image Assessment) is a convolutional neural network (CNN) adapted from state-of-the-art object recognition architectures (like VGG16, Inception-v2, and MobileNet). Instead of outputting a single score, the network's final layer is modified to predict a probability distribution over a set of discrete quality score bins. This is achieved by replacing the standard classification/regression head with a fully-connected layer followed by a softmax activation, outputting 10 probabilities corresponding to 10 quality score bins.
The training objective is crucial: instead of a standard cross-entropy loss, the authors employ a loss function based on the Earth Mover's Distance (EMD). EMD is particularly well-suited for ordered classes (like image quality scores) because it penalizes misclassifications based on the "distance" between score bins. This encourages the model to learn the relationships between different quality levels. The authors also explore using the predicted score distributions to estimate not only the mean but also the standard deviation of human ratings, which they link to image unconventionality.
Datasets
The paper utilizes three publicly available datasets for training and evaluation:
- AVA (Aesthetic Visual Analysis) [1]: A large-scale dataset of approximately 255,000 images, rated by amateur photographers. It provides histograms of human ratings, allowing for the prediction of score distributions.
- TID2013 [2]: A dataset of 3000 images with 25 reference images and 24 types of distortions at 5 levels each. It provides mean and standard deviation of opinion scores.
- LIVE (In the Wild Image Quality Challenge Database) [26]: A dataset of 1162 photos captured by mobile devices, with mean and standard deviation of opinion scores.
The authors note that TID2013 and LIVE only provide mean and standard deviation, requiring approximation of score distributions through maximum entropy optimization.
Key Insights and Ideas
- Predicting Distributions, Not Just Means: The most significant insight is that predicting the distribution of human scores, rather than just the average, provides a richer and more accurate representation of image quality. This acknowledges the inherent subjectivity and variability in human perception.
- EMD Loss for Ordered Quality: Using the Earth Mover's Distance as a loss function is a key innovation for ordered-class prediction tasks like image quality assessment. It effectively leverages the ordinal nature of scores, leading to better performance than standard cross-entropy.
- Unconventionality and Standard Deviation: The paper suggests a strong correlation between the standard deviation of human ratings and the "unconventionality" of an image's subject matter or style. This opens up new avenues for understanding aesthetic appeal beyond simple quality scores.
- No-Reference Quality for Enhancement: The ability to predict quality without a reference image is crucial for real-world applications. NIMA's quality scores can be directly used to guide and optimize image enhancement algorithms, leading to perceptually superior results.
Key Results
- State-of-the-Art Performance: NIMA achieves state-of-the-art performance on aesthetic quality assessment on the AVA dataset, outperforming previous methods in terms of accuracy and correlation.
- High Correlation: The model demonstrates high correlation (LCC and SRCC) with ground truth mean scores on both AVA and TID2013 datasets, often comparable to or exceeding existing methods.
- Efficient Architectures: The paper shows that NIMA can be implemented using various efficient CNN architectures (MobileNet, Inception-v2, VGG16), offering a trade-off between performance and computational cost. NIMA(MobileNet) is particularly efficient for mobile applications.
- Effective Image Enhancement: The paper demonstrates the practical utility of NIMA by showing how its predicted scores can effectively tune parameters for tone enhancement and image denoising algorithms, resulting in perceptually improved images.
Limitations and Future Work
- Approximating Distributions: For datasets like TID2013 and LIVE that only provide mean and standard deviation, the score distributions are approximated. While effective, this approximation might limit the full potential of the distribution-based approach.
- Computational Cost: While NIMA is generally efficient, especially with MobileNet, some architectures (like VGG16) can still be computationally intensive for real-time applications.
- Future Work: The authors suggest exploring the trained models for other image enhancement applications and investigating the use of NIMA as a direct loss function for enhancement operators when derivatives are well-defined. They also hint at further research into the relationship between score standard deviation and image unconventionality.
Summary updated