TL;DR
This paper statistically evaluates 21 no-reference image quality assessment (NR-IQA) metrics on remote sensing images degraded by filtering, noise, and motion blur. They found that degradation type significantly impacts metric performance, and no single metric is universally robust, leading to specific recommendations for different degradation types and a general-purpose metric for unknown degradations.
What Problem Does This Paper Address?
Assessing the quality of digital images is crucial for various applications, especially when the original, pristine image isn't available for comparison (no-reference or NR assessment). This is common in remote sensing, where images might be degraded during acquisition, transmission, or processing (e.g., denoising, deblurring). Existing NR-IQA metrics, often developed for natural images, may not perform reliably on remote sensing imagery due to its unique characteristics (e.g., multi-spectral bands, different scene content like urban, rural, harbor). This paper aims to bridge this gap by rigorously evaluating a broad set of NR-IQA metrics on remote sensing data under controlled degradation conditions to identify robust and suitable metrics.
Methods
The researchers evaluated 21 commonly used NR-IQA metrics. Their evaluation involved a three-pronged approach:
- Robustness Assessment: They tested each metric against three criteria:
- Prediction Accuracy: Using one-way ANOVA, they checked if a metric's score was primarily influenced by image degradation rather than image content (urban, rural, harbor). Metrics sensitive to content were deemed unsuitable.
- Prediction Monotonicity: They analyzed scatter plots to ensure metrics showed a consistent trend (increasing or decreasing) with increasing degradation levels, mimicking human perception.
- Prediction Consistency: Pearson linear correlation coefficient (PLCC) was used to measure how well metric scores correlated with degradation levels.
- Factor Analysis (FA): For metrics that passed the robustness tests, FA was employed to group similar metrics into components (clusters). This helps identify redundancy and understand underlying quality dimensions.
- Representative Metric Selection: Within each identified component, the metric with the highest loading coefficient was chosen as the representative metric for that component.
Datasets
The study used Quickbird remote sensing images covering San Francisco, CA, USA. Twelve subset images (256x256 pixels) were selected, categorized into three types of general content: urban, rural, and harbor. These images were then subjected to three types of degradation: average filtering, Gaussian white noise, and linear motion degradation, each at 40 distinct levels, resulting in a total of 1440 degraded images for evaluation. No new benchmark datasets were introduced; existing Quickbird imagery was utilized.
Key Insights and Ideas
- Degradation Type Dominates Content: The study found that the type of degradation (e.g., blur vs. noise) has a much stronger influence on how an NR-IQA metric performs than the content of the image (urban vs. rural). This is a critical finding for applying these metrics in real-world scenarios.
- No Universal Metric: No single NR-IQA metric proved effective across all degradation types. A metric that performs well for noise might fail for blur, and vice-versa. This highlights the need for context-specific metric selection.
- Training-Based Metrics Struggle: Metrics that rely on training with specific datasets (common in some NR-IQA approaches) were found to be unsuitable for remote sensing images. Their performance was highly dependent on the training samples, making them unreliable for general remote sensing applications.
- Opposite Trends: Some metrics showed opposite trends (e.g., increasing quality score with increasing degradation) depending on the degradation type, further emphasizing their context-dependent nature.
- Robustness is Key: Only a subset of the tested metrics (seven out of 21) met the criteria for robustness across different degradation types and image contents.
Key Results
- Robustness Criteria: Out of 21 NR-IQA metrics, only 7 were found to be robust enough to meet the requirements of prediction accuracy, monotonicity, and consistency across the tested conditions.
- Specific Recommendations:
- For average filter degradation: Edge Intensity (EI) and Just Noticeable Distortion (JND) are recommended.
- For Gaussian white noise degradation: EI, Blind Image Quality Assessment through Anisotropy (BIQAA), and Mean Metric (MM) are suggested.
- For linear motion degradation: Laplacian Derivative (LD), JND, and Standard Deviation (SD) are recommended.
- General Recommendation: For evaluating image quality when the degradation type is unknown, Edge Intensity (EI) is proposed as the most suitable metric.
- Factor Analysis Insights: Factor analysis revealed underlying components of image quality. For instance, for average filter degradation, EI, AG, FISH, and LD loaded highly on one component, while JND and EM loaded on another.
Limitations and Future Work
- Limited Degradation Types: The study focused on three specific degradation types (average filtering, Gaussian white noise, linear motion). Future work could explore a wider range of degradations relevant to remote sensing, such as compression artifacts, atmospheric distortions, or sensor noise.
- Specific Sensor Data: The evaluation was conducted on Quickbird imagery. While representative, testing on data from other remote sensing sensors with different spectral bands and resolutions could further validate the findings.
- Subjective Ground Truth: While the study focuses on objective NR-IQA, a direct comparison with subjective human evaluations on these specific remote sensing datasets would strengthen the conclusions.
- Training-Based Metric Re-evaluation: The paper dismisses training-based metrics due to their poor performance. Future research could investigate if these metrics can be adapted or retrained for remote sensing data to improve their applicability.
Summary updated