TL;DR
This paper introduces a new benchmark and models for assessing the quality and predicting distortion-aware saliency in AI-generated omnidirectional images (AIGODIs). They propose BLIP2OIQA and BLIP2OISal, which leverage a shared encoder based on BLIP-2, to achieve state-of-the-art results in evaluating human visual experience and identifying problematic regions, enabling an automated optimization process for AIGODIs.
What Problem Does This Paper Address?
The rapid rise of AI-generated content (AIGC) has led to impressive AI-generated images (AIGIs). Omnidirectional images (ODIs), crucial for VR/AR applications, are no exception. However, AIGODIs often suffer from unique quality issues beyond typical image degradations, such as unrealistic compositions and prompt-image mismatches. Existing quality assessment methods are insufficient for these specific challenges, and research on optimizing AIGODIs is scarce. This paper addresses this gap by focusing on evaluating the human visual experience and predicting "distortion-aware" saliency (regions with significant flaws) in AIGODIs, paving the way for their improvement.
Methods
The core of this work lies in two novel models, BLIP2OIQA and BLIP2OISal, built upon the BLIP-2 architecture. Both models utilize a shared-wise encoder that fuses image and text prompt information to extract a rich text-image fused feature.
- BLIP2OIQA (Quality Assessment): This model employs a quality decoder that processes viewport-based features (the omnidirectional image is split into six viewports). It uses attention mechanisms (self-attention for intra-viewport and cross-attention for inter-viewport consistency) to aggregate these features and then maps them to predict human visual experience scores across three dimensions: quality, comfortability, and correspondence.
- BLIP2OISal (Distortion-aware Saliency Prediction): This model uses a saliency decoder. It combines the text-image fused feature with multi-layer image features extracted from the BLIP-2 image encoder. Through feature fusion (FF) and hierarchical feature refinement (HFR) modules, it progressively integrates spatial and textual information to predict distortion-aware saliency maps.
An automatic optimization process is also proposed, where BLIP2OIQA identifies unsatisfactory images, and BLIP2OISal pinpoints distortion regions. These identified regions are then used to guide an inpainting process to refine the AIGODIs.
Datasets
The primary dataset introduced in this paper is OHF2024. This comprehensive dataset contains:
- 600 AI-generated omnidirectional images.
- 1,800 Mean Opinion Scores (MOSs) from human subjects, evaluated across three perspectives: quality, comfortability, and correspondence.
- 600 distortion-aware saliency maps derived from human annotations.
The dataset was constructed using five different AI generation models based on 50 diverse scene prompts, aiming to capture a wide range of AIGODI characteristics and quality issues.
Key Insights and Ideas
- Multidimensional Visual Experience: Human perception of AIGODIs isn't just about "quality." The paper highlights the importance of considering "comfortability" (immersive experience, lack of sickness) and "correspondence" (alignment with the text prompt) as distinct but related evaluation dimensions.
- Distortion-Aware Saliency is Crucial: Unlike natural saliency, which focuses on visually interesting regions, AIGODIs require identifying areas with actual degradations. This "distortion-aware saliency" is key for targeted optimization.
- Text Prompt is Highly Informative: The study found a strong correlation between text prompts and distortion-aware saliency in AIGODIs, suggesting that understanding the prompt is vital for predicting where flaws might occur.
- Viewport-Based Processing for ODIs: Splitting omnidirectional images into viewports and processing them with attention mechanisms helps capture the nuances of human visual experience in immersive environments.
- Synergy of Quality and Saliency for Optimization: Combining quality assessment scores with distortion-aware saliency maps provides a powerful feedback loop for effectively enhancing AIGODI quality.
Key Results
- State-of-the-Art Performance: The proposed BLIP2OIQA model achieved state-of-the-art (SOTA) results on the OHF2024 dataset for evaluating human visual experience across all three dimensions (quality, comfortability, correspondence), significantly outperforming existing NR-IQA models.
- SOTA Saliency Prediction: BLIP2OISal also achieved SOTA performance on distortion-aware saliency prediction for AIGODIs, outperforming traditional and deep learning-based saliency models on metrics like AUC, NSS, CC, SIM, and KLD.
- Effective Optimization: The proposed optimization process, guided by BLIP2OIQA and BLIP2OISal, demonstrated significant improvements in both human ratings and predicted scores for AIGODIs, even for high-quality images.
- Cross-Model Generalization: BLIP2OIQA showed good adaptability in evaluating ODIs generated by diverse AIGC models.
Limitations and Future Work
- Limited AIGODI Models: The optimization process was demonstrated using a modified MVDiffusion model due to the limited availability of AIGODI models. Future work could explore optimization across a wider range of generation techniques.
- Computational Cost: While efforts were made to reduce computational costs (e.g., freezing parts of the encoder), the models still involve complex architectures.
- Future Work: The authors suggest releasing their database and code to facilitate further research in this area. They also implicitly point towards exploring more sophisticated optimization strategies and potentially extending these methods to other forms of AI-generated content.
Summary updated