TL;DR
This paper introduces the first comprehensive framework for assessing the quality and predicting distortion-aware saliency in AI-generated omnidirectional images (AIGODIs). They present a new dataset (OHF2024) and two models, BLIP2OIQA and BLIP2OISal, which achieve state-of-the-art results and enable an automated optimization process to enhance AIGODI visual quality.
What Problem Does This Paper Address?
The rapid advancement of AI image generation has led to a surge in AI-generated omnidirectional images (AIGODIs), which hold immense potential for immersive applications like VR and AR. However, these AIGODIs often suffer from unique quality issues and degradations that are not well-understood or addressed by existing methods. Current research on image quality assessment (IQA) and saliency prediction is largely focused on natural or standard 2D images, and these methods don't effectively generalize to the distinct characteristics and applications of AIGODIs. This gap hinders the development and deployment of high-quality AIGODIs for real-world use. The authors aim to bridge this gap by providing tools to evaluate AIGODI visual experience, identify problematic regions, and ultimately improve their quality.
Methods
The core of this work lies in a novel, two-pronged approach leveraging the BLIP-2 vision-language model.
- BLIP2OIQA (Quality Assessment): This model evaluates the human visual experience of AIGODIs across three dimensions: quality (low-level visual attributes), comfortability (overall immersive experience), and correspondence (alignment with the text prompt). It employs a shared-wise encoder to fuse image and text information, then uses a quality decoder that incorporates viewport-based feature extraction and attention mechanisms to capture both intra- and inter-viewport quality aspects.
- BLIP2OISal (Distortion-Aware Saliency Prediction): This model predicts salient regions that are specifically affected by distortions, rather than just general visual saliency. It also uses the shared-wise encoder and then a saliency decoder that integrates multi-layer image features with text-image fused features to identify these critical areas.
- Optimization Process: The insights from BLIP2OIQA and BLIP2OISal are combined for an automated optimization pipeline. Unsatisfactory images are identified by BLIP2OIQA, and BLIP2OISal highlights distorted regions. These regions are then used to guide an inpainting process (using a modified MVDiffusion model) to refine the image.
Datasets
The paper introduces a new, comprehensive dataset:
- OHF2024: This dataset contains 600 AI-generated omnidirectional images created using five different models based on 50 diverse scene prompts. It includes:
- 1,800 Mean Opinion Scores (MOSs) from human subjects across three perspectives: quality, comfortability, and correspondence.
- 600 distortion-aware saliency maps derived from human annotations.
- The dataset is designed to capture a wide range of visual experiences and distortions specific to AIGODIs.
Key Insights and Ideas
- Multidimensional Visual Experience: Human perception of AIGODIs is not a single score but a complex interplay of quality, comfortability, and correspondence. These dimensions are distinct and require separate evaluation.
- Distortion-Aware Saliency is Crucial: Simply predicting general visual saliency isn't enough for AIGODIs. Identifying regions with actual degradations is key for effective optimization.
- Text Prompt Influence: The text prompt used to generate an AIGODI significantly influences distortion-aware saliency, highlighting the need to integrate text information into saliency prediction models.
- Viewport-Based Processing: For omnidirectional images, considering individual viewports and their interrelationships is vital for accurately assessing visual experience, especially for comfortability and correspondence.
- BLIP-2 as a Strong Foundation: Leveraging the pre-trained BLIP-2 model provides a powerful foundation for both quality assessment and saliency prediction tasks, enabling effective feature fusion.
Key Results
- SOTA Performance: BLIP2OIQA achieved state-of-the-art (SOTA) performance on human visual experience evaluation for AIGODIs across all three dimensions (quality, comfortability, correspondence), showing significant improvements over existing methods, especially in comfortability and correspondence.
- SOTA Saliency Prediction: BLIP2OISal also achieved SOTA results in distortion-aware saliency prediction for AIGODIs, outperforming traditional and deep learning-based saliency models.
- Effective Optimization: The proposed optimization process, guided by BLIP2OIQA and BLIP2OISal, demonstrated significant improvements in visual quality for AIGODIs, as validated by both human ratings and the BLIP2OIQA model itself.
- Generalizability: BLIP2OIQA showed good adaptability in evaluating AIGODIs generated by diverse models.
Limitations and Future Work
- Model Specificity: While the models are trained on a diverse dataset, their performance might vary on AIGODIs generated by entirely novel or significantly different generation techniques.
- Optimization Scope: The optimization process was demonstrated using a modified MVDiffusion model. Future work could explore integrating this optimization with other AIGODI generation pipelines.
- Dataset Size: While OHF2024 is comprehensive, larger datasets could further enhance model robustness and generalization.
- Real-time Applications: Further research could focus on optimizing these models for real-time performance in interactive VR/AR applications.
Summary updated