TL;DR
Researchers have introduced ESIQAD, the first dedicated database for assessing the quality of egocentric spatial images (like those captured by the Apple Vision Pro). Alongside this, they developed ESIQAnet, a state-of-the-art model leveraging the efficient Mamba2 architecture and binocular cross-attention to predict how humans perceive image quality across 2D and immersive 3D displays.
What Problem Does This Paper Address?
With the rise of "spatial computing" and devices like the Apple Vision Pro and Meta Quest 3, we are seeing a surge in egocentric spatial images—photos taken from a first-person perspective with depth information.
Traditional Image Quality Assessment (IQA) metrics struggle here for three reasons:
- Stereoscopic Nature: These images consist of left and right views that the brain fuses into 3D. Standard 2D metrics ignore this interaction.
- Immersive Context: Egocentric images have a much wider Field of View (FoV) and different distortion patterns (like motion blur or exposure issues from head movement) compared to traditional photography.
- Display Variability: The same spatial image might be viewed as a flat 2D photo, a 3D window, or a fully immersive 3D environment, each resulting in a different Quality of Experience (QoE).
Until now, there was no large-scale dataset or specialized model to evaluate the perceptual quality of this specific type of content.
Methods
The authors propose ESIQAnet, a multi-stage feature fusion model. Its architecture is notable for several reasons:
- Mamba2 Backbone: Instead of using standard Transformers or CNNs, the model uses Visual State Space Duality (VSSD) blocks based on the Mamba2 architecture. This allows for efficient modeling of spatial relationships with lower computational overhead.
- Cross-Attention Fusion: To mimic how the human brain integrates visual information from both eyes, the model uses a cross-attention mechanism. It treats the left view as a "query" and the right view as "keys/values" to capture disparity and binocular interactions.
- Transposed Attention: This block refines features by looking at interactions across channels rather than just spatial patches, helping the model prioritize the most relevant visual information for quality.
- Multi-Stage Regression: Features from different depths of the network are concatenated and fed into a final regressor to produce a quality score.
Datasets
The paper introduces the Egocentric Spatial Images Quality Assessment Database (ESIQAD):
- Content: 500 egocentric spatial images. 400 were captured directly with an Apple Vision Pro; 100 were synthesized using the iPhone 12’s dual-camera setup.
- Ground Truth: Mean Opinion Scores (MOS) collected from 22 participants.
- Display Modes: Each image was rated under three conditions: 2D display (iMac), 3D-window (Vision Pro), and 3D-immersive (Vision Pro).
- Diversity: The images cover various indoor/outdoor scenes, lighting conditions, and natural degradations like noise and blur.
Key Insights and Ideas
- The "3D Boost": Subjective scores showed that participants generally preferred the 3D-immersive and 3D-window modes over 2D viewing. The sense of depth and realism actually "masks" some minor technical flaws, leading to higher quality ratings.
- The Dizziness Trade-off: While the 3D-immersive mode provided the highest perceived quality, it also induced significantly more dizziness than the 3D-window mode.
- Synthesized vs. Captured: Interestingly, synthesized images from the iPhone often received higher MOS than raw Vision Pro captures. This is likely due to the iPhone's aggressive internal post-processing (color enhancement and higher fixed resolution) which humans find aesthetically pleasing, even if less "realistic."
- Mamba Efficiency: The results suggest that State Space Models (SSMs) like Mamba are highly effective for IQA tasks, outperforming traditional heavy-weight Transformers in both accuracy and efficiency.
Key Results
ESIQAnet was tested against 22 state-of-the-art IQA models (including BRISQUE, NIQE, and MANIQA).
- Superior Accuracy: ESIQAnet-B (the "Base" version) achieved the highest performance across all metrics. In the 3D-immersive mode, it showed a 5.33% improvement in SRCC (Spearman Rank Correlation Coefficient) over the previous best model.
- Generalization: The model remained robust when tested separately on the "captured" vs. "synthesized" subsets of the database.
- Ablation Success: Removing the Cross-Attention or Transposed Attention blocks led to a measurable drop in performance, proving that binocular fusion is critical for spatial IQA.
Limitations and Future Work
- Device Diversity: The current dataset is limited to Apple hardware (Vision Pro and iPhone). The authors acknowledge the need to include images from other devices like the Meta Quest 3 or Varjo XR-3 to ensure the model generalizes across different sensor types.
- Motion and Video: This study focuses on static images. Egocentric video introduces complex temporal distortions and a much higher risk of motion sickness, which will require new modeling approaches.
- Subjective Scale: The study used 22 participants; while statistically significant for this field, larger-scale crowdsourced studies could further refine the MOS.
Summary updated