ESIQA: Perceptual Quality Assessment of Vision-Pro-based Egocentric Spatial Images

Xilei Zhu, Liu Yang, Huiyu Duan, Xiongkuo Min, Guangtao Zhai, Patrick Le Callet

Abstract

The emergence of spatial computing, led by devices such as the Apple Vision Pro, has introduced a new medium: egocentric spatial images. These first-person, binocular captures present unique challenges for traditional Image Quality Assessment (IQA) due to their stereoscopic nature, wide field of view, and varied display modes. This paper introduces the Egocentric Spatial Images Quality Assessment Database (ESIQAD), the first large-scale dataset of its kind, and ESIQAnet, a novel quality assessment model. ESIQAnet utilizes the Mamba2 architecture and binocular cross-attention to effectively model human perception across 2D and immersive 3D environments. Experimental results demonstrate that ESIQAnet significantly outperforms existing state-of-the-art IQA metrics.

Introduction

With the rise of devices like the Apple Vision Pro and Meta Quest 3, "spatial computing" is becoming a reality. Central to this experience are egocentric spatial images—photos taken from a first-person perspective that include depth information. Traditional IQA metrics struggle with these images for three primary reasons: their stereoscopic nature (left and right views fused by the brain), the immersive context (wide field of view and unique head-movement distortions), and display variability (viewing as 2D, 3D windows, or immersive 3D environments).

Overview of the egocentric spatial image pipeline, illustrating real scene capturing, binocular views, and the transition to 2D vs. 3D display modes.

The ESIQAD Dataset

To address the lack of specialized data, the authors developed the Egocentric Spatial Images Quality Assessment Database (ESIQAD). This dataset contains 500 images: 400 captured directly with an Apple Vision Pro and 100 synthesized using an iPhone 12’s dual-camera setup. Mean Opinion Scores (MOS) were collected from 22 participants across three viewing conditions: 2D (iMac), 3D-window (Vision Pro), and 3D-immersive (Vision Pro). The dataset covers a wide range of indoor and outdoor scenes with natural degradations like noise and blur.

Methodology: ESIQAnet

The proposed model, ESIQAnet, is a multi-stage feature fusion network designed to mimic human binocular perception. Its architecture features several innovations:

  • Mamba2 Backbone: Utilizing Visual State Space Duality (VSSD) blocks based on the Mamba2 architecture, the model achieves efficient spatial relationship modeling with lower computational overhead than standard Transformers.
  • Binocular Cross-Attention: To simulate how the brain integrates visual information, a cross-attention mechanism treats the left view as a "query" and the right view as "keys/values," capturing disparity and binocular interactions.
  • Transposed Attention: This component refines features by analyzing interactions across channels, allowing the model to prioritize relevant visual information for quality assessment.

Key Insights and Results

Subjective analysis revealed a "3D Boost": participants generally preferred 3D-immersive and 3D-window modes over 2D viewing, as depth perception can mask minor technical flaws. However, a trade-off exists, as immersive modes induced more dizziness. Interestingly, synthesized iPhone images often scored higher than raw Vision Pro captures due to aggressive post-processing that users found aesthetically pleasing.

In quantitative evaluations, ESIQAnet-B outperformed 22 state-of-the-art IQA models. In 3D-immersive mode, it achieved a 5.33% improvement in SRCC (Spearman Rank Correlation Coefficient) over the previous best-performing model. Ablation studies confirmed that both binocular fusion and transposed attention are critical for high performance.

Conclusion and Future Work

ESIQAnet and the ESIQAD dataset represent a significant step forward in evaluating spatial computing content. While currently focused on Apple hardware and static images, future work will expand to other devices (e.g., Meta Quest 3) and address the complexities of egocentric video, including temporal distortions and motion sickness modeling.

Summary updated

BibTeX
@article{zhu2025esiqa,
  author = {Xilei Zhu and Liu Yang and Huiyu Duan and Xiongkuo Min and Guangtao Zhai and Patrick Le Callet},
  title = {{ESIQA: Perceptual Quality Assessment of Vision-Pro-based Egocentric Spatial Images}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2025},
  doi = {10.1109/tvcg.2025.3549174},
}
Figure