Introduction
Augmented Reality (AR) presents a unique perceptual challenge: the simultaneous presentation of virtual content and the physical world. Unlike traditional 2D displays or fully immersive Virtual Reality (VR), AR requires the human visual system to negotiate between two distinct layers of information. This overlap often results in "visual confusion," where the saliency of an object is determined not just by its own features, but by its interaction with the background.
Predicting where a user will look in these environments is critical for optimizing display performance, user interface design, and safety. However, existing saliency models—largely trained on single-layer natural images—struggle to account for the transparency and layering inherent to AR. This paper introduces the Saliency in AR Dataset (SARD) and a novel deep learning architecture, VQSal-AR, designed specifically to address these challenges.
The Challenge: Visual Confusion and Transparency
In AR, virtual content (AR) is superimposed onto a real-world background (BG). The visibility of these layers is governed by the mixing level, or transparency ($\alpha$). The researchers identified that human attention in AR is not a simple linear combination of the two layers. Instead, it is a "tug-of-war" influenced by the opacity of the virtual objects.
At high opacity ($\alpha=0.75$), users focus predominantly on the virtual content. Conversely, at low opacity ($\alpha=0.25$), the background features dominate. In the middle ground ($\alpha=0.5$), saliency maps exhibit a complex fusion of features from both layers. Understanding this non-linear interaction is the key to accurate attention prediction in AR.
Methodology
The study approached the problem through two primary contributions: a specialized dataset and a multi-stream model architecture.
The SARD Dataset
To study these interactions under controlled conditions, the researchers utilized VR headsets to simulate AR environments. This allowed for precise manipulation of background and overlay elements while tracking the eye movements of 60 subjects. The resulting Saliency in AR Dataset (SARD) contains 450 background images and 450 AR images, combined to create 1,350 unique superimposed scenarios across three transparency levels. This represents the first large-scale dataset dedicated to the nuances of AR saliency.
VQSal and VQSal-AR
The proposed model, VQSal (Vector Quantized Saliency), departs from traditional methods that rely on supervised pretraining for object recognition. Instead, it utilizes Vector Quantization (VQ) to break images into a "codebook" of discrete visual tokens. This unsupervised approach allows the model to learn a vocabulary of perceptually significant patterns, which the authors found to be more effective for saliency prediction than standard pixel-based features.
To handle the specific architecture of AR, the model was extended into VQSal-AR. This version employs a three-stream fusion network. Rather than processing only the final "mixed" image, the model uses three separate decoders to analyze the AR content, the background, and the superimposed result simultaneously. This allows the network to explicitly model how the two layers compete for the user’s attention.
Key Results
The VQSal model first demonstrated its robustness by achieving state-of-the-art (SOTA) performance on the SALICON benchmark, a standard for general saliency prediction (AUC: 0.863, CC: 0.869).
When applied to the AR-specific challenges of the SARD dataset, VQSal-AR significantly outperformed 13 classical and 6 deep-learning-based models. It achieved an AUC of 0.903 and a Correlation Coefficient (CC) of 0.893. Ablation studies confirmed that the "Vector Quantization" pretraining was essential; removing it led to a significant drop in performance, validating the "visual token" approach as the core driver of the model's success.
Conclusion and Future Work
This research provides a foundational framework for understanding human attention in layered visual environments. By introducing the SARD dataset and the VQSal-AR model, the authors have bridged a significant gap in AR interface optimization.
However, the current study is limited to static images. Real-world AR is inherently dynamic, involving head movements and animated virtual content. Future research will focus on augmented video saliency, incorporating temporal factors and motion-induced saliency. Such advancements will be vital for high-stakes applications, including surgical navigation and AR-assisted driving, where understanding a user's focus in real-time is a matter of safety and precision.
Summary updated