EyeAR: Refocusable Augmented Reality Content through Eye Measurements

Damien Rompapas, Aitor Rovira, Alexander Plopski, Christian Sandor, Takafumi Taketomi, Goshiro Yamamoto, Hirokazu Kato, Sei Ikeda

TL;DR

This paper introduces EyeAR, a novel approach for creating more realistic Augmented Reality (AR) content by dynamically adjusting virtual object focus based on real-time user eye measurements. By matching the Depth of Field (DoF) of virtual objects to the user's eye, EyeAR significantly improves the perceived realism of AR experiences, particularly for optical see-through HMDs.

What Problem Does This Paper Address?

Augmented Reality (AR) aims to seamlessly blend computer-generated imagery (CG) with the real world. A major hurdle in achieving photorealistic AR, especially with optical see-through Head-Mounted Displays (OST-HMDs), is the mismatch between the user's natural eye focus (accommodation) and the fixed focal plane of the display. This mismatch leads to a lack of realistic Depth of Field (DoF) effects for virtual objects, making them appear artificial and less integrated with the real environment. Existing solutions like Light Field Displays (LFDs) are computationally expensive and bulky, while simpler OST-HMDs lack the ability to optically create these crucial focus cues. This paper tackles the challenge of generating realistic DoF for virtual content on standard, single-focal-plane OST-HMDs by actively adapting the CG to the user's eye state.

Methods

The core of EyeAR is a closed-loop system that continuously measures the user's eye state and uses this information to render CG with appropriate DoF. The key steps are:

  1. Measure Eye: The system uses an autorefractometer to measure the user's pupil size and focal distance (accommodation) at 5 Hz. This data is crucial for determining the correct DoF parameters.
  2. Render CG with Correct DoF: The measured eye parameters are fed into a distributed ray tracer. This tracer generates CG that mimics the user's eye's focal length and aperture, thereby creating a realistic DoF for virtual objects. The model used for the eye is a camera with continuously adjusted focal length and aperture, moving beyond the simpler pinhole camera model.
  3. Correct Screen-Object Disparity: Acknowledging that the virtual image plane of the display might not perfectly align with the user's focus distance, the system aims to correct this disparity. However, due to limitations in their prototype's speed and potential for artifacts, this correction was minimized in the experimental setup by carefully controlling the scene's depth variance.
  4. User Observes Graphics: The rendered CG is displayed to the user.
  5. Loop: The process repeats, continuously adapting to the user's eye movements.

Datasets

Not discussed in the paper. The experiment uses a custom-built physical setup with 3D printed pillars and a real scene.

Key Insights and Ideas

  • Eye-centric rendering is key: The paper strongly advocates for rendering CG based on actual user eye measurements, rather than relying on simplified models like the pinhole camera. This is the fundamental insight driving EyeAR's improved realism.
  • DoF mismatch is a major realism killer: The experiment clearly demonstrates that when the DoF of virtual objects doesn't match the user's eye focus, it's easier to distinguish them from real objects.
  • Pinhole camera model is insufficient for AR: The results show that CG rendered with a pinhole eye model often appears "too sharp," making it stand out unnaturally against the real world.
  • Real-time eye tracking is crucial for future AR: The limitations encountered (slow update rates, disparity) highlight the need for faster and more integrated eye-tracking solutions in future HMDs.

Key Results

The study's results strongly support Hypothesis 1 (H1): participants were significantly less likely to detect the virtual pillar when the autorefractometer was "on" (i.e., EyeAR was used) compared to when it was "off" (simulating a standard AR display without eye-based DoF adjustment).

  • Overall Detection Rate: When the autorefractometer was off, participants correctly identified the virtual pillar 81.9% of the time. With the autorefractometer on, this dropped to 44.4%.
  • Pillar-Specific Performance: The green pillar (front) and red pillar (back) were identified significantly above chance even with EyeAR, suggesting limitations in the system's effectiveness at extreme depths or due to display disparity. However, the blue pillar (at the display's focal plane) was identified at chance level (33.3%) when EyeAR was active, indicating successful integration in this specific scenario.
  • Learning Effect: Participants improved their ability to guess the virtual pillar over successive trials, suggesting a learning effect that might have influenced the results.
  • Gender Significance: Gender was found to be a significant factor in detection rates, which was an unexpected finding.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Screen-Object Disparity: The fixed focal plane of the display and the distance to virtual objects created a disparity that likely contributed to detection, especially for front and back pillars. Future work aims to address this by either adjusting the display's position or developing more sophisticated rendering techniques.
  • Autorefractometer Update Rate: The 5 Hz update rate of the autorefractometer, combined with interpolation and rendering delays, led to noticeable latency. Faster eye-tracking solutions (e.g., using cameras) are needed for smoother experiences.
  • Pillar Color and Texture: While the study found color not to be a significant factor, differences in texture and illumination could have contributed to detection. Future work will explore more complex scenes with varied materials and lighting.
  • Generalizability: The study used simple, uniform pillars. Future work will investigate EyeAR's performance with more complex scenes and objects.
  • Portability and Integration: Miniaturizing the autorefractometer or integrating eye-tracking cameras into HMDs is a key goal for practical EyeAR implementation.
  • AR Turing Test Standardization: The authors aim to refine their methodology for conducting AR Turing Tests to make them more robust and standardized.
  • Age and Gender Effects: Further investigation into how age and gender influence the perception of EyeAR is planned.

Summary updated

BibTeX
@article{rompapas2017eyear,
  author = {Damien Rompapas and Aitor Rovira and Alexander Plopski and Christian Sandor and Takafumi Taketomi and Goshiro Yamamoto and Hirokazu Kato and Sei Ikeda},
  title = {{EyeAR: Refocusable Augmented Reality Content through Eye Measurements}},
  journal = {Multimodal Technol. Interact.},
  year = {2017},
  doi = {10.3390/mti1040022},
}
Figure