Vision in natural and virtual environments

Mary M. Hayhoe, Dana H. Ballard, Jochen Triesch, Hiroyuki Shinoda, Pilar Aivar, Brian T. SullivanEtra 2002 · DOI

TL;DR

This paper argues that human vision in natural environments is not a passive, bottom-up process of feature extraction, but rather an active, task-specific "visual routine" driven by internal goals. Using virtual reality and eye-tracking, the authors demonstrate that observers selectively attend to and process visual information only when it's immediately relevant to their current task, challenging traditional views of vision.

What Problem Does This Paper Address?

For decades, visual perception research has largely focused on how the visual system processes static, isolated stimuli in controlled laboratory settings. This approach, while yielding valuable insights, often overlooks the dynamic, goal-directed nature of vision in everyday life. The problem this paper addresses is understanding how the visual system functions when embedded within complex, ongoing behavior, where observers actively seek and utilize information to achieve specific goals. This is crucial for developing more effective human-computer interfaces, understanding visual impairments, and building a more complete theory of visual perception. The authors highlight that traditional experimental setups often simplify the challenges faced in natural environments, such as dealing with complex, unsegmented visual arrays and global image motion generated by the observer's own movements.

Methods

The core methodology involves creating sophisticated virtual environments that allow for precise experimental control while simulating naturalistic tasks. Key techniques include:

  • Head-mounted eye-tracking: Using devices like the ASL series 501, researchers meticulously monitored participants' gaze (eye position and fixation duration) in real-time. This allowed for a detailed analysis of where observers looked and for how long during specific tasks.
  • Virtual Reality (VR) environments: The researchers developed custom VR setups (e.g., using Silicon Graphics Onyx II with high-resolution displays and head-mounted displays like Virtual Research V8) to present controlled visual stimuli and scenarios.
  • Haptic Feedback: In some experiments, haptic devices (Phantom force feedback) were integrated to provide realistic touch sensations, further enhancing the naturalness of the virtual environment and allowing for the study of combined visual-haptic tasks.
  • Controlled Stimulus Manipulation: Within these VR environments, specific visual properties (like object size) or scene elements (like traffic signs) could be altered dynamically at critical moments during task performance. This allowed researchers to probe what information was being actively processed.
  • Behavioral Analysis: Detailed analysis of recorded video data, eye-movement data, and participant reports was used to infer the strategies and information processing employed by observers.

Datasets

Not discussed in the paper. The paper focuses on experiments conducted in custom-built virtual environments rather than using pre-existing datasets.

Key Insights and Ideas

  • Vision as Active "Visual Routines": The paper strongly advocates for the concept of "visual routines" – task-specific procedures applied on demand – rather than passive, bottom-up feature extraction. This means vision is an active, goal-directed process.
  • Task-Specificity of Information Processing: Visual information, even simple features like object size, is not automatically processed or continuously available. Instead, it's acquired and discarded dynamically based on the immediate demands of the task.
  • The "Scheduling Problem": A significant challenge in active vision is determining when to deploy these visual routines. This involves balancing the selectivity of task-specific computations with the need to remain responsive to novel stimuli.
  • Learned Regularities Guide Attention: Observers leverage learned knowledge about the probabilistic structure of their environment to initiate search procedures and schedule visual attention, particularly in complex dynamic scenarios like driving.
  • Change Blindness and Task Relevance: The inability to notice changes (change blindness) is not necessarily a failure of representation but rather a consequence of the information not being relevant to the current task. If a change occurs in information not currently being used, it may go unnoticed.
  • Visual Memory for Spatial Structure: While transient information might be lost, there's evidence for retained visual memory of spatial structure, which is crucial for guiding eye and hand movements over longer periods.

Key Results

  • Size Information is Task-Dependent: In a block-copying task, participants were significantly more likely to notice a change in block height when that information was relevant to the task (Condition 3) compared to when it was not (Condition 1). This suggests size information was actively processed only when needed.
  • Driving Behavior is Goal-Driven: In a driving simulator, participants were much more likely to detect a Stop sign when instructed to obey traffic rules (F+S condition) than when simply following the car ahead (F condition). Furthermore, fixations were more concentrated around intersections in the F+S condition, indicating learned strategies for attention deployment.
  • Spatial Memory Guides Targeting: In the Baufix environment, a significant percentage of saccades to pick up pieces landed on the previous location of the piece, even after the pieces had been shuffled while the observer looked away. This demonstrates the use of spatial memory for targeting.

Limitations and Future Work

  • Generalizability of Virtual Environments: While virtual environments offer control, the authors acknowledge the need to validate findings in real environments to ensure performance translates.
  • Complexity of Real-World Tasks: The paper focuses on specific tasks (block copying, driving, toy construction). Future work could explore a wider range of more complex, multi-goal tasks.
  • Understanding the "Scheduling" Mechanism: While the paper identifies the need for scheduling visual routines, the precise mechanisms by which this scheduling occurs remain an open area for further investigation.
  • Integration of Visual and Motor Control: The paper touches on eye-hand coordination, but a deeper understanding of how visual information is integrated with motor planning and execution in natural tasks is a promising direction.

Summary updated

BibTeX
@inproceedings{hayhoe2002vision,
  author = {Mary M. Hayhoe and Dana H. Ballard and Jochen Triesch and Hiroyuki Shinoda and Pilar Aivar and Brian T. Sullivan},
  title = {{Vision in natural and virtual environments}},
  booktitle = {Etra},
  year = {2002},
  doi = {10.1145/507072.507074},
}
Figure