What you see is what you need

Jochen Triesch, D.h. Ballard, Mary Hayhoe, Brian SullivanJournal of Vision 2003 · DOI

TL;DR

This paper investigates how task demands influence our ability to detect changes in our visual environment, even when we're actively engaged with an object. They found that simply making an object's attribute relevant to the task isn't enough; it needs to be relevant at the precise moment of the change for detection to be likely. This suggests our visual system is highly "just-in-time" and goal-driven, rather than passively processing all visual information.

What Problem Does This Paper Address?

The phenomenon of "change blindness" describes our surprising inability to notice significant changes in a visual scene, especially when these changes occur during brief disruptions like eye blinks or saccades. While previous research has explored this, much of it used explicit change detection tasks where participants were actively looking for changes. This paper aims to understand change blindness in more naturalistic, implicit settings, where participants are unaware changes might occur. They want to disentangle whether change blindness is solely due to limitations in visual short-term memory or if attentional and task-specific demands play a more crucial role, particularly in ongoing, goal-directed behaviors.

Methods

The researchers employed a virtual reality (VR) setup to create a controlled yet naturalistic environment. Subjects performed an object sorting task in VR, where they had to pick up bricks and place them on conveyor belts. Crucially, the height of some bricks was changed mid-action. The VR system allowed for precise control over the visual scene, high-fidelity rendering, and simultaneous recording of eye movements and hand interactions using haptic feedback devices.

They manipulated the task in three conditions, varying when the brick's height was relevant:

  1. Condition 1: Size irrelevant for both picking up and placing.
  2. Condition 2: Size relevant only for picking up (identifying tall vs. short bricks).
  3. Condition 3: Size relevant for both picking up and placing (e.g., tall bricks to one belt, short to another).

Changes were masked by the natural saccadic eye movements subjects made when moving bricks.

Datasets

Not discussed in the paper. The experiment uses a custom-built virtual reality environment and task.

Key Insights and Ideas

  • "Just-in-Time" Vision: The most significant finding is that our visual system seems to compute information "just in time" for the current goal. Simply making a feature (like brick height) relevant isn't enough; it must be relevant at the exact moment the change occurs for us to notice it.
  • Task Relevance Timing is Key: The difference between Condition 2 (size relevant for pick-up) and Condition 3 (size relevant for pick-up and put-down) highlights this. While size was relevant in both, noticing was much higher when it was relevant for both stages of the action.
  • Change Blindness Beyond Short-Term Memory: The simplicity of the objects and task suggests that change blindness can occur even with minimal visual short-term memory load, implying a failure to engage processing rather than a lack of capacity.
  • Tracking Isn't Enough: Surprisingly, even when subjects were visually tracking the object at the moment of change, they still frequently failed to notice it. This challenges the intuitive idea that direct fixation guarantees perception.
  • Graded Differences: Noticing changes wasn't an all-or-nothing phenomenon. The differences between conditions were graded, suggesting a continuous modulation of visual processing rather than discrete states.

Key Results

  • Condition 3 showed significantly higher change detection rates compared to Conditions 1 and 2, both in spontaneous reports and end-of-experiment questionnaires.
  • In Condition 1, subjects reported noticing very few changes (88% of subjects reported no changes spontaneously).
  • In Condition 3, subjects noticed many more changes, with the average number of unnoticed changes before the first noticed one dropping from 7.2 (Condition 1) and 6.5 (Condition 2) to just 1.0.
  • Gaze analysis revealed no significant differences in gaze behavior (tracking, saccades, blinks) across the three conditions at the moment of change. This rules out differences in gaze strategy as the primary explanation for the varying detection rates.
  • Fixation durations during the "put-down" phase (after the change) were similar for noticed and unnoticed changes, suggesting that prolonged attention after the event doesn't necessarily indicate prior detection. However, noticed changes did lead to a significant increase in overall fixation time during put-down.

Limitations and Future Work

  • Reporting Bias: The authors acknowledge the difficulty in obtaining accurate reports of noticed changes when subjects are unaware changes might happen. They used both spontaneous verbal reports and end-of-experiment questionnaires, noting that questionnaire responses were consistently higher.
  • Task Complexity Hypothesis: They suggest that an alternative explanation for their findings could be a general increase in "perceptual arousal" due to task complexity, rather than specific task relevance. They propose future experiments to differentiate between these hypotheses.
  • Future Directions: The paper raises several open questions: How do people select their moment-to-moment goals? What tactics do they use to manage multiple goals? What are the neural correlates of these dynamic processing shifts? They also suggest further work to explore the "task complexity" hypothesis.

Summary updated

Abstract

We studied the role of attention and task demands for implicit change detection. Subjects engaged in an object sorting task performed in a virtual reality environment, where we changed the properties of an object while the subject was manipulating it. The task assures that subjects are looking at the changed object immediately before and after the change. Our results demonstrate that in this situation subjects' ability to notice changes to the object strongly depends on momentary task demands. Surprisingly, frequent noticing is not guaranteed by task relevance of the changed object attribute per se, but the changed object attribute needs to be task relevant at exactly the right times. Also, the simplicity of the used objects indicates that change blindness occurs in situations where the visual short term memory load is minimal, suggesting a potential dissociation between short term memory limitations and change blindness. Finally, we found that changes may even go unnoticed if subjects are visually tracking the object at the moment of change. Our experiments suggest a highly purposive and task specific nature of human vision, where information extracted from the fixation point is used for certain computations only "just in time" when needed to solve the current goal.
BibTeX
@article{triesch2003what,
  author = {Jochen Triesch and D.H. Ballard and Mary Hayhoe and Brian Sullivan},
  title = {{What you see is what you need}},
  journal = {Journal of Vision},
  year = {2003},
  doi = {10.1167/3.1.9},
}

Loading related content...

Figure