Disentangling presentation and processing times in the brain

Laurent Caplette, Robin A. A. Ince, Karim Jerbi, Frédéric Gosselin

TL;DR

This paper introduces a novel "reverse correlation paradigm" to precisely track how the brain processes visual information that arrives at different moments during a single eye fixation. They found that the brain doesn't just passively receive information but actively samples and processes it in a task-dependent manner, suggesting top-down control over visual perception.

What Problem Does This Paper Address?

We often perceive visual object recognition as instantaneous. However, this is an illusion. Both the processing of visual information and the eye's fixation on an object take hundreds of milliseconds. This means that over the course of a single fixation, our eyes receive a continuous stream of visual input. The crucial question is: how does the brain handle this continuous stream? Does it process all incoming information equally, or does it selectively attend to and process information that arrives at different moments within that fixation? Traditional methods often struggle to disentangle the timing of stimulus presentation from the timing of brain processing, making it difficult to understand how the brain dynamically samples and integrates information over time.

Methods

The core innovation here is a novel reverse correlation paradigm. Instead of presenting a whole stimulus and observing the brain's response, they:

  1. Randomly sampled face features: They presented parts of a face (eyes, mouth) gradually over 200 ms, revealing them at random moments. This allowed them to control when specific visual information was available to the retina.
  2. Recorded EEG: They precisely measured brain activity (EEG) while participants performed tasks (gender or expression recognition).
  3. Reverse correlation: They then correlated the EEG activity at each time point and electrode with the specific features that were visible at each "stimulus moment" (the moment information was presented). This allowed them to reconstruct how the brain's processing varied depending on when the information was initially received.

This method allows for millisecond-precision analysis of the processing time course of specific visual information.

Datasets

  • Karolinska Directed Emotional Faces (KDEF) database: Used to select 264 color images of faces.

Key Insights and Ideas

  • Active, not passive, visual processing: The brain doesn't just passively absorb visual input. It actively samples and processes information that arrives at different moments within a single eye fixation.
  • "Gating" mechanisms: Certain neural events, like the N170 ERP component, seem to act as gates, controlling when information is processed and transferred across hemispheres. This is particularly evident in how information from the eyes is processed for the gender task.
  • Temporal compression and integration: The brain appears to integrate information over time, sometimes compressing the processing of information received at different moments into a similar processing window.
  • Task-dependent processing: The way the brain processes information arriving at different moments is significantly influenced by the task the participant is performing. This points to top-down control.
  • Behavioral relevance: Variations in how the brain processes information at different moments are directly related to how that information is used for behavioral responses.

Key Results

  • Differential processing based on arrival time: The study found significant differences in EEG activity depending on the moment information was presented to the retina. This processing varied across electrodes and latencies.
  • N170's role in gating: Disruptions around 160-200 ms after face onset, linked to the N170, suggest its involvement in gating information processing, especially for task-relevant features like the eyes in the gender task.
  • Temporal compression: In some cases, information received earlier was "maintained" for longer, leading to a temporal compression where all information was processed around the same time, regardless of its arrival.
  • Task modulation: Significant interactions between stimulus moment and task were observed, indicating that the processing of information arriving at different times was not uniform across tasks.
  • Behavioral correlation: Mutual information analysis showed that variations in EEG activity across stimulus moments were related to variations in behavioral accuracy.

Limitations and Future Work

  • Generalizability: While faces were used as stimuli, future work could explore this paradigm with other types of visual stimuli, including dynamic ones like facial expressions or naturalistic movies, to understand how evolving information is integrated.
  • Spatial Resolution: The study used EEG, which has limited spatial resolution. Future research using more spatially resolved methods like MEG could provide a deeper understanding of how information is processed, accumulated, integrated, and transferred across different brain regions.
  • Bottom-up vs. Top-down: While the study provides strong evidence for top-down modulations, the authors acknowledge that bottom-up effects (like adaptation or priming) might still play a role, though their experiments suggest these are not the primary drivers of the observed differences.

The authors propose that their novel method offers a promising avenue for future research into the accumulation and integration of information during object recognition, allowing visualization of simultaneous processing of information received at different times.

Summary updated

Abstract

Abstract Visual object recognition seems to occur almost instantaneously. However, not only does it require hundreds of milliseconds of processing, but our eyes also typically fixate the object for hundreds of milliseconds. Consequently, information reaching our eyes at different moments is processed in the brain together. Moreover, information received at different moments during fixation is likely to be processed differently, notably because different features might be selectively attended at different moments. Here, we introduce a novel reverse correlation paradigm that allows us to uncover with millisecond precision the processing time course of specific information received on the retina at specific moments. Using faces as stimuli, we observed that processing at several electrodes and latencies was different depending on the moment at which information was received. Some of these variations were caused by a disruption occurring 160-200 ms after the face onset, suggesting a role of the N170 ERP component in gating information processing; others hinted at temporal compression and integration mechanisms. Importantly, the observed differences were not explained by simple adaptation or repetition priming, they were modulated by the task, and they were correlated with differences in behavior. These results suggest that top-down routines of information sampling are applied to the continuous visual input, even within a single eye fixation.
BibTeX
@article{caplette2020disentangling,
  author = {Laurent Caplette and Robin A. A. Ince and Karim Jerbi and Frédéric Gosselin},
  title = {{Disentangling presentation and processing times in the brain}},
  journal = {NeuroImage},
  year = {2020},
  doi = {10.1016/j.neuroimage.2020.116994},
}
Figure