Attentionally modulated subjective images reconstructed from brain activity

Tomoyasu Horikawa, Yukiyasu KamitanibioRxiv 2020 · DOI

TL;DR

This paper demonstrates that we can reconstruct images from brain activity that reflect what a person is attending to, even when multiple images are present. By using deep neural networks and fMRI, the researchers show that attention biases neural representations, leading to reconstructions that align with subjective visual experience, not just the raw visual input.

What Problem Does This Paper Address?

The field of brain-based image reconstruction has made strides in decoding visual information from brain activity, essentially "seeing what you see." However, these methods often focus on reconstructing the presented stimulus accurately. This overlooks a crucial aspect of human vision: attention. Our subjective experience of the world isn't just a passive reception of sensory input; it's actively shaped by what we choose to focus on. This paper tackles the question of how top-down attention influences neural representations of visual stimuli and whether this influence can be captured and reflected in reconstructed images.

Methods

The core of this research relies on a "deep image reconstruction" approach. This method leverages the hierarchical structure of deep neural networks (DNNs), specifically a VGG19 model, which has been shown to have a correspondence with visual cortex representations.

  1. Training Decoders: fMRI data from subjects viewing single natural images was used to train linear regression models (decoders). These decoders learn to predict DNN features from fMRI voxel patterns in the visual cortex.
  2. Image Reconstruction: For attention trials, where subjects viewed superimposed images and were instructed to attend to one, the trained decoders were used to extract DNN features from the fMRI data. These decoded features were then fed into a generative adversarial network (GAN)-based deep generator network (DGN). The DGN was optimized to produce an image whose DNN features closely matched the decoded brain features, effectively reconstructing the attended visual experience.
  3. Behavioral Evaluation: To validate the reconstructions, human raters were asked to identify which of two candidate images (attended vs. unattended) was more similar to the reconstructed image.

Datasets

The study utilized natural image datasets for training and testing. Specifically, they used images from the ImageNet database (2011, fall release) for stimuli. For training the DNN models, they used a large dataset of natural images. The fMRI data was collected from five healthy subjects across two experimental sessions: a training session and a test session. Some data for the first three subjects was reused from a previous study, while new data was collected for the last two subjects.

Key Insights and Ideas

  • Attention Biases Reconstruction: The most striking finding is that reconstructions from brain activity during attention trials consistently resembled the attended image more than the unattended one, even when both were physically present. This highlights that brain activity reflects subjective experience, not just raw sensory input.
  • Hierarchical Influence of Attention: Attentional modulation wasn't confined to early visual areas; it was observed across a broad range of hierarchical visual representations within the DNN, mirroring the brain-DNN correspondence.
  • DNNs as Models for Visual Experience: The study reinforces the idea that DNNs can serve as valuable models for understanding how the brain represents visual information and subjective experience. The hierarchical nature of DNNs allows for a more nuanced examination of how attention affects these representations at different levels.
  • Subjective Experience vs. Objective Stimulus: The results demonstrate that top-down attention can actively override stimulus-driven responses, shaping neural representations to align with what we perceive, not just what is physically presented.

Key Results

  • Behavioral Validation: Human raters could identify the attended image from the reconstruction with significantly higher accuracy than chance (56.2% for attention trials vs. 83% for single-image trials).
  • Contrast Modeling: Reconstructions could be modeled by superimposed images with contrasts biased towards the attended image, reflecting the subjective experience of enhanced contrast due to attention. This bias was observed to be around 60% vs. 40% for attended vs. unattended stimuli.
  • Brain-DNN Correspondence: Attentional modulations were found to mirror the brain-DNN correspondence, with similar patterns observed across different visual areas and DNN layers.
  • Feature Specificity: While early visual areas (V1-V3) were crucial for general stimulus representation, middle-to-higher DNN layers showed a stronger link to attentional modulation.

Limitations and Future Work

  • Lack of Explicit Attention Strategy: The authors acknowledge that the absence of explicit instructions on how subjects should direct their attention might explain some of the variability observed across subjects. Future work could explore how different attention strategies impact reconstructions.
  • Individual Differences: Significant performance differences were noted across subjects, suggesting individual variations in attentional control capabilities. Further research could investigate the psychological and neural correlates of these differences.
  • Biased Contrast as Approximation: While biased image contrast effectively modeled attentional modulation, the authors note it's a "rough approximation." Future work could explore more sophisticated models that capture the full complexity of attentional effects.
  • Brain-Like DNNs: The authors suggest that using more "brain-like" DNNs could potentially enhance the ability to reveal finer-grained contents of attentionally modulated visual experience.
  • Real-time Applications: The ability to reconstruct subjective images from single-trial fMRI data opens possibilities for real-time applications, such as brain-based communication and creation.

Summary updated

Abstract

Summary Visual image reconstruction from brain activity produces images whose features are consistent with the neural representations in the visual cortex given arbitrary visual instances [1–3], presumably reflecting the person’s visual experience. Previous reconstruction studies have been concerned either with how stimulus images are faithfully reconstructed or with whether mentally imagined contents can be reconstructed in the absence of external stimuli. However, many lines of vision research have demonstrated that even stimulus perception is shaped both by stimulus-induced processes and top-down processes. In particular, attention (or the lack of it) is known to profoundly affect visual experience [4–8] and brain activity [9–21]. Here, to investigate how top-down attention impacts the neural representation of visual images and the reconstructions, we use a state-of-the-art method (deep image reconstruction [3]) to reconstruct visual images from fMRI activity measured while subjects attend to one of two images superimposed with equally weighted contrasts. Deep image reconstruction exploits the hierarchical correspondence between the brain and a deep neural network (DNN) to translate (decode) brain activity into DNN features of multiple layers, and then create images that are consistent with the decoded DNN features [3, 22, 23]. Using the deep image reconstruction model trained on fMRI responses to single natural images, we decode brain activity during the attention trials. Behavioral evaluations show that the reconstructions resemble the attended rather than the unattended images. The reconstructions can be modeled by superimposed images with contrasts biased to the attended one, which are comparable to the appearance of the stimuli under attention measured in a separate session. Attentional modulations are found in a broad range of hierarchical visual representations and mirror the brain–DNN correspondence. Our results demonstrate that top-down attention counters stimulus-induced responses and modulate neural representations to render reconstructions in accordance with subjective appearance. The reconstructions appear to reflect the content of visual experience and volitional control, opening a new possibility of brain-based communication and creation.
BibTeX
@misc{horikawa2020attentionally,
  author = {Tomoyasu Horikawa and Yukiyasu Kamitani},
  title = {{Attentionally modulated subjective images reconstructed from brain activity}},
  year = {2020},
  doi = {10.1101/2020.12.27.424510},
  howpublished = {bioRxiv},
}
Figure