Voluntary control of semantic neural representations by imagery with conflicting visual stimulation

Ryohei Fukuma, Takufumi Yanagisawa, Shinji Nishimoto, Hidenori Sugano, Kentaro Tamura, Shota Yamamoto, Yasushi Iimura, Yuya Fujita, Satoru Oshino, Naoki Tani, Naoko Koide‐Majima, Yukiyasu Kamitani, Haruhiko Kishima

Abstract

Neural representations of visual perception are continuously shaped by top-down signals such as mental imagery and attention. However, the dynamics of these representations when imagery and perception semantically conflict remain poorly understood. This study investigates whether individuals can voluntarily modulate neural representations toward an imagined category while simultaneously viewing a conflicting visual stimulus. Using a novel closed-loop representational Brain-Computer Interface (rBCI) and electrocorticogram (ECoG) recordings from 21 patients, we demonstrate that mental imagery can successfully "pull" neural activity toward a target semantic category. Our findings reveal that this modulation is both category-dependent and temporally asymmetrical, providing new insights into the competitive interaction between sensory input and internal cognition.

Introduction

Our brains are the site of a constant tug-of-war between "bottom-up" sensory information—what we see in the world—and "top-down" cognitive signals—what we think, remember, or imagine. While it is well-established that attention can sharpen our perception of a stimulus, it has been less clear how the brain handles total semantic conflict. For instance, if you are looking at a landscape but intensely imagining a written word, does your brain’s internal representation shift toward the word, or does the physical reality of the landscape dominate?

This research explores whether we can voluntarily control these semantic neural representations and how imagery and perception compete for dominance within the visual cortex. To test this, the researchers developed a "representational Brain-Computer Interface" (rBCI) that provides real-time feedback based on the semantic content of a user's brain activity.

Methods

Participants and Data Acquisition

The study involved 21 patients with epilepsy who had electrocorticogram (ECoG) electrodes implanted for clinical monitoring. ECoG was chosen for its superior signal-to-noise ratio and temporal resolution compared to non-invasive methods like fMRI or EEG, allowing for the precise tracking of high-gamma (80–150 Hz) power associated with visual processing.

Semantic Vector Space and Decoding

To quantify the "meaning" of visual stimuli, the researchers employed a skip-gram model (a word-embedding algorithm) to map human-provided descriptions of 3,600 video scenes into a 1,000-dimensional semantic vector space.

A ridge regression decoder was trained to predict these semantic vectors based on the high-gamma power of the ECoG signals. This allowed the researchers to translate raw brain activity into a coordinate within the semantic map.

The Representational BCI (rBCI) and Conflict Task

The core of the experiment was the rBCI, a closed-loop system where subjects were shown a feedback image selected in real-time based on their decoded brain activity. The subjects' task was to "steer" the feedback image toward a target category (e.g., "word," "landscape," or "face") using only mental imagery.

In a separate "conflict task," subjects were instructed to view one category of image while imagining another. This allowed the team to measure the exact shift in neural representation caused by imagery in the presence of distracting, conflicting visual input.

Key Insights

Imagery as a Competitive Signal

The study confirms that mental imagery is not merely a faint echo of perception; it is a robust signal capable of actively shifting the neural representation even when it contradicts physical sensory input. In the conflict task, the "semantic distance" between the brain's state and the imagined category significantly decreased, proving that imagery was successfully overriding the perceived image.

The Role of Real-Time Feedback

Decoding accuracy was significantly higher in the closed-loop (BCI) condition than in open-loop settings. This suggests that the brain uses the visual feedback to "tune" its imagery strategy, effectively learning how to generate a more distinct neural signal to achieve the target goal.

Semantic Asymmetry and Anatomy

The interaction between imagery and perception is not symmetrical. For example, the shift toward imagining "words" while seeing "landscapes" occurred more rapidly than the reverse. Anatomically, while early visual areas (V1-V4) remained more tethered to the actual perceived image, higher visual areas in the ventral stream proved more susceptible to modulation by imagery.

Results

  • Decoding Performance: The system achieved a classification accuracy of approximately 70-73% for category pairs (e.g., word vs. landscape) based on ECoG signals.
  • BCI Success: All four subjects participating in the real-time rBCI task successfully controlled the system, achieving three-choice accuracy (word/landscape/face) significantly above the 33.3% chance level, with some subjects reaching ~50%.
  • Neural Modulation: The decrease in semantic distance (measured via Fisher z-transformed correlation) confirmed that voluntary imagery successfully moved the neural state toward the target semantic vector despite conflicting visual stimuli.

Discussion and Future Work

While the study demonstrates the power of rBCI, it also highlighted challenges. Certain categories, like "human faces," were harder to control than "words" or "landscapes," likely due to the prevalence of faces across various feedback images. Additionally, because the study relied on clinical patients, the sample size for the BCI task was limited to four individuals.

Looking forward, this research paves the way for advanced communication tools for patients with locked-in syndrome or ALS. Rather than simple thought-to-text, a refined rBCI could allow users to navigate a vast semantic space, providing a more intuitive and flexible way to express complex thoughts by modulating the brain's internal representational maps.

Summary updated

Abstract

Neural representations of visual perception are affected by mental imagery and attention. Although attention is known to modulate neural representations, it is unknown how imagery changes neural representations when imagined and perceived images semantically conflict. We hypothesized that imagining an image would activate a neural representation during its perception even while watching a conflicting image. To test this hypothesis, we developed a closed-loop system to show images inferred from electrocorticograms using a visual semantic space. The successful control of the feedback images demonstrated that the semantic vector inferred from electrocorticograms became closer to the vector of the imagined category, even while watching images from different categories. Moreover, modulation of the inferred vectors by mental imagery depended on both the image category and time from the initiation of imagery. The closed-loop control of the semantic vectors revealed an asymmetrical interaction between visual perception and imagery.
BibTeX
@inproceedings{fukuma2022voluntary,
  author = {Ryohei Fukuma and Takufumi Yanagisawa and Shinji Nishimoto and Hidenori Sugano and Kentaro Tamura and Shota Yamamoto and Yasushi Iimura and Yuya Fujita and Satoru Oshino and Naoki Tani and Naoko Koide‐Majima and Yukiyasu Kamitani and Haruhiko Kishima},
  title = {{Voluntary control of semantic neural representations by imagery with conflicting visual stimulation}},
  booktitle = {Communications Biology},
  year = {2022},
  url = {https://openreview.net/forum?id=kwnrMaYJmi},
}
Figure