Abstract
Neural representations of visual perception are continuously shaped by top-down signals such as mental imagery and attention. However, the dynamics of these representations when imagery and perception semantically conflict remain poorly understood. This study investigates whether individuals can voluntarily modulate neural representations toward an imagined category while simultaneously viewing a conflicting visual stimulus. Using a novel closed-loop representational Brain-Computer Interface (rBCI) and electrocorticogram (ECoG) recordings from 21 patients, we demonstrate that mental imagery can successfully "pull" neural activity toward a target semantic category. Our findings reveal that this modulation is both category-dependent and temporally asymmetrical, providing new insights into the competitive interaction between sensory input and internal cognition.
Introduction
Our brains are the site of a constant tug-of-war between "bottom-up" sensory information—what we see in the world—and "top-down" cognitive signals—what we think, remember, or imagine. While it is well-established that attention can sharpen our perception of a stimulus, it has been less clear how the brain handles total semantic conflict. For instance, if you are looking at a landscape but intensely imagining a written word, does your brain’s internal representation shift toward the word, or does the physical reality of the landscape dominate?
This research explores whether we can voluntarily control these semantic neural representations and how imagery and perception compete for dominance within the visual cortex. To test this, the researchers developed a "representational Brain-Computer Interface" (rBCI) that provides real-time feedback based on the semantic content of a user's brain activity.
Methods
Participants and Data Acquisition
The study involved 21 patients with epilepsy who had electrocorticogram (ECoG) electrodes implanted for clinical monitoring. ECoG was chosen for its superior signal-to-noise ratio and temporal resolution compared to non-invasive methods like fMRI or EEG, allowing for the precise tracking of high-gamma (80–150 Hz) power associated with visual processing.
Semantic Vector Space and Decoding
To quantify the "meaning" of visual stimuli, the researchers employed a skip-gram model (a word-embedding algorithm) to map human-provided descriptions of 3,600 video scenes into a 1,000-dimensional semantic vector space.
A ridge regression decoder was trained to predict these semantic vectors based on the high-gamma power of the ECoG signals. This allowed the researchers to translate raw brain activity into a coordinate within the semantic map.
The Representational BCI (rBCI) and Conflict Task
The core of the experiment was the rBCI, a closed-loop system where subjects were shown a feedback image selected in real-time based on their decoded brain activity. The subjects' task was to "steer" the feedback image toward a target category (e.g., "word," "landscape," or "face") using only mental imagery.
In a separate "conflict task," subjects were instructed to view one category of image while imagining another. This allowed the team to measure the exact shift in neural representation caused by imagery in the presence of distracting, conflicting visual input.
Key Insights
Imagery as a Competitive Signal
The study confirms that mental imagery is not merely a faint echo of perception; it is a robust signal capable of actively shifting the neural representation even when it contradicts physical sensory input. In the conflict task, the "semantic distance" between the brain's state and the imagined category significantly decreased, proving that imagery was successfully overriding the perceived image.
The Role of Real-Time Feedback
Decoding accuracy was significantly higher in the closed-loop (BCI) condition than in open-loop settings. This suggests that the brain uses the visual feedback to "tune" its imagery strategy, effectively learning how to generate a more distinct neural signal to achieve the target goal.
Semantic Asymmetry and Anatomy
The interaction between imagery and perception is not symmetrical. For example, the shift toward imagining "words" while seeing "landscapes" occurred more rapidly than the reverse. Anatomically, while early visual areas (V1-V4) remained more tethered to the actual perceived image, higher visual areas in the ventral stream proved more susceptible to modulation by imagery.
Results
- Decoding Performance: The system achieved a classification accuracy of approximately 70-73% for category pairs (e.g., word vs. landscape) based on ECoG signals.
- BCI Success: All four subjects participating in the real-time rBCI task successfully controlled the system, achieving three-choice accuracy (word/landscape/face) significantly above the 33.3% chance level, with some subjects reaching ~50%.
- Neural Modulation: The decrease in semantic distance (measured via Fisher z-transformed correlation) confirmed that voluntary imagery successfully moved the neural state toward the target semantic vector despite conflicting visual stimuli.
Discussion and Future Work
While the study demonstrates the power of rBCI, it also highlighted challenges. Certain categories, like "human faces," were harder to control than "words" or "landscapes," likely due to the prevalence of faces across various feedback images. Additionally, because the study relied on clinical patients, the sample size for the BCI task was limited to four individuals.
Looking forward, this research paves the way for advanced communication tools for patients with locked-in syndrome or ALS. Rather than simple thought-to-text, a refined rBCI could allow users to navigate a vast semantic space, providing a more intuitive and flexible way to express complex thoughts by modulating the brain's internal representational maps.
Summary updated