Abstract
Neural decoding via the latent space of deep neural network models can infer perceived and imagined images from neural activities, even when the image is novel to the decoder. While brain-computer interfaces (BCIs) leveraging these latent spaces have the potential to allow subjects to retrieve intended information from massive datasets, such systems have not been realized until now. In this study, we implemented a closed-loop BCI to retrieve images from a pool of 2.3 million candidates based on latent vectors inferred from electrocorticographic (ECoG) signals. Using the latent space of the Contrastive Language-Image Pretraining (CLIP) model, subjects achieved retrieval accuracies exceeding 80%. Conversely, retrieval failed when using the AlexNet latent space. These results demonstrate that humans can navigate large-scale semantic data via a closed-loop BCI when utilizing an appropriate, brain-aligned latent space.
Introduction
Traditional brain-computer interfaces (BCIs) have historically focused on motor cortical activity, translating the intent to move into digital commands. While life-changing for many, these systems are less effective for patients with conditions like advanced Amyotrophic Lateral Sclerosis (ALS), where motor signals eventually deteriorate. Furthermore, "semantic search"—the ability to find an image or concept based on a mental representation—is difficult to automate because internal mental images are high-dimensional and often lack the clarity of direct perception.
This research addresses these challenges by shifting the focus from motor commands to visual-semantic decoding. By leveraging the shared "language" of modern deep learning models, the researchers developed a system that allows users to navigate massive, real-world datasets using only their thoughts.
Methodology
Neural Recording and Decoding
The study utilized electrocorticography (ECoG) to record high-frequency brain signals (high-γ power) from electrodes implanted in the visual, temporal, and parietal cortices of eleven epilepsy patients. A ridge regression model was trained to map these brain signals to the latent vectors (mathematical representations) of images within two different AI architectures:
- CLIP: A model trained to align images with natural language descriptions.
- AlexNet: A classic convolutional neural network designed for image classification.
The Closed-Loop Retrieval Task
The core innovation of this study is the closed-loop feedback cycle. During the online task, the system decoded the subject's brain activity every 250ms to infer a latent vector. This vector was then used to search a database of 2,283,429 high-quality images (sourced from Unsplash) to find the closest match.
When the system displayed the retrieved "feedback" image, the subject could cognitively modulate their mental imagery to "nudge" the system closer to the target category. For example, if the subject intended to find a "tool" but the system displayed a "human," the subject would intensify their mental imagery of a tool to correct the trajectory of the search.
Key Results
High-Accuracy Retrieval at Scale
The system successfully retrieved intended images from the pool of 2.3 million candidates, a scale far beyond previous BCI studies. Using the CLIP latent space, one subject (EC01) achieved 82.5% accuracy, while another (EA02) reached 92.5% accuracy after several days of training. The system maintained a low latency of approximately 174ms, enabling a fluid and responsive interaction.
The Critical Role of Latent Space
A major finding was that the choice of AI model determines the BCI's controllability. While both CLIP and AlexNet were effective at decoding images that subjects were actively viewing, only CLIP supported volitional imagery. In the imagery task, AlexNet’s performance dropped to near-chance levels (~50%).
Interestingly, the researchers discovered an "opposite direction" phenomenon: in a modulation task, imagining an object moved the decoded CLIP vector toward the target, but moved the AlexNet vector away from it. This suggests that CLIP’s latent space, which integrates visual and linguistic information, is more aligned with human volitional thought than the purely visual features of AlexNet.
Learning to Navigate
The study also demonstrated a clear learning curve. Subject EA02 improved their accuracy from 52.5% on the first day to 92.5% by the sixth day. This suggests that users can "learn" to use a semantic BCI through feedback, much like learning to operate a physical prosthetic.
Discussion and Limitations
While the results are promising, the study is limited by its small sample size and the invasive nature of ECoG recordings, which restricts the technology's current use to clinical settings. Additionally, individual differences in imagery vividness (such as aphantasia) likely influence BCI performance.
Future work will explore adapting these visual-semantic decoders for non-invasive methods like fMRI or EEG. Although these methods currently lack the signal-to-noise ratio of ECoG, the success of the CLIP-based closed-loop system provides a roadmap for more accessible, high-dimensional neural decoding. By aligning AI latent spaces with human cognition, we can create more intuitive interfaces for those who have lost the ability to communicate through traditional means.
Summary updated