Abstract
Brain-computer interfaces enable active communication and execution of a pre-defined set of commands, such as typing a letter or moving a cursor. However, they have thus far not been able to infer more complex intentions or adapt more complex output based on brain signals. Here, we present neuroadaptive generative modelling, which uses a participant's brain signals as feedback to adapt a boundless generative model and generate new information matching the participant's intentions. We report an experiment validating the paradigm in generating images of human faces. In the experiment, participants were asked to specifically focus on perceptual categories, such as old or young people, while being presented with computer-generated, photorealistic faces with varying visual features. Their EEG signals associated with the images were then used as a feedback signal to update a model of the user's intentions, from which new images were generated using a generative adversarial network. A double-blind follow-up with the participant evaluating the output shows that neuroadaptive modelling can be utilised to produce images matching the perceptual category features. The approach demonstrates brain-based creative augmentation between computers and humans for producing new information matching the human operator's perceptual categories.
Introduction
Traditional Brain-Computer Interfaces (BCIs) typically rely on explicit mental tasks to trigger pre-defined commands. While effective for basic control, these systems struggle to capture the nuance of complex, subjective human intentions. This research, published in Scientific Reports, introduces a "neuroadaptive" framework that bridges the gap between passive brain activity and generative synthesis. By monitoring a user's natural perceptual responses, the system can guide a Generative Adversarial Network (GAN) to create entirely new, photorealistic images that align with the user's mental target.
Methodology: The Generate-Perceive-Adapt Loop
The researchers implemented a closed-loop system designed to navigate the high-dimensional latent space of a generative model using EEG feedback. The process is divided into three iterative stages:
- Generate: A Progressive GAN, pre-trained on the CelebA-HQ dataset, synthesizes high-resolution face images from latent vectors ($z$).
- Perceive: Participants are presented with a Rapid Serial Visual Presentation (RSVP) of these faces at a rate of two images per second. During this time, they are tasked with looking for a specific category (e.g., "young people" or "smiling faces").
- Adapt: The system monitors for the P300 event-related potential—a neural signature of recognition that occurs when a participant encounters a task-relevant stimulus.
To translate these neural signals into visual updates, the system employs Regularized Linear Discriminant Analysis (LDA) to classify the presence of a P300 response. A variation of the Rocchio algorithm then updates the "intention model" by calculating a new latent vector based on a weighted average of the vectors that triggered positive brain responses.
Key Insights: Navigating Latent Space
The primary innovation lies in using the brain as a "compass" rather than a direct source for reconstruction. Instead of attempting to decode an image directly from noisy EEG data, the system uses the brain's "Aha!" moment to find the correct coordinates within the GAN’s pre-existing knowledge of what faces look like.
This approach allows for passive interaction. Users do not need to perform artificial mental exercises; they simply look for what they want. The study confirmed that as more images are viewed, the model converges toward the mental target, demonstrating that the P300 signal provides sufficient information for high-dimensional optimization.
Results and Validation
The system's performance was validated through a double-blind follow-up study where participants rated the generated outputs:
- High Precision: Images generated from positive brain feedback matched the intended category in 90% of cases.
- Strong Contrast: Images derived from negative feedback (ignored stimuli) matched the category only 2.5% of the time, while random feedback yielded a 42.3% match rate.
- Qualitative Success: On a 5-point Likert scale, positive images received an average score of 4.61, indicating high representativeness of the target category.
Limitations and Future Outlook
While the current study utilized offline processing, the ultimate goal is real-time, closed-loop generation. This will require significant computational optimization to update GANs on the fly. Furthermore, the system is constrained by the biases of its underlying generative model. Future research aims to scale this methodology beyond objective categories to purely subjective concepts, such as "trustworthiness" or "attractiveness."
Summary updated