Non-invasive neural recordings—such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG), and functional near-infrared spectroscopy (fNIRS)—are notoriously noisy. If you try to decode what a person is looking at, listening to, or feeling from a single brain recording, you run into massive hurdles: physical artifacts, sudden lapses in attention, and highly subjective cognitive biases.
Historically, researchers attempted to overcome this by building highly customized, single-subject models. However, these models often overfit to individual quirks and fail to capture the invariant, underlying representation of the stimulus itself. A powerful alternative paradigm has emerged: instead of relying on a single brain, we can aggregate neural data across multiple individuals to build a far more robust, collective decoder.
The Intuition: Constructive vs. Destructive Interference
Why does pooling data from multiple brains work so effectively? Think of a group of people watching the same movie or listening to the same speech. While their internal thoughts, distractions, and sensor noise are completely uncorrelated, their sensory and cognitive processing of the external stimulus is highly synchronized.
When we aggregate neural data across subjects, this uncorrelated, idiosyncratic noise cancels itself out (destructive interference), while the shared, stimulus-driven signal reinforces itself (constructive interference).
This effect was clearly demonstrated by Di Zhou et al. in Neural Entrainment to Natural Speech Envelope Based on Subject Aligned EEG Signals. By aligning and averaging EEG responses across multiple individuals listening to the same story, the authors suppressed idiosyncratic physiological noise and effectively doubled the accuracy of speech envelope reconstruction compared to traditional single-subject analysis. This massive boost in the signal-to-noise ratio allowed them to detect subtle neural patterns, like high-frequency gamma band tracking, that are typically completely obscured in single-subject data.
Aligning Diverse Brains into Common Spaces
To combine neural data across physically and anatomically different brains, models must map individual signals into a unified space. This aggregation can happen at different stages of the machine learning pipeline: at the raw signal level, within a shared latent space, or at the final prediction output.
Signal-Level and Latent Space Aggregation
To combine signals early on, researchers map different brains into a shared coordinate system. For instance, Cara E. Van Uden et al. utilized a technique called hyperalignment in Modeling Semantic Encoding in a Common Neural Representational Space to pool fMRI data across subjects into a common representational space. The resulting universal semantic encoding models actually outperformed models trained on a single person's own data, providing a more precise mapping of semantic tuning.
Similarly, André Ofner et al. designed a multi-view generative model in Shared Generative Representation of Auditory Concepts and EEG to Reconstruct Perceived and Imagined Music to map both EEG and musical features into a shared latent space. Training on aggregated data helped the system filter out individual-specific noise and isolate universal rhythmic signatures, significantly improving the temporal alignment of reconstructed music.
Prediction-Level Ensembling
Alternatively, we can train separate decoders for each individual and aggregate their final predictions. In From Brain Space to Distributional Space: The Perilous Journeys of fMRI Decoding, Gosse Minnema et al. showed that while individual fMRI decoders struggle to accurately reconstruct distributional semantic vector spaces, averaging the predicted vectors across multiple subjects leverages shared semantic representations to cancel out idiosyncratic noise, yielding highly robust global vector reconstructions.
Taking this further, Takahiro Ogawa et al. developed a mathematical framework called KESLME in Multi-classifier Decision: Integration of Multiple Brain Activity-based Classifications to fuse noisy individual EEG classifications. By aggregating decisions across multiple participants to compensate for individual lapses in attention, their system achieved near-perfect image classification accuracy without ever processing the image pixels directly.
Brainsourcing: The Crowd as a Distributed Sensor
This phenomenon has enabled "brainsourcing"—using a crowd of human brains as a distributed, passive sensor network to label content automatically without requiring active user input.
[ Stimulus: Video/Image ]
│
├─► Brain 1 (EEG/fNIRS) ──► Noisy Prediction ──┐
├─► Brain 2 (EEG/fNIRS) ──► Noisy Prediction ──┼─► [ Consensus Aggregation ] ──► Stable Consensus Label
└─► Brain N (EEG/fNIRS) ──► Noisy Prediction ──┘
The power of this consensus-based approach spans several domains:
- Abstract Concepts: In Understanding action concepts from videos and brain activity through subjects’ consensus, Jacopo Cavazza et al. demonstrated that averaging neural signal predictions across participants filters out subjective biases, revealing a shared conceptual representation that serves as a robust ground truth for training models on abstract action categories.
- Media Classification: A similar group-level framework was exploited by Pouya Ghaemmaghami et al. in Movie Genre Classification by Exploiting MEG Brain Signals, proving that aggregating neural response data across a group transcends individual variations in perception to significantly improve content classification.
- Affective Annotation: For emotional labeling, Kalle Mäkelä et al. utilized fNIRS in Crowdsourcing Affective Annotations Via fNIRS-BCI to generate reliable emotional labels for images from passive crowd signals. Expanding on this, Yoelvis Moreno-Alcayde et al. implemented a multi-user voting strategy on EEG-based predictions in Affective annotation of videos from EEG-based crowdsourcing, achieving up to 80% accuracy in predicting valence and arousal with a crowd of 14 participants.
- Temporal Attention: Beyond static labels, Yoelvis Moreno-Alcayde et al. showed in Brainsourcing for temporal visual attention estimation that aggregating collective EEG signals allows a system to accurately pinpoint exactly when a video is most attention-grabbing, mapping temporal visual attention without requiring eye-trackers or manual annotations.
Scaling AI Models with Multi-Brain Tuning
The benefits of cross-subject aggregation extend back into artificial intelligence itself. Instead of just using brains to decode stimuli, we can use multi-subject data to train better, more generalizable machine learning models.
In Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models, Omer Moussa et al. introduced a "multi-brain-tuning" paradigm where speech AI models are fine-tuned to jointly predict brain activity from an entire group of people.
By forcing the model to align with an aggregate population rather than a single individual, the model learns to ignore individual-specific noise and focus on universal, population-level processing mechanisms. This approach made the brain-alignment models five times more data-efficient and allowed them to successfully generalize to entirely new individuals whose brain data was never seen during training, paving the way for scalable, universal cognitive models.
Key Foundational Work
Shared Generative Representation of Auditory Concepts and EEG to Reconstruct Perceived and Imagined Music (2018) — contributed 2 concept elements to this direction.
[Papers] Multi-classifier Decision: Integration of Multiple Brain Activity-based Classifications (2018) — contributed 1 concept elements to this direction.
Movie Genre Classification by Exploiting MEG Brain Signals (2015) — contributed 1 concept elements to this direction.