Aligning Human Brain Activity with Artificial Latent Spaces

23 concepts24 papers1 proposal

When we look at an image, listen to a story, or imagine a concept, our brains generate complex, high-dimensional patterns of biological activity. In computational neuroscience and machine learning, a major goal is to decode these patterns—translating functional Magnetic Resonance Imaging (fMRI), Electroencephalography (eeg), or Magnetoencephalography (meg) signals back into natural language or visual images.

However, doing this directly presents a fundamental challenge known as the modality gap. Biological brain signals are continuous, noisy, spatio-temporal, and highly variable across individuals. In contrast, modern generative models operate on discrete tokens or highly structured, deterministic vector spaces. Directly mapping raw neural pulses to text tokens or image pixels is an ill-posed inverse problem; there is no inherent shared structure between a voltage fluctuation in the cortex and a word in a vocabulary.

To solve this, researchers do not try to translate brain signals directly into outputs. Instead, they build a shared latent space that acts as a semantic bridge, aligning the internal representations of the human brain with the latent spaces of pre-trained artificial intelligence models.

The Core Intuition: Shared Semantic Spaces

To understand why this works, consider how modern AI models represent meaning. A multimodal model like CLIP projects both text and images into a unified embedding space where distance corresponds to semantic similarity: the vector for the word "dog" sits close to the vector for an image of a dog.

The core insight of recent neural decoding research is that we can treat human brain activity as just another modality to be projected into this same shared space.

[ Biological Brain Activity ] ---> ( Brain Encoder ) ---\
                                                         \
[ Natural Language Tokens   ] ---> ( Text Encoder )  -----> [ Shared Semantic Latent Space ]
                                                         /
[ Visual Image Pixels       ] ---> ( Image Encoder ) ---/

Early foundations for this idea were laid by researchers like Pereira et al., who mapped fMRI voxel patterns to a latent topic model trained on Wikipedia, allowing them to generate clouds of semantically related words from brain scans. As natural language processing advanced, Matsuo et al. highlighted that the lack of direct structural alignment between physiological signals and abstract text spaces was the primary barrier to open-vocabulary decoding.

By finding a mathematical mapping—often a learned linear projection or a shallow neural network—that translates brain recordings into an established artificial latent space, researchers can bypass the need to teach a model how to generate text or images from scratch.

Leveraging Pre-trained Anchors for Generation

Once brain signals are successfully projected into an artificial latent space (such as CLIP or an LLM's embedding space), we can unlock the capabilities of frozen, state-of-the-art generative models.

EEG-to-Image Reconstruction

In visual decoding, frameworks like DreamDiffusion, developed by Bai et al., use contrastive learning to align non-invasive EEG signals with the pre-trained latent space of CLIP. Because Stable Diffusion is already conditioned to generate high-quality images from CLIP embeddings, the aligned EEG representations can be fed directly into the diffusion model.

Similarly, Ferrante et al. utilized knowledge distillation from CLIP into a specialized convolutional network to decode visual representations from EEG, achieving highly accurate semantic reconstructions. To capture richer structural details beyond simple category matching, Zhang et al. expanded this alignment by mapping EEG signals to a combination of text descriptions, images, and depth maps simultaneously.

When dealing with fMRI, Du et al. demonstrated with MindDiffuser that aligning brain activity to both the semantic features (CLIP) and structural features (Generative Adversarial Networks) allows for the reconstruction of images that preserve both the meaning and the precise geometric layout of the original visual stimulus.

Brain Signals as Soft Prompts for LLMs

For text generation, instead of training a language model to understand raw brain waves, researchers treat the aligned neural signals as "soft prompts" or prefix tokens for Large Language Models (LLMs).

In the Thought2Text framework by Mishra et al., EEG signals are aligned with visual and linguistic features, allowing an LLM to generate coherent descriptions of what a subject is viewing. For auditory decoding, Chen et al. introduced BP-GPT, which converts 3D fMRI voxel data into sequential prompts for a language model, using a specialized alignment token to synchronize the timing of brain responses with the flow of a narrated story.

Taking this a step further, Jiang et al. developed NeuroLM, a foundation model that tokenizes EEG signals and treats them directly as a "foreign language," aligning them natively with the internal token space of an LLM to perform multiple downstream tasks without task-specific architectures.

Taming Biological Noise and Subject Variability

Aligning these spaces is difficult because biological data is incredibly messy. Two major hurdles prevent clean alignment: the low signal-to-noise ratio of non-invasive recordings and the anatomical differences between individual human brains.

To address noisy, spatio-temporal data, modern architectures rely on self-supervised pre-training. For instance, Wang et al. developed a framework (CET-MAE) that uses a high-ratio masked autoencoder to force the network to learn the underlying temporal structure of EEG signals before attempting to align them with text via contrastive learning.

[ Raw, Noisy EEG Signal ] 
          │
          ▼
[ Mask out 70-80% of signal ] ──> [ Masked Autoencoder ] ──> [ Robust Temporal Features ]
                                                                        │
                                                                        ▼
                                                            [ Contrastive Alignment ]
                                                                        │
                                                                        ▼
                                                            [ Target Semantic Space ]

To handle the fact that every brain is shaped differently, researchers use intermediate alignment spaces. Sun et al. developed FAA-GAN, which maps the unique fMRI responses of different subjects into a single, shared "functional common space" before attempting cross-modal mapping to visual features.

When paired data (such as simultaneous recordings of brain activity and complex physical actions) is entirely missing, text can serve as an explicit intermediary. Zhu et al. demonstrated this in fMRI2GES, using text embeddings as a "semantic bridge" to map raw brain activity to continuous physical co-speech gestures, bypassing the lack of direct brain-to-gesture datasets.

Do Artificial and Biological Substrates Truly Align?

The success of these cross-modal alignment techniques raises a deeper neuroscientific question: do deep neural networks and human brains organize semantic information in fundamentally similar ways?

Accumulating evidence suggests a surprising degree of convergence. Artemova et al. surveyed the intersection of neurolinguistics and NLP, noting that the high-dimensional vector spaces of modern language models increasingly mirror biological language processing patterns.

Specifically, Jat et al. demonstrated that the internal representations of models like BERT correlate strongly with Magnetoencephalography (MEG) recordings of humans reading sentences, allowing artificial vectors to predict physical brain activity. Furthermore, Oota showed that generic, off-the-shelf word embeddings (like GloVe or Word2Vec) can predict spatial patterns of fMRI brain activation for concrete nouns without requiring laboriously hand-crafted features, confirming that artificial vector spaces capture real organizational principles of human cortical topography.

This biological-artificial parallel can be optimized intentionally. Rajabi et al. demonstrated that if you fine-tune an artificial image encoder to better match human visual perception judgments, its latent space becomes significantly easier to align with EEG, MEG, and fMRI signals. By making the artificial substrate more "human-aligned," the downstream task of decoding complex thoughts from the brain becomes inherently more accurate.

Key Foundational Work

Generating Natural Language Descriptions for Semantic Representations of Human Brain Activity (2016) — contributed 1 concept elements to this direction.
Generating Text from Functional Brain Images (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modal_semantic_gap_cross_modal_sem_08cd7476,
  title = {cross-modal semantic gap (cross-modal semantic alignment)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-semantic-gap-cross-modal-semantic-alignment/rdr_08cd74766ae846c8a340349f29cc1f5e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_modal_semantic_gap_cross_modal_sem_08cd7476_v1,
  title = {cross-modal semantic gap (cross-modal semantic alignment)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-semantic-gap-cross-modal-semantic-alignment/rdr_08cd74766ae846c8a340349f29cc1f5e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake