Linear Representational Mapping Between Deep Networks and Brains

122 concepts106 papers1 proposal

How do we determine if the internal processing of a deep neural network mirrors the human brain? We cannot simply compare raw weights or individual activations directly; their coordinate systems are fundamentally incompatible. An artificial network might represent a concept within a 4096-dimensional vector space, while a human brain distributes that same concept across thousands of fMRI voxels or electrophysiological recording sites.

The breakthrough solution is linear representational mapping (often called linear neural encoding). Instead of searching for an exact one-to-one match between artificial and biological neurons, researchers train a simple linear regression model to project the high-dimensional internal activations of an AI model directly onto biological brain activity patterns.

The Intuition Behind the Linear Constraint

Why must the mapping be strictly linear? This constraint is the core conceptual foundation of the entire field. A sufficiently powerful non-linear model—such as another deep neural network—can transform almost any high-dimensional distribution into any other, effectively "manufacturing" an alignment that does not inherently exist.

A linear transformation, by contrast, can only rotate, scale, and shift the representational space. It cannot alter the underlying topology or create new relational structures. Therefore, if a simple linear model can accurately predict biological brain activity from artificial network states, it proves that the geometric structure of the information is preserved across both systems.

As noted by Antonia Karamolegkou et al. in Mapping Brains with Language Models: A Survey, while linear transformations preserve the interpretability of the shared representational space, they act as a strict mathematical test of whether deep networks develop internal representations that mirror the functional organization of the brain.

The Mechanics of Voxel-Wise Encoding

To build a linear bridge, researchers expose both the artificial model and human participants to the identical set of naturalistic stimuli (such as text, images, or audio).

Let $X \in \mathbb{R}^{N \times D}$ represent the matrix of internal activations extracted from a specific layer of a neural network across $N$ stimuli, where $D$ is the latent dimensionality of that layer. Let $Y \in \mathbb{R}^{N \times V}$ represent the recorded biological brain responses (such as fMRI BOLD signals across $V$ voxels, or electrophysiological firing rates) for the same $N$ stimuli.

We solve for a transformation matrix $W \in \mathbb{R}^{D \times V}$ using ridge regression (L2 regularization) to handle the high dimensionality and multicollinearity of the feature space:

$$\min_W \|Y - XW\|_2^2 + \lambda \|W\|_2^2$$

Once $W$ is learned on a training set, its predictive power is evaluated on held-out stimuli. If the predicted brain activity $\hat{Y} = X_{\text{test}}W$ correlates significantly with the actual observed brain activity $Y_{\text{test}}$, we conclude that the model's layer linearly encodes the features processed by that specific biological region.

To streamline this process, open-source toolboxes like DNNBrain, developed by Xiayu Chen et al. (DNNBrain: A Unifying Toolbox for Mapping Deep Neural Networks and Brains), and Net2Brain, developed by Domenic Bersch et al. (Net2Brain: a toolbox to compare artificial vision models with human brain responses), have standardized these pipelines to allow rapid cross-system comparisons across hundreds of architectures.

Mapping Hierarchical and Multimodal Alignments

When this linear mapping is applied layer-by-layer, a striking pattern emerges: the structural hierarchy of deep networks spontaneously mirrors the anatomical hierarchy of biological sensory processing.

In speech and language processing, Gopala Anumanchipalli et al. (Dissecting neural computations in the human auditory pathway using deep neural networks for speech) and Gavin Mischler et al. (Parallel hierarchical encoding of linguistic representations in the human auditory cortex and recurrent automatic speech recognition systems) demonstrated that early layers of speech models map cleanly to primary auditory regions, whereas deeper layers map to higher-level phonetic and lexical cortical areas.

This alignment is temporal as well as spatial. Ariel Goldstein (Temporal structure of natural language processing in the human brain corresponds to layered hierarchy of large language models) demonstrated that deeper layers of Large Language Models (LLMs) linearly predict brain activity at increasingly later time points during comprehension, revealing a shared temporal-hierarchical logic.

This mapping paradigm extends seamlessly across modalities:

How Training Objectives Shape Brain Geometry

Not all artificial representations align equally well with the brain. The choice of optimization objective fundamentally dictates whether a model's latent space converges toward or diverges from biological reality.

In a landmark study, Martin Schrimpf et al. (The neural architecture of language: Integrative modeling converges on predictive processing) discovered that models optimized for next-word prediction (such as GPT architectures) achieve the highest linear predictive accuracy for human language regions, providing computationally explicit evidence that the human language system is fundamentally optimized for predictive processing.

However, optimizing purely for engineering benchmarks can break this alignment. Drew Linsley et al. (Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex) observed that as modern vision networks achieve superhuman performance on ImageNet, their linear alignment with the primate visual cortex actually decreases, indicating that engineering shortcuts can diverge from biological constraints.

Conversely, high-level cognitive objectives can restore alignment. Khai Loong Aw (Training language models to summarize narratives improves brain alignment) demonstrated that training language models on narrative summarization significantly enhances their fMRI predictivity, forcing the model to build global context representations that mirror human narrative comprehension.

Closing the Loop: Encoding vs. Decoding

While encoding models map from artificial features to the brain ($X \to Y$), the inverse process—decoding ($Y \to X$)—validates the semantic fidelity of the shared space by reconstructing stimuli directly from neural activity.

                  ENCODING (X → Y)
       ┌────────────────────────────────────┐
       │                                    ▼
┌──────────────┐                     ┌──────────────┐
│  Artificial  │                     │  Biological  │
│ Latent Space │                     │ Neural Space │
└──────────────┘                     └──────────────┘
       ▲                                    │
       └────────────────────────────────────┘
                  DECODING (Y → X)

For example, Rufin VanRullen (Reconstructing faces from fMRI patterns using deep generative neural networks) used a linear regression model to translate high-dimensional fMRI voxel activation patterns into the specific coordinates of a generative model's latent space (VAE-GAN), allowing the high-fidelity reconstruction of faces seen or imagined by a subject.

Similarly, Mitja Nikolaus et al. (Modality-Agnostic fMRI Decoding of Vision and Language) utilized linear regression to project brain activity into pre-existing text and image embedding spaces, enabling the decoding of concepts regardless of whether the subject saw an image or read a text description.

This bidirectional flow confirms that linear representational mapping is not merely a statistical artifact. Instead, it serves as a functional, interpretable pathway demonstrating that deep neural networks and biological brains solve complex perceptual and cognitive problems using a shared geometric logic.

Key Foundational Work

CNN-based Encoding and Decoding of Visual Object Recognition in Space and Time (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_representational_mapping_2bc1bbe8,
  title = {linear representational mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-representational-mapping/rdr_2bc1bbe8e16d4ee1a9b55541a1067ed3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_representational_mapping_2bc1bbe8_v1,
  title = {linear representational mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-representational-mapping/rdr_2bc1bbe8e16d4ee1a9b55541a1067ed3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake