Variance Partitioning for Representational Analysis

30 concepts32 papers1 proposal

When studying complex systems like the human brain or deep neural networks, we often want to know what kind of information a specific layer or region represents. For instance, does a "scene-selective" brain area respond to low-level 2D pixel features, mid-level 3D geometry, or high-level semantic categories?

The fundamental challenge is that these features are deeply entangled in natural data. Images of wide-open outdoor landscapes (high-level semantics) naturally contain specific high-frequency spatial layouts (mid-level) and distinct color distributions (low-level). If a computational model based on 3D layout successfully predicts brain activity, we cannot immediately conclude that the brain is processing 3D layout. The model might simply be succeeding because it correlates with low-level features. As Mark D. Lescroart et al. demonstrated in Fourier power, subjective distance, and object categories all provide plausible models of BOLD responses in scene-selective visual areas, competing theories of vision often explain brain activity equally well precisely because they describe the same shared information.

To move past aggregate correlation measures—which Hamed Nili et al. note struggle to isolate specific driving features in A Toolbox for Representational Similarity Analysis—researchers need a way to untangle these correlated predictors. This is where variance partitioning comes in.

The Intuition of Variance Partitioning

Variance partitioning is a statistical technique that decomposes the total explained variance of a target signal into unique and shared contributions from multiple predictor models.

Imagine you have a target variable $Y$ (such as fMRI responses in a visual area) and two correlated models: Model $A$ (low-level edges) and Model $B$ (high-level object categories). If we fit each model individually, they will both explain a certain percentage of the variance ($R^2_A$ and $R^2_B$). Because the models are correlated, their explained variances overlap.

To find out what Model $B$ uniquely contributes over and above Model $A$, we look at the predictive gain. We fit a combined model containing both $A$ and $B$, yielding a total explained variance $R^2_{A+B}$. The unique contribution of Model $B$ is simply the difference:

$$\text{Unique}(B) = R^2_{A+B} - R^2_A$$

Similarly, the unique contribution of Model $A$ is:

$$\text{Unique}(A) = R^2_{A+B} - R^2_B$$

The remaining explained variance is the shared variance, captured by both models due to their collinearity:

$$\text{Shared} = R^2_A + R^2_B - R^2_{A+B}$$

By calculating these intersections and differences, as explained by Wendy A. de Heer et al. in The Hierarchical Cortical Organization of Human Speech Processing, we can determine whether a model captures novel information or is merely a redundant reflection of simpler baseline features. To ensure these estimates generalize robustly without overfitting, researchers often employ cross-validated variance partitioning, a refinement highlighted by Bruno L. Giordano et al. in Intermediate acoustic-to-semantic representations link behavioral and neural responses to natural sounds.

Disentangling Biological Hierarchies

In cognitive neuroscience, variance partitioning has become essential for mapping how sensory pathways transform raw inputs into abstract concepts.

For example, when listening to a story, the brain processes speech through a rapid succession of acoustic, phonemic, and semantic stages. Xue Gong et al. used variance partitioning in Phonemic segmentation of narrative speech in human cerebral cortex to show that specific cortical regions process adjacent pairs of phonemes (diphones) uniquely, explaining neural variance that cannot be accounted for by raw acoustics or downstream semantics. Similarly, SUBBA REDDY OOTA isolated the unique contribution of syntactic formalisms in How does the brain process syntactic structure while listening?, proving that the brain tracks structural linguistic features independently of semantic embeddings.

A similar logic applies to vision. Markus W. Badwal et al. used the technique in The scope and limits of fine-grained image and category information in the ventral visual pathway to demonstrate that while image-specific visual features dominate the ventral visual stream, certain regions like the lateral occipital complex retain a small but statistically unique portion of variance dedicated strictly to fine-grained category information. By controlling for shape descriptors like silhouette and curvature, as explored by Andrea Leo et al. in Shape coding in occipito-temporal cortex relies on object silhouette, curvature, and medial axis, variance partitioning allows researchers to isolate the exact feature dimensions driving neural representation.

Probing the Internal Representations of AI

The utility of variance partitioning extends directly into artificial intelligence. Deep neural networks are often criticized as "black boxes" where internal representations are as entangled as those in the human brain.

Mathieu Aubry et al. leveraged variance decomposition in Understanding Deep Features with Computer-Generated Imagery to analyze how convolutional neural networks represent independent scene factors like viewpoint, category, and lighting across different layers, mapping out how networks gradually disentangle these variables.

Furthermore, when we use large language models (LLMs) or self-supervised speech models to predict brain activity, we want to know why they align so well. Aditya R. Vaidya et al. applied variance partitioning in Self-Supervised Models of Audio Effectively Explain Human Cortical Responses to Speech to show that self-supervised audio models capture novel structural information from speech that goes far beyond hand-crafted acoustic baselines.

Conversely, variance partitioning can expose when an AI model relies on superficial shortcuts. Carolin Schuster et al. partitioned the variance of BERT's internal states in From BERT's Point of View: Revealing the Prevailing Contextual Differences to discover that simple word position within a sentence explains a dominant share of its representational structure, often overshadowing complex syntax.

Methodological Implementations

To scale this analysis across hundreds of models and conditions, researchers have developed specialized computational pipelines. Toolboxes like Net2Brain, introduced by Domenic Bersch et al. in Net2Brain: a toolbox to compare artificial vision models with human brain responses, automate the process of extracting representations from hundreds of artificial networks and partitioning their shared versus unique contributions to biological brain data. When dealing with highly complex, multi-featured models, techniques like banded ridge regression—utilized by Giuseppe Marrazzo et al. in Neural encoding of biomechanically (im)possible human movements in occipitotemporal cortex—are used to efficiently partition variance across multiple competing feature spaces simultaneously.

Ultimately, whether applied to urban planning patterns (Hongchao Xu et al. in Quantifying the Relationship between 2D/3D Building Patterns and Land Surface Temperature: Study on the Metropolitan Shanghai) or clinical biomarkers (Derek B. Archer et al. in The relationship between white matter microstructure and self-perceived cognitive decline), variance partitioning provides a rigorous mathematical lens to move from mere correlation to precise feature attribution.

Key Foundational Work

Applying artificial vision models to human scene understanding (2015) — contributed 1 concept elements to this direction.
Fourier power, subjective distance, and object categories all provide plausible models of BOLD responses in scene-selective visual areas (2015) — contributed 1 concept elements to this direction.
Visual dictionaries as intermediate features in the human brain (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_variance_partitioning_ddf2bb44,
  title = {variance partitioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variance-partitioning/rdr_ddf2bb44b49942b18ce829da0e8fd129},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_variance_partitioning_ddf2bb44_v1,
  title = {variance partitioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variance-partitioning/rdr_ddf2bb44b49942b18ce829da0e8fd129/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake