When a modern AI model predicts human brain activity, it is tempting to conclude that the model and the brain "work the same way." However, this conclusion is often premature. Because visual and linguistic features are highly correlated in the natural world—for instance, the visual texture of fur often co-occurs with the semantic category "animal"—a model might achieve a high "brain score" simply by capturing low-level statistical artifacts rather than high-level cognitive principles.
To solve this, researchers have converged on a statistical framework known as variance partitioning (or variance decomposition). This method allows us to move beyond aggregate similarity and isolate the unique explanatory power of specific features, effectively "unravelling" the complex representations within both biological and artificial systems.
The Problem of Redundant Information
In computational neuroscience, we often compare a target (like fMRI activity) against multiple predictor models. If Model A (edges) and Model B (objects) both correlate with the same brain region, we cannot easily tell if that region cares about objects or just the edges that compose them.
As Ebrahim Feghhi and Nima Hadidi argue in What Are Large Language Models Mapping to in the Brain?, high alignment scores can be a "statistical illusion" driven by simple metadata like sentence length or temporal autocorrelation. Without isolating unique variance, we risk over-attributing human-like intelligence to models that are merely tracking basic signal properties.
How Variance Partitioning Works
Intuition for this method is best found in a Venn diagram. Imagine the total variance of brain activity as a circle. Model A explains one portion, and Model B explains another. These portions often overlap—this is shared variance. Variance partitioning mathematically peels these apart to find the unique variance: the information explained by Model A that cannot be explained by Model B, and vice versa.
Researchers like Christopher Baldassano and Li Fei-Fei used this to show that human scene categorization is driven by affordances—what a space allows us to do—rather than just the objects present. In Affordances Provide a Fundamental Categorization Principle for Visual Scenes, they demonstrated that even when you control for visual features, the "action-possibility" of a room uniquely explains how our brains group different environments.
Disentangling Task and Structure
This approach has been instrumental in mapping the functional specialization of the visual cortex. Kshitij Dwivedi and Gemma Roig have repeatedly applied variance partitioning to show that different brain regions are tuned to different computational "jobs."
In Unravelling Representations in Scene-selective Brain Regions Using Scene Parsing Deep Neural Networks, they isolated the unique contributions of "parsing" (identifying floors and walls) versus "classification" (naming the room). They found that the Occipital Place Area (OPA) is uniquely sensitive to floors—a finding that suggests its primary role is supporting navigation rather than simple recognition.
A similar logic applies to speech. In The Hierarchical Cortical Organization of Human Speech Processing, Wendy A. de Heer and Jack L. Gallant used variance partitioning to separate acoustic features from semantic ones. This allowed them to prove that the brain processes these features in a parallel, hierarchical gradient across both hemispheres, rather than in the strictly lateralized manner previously assumed.
Mathematical Foundations and Tooling
While the concept is intuitive, the math must be rigorous to avoid "bleeding" between categories. Hai Shu has developed frameworks like D-GCCA and D-CDLF to ensure that shared and unique factors are truly orthogonal. These methods allow researchers to quantify variance at the level of individual variables, linking global brain patterns to specific biological mechanisms.
To make these techniques accessible, toolboxes like Net2Brain, developed by Domenic Bersch and colleagues, and the RSA Toolbox by Nikolaus Kriegeskorte, provide standardized pipelines for performing these decompositions across hundreds of AI models.
Beyond the Brain: A General Tool for Complexity
The power of variance partitioning extends beyond neuroscience into any field dealing with coupled, complex systems:
- Ecology: Jianming Deng used it to determine that while temperature and nutrients both affect algae blooms, nutrients provide the primary "unique" drive, while temperature acts as a shared "multiplier."
- Climate Science: Carl‐Friedrich Schleussner applied commonality analysis to isolate the specific influence of Atlantic currents on global temperature swings.
- Individual Differences: Matteo Visconti di Oleggio Castello used distributional variance decomposition to show that while we share a "semantic map," about 22% of our conceptual representation is entirely unique to our personal experience.
By forcing models to compete for "unique" explanatory credit, variance partitioning ensures that our scientific theories are not just capturing correlations, but are identifying the specific, independent dimensions that define how a system—be it a brain, an ecosystem, or an AI—actually functions.