Representation Biases: Variance Is Not Always a Good Proxy for Importance

Andrew K. Lampinen, Stephanie C. Y. Chan, Yuxuan Li, Katherine L. HermanneNeuro 2026 · DOI

Abstract

A central approach in neuroscience is to analyze neural representations as a means to understand a system's function, through the use of methods like principal component analysis, regression, and representational similarity analysis. These analyses often rest on a tacit "linking assumption": that the features explaining the most variance in neural activity are the most important for the system's computation. Here, we challenge this assumption. We review recent work in machine learning demonstrating "representation biases"—the fact that learned representations can be biased toward certain features over others. For example, learned representations heavily overrepresent simple (linear) features while representing complex (nonlinear) features much more weakly, even when both are equally critical for the system's computations. We review the origins of these biases in learning dynamics and patterns of computation. We then discuss their consequences for neuroscience. We show that if a subset of features dominates the representations, standard analytic techniques can yield highly biased inferences—for example, resulting in the mistaken conclusion that a system is simpler than it really is or that two systems are more similar than they really are. We discuss some connections between these findings and recent empirical developments in neuroscience. Finally, we present homomorphic encryption as a conceptual case study of the potential for a total dissociation between representational geometry and computation. We conclude that achieving a complete understanding of neural systems requires moving beyond high-variance signals, as critical computational mechanisms may be hidden in low-variance components.

The Problem: The Variance-Importance Linking Assumption

Computational neuroscience relies heavily on a "geometry-first" approach. Researchers use tools like Principal Component Analysis (PCA), Representational Similarity Analysis (RSA), and linear encoding models to map neural activity to function. These methods rest on a tacit assumption: if a feature is computationally important, it should be prominently represented in the "shape" or variance of the neural data.

Figure 1: The linking assumption that variance equals importance can lead to biased inferences when simple features dominate the representational space.

The authors argue that this assumption is a potential trap. If a system is biased toward representing simple, linear features with high variance while burying complex, nonlinear computations in low-variance "noise," our standard analytic toolkit will produce a skewed picture. We might conclude a system is simpler than it actually is, or mistakenly claim two systems are functionally identical just because their high-variance "geometry" looks similar.

Methodology and Experimental Design

The paper synthesizes recent work in machine learning to probe the relationship between representation and computation. The authors utilize Deep Multilayer Perceptrons (MLPs) trained on multitask objectives using synthetic binary datasets.

Figure 2: Experimental setup using synthetic datasets to compare how neural networks represent simple linear features versus complex nonlinear features.

The key is the "ground truth" setup: the models are forced to learn two types of features simultaneously:

  1. "Easy" features: Simple linear readouts (e.g., identifying the value of a single input bit).
  2. "Hard" features: Complex nonlinear functions (e.g., 4-parity, a difficult version of XOR).

Because the researchers define the tasks, they know both features are equally necessary for the model to succeed. They then apply standard neuroscience metrics (PCA, RSA, and $R^2$ variance explained) to the model’s internal layers to see if the "importance" of these features is reflected in their representational strength.

Key Insights: The Simplicity Bias

  • Variance Disparity: Learned representations heavily favor simple features. In the penultimate layer of the models, "easy" features explained over 50% of the variance, while "hard" features—despite being equally necessary for the output—explained only ~5%.
  • Learning Dynamics: Simple features are typically learned faster. This "head start" allows them to dominate the representational space early on, a bias that persists even after the model has mastered the harder tasks.
  • The "Dilution" of Nonlinearity: There is essentially only one way to represent a linear feature. However, there are many ways to represent a nonlinear feature like XOR. In high-dimensional space, these diverse solutions "dilute" the variance across many dimensions, making the signal appear weak and disorganized to tools like PCA.
  • The Streetlight Effect: The authors warn of a "streetlight effect" in neuroscience, where researchers focus on the "brightest" signals (high variance) and ignore the "shadows" (low variance) where the most sophisticated computations might actually be happening.

Consequences for Neuroscience

The study demonstrates that standard tools can be highly misleading:

  • PCA Blindness: Top principal components (PCs) showed clear clusters for easy tasks but appeared as unstructured noise for hard tasks.
  • RSA Misleadingness: When comparing a multitask model to a single-task model, the multitask model appeared almost identical to the "easy-only" model and "very different" from the "hard-only" model, despite successfully performing the hard task.
  • Homomorphic Encryption: The paper points to homomorphic encryption as the ultimate proof that computation can be totally dissociated from representational geometry. In such a system, activity looks like random noise, yet it performs perfect, complex logic.

Limitations and Future Directions

While artificial neural networks are common models for brain function, biological brains may have different structural constraints. Future work should prioritize:

  1. Metrics Beyond Variance: Developing metrics invariant to signal magnitude.
  2. Causal Interventions: Using perturbations to identify which low-variance signals drive behavior.
  3. Behavioral Characterization: A strong focus on completely characterizing behavior is essential to avoid missing hidden computational mechanisms.

Summary updated

Abstract

A central approach in neuroscience is to analyze neural representations as a means to understand a system's function, through the use of methods like principal component analysis, regression, and representational similarity analysis. These analyses often rest on a tacit "linking assumption": that the features explaining the most variance in neural activity are the most important for the system's computation. Here, we challenge this assumption. We review recent work in machine learning demonstrating "representation biases"-the fact that learned representations can be biased toward certain features over others. For example, learned representations heavily overrepresent simple (linear) features while representing complex (nonlinear) features much more weakly, even when both are equally critical for the system's computations. We review the origins of these biases in learning dynamics and patterns of computation. We then discuss their consequences for neuroscience. We show that if a subset of features dominates the representations, standard analytic techniques can yield highly biased inferences-for example, resulting in the mistaken conclusion that a system is simpler than it really is or that two systems are more similar than they really are. We discuss some connections between these findings and recent empirical developments in neuroscience. Finally, we present homomorphic encryption as a conceptual case study of the potential for a total dissociation between representational geometry and computation. We conclude that achieving a complete understanding of neural systems requires moving beyond high-variance signals, as critical computational mechanisms may be hidden in low-variance components.
BibTeX
@inproceedings{lampinen2026representation,
  author = {Andrew K. Lampinen and Stephanie C. Y. Chan and Yuxuan Li and Katherine L. Hermann},
  title = {{Representation Biases: Variance Is Not Always a Good Proxy for Importance}},
  booktitle = {eNeuro},
  year = {2026},
  doi = {10.1523/eneuro.0461-25.2026},
}
Figure