The Representational Alignment Gap in Artificial Vision

64 concepts57 papers1 proposal

Deep neural networks (DNNs) have achieved human-level performance on visual tasks, yet they often arrive at their conclusions through mechanisms that diverge sharply from biological reality. This discrepancy, known as the representational alignment gap, describes the fundamental difference between how artificial systems and biological brains abstract information from the environment. While a model might correctly label an image, its internal "logic"—the features it prioritizes and the information it discards—remains stubbornly non-biological.

Measuring the Common Currency of Representations

To understand why a gap exists, researchers first needed a way to compare two vastly different systems: the silicon-based activations of a computer and the carbon-based firing patterns of a brain. Nikolaus Kriegeskorte introduced Representational Similarity Analysis (RSA) as a solution. By using Representational Dissimilarity Matrices (RDMs), Relating population-code representations between man, monkey, and computational models demonstrates that we can compare species and modalities by looking at the relative distances between object representations rather than the raw signals themselves.

Using this framework, the authors found that while human and monkey brains show a striking match in their categorical structure (e.g., distinguishing animate from inanimate objects), most computational models fail to capture this inherent biological hierarchy.

The Paradox of Superhuman Performance

A common assumption in AI research was that increasing task performance would naturally lead to more brain-like models. However, Linsley et al. revealed a startling reversal in Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex. As models become "superhuman" on benchmarks like ImageNet, they actually become less predictive of neural activity in the primate visual cortex.

This suggests that modern AI achieves high accuracy by exploiting statistical shortcuts—such as high-frequency textures—that the human brain ignores. Geirhos et al. quantified this by measuring "error consistency," showing that while CNNs are consistent with each other, they share almost no common error patterns with humans in Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency.

Idiosyncratic Invariances and Model Metamers

One of the most intuitive ways to "see" the alignment gap is through model metamers. These are synthetic stimuli that a model perceives as identical to a natural image but which appear as meaningless noise to a human. Feather et al. used these stimuli in Model metamers reveal divergent invariances between biological and artificial neural networks to show that models develop "idiosyncratic invariances."

Essentially, the models discard information that humans find essential for recognition, while remaining sensitive to pixel-level noise that humans cannot perceive. This representational divergence is further evidenced by the "decodability" problem: Horikawa et al. found that a significant portion of artificial "neurons" in DNNs cannot be decoded from biological brain signals, suggesting they are mathematical artifacts rather than biologically relevant features in Characterization of deep neural network features by decodability from human brain activity.

Structural and Functional Mismatches

The gap is not just in the software (the features) but also the hardware (the architecture). Biological vision is characterized by circular, sparse connectivity and complex inhibitory mechanisms. Gondur et al. argue in A tale of two tails: Preferred and anti-preferred natural stimuli in visual cortex that primate neurons are "two-tailed" encoders, actively suppressed by specific "anti-preferred" stimuli—a mechanism missing from the one-tailed activation functions of standard DNNs.

Furthermore, standard convolutional kernels are rigid squares, unlike the flexible receptive fields of the brain. Jea Kwon proposed $L_p$-convolution in Brain-inspired Lp-Convolution benefits large kernels and aligns better with visual cortex to bridge this gap. By applying trainable, biologically-plausible masks to kernels, the authors demonstrated that models could achieve better alignment with the visual cortex while maintaining performance.

Closing the Gap via Direct Alignment

If optimizing for task performance alone fails to produce biological alignment, researchers have turned to training models directly on neural data. Julie D. Golomb introduced ReAlnet in Achieving more human brain-like vision via human EEG representational alignment., which uses non-invasive EEG data to force models to adopt human-like representational strategies. Similarly, the Algonauts Project provides a massive benchmark for researchers to test how well their models predict fMRI responses to natural scenes, highlighting that current models still only capture a fraction of the predictable neural signal.

Ultimately, bridging the representational alignment gap is about more than just making AI "better." It is about ensuring that artificial intelligence processes the world using the same robust, conceptual, and efficient principles that have evolved in biological systems over millions of years.

Key Foundational Work

Relating population-code representations between man, monkey, and computational models (2009) — contributed 1 concept elements to this direction.
Neurobiology, Psychophysics, and Computational Models of Visual Attention (1993) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_representational_alignment_gap_fa27e2a7,
  title = {representational alignment gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representational-alignment-gap/rdr_fa27e2a79fcb4048b7cc74f2bff8f636},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_representational_alignment_gap_fa27e2a7_v1,
  title = {representational alignment gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representational-alignment-gap/rdr_fa27e2a79fcb4048b7cc74f2bff8f636/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake