Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons

Irina Higgins, Le Chang, Victoria Langston, Demis Hassabis, Christopher Summerfield, Doris Y. Tsao, Matthew Botvinick

Abstract

Deep supervised neural networks trained to classify objects have emerged as popular models of computation in the primate ventral stream. These models represent information with a high-dimensional distributed population code, implying that inferotemporal (IT) responses are also too complex to interpret at the single-neuron level. We challenge this view by modelling neural responses to faces in the macaque IT with a deep unsupervised generative model, beta-VAE. Unlike deep classifiers, beta-VAE "disentangles" sensory data into interpretable latent factors, such as gender or hair length. We found a remarkable correspondence between the generative factors discovered by the model and those coded by single IT neurons. Moreover, we were able to reconstruct face images using the signals from just a handful of cells. This suggests that the ventral visual stream may be optimising the disentangling objective, producing a neural code that is low-dimensional and semantically interpretable at the single-unit level.

Introduction

For decades, systems neuroscience has been divided between two competing paradigms regarding the "basic unit" of neural representation. The classic neuron doctrine posits that individual cells are the fundamental building blocks of perception, each tuned to specific features. Conversely, the population doctrine suggests that meaningful information is multiplexed across vast ensembles of neurons, rendering individual cells largely uninterpretable in isolation.

The recent dominance of deep supervised neural networks (such as VGG-Face) in modeling the primate ventral stream has bolstered the population doctrine. These models, while highly accurate at classification, generate high-dimensional, "tangled" representations where features are spread across many units. However, biological systems rarely learn from millions of labeled examples. Instead, they learn through unsupervised interaction with their environment. This study investigates whether an unsupervised model designed to find the underlying structure of data—a process known as "disentanglement"—provides a more accurate description of biological neural coding.

The Disentanglement Hypothesis and $\beta$-VAE

The researchers utilized the $\beta$-Variational Autoencoder ($\beta$-VAE) as their primary computational model. Unlike standard autoencoders that focus solely on data compression and reconstruction, the $\beta$-VAE introduces a constraint (the $\beta$ parameter) that encourages the model to discover independent, semantically meaningful factors of variation. These factors might include attributes like head rotation, gender, or the presence of a smile.

To test the biological validity of this model, it was compared against several benchmarks:

  • Supervised Models: VGG-Face (a state-of-the-art classifier).
  • Standard Generative Models: Basic Autoencoders (AE) and standard VAEs.
  • Linear Baselines: Principal Component Analysis (PCA) and Independent Component Analysis (ICA).
  • Hand-crafted Models: The Active Appearance Model (AAM), previously the gold standard for modeling face-processing neurons.

The core metric for comparison was "alignment," which measures the degree to which the variance of a single biological neuron can be explained by a single latent unit in the model.

Experimental Framework

The study integrated electrophysiological recordings with computational modeling using the following data:

  • Neural Data: Recordings from 159 neurons in the macaque face patch AM, a region at the apex of the ventral visual stream.
  • Image Data: A set of 2,100 natural grayscale face images presented to both the monkeys and the models.
  • Generalization Test: 62 "held-out" face images were used to evaluate the models' ability to reconstruct novel stimuli based solely on neural activity.

Key Results

1. Ceiling-Level Alignment

The $\beta$-VAE demonstrated a striking one-to-one correspondence with biological neurons. The alignment between the model's latent units and single IT neurons reached the theoretical ceiling, meaning the model explained the neural data as well as the neural data explained itself. This significantly outperformed the hand-crafted AAM and all supervised deep learning baselines ($p < 0.01$).

2. Single-Neuron Interpretability

Contrary to the view that single neurons are uninterpretable, the study found that many IT neurons are tuned to specific, semantically meaningful axes discovered by the $\beta$-VAE. Neurons were found to respond specifically to features like "hair thickness" or "smile intensity," matching the disentangled factors identified by the unsupervised model.

3. Efficiency and Reconstruction

Because the neural code is disentangled and low-dimensional, it is remarkably efficient. The researchers successfully reconstructed high-quality face images from neural activity using as few as 12 neurons. When decoding novel faces, the $\beta$-VAE produced more identifiable reconstructions than standard autoencoders, even though the latter are explicitly optimized for reconstruction rather than disentanglement.

Discussion and Implications

These findings suggest that the ventral visual stream may be optimizing for a disentangling objective rather than simple classification. By building a generative model of the world where independent physical properties are separated, the brain creates a representation that is both semantically interpretable and computationally efficient.

While this study focuses on the specialized domain of face processing, it challenges the necessity of the population doctrine for understanding high-level vision. If the brain's goal is to disentangle the underlying factors of the sensory world, then the "neuron doctrine" may be more relevant than previously thought. Future research will be required to determine if these principles of semantic disentanglement generalize to other object categories and sensory modalities.

Summary updated

Abstract

Deep supervised neural networks trained to classify objects have emerged as popular models of computation in the primate ventral stream. These models represent information with a high-dimensional distributed population code, implying that inferotemporal (IT) responses are also too complex to interpret at the single-neuron level. We challenge this view by modelling neural responses to faces in the macaque IT with a deep unsupervised generative model, beta-VAE. Unlike deep classifiers, beta-VAE "disentangles" sensory data into interpretable latent factors, such as gender or hair length. We found a remarkable correspondence between the generative factors discovered by the model and those coded by single IT neurons. Moreover, we were able to reconstruct face images using the signals from just a handful of cells. This suggests that the ventral visual stream may be optimising the disentangling objective, producing a neural code that is low-dimensional and semantically interpretable at the single-unit level.
BibTeX
@inproceedings{higgins2021unsupervised,
  author = {Irina Higgins and Le Chang and Victoria Langston and Demis Hassabis and Christopher Summerfield and Doris Y. Tsao and Matthew Botvinick},
  title = {{Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons}},
  booktitle = {Nature Communications},
  year = {2021},
  doi = {10.1038/s41467-021-26751-5},
}
Figure