Abstract
In order to better understand how the brain perceives faces, it is important to know what objective drives learning in the ventral visual stream. To answer this question, we model neural responses to faces in the macaque inferotemporal (IT) cortex with a deep self-supervised generative model, $\beta$-VAE, which disentangles sensory data into interpretable latent factors, such as gender or age. Our results demonstrate a strong correspondence between the generative factors discovered by $\beta$-VAE and those coded by single IT neurons, beyond that found for the baselines, including the handcrafted state-of-the-art model of face perception, the Active Appearance Model, and deep classifiers. Moreover, $\beta$-VAE is able to reconstruct novel face images using signals from just a handful of cells. Together our results imply that optimising the disentangling objective leads to representations that closely resemble those in the IT at the single unit level. This points at disentangling as a plausible learning objective for the visual brain.
Introduction: The Search for the Brain's Objective Function
For decades, neuroscientists have sought the "objective function" of the visual brain: the fundamental goal driving the ventral visual stream to learn its internal representations. While deep supervised classifiers (like those used in facial recognition) can predict the population activity of neurons, they often fail at the single-neuron level. In the brain's "face patches," individual neurons seem to code for specific, interpretable axes of variation—for instance, one neuron might respond specifically to hair length, while another tracks face width.
Traditional handcrafted models like the Active Appearance Model (AAM) capture this low-dimensional coding well but lack a biologically plausible explanation for how these features are learned from raw, unlabeled sensory input. The challenge lies in finding a model that is both self-supervised and matches the specific, sparse coding found in single biological neurons.
Methods: Modeling with $\beta$-VAE
The researchers compared neural recordings from the macaque inferotemporal (IT) cortex to several computational models, most notably the $\beta$-Variational Autoencoder ($\beta$-VAE).
Unlike a standard VAE, the $\beta$-VAE introduces a hyperparameter ($\beta$) that penalizes the model if its "latent units" (the internal variables it uses to describe a face) are not statistically independent. This pressure forces the model to "disentangle" features—separating "smile" from "gender" or "skin tone," for example. The study used a metric called "alignment" to quantify the one-to-one mapping between a single model unit and a single biological neuron, comparing $\beta$-VAE against baselines including supervised deep classifiers (VGG-Face), PCA, ICA, and standard Autoencoders.
Results: Single-Neuron Alignment and Semantic Mapping
The most striking result was the superior alignment of $\beta$-VAE with biological neurons. The model achieved significantly higher alignment scores than all other baselines, including supervised classifiers and the handcrafted AAM.
The researchers found that the better a model is at disentangling (measured by the Unsupervised Disentanglement Ranking), the better it aligns with biological neurons ($r = 0.96$). Furthermore, human raters were able to consistently label the $\beta$-VAE latent units (e.g., "hairstyle," "age," "smile") with 32.1% agreement, significantly above chance. These interpretable units mapped directly onto the tuning of real neurons. This high-fidelity mapping allowed the authors to reconstruct high-quality face images from the activity of just 12 neurons by decoding their signals through the $\beta$-VAE latents.
Discussion: Disentangling as a Biological Principle
These findings suggest that the brain’s learning objective may be to discover the independent "latent factors" that explain the visual world, rather than just learning to classify objects. The $\beta$-VAE learns these brain-like representations without any external labels, making it a much more plausible model for how visual expertise develops in the brain.
However, limitations remain. The $\beta$-VAE only discovered a subset of the dimensions the brain uses, likely due to the limited sample of 2,100 images. Future work is needed to determine if this "disentangling" objective extends beyond face processing to general object recognition or abstract concepts, and to incorporate temporal dynamics and feedback loops present in biological systems.
Summary updated