Encoding of facial features by single neurons in the human amygdala and hippocampus

Runnan Cao, Xin Li, Nicholas Brandmeir, Shuo Wang

Abstract

The human amygdala and hippocampus are known to play critical roles in face processing, yet the cellular mechanisms by which they encode specific facial features and guide eye movements remain poorly understood. This study identifies a specialized population of neurons in the human medial temporal lobe (MTL) that selectively differentiates between fixations on the eyes and the mouth. We demonstrate that these feature-selective neurons are not dependent on fixation order and do not represent distinct morphological neuronal types. Furthermore, we identified neurons that encode saccades toward specific facial features, with population decoding revealing distinct temporal dynamics between the amygdala and hippocampus. Finally, we show that these neurons encode regions of high saliency for face recognition—as validated by deep neural networks—and correlate with subjective social trait judgments. These findings suggest that feature-selective neurons in the MTL are fundamental to social perception and the visual exploration of faces.

Introduction

Navigating social environments requires the rapid and accurate processing of facial information. While previous research has established that the amygdala and hippocampus (MTL) respond to whole faces or specific emotional expressions, it has been unclear how individual neurons handle the visual scanning process. Specifically, how does the brain encode the saliency of facial features to guide saccades (rapid eye movements) and fixations? This study sought to characterize the cellular encoding of facial features, specifically the eyes and mouth, and determine how this activity relates to both identification and social judgment.

Methods

To investigate these mechanisms, researchers recorded single-neuron activity from five neurosurgical patients undergoing epilepsy monitoring. Micro-depth electrodes were used to record from 422 single units across the amygdala and hippocampus.

Behavioral Task and Eye Tracking

Patients performed a "one-back" task while viewing 500 natural images of celebrities from the CelebA dataset. The images were balanced for gender and race. During the task, patients' eye movements were tracked at 500 Hz using an infrared system to precisely correlate neuronal firing with specific points of fixation.

Computational Analysis

The team developed two primary metrics: the Fixation Selectivity Index (FSI) and the Saccade Selectivity Index (SSI) to quantify changes in firing rates based on the facial feature being targeted. To understand the broader implications of this encoding, the researchers utilized a VGG-16 Deep Neural Network trained on face recognition to generate saliency maps, identifying which pixels were most critical for artificial identification. Finally, patients provided subjective ratings for eight social traits (e.g., warmth, competence) for each face, allowing for a correlation between neuronal activity and social perception.

Key Results

Feature Selectivity and Anatomical Distribution

Of the 365 neurons analyzed for selectivity, 74 (approximately 20%) were identified as feature-selective. A significant majority of these neurons (74%) showed a preference for the eyes over the mouth. The study revealed a distinct anatomical division of labor: the amygdala was significantly more "eye-centric" than the hippocampus, containing a higher percentage of feature-selective neurons (26.4% in the amygdala vs. 14.4% in the hippocampus).

Predictive Saccade Encoding

The researchers identified 29 neurons (8%) that differentiated saccades toward the eyes versus the mouth. Notably, some of these neurons began firing before the eye movement was initiated. This suggests that these cells are not merely reactive to visual stimuli but are involved in the planning and execution of visual exploration. Population decoding confirmed that the amygdala encodes these saccade targets earlier (pre-onset) than the hippocampus (post-onset), suggesting a hierarchical flow of information.

Alignment with Artificial Intelligence

One of the study's most striking findings was the alignment between biological neuronal activity and deep learning models. Neurons fired most robustly when patients fixated on regions that the VGG-16 model identified as highly salient for face recognition. This suggests that the human MTL utilizes a feature-encoding strategy optimized for identification, similar to high-performing computer vision architectures.

Social Trait Perception

Beyond identification, these neurons appear to encode social meaning. Firing rates during eye fixations were significantly correlated with subjective social judgments. For example, higher firing rates during eye fixations were associated with lower perceived "warmth" and "practicality" (Pearson correlation $r = -0.03, P = 0.001$). This indicates that the MTL's response to facial features provides a neural basis for implicit social evaluation.

Discussion

This study identifies a new class of neurons in the human MTL that selectively encode facial features and guide visual attention. The finding that the amygdala leads the hippocampus in saccade encoding suggests it plays a primary role in directing attention to salient social cues.

While the study is limited by the small sample size inherent in invasive human electrophysiology, the results provide a cellular framework for understanding social processing. These "feature cells" may be critical for understanding conditions like autism spectrum disorder, where atypical eye-fixation patterns are a hallmark. Future work will explore whether these neurons respond to abstract or cartoon representations of faces, further clarifying the boundaries of social feature encoding.

Summary updated

Abstract

Abstract The human amygdala and hippocampus play a key role in face processing. However, it has been unknown how the neurons in the human amygdala and hippocampus encode facial feature information and directs eye movements to salient facial features such as the eyes and mouth. In this study, we identified a population of neurons that differentiated fixations on the eyes vs. mouth. The response of these feature-selective neurons was not dependent on fixation order, and eye-preferring and mouth-preferring neurons were not of different neuronal types. We found another population of neurons that differentiated saccades to the eyes vs. mouth. Population decoding confirmed our results and further revealed the temporal dynamics of face feature coding. Interestingly, we found that the amygdala and hippocampus played a different role in encoding face features. Lastly, we revealed two functional roles of feature-selective neurons that they encoded the salient region for face recognition and they encoded perceived social trait judgment. Together, we revealed and characterized a new class of neurons that encoded facial features. These neurons may play an important role in social perception and recognition of faces.
BibTeX
@inproceedings{cao2021encoding,
  author = {Runnan Cao and Xin Li and Nicholas Brandmeir and Shuo Wang},
  title = {{Encoding of facial features by single neurons in the human amygdala and hippocampus}},
  booktitle = {Communications},
  year = {2021},
  doi = {10.1038/s42003-021-02917-1},
}
Figure