Decoding of human identity by computer vision and neuronal vision

Yipeng Zhang, Zahra M. Aghajan, Matias J. Ison, Qiujing Lu, Hanlin Tang, Güldamla Kalender, Tonmoy Monsoor, Jie Zheng, Gabriel Kreiman, Vwani Roychowdhury, Itzhak FriedScientific 2023 · DOI

Abstract

Extracting meaning from dynamic visual information is a core challenge for both biological and artificial intelligence. While computer vision (CV) has achieved high accuracy in identity recognition, the human brain's mechanism for this task—traditionally linked to sparse "concept cells"—remains partially understood. This study recorded neuronal activity from epilepsy patients watching the TV series "24" to compare CV performance with "neuronal vision." Results show that identity can be robustly decoded from small populations of neurons across multiple brain regions, including those beyond the medial temporal lobe. Furthermore, these neural representations are modulated by subjective memory and narrative associations, highlighting the integrated nature of biological identity decoding.

Introduction

The ability to recognize a specific individual across varying contexts, lighting, and expressions is a hallmark of advanced intelligence. In neuroscience, the "concept cell" hypothesis suggests that specific neurons fire selectively for unique identities. However, the sparsity of these cells makes them difficult to study in naturalistic settings. This research shifts the focus toward population-level coding, investigating whether the collective activity of small neuronal groups can decode identity as effectively as modern deep learning models.

Methodology

The study utilized data from nine neurosurgical patients with depth electrodes. Participants watched a 42-minute episode of the TV series "24" while researchers recorded activity from 385 neurons across various brain regions.

The study used naturalistic stimuli from the TV series '24' to evaluate identity decoding in a dynamic environment.

A dual-track analysis was performed:

  1. Computer Vision Pipeline: A semi-supervised system using YOLO-v3, SORT, FaceNet, and ResNet was developed to provide ground-truth labels for character appearances in every frame.
  2. Neuronal Decoding: Deep learning models (LSTM and CNN) were trained to decode character presence directly from neural firing rates.
  3. Knockout Analysis: Researchers systematically removed data from specific brain regions to determine their relative contribution to decoding accuracy.

Results

The CV algorithm demonstrated high reliability, achieving an overall accuracy of approximately 86% when compared to manual human labeling.

Confusion matrices illustrating the performance of the computer vision algorithm for the four main characters and overall detection.

The neuronal decoders significantly outperformed baseline models, with the LSTM architecture performing seven times better than a Naive Bayes distribution. Crucially, "neuronal vision" showed superior recall compared to CV when benchmarked against human-annotated data. The "knockout" analysis revealed a synergistic effect: the performance drop from removing an entire brain region was greater than the sum of individual electrode removals, suggesting that inter-neuronal interactions are critical for identity representation.

Discussion

The findings extend identity representation beyond the medial temporal lobe (MTL) to the occipital cortex, superior temporal gyrus, and anterior cingulate. The study also observed "memory reinstatement"—decoder confidence was significantly higher for clips that participants successfully remembered during a post-viewing task. Additionally, the models exhibited "narrative mapping," where characters frequently appearing together in the plot showed higher co-activation in the neural decoder, suggesting that social associations are baked into the visual representation of identity.

Conclusion

This research demonstrates that the human brain encodes identity through the coordinated activity of relatively small neuronal populations. By bridging computer vision and electrophysiology, the study provides a framework for understanding how the brain transforms dynamic sensory input into stable conceptual representations, influenced by memory and narrative context.

Summary updated

Abstract

Extracting meaning from a dynamic and variable flow of incoming information is a major goal of both natural and artificial intelligence. Computer vision (CV) guided by deep learning (DL) has made significant strides in recognizing a specific identity despite highly variable attributes. This is the same challenge faced by the nervous system and partially addressed by the concept cells-neurons exhibiting selective firing in response to specific persons/places, described in the human medial temporal lobe (MTL) ⁠. Yet, access to neurons representing a particular concept is limited due to these neurons' sparse coding. It is conceivable, however, that the information required for such decoding is present in relatively small neuronal populations. To evaluate how well neuronal populations encode identity information in natural settings, we recorded neuronal activity from multiple brain regions of nine neurosurgical epilepsy patients implanted with depth electrodes, while the subjects watched an episode of the TV series "24". First, we devised a minimally supervised CV algorithm (with comparable performance against manually-labeled data) to detect the most prevalent characters (above 1% overall appearance) in each frame. Next, we implemented DL models that used the time-varying population neural data as inputs and decoded the visual presence of the four main characters throughout the episode. This methodology allowed us to compare "computer vision" with "neuronal vision"-footprints associated with each character present in the activity of a subset of neurons-and identify the brain regions that contributed to this decoding process. We then tested the DL models during a recognition memory task following movie viewing where subjects were asked to recognize clip segments from the presented episode. DL model activations were not only modulated by the presence of the corresponding characters but also by participants' subjective memory of whether they had seen the clip segment, and by the associative strengths of the characters in the narrative plot. The described approach can offer novel ways to probe the representation of concepts in time-evolving dynamic behavioral tasks. Further, the results suggest that the information required to robustly decode concepts is present in the population activity of only tens of neurons even in brain regions beyond MTL.
BibTeX
@inproceedings{zhang2023decoding,
  author = {Yipeng Zhang and Zahra M. Aghajan and Matias J. Ison and Qiujing Lu and Hanlin Tang and Güldamla Kalender and Tonmoy Monsoor and Jie Zheng and Gabriel Kreiman and Vwani Roychowdhury and Itzhak Fried},
  title = {{Decoding of human identity by computer vision and neuronal vision}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-022-26946-w},
}
Figure