Coupling Visual Semantics of Artificial Neural Networks and Human Brain Function via Synchronized Activations

Lin Zhao, Haixing Dai, Zihao Wu, Zhenxiang Xiao, Lu Zhang, David Weizhong Liu, Xintao Hu, Xi Jiang, Sheng Li, Dajiang Zhu, Tianming Liu

TL;DR

Researchers have developed Sync-ACT, a novel framework that aligns the internal "thoughts" of AI models with actual human brain activity. By synchronizing the activations of artificial neurons with functional brain networks (FBNs) recorded during movie-watching, the team can now "cross-annotate" AI filters with biological descriptions and has discovered that AI models which more closely mimic human brain patterns tend to perform better at real-world tasks.

What Problem Does This Paper Address?

While Artificial Neural Networks (ANNs) were originally inspired by the human brain, the two fields have largely evolved in isolation. We know that Convolutional Neural Networks (CNNs) are great at recognizing images, but we struggle to explain why a specific artificial neuron fires or if its logic bears any resemblance to human biology.

The challenge is twofold:

  1. Domain Gap: Brain activity (fMRI) and AI activations are mathematically very different.
  2. Lack of Context: Standard fMRI studies often use static images, which don't capture the dynamic, "naturalistic" way humans actually perceive the world.

The authors seek to bridge this gap by using movies as a common stimulus to see if AI and humans "see" the same way.

Methods

The authors introduce the Synchronized Activations (Sync-ACT) framework. The process works like a high-tech matching game:

  • The Stimulus: Both the AI and human subjects "watch" the same movie. Because video frames are intrinsically ordered in time, time becomes the common axis to link the two domains.
  • The AI Side: Movie frames are fed into pre-trained CNNs (like ResNet or DenseNet). The researchers record the maximum activation of each convolutional filter over time, creating a "temporal activation series" for every artificial neuron.
  • The Brain Side: Using naturalistic fMRI (nfMRI) data, the team uses a deep learning Encoder-Decoder (specifically using Multi-Head Self-Attention) to compress messy 4D brain data into 64 distinct Functional Brain Networks (FBNs).
  • The Coupling: By calculating the Pearson Correlation Coefficient (PCC) between the AI's activation series and the brain's FBN series, they identify which artificial filters "sync up" with which biological networks.
  • Cross-Annotation: Once paired, they use existing neuroscientific meta-analyses to label the AI filters with human-readable concepts like "face recognition," "navigation," or "color perception."

Datasets

The study utilizes two major publicly available datasets:

  • HCP 7T (Human Connectome Project): A high-quality dataset featuring 184 subjects watching short independent films and Hollywood movie excerpts.
  • StudyForrest: A validation dataset featuring 15 subjects watching the full 2-hour movie Forrest Gump.

Key Insights and Ideas

  • Biological Grounding for AI: This is the first time researchers have been able to semantically annotate AI neurons with biologically meaningful descriptions derived directly from human brain imaging.
  • The "Brain-Performance" Link: Perhaps the most striking finding is a strong linear relationship ($R^2 \approx 0.75$) between an AI's similarity to the human brain and its accuracy on ImageNet classification. Essentially, biology has already "solved" vision; the closer an AI gets to that biological solution, the better it performs.
  • Hierarchical Alignment: The correlation with brain activity is not uniform. Higher-level layers in CNNs (which handle complex concepts) show significantly stronger synchronization with the brain than lower-level layers (which handle simple edges and textures). This mirrors the hierarchical structure of the human visual cortex.

Key Results

  • Significant Correlation: Most CNN models showed statistically significant correlations (PCC > 0.2) with human brain networks across different movie runs.
  • Model Comparison: Advanced models like ResNet-50 and DenseNet-161 exhibited higher brain-similarity scores than older architectures like AlexNet or VGG-16.
  • Predictive Accuracy: The researchers demonstrated that you could potentially predict how well a new AI architecture will perform on a vision task simply by measuring its "brain-sync" score.
  • Qualitative Success: The paper provides visual evidence (e.g., Figure 4) where an FBN associated with "navigation and place" correctly pairs with a CNN filter that activates for "natural scenes" and "rocks."

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Activation Metrics: Currently, they only use the "maximum value" in a feature map to represent an AI neuron's activation. This might be too simplistic to capture the full nuance of neural firing.
  • Label Granularity: The semantic labels from fMRI meta-analysis can be "coarse-grained" (e.g., "Bank Vault"). Future work could involve finer-grained natural language descriptions.
  • Broadening the Scope: The study focused primarily on CNNs for image classification. The authors suggest that Vision Transformers (ViTs) and models for other tasks (like audio or NLP) should be investigated using the same Sync-ACT paradigm.
  • NAS Integration: They propose using brain-similarity as a reward metric for Neural Architecture Search (NAS) to design "brain-inspired" AI from the ground up.

Summary updated

Abstract

Artificial neural networks (ANNs), originally inspired by biological neural networks (BNNs), have achieved remarkable successes in many tasks, such as visual representation learning. However, whether there exists semantic correlations/connections between the visual representations in ANNs and those in BNNs remains largely unexplored due to both the lack of an effective tool to link and couple two different domains, and the lack of a general and effective framework for representing the visual semantics in BNNs such as human functional brain networks (FBNs). To answer this question, we propose a novel computational framework, synchronized activations (Sync-ACTs), to couple the visual representation spaces and semantics between ANNs and BNNs in human brain based on naturalistic functional magnetic resonance imaging (nfMRI) data. With this approach, we are able to semantically annotate the neurons in ANNs with biologically meaningful descriptions derived from human brain imaging for the first time. We evaluated the Sync-ACT framework on two publicly available movie-watching nfMRI data sets. The experiments demonstrate 1) the significant correlation and similarity of the semantics between the visual representations in FBNs and those in a variety of convolutional neural networks (CNNs) models and 2) the close relationship between CNN’s visual representation similarity to BNNs and its performance in image classification tasks. Overall, our study introduces a general and effective paradigm to couple the ANNs and BNNs and provides novel insights for future studies such as brain-inspired artificial intelligence.
BibTeX
@article{zhao2024coupling,
  author = {Lin Zhao and Haixing Dai and Zihao Wu and Zhenxiang Xiao and Lu Zhang and David Weizhong Liu and Xintao Hu and Xi Jiang and Sheng Li and Dajiang Zhu and Tianming Liu},
  title = {{Coupling Visual Semantics of Artificial Neural Networks and Human Brain Function via Synchronized Activations}},
  journal = {IEEE Trans. Cogn. Dev. Syst.},
  year = {2024},
  doi = {10.1109/tcds.2023.3287184},
}
Figure