Coupling Visual Semantics of Artificial Neural Networks and Human Brain Function via Synchronized Activations

Lin Zhao, Haixing Dai, Zihao Wu, Zhenxiang Xiao, Lu Zhang, David Weizhong Liu, Xintao Hu, Xi Jiang, Sheng Li, Dajiang Zhu, Tianming Liu

Introduction

Artificial Neural Networks (ANNs) were conceptually inspired by biological neural systems, yet the development of these two fields has largely diverged. While Convolutional Neural Networks (CNNs) excel at visual recognition, the internal logic governing their filters often remains opaque. Conversely, functional Magnetic Resonance Imaging (fMRI) provides a window into human brain activity but often lacks the semantic context needed to map specific neural responses to complex visual features.

To bridge this gap, researchers have developed Synchronized Activations (Sync-ACT), a framework that aligns the internal representations of AI models with human brain activity recorded during naturalistic movie-watching. By using time as a common axis, Sync-ACT enables the "cross-annotation" of artificial neurons with biological descriptions and reveals a profound link between an AI's biological plausibility and its task performance.

Methodology: The Sync-ACT Framework

The core of Sync-ACT is the synchronization of two distinct data streams—artificial and biological—using a shared naturalistic stimulus.

Overview of the Sync-ACT framework, illustrating the parallel processing of movie frames through a CNN and fMRI volumes through an encoder to produce synchronized temporal activations.

1. Temporal Activation Extraction

Both the AI model and human subjects are presented with the same movie sequences. For the AI, movie frames are processed by pre-trained CNNs (e.g., ResNet, DenseNet). The maximum activation of each convolutional filter is recorded at each time step, generating a "temporal activation series" for every artificial neuron.

2. Functional Brain Network (FBN) Encoding

On the biological side, 4D fMRI data is processed using a deep learning Encoder-Decoder architecture equipped with Multi-Head Self-Attention. This model compresses complex brain signals into 64 distinct Functional Brain Networks (FBNs), each with its own temporal activation series.

3. Coupling and Cross-Annotation

The framework calculates the Pearson Correlation Coefficient (PCC) between the AI's filter activations and the brain's FBN activations. High PCC values indicate synchronization. Once paired, AI filters are annotated with human-readable semantic labels (e.g., "face recognition," "navigation") derived from established neuroscientific meta-analyses of the corresponding FBNs.

Key Results and Discussion

The application of Sync-ACT across multiple datasets, including the HCP 7T and StudyForrest projects, yielded several significant insights into the relationship between artificial and biological vision.

A pivotal discovery of this research is the strong linear relationship between an AI model's similarity to the human brain and its objective performance. As shown in the figure below, there is a high correlation ($R^2 \approx 0.75$) between the average PCC (brain-sync score) and Top-1 accuracy on ImageNet classification.

Scatter plots across four experimental runs demonstrating the strong linear relationship between a model's brain similarity (PCC) and its classification accuracy (Top-1 Acc).

This suggests that architectures which more closely approximate biological visual processing strategies tend to be more effective at real-world recognition tasks. Advanced models like DenseNet-161 and ResNet-50 consistently showed higher brain-similarity scores compared to older architectures like VGG-16.

Hierarchical Alignment

The study also confirmed a hierarchical alignment: higher-level CNN layers, which process complex semantic concepts, show significantly stronger synchronization with the human brain than lower-level layers responsible for basic edges and textures. This mirrors the known hierarchical organization of the human visual cortex.

Conclusion and Future Directions

Sync-ACT provides a novel methodology for grounding AI interpretability in biological reality. By demonstrating that "brain-like" models perform better, this work opens the door for using brain-similarity as a metric in Neural Architecture Search (NAS) to design the next generation of bio-inspired AI. Future work will look to expand this paradigm to Vision Transformers (ViTs) and incorporate finer-grained natural language descriptions for even more precise semantic mapping.

Summary updated

BibTeX
@article{zhao2024coupling,
  author = {Lin Zhao and Haixing Dai and Zihao Wu and Zhenxiang Xiao and Lu Zhang and David Weizhong Liu and Xintao Hu and Xi Jiang and Sheng Li and Dajiang Zhu and Tianming Liu},
  title = {{Coupling Visual Semantics of Artificial Neural Networks and Human Brain Function via Synchronized Activations}},
  journal = {IEEE Trans. Cogn. Dev. Syst.},
  year = {2024},
  doi = {10.1109/tcds.2023.3287184},
}
Figure