Abstract: Transfer learning improves the performance of the target task by leveraging the data of a specific source task: the closer the relationship between the source and the target tasks, the greater the performance improvement by transfer learning. In neuroscience, the relationship between cognitive tasks is usually represented by similarity of activated brain regions or neural representation. However, no study has linked transfer learning and neuroscience to reveal the relationship between cognitive tasks. In this study, we propose a transfer learning framework to reflect the relationship between cognitive tasks, and compare the task relations reflected by transfer learning and by the overlaps of brain regions (e.g., neurosynth). Our results of transfer learning create cognitive taskonomy to reflect the relationship between cognitive tasks which is well in line with the task relations derived from neurosynth. Transfer learning performs better in task decoding with fMRI data if the source and target cognitive tasks activate similar brain regions. Our study uncovers the relationship of multiple cognitive tasks and provides guidance for source task selection in transfer learning for neural decoding based on small-sample data.
Introduction
In the world of machine learning, we know that a model trained to recognize cats will likely learn features useful for recognizing dogs. This is the essence of transfer learning: leveraging knowledge from a "source" task to jumpstart learning on a "target" task. But does this logic hold up when the "data" is the human brain?
Decoding brain states from functional Magnetic Resonance Imaging (fMRI) is a cornerstone of modern neuroscience, but it faces a major hurdle: the small-sample problem. Collecting high-quality fMRI data is expensive and time-consuming, often leaving researchers with insufficient data to train deep learning models from scratch. This study explores whether the relationships defined by AI performance match the biological relationships defined by brain activity, creating a "cognitive taskonomy" that bridges artificial and biological intelligence.
Methods
The researchers developed a computational framework to calculate a task affinity matrix through a three-step process:
- Data Processing: Brains were parcellated into 90 regions using the Automated Anatomical Labeling (AAL) atlas, converting complex 3D images into manageable feature vectors.
- Transfer Learning: Seven independent neural networks were trained. A pre-trained encoder from a source task was then used with a new decoder for a target task to measure transferability across varying amounts of target data.
- Cognitive Taskonomy: The resulting AI-derived affinities were compared to biological data from Neurosynth, a massive meta-analysis database that maps brain region activations across thousands of studies.
Datasets
The study utilized the Human Connectome Project (HCP) S1200 Release, a gold-standard dataset featuring over 1,000 subjects. The data covered 23 subtasks categorized into 7 domains: Working Memory, Motor, Relational Processing, Language, Emotion, Social Cognition, and Gambling. Preprocessing involved turning 3D fMRI scans into BOLD signal time series across defined regions of interest (ROIs).
Key Insights and Ideas
The central finding of the study is the "Biological Mirror": the AI's affinity matrix (how well tasks transfer) looks remarkably like the correlation matrix from Neurosynth (how much tasks overlap in brain activation).
- The "Social-Emotion-Gambling" Cluster: These tasks showed high positive transferability, suggesting they share a common high-order neural representation that the AI can capture and repurpose.
- Working Memory as an Island: Interestingly, Working Memory was largely non-transferable. The authors suggest this is because working memory is processed in a highly distributed way, making it a unique challenge for simple transfer methods.
- Efficiency at Scale: With the right source task, the framework achieved over 90% decoding accuracy using only 1% of the target domain data, demonstrating a powerful solution to the small-sample problem.
Key Results
The study demonstrated a strong Pearson correlation between the transfer learning affinity matrix and Neurosynth brain region overlap maps, particularly for higher-order cognitive tasks. Transfer learning significantly outperformed training from scratch on small datasets, while the framework successfully identified and quantified negative transfer between unrelated tasks like Emotion and Motor.
Limitations and Future Work
While the study used relatively simple fully connected layers, future work could employ more complex architectures like Convolutional Neural Networks (CNNs) or Transformers to capture spatial and temporal brain dynamics. Additionally, moving beyond discrete classification to predict continuous behavioral data or raw fMRI signals would provide deeper insights into the hierarchy of human cognitive processing.
Summary updated