Towards Zero-Shot Task-Generalizable Learning on FMRI

Jiyao Wang, Nicha C. Dvornek, Peiyu Duan, Lawrence H. Staib, James S. Duncan

Abstract

Functional MRI (fMRI) provides a window into brain activity across diverse cognitive states, yet developing models that generalize across different tasks remains a significant challenge in neuroimaging. This paper introduces the Task-Aware Graph Attention Network (TA-GAT), a framework designed to aggregate multi-task fMRI data while preserving task-specific context. By utilizing a task memory bank and an orthogonal projection loss, TA-GAT achieves superior performance in gender classification and cognitive score prediction. Crucially, the model demonstrates robust zero-shot generalization, maintaining high accuracy even when encountering brain activity patterns from tasks not seen during training.

Introduction

Functional MRI (fMRI) captures brain activity by measuring the Blood Oxygen Level Dependent (BOLD) signal. Traditionally, this data is categorized into resting-state (where subjects lie still) or task-based (where subjects perform specific cognitive activities). While task-based fMRI is rich with information, it is notoriously difficult to use for training general-purpose models. Because different tasks—such as gambling, language processing, or motor exercises—activate the brain in distinct ways, simply aggregating this data into a single model often creates a "messy" inductive bias. Most current models either focus on a single task or ignore the task context entirely, limiting their ability to generalize. The authors of this study address a fundamental question: How can we aggregate diverse task-based fMRI data while maintaining the unique context of each task to improve generalization?

Methodology

The researchers developed the Task-Aware Graph Attention Network (TA-GAT) to bridge the gap between task-specific nuances and generalizable brain features. The pipeline consists of three primary stages:

1. Graph Construction

The brain is modeled as a graph where 268 regions of interest (ROIs), defined by the Shen atlas, serve as nodes. The edges between these nodes represent functional connectivity, calculated based on the correlations in activity between these regions over time.

2. The TA-GAT Architecture

  • GNN Encoder: A Graph Attention Network (GAT) extracts general features from the brain graph, capturing the underlying spatial relationships of neural activity.
  • Task Memory Bank: This component stores learnable mathematical representations (embeddings) for each specific task. When the model processes a specific scan (e.g., a "Motor" task), it retrieves the corresponding "Motor" embedding from the bank and fuses it with the general brain features extracted by the encoder.
  • Downstream MLPs: The combined information is fed into Multi-Layer Perceptrons (MLPs) to perform specific tasks, such as gender classification or predicting cognitive scores (regression).

3. Orthogonal Projection Loss ($L_{ortho}$)

To prevent task embeddings from becoming redundant or overlapping, the researchers applied an orthogonal projection loss. This forces the task representations in the memory bank to be mathematically perpendicular to one another. This ensures the model captures distinct, non-overlapping contextual information for each task, which is vital for distinguishing between different cognitive states.

Experimental Setup

The study utilized the WU-Minn Human Connectome Project (HCP) 1200 Subjects Data Release, focusing on a subset of 1,008 subjects. The dataset included seven distinct fMRI paradigms: Emotion, Gambling, Language, Motor, Relational, Social, and Working Memory. The models were evaluated on their ability to predict gender and age-adjusted total cognitive function composite scores.

To test generalization, the researchers employed a "leave-one-task-out" validation strategy. The model was trained on six tasks and then tested on the seventh, completely unseen task to evaluate its zero-shot capabilities.

Results and Discussion

Improved Accuracy and Generalization

TA-GAT consistently outperformed standard models that lacked task-awareness. In the "Emotion" task, for instance, gender classification accuracy improved from approximately 78.8% to 82.4%.

Zero-Shot Success

The most significant finding was the model's performance on unseen tasks. In the zero-shot "Emotion" task test, accuracy rose from 78.1% (without task awareness) to 82.2%. This suggests that by learning the "context" of various tasks, the model develops a more robust understanding of brain activity that can be applied to entirely new cognitive paradigms.

The Role of Orthogonality

Ablation studies confirmed that the orthogonal projection loss is essential. Removing this loss significantly hindered the model's ability to generalize, proving that maintaining clear boundaries between task representations is a key requirement for robust multi-task learning in neuroimaging.

Conclusion and Future Work

TA-GAT represents a significant step toward a universal foundation model for fMRI. By treating task context as a first-class citizen in the architecture, the model moves beyond simple data aggregation. However, limitations remain: the study was limited to the seven tasks available in the HCP dataset, and the ROI-based approach may overlook fine-grained information at the individual voxel level. Future research will focus on extending this framework to unsupervised representation learning, potentially allowing models to "discover" task contexts autonomously without the need for explicit labels.

Summary updated

BibTeX
@inproceedings{wang2025towards,
  author = {Jiyao Wang and Nicha C. Dvornek and Peiyu Duan and Lawrence H. Staib and James S. Duncan},
  title = {{Towards Zero-Shot Task-Generalizable Learning on FMRI}},
  booktitle = {ISBI},
  year = {2025},
  doi = {10.1109/isbi60581.2025.10981094},
}
Figure