Chat2Brain: A Method for Mapping Open-Ended Semantic Queries to Brain Activation Maps

Yaonai Wei, Tianyang Zhong, Songyao Zhang, Xiao Li, Tuo Zhang, Lin Zhao, Zhengliang Liu, Muheng Shang, Tianming Liu, Han Zhang, Chong Ma, Lei Du, Junwei Han

Abstract

Chat2Brain is a novel framework that leverages Large Language Models (LLMs) to map complex, open-ended natural language queries to 3D brain activation maps. By employing LLMs for data augmentation (ChatAUG) and iterative semantic refinement (Text2Semantic), the system overcomes limitations of traditional keyword-based meta-analysis tools, such as semantic ambiguity and data scarcity. Experimental results demonstrate significant improvements in AUC and mIoU, particularly in noisy query environments, while maintaining high anatomical plausibility.

Introduction

In neuroscience, meta-analysis is essential for synthesizing thousands of research papers to identify brain regions associated with specific cognitive tasks, such as "working memory" or "language processing." Traditional tools like Neurosynth or NeuroQuery rely primarily on keyword matching. However, these methods face significant challenges: semantic ambiguity where different researchers use varying terms for identical concepts, the inability to parse complex or poorly phrased real-world queries, and a scarcity of high-quality labeled data pairing text with precise 3D brain maps.

Chat2Brain addresses these gaps by utilizing the reasoning and linguistic capabilities of LLMs to translate unstructured human language into structured semantic queries. These queries are then processed by a deep learning model to generate accurate 3D brain activation maps.

Methodology

The Chat2Brain framework introduces two primary innovations: ChatAUG for training and Text2Semantic for inference.

ChatAUG: Data Augmentation

To mitigate data scarcity, the authors utilize ChatGPT to augment the training set. Starting with research paper titles, ChatGPT generates five types of related text: synonymous titles, potential abstracts, experimental design descriptions, and keywords. This process enables the mapping model to learn diverse semantic connections between language and spatial brain activations.

Text2Semantic: Inference and Refinement

During inference, user queries undergo a two-stage "Text2Semantic" process. First, the system identifies similar samples in the research corpus to construct a "dynamic prompt" for the LLM. Second, an iterative optimization loop refines the query into a core "semantic query," ensuring the input to the mapping model remains concise, accurate, and grounded in the neuroscience corpus.

The Text2Semantic process involves dynamic prompting and iterative optimization to refine open-ended queries into structured semantic inputs.

The Mapping Model

The core architecture, inspired by Text2Brain, employs a SciBERT encoder to transform text into a 768-dimensional vector. This vector is then processed by a 3D Convolutional Neural Network (CNN) using transposed convolutions to "up-sample" the data into a 3D volumetric brain map consisting of 40x48x40 voxels.

Results and Discussion

The framework was evaluated using the NeuroQuery database (13,460 articles) across standard and non-standard (noisy) query environments.

Quantitative Performance

In standard environments, Chat2Brain achieved an 8.7% improvement in AUC and a 12.2% improvement in mean Intersection over Union (mIoU) compared to the baseline Text2Brain model. In non-standard environments with disrupted queries, the LLM-based refinement led to a massive performance leap, showing up to a 56.8% improvement in mIoU.

Qualitative Analysis and Anatomical Plausibility

Qualitative results indicate that Chat2Brain produces brain activation maps with higher anatomical plausibility. For instance, in language tasks, the model correctly identifies specific regions like the Broca area, whereas previous models often predicted activations in incorrect hemispheres or locations.

Qualitative comparison showing Chat2Brain (yellow) more accurately matching ground truth (red) compared to the baseline model (blue).

Conclusion

Chat2Brain demonstrates that LLMs can serve as sophisticated semantic pre-processors, cleaning and structuring human language for specialized downstream tasks. The use of synthetic data through ChatAUG helps models generalize despite sparse real-world data, while iterative grounding ensures scientific accuracy. Future work may involve developing domain-specific LLMs tailored for neuroscience and incorporating raw fMRI data to improve the resolution of predicted brain activity.

Summary updated

BibTeX
@inproceedings{wei2023chat,
  author = {Yaonai Wei and Tianyang Zhong and Songyao Zhang and Xiao Li and Tuo Zhang and Lin Zhao and Zhengliang Liu and Muheng Shang and Tianming Liu and Han Zhang and Chong Ma and Lei Du and Junwei Han},
  title = {{Chat2Brain: A Method for Mapping Open-Ended Semantic Queries to Brain Activation Maps}},
  booktitle = {BIBM},
  year = {2023},
  doi = {10.1109/bibm58861.2023.10385933},
}

Loading related content...

Figure