Abstract
Chat2Brain is a novel framework that leverages Large Language Models (LLMs) to map complex, open-ended natural language queries to 3D brain activation maps. By employing LLMs for data augmentation (ChatAUG) and iterative semantic refinement (Text2Semantic), the system overcomes limitations of traditional keyword-based meta-analysis tools, such as semantic ambiguity and data scarcity. Experimental results demonstrate significant improvements in AUC and mIoU, particularly in noisy query environments, while maintaining high anatomical plausibility.
Introduction
In neuroscience, meta-analysis is essential for synthesizing thousands of research papers to identify brain regions associated with specific cognitive tasks, such as "working memory" or "language processing." Traditional tools like Neurosynth or NeuroQuery rely primarily on keyword matching. However, these methods face significant challenges: semantic ambiguity where different researchers use varying terms for identical concepts, the inability to parse complex or poorly phrased real-world queries, and a scarcity of high-quality labeled data pairing text with precise 3D brain maps.
Chat2Brain addresses these gaps by utilizing the reasoning and linguistic capabilities of LLMs to translate unstructured human language into structured semantic queries. These queries are then processed by a deep learning model to generate accurate 3D brain activation maps.
Methodology
The Chat2Brain framework introduces two primary innovations: ChatAUG for training and Text2Semantic for inference.
ChatAUG: Data Augmentation
To mitigate data scarcity, the authors utilize ChatGPT to augment the training set. Starting with research paper titles, ChatGPT generates five types of related text: synonymous titles, potential abstracts, experimental design descriptions, and keywords. This process enables the mapping model to learn diverse semantic connections between language and spatial brain activations.
Text2Semantic: Inference and Refinement
During inference, user queries undergo a two-stage "Text2Semantic" process. First, the system identifies similar samples in the research corpus to construct a "dynamic prompt" for the LLM. Second, an iterative optimization loop refines the query into a core "semantic query," ensuring the input to the mapping model remains concise, accurate, and grounded in the neuroscience corpus.
The Mapping Model
The core architecture, inspired by Text2Brain, employs a SciBERT encoder to transform text into a 768-dimensional vector. This vector is then processed by a 3D Convolutional Neural Network (CNN) using transposed convolutions to "up-sample" the data into a 3D volumetric brain map consisting of 40x48x40 voxels.
Results and Discussion
The framework was evaluated using the NeuroQuery database (13,460 articles) across standard and non-standard (noisy) query environments.
Quantitative Performance
In standard environments, Chat2Brain achieved an 8.7% improvement in AUC and a 12.2% improvement in mean Intersection over Union (mIoU) compared to the baseline Text2Brain model. In non-standard environments with disrupted queries, the LLM-based refinement led to a massive performance leap, showing up to a 56.8% improvement in mIoU.
Qualitative Analysis and Anatomical Plausibility
Qualitative results indicate that Chat2Brain produces brain activation maps with higher anatomical plausibility. For instance, in language tasks, the model correctly identifies specific regions like the Broca area, whereas previous models often predicted activations in incorrect hemispheres or locations.
Conclusion
Chat2Brain demonstrates that LLMs can serve as sophisticated semantic pre-processors, cleaning and structuring human language for specialized downstream tasks. The use of synthetic data through ChatAUG helps models generalize despite sparse real-world data, while iterative grounding ensures scientific accuracy. Future work may involve developing domain-specific LLMs tailored for neuroscience and incorporating raw fMRI data to improve the resolution of predicted brain activity.
Summary updated