Text2Brain: Synthesis of Brain Activation Maps from Free-Form Text Query

Gia H. Ngo, Minh Nguyen, Nancy F. Chen, Mert R. Sabuncu

TL;DR

This paper introduces Text2Brain, a novel neural network that synthesizes brain activation maps from free-form text queries. By leveraging a transformer-based text encoder and a 3D convolutional neural network, Text2Brain can generate anatomically plausible brain activity patterns, offering a more flexible and powerful alternative to traditional keyword-based meta-analysis.

What Problem Does This Paper Address?

Neuroimaging research has generated a vast amount of data, but synthesizing this knowledge into actionable insights remains a challenge. Individual studies are often limited by sample size and the specific cognitive processes they investigate. Traditional meta-analysis, which aggregates findings from multiple studies, typically relies on manual selection of relevant experiments and is restricted to rigid keyword queries. This limits the ability to explore complex or nuanced cognitive concepts and discover novel relationships within the existing literature. The authors aim to bridge this gap by enabling the generation of brain activation maps from open-ended, natural language descriptions of cognitive tasks or concepts.

Methods

The core of Text2Brain is a neural network architecture that combines a transformer-based text encoder with a 3D convolutional neural network (CNN) image decoder.

  • Text Encoder: The model uses a SciBERT-based transformer to encode input text queries into rich vector representations. Transformers are adept at capturing contextual information and semantic similarity, allowing Text2Brain to understand nuances in language and map synonymous terms to similar brain activation patterns. This is a departure from older methods that relied on simple keyword co-occurrence.
  • 3D CNN Decoder: This component takes the encoded text representation and generates a 3D brain activation map. The decoder is composed of transposed 3D convolutional layers, enabling it to reconstruct both coarse and fine details of brain activity.
  • Training Strategy: Text2Brain is trained on a large dataset of neuroimaging studies. To encourage generalization and robust learning, an augmented data sampling strategy is employed. This involves sampling activation maps from various sources within an article (table-specific maps and article-average maps) and pairing them with different types of textual descriptions (titles, abstracts, keywords, discussion sentences). This "many-to-one" mapping approach helps the model learn associations between diverse textual inputs and specific brain patterns.

Datasets

  • Training Data: The model was trained on data extracted from over 13,000 published neuroimaging studies. For each study, full text and activation coordinates were collected.
  • Evaluation Data:
    • Article Titles: 1000 articles were sampled for testing, with their titles used as free-form queries to predict article-average activation maps.
  • Human Connectome Project (HCP) Task Contrasts: Data from 86 task contrasts across 7 domains were used. Concise contrast descriptions from the Individual Brain Charting (IBC) project were used as queries, with HCP group-average contrast maps serving as ground truth.

Key Insights and Ideas

  • Beyond Keywords: Text2Brain moves beyond rigid keyword matching by leveraging the semantic understanding capabilities of transformers. This allows for more flexible and intuitive querying of neuroimaging literature.
  • Implicit Semantic Similarity: The model can infer relationships between words and concepts even if they are not explicitly stated, enabling it to generalize to synonymous terms and novel descriptions.
  • Synthesizing Knowledge: The ability to generate activation maps from text offers a powerful new way to synthesize existing neuroimaging findings, potentially uncovering novel hypotheses and relationships.
  • Educational Tool Potential: The system's ability to translate textual descriptions into visual brain maps makes it a valuable tool for education and understanding complex cognitive neuroscience concepts.

Key Results

  • Article Title Prediction: Text2Brain significantly outperformed two established baselines (Neurosynth and Neuroquery) in predicting article-average activation maps from article titles, achieving a higher Area Under the Dice Curve (AUC) score (0.0576 vs. 0.0478 and 0.0464, respectively).
  • Task Contrast Prediction: Text2Brain also showed superior performance in predicting activation maps from contrast descriptions, outperforming baselines with a mean AUC of 0.082 compared to 0.0755 (Neuroquery) and 0.047 (Neurosynth).
  • Handling Synonymous Queries: The model demonstrated robust performance in predicting the "self-generated thought" activation map using different synonymous query terms, closely matching the ground truth and outperforming baselines.
  • Avoiding Failure Cases: Unlike Neurosynth and Neuroquery, Text2Brain was able to generate predictions even for queries not explicitly present in its training vocabulary, such as article titles or contrast descriptions.

Limitations and Future Work

  • Interpretability: The authors acknowledge that further work is needed to improve the interpretability of the model, specifically to attribute regions of activation in the generated map to specific words in the input query.
  • Efficient Matching: Future work will focus on efficiently matching generated activation maps with relevant research texts.
  • Generalization to Novel Concepts: While Text2Brain shows promise in generalizing to synonymous terms, its ability to predict activation for entirely novel or abstract concepts not well-represented in the training data could be further explored.

Summary updated

Abstract

Most neuroimaging experiments are under-powered, limited by the number of subjects and cognitive processes that an individual study can investigate. Nonetheless, over decades of research, neuroscience has accumulated an extensive wealth of results. It remains a challenge to digest this growing knowledge base and obtain new insights since existing meta-analytic tools are limited to keyword queries. In this work, we propose Text2Brain, a neural network approach for coordinate-based meta-analysis of neuroimaging studies to synthesize brain activation maps from open-ended text queries. Combining a transformer-based text encoder and a 3D image generator, Text2Brain was trained on variable-length text snippets and their corresponding activation maps sampled from 13,000 published neuroimaging studies. We demonstrate that Text2Brain can synthesize anatomically-plausible neural activation patterns from free-form textual descriptions of cognitive concepts. Text2Brain is available at https://braininterpreter.com as a web-based tool for retrieving established priors and generating new hypotheses for neuroscience research.
BibTeX
@inproceedings{ngo2021text,
  author = {Gia H. Ngo and Minh Nguyen and Nancy F. Chen and Mert R. Sabuncu},
  title = {{Text2Brain: Synthesis of Brain Activation Maps from Free-Form Text Query}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  doi = {10.1007/978-3-030-87234-2\_57},
}
Figure