NeuroQuery, comprehensive meta-analysis of human brain mapping

Jérôme Dockès, Russell A. Poldrack, Romain Primet, Hande Gözükan, Tal Yarkoni, Fabian M. Suchanek, Bertrand Thirion, Gaël VaroquauxeLife 2020 · DOI

TL;DR

NeuroQuery offers a novel approach to brain mapping meta-analysis by shifting from inference to prediction. It builds a comprehensive model of neuroscience terms and their associated brain regions using a large corpus of full-text articles, enabling it to generate brain maps for arbitrary queries, including rare concepts and complex descriptions.

What Problem Does This Paper Address?

The vast and ever-growing body of neuroscience literature, particularly from brain imaging studies, presents a significant challenge for researchers trying to synthesize findings and gain a global understanding of brain organization. A key hurdle is the lack of a standardized, comprehensive vocabulary for describing cognitive processes, leading to diverse terminology and making it difficult to relate findings across studies. Existing meta-analysis tools often rely on simple keyword matching or statistical tests on frequently occurring terms, limiting their ability to handle rare concepts, complex queries, or nuanced descriptions of experimental paradigms. This makes it hard to generate hypotheses or interpret results for less common or multifaceted research questions.

Methods

NeuroQuery introduces a predictive meta-analysis framework that moves beyond traditional inference-based methods. Its core innovation lies in a multivariate model that predicts the spatial distribution of neurological observations based on textual descriptions. This model is trained on a large corpus of full-text neuroimaging publications.

Key methodological components include:

  • Semantic Smoothing: To address the issue of inconsistent terminology and rare words, NeuroQuery leverages distributional semantics and co-occurrence statistics from the literature. It uses Non-negative Matrix Factorization (NMF) to learn a denoised term-to-term similarity matrix, allowing it to infer relationships between related concepts. This helps in mapping rare or difficult terms by leveraging information from more frequently studied, related concepts.
  • Reweighted Ridge Regression and Feature Selection: To handle the high-dimensional and sparse nature of TFIDF features derived from text, NeuroQuery employs a reweighted ridge regression approach. This method adaptively regularizes features, effectively selecting a smaller set of informative terms that have a strong statistical link with brain activity. This process is crucial for preventing noise and improving predictive power.
  • Predictive Modeling: Unlike traditional meta-analyses that perform statistical tests to establish consensus, NeuroQuery's model is designed for out-of-sample prediction. It learns to map text queries to brain activation maps, enabling it to generate maps for novel combinations of terms or even free-text descriptions.

Datasets

  • Training Data: The primary dataset consists of 13,459 full-text neuroimaging publications. This corpus is significantly larger than those used by previous tools like NeuroSynth, which primarily relied on abstracts.
  • Vocabulary: A comprehensive vocabulary of 7,547 neuroscience terms and phrases was curated from multiple sources, including MeSH, Cognitive Atlas, NeuroNames, NIF, and terms from NeuroSynth. Terms occurring in less than 5 out of 10,000 articles were excluded.
  • Evaluation Data: The paper uses various datasets and benchmarks for validation, including:
  • The Individual Brain Charting (IBC) dataset for comparing NeuroQuery maps with experimental fMRI data.
  • Publicly available Coordinate-Based Meta-Analysis (CBMA) maps from the BrainPedia study.
  • Regions from the Harvard-Oxford anatomical atlas.
  • NeuroSynth's own "association test" maps as a reference for well-established terms.

Key Insights and Ideas

  • Prediction over Inference: The fundamental shift from statistical inference to predictive modeling is a paradigm change for meta-analysis, allowing for broader applications beyond simply confirming existing findings.
  • Leveraging Full Text: Utilizing the full text of research articles, rather than just abstracts, provides a richer representation of studies, significantly increasing the number of term occurrences and thus improving statistical power and vocabulary coverage.
  • Semantic Smoothing for Rare Terms: The ability to infer relationships between terms through semantic smoothing is crucial for mapping concepts that are infrequently studied or described with diverse terminology. This allows NeuroQuery to generate meaningful maps for terms that would be unmappable by traditional methods.
  • Arbitrary Queries: The framework's capacity to handle arbitrary text queries, including complex descriptions and combinations of terms, democratizes meta-analysis, making it accessible for hypothesis generation and exploration of novel research questions.

Key Results

  • Improved Coverage: NeuroQuery's vocabulary is significantly larger than NeuroSynth's (7547 terms vs. 3228 terms), enabling meta-analyses for a wider range of concepts.
  • Enhanced Prediction Accuracy: In out-of-sample prediction tasks, NeuroQuery consistently outperformed NeuroSynth, GCLDA, and a naive baseline, achieving higher log-likelihoods for predicting reported coordinates. For example, it achieved a median Pearson correlation of 0.85 when predicting maps for unseen term combinations.
  • Robustness to Rare Terms: NeuroQuery demonstrated superior performance in mapping rare or difficult concepts compared to NeuroSynth and GCLDA, often producing usable maps where other tools failed.
  • High Agreement with References: NeuroQuery's maps showed strong agreement with established CBMA maps (median AUC of 0.80 with BrainPedia) and anatomical atlases (median AUC above 0.9 with Harvard-Oxford atlas). When compared to NeuroSynth for well-activated terms, NeuroQuery achieved a median AUC of 0.90.
  • Efficient Feature Selection: The reweighted ridge regression and feature selection process efficiently identified approximately 200 informative features, reducing computational burden and improving prediction time.

Limitations and Future Work

  • Not a Statistical Test: NeuroQuery produces predictions, not statistical tests. Its maps cannot be used to reject null hypotheses, and their interpretation requires careful consideration of the underlying model.
  • Potential for Misleading Predictions: While generally accurate, individual predictions can be wrong. The tool provides warnings for potentially unreliable results, but users should still exercise critical judgment.
  • Additive Combination Model: The model assumes an additive combination of terms, which might not perfectly capture all complex interactions between concepts. This can lead to dominance by more prominent terms (e.g., "visual" in "visual sentence comprehension").
  • Confirmation Bias: Like all meta-analyses, NeuroQuery is susceptible to biases present in the primary literature, such as confirmation bias.
  • Future Work: The authors suggest further improvements in coordinate extraction, handling of Talairach vs. MNI space, and exploring more sophisticated models for combining terms. The tool is made openly available online and as a Python package to encourage further development and application.

Summary updated

Abstract

Reaching a global view of brain organization requires assembling evidence on widely different mental processes and mechanisms. The variety of human neuroscience concepts and terminology poses a fundamental challenge to relating brain imaging results across the scientific literature. Existing meta-analysis methods perform statistical tests on sets of publications associated with a particular concept. Thus, large-scale meta-analyses only tackle single terms that occur frequently. We propose a new paradigm, focusing on prediction rather than inference. Our multivariate model <i>predicts</i> the spatial distribution of neurological observations, given text describing an experiment, cognitive process, or disease. This approach handles text of arbitrary length and terms that are too rare for standard meta-analysis. We capture the relationships and neural correlates of 7547 neuroscience terms across 13 459 neuroimaging publications. The resulting meta-analytic tool, neuroquery.org, can ground hypothesis generation and data-analysis priors on a comprehensive view of published findings on the brain.
BibTeX
@inproceedings{dockes2020neuroquery,
  author = {Jérôme Dockès and Russell A. Poldrack and Romain Primet and Hande Gözükan and Tal Yarkoni and Fabian M. Suchanek and Bertrand Thirion and Gaël Varoquaux},
  title = {{NeuroQuery, comprehensive meta-analysis of human brain mapping}},
  booktitle = {eLife},
  year = {2020},
  doi = {10.7554/elife.53385},
}
Figure