Diffusion MRI microstructure models with in vivo human brain Connectome data: results from a multi‐group comparison

Uran Ferizi, Benoît Scherrer, Torben Schneider, Mohammad Alipoor, Odin Eufracio, Rutger Fick, Rachid Deriche, Markus Nilsson, Ana K. Loya-Olivas, Mariano Rivera, Dirk H. J. Poot, Alonso Ramírez-Manzanares, José L. Marroquín, Ariel Rokem, Christian De Potter, Robert F. Dougherty, Ken Sakaie, Claudia A. M. Gandini Wheeler‐Kingshott, Simon K. Warfield, Thomas Witzel, Lawrence L. Wald, José G. Raya, Daniel C. AlexanderNMR 2017 · DOI

TL;DR

This paper presents the results of the "White Matter Modeling Challenge," a competition designed to compare various mathematical models used to interpret diffusion MRI data. The challenge revealed that simpler, biophysical "tissue models" often outperform more complex "signal models" in predicting unseen data, and that removing signal outliers can improve performance.

What Problem Does This Paper Address?

Diffusion-weighted Magnetic Resonance Imaging (DW-MRI) is a powerful tool for non-invasively studying the brain's white matter microstructure. However, a vast array of mathematical models exist to interpret the complex signals generated by DW-MRI. These models range from those that describe the physical properties of tissue (tissue models) to those that focus solely on fitting the observed signal (signal models). Despite the proliferation of these models, there's a significant lack of standardized, head-to-head comparisons on real-world human brain data. This makes it difficult to determine which models are most accurate, robust, and ultimately useful for clinical applications. This paper addresses this gap by organizing a challenge to rigorously compare a diverse set of DW-MRI models.

Methods

The core of this paper is the "White Matter Modeling Challenge." Participants were given three-quarters of a rich, in-vivo DW-MRI dataset and tasked with building models to predict the remaining quarter. The models were evaluated based on their ability to accurately predict the unseen signal, using the sum of squared differences (SSE) as the primary metric, corrected for Rician noise. The challenge encouraged a wide variety of modeling approaches, including different noise assumptions, fitting algorithms, and parameter estimation strategies. The paper also provides detailed descriptions of each competing model in an appendix.

Datasets

The study utilized a high-quality, in-vivo DW-MRI dataset acquired from a single healthy volunteer using the Human Connectome scanner. This scanner is notable for its high gradient strength (up to 300 mT/m), allowing for a very rich sampling of the diffusion measurement space. The dataset was split into a training set (three-quarters) and a testing set (one-quarter). The data was acquired across two regions of interest: the genu of the corpus callosum (relatively straight fibers) and the fornix (more complex fiber architecture).

Key Insights and Ideas

  • Tissue Models Shine: Surprisingly, biophysical "tissue models" (which explicitly describe tissue microstructure) generally outperformed more flexible "signal models" (which focus on fitting the signal without explicit biophysical assumptions). This challenges the common perception that signal models offer superior flexibility.
  • Overfitting is a Risk: The superior performance of tissue models suggests that the flexibility of signal models can lead to overfitting the training data, hindering their ability to generalize to unseen data.
  • Noise Modeling Nuances: Assuming a non-Gaussian noise model provided little benefit over a Gaussian assumption, likely due to the high signal-to-noise ratio (SNR) of the acquired data.
  • Outlier Removal Matters: Preprocessing the training data by removing signal outliers significantly improved model performance for some models and was a common strategy among the top performers.
  • Prediction vs. Explanation: While models that predict unseen data well are valuable for rejecting poor models, this doesn't automatically mean they are mechanistically closer to the true tissue structure. Further validation is needed.

Key Results

The challenge involved 14 distinct models from various research groups. The top-ranking models, which demonstrated the best prediction accuracy, were primarily tissue models. The paper highlights that the ranking was largely dominated by performance on shells with b-values between 750 and 1400 s/mm². While specific quantitative performance metrics for each model are presented in figures and tables, the overarching finding is the consistent performance of certain tissue models across different regions and b-value ranges.

Limitations and Future Work

The authors acknowledge several limitations. The acquisition protocol was exceptionally rich and not representative of typical clinical scans, which might influence model performance. The study focused on isolated voxels, and future work could extend this to whole-brain analysis. The computational demand of models was not considered, which could impact their practical applicability. Future challenges could explore more typical clinical acquisition protocols and investigate the generalizability of findings across subjects. The authors also suggest that future work should focus on validating the parameters estimated by these models against independent measurements, such as histology, though they note the inherent difficulties in obtaining ground truth. The challenge data and model details are made publicly available to encourage further research and model development.

Summary updated

Abstract

A large number of mathematical models have been proposed to describe the measured signal in diffusion-weighted (DW) magnetic resonance imaging (MRI). However, model comparison to date focuses only on specific subclasses, e.g. compartment models or signal models, and little or no information is available in the literature on how performance varies among the different types of models. To address this deficiency, we organized the 'White Matter Modeling Challenge' during the International Symposium on Biomedical Imaging (ISBI) 2015 conference. This competition aimed to compare a range of different kinds of models in their ability to explain a large range of measurable in vivo DW human brain data. Specifically, we assessed the ability of models to predict the DW signal accurately for new diffusion gradients and b values. We did not evaluate the accuracy of estimated model parameters, as a ground truth is hard to obtain. We used the Connectome scanner at the Massachusetts General Hospital, using gradient strengths of up to 300 mT/m and a broad set of diffusion times. We focused on assessing the DW signal prediction in two regions: the genu in the corpus callosum, where the fibres are relatively straight and parallel, and the fornix, where the configuration of fibres is more complex. The challenge participants had access to three-quarters of the dataset and their models were ranked on their ability to predict the remaining unseen quarter of the data. The challenge provided a unique opportunity for a quantitative comparison of diverse methods from multiple groups worldwide. The comparison of the challenge entries reveals interesting trends that could potentially influence the next generation of diffusion-based quantitative MRI techniques. The first is that signal models do not necessarily outperform tissue models; in fact, of those tested, tissue models rank highest on average. The second is that assuming a non-Gaussian (rather than purely Gaussian) noise model provides little improvement in prediction of unseen data, although it is possible that this may still have a beneficial effect on estimated parameter values. The third is that preprocessing the training data, here by omitting signal outliers, and using signal-predicting strategies, such as bootstrapping or cross-validation, could benefit the model fitting. The analysis in this study provides a benchmark for other models and the data remain available to build up a more complete comparison in the future.
BibTeX
@inproceedings{ferizi2017diffusion,
  author = {Uran Ferizi and Benoît Scherrer and Torben Schneider and Mohammad Alipoor and Odin Eufracio and Rutger Fick and Rachid Deriche and Markus Nilsson and Ana K. Loya-Olivas and Mariano Rivera and Dirk H. J. Poot and Alonso Ramírez-Manzanares and José L. Marroquín and Ariel Rokem and Christian De Potter and Robert F. Dougherty and Ken Sakaie and Claudia A. M. Gandini Wheeler‐Kingshott and Simon K. Warfield and Thomas Witzel and Lawrence L. Wald and José G. Raya and Daniel C. Alexander},
  title = {{Diffusion MRI microstructure models with in vivo human brain Connectome data: results from a multi‐group comparison}},
  booktitle = {NMR},
  year = {2017},
  doi = {10.1002/nbm.3734},
}
Figure