Accelerating Fibre Orientation Estimation from Diffusion Weighted Magnetic Resonance Imaging Using GPUs

M. Hernández, Ginés D. Guerrero, José M. Cecilia, José M. Garcı́a, Alberto Inuggi, Saâd Jbabdi, Timothy E.j. Behrens, Stamatios N. SotiropoulosPLoS 2013 · DOI

TL;DR

Researchers have developed a GPU-accelerated version of the popular "ball & stick" model for diffusion MRI, achieving speedups of over 100x compared to traditional CPU implementations. By parallelizing complex Bayesian inference at both the voxel and sub-voxel levels, the processing time for a full brain dataset drops from over 24 hours to just a few minutes, making high-resolution clinical tractography significantly more feasible.

What Problem Does This Paper Address?

Diffusion-weighted Magnetic Resonance Imaging (DW-MRI) is the only non-invasive way to study the structural connectivity of the living human brain. By measuring how water molecules diffuse along axonal "cables," researchers can estimate the orientation of white matter fibers.

However, the gold-standard method for this—the "ball & stick" model implemented in the FSL bedpostX toolbox—is computationally punishing. It uses a Bayesian inference framework called Markov Chain Monte Carlo (MCMC) to estimate fiber orientations in every single volume element (voxel) of the brain. Because a single dataset can contain millions of voxels and requires thousands of MCMC iterations, a standard single-core CPU can take more than a day to process one patient. This bottleneck hinders both large-scale research studies and real-time clinical applications like neurosurgical planning.

Methods

The authors ported the FSL bedpostX algorithm to NVIDIA’s CUDA (Compute Unified Device Architecture) platform. Their approach utilizes two levels of parallelism:

  1. Voxel-level Parallelism: Since the fiber orientation in one voxel doesn't depend on its neighbor, the authors assigned different voxels to different GPU threads.
  2. Sub-voxel Parallelism: Within the MCMC kernel, the most expensive step is calculating the "likelihood"—predicting the MRI signal across hundreds of different gradient directions. The authors parallelized this specific calculation across threads within a block, allowing a single voxel's math to be shared by multiple cores.

The implementation consists of two main CUDA kernels:

  • Levenberg-Marquardt (LM) Kernel: A deterministic fit used to provide a smart starting point for the MCMC.
  • MCMC Kernel: The core Bayesian engine that draws samples to build a probability distribution of fiber orientations.

Datasets

The team evaluated their design using two primary datasets:

  • Standard Resolution: Acquired on a 3T Siemens Trio system (30 slices, 2mm isotropic resolution, 256 gradient directions).
  • High Resolution: Acquired via the Human Connectome Project on a 3T Connectom Skyra system (102 slices, 1.25mm isotropic resolution, 256 gradient directions).
  • Validation Data: They also used specific voxels from the corpus callosum, centrum semiovale, and grey matter to ensure the GPU and CPU produced statistically identical results.

Key Insights and Ideas

  • Breaking the Sequential Nature of MCMC: While MCMC is inherently sequential (iteration $N$ depends on $N-1$), the authors realized the internal operations of each iteration—specifically the signal prediction for $K$ gradient directions—are "embarrassingly parallel."
  • Occupancy Optimization: The authors found that a thread block size of $Q=64$ was the "sweet spot." It was large enough to hide memory access latencies but small enough to avoid exhausting the GPU's register file, which is a common pitfall in complex CUDA kernels.
  • Deterministic Initialization: By using a parallelized Levenberg-Marquardt fit first, they ensured the MCMC reached convergence faster, further reducing the total workload.

Key Results

  • Massive Speedups: A single NVIDIA Tesla C2050 GPU achieved a 112x speedup over a single-core Intel Xeon CPU for the full bedpostX pipeline.
  • Scalability: When moving to a cluster environment, a multi-GPU setup (102 GPUs) outperformed a multi-CPU cluster by up to 120x.
  • Time Savings: Processing that previously took 24–48 hours was completed in under 15 minutes.
  • Accuracy: The researchers confirmed that the GPU implementation produced model estimates (diffusivity, baseline signal, and fiber fractions) that were statistically indistinguishable from the original CPU-based FSL code.

Limitations and Future Work

The primary limitation is that the current implementation is tied to NVIDIA’s proprietary CUDA platform, meaning it cannot run on other GPU hardware (like AMD or Intel) without being ported to a language like OpenCL.

Additionally, the authors noted that while the GPU handles the heavy lifting, the CPU remains largely idle during the process, acting only as a "monitor." Future iterations could explore heterogeneous computing, where the CPU and GPU share the workload more dynamically to squeeze out even more performance. Finally, the authors suggest that this framework could be extended to even more complex, multi-shell diffusion models that are currently too slow for practical use.

Summary updated

Abstract

With the performance of central processing units (CPUs) having effectively reached a limit, parallel processing offers an alternative for applications with high computational demands. Modern graphics processing units (GPUs) are massively parallel processors that can execute simultaneously thousands of light-weight processes. In this study, we propose and implement a parallel GPU-based design of a popular method that is used for the analysis of brain magnetic resonance imaging (MRI). More specifically, we are concerned with a model-based approach for extracting tissue structural information from diffusion-weighted (DW) MRI data. DW-MRI offers, through tractography approaches, the only way to study brain structural connectivity, non-invasively and in-vivo. We parallelise the Bayesian inference framework for the ball & stick model, as it is implemented in the tractography toolbox of the popular FSL software package (University of Oxford). For our implementation, we utilise the Compute Unified Device Architecture (CUDA) programming model. We show that the parameter estimation, performed through Markov Chain Monte Carlo (MCMC), is accelerated by at least two orders of magnitude, when comparing a single GPU with the respective sequential single-core CPU version. We also illustrate similar speed-up factors (up to 120x) when comparing a multi-GPU with a multi-CPU implementation.
BibTeX
@inproceedings{hernandez2013accelerating,
  author = {M. Hernández and Ginés D. Guerrero and José M. Cecilia and José M. Garcı́a and Alberto Inuggi and Saâd Jbabdi and Timothy E.J. Behrens and Stamatios N. Sotiropoulos},
  title = {{Accelerating Fibre Orientation Estimation from Diffusion Weighted Magnetic Resonance Imaging Using GPUs}},
  booktitle = {PLoS},
  year = {2013},
  doi = {10.1371/journal.pone.0061892},
}
Figure