Scalable, reproducible, and cost-effective processing of large-scale medical imaging datasets

Michael E. Kim, Karthik Ramadass, Chenyu Gao, Praitayini Kanakaraj, Nancy R. Newlin, Gaurav Rudravaram, Kurt G. Schilling, Blake E. Dewey, Derek B. Archer, Timothy J. Hohman, Zhiyuan Li, Shunxing Bao, Bennett A. Landman, Nazirah Mohd KhairiOpenReview 2024 · DOI · OpenReview

TL;DR

This paper introduces a novel, BIDS-compliant system for efficiently and cost-effectively processing large-scale medical imaging datasets. By leveraging a combination of local high-performance computing (HPC) resources, containerization, and a semi-automated workflow, the authors demonstrate a method that is nearly 20 times cheaper than cloud-based solutions while maintaining comparable processing times and significantly faster data transfer speeds.

What Problem Does This Paper Address?

The explosion of large-scale medical imaging studies, particularly in neuroimaging, presents a significant challenge for researchers. Processing these massive datasets is computationally intensive, requires substantial data throughput, and often incurs high financial costs, especially when relying on cloud services. Furthermore, ensuring consistency and reproducibility across a team of researchers working with diverse data and software versions is a complex undertaking. Existing solutions often struggle to balance cost, speed, scalability, and ease of use, hindering the pace of scientific discovery.

Methods

The core of the proposed solution is a BIDS-compliant data processing pipeline built on several key components:

  • BIDS Compliance: All datasets are organized according to the Brain Imaging Data Structure (BIDS) standard, ensuring consistency and compatibility with existing neuroimaging tools.
  • Containerization (Singularity): Processing pipelines are containerized using Singularity images. This approach allows for reproducible execution of complex software environments without requiring administrative OS permissions, making it accessible to all team members.
  • Semi-Automated Workflow: A system is developed to automatically query for available data, generate processing scripts, and create SLURM job array scripts for submission to the HPC cluster. This automates much of the tedious manual work involved in large-scale processing.
  • Heterogeneous Computing Resources: The system utilizes Vanderbilt University's Advanced Computing Center for Research and Education (ACCRE) cluster, a high-performance computing environment. This allows for parallelized computation at a significantly lower cost than cloud alternatives.
  • Dual Storage System: Data is stored on two separate servers: one for general datasets and another for GDPR-compliant data. Symbolic links are used to provide authorized access to the high-security server.
  • Fast Near-line Storage: The system prioritizes fast data transfer speeds between storage and compute nodes, crucial for handling large imaging files.
  • Cost-Effective Archival: Data is backed up to Amazon Glacier Deep Archive for long-term, low-cost storage.

Datasets

The paper utilizes data from 20 separate national-scale studies, including T1-weighted and diffusion-weighted MRI data. Specific datasets mentioned include:

  • MASIVar dataset (used for the experiment)
  • ADNI (Alzheimer's Disease Neuroimaging Initiative)
  • NACC (National Alzheimer's Coordinating Center)
  • UKBB (UK Biobank)

The paper does not introduce new benchmark datasets but rather focuses on processing existing, large-scale datasets.

Key Insights and Ideas

  • HPC + Containerization = Cost-Effective Scalability: The combination of readily available HPC resources and containerization (Singularity) offers a powerful and affordable alternative to cloud computing for large-scale neuroimaging analysis.
  • BIDS is Crucial for Reproducibility: Adhering to the BIDS standard is fundamental for ensuring that processing pipelines are consistent, reproducible, and easily integrated with other tools and workflows.
  • Data Transfer Speed Matters: The paper highlights that while computation time can be similar across environments, the efficiency of data transfer significantly impacts overall processing time, especially for large files.
  • Semi-Automation Reduces Bottlenecks: Automating the querying and script generation process streamlines the workflow, allowing researchers to focus on analysis rather than manual setup.
  • Cost is a Major Hurdle: The significant cost difference between HPC and cloud solutions underscores the importance of developing cost-effective strategies for large-scale data processing.

Key Results

  • Cost-Effectiveness: The proposed method is nearly 20 times more cost-effective than using AWS cloud computation for processing. The direct cost for processing a small dataset was $0.36 on ACCRE HPC versus $6.59 on AWS.
  • Processing Time: Computation time for Freesurfer was comparable across environments: 375.5 minutes/image on ACCRE HPC, 355.2 minutes/image on AWS, and 386.0 minutes/image on local computation.
  • Data Transfer Speed: The system achieved an average data throughput of 0.60 Gb/s from storage to compute on ACCRE HPC, significantly faster than AWS cloud (0.33 Gb/s). Local computation achieved the fastest at 0.81 Gb/s.
  • Latency: ACCRE HPC exhibited low latency (0.16 ms), while AWS cloud had considerably higher latency (19.56 ms).

Limitations and Future Work

  • CLI Dependence: The data archival solution relies on a command-line interface (CLI), which may be a barrier for researchers unfamiliar with command-line tools.
  • Datalad Integration: While Datalad is acknowledged as a useful tool for tracking data structure changes, it was not incorporated due to the dynamic nature of the current database. The authors suggest reconsidering its integration as the database becomes more stable.
  • Future Work: The authors plan to release their processing pipelines as BIDS-Apps, which would offer an alternative deployment method for users who prefer BIDS-Apps and Docker. They also note that the method can be adapted to other imaging modalities.

Summary updated

Abstract

Curating, processing, and combining large-scale medical imaging datasets from national studies is a non-trivial task due to the intense computation and data throughput required, variability of acquired data, and associated financial overhead. Existing platforms or tools for large-scale data curation, processing, and storage have difficulty achieving a viable cost-to-scale ratio of computation speed for research purposes, either being too slow or too expensive. Additionally, management and consistency of processing large data in a team-driven manner is a non-trivial task. We design a BIDS-compliant method for an efficient and robust data processing pipeline of large-scale diffusion-weighted and T1-weighted MRI data compatible with low-cost, high-efficiency computing systems. Our method accomplishes automated querying of data available for processing and process running in a consistent and reproducible manner that has long-term stability, while using heterogenous low-cost computational resources and storage systems for efficient processing and data transfer. We demonstrate how our organizational structure permits efficiency in a semi-automated data processing pipeline and show how our method is comparable in processing time to cloud-based computation while being almost 20 times more cost-effective. Our design allows for fast data throughput speeds and low latency to reduce the time for data transfer between storage servers and computation servers, achieving an average of 0.60 Gb/s compared to 0.33 Gb/s for using cloud-based processing methods. The design of our workflow engine permits quick process running while maintaining flexibility to adapt to newly acquired data.
BibTeX
@misc{kim2024scalable,
  author = {Michael E. Kim and Karthik Ramadass and Chenyu Gao and Praitayini Kanakaraj and Nancy R. Newlin and Gaurav Rudravaram and Kurt G. Schilling and Blake E. Dewey and Derek B. Archer and Timothy J. Hohman and Zhiyuan Li and Shunxing Bao and Bennett A. Landman and Nazirah Mohd Khairi},
  title = {{Scalable, reproducible, and cost-effective processing of large-scale medical imaging datasets}},
  year = {2024},
  eprint = {2408.14611},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.14611},
  howpublished = {OpenReview},
}
Figure