A large, curated, open-source stroke neuroimaging dataset to improve lesion segmentation algorithms

Sook‐Lei Liew, Bethany Lo, Miranda R. Donnelly, Artemis Zavaliangos‐Petropulu, Jessica N. Jeong, Giuseppe Barisano, A. Hutton, Julia Pia Simon, Julia M. Juliano, Anisha Suri, Tyler Ard, Nerisa Banaj, Michael R. Borich, Lara A. Boyd, Amy Brodtmann, Cathrin M. Buetefisch, Lei Cao, Jessica M. Cassidy, Valentina Ciullo, Adriana Bastos Conforto, Steven C. Cramer, Rosalía Dacosta‐Aguayo, Ezequiel de la Rosa, Martin Domín, Adrienne N. Dula, Wuwei Feng, Alexandre R. Franco, Fatemeh Geranmayeh, Alexandre Gramfort, Chris M. Gregory, Colleen A. Hanlon, Brenton Hordacre, Steven A. Kautz, Mohamed Salah Khlif, Hosung Kim, Jan S. Kirschke, Jingchun Liu, Martín Lotze, Bradley J. MacIntosh, María Mataró, Feroze B. Mohamed, Jan Egil Nordvik, Gilsoon Park, Amy Pienta, Fabrizio Piras, Shane M. Redman, Kate Revill, Mauricio Reyes, Andrew D. Robertson, Na Jin Seo, Surjo R. Soekadar, Gianfranco Spalletta, Alison Sweet, Maria Teleńczuk, Gregory Thielman, Lars T. Westlye, Carolee J. Winstein, George F. Wittenberg, Kristin A. Wong, Chunshui YumedRxiv 2021 · DOI

Abstract

Accurate lesion segmentation is critical in stroke rehabilitation research for the quantification of lesion burden and accurate image processing. Current automated lesion segmentation methods for T1-weighted (T1w) MRIs, commonly used in rehabilitation research, lack accuracy and reliability. Manual segmentation remains the gold standard, but it is time-consuming, subjective, and requires significant neuroanatomical expertise. We previously released a large, open-source dataset of stroke T1w MRIs and manually segmented lesion masks (ATLAS v1.2, N=304) to encourage the development of better algorithms. However, many methods developed with ATLAS v1.2 report low accuracy, are not publicly accessible or are improperly validated, limiting their utility to the field. Here we present ATLAS v2.0 (N=955), a larger dataset of T1w stroke MRIs and manually segmented lesion masks that includes both training (public) and test (hidden) data. Algorithm development using this larger sample should lead to more robust solutions, and the hidden test data allows for unbiased performance evaluation via segmentation challenges. We anticipate that ATLAS v2.0 will lead to improved algorithms, facilitating large-scale stroke rehabilitation research.

Introduction: The Challenge of Automated Segmentation

In stroke rehabilitation research, identifying the precise location and volume of a lesion (lesion burden) is essential for predicting patient outcomes. While manual segmentation by experts is the current "gold standard," it is labor-intensive and requires specialized neuroanatomical knowledge. Automated algorithms offer a potential solution, but they frequently struggle with T1-weighted (T1w) MRIs—the high-resolution standard in rehabilitation—due to the lack of contrast found in acute-phase multispectral imaging.

Previous efforts to automate this process were hindered by small datasets and a lack of "hidden" test sets, which often led to model overfitting. Consequently, many existing tools fail to generalize to new, real-world scans.

The ATLAS v2.0 Dataset

To overcome these hurdles, we developed ATLAS v2.0 (Anatomical Tracings of Lesions After Stroke), a curated dataset of 955 T1w MRIs and manually segmented lesion masks. This release triples the size of the previous version and standardizes data from 33 research cohorts across 20 institutions worldwide.

Rigorous Standardization and Quality Control

To ensure the dataset is robust for machine learning, we employed a strict manual pipeline:

  • Expert Tracing: Trained tracers used ITK-SNAP to delineate lesions in native space.
  • Quality Control: Every mask underwent review and editing by at least two additional researchers to ensure boundary accuracy.
  • Preprocessing: The team released both raw data and a preprocessed version featuring intensity normalization, registration to the MNI-152 template, and "defacing" to protect participant privacy.
  • BIDS Compliance: Data is organized according to the Brain Imaging Data Structure (BIDS), ensuring compatibility with standard neuroimaging software.

Dataset Composition

The dataset is structured to facilitate both training and unbiased evaluation:

  • Public Training Set (n=655): Includes T1w MRIs and corresponding manual lesion masks.
  • Hidden Test Set (n=300): Only T1w scans are public; masks are withheld for fair performance evaluation in challenges.
  • Generalizability Set (n=135): A completely hidden set from entirely new cohorts used to test how well algorithms handle unseen data.

Key Insights: Bridging the Reliability Gap

A major finding in current research is the "reliability gap." While clinical utility typically requires a Dice coefficient (a measure of spatial overlap) of at least 0.85, most existing public methods only achieve scores between 0.5 and 0.7.

Comparison of existing automated lesion segmentation methods and their reported performance metrics.

As shown in the table above, there is significant variability in how current methods are validated and the performance they report. ATLAS v2.0 addresses this by introducing hidden test sets to enforce scientific honesty and ensure models generalize across diverse stroke presentations, including unilateral, bilateral, and brainstem lesions.

Conclusion and Future Work

ATLAS v2.0 is now the largest open-source dataset of its kind. By providing a massive, curated training set alongside hidden test data, we aim to bridge the gap between current automated performance and the reliability needed for clinical research. While currently focused on T1w MRIs, future work will involve using this data to fuel international segmentation challenges, encouraging the development of 3D convolutional neural networks (CNNs) that can finally match human performance in the clinic.

Summary updated

Abstract

Abstract Accurate lesion segmentation is critical in stroke rehabilitation research for the quantification of lesion burden and accurate image processing. Current automated lesion segmentation methods for T1-weighted (T1w) MRIs, commonly used in rehabilitation research, lack accuracy and reliability. Manual segmentation remains the gold standard, but it is time-consuming, subjective, and requires significant neuroanatomical expertise. We previously released a large, open-source dataset of stroke T1w MRIs and manually segmented lesion masks (ATLAS v1.2, N=304) to encourage the development of better algorithms. However, many methods developed with ATLAS v1.2 report low accuracy, are not publicly accessible or are improperly validated, limiting their utility to the field. Here we present ATLAS v2.0 (N=955), a larger dataset of T1w stroke MRIs and manually segmented lesion masks that includes both training (public) and test (hidden) data. Algorithm development using this larger sample should lead to more robust solutions, and the hidden test data allows for unbiased performance evaluation via segmentation challenges. We anticipate that ATLAS v2.0 will lead to improved algorithms, facilitating large-scale stroke rehabilitation research.
BibTeX
@misc{liew2021large,
  author = {Sook‐Lei Liew and Bethany Lo and Miranda R. Donnelly and Artemis Zavaliangos‐Petropulu and Jessica N. Jeong and Giuseppe Barisano and A. Hutton and Julia Pia Simon and Julia M. Juliano and Anisha Suri and Tyler Ard and Nerisa Banaj and Michael R. Borich and Lara A. Boyd and Amy Brodtmann and Cathrin M. Buetefisch and Lei Cao and Jessica M. Cassidy and Valentina Ciullo and Adriana Bastos Conforto and Steven C. Cramer and Rosalía Dacosta‐Aguayo and Ezequiel de la Rosa and Martin Domín and Adrienne N. Dula and Wuwei Feng and Alexandre R. Franco and Fatemeh Geranmayeh and Alexandre Gramfort and Chris M. Gregory and Colleen A. Hanlon and Brenton Hordacre and Steven A. Kautz and Mohamed Salah Khlif and Hosung Kim and Jan S. Kirschke and Jingchun Liu and Martín Lotze and Bradley J. MacIntosh and María Mataró and Feroze B. Mohamed and Jan Egil Nordvik and Gilsoon Park and Amy Pienta and Fabrizio Piras and Shane M. Redman and Kate Revill and Mauricio Reyes and Andrew D. Robertson and Na Jin Seo and Surjo R. Soekadar and Gianfranco Spalletta and Alison Sweet and Maria Teleńczuk and Gregory Thielman and Lars T. Westlye and Carolee J. Winstein and George F. Wittenberg and Kristin A. Wong and Chunshui Yu},
  title = {{A large, curated, open-source stroke neuroimaging dataset to improve lesion segmentation algorithms}},
  year = {2021},
  doi = {10.1101/2021.12.09.21267554},
  howpublished = {medRxiv},
}
Figure