VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021

Carole H. Sudre, Kimberlin M. H. van Wijnen, Florian Dubost, Hieab Adams, David Atkinson, Frederik Barkhof, Mahlet A. Birhanu, Esther E. Bron, Robin Camarasa, Nish Chaturvedi, Yuan Chen, Zihao Chen, Shuai Chen, Qi Dou, Tavia E. Evans, Ivan Ezhov, Haojun Gao, Marta Gironés-Sangüesa, Juan Domingo Gispert, Beatriz Gomez Anson, Alun D. Hughes, Mohammad Arfan Ikram, Silvia Ingala, Hans Rolf Jäger, Florian Kofler, Hugo J. Kuijf, Denis Kutnar, Minho Lee, Bo Li, Luigi Lorenzini, Bjoern H. Menze, José Luis Molinuevo, Yiwei Pan, Élodie Puybareau, Rafael Rehwald, Ruisheng Su, Pengcheng Shi, Lorna Smith, Therese Tillin, Guillaume Tochon, Hélène Urien, Bas H. M. van der Velden, Isabelle F. van der Velpen, Benedikt Wiestler, Frank J. Wolters, Pinar Yilmaz, Marius de Groot, Meike W. Vernooij, Marleen de BruijneMedical Image Analysis 2024 · OpenReview

TL;DR

This paper presents the "Where is VALDO?" challenge, a competition focused on automatically detecting and segmenting small, often sparse, markers of cerebral small vessel disease (CSVD) in brain MRI scans. While promising results were achieved for enlarged perivascular spaces (EPVS) and microbleeds, lacune detection remains a significant challenge, highlighting the need for further research in this area.

What Problem Does This Paper Address?

Cerebral small vessel disease (CSVD) is a major contributor to stroke, dementia, and cognitive decline. Key imaging markers of CSVD, such as enlarged perivascular spaces (EPVS), cerebral microbleeds, and lacunes, are crucial for diagnosis and prognosis. However, manually assessing these markers is time-consuming and prone to significant variability between human annotators. Automated methods could revolutionize CSVD research and clinical assessment, but their reliability and performance for these subtle markers are not well-established. This challenge aimed to spur the development of robust automated methods for detecting and segmenting these challenging lesions.

Methods

The "Where is VALDO?" challenge was structured into three tasks, each focusing on a specific CSVD marker: Task 1 (EPVS), Task 2 (Microbleeds), and Task 3 (Lacunes). Participants were tasked with developing methods for both detection and segmentation. The challenge leveraged a multi-cohort dataset with inherent data imbalance and noisy labels, reflecting real-world scenarios. Most participating teams employed deep learning approaches, with the U-Net architecture being a popular choice. The evaluation metrics included F1 score and Mean Dice score for detection and segmentation, respectively, along with absolute error metrics. Uncertainty estimation was also a component for Task 3.

Datasets

The challenge utilized data from three retrospective population-based cohorts:

  • SABRE (Southall and Brent Revisited): A tri-ethnic cohort in West London.
  • Rotterdam Scan Study (RSS): Part of the larger Rotterdam Study, focusing on elderly individuals.
  • ALFA (Alzheimer's and Families) study: A cohort enriched for genetic predisposition to Alzheimer's Disease.

These datasets were used for training, validation, and testing, with a 6:10 ratio for training to testing data.

Key Insights and Ideas

  • Task-Specific Performance: Performance varied significantly across the three tasks, with EPVS and microbleeds showing more promising results than lacunes.
  • Deep Learning Dominance: The vast majority of successful approaches relied on deep learning, particularly U-Net architectures.
  • Data Imbalance is Key: The challenge's design, incorporating data imbalance and noisy labels, reflects real-world clinical data and pushed for more robust algorithms.
  • Segmentation vs. Detection: While most solutions focused on segmentation, the challenge highlighted that detection performance was often the bottleneck, especially for lacunes.
  • Noisy Labels are a Hurdle: The use of weak and noisy labels presented a significant challenge, particularly for lacune detection.

Key Results

  • Task 1 (EPVS): Promising results were observed, with some methods showing potential for population-level studies. The best performing methods achieved reasonable F1 scores and Mean Dice scores.
  • Task 2 (Microbleeds): Similar to EPVS, good performance was achieved, with some teams demonstrating strong detection and segmentation capabilities.
  • Task 3 (Lacunes): Performance for lacune detection and segmentation was generally poor, with many teams overcalling lacunes. This task highlighted the significant challenges in accurately identifying and segmenting these lesions.
  • Variability: A notable finding was the large variability in performance across individual cases, even for the best-performing teams. This inconsistency could limit the direct clinical adoption of current methods without human oversight.
  • Computational Cost: There was no strong correlation between computational resources (memory, time) and overall performance, suggesting that efficiency does not necessarily equate to accuracy.

Limitations and Future Work

  • Lacune Detection: The poor performance on Task 3 indicates a need for further research into novel approaches for lacune detection and segmentation, potentially incorporating more sophisticated feature engineering or different network architectures.
  • Clinical Viability: The significant variability in performance across cases for all tasks suggests that current methods may require extensive manual review before clinical deployment. Future work could focus on improving robustness and consistency.
  • Dataset Bias: The data was sourced from population cohorts, and the generalizability to patient populations (e.g., those with dementia) needs further investigation.
  • Uncertainty Estimation: While explored for lacunes, systematic assessment of prediction confidence across all tasks could be valuable for practical implementation.
  • Weak Supervision: Most teams opted for dense labeling despite the availability of weak labels, suggesting that leveraging weak supervision for these challenging tasks might be an underexplored avenue.

Summary updated

Abstract

Highlights•12 teams participated to the Where is Valdo? Challenge.•For all tasks, the performance varied greatly across individual cases.•For Task 1-EPVS and Task 2-Microbleeds, best methods could be used in population studies.•Performance of similar deep learning architectures is greatly impacted by training choices•The performance was mostly driven by the ability to detect individual markers.
BibTeX
@article{sudre2024vascular,
  author = {Carole H. Sudre and Kimberlin M. H. van Wijnen and Florian Dubost and Hieab Adams and David Atkinson and Frederik Barkhof and Mahlet A. Birhanu and Esther E. Bron and Robin Camarasa and Nish Chaturvedi and Yuan Chen and Zihao Chen and Shuai Chen and Qi Dou and Tavia E. Evans and Ivan Ezhov and Haojun Gao and Marta Gironés-Sangüesa and Juan Domingo Gispert and Beatriz Gomez Anson and Alun D. Hughes and Mohammad Arfan Ikram and Silvia Ingala and Hans Rolf Jäger and Florian Kofler and Hugo J. Kuijf and Denis Kutnar and Minho Lee and Bo Li and Luigi Lorenzini and Bjoern H. Menze and José Luis Molinuevo and Yiwei Pan and Élodie Puybareau and Rafael Rehwald and Ruisheng Su and Pengcheng Shi and Lorna Smith and Therese Tillin and Guillaume Tochon and Hélène Urien and Bas H. M. van der Velden and Isabelle F. van der Velpen and Benedikt Wiestler and Frank J. Wolters and Pinar Yilmaz and Marius de Groot and Meike W. Vernooij and Marleen de Bruijne},
  title = {{VAscular Lesions Detection and segmentatiOn challenge at MICCAI 2021}},
  journal = {Medical Image Analysis},
  year = {2024},
  url = {https://openreview.net/forum?id=peQhLUbUlW},
}
Figure