TL;DR
This paper introduces the CROWN challenge, a benchmark for automated analysis of the Circle of Willis (CoW) from TOF-MRA images. It highlights the current limitations of manual assessment and evaluates various automated techniques for classifying CoW anatomical variants and quantifying artery diameters and bifurcation angles, finding that while progress has been made, current methods still fall short of clinical standards.
What Problem Does This Paper Address?
Intracranial aneurysms (IAs), which often form on the Circle of Willis (CoW), pose a significant health risk. Early detection and risk assessment are crucial, but manually analyzing the complex anatomy of the CoW from TOF-MRA images is time-consuming and prone to errors. Existing studies on imaging markers for IA development lack robust evidence, partly due to the difficulty in consistently assessing CoW morphology. This paper addresses the need for automated methods to accurately classify CoW variants and quantify its arterial structures, aiming to establish a benchmark for evaluating and improving these techniques.
Methods
The CROWN challenge was designed with two main tasks:
- Task 1: Classification of CoW Variants: Participants were asked to classify anatomical variants of the CoW based on the Lippert and Pabst system, which categorizes both the anterior and posterior parts of the CoW.
- Task 2: Quantification of Artery Diameters and Bifurcation Angles: This task involved a regression problem to automatically measure the diameters of 15 major CoW arteries and the angles of 10 artery bifurcations.
The challenge provided 300 TOF-MRA scans for training and 300 for testing, all manually annotated. Submissions were evaluated using balanced accuracy (BAC) for classification and Mean Absolute Error (MAE) and Pearson correlation coefficient for quantification.
Datasets
- Training Data: 300 TOF-MRA scans with manual annotations for CoW variants, artery diameters, and bifurcation angles.
- Test Data: 300 TOF-MRA scans with manual annotations for evaluation.
- External Data: 30 TOF-MRA scans from the TopCoW'23 challenge were used to assess the generalizability of the submitted solutions.
The CROWN challenge itself serves as a newly introduced benchmark for automated CoW analysis, building upon existing classification schemes like Lippert's system.
Key Insights and Ideas
- Manual Annotation is Challenging: The study highlights significant intra- and inter-annotator variability, particularly in classifying subtle variations like hypoplasia, underscoring the need for objective automated methods.
- Diversity in Approaches: The leading teams employed a range of techniques, including deep learning (CNNs, GNNs) and atlas-based registration, suggesting that no single method is universally superior.
- No End-to-End Solutions: Despite encouragement, no participating team submitted a fully end-to-end machine learning solution, indicating the complexity of directly mapping raw images to desired outputs.
- Generalizability is Key: Performance varied significantly when tested on external data, emphasizing the importance of robust models that can generalize across different datasets and acquisition parameters.
- Minority Classes are Difficult: Both classification and quantification tasks struggled with minority classes or smaller arteries, where performance metrics were notably lower.
Key Results
- Classification (Task 1): The best performing team achieved an average balanced accuracy of 0.26 for anterior and 0.27 for posterior CoW classification. These scores are considered low, indicating the difficulty of the task.
- Quantification (Task 2): The top team achieved an average MAE of 0.44 mm for artery diameters and 28 degrees for bifurcation angles. While progress was made, these results still fall short of clinical standards and inter-annotator agreement.
- Generalizability: Performance on external data varied, with some methods showing better robustness than others. Team Snaillab's quantification method demonstrated consistent performance on external data.
Limitations and Future Work
- Limited Participation: The challenge had a relatively small number of final submissions, which may limit the robustness of the conclusions.
- Rare Variants Excluded: Rare CoW variants were excluded from the dataset to ensure sufficient training instances, which limits the evaluation of methods on these less common anatomies.
- Semi-Automatic Annotations: The reference standard for quantification was derived from semi-automatic methods, which may introduce some level of error compared to fully manual annotations.
- Future Directions: The authors suggest exploring alternative approaches like vision transformers, which might better capture global features for classification. They also emphasize the need for more publicly available, annotated datasets to advance research in this area. The challenge remains open for future submissions to foster further development.
Summary updated