Evaluation of current algorithms for segmentation of scar tissue from late Gadolinium enhancement cardiovascular magnetic resonance of the left atrium: an open-access grand challenge

Rashed Karim, R. James Housden, Mayuragoban Balasubramaniam, Zhong Chen, Daniel J. Perry, Ayesha Uddin, Yosra Al-Beyatti, Ebrahim Palkhi, Prince Acheampong, Samantha Obom, Anja Hennemuth, Yingli Lu, Wenjia Bai, Wenzhe Shi, Yi Gao, Heinz-Otto Peitgen, Perry Radau, Reza Razavi, Allen Tannenbaum, Daniel Rueckert, Joshua Cates, Tobias Schaeffter, Dana C. Peters, Rob MacLeod, Kawal RhodeJournal of Cardiovascular Magnetic Resonance 2013 · DOI

Abstract

The study concludes that currently no algorithm is deemed clearly better than others. There is scope for further algorithmic developments in LA fibrosis and scar quantification from LGE CMR images. Benchmarking of future scar segmentation algorithms is thus important. The proposed benchmarking framework is made available as open-source and new participants can evaluate their algorithms via a web-based interface.

Introduction: The Challenge of Atrial Scarring

Atrial Fibrillation (AF) is a common heart rhythm disorder often treated with radiofrequency catheter ablation (RFCA)—a procedure that intentionally scars heart tissue to block irregular electrical signals. To plan these treatments and evaluate their success, clinicians need to see where existing fibrosis (pre-ablation) and new scars (post-ablation) are located.

However, segmenting the left atrium (LA) is notoriously difficult. The atrial wall is incredibly thin (~2.5mm), and Late Gadolinium Enhancement (LGE) CMR images are often plagued by low signal-to-noise ratios, motion blur, and "fake" enhancements from nearby structures like the aorta.

Representative LGE CMR slices showing the left atrium (LA), right pulmonary vein (RPV), and aorta (AO), highlighting the challenge of low contrast and thin tissue walls.

The cDEMRIS Grand Challenge

To address the lack of a common "yardstick" for the field, the authors organized the cDEMRIS Grand Challenge at ISBI 2012. Seven international institutions were invited to test eight distinct algorithms on a shared dataset of 60 LGE CMR image datasets (30 pre-ablation and 30 post-ablation).

The algorithms ranged in complexity from simple thresholding-based intensity cut-offs to advanced energy minimization techniques like Graph-cuts and Markov Random Fields (MRF). To evaluate these, the authors created a "pseudo-ground truth" by merging manual segmentations from three expert observers using the STAPLE algorithm.

Key Insights: Anatomy and Thresholding

A major finding was the "Pre-vs-Post" gap: segmenting dense post-ablation scars is significantly easier than identifying diffuse pre-ablation fibrosis. Furthermore, traditional clinical standards like "n-Standard Deviations (SD)" or "Full-Width-at-Half-Maximum (FWHM)" performed poorly because they are too rigid to handle variability in image quality across different scanners.

The most successful algorithms, such as the Yale method, often relied on anatomical knowledge to filter out false positives. By strictly defining the "search space" through manual or semi-automatic delineation of the atrial wall, they avoided false positives from nearby enhancing structures like the mitral valve or aorta.

The Yale method's threshold criterion. (a) Threshold selection for pre- and post-ablation images. (b) Exclusion of artifacts from the mitral valve (MV), aorta (AAo, DAo), and right atrial (RA) wall.

Results and Benchmarking

Performance was measured using the Dice Similarity Coefficient (DSC), where 100 represents a perfect match with the experts:

  • Post-Ablation (Scar): Algorithms performed relatively well, with median Dice scores ranging from 72 to 85. The Mevis (85) and Yale (84) methods were the top performers.
  • Pre-Ablation (Fibrosis): Performance dropped significantly, with median Dice scores between 17 and 48. Yale (48) and Utah B (45) led this category.

While modern algorithms significantly outperformed clinical baselines, no single "silver bullet" emerged. An algorithm might excel in volume accuracy but fail in surface overlap, highlighting the need for multi-metric evaluation in medical imaging.

Conclusion

The field has moved past simple thresholding, yet a fully automated, robust solution for the thin atrial wall remains elusive. The primary limitation of top-performing methods is their reliance on human intervention for wall delineation. The authors have made their benchmarking framework open-source, inviting the community to continue the challenge and push toward more reliable clinical tools.

Summary updated

Abstract

The study concludes that currently no algorithm is deemed clearly better than others. There is scope for further algorithmic developments in LA fibrosis and scar quantification from LGE CMR images. Benchmarking of future scar segmentation algorithms is thus important. The proposed benchmarking framework is made available as open-source and new participants can evaluate their algorithms via a web-based interface.
BibTeX
@article{karim2013evaluation,
  author = {Rashed Karim and R. James Housden and Mayuragoban Balasubramaniam and Zhong Chen and Daniel J. Perry and Ayesha Uddin and Yosra Al-Beyatti and Ebrahim Palkhi and Prince Acheampong and Samantha Obom and Anja Hennemuth and Yingli Lu and Wenjia Bai and Wenzhe Shi and Yi Gao and Heinz-Otto Peitgen and Perry Radau and Reza Razavi and Allen Tannenbaum and Daniel Rueckert and Joshua Cates and Tobias Schaeffter and Dana C. Peters and Rob MacLeod and Kawal Rhode},
  title = {{Evaluation of current algorithms for segmentation of scar tissue from late Gadolinium enhancement cardiovascular magnetic resonance of the left atrium: an open-access grand challenge}},
  journal = {Journal of Cardiovascular Magnetic Resonance},
  year = {2013},
  doi = {10.1186/1532-429x-15-105},
}
Figure