Development and external evaluation of a self-learning auto-segmentation model for Colorectal Cancer Liver Metastases Assessment (COALA)

Jacqueline I. Bereska, Michiel Zeeuw, Lodewijk J. Wagenaar, Håvard Bjørke Jenssen, Nina J. Wesdorp, D. van der Meulen, Leonard Bereska, Efstratios Gavves, Boris V. Janssen, Marc G. Besselink, Henk A. Marquering, Jan‐Hein T. M. van Waesberghe, Davit L. Aghayan, Egidijus Pelanis, Janneke van den Bergh, I. Nota, Shira Moos, Gunter Kemmerich, Trygve Syversveen, Finn Kristian Kolrud, Joost Huiskens, Rutger‐Jan Swijnenburg, Cornelis J.a. Punt, Jaap Stoker, Bjørn Edwin, Åsmund Avdem Fretland, Geert Kazemier, Inez M. Verpalen, Giovanni Marchegiani, Daniela Bassi, Riccardo Boetto, Mattia Ballo, Riccardo Carandina, Filippo Crimì, Matteo Fassan, Arantza Fariña Sarasqueta, Caroline S. Verbeke, Knut Jørgen Labori, Mirko D’Onofrio, Giulia Zamboni, Riccardo di Robertis, Claudio Luchini, Alberto Balduzzi, Giuseppe Malleo, Roberto Salvia, Christopher L. Wolfgang, Ammar A. Javed, Kathryn Colborn, Marco Del Chiaro, Jeffrey B. Kaplan, Toshimasa Clark, Thomas F. Stoop, Ioana Gabriela Lupescu, Mugur Grasu, Cristian Anghel, Mihai Dan Pomohaci, Philipp Mayer, Benedict Kinny‐Köster, Martin Loos, Christoph Michalski, Martinus J. van Amerongen, Marinde J. G. Bond, T. Chapelle, Ronald M. van Dam, Marc Engelbrecht, Michael F. Gerhards, Dirk J. Grünhagen, Thomas M. van Gulik, John J. Hermans, Koert P. de Jong, Joost M. Klaase, Niels F. M. Kok, Wouter K. G. Leclercq, Mike S.l. Liem, Krijn P. van Lienden, I. Quintus Molenaar, Gijs A Patijn, Arjen M. Rijken, Theo Ruers, Cornelis Verhoef, Johannes H.w. de WiltInsights 2024 · DOI

TL;DR

Researchers have developed COALA, a self-learning AI model that automates the segmentation of colorectal cancer liver metastases (CRLM) on CT scans. By using a "teacher-student" training approach to leverage a large dataset, the model achieves near-human accuracy, paving the way for Total Tumor Volume (TTV) to finally become a practical clinical biomarker for cancer prognosis.

What Problem Does This Paper Address?

In patients with colorectal cancer, the liver is a frequent site for metastasis. Currently, doctors evaluate treatment response using RECIST 1.1, which involves measuring the diameter of a few "target" lesions. However, this is a 1D approximation of a 3D problem. Total Tumor Volume (TTV)—the sum of all metastatic volumes—is a much more powerful predictor of survival and recurrence.

The bottleneck? Manual segmentation is a nightmare. It is labor-intensive, time-consuming, and highly subjective, leading to significant variation between different radiologists. Because of this "manual tax," TTV hasn't been adopted in routine clinical practice. The authors of this paper sought to break this bottleneck by building a robust, automated segmentation tool that works across different hospitals and scanners.

Methods

The researchers employed a self-learning (teacher-student) framework to maximize the utility of their data:

  1. The Teacher: A model was first trained on a high-quality "gold standard" subset of 99 CT scans, where three radiologists provided manual segmentations that were merged using the STAPLE algorithm (a method to find the most likely consensus between experts).
  2. The Student: This teacher model then "labeled" an additional 663 unsegmented CT scans. The student model was then trained on this massive combined dataset (the original 99 manual scans + the 663 "pseudo-labeled" scans).
  3. The Architecture: They used nnU-Net, a self-configuring 3D U-Net cascade that is currently the state-of-the-art for medical image segmentation.
  4. Contextual Awareness: To help the model distinguish tumors from other structures, they included segmentations of 12 surrounding organs (like the spleen, kidneys, and stomach) as contextual information.

Datasets

The study utilized two primary datasets from major clinical trials:

  • INTERNAL (CAIRO5 trial): 783 portal venous phase CT scans from 373 patients. This was used for training and internal testing.
  • EXTERNAL (Oslo-COMET trial): 50 CT scans from 35 patients. This served as a completely independent "blind" test to see if the model could generalize to a different hospital's data and patient population.
  • Benchmarks: The authors have made their internal test set and the fully trained model publicly available on GitHub to encourage further research.

Key Insights and Ideas

  • Self-Learning Scales Expertise: The "aha moment" here is that the student model can actually outperform the teacher. By training on a much larger (albeit noisier) dataset generated by the teacher, the student learns a more robust representation of the anatomy, reducing the impact of individual observer biases.
  • Consensus is King: Rather than training on one person's opinion, using the STAPLE algorithm to combine three radiologists' inputs created a more reliable "ground truth."
  • Generalization is the Real Test: Many AI models fail when they leave the hospital where they were born. COALA maintained high performance on the Oslo-COMET dataset, proving it isn't just "memorizing" the specific quirks of the Amsterdam UMC scanners.
  • Surrounding Anatomy Matters: By teaching the model what a kidney or a stomach looks like, the researchers reduced "false positives" where the AI might otherwise mistake a nearby organ for a tumor.

Key Results

The COALA model demonstrated "excellent" agreement with expert radiologists:

  • DICE Score: The model achieved a mean DICE score (an overlap metric where 1.0 is perfect) of 0.85 on the internal test set and 0.83 on the external test set. For context, previous similar models often struggled to break 0.70 in external evaluations.
  • Volumetric Accuracy: The Intraclass Correlation Coefficient (ICC) for TTV was 0.97 for both test sets. In the world of medical imaging, an ICC above 0.90 is considered "near-perfect" agreement.
  • Robustness: The model performed consistently well regardless of whether the scan was taken before or after neoadjuvant therapy (chemotherapy), which often changes the appearance and density of tumors.

Limitations and Future Work

While promising, the authors note a few hurdles:

  • Retrospective Bias: The study used historical data. A prospective study (using the model in real-time clinical workflows) is needed to prove it actually improves patient outcomes.
  • Ground Truth "Cleaning": The radiologists created the ground truth by correcting an initial AI segmentation rather than drawing from scratch. This might slightly inflate the agreement scores.
  • Dataset Diversity: While the external test was successful, it had fewer metastases per patient than the training set. Testing on even more diverse "real-world" datasets from non-academic centers would be a valuable next step.

Ultimately, COALA represents a significant step toward making 3D tumor volume a standard part of cancer care, moving us closer to truly personalized oncology.

Summary updated

Abstract

Developed colorectal liver metastases segmentation model to facilitate total tumor volume assessment. Model achieved high performance on internal and external test sets. Model can improve prognostic stratification and treatment planning for colorectal liver metastases.
BibTeX
@inproceedings{bereska2024development,
  author = {Jacqueline I. Bereska and Michiel Zeeuw and Lodewijk J. Wagenaar and Håvard Bjørke Jenssen and Nina J. Wesdorp and D. van der Meulen and Leonard Bereska and Efstratios Gavves and Boris V. Janssen and Marc G. Besselink and Henk A. Marquering and Jan‐Hein T. M. van Waesberghe and Davit L. Aghayan and Egidijus Pelanis and Janneke van den Bergh and I. Nota and Shira Moos and Gunter Kemmerich and Trygve Syversveen and Finn Kristian Kolrud and Joost Huiskens and Rutger‐Jan Swijnenburg and Cornelis J.A. Punt and Jaap Stoker and Bjørn Edwin and Åsmund Avdem Fretland and Geert Kazemier and Inez M. Verpalen and Giovanni Marchegiani and Daniela Bassi and Riccardo Boetto and Mattia Ballo and Riccardo Carandina and Filippo Crimì and Matteo Fassan and Arantza Fariña Sarasqueta and Caroline S. Verbeke and Knut Jørgen Labori and Åsmund Avdem Fretland and Mirko D’Onofrio and Giulia Zamboni and Riccardo di Robertis and Claudio Luchini and Alberto Balduzzi and Giuseppe Malleo and Roberto Salvia and Christopher L. Wolfgang and Ammar A. Javed and Kathryn Colborn and Marco Del Chiaro and Jeffrey B. Kaplan and Toshimasa Clark and Thomas F. Stoop and Ioana Gabriela Lupescu and Mugur Grasu and Cristian Anghel and Mihai Dan Pomohaci and Philipp Mayer and Benedict Kinny‐Köster and Martin Loos and Christoph Michalski and Martinus J. van Amerongen and Marinde J. G. Bond and T. Chapelle and Ronald M. van Dam and Marc Engelbrecht and Michael F. Gerhards and Dirk J. Grünhagen and Thomas M. van Gulik and John J. Hermans and Koert P. de Jong and Joost M. Klaase and Niels F. M. Kok and Wouter K. G. Leclercq and Mike S.L. Liem and Krijn P. van Lienden and I. Quintus Molenaar and Gijs A Patijn and Arjen M. Rijken and Theo Ruers and Cornelis Verhoef and Johannes H.W. de Wilt and Johannes H. W. de Wilt},
  title = {{Development and external evaluation of a self-learning auto-segmentation model for Colorectal Cancer Liver Metastases Assessment (COALA)}},
  booktitle = {Insights},
  year = {2024},
  doi = {10.1186/s13244-024-01820-7},
}
Figure