Improving assessment of lesions in longitudinal CT scans: a bi-institutional reader study on an AI-assisted registration and volumetric segmentation workflow

Alessa Hering, Max Westphal, Annika Gerken, Haidara Almansour, Michael Maurer, Benjamin P. Geisler, Temke Kohlbrandt, Thomas Eigentler, Teresa Amaral, Nikolas Lessmann, Sergios Gatidis, Horst K. Hahn, Konstantin Nikolaou, Ahmed E. Othman, Jan Hendrik Moltz, Felix PeisenInt. J. Comput. Assist. Radiol. Surg 2024 · DOI

TL;DR

This study introduces and validates an AI-assisted workflow for segmenting lymph node and soft tissue metastases in longitudinal CT scans. The AI approach significantly speeds up the assessment process, maintains comparable segmentation quality to manual methods, and crucially, reduces variability between different readers, paving the way for more standardized and efficient cancer follow-up.

What Problem Does This Paper Address?

Accurately tracking the growth or shrinkage of metastatic lesions in CT scans is vital for evaluating cancer treatment effectiveness. Traditional methods often rely on measuring lesion diameters (e.g., RECIST criteria), which can be time-consuming, prone to errors, and may not fully capture changes in tumor burden due to heterogeneity. Volumetric segmentation offers a more comprehensive assessment, but manual segmentation is laborious and subject to significant inter-reader variability. This makes it challenging to compare results across different institutions or even between different radiologists within the same institution. The paper addresses the need for a faster, more consistent, and reliable method for volumetric lesion segmentation in longitudinal CT scans, specifically for lymph node and soft tissue metastases, which are common in melanoma patients and notoriously difficult to segment due to their varied locations and shapes.

Methods

The core of this work is a bi-institutional reader study evaluating a recently developed AI-assisted workflow. This workflow combines automated registration of baseline and follow-up CT scans with a U-Net-based deep learning model for volumetric segmentation of lesions. The AI pipeline first registers the scans, then propagates a region of interest (ROI) from the baseline scan to the follow-up scan to guide the segmentation. The AI then automatically segments lesions within this guided region.

The study design involved three radiologists from two institutions independently performing both manual and AI-assisted segmentations on a testing dataset. For the AI-assisted workflow, radiologists had the option to accept the AI's segmentation, make manual corrections, or reject it entirely and perform manual segmentation. This "human-in-the-loop" approach allows for validation of the AI's performance and assessment of user interaction. The study specifically focused on the user interaction during the follow-up scan segmentation.

Datasets

The study utilized a dataset derived from the Central Malignant Melanoma Registry in Germany.

  • Training and Validation Dataset: This dataset comprised 4308 lesions (2603 soft tissue, 1705 lymph node) from 214 patients. This was used to develop and train the AI pipeline.
  • Testing Dataset: This dataset consisted of 126 soft tissue and 135 lymph node lesions from 58 patients who had undergone their first follow-up CT after therapy initiation. These patients and lesions were not part of the training/validation sets. Lesions in the testing set were stratified by diameter (<10 mm, 10-20 mm, >20 mm).

Key Insights and Ideas

  • AI as a Time Saver, Not a Replacement: The AI workflow significantly reduces the time radiologists spend on segmentation, but it doesn't eliminate human oversight. The "human-in-the-loop" approach allows for corrections and validation, ensuring accuracy while leveraging AI's speed.
  • Bridging the Gap for Small Lesions: The AI-assisted approach shows a notable improvement in reducing inter-reader variability, particularly for small lesions (<10 mm), which are often the most challenging to segment consistently. This is a critical finding for early detection and monitoring.
  • Standardization Potential: By reducing inter-reader variability, the AI workflow has the potential to standardize lesion assessment across different readers and institutions, making radiomic analyses and treatment response evaluations more reliable and comparable.

Key Results

  • Time Savings: The AI-assisted workflow reduced user interaction time by 33% (average 222 seconds per patient) compared to manual segmentation (average 336 seconds per patient).
  • Segmentation Quality: The AI-assisted segmentation achieved comparable quality to manual segmentation, with median Dice scores of 0.80-0.84 for AI-assisted versus 0.81-0.82 for manual segmentation, when compared to a reference standard. The quality was particularly promising for small lesions.
  • Reduced Inter-Reader Variability: The AI-assisted workflow significantly decreased inter-reader variability. Median Dice scores for inter-reader agreement were 0.85-1.0 for AI-assisted versus 0.80-0.82 for manual segmentation. The overall intraclass correlation coefficient (ICC) improved from 0.80 (manual) to 0.84 (AI-assisted).

Limitations and Future Work

  • Potential Bias in Training Data: The authors acknowledge a potential bias because the reference reader who annotated the training data also participated in the reader study. This might have influenced the AI to adopt a specific annotation style.
  • Baseline Segmentation Not Evaluated: The study did not evaluate the AI's performance on baseline segmentations, which could have provided additional insights.
  • Future Directions: The authors suggest that further training of radiologists with the AI-assisted workflow could lead to even greater improvements in outcomes. They also highlight the need for AI-assisted one-click segmentation or fully automatic detection for baseline scans to further streamline the process. The study's findings are generalized to new patients and readers with similar characteristics, but including more patients and readers in future studies would further validate these results.

Summary updated

BibTeX
@article{hering2024improving,
  author = {Alessa Hering and Max Westphal and Annika Gerken and Haidara Almansour and Michael Maurer and Benjamin P. Geisler and Temke Kohlbrandt and Thomas Eigentler and Teresa Amaral and Nikolas Lessmann and Sergios Gatidis and Horst K. Hahn and Konstantin Nikolaou and Ahmed E. Othman and Jan Hendrik Moltz and Felix Peisen},
  title = {{Improving assessment of lesions in longitudinal CT scans: a bi-institutional reader study on an AI-assisted registration and volumetric segmentation workflow}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2024},
  doi = {10.1007/s11548-024-03181-4},
}
Figure