Quality Monitoring of Federated Covid-19 Lesion Segmentation

Camila González, Christian L. Harder, Amin Ranem, Ricarda Fischbach, Isabel J. Kaltenborn, Armin Dadras, Andreas M. Bucher, Anirban Mukhopadhyay

TL;DR

This paper tackles the challenge of ensuring the quality of AI-driven COVID-19 lesion segmentation in federated learning settings. They propose a set of lightweight, locally computable metrics that can be aggregated for central monitoring, enabling the detection of low-quality segmentations without requiring extensive manual review. Their linear model demonstrates promising performance in identifying these faulty segmentations, crucial for reliable clinical deployment.

What Problem Does This Paper Address?

Deep learning models have shown great promise in segmenting COVID-19 related lesions in CT scans, which is vital for quantifying disease severity and aiding clinical decisions. Federated learning offers a way to train these models on diverse datasets from multiple institutions while preserving patient privacy. However, a significant challenge arises from the silent failure of these models: they can produce low-quality segmentations with high confidence, especially when faced with variations in scanner models and acquisition protocols across different hospitals. Manually reviewing every segmentation output is not feasible due to the sheer volume and the need for expert radiologists. This paper addresses the critical need for an automated, efficient, and reliable quality control mechanism for these federated segmentation systems, particularly for diffuse lung lesions which are hard to visually inspect.

Methods

The authors propose a system for automated quality assurance of segmentation masks that relies on lightweight metrics computable locally at each participating hospital. These metrics are designed to predict the quality of segmentation (specifically, the Dice coefficient) without requiring ground truth annotations. The four key features they developed are:

  • Connected Components: Analyzing the number of disconnected components in a segmentation mask, hypothesizing that failed segmentations might be more fragmented.
  • Intensity Mode: Examining the intensity values within the CT scan, using a Gaussian distribution fit to identify tissue unlikely to be infected.
  • Segmentation Smoothness: Measuring the overlap between consecutive slices to assess how consistent the segmentation is across slices.
  • Lesions within Lungs: Utilizing a pre-trained lung segmentation model to determine the proportion of segmented lesion tissue that falls within the lung boundaries.

These features are then used to train two types of models: regression models (Ridge Regression, Support Vector Regression, Multi-Layer Perceptron) to directly predict the Dice coefficient, and classification models (Support Vector Machine, Logistic Regression) to categorize segmentation quality into bins. The focus is on detecting "failed" segmentations (defined as Dice < 0.6).

Datasets

The study utilized a combination of datasets:

  • Challenge Dataset: This refers to the COVID-19 Lung Lesion Segmentation Challenge dataset, used for training and evaluation.
  • In-house Dataset: Data from the authors' institution, also used for training and evaluation.
  • MosMed Dataset: A dataset of chest CT scans with COVID-19 related findings, used as an out-of-distribution (OOD) dataset for evaluation.

The in-distribution (ID) datasets (Challenge and in-house) were split into training and testing sets. The MosMed dataset served as a crucial benchmark for evaluating the models' generalization capabilities on unseen data distributions.

Key Insights and Ideas

  • Federated Learning Needs Quality Control: The paper strongly emphasizes that while federated learning is powerful for data privacy and diversity, it's incomplete without robust quality monitoring.
  • Lightweight Metrics are Key: The core idea is that complex, computationally expensive metrics or those requiring ground truth are impractical for federated settings. The proposed features are designed for efficiency and local computation.
  • Beyond Visual Inspection: For diffuse lung lesions, visual inspection is insufficient. Automated metrics are essential for detecting subtle quality degradations.
  • Classifiers Outperform Regressors for Detection: The study found that classification models (LR and SVM) were more effective at detecting failed segmentations compared to regression models, likely due to better handling of imbalanced classes.
  • Feature Importance: While all features contributed, "Intensity Mode" was found to be the least impactful individually, suggesting that a combination of spatial and intensity-based features is optimal.

Key Results

The study evaluated several models for their ability to detect failed segmentations (sensitivity), identify correct quality intervals (specificity), and predict segmentation quality directly (Mean Absolute Error).

  • Sensitivity: The Logistic Regression (LR) model demonstrated the highest sensitivity, detecting over 60% of failed segmentations on the larger Challenge and MosMed datasets. While it missed one failed segmentation on the in-house dataset, its performance on OOD data was notable.
  • Specificity: All models achieved high specificity (over 0.8) across datasets, meaning they were good at correctly identifying high-quality segmentations.
  • MAE: Regression models achieved lower Mean Absolute Error, but this was attributed to overfitting on good-quality segmentations, potentially hindering their sensitivity for detecting failures.
  • LR Model Performance: The LR model, using the four proposed features, achieved a sensitivity range of 0.22 to 1.0 with 95% confidence intervals on the tested datasets.

Limitations and Future Work

The authors acknowledge several limitations:

  • Retrospective Study: The evaluation was performed retrospectively on a statically trained nnU-Net. A prospective study on a truly federated system with real-time deployment would better emulate real-world scenarios.
  • Dataset Bias: The CT data was acquired from ICU patients, which may not be representative of the general COVID-19 population. This could also mean that a metric other than Dice might be more suitable for a broader population, as Dice's expressiveness depends on lesion size.
  • Inter-Annotator Variability: Different expert groups annotated the datasets, potentially leading to variations in findings and impacting performance comparisons.

Future work suggested includes:

  • Expanding the catalogue of quality metrics.
  • Assessing the effectiveness of the proposed methods in a model deployed across multiple hospitals in a federated system.
  • Investigating metrics beyond Dice for general populations.

Summary updated

Abstract

Federated Learning is the most promising way to train robust Deep Learning models for the segmentation of Covid-19-related findings in chest CTs. By learning in a decentralized fashion, heterogeneous data can be leveraged from a variety of sources and acquisition protocols whilst ensuring patient privacy. It is, however, crucial to continuously monitor the performance of the model. Yet when it comes to the segmentation of diffuse lung lesions, a quick visual inspection is not enough to assess the quality, and thorough monitoring of all network outputs by expert radiologists is not feasible. In this work, we present an array of lightweight metrics that can be calculated locally in each hospital and then aggregated for central monitoring of a federated system. Our linear model detects over 70% of low-quality segmentations on an out-of-distribution dataset and thus reliably signals a decline in model performance.
BibTeX
@inproceedings{gonzalez2022quality,
  author = {Camila González and Christian L. Harder and Amin Ranem and Ricarda Fischbach and Isabel J. Kaltenborn and Armin Dadras and Andreas M. Bucher and Anirban Mukhopadhyay},
  title = {{Quality Monitoring of Federated Covid-19 Lesion Segmentation}},
  booktitle = {Bildverarbeitung für die Medizin},
  year = {2022},
  url = {https://openreview.net/forum?id=WhVqVUBNeW},
}
Figure