Quality monitoring of federated Covid-19 lesion segmentation

Camila González, Christian Harder, Amin Ranem, Ricarda Fischbach, Isabel Kaltenborn, Armin Dadras, Andreas Bucher, Anirban Mukhopadhyay

TL;DR

Federated learning allows hospitals to train AI models collaboratively without sharing sensitive patient data, but it makes monitoring model performance a major challenge. This paper introduces a set of lightweight, "ground-truth-free" metrics that can be calculated locally to detect over 70% of poor-quality Covid-19 lesion segmentations, enabling a central monitoring dashboard that respects privacy.

What Problem Does This Paper Address?

Deep learning models for segmenting Covid-19 lesions in CT scans often suffer from "silent failures." When a model trained at one hospital encounters data from another hospital with different scanner settings or protocols (out-of-distribution data), its performance can drop significantly—often while the model remains highly confident in its incorrect predictions.

In a Federated Learning (FL) setup, this is particularly tricky. Because data stays local to preserve privacy, central researchers cannot simply inspect the outputs. Furthermore, manual review by expert radiologists for every single scan is logistically impossible. There is a dire need for an automated, privacy-preserving way to flag low-quality segmentations before they impact clinical decisions.

Methods

The authors propose a system where four lightweight features are extracted from the segmentation masks locally at each hospital. These features do not require "ground truth" (expert-labeled) masks:

  1. Connected Components: Measures how fragmented the segmentation is. Failed segmentations often appear as many tiny, disconnected blobs.
  2. Intensity Mode: Checks if the Hounsfield Unit (HU) values in the segmented area are physically plausible for a Covid-19 infection by fitting a Gaussian distribution over the largest component.
  3. Segmentation Smoothness: Compares the overlap (2D Dice) between consecutive slices. A sudden, jagged change in the mask between slices often indicates an error.
  4. Lesions within Lungs: Uses a separate, pre-trained lung segmentation model to ensure the predicted lesions actually fall inside the lung boundaries.
The proposed federated monitoring workflow where local quality scores are aggregated for a central dashboard.

These features are fed into simple machine learning models—such as Logistic Regression (LR) and Support Vector Machines (SVM)—to predict the segmentation quality (represented by a Dice score). Only the aggregated, privacy-preserving quality scores leave the hospital to be visualized on a central dashboard.

Datasets

The study utilized three distinct datasets to test the models under both in-distribution (ID) and out-of-distribution (OOD) conditions:

  • COVID-19 Lung Lesion Segmentation Challenge: 199 cases used for training and testing (ID).
  • In-house Dataset: 50 cases from a university hospital used for training and testing (ID).
  • MosMed: 50 cases used exclusively for testing to evaluate performance on OOD data.

The primary segmentation model being monitored was an nnU-Net.

Key Insights and Ideas

  • Plausibility as a Proxy for Quality: You don't always need to know the "right answer" to know an answer is wrong. By checking for physical and geometric plausibility (e.g., "is this lesion outside the lung?"), we can catch the majority of catastrophic failures.
  • Simple Models for Meta-Monitoring: Surprisingly, simple linear models like Logistic Regression outperformed complex Multi-Layer Perceptrons (MLPs) in detecting failures. This is likely because the linear models could be more easily tuned with class weights to handle the "needle in a haystack" problem of finding rare, low-quality segmentations in a sea of good ones.
  • The "Intensity Mode" Surprise: While intuitively useful, the ablation study showed that the "Intensity Mode" feature was the least useful on its own. However, the combination of all four features provided the best overall balance between catching errors (sensitivity) and avoiding false alarms (specificity).
Fig. 2. Exemplary subjects and slides for the four features used to assess seg- mentation quality.

Key Results

  • High Sensitivity on OOD Data: The Logistic Regression model detected 76% of failed segmentations (defined as Dice < 0.6) on the out-of-distribution MosMed dataset.
  • Reliable Monitoring: The models maintained a high specificity (over 0.80) across all datasets, meaning the system is unlikely to flood a central dashboard with false "substandard" alerts.
  • Statistical Significance: Using a p-value test (0.05), the authors confirmed that the sensitivity of their LR model is reliably above 0.28, even in the worst-case scenarios.

Limitations and Future Work

  • Retrospective Nature: The study was conducted on static, pre-existing datasets. A prospective study in a live, multi-hospital federated environment is needed to see how it handles real-time data drift.
  • Demographic Bias: The training data was heavily weighted toward ICU patients, which may not represent the broader population with milder Covid-19 cases.
  • Metric Limitations: The authors acknowledge that the Dice coefficient, while standard, is highly sensitive to lesion size. Future work could explore alternative metrics better suited for small, diffuse lesions.
  • Expanding the Catalog: The authors suggest that adding more lightweight metrics to the "catalogue" could further improve the detection of subtle failures that currently maintain a "plausible" shape.

Summary updated

Abstract

Federated Learning is the most promising way to train robust Deep Learning models for the segmentation of Covid-19-related findings in chest CTs. By learning in a decentralized fashion, heterogeneous data can be leveraged from a variety of sources and acquisition protocols whilst ensuring patient privacy. It is, however, crucial to continuously monitor the performance of the model. Yet when it comes to the segmentation of diffuse lung lesions, a quick visual inspection is not enough to assess the quality, and thorough monitoring of all network outputs by expert radiologists is not feasible. In this work, we present an array of lightweight metrics that can be calculated locally in each hospital and then aggregated for central monitoring of a federated system. Our linear model detects over 70% of low-quality segmentations on an out-of-distribution dataset and thus reliably signals a decline in model performance.
BibTeX
@inproceedings{gonzalez2022quality,
  author = {Camila González and Christian Harder and Amin Ranem and Ricarda Fischbach and Isabel Kaltenborn and Armin Dadras and Andreas Bucher and Anirban Mukhopadhyay},
  title = {{Quality monitoring of federated Covid-19 lesion segmentation}},
  booktitle = {Bildverarbeitung für die Medizin},
  year = {2022},
  url = {https://openreview.net/forum?id=WhVqVUBNeW},
}
Figure