COVID-BLUeS - a Prospective Study on the Value of AI in Lung Ultrasound Analysis

Nina Wiedemann, Dianne de Korte-de Boer, Matthias Richter, Sjors van de Weijer, Charlotte Buhre, Franz A. M. Eggert, Sophie Aarnoudse, Lotte Grevendonk, Steffen Röber, Carlijn M. E. Remie, Wolfgang Buhre, Ronald Henry, Jannis Born

Introduction

During the COVID-19 pandemic, Lung Ultrasound (LUS) emerged as a vital diagnostic tool. As a portable, non-irradiating, and cost-effective alternative to Chest X-rays and CT scans, it provided a safe method for assessing pulmonary involvement. The machine learning community responded rapidly, producing numerous AI models for automated LUS analysis, with some studies reporting diagnostic accuracies exceeding 98%.

However, these high performance metrics often stemmed from models trained on heterogeneous, publicly scraped datasets. These datasets frequently suffer from significant biases, such as the inclusion of educational videos depicting only textbook pathologies or flawed experimental designs that split frames from the same video into both training and testing sets. This study introduces the COVID-BLUeS dataset to address a critical question: how do these AI models actually perform when deployed in a realistic, prospective clinical environment?

The COVID-BLUeS Dataset

To provide a rigorous benchmark, the authors collected and open-sourced the COVID-BLUeS dataset. Unlike previous collections, this dataset was gathered prospectively at Maastricht UMC+ and includes 371 videos from 63 patients (33 COVID-positive, 30 negative).

The dataset is characterized by high standardization:

  • Protocol: All scans followed the BLUE protocol, targeting six specific thoracic locations.
  • Hardware: Data was acquired using a single ultrasound device type to minimize hardware-induced variance.
  • Metadata: Each entry is paired with rich clinical metadata, including Complete Blood Counts (CBC), symptoms, and definitive PCR ground truth.

Methodology

The researchers evaluated three primary paradigms of LUS AI to determine their clinical utility:

  1. Image Classification: Utilizing POCOVID-Net (a VGG-16 based CNN) for direct frame-level classification.
  2. Pattern Localization: Using the CovEcho model (YOLOv5) to identify specific LUS landmarks, such as A-lines and B-lines.
  3. Semantic Segmentation: Employing the ICLUS model ensemble (U-Net and DeepLabV3+) to generate pixel-wise masks of lung pathologies.

For localization and segmentation, a two-stage approach was used: features such as the count or total area of detected patterns were extracted and fed into traditional classifiers (Logistic Regression, Random Forest, SVM) to predict PCR results. Additionally, the authors pioneered a multi-modal approach, using a late-fusion neural network to combine image embeddings with tabular clinical variables like blood markers and symptoms.

Results and Discussion

The Reality of AI Performance

The study revealed a significant performance gap between laboratory results and clinical reality. The POCOVID-Net model, which reported 87.8% accuracy on scraped data, achieved only 65% accuracy in the prospective COVID-BLUeS setting. This "99% accuracy illusion" suggests that many existing models have overfit to the biases of public datasets rather than learning generalizable clinical features.

Human vs. Machine

Interestingly, human experts grading LUS severity also struggled, peaking at 65% accuracy when attempting to distinguish COVID-positive from negative patients based solely on ultrasound. The best-performing AI models significantly outperformed this human baseline, suggesting that while the task is difficult, automated systems can extract more nuanced information than the naked eye.

The Superiority of Segmentation and Clinical Data

The study found that two-stage models (segmentation followed by classification) outperformed end-to-end CNN classifiers. The ICLUS segmentation model achieved 79% accuracy, likely because it aggregates structural data across entire videos, whereas standard classifiers are often confused by uninformative frames.

However, the most striking result came from non-imaging data. A Random Forest trained exclusively on clinical variables and blood work (specifically LDH and CRP levels) achieved 84% accuracy. When LUS images were combined with these clinical variables in a multi-modal architecture, the accuracy reached 83%, confirming that clinical context is as important as imaging in diagnostic triage.

Conclusion

The COVID-BLUeS study highlights the necessity of prospective validation in medical AI. It demonstrates that while pure image classification is prone to bias, multi-modal models and segmentation-based approaches offer a robust path forward.

Future research should move beyond frame-by-frame analysis toward spatio-temporal models that account for the video-level dynamics of ultrasound. Furthermore, the authors advocate for the routine integration of spatial metadata (knowing which lung area is being scanned) and clinical variables to transform AI from a research curiosity into a genuine clinical decision support tool.

Summary updated

BibTeX
@article{wiedemann2025covid,
  author = {Nina Wiedemann and Dianne de Korte-de Boer and Matthias Richter and Sjors van de Weijer and Charlotte Buhre and Franz A. M. Eggert and Sophie Aarnoudse and Lotte Grevendonk and Steffen Röber and Carlijn M. E. Remie and Wolfgang Buhre and Ronald Henry and Jannis Born},
  title = {{COVID-BLUeS - a Prospective Study on the Value of AI in Lung Ultrasound Analysis}},
  journal = {IEEE J. Biomed. Health Informatics},
  year = {2025},
  doi = {10.1109/jbhi.2025.3543686},
}

Loading related content...

Figure