Introduction
Distinguishing COVID-19 from other types of viral or bacterial pneumonia on Computed Tomography (CT) scans is a significant clinical challenge. While certain visual patterns are typical of COVID-19, they are often not pathognomonic, leading to diagnostic ambiguity and high inter-rater variability among clinicians. This study investigates whether a "human-in-the-loop" approach—integrating standardized structured reports, quantitative radiomic features, and machine learning (ML) algorithms—can enhance diagnostic accuracy and consistency.
Methods
The researchers implemented a multi-center pipeline across three Brazilian academic hospitals, utilizing a dataset of 128 subjects (64 COVID-19 cases and 64 pre-pandemic pneumonia controls). The data was partitioned into a training cohort (n=73) and an independent validation cohort (n=55).
The methodology employed two primary data streams:
- Structured Reports: Radiologists completed a 10-item binary checklist regarding lesion morphology and distribution. These qualitative observations were processed using an XgBoost algorithm.
- Radiomics: Lung lesions were automatically segmented using a 2D convolutional neural network. From these regions of interest (ROIs), 93 quantitative features (texture, shape, and intensity) were extracted. To ensure robustness, features were filtered based on their reproducibility (Intraclass Correlation Coefficient > 0.90) across various image perturbations, such as erosion and different kernels.
The final evaluation compared the performance of two experienced physicians (a radiologist and a pneumologist) working independently versus their performance when assisted by the ML models 30 days later.
Results and Key Insights
The integration of ML significantly improved diagnostic performance, particularly in terms of specificity. While humans are generally sensitive to detecting pneumonia, they often struggle with specificity—the ability to correctly identify the specific cause.
- Diagnostic Accuracy: Physicians alone achieved a mean sensitivity of 83.4% and a specificity of 64.3%. With AI support, sensitivity rose to 87.1%, and specificity increased dramatically to 91.1%.
- Model Performance: On the validation set, the Radiomics model and the Structured Report model achieved AUCs of 0.84 and 0.82, respectively. The radiologist, when assisted by AI, reached a peak specificity of 96.4% and an AUC of 0.97.
- Inter-rater Reliability: A critical finding was the improvement in Cohen’s Kappa coefficient. ML assistance moved the agreement between the radiologist and the pneumologist from "moderate" to "substantial," suggesting that AI can help harmonize diagnostic standards across different medical specialties.
Discussion and Limitations
The study highlights that standardizing clinical descriptions through structured reports makes qualitative data "minable" for AI, acting as a powerful "specificity filter." However, several limitations remain. The sample size of 128 subjects is relatively small, resulting in wide confidence intervals. Additionally, the models were specifically trained to distinguish COVID-19 from other pneumonias and may not generalize to other lung pathologies like cancer. Future research will focus on fusing the radiomics and structured report models into a single unified score and validating the system in larger, prospective clinical trials.
Summary updated