Real-world evaluation of AI-driven COVID-19 triage for emergency admissions: External validation & operational assessment of lab-free and high-throughput screening solutions

Andrew A. S. Soltan, Jenny Yang, Ravi Pattanshetty, Alex Novak, Omid Rohanian, Sally Beer, Marina Soltan, David Thickett, Rory Fairhead, Tingting Zhu, David W. Eyre, David A. CliftonmedRxiv 2021 · DOI

Abstract

Uncertainty in patients’ COVID-19 status contributes to treatment delays, nosocomial transmission, and operational pressures in hospitals. While laboratory PCR tests remain the gold standard, their 12-24h turnaround time creates significant bottlenecks. This study evaluates CURIAL-Lab and CURIAL-Rapide, AI-driven triage models that use routine clinical data to screen for COVID-19 within one hour of arrival. Validated across four NHS trusts with 72,223 patients, the models demonstrated high generalizability (AUROCs 0.836-0.881) and a high Negative Predictive Value (99.7%). Prospective deployment of CURIAL-Rapide alongside point-of-care blood analysis reduced time-to-result to 45 minutes, significantly faster than both LFD and PCR testing, potentially reducing the number of COVID-negative patients triaged to high-risk clinical areas by 58.5%.

Introduction

In Emergency Departments (ED), rapid "streaming" of patients into appropriate clinical areas is vital for infection control. However, clinicians often face a choice between slow, accurate PCR tests and fast, but insensitive, lateral flow devices (LFDs). This study presents a real-world evaluation of the CURIAL AI models, designed to act as high-performance "tests-of-exclusion" using data already routinely collected during emergency admissions: vital signs and standard blood work.

Methods

The research team refined the original CURIAL-1.0 model into two streamlined versions using XGBoost classifiers to optimize for speed and generalizability:

  • CURIAL-Lab: Utilizes vital signs and a standard suite of laboratory blood tests (FBC, U&Es, LFTs, and CRP).
  • CURIAL-Rapide: A minimalist model requiring only vital signs and a Full Blood Count (FBC).
Constituents of the CURIAL-Lab and CURIAL-Rapide feature sets.

To achieve a "lab-free" workflow, CURIAL-Rapide was deployed alongside a point-of-care hematology analyzer (SightDiagnostics OLO), providing results at the bedside. The models were validated externally across three independent NHS trusts and prospectively at the John Radcliffe Hospital in Oxford.

Results

The models showed consistent performance across all validation sites. CURIAL-Lab achieved AUROCs between 0.858 and 0.881, while CURIAL-Rapide achieved 0.836 to 0.854.

Key operational findings include:

  • Superior Sensitivity: In a retrospective cohort of 3,207 patients, CURIAL-Rapide reached 88.2% sensitivity, far exceeding the 56.9% sensitivity of LFDs used in routine care.
  • Rapid Turnaround: During prospective deployment, the median time to a CURIAL-Rapide result was 45 minutes. This was 16 minutes faster than LFD results and approximately 7 hours faster than PCR results.
  • High Confidence Exclusion: The model achieved an NPV of 99.7% in real-world use, meaning a negative AI result almost certainly excluded COVID-19.

Discussion and Impact

AI-driven triage offers a scalable solution to the operational challenges of COVID-19 screening. By providing rapid, high-confidence negative results, CURIAL-Rapide can significantly reduce the strain on "amber" (suspected infection) zones. The study found that the AI could have correctly excluded COVID-19 for 58.5% of patients whom clinicians had manually triaged to suspected areas. This approach not only improves hospital flow but also enhances patient safety by reducing unnecessary exposure in high-risk zones. Future work will explore adapting these models for other seasonal respiratory viruses like Influenza and RSV.

Summary updated

Abstract

Abstract Background Uncertainty in patients’ COVID-19 status contributes to treatment delays, nosocomial transmission, and operational pressures in hospitals. However, typical turnaround times for batch-processed laboratory PCR tests remain 12-24h. Although rapid antigen lateral flow testing (LFD) has been widely adopted in UK emergency care settings, sensitivity is limited. We recently demonstrated that AI-driven triage (CURIAL-1.0) allows high-throughput COVID-19 screening using clinical data routinely available within 1h of arrival to hospital. Here we aimed to determine operational and safety improvements over standard-care, performing external/prospective evaluation across four NHS trusts with updated algorithms optimised for generalisability and speed, and deploying a novel lab-free screening pathway in a UK emergency department. Methods We rationalised predictors in CURIAL-1.0 to optimise separately for generalisability and speed, developing CURIAL-Lab with vital signs and routine laboratory blood predictors (FBC, U&E, LFT, CRP) and CURIAL-Rapide with vital signs and FBC alone. Models were calibrated during training to 90% sensitivity and validated externally for unscheduled admissions to Portsmouth University Hospitals, University Hospitals Birmingham and Bedfordshire Hospitals NHS trusts, and prospectively during the second-wave of the UK COVID-19 epidemic at Oxford University Hospitals (OUH). Predictions were generated using first-performed blood tests and vital signs and compared against confirmatory viral nucleic acid testing. Next, we retrospectively evaluated a novel clinical pathway triaging patients to COVID-19-suspected clinical areas where either model prediction or LFD results were positive, comparing sensitivity and NPV with LFD results alone. Lastly, we deployed CURIAL-Rapide alongside an approved point-of-care FBC analyser (OLO; SightDiagnostics, Israel) to provide lab-free COVID-19 screening in the John Radcliffe Hospital’s Emergency Department (Oxford, UK), as trust-approved service improvement. Our primary improvement outcome was time-to-result availability; secondary outcomes were sensitivity, specificity, PPV, and NPV assessed against a PCR reference standard. We compared CURIAL-Rapide’s performance with clinician triage and LFD results within standard-care. Results 72,223 patients met eligibility criteria across external and prospective validation sites. Model performance was consistent across trusts (CURIAL-Lab: AUROCs range 0.858-0.881; CURIAL-Rapide 0.836-0.854), with highest sensitivity achieved at Portsmouth University Hospitals (CURIAL-Lab:84.1% [95% Wilson’s score CIs 82.5-85.7]; CURIAL-Rapide:83.5% [81.8 - 85.1]) at specificities of 71.3% (95% Wilson’s score CIs: 70.9 - 71.8) and 63.6% (63.1 - 64.1). For 3,207 patients receiving LFD-triage within routine care for OUH admissions between December 23, 2021 and March 6, 2021, a combined clinical pathway increased sensitivity from 56.9% for LFDs alone (95% CI 51.7-62.0) to 88.2% with CURIAL-Rapide (84.4-91.1; AUROC 0.919) and 85.6% with CURIAL-Lab (81.6-88.9; AUROC 0.925). 520 patients were prospectively enrolled for point-of-care FBC analysis between February 18, 2021 and May 10, 2021, of whom 436 received confirmatory PCR testing within routine care and 10 (2.3%) tested positive. Median time from patient arrival to availability of CURIAL-Rapide result was 45:00 min (32-64), 16 minutes (26.3%) sooner than LFD results (61:00 min, 37-99; log-rank p<0.0001), and 6:52 h (90.2%) sooner than PCR results (7:37 h, 6:05-15:39; p<0.0001). Sensitivity and specificity of CURIAL-Rapide were 87.5% (52.9-97.8) and 85.4% (81.3-88.7), therefore achieving high NPV (99.7%, 98.2-99.9). CURIAL-Rapide correctly excluded COVID-19 for 58.5% of negative patients who were triaged by a clinician to ‘COVID-19-suspected’ (amber) areas. Impact CURIAL-Lab & CURIAL-Rapide are generalisable, high-throughput screening tests for COVID-19, rapidly excluding the illness with higher NPV than LFDs. CURIAL-Rapide can be used in combination with near-patient FBC analysis for rapid, lab-free screening, and may reduce the number of COVID-19-negative patients triaged to enhanced precautions (‘amber’) clinical areas.
BibTeX
@misc{soltan2021real,
  author = {Andrew A. S. Soltan and Jenny Yang and Ravi Pattanshetty and Alex Novak and Yang Yang and Omid Rohanian and Sally Beer and Marina Soltan and David Thickett and Rory Fairhead and Tingting Zhu and David W. Eyre and David A. Clifton and David A. Clifton},
  title = {{Real-world evaluation of AI-driven COVID-19 triage for emergency admissions: External validation \& operational assessment of lab-free and high-throughput screening solutions}},
  year = {2021},
  doi = {10.1101/2021.08.24.21262376},
  howpublished = {medRxiv},
}
Figure