Real-world Independent Testing of e-ASPECTS Software (RITeS): statistical analysis plan

Grant Mair, Francesca M. Chappell, Chloé Martin, David Dye, Philip M. Bath, Keith W. Muir, Rüdiger von Kummer, Rustam Al‐Shahi Salman, Peter Sandercock, Malcolm Macleod, Nikola Sprigg, Phil White, Joanna M. WardlawAMRC 2020 · DOI

The RITeS (Real-world Independent Testing of e-ASPECTS Software) study establishes a rigorous framework to independently validate AI-driven stroke diagnostic tools. By utilizing a massive, multi-center dataset, the study aims to provide a comprehensive "stress test" for automated imaging software in clinical environments.

The Challenge of Emergency Stroke Triage

In acute stroke management, "time is brain." Clinicians rely on Non-Contrast CT (NCCT) scans to calculate the ASPECTS score (a 10-point scale assessing early ischemic changes) and CT Angiography (CTA) to identify arterial occlusions. However, interpreting these scans rapidly and accurately is challenging, particularly for non-experts.

Artificial Intelligence (AI) tools like Brainomix’s e-ASPECTS and e-CTA offer automated scoring and visual "heatmaps." While initial results are promising, there is a critical need for independent, large-scale validation that accounts for real-world complexities such as poor image quality, "stroke mimics" (conditions that look like strokes but aren't), and diverse patient populations.

Study Methodology

The RITeS Statistical Analysis Plan (SAP) details how the software will be evaluated:

  • Reference Standards: AI outputs will be compared against expert human ratings and "Diagnostic Ground Truth" (follow-up imaging and final clinical diagnosis).
  • Statistical Metrics: The study will use Bland-Altman plots to visualize agreement and Krippendorff’s Alpha (K-alpha) for inter-rater reliability, offering more robustness than standard Cohen’s Kappa when handling multiple observers and missing data.
  • Intention-to-Analyse: Technical failures (where the software fails to process a scan) will be included in the analysis to provide a realistic assessment of clinical utility.
  • Clinical Impact: A physician questionnaire will evaluate whether AI overlays influence diagnostic confidence or treatment decisions.

Datasets

The study leverages an extensive imaging bank pooled from 10 national and international clinical stroke trials and registries, encompassing over 6,600 patients.

Summary of the 10 clinical stroke trials and registries used in the RITeS study.

This diverse dataset ensures the AI is tested against various scanners, slice thicknesses, and pathologies, including both ischemic (clot-based) and hemorrhagic (bleeding) strokes.

Key Insights and Success Criteria

  • Distinguishing Mimics: A core objective is determining if AI can differentiate acute strokes from "mimics" like tumors or old lesions.
  • Analyzing Failure Modes: Researchers will use multivariable regression to identify why software might fail (e.g., patient movement or specific brain regions).
  • Success Thresholds: The software is considered "non-inferior" if ASPECTS scores remain within a 5% margin of expert ratings. For clinical equivalence, the limit of agreement is set at ±2 ASPECTS points—a threshold meaningful for treatment decisions.

Limitations

While the dataset is vast, it is retrospective. Future research may require prospective "shadow testing" in active emergency departments. Additionally, because AI software evolves rapidly, the version tested may be superseded by the time results are published.

Summary updated

Abstract

<b>Background:</b> Artificial intelligence-based software may automatically detect ischaemic stroke lesions and provide an Alberta Stroke Program Early CT score (ASPECTS) on CT, and identify arterial occlusion and provide a collateral score on CTA. Large-scale independent testing will inform clinical use, but is lacking. We aim to test e-ASPECTS and e-CTA (Brainomix, Oxford UK) using CT scans obtained from a range of clinical studies. <b>Methods:</b> Using prospectively collected baseline CT and CTA scans from 10 national/international clinical stroke trials or registries (total >6600 patients), we will select a large clinically representative sample for testing e-ASPECTS and e-CTA compared to previously acquired independent expert human interpretation (reference standard). Our primary aims are to test agreement between software-derived and masked human expert ASPECTS, and the diagnostic accuracy of e-ASPECTS for identifying all causes of stroke symptoms using follow-up imaging and final clinical opinion as diagnostic ground truth. Our secondary aims are to test when and why e-ASPECTS is more or less accurate, or succeeds/fails to produce results, agreement between e-CTA and human expert CTA interpretation, and repeatability of e-ASPECTS/e-CTA results. All testing will be conducted on an intention-to-analyse basis. We will assess agreement between software and expert-human ratings and test the diagnostic accuracy of software. <b>Conclusions:</b> RITeS will provide comprehensive, robust and representative testing of e-ASPECTS and e-CTA against the current gold-standard, expert-human interpretation.
BibTeX
@inproceedings{mair2020real,
  author = {Grant Mair and Francesca M. Chappell and Chloé Martin and David Dye and Philip M. Bath and Keith W. Muir and Rüdiger von Kummer and Rustam Al‐Shahi Salman and Peter Sandercock and Malcolm Macleod and Nikola Sprigg and Phil White and Joanna M. Wardlaw},
  title = {{Real-world Independent Testing of e-ASPECTS Software (RITeS): statistical analysis plan}},
  booktitle = {AMRC},
  year = {2020},
  doi = {10.12688/amrcopenres.12904.1},
}
Figure