The RITeS (Real-world Independent Testing of e-ASPECTS Software) study establishes a rigorous framework to independently validate AI-driven stroke diagnostic tools. By utilizing a massive, multi-center dataset, the study aims to provide a comprehensive "stress test" for automated imaging software in clinical environments.
The Challenge of Emergency Stroke Triage
In acute stroke management, "time is brain." Clinicians rely on Non-Contrast CT (NCCT) scans to calculate the ASPECTS score (a 10-point scale assessing early ischemic changes) and CT Angiography (CTA) to identify arterial occlusions. However, interpreting these scans rapidly and accurately is challenging, particularly for non-experts.
Artificial Intelligence (AI) tools like Brainomix’s e-ASPECTS and e-CTA offer automated scoring and visual "heatmaps." While initial results are promising, there is a critical need for independent, large-scale validation that accounts for real-world complexities such as poor image quality, "stroke mimics" (conditions that look like strokes but aren't), and diverse patient populations.
Study Methodology
The RITeS Statistical Analysis Plan (SAP) details how the software will be evaluated:
- Reference Standards: AI outputs will be compared against expert human ratings and "Diagnostic Ground Truth" (follow-up imaging and final clinical diagnosis).
- Statistical Metrics: The study will use Bland-Altman plots to visualize agreement and Krippendorff’s Alpha (K-alpha) for inter-rater reliability, offering more robustness than standard Cohen’s Kappa when handling multiple observers and missing data.
- Intention-to-Analyse: Technical failures (where the software fails to process a scan) will be included in the analysis to provide a realistic assessment of clinical utility.
- Clinical Impact: A physician questionnaire will evaluate whether AI overlays influence diagnostic confidence or treatment decisions.
Datasets
The study leverages an extensive imaging bank pooled from 10 national and international clinical stroke trials and registries, encompassing over 6,600 patients.
This diverse dataset ensures the AI is tested against various scanners, slice thicknesses, and pathologies, including both ischemic (clot-based) and hemorrhagic (bleeding) strokes.
Key Insights and Success Criteria
- Distinguishing Mimics: A core objective is determining if AI can differentiate acute strokes from "mimics" like tumors or old lesions.
- Analyzing Failure Modes: Researchers will use multivariable regression to identify why software might fail (e.g., patient movement or specific brain regions).
- Success Thresholds: The software is considered "non-inferior" if ASPECTS scores remain within a 5% margin of expert ratings. For clinical equivalence, the limit of agreement is set at ±2 ASPECTS points—a threshold meaningful for treatment decisions.
Limitations
While the dataset is vast, it is retrospective. Future research may require prospective "shadow testing" in active emergency departments. Additionally, because AI software evolves rapidly, the version tested may be superseded by the time results are published.
Summary updated