A diagnostic and economic evaluation of the complex artificial intelligence algorithm aimed to detect 10 pathologies on the chest CT images

Valeria Chernina, Mikhail Belyaev, A. Yu. Silin, Ivan O. Avetisov, Ilya Pyatnitskiy, Ea Petrash, Maria V. Basova, В. Е. Синицын, V. V. Omelyanovsky, Victor А. GombolevskiymedRxiv 2023 · DOI

TL;DR

This study demonstrates that a multi-task AI algorithm can identify pathologies missed by radiologists in over 50% of chest CT scans. Beyond improving diagnostic accuracy, the implementation of such AI "assistants" offers a significant economic incentive, potentially generating 3.6 times more revenue for clinics by identifying necessary follow-up procedures that would otherwise go unbilled.

What Problem Does This Paper Address?

Radiology departments worldwide face a growing crisis of volume and burnout. When a radiologist reviews a chest CT, they are often looking for a specific answer to a clinical question (e.g., "Does this patient have COVID-19?"). In the process, clinically significant "incidental findings"—such as early-stage lung nodules, coronary calcification, or signs of osteoporosis—are frequently overlooked.

Missing these findings isn't just a patient safety issue; it’s an economic one. Each missed diagnosis represents a "lost" pathway for necessary medical care, including specialist consultations and follow-up imaging. The authors aim to quantify both the diagnostic gap and the "Lost Potential Revenue" (LPR) that could be recovered by using AI as a secondary auditor.

Methods

The researchers conducted a retrospective, observational study at a private medical center in Moscow. They analyzed 160 non-contrast chest CT scans using a complex AI algorithm called "Multi-IRA" (developed by IRA Labs).

What makes this "complex" is its multi-task nature: rather than looking for a single disease, it simultaneously screens for 10 distinct pathologies:

  1. COVID-19 pneumonia
  2. Lung nodules
  3. Pleural effusion
  4. Pulmonary emphysema
  5. Thoracic aortic dilatation
  6. Pulmonary trunk dilatation
  7. Coronary artery calcification
  8. Adrenal hyperplasia
  9. Osteoporosis (density changes)
  10. Vertebral compression fractures

The study compared the original radiologist reports against the AI's findings, which were then validated by two independent expert radiologists to eliminate false positives. To calculate the economic impact, the authors mapped every missed finding to the standard clinical guidelines for follow-up care (the "second stage" of diagnostics) and applied the hospital's price list to determine the LPR.

Datasets

The primary evaluation was performed on a dataset of 160 native chest CT scans from adult patients (mean age 49.5) at "Yauza Hospital" LLC, Moscow.

While the study itself was a retrospective analysis of this clinical data, the "Multi-IRA" algorithm had been previously trained and validated on developer-independent, non-public datasets within the framework of the "Moscow Experiment"—one of the world's largest regional projects for testing medical AI. The algorithm was required to meet a minimum accuracy threshold of 0.81 ROC AUC for each pathology before being used in this study.

Key Insights and Ideas

  • The "Incidental" Goldmine: The most striking insight is that radiologists are highly efficient at finding what they are looking for, but struggle with what they aren't. AI excels at this "background" screening, turning a single scan into a comprehensive health check.
  • Economic Alignment: Often, AI is viewed as a cost center. This paper reframes it as a revenue generator. By identifying significant pathologies that require follow-up, the AI naturally feeds the clinical pipeline for consultations and further imaging.
  • The Audit Effect: The study suggests that AI is most effective as an "assistant" or auditor. It identified that 51% of reports were missing at least one pathology. This highlights a massive opportunity for quality control in private healthcare settings.
  • Pathology Specificity: Some conditions are missed more than others. The largest number of clinically significant missed findings occurred in cases of osteoporosis and adrenal masses, suggesting these are areas where human perception is particularly prone to "satisfaction of search" bias.

Key Results

  • Diagnostic Gap: Out of 160 CTs, AI identified pathologies in 90 cases (56%). Crucially, 51% of all studies (81/160) had at least one pathology missed in the original radiologist's report.
  • Revenue Potential: The total Lost Potential Revenue (LPR) for the 81 patients with missed findings was estimated at RUB 2,847,760 (~$37,251).
  • Efficiency Multiplier: The authors concluded that using AI brings 3.6 times more economic benefit compared to the standard model without AI.
  • Significant vs. Insignificant: Even when filtering for only "clinically significant" missed findings, the LPR remained substantial at RUB 770,855 (~$10,083) for the cohort.

Limitations and Future Work

The authors acknowledge several limitations:

  • Pilot Scale: The study used a relatively small sample size (160 scans) from a single private center, which may not represent public health settings or different demographic mixes.
  • Economic Assumptions: The LPR calculation assumes 100% patient compliance—that every patient notified of a missed finding would actually return for the recommended follow-up. In reality, conversion rates are likely lower.
  • Cost of Implementation: The study did not factor in the cost of the AI license itself or the time required for radiologists to verify AI flags.
  • Retrospective Design: A prospective study is needed to see how radiologists interact with the AI in real-time and whether "AI-assisted" reporting actually changes patient outcomes over the long term.

Summary updated

Abstract

Abstract Background Artificial intelligence (AI) technologies can help solve the significant problem of missed findings in radiology studies. An important issue is assessing the economic benefits of implementing AI. Aim to evaluate the frequency of missed pathologies detection and the economic potential of AI technology for chest CT, validated by expert radiologists, compared with radiologists without access to AI in a private medical center. Methods An observational, single-center retrospective study was conducted. The study included chest CTs without IV contrast performed from 01.06.2022 to 31.07.2022 in "Yauza Hospital" LLC, Moscow. The CTs were processed using a complex AI algorithm for ten pathologies: pulmonary infiltrates, typical for viral pneumonia (COVID-19 in pandemic conditions); lung nodules; pleural effusion; pulmonary emphysema; thoracic aortic dilatation; pulmonary trunk dilatation; coronary artery calcification; adrenal hyperplasia; osteoporosis (vertebral body height and density changes). Two experts analyzed CTs and compared results with AI. Further routing was determined according to clinical guidelines for all findings initially detected and missed by radiologists. The lost potential revenue (LPR) was calculated for each patient according to the hospital price list. Results From the final 160 CTs, the AI identified 90 studies (56%) with pathologies, of which 81 studies (51%) were missing at least one pathology in the report. The "second-stage" LPR for all pathologies from 81 patients was RUB 2,847,760 ($37,251 or CNY 256,218). LPR only for those pathologies missed by radiologists but detected by AI was RUB 2,065,360 ($27,017 or CNY 185,824). Conclusion Using AI for chest CTs as an "assistant" to the radiologist can significantly reduce the number of missed abnormalities. AI usage can bring 3.6 times more benefits compared to the standard model without AI. The use of complex AI for chest CT can be cost-effective.
BibTeX
@misc{chernina2023diagnostic,
  author = {Valeria Chernina and Mikhail Belyaev and A. Yu. Silin and Ivan O. Avetisov and Ilya Pyatnitskiy and EA Petrash and Maria V. Basova and В. Е. Синицын and V. V. Omelyanovsky and Victor А. Gombolevskiy},
  title = {{A diagnostic and economic evaluation of the complex artificial intelligence algorithm aimed to detect 10 pathologies on the chest CT images}},
  year = {2023},
  doi = {10.1101/2023.04.19.23288584},
  howpublished = {medRxiv},
}
Figure