Constructing and implementing a performance evaluation indicator set for artificial intelligence decision support systems in pediatric outpatient clinics: an observational study

Yingwen Wang, Weijia Fu, Yuejie Zhang, Daoyang Wang, Ying Gu, Weibing Wang, Hong Xu, Xiaoling Ge, Chengjie Ye, Jinwu Fang, Ling Su, Jiayu Wang, He Wen, Xiaobo Zhang, Rui FengScientific 2024 · DOI

TL;DR

This paper addresses the critical need for a standardized, comprehensive evaluation framework for AI decision support systems (AI-DSS) in pediatric outpatient clinics. The authors developed and implemented a 1-3-5 indicator set (1 clinic, 3 dimensions, 5 components) using a Delphi method and combined subjective and objective weighting to assess AI-DSS performance, aiming for continuous monitoring and improvement.

What Problem Does This Paper Address?

The rapid integration of Artificial Intelligence (AI) into healthcare, particularly AI Decision Support Systems (AI-DSS), presents a significant challenge: how do we effectively measure their performance and ensure they deliver real-world value? This is especially true in complex pediatric outpatient settings, where diverse patient needs and clinical scenarios abound. While AI-DSS can offer benefits like improved diagnosis and reduced wait times, their cost and potential for disruption necessitate rigorous evaluation. Existing evaluation frameworks are often too general, focus on technical prowess over practical application, or lack stakeholder input. This paper aims to fill this gap by creating a specialized, systematic, and continuous evaluation system for AI-DSS in pediatric outpatient clinics.

Methods

The study employed a two-stage approach:

  1. Indicator Set Development:
    • Literature Review: A systematic review of AI decision-making systems in healthcare was conducted, along with literature on information system evaluation.
  2. Focus Group Interviews: Eleven experts involved in AI-DSS development and utilization shared their insights.
  3. Delphi Method: A national panel of 49 experts (physicians, nurses, radiologists, pharmacists, AI professors) participated in two rounds of consultations. They rated the importance of draft indicators on a 5-point Likert scale and provided feedback for additions, deletions, or modifications.
  4. SMART Model Adaptation: Indicators were initially categorized using the Society-Management-Application-Result-Technology (SMART) model, which was adapted to a "1-3-5" framework (1 pediatric outpatient clinic, 3 dimensions: organizational performance, societal performance, user experience performance, and 5 components of technology utility).
  5. Weight Analysis:
    • Subjective Weights: The Analytic Hierarchy Process (AHP) was used, based on expert opinions from the final Delphi round, to determine the importance of each indicator.
  6. Objective Weights: The Entropy Weight Method (EWM) was applied, using data directly extracted from the hospital information system to objectively measure indicator importance.
  7. Combined Weights: Subjective and objective weights were synthesized using a geometric average method to create a final, comprehensive weight for each indicator.

Datasets

The study did not introduce new benchmark datasets. Instead, it utilized data directly from the hospital information system of the Children's Hospital of Fudan University for the objective weight calculation. This data was collected quarterly over a specific period (March 31, 2023, June 30, 2023, and September 28, 2023).

Key Insights and Ideas

  • The "1-3-5" Framework: Moving beyond generic evaluation, the paper proposes a structured framework tailored to the specific context of pediatric outpatient AI-DSS, emphasizing three core dimensions and five technology utility components. This specificity is crucial for meaningful assessment.
  • Bridging Subjective and Objective Evaluation: The combination of expert opinion (AHP) and real-world data (EWM) for weighting indicators offers a more robust and balanced assessment than relying on either method alone. This approach acknowledges both expert knowledge and empirical performance.
  • Organizational Performance Dominance: Interestingly, "Organizational performance" emerged as the most heavily weighted indicator across both objective and combined weightings. This suggests that the practical integration, efficiency, and impact on hospital operations are paramount for successful AI-DSS implementation in this setting.
  • Data-Driven Weighting Challenges: The paper notes that the entropy weight method can result in very low weights for indicators with minimal data variation over time. This highlights the need for careful consideration of data collection frequency and indicator selection to ensure they truly reflect dynamic performance.

Key Results

  • Final Indicator Set: A comprehensive set of 3 first-class, 15 second-class, and 47 third-class indicators was established.
  • Expert Consensus: High authority coefficients (0.834 and 0.846) and statistically significant Kendall's coordination coefficients (0.135 in Round 1, 0.312 in Round 2) indicate good expert agreement.
  • Indicator Weighting:
    • Objective/Combined Weights: Indicator I-1 (Organizational performance) received the highest weight, followed by I-2 (Societal performance) and I-3 (User experience performance).
  • Subjective Weights: 'Societal performance' held the most weight, followed by 'Organizational performance' and 'User experience performance'.
  • Implementation: The established indicator set was integrated into the hospital information system for pilot implementation, enabling dynamic monitoring.

Limitations and Future Work

  • Delphi Method Limitations: The study acknowledges that the Delphi method, especially when conducted via email, might limit real-time discussion depth and exploration of alternative viewpoints.
  • Subjective Weight Biases: Expert opinions can be influenced by biases, incomplete information, or professional interests, potentially affecting the validity of subjective weightings.
  • Entropy Weight Method Data Requirements: The entropy weight method requires long-term observation or frequent data collection. The quarterly data collection in this study might not fully capture dynamic performance for all indicators, potentially leading to bias.
  • Future Work: The authors emphasize the need for long-term data collection to optimize the weight proportions of the established indicators. Further exploration of the entropy weight method with varied data collection frequencies (e.g., monthly, biannually) is suggested to enhance robustness and minimize bias.

Summary updated

Abstract

Artificial intelligence (AI) decision support systems in pediatric healthcare have a complex application background. As an AI decision support system (AI-DSS) can be costly, once applied, it is crucial to focus on its performance, interpret its success, and then monitor and update it to ensure ongoing success consistently. Therefore, a set of evaluation indicators was explicitly developed for AI-DSS in pediatric healthcare, enabling continuous and systematic performance monitoring. The study unfolded in two stages. The first stage encompassed establishing the evaluation indicator set through a literature review, a focus group interview, and expert consultation using the Delphi method. In the second stage, weight analysis was conducted. Subjective weights were calculated based on expert opinions through analytic hierarchy process, while objective weights were determined using the entropy weight method. Subsequently, subject and object weights were synthesized to form the combined weight. In the two rounds of expert consultation, the authority coefficients were 0.834 and 0.846, Kendall's coordination coefficient was 0.135 in Round 1 and 0.312 in Round 2. The final evaluation indicator set has three first-class indicators, fifteen second-class indicators, and forty-seven third-class indicators. Indicator I-1(Organizational performance) carries the highest weight, followed by Indicator I-2(Societal performance) and Indicator I-3(User experience performance) in the objective and combined weights. Conversely, 'Societal performance' holds the most weight among the subjective weights, followed by 'Organizational performance' and 'User experience performance'. In this study, a comprehensive and specialized set of evaluation indicators for the AI-DSS in the pediatric outpatient clinic was established, and then implemented. Continuous evaluation still requires long-term data collection to optimize the weight proportions of the established indicators.
BibTeX
@inproceedings{wang2024constructing,
  author = {Yingwen Wang and Weijia Fu and Yuejie Zhang and Daoyang Wang and Ying Gu and Weibing Wang and Hong Xu and Xiaoling Ge and Chengjie Ye and Jinwu Fang and Ling Su and Jiayu Wang and He Wen and Xiaobo Zhang and Rui Feng},
  title = {{Constructing and implementing a performance evaluation indicator set for artificial intelligence decision support systems in pediatric outpatient clinics: an observational study}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-64893-w},
}
Figure