TL;DR
This paper addresses the critical need for a standardized, comprehensive evaluation framework for AI decision support systems (AI-DSS) in pediatric outpatient clinics. The authors developed and implemented a 1-3-5 indicator set (1 clinic, 3 dimensions, 5 components) using a Delphi method and combined subjective and objective weighting to assess AI-DSS performance, aiming for continuous monitoring and improvement.
What Problem Does This Paper Address?
The rapid integration of Artificial Intelligence (AI) into healthcare, particularly AI Decision Support Systems (AI-DSS), presents a significant challenge: how do we effectively measure their performance and ensure they deliver real-world value? This is especially true in complex pediatric outpatient settings, where diverse patient needs and clinical scenarios abound. While AI-DSS can offer benefits like improved diagnosis and reduced wait times, their cost and potential for disruption necessitate rigorous evaluation. Existing evaluation frameworks are often too general, focus on technical prowess over practical application, or lack stakeholder input. This paper aims to fill this gap by creating a specialized, systematic, and continuous evaluation system for AI-DSS in pediatric outpatient clinics.
Methods
The study employed a two-stage approach:
- Indicator Set Development:
- Literature Review: A systematic review of AI decision-making systems in healthcare was conducted, along with literature on information system evaluation.
- Focus Group Interviews: Eleven experts involved in AI-DSS development and utilization shared their insights.
- Delphi Method: A national panel of 49 experts (physicians, nurses, radiologists, pharmacists, AI professors) participated in two rounds of consultations. They rated the importance of draft indicators on a 5-point Likert scale and provided feedback for additions, deletions, or modifications.
- SMART Model Adaptation: Indicators were initially categorized using the Society-Management-Application-Result-Technology (SMART) model, which was adapted to a "1-3-5" framework (1 pediatric outpatient clinic, 3 dimensions: organizational performance, societal performance, user experience performance, and 5 components of technology utility).
- Weight Analysis:
- Subjective Weights: The Analytic Hierarchy Process (AHP) was used, based on expert opinions from the final Delphi round, to determine the importance of each indicator.
- Objective Weights: The Entropy Weight Method (EWM) was applied, using data directly extracted from the hospital information system to objectively measure indicator importance.
- Combined Weights: Subjective and objective weights were synthesized using a geometric average method to create a final, comprehensive weight for each indicator.
Datasets
The study did not introduce new benchmark datasets. Instead, it utilized data directly from the hospital information system of the Children's Hospital of Fudan University for the objective weight calculation. This data was collected quarterly over a specific period (March 31, 2023, June 30, 2023, and September 28, 2023).
Key Insights and Ideas
- The "1-3-5" Framework: Moving beyond generic evaluation, the paper proposes a structured framework tailored to the specific context of pediatric outpatient AI-DSS, emphasizing three core dimensions and five technology utility components. This specificity is crucial for meaningful assessment.
- Bridging Subjective and Objective Evaluation: The combination of expert opinion (AHP) and real-world data (EWM) for weighting indicators offers a more robust and balanced assessment than relying on either method alone. This approach acknowledges both expert knowledge and empirical performance.
- Organizational Performance Dominance: Interestingly, "Organizational performance" emerged as the most heavily weighted indicator across both objective and combined weightings. This suggests that the practical integration, efficiency, and impact on hospital operations are paramount for successful AI-DSS implementation in this setting.
- Data-Driven Weighting Challenges: The paper notes that the entropy weight method can result in very low weights for indicators with minimal data variation over time. This highlights the need for careful consideration of data collection frequency and indicator selection to ensure they truly reflect dynamic performance.
Key Results
- Final Indicator Set: A comprehensive set of 3 first-class, 15 second-class, and 47 third-class indicators was established.
- Expert Consensus: High authority coefficients (0.834 and 0.846) and statistically significant Kendall's coordination coefficients (0.135 in Round 1, 0.312 in Round 2) indicate good expert agreement.
- Indicator Weighting:
- Objective/Combined Weights: Indicator I-1 (Organizational performance) received the highest weight, followed by I-2 (Societal performance) and I-3 (User experience performance).
- Subjective Weights: 'Societal performance' held the most weight, followed by 'Organizational performance' and 'User experience performance'.
- Implementation: The established indicator set was integrated into the hospital information system for pilot implementation, enabling dynamic monitoring.
Limitations and Future Work
- Delphi Method Limitations: The study acknowledges that the Delphi method, especially when conducted via email, might limit real-time discussion depth and exploration of alternative viewpoints.
- Subjective Weight Biases: Expert opinions can be influenced by biases, incomplete information, or professional interests, potentially affecting the validity of subjective weightings.
- Entropy Weight Method Data Requirements: The entropy weight method requires long-term observation or frequent data collection. The quarterly data collection in this study might not fully capture dynamic performance for all indicators, potentially leading to bias.
- Future Work: The authors emphasize the need for long-term data collection to optimize the weight proportions of the established indicators. Further exploration of the entropy weight method with varied data collection frequencies (e.g., monthly, biannually) is suggested to enhance robustness and minimize bias.
Summary updated