Development of a quantitative index system for evaluating the quality of electronic medical records in disease risk intelligent prediction

Jiayin Zhou, Jie Hao, Mingkun Tang, Haixia Sun, Jiayang Wang, Jiao Li, Qing QianBMC Medical Informatics Decis. Mak 2024 · DOI

TL;DR

This paper introduces a novel, quantitative index system to rigorously assess the quality of Electronic Medical Records (EMRs) for disease risk prediction using Machine Learning (ML). By developing and validating this system, the authors provide a much-needed framework to ensure EMR data is reliable, ultimately leading to more accurate predictions and better patient care.

What Problem Does This Paper Address?

The digital health revolution has seen a surge in the use of Electronic Medical Records (EMRs) for proactive health management, particularly in predicting disease risks with Machine Learning (ML). However, the effectiveness of these ML models is heavily dependent on the quality of the EMR data they are trained on. Existing methods for evaluating EMR data quality are often qualitative, subjective, or lack a comprehensive approach tailored to the specific needs of ML-based disease risk prediction. This leaves researchers and data managers without a clear, objective standard to ensure EMR data is fit for purpose, potentially leading to biased models and unreliable predictions. This paper aims to bridge this gap by developing and validating a quantitative index system specifically designed for this critical task.

Methods

The researchers developed their quantitative index system through a multi-step process:

  1. Literature Review: An initial set of indicators was compiled by reviewing existing studies on EMR and ML applications in disease risk prediction.
  2. Delphi Method: This iterative expert consultation process was used to structure the indicators into a three-level hierarchy (first-level for broad dimensions, second-level for specific EMR quality aspects, and third-level for granular requirements) and refine their definitions.
  3. Analytic Hierarchy Process (AHP): AHP was employed to quantitatively determine the weights of the indicators at each level. This method breaks down complex decision-making into pairwise comparisons, allowing for the quantification of subjective expert opinions into objective weights.
  4. Empirical Validation: The developed index system was tested using real-world EMR data from the MIMIC-III database. Datasets were constructed, scored using the index system, and then used to train ML models for disease risk prediction (sepsis). The correlation between the EMR data quality scores and the ML model performance was analyzed to validate the system's effectiveness.

Datasets

The study utilized the MIMIC-III clinical database (v1.4), a large, freely accessible dataset containing comprehensive health-related data from over 46,000 patients admitted to intensive care units (ICUs) between 2001 and 2012. For their empirical validation, the researchers constructed five distinct datasets from MIMIC-III, focusing on patients at high risk of sepsis. These datasets were categorized by specific patient groups: elderly (>80 years old), long-stay (>30 days), ischemic stroke, acute renal failure (ARF), and cirrhosis (CIR).

Key Insights and Ideas

  • Quantifying EMR Quality is Crucial for ML: The study strongly reinforces that EMR data quality isn't just a background concern; it directly impacts the performance of ML models used for disease risk prediction.
  • A Hierarchical, Multi-Level Approach is Effective: Breaking down data quality into first, second, and third-level indicators provides a structured and comprehensive way to assess different facets of EMR data.
  • Expert Consensus and Quantitative Weighting are Key: Combining expert knowledge (Delphi) with a structured weighting method (AHP) creates a robust and defensible index system.
  • Real-World Validation is Essential: The empirical study demonstrating a positive correlation between higher data quality scores and better ML model performance is critical for establishing the system's practical utility.

Key Results

The developed index system comprises four first-level, 11 second-level, and 33 third-level indicators. Through empirical validation using the MIMIC-III datasets for sepsis prediction:

  • A positive correlation was observed between the scores assigned by the index system and the predictive performance of ML models (LR, SVM, RF). Datasets with higher quality scores generally led to better model accuracy, precision, and AUC.
  • The LLOS (Long Length of Stay) dataset achieved the highest overall quality score (0.966), while the ARF (Acute Renal Failure) dataset had the lowest (0.907).
  • Across all models and datasets, SVM-Precision showed the strongest positive correlation with the dataset quality scores, indicating its particular sensitivity to data quality.
  • The LR-Precision correlation was an exception, showing a weaker relationship, suggesting potential model-specific sensitivities to data quality.

Limitations and Future Work

The authors acknowledge several limitations:

  • Third-Level Indicator Weighting: The calculation of weights for third-level indicators was based on simple percentages, which might not fully capture the nuanced importance of each indicator.
  • Single Data Resource: The empirical study was conducted using only the MIMIC-III database. This limits the generalizability of the findings, and the authors suggest future validation studies using diverse EMR data resources.
  • Comparison of Data Resources: Validating the system's ability to compare different data resources might require more varied datasets.

Future work should focus on addressing these limitations by incorporating more sophisticated weighting methods for third-level indicators and conducting validation across multiple, diverse EMR datasets to further solidify the system's robustness and broad applicability.

Summary updated

BibTeX
@article{zhou2024development,
  author = {Jiayin Zhou and Jie Hao and Mingkun Tang and Haixia Sun and Jiayang Wang and Jiao Li and Qing Qian},
  title = {{Development of a quantitative index system for evaluating the quality of electronic medical records in disease risk intelligent prediction}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2024},
  doi = {10.1186/s12911-024-02533-z},
}
Figure