The Benefit of Augmenting Open Data with Clinical Data-Warehouse EHR for Forecasting SARS-CoV-2 Hospitalizations in Bordeaux Area, France

Thomas Ferté, Vianney Jouhet, Romain Griffier, Boris P. Hejblum, Rodolphe ThiébautSSRN 2022 · DOI

TL;DR

This study demonstrates that integrating Electronic Health Records (EHRs) from a hospital data warehouse with publicly available data significantly improves the accuracy of forecasting COVID-19 hospitalizations. By leveraging granular, real-time clinical data, forecasting models can better anticipate epidemic surges and inform public health responses.

What Problem Does This Paper Address?

Accurate forecasting of infectious disease outbreaks, particularly COVID-19, is crucial for effective healthcare system management and public health policy. While numerous forecasting models have been developed, many struggle with accuracy, especially in predicting sudden shifts in epidemic trends (slope shifts). Existing models often rely on aggregated, publicly available data (like national case counts or weather patterns), which may lack the granularity and timeliness needed for precise local predictions. The challenge lies in bridging the gap between broad epidemiological trends and the specific, real-time dynamics within a hospital setting. This paper addresses the question of whether incorporating detailed clinical data from a hospital's data warehouse can enhance the predictive power of these models, particularly for short-term hospitalization forecasts.

Methods

The researchers developed and evaluated several forecasting models, primarily focusing on elastic-net penalized linear regression. This statistical technique is well-suited for high-dimensional data as it performs feature selection by shrinking the coefficients of less important features to zero. They employed a rigorous modeling strategy:

  • Feature Engineering: They created various features from the raw data, including rolling means, minimums, maximums, and first derivatives over different time windows (e.g., last 7 days). Smoothing techniques were also applied to handle outliers and weekly variations.
  • Model Comparison: They compared different data sources, feature engineering approaches, and machine learning models (including random forest and Fréchet random forest).
  • Adaptive Training: The model was retrained daily to adapt to the evolving dynamics of the epidemic, a crucial aspect given the non-stationary nature of infectious disease data influenced by factors like vaccination and new variants.
  • Evaluation Metrics: Performance was assessed using Median Absolute Error (MAE) and Median Relative Error (MRE), chosen for their robustness to extreme values common in low-count scenarios.

The novelty lies in the systematic comparison of forecasting performance with and without the inclusion of detailed EHR data, and the exploration of how different data sources and feature engineering techniques contribute to improved accuracy.

Datasets

The study utilized data from two primary sources for the period of May 16, 2020, to January 17, 2022:

  • Open Data:
    • Epidemiological Data: From Santé Publique France, including hospitalizations, RT-PCR test results (positive and total), dominant variants, and vaccination numbers.
  • Weather Data: From the National Oceanic and Atmospheric Administration (NOAA) Integrated Surface Database, including temperature, wind speed, humidity, and dew point. An index for COVID-19 climate transmissibility (IPTCC) was computed.
  • Clinical Data:
    • Bordeaux Hospital Data Warehouse: This included aggregated data from the hospital's Electronic Health Records (EHRs), such as hospital admissions, ICU admissions, discharge data, ambulance service notes, and emergency unit notes. Concepts related to COVID-19 (e.g., cough, dyspnea) were extracted from these notes.

This study does not introduce a new benchmark dataset but rather leverages existing public data and a hospital's internal data warehouse to create a more comprehensive forecasting framework.

Key Insights and Ideas

  • EHRs are Game-Changers for Local Forecasting: The most significant insight is that granular clinical data from a hospital's EHR system provides a substantial boost to forecasting accuracy, particularly for short-term predictions (7 and 14 days). This highlights the value of local, real-time data over aggregated national or regional statistics.
  • Timeliness Matters: The data warehouse provided information with a shorter delay (1 day) compared to public data sources (up to 5 days for hospitalizations and RT-PCR). This "data availability advantage" directly translates to more timely and potentially more accurate forecasts.
  • Model Adaptability is Key: The epidemic's dynamics are constantly changing due to interventions, variants, and population behavior. Daily retraining of the model, coupled with elastic-net's feature selection, allows the model to adapt to these shifts, improving its robustness.
  • Not All Data is Equally Useful: While weather data showed some improvement, the impact of vaccine and variant data was less pronounced in this specific model. This could be due to the monotonic nature of vaccination curves, potential biases in variant reporting, or the possibility that this information was already implicitly captured by other features like RT-PCR and hospitalization trends.
  • Linear Models Can Be Powerful: Despite the complexity of epidemic dynamics, a well-tuned linear regression model with elastic-net regularization outperformed more complex models like random forests in this context. This suggests that for short-term forecasting, capturing linear relationships and performing robust feature selection can be highly effective.

Key Results

  • The best performing model, an elastic-net penalized linear regression using all available data, achieved a median relative error (MRE) of 0.136 (13.6%) at 7 days and 0.198 (19.8%) at 14 days.
  • The inclusion of EHR data from the Bordeaux hospital data warehouse improved the median relative error by 10.9% at 7 days and 19.8% at 14 days.
  • Adding weather data improved the 7- and 14-day MAE by 1.28 and 5.58 hospitalizations, respectively.
  • The model's performance was notably better when using EHR data, as illustrated by its ability to anticipate the April peak and better forecast the winter increase, while being more robust to specific hospital clusters.
  • Graphical evaluation indicated that the remaining forecast error was primarily due to delays in detecting slope shifts in the epidemic curve.

Limitations and Future Work

  • Slope Shift Detection: The authors acknowledge that the model struggles to accurately predict sudden changes in the epidemic's trajectory (slope shifts). This is a common challenge in time-series forecasting.
  • Non-Stationarity and External Factors: The epidemic's dynamics are inherently non-stationary due to interventions like vaccination and the emergence of new variants. While the adaptive model helps, capturing these complex, evolving relationships remains a challenge.
  • Linear Model Assumptions: The linear regression model might not fully capture highly complex, non-linear relationships within the data.
  • Data Delays and Bias: Delays in data availability (e.g., billing codes for hospitalizations) and potential measurement biases (e.g., variant identification) can affect model performance.

Future work could explore more sophisticated mechanistic models that explicitly incorporate epidemiological principles, or investigate other machine learning approaches (like reservoir computing) that might better capture complex dynamics. Further research into how to better integrate the impact of vaccination and variants, or how to improve slope shift detection, would also be valuable. The authors suggest that the development of hospital data warehouses is a promising avenue for improving epidemic forecasting at both larger and finer scales.

Summary updated

BibTeX
@misc{ferte2022benefit,
  author = {Thomas Ferté and Vianney Jouhet and Romain Griffier and Boris P. Hejblum and Rodolphe Thiébaut},
  title = {{The Benefit of Augmenting Open Data with Clinical Data-Warehouse EHR for Forecasting SARS-CoV-2 Hospitalizations in Bordeaux Area, France}},
  year = {2022},
  doi = {10.2139/ssrn.4071506},
  howpublished = {SSRN},
}
Figure