Online Transfer Learning for RSV Case Detection

Yiming Sun, Yuhe Gao, Runxue Bao, Gregory F. Cooper, Jessi Espino, Harry Hochheiser, Marian G. Michaels, John M. Aronis, Chenxi Song, Ye Ye

TL;DR

Researchers have developed Multi-Source Adaptive Weighting (MSAW), a new online transfer learning framework that detects Respiratory Syncytial Virus (RSV) cases in real-time. By dynamically balancing historical data from past years with incoming patient data, the model achieves superior accuracy (0.870 AUROC) compared to traditional static models, effectively handling the "concept drift" inherent in seasonal infectious diseases.

What Problem Does This Paper Address?

Detecting infectious diseases like RSV using Electronic Health Records (EHRs) is a "moving target" problem. Traditional machine learning models are usually "static"—they are trained on a fixed dataset and deployed. However, in healthcare, data arrives sequentially, and disease patterns change over time due to new virus strains, changes in hospital protocols, or environmental factors (a phenomenon known as concept drift).

Furthermore, when a new RSV season begins, there is a "cold start" problem: there isn't enough new data to train a reliable local model, but relying solely on old data might be misleading. The authors tackle the challenge of how to effectively "transfer" knowledge from multiple past seasons to a current, evolving season without being bogged down by outdated or irrelevant historical information.

Methods

The authors introduce Multi-Source Adaptive Weighting (MSAW). Here is how the engine works:

  1. Base Classifiers: The system uses Naive Bayes as the base learner. While simple, it is computationally efficient and performs well even with the limited data available at the start of a new season.
  2. The Ensemble: The model maintains an ensemble consisting of several "source" models (each trained on a specific past year of data) and one "target" model (which learns incrementally from the current year's data).
  3. Dynamic Weighting: This is the "secret sauce." Instead of giving every year equal importance, MSAW adjusts weights in real-time:
  4. Accuracy-Based Penalty: If a specific source model misclassifies a new patient, its weight is penalized (reduced).
  5. Volume-Adaptive Growth: As more data is collected in the current season, the weight of the "target" model is automatically increased. This allows the system to trust historical data early on but gradually shift its trust to the current season's specific patterns as they emerge.

Datasets

The study utilized a massive dataset from the University of Pittsburgh Medical Center (UPMC):

  • Scope: 2,263,360 patient encounters across five hospitals over nine years (2011–2020).
  • Features: 7,614 distinct features, including demographics, laboratory results, and clinical findings extracted from free-text notes using Natural Language Processing (MetaMap).
  • Segmentation: Data was divided into "seasons" (June to May) to align with the natural epidemiological cycle of RSV.

Key Insights and Ideas

  • Trust is Earned, Not Given: The most significant insight is that temporal relevance isn't always linear. While we might assume the most recent year is the best predictor for the current one, the data showed that some older years were actually more similar to the target year than immediate predecessors. MSAW’s ability to autonomously "discover" which past years to trust is its biggest advantage.
  • The Power of Clinical Notes: By using NLP to extract clinical concepts (like "increased work of breathing"), the model captured nuances that structured data (like simple ICD codes) might miss.
  • Balancing History and Present: The volume-adaptive factor ($\beta$) ensures the model doesn't "overfit" to the past. It provides a mathematical bridge that allows the local, current-season model to eventually take the lead once it has seen enough cases.

Key Results

The MSAW method was evaluated on the 2019–2020 RSV season and compared against several baselines:

  • MSAW Performance: Achieved an AUROC of 0.870, the highest among all tested methods.
  • Comparison to Static Weighting: It significantly outperformed "Equal Weights" (0.728) and "Time-based Weights" (0.761), proving that dynamic adjustment is better than human-defined rules for weighting history.
  • Comparison to Standard Online Learning: Even a model that was pre-trained on all source data and then updated (0.864) couldn't beat the multi-source ensemble approach of MSAW.
  • Feature Stability: The researchers found that "Age < 5" and "Increased work of breathing" were the most consistent and reliable predictors of RSV across nearly a decade of data.

Limitations and Future Work

  • Model Simplicity: The study relied on Naive Bayes. While effective for online learning, the authors acknowledge that more complex architectures, like Neural Networks or Gradient Boosting, might capture even more intricate patterns.
  • Label Reliability: The "ground truth" for RSV was based on ICD codes at discharge rather than universal lab testing. This means some RSV-positive cases might have been missed if they weren't coded correctly.
  • Single-System Data: The data comes entirely from UPMC. Future work is needed to see if these weights and patterns hold up across different geographic regions or different healthcare systems.
  • Refining the Penalty: The current system penalizes all features equally when a model fails; the authors suggest exploring "feature-specific" weighting in the future.

Summary updated

Abstract

Transfer learning has become a pivotal technique in machine learning and has proven to be effective in various realworld applications. However, utilizing this technique for classification tasks with sequential data often faces challenges, primarily attributed to the scarcity of class labels. To address this challenge, we introduce Multi-Source Adaptive Weighting (MSAW), an online multi-source transfer learning method. MSAW integrates a dynamic weighting mechanism into an ensemble framework, enabling automatic adjustment of weights based on the relevance and contribution of each source (representing historical knowledge) and target model (learning from newly acquired data). We demonstrate the effectiveness of MSAW by applying it to detect Respiratory Syncytial Virus cases within Emergency Department visits, utilizing multiple years of electronic health records from the University of Pittsburgh Medical Center. Our method demonstrates performance improvements over many baselines, including refining pre-trained models with online learning as well as three static weighting approaches, showing MSAW's capacity to integrate historical knowledge with progressively accumulated new data. This study indicates the potential of online transfer learning in healthcare, particularly for developing machine learning models that dynamically adapt to evolving situations where new data is incrementally accumulated.
BibTeX
@inproceedings{sun2024online,
  author = {Yiming Sun and Yuhe Gao and Runxue Bao and Gregory F. Cooper and Jessi Espino and Harry Hochheiser and Marian G. Michaels and John M. Aronis and Chenxi Song and Ye Ye},
  title = {{Online Transfer Learning for RSV Case Detection}},
  booktitle = {ICHI},
  year = {2024},
  url = {https://openreview.net/forum?id=iEFU82ZxiA},
}

Loading related content...

Figure