Online Transfer Learning for RSV Case Detection

Yiming Sun, Yuhe Gao, Runxue Bao, Gregory F. Cooper, Jessi Espino, Harry Hochheiser, Marian G. Michaels, John M. Aronis, Ye Ye

Abstract

Transfer learning has become a pivotal technique in machine learning and has proven to be effective in various real-world applications. However, utilizing this technique for classification tasks with sequential data often faces challenges, primarily attributed to the scarcity of class labels. To address this challenge, we introduce Multi-Source Adaptive Weighting (MSAW), an online multi-source transfer learning method. MSAW integrates a dynamic weighting mechanism into an ensemble framework, enabling automatic adjustment of weights based on the relevance and contribution of each source (representing historical knowledge) and target model (learning from newly acquired data). We demonstrate the effectiveness of MSAW by applying it to detect Respiratory Syncytial Virus cases within Emergency Department visits, utilizing multiple years of electronic health records from the University of Pittsburgh Medical Center. Our method demonstrates performance improvements over many baselines, including refining pre-trained models with online learning as well as three static weighting approaches, showing MSAW's capacity to integrate historical knowledge with progressively accumulated new data. This study indicates the potential of online transfer learning in healthcare, particularly for developing machine learning models that dynamically adapt to evolving situations where new data is incrementally accumulated.

Introduction

Detecting infectious diseases like Respiratory Syncytial Virus (RSV) using Electronic Health Records (EHRs) is a "moving target" problem. Traditional machine learning models are typically static, trained on fixed datasets and deployed without the ability to adapt to "concept drift"—changes in disease patterns, hospital protocols, or environmental factors over time. Furthermore, at the start of a new RSV season, clinicians face a "cold start" problem: there is insufficient local data to train a reliable new model, yet relying solely on historical data may ignore current trends. The challenge lies in effectively transferring knowledge from multiple past seasons to a current, evolving season without being misled by outdated information.

The MSAW Framework

To address these challenges, we developed Multi-Source Adaptive Weighting (MSAW), an online multi-source transfer learning method. MSAW utilizes an ensemble framework that integrates knowledge from multiple historical "source" models and a "target" model that learns incrementally from new data.

The MSAW ensemble architecture, combining multiple historical source models with a dynamic target model.

The core of MSAW is its dynamic weighting mechanism. Unlike static ensembles, MSAW adjusts the contribution of each model in real-time based on two primary factors:

  1. Accuracy-Based Penalty: If a specific source model misclassifies a new patient, its weight is penalized and reduced. This ensures the ensemble prioritizes the most relevant historical knowledge for the current context.
  2. Volume-Adaptive Growth: As more data is accumulated in the current (target) season, the weight of the target model is automatically increased. This allows the system to trust historical data early on but gradually shift its trust to the current season's specific patterns as they emerge.

Data and Experimental Setup

The study utilized a massive dataset from the University of Pittsburgh Medical Center (UPMC), encompassing 2,263,360 patient encounters across five hospitals over nine years (2011–2020). The data was segmented into annual seasons (June to May) to align with the natural epidemiological cycle of RSV. Features included demographics, laboratory results, and clinical findings extracted from free-text notes using Natural Language Processing (NLP) via MetaMap. This approach captured 7,614 distinct features, providing a rich context for case detection that structured data alone might miss.

Results and Feature Analysis

MSAW was evaluated on the 2019–2020 RSV season and achieved an AUROC of 0.870, the highest among all tested methods. It significantly outperformed static weighting approaches, such as equal weighting (0.728) and time-based weighting (0.761), proving that dynamic adjustment is superior to human-defined rules for weighting history.

Analysis of feature importance across seasons, highlighting stable predictors like age and respiratory distress.

A key insight from the study was the identification of stable clinical predictors. As shown in the feature analysis, "Age < 5" and "Increased work of breathing" remained highly consistent and reliable indicators of RSV across nearly a decade of data. The use of NLP to extract clinical concepts like "increased work of breathing" proved vital, as these nuances are often missed by simple ICD codes.

Conclusion and Future Work

The MSAW method demonstrates the potential of online transfer learning to create adaptive diagnostic tools in healthcare. By balancing historical wisdom with emerging data, MSAW effectively mitigates the risks of concept drift and the cold start problem. Future research will explore more complex architectures, such as Neural Networks, and investigate the generalizability of these findings across different geographic regions and healthcare systems.

Summary updated

Abstract

Transfer learning has become a pivotal technique in machine learning and has proven to be effective in various real-world applications. However, utilizing this technique for classification tasks with sequential data often faces challenges, primarily attributed to the scarcity of class labels. To address this challenge, we introduce Multi-Source Adaptive Weighting (MSAW), an online multi-source transfer learning method. MSAW integrates a dynamic weighting mechanism into an ensemble framework, enabling automatic adjustment of weights based on the relevance and contribution of each source (representing historical knowledge) and target model (learning from newly acquired data). We demonstrate the effectiveness of MSAW by applying it to detect Respiratory Syncytial Virus cases within Emergency Department visits, utilizing multiple years of electronic health records from the University of Pittsburgh Medical Center. Our method demonstrates performance improvements over many baselines, including refining pre-trained models with online learning as well as three static weighting approaches, showing MSAW's capacity to integrate historical knowledge with progressively accumulated new data. This study indicates the potential of online transfer learning in healthcare, particularly for developing machine learning models that dynamically adapt to evolving situations where new data is incrementally accumulated.
BibTeX
@inproceedings{sun2024online,
  author = {Yiming Sun and Yuhe Gao and Runxue Bao and Gregory F. Cooper and Jessi Espino and Harry Hochheiser and Marian G. Michaels and John M. Aronis and Ye Ye},
  title = {{Online Transfer Learning for RSV Case Detection}},
  booktitle = {ICHI},
  year = {2024},
  url = {https://openreview.net/forum?id=iEFU82ZxiA},
}
Figure