A Comparative Study on Application of Class-Imbalance Learning for Severity Prediction of Adverse Events Following Immunization

Ning Chen, Zhengke Sun, Tong JiaarXiv 2022 · OpenReview

Abstract

In collaboration with the Liaoning CDC, China, we propose a prediction system to predict the subsequent hospitalization of children with adverse reactions based on data on adverse events following immunization (AEFI). We extracted multiple features from the data and selected "hospitalization or not" as the target for classification. Since the data are imbalanced, we used various class-imbalance learning methods for training and improved the RUSBoost algorithm. Experimental results show that the improved RUSBoost has the highest Area Under the ROC Curve (AUC) on the target among these algorithms. Additionally, we compared these class-imbalance learning methods with common machine learning algorithms. We combined the improved RUSBoost with dynamic web resource development techniques to build an evaluation system with information entry and vaccination response prediction capabilities for relevant medical practitioners.

Introduction: Addressing the Challenge of AEFI Severity

Vaccination is a cornerstone of public health, but Adverse Events Following Immunization (AEFI) can occur. While most reactions are mild, a tiny fraction are severe enough to require hospitalization. For clinicians, the challenge is identifying these high-risk cases early to maintain public trust and ensure prompt care.

From a data science perspective, this is a classic class-imbalance problem. In the study's dataset, severe reactions are extremely rare compared to mild ones. Standard machine learning algorithms are often "lazy" in these scenarios; they may achieve high raw accuracy simply by predicting that no one will be hospitalized, failing entirely to catch the rare cases that actually matter. This research seeks to identify a model that prioritizes finding the minority class (hospitalization) without generating excessive false alarms.

Data Processing and Feature Engineering

The study utilized a real-world dataset of childhood AEFI reports from 2009 to 2014 in Shenyang, China, provided by the Liaoning CDC. After cleaning, 1,315 records were selected for analysis. The team performed extensive feature engineering to transform raw CDC reports into predictive features.

Feature engineering workflow for AEFI data

As illustrated in Figure 1, raw features such as date of birth and vaccination date were processed to calculate "Age at vaccination," while discovery and visit dates were used to determine the "Hospital visits interval." Other features included vaccine type, dose number, injection site, and specific reaction symptoms like fever or local induration.

Algorithmic Innovation: The Improved RUSBoost Model

The researchers compared several standard machine learning models (e.g., Random Forest, Logistic Regression, XGBoost) against specialized imbalance-learning algorithms including EasyEnsemble, Balanced Random Forest (BRF), and RUSBoost.

The core innovation of this work is the Improved RUSBoost. The authors hypothesized that the "support vectors" identified by a Support Vector Classifier (SVC)—the data points closest to the decision boundary—are the most critical for accurate classification. They used an SVC to pre-classify the data, identified these support vectors, and increased their weights before feeding them into the RUSBoost algorithm. This forces the model to pay extra attention to the most ambiguous and difficult cases during training. The researchers found an 83.9% overlap between the support vectors of an SVC and the samples that RUSBoost found difficult to classify, validating this hybrid approach.

Results and Performance Comparison

The Improved RUSBoost model outperformed both standard ML models and unimproved imbalance-learning algorithms. While standard models like Logistic Regression showed high raw accuracy (96.9%), their Area Under the Curve (AUC) was a dismal 0.557—barely better than a coin flip for identifying severe cases.

Confusion matrices for different classification models

As shown in the confusion matrices in Figure 6, the models demonstrate varying capabilities in identifying the minority "YES" class. The Improved RUSBoost achieved an AUC of 0.83, matching the Balanced Random Forest but significantly outperforming standard SVC (0.585) and AdaBoost (0.559). Furthermore, the hybrid model achieved a high F1-score (0.90) and G-mean (0.77), indicating a superior balance between catching true hospitalizations and avoiding false positives.

Conclusion and Clinical Implementation

Beyond algorithmic development, the authors implemented their findings into a functional web-based evaluation system. This allows medical practitioners to input patient data and receive a severity prediction in real-time, facilitating prompt clinical intervention. Future work will focus on expanding the feature set with additional clinical indicators and optimizing the software architecture for robust daily clinical use.

Summary updated

Abstract

In collaboration with the Liaoning CDC, China, we propose a prediction system to predict the subsequent hospitalization of children with adverse reactions based on data on adverse events following immunization. We extracted multiple features from the data, and selected "hospitalization or not" as the target for classification. Since the data are imbalanced, we used various class-imbalance learning methods for training and improved the RUSBoost algorithm. Experimental results show that the improved RUSBoost has the highest Area Under the ROC Curve on the target among these algorithms. Additionally, we compared these class-imbalance learning methods with some common machine learning algorithms. We combined the improved RUSBoost with dynamic web resource development techniques to build an evaluation system with information entry and vaccination response prediction capabilities for relevant medical practitioners.
BibTeX
@misc{chen2022comparative,
  author = {Ning Chen and Zhengke Sun and Tong Jia},
  title = {{A Comparative Study on Application of Class-Imbalance Learning for Severity Prediction of Adverse Events Following Immunization}},
  year = {2022},
  eprint = {2206.09752},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.09752},
}
Figure