Multi-instance Domain Adaptation for Vaccine Adverse Event Detection

Junxiang Wang, Liang Zhao

TL;DR

Researchers have developed a new framework called MIDA (Multi-instance Domain Adaptation) that combines slow-but-accurate formal medical reports with fast-but-noisy social media data to detect vaccine side effects. By using a novel "mixed instance kernel" and advanced optimization techniques, MIDA successfully bridges the gap between formal medical jargon and informal "Twitter-speak" to identify adverse events more accurately than existing models.

What Problem Does This Paper Address?

Public health officials monitor vaccine safety through systems like the Vaccine Adverse Event Reporting System (VAERS). While accurate, these formal reports often suffer from significant delays. Social media (like Twitter) offers a "real-time" alternative, but it presents three major hurdles for machine learning:

  1. The Language Gap: Doctors use terms like "parotid gland malaise," while patients tweet "Ouch! my face hurts."
  2. Structural Differences: A formal report is a single, focused text. A Twitter user, however, provides a "bag" of multiple tweets, only one of which might actually describe a side effect.
  3. Class Imbalance: VAERS data consists almost entirely of positive cases (people reporting side effects), whereas the vast majority of social media users are healthy (negative cases).

Existing domain adaptation methods usually assume both datasets have the same structure (e.g., one document per user), making them ineffective for this specific "multi-instance" challenge.

Methods

The authors propose the MIDA framework to maximize the synergy between these two disparate worlds. Key technical components include:

  • The Max Rule: In the multi-instance learning (MIL) setting, a user is labeled "positive" if at least one of their tweets indicates an adverse event. MIDA uses a logistic regression classifier to estimate the probability of each individual tweet being positive.
  • Generalized MMD (Maximum Mean Discrepancy): To bridge the domain gap, the authors use a generalized MMD criterion. This measures the distance between the formal reports and the tweets in a shared latent semantic space, effectively "aligning" the two languages.
  • Mixed Instance Kernels: This is the core innovation. Since formal reports are single instances and Twitter users are "bags" of instances, MIDA uses a specialized kernel to calculate distances between these different structures.
  • Optimization (ADMM + CCP): The resulting mathematical problem is non-convex and difficult to solve. The authors developed an algorithm using the Alternating Direction Method of Multipliers (ADMM) combined with the Convex-Concave Procedure (CCP) to ensure the model converges to an accurate solution efficiently.

Datasets

The study utilized two primary data sources:

  • VAERS (Source Domain): 2,500 formal reports from the 2016 VAERS database, focusing on symptom descriptions.
  • Twitter (Target Domain): A custom dataset of 41,438 tweets from 1,572 users (566 labeled positive for flu shot side effects, 1,006 negative). The tweets were collected between 2011 and 2015 and filtered using the LibShortText tool to identify users who had actually received a vaccination.

Key Insights and Ideas

  • Structure Matters: The "Aha!" moment here is recognizing that domain adaptation isn't just about matching word frequencies; it's about matching structures. By treating Twitter users as "bags" of instances, the model avoids being distracted by a user's irrelevant tweets (e.g., what they had for lunch) and focuses on the one tweet that matches the medical patterns found in VAERS.
  • Semantic Alignment: Even though the words are different, the patterns of symptoms are consistent. MIDA proves that you can use high-quality medical data to "supervise" the learning of noisy social media data without needing massive amounts of manually labeled tweets.
  • Pruning the Noise: The authors found that "initial instance pruning" (using techniques like Kernel Density Estimation) was vital. By removing tweets that were obviously irrelevant before running the main model, they significantly reduced computational overhead.

Key Results

MIDA was tested against several baselines, including standard multi-instance learning (miFV, miVLAD) and transfer learning (JDA, ARTL) models.

  • Superior Accuracy: MIDA achieved an Accuracy (ACC) of 0.7767 and an AUC of 0.8530, outperforming all baselines.
  • Precision Boost: MIDA showed a significant lead in Precision (0.7735), which is crucial for public health monitoring to avoid "crying wolf" with false positives.
  • Scalability: Despite the complex optimization, the ADMM-based approach allowed the model to scale linearly with the number of users and keywords, making it practical for large-scale social media monitoring.

Limitations and Future Work

  • Computational Intensity: While ADMM helps, MIDA is still more computationally expensive than simpler models like ARTL or GTL.
  • Dependency on Keywords: The model currently relies on a predefined set of keywords (234 in this study) to encode the text. Future work could involve using deep learning (like Word2Vec or BERT) to learn these features automatically.
  • Generalization: The authors suggest that MIDA could be extended to handle multiple source domains (e.g., combining VAERS with clinical trial data) to further improve detection.

Summary updated

Abstract

Detection of vaccine adverse events is crucial to the discovery and improvement of problematic vaccines. To achieve it, traditionally formal reporting systems like VAERS support accurate but delayed surveillance, while recently social media have been mined for timely but noisy observations. Utilizing the complementary strengths of these two domains to boost the detection performance looks good but cannot be effectively achieved by existing methods due to significant differences between their data characteristics, including: 1) formal language v.s. informal language, 2) single-message per user v.s. multi-messages per user, and 3) one class v.s. binary class. In this paper, we propose a novel generic framework named Multi-instance Domain Adaptation (MIDA) to maximize the synergy between these two domains in the vaccine adverse event detection task for social media users. Specifically, we propose a generalized Maximum Mean Discrepancy (MMD) criterion to measure the semantic distances between the heterogeneous messages from these two domains in their shared latent semantic space. Then these message-level generalized MMD distances are synthesized by newly proposed mixed instance kernels to user-level distances. We finally minimize the distances between the samples of the partially-matched classes from these two domains. In order to solve the non-convex optimization problem, an efficient Alternating Direction Method of Multipliers (ADMM) based algorithm combined with the Convex-Concave Procedure (CCP) is developed to optimize parameters accurately. Extensive experiments demonstrated that our model outperformed the baselines by a large margin under six metrics. Case studies showed that formal reports and extracted adverse-relevant tweets by MIDA shared a similarity of keyword and description patterns.
BibTeX
@inproceedings{wang2018multi,
  author = {Junxiang Wang and Liang Zhao},
  title = {{Multi-instance Domain Adaptation for Vaccine Adverse Event Detection}},
  booktitle = {The Web Conference},
  year = {2018},
  doi = {10.1145/3178876.3186051},
}

Loading related content...

Figure