TL;DR
This paper tackles the challenge of identifying rare vaccine side effects from patient self-reports, which are often unstructured and contain varied language. They propose an autoregressive model for symptom detection and introduce a novel data augmentation technique using symptom definitions to improve performance on these rare "long-tail" symptoms.
What Problem Does This Paper Address?
Detecting and understanding vaccine side effects is crucial, especially for newly developed vaccines. While systems like VAERS collect patient reports, these reports are highly unstructured. Patients use diverse language, leading to challenges in accurately identifying and standardizing reported symptoms. Traditional classification methods struggle with this due to the sheer number of potential symptoms and the fact that many are reported very infrequently (the "long-tail" problem). This makes it difficult to train robust models that can reliably detect these rare but potentially important side effects.
Methods
The authors frame the problem as an "entity retrieval" task, where the goal is to extract symptom entities from patient descriptions. They adopt an autoregressive entity retrieval model, similar to the GENRE model, which generates a sequence of symptom names rather than classifying each symptom independently. This approach is better suited for handling the variable number of symptoms per report.
To combat the data sparsity issue for rare symptoms, they introduce a data augmentation technique. This involves generating synthetic training data by using symptom definitions. These definitions are obtained in two ways:
- Pre-trained Language Models (GPT-3): They prompt GPT-3 to generate definitions for rare symptoms.
- UMLS Medical Dictionary: They leverage the Unified Medical Language System, a comprehensive biomedical knowledge graph, to extract definitions.
These definitions are then treated as synthetic patient reports, with the symptom name serving as the label. They also create examples with multiple symptoms by concatenating definitions.
Datasets
The primary dataset used is the Vaccine Adverse Event Reporting System (VAERS), specifically data from 2019-2021. The dataset is split into training, validation, and test sets. They also create specialized test sets:
- Full: The complete test set.
- CUI-mapped: A subset where symptoms are mapped to Concept Unique Identifiers (CUIs) from UMLS, allowing comparison with methods that detect UMLS CUIs.
- Long-tail: A test set specifically containing only rare symptoms.
Key Insights and Ideas
- Generative models outperform classification for this task: The autoregressive entity retrieval approach significantly outperforms traditional multi-label classification models, highlighting the suitability of generative methods for unstructured text with variable outputs.
- Data augmentation is crucial for rare symptoms: The proposed data augmentation strategy, using symptom definitions, demonstrably improves performance on long-tail entities without negatively impacting performance on common symptoms. This is a key finding for addressing the long-tail problem in real-world data.
- Symptom definitions as synthetic data: The idea of using symptom definitions as a source of synthetic training data is an innovative way to leverage existing knowledge resources to overcome data scarcity.
Key Results
The generative model (GENRE) achieved strong performance, with F1 scores of over 79.8% (Macro) and 76.6% (Micro) on the full test set, significantly outperforming baselines like string matching and MetaMap.
Crucially, on the long-tail test set, augmenting the training data with both UMLS and GPT-3 definitions led to substantial improvements:
- Macro F1: Increased by 2.62%.
- Micro F1: Increased by 3.31%.
This indicates that the augmentation strategy is effective in improving the model's ability to detect rare symptoms.
Limitations and Future Work
The authors acknowledge that their data augmentation approach has limitations. A key one is that each symptom definition typically corresponds to a single symptom, whereas real patients often experience multiple symptoms. While they attempted to mitigate this by concatenating definitions, it's a simplification of real-world complexity.
Future work could explore more sophisticated methods for generating realistic multi-symptom synthetic reports. Additionally, exploring other approaches for learning with limited data is suggested as a promising avenue.
Summary updated