TL;DR
This paper explores using LLM-generated synthetic data to augment imbalanced health-related social media datasets for text classification. Despite efforts to create diverse synthetic examples, the generated texts lacked sufficient diversity compared to real-world data, leading to minimal or even negative impacts on model performance. This highlights the challenges of relying solely on LLM generation for data augmentation in nuanced domains.
What Problem Does This Paper Address?
The paper tackles the challenge of imbalanced datasets in health-related social media text classification. Specifically, Tasks 5 and 6 of the SMM4H 2024 challenge involved datasets with a significant scarcity of positive examples. This imbalance is a common problem in real-world applications, where rare events or specific conditions are underrepresented in collected data. Such imbalances can lead to machine learning models that are biased towards the majority class and perform poorly on identifying the minority class, which is often the class of greatest interest (e.g., identifying genuine medical diagnoses or specific age groups for health research).
Methods
The authors employed a two-pronged approach:
- LLM-based Data Augmentation: They utilized OpenAI's GPT-4 model with in-context learning. The prompts included label definitions and a few examples of both positive and negative class texts. The goal was to generate synthetic "positive" examples to balance the datasets.
- Text Classification Model Development: They used the DeBERTa v3 Large model for text classification. For each task, they trained two versions: one using the original imbalanced data and another augmented with the LLM-generated synthetic data.
- LLM-Generated Text Assessment: To understand why the augmentation might not be working, they performed two analyses on the generated texts:
- Clustering of Text Embeddings: Using sentence transformers and t-SNE, they visualized the distribution of original and synthetic text embeddings to assess semantic diversity.
- N-gram Analysis: They compared the number of unique n-grams (unigrams, bigrams, trigrams) in the original and synthetic datasets to quantify vocabulary diversity.
Datasets
The paper focuses on two datasets from the SMM4H 2024 challenge:
- Task 5: Binary classification of tweets related to children's medical disorders, distinguishing between genuine diagnoses and mere mentions. The dataset included 7,398 training, 389 validation, and 1,947 testing tweets.
- Task 6: Extraction of self-reported exact ages from Twitter and Reddit posts. The dataset comprised 8,800 labeled tweets and 100,000 unlabeled Reddit posts for training, with separate validation and testing sets.
Key Insights and Ideas
- LLM-generated data isn't a silver bullet for imbalance: While LLMs can generate text, simply generating more examples doesn't guarantee improved model performance, especially if the generated data lacks the richness of real-world data.
- Diversity is key, and LLMs can struggle with it: The core issue identified was the lack of diversity in the GPT-4 generated texts. The embeddings and n-gram analysis revealed that the synthetic data was more clustered and less varied than the original, human-generated text.
- Health-related social media is nuanced: The specific domain of health-related social media, with its unique language, context, and potential for subtle expressions, might be particularly challenging for LLMs to replicate authentically and diversely.
Key Results
- Task 5: The model trained with LLM-generated data showed a slight improvement in F1-score (0.009), increasing from 0.924 to 0.933.
- Task 6: Conversely, the model trained with LLM-generated data experienced a slight deterioration in F1-score (0.013), decreasing from 0.936 to 0.923.
- N-gram Analysis: The generated texts consistently exhibited a significantly lower number of unique n-grams compared to the original datasets across all n values (1, 2, and 3).
- Clustering Analysis: Visualizations showed that GPT-4 generated text embeddings formed localized clusters with limited spread, particularly for Label 0 sentences, indicating a lack of semantic diversity.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Limited LLM Exploration: Only GPT-4 was used for generation. Future work could explore other LLMs or multiple LLMs to increase diversity.
- Prompt Engineering: The current prompt structure might not be optimal. Future research could investigate more sophisticated prompting techniques.
- Diversity Enhancement Strategies: The paper suggests exploring methods to increase diversity, such as:
- Specifying poster attributes or disease characteristics in prompts.
- Using multiple language models.
- Presenting more original examples in the few-shot prompts.
- Adjusting hyperparameters like temperature to encourage more varied outputs.
- Domain Specificity: The findings suggest that creating diverse and effective data for training language models, especially in specialized domains like health-related social media, requires careful consideration beyond simple generation.
Summary updated