TL;DR
This paper tackles the challenge of imbalanced datasets in identifying "Wellness Dimensions" (WD) from social media text. They propose using prompt-based generative NLP models, specifically ChatGPT, to augment Reddit posts, creating a more balanced dataset for training mental health screening models. Their approach significantly improves classification performance, demonstrating the power of targeted data augmentation.
What Problem Does This Paper Address?
Social media platforms like Reddit have become a rich source of self-narrated text, offering insights into individuals' well-being. However, identifying specific "Wellness Dimensions" (WD) – aspects like physical, intellectual, social, and emotional well-being – from this data is hampered by imbalanced datasets. This means some WD are underrepresented, making it difficult for machine learning models to accurately detect them. This imbalance can hinder the development of effective tools for early mental health screening, which is crucial given the increasing need for accessible mental health support. The authors aim to bridge this gap by creating a more balanced and representative dataset through data augmentation.
Methods
The core of this paper's methodology lies in using generative NLP models for data augmentation. They experimented with several pre-trained models, including different versions of ChatGPT (gpt-3.5-turbo, gpt-3.5-turbo-0301) and other GPT-3 models (text-curie-001, text-davinci-003).
A key innovation is their prompt design. They developed specific prompts to guide the models in generating:
- Similar text: Generating new posts that are semantically and syntactically similar to existing ones, while adhering to a specific Wellness Dimension (e.g., "Physical Aspect").
- Explanations: Generating short explanations for why a given text snippet relates to a particular Wellness Dimension. This "few-shot learning" approach, where examples are provided, helps the model understand the nuances of each WD.
They also employed traditional data augmentation techniques like Easy Data Augmentation (EDA) and Backtranslation for comparison. To evaluate the quality of the augmented data, they used a multi-faceted approach:
- ROUGE scores: To measure overlap between original and augmented text.
- Semantic similarity: Using embeddings from various pre-trained language models and cosine similarity.
- Syntactic similarity: Analyzing syntactic trees and Part-of-Speech (POS) tag sequences.
Finally, they trained a BERT classifier on both the original and augmented datasets to assess the impact of augmentation on classification performance.
Datasets
The primary dataset used is a newly constructed and annotated Reddit corpus. This corpus contains 3,092 instances and 72,813 words, labeled for four Wellness Dimensions: Physical Aspect (PA), Intellectual and Vocational Aspect (IVA), Social Aspect (SA), and Spiritual and Emotional Aspect (SEA). The annotation process involved experts and postgraduate students, with inter-annotator agreement validated using Fleiss' Kappa. The authors publicly release this dataset.
Key Insights and Ideas
- Generative models can effectively augment imbalanced text data: The study demonstrates that models like ChatGPT can generate diverse and contextually relevant text for underrepresented classes, significantly improving dataset balance.
- Prompt engineering is crucial for targeted augmentation: The specific prompts designed to generate both similar text and explanations were key to guiding the models and ensuring the augmented data was useful for the classification task.
- Lower textual similarity doesn't always mean worse performance: Surprisingly, the augmented data that showed the least ROUGE, semantic, and syntactic similarity to the original text (generated by the top ChatGPT models) led to the highest classification performance. This suggests that diversity, rather than strict similarity, might be more beneficial for training robust classifiers.
- Data augmentation significantly boosts classifier performance: The improvements in F-score and Matthew's Correlation Coefficient (MCC) highlight the practical impact of this augmentation strategy on real-world applications like mental health screening.
Key Results
- The gpt-3.5-turbo model, when used for data augmentation, achieved the best classification performance.
- Augmenting the dataset with the best generative model resulted in an improvement of up to 13.11% in F-score and 15.95% in Matthew's Correlation Coefficient (MCC) compared to using the original imbalanced dataset.
- The augmented dataset, with 4376 records (1094 per class), achieved a more balanced distribution than the original imbalanced dataset.
- While traditional augmentation methods (EDA, BT) and other GPT models showed some improvement, the top ChatGPT models provided the most significant gains in classification accuracy.
Limitations and Future Work
- Subjectivity of Annotation: The authors acknowledge that annotation is subjective and may introduce biases. They also note a slight confusion between PA and SEA during annotation.
- Privacy Concerns: While user IDs are anonymized and posts obfuscated, the use of social media data for mental health analysis raises ethical considerations. The authors emphasize the need for human intervention and moderator oversight.
- Future Work: The authors plan to explore different parameter settings and prompts for data generation, develop improved classifiers for well-balanced datasets, and evaluate the classification performance of the generated short explanations. They also aim to investigate the impact of augmentation on short explanation text.
Summary updated